Efficient 3D Gaussian world modeling for robotics

GaussianDream++

A compact, policy-native world model for physically grounded Vision-Language-Action control.

Yuqing Jiang^*, Zijian Zhang^*, Weitao Zhou, Junjie He, Lei Yang, Haifang Qing, Si Liu, Ding Zhao, Ping Luo, Haibao Yu^†

Tuojing Intelligence · University of Chinese Academy of Sciences · Institute of Automation, CAS
Tsinghua University · HKUST (Guangzhou) · NTU · Beihang University · Carnegie Mellon University · HKU

GitHub Cite the paper

^* Equal contribution ^† Corresponding author

Policy-native world representation

Overview of the GaussianDream++ framework

World State Tokens and World Prediction Tokens bring current and future Gaussian supervision directly into the VLA backbone.

The upgrade

Structured 3D supervision, carried by just 20 world tokens.

GaussianDream++ internalizes world modeling inside the native VLA pathway. It preserves the benefits of current-world reconstruction and future prediction while removing the dedicated runtime geometry pathway.

01 / Method

Current state and future evolution share one compact representation.

Two role-specific token groups are contextualized with visual and language tokens, then directly condition the Action Expert. Dense Gaussian decoding exists only during training.

World State Tokens

16 policy-native tokens encode a spatially grounded summary of the current physical scene.

World Prediction Tokens

4 compact tokens represent short-horizon scene evolution and interaction-relevant motion.

Gaussian supervision

A lightweight World Representation Head decodes Current World and Future Prediction over shared Gaussian primitives.

Comparison of GaussianDream++ representations and baselines — GaussianDream++ separates current-state and future-prediction roles while keeping the deployed representation compact.

Asymmetric by design

Train with dense worlds. Deploy with native tokens.

During training, the World Representation Head converts token representations into geometry, appearance, visibility, and motion targets. At deployment, the dense branch disappears and the same tokens continue to condition action generation through native attention.

TRAINGaussian decoder · renderer · auxiliary losses

DEPLOY20 world tokens · original VLA action pathway

GaussianDream++ qualitative visualization — Qualitative world representation

20policy-native
world tokens

02 / Results

More robust manipulation with a smaller deployed world representation.

Under the matched GaussianDream-family protocol, GaussianDream++ improves both standard manipulation and zero-shot robustness, especially under camera and layout shifts.

98.6%LIBERO average success+0.2 over GaussianDream

87.8%LIBERO-Plus overall+0.8 over GaussianDream

80.1%LIBERO-Plus Camera+2.8 over GaussianDream

90.0%LIBERO-Plus Layout+1.6 over GaussianDream

Real-world robotic manipulation experiments — Real-robot evaluation improves average success from 29.2% to 52.5% over the reproduced π₀.₅ baseline.

No runtime world model

Dense Gaussian worlds stay on the training side.

Removed after training
World Representation Head · Gaussian renderer · auxiliary branches · runtime VGGT/TGE pathway

Kept at deployment
20 world tokens inside the PaliGemma prefix, directly conditioning the Action Expert

03 / Resources

Explore the GaussianDream family.

Code↗ Project page source↗

BibTeX

@article{gaussiandreamplusplus,
  title   = {GaussianDream++: Efficient 3D Gaussian World Modeling for Robotic Manipulation},
  author  = {Jiang, Yuqing and Zhang, Zijian and Zhou, Weitao and He, Junjie and Yang, Lei and Qing, Haifang and Liu, Si and Zhao, Ding and Luo, Ping and Yu, Haibao},
  year    = {2026},
  url     = {https://github.com/TuojingAI/GaussianDream}
}