Training-time 3D Gaussian enhancement for robotics

GaussianWAM

Distilling Geometry and Semantics from 3D Gaussian Fields into World-Action Models

Zijian Zhang^*, Yuqing Jiang^*, Weitao Zhou^†, Minglei Li, Jinhao Zhang, Yao Mu, Xiaofan Li, Hao Zhao, Haibao Yu^†

Tuojing Intelligence · University of Chinese Academy of Sciences · Institute of Automation, CAS
Tsinghua University · Simple AI · HIT Shenzhen · Shanghai Jiao Tong University · Zhejiang University · HKU

GitHub Cite the paper

^* Equal contribution ^† Corresponding author

A shared spatial carrier

Geometry, semantics, and coverage are organized in one 3D Gaussian field, then distilled into the original WAM representation.

The idea

Teach WAMs where and what without changing how they run.

GaussianWAM uses a 3D Gaussian field to spatially align geometric and visual-semantic supervision during training. The Gaussian teacher, foundation models, and auxiliary heads are removed before deployment, leaving the original inference path unchanged.

01 / Method

One field, three signals, zero deployment overhead.

Multi-view observations are lifted into a shared 3D coordinate system. Gaussian rendering turns that field into dense targets aligned with the WAM visual tokens.

Build the teacher

Frozen geometry and vision foundation models provide depth, camera parameters, and dense semantic features from synchronized views.

Bind in 3D

Geometry and semantics are attached to the same Gaussian primitives, creating a spatially coherent teacher representation.

Distill into WAM

Rendered semantic, depth, and alpha targets supervise current-observation representations during policy training.

Comparison of WAM enhancement paradigms — GaussianWAM organizes complementary teacher signals in 3D while preserving the original policy architecture.

Spatially grounded representations

Semantics become useful when they know where they live.

Rather than distilling heterogeneous teachers independently in 2D, GaussianWAM associates semantic features with the geometry that supports them. The resulting supervision encourages action-relevant visual tokens to retain object-level meaning and cross-view structure.

Semanticwhat the region represents

Depthwhere the region is in 3D

Coveragewhich locations are supported

Semantic representation visualization — Semantic representation alignment

Depth visualization from a robotic scene — Geometry-aware supervision

02 / Results

Consistent gains across WAM backbones.

GaussianWAM improves both a dual-expert FastWAM-style model and a unified Cosmos Policy transformer, with the strongest gain under the LIBERO-Plus distribution shift.

71.29%FastWAM on LIBERO-Plus+19.24 points

77.30%Cosmos Policy on LIBERO-Plus+5.78 points

97.60%FastWAM on LIBERO+1.00 point

40.00%Real-world FastWAMfrom 30.00%

Real-world robotic manipulation experiments — Real-world manipulation experiments show positive transfer beyond the training benchmarks.

Training only

Nothing extra at deployment.

During training
Gaussian field · VGGT · CLIP · auxiliary heads

During inference
Original WAM forward path · no extra modules · no extra compute

03 / Resources

Explore GaussianWAM.

Code↗ Project page source↗

BibTeX

@article{gaussianwam,
  title   = {GaussianWAM: Distilling Geometry and Semantics from 3D Gaussian Fields into World-Action Models},
  author  = {Zhang, Zijian and Jiang, Yuqing and Zhou, Weitao and Li, Minglei and Zhang, Jinhao and Mu, Yao and Li, Xiaofan and Zhao, Hao and Yu, Haibao},
  year    = {2026},
  url     = {https://github.com/TuojingAI/GaussianWAM}
}