Future Representation Distillation for Imagination-free World Action Models
Abstract
DreamFree repurposes a pretrained video diffusion model as a predictive visual encoder through representation distillation, enabling real-time robot action prediction without generating future videos and improving control and learning from video demonstrations.
World Action Models (WAMs) use the predictive knowledge of pretrained video diffusion models for robot control through future video prediction. However, iterative video denoising is costly, and inaccurate visual predictions can compromise action prediction. Our key idea is to make this predictive knowledge available through visual representations computed from the current observation, enabling action prediction without generating future videos. We introduce DreamFree, a WAM that repurposes a pretrained video diffusion model as a predictive visual encoder through representation distillation. The visual encoder learns to predict a frozen video diffusion model's representation using only the current observation. An action module is then trained to predict actions by extracting action-relevant information from the visual encoder. Across real-world and simulated tasks, DreamFree achieves higher success rates than state-of-the-art VLAs and WAMs, while performing real-time closed-loop control at 11Hz with a 14B video diffusion backbone. Experiments show that DreamFree can translate stronger video priors and additional video data into better robot control and enable learning new robot tasks from video demonstrations, including videos from a different embodiment, without action annotation for the new tasks.
Community
Can we scale up imagination-free WAMs (e.g., Fast-WAM)?
Excited to share DreamFree — an imagination-free WAM that improves with stronger video models and more video data, running at 11 Hz with a 14B video backbone.
Video models learn how the world changes, providing useful knowledge for robot control. But generating future videos is costly, and prediction errors can affect actions.
Imagination-free WAMs, e.g., Fast-WAM, enable robot control without generating future videos. But for Fast-WAM, stronger video priors don't automatically mean better control. In our experiments, scaling from 5B to 14B did not improve success.
How can we make imagination-free WAMs benefit from stronger video models?
Our key idea: turn the pretrained video model into a predictive visual encoder
A frozen video teacher sees the future. The encoder learns to predict the teacher’s representation from the current view. This teaches the encoder to anticipate scene changes without generating future videos. Training the encoder needs only videos, without using action labels.
Next, we train the action DiT using robot demonstrations with action labels. From the encoder’s visual representation, a readout learns to extract action-relevant information. An action decoder uses this information to predict robot actions.
What we find:
Real-world performance: Trained on DROID alone, DreamFree achieves 64% success across 10 real-world tasks, outperforming DreamZero at 56% and π0.5 at 16%. We evaluate the policies without task-specific fine-tuning.
Fast inference: A 14B video backbone runs at 11 Hz on a single H200, with 17.8× faster inference than DreamZero.
Memory efficiency: With a 14B video backbone, DreamFree fits on a single A6000 GPU with 48 GB of memory, without future-video generation or classifier-free guidance at inference.
Better scaling: Scaling the video backbone from 5B to 14B significantly improves performance, enabling real-world manipulation on DROID and bimanual YAM. More video data further improves control without additional action labels.
Video-only task transfer: DreamFree learns new task sequences from videos, including demonstrations from a different robot embodiment, without action labels for the new tasks.
Get this paper in your agent:
hf papers read 2611.04527 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper