Doppel-8: one video world model across eight embodiments
Model weights for Render the Robot, Generate the World: One Video World Model Across Embodiments (Yash Jangir, Karan Mirakhor, Renos Zabounidis, Muhammad Zubair Irshad, Sergey Zakharov, Yonatan Bisk, Homanga Bharadhwaj; JHU / CMU / TRI). Project page: https://doppelwm.github.io. Code: https://github.com/DoppelWM/Doppel.
Doppel is an action-conditioned video world model. Given the first camera frame, a rendering of the embodiment along a proposed trajectory (seen from the same camera), and the numerical trajectory itself, it predicts the video of what happens: how the arm, hand or body moves and how the objects it touches respond.
This repository holds Doppel-8, the model trained jointly on eight datasets / eight embodiments (robot arms, a bimanual robot, a humanoid with dexterous hands, human hands and human bodies) with one shared action encoder and no embodiment label. The renderer tells the model which body it is looking at.
| Backbone | Wan 2.1 image-to-video 14B (480P, Diffusers format), all 40 DiT blocks fine-tuned |
| Conditioning | render-action adapter ("REDAR", 30.6M params): render encoder + action encoder + render-to-action cross-attention, injected per video token through the flow-timestep modulation |
| Checkpoint | epoch 2489 of 2490 (189,240 optimizer steps), EMA weights (decay 0.995) |
| Final validation loss (EMA) | 0.0632 |
| Training data | 803,602 training clips from 68,748 episodes, about 670 hours of video |
| Output | 17 RGB frames at 240x432 |
What is in this repository
This repo is an overlay on the public base model: it holds only the tensors that differ from
Wan-AI/Wan2.1-I2V-14B-480P-Diffusers (revision b184e23a8a16b20f108f727c902e769e873ffc73), plus the new adapter.
Because every DiT block was fine-tuned, the overlay covers 16.34B of the transformer's 16.43B parameters (99.5%);
the rest (patch embedding, time/text/image embedders, output scale-shift table) and the VAE, CLIP image encoder,
UMT5 text encoder, tokenizer and scheduler config come from the base repo.
README.md this card
config.json model, input, inference and training settings (no internal paths)
action_padding_layout.json per-embodiment action layout inside the shared 159-slot vector
action_stats/<embodiment>.json per-embodiment z-score statistics used in training (8 files)
transformer_overlay/
model.safetensors.index.json weight map
model-0000{1..7}-of-00007.safetensors
SHA256SUMS sha256 of every file above
| Tensors | Count | Params | Stored dtype |
|---|---|---|---|
blocks.{0..39}.* attention, FFN, norm2 |
1,240 | 16,152.3M | bfloat16 |
blocks.{0..39}.scale_shift_table |
40 | 1.2M | float32 |
condition_embedder.time_proj.* |
2 | 157.3M | float32 |
proj_out.* |
2 | 0.3M | bfloat16 |
action_adapter_v4.* (render-action adapter) |
72 | 30.6M | float32 |
| Total | 1,356 | 16,341.7M | 33.06 GB |
Precision. Training kept fp32 master weights and an fp32 EMA; the model runs in bf16 with a small set of
modules kept in fp32 (time_proj, scale_shift_table, the adapter, plus base time_embedder and norm_out). Each
tensor here is stored in the dtype it has at inference, so loading this overlay gives a model that is bit-identical
to the one used for every number in the paper (see Verification). The fp32 EMA master weights (65.4 GB) are not
included.
The adapter keys use the training-code name action_adapter_v4.*; the doppel code loads them into
render_action_adapter (same sub-module names) automatically.
The auxiliary point-track head (tracks_head.*) of the architecture was disabled in this run (loss weight 0); it is
not shipped and is never used for video generation.
Training data
One model, eight datasets, sampled in proportion to their training clips (about 60 passes over each).
| Dataset | Embodiment (what the renderer draws) | Action | Train episodes | Train clips | Test shift |
|---|---|---|---|---|---|
| DROID | Franka FR3 + 2F-85 | 8-D: 7 joints + gripper | 16,726 | 83,511 | unseen lab |
| BridgeData V2 | WidowX 250s | 8-D: 6 joints + gripper | 12,387 | 61,935 | unseen kitchen |
| RoboMIND | UR5e + Robotiq 2F-85 | 8-D: 6 joints + gripper | 14,801 | 74,005 | unseen objects |
| ABC-130K | bimanual YAM | 14-D: 2 x (6 joints + gripper) | 9,115 | 274,523 | unseen tasks |
| T-Rex | Vega-1 humanoid + Sharpa hands | 58-D: 2 x (7 arm + 22 hand joints) | 4,784 | 58,120 | unseen backgrounds |
| Ego-Exo4D | human body (SMPL-H) | 159-D body + hands axis-angle | 2,041 | 213,292 | unseen tasks |
| GigaHands | two human hands (MANO) | 108-D: 2 x (wrist 6-D rot + xyz + 15 joints) | 8,491 | 31,218 | unseen activities |
| OakInk2 | human hands (SMPL-X) | 159-D (42 constant slots zeroed) | 403 | 6,998 | unseen tasks |
| All eight | 68,748 | 803,602 |
Renderings: robots are drawn with per-robot Dr. Robot Gaussian-splat models along the logged joint trajectory; humans and hands from their fitted SMPL-H / SMPL-X / MANO models, all from the calibrated camera of the clip.
Training data licenses
These weights are released under CC BY-NC 4.0 (non-commercial). They were trained on data and body models with non-commercial or share-alike terms (GigaHands, OakInk2, and the MPI SMPL-H / SMPL-X / MANO models used for the human renderings), so the weights carry a non-commercial licence. The base model Wan 2.1 is Apache-2.0. Using the human embodiments also requires your own SMPL-H / SMPL-X / MANO licence from MPI; no body-model files are included here.
| Dataset | License / terms |
|---|---|
| DROID | CC BY 4.0 |
| BridgeData V2 | CC BY 4.0 |
| RoboMIND | Apache-2.0 (gated on Hugging Face) |
| ABC-130K | Apache-2.0 (gated on Hugging Face) |
| T-Rex | MIT |
| Ego-Exo4D | Ego-Exo4D license agreement (video); Ego-Exo4D Human Meshes CC BY 4.0 |
| GigaHands | CC BY-NC 4.0 |
| OakInk2 | CC BY-SA 4.0 |
No training videos or frames are included in this repository.
Inputs and outputs
- First frame: one RGB image, resized (not aspect-preserving) to 240x432. Used as the I2V conditioning frame and CLIP-encoded.
- Rendering: 17 frames of the embodiment drawn along the trajectory from the same camera, black background, 240x432.
- Actions:
(17, 159)float32. For an embodiment with native width D: take its per-frame state, unwrap the root axis-angle over the clip (SMPL-X / SMPL-H only), z-score withaction_stats/<embodiment>.json((a - mean) / std), set the masked slots to 0, then zero-extend to 159 slots. The exact layout of each embodiment is inaction_padding_layout.json. No embodiment id is passed. - Prompt: empty string. Text prompts were ignored in training.
- Output: 17 RGB frames, 240x432. Frame 0 reproduces the input frame; frames 1..16 are the prediction. The 17 frames are spread evenly over the clip window, so the time step depends on the clip length and the dataset's native rate (5 to 30 Hz).
Inference settings used for every reported number: 50 flow-matching Euler steps (base scheduler config),
guidance scale 1.0 (no CFG), max_sequence_length=512, bf16 autocast, VAE tiling and slicing on. One 80 GB A100
runs the full pipeline without offloading, about 30 s per clip.
Usage
With the code from github.com/DoppelWM/Doppel:
hf download Wan-AI/Wan2.1-I2V-14B-480P-Diffusers --revision b184e23a8a16b20f108f727c902e769e873ffc73 \
--local-dir checkpoints/wan2.1-i2v-14b-480p
hf download sourlreapwr/doppel-emb8 --local-dir checkpoints/doppel-emb8
from doppel.inference import DoppelWorldModel
wm = DoppelWorldModel.from_pretrained(
checkpoint="checkpoints/doppel-emb8",
config="checkpoints/doppel-emb8/config.json",
model_path="checkpoints/wan2.1-i2v-14b-480p")
video = wm.predict(image="frame0.png", render="render.mp4", actions=actions, robot_type="panda", seed=0)
# uint8 (17, 240, 432, 3); frame 0 is the input frame
actions are the raw per-frame states, shape (17, D); see the repository README for the layout of each embodiment.
Results
Numbers are averages over frames 1-16 on 200 clips per split, scored at this checkpoint.
Robot datasets, validation (unseen trajectories) / test (held-out shift):
| Dataset | Val PSNR | Val SSIM | Val LPIPS | Test PSNR | Test SSIM | Test LPIPS |
|---|---|---|---|---|---|---|
| DROID | 24.82 | 0.870 | 0.062 | 21.46 | 0.806 | 0.108 |
| BridgeData V2 | 26.54 | 0.873 | 0.049 | 22.24 | 0.784 | 0.099 |
| RoboMIND | 31.30 | 0.930 | 0.036 | 30.66 | 0.921 | 0.038 |
On DROID and RoboMIND this one model matches or beats the per-dataset render+action model and the three-robot model on every image metric in both splits; on Bridge it is within 0.08 dB of the best model on validation and second on the test split, behind the three-robot model (22.47 dB). Against an action-only model trained on the same dataset it is +1.9 / +0.3 / +0.8 dB on validation (DROID / Bridge / RoboMIND) and +1.4 / +1.7 / +0.6 dB on test.
The five other datasets (same weights):
| Dataset | Val PSNR | Val SSIM | Val LPIPS | Test PSNR | Test SSIM | Test LPIPS |
|---|---|---|---|---|---|---|
| ABC-130K | 24.65 | 0.850 | 0.096 | 25.22 | 0.861 | 0.089 |
| T-Rex | 28.33 | 0.871 | 0.050 | 28.35 | 0.867 | 0.051 |
| Ego-Exo4D | 27.94 | 0.895 | 0.045 | 27.61 | 0.892 | 0.044 |
| GigaHands | 25.62 | 0.885 | 0.053 | 25.78 | 0.897 | 0.049 |
| OakInk2 | 29.01 | 0.918 | 0.036 | 29.09 | 0.921 | 0.033 |
Intended use
- Research on action-conditioned video prediction, world models and policy evaluation for robot manipulation and human motion, in settings like the eight training datasets.
- Predicting short (17-frame) futures from a real first frame and a trajectory you can render: a robot with a calibrated camera and a renderable model, or a human body / hand pose sequence.
Limitations
- Short horizon, low resolution: 17 frames at 240x432. Longer rollouts need chaining, which this model was not trained for.
- Needs a calibrated rendering: the rendering must be aligned with the camera of the first frame. Misaligned renders (wrong extrinsics, wrong robot model) degrade predictions; the model has no other way to know the body.
- Embodiments outside training are untested. The action layout is fixed per embodiment
(
action_padding_layout.json); a new robot needs its own statistics and was never seen by the action encoder. - Distribution shift: test-split scores (new labs, kitchens, objects, tasks) are lower than validation. On the Bridge held-out kitchen, scores fell slightly late in training (22.57 dB at epoch 1229, 22.24 at 2489) while in-distribution Bridge improved.
- Physics is learned, not simulated: object motion can be implausible, objects can morph, appear or vanish, and the camera can drift, especially for in-the-wild video unlike the training data.
- Not for safety-critical decisions: predictions are samples from a generative model, not measurements.
- People in the data: three datasets show real people. Do not use the model to generate misleading video of identifiable people.
License
- Base model: Wan 2.1 I2V 14B, Apache-2.0.
- These weights: CC BY-NC 4.0 (https://creativecommons.org/licenses/by-nc/4.0/), see Training data licenses above.
Citation
@article{doppel2026,
title = {Render the Robot, Generate the World: One Video World Model Across Embodiments},
author = {Jangir, Yash and Mirakhor, Karan and Zabounidis, Renos and Irshad, Muhammad Zubair and
Zakharov, Sergey and Bisk, Yonatan and Bharadhwaj, Homanga},
year = {2026}
}
Verification
Checked before release (2026-11-05) against the original training checkpoint (EMA, fp32, 65.4 GB, sha256
42803a1aa2dc5c09eeb30bac1bf55e196ae4bc8762b08912e76cab40b330bc48):
- Weights: the evaluator's model was built from the base transformer and the original checkpoint was loaded the
way every paper number was produced. All 1,356 tensors in this overlay are bit-identical to the parameters
that model held. The 114 fp32 tensors (adapter,
time_proj,scale_shift_table) are also bit-identical to the fp32 checkpoint. The 1,242 bf16 tensors differ from the fp32 masters only by bf16 rounding: max relative error 3.9e-3 (2^-8), relative RMS error 1.5e-3 to 1.8e-3 per tensor group. - End to end: one evaluation clip (DROID, seed 374915573), 50 steps on one A100, scored with the paper's evaluator: PSNR 28.5735 dB with this overlay vs 28.5666 dB with the original checkpoint in the same session vs 28.5727 dB stored from the paper run. SSIM, LPIPS and SCS agree to 7e-5, 7e-5 and 4e-4. The +-0.007 dB spread is GPU run-to-run nondeterminism: two runs of the original checkpoint differ by the same amount.
- Downloads last month
- 2
Model tree for sourlreapwr/doppel-emb8
Base model
Wan-AI/Wan2.1-I2V-14B-480P-Diffusers