DROID action-conditioned rollouts (left) and RealEstate10K camera-conditioned rollouts (right), shown at 2× speed. Click either grid for the full-resolution video, or see the project page for all 100 rollouts.
## Model Summary MiniWorld uses a block-causal Video Diffusion Transformer trained with Rectified Flow in the latent space of the Wan2.2 VAE. During inference, MiniWorld performs streaming generation with a rolling KV cache and pipelined asynchronous denoising, enabling long-horizon generation under bounded online computation. Key components: - **Block-causal Video DiT** with bidirectional attention inside each chunk and causal attention across chunks. - **Unified conditioning** for robot actions and camera poses through AdaLN-LoRA modulation. - **Chunk-oriented Probability Propagation (CoPP)** for stable non-decreasing diffusion schedules. - **Continued long-context training** from short clips to 253-frame sequences. - **Structured rolling KV cache** with a persistent sink and FIFO history. - **Pipelined asynchronous denoising** for a quality-throughput trade-off at inference time. The complete model can be trained in several days on a single 8-GPU server. ## Released Checkpoints Sampling requires matching the checkpoint with the corresponding dataset and model scale. | Dataset | Model | Status | Checkpoint | | --- | --- | --- | --- | | DROID | MiniWorld-0.5B | Available | [MiniWorld_0_5b_droid.pt](https://huggingface.co/zhaoyian01/MiniWorld/resolve/main/MiniWorld_0_5b_droid.pt) | | DROID | MiniWorld-1B | Available | [MiniWorld_1b_droid.pt](https://huggingface.co/zhaoyian01/MiniWorld/resolve/main/MiniWorld_1b_droid.pt) | | DROID | MiniWorld-3B | Coming soon | -- | | RealEstate10K | MiniWorld-0.5B | Available | [MiniWorld_0_5b_re10k.pt](https://huggingface.co/zhaoyian01/MiniWorld/resolve/main/MiniWorld_0_5b_re10k.pt) | | RealEstate10K | MiniWorld-1B | Available | [MiniWorld_1b_re10k.pt](https://huggingface.co/zhaoyian01/MiniWorld/resolve/main/MiniWorld_1b_re10k.pt) | | RealEstate10K | MiniWorld-3B | Coming soon | -- | Download a single checkpoint with: ```bash hf download zhaoyian01/MiniWorld \ --include "MiniWorld_1b_droid.pt" \ --local-dir checkpoints/miniworld ``` ## Model Configurations `MODEL` is the identifier expected by the training and sampling scripts in the GitHub repository. | Model | `MODEL` | Depth | Width | Heads | Parameters | | --- | --- | ---: | ---: | ---: | ---: | | MiniWorld-B | `B` | 12 | 768 | 12 | 0.12B | | MiniWorld-L | `L` | 24 | 1024 | 16 | 0.39B | | MiniWorld-0.5B | `0.5B` | 28 | 1152 | 16 | 0.55B | | MiniWorld-1B | `1B` | 28 | 1536 | 12 | 1B | | MiniWorld-3B | `3B` | 32 | 2560 | 20 | 3B | ## Intended Use MiniWorld is intended for research on streaming video world models, including: - action-conditioned robot world modeling, - camera-pose-conditioned scene prediction, - long-horizon autoregressive video generation, - temporal memory and KV-cache mechanisms, - train-test alignment for streaming diffusion models. MiniWorld is a research baseline and is not intended as a general-purpose text-to-video model. ## Requirements Inference requires the MiniWorld codebase and the pretrained Wan2.2 VAE: - Linux with an NVIDIA CUDA GPU - Python 3.11 - CUDA-compatible PyTorch 2.x - FlashAttention - Wan2.2 VAE checkpoint from `Wan-AI/Wan2.2-TI2V-5B` Download the VAE: ```bash hf download Wan-AI/Wan2.2-TI2V-5B \ --include "Wan2.2_VAE.pth" \ --local-dir checkpoints/wan2.2 ``` ## Usage Clone the [MiniWorld codebase](https://github.com/zhao-yian/MiniWorld), install its requirements, then download the desired checkpoint. All commands are run from the repository root. The default sampler uses one observed frame as initial context, eight in-flight chunks and a 24-chunk rolling KV cache (a 64-frame active attention window), one persistent sink frame, 100 denoising steps with classifier-free guidance at scale 2.0, and a 64-latent-frame rollout corresponding to 253 RGB frames. Generated videos are saved to `${SAMPLE_DIR}/pred/`. ### DROID action-conditioned generation ```bash DATA_ROOT=/path/to/droid_lerobot \ CKPT=/path/to/MiniWorld_1b_droid.pt \ VAE_CKPT=checkpoints/wan2.2/Wan2.2_VAE.pth \ MODEL=1B \ bash scripts/sample_droid.sh ``` ### RealEstate10K camera-conditioned generation ```bash DATA_ROOT=/path/to/re10k/videos \ POSE_DIR=/path/to/re10k/poses \ CKPT=/path/to/MiniWorld_1b_re10k.pt \ VAE_CKPT=checkpoints/wan2.2/Wan2.2_VAE.pth \ MODEL=1B \ bash scripts/sample_re10k.sh ``` ### Common inference controls ```bash GPU=0 \ TOTAL_LEN=96 \ CFG_SCALE=2.0 \ SAMPLE_NUM_VIDEOS=10 \ STREAM_INFLIGHT_CHUNKS=8 \ STREAM_MAX_CACHE_CHUNKS=24 \ STREAM_SINK_SIZE=1 \ bash scripts/sample_droid.sh ``` `TOTAL_LEN` sets the rollout length in latent frames and can exceed the trained window, since streaming keeps the attention span bounded; `TOTAL_LEN=96` yields 381 RGB frames from a 64-frame checkpoint. MiniWorld is a streaming model and does not assume a fixed generation horizon. ### Custom camera trajectories A RealEstate10K checkpoint can also animate a single image along a procedural camera trajectory, without any dataset on disk: ```bash PYTHONPATH=. python -m miniworld.sample \ --dataset re10k \ --init_image /path/to/first_frame.png \ --custom_camera_trajectory orbit_right \ --checkpoint /path/to/MiniWorld_1b_re10k.pt \ --vae_checkpoint checkpoints/wan2.2/Wan2.2_VAE.pth \ --sample_dir samples/re10k_orbit_right \ --wm_model 1B \ --total_len 64 \ --sample_num_videos 1 \ --trajectory_magnitude 3.0 ``` These checkpoints are trained on raw (unnormalized) translations, so `--trajectory_magnitude` is worth tuning: `1.0` is almost static, `3.0` is a good default at `--total_len 64`, and values above `5.0` degrade the second half of the rollout. Scale it with the rollout length to keep the same apparent speed. See the GitHub README for the full list of trajectories. ## Limitations MiniWorld is a research model trained and evaluated at modest resolution and on limited domains. It may exhibit long-horizon drift, geometric errors, temporal inconsistencies, and failures under out-of-distribution actions, poses, scenes, or camera motions. It should not be used for safety-critical simulation or as a faithful physical simulator. ## Citation The paper is available on arXiv: [arXiv:2608.01127](https://arxiv.org/abs/2608.01127). If you find MiniWorld useful in your research, please cite: ```bibtex @article{zhao2026miniworld, title = {MiniWorld: Democratizing the Training of Video World Models from Scratch}, author = {Zhao, Yian and Zheng, Ruochong and Guo, Hongcan and Yan, Yu and Zhang, Jian and Chen, Jie}, journal = {arXiv preprint arXiv:2608.01127}, year = {2026} } ``` ## License These checkpoints are released under the Apache 2.0 license. Please also follow the licenses and usage terms of the underlying datasets (DROID, RealEstate10K) and of the Wan2.2 VAE.