Spaces:
Running
Running
| title: README | |
| emoji: ⛏️ | |
| colorFrom: green | |
| colorTo: blue | |
| sdk: static | |
| pinned: false | |
| # FlowCraft | |
| Action-conditioned video world models for Minecraft. FlowCraft models are | |
| causal diffusion transformers trained with flow matching in the latent space of | |
| a fine-tuned WAN 2.2 VAE. They generate autoregressively, one latent at a time, | |
| conditioned on the latents and actions so far. | |
| This organisation holds the data, the VAEs and the model checkpoints from | |
| *How to Train Your Video World Model for Long-Horizon Rollout* | |
| (arXiv link coming soon). The paper compares pre-training recipes | |
| at equal compute, then post-trains three of them (Diffusion Forcing, | |
| clean-prefix teacher forcing and chunked teacher forcing): flow map finetuning | |
| followed by Self Forcing against the model's own pre-trained teacher. Every | |
| stage is evaluated on 1000-frame rollouts, far beyond the attention window. | |
| **How to Train Your Video World Model for Long-Horizon Rollout** | |
| Aidan Scannell\*, Samuel Garcin\*, Paul Chang, Peter Bell and Amos Storkey | |
| <sub>Toyota Research Institute · University of Edinburgh · Verda. \*Equal contribution.</sub> | |
| Code: [github.com/aidanscannell/flowcraft](https://github.com/aidanscannell/flowcraft) | |
| ## What is here | |
| | Resolution | Latents | VAE | Models | | |
| |---|---|---|---| | |
| | 128×224 | [`minecraft-latents-128-224-v2`](https://huggingface.co/datasets/flow-craft/minecraft-latents-128-224-v2) (142 GB) | [`wan-vae-minecraft-128-224`](https://huggingface.co/flow-craft/wan-vae-minecraft-128-224) | all 16 checkpoints | | |
| | 352×640 | [`minecraft-latents-352-640-v2`](https://huggingface.co/datasets/flow-craft/minecraft-latents-352-640-v2) (1.1 TB) | [`wan-vae-minecraft-352-640`](https://huggingface.co/flow-craft/wan-vae-minecraft-352-640) | none yet | | |
| Collections: [Models](https://huggingface.co/collections/flow-craft/flowcraft-models-6abb08191dd495f698f68b9a) · | |
| [VAEs](https://huggingface.co/collections/flow-craft/flowcraft-vaes-6abb08321dd495f698f69207) · | |
| [Datasets](https://huggingface.co/collections/flow-craft/flowcraft-datasets-6abb0833a001f08539186907) | |
| The latents are the gameplay recordings from OpenAI's | |
| [Video Pre-Training (VPT)](https://github.com/openai/Video-Pre-Training) | |
| contractor data, encoded once with the VAE of the same resolution, with the | |
| per-frame actions alongside. You never need the raw videos to train or evaluate. | |
| ## Checkpoint names | |
| `{pre,post}-train-<recipe>[-quarter-budget]` | |
| - **pre-train**: trained from scratch with the given recipe. | |
| - **post-train**: the final post-trained model. Stage 1 finetunes the | |
| pre-trained model into a few-step flow map; stage 2 is Self Forcing with | |
| distribution matching distillation, with the pre-trained model of the same | |
| recipe as the teacher (`post-train-df` comes from `pre-train-df`, and so on). | |
| Only stage 2 is released. | |
| - **Budget**: full is ~680 EFLOP (350k steps for single-stream recipes, 185k for | |
| the two-stream ones: `pf`, `chunk`, `jump`). `-quarter-budget` is 166 EFLOP | |
| (85k / 45k steps). | |
| | Recipe | Paper name | What it trains on | | |
| |---|---|---| | |
| | `df` | Diffusion Forcing | An independent flow time for every latent, with a loss on each. Uniform flow-time prior. | | |
| | `pf` | Clean-prefix teacher forcing | Every latent against its own clean prefix, all in one pass over a doubled clean/noisy sequence. | | |
| | `chunk` | Chunked teacher forcing (B=4) | As `pf`, but targets come in blocks of 4 latents, each seeing the clean prefix and the earlier members of its block at their own flow times. | | |
| | `jump` | Jump prediction (h<sub>max</sub>=3) | As `pf`, but up to 3 latents between a target and its clean prefix are deleted instead of noised. | | |
| | `ctx` | Context-length sampling | One context length per clip: the latents before it clean, a loss on every latent after it. | | |
| | `ctx-notail` | ctx − tail | As `ctx`, with a loss only on the first latent after the cut. | | |
| All recipes except `df` use a logit-normal flow-time prior. In pre-training, | |
| every recipe except `df` also uses **history corruption**: on a fraction p=0.1 of clips the clean | |
| history is presented at flow times τ ~ U[τ<sub>min</sub>, 1], with τ<sub>min</sub>=0.7. | |
| A name without a suffix uses these defaults, so `pre-train-pf` and | |
| `pre-train-pf-quarter-budget` differ only in budget. The suffixed variants | |
| sweep the history corruption (quarter budget only): | |
| | Suffix | History corruption | | |
| |---|---| | |
| | `pf-clean` | none (p=0) | | |
| | `pf` | p=0.1, τ<sub>min</sub>=0.7 (default) | | |
| | `pf-t05` | p=0.1, τ<sub>min</sub>=0.5 | | |
| | `pf-t00` | p=0.1, τ<sub>min</sub>=0 | | |
| Every checkpoint repo carries its full training config (`config.json`), the | |
| step, and the sha256 of the original training checkpoint (`meta.json`). | |
| ## Quickstart | |
| ```sh | |
| git clone https://github.com/aidanscannell/flowcraft && cd flowcraft | |
| uv sync --extra gpu --extra cu130 | |
| # Evaluate a released model (downloads the weights, VAE and test split) | |
| uv run fc-eval ckpt_path=flow-craft/pre-train-pf | |
| ``` | |
| ```python | |
| from omegaconf import OmegaConf | |
| from flowcraft.config import ExperimentConfig | |
| from flowcraft.model import build_world_model | |
| from flowcraft.train.checkpoint import CheckpointManager, load_checkpoint_config | |
| ref = "flow-craft/pre-train-pf@iclr2027-submission" | |
| cfg = OmegaConf.merge(OmegaConf.structured(ExperimentConfig), load_checkpoint_config(ref)) | |
| model = build_world_model(cfg) | |
| CheckpointManager(cfg.train.ckpt, "/tmp/unused").load(ref, model) | |
| ``` | |
| Only need a few tasks of the data? | |
| ```python | |
| from huggingface_hub import snapshot_download | |
| snapshot_download( | |
| "flow-craft/minecraft-latents-128-224-v2", repo_type="dataset", | |
| allow_patterns=["build-house/**", "manifest.json", "splits/**"], | |
| ) | |
| ``` | |
| ## Reproducing the paper | |
| Every checkpoint has an **`iclr2027-submission`** tag: the exact version the | |
| paper evaluated. Pin it (`@iclr2027-submission`) for reproduction; `main` may | |
| move. | |
| ## License | |
| - **Code**: [MIT](https://github.com/aidanscannell/flowcraft/blob/main/LICENSE). | |
| - **Checkpoints and VAEs**: [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0). | |
| The VAEs are fine-tuned from [WAN 2.2](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B-Diffusers) | |
| (Apache-2.0), and every checkpoint contains one. | |
| - **Datasets**: our encoding and packaging are MIT. The gameplay footage comes | |
| from OpenAI's [VPT](https://github.com/openai/Video-Pre-Training) contractor | |
| data and is subject to OpenAI's terms. Minecraft is a trademark of Mojang | |
| Synergies AB; this project is not affiliated with Mojang or Microsoft. | |
| ## Citation | |
| If you find FlowCraft, its checkpoints or its data useful, please consider citing us: | |
| ```bibtex | |
| @article{scannell2026flowcraft, | |
| title = {How to Train Your Video World Model for Long-Horizon Rollout}, | |
| author = {Scannell, Aidan and Garcin, Samuel and Chang, Paul and Bell, Peter and Storkey, Amos}, | |
| journal = {arXiv preprint arXiv:<ARXIV-ID>}, | |
| year = {2026} | |
| } | |
| ``` | |