zhaoyian01/Frechet_Forcing

Fréchet Forcing

Fréchet Forcing: Training Long-Horizon Streaming World Models from Short Clips

Fréchet Forcing is a teacher-free, critic-free post-training framework for long-horizon streaming video world models. Starting from MiniWorld-1B (pretrained on 253-frame clips), Fréchet Forcing fine-tunes the world model together with its deployed multi-step streaming sampler on 381-frame real clips for at most 36 GPU-hours (225 steps) — sustaining coherent 4,000+ frame rollouts (>10× the fine-tuning horizon) with no teacher, no online critic, and no long training videos.

The same framework supports two control modalities:

• DROID: low-level robot actions (end-effector Cartesian pose and gripper position) for embodied world modeling.
• Sekai: camera trajectories (Plücker ray encodings) for controllable 3D scene exploration.

This Hugging Face repository hosts the Fréchet Forcing (MiniWorld-1B-FF) checkpoints. Code, training scripts, and long-horizon evaluation utilities live in the GitHub repository.

Qualitative Results

Each tile is a 4,001-frame (~4.2 min) streaming rollout from the MiniWorld-1B-FF checkpoint, generated chunk by chunk from a single observed real frame plus the action or camera trajectory stream.

_{DROID 4,001-frame action-conditioned rollouts (left) and Sekai 4,001-frame camera-conditioned rollouts (right). Click either grid for the video, or see the project page for interactive comparisons and individual rollouts.}

Model Summary

Fréchet Forcing fine-tunes a block-causal Video Diffusion Transformer (MiniWorld-1B) in the latent space of the Wan2.2 VAE together with its deployed staircase streaming sampler under three unit-weighted constraints derived solely from short real clips:

• Generative Anchoring ($\mathcal{L}_{\text{anc}}$): Applies the flow-matching loss on a 4-chunk window conditioned on a self-generated history ($c \sim U[1, 40]$ chunks re-noised to $t_a = 0.7$ and denoised along the deployed staircase ladder) ending on a single clean ground-truth chunk, preserving action-conditioned generation while eliminating chunk-boundary seam redraw.
• Context Fidelity ($\mathcal{L}_{\text{ctx}}$): Matches the Fréchet distance (FD) of pooled spatiotemporal VAE latent moments over the gradient-carrying tail of a 40-chunk free rollout, centred on each clip's own ground truth, keeping every rollout faithful to its initial scene.
• Reality Matching ($\mathcal{L}_{\text{real}}$): Matches the Fréchet distance of decoded rollout frames against real training frames in frozen perceptual feature space (DINOv2 ViT-B/14), preventing degenerate textures and exposure drift.
• Self-normalised EMA Fréchet Distance: Running moment estimation ($\beta = 0.99$) with self-normalised scaling $\mathrm{FD} / \operatorname{sg}(\mathrm{FD})$ enables batch-size-one training per GPU without storing past features, a teacher network, or an online critic.

At inference, MiniWorld-1B-FF performs streaming generation with a 24-chunk rolling KV cache, 1 persistent sink latent frame, 8 in-flight chunks, and an 8-step staircase sampler (CFG scale 1.0).

Released Checkpoints

Sampling requires matching the checkpoint with the corresponding dataset and model scale.

Dataset	Model	Status	Checkpoint
DROID	MiniWorld-1B-FF	Available	miniworld1b_droid_frechet_forcing.pt
Sekai	MiniWorld-1B-FF	Coming soon	--

Download the DROID checkpoint with:

hf download zhaoyian01/Frechet_Forcing \
  --include "miniworld1b_droid_frechet_forcing.pt" \
  --local-dir checkpoints

Model Configurations

MODEL (--wm_model) is the architecture identifier expected by the training and sampling scripts in the GitHub repository.

Model	MODEL	Depth	Width	Heads	Parameters	Resolution	Chunk Size	Sampler Steps
MiniWorld-1B-FF	1B	28	1536	12	1B	240×320	2 latents (8 frames)	8

Intended Use

Fréchet Forcing (MiniWorld-1B-FF) is intended for research on long-horizon streaming video world models, including:

• action-conditioned robot manipulation world modeling over thousands of frames,
• camera-pose-conditioned 3D scene exploration,
• teacher-free and critic-free post-training for autoregressive diffusion models,
• long-horizon error accumulation, context fidelity, and perceptual distribution matching,
• KV-cached streaming video generation under bounded online computation.

This model is a research checkpoint trained on specific domains and is not intended as a general-purpose text-to-video generator.

Requirements

Inference requires the Fréchet Forcing codebase and the pretrained Wan2.2 VAE:

• Linux with an NVIDIA CUDA GPU
• Python 3.10+
• CUDA-compatible PyTorch 2.x
• FlashAttention
• Wan2.2 VAE checkpoint from Wan-AI/Wan2.2-TI2V-5B

Download the VAE:

hf download Wan-AI/Wan2.2-TI2V-5B \
  --include "Wan2.2_VAE.pth" \
  --local-dir checkpoints

Usage

Clone the Fréchet Forcing codebase, install its requirements, then download the checkpoint. All commands are run from the repository root.

The default sampler uses one observed latent frame as initial context, eight in-flight chunks and a 24-chunk rolling KV cache, one persistent sink latent frame, 8 denoising steps per chunk with staircase step 1 and CFG scale 1.0, and a 240-latent-frame rollout corresponding to 961 RGB frames (60 seconds at 16 fps). Generated videos are saved to ${SAMPLE_DIR}/pred/.

DROID action-conditioned generation

DATA_ROOT=/path/to/droid_lerobot \
CKPT=checkpoints/miniworld1b_droid_frechet_forcing.pt \
VAE_CKPT=checkpoints/Wan2.2_VAE.pth \
SAMPLE_DIR=outputs/samples_droid \
bash scripts/sample_droid.sh

Common inference controls

Pass additional flags to scripts/sample_droid.sh or invoke miniworld.sample directly to control the rollout horizon (--total_len in latent frames, where $T_{\text{rgb}} = 4 \times (T_{\text{lat}} - 1) + 1$):

# Generate 4,001-frame (~4.2 min) ultra-long streaming rollouts (1,001 latent frames)
DATA_ROOT=/path/to/droid_lerobot \
CKPT=checkpoints/miniworld1b_droid_frechet_forcing.pt \
VAE_CKPT=checkpoints/Wan2.2_VAE.pth \
bash scripts/sample_droid.sh \
  --total_len 1001 \
  --sample_num_videos 8

Long-horizon evaluation

Evaluate long-horizon DROID rollouts with forward-backward episode replay:

python3 scripts/eval_long_droid.py \
  --checkpoint checkpoints/miniworld1b_droid_frechet_forcing.pt \
  --vae_checkpoint checkpoints/Wan2.2_VAE.pth \
  --data_root /path/to/droid_lerobot \
  --horizons_sec "16,60,120" \
  --out_dir outputs/eval_droid_long

Limitations

MiniWorld-1B-FF is a research world model trained and evaluated at $240\times320$ resolution on specific domains (DROID and Sekai). Because the distributional objectives operate at the population level over short training rollouts and use coarse perceptual features, a minority of ultra-long rollouts (e.g., beyond 2,500–4,000 frames) may exhibit gradual exposure drift, periodic surface textures, or localized object artifacts, and revisited regions are not guaranteed to maintain long-range 3D geometric consistency after leaving the rolling KV cache. It should not be used for safety-critical control or as a faithful physical simulator.

Citation

If you find Fréchet Forcing useful in your research, please cite:

@article{zhao2026frechetforcing,
  title   = {Fr{\'e}chet Forcing: Training Long-Horizon Streaming World Models from Short Clips},
  author  = {Zhao, Yian and Yan, Yu and Guo, Hongcan and Zheng, Ruochong and Zhang, Jian and Chen, Jie},
  journal = {arXiv preprint},
  year    = {2026}
}

License

These checkpoints are released under the Apache 2.0 license. Please also follow the licenses and usage terms of the underlying datasets (DROID, Sekai) and of the Wan2.2 VAE.