CaT-Movie-5B (Renderer Checkpoint)

Project Page GitHub Code HF Collection Dataset CaT-MBench License

Finetuned Wan2.2-TI2V-5B multi-shot renderer weights (step-3000.safetensors, 9.44 GiB, BF16) for CaT-Movie: Balancing Global Planning and Local Rendering for Long Multi-Shot Video Generation.

CaT-Movie Overview: Global Storyboard Planner, Prompt Compiler & Anchor Recall, and Local Window Renderer

Architecture & Usage Contract

Important: step-3000.safetensors is not a drop-in checkpoint for stock Wan2.2 generate.py. Run inference through run/render_story.py with configs/infer/cat_movie_5b.py so CaT-Movie's compiled prompts, window attention masks, and anchor recall slots are enabled.

  • Backbone: Wan-AI/Wan2.2-TI2V-5B (1280Γ—704, 24 fps, umT5-XXL, Wan2.2 16ch VAE; all 5.0B transformer blocks trained for 3,000 rectified-flow updates on 8Γ— A100).
  • Shot-Wise Cross-Attention: Each target shot in a window (8 s training / 7 s test cap) attends exclusively to its compiled per-shot prompt c_k.
  • Reference-Routed Sparse Self-Attention (holocine_first_frame + routed_self_only_v1): Each shot attends to itself, co-generated window first latent frames, and up to 10 storyboard-indexed clean anchor views (history_policy = plan_selection) placed at negative temporal RoPE offsets (Ξ”t = -5) with per-reference identity lines (frame_entities_v1).
  • Planner: Pairs cleanly with a training-free structured planner (Qwen3.8-27B via vLLM; setup in the GitHub repository).

Quickstart: Render a Story

export MSPC_SHARED_ROOT=/path/to/cat_movie_workspace

# 1. Download base Wan2.2-TI2V-5B and CaT-Movie-5B weights
hf download Wan-AI/Wan2.2-TI2V-5B --local-dir "$MSPC_SHARED_ROOT/checkpoints/Wan2.2-TI2V-5B"
hf download JinPLU/CaT-Movie step-3000.safetensors --local-dir "$MSPC_SHARED_ROOT/checkpoints/CaT-Movie"

# 2. From a clone of JinPLu/CaT-Movie:
export MSPC_CODE_ROOT="$(pwd)"
PYTHONPATH=.:src python run/render_story.py \
  --config configs/infer/cat_movie_5b.py \
  --storyboard examples/sample_storyboard.json \
  --story-id cinedance-008935-b0001-shot01 \
  --output outputs/demo/cinedance-008935-b0001-shot01 \
  --seed 42 --segment-seed-stride 1009 --segment-max-shots none \
  --segment-max-frames 112 --history-policy plan_selection

Files in This Repository

File Description Size
step-3000.safetensors CaT-Movie Wan2.2-TI2V-5B renderer weights (step 3,000, BF16) 9.44 GiB
LICENSE CC BY-NC-SA 4.0 (non-commercial research weights) β€”

Main Quantitative Results (CaT-MBench, 92 Test Stories)

Inter-shot consistency vs. transition control on CaT-MBench, breaking the baseline Pareto frontier

Evaluated with seed 42 via TransNetV2 (Cut F1 @ 0.125 s, Execution over 674 setup-changing cuts), Gemini 3.8 Flash (6 native-res frames/pair for Consistency; 2 fps whole video for Establish/Persist under the cut gate), and ViCLIP (Text Alignment). Baselines use annotated reference storyboards in native prompt formats; CaT-Movie renders its Qwen3.8-27B planner's storyboard from the shots request (bold = best, italic = second best):

Method Cut F1 ↑ Exec. ↑ Char. ↑ Obj. ↑ Scene ↑ Est. ↑ Persist ↑ Align. ↑ Avg. ↑
Wan2.1, per shot (1.3B) 0.998 0.994 0.219 0.169 0.310 0.274 0.694 0.181 0.474
Wan2.2, per shot (5B) 0.999 0.996 0.227 0.275 0.310 0.387 0.701 0.174 0.497
StoryMem (A14B) 0.928 0.719 0.631 0.453 0.687 0.484 0.545 0.179 0.527
StoryDiffusion (A14B) 0.905 0.825 0.300 0.265 0.245 0.597 0.535 0.171 0.468
IC-LoRA (A14B) 0.976 0.961 0.168 0.133 0.406 0.452 0.663 0.176 0.484
MultiShotMaster (1.3B) 0.994 0.990 0.538 0.254 0.582 0.371 0.587 0.178 0.527
HoloCine (A14B) 0.748 0.967 0.517 0.512 0.586 0.694 0.691 0.195 0.571
CaT-Movie (Ours) (5B) 0.995 0.974 0.651 0.497 0.630 0.468 0.785 0.186 0.597

(Backbones: Wan2.1-T2V-1.3B, Wan2.2-TI2V-5B, Wan2.2-I2V-A14B for StoryMem/StoryDiffusion/IC-LoRA, Wan2.2-T2V-A14B for HoloCine. Columns shorten Execution, Character, Object, Scene, Establish, Text Alignment.)

Compute & Limitations

  • Hardware: Single GPU with β‰₯ 40 GB VRAM (BF16) for 1280Γ—704, 50 flow UniPC steps (shift = 5, CFG = 5).
  • Trade-offs: CaT-Movie lifts within-shot change execution (Establish) over its 5B base (0.387 β†’ 0.468) alongside 0.785 Persist, though Establish tracks base model size across 5B vs. 14B models (0.45–0.69). Multi-window anchor recall assumes returning entities remain clear in an earlier keyframe.
CaT-Movie Release Ecosystem (GitHub & 8 HF Repositories)
Repository Tier Contents
JinPLu/CaT-Movie (GitHub) Core Planner, training, inference, curation & CaT-MBench code (Apache-2.0)
JinPLU/CaT-Movie (this repo) Core Renderer weights (step-3000.safetensors)
JinPLU/CaT-Movie-Data Core 75,461 blocks, Qwen3.7-Flash schemas, grounding, 96,000 train windows & SAM 3.1 weights
JinPLU/CaT-Movie-T5-Cache Core Precomputed umT5-XXL BF16 caches (332,516 unique prompts)
JinPLU/CaT-Movie-VAE-Caches Core Precomputed VAE target latents (Wan2.2 704p + Wan2.1 480p)
JinPLU/CaT-Movie-Source-Videos Supporting Complete 18,173 block-yielding source MP4 clips
JinPLU/CaT-MBench Core 122 stories, cut-gated GPT-6.1-sol VQA bank, Gemini 3.8 Flash verdicts & 557 human labels
JinPLU/CaT-MBench-Videos Supporting 736 generated test videos (92 test stories Γ— 8 methods)
JinPLU/CaT-MBench-Author-Labels Archive Early Nov 6 single-author calibration archive (superseded by CaT-MBench/human/)

Licensing & Citation

Released under CC BY-NC-SA 4.0 strictly for non-commercial academic research (derived from Wan-AI/Wan2.2-TI2V-5B, Apache-2.0; codebase Apache-2.0).

@misc{catmovie2026,
  title         = {CaT-Movie: Balancing Global Planning and Local Rendering for Long Multi-Shot Video Generation},
  author        = {Jinpeng Lu and Haoyuan Shi and Yuanshen Guan and Jiyuan Xia and Bin Wu and Yinda Chen and Zhiwei Xiong},
  year          = {2026},
  url           = {https://jinplu.github.io/CaT-Movie/}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for JinPLU/CaT-Movie

Finetuned
(145)
this model

Collection including JinPLU/CaT-Movie