wantrack_synth_sparse_warmstart_2000

WanTrack (MotionStream-aligned, sparse-track-conditioned) DiT overfitted on the 20-clip synthetic-toy dataset (see noctuashap/wantrack_synth_toy). This is the transformer only; VAE / T5 / CLIP come from the Fun-InP base.

What's included

• transformer/config.json -- TrackWanTransformer3DModel config (in_channels=52, id_dim=64, zero_init_head=false, track_channels=16).
• transformer/model.safetensors -- fine-tuned transformer weights (~5.9 GB).

Training recipe (2000 steps, from control_init base)

Two-stage on the 20-clip synth-toy dataset (single shared first frame, 20 diverse I2V motions):

1. Sparse-fixed for 1000 steps: WANTRACK_SPARSE=1 WANTRACK_EXTRA_RANDOM=20 WANTRACK_FIXED_SAMPLE=1 -> 1 track per SAM object + 20 background extras, deterministic per-video.
2. Sparse-random warmstart from step 1000 for another 1000 steps: same recipe but WANTRACK_FIXED_SAMPLE=0 (fresh random subset every step, random sinusoidal track IDs).

Other paper-alignment changes from the WanTrack repo (hao-ai-lab/FastVideo, branch trackwan_bidir): bias=False on track_encoder.temporal_conv / proj, flow_shift=6, joint text+motion CFG (w_t=3.0, w_m=1.5) at inference.

At step 2000 average |gen - no_track| divergence was ~3.2% (vs ~1% for cold-start random, ~9% for a pure-fixed run that memorizes track-video pairing).

How to use

from fastvideo.models.dits.trackwan.model import TrackWanTransformer3DModel
tr = TrackWanTransformer3DModel.from_pretrained("noctuashap/wantrack_synth_sparse_warmstart_2000", subfolder="transformer")
# combine with weizhou03/Wan2.1-Fun-1.3B-InP-Diffusers VAE/T5/CLIP for the full pipeline.

Or use the app_action.py gradio app in examples/inference/gradio/trackwan/ from the repo above.