Request access to the LayerGen weights

Access is reviewed manually. Please tell us who you are and how you intend to use these research checkpoints.

LayerGen — Harmonize (grounded, fg-pred type-4) · `harmonize_grounded_fgpred_type4_gbs32_4n_b200`

Single-task video harmonization / compositing checkpoint: given a foreground layer and a background layer, it generates the harmonized composite video (relit / colour-matched / shadow- consistent). Built on Wan2.2-14B as a multi-stream diffusion transformer.

Task: harmonize (foreground + background → composite)
fg-pred type-4: this variant also predicts the pristine foreground and uses a dedicated 5th stream type (id 4) for the corrupted-foreground observation — so it requires FASTVIDEO_FG_CORRUPT_TYPE_ID=4 at inference (see below).
Base model: Wan2.2-14B diffusers WanPipeline (40-layer / 5120-dim DiT, AutoencoderKLWan VAE, UMT5EncoderModel)
Trained at: 480×832, 41 frames (11 latent frames), flow-shift 3.0 · 4-node B200, gbs32 · grounded editcap2 captions

Checkpoints in this repo

Path	Which weights	Step
`checkpoint-4000/`	EMA (exponential moving average)	4000
`nonema/checkpoint-4000/`	raw / non-EMA (the "pure" training weights)	4000

Both are model-only DCP exports (roles.* = the trained DiT only, ~53 GB fp32 each; the frozen Wan VAE / text encoder come from init_from). Each dir is a self-contained checkpoint-<step>/ with dcp/ inside, plus a metadata.json (embedded training config). EMA weights are usually the better choice for sampling; the non-EMA weights are provided for completeness / ablation.

Requirements

FastVideo with the layer-decomp stack (fastvideo/train/entrypoint/infer_layer_decomp.py, fastvideo.train.models.layer_decomp.LayerDecompWanModel). Reference: branch prompt-fix-infer, commit b7ccac76. (Currently internal research code — you need access to it to run inference.)
A local Wan2.2-14B diffusers model for the VAE + architecture; point init_from in the config at it.
One 80 GB GPU (A100-80G / H100 / H200 / B200). FlashAttention recommended (TORCH_SDPA also works).

Inference

# Point configs/infer_harmonize.yaml:  init_from -> your local Wan2.2-14B diffusers dir,
#                                       data_path -> your encoded latent pool (see encode_evals.py).
export FASTVIDEO_ATTENTION_BACKEND=FLASH_ATTN        # or TORCH_SDPA
export FASTVIDEO_COMBINED_INFER=0                    # single-task checkpoint
export FASTVIDEO_FG_CORRUPT_TYPE_ID=4                # REQUIRED: 5-type stream layout (fg-corrupt = its own type id 4)
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
export TOKENIZERS_PARALLELISM=false WANDB_MODE=disabled

# EMA weights:      --checkpoint ./checkpoint-4000
# non-EMA weights:  --checkpoint ./nonema/checkpoint-4000
torchrun --standalone --nnodes 1 --nproc_per_node 1 \
  -m fastvideo.train.entrypoint.infer_layer_decomp \
  --config configs/infer_harmonize.yaml \
  --checkpoint ./checkpoint-4000 \
  --out ./out --steps 30 --model-only \
  --training.distributed.num_gpus 1 \
  --training.distributed.hsdp_replicate_dim 1 \
  --training.distributed.hsdp_shard_dim 1 \
  --training.distributed.sp_size 1

⚠️ You must set FASTVIDEO_FG_CORRUPT_TYPE_ID=4. The DiT sizes its stream_type_embedding from this env var at import; with the default (2) the DCP load fails with a shape mismatch ([5,5120] vs [4,5120]).

Outputs are written under ./out/samples/<clip_id>/ (per-stream mp4s: fg/bg conditions, predicted composite, predicted fg, plus a per-clip record.json).

Longer clips: trained at 41 frames. For 61f/81f set FASTVIDEO_ROPE_T_INTERP=(41-1)/(T_infer-1) and bump num_latent_t (16 for 61f, 21 for 81f).

Provenance

Run harmonize_grounded_fgpred_type4_gbs32_4n_b200 (wandb har_grnd_fg_type4_b200), checkpoints at step 4000 (EMA + non-EMA). Part of the LayerGen layer-decomposition project.

License

Research preview released for reproducibility. Built on Wan2.2 (see the Wan-AI license for the base model). Contact the authors for reuse terms.

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support