Impulse2000/svi-model-pro-5b

SVI-Pro LoRA for Wan2.2-TI2V-5B

Infinite-length video generation with error recycling — the 5B member of the Stable Video Infinity family

🎯 What this is

An SVI-Pro-style error-recycling LoRA that gives Wan2.2-TI2V-5B infinite-length video generation through clip chaining. Upstream Stable Video Infinity (SVI) ships SVI 2.0 Pro only for Wan2.2-I2V-A14B; the 5B TI2V model was missing from the family. This LoRA adds it — trained from scratch, including a re-implementation of the paper's Error-Recycling Fine-Tuning (ERFT) algorithm for a single-DiT model.

Code, training scripts and full implementation plan: JamesClarke7283/Stable-Video-Infinity-Reimplemented

Where this sits in the family:

	SVI 1.0 (Wan2.1 14B)	SVI 2.0 / 2.0 Pro (Wan2.2-A14B)	this (Wan2.2-TI2V-5B)
Shipped upstream	✅	✅	❌ — this repo
DiT	single	2 experts (high/low noise)	single
Conditioning	first-frame hold	y-concat (extra input channels)	fused N-frame clean hold (no extra channels available)
VRAM	14B class	27B MoE	5B — ≥24 GB (base model card lists an RTX 4090)
Res / fps	480×832 (Wan2.1 14B spec)	480×832, 15 fps (defaults in the shipped inference_svi_2.0_pro.py)	720P, 24 fps

✨ Features

Tasks	T2V and I2V — Wan2.2-TI2V-5B is a joint TI2V model, and this LoRA covers both: --task i2v (your image is the anchor) and --task t2v (clip 1's first latent becomes the anchor)
Length	Unbounded — chain any number of clips; 121 frames/clip at 24 fps ≈ 5 s per clip
Spec	720P (1280×704 landscape / 704×1280 portrait), 24 fps — the 5B model's maximum specification
Conditioning	Anchor latent (identity) + motion latent (continuity), held clean at t=0 and re-clamped after every sampling step
Cost	No extra tokens — the anchor/motion frames overwrite existing latent slots instead of being appended, one LoRA, one DiT, no decode/re-encode of previous clips. The only per-step delta is the timestep modulation being computed per frame rather than per sample
Size	615 MiB, rank 128, bf16

🔧 How it works

The 14B SVI 2.0 Pro conditions on extra DiT input channels: y = concat([anchor_latent, motion_latent, padding]). That trick cannot be ported to the 5B model — its DiT has no conditioning channels at all (in_dim == out_dim == z_dim == 48).

Instead, this LoRA learns a generalized version of the 5B's own native single-frame hold — an N-frame clean hold:

1. cond_latents = concat([anchor_latent?, motion_latent?], dim=2) — 0, 1 or 2 latent frames
2. Those frames are written into the initial noise and clamped back after every sampling step
3. A per-frame timestep map marks frames 0..num_cond-1 as t=0 (generalizing the stock seperated_timestep path, which hard-codes exactly one such frame)

Clip position	Task	cond_latents
First clip	I2V	[anchor] — identical to native TI2V I2V
Later clips	I2V	[anchor, motion]
First clip	T2V	[] — identical to native T2V
Later clips	T2V	[anchor_1, motion], anchor from clip 1's first latent

• anchor = the user's first frame (I2V) or clip 1's first latent (T2V) — shared across all clips, which is what preserves identity over long horizons.
• motion = the previous clip's last latent, handed over in latent space. Previous clips are never decoded and re-encoded, so no VAE round-trip error accumulates.
• Stitching drops the first 5 pixel frames of each later clip (the duplicated anchor/motion tail).

Because all of this is pipeline-level, it is learnable by a LoRA with no DiT surgery.

Error-Recycling Fine-Tuning (ERFT)

Re-implemented from the paper (arXiv 2510.09212) for the single-DiT 5B model. At each training step the DiT's own errors are injected back into its input, so it learns to correct drift rather than merely avoid it:

• Error replay banks — 50 timestep grids × up to 500 curated errors, filled by the bidirectional one-step curation of Eq. 4 and sampled during training
• Injection — latent error p=0.9, conditioning error p=0.9, noise error p=0.01, with a clean-input override at p=0.5 to preserve base generation ability
• Target — the model sees corrupted input but must predict velocity toward clean data
• The noise bank is stored implicitly (E_noise = -((1-σ)/σ)·E_clean), halving host RAM at full paper scale

🚀 Usage

git clone https://github.com/JamesClarke7283/Stable-Video-Infinity-Reimplemented
cd Stable-Video-Infinity-Reimplemented

# Python 3.10+; uv is what this project uses.
uv venv
# RTX 50-series (Blackwell, sm_120) needs CUDA >= 12.8 wheels — install torch
# FIRST so `pip install -e .` sees torch>=2.0 already satisfied and keeps it.
uv pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
uv pip install -e .

# the base weights (DiT + Wan2.2_VAE.pth + umt5) auto-download into ./models
export DIFFSYNTH_DOWNLOAD_SOURCE=huggingface

Then put this LoRA file anywhere and pass it as --lora_path:

Image-to-video (anchor = your image):

python inference_svi_pro_5b.py \
  --task i2v \
  --ref_image_path anchor.png \
  --prompt_path prompts.txt \
  --lora_path SVI_Pro-Wan2.2-TI2V-5B_lora.safetensors \
  --num_clips 20 \
  --height 704 --width 1280 --frames_per_clip 121 --fps 24 \
  --tiled

Text-to-video (generated anchor):

python inference_svi_pro_5b.py \
  --task t2v --t2v_anchor_mode generated \
  --prompt "a majestic sailing ship on turquoise ocean waves, cinematic wide shot" \
  --lora_path SVI_Pro-Wan2.2-TI2V-5B_lora.safetensors \
  --num_clips 8 \
  --tiled

prompts.txt is a Python list; clip k consumes prompt k (cycled), giving you streaming storylines. Output is one mp4 per clip plus a stitched *_streaming_final.mp4, written at 24 fps.

Key defaults (leave them alone)

Argument	Value	Why
--lora_alpha	1.0	LoRA strength here is test-time error-recycling intensity; the paper measures degradation at ≤0.8
--num_inference_steps	50	Matches the timestep grids the error banks were built on
--cfg_scale	5.0
--sigma_shift	5.0	Wan flow-matching shift
--num_motion_latent	1	What the LoRA was trained with
--num_overlap_frame	5	Duplicated frames dropped at each seam
--fps	24	Native/max spec of the 5B model

Height/width must be divisible by 32; frames_per_clip % 4 == 1.

📦 Training configuration

Trained on 1× RTX 5090 32 GB, ~50 k steps over ~160 h.

Algorithm	SVI Error-Recycling Fine-Tuning (ERFT), SVI 2.0 Pro semantics
Base model	Wan-AI/Wan2.2-TI2V-5B (single DiT, full timestep range — no high/low-noise expert split)
LoRA	rank 128, alpha 128, targets q,k,v,o,ffn.0,ffn.2 → 300 modules / 600 tensors, 322 M params, bf16
Resolution	720P, 704×1280, 121 frames/clip @ 24 fps (max spec, training = inference)
Steps / epochs	50 000 steps, 10 epochs
Optimizer	AdamW, lr 1e-4 constant to ~step 29 k, then cosine to 1e-5 at 50 k; weight decay 0.01, grad-clip 1.0
Precision	bf16 models, fp32 loss; gradient checkpointing + DiT torch.compile
Error banks	50 grids, Z=500, warmup 50 iters, spatial pool 3 (720P banks need ~25 GB host RAM at paper scale)
Task mix	I2V-chained 0.45 / I2V-first 0.15 / T2V-chained 0.30 / T2V-pure 0.10
Anchor augmentation	p=0.5 a random frame instead of the video's first — encourages anchor-guided restoration

Data: 5 555 MixKit stock videos (5 000 train / 555 holdout) out of the 6 428-video pool probed — all 1280×720, filtered to ≥ 242 frames (10.1 s) and normalized to 24 fps, so every sample carries a full previous-clip + window pair. Latents and text embeddings were pre-cached, leaving a DiT-only train pass (43 s/step → 12.1 s/step median, 25.2 GB peak VRAM). Train-loop throughput and memory above are read from the run's own metrics.csv.

Deviations from the paper's Table 9

Deliberate, and stated so results can be compared honestly:

Paper	Here	Why
lr 2e-5	1e-4	The shipped SVI training scripts and DiffSynth's own Wan2.2-5B LoRA recipes both use 1e-4; a 1e-4/2e-5 ablation was out of compute budget
480×832, 81 frames, 5 motion frames	704×1280, 121 frames, 1 motion latent	Train at the 5B's maximum spec instead of the 14B's training spec; SVI 2.0 Pro itself uses a single motion latent
Warmup 20 iters	50	Longer warmup before bank curation starts; harmless
50-step timestep grid, Z=500, p_vid/p_img/p_noi/p_clean	same	Paper-scale banks, with E_noise derived rather than stored to fit host RAM

📊 Evaluation

Two 8-clip chained runs (38.8 s, 933 frames @ 24 fps), 50 steps, CFG 5.0, identical per-clip seeds, base Wan2.2-TI2V-5B vs this LoRA on the same code path. Per-frame metrics from scripts/analyze_video.py; the luminance trend was independently reproduced with ffmpeg signalstats (YAVG, quoted below — its absolute values run lower than the RGB-mean luma by construction, but the drift direction and magnitude match).

T2V — one fixed 8-prompt stream, no anchor image

Prompts: sailing ship, steady → wind picks up → larger swells → seagulls → camera orbits → golden light → golden hour → sunset.

	Base 5B chaining	+ this LoRA
Frame-to-frame change (mean |Δ|, frames sampled 1 s apart)	12.6	14.8
Mean luminance YAVG, clip 1 → clip 8	184.8 → 185.5 (flat)	172.4 → 127.1 (−26 %)
YAVG per clip (1→8)	184, 183, 174, 169, 181, 172, 170, 184 (no trend)	173, 170, 169, 157, 149, 130, 135, 127 (monotone decline)
Seam PSNR (7 clip joins)	11.8 – 25.5	9.6 – 26.0
PSNR vs first frame, first → last clip end	25.4 → 22.0	24.5 → 12.4

I2V — anchored on a real photo (Siamese cat on a sofa)

	Base 5B chaining	+ this LoRA
Frame-to-frame change	23.4	15.8
Mean luminance YAVG, clip 1 → clip 8	127.4 → 150.7 (+23, brightening)	127.4 → 134.3 (+6.9)
YAVG per clip (1→8)	127, 127, 119, 128, 136, 139, 132, 155	127, 128, 123, 124, 124, 118, 126, 138
Seam PSNR (7 clip joins)	10.1 – 15.9 (mean 13.2)	12.3 – 28.8 (mean 19.7)
PSNR vs first frame, first → last clip end	34.3 → 10.2	33.5 → 12.1

Reading this honestly. The two directions disagree, so the picture is mixed rather than a clean win:

• T2V: the LoRA adds motion and keeps the subject identifiable (base chaining is visibly frozen: identity drifts while the scene barely changes), but pays a ~26 % luminance loss.
• I2V: the anchor does most of the consistency work — the base model is already fairly stable here. The LoRA's clearest win is seam quality: mean seam PSNR rises from 13.2 to 19.7, so the joins are noticeably smoother. Luminance drift is cut to roughly a third (+6.9 vs +23 over 8 clips). Frame-to-frame change is also much lower (15.8 vs 23.4), and on this subject that reads as less uncontrolled motion — the pose sequence is still present, just less erratic.
• The darkening is T2V-specific in these runs. On I2V the LoRA did not darken at all; it brightened, like base, just less. These are single 8-clip samples per setting, so treat the directions as indicative, not as a measured law.

The paper documents the closest known failure mode (Sec. B.4: color shift when the test style diverges from the training distribution) and recommends broader training data as the fix. If you hit drift, shorter chains (≤4 clips) stay in range; the intermediate checkpoint step-28800 had far more uniform seam continuity on T2V (PSNR 19.2–23.3 across all 7 seams) at the cost of some motion.

📁 Files

File	Description
SVI_Pro-Wan2.2-TI2V-5B_lora.safetensors	The LoRA — 600 bf16 tensors, rank 128, blocks.{0..29}.* keys, load with pipe.load_lora(pipe.dit, path, alpha=1)

SHA256 13b351bbd9a2c8d6e2d732de6ef3962e115b8810ed817b7a63e658da12f16e7b

These are the step 49 600 weights (epoch 9, val 0.158972 — the lowest clean-input validation loss of the run), out of a 50 000-step / 10-epoch schedule. That checkpoint was selected on a clean-input probe, which goes flat while error recycling is still improving, so treat it as a reasonable default rather than a proven optimum: an earlier checkpoint (step-28800) traded motion for better seam continuity in the eval above.

📝 Citation

@article{li2025stable,
  title={Stable Video Infinity: Infinite-Length Video Generation with Error Recycling},
  author={Li, Wuyang and Pan, Wentao and Luan, Po-Chien and Gao, Yang and Alahi, Alexandre},
  journal={arXiv preprint arXiv:2510.09212},
  year={2025}
}

⚖️ Licenses & provenance

• This LoRA — Apache-2.0
• Algorithm — Stable Video Infinity (VITA @ EPFL, Apache-2.0)
• Framework — DiffSynth-Studio (ModelScope Team, Apache-2.0), the svi_wan22 branch
• Base model — Wan2.2-TI2V-5B (Wan-AI / Alibaba, Apache-2.0)
• Training data — MixKit free stock video (MixKit license)

Not affiliated with or endorsed by the SVI authors, ModelScope, or Alibaba.