PhysAlign: Physics-Coherent Image-to-Video Generation through Feature and 3D Representation Alignment
Paper • 2603.13770 • Published
import torch
from diffusers import DiffusionPipeline
from diffusers.utils import load_image, export_to_video
# switch to "mps" for apple devices
pipe = DiffusionPipeline.from_pretrained("Wan-AI/Wan2.2-I2V-A14B", dtype=torch.bfloat16, device_map="cuda")
pipe.load_lora_weights("yrlyrl/wan2.2-i2v-a14b-physalign-lora")
prompt = "A man with short gray hair plays a red electric guitar."
input_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/guitar-man.png")
image = pipe(image=input_image, prompt=prompt).frames[0]
export_to_video(output, "output.mp4")Wan2.2-I2V-A14B · PhysAlign LoRA (faithful reproduction, pure-Gram)
LoRA post-training of Wan2.2-I2V-A14B with a V-JEPA2 feature-alignment (Gram) loss on synthetic physics videos (PhyCo Kubric), reproducing the PhysAlign recipe (arXiv:2603.13770) without the 3D depth loss. Improves the intrinsic physical consistency of generated image-to-video.
Results (Physics-IQ, official scorer, full 198)
| Model | Physics-IQ ↑ | PhyCo cont. ST-IoU ↑ | PisaBench IoU ↑ / Chamfer ↓ |
|---|---|---|---|
| Wan2.2-I2V-A14B base | 31.75 | 0.097 | 0.018 / 0.275 |
| + this LoRA (a14b_2x) | 37.04 (+5.29) | 0.311 (3.2×) | 0.070 (3.9×) / 0.263 |
| PhysAlign paper (Full, w/ 3D) | 38.1 | — | — |
Pure Gram alignment (no 3D) reaches near paper-level gain, confirming the paper's ablation that the geometry loss is not the bottleneck.
What's in here
The A14B is a dual-expert MoE (high-noise + low-noise). PhysAlign requires training both experts, each on its own timestep range:
low_noise_expert/pytorch_lora_weights.safetensors— LoRA fortransformer_2(σ < 0.9)high_noise_expert/pytorch_lora_weights.safetensors— LoRA fortransformer(σ ≥ 0.9)config_*.yaml— the exact training configs
LoRA rank 32 on q/k/v/o + ffn.0/ffn.2.
The 4 faithful-reproduction fixes (vs a naive single-expert run, +0.58 → +5.29)
- Train BOTH MoE experts separately (the high-noise expert governs coarse motion/physics) — the dominant fix.
- Multi-block alignment at DiT blocks 12/16/20/24 (not a single block).
- Teacher input keeps aspect ratio (V-JEPA2 resize to height 160, not a 256² square stretch).
- Student→teacher resampling, Gram computed at the teacher's grid resolution.
Pure Gram loss: L = L_FM + 0.25 · L_Gram, margin 0.1, constant LR 1e-4, 3000 steps/expert.
Usage (sketch)
# load the low-noise LoRA onto transformer_2 and the high-noise LoRA onto transformer
pipe.transformer_2.load_lora_adapter("low_noise_expert/pytorch_lora_weights.safetensors")
pipe.transformer.load_lora_adapter("high_noise_expert/pytorch_lora_weights.safetensors")
# then run I2V as usual (432×768, 121 frames @24fps in our eval)
Teacher (V-JEPA2) and the projector are training-only and not needed at inference — only the base model + these two LoRAs.
- Downloads last month
- -
Model tree for yrlyrl/wan2.2-i2v-a14b-physalign-lora
Base model
Wan-AI/Wan2.2-I2V-A14B