Viggle-Animate / MODIFICATIONS.md
yycc's picture
docs: head-to-head vs Wan2.2-Animate, with clips
9dfbf10 verified
|
Raw
History Blame Contribute Delete
4.66 kB

Modified files

Section III.2 of the MiniMax H3 Community License Agreement requires that modified files carry a prominent notice saying so. This file is that notice.

Everything below is derived from MiniMaxAI/MiniMax-H3.

transformer/ β€” modified

Every weight file in transformer/ has been modified. It started as the base model's transformer_ref/ (the ref2va transformer, 33.1 B parameters) and every parameter was updated by a full finetune on a character-replacement objective. The architecture, config.json and tensor names are unchanged, so it is a drop-in replacement for the base transformer_ref/; the numbers in it are not the base model's numbers.

The file layout also differs: the finetune was written as one 61.7 GiB safetensors file and re-sharded here into 14 parts, because HuggingFace rejects single files above 50 GB. The 638 tensors and their contents are unchanged by that re-sharding.

lora/pytorch_lora_weights.safetensors β€” new

Not a MiniMax file. A rank-128 LoRA over 302 linear layers of transformer/, trained by us with DMD2 distillation. It is a delta on the finetuned transformer above, not on the base model β€” loading it onto stock transformer_ref/ produces garbage.

assets/fixed_embed_fwd_anyframe.pt β€” new

Not a MiniMax file. A frozen 362 Γ— 5120 text-conditioning tensor we computed once with the base model's own text encoder, so that inference never has to load Qwen3-VL. It is an output of the base model's encoder in the sense of Section I.12, computed from the prompt in assets/fixed_prompt.txt.

assets/fixed_prompt.txt β€” new

Not a MiniMax file. The prompt text the tensor above was computed from, included so that what conditions every render is readable rather than opaque.

inference/sample.py, examples/demo.sh β€” new

Not MiniMax files. Written by us against the public diffusers API.

LICENSE-CODE, NOTICE β€” new

Not MiniMax files. LICENSE-CODE is the Apache 2.0 text, and it covers inference/ and examples/ only β€” each file there carries an SPDX-License-Identifier: Apache-2.0 header. NOTICE records that the weights are not Apache 2.0. LICENSE is the MiniMax H3 Community License Agreement itself, included unmodified as Section III.1 requires.

examples/media/ β€” new

Two kinds of media here, with different provenance.

The demo β€” reference.png, output.mp4, before-after.png β€” derives from assets/ref2va.mp4, a video MiniMax published with the base model. That clip is itself a MiniMax H3 generation rather than camera footage. All three files are a 512 Γ— 768 portrait crop of it (crop=512:768:389:0, no scaling):

  • reference.png β€” the crop's first frame with the young man repainted as an invented elderly woman. Produced with OpenAI's gpt-image-2; the character is fictional and is not a real person or an existing property.
  • output.mp4 β€” that reference propagated across 124 frames by this model.
  • before-after.png β€” frames from the driving crop above frames from the output.

The driving clip itself is not bundled. examples/demo.sh rebuilds it, with the documented crop, from your own copy of the base model.

The comparison clips β€” compare-prop.mp4, compare-costume.mp4, compare-style.mp4 β€” are a different matter. Their driving videos are real filmed footage that we hold the rights to, and they are among the clips this model was evaluated against. Each file is a four-panel stack: painted reference, driving video, this model, Wan2.2-Animate-14B. The Wan2.2-Animate panels were rendered by us from the official Wan-AI/Wan2.2-Animate-14B weights and code, unmodified, at the replacement-mode settings its own README documents (20 steps, sample_shift 5.0, --refert_num 1 --replace_flag --use_relighting_lora, preprocessing at --w_len 1 --h_len 1) β€” that panel is Wan's output, not ours.

Wan's generate.py hardcodes a 30 fps output timebase regardless of the source, so its raw files claim 4.10 s for motion that is 24 fps. The panels are retimed (setpts), not resampled, so no frames are dropped and both models play at the same speed. Every panel is letterboxed into the driving clip's own geometry; nothing is stretched.

Not redistributed here

The VAE, audio VAE, schedulers, text encoder, tokenizer and processor are not included in this repository and are not modified. They are loaded at runtime from your own copy of MiniMaxAI/MiniMax-H3, which you must download separately and under its own license terms.