Viggle-Animate / MODIFICATIONS.md
yycc's picture
docs: head-to-head vs Wan2.2-Animate, with clips
9dfbf10 verified
|
Raw
History Blame Contribute Delete
4.66 kB
# Modified files
Section III.2 of the MiniMax H3 Community License Agreement requires that modified
files carry a prominent notice saying so. This file is that notice.
Everything below is derived from
[`MiniMaxAI/MiniMax-H3`](https://huggingface.co/MiniMaxAI/MiniMax-H3).
## `transformer/` β€” modified
**Every weight file in `transformer/` has been modified.** It started as the base
model's `transformer_ref/` (the `ref2va` transformer, 33.1 B parameters) and every
parameter was updated by a full finetune on a character-replacement objective. The
architecture, `config.json` and tensor names are unchanged, so it is a drop-in
replacement for the base `transformer_ref/`; the numbers in it are not the base
model's numbers.
The file layout also differs: the finetune was written as one 61.7 GiB safetensors
file and re-sharded here into 14 parts, because HuggingFace rejects single files
above 50 GB. The 638 tensors and their contents are unchanged by that re-sharding.
## `lora/pytorch_lora_weights.safetensors` β€” new
Not a MiniMax file. A rank-128 LoRA over 302 linear layers of `transformer/`,
trained by us with DMD2 distillation. It is a delta on the finetuned transformer
above, not on the base model β€” loading it onto stock `transformer_ref/` produces
garbage.
## `assets/fixed_embed_fwd_anyframe.pt` β€” new
Not a MiniMax file. A frozen 362 Γ— 5120 text-conditioning tensor we computed once
with the base model's own text encoder, so that inference never has to load
Qwen3-VL. It is an *output* of the base model's encoder in the sense of Section
I.12, computed from the prompt in `assets/fixed_prompt.txt`.
## `assets/fixed_prompt.txt` β€” new
Not a MiniMax file. The prompt text the tensor above was computed from, included so
that what conditions every render is readable rather than opaque.
## `inference/sample.py`, `examples/demo.sh` β€” new
Not MiniMax files. Written by us against the public `diffusers` API.
## `LICENSE-CODE`, `NOTICE` β€” new
Not MiniMax files. `LICENSE-CODE` is the Apache 2.0 text, and it covers `inference/` and
`examples/` only β€” each file there carries an `SPDX-License-Identifier: Apache-2.0` header.
`NOTICE` records that the weights are *not* Apache 2.0. `LICENSE` is the MiniMax H3
Community License Agreement itself, included unmodified as Section III.1 requires.
## `examples/media/` β€” new
Two kinds of media here, with different provenance.
**The demo** β€” `reference.png`, `output.mp4`, `before-after.png` β€” derives from
`assets/ref2va.mp4`, a video MiniMax published with the base model. That clip is itself a
MiniMax H3 generation rather than camera footage. All three files are a 512 Γ— 768 portrait
crop of it (`crop=512:768:389:0`, no scaling):
- `reference.png` β€” the crop's first frame with the young man repainted as an invented
elderly woman. Produced with OpenAI's `gpt-image-2`; the character is fictional and is
not a real person or an existing property.
- `output.mp4` β€” that reference propagated across 124 frames by this model.
- `before-after.png` β€” frames from the driving crop above frames from the output.
The driving clip itself is **not** bundled. `examples/demo.sh` rebuilds it, with the
documented crop, from your own copy of the base model.
**The comparison clips** β€” `compare-prop.mp4`, `compare-costume.mp4`, `compare-style.mp4` β€” are a
different matter. Their driving videos are
real filmed footage that we hold the rights to, and they are among the clips this model was
evaluated against. Each file is a four-panel stack: painted reference, driving video, this model,
Wan2.2-Animate-14B. The Wan2.2-Animate panels were rendered by us from the official
[`Wan-AI/Wan2.2-Animate-14B`](https://huggingface.co/Wan-AI/Wan2.2-Animate-14B) weights and code,
unmodified, at the replacement-mode settings its own README documents (20 steps, `sample_shift 5.0`,
`--refert_num 1 --replace_flag --use_relighting_lora`, preprocessing at `--w_len 1 --h_len 1`) β€”
that panel is Wan's output, not ours.
Wan's `generate.py` hardcodes a 30 fps output timebase regardless of the source, so its raw files
claim 4.10 s for motion that is 24 fps. The panels are retimed (`setpts`), not resampled, so no
frames are dropped and both models play at the same speed. Every panel is letterboxed into the
driving clip's own geometry; nothing is stretched.
## Not redistributed here
The VAE, audio VAE, schedulers, text encoder, tokenizer and processor are **not**
included in this repository and are not modified. They are loaded at runtime from
your own copy of `MiniMaxAI/MiniMax-H3`, which you must download separately and
under its own license terms.