docs: head-to-head vs Wan2.2-Animate, with clips

.gitattributes

@@ -54,3 +54,5 @@ examples/media/swap-wushu-animals.mp4 filter=lfs diff=lfs merge=lfs -text
 examples/media/swap-wushu-robots.mp4 filter=lfs diff=lfs merge=lfs -text
 examples/media/hero-corgi.mp4 filter=lfs diff=lfs merge=lfs -text
 examples/media/hero-duo.mp4 filter=lfs diff=lfs merge=lfs -text
+examples/media/compare-cosplay.mp4 filter=lfs diff=lfs merge=lfs -text
+examples/media/compare-gamechar.mp4 filter=lfs diff=lfs merge=lfs -text

README.md

@@ -11,30 +11,6 @@ tags:
   - video-to-video
   - distillation
   - dmd
-extra_gated_prompt: >-
-  Viggle-Animate is a Model Derivative of MiniMax H3 and is distributed under the
-  MiniMax H3 Community License Agreement, not an open-source license. Two terms in
-  particular are easy to miss.
-
-
-  **Territory.** The Agreement grants rights worldwide EXCEPT in the European Union,
-  the United Kingdom, the Republic of Korea and the United States of America. In
-  those Excluded Territories it grants you nothing, and Section V.4 forbids using,
-  reproducing, modifying, distributing or displaying the Works or their Outputs
-  there. If you are in an Excluded Territory, contact MiniMax at api@minimax.io for
-  a separate license before downloading.
-
-
-  **Acceptable use.** This model replaces the person in a video with whoever you paint
-  into its first frame. Do not use it on anyone who has not consented, and read the
-  Acceptable Use Policy in Exhibit A of the Agreement.
-extra_gated_fields:
-  Name: text
-  Organization: text
-  Country: country
-  I am not located in the EU, UK, South Korea or the USA, or I hold a separate license from MiniMax: checkbox
-  I have read and agree to the MiniMax H3 Community License Agreement and its Acceptable Use Policy: checkbox
-  I will not use this model on people who have not consented: checkbox
 ---
 
 # Viggle-Animate
@@ -47,8 +23,37 @@ extra_gated_fields:
 
 Left panel is the driving video, right panel is this model at `--steps 4`. The only other input
 is that same clip's own first frame with the character painted over — same camera, same pose,
-same background, same light. No pose skeletons, no segmentation masks, no background plates, no
-text prompt.
+same background, same light.
+
+<div align="center">
+<div style="display:flex;gap:10px;flex-wrap:wrap;justify-content:center;margin:20px 0 6px;text-align:left">
+
+<div style="flex:1 1 170px;border:1px solid rgba(128,128,128,.35);border-radius:10px;padding:12px 14px">
+<div style="font-size:1.7em;font-weight:700;line-height:1.15">26 s</div>
+<b>per render</b><br>
+<small style="opacity:.7">124 frames at 24 fps on one B200 · 4 sampling steps, 3 forward passes</small>
+</div>
+
+<div style="flex:1 1 170px;border:1px solid rgba(128,128,128,.35);border-radius:10px;padding:12px 14px">
+<div style="font-size:1.7em;font-weight:700;line-height:1.15">2</div>
+<b>inputs</b><br>
+<small style="opacity:.7">a driving clip and its own first frame, repainted · no pose, mask, depth or prompt</small>
+</div>
+
+<div style="flex:1 1 170px;border:1px solid rgba(128,128,128,.35);border-radius:10px;padding:12px 14px">
+<div style="font-size:1.7em;font-weight:700;line-height:1.15">anything</div>
+<b>you can paint</b><br>
+<small style="opacity:.7">animals, machines, flat 2D, two at once — the model is never told what it is</small>
+</div>
+
+<div style="flex:1 1 170px;border:1px solid rgba(128,128,128,.35);border-radius:10px;padding:12px 14px">
+<div style="font-size:1.7em;font-weight:700;line-height:1.15">1</div>
+<b>take each</b><br>
+<small style="opacity:.7">every clip on this page is one paint, one render, <code>--steps 4 --seed 42</code> · no best-of-N</small>
+</div>
+
+</div>
+</div>
 
 <div align="center">
 <div style="display:inline-block;text-align:left;border:1px solid rgba(128,128,128,.35);border-radius:10px;padding:16px 18px;margin:6px 0">
@@ -102,44 +107,28 @@ replacement recipe — 20 steps, `sample_shift 5.0`, `--refert_num 1 --replace_f
 | Parameters | 33.1 B | 17.3 B |
 
 **6.1× faster per render, 10.3× on sampling alone**, on a model with 1.9× the parameters. Wan's
-preprocessing pass is not counted in its 160 s. The win is step distillation, not a faster
-backbone: per forward pass we are 4.5 s to Wan's 3.5 s.
+preprocessing pass is not counted in its 160 s.
 
 Four panels each: **painted reference · driving video · this model · Wan2.2-Animate-14B.**
 
 <video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/compare-prop.mp4"></video>
 
-A light-up toy held in both hands. Wan renders it as a featureless red blob, then as a blown-out
-white rectangle for the rest of the clip.
-
 <video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/compare-costume.mp4"></video>
 
-A white bow headwrap, floral-print leggings and navy slippers, driven by a bearded man. Wan flattens
-the leggings to plain white, drops the bow and the slippers, and lets the driver's beard through.
+<video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/compare-cosplay.mp4"></video>
 
-<video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/compare-style.mp4"></video>
+<video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/compare-gamechar.mp4"></video>
 
-A flat 2D anime character. Wan drifts toward a semi-realistic face and recolours the outfit.
-
-The two pipelines carry appearance very differently. In replacement mode Wan regenerates everything
-inside a per-frame mask over the driving performer, and what it has to rebuild that region from is
-one CLIP embedding plus one VAE latent of the character image, against a pose-skeleton video, 512×512
-face crops through a motion encoder, and a background plate with the region painted out — four of the
-five conditioning streams derived from the driving clip. At the documented `--w_len 1 --h_len 1` the
-painted-out region is the performer's whole bounding box, so a held prop sits inside it and is not
-represented in any of those streams. We condition on the painted frame and the driving video, with no
-pose extraction, no segmentation, no background plate and no prompt.
+<video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/compare-style.mp4"></video>
 
-These three are selected. Across the 24 clips we ran head to head, Wan came out ahead on one — a
-tight talking-head shot, where its explicit face channel tracks expression that we render flat. That
-is the same weakness listed under Limitations below. Roughly six were a wash.
+Five selected from 24 clips run head to head. Wan came out ahead on one — a tight talking-head shot,
+where its explicit face channel tracks expression that we render flat; roughly six were a wash.
 
 ## What you can paint in
 
 The model is never told what the new character is — it sees one edited frame and carries it through
 five seconds of motion it did not choreograph. So the question is how far from a person that
-character can get. Three panels each: **painted reference · driving video · this model.** All ten
-are single attempts at both stages: one paint, one render, `--steps 4 --seed 42`, no best-of-N.
+character can get. Three panels each: **painted reference · driving video · this model.**
 
 **Animals.** Ears, eye patches and flippers move on limbs the driving clip does not have — the paint
 places them, the render animates them as if they were arms and a head.
@@ -178,12 +167,10 @@ Two parts, both derived from [`MiniMaxAI/MiniMax-H3`](https://huggingface.co/Min
 | `transformer/` | 33.1 B, bf16, 14 shards. A **full finetune** of the base `transformer_ref` on a character-replacement objective |
 | `lora/` | rank 128 over 302 linear layers, 2.5 GB. A **DMD2-distilled** delta on that finetune — this is what turns 30 sampling steps into 4 |
 
-The LoRA is a delta on the *finetuned* transformer. Loading it onto stock `transformer_ref`
-produces garbage. There is no text prompt: conditioning is a frozen embedding computed once
-([`assets/fixed_prompt.txt`](assets/fixed_prompt.txt)), so the text encoder never loads. All of
-the art direction happens in the frame you paint.
+The LoRA is a delta on the *finetuned* transformer — loading it onto stock `transformer_ref` produces
+garbage. The frozen conditioning embedding is [`assets/fixed_prompt.txt`](assets/fixed_prompt.txt).
 
-## Install
+## Quickstart
 
 This repository ships only the transformer and the LoRA — the VAE, audio VAE and schedulers load
 from your own copy of the base model. Inference touches 11 GB of its 269 GB:
@@ -196,6 +183,10 @@ hf download MiniMaxAI/MiniMax-H3 --local-dir ./MiniMax-H3 \
 hf download Viggle/Viggle-Animate --local-dir ./Viggle-Animate
 
 pip install torch "git+https://github.com/huggingface/diffusers@d6726f3" av
+
+python Viggle-Animate/inference/sample.py \
+  --model-dir ./MiniMax-H3 \
+  --cond driving.mp4 --ref repainted_first_frame.png --out swapped.mp4
 ```
 
 `d6726f3` is the tested `diffusers` commit; the upstream `minimax_h3` modular pipeline is enough,
@@ -205,14 +196,6 @@ no fork or patch. The last `--include` is the clip [`examples/demo.sh`](examples
 124-frame render peaks at 80.1 GiB allocated. Use a card with ≥ 96 GB, or pass `--offload` to
 stream blocks from CPU (~12 GB resident, much slower).
 
-## Run
-
-```bash
-python Viggle-Animate/inference/sample.py \
-  --model-dir ./MiniMax-H3 \
-  --cond driving.mp4 --ref repainted_first_frame.png --out swapped.mp4
-```
-
 Defaults are the evaluated configuration: `--steps 4 --flow-shift 3 --num-frames 124` (≈ 5.2 s at
 24 fps) `--seed 42`. Output geometry follows the driving clip and must be a multiple of 32 on both
 axes. Weights load in ~21 s, once per process.
@@ -234,18 +217,15 @@ in **40–60 s**. That quantization path is not shipped here.
 - **Everything you want must survive the paint step.** The model will not invent a costume detail,
   a prop or a face the reference does not show.
 - **More steps is not better.** `--steps 4` is the operating point, not a degraded `--steps 12`. It
-  renders sharper than its own teacher — Laplacian variance 16–50 % higher — which reads as detail
-  sometimes and over-sharpening other times, and worsens with more steps.
-- **The reference defines only the views it contains.** A character painted from behind has no
-  front, and when the driving clip rotates to reveal one the model fills it from its own prior
-  rather than from your character. On a figure skater who spins to face camera, a swapped-in cat is
-  a cat while her back is turned, a generic woman at the moment she faces us, and a cat again when
-  she turns away. Prefer driving clips that keep one side to camera, or paint a frame that shows the
-  view the clip will eventually reach.
-- **A silhouette far from the driver's survives as colour, not as geometry.** A LEGO minifigure on
-  that same skating clip kept its palette and its yellow claw hands, but the moulded plastic body
-  reverted to human anatomy. The airliner above works because the paint bound its wings to the arms
-  — geometry the *driving pose* can still explain.
+  renders sharper than its teacher (Laplacian variance 16–50 % higher) — detail sometimes,
+  over-sharpening other times, worse the further you push it.
+- **The paint carries appearance; the driving video carries geometry.** Two consequences. A view the
+  reference never shows is filled from the model's prior: a cat painted from behind is a cat while
+  the skater's back is turned, a generic woman at the frame she faces camera, a cat again after. And
+  a silhouette the driving pose cannot explain survives as colour only — a LEGO minifigure kept its
+  palette and yellow claw hands but reverted to human anatomy, while the airliner held because the
+  paint bound its wings to the arms. Prefer clips that keep one side to camera, and bind new limbs
+  to real ones.
 - **No facial-landmark channel**, so tight close-up dialogue is the weakest case — Wan drives the
   face from explicit landmarks and is visibly more expressive there.
 - **Out of distribution is not the same as broken.** Training was one subject filling a portrait
@@ -257,13 +237,12 @@ in **40–60 s**. That quantization path is not shipped here.
 
 ## License
 
-- **Code** — [`inference/`](inference) and [`examples/`](examples) are Apache 2.0
-  ([`LICENSE-CODE`](LICENSE-CODE)).
-- **Weights** — `transformer/` and `lora/` are Model Derivatives of MiniMax H3 and stay under the
-  [MiniMax H3 Community License](LICENSE), which covers distillation explicitly. It is not an
-  open-source license and grants **no rights in the EU, UK, South Korea or the USA**. Read it
-  before redistributing or shipping a product. Our changes are in
-  [`MODIFICATIONS.md`](MODIFICATIONS.md).
+The weights are a Model Derivative of MiniMax H3, so the
+[MiniMax H3 Community License](LICENSE) applies to them — read it before you redistribute them or
+ship a product on them. Our changes are listed in [`MODIFICATIONS.md`](MODIFICATIONS.md).
+
+The code in [`inference/`](inference) and [`examples/`](examples) is Apache 2.0
+([`LICENSE-CODE`](LICENSE-CODE)).
 
 ## Intended use
 

examples/media/compare-cosplay.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6bd63ca750de511cbc6391664def22a9671f79d6885729c8d20bc92978fc8843
+size 1099913

examples/media/compare-gamechar.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:69e34a0ec75f588dfa3c2cdf882fab5bb7107e357e64ceb0d1881d8dbc23452c
+size 858158