---
license: other
license_name: minimax-h3-community-license
license_link: LICENSE
base_model: MiniMaxAI/MiniMax-H3
pipeline_tag: video-to-video
tags:
- video-editing
- character-replacement
- video-to-video
- distillation
- dmd
---
# Viggle-Animate
### Character Replacement in Video from a Single Repainted Frame
**[Try the demo](https://huggingface.co/spaces/Viggle/viggle-animate)** ·
**[viggle.ai/h3](https://viggle.ai/h3)** ·
Built on **[MiniMaxAI/MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3)**
**Viggle-Animate replaces the character in a video with whatever you paint into one of its own
frames** — motion, camera and timing untouched. You prepare that one frame in an image editor;
from there the video stage runs no pose estimator, segmenter, face tracker or text encoder. Two
inputs, three forward passes, 26 seconds a shot on one GPU.
**It is strongest where replacement is hardest: fast motion, and pose transfer accurate enough to
follow it.** Whipping heads, full kicks, jumps — tracked frame for frame, not smeared through.
## Abstract
Controlled character replacement is usually built on intermediate representations — pose skeletons,
segmentation masks, background plates, face crops. Each needs its own extractor, and each extractor
is another model to run and another place to lose information. Recent work drops the skeleton but
keeps a mask channel. **Viggle-Animate uses neither.** Its two inputs are a driving video and one of
that video's own frames with the character repainted, and its only task is to propagate that edit
across the shot.
Because the reference is a frame of the clip, its pose, camera, framing and lighting already agree
with the footage, and nothing downstream has to align them again. The model is never told what the
new character is: no class, no identity encoder, and no user-provided text prompt.
**Viggle-Animate is a 33.1 B full finetune of MiniMax-H3's `ref2va` transformer, jointly distilled
with DMD to three forward passes.** In a matched comparison on the same machine and B200 GPU, using
the same source videos, output resolution and frame count, it renders 124 frames in 26 s, 6.1×
faster per clip than Wan2.2-Animate-14B.
## Method
Character replacement asks two questions at once: *what does the new character look like*, and *how
does it move through this shot*. Systems that condition on a standalone character photograph must
answer both, and reconciling a photograph with footage it was never part of is what the scaffolding
exists for.
State-of-the-art image models have finished that job. Give `gpt-image` a frame and an instruction
and it replaces the character while following the prompt exactly — transferring the pose, matching
the lighting, preserving the background. The hard reconciliation is already solved, once, on one
image. This model is the second half of that pipeline, not the whole of it.
Appearance enters only through the repainted frame; geometry enters only through the driving video.
**The text encoder is never loaded.** Conditioning is one frozen embedding shipped with the weights
([`assets/fixed_prompt.txt`](assets/fixed_prompt.txt)), identical for every render.
Left panel is the driving video, right panel is this model:
**It is fast twice over.** Once the repainted frame exists there is nothing else to run — no pose
estimator, no segmenter, no face tracker, no text encoder. And the sampler is distilled, so a
finished clip is three forward passes rather than thirty. The two compound: one model, one GPU,
and no orchestration to get wrong.
The distillation is **joint, across two teachers split by noise level.** Our finetune supervises the
high-noise end of the schedule, where the replacement itself is decided — it is the model that gets
the swap right. The original MiniMax-H3 supervises the low-noise end, where detail and texture are
decided — it is the model with the better image quality. Distilling each end against the teacher that
owns it keeps both properties in one student, instead of inheriting the finetune's visual
regressions along with its replacement ability.
**It generalizes past humans**, because nothing in the loop assumes one. A pose skeleton has a neck
and two arms; a mask has a person-shaped hole. We have neither, so the model holds no representation
that a character must be a person. What it can animate is bounded by what you can paint.
## Efficiency
26 s
per render
124 frames at 24 fps, 480×832, a single B200
3
forward passes
--steps 4 names four sigma boundaries, so three passes
2
inputs
a clip, and one of its own frames repainted
0
other models
in the video stage — no pose estimator, segmenter, face tracker or text encoder
One B200, 480×832, 124 frames at 24 fps, bf16, no compile, no offload. Wan ran its documented
replacement recipe — 20 steps, `sample_shift 5.0`, `--refert_num 1 --replace_flag
--use_relighting_lora`, `--w_len 1 --h_len 1` — after its own preprocessing pass.
| | Viggle-Animate | Wan2.2-Animate-14B |
|---|---|---|
| Inputs | driving video + one repainted frame | driving video + character image, then a **preprocessing pass** producing pose, face, mask and background tracks |
| Render, after weights load | **26 s** | 160 s |
| — of which sampling | **13.6 s** | 140 s |
| Forward passes | **3** | 40 (20 steps × 2 chunks) |
| Parameters | 33.1 B | 17.3 B |
**6.1× faster per render, 10.3× on sampling alone.** Wan's preprocessing pass is not counted in
its 160 s.
### Qualitative comparison
Four panels each: **painted reference · driving video · this model · Wan2.2-Animate-14B**, the last
at its documented replacement settings. The gap is widest under fast motion: where the comparison
smears, this model stays sharp and lands the pose on the right frame.
## Generalization
The model is never told what it is animating, so how far the character can get from a person is an
empirical question rather than a list of supported categories. Three panels each: **painted
reference · driving video · this model.** Every clip below is one paint and one render at the
shipped defaults, `--seed 42` — no best-of-N.
**Animals.** Ears, eye patches and flippers move on limbs the driving clip does not have — the paint
places them, the render animates them as if they had always been arms and a head.
**Not humanoid.** The airliner is the hardest case we have: the paint binds wings to arms and landing
gear to legs, and the model's job is to keep that binding for 124 frames. The robot has to relight
specular metal as it turns.
**Stylized.** The clay figure holds its style boundary for the whole clip.
**More than one character.** The work moves into the paint prompt, which has to bind each one to a
position — "the one on the left". The last clip is a wide arena shot, each figure a few dozen pixels
tall.
## Limitations
**It inherits the image edit.** What the paint does not show, the model will not add, and where
paint and video disagree the video wins. Appearance comes from the paint but shape comes from the
driving pose: a LEGO minifigure kept its palette and yellow claw hands, yet reverted to human
anatomy — the airliner held because the paint tied its wings to real arms.
**Lip-sync is weak.** Mouth shapes do not track speech closely in close-ups. Identity and expression
hold; it is the sync that lags, and we believe that is a training-data limit rather than anything
structural.
**Complex scenes are harder than single subjects.** Several characters at once, close interaction
between them, and shots that cut are all cases where quality drops off — enough that we would not
call them solved.
**We are training a substantially better model right now**, aimed squarely at these three. This
release is the version we can ship today, not the ceiling.
## What this repository contains
Two parts, both derived from [`MiniMaxAI/MiniMax-H3`](https://huggingface.co/MiniMaxAI/MiniMax-H3)'s
`ref2va` transformer:
| | |
|---|---|
| `transformer/` | 33.1 B, bf16, 14 shards. A **full finetune** of the base `transformer_ref` on a character-replacement objective |
| `lora/` | rank 128 over 302 linear layers, 2.5 GB. A **DMD2-distilled** delta on that finetune — this is what collapses the sampler to three forward passes |
The LoRA is a delta on the *finetuned* transformer — loading it onto stock `transformer_ref` produces
garbage.
## Quickstart
This repository ships only the transformer and the LoRA — the VAE, audio VAE and schedulers load
from your own copy of the base model. Inference touches 11 GB of its 269 GB:
```bash
hf download MiniMaxAI/MiniMax-H3 --local-dir ./MiniMax-H3 \
--include "modular_model_index.json" "vae/*" "audio_vae/*" \
"scheduler/*" "audio_scheduler/*" "assets/ref2va.mp4"
hf download Viggle/Viggle-Animate --local-dir ./Viggle-Animate
pip install torch "git+https://github.com/huggingface/diffusers@d6726f3" av
python Viggle-Animate/inference/sample.py \
--model-dir ./MiniMax-H3 \
--cond driving.mp4 --ref ref.png --out swapped.mp4
```
`d6726f3` is the tested `diffusers` commit; the upstream `minimax_h3` modular pipeline is enough,
no fork or patch. The last `--include` is the clip [`examples/demo.sh`](examples/demo.sh) needs.
**One 80 GB card is not enough at bf16** — the transformer is 62 GiB resident and a 480×832 /
124-frame render peaks at 80.1 GiB allocated. Use a card with ≥ 96 GB, or pass `--offload` to
stream blocks from CPU (~12 GB resident, much slower).
**It also runs quantized on consumer hardware.** We deploy it on a single **RTX 5090 (32 GB)**:
NVFP4 weights — 4.5 bits/param, dispatching to the real sm_120 cutlass block-scaled kernel, 2.70×
bf16 per compiled linear — plus a low-rank `adaln_proj` and `torch.compile`. Quantization alone is
not enough for 32 GB: 13.0 B of the 33.1 B parameters sit in `adaln_proj`, which the linear-layer
quantizer does not touch. That deployment path is not shipped in this repository.
Defaults are the evaluated configuration: `--steps 4 --flow-shift 3 --num-frames 124` (≈ 5.2 s at
24 fps) `--seed 42`. Output geometry follows the driving clip and must be a multiple of 32 on both
axes. Weights load in ~21 s, once per process. **Four steps is the operating point, not a shortcut**
— the distilled model already renders sharper than its teacher, and raising the step count tips that
into over-sharpening. `--steps 4` is four sigma boundaries and therefore three forward passes. The
driving clip's audio is dropped at input; the model emits its own track, and the fixed prompt asks
for silence.
Pull a frame with `ffmpeg -ss 1.5 -i driving.mp4 -frames:v 1 ref.png` and edit it at the same
resolution. **It does not have to be the first frame.** The still is passed to the model with no
frame index, so nothing downstream knows where in the clip it came from — pick whichever frame shows
the character most clearly, front-on and unoccluded. Name the change, and pin down what must *not*
change: pose, hands, props, framing, background, light. An editor that quietly reframes the shot will
fight the driving motion. Prefer clips that keep one side to camera, and bind any new limb to a real
one.
[`examples/demo.sh`](examples/demo.sh) runs a swap end to end on a clip that ships with the base
model, so it needs no media from you — and it is the fastest way to check that the LoRA loaded.
## Citation
```bibtex
@misc{viggle2026animate,
title = {Viggle-Animate: Character Replacement in Video from a Single Repainted Frame},
author = {Viggle Research},
year = {2026},
url = {https://huggingface.co/Viggle/Viggle-Animate}
}
```
## License
The weights are a Model Derivative of MiniMax H3, so the
[MiniMax H3 Community License](LICENSE) applies to them — read it before you redistribute them or
ship a product on them. Our changes are listed in [`MODIFICATIONS.md`](MODIFICATIONS.md).
The code in [`inference/`](inference) and [`examples/`](examples) is Apache 2.0
([`LICENSE-CODE`](LICENSE-CODE)).
Music in the teaser at the top of this page: "Electrodoodle" by Kevin MacLeod
([incompetech.com](https://incompetech.com)), licensed under
[Creative Commons: By Attribution 4.0](http://creativecommons.org/licenses/by/4.0/).
## Intended use
This model exists to put a consenting performer into footage they did not shoot, and it will just
as readily put someone into footage they never agreed to appear in. Note where that decision is
made: **the identity comes from the frame you paint**, so an image editor's safeguards are upstream
of this model and none of them are in it. It cannot verify identity or consent. Do not run it on
people who have not agreed to it, label what you generate as AI-generated, and see Section V.5 of
the Agreement if you offer this as a service.
Powered by MiniMax H3.