README.md

---
license: other
license_name: h3-keyframe-animation-adapter-license-1.0.0
license_link: LICENSE
base_model: MiniMaxAI/MiniMax-H3
base_model_relation: adapter
library_name: diffusers
pipeline_tag: image-to-video
tags:
  - lora
  - minimax-h3
  - animation
  - keyframe-animation
  - inbetweening
  - hand-drawn
  - video
---

# H3 Keyframe Animation Adapters

Three LoRA adapters that teach [MiniMax H3](https://github.com/MiniMax-AI) to move a
drawing the way a hand-drawn animator moves it — a consistent cadence held across
styles, rather than the drift between vector interpolation, digital easing, and
limited animation that the base model slides through unprompted.

Companion to the paper **[Animating on Twos](https://alvdansen.github.io/animating-on-twos/)**
(Carlson & Bielec, Alvdansen Labs, August 2026).

---

## The three adapters

Same architecture, same training schedule. What separates them is the **conditioning
contract**: how many reference drawings each takes, and what the distance between those
drawings is asked to mean. In every clip below, the **left half is the untrained baseline**
and the **right half is the adapter at step 12,000** — same inputs, same seed.

### hero — 1 reference

*Draw the natural next hero key.*

![hero: untrained fl2va on the left, step 12,000 on the right](https://alvdansen.github.io/animating-on-twos/figures/hero_000_fl_vs_step12000.gif)

Conditioned on the **current hero key**, and nothing else. There is deliberately no second
reference: where the action goes next is the question being asked, so showing the model a
destination would be showing it the answer.

### tween — 2 references

*Draw the next inbetween, advancing the motion one small step.*

![tween: untrained fl2va on the left, step 12,000 on the right](https://alvdansen.github.io/animating-on-twos/figures/tween_000_fl_vs_step12000.gif)

Conditioned on a **rolling current frame** plus the beat's **distant end extreme, held
fixed** for the whole chain. That far reference is *context*, not a target — it tells the
next inbetween which way to lean. Chain it by feeding each result back as the near
reference while the far one never moves.

### sequence — 2 references

*Surface a short, self-contained held sequence.*

![sequence: untrained fl2va on the left, step 12,000 on the right](https://alvdansen.github.io/animating-on-twos/figures/seq_000_fl_vs_step12000_seed7.gif)

Conditioned on the **window's first drawing** and the **window's own natural end**. Unlike
the other two, this adapter is not teaching the model a new mapping — hand-drawn held
animation already exists somewhere in the base distribution, and its job is to surface that
more consistently. It converges early and changes little across late checkpoints.

---

## Quick start

A ready-to-load ComfyUI graph, the reference it runs on, and the prompt that drives it.

| File | |
| --- | --- |
| [`workflows/h3_seq_r2v.json`](workflows/h3_seq_r2v.json) | The graph. Drag it onto the ComfyUI canvas. |
| [`examples/quickstart/reference.png`](examples/quickstart/reference.png) | The single reference drawing it conditions on. |
| [`examples/quickstart/prompt.txt`](examples/quickstart/prompt.txt) | The caption, in the dialect below. |

Put the reference in `ComfyUI/input/` and the adapter in `ComfyUI/models/loras/`, then load
the graph. It is laid out in labelled stages — models, references, prompt, sampling — and
carries its own notes on the settings that matter.

The graph ships configured for **one reference** (the second image slot is bypassed), which
is the *hero* contract. For *tween* or *sequence*, un-bypass the second `LoadImage` and swap
the adapter in the `LoraLoaderModelOnly` node.

---

## ⚠ Load the converted files

These adapters were trained against the diffusers module tree, which exposes separate
`to_q` / `to_k` / `to_v` projections. MiniMax H3 serves a single **fused `qkv_proj`**.

A raw PEFT adapter loaded against H3 matches **zero keys, does nothing, and warns about
nothing.** You get a clean render that the adapter never touched.

The `.safetensors` files in this repo are **already converted** — 400 tensors, 100
`qkv_proj` keys, no `to_q`/`to_k`/`to_v` — so load them as they are. If you convert
another checkpoint yourself, use the included `h3_lora_to_comfy.py` and check the result
by confirming its key names contain `qkv_proj`.

---

## ⚠ Not on Turbo variants

Run these on the **full H3 model**, not a Turbo or otherwise step-distilled repack of it.

These adapters exist to hold a hand-drawn line and a deliberate cadence, and the
step-distilled variants do not preserve the linework. Whatever you gain in speed you lose
in exactly the thing the adapters were trained to protect.

Thanks to [@shadowworksltd](https://huggingface.co/shadowworksltd) for raising this.

---

## Contents

| File | What it is |
| --- | --- |
| `adapters/h3_hero_step12000.safetensors` | *hero* adapter, final checkpoint, converted |
| `adapters/h3_tween_step12000.safetensors` | *tween* adapter, final checkpoint, converted |
| `adapters/h3_seq_step12000.safetensors` | *sequence* adapter, final checkpoint, converted |
| `h3_lora_to_comfy.py` | Raw PEFT → fused-QKV converter |
| `examples/` | A small number of renders, selected by the author |
| `LICENSE`, `NOTICE` | Terms, and the base-model obligations we cannot waive |

Each adapter is rank 64 / alpha 64, F32, ~1.88 GB, trained for 12,000 steps.

---

## The conditioning contract

| Adapter | The act it supports | Conditioning | What the second reference's distance means |
| --- | --- | --- | --- |
| **hero** | Draw the **natural next hero key** | **1 reference**, the current hero key | *No second reference.* Where the action goes next is the question being asked; showing the model a destination would be showing it the answer. |
| **tween** | Draw the **next inbetween**, advancing the motion one small step | **2 references**: a rolling current frame, plus the beat's **distant end extreme, held fixed** for the whole chain | *Far, deliberately.* The end extreme provides **context**: it tells the next inbetween which way to lean, without being the thing to interpolate to. |
| **sequence** | Surface a short, self-contained **held sequence** already latent in the model | **2 references**: the window's first drawing and the window's **own natural end** | *One window* to fill: a complete keyframed clip on twos. |

*hero* and *tween* **teach** the model a mapping it does not reliably have. *sequence*
works differently — hand-drawn-looking held animation already exists somewhere in the
base distribution, and that adapter's job is to **surface it more consistently**. Expect
*sequence* to converge early and change little across late checkpoints, where a teaching
adapter keeps moving.

All three drive the **reference-conditioning path** (N soft reference images), not the
first/last-frame path (hard endpoints).

---

## Inference settings

These are the settings that produced the paper's figures, read out of the render code
rather than recommended in the abstract. Everything not listed here is a ComfyUI default.

| | hero | tween | sequence |
| --- | --- | --- | --- |
| Base checkpoint | `minimax_h3_ref2va_int8_convrot` (**unpruned**, not Turbo) | same | same |
| Conditioning node | `MiniMaxH3ReferenceToVideo` | same | same |
| References | **1** | **2** | **2** |
| Reference keys | `ref_images.ref_image_0` | `…_0`, `…_1` | `…_0`, `…_1` |
| `ref_image_size` | `max` | `max` | `max` |
| LoRA loader | `LoraLoaderModelOnly` | same | same |
| LoRA strength | **1.0** | **1.0** | **1.0** |
| Sampler / scheduler | `euler` / `simple` | same | same |
| Steps | **30–100** | **30–100** | **30–100** |
| Guidance | **none** — `BasicGuider`, no CFG input exists | same | same |
| Sigma shift | video **12.0**, audio **3.0** | same | same |
| Denoise | 1.0 | 1.0 | 1.0 |
| Resolution | **1344 × 768** | same | same |
| Frames | **22** | **22** | **22** |
| Seed (paper figures) | 42 | 42 | 7 |
| Output | 24 fps | same | same |
| Text encoder | `qwen3vl_32b_minimax_h3_int8_convrot`, `CLIPLoader` type `minimax` | same | same |
| Video / audio VAE | `..._video_vae_fp16` / `..._audio_vae_fp32`, untiled | same | same |

**Notes that matter more than the numbers:**

- **Strength 1.0 is the trained strength.** Rank and alpha are both 64, so the applied
  scale is exactly 1.0. Do not walk it back at inference.
- **The adapters were trained against the unpruned reference partition**, which is what
  the settings above record. They also run on pruned repacks.
- **The LoRA goes on the model only.** `LoraLoaderModelOnly`, not `LoraLoader` — the
  Qwen3-VL text encoder takes no LoRA.
- **There is no negative prompt, and no CFG scale to set.** The checkpoints are
  CFG-distilled; the graph has no negative input at all.
- **Steps run 30–100.** The figures in the paper were sampled lower than that (25 for
  hero and tween, 50 for sequence) because those matched the training-time sample config
  — a sampling choice for the review grid, not a recommendation.
- **Legal clip lengths follow `frames = 17n + 5`**, so the shortest legal clip is 5.
  Every render above is 22 (`n = 1`), which for *sequence* is 11 drawings held two
  frames each. Note that *hero* and *tween* were **trained** on a 5-frame uniform freeze
  but **rendered** at 22 — the cache-encode length and the render length are different
  numbers and conflating them is a documented crash.

---

## The caption dialect

The caption is the inference interface. These are the portable shapes; fill the angle
brackets.

**hero / early tween — three sections.**
```
Create the next keyframe in the sequence. SCENE: <framing + subject-type + medium + props>. MOVEMENT: <what moves, how, where it lands>. STATIC: <what holds, ending with framing + ground>.
```
*hero*'s `MOVEMENT` carries a full **arc** — start-state, path, end-state — because the
model must bridge a whole beat.

**tween — five sections.**
```
… SCENE: … FIRST FRAME: <pose in the near reference>. TARGET END FRAME: <pose in the far reference, phrased as a destination, never comparative>. MOVEMENT: <one step only>, landing <progress>. STATIC: <what holds, ending with framing + ground>.
```
The trailing `landing <progress>` clause is the **step-size lever**, drawn from a fixed
vocabulary (fractional advances, plus *hold / anticipation / overshoot / arrival*).
`TARGET END FRAME` must **never** use comparative language ("unchanged", "still the
same") — the rolling reference changes under it at inference while it stays fixed, so
any comparison goes stale.

**sequence — alignment line first.**
```
Picture 1 aligns with the 0.00-second mark of the target video; Picture 2 aligns with the <computed>-second mark. Subject: <medium + subject-type + ground>, animated on twos. Action: <motion path in beat order, held drawings changing on a rhythm>. Camera: <angle/move + the reason for it>. Preserve: <every axis that must not drift>, and the on-twos rhythm.
```
Put "on twos" in `Subject` and honour it in `Action`.

### How the training captions were written

These describe the dialect the adapters were captioned in, so you can match it. They are
conventions from the training data, not tested constraints.

- **No negations.** The checkpoints are CFG-distilled, so there is no negative branch and
  every token is positive. Captions state the desired condition rather than excluding an
  unwanted one.
- **No character, show, or franchise names.** Identity comes in through the reference
  image; captions describe subject-type, medium, motion, and framing in general terms.
- **The medium string is byte-identical** across a shot, drawn from a small fixed set —
  `clean line on white`, `flat black-and-white animatic`, `flat cel colour on white`.
- **`Camera` carries a reason clause**, e.g. "a low three-quarter angle, so the silhouette
  stays readable", rather than a bare angle.
- **`Preserve` names each axis** that should not drift.
- **Craft vocabulary appears where it is literally true of the clip** — *drag, settle,
  follow-through, anticipation, weight shift, leading edge*, and *on twos*, which is used
  when the clip genuinely holds each drawing two frames.

---

## License

The adapters are released under the **H3 Keyframe Animation Adapter License 1.0.0**
(see `LICENSE`) — a PolyForm Small Business derivative. Source-available, not open
source.

**Free, with no need to ask**, for individuals at any income (personal, creative,
freelance, educational), researchers and educational institutions, nonprofits and
charities, and organizations that with their affiliates earned under **USD 2,000,000**
gross revenue last fiscal year **and** have raised under **USD 2,000,000** in total
capital. Crossing either threshold means a commercial license, with a 90-day grace
period — past use is not retroactively a violation. Contact **minta@promptcrafted.com**.

### The base model's license binds you directly

These adapters do nothing without MiniMax H3, which we do not own and cannot
sublicense. **You must obtain and accept your own grant from MiniMax.** Any access this
project's authors have is personal to them and does not transfer. That license carries
obligations we cannot waive, including **Excluded Territories** — which as read include
the European Union, the United Kingdom, the Republic of Korea, and the United States —
an attribution requirement on commercial products, a revenue threshold independent of
ours, and a prohibition on using outputs to train other models. Read it in full. `NOTICE`
has the detail.

The paper text and figures are CC BY 4.0 and the paper repo's code is MIT — a different,
more permissive scheme than the one covering these weights.

---

## Citation

```bibtex
@article{carlson2026animating,
  title   = {Animating on Twos: Training Keyframe-Animation Adapters on a Pretrained Video Model},
  author  = {Carlson, Minta and Bielec, Timothy},
  year    = {2026},
  month   = {August},
  journal = {Alvdansen Labs},
  url     = {https://alvdansen.github.io/animating-on-twos/}
}
```