README.md

---
license: mit
base_model: MiniMaxAI/MiniMax-H3
base_model_relation: adapter
tags:
  - minimax-h3
  - lora
  - video
  - comfyui
  - motion
library_name: diffusers
pipeline_tag: image-to-video
---

# MiniMax-H3 motion adapter (pilot, r16)

A small LoRA for MiniMax-H3 that makes the de-rope pass in
[ComfyUI-MAINodes](https://github.com/matlowai/ComfyUI-MAINodes) behave
better on fast motion. It is a pilot: rank 16, trained for a few hours on
38 clips we generated ourselves. It works, it is measurable, and it has
clear rough edges listed below. We are publishing it as an intermediate
release while we try something considerably more ambitious that may or may
not work (an all-in-one adapter that internalises the whole two-pass
pipeline). If that lands it will replace this file; if it does not, this
one still does a useful job. Part of that work is finding out whether an
adapter can be made to actually assign more compute to the burst on its
own (a variable-rate attention behaviour learned inside the model, rather
than the nodes stretching the clock for it); this pilot does not do that,
see the mechanics note below.

## What it does

MAINodes' Motion Lab regenerates the fast span of a clip on a stretched
clock (the "de-rope") and recovers it back to real time. That pass has a
known bad habit: on very fast motion the base model over-produces, and the
result alternates between advance and snap frame to frame. This adapter
was trained on that exact task (frames held out of the middle of a burst,
the model asked to fill them back in), and it teaches the base model to
spend the extra clock on smoothness instead of invention. To be precise
about the mechanics: the LoRA is a plain rank-16 weight delta and adds no
compute and needs no custom node. The extra clock comes from the nodes
(the de-rope stretches the burst into more latent tokens); the LoRA
changes how the model behaves on that stretched clock, because it was
trained in that exact situation.

Measured on four clips the adapter never saw during training (a fight,
an anime sword ring, a character swap, a dialogue close-up), against the
same graph without the adapter and against a same-seed noise floor:

- advance/snap alternation improved on 4 of 4 clips (4.6 to 6.1 times the
  floor)
- frame-to-frame over-production dropped from 1.76 to 2.93 down to 1.09
  to 1.23 on all four
- blind operator picks, adapter vs no adapter, same seed: 7 for, 0
  against, 1 no preference
- it transfers to the Ref2VA checkpoint (span, alternation and jitter
  wins of the same size on the two clips we tried), so one file serves
  both fl2va and ref2va graphs
- one end-to-end run of the shipped graph on a scene it had never seen
  (the fight choreography moved to a neon rooftop, same seed for both
  arms): alternation 0.24 to 0.19, jitter 0.09 to 0.04, fewer invented
  objects, and it kept the plate's setting where the no-adapter pass
  rewrote the signage. One clip, and each graph at its own shipped inject
  (0.70 vs 0.45), so read it as a smoke test, not a study

Where it is not great (also measured):

- it costs about 1 dB of anchor fidelity on native keyframes and it
  over-corrects calm chains (it smooths things that did not need
  smoothing); keep it off on calm content
- it mutes strong colour and thins particle effects. A prismatic creature
  came out as a plain calico on the Ref2VA checkpoint (mean saturation 46
  to 38 at denoise 0.40, 54 to 40 at 0.70) and again on the fl2va
  checkpoint when the whole clip was de-roped at 3x. Worst on Ref2VA and
  on full-clip passes; a windowed de-rope on the same creature kept its
  colour. If the subject is the colour, lower the strength or accept the
  trade
- it is anime-heavy in effect: on the one very fast anime clip we
  measured it wins clearly at every setting; on live-action character
  work it needs the lower inject values below

## Settings that work

- Strength 1.0. Strength and inject turned out to be the same dial. 1.0
  wins on alternation in every paired cell we ran (8 of 8), while 0.75
  invents fewer objects (7 of 8), so 1.0 is the smoothness setting and
  0.75 the lower-invention one; the old 0.75 to 0.80 band was
  in-distribution tuning on one fight plate and does not hold elsewhere.
- Inject is the dial. Starting points: 0.45 for character or dialogue
  where the base model over-produces hard; 0.30 where identity or props
  are the deliverable and the base is only mildly inflated; on very fast
  anime it does not matter (every setting wins).
- Keep the tail guide on. Turning it off costs identity about 3 times the
  floor across seeds and checkpoints, adapter or not.
- Apply it to the de-rope pass only, not to the first (text-to-video)
  pass.

## Use in ComfyUI

Put the file in `models/loras/minimax_h3/`. Load it with a stock
`LoraLoaderModelOnly` at strength 1.0 on the model that feeds the de-rope
pass. A ready graph is in ComfyUI-MAINodes:
`examples/motion_pipeline_adapter_api.json` (API format, loads through the
API and through the frontend's API import). The stock
`LoraLoaderModelOnly` is fine for this file; the key mismatch that bites
turbo LoRAs does not apply here.

## Training facts

- Base: MiniMax-H3 (fl2va), rank 16, alpha 16, 375 steps, lr 1e-4,
  adamw8bit, bf16, about 4.6 h on one GPU
- Data: 38 holdout-infilling items cut from 15 clips we generated with
  MiniMax-H3 (physics scenes, fights, chains, a dialogue plate); no real
  footage, no third-party footage
- Task: hold out the frames inside a motion burst, keep the rest as clean
  context (kept_label 1.0), predict the held tokens
- Trainer: our ai-toolkit extension for H3 (span-aware packing verified
  bit-exact against ComfyUI)

## Base model and licence

This is a LoRA for [MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3)
by MiniMax; it does nothing without their base weights and is trained on
clips generated with them. The adapter weights and the graph here are
released under the MIT licence (`LICENSE`); use of the base model, and so
of this adapter with it, is subject to the MiniMax H3 Community License
Agreement, included as `LICENSE-MiniMax-H3` (copied from the MiniMax-H3
repository at commit 42ed227e). Thanks to MiniMax for releasing H3.

## Files

- `minimax_h3_motion_adapter_pilot_r16.safetensors` (63 MB)
- `motion_pipeline_adapter_api.json` (the same graph as in the node pack)
- `LICENSE` (this repository, MIT) and `LICENSE-MiniMax-H3` (the base
  model's community licence)

Examples and before/after clips, including the colour-muting case, are on
the demo page: https://matlowai.github.io/ComfyUI-MAINodes/#adapter
The node pack, the example graph and the tuning notes are at
https://github.com/matlowai/ComfyUI-MAINodes (see README, "The motion
adapter (pilot)", and TUNING.md).