| --- |
| license: mit |
| base_model: MiniMaxAI/MiniMax-H3 |
| base_model_relation: adapter |
| tags: |
| - minimax-h3 |
| - lora |
| - video |
| - comfyui |
| - motion |
| library_name: diffusers |
| pipeline_tag: image-to-video |
| --- |
| |
| # MiniMax-H3 motion adapter (pilot, r16) |
|
|
| A small LoRA for MiniMax-H3 that makes the de-rope pass in |
| [ComfyUI-MAINodes](https://github.com/matlowai/ComfyUI-MAINodes) behave |
| better on fast motion. It is a pilot: rank 16, trained for a few hours on |
| 38 clips we generated ourselves. It works, it is measurable, and it has |
| clear rough edges listed below. We are publishing it as an intermediate |
| release while we try something considerably more ambitious that may or may |
| not work (an all-in-one adapter that internalises the whole two-pass |
| pipeline). If that lands it will replace this file; if it does not, this |
| one still does a useful job. Part of that work is finding out whether an |
| adapter can be made to actually assign more compute to the burst on its |
| own (a variable-rate attention behaviour learned inside the model, rather |
| than the nodes stretching the clock for it); this pilot does not do that, |
| see the mechanics note below. |
|
|
| ## What it does |
|
|
| MAINodes' Motion Lab regenerates the fast span of a clip on a stretched |
| clock (the "de-rope") and recovers it back to real time. That pass has a |
| known bad habit: on very fast motion the base model over-produces, and the |
| result alternates between advance and snap frame to frame. This adapter |
| was trained on that exact task (frames held out of the middle of a burst, |
| the model asked to fill them back in), and it teaches the base model to |
| spend the extra clock on smoothness instead of invention. To be precise |
| about the mechanics: the LoRA is a plain rank-16 weight delta and adds no |
| compute and needs no custom node. The extra clock comes from the nodes |
| (the de-rope stretches the burst into more latent tokens); the LoRA |
| changes how the model behaves on that stretched clock, because it was |
| trained in that exact situation. |
|
|
| Measured on four clips the adapter never saw during training (a fight, |
| an anime sword ring, a character swap, a dialogue close-up), against the |
| same graph without the adapter and against a same-seed noise floor: |
|
|
| - advance/snap alternation improved on 4 of 4 clips (4.6 to 6.1 times the |
| floor) |
| - frame-to-frame over-production dropped from 1.76 to 2.93 down to 1.09 |
| to 1.23 on all four |
| - blind operator picks, adapter vs no adapter, same seed: 7 for, 0 |
| against, 1 no preference |
| - it transfers to the Ref2VA checkpoint (span, alternation and jitter |
| wins of the same size on the two clips we tried), so one file serves |
| both fl2va and ref2va graphs |
| - one end-to-end run of the shipped graph on a scene it had never seen |
| (the fight choreography moved to a neon rooftop, same seed for both |
| arms): alternation 0.24 to 0.19, jitter 0.09 to 0.04, fewer invented |
| objects, and it kept the plate's setting where the no-adapter pass |
| rewrote the signage. One clip, and each graph at its own shipped inject |
| (0.70 vs 0.45), so read it as a smoke test, not a study |
|
|
| Where it is not great (also measured): |
|
|
| - it costs about 1 dB of anchor fidelity on native keyframes and it |
| over-corrects calm chains (it smooths things that did not need |
| smoothing); keep it off on calm content |
| - it mutes strong colour and thins particle effects. A prismatic creature |
| came out as a plain calico on the Ref2VA checkpoint (mean saturation 46 |
| to 38 at denoise 0.40, 54 to 40 at 0.70) and again on the fl2va |
| checkpoint when the whole clip was de-roped at 3x. Worst on Ref2VA and |
| on full-clip passes; a windowed de-rope on the same creature kept its |
| colour. If the subject is the colour, lower the strength or accept the |
| trade |
| - it is anime-heavy in effect: on the one very fast anime clip we |
| measured it wins clearly at every setting; on live-action character |
| work it needs the lower inject values below |
|
|
| ## Settings that work |
|
|
| - Strength 1.0. Strength and inject turned out to be the same dial. 1.0 |
| wins on alternation in every paired cell we ran (8 of 8), while 0.75 |
| invents fewer objects (7 of 8), so 1.0 is the smoothness setting and |
| 0.75 the lower-invention one; the old 0.75 to 0.80 band was |
| in-distribution tuning on one fight plate and does not hold elsewhere. |
| - Inject is the dial. Starting points: 0.45 for character or dialogue |
| where the base model over-produces hard; 0.30 where identity or props |
| are the deliverable and the base is only mildly inflated; on very fast |
| anime it does not matter (every setting wins). |
| - Keep the tail guide on. Turning it off costs identity about 3 times the |
| floor across seeds and checkpoints, adapter or not. |
| - Apply it to the de-rope pass only, not to the first (text-to-video) |
| pass. |
|
|
| ## Use in ComfyUI |
|
|
| Put the file in `models/loras/minimax_h3/`. Load it with a stock |
| `LoraLoaderModelOnly` at strength 1.0 on the model that feeds the de-rope |
| pass. A ready graph is in ComfyUI-MAINodes: |
| `examples/motion_pipeline_adapter_api.json` (API format, loads through the |
| API and through the frontend's API import). The stock |
| `LoraLoaderModelOnly` is fine for this file; the key mismatch that bites |
| turbo LoRAs does not apply here. |
|
|
| ## Training facts |
|
|
| - Base: MiniMax-H3 (fl2va), rank 16, alpha 16, 375 steps, lr 1e-4, |
| adamw8bit, bf16, about 4.6 h on one GPU |
| - Data: 38 holdout-infilling items cut from 15 clips we generated with |
| MiniMax-H3 (physics scenes, fights, chains, a dialogue plate); no real |
| footage, no third-party footage |
| - Task: hold out the frames inside a motion burst, keep the rest as clean |
| context (kept_label 1.0), predict the held tokens |
| - Trainer: our ai-toolkit extension for H3 (span-aware packing verified |
| bit-exact against ComfyUI) |
| |
| ## Base model and licence |
| |
| This is a LoRA for [MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3) |
| by MiniMax; it does nothing without their base weights and is trained on |
| clips generated with them. The adapter weights and the graph here are |
| released under the MIT licence (`LICENSE`); use of the base model, and so |
| of this adapter with it, is subject to the MiniMax H3 Community License |
| Agreement, included as `LICENSE-MiniMax-H3` (copied from the MiniMax-H3 |
| repository at commit 42ed227e). Thanks to MiniMax for releasing H3. |
| |
| ## Files |
| |
| - `minimax_h3_motion_adapter_pilot_r16.safetensors` (63 MB) |
| - `motion_pipeline_adapter_api.json` (the same graph as in the node pack) |
| - `LICENSE` (this repository, MIT) and `LICENSE-MiniMax-H3` (the base |
| model's community licence) |
|
|
| Examples and before/after clips, including the colour-muting case, are on |
| the demo page: https://matlowai.github.io/ComfyUI-MAINodes/#adapter |
| The node pack, the example graph and the tuning notes are at |
| https://github.com/matlowai/ComfyUI-MAINodes (see README, "The motion |
| adapter (pilot)", and TUNING.md). |
|
|
