File size: 10,144 Bytes

---
license: other
license_name: minimax-h3-community-license-agreement
license_link: https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/LICENSE
base_model:
  - MiniMaxAI/MiniMax-H3
  - Comfy-Org/MiniMax-H3
base_model_relation: merge
library_name: diffusers
pipeline_tag: image-text-to-video
tags:
  - minimax-h3
  - modular-diffusers
  - ref2va
  - fl2va
  - merge
  - synchronized-audio-video
  - experimental
inference: false
---

# MiniMax-H3, one transformer for references *and* keyframes

MiniMax-H3 ships **two** 37.5 GB transformer partitions: `fl2va` for first/last-frame conditioning and `ref2va`
for reference conditioning. This is **one** checkpoint that serves both, so a deployment holds 37.5 GB instead of
75 GB. It is the pruned `fl2va` partition with a rank-1024 approximation of the `ref2va − fl2va` weight delta
**fused into the weights**.

> [!WARNING]
> Experimental and mechanically derived — the delta was extracted from two released checkpoints by SVD, not
> trained. This is **not** `transformer_ref`: against the real thing it reaches video-latent cosine 0.875 / 0.897 /
> 0.691 on three matched reference requests, short of the 0.99 that would make it a drop-in replacement. Use it when
> one-partition deployment is worth that gap.

> [!IMPORTANT]
> Governed by the [MiniMax H3 Community License Agreement](https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/LICENSE),
> which carries territory exclusions and redistribution conditions. The upstream `LICENSE` is included and this card
> is the modification notice.

## Inference

Both workflows resolve to the weights in this repo, and each loads only its own slot — pass a `workflow=`, or the
components of *both* transformer slots get pulled:

```python
import torch
from diffusers import ComponentsManager, ModularPipeline

REPO = "diffusers-modular/MiniMax-H3-Pruned-Ref-Delta-Fused-r1024"

manager = ComponentsManager()
pipe = ModularPipeline.from_pretrained(REPO, workflow="ref2va", components_manager=manager,
                                      trust_remote_code=True)
pipe.load_components(dtype=torch.bfloat16, trust_remote_code=True)
manager.enable_auto_cpu_offload(device="cuda")   # 37.5 GB DiT + 62 GB conditioner + two VAEs
```

**References** (`workflow="ref2va"`) — image, video and/or audio, in the order they are passed:

```python
from diffusers.modular_pipelines.minimax_h3 import MiniMaxH3ImageReference

out = pipe(
    prompt="The woman from the reference stands in a snowy park at dusk, catching snowflakes and laughing.",
    references=[MiniMaxH3ImageReference.from_file("subject.png")],
    height=544, width=960, num_frames=124, num_inference_steps=20,
    generator=torch.Generator("cpu").manual_seed(42),
)
```

**Keyframes** (`workflow="fl2va"`) — a first and/or last frame:

```python
from PIL import Image

out = pipe(                       # same repo, reloaded with workflow="fl2va"
    prompt="She lifts the teacup and takes a slow sip, then smiles; quiet kitchen room tone.",
    image=Image.open("first.png"), last_image=Image.open("last.png"),
    height=544, width=960, num_frames=124, num_inference_steps=20,
    generator=torch.Generator("cpu").manual_seed(42),
)
video, audio, rate = out["videos"][0], out["audio"][0], out["sampling_rate"]
```

### Keyframes and references in the same generation

`workflow="combined"` — this repo ships the blocks for it:

```python
pipe = ModularPipeline.from_pretrained(REPO, workflow="combined", trust_remote_code=True)
pipe.load_components(dtype=torch.bfloat16, trust_remote_code=True)

out = pipe(
    prompt="She lifts the teacup and takes a slow sip, then smiles; warm kitchen light",
    references=[MiniMaxH3ImageReference.from_file("subject.png")],   # who / what
    image=Image.open("first.png"), last_image=Image.open("last.png"),  # where it starts and ends
    height=544, width=960, num_frames=124, num_inference_steps=20,
    generator=torch.Generator("cpu").manual_seed(42),
)
```

MiniMax-H3 denoises **one packed sequence**, and that sequence can hold keyframe conditioning rows and reference
conditioning rows at the same time. `diffusers`' shipped blocks cannot express it: their conditional steps dispatch
either/or — `select_block` checks `references` first — so a request carrying both is accepted and the **keyframes are
silently dropped**, with no error and no warning.

`MiniMaxH3CombinedBlocks` (in `combined_blocks.py`) is the stock blockset with three of its conditional steps replaced
by ones that know a fourth shape. It reserves `[text | keyframe conditions | reference blocks | target audio | target
video]`, and because the references push the target timeline out, the keyframe anchors ride on the timeline their
spans leave behind. The `t2va`, `fl2va` and `ref2va` workflows are untouched and take exactly the same path as before;
the denoising loop needed no change at all, since the conditioning rows are simply the leading rows of the sequence.

Two things keep it honest: the layout reproduces **both** shipped `diffusers` builders bit for bit in their degenerate
cases — no keyframes gives the `ref2va` layout, no references gives the `fl2va` one, `position_ids` compared in
float64 — and the stock prepare-latents step asserts that the conditioning rows encoded equal the rows the layout
reserved, so a wrong layout raises instead of quietly degrading.

Measured on this checkpoint — with this layout, driven through the same underlying steps, before it was packaged as
the blockset above: a combined request (first frame + last frame + an image reference) agrees with the real
`ref2va` partition at video-latent cosine **0.972**, closer than any reference-only request reaches, because keyframes
anchoring both ends leave the delta less to carry. Adding an audio reference on top moves the video only 0.965 while
rewriting the soundtrack to 0.403 — each conditioning doing its own job.

### Few-step generation — keep the turbo LoRA live

```python
pipe.load_lora_weights("larryvrh/MiniMax-H3-Turbo-Lora", adapter_name="turbo",
                       load_into_transformer_ref=True)   # `load_into_transformer_ref` only for ref2va
```

**Do not fuse it.** A distill LoRA's deltas are 2–5e-4 of the weights they modify, and folding them into bf16 keeps
only **0.67–0.79** — a third of turbo's AdaLN modulation is lost to rounding. Left live it costs ~7–15% wall time.
For the same reason, an AoT-compiled transformer cannot be combined with a live adapter: the graph is captured from
the base modules and runs straight past it.

Requires `diffusers` with `MiniMaxH3LoraLoaderMixin` (PR
[#14408](https://github.com/huggingface/diffusers/pull/14408)) and `trust_remote_code=True` — the pruned layout
ships a `MiniMaxH3PrunedTransformer3DModel` whose AdaLN is 8 wide, where the stock class expects 2688.

## How it was made

1. **Take the difference between the twins.** `ref2va − fl2va`, tensor by tensor. Both partitions are architecturally
   identical, so this is what makes one able to use references.
2. **Compress it.** [ethanfel's](https://huggingface.co/ethanfel/MiniMax-H3-Pruned-Ref2VA-Delta-LoRAs-Experimental)
   randomized-SVD extraction at **rank 1024** (9.4 GB), applied at strength 1.0.
3. **Apply the parts that cannot be compressed exactly.** 267 patches: 211 RMSNorm deltas, 56 biases, and
   `adaln_t_table` — the timestep coordinate table, which differs between the partitions and which the earlier
   rank-256 extractions omit.
4. **Fuse.** `fuse_lora`, then unload the adapter, leaving an ordinary checkpoint. The delta is 0.02–1.9 of the
   weights it lands in, so it survives a bf16 fold at 1.00 (unlike a distill LoRA — see above).
5. **Keep the AdaLN affine map.** `adaln_basis` / `adaln_mean` ship as buffers, so LoRAs trained on the released
   2688-wide AdaLN still project onto this 8-wide one.

Verified by re-downloading this repo and generating: bit-identical (`torch.equal` on video and audio latents) to the
local build it was made from.

## Measured

Video-latent cosine against the true `ref2va` partition, generated on the same GPU (identical weights on a
different GPU only agree to 0.959–0.987, so cross-machine anchors are not usable at this precision). Three
seed-matched requests: photoreal image reference, stylized image reference, video reference.

| applied to pruned FL2VA | size | photoreal | stylized | video ref |
|---|---|---|---|---|
| **rank-1024 delta (this repo)** | 9.4 GB | **0.875** | **0.897** | **0.691** |
| rank-256 delta | 2.4 GB | 0.772 | 0.798 | 0.527 |
| AdaLN-only, exact | 95 MiB | 0.678 | 0.803 | 0.505 |
| nothing | 0 | 0.668 | 0.795 | 0.480 |

Rank is what matters, and the bulk is load-bearing: the exact AdaLN half — the tempting 95 MiB shortcut — lands on
the no-delta baseline, so the trunk attention and MLP deltas are carrying the result.

**Keyframes still work.** Against the *pristine* `fl2va` partition on the same first+last-frame request, this
checkpoint reaches video cosine **0.930** and is visually indistinguishable. Audio is the weaker half (0.662, and
it runs louder). Few-step, with turbo live: 64–179 s per 5 s clip against 174–478 s for 20 steps on the true
partition, at the VRAM of a single partition.

## Credit

Delta extraction [ethanfel](https://huggingface.co/ethanfel/MiniMax-H3-Pruned-Ref2VA-Delta-LoRAs-Experimental);
approach established by [Kijai](https://huggingface.co/Kijai/MiniMax-H3-experimental); pruned repack
[Comfy-Org](https://huggingface.co/Comfy-Org/MiniMax-H3); turbo LoRA
[larryvrh](https://huggingface.co/larryvrh/MiniMax-H3-Turbo-Lora); base model and license
[MiniMaxAI](https://huggingface.co/MiniMaxAI/MiniMax-H3). Related community work on the same question:
[lihaoyun6](https://huggingface.co/lihaoyun6/MiniMax-H3-Ref-Patch) (exact-diff-only patch),
[smhfacct](https://huggingface.co/smhfacct/Minimax-H3-fl2va-ref2va-hybrid-models) (AdaLN block swap),
[PulpCut](https://huggingface.co/PulpCut/MiniMax-H3-Ref2VA-Turbo-INT8-ConvRot) (turbo merged into Ref2VA). The
packed-sequence order a combined request uses follows ComfyUI's implementation of this model.