README.md

---
license: other
license_name: minimax-h3-community-license-agreement
license_link: https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/LICENSE
base_model:
  - MiniMaxAI/MiniMax-H3
  - Comfy-Org/MiniMax-H3
base_model_relation: merge
library_name: diffusers
pipeline_tag: image-text-to-video
tags:
  - minimax-h3
  - modular-diffusers
  - ref2va
  - fl2va
  - merge
  - synchronized-audio-video
  - experimental
inference: false
---

# MiniMax-H3, one transformer for references *and* keyframes

MiniMax-H3 ships **two** 37.5 GB transformer partitions: `fl2va` for first/last-frame conditioning and `ref2va`
for reference conditioning. This is **one** checkpoint that serves both, so a deployment holds 37.5 GB instead of
75 GB. It is the pruned `fl2va` partition with a rank-1024 approximation of the `ref2va − fl2va` weight delta
**fused into the weights**.

> [!WARNING]
> Experimental and mechanically derived — the delta was extracted from two released checkpoints by SVD, not
> trained. This is **not** `transformer_ref`: against the real thing it reaches video-latent cosine 0.875 / 0.897 /
> 0.691 on three matched reference requests, short of the 0.99 that would make it a drop-in replacement. Use it when
> one-partition deployment is worth that gap.

> [!IMPORTANT]
> Governed by the [MiniMax H3 Community License Agreement](https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/LICENSE),
> which carries territory exclusions and redistribution conditions. The upstream `LICENSE` is included and this card
> is the modification notice.

## Inference

Both workflows resolve to the weights in this repo, and each loads only its own slot:

```python
import torch
from diffusers import ComponentsManager, ModularPipeline

REPO = "diffusers-modular/MiniMax-H3-Pruned-Ref-Delta-Fused-r1024"

manager = ComponentsManager()
pipe = ModularPipeline.from_pretrained(REPO, workflow="ref2va", components_manager=manager,
                                      trust_remote_code=True)
pipe.load_components(dtype=torch.bfloat16, trust_remote_code=True)
manager.enable_auto_cpu_offload(device="cuda")   # 37.5 GB DiT + 62 GB conditioner + two VAEs
```

**References** (`workflow="ref2va"`) — image, video and/or audio, in the order they are passed:

```python
from diffusers.modular_pipelines.minimax_h3 import MiniMaxH3ImageReference

out = pipe(
    prompt="The woman from the reference stands in a snowy park at dusk, catching snowflakes and laughing.",
    references=[MiniMaxH3ImageReference.from_file("subject.png")],
    height=544, width=960, num_frames=124, num_inference_steps=20,
    generator=torch.Generator("cpu").manual_seed(42),
)
```

**Keyframes** (`workflow="fl2va"`) — a first and/or last frame:

```python
from PIL import Image

out = pipe(                       # same repo, reloaded with workflow="fl2va"
    prompt="She lifts the teacup and takes a slow sip, then smiles; quiet kitchen room tone.",
    image=Image.open("first.png"), last_image=Image.open("last.png"),
    height=544, width=960, num_frames=124, num_inference_steps=20,
    generator=torch.Generator("cpu").manual_seed(42),
)
video, audio, rate = out["videos"][0], out["audio"][0], out["sampling_rate"]
```

One generation carries **one** shape: the blocks dispatch on their inputs and `references` wins, so passing both
silently drops the keyframes.

### Few-step generation — keep the turbo LoRA live

```python
pipe.load_lora_weights("larryvrh/MiniMax-H3-Turbo-Lora", adapter_name="turbo",
                       load_into_transformer_ref=True)   # `load_into_transformer_ref` only for ref2va
```

**Do not fuse it.** A distill LoRA's deltas are 2–5e-4 of the weights they modify, and folding them into bf16 keeps
only **0.67–0.79** — a third of turbo's AdaLN modulation is lost to rounding. Left live it costs ~7–15% wall time.
For the same reason, an AoT-compiled transformer cannot be combined with a live adapter: the graph is captured from
the base modules and runs straight past it.

Requires `diffusers` with `MiniMaxH3LoraLoaderMixin` (PR
[#14408](https://github.com/huggingface/diffusers/pull/14408)) and `trust_remote_code=True` — the pruned layout
ships a `MiniMaxH3PrunedTransformer3DModel` whose AdaLN is 8 wide, where the stock class expects 2688.

## How it was made

1. **Take the difference between the twins.** `ref2va − fl2va`, tensor by tensor. Both partitions are architecturally
   identical, so this is what makes one able to use references.
2. **Compress it.** [ethanfel's](https://huggingface.co/ethanfel/MiniMax-H3-Pruned-Ref2VA-Delta-LoRAs-Experimental)
   randomized-SVD extraction at **rank 1024** (9.4 GB), applied at strength 1.0.
3. **Apply the parts that cannot be compressed exactly.** 267 patches: 211 RMSNorm deltas, 56 biases, and
   `adaln_t_table` — the timestep coordinate table, which differs between the partitions and which the earlier
   rank-256 extractions omit.
4. **Fuse.** `fuse_lora`, then unload the adapter, leaving an ordinary checkpoint. The delta is 0.02–1.9 of the
   weights it lands in, so it survives a bf16 fold at 1.00 (unlike a distill LoRA — see above).
5. **Keep the AdaLN affine map.** `adaln_basis` / `adaln_mean` ship as buffers, so LoRAs trained on the released
   2688-wide AdaLN still project onto this 8-wide one.

Verified by re-downloading this repo and generating: bit-identical (`torch.equal` on video and audio latents) to the
local build it was made from.

## Measured

Video-latent cosine against the true `ref2va` partition, generated on the same GPU (identical weights on a
different GPU only agree to 0.959–0.987, so cross-machine anchors are not usable at this precision). Three
seed-matched requests: photoreal image reference, stylized image reference, video reference.

| applied to pruned FL2VA | size | photoreal | stylized | video ref |
|---|---|---|---|---|
| **rank-1024 delta (this repo)** | 9.4 GB | **0.875** | **0.897** | **0.691** |
| rank-256 delta | 2.4 GB | 0.772 | 0.798 | 0.527 |
| AdaLN-only, exact | 95 MiB | 0.678 | 0.803 | 0.505 |
| nothing | 0 | 0.668 | 0.795 | 0.480 |

Rank is what matters, and the bulk is load-bearing: the exact AdaLN half — the tempting 95 MiB shortcut — lands on
the no-delta baseline, so the trunk attention and MLP deltas are carrying the result.

**Keyframes still work.** Against the *pristine* `fl2va` partition on the same first+last-frame request, this
checkpoint reaches video cosine **0.930** and is visually indistinguishable. Audio is the weaker half (0.662, and
it runs louder). Few-step, with turbo live: 64–179 s per 5 s clip against 174–478 s for 20 steps on the true
partition, at the VRAM of a single partition.

## Credit

Delta extraction [ethanfel](https://huggingface.co/ethanfel/MiniMax-H3-Pruned-Ref2VA-Delta-LoRAs-Experimental);
approach established by [Kijai](https://huggingface.co/Kijai/MiniMax-H3-experimental); pruned repack
[Comfy-Org](https://huggingface.co/Comfy-Org/MiniMax-H3); turbo LoRA
[larryvrh](https://huggingface.co/larryvrh/MiniMax-H3-Turbo-Lora); base model and license
[MiniMaxAI](https://huggingface.co/MiniMaxAI/MiniMax-H3). Related community work on the same question:
[lihaoyun6](https://huggingface.co/lihaoyun6/MiniMax-H3-Ref-Patch) (exact-diff-only patch),
[smhfacct](https://huggingface.co/smhfacct/Minimax-H3-fl2va-ref2va-hybrid-models) (AdaLN block swap),
[PulpCut](https://huggingface.co/PulpCut/MiniMax-H3-Ref2VA-Turbo-INT8-ConvRot) (turbo merged into Ref2VA).