README.md

---
title: MiniMax-H3 Reference
emoji: 🎭
colorFrom: pink
colorTo: purple
sdk: gradio
sdk_version: 6.20.0
app_file: app.py
pinned: false
short_description: Unquantized MiniMax-H3 from image, audio, video refs
suggested_hardware: zero-a10g
---

# MiniMax-H3 — omni-references, unquantized, split across two Spaces

Joint video **and** soundtrack out of a single denoising pass, conditioned on an ordered list of image, video and
audio references, at **bfloat16 with no quantization anywhere**.

This Space is the denoising half of the `ref2va` task: the 61.73 GiB `transformer_ref` partition and the two
autoencoders. The 62.14 GiB Qwen3-VL conditioner runs in
[`minimax-h3-conditioner`](https://huggingface.co/spaces/diffusers-internal-dev/minimax-h3-conditioner), which this
Space calls over the gradio API for every request — the same conditioner Space, and the same resident weights, that
the keyframe half [`minimax-h3-generator`](https://huggingface.co/spaces/diffusers-internal-dev/minimax-h3-generator)
uses.

## Why split

MiniMax-H3 is 195.9 GiB in bfloat16 and a ZeroGPU Space is evicted at **150 GB of storage**. An unquantized single
Space is therefore impossible. Cut the `MiniMaxH3Ref2VABlocks` sequence at its `text_encoder` step and both halves
fit unquantized:

| Space | Subfolders | Download | Resident |
|---|---|---|---|
| [`minimax-h3-conditioner`](https://huggingface.co/spaces/diffusers-internal-dev/minimax-h3-conditioner) | `text_encoder/` + `tokenizer/` + `processor/` | 66.7 GB | 62.15 GiB bf16 |
| this one | `transformer_ref/` + `vae/` + `audio_vae/` | 77.3 GB | 61.73 GiB bf16 + 10.43 GiB float32 |

## References

A request carries up to **12** references — at most 9 images, 3 videos and 3 audio clips — **in the order the model
reads them**. The order is semantic: it numbers the labels of MiniMax-H3's prompt presentation (`<Picture 1>`,
`<Video 1>`, `<Audio 1>`) and it advances the shared audio/video rotary clock, so the same references in a different
order are a different request. This demo lays the slots out as one tab per modality in reading order — images, then
audio, then video — and assembles the request that way. The Images tab opens with two slots and **+ Add another
image** reveals the rest, up to the model's own nine; the audio and video tabs hold one each. A reference left in a
tab that is not the open one is still part of the request; the tabs lay the slots out, they do not choose between
them.

Rules the model imposes, enforced here before anything is uploaded:

* an audio reference cannot be the only one; it needs an image or a video alongside it,
* a reference video runs 2 to 15 seconds, and brings its own soundtrack with it,
* the generated duration may be left to the references, but only when exactly one of them carries a soundtrack —
  which is why the duration slider disappears when a single reference can set it, and comes back when two can or
  when the one that could is out of range.

## How the split is expressed

`MiniMaxH3Ref2VABlocks` is a `SequentialPipelineBlocks` of eight steps:

```
setup -> text_encoder -> reference_encoder -> prepare_layout -> prepare_latents -> set_timesteps -> denoise -> decode
```

`h3_split_blocks.py` subclasses it with the `text_encoder` step removed. Dropping the step drops the three
components it declares, so `load_components` resolves `transformer_ref` / `vae` / `audio_vae` / the two schedulers
out of the shared `modular_model_index.json` and never fetches the conditioner — and `prompt_embeds` and
`text_token_tags` become ordinary required inputs of the pipeline call:

```py
pipe = MiniMaxH3Ref2VAGeneratorBlocks().init_pipeline("diffusers-internal-dev/MiniMax-H3")
pipe.load_components(dtype=torch.bfloat16)
state = pipe(prompt_embeds=..., text_token_tags=..., references=[...], height=544, width=960, num_frames=124,
             num_inference_steps=28)
```

Only **text** encoding is remote. `reference_encoder` is the `ref2va` blockset's own encoder step — it runs the video
VAE over the image and video references and the audio VAE over the soundtracks, and it is where the references'
latent geometry is resolved — so it stays on this side, next to the autoencoders the conditioner Space does not hold.

The wire format is the same two tensors as the keyframe half: `(1, num_text_tokens, 5120)` bfloat16 and
`(num_text_tokens,)` int64, carried as one safetensors file with the resolved `height` / `width` / `num_frames` in
its metadata header. What differs is only what the conditioner is shown, so the references travel to it as files:
`ref2va`'s presentation puts a vision block in front of the prompt for every image and every merged video frame
pair. An audio reference contributes its `"<Audio j>: "` label and nothing else — a waveform never reaches the
conditioner — but it still goes over, because a single audio-bearing reference is what resolves `num_frames` when
the request leaves it open.

The `setup` step runs on **both** halves. It owns no component (PIL, PyAV-decoded media and arithmetic) and it
resolves the canvas, the `17 * n + 5` frame count and the references prepared at their own resolutions. It is
deterministic over the same files, and the conditioner returns the plan it resolved so this Space pins the same
canvas and frame count rather than re-deriving them.

## AoTI-compiled blocks

With `H3_AOTI=1` the 50 repeated transformer blocks run from a compiled package,
`diffusers-internal-dev/minimax-h3-aoti:bf16/torch2.11/sm120/dynamic` — a single dynamic-sequence artifact that serves
every canvas, duration, reference set and prompt length.

It is the **same package the `transformer/` partition runs**, and nothing about it is partition-specific. The two
`config.json` files are identical field for field, and the package carries no weights at all: `LazyAOTIModel` binds
each block's own live `state_dict()` by name on its first forward. Patching it in is startup CPU work and costs no
GPU time.

It removes a near-constant ~0.5 s/step — 50 blocks' worth of kernel-launch overhead plus the norm / rotary / AdaLN
epilogues around the matmuls — and cannot touch the matmuls themselves, so it pays best where the block is not
compute bound. `ref2va` packs the reference rows in front of the generated ones, which makes the sequence longer
than a keyframe request at the same canvas and moves it further toward compute bound.

## Nothing is paid for with GPU time

The 77.3 GB download and the load happen at **startup**: `import spaces` at module top patches `torch.cuda` before
any GPU is attached, so nothing about the load needs a card. The conditioner round trip is a network call on this
Space's CPU. A `@spaces.GPU` call is therefore only the placement (once), the two reference encoders, the denoise
loop and the two decoders.

One thing does *not* happen at startup: the move onto the card. `spaces`' startup `torch.pack()` writes every
startup-resident CUDA tensor to a second copy on disk before deleting the downloaded originals, and 77.3 GB of
weights plus a 77.3 GB pack is 154.6 GB against a 150 GB quota — the Space is evicted mid-pack with `OSError:
[Errno 28] No space left on device`. Placement therefore happens on the **first GPU call**, `PIPE.to("cuda")` at the
top of the `@spaces.GPU` function: about 10 s of PCIe once, then a no-op walk, and the denoise loop runs with
everything resident and no offloading at all.

The references are decoded inside that call too, from their paths rather than as decoded media. A `@spaces.GPU`
argument crosses a process boundary by pickling, and a 5 s 1344x768 reference video is 370 MB of frames once PyAV
has expanded it.

## Generation constraints

Fixed by the checkpoint: 24 fps, a 768 pixel short edge, 5 to 15 s, `num_frames` snapped up to the next `17 * n + 5`,
no CFG and no negative prompt (it is guidance-distilled, so every step is one forward pass). The duration slider
stops at 14 s because it is the *snapped* count that has to hold for the ceiling: 15 s is 360 frames, which rounds
up to 362, i.e. 15.083 s, and is refused.

## Space variables

| Variable | Default | Meaning |
|---|---|---|
| `H3_CONDITIONER` | `diffusers-internal-dev/minimax-h3-conditioner` | The Space this one asks for embeddings. |
| `H3_AOTI` | `0` | `1` loads the compiled block package. |
| `H3_PLACEMENT` | `lazy` | `lazy` moves all 72.16 GiB onto the card on the first GPU call and leaves it there; `offload` hands placement to `ComponentsManager.enable_auto_cpu_offload` instead. |
| `H3_ATTENTION` | `_native_cudnn` | cuDNN's fused kernel, 10–20% faster than the SDPA default and needs nothing installed. flash-attention 3 is sm90-only and this pool is sm120. |
| `H3_GPU_DURATION` | `900` | Seconds per request; the pool applies a 1.5 duration factor. |
| `H3_GPU_SIZE` | `xlarge` | ZeroGPU allocation size. `large` does not fit. |

## Required secret

`HF_TOKEN` — `diffusers-internal-dev/MiniMax-H3` is private, and so is the conditioner Space this one calls.

## Where diffusers comes from

MiniMax-H3 is modular-only and not in a released `diffusers`, so the integration branch's `src/diffusers` tree is
vendored here as a top-level `diffusers/` package; the working directory comes first on `sys.path`, so there is no
install step. `requirements.txt` only carries what that tree imports.

Two of those are `ref2va`-only and easy to miss. PyAV decodes a reference video or audio file as the reference is
built, and **`torchaudio`** resamples a soundtrack that is not already at the audio VAE's 32 kHz — a 32 kHz
reference skips the resample entirely, so the dependency only shows up once someone brings audio at another rate:

```
ImportError: Resampling a MiniMax-H3 reference soundtrack from 24000 Hz to 32000 Hz needs `torchaudio`.
```

The conditioner Space needs it as well: its `setup` step prepares the very same waveforms this one does.