README.md

---
title: MiniMax-H3 Reference
emoji: 🎭
colorFrom: pink
colorTo: purple
sdk: gradio
sdk_version: 6.20.0
app_file: app.py
pinned: true
short_description: Unquantized MiniMax-H3 from image, audio, video refs
suggested_hardware: zero-a10g
---

# MiniMax-H3 — omni-references, unquantized, split across two Spaces

Joint video **and** soundtrack out of a single denoising pass, conditioned on an ordered list of image, video and
audio references, at **bfloat16 with no quantization anywhere**.

This Space is the denoising half of the `ref2va` task: the 61.73 GiB `transformer_ref` partition and the two
autoencoders. The 62.14 GiB Qwen3-VL conditioner runs in
[`qwen3vl-conditioner`](https://huggingface.co/spaces/multimodalart/qwen3vl-conditioner), which this
Space calls over the gradio API for every request — the same conditioner Space, and the same resident weights, that
the keyframe half [`minimax-h3`](https://huggingface.co/spaces/multimodalart/minimax-h3)
uses.

## Why split

MiniMax-H3 is 195.9 GiB in bfloat16 and a ZeroGPU Space is evicted at **150 GB of storage**. An unquantized single
Space is therefore impossible. Cut the `ref2va` branch of `MiniMaxH3Blocks` at its `text_encoder` step and both
halves fit unquantized:

| Space | Subfolders | Download | Resident |
|---|---|---|---|
| [`qwen3vl-conditioner`](https://huggingface.co/spaces/multimodalart/qwen3vl-conditioner) | `text_encoder/` + `tokenizer/` + `processor/` | 66.7 GB | 62.15 GiB bf16 |
| this one | `transformer_ref/` + `vae/` + `audio_vae/` | 77.3 GB | 61.73 GiB bf16 + 10.43 GiB float32 |

## References

A request carries up to **12** references — at most 9 images, 3 videos and 3 audio clips — **in the order the model
reads them**. The order is semantic: it numbers the labels of MiniMax-H3's prompt presentation (`<Picture 1>`,
`<Video 1>`, `<Audio 1>`) and it advances the shared audio/video rotary clock, so the same references in a different
order are a different request. This demo lays the slots out as one tab per modality in reading order — images, then
audio, then video — and assembles the request that way. The Images tab opens with two slots and **+ Add another
image** reveals the rest, up to the model's own nine; the audio and video tabs hold one each. A reference left in a
tab that is not the open one is still part of the request; the tabs lay the slots out, they do not choose between
them.

Rules the model imposes, enforced here before anything is uploaded:

* an audio reference cannot be the only one; it needs an image or a video alongside it,
* a reference video runs 2 to 15 seconds, and brings its own soundtrack with it,
* the generated duration may be left to the references, but only when exactly one of them carries a soundtrack —
  which is why the duration slider disappears when a single reference can set it, and comes back when two can or
  when the one that could is out of range.

## Example assets

`examples/subject.png` is [a studio portrait by Oliver
Dohrn](https://www.pexels.com/photo/casual-portrait-of-smiling-woman-in-studio-31747504/) cropped to head and
shoulders — Pexels licence, free to use. `examples/motion.mp4` is a synthetic clip from the parity fixtures, and
`examples/voice.wav` is utterance `1462-170145-0022` of
[LibriSpeech](https://www.openslr.org/12) `dev-clean` — CC BY 4.0, read from a public-domain LibriVox recording. It
is 16 kHz mono on purpose: the audio VAE wants 32 kHz, so the example exercises the `torchaudio` resample the
`ref2va` path needs.

## How the split is expressed

`MiniMaxH3Blocks` is one `SequentialPipelineBlocks` whose branches are picked per request — and per `workflow=` —
from the inputs, `ref2va` being the branch `references` selects:

```
setup -> text_encoder -> reference_encoder -> denoise -> after_denoise -> decode
```

where `denoise` is itself `prepare_layout -> prepare_latents -> set_timesteps -> denoise`, against the
`transformer_ref` partition.

`h3_split_blocks.py` subclasses it with the `text_encoder` step removed. Dropping the step drops the three
components it declares, so `load_components` resolves `transformer_ref` / `vae` / `audio_vae` / the two schedulers
out of the shared `modular_model_index.json` and never fetches the conditioner — and `prompt_embeds` and
`text_token_tags` become ordinary required inputs of the pipeline call:

```py
pipe = MiniMaxH3Ref2VAGeneratorBlocks().init_pipeline("MiniMaxAI/MiniMax-H3")
pipe.load_components(dtype=torch.bfloat16)
state = pipe(prompt_embeds=..., text_token_tags=..., references=[...], height=544, width=960, num_frames=124,
             num_inference_steps=28)
```

Only **text** encoding is remote. `reference_encoder` is the `ref2va` branch's own encoder step — it runs the video
VAE over the image and video references and the audio VAE over the soundtracks, and it is where the references'
latent geometry is resolved — so it stays on this side, next to the autoencoders the conditioner Space does not hold.

The wire format is the same two tensors as the keyframe half: `(1, num_text_tokens, 5120)` bfloat16 and
`(num_text_tokens,)` int64, carried as one safetensors file with the resolved `height` / `width` / `num_frames` in
its metadata header. What differs is only what the conditioner is shown, so the references travel to it as files:
`ref2va`'s presentation puts a vision block in front of the prompt for every image and every merged video frame
pair. An audio reference contributes its `"<Audio j>: "` label and nothing else — a waveform never reaches the
conditioner — but it still goes over, because a single audio-bearing reference is what resolves `num_frames` when
the request leaves it open.

The `setup` step runs on **both** halves. It owns no component (PIL, PyAV-decoded media and arithmetic) and it
resolves the canvas, the `17 * n + 5` frame count and the references prepared at their own resolutions. It is
deterministic over the same files, and the conditioner returns the plan it resolved so this Space pins the same
canvas and frame count rather than re-deriving them.

## AoTI-compiled blocks

With `H3_AOTI=1` the 50 repeated transformer blocks run from a compiled package,
[`multimodalart/minimax-h3-aoti`](https://huggingface.co/multimodalart/minimax-h3-aoti)`:bf16/torch2.11/sm120/dynamic` — a single dynamic-sequence artifact that serves
every canvas, duration, reference set and prompt length.

It is the **same package the `transformer/` partition runs**, and nothing about it is partition-specific. The two
`config.json` files are identical field for field, and the package carries no weights at all: `LazyAOTIModel` binds
each block's own live `state_dict()` by name on its first forward. Patching it in is startup CPU work and costs no
GPU time.

It removes a near-constant ~0.5 s/step — 50 blocks' worth of kernel-launch overhead plus the norm / rotary / AdaLN
epilogues around the matmuls — and cannot touch the matmuls themselves, so it pays best where the block is not
compute bound. `ref2va` packs the reference rows in front of the generated ones, which makes the sequence longer
than a keyframe request at the same canvas and moves it further toward compute bound.

## Nothing is paid for with GPU time

The 77.3 GB download and the load happen at **startup**: `import spaces` at module top patches `torch.cuda` before
any GPU is attached, so nothing about the load needs a card. The conditioner round trip is a network call on this
Space's CPU. A `@spaces.GPU` call is therefore only the placement (once), the two reference encoders, the denoise
loop and the two decoders.

One thing does *not* happen at startup: the move onto the card. `spaces`' startup `torch.pack()` writes every
startup-resident CUDA tensor to a second copy on disk before deleting the downloaded originals, and 77.3 GB of
weights plus a 77.3 GB pack is 154.6 GB against a 150 GB quota — the Space is evicted mid-pack with `OSError:
[Errno 28] No space left on device`. Placement therefore happens on the **first GPU call**, `PIPE.to("cuda")` at the
top of the `@spaces.GPU` function: about 10 s of PCIe once, then a no-op walk, and the denoise loop runs with
everything resident and no offloading at all.

The references are decoded inside that call too, from their paths rather than as decoded media. A `@spaces.GPU`
argument crosses a process boundary by pickling, and a 5 s 1344x768 reference video is 370 MB of frames once PyAV
has expanded it.

## Generation constraints

Fixed by the checkpoint: 24 fps, a 768 pixel short edge, 5 to 15 s, `num_frames` snapped up to the next `17 * n + 5`,
no CFG and no negative prompt (it is guidance-distilled, so every step is one forward pass). The duration slider
stops at 14 s because it is the *snapped* count that has to hold for the ceiling: 15 s is 360 frames, which rounds
up to 362, i.e. 15.083 s, and is refused.

## GPU time is reserved per request, not per Space

MiniMax-H3 attends over one packed sequence, so what a step costs is a function of that sequence's length alone —
and on this half the references dominate it. A single 1344x768 image reference is ~7168 conditioning rows plus the
vision block it puts in front of the prompt; a 2.5 s video reference is another ~17000. The same 960x544, 124-frame
request runs 2.4 s/step with no references and 16 s/step with an image and a video.

`get_duration` prices that before the call instead of reserving a flat ceiling for everything. It takes the
arguments of the `@spaces.GPU` function, so it has the conditioner's own `text_token_tags` (exact) and the reference
files (measured from metadata, no decode), and evaluates

```
S = text rows + reference rows + target rows
seconds = placement + reference encode + steps * (LINEAR * S + QUADRATIC * S**2) * SAFETY + decode + pad
```

fitted on the `t2va` half and checked against live `ref2va` requests to about 10%. It matters beyond tidiness: the
pool reserves whatever number it is given, and a flat 900 s is what makes a busy account fail admission with *"You
have too many ZeroGPU credits allocated to running tasks."* A typical single-image request now reserves ~460 s.

Every request carries the full placement allowance, because nothing on this side knows whether the worker it lands
on is cold and a cold one pays the lazy 72.16 GiB `PIPE.to("cuda")` inside its first GPU call.

## Space variables

| Variable | Default | Meaning |
|---|---|---|
| `H3_CONDITIONER` | `multimodalart/qwen3vl-conditioner` | The public Space this one asks for embeddings; the client passes no token, so the call runs on the caller's own quota. |
| `H3_MODEL_REPO` | `MiniMaxAI/MiniMax-H3` | The diffusers-layout checkpoint. Public. |
| `H3_AOTI` | `0` | `1` loads the compiled block package. |
| `H3_PLACEMENT` | `lazy` | `lazy` moves all 72.16 GiB onto the card on the first GPU call and leaves it there; `offload` hands placement to `ComponentsManager.enable_auto_cpu_offload` instead. |
| `H3_ATTENTION` | `_native_cudnn` | cuDNN's fused kernel, 10–20% faster than the SDPA default and needs nothing installed. flash-attention 3 is sm90-only and this pool is sm120. The two VAEs are pinned to torch SDPA instead: they are float32, which cuDNN has no kernel for. |
| `H3_GPU_DURATION_MIN` / `_MAX` | `120` / `1500` | Bounds on what `get_duration` may reserve. |
| `H3_PLACEMENT_ALLOWANCE` | `90` | Seconds of the reservation set aside for a cold worker's placement. |
| `H3_GPU_SIZE` | `xlarge` | ZeroGPU allocation size. `large` does not fit. |

## Whose GPU quota pays

Two cards are booked per request — this Space's denoise loop and the conditioner's forward — and both are billed to the
requesting user, with nothing here arranging it: `gradio_client` attaches the caller's own `x-ip-token` to every
outgoing call, reading it off gradio's `LocalContext` inside the event listener (`Client.send_data` ->
`add_zero_gpu_headers`), and ZeroGPU charges the booking to whatever that token identifies.

A caller with no token to forward — a `gradio_client` script rather than a browser — leaves the conditioner's booking
attributed to this Space's pod IP and its small shared quota. An unattributed caller may book at most 120 credits at a
time and an `xlarge` booking costs twice its seconds, so the conditioner books the encode (45 s) and a prompt upsample
(60 s) as two separate calls, each within that ceiling.

## Secrets

None are required. Everything this Space downloads is public — the
[`MiniMaxAI/MiniMax-H3`](https://huggingface.co/MiniMaxAI/MiniMax-H3) checkpoint and the
[`multimodalart/minimax-h3-aoti`](https://huggingface.co/multimodalart/minimax-h3-aoti) packages — and the conditioner
is a public Space called on the requesting user's own ZeroGPU token, never on an org token.

## Where diffusers comes from

MiniMax-H3 is modular-only and not in a released `diffusers`, so `requirements.txt` installs it from the canonical
pull request, [huggingface/diffusers#14371](https://github.com/huggingface/diffusers/pull/14371), pinned to the commit
`665f5782` (`refs/pull/14371/head`) rather than to the moving `minimax-h3-refactor` branch.

That PR is a WIP, so it needs re-pinning whenever it updates, and `h3_split_blocks.py` — which subclasses its block
classes to cut the pipeline in two — has to be re-checked against the new head at the same time.

Two of those are `ref2va`-only and easy to miss. PyAV decodes a reference video or audio file as the reference is
built, and **`torchaudio`** resamples a soundtrack that is not already at the audio VAE's 32 kHz — a 32 kHz
reference skips the resample entirely, so the dependency only shows up once someone brings audio at another rate:

```
ImportError: Resampling a MiniMax-H3 reference soundtrack from 24000 Hz to 32000 Hz needs `torchaudio`.
```

The conditioner Space needs it as well: its `setup` step normalizes the very same waveforms this one does.