README.md

metadata

title: MiniMax-H3 Reference
emoji: 🎭
colorFrom: pink
colorTo: purple
sdk: gradio
sdk_version: 6.20.0
app_file: app.py
pinned: false
short_description: Unquantized MiniMax-H3 from image, audio, video refs
suggested_hardware: zero-a10g

MiniMax-H3 — omni-references, unquantized, split across two Spaces

Joint video and soundtrack out of a single denoising pass, conditioned on an ordered list of image, video and audio references, at bfloat16 with no quantization anywhere.

This Space is the denoising half of the ref2va task: the 61.73 GiB transformer_ref partition and the two autoencoders. The 62.14 GiB Qwen3-VL conditioner runs in minimax-h3-conditioner, which this Space calls over the gradio API for every request — the same conditioner Space, and the same resident weights, that the keyframe half minimax-h3-generator uses.

Why split

MiniMax-H3 is 195.9 GiB in bfloat16 and a ZeroGPU Space is evicted at 150 GB of storage. An unquantized single Space is therefore impossible. Cut the MiniMaxH3Ref2VABlocks sequence at its text_encoder step and both halves fit unquantized:

Space	Subfolders	Download	Resident
minimax-h3-conditioner	text_encoder/ + tokenizer/ + processor/	66.7 GB	62.15 GiB bf16
this one	transformer_ref/ + vae/ + audio_vae/	77.3 GB	61.73 GiB bf16 + 10.43 GiB float32

References

A request carries up to 12 references — at most 9 images, 3 videos and 3 audio clips — in the order the model reads them. The order is semantic: it numbers the labels of MiniMax-H3's prompt presentation (<Picture 1>, <Video 1>, <Audio 1>) and it advances the shared audio/video rotary clock, so the same references in a different order are a different request. This demo lays the slots out as one tab per modality in reading order — images, then audio, then video — and assembles the request that way. The Images tab opens with two slots and + Add another image reveals the rest, up to the model's own nine; the audio and video tabs hold one each. A reference left in a tab that is not the open one is still part of the request; the tabs lay the slots out, they do not choose between them.

Rules the model imposes, enforced here before anything is uploaded:

• an audio reference cannot be the only one; it needs an image or a video alongside it,
• a reference video runs 2 to 15 seconds, and brings its own soundtrack with it,
• the generated duration may be left to the references, but only when exactly one of them carries a soundtrack — which is why the duration slider disappears when a single reference can set it, and comes back when two can or when the one that could is out of range.

How the split is expressed

MiniMaxH3Ref2VABlocks is a SequentialPipelineBlocks of eight steps:

setup -> text_encoder -> reference_encoder -> prepare_layout -> prepare_latents -> set_timesteps -> denoise -> decode

h3_split_blocks.py subclasses it with the text_encoder step removed. Dropping the step drops the three components it declares, so load_components resolves transformer_ref / vae / audio_vae / the two schedulers out of the shared modular_model_index.json and never fetches the conditioner — and prompt_embeds and text_token_tags become ordinary required inputs of the pipeline call:

pipe = MiniMaxH3Ref2VAGeneratorBlocks().init_pipeline("diffusers-internal-dev/MiniMax-H3")
pipe.load_components(dtype=torch.bfloat16)
state = pipe(prompt_embeds=..., text_token_tags=..., references=[...], height=544, width=960, num_frames=124,
             num_inference_steps=28)

Only text encoding is remote. reference_encoder is the ref2va blockset's own encoder step — it runs the video VAE over the image and video references and the audio VAE over the soundtracks, and it is where the references' latent geometry is resolved — so it stays on this side, next to the autoencoders the conditioner Space does not hold.

The wire format is the same two tensors as the keyframe half: (1, num_text_tokens, 5120) bfloat16 and (num_text_tokens,) int64, carried as one safetensors file with the resolved height / width / num_frames in its metadata header. What differs is only what the conditioner is shown, so the references travel to it as files: ref2va's presentation puts a vision block in front of the prompt for every image and every merged video frame pair. An audio reference contributes its "<Audio j>: " label and nothing else — a waveform never reaches the conditioner — but it still goes over, because a single audio-bearing reference is what resolves num_frames when the request leaves it open.

The setup step runs on both halves. It owns no component (PIL, PyAV-decoded media and arithmetic) and it resolves the canvas, the 17 * n + 5 frame count and the references prepared at their own resolutions. It is deterministic over the same files, and the conditioner returns the plan it resolved so this Space pins the same canvas and frame count rather than re-deriving them.

AoTI-compiled blocks

With H3_AOTI=1 the 50 repeated transformer blocks run from a compiled package, diffusers-internal-dev/minimax-h3-aoti:bf16/torch2.11/sm120/dynamic — a single dynamic-sequence artifact that serves every canvas, duration, reference set and prompt length.

It is the same package the transformer/ partition runs, and nothing about it is partition-specific. The two config.json files are identical field for field, and the package carries no weights at all: LazyAOTIModel binds each block's own live state_dict() by name on its first forward. Patching it in is startup CPU work and costs no GPU time.

It removes a near-constant ~0.5 s/step — 50 blocks' worth of kernel-launch overhead plus the norm / rotary / AdaLN epilogues around the matmuls — and cannot touch the matmuls themselves, so it pays best where the block is not compute bound. ref2va packs the reference rows in front of the generated ones, which makes the sequence longer than a keyframe request at the same canvas and moves it further toward compute bound.

Nothing is paid for with GPU time

The 77.3 GB download and the load happen at startup: import spaces at module top patches torch.cuda before any GPU is attached, so nothing about the load needs a card. The conditioner round trip is a network call on this Space's CPU. A @spaces.GPU call is therefore only the placement (once), the two reference encoders, the denoise loop and the two decoders.

One thing does not happen at startup: the move onto the card. spaces' startup torch.pack() writes every startup-resident CUDA tensor to a second copy on disk before deleting the downloaded originals, and 77.3 GB of weights plus a 77.3 GB pack is 154.6 GB against a 150 GB quota — the Space is evicted mid-pack with OSError: [Errno 28] No space left on device. Placement therefore happens on the first GPU call, PIPE.to("cuda") at the top of the @spaces.GPU function: about 10 s of PCIe once, then a no-op walk, and the denoise loop runs with everything resident and no offloading at all.

The references are decoded inside that call too, from their paths rather than as decoded media. A @spaces.GPU argument crosses a process boundary by pickling, and a 5 s 1344x768 reference video is 370 MB of frames once PyAV has expanded it.

Generation constraints

Fixed by the checkpoint: 24 fps, a 768 pixel short edge, 5 to 15 s, num_frames snapped up to the next 17 * n + 5, no CFG and no negative prompt (it is guidance-distilled, so every step is one forward pass). The duration slider stops at 14 s because it is the snapped count that has to hold for the ceiling: 15 s is 360 frames, which rounds up to 362, i.e. 15.083 s, and is refused.

Space variables

Variable	Default	Meaning
H3_CONDITIONER	diffusers-internal-dev/minimax-h3-conditioner	The Space this one asks for embeddings.
H3_AOTI	0	1 loads the compiled block package.
H3_PLACEMENT	lazy	lazy moves all 72.16 GiB onto the card on the first GPU call and leaves it there; offload hands placement to ComponentsManager.enable_auto_cpu_offload instead.
H3_ATTENTION	_native_cudnn	cuDNN's fused kernel, 10–20% faster than the SDPA default and needs nothing installed. flash-attention 3 is sm90-only and this pool is sm120.
H3_GPU_DURATION	900	Seconds per request; the pool applies a 1.5 duration factor.
H3_GPU_SIZE	xlarge	ZeroGPU allocation size. large does not fit.

Required secret

HF_TOKEN — diffusers-internal-dev/MiniMax-H3 is private, and so is the conditioner Space this one calls.

Where diffusers comes from

MiniMax-H3 is modular-only and not in a released diffusers, so the integration branch's src/diffusers tree is vendored here as a top-level diffusers/ package; the working directory comes first on sys.path, so there is no install step. requirements.txt only carries what that tree imports.