README.md

---
pipeline_tag: image-text-to-video
library_name: vllm-omni
license: other
license_name: minimax-h3-community-license-agreement
license_link: LICENSE
base_model: MiniMaxAI/MiniMax-H3
base_model_relation: quantized
tags:
  - minimax-h3
  - text-to-video
  - image-to-video
  - text-to-audio-video
  - fp8
  - modelopt
  - mixed-precision
  - vllm-omni
---

# MiniMax-H3 ModelOpt Mixed Dynamic FP8

This repository contains mixed BF16/FP8 derivatives of both MiniMax-H3 video
generation partitions:

- **FL2VA**, at the repository root, supports T2VA/FL2VA and targets an
  effective precision of 9.0 bits.
- **Ref2VA**, under `Ref2VA/`, supports reference-conditioned video and audio
  generation and targets an effective transformer precision of 9.25 bits.

The checkpoints target single-GPU vLLM-Omni serving. CPU component offload is
recommended on GPUs that cannot hold all resident components.

> **License:** these derivatives remain subject to the MiniMax H3 Community
> License Agreement in [LICENSE](LICENSE), including its territorial,
> redistribution, notice, and acceptable-use requirements. Review that license
> before using or redistributing this model.

## Runtime format

| Setting | Value |
|---|---|
| Weight format | FP8 E4M3 for selected Linear weights; BF16 otherwise |
| Weight scaling | Per-output-channel |
| Activation scaling | Dynamic per-token |
| ModelOpt algorithm | `FP8_PER_CHANNEL_PER_TOKEN` |
| Kernel | CUTLASS FP8 in vLLM-Omni |

“Mixed9” and “Mixed9.25” are parameter-weighted effective precisions. They do
not denote scalar 9-bit dtypes.

## FL2VA Mixed9

The root checkpoint was selected using BF16-vs-dynamic-FP8 relative MSE.

| Component | Achieved precision | FP8 groups | FP8 parameters | BF16 groups | BF16 parameters |
|---|---:|---:|---:|---:|---:|
| H3 transformer | 8.997527 bits | 203 | 28,988,080,128 | 63 | 4,129,456,128 |
| Qwen3-VL language encoder | 8.994409 bits | 166 | 21,349,007,360 | 34 | 3,030,384,640 |

### FL2VA transformer scope

Kept in BF16:

- `proj_in`, `audio_proj_in`, `context_embedder`, both timestep embedding
  linears, `proj_out`, and `audio_proj_out`.
- Attention output projections in transformer blocks 29-48.
- FFN input/gate-up projections in blocks 28, 31, 32, 33, 41, 45, and 46.
- FFN output projections in blocks 5, 6, 8-10, 12-14, 19, 20, and 30-48.
- Biases, normalization parameters, RoPE buffers, and other non-linear state.

Quantized to dynamic FP8:

- All Q/K/V projections and AdaLN linears in transformer blocks 0-49.
- All Linear projections in both token-refiner blocks and `norm_out.linear`.
- Attention and FFN projections not listed in the BF16 sets above.

### Shared Qwen3-VL text encoder scope

Kept in BF16:

- Token embeddings, RMSNorms, rotary state, all non-linear parameters, and the
  complete Qwen3-VL vision encoder.
- Attention `o_proj` in language layers 24, 25, 30, 31, 35, 37, 38, 40, and
  42-49.
- MLP `down_proj` in language layers 17-19, 21-30, 34, 36, 46, 48, and 49.

Quantized to dynamic FP8:

- All Q/K/V and MLP gate/up projections in language layers 0-49.
- Every attention `o_proj` and MLP `down_proj` not listed above.

The video VAE, audio VAE, tokenizer, processor, embeddings, normalization
layers, and vision encoder remain unquantized.

Exact selections and sensitivity scores are stored in:

- `transformer/transformer_mixed_precision_config.json`
- `transformer/transformer_sensitivity_ranking.tsv`
- `text_encoder/text_encoder_mixed_precision_config.json`
- `text_encoder/text_encoder_sensitivity_ranking.tsv`

## Ref2VA GlobalGrad Mixed9.25

The Ref2VA transformer was independently selected from BF16 using an
output-probed global-gradient sensitivity score. This is a custom search
inspired by mixed-precision AutoQuant; it is not an NVIDIA ModelOpt AutoQuant
export. ModelOpt performs the dynamic FP8 conversion and export.

| Setting | Value |
|---|---:|
| Target effective precision | 9.25 bits |
| Achieved effective precision | **9.248862 bits** |
| FP8 groups / Linear modules | 212 / 296 |
| FP8 candidate parameters | 27,947,630,592 |
| BF16 groups / Linear modules | 54 / 74 |
| BF16 candidate parameters | 5,169,905,664 |

### Ref2VA transformer scope

| Projection family | Dynamic FP8 blocks | BF16 blocks |
|---|---|---|
| Q/K/V | 0, 2-40 | 1, 41-49 |
| Attention output | 0-4, 6-38 | 5, 39-49 |
| FFN input/gate-up | 1-38, 40 | 0, 39, 41-49 |
| FFN output | 0, 2-40 | 1, 41-49 |
| AdaLN Linear | 2-47 | 0-1, 48-49 |

Also quantized to dynamic FP8:

- All Q/K/V, attention-output, and FFN Linear projections in both token-refiner
  blocks.
- `norm_out.linear`.

Always retained in BF16:

- `context_embedder`, `proj_in`, `audio_proj_in`, both timestep embedding
  linears, `proj_out`, and `audio_proj_out`.
- Biases, normalization parameters, RoPE buffers, and all other non-linear
  state.

Ref2VA reuses the Mixed9 Qwen3-VL text encoder described above. Its VAEs,
tokenizer, processor, embeddings, normalization layers, and vision encoder are
not quantized. The exact transformer decisions are stored in:

- `Ref2VA/transformer/transformer_mixed_precision_config.json`
- `Ref2VA/transformer/transformer_sensitivity_ranking.tsv`

## Similarity samples

These are deterministic short regression samples, not comprehensive
perceptual-quality benchmarks.

| Partition | Task | Resolution | Steps | Seed | Video SSIM vs BF16 | Video PSNR | Audio spectral cosine |
|---|---|---:|---:|---:|---:|---:|---:|
| FL2VA Mixed9 | T2VA | 672 x 384 | 10 | 1101 | **0.858496** | **24.7968 dB** | not measured |
| Ref2VA Mixed9.25 | Ref2VA | 672 x 384 | 10 | 3101 | **0.758005** | **22.234802 dB** | **0.996300151** |

For the Ref2VA sample, FP8 peak GPU memory was 84,842 MiB versus 131,308 MiB
for BF16, saving 46,466 MiB (35.39%). The request used the same extracted
reference frame/audio and saved BF16 baseline, produced 107 frames, and used a
4-second requested duration. Full machine-readable results are included in:

- `evaluation/t2va_bf16_similarity.json`
- `evaluation/ref2va_globalgrad9p25_bf16_similarity.json`

## vLLM-Omni serving

This checkpoint requires vLLM-Omni with MiniMax-H3 ModelOpt mixed-FP8 loading
support.

FL2VA/T2VA:

~~~bash
CUDA_VISIBLE_DEVICES=0 \
vllm-omni serve feizhai123/MiniMax-H3-ModelOpt-Mixed9-Dynamic-FP8 \
  --omni \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --enforce-eager \
  --force-cutlass-fp8 \
  --enable-cpu-offload \
  --stage-init-timeout 1800 \
  --init-timeout 2400
~~~

Ref2VA:

~~~bash
hf download feizhai123/MiniMax-H3-ModelOpt-Mixed9-Dynamic-FP8 \
  --local-dir ./MiniMax-H3-ModelOpt-Mixed9-Dynamic-FP8

CUDA_VISIBLE_DEVICES=0 \
vllm-omni serve ./MiniMax-H3-ModelOpt-Mixed9-Dynamic-FP8/Ref2VA \
  --omni \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --enforce-eager \
  --force-cutlass-fp8 \
  --enable-cpu-offload \
  --stage-init-timeout 1800 \
  --init-timeout 2400
~~~

## Modification notice

Selected H3 transformer and Qwen3-VL language-model Linear weights were
modified from the original MiniMax-H3 checkpoints by mixed BF16/FP8
quantization. The VAEs, Qwen3-VL vision encoder, tokenizer, processor, and
other explicitly retained parameters remain in their original precision.