| --- |
| license: other |
| license_name: minimax-h3-community-license-agreement |
| license_link: https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/LICENSE |
| tags: |
| - comfyui |
| - nvfp4 |
| - quantized |
| - video |
| - text-to-video |
| base_model: MiniMaxAI/MiniMax-H3 |
| base_model_relation: quantized |
| --- |
| |
| # MiniMax H3 — NVFP4 |
|
|
| NVFP4 quantizations of the **MiniMax-H3** diffusion transformer for ComfyUI — both task |
| variants, `ref2va` (reference images → video) and `fl2va` (first/last frame → video). |
|
|
| - Original model: https://huggingface.co/MiniMaxAI/MiniMax-H3 |
| - ComfyUI repackage these derive from: https://huggingface.co/Comfy-Org/MiniMax-H3 |
|
|
| **NVFP4 requires an NVIDIA Blackwell GPU** (RTX 50-series, RTX PRO 6000, B200). On Ada, |
| Hopper or older the NVFP4 path is emulated — use Comfy-Org's `int8_convrot` files instead. |
|
|
| ## Which file do I want? |
|
|
| First pick the **task**, then the file: |
|
|
| - **`ref2va`** — up to 9 reference images (+ videos/audio) → video. Identity-driven. |
| - **`fl2va`** — first and/or last frame → video. Interpolates between endpoints; also how |
| you chain clips past one generation by feeding the previous clip's last frame in. |
|
|
| | file | size | s/it | VRAM (DiT) | notes | |
| |---|---|---|---|---| |
| | **`minimax_h3_ref2va_pruned_nvfp4.safetensors`** | **12.5 GB** | **1.90** | 11.9 GB | smallest/fastest ref2va | |
| | **`minimax_h3_fl2va_pruned_nvfp4.safetensors`** | **12.5 GB** | — | ~11.9 GB | smallest/fastest fl2va | |
| | `minimax_h3_ref2va_nvfp4_mixed.safetensors` | 24.4 GB | 1.92 | ~20 GB | from unpruned bf16 | |
| | `minimax_h3_ref2va_nvfp4_full.safetensors` | 18.7 GB | 1.91 | ~16 GB | experimental | |
| | `minimax_h3_ref2va_pruned_nvfp4_convrot_int8.safetensors` | 20.1 GB | — | ~20 GB | **mixed-precision, by [rockerBOO](https://huggingface.co/rockerBOO/minimax-h3-nvfp4)** | |
| | `minimax_h3_fl2va_pruned_nvfp4_convrot_int8.safetensors` | 20.1 GB | — | ~20 GB | **mixed-precision, by [rockerBOO](https://huggingface.co/rockerBOO/minimax-h3-nvfp4)** | |
|
|
| Take a **`pruned_nvfp4`** file when footprint and speed matter: half the size of the |
| alternatives at equal-or-better speed, with the modulation path left at full precision. If |
| you have VRAM to spare and want maximum fidelity, read "Honest limitations" first — 4-bit |
| weights appear to cost some motion quality relative to Comfy-Org's 8-bit `int8_convrot`. |
| |
| The two `_convrot_int8` files are **not ours** — they are |
| [rockerBOO](https://huggingface.co/rockerBOO/minimax-h3-nvfp4)'s work, mirrored here for |
| convenience under the same MiniMax community license. A mixed-precision profile quantized in |
| a single pass from bf16: NVFP4 for `blocks.2-46` mlp, FP8 for `attn.qkv_proj`/`adaln_proj`, |
| BF16 left on the layers stored F32 upstream, and `attn.qkv_proj` swapped to INT8 ConvRot |
| quantized fresh rather than restacked. Bigger than ours, better motivated on fidelity. |
| Original repo: https://huggingface.co/rockerBOO/minimax-h3-nvfp4 |
|
|
| The two pruned files are the same size and were produced by the same script over the same |
| 200 quantized layers — they differ only in task head. `fl2va` s/it is unmeasured but should |
| match `ref2va`; the architectures are identical. |
|
|
| ## Why the pruned base is the right one to quantize |
|
|
| Comfy-Org's `pruned` checkpoint is not lossily pruned — it is a **structural refactor of |
| AdaLN**, and understanding it explains the whole table above. |
|
|
| In the bf16 model, AdaLN modulation dominates the parameter count: |
|
|
| | group | bf16 | pruned | |
| |---|---|---| |
| | `adaln_proj` | **13.04B (39.4%)** | **0.04B (0.2%)** | |
| | `mlp` | 12.02B | 11.56B | |
| | `attn` | 8.02B | 7.71B | |
| | token_refiner / norms / embedders | 0.05B | 0.80B | |
| | **total** | **33.12B** | **20.11B** | |
| |
| The bf16 model projects a 5376-dim conditioning vector into modulation parameters per block. |
| The pruned model replaces this with an 8-dim timestep table (`adaln_t_table`, shape |
| `[1025, 8]`) feeding `adaln_proj.linear` of shape `[96768, 8]`. Because modulation depends |
| only on the timestep, that 5376-wide projection was almost entirely redundant — 13.04B |
| parameters collapse to 0.04B, a ~326x reduction. |
|
|
| This matters for quantization because AdaLN is the part you least want to quantize: it emits |
| the scale and shift applied to every residual stream, so error there is multiplicative and |
| compounds across all 50 blocks and every sampling step. In the bf16 model you face a bad |
| choice — protect AdaLN and produce a ~36 GB file (larger than the 34 GB int8 it should beat), |
| or quantize 39% of the model and hope. **In the pruned model the problem disappears**: AdaLN |
| is already tiny, so you keep it at full precision for free and quantize only attn+mlp, which |
| are error-tolerant. |
|
|
| Comfy-Org's `pruned_int8_convrot` quantizes exactly those 200 attn/mlp layers to |
| int8_convrot and leaves everything else alone. `pruned_nvfp4` takes that same set to NVFP4. |
|
|
| ## Measured |
|
|
| RTX PRO 6000 Blackwell (96 GB), ComfyUI 0.30.0, ref2va, 864x480, 39 frames, 20 steps, |
| `res_multistep` / `beta`, three matched seeds. **These are speed/size numbers only** — the |
| quality comparison from the same runs is retracted (see "Honest limitations"). Note also that |
| `beta` was the wrong scheduler; ComfyUI's official H3 templates all use `simple`. |
|
|
| | model | size | staged VRAM | s/it | |
| |---|---|---|---| |
| | `pruned_int8_convrot` (Comfy-Org) | 21.0 GB | 19,995 MB | 2.17 | |
| | **`pruned_nvfp4` (this repo)** | **12.5 GB** | **11,944 MB** | **1.90** | |
| |
| **-40% file size, -8.0 GB VRAM, -12.4% sampling time.** |
| |
| At ~12 GB for the DiT, a 32 GB card (RTX 5090) becomes viable if the text encoder is |
| offloaded to CPU after encoding — it runs once per prompt, not once per sampling step. |
| |
| ## How `pruned_nvfp4` was built |
| |
| **Single pass from `pruned_bf16`.** Comfy-Org's `minimax_h3_{ref2va,fl2va}_pruned_bf16` |
| (40.2 GB) is full-precision weights with AdaLN already refactored into the timestep table, so |
| the 200 attn/mlp layers are quantized once, straight from bf16: |
|
|
| ```python |
| nq, nparams = ckt.TensorCoreNVFP4Layout.quantize(w.contiguous()) # bf16 -> NVFP4 |
| tensors = ckt.TensorCoreNVFP4Layout.state_dict_tensors(nq, nparams) |
| ``` |
|
|
| The layer set is `blocks.<N>.{attn.qkv_proj, attn.out_proj, mlp.fc1, mlp.fc2}.weight` — 50 |
| blocks x 4 = 200, verified set-identical to the layers Comfy-Org quantize in their |
| `pruned_int8_convrot` release. Everything else (AdaLN, norms, embedders, token_refiner, all |
| biases) is copied verbatim at source precision: 332 tensors, 1132 in the output. |
| |
| Per-layer config lives in a `comfy_quant` uint8 tensor holding JSON, written as |
| `{"format": "nvfp4"}`; there is no file-level `__metadata__`. |
|
|
| Full script: `bf16_to_nvfp4.py` in this repo. 200 layers in ~5 seconds on one GPU. |
|
|
| <details> |
| <summary>Earlier builds requantized from int8 (superseded)</summary> |
|
|
| Before Comfy-Org published a pruned bf16 checkpoint, the only pruned source was |
| `pruned_int8_convrot`, so these files were built bf16 -> int8_convrot (Comfy-Org) -> NVFP4 — |
| two lossy passes whose error compounds. That path needed a real dequantize -> requantize, |
| because the StarNodes model converter passes non-floating-point tensors through untouched: |
| |
| ```python |
| if not tensor.dtype.is_floating_point: |
| return tensor # already-int8 weights are copied verbatim |
| ``` |
| |
| so running it on an int8 checkpoint silently produced a byte-identical file. The old script |
| (`pruned_to_nvfp4.py`) is kept for reference. Both files were rebuilt single-pass on |
| 2026-08-05; the output key set and dtype census are unchanged, only the error path is. |
| </details> |
| |
| ## Prompting: H3 wants a structured IR, not prose |
| |
| **Read this before blaming the weights for bad output.** H3 was trained on the structured |
| output of **H3-Context-IR**, a preprocessing model that rewrites a plain request into |
| labelled sections; MiniMax's model card calls it "critical to the quality of the final |
| output". ComfyUI passes your raw string straight to the DiT, so you must write that |
| structure yourself. |
| |
| Official guides: |
| [base](https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/docs/VIDEO_PROMPT_WRITING_GUIDE_base_en.md) · |
| [ref](https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/docs/VIDEO_PROMPT_WRITING_GUIDE_ref_en.md) |
| |
| ```text |
| For the target video, at 0.00 seconds into the target video, <Picture 1> (from [Shot 1]) is fully referenced. |
| |
| integrated_multimodal_description: [Shot 1] Live-action, cinematic, the young woman shown in |
| <Picture 1> remains beside the rain-covered train window, preserving her appearance and the |
| carriage layout. The camera trucks right with small amplitude at slow speed as she lifts her |
| gaze toward the passing city lights. The quiet, breathy young woman (S1) says: |
| <d>[English] I get off at the next station.</d> She folds the letter along its existing crease. |
| |
| overall_soundscape: The train wheels produce a steady metallic rhythm beneath a low |
| ventilation hum. Rain ticks against the window while paper rustles softly in her hands. |
|
|
| non_diegetic_music: Sustained cello notes at a slow tempo with widely spaced piano tones. |
| ``` |
| |
| **Dialogue must be explicit or you get gibberish.** Speech is generated jointly with video, |
| so saying *that* someone speaks without giving the words yields correct prosody and mouth |
| shapes with no lexical content. Speaker identity, action and delivery go *outside* `<d>`; |
| only the language tag and verbatim words go *inside*. Use stable IDs `(S1)`, `(S2)`, and |
| `(S1,S2)` for simultaneous speech. |
| |
| Other essentials: `[Shot 1]` carries no timestamp, later shots use `[Shot N] At MM:SS.mmm`; |
| aim for 350-500 words of description; write camera motion as type + amplitude + speed; |
| reference tags must appear in the order the inputs were connected. ref2va accepts up to 9 |
| reference images, and 3-4 varied shots hold identity far better than one. |
| |
| ## Companion files (mirrored, not ours) |
| |
| | file | precision | origin | |
| |---|---|---| |
| | `text_encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors` | NVFP4-AWQ | **Comfy-Org, unmodified** | |
| | `vae/minimax_h3_video_vae_fp16.safetensors` | FP16 | **Comfy-Org, unmodified** | |
| | `vae/minimax_h3_audio_vae_fp32.safetensors` | FP32 | **Comfy-Org, unmodified** | |
| |
| **The NVFP4 text encoder is Comfy-Org's work, not ours.** Only the |
| `minimax_h3_*_nvfp4*.safetensors` diffusion models here are new. |
| |
| VAEs are deliberately **not** quantized: they are small, run once per generation rather than |
| per step, and decode straight to pixels and audio samples where error is immediately |
| visible. The text encoder being NVFP4 buys VRAM, not speed — it also runs once per prompt. |
| |
| ## Usage |
| |
| ``` |
| 📂 ComfyUI/models/ |
| ├── 📂 diffusion_models/ |
| │ └── minimax_h3_ref2va_pruned_nvfp4.safetensors (or the fl2va file) |
| ├── 📂 text_encoders/ |
| │ └── qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors |
| └── 📂 vae/ |
| ├── minimax_h3_video_vae_fp16.safetensors |
| └── minimax_h3_audio_vae_fp32.safetensors |
| ``` |
| |
| Recommended stack total: **33.7 GB**. Use the official |
| [R2V template](https://github.com/Comfy-Org/workflow_templates/blob/main/templates/video_minimax_h3_r2v.json), |
| swapping the diffusion model — or the |
| [FL2V template](https://github.com/Comfy-Org/workflow_templates/blob/main/templates/video_minimax_h3_fl2v.json) |
| for the `fl2va` file. Requires ComfyUI >= 0.30.0 (native H3 support in |
| `comfy/ldm/minimax/`). `CLIPLoader` type must be `minimax`; sampler `res_multistep`; |
| frame `length` must satisfy 17n+5. |
|
|
| ## Honest limitations |
|
|
| - **NVFP4 is 4 bits/weight and that appears to cost visible quality vs int8_convrot.** In |
| side-by-side playback of 15s clips at 1152x640, the int8 build showed noticeably less |
| mid-motion artifacting and held object shape better through fast pans. If you have the |
| VRAM (~20 GB vs ~12 GB) and can spend ~14% more time per step, Comfy-Org's |
| `pruned_int8_convrot` may be the better choice. This repo's value is smallest-footprint |
| and fastest, not highest-fidelity. |
| - **That observation predates the single-pass rebuild and has not been repeated since.** It |
| was made against the doubly-quantized files, so some of the smearing may have been |
| compounding error rather than 4-bit weights. Treat the int8-is-better lead as untested |
| against the current files. |
| - That comparison is **not controlled**: the two runs also differed in text encoder |
| (nvfp4_awq vs int8_convrot), it was n=1, and matched seeds do not produce matched |
| trajectories across different quantization. Treat it as a lead, not a result. |
| - The earlier claim of "no visible degradation across three matched seeds" is **retracted** — |
| it was measured at 480x864 / 39 frames, too small and too short to show what appears at |
| 768p over 15s. Contact sheets in particular hide motion artifacts; judge on playback. |
| - 15s (362 frames) generates fine; see the VRAM note below. |
| - The **`fl2va` file is unbenchmarked** — it was produced by the same script over the same |
| layer set as `ref2va`, but has not been run end-to-end. |
| - Benchmarks are single-GPU, one card, one resolution, and were measured on the older |
| doubly-quantized files. Size and layout are unchanged by the rebuild, so s/it and VRAM |
| should carry over, but they have not been re-measured. |
| |
| Failure cases are welcome in the discussions tab — concrete artifacts beat aggregate scores. |
| |
| ## VRAM / length ceiling |
| |
| Measured on RTX PRO 6000 (96 GB), NVFP4 pruned DiT, 20 steps. The budget that matters is |
| **pixels x frames**, not resolution alone: |
| |
| | size | px | frames | result | |
| |---|---|---|---| |
| | 768x960 | 737,280 | 362 | ok, 29.51 s/it | |
| | 1152x640 | 737,280 | 362 | ok, 28.73 s/it | |
| | **1344x768** | **1,032,192** | **362** | **OOM — container hard-killed** | |
| |
| The ComfyUI template default (1344x768) is safe at its default 124 frames but **not** at 362. |
| OOM here is a SIGKILL: no traceback, no `/history` entry, and ComfyUI restarts with its queue |
| wiped. A job that silently vanishes from `/queue` is this. |
| |
| ## License |
| |
| Inherits the MiniMax-H3 Community License Agreement from the original model. |
| |
