--- library_name: cortiq license: apache-2.0 base_model: - Comfy-Org/MiniMax-H3 - larryvrh/MiniMax-H3-Turbo-Lora base_model_relation: quantized pipeline_tag: text-to-video tags: - cmf - cortiq - video - audio - 4-bit --- # MiniMax-H3 Turbo — one 23.5 GB file, no Python [MiniMax-H3](https://huggingface.co/Comfy-Org/MiniMax-H3) renders video and synchronized stereo audio from one prompt, in one transformer, on two flow schedules. [larryvrh's Turbo LoRA](https://huggingface.co/larryvrh/MiniMax-H3-Turbo-Lora) brings it to four sampling steps. This is both of them in the [CMF container](https://github.com/infosave2007/cmf) — the DiT, the Qwen3-VL prompt encoder, the video VAE decoder and the audio vocoder in a single memory-mapped file — running on `cortiq`, a Rust binary with no ML framework underneath. | | reference checkout | here | |---|---|---| | diffusion model | 66.3 GB (bf16) | — | | prompt encoder | 51.5 GB (bf16) | — | | video + audio VAE | 5.8 GB | — | | Turbo LoRA | 0.8 GB | — | | **total** | **124.4 GB, four files + a ComfyUI checkout** | **23.5 GB, one file** | 47.83 B parameters, 2 361 tensors, `cortiq verify` clean. ## What comes out ![A corgi in a chef hat over a pan, four-step render](https://huggingface.co/infosave/MiniMax-H3-Turbo-cmf/resolve/main/samples/corgi_512x288_4step.gif) *"A corgi in a chef hat flipping a pancake, sizzling sounds and a cheerful bark."* — 512×288, 39 frames at 24 fps, seed 42, **four steps**, nothing but the prompt. The GIF is silent; the audio is the point, so take the **[mp4](https://huggingface.co/infosave/MiniMax-H3-Turbo-cmf/resolve/main/samples/corgi_512x288_4step.mp4)**. It is not a second model: the same transformer denoises both streams in one packed sequence, on two different flow schedules. [`samples/`](https://huggingface.co/infosave/MiniMax-H3-Turbo-cmf/tree/main/samples) also holds the AVI `cortiq animate` actually wrote and its `.wav` — the mp4 and the GIF are remuxes for the browser, and the runtime itself never touches ffmpeg. The LoRA is not a separate download: it is merged into the weights, so the file IS the 4-step model. **Text-to-video and keyframe-to-video.** Prompt in, video and audio out; or give it a first and/or last frame and it continues from there. The release's third path — `ref2va`, conditioning on reference images, clips and audio — is not ported. | file | size | | |---|---|---| | `mmh3-turbo-fl2va-q4tp.cmf` | 23.94 GB | **use this one** | | `mmh3-turbo-fl2va-q2tp.cmf` | 18.74 GB | two bits on the gate/up planes. Smaller, faster, and it stops following the prompt — kept for anyone who wants to push on it, not for rendering. See below | ## Keyframe to video ![The corgi flipping the pancake, started from one frame](https://huggingface.co/infosave/MiniMax-H3-Turbo-cmf/resolve/main/samples/i2v_corgi_flip.gif) ```bash cortiq animate mmh3-turbo-fl2va-q4tp.cmf \ --prompt "the corgi lifts the pan and flips the pancake high, sizzling" \ --first-frame keyframe.ppm --out flip.avi ``` One picture conditions the run twice, and both halves matter. Its VAE latent becomes a row the DiT holds at a timestep of its own near 1 — a condition, not noise being removed — and never denoises. The picture ITSELF goes to the prompt encoder through Qwen3-VL's vision tower, as `": "` and a vision block: at 512×288 that is 144 tokens of the 168 the prompt above carries. Leave one out and the model is conditioned on something the reference never conditions on. `--last-frame` anchors the other end. The first frame is a geometry anchor and is stretched to the canvas; the last one follows and is cover-cropped, which is what the reference does with each. Frames come in as binary P6 PPM. ## Running it ### 1. Get the runtime `cortiq` is one Rust binary. Either install it — ```bash cargo install cortiq-cli # needs Rust 1.85+; brings the GPU backend ``` — or take a prebuilt archive from the [latest release](https://github.com/infosave2007/cmf/releases/latest) (Linux x86-64, macOS on Apple Silicon and Intel, Windows x86-64 and ARM64; each ships a `.sha256`). Nothing else is required: no Python, no PyTorch, no CUDA toolkit, no ffmpeg. Check it took: ```bash cortiq --version ``` ### 2. Get the weights One file, 23.5 GB. ```bash pip install -U "huggingface_hub[cli]" # only to fetch the file hf download infosave/MiniMax-H3-Turbo-cmf mmh3-turbo-q4tp.cmf --local-dir . ``` Confirm it arrived whole — the container carries a hash per tensor: ```bash cortiq verify mmh3-turbo-q4tp.cmf # → ✓ all tensor hashes match cortiq info mmh3-turbo-q4tp.cmf # → arch, layers, 47.83B params ``` ### 3. Render ```bash cortiq animate mmh3-turbo-q4tp.cmf \ --prompt "A corgi in a chef hat flipping a pancake, sizzling sounds and a cheerful bark." \ --width 512 --height 288 --frames 39 --steps 4 --seed 42 \ --out corgi.avi ``` That writes `corgi.avi` — MJPEG video with PCM stereo, playable in VLC, mpv, QuickTime and Windows Media Player — and `corgi.wav` beside it. The JPEG encoder and the RIFF muxer are inside the binary: a pipeline that ends in a shell-out to a 20 MB dependency is not a pipeline you can ship. If you want an mp4 for a browser, remux it yourself; the model never needs one. **On a GPU.** Nothing to opt into any more: `cortiq` probes this file's own first qkv weight against the host on startup and takes the device arm only if they agree, so the arm that renders is the arm that was checked. ```bash cortiq animate mmh3-turbo-q4tp.cmf --prompt "…" --out corgi.avi ``` On one RTX 5090, 512×288, 39 frames: **60.2 s at the default four steps** (91.6 s at eight, 42.8 at two). `RUST_LOG=info` prints a per-stage breakdown for every run, and the full table is under *What it costs to run* below. The whole pipeline stays on the card — the DiT block, both VAE decoders, and the vocoder's dilated convolutions — so nothing but the finished frames crosses the bus. `CMF_MMH3_GPU=0` forces the host path if you want to compare. **Two cards.** A render does not split across them, and should not: the DiT block and both decoders are already resident, so a second card would only add a bus crossing to a pipeline that no longer has one. Two cards double your *clips*, not your clip — run two processes, one pinned to each: ```bash CMF_GPU_ADAPTER=0 cortiq animate model.cmf --prompt "…" --seed 1 --out a.avi & CMF_GPU_ADAPTER=1 cortiq animate model.cmf --prompt "…" --seed 2 --out b.avi & wait ``` `cortiq gpu` lists the cards and their indices. For text models the same binary both splits and replicates across cards — see [docs/MULTI_GPU.md](https://github.com/infosave2007/cmf/blob/master/docs/MULTI_GPU.md). ### Options that matter | flag | default | what it does | |---|---|---| | `--width` / `--height` | 512 × 288 | multiples of 32. The trained short edge is 768; below ~256 the model drifts off-distribution | | `--frames` | 39 | at 24 fps, snapped **up** to the model's 17k+5 grid: 5, 22, 39, 56, … 124. 124 ≈ 5 s, and 124–362 is the validated range | | `--steps` | 4 | what the Turbo LoRA is trained for. More still helps a little | | `--seed` | 42 | same seed, same prompt, same size → the same clip, byte for byte. This is now true on the GPU too: the op arbitration used to alternate arms on real data while it made up its mind, and two runs of one binary could differ | | `--quality` | 92 | JPEG quality of the AVI's frames | | `--stock-sampler` | off | integrate the audio on the video's clock, as a single-schedule sampler does. Wrong at 4 steps — it is here to hear how wrong | | environment | what it does | |---|---| | `CMF_MMH3_GPU=1` / `=0` | force the device or the host path instead of letting the parity probe choose | | `CMF_GPU_PROBE=0` | pin the op arbitration (already the default for `animate`, so a seed reproduces) | | `CMF_THREADS=n` | cap the worker pool (defaults to the machine's cores) | | `CMF_ANIM_PROF=1` | per-step rms of both latent streams and both velocities | ### What it needs RAM at least the file's size — 24 GB — or every step faults on non-resident pages; the weights are memory-mapped, not read. Disk: 24 GB. A GPU is optional and wants ~14 GB of VRAM for the DiT's planes. No network access at run time. ## Two bits: smaller, faster, and answering a different question The obvious next cut is the DeepSeek-V4 policy: gate/up at two bits, everything else at four. It builds — 23.94 GB down to **18.74**, and with a device kernel of its own it renders *faster* than the four-bit file — 217.3 s against 258.4 when the pair was measured, because there is less weight to move. (Both numbers are from the build of that day; the four-bit file renders the same clip in 60.2 s now. The ratio is what carries over, not the seconds.) `cortiq verify` passes. The file is here. It also stops following the prompt, which is why it is not the one to reach for. | | | |---|---| | ![four bits](https://huggingface.co/infosave/MiniMax-H3-Turbo-cmf/resolve/main/samples/ab_q4tp.gif) | ![two bits](https://huggingface.co/infosave/MiniMax-H3-Turbo-cmf/resolve/main/samples/ab_q2tp.gif) | | `q4tp` — 23.94 GB | `q2tp` — 18.74 GB | Same prompt, same seed, same four steps. On the left the corgi is behind a pan with batter in it, drawn flat and clean, which is what was asked for. On the right it is a different animal in a different style with **no pan and no pancake at all**, over a washed-out ground with visible texture noise. That is not a quantizer trading detail for size; that is a model answering a different question. The likely culprit is where the two bits landed. Half this file is the PROMPT ENCODER, and the policy put two bits on its gate/up planes along with the DiT's — so the loss falls on the part that decides what the clip is about, not on the part that draws it. A two-bit build confined to the DiT would save ~2.9 GB instead of 5.2 and is the version worth measuring next — the packer's policy is one predicate, `is_wide_plane`, if you want to try it. The file above is published so that experiment starts from something rather than nothing; four bits is what to render with. ## What it costs to run The file is memory-mapped, so plan on RAM at least its size or every step touches non-resident pages. One RTX 5090, 4 steps, 39 frames. `RUST_LOG=info` prints this breakdown for every run: | | text | denoise | video VAE | audio VAE | total | |---|---|---|---|---|---| | 512×288 | 2.5 s | 34.8 s | 16.6 s | 4.3 s | **60.2 s** | | 512×256 | 2.3 s | 31.4 s | 8.4 s | 4.3 s | **48.5 s** | | 256×160, 22 frames | | | | | **15.9 s** | | 512×288, host only (`CMF_MMH3_GPU=0`) | 2.6 s | 363.5 s | 271.5 s | 7.1 s | **646.1 s** | The card is **10.7× the host path** on the same machine, same seed. At 8 steps a 512×288 clip is **91.6 s**; at 2 it is 42.8. The same 4-step render took 172 s when this card was written — the pipeline has since moved onto the card end to end, both VAE decoders with it. Nearly all of the decode is the video VAE — the vocoder is 4.3 s of it. The packed sequence is `[text | audio | video]` and everything attends to everything, so cost grows with the token count and then with its square: a 512×288 second is five times the tokens of a 256×160 one. **A free 2× on the decoder, if you want it.** The video VAE decodes in 256-pixel tiles, always, and grows the OVERLAP rather than the tile count — so a 288-pixel edge is covered by two 256-pixel tiles overlapping by 224, and you pay for 512 rows to get 288. An edge of exactly 256 is one tile. 512×256 therefore decodes three tiles where 512×288 decodes six, for 89% of the pixels. Measured on the current build: the video VAE goes 16.6 s → 8.4 s and the whole render 60.2 s → 48.5 s. The schedule is the reference's and this port reproduces it exactly; picking an edge that lands on it is free. **Host and device do not agree to the last bit, and neither is wrong.** The host arm quantizes activations to int8 (`CMF_SDOT`) where the device dequantizes to f32, so the two renders differ by a few per cent in latent rms and visibly in fine texture. Set `CMF_SDOT=0` on both sides to compare arithmetic instead of that approximation. **Why the device is opt-in.** Getting it right took three fixes, and one thing is still held back. The engine's blocked f32 GEMM cached its weight-side device buffer **by pointer address**. Every batched attention allocates one k/v scratch pair per call and refills it per head — same address, different matrix — so head 0's keys came back for every head, on the GPU only, silently. It is keyed on a content fingerprint now. The same GEMM also took every job over 4 M MACs on sight with no CPU arm to lose to, which on this model's decoder was three times *slower* than the host it displaced; it goes through the same measure-don't-assume probe as every other op class now, and on this stack the probe hands that work back (0.24 ms device against 0.13 host) while sending the weight GEMMs to the card (25.8 ms against 92.0). Still held: **the cooperative-matrix kernel runs this model out of f16 range.** At 256×160 the render is correct; at 512×288 the audio stream goes NaN on the second sampling step and the video follows. Bisected — `CMF_BAKE_GPU=0` does not help, `CMF_COOP=0` does — so `cortiq animate` pins `CMF_COOP=0`. That hold is specific to this model, not a verdict on the kernel: the image model on the same card and the same kernel renders 20.5 s without it against 14.8 with, and the two agree to 42.6 dB — the price of f16 operands, which the kernel documents, not a fault. MiniMax-H3's activations are simply larger. Giving that kernel a scale is the next real speedup here. ## What the conversion did **The adaLN collapse.** Forty per cent of the released DiT is one matrix per block: `adaln_proj.linear` is `[96768, 2688]`, 520 MB at bf16, **13 B of the model's 33 B parameters** — for a map whose input is one number, the timestep. Its output over the whole schedule is a one-dimensional curve in R^96768, and Comfy-Org's `pruned` checkpoints already ship it as one: an `adaln_t_table` of `[1025, 8]` shared by every block and per-block weights of `[96768, 8]`. Measured against the full matrix on block 0 (`tools/mmh3_fetch.py check`, which range-reads 520 MB out of the 66 GB file rather than downloading it): ``` adaln max|Δ| 8.0e-4 rms 8.7e-5 against a signal of rms 0.464 time-curve singular values 1..12, relative: 1.00e0 2.96e-1 1.05e-1 6.63e-2 6.60e-3 2.11e-3 5.61e-4 2.92e-4 3.67e-5 2.73e-5 1.32e-5 1.34e-6 ``` The ninth singular value is already 3.7e-5 of the first. Rank eight is not an approximation anyone should feel nervous about; the 26 GB is redundant. The Turbo LoRA is written against the FULL matrix (`lora_A` is `[16, 2688]`), which is why the ComfyUI node re-injects the time conditioning at run time when the base is pruned. `cortiq animate-pack` does it once, at conversion: ``` adaln(t) = W_p · u(t) + b + B · (A · silu(e(t))) = [W_p | B] · [u(t) ; A · silu(e(t))] ``` — a rank-24 curve, driven by a `[1025, 24]` table per block. 4.6 MB a block instead of 520, with the LoRA already inside it. **The rest.** - **Backbone** — bf16 → `q4tp`, 4.16 bits a weight with a predicted per-row scale ladder. The LoRA's rank-64 update is merged before quantizing. - **Prompt encoder** — Qwen3-VL-32B truncated to 50 layers, 51.5 GB → 12.2 GB. It is the largest single component of the file and it runs once per generation. - **Video VAE — decoder only.** It is a ViT3D, not a conv stack: 36 transformer blocks over the latent grid and one linear that expands each cell into a 4×16×16 block of pixels. The 3-D causal CNN encoder is a third of the checkpoint and text-to-video never runs it. - **Audio VAE — decoder only**, f16. Quantizing a vocoder buys 45 MB and costs audible hiss. Its 254 kaiser-sinc resampling filters are read from the checkpoint rather than re-derived — the design formula is in the code as a fallback, but a filter you compute is a filter that can drift from the one the weights were trained against. - Integrity: 47.83 B parameters over 2 361 tensors; `cortiq verify` checks every one against the directory's hashes. ## On parity Established, not assumed, and separately for each of the four stacks. The reference is ComfyUI's own module, run on a toy checkpoint carrying the release's real tensor names and the release's real schedules — `tools/` builds them, `tools/mmh3_toy_gate.sh` runs the diff. The packs are exact f32 on purpose: `q4tp`'s noise floor sits an order of magnitude above the arithmetic difference these are looking for, so quantizing here would pass a broken port. | stack | worst | rms | signal rms | |---|---|---|---| | DiT — video velocity | 8.8e-5 | 2.1e-5 | 0.515 | | DiT — audio velocity | 5.2e-5 | 2.5e-5 | 0.409 | | DiT — token refiner | 8.3e-7 | 2.6e-7 | 1.003 | | Qwen3-VL encoder | 1.1e-6 | 3.3e-7 | 0.812 | | video VAE decoder | 4.2e-7 | 4.0e-8 | 0.470 | | audio VAE decoder | 1.7e-9 | 3.5e-10 | 8.9e-4 | A dozen conventions in this model pass at one token and fail differently at a hundred, which is why the toys are not one-vector unit tests: the packed layout's cursor, the video time axis's 1,4,4,4,4 span pattern, which 96 of 128 head dimensions rotate, the adaLN row order (timestep-major, modality-minor), the video VAE's 256-pixel tiling — global attention makes a tile a different computation from a whole frame, so the tiling is part of the output, not a memory strategy — and the audio stream's separate clock. ## Two clocks The video and audio latents ride different flow schedules (shift 12 and 3). The sampler walks the video grid, which at four steps is `1, 0.973, 0.923, 0.8, 0`, and integrates the audio on its own remap of it. Stepping both on the video grid is what a stock sampler does; it is fine at twenty steps and wrong at four, because over the last interval Δσ_a and Δσ_v differ by a factor of three and no per-step slope correction survives a step that large. `--stock-sampler` reproduces the broken behaviour if you want to hear it. ## Provenance Weights derive from MiniMax's H3 release as repackaged by Comfy-Org, and from larryvrh's Turbo LoRA; both remain under their own licences. The Turbo LoRA is a **preview** — its own card notes plastic-looking skin and over-sharp grain at `ckpt850`, and nothing here changes that. The CMF container and the cortiq runtime are Apache-2.0 (see the repository's LICENSE and PATENTS.md).