card: point the opening at the 32-step soul render, add the Russian one

8c5652c verified about 21 hours ago

10 kB

library_name: cortiq
license: apache-2.0
base_model:
  - MiniMaxAI/MiniMax-Music3
base_model_relation: quantized
pipeline_tag: text-to-audio
tags:
  - cmf
  - cortiq
  - music
  - audio
  - 4-bit

MiniMax-Music-3 → CMF — 20.3 GB of weights in one 5.55 GB file

cortiq music minimax-music3-q4tp.cmf \
  --prompt "Classic 1960s soul, passionate male tenor with rich vibrato, \
lush female backing vocals, gospel choir harmonies, vintage Motown and \
Stax atmosphere, groovy bassline, warm Hammond organ, horn section, \
analog tape saturation, romantic nighttime mood." \
  --lyrics "[verse]
Baby when the midnight comes around
I still hear your footsteps on the ground
[chorus]
Oh, come back to me" \
  --seconds 20 --steps 32 --seed 7 --out song.wav

Listen to that command's output. One binary, one file, no Python.

The lyrics are not English-only — the same command in Russian, caption and all. And steps buy audible quality: the same soul prompt at 16 is where the sibilance stops being distracting, 32 is where it settles.

MiniMax-Music-3 generates music with vocals from a caption and lyrics. This is its Comfy-Org repackage converted into the CMF container: one memory-mapped file holding the AR stack, the DiT and the vocoder, read by cortiq, a Rust binary with no ML framework underneath.

	source	here
AR stack (Qwen3-8B + RVQ depth decoder)	15.56 GiB bf16	—
flow-matching DiT	4.58 GiB fp16	—
DAV vocoder	207 MiB	—
total	20.34 GiB, three files	5.55 GiB, one file

915 tensors, cortiq verify clean. The tokenizer travels inside it.

What is in the file

stack	packed as	status
AR — Qwen3-8B backbone (36 layers, 32 q heads / 8 kv of 128), three embedding tables, pruned audio head, 4-layer RVQ depth decoder	`q4tp`, norms exact	implemented, tested
DiT — 36 layers, 32×64 heads, GEGLU 8192, flow matching	`q4tp`, norms and both 1×1 convs exact	implemented, tested
DAV vocoder — ×8 ×8 ×4 ×2 to 44.1 kHz stereo	exact throughout	implemented, tested

The vocoder is not quantized on purpose. It buys tens of megabytes and costs audible hiss — a lesson the H3 conversion already paid for once.

What the conversion had to get right

None of these are visible in a tensor name, and each is wrong in a way that still produces plausible output. They came from reading ComfyUI's comfy/ldm/minimax_music/, and several of them corrected a guess made from the weights alone.

The vocoder's residual dilations are 1, 3, 9 — not BigVGAN's usual 1, 3, 5.

Its Snake activation reads α verbatim. The H3 vocoder in the same engine keeps α and β in log scale and exponentiates on load; feeding one model's parameters to the other's loader silently raises the activation to an exponent. Music-3 gets its own plain Snake rather than a shared loader.

The 128 latent channels are a stereo pair of 64. decode() folds [b,128,t] to [b·2,64,t] and unfolds to [b,2,-1]. The vocoder config's latent_channels: 128 against dec_in_proj [1024,64,1] reads like evidence of a second VAE sitting between them. There is none — it is left and right.

The DiT's input is [x | zeros_like(x) | condition] on the channel axis, 128 + 128 + 2048 = 2304, which is what preprocess_conv's width was saying. The middle plane is a slot the reference leaves empty, not padding.

Both 1×1 convs are residual (conv(x) + x), the timestep embedding is prepended as a token — carried through all 36 blocks, dropped before project_out, and shifting every latent frame's rotary position by one — and the output is negated. A sampler stepping the wrong way still moves and still decodes.

RoPE covers only the first 32 of each head's 64 dims, split-half: the pair for i < 16 is x[i], x[i+16].

The two fused projections are not the same shape. Every projection in the AR checkpoint is fused where the engine wants them split, and the LM's qkv_proj is a GQA fuse — 32 query heads and 8 key/value heads of 128, hence 6144 rows rather than 3×4096 — while the depth decoder's is 12288 = 3×4096 with no grouping. Splitting the first as if it were the second gives a model that loads, runs and is wrong, so the packer DERIVES the key/value head count from the row count instead of assuming it.

What is verified, and how

Not "it produced something". Each stack is gated on quantities the reference fixes exactly, so a stack that has quietly lost an input fails rather than degrades.

Vocoder — 12 latent frames must decode to exactly 6144 samples per side (512 per frame), inside a tanh range, above silence, and left must differ from right. That last one is what catches the stereo fold being read as one wide latent, which decodes noise at half the length. Measured: 6144 samples/side, rms 0.065, L−R max 0.177.

DiT — the velocity must be [128, n], finite, and must respond: zeroing the condition has to change it (catches a wrong concatenation) and moving the timestep has to change it (catches a dropped token). Measured: rms 0.563, d/dcond 0.254, d/dt 2.445.

Depth decoder — it must be CAUSAL, since that is the only thing its attention mask does and losing it lets a codebook level attend to its own answer while the model keeps sampling plausible codes. Measured: 8 codebooks, head spread 6.188.

The chain — noise → sampler → vocoder has to land on the sample count all three agree about, frames × 512 × 2, because the σ walk, the DiT's timestep convention and the vocoder's hop all feed it.

And then the ear, which is the only judge of the last mile. The sample above is what came out; by numbers it is 3382 zero crossings a second (music sits in the low thousands, white noise above ten), band energy 0.42/0.25/0.25/0.08 from low to high, and an envelope with real onsets — sd/mean 0.71, 124 of 149 windows active.

How it generates

The AR stack does not encode the prompt, it generates the conditioning. A Qwen3-8B backbone is prefilled at batch two — the words, and a copy whose middle is replaced by <|audio_cfg|> — then sampled one audio frame at a time at 25 fps: c0 from the pruned head under classifier-free guidance at 1.5, with the top-k mask taken from the CONDITIONED logits, then seven more codebooks through the depth decoder, each fed back through its own embedding table. The eight hidden states of that frame are what the DiT sees, softmax-mixed by cond_layer_logits.

Then an ordinary Euler flow walk over the latent — σ from 1 to 0, the DiT asked at 1 − σ, windowed 689 frames at a time with a 344 hop and the overlap averaged, exactly as the reference does it — and the vocoder turns each latent frame into 512 stereo samples.

Two places where this deliberately is not the reference, both marked in the source:

The top-k sampler is a plain xorshift, not torch's seeded Generator. Reproducing torch.multinomial bit-for-bit is its own project, and nothing here needs one seed to mean the same song across implementations — only that a seed means one song in this one.
The lyrics normaliser skips the reference's markdown scrubbing. That step only ever removes characters a caption should not carry.

What it costs

Every render prints where its time went. On an RTX 3090, 4 s at 4 steps:

stage	GPU	CPU only
AR	52.3 s	44.2 s
denoise	32.9 s	36.2 s
vocoder	18.2 s	31.3 s
total	103.5 s	111.8 s

The vocoder was 54.0 s until its convolutions stopped shipping their column matrix across the bus — the host built it, transposed it into a second buffer of the same size and uploaded that, up to 2.37 GB for a 20-second song, when the input it expands from is k times smaller. It is expanded on the card now.

Two things that table will not tell you. The CPU column is a 256-core EPYC, so an ordinary machine's fallback is far slower than this and the device gap far wider. And the shape that matters for real songs is not this one: attention over latent frames is quadratic, so a 20-second render at 32 steps spends about 80% of its time in the denoise, around 32-40 s a step. CMF_MUSIC3_PROF=1 splits a step four ways if you want to see it.

Earlier, on an Apple M4, 5 s at 8 steps took 206 s: AR 0.45 s/frame, denoise 6.5 s/step over 430 latent frames, vocoder 19 s — and the vocoder was 75 s before it was handed the thread pool.

Running it

cargo install cortiq-cli          # 0.5.74+
hf download infosave/MiniMax-Music-3-cmf minimax-music3-q4tp.cmf --local-dir .
cortiq verify minimax-music3-q4tp.cmf     # → ✓ all tensor hashes match
cortiq music minimax-music3-q4tp.cmf --prompt "..." --lyrics "..." \
  --seconds 20 --steps 32 --seed 7 --out song.wav

--seconds is a ceiling: the model can stop earlier. Same seed, same prompt, same song. Repacking from the original sources:

cortiq animate-pack \
  --music-te  minimax_music3_text_encoder_pruned_bf16.safetensors \
  --music-dit minimax_music3_dit_fp16.safetensors \
  --music-vae minimax_music3_dav.safetensors \
  --quant q4tp --out minimax-music3-q4tp.cmf

Provenance

Weights derive from MiniMax's Music-3 release as repackaged by Comfy-Org, under their own licence. The conversion conventions were read from ComfyUI's comfy/ldm/minimax_music/ and MiniMax's own configs. The CMF container and the cortiq runtime are Apache-2.0.