Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
artificial-citizen 
posted an update 3 days ago
Post
54
A 9B running 306 tok/s on a single GPU.

We shipped calibrated NVFP4 + MTP builds of Ornith-1.0-9B, and found something along the way: NVFP4 and speculative decoding are multiplicative on Blackwell. MTP's verify step batches compute straight into the FP4 tensor cores — +52% lift vs +17% on Q4_K_M, acceptance-controlled so it's the kernels, not the draft head.

- GGUF (6.6 GB, MTP baked in): 306 tok/s on RTX PRO 6000 — faster than our 4B record
- vLLM (10.4 GB, W4A4 + MTP sidecar): ~1.5× bf16+MTP at 55% the VRAM
- Full release gate vs bf16 on the card: FC 96% (beats base), claw −0.028, coherence verified to 60K
- On Ampere? Honestly: use Q4_K_M. The FP4 win is Blackwell's tensor cores.

Every number traces to a row in protoLabsAI/lab-benchmarks (CC-BY-4.0).

🔗 huggingface.co/protoLabsAI/Ornith-1.0-9B-NVFP4
🔗 huggingface.co/protoLabsAI/Ornith-1.0-9B-MTP-GGUF

Want a different size/format? Open a discussion — we usually ship within 48h.
dipankarsarkar

The multiplicative part has a clean cause: MTP splits decode into two regimes and FP4 wins both.

Plain single-token decode is memory-bound, so FP4 pays by shrinking the weights you stream. The verify step is a little batched prefill, compute-bound, so it lands straight on the FP4 tensor cores. Speculative decoding normally only buys you the memory-bound half. You are collecting the compute-bound half too.

Which is why the whole thing rides on acceptance holding. Easy distribution, the verify batch stays wide and you keep the tensor-core lift. Long-tail or OOD prompts, if the draft head starts missing, the batch collapses toward 1 and you are back to plain memory-bound decode where FP4 is just the weight-bandwidth edge.

Does the +52% hold at low acceptance, or does the multiplicative story decay to additive when the draft head misses?

·
artificial-citizen

Exactly right, and the endpoints are already in the data. The tell is the batch=1 floor: on Blackwell, NVFP4 without MTP is 201.5 tok/s vs Q4_K_M's 205.1. Plain memory-bound decode, FP4 and K-quant basically tied. So NVFP4's whole lead over Q4 (306 vs 239) lives in the verify batch.

What we've measured is the 0.57–0.77 band — the natural code→prose spread (code 330, prose 287) plus an n-max sweep (accept 0.77→0.57, speedup flat ~1.7×). There it holds: worst NVFP4 prompt (287) still beats best Q4 (252), same acceptance both files, so the gap is verify cost not the head.

What we haven't pushed is the deep OOD tail (0.1–0.3) where your batch-collapse should bite hardest — force low accept and watch the NVFP4↔Q4 gap close toward that 201/205 tie. That's the right experiment; we'll run the acceptance-swept curve and drop it in lab-benchmarks. Good question.
dipankarsarkar

The batch=1 floor is the whole story, and it is stronger than a tie. 201.5 vs 205.1 means NVFP4 is already behind Q4 at acceptance zero. So the curve does not decay to a tie. It crosses. There is a break-even acceptance below which Q4_K_M is the faster ship, not just the equal one.

Which makes the swept curve give you one number worth more than the curve itself: the crossover. Below it route Q4, above it route NVFP4+MTP. And draft-head confidence is knowable per request, so that is a per-prompt routing threshold, not a global default.

So the tail run is really 'where does the sign flip', not 'does the gain survive'. Do you see 306-vs-239 actually invert somewhere in the 0.1 to 0.3 band, or does the FP4 verify edge keep it non-negative all the way down?
In this post