Masked MTP drafts
With LLMs it seems that progress is not intuitive: acceptance dropped, and the tok/s got higher. Full draft head: 66.7% acceptance, 117.1 tok/s. Trimmed head: 63.6%, 121.7 tok/s. +3.9% end to end, on an early K=3 held-out run.
I knew it would release some compute before I ran it, it simply wasn't intuitive. The sequence was ordinary: I measured, saw I'm compute bound, checked, and saw the draft head is pushing the limit. I can't do both, the full head read and the step. So I looked for ways to shrink the compute. And masking worked. The surprising part was: shrinking by almost 7x keeps the acceptance pretty close, even though it's not even close to the full vocab options. 66.7% to 63.6%. That's a small drop for cutting the head from 248,320 rows to 32,768. Acceptable.
Not my idea. It comes from FR-Spec (ACL 2025). In short: when speculative decoding is used, the draft model's vocabulary can be trimmed to a frequency-ranked subset of the full vocab. The verification is the same, but it simply can't offer tokens outside this subset. It simply dies immediately if it tries to, and the next token is generated from the full model. There's less options to consider, so the acceptance drops a little, but the compute is much cheaper. The only difference is that the paper's authors measured it on a separate draft model, smaller, which I also tried, but it requires training and the head is already there, so why not just use it. So I wired it for native MTP.
That read is especially expensive on this model, and its family. Qwen3.8-27B has a 248,320-token vocabulary, and the MTP head produces a logit for all of it, every draft step, but on average the model will never say most of them. Keep the top 32,768 by what the model actually emits and the head read shrinks 248,320 / 32,768 = 7.6x. That's the trick. When each draft step gets that much cheaper you can afford a few percent acceptance drop.
So the credit really is: FR-Spec is the paper's idea. My part is wiring it for native MTP, the multi-token-prediction head that ships inside the checkpoint and not a separate draft model, and publishing a card you can actually download and run. And this write-up.
Important: it's not what people mostly use, it's specifically what the model probably emits. The ranks come from the model's own generations, 163k tokens of agentic output and 154k of prose, with external text used as prompts only.
It kept its truth on re-measure too. memra v0.86.2, interleaved x5 against the full embedded head: +5.1% on an RTX PRO 6000, 137.9 vs 131.2 decode p50. +6.4% on an RTX 5090 Laptop, 73.6 vs 69.2. And the full head still accepts more, 0.76 vs 0.74. Loses anyway. But - measured on three Qwen 3.6 models and two Gemmas, plus the 3.8 here. The per-model drafts, rank files and board rows are public on memra-bench. Two things that sweep taught me: ranks don't transfer between models, I measured -12 acceptance points when I tried; and requantizing the trimmed head to NVFP4 costs zero measured acceptance. I can't claim more without measuring.
Two ways to run it in memra, because checkpoints come in two shapes.
GGUF wants a separate pre-trimmed draft file. You attach it:
MEMRA_MTP_DRAFT=mtp-Qwen3.8-27B-NVFP4-frspec-sxc32768.gguf \
memra-server --model Qwen3.8-27B-NVFP4-Q5K-mtp.gguf
Safetensors doesn't need a separate file. Drop a .txt ranks card next to the model and memra trims the head weight at load:
MEMRA_FRSPEC_TRIM=<ranks.txt>
Why two mechanisms for one trick? GGUF is what people download, so the trim is baked into the file and nobody needs conversion tooling. The safetensors path leaves the official checkpoint alone and masks at load, so trying a different flavor means swapping one text file. The .txt is literally the list that produced the trimmed GGUF.
If memra isn't your engine, steal the idea anyway. I opened a llama.cpp issue for it: Research: FR-Spec-style draft-vocab trimming for native MTP speculative decoding. Research plus a branch, to be clear, not something I shipped there. Most of the plumbing already exists in their EAGLE-3 support: a d2t index mapping, scatter-after-matmul. That's exactly what a frequency-ranked subset needs. Native trimmed MTP just has to reuse it.
Everything downloadable sits on the Hugging Face card: pre-trimmed draft GGUFs in agentic, prose and mixed flavors, plus the ranks lists for the safetensors path. 32,768 ranked token ids each, and the agentic one is my serving default. Based on my usage. And it makes a difference.
- Draft GGUF:
mtp-Qwen3.8-27B-NVFP4-frspec-sxc32768.gguf, plus prose and mixed flavors. - Ranks lists:
q38-ranks-sxc32768.gguf.txt(agentic),q38-ranks-prose-32768.txt,q38-ranks-mixed-32768.txt.
For short generic stuff it doesn't really matter. My quick checks put all three inside noise of each other, acceptance around 0.40 on prose text and 0.59 to 0.62 on code, whichever ranking is loaded. One good read is that the top of the ranking is almost the same across domains and the differences only live in the tail. Agentic workloads take the agentic list, prose takes prose, and if you have no idea, mixed exists so you can stop thinking about it.
I'm probably already measuring the next head by the time you read this.