cortexist/gemma-4-12B-it-assistant-MTP-GGUF

Gemma 4 12B — MTP Assistant (speculative drafter) · GGUF

GGUF builds of google/gemma-4-12B-it-assistant — the Multi-Token Prediction (MTP) draft head for Gemma 4 12B — for use as a speculative-decoding drafter in the cortexist/llama.cpp fork.

⚠️ This is not a standalone model. It is a small draft head that must be loaded alongside the Gemma 4 12B target model. On its own it produces nothing useful. Paired with the target it does lossless speculative decoding — the greedy output is byte-for-byte identical to running the target alone, just faster when the target's decode is memory-bandwidth-bound.

Files

File	Precision	Size	Notes
gemma-4-12B-it-assistant-MTP-Q4_K_M.gguf	Q4_K_M	~0.31 GB	token_embd Q6_K, rest Q4_K — recommended
gemma-4-12B-it-assistant-MTP-F16.gguf	F16	~0.82 GB	full-precision conversion

Acceptance is essentially identical between the two quants (the head only proposes token ids the target then verifies), so Q4_K_M is the practical choice.

Requirements

• The cortexist/llama.cpp fork. Upstream llama.cpp does not currently support this head (the converter for the gemma4_unified_assistant checkpoint and the gemma4_assistant runtime graph live in the fork).
• A Gemma 4 12B target GGUF — e.g. your own conversion of google/gemma-4-12B-it. The drafter's backbone width (3840) must match the target's embedding length.

Usage

./build/bin/llama-server \
  -m  gemma-4-12B-it-Q4_K_M.gguf \
  -md gemma-4-12B-it-assistant-MTP-Q4_K_M.gguf \
  --spec-type draft-mtp --spec-draft-block-size 2 \
  -ngl 99 -ngld 99 -fa on \
  -ctk f16 -ctv f16 -ctkd f16 -ctvd f16 \
  -c 4096 --host 0.0.0.0 --port 8080

• -md / --model-draft points at the drafter (mainline-style; --mtp-head is an accepted alias).
• --spec-type draft-mtp enables the MTP path; --spec-draft-block-size B makes the head draft B − 1 tokens per round (B = 2 is the sweet spot — see below).
• Works the same in llama-cli and llama-speculative-simple.

How it works

The drafter is a small 4-layer, query-only transformer that cross-attends the target model's own KV cache and seeds from the target's last backbone hidden state. Each round it proposes B − 1 tokens; the target verifies them in a single batched forward pass and accepts the longest correct prefix. Because the target does the accepting, output is identical to non-speculative greedy decoding — speculation only changes speed, never content.

Benchmarks — when does MTP actually help?

Speculative decoding pays off only when the target's verify step is memory-bandwidth-bound, so batching the drafted tokens through it is nearly free. On a fast, compute-bound GPU there's no spare bandwidth to exploit and the extra draft + wider-verify compute is a net cost. The 12B drafter shows this split cleanly.

Methodology: scripts/bench-gemma4-mtp.sh from the fork — Gemma 4 12B Q4_K_M target + this drafter, B = 2, -fa on, -ngl 99, 3 prompts × 256 tokens, greedy (temperature 0, deterministic), throughput and acceptance read from the server's own timings. "Baseline" = target with no draft head loaded.

Jetson Orin NX (bandwidth-bound → MTP wins)

KV cache	Baseline tk/s	MTP tk/s	Speedup	Draft accept
f16	7.53	9.89	1.31×	74.8%
turbo3	7.24	8.17	1.13×	72.0%

NVIDIA RTX A5000 (compute-bound → MTP loses)

KV cache	Baseline tk/s	MTP tk/s	Speedup	Draft accept
f16	66.5	61.7	0.93×	75.6%
turbo3	58.2	47.8	0.82×	74.4%

Takeaways

• On the edge (Orin NX), MTP is a real ~1.31× win — and f16 KV + MTP is the best config.
• On a fast desktop GPU (A5000), MTP is a net loss (0.93×): the 12B decode is already compute-bound, so the draft/verify overhead isn't repaid.
• Draft acceptance is ~72–76% on both machines — acceptance is a property of the model + prompt, independent of hardware. Only the throughput payoff flips sign.
• turbo3 KV helps MTP less than f16 (and is even slightly slower at the baseline at this context length): its win is KV-cache VRAM at long context, while its per-step dequant adds compute — counterproductive on an already compute-bound verify. Use turbo3 to fit long context, not to go faster.

If you're deploying on a bandwidth-bound edge device, enable MTP. On a high-end discrete GPU that isn't bandwidth-starved on a model this size, plain decoding is simpler and faster.

Conversion

Converted from the official google/gemma-4-12B-it-assistant safetensors with the fork's convert_hf_to_gguf.py (registered as Gemma4UnifiedAssistantForCausalLM → gemma4_assistant arch), then llama-quantize for Q4_K_M.

License

These are derivative GGUF builds of google/gemma-4-12B-it-assistant. Use is governed by the Gemma 4 license / Terms of Use. All credit for the underlying model belongs to Google DeepMind; cortexist only performed the GGUF conversion/quantization and benchmarking.