| --- |
| license: mit |
| base_model: deepreinforce-ai/Ornith-1.0-9B |
| tags: |
| - speculative-decoding |
| - mtp |
| - multi-token-prediction |
| - qwen3.5 |
| - vllm |
| library_name: vllm |
| --- |
| |
| # Ornith-1.0-9B MTP head |
|
|
| An **MTP (Multi-Token-Prediction) speculative-decode head** for |
| [`deepreinforce-ai/Ornith-1.0-9B`](https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B). |
| Ornith-1.0-9B shipped without the `mtp.*` tensors its Qwen3.5-9B base carries, so it serves |
| with no native speculative speedup. This is that head, re-aligned to Ornith's hidden states. |
|
|
| Merge it into Ornith-1.0-9B (one command, below) and serve with vLLM's `mtp` method for a |
| **lossless +49–57% single-stream decode speedup**. Lossless by construction: the base model |
| verifies every drafted token, so the output distribution is unchanged — the head only buys |
| throughput. |
|
|
| ## Results |
|
|
| Measured on a single RTX PRO 6000 Blackwell, vLLM 0.22.1, `num_speculative_tokens=1`. |
| Acceptance is reported on two prompt distributions (coding, and WildBench/ToolACE-style), |
| sampled at T=0.7. |
|
|
| | Head | Accept (coding) | Accept (corpus) | tok/s | Notes | |
| |---|:---:|:---:|:---:|---| |
| | none (plain Ornith-9B) | — | — | ~75 | no MTP | |
| | graft (Qwen head, zero training) | 0.763 | 0.742 | ~117 | free, reproducible below | |
| | **this head (KL-distilled)** | **0.765** | **0.762** | **~121** | best | |
|
|
| Two findings came out of building it: |
|
|
| 1. **The graft is nearly free.** Copying Qwen3.5-9B's MTP head onto Ornith verbatim already |
| gives ~0.74–0.76 acceptance — the fine-tune is light enough that the base head transfers. |
| You can reproduce that head in one command (no download needed); see below. |
| 2. **The training objective is what matters.** Re-distilling the head with hard |
| cross-entropy on sampled tokens *regressed* acceptance (it sharpens the argmax but |
| miscalibrates the distribution). MTP acceptance is rejection sampling against the target, |
| which rewards a draft distribution that *matches* the target — so this head is trained |
| with **KL divergence to the target's own next-token distribution**. Same data, same |
| schedule; only the loss changed. |
|
|
| ## Use |
|
|
| ```bash |
| # 1. merge this head into Ornith-1.0-9B (verbatim tensor copy; ~0.5 GB head, base untouched) |
| hf download protoLabsAI/Ornith-1.0-9B-MTP --local-dir ./ornith-mtp-head |
| python recipe/graft.py \ |
| --donor ./ornith-mtp-head \ |
| --target deepreinforce-ai/Ornith-1.0-9B \ |
| --out ./Ornith-1.0-9B-MTP |
| |
| # 2. serve with vLLM's native MTP method |
| vllm serve ./Ornith-1.0-9B-MTP \ |
| --speculative-config '{"method":"mtp","num_speculative_tokens":1}' |
| ``` |
|
|
| ### Reproduce the zero-training graft (no download) |
|
|
| ```bash |
| python recipe/graft.py --donor Qwen/Qwen3.5-9B \ |
| --target deepreinforce-ai/Ornith-1.0-9B --out ./Ornith-1.0-9B-MTP-graft |
| ``` |
|
|
| ## Recipe |
|
|
| The full, donor-agnostic toolkit is in [`recipe/`](./recipe) — it retargets any Qwen3.5 |
| fine-tune: `graft.py` (transplant the head), `gen_corpus.py` (self-distillation: the target's |
| own generations, no external data), `distill.py` (`loss: kl`, freeze base / train only the |
| head), `eval_head.py` (offline acceptance proxy), `validate.sh`. The recipe is the product. |
|
|
| ## Provenance & license |
|
|
| - Base model: `deepreinforce-ai/Ornith-1.0-9B` (MIT). This head is a derivative; merging it |
| produces a derivative of Ornith-1.0-9B — its MIT terms carry. |
| - The head was initialized from `Qwen/Qwen3.5-9B`'s `mtp.*` tensors, then re-trained on |
| Ornith-9B's own generations. |
| - Released under **MIT** by [protoLabs.studio](https://protolabs.studio). Open core: free to |
| fork, no paywall on the weights or the recipe. |
|
|
| ## Works on quantized bases too |
|
|
| Verified on [`Ornith-1.0-9B-NVFP4`](https://huggingface.co/protoLabsAI/Ornith-1.0-9B-NVFP4) |
| (calibrated W4A4): acceptance **0.76** on real text vs 0.762 on bf16 — quantizing the target |
| costs the draft head nothing, and NVFP4+MTP measures ~1.5x bf16+MTP under identical load. |
| Same one-command merge; or use the NVFP4 repo, which ships this sidecar in-box. |
|
|
| Benchmark rows: [`protoLabsAI/lab-benchmarks`](https://huggingface.co/datasets/protoLabsAI/lab-benchmarks). |
| Want a different variant? Open a Community discussion — usually ~48h. |
|
|
