--- license: apache-2.0 base_model: - Qwen/Qwen3.5-9B - nightmedia/Qwen3.5-9B-Holodeck-Lounge - Jackrong/Qwopus3.5-9B-Coder tags: - multi-token-prediction - speculative-decoding - qwen3.5 - mtp - merge --- # Qwen3.5-9B-Holodeck-Lounge (MTP-restored) > **TL;DR** A working **native Multi-Token Prediction (MTP) head** on top of the > Holodeck-Lounge merge. The upstream merge lineage shipped **without a > functional MTP head** (the `mtp.fc.weight` tensor is absent after merge), so > we restored it by sourcing only that **single missing tensor** — > `mtp.fc.weight` — from > [Jackrong/Qwopus3.5-9B-Coder](https://huggingface.co/Jackrong/Qwopus3.5-9B-Coder). > The other **14 of the MTP head's 15 layers** (the `mtp.layers.*` blocks) were > already present in the merge. We also patched in the > [froggeric/Qwen-Fixed-Chat-Templates](https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates) > fixed jinja template (**v22.1**), which improves reasoning > structure and MTP acceptance in our evals. This model is the reference donor for our > [head fine-tune experiments](https://huggingface.co/Wiself/Holodeck-Lounge-Regressed-MTP). ## Why this model exists - [nightmedia/Qwen3.5-9B-Holodeck-Lounge](https://huggingface.co/nightmedia/Qwen3.5-9B-Holodeck-Lounge) is a fine creative-fiction merge. Excellent writing model — but its merge lineage (13 base models, see the upstream card for the full recipe) does not carry a working MTP head: `mtp.fc.weight` is missing, so native speculative decoding is impossible from the source. - **What we did:** the merge already carried **14 of the MTP head's 15 layers** (the `mtp.layers.*` transformer blocks); only the input-fusion projection `mtp.fc.weight` was absent. We sourced that **single tensor** from **Jackrong/Qwopus3.5-9B-Coder**, transplanted it into the Holodeck-Lounge backbone, and verified it loads and runs end-to-end in llama.cpp (`draft-mtp`) and vLLM (`qwen3_5` MTP method). Backbone weights untouched. - **Chat template:** baked in the fixed jinja template v22.1 from [froggeric/Qwen-Fixed-Chat-Templates](https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates) (Qwen 3.5/3.6/3.8), improving reasoning structure, organisation, coherence, and MTP acceptance rate in our evals. ## Native MTP head - Single-layer, DeepSeek-style MTP head (as in the [DeepSeek-V3 Technical Report](https://arxiv.org/abs/2412.19437)), **3 draft tokens**, fixed adjacent context. - The 14 `mtp.layers.*` blocks come from the Holodeck-Lounge merge; only the `mtp.fc.weight` fusion projection was sourced from Jackrong/Qwopus3.5-9B-Coder (a model whose checkpoint carries the same native Qwen3.5 MTP architecture). - The donor projection plus the 14 host layers reach the acceptance range below — higher than any of our head-only fine-tunes ([related experiment →](https://huggingface.co/Wiself/Holodeck-Lounge-Regressed-MTP)). ## Evaluation | Metric | Value | |---|---| | llama.cpp greedy / target-only acceptance | **59.0%** | | vLLM rejection-sampling acceptance | **58.55%** | | RS pos0 / pos1 / pos2 | 77.2% / 57.0% / 41.4% | | Mean accepted length | 2.756 | Eval: 30 prompts, temperature 0.7, max 256 tokens, concurrency 8. llama.cpp numbers are greedy target-only acceptance (reasoning off); vLLM RS acceptance from `vllm:spec_decode_num_accepted_tokens / _num_draft_tokens`. With reasoning enabled per the Usage section, measured acceptance reaches the **60–69%** range (n=30, same eval set). > **Eval caveat:** n=30 is a small sample — treat the 3-decimal precision as > indicative; confidence intervals are wide. ## Real-world usage With the recommended settings — `draft-mtp` in llama.cpp, `qwen3_5` MTP in vLLM — the restored head has consistently delivered a **60–69% draft-token acceptance rate** (65+% typical) in our testing. Real-world mileage will vary with workload and hardware; treat 65% as a strong baseline, not a guarantee. The point of this model isn't raw speed, but **option**: MTP when it pays off, plain greedy decoding when it doesn't. Where the win matters most: - Bandwidth-constrained serving, where accepted draft tokens cut tokens-per-request at the wire. - High-concurrency, latency-sensitive inference, where saved decode steps shrink queue and p95. - Settings where every accepted draft token compounds — long generations, reasoning-heavy prompts, tight SLOs. The clean part: the host checkpoint already carried **14 of the MTP head's 15 layers**, so this model ships at essentially the same size as its host — it was always that big. The single-tensor transplant just completes the missing `mtp.fc.weight` fusion layer, so you get the choice in one model, at no real size cost. Thanks to nightmedia for the Holodeck-Lounge backbone (which already carried 14 of the MTP head's 15 layers), to Jackrong/Qwopus3.5-9B-Coder for the `mtp.fc.weight` tensor that completed the restore, and to froggeric for the Qwen-Fixed-Chat-Templates v22.1 baked in here. ## Usage (recommended) For best performance, use the baked-in froggeric template with reasoning enabled and preserved: ```bash llama-server -m model.gguf \ --jinja --chat-template-file chat_template.jinja \ --reasoning-format deepseek --reasoning on --reasoning-preserve \ --spec-type draft-mtp --spec-draft-n-max 3 ``` - **llama.cpp MTP:** `--spec-type draft-mtp --spec-draft-n-max 3` - **vLLM MTP:** `--speculative-config '{"method":"mtp","num_speculative_tokens":3}'` ## Provenance & credits | Component | Source | |---|---| | Base architecture | [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) (Qwen team) | | Backbone merge | [nightmedia/Qwen3.5-9B-Holodeck-Lounge](https://huggingface.co/nightmedia/Qwen3.5-9B-Holodeck-Lounge) — 13-model creative-writing merge (DavidAU heretic family, armand0e, microsoft/Fara1.5-9B, Jackrong, et al.; see upstream card) | | MTP head — 14 `mtp.layers.*` blocks + `mtp.norm`/`mtp.pre_fc_norm_*` | Present in [nightmedia/Qwen3.5-9B-Holodeck-Lounge](https://huggingface.co/nightmedia/Qwen3.5-9B-Holodeck-Lounge) merge | | MTP head — `mtp.fc.weight` (input-fusion projection, the missing 15th layer) | [Jackrong/Qwopus3.5-9B-Coder](https://huggingface.co/Jackrong/Qwopus3.5-9B-Coder) | | Chat template (v22.1) | [froggeric/Qwen-Fixed-Chat-Templates](https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates) | License: **apache-2.0**. License chain (all components apache-2.0, verified): Qwen/Qwen3.5-9B → nightmedia merge → Jackrong MTP head → froggeric template. ## Limitations - Acceptance trails dedicated trained draft heads such as [EAGLE-3](https://arxiv.org/abs/2503.01840) (≈61% on Qwen3.5-9B). - Deepest draft positions (pos2) accept at lower rates than pos0/pos1. - The transplanted head is inference-accelerator only; it does not affect backbone text quality. - The head is only directly usable as a built-in MTP head for this architecture (same hidden size + vocab); as a standalone GGUF it can also be used as a sidecar draft (`--model-draft mtp-*.gguf`). **AI was used to draft this report.**