Wiself/Holodeck-Lounge-Regressed-MTP

Holodeck-Lounge — MTP Head Fine-Tune (Learning Experiment)

TL;DR — a documented negative result. This model is the MTP-restored Holodeck-Lounge (a working native single-layer DeepSeek-style MTP head: the merge already carried 14 of the head's 15 layers, and we restored it by sourcing only the missing mtp.fc.weight tensor from Jackrong/Qwopus3.5-9B-Coder, with the froggeric v22.1 chat template baked in) — with only that MTP head re-trained in isolation (backbone frozen) on self-distilled data. The fine-tune regressed MTP acceptance versus the donor head — 49.75% vs 58.55% under rejection-sampling acceptance. A failure consistent with what the literature predicts. It is shared as a learning artifact, not a production model.

📚 Directly related work

This experiment is a direct, small-scale test of claims from these papers:

Paper	Claim we tested
DeepSeek-V3 Technical Report	Native single-layer MTP head for speculative decoding (the exact head we fine-tuned)
Better & Faster LLMs via Multi-token Prediction	Multi-token prediction foundations and acceptance behavior
On Multi-Token Prediction for Efficient LLM Inference	Head-only fine-tunes regress on hidden-layer-specialized backbones; LR must scale down with model size
AdaMTP: An Adaptive Training Paradigm for MTP	Fine-tune corpora that shift distribution can regress MTP below the base — even NTP fine-tunes do
Multi-Token Prediction via Self-Distillation	Distill teacher distributions, not CE on self-generated tokens
MTP-D: Self-Distillation for Multi-Token Prediction	Head-only MTP distillation (gradient-detached KL to main-head Top-10k logits, stop-grad)
FastMTP	Single-layer MTP head fine-tune on 389K self-distilled samples (LR 5e-5, 3 epochs)
Bebop: Breaking Entropy Bounds (Qwen team)	Acceptance degradation during post-training is driven by target-model entropy, not weight mismatch; a one-time TV-loss adaptation on small data keeps MTP effective without retraining
EAGLE-3	The draft-head baseline that beats native MTP heads (≈61% on Qwen3.5-9B)

⚠️ REGRESSION WARNING — research artifact, not for production

• This model's MTP head regressed versus the donor head — 59.0% / 58.55% at our standard eval (reasoning off). The original reaches 60–69% only with reasoning enabled; this fine-tune does not, in any configuration.
• The native MTP head here has lower acceptance than the donor head, offers no quality or speed benefit, and the shared backbone is untouched.
• Use the original model (Wiself/Holodeck-Lounge-MTP) for any real work.
• This model exists to document why such fine-tunes fail — and how to avoid it.

What we did

• Base: Wiself/Holodeck-Lounge-MTP — Holodeck-Lounge backbone (≈9.6B params, qwen3_5, 32 layers, hidden 4096) + transplanted single-layer MTP head (3 draft tokens) + froggeric v22.1 chat template.
• Fine-tuned only the MTP head (~300M params), backbone fully frozen, for 1 epoch on 5,000 self-distilled samples (greedy generations from the model itself).
• Loss: lk_hybrid (adaptive λ·KL + (1−λ)·TV, internal η=3, fused Triton impl).
• Signal caveat: the self-distilled targets were greedy hard labels, not teacher logits — 2602.06019 and MTP-D both distill teacher distributions (KL), which we did not.
• LR 2e-4, cosine schedule, 100 warmup steps, seq len 1536, RTX 5090 (32 GB), via vllm-project/speculators scripts/train.py.

Results — trained vs donor

Metric	Donor	Trained
llama.cpp greedy / target-only acceptance	59.0%	48.69%
vLLM rejection-sampling acceptance	58.55%	49.75%
RS pos0 / pos1 / pos2	77.2% / 57.0% / 41.4%	72.2% / 47.5% / 29.5%
Mean accepted length	2.756	2.493

Eval: 30 prompts, temperature 0.7, max 256 tokens, concurrency 8. llama.cpp numbers are greedy target-only acceptance (reasoning off); vLLM RS acceptance from vllm:spec_decode_num_accepted_tokens / _num_draft_tokens.

Read: the trained head loses roughly 10 points on aggregate acceptance; the deepest draft position (pos2) collapses from 41% to 30%.

Eval caveat: n=30 is a small sample — treat the 3-decimal precision as indicative; confidence intervals are wide.

Why it regressed — what the literature predicts

1. Learning rate too hot. On MTP for Efficient LLM Inference reports LR 6e-6 at 2.8B and LR shrinking with model size; a ~9B head-only fine-tune extrapolates to ~1e-6–3e-6. Our 2e-4 is 1–2 orders of magnitude hotter, destroying the head's pre-existing distribution alignment.
2. Corpus distribution shift. AdaMTP found that on Qwen2.5-7B all fine-tuning paradigms — including plain NTP — regressed below the base model, attributed to the fine-tune corpus, not the MTP paradigm. Our 5k self-distilled corpus diverges from the model's pretraining/eval distribution the same way.
3. Heads-only ceiling. 2502.09419 shows hidden layers are NTP-specialized; head-only adaptation has a hard ceiling and cannot match a head co-trained with the backbone.
4. We deviated from the proven recipes. Self-distillation (2602.06019) uses teacher likelihoods at LR 1e-5; MTP-D uses KL-to-main-head with stop-grad; FastMTP uses weighted CE at LR 5e-5 for 3 epochs — on ~389K samples, ~80× more than ours. We used a hot lk_hybrid loss (λ·KL + (1−λ)·TV) on self-generated tokens for 1 epoch — a maximally hostile combination.
5. Target entropy, not just weights. Bebop (Qwen team) shows MTP acceptance collapse during post-training is dominated by the target model's entropy, and prescribes a one-time TV-loss adaptation on a small corpus, then freeze. A hot LR over greedy hard targets is the regime their TV loss is designed to avoid.

Caveat — the experiment is confounded. With LR 2e-4 + greedy hard targets + 5k samples + 1 epoch, we cannot isolate which factor dominated. This documents combined failure modes, not a clean ablation.

How to use

Honestly: don't. This head is strictly worse than the donor's. The backbone text quality is unchanged (the MTP head only affects speculative decoding), so this is safe, but it is strictly slower to verify than the original. Use the original at Wiself/Holodeck-Lounge-MTP for MTP acceleration — it delivers a 60–69% acceptance rate (65+% typical) with the recommended settings (see its Real-world usage section), which this fine-tune does not.

Training details

Item	Value
Base model	MTP-restored Holodeck-Lounge (see Wiself/Holodeck-Lounge-MTP)
Fine-tuned part	MTP head only (~300M params), backbone frozen
MTP head donor	Jackrong/Qwopus3.5-9B-Coder (mtp.fc.weight only — the missing 15th layer; 14 mtp.layers.* blocks already present in the merge)
Chat template	froggeric/Qwen-Fixed-Chat-Templates v22.1
Loss	lk_hybrid (λ·KL + (1−λ)·TV, η=3)
LR	2e-4, cosine, 100 warmup
Epochs	1
Samples	5,000 (self-distilled, greedy)
Seq len	1536
Hardware	RTX 5090 (32 GB)
Framework	vllm-project/speculators train.py (torchrun)
Base license	apache-2.0

Provenance & credits

Component	Source
Base architecture	Qwen/Qwen3.5-9B (Qwen team)
Backbone merge	nightmedia/Qwen3.5-9B-Holodeck-Lounge — 13-model creative-writing merge
MTP head — 14 mtp.layers.* blocks + mtp.norm/mtp.pre_fc_norm_*	Present in nightmedia/Qwen3.5-9B-Holodeck-Lounge merge
MTP head — mtp.fc.weight (input-fusion projection, the missing 15th layer)	Jackrong/Qwopus3.5-9B-Coder
Chat template (v22.1)	froggeric/Qwen-Fixed-Chat-Templates

AI was used to draft this report.