--- license: apache-2.0 base_model: NullSense/Nanbeige4.2-3B-FP8-Dynamic base_model_relation: adapter pipeline_tag: text-generation language: [en, zh] tags: [eagle3, speculative-decoding, draft-model, speculators, vllm, looped-transformer, nanbeige] --- # Nanbeige4.2-3B-EAGLE3 EAGLE-3 draft model (speculator) for [Nanbeige/Nanbeige4.2-3B](https://huggingface.co/Nanbeige/Nanbeige4.2-3B), trained against my [FP8-Dynamic quant](https://huggingface.co/NullSense/Nanbeige4.2-3B-FP8-Dynamic) as the verifier. This is the thinking-aware revision: the target defaults to reasoning (`enable_thinking: true`), so I regenerated the training corpus on-policy with thinking on and retrained. Against the prior head, batch-1 decode acceptance in thinking mode rises from 0.33 to 0.41 (+23%) and thinking-mode throughput from 244 to 272 tok/s (+12%) on an RTX 5090, with non-thinking gaining too. Speculative decoding is lossless: outputs are verified by the target model, so quality is exactly the verifier's. I have not found another published draft model for a looped or weight-shared LLM. ## The acceptance gap this fixes The first head reached val acceptance 0.622 but its 21k-conversation corpus was almost entirely non-thinking generations, while production serves reasoning by default. A drafter learns the distribution it trains on, so the prior head accepted *fewer* draft tokens in thinking mode (0.33) than in plain mode (0.41): the exact traffic it serves most was the traffic it fit worst. EAGLE-3 acceptance scales with training data that matches the serving distribution, so the fix is data, not architecture. ## The looped-arch part (why this needed new wiring) The target executes 22 layers twice per token (`num_loops: 2`), so the "layer axis" the draft taps is the 44-step unrolled virtual depth, not the 22 physical layers. Three things follow, all carried by this repo: 1. **Aux feature taps are virtual-depth ids**: `eagle_aux_hidden_state_layer_ids = [3, 23, 41]`: early loop-1, early loop-2, late loop-2. Extraction for training adds the last virtual layer (44) for the loss target. 2. **Serve-time layer naming must offset by virtual depth**: vLLM's Eagle3 draft names its layer at `num_hidden_layers` (22); this collides with the target's loop-2 layer 22, and stamps `target_layer_count = 22`, which mis-indexes the draft's `layer_types`. The bundled `vllm_plugin/` fixes both (offset 44, stamp 44). 3. Speculation pays double on this arch: the target reads every weight twice per token, and verification amortizes both passes across the whole draft block. ## Training [speculators](https://github.com/vllm-project/speculators) online training (v0.6.0.dev), hidden states extracted on-the-fly from the FP8 verifier served by vLLM v0.25.1. The corpus is 28,648 conversations, of which 12,886 (45%) carry reasoning traces: - **21,299 workload conversations** (my serving mix: summarization, RAG answer generation, dictation rewrite, freeform chat, plus a code slice), replayed to keep the prior head's coverage and prevent forgetting. - **7,349 on-policy thinking regenerations**: GSM8K-style reasoning and multi-turn chat prompts, answered by the FP8 target itself with thinking on (T=0.6, top-p 0.95), so the draft sees the target's own reasoning tokens, not a different model's. 4 epochs, total-seq-len 4096, draft vocab 32k, single RTX 5090 shared between extract-server and trainer (~29 GB total). This is a full retrain on the combined corpus, not a warm-start on the old head. The continual-learning result is that retraining on D_old plus D_new is the upper bound, and the high replay ratio already protects the prior coverage. ![Training curves](assets/eagle_train_curves.png) | epoch | val loss | val cond_acc (pos 0 / 1 / 2) | |---|---|---| | 1 | 5.88 | 0.584 / 0.589 / 0.608 | | 2 | 4.99 | 0.625 / 0.632 / 0.650 | | 3 | 4.29 | 0.659 / 0.667 / 0.684 | | **4 (this repo)** | **3.85** | **0.685 / 0.697 / 0.715** | The val split now includes thinking traces, so it's harder than the prior head's split. This head clears 0.622 by epoch 2 anyway, and acceptance was still climbing at epoch 4. ## Measured serving results (RTX 5090, vLLM v0.25.1, batch-1, T=0.6) Both heads measured back-to-back under identical conditions. Real reasoning and code prompts, thinking mode is the production default: | mode | metric | prior head | **this head** | |---|---|---|---| | thinking | draft acceptance | 0.332 | **0.409** (+23%) | | thinking | output tok/s | 244 | **272** (+12%) | | non-thinking | draft acceptance | 0.413 | **0.445** | | non-thinking | output tok/s | 277 | **289** (+5%) | Synthetic GuideLLM sweep (synchronous, 256 output tokens, both heads re-measured under the same unlocked clocks) shows the same direction across context lengths: | input length | prior head tok/s | **this head tok/s** | |---|---|---| | 512 | 198 | **280** (+41%) | | 2048 | 152 | **171** (+12%) | | 8192 | 75 | **83** (+11%) | Thinking-mode serve acceptance by draft position: 0.61 / 0.37 / 0.25, about 2.2 tokens per verify cycle. Position 2 (the 3rd draft) still pays at 0.25; a 4th rarely does, which is why `num_speculative_tokens` stays at 3. One workload still belongs to ngram speculation, and it is a property of the traffic, not the head: long-document summarization with verbatim copy-spans (2-14k-char inputs) favors prompt-lookup drafting over a learned 3-token draft. For bulk long-doc jobs, run the ngram config; eagle3 stays the default for reasoning, agentic, RAG, chat, and code traffic. ![Long-document spec comparison](assets/chart_spec_longdoc.png) Unlike ngram speculation on this stack, eagle3 keeps async scheduling, Model Runner V2, and full CUDA-graph decode. One caveat that cost me an afternoon: **use `TRITON_ATTN` as the attention backend**: FlashInfer + spec-decode downgrades CUDA graphs to piecewise on SM120 and the head then measures *slower* than no speculation at all. ## Serving (vLLM) ```bash pip install --no-deps ./vllm_plugin # arch + looped-draft fixes (not upstream yet) VLLM_ATTENTION_BACKEND=TRITON_ATTN \ vllm serve NullSense/Nanbeige4.2-3B-FP8-Dynamic --trust-remote-code \ --max-model-len 65536 --kv-cache-dtype fp8 \ --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml \ --speculative-config '{"method":"eagle3","model":"NullSense/Nanbeige4.2-3B-EAGLE3","num_speculative_tokens":3}' ``` - `num_speculative_tokens 3`: position-2 acceptance is 0.25, position 3 rarely pays its latency. - The head was trained against the FP8-Dynamic verifier. It should transfer to the bf16 original and the other quants in the family (same tokenizer, near-identical distributions) but I have only measured it on FP8-Dynamic. Download just this artifact: ```bash hf download NullSense/Nanbeige4.2-3B-EAGLE3 --local-dir Nanbeige4.2-3B-EAGLE3 ``` ## Limitations - Acceptance measured at T=0.6 on my workload mix. The corpus is summarize/RAG/rewrite weighted with reasoning and code slices; traffic far from that mix may accept differently. - Requires the bundled plugin until the arch lands upstream ([vLLM PR #49433](https://github.com/vllm-project/vllm/pull/49433)); the two looped-draft serve fixes in it are not upstream anywhere yet. - Draft vocab is 32k (pruned from 166k); rare-token drafting relies on the fallback mapping. - English+Chinese target; training data English-only. ## The family | artifact | role | |---|---| | [FP8-Dynamic](https://huggingface.co/NullSense/Nanbeige4.2-3B-FP8-Dynamic) | verifier / recommended serving quant | | [NVFP4-FP8-LoopShield](https://huggingface.co/NullSense/Nanbeige4.2-3B-NVFP4-FP8-LoopShield) | 4-bit quant, reasoning at FP8 parity | | [NVFP4A16](https://huggingface.co/NullSense/Nanbeige4.2-3B-NVFP4A16) | fastest quant, summarization-scoped | | [EAGLE3 (this repo)](https://huggingface.co/NullSense/Nanbeige4.2-3B-EAGLE3) | draft model: faster decode on top of any of them | ## Links & provenance - Base model: [Nanbeige/Nanbeige4.2-3B](https://huggingface.co/Nanbeige/Nanbeige4.2-3B) - Trained with [speculators](https://github.com/vllm-project/speculators) (online extract_hidden_states pipeline, vLLM v0.25.1) - Arch serving support: [vLLM PR #49433](https://github.com/vllm-project/vllm/pull/49433) plus this repo's `vllm_plugin/` (registers the arch out-of-tree and patches the Eagle3 draft for looped targets) - Training config: `train_command.txt`; per-epoch validation: `val_metrics.json` ## Citation ```bibtex @misc{peciukonis2026nanbeige42eagle3, author = {Pe{\v{c}}iukonis, Matas (NullSense)}, title = {Nanbeige4.2-3B-EAGLE3: an EAGLE-3 draft model for a looped transformer via virtual-depth feature taps}, year = {2026}, howpublished = {Hugging Face}, url = {https://huggingface.co/NullSense/Nanbeige4.2-3B-EAGLE3}, note = {Virtual-depth aux taps (3/23/41 of 44) and looped-target draft-naming fixes; thinking-aware on-policy retrain for reasoning-default serving.} } ```