RESULTS — EAGLE-3 draft for North-Mini-Code-1.0 (cohere2_moe)
1. Best trained draft (champion)
| metric | value |
|---|---|
| exp | exp7_bigdata_ep10 |
| tau (Σ held-out eval acc, 7 positions) | 4.25 (acc0=0.71) |
| config | LlamaForCausalLMEagle3, 1 layer, hidden 2048, target_hidden 2048, FFN 12288, draft_vocab 32000, aux layers [1,23,45] |
| data | ~8.3k code-instruction samples (magicoder-evol-instruct), 10 epochs, lr 1e-4 |
| draft size | 366 MB (bf16) |
Lever ladder findings (offline held-out tau):
- baseline FFN8192/4ep = 3.82 → lr3e-4 = 2.00 (too high ✗) → FFN12288 ≈ +0 (noise) → 10ep = 4.02 → 20ep = 4.10 (epochs saturate) → +more data (8.3k) = 4.25 (winner).
- More data beat more epochs. Next lever = self-distillation on the model's own generations.
2. Real serving in vLLM (North-Mini-Code + our EAGLE draft)
Brought up on vLLM main (0.23.1rc1.dev, nightly) per Cohere guidance, on an A100-80 (driver 580/CUDA13),
with cohere_melody + --tool-call-parser/--reasoning-parser cohere_command4. Required two fixes:
--hf-overrides '{"first_k_dense_replace":1}'— transformers 5.12 drops this field, so vLLM mis-built dense layer-0 as MoE (weight KeyError). Override restores it.- Patched
vllm/model_executor/models/cohere2_moe.pyto implement the EAGLE3 interface (SupportsEagle3: capture aux hidden states from layers [1,23,45],set/get_aux_hidden_state_layers). vLLM's cohere2_moe shipped without it → "Model does not support EAGLE3 interface".
3. vibetest quality (14 qa scenarios, OpenAI endpoint)
| harness set | base (no spec) | + EAGLE |
|---|---|---|
| GPQA-physics G1–G8 (PhD MCQ) | 4/8 | 3/8 |
| Security/exploit X1–X6 | 6/6 | 6/6 |
| total | 10/14 | 9/14 |
EAGLE-3 is distribution-lossless in theory; the ±1 difference is bf16 spec-decode greedy-path divergence on borderline PhD-physics ties (off-domain for a code model), not a real regression. Security set is identical 6/6.
4. EAGLE speed in vLLM (the honest result)
| base | + EAGLE | |
|---|---|---|
| gen throughput | ~118 tok/s | ~38 tok/s |
| mean acceptance length | — | 1.28 |
| per-position accept | — | 0.22, 0.03, 0.001, 0, 0 |
EAGLE was net SLOWER here. With acceptance length only 1.28, the draft+verify overhead
(5 spec tokens) is not repaid → ~3× slowdown. EAGLE only wins when acceptance >2.
Why acceptance is low (≠ training acc0 0.71): train↔serve hidden-state mismatch. The draft was trained OFFLINE on hidden states dumped from HuggingFace transformers; vLLM feeds it its own internal aux hidden states (different fused add-norm / residual representation). Plus domain shift (code-trained draft vs physics/security eval prompts). The offline metric (tau) is real for the offline setup but does not transfer to vLLM's serving representation.
Fix / next step: ONLINE EAGLE-3 training in vLLM/SpecForge (Speculators), where the draft learns on the exact serving-time hidden states — this is the officially recommended path and is what closes the acceptance gap. Alternatively, align the offline aux-hidden-state capture to vLLM's representation.
5. Deliverable
- Champion draft (exp7, tau=4.25) ready to push:
alexdragannikolich/North-Mini-Code-1.0-EAGLE3(needs HF token). - Full harness reproducible: gen_hidden_states.py (offline), SpecForge train_eagle3 (offline), vLLM main serve + cohere2_moe EAGLE3 patch, vibetest qa.