EAGLE-3 drafts for North-Mini-Code-1.0 (champion exp7 tau=4.25 + all experiments)

c7646de verified 21 days ago

1.46 kB

license: cc-by-nc-4.0
base_model: CohereLabs/North-Mini-Code-1.0
tags:
  - eagle3
  - speculative-decoding
  - cohere2_moe

North-Mini-Code-1.0 — EAGLE-3 draft head

EAGLE-3 draft model for CohereLabs/North-Mini-Code-1.0 (cohere2_moe, 30B/3B MoE, 49 layers), trained with SpecForge (offline) for lossless speculative decoding.

Draft: 1 Llama-style decoder layer, hidden 2048, FFN 12288, draft_vocab 32000 (freq-reduced from 262144).
Aux hidden-state layers: [1, 23, 45].
Training: offline, ~8.3k code-instruction samples (magicoder-evol-instruct), 10 epochs, lr 1e-4.
Offline held-out acceptance (Σ over 7 positions): τ = 4.25 (pos-0 acc 0.71).

Serving in vLLM

Needs vLLM main + --hf-overrides '{"first_k_dense_replace":1}', and a patch adding the EAGLE3 interface (SupportsEagle3) to cohere2_moe.py. See repo notes.

vllm serve CohereLabs/North-Mini-Code-1.0 \
  --speculative-config '{"method":"eagle3","model":"<this-repo>","num_speculative_tokens":5}' \
  --hf-overrides '{"first_k_dense_replace":1}' \
  --reasoning-parser cohere_command4 --tool-call-parser cohere_command4 --enable-auto-tool-choice

Note: this draft was trained offline on HuggingFace-transformers hidden states; real vLLM acceptance is modest (~1.28) due to train/serve hidden-state representation mismatch. For best speedup, retrain online in vLLM/SpecForge so the draft matches serving-time hidden states.