| --- |
| license: cc-by-nc-4.0 |
| base_model: CohereLabs/North-Mini-Code-1.0 |
| tags: |
| - eagle3 |
| - speculative-decoding |
| - cohere2_moe |
| --- |
| |
| # North-Mini-Code-1.0 — EAGLE-3 draft head |
|
|
| EAGLE-3 draft model for `CohereLabs/North-Mini-Code-1.0` (`cohere2_moe`, 30B/3B MoE, 49 layers), |
| trained with SpecForge (offline) for lossless speculative decoding. |
|
|
| - **Draft**: 1 Llama-style decoder layer, hidden 2048, FFN 12288, draft_vocab 32000 (freq-reduced from 262144). |
| - **Aux hidden-state layers**: [1, 23, 45]. |
| - **Training**: offline, ~8.3k code-instruction samples (magicoder-evol-instruct), 10 epochs, lr 1e-4. |
| - **Offline held-out acceptance** (Σ over 7 positions): τ = 4.25 (pos-0 acc 0.71). |
| |
| ## Serving in vLLM |
| Needs vLLM main + `--hf-overrides '{"first_k_dense_replace":1}'`, and a patch adding the EAGLE3 |
| interface (`SupportsEagle3`) to `cohere2_moe.py`. See repo notes. |
|
|
| ```bash |
| vllm serve CohereLabs/North-Mini-Code-1.0 \ |
| --speculative-config '{"method":"eagle3","model":"<this-repo>","num_speculative_tokens":5}' \ |
| --hf-overrides '{"first_k_dense_replace":1}' \ |
| --reasoning-parser cohere_command4 --tool-call-parser cohere_command4 --enable-auto-tool-choice |
| ``` |
|
|
| > Note: this draft was trained **offline** on HuggingFace-transformers hidden states; real vLLM |
| > acceptance is modest (~1.28) due to train/serve hidden-state representation mismatch. For best |
| > speedup, retrain **online** in vLLM/SpecForge so the draft matches serving-time hidden states. |
|
|
