File size: 740 Bytes
44b084b 2090264 8c7fcb5 2090264 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | ---
license: apache-2.0
---
Eagle model for [CohereLabs/North-mini-code-1.0](https://huggingface.co/CohereLabs/North-Mini-Code-1.0).
Key highlights include:
- three layers with dense transformer blocks
- all draft layers use sliding window attention
- compatible with fp8/w4a16 target model checkpoints
## vllm
To start vllm server with this draft model, run the following
```
vllm serve CohereLabs/North-Mini-Code-1.0 \
-tp 2 \
--max-model-len 320000 \
--tool-call-parser cohere_command4 \
--reasoning-parser cohere_command4 \
--enable-auto-tool-choice \
--speculative_config "{\"method\": \"eagle\", \"model\": \"CohereLabs/North-Mini-Code-1.0-eagle\", \"num_speculative_tokens\": 3, \"draft_tensor_parallel_size\": 2}"
``` |