Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,22 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
---
|
| 4 |
+
Eagle model for [CohereLabs/North-mini-code-1.0](https://huggingface.co/CohereLabs/North-Mini-Code-1.0).
|
| 5 |
+
|
| 6 |
+
|
| 7 |
+
Key highlights include:
|
| 8 |
+
- three layers with dense transformer blocks
|
| 9 |
+
- all draft layers use sliding window attention
|
| 10 |
+
- compatible with fp8/w4a4 target model checkpoints
|
| 11 |
+
|
| 12 |
+
## vllm
|
| 13 |
+
To start vllm server with this draft model, run the following
|
| 14 |
+
```
|
| 15 |
+
vllm serve CohereLabs/North-Mini-Code-1.0 \
|
| 16 |
+
-tp 2 \
|
| 17 |
+
--max-model-len 320000 \
|
| 18 |
+
--tool-call-parser cohere_command4 \
|
| 19 |
+
--reasoning-parser cohere_command4 \
|
| 20 |
+
--enable-auto-tool-choice \
|
| 21 |
+
--speculative_config "{\"method\": \"eagle\", \"model\": \"CohereLabs/North-Mini-Code-1.0-eagle\", \"num_speculative_tokens\": 3, \"draft_tensor_parallel_size\": 2}"
|
| 22 |
+
```
|