Update README.md
Browse files
README.md
CHANGED
|
@@ -42,9 +42,9 @@ tags:
|
|
| 42 |
pad_token_id: 151643
|
| 43 |
eos_token_id: 151643
|
| 44 |
}
|
| 45 |
-
- Compatible with Qwen
|
| 46 |
- It needs 100k example to fully adapt routing features for RAG Routing model . So check Qwen modification tokenizer rules
|
| 47 |
-
- A
|
| 48 |
<|fim_prefix|>,
|
| 49 |
<|fim_middle|>,
|
| 50 |
<f|im_suffix|>
|
|
|
|
| 42 |
pad_token_id: 151643
|
| 43 |
eos_token_id: 151643
|
| 44 |
}
|
| 45 |
+
- Compatible with Qwen 2.5 and DeekSeek R1 • Optimized for Code . Use resize function without adaptation, see examples below
|
| 46 |
- It needs 100k example to fully adapt routing features for RAG Routing model . So check Qwen modification tokenizer rules
|
| 47 |
+
- A DeekSeek R1-based tokenizer enhanced with FIM markers from Microsoft datasets and other tags for ML (
|
| 48 |
<|fim_prefix|>,
|
| 49 |
<|fim_middle|>,
|
| 50 |
<f|im_suffix|>
|