Update README.md
Browse files
README.md
CHANGED
|
@@ -6,7 +6,7 @@ license_link: LICENSE
|
|
| 6 |
|
| 7 |
# 💎 JiRack Tokenizer
|
| 8 |
- Compatible with Llama 3 • Optimized for Code
|
| 9 |
-
- A Llama 3-based tokenizer enhanced with FIM markers (
|
| 10 |
<|fim_prefix|>,
|
| 11 |
<|fim_middle|>,
|
| 12 |
<f|im_suffix|>
|
|
|
|
| 6 |
|
| 7 |
# 💎 JiRack Tokenizer
|
| 8 |
- Compatible with Llama 3 • Optimized for Code
|
| 9 |
+
- A Llama 3-based tokenizer enhanced with FIM markers from Microsoft datasets (
|
| 10 |
<|fim_prefix|>,
|
| 11 |
<|fim_middle|>,
|
| 12 |
<f|im_suffix|>
|