Update README.md
Browse files
README.md
CHANGED
|
@@ -7,9 +7,9 @@ license_link: LICENSE
|
|
| 7 |
# 💎 JiRack Tokenizer
|
| 8 |
- Compatible with Llama 3 • Optimized for Code
|
| 9 |
- A Llama 3-based tokenizer enhanced with FIM markers (
|
| 10 |
-
<fim_prefix>,
|
| 11 |
-
<fim_middle>,
|
| 12 |
-
<
|
| 13 |
)
|
| 14 |
- Added Robotics & Embodiment tags (
|
| 15 |
"<|action_start|>",
|
|
|
|
| 7 |
# 💎 JiRack Tokenizer
|
| 8 |
- Compatible with Llama 3 • Optimized for Code
|
| 9 |
- A Llama 3-based tokenizer enhanced with FIM markers (
|
| 10 |
+
<|fim_prefix|>,
|
| 11 |
+
<|fim_middle|>,
|
| 12 |
+
<f|im_suffix|>
|
| 13 |
)
|
| 14 |
- Added Robotics & Embodiment tags (
|
| 15 |
"<|action_start|>",
|