Update README.md
Browse files
README.md
CHANGED
|
@@ -6,7 +6,11 @@ license_link: LICENSE
|
|
| 6 |
|
| 7 |
# 💎 JiRack Tokenizer
|
| 8 |
- Compatible with Llama 3 • Optimized for Code
|
| 9 |
-
- A Llama 3-based tokenizer enhanced with FIM markers (
|
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
- Added Robotics & Embodiment tags (
|
| 11 |
"<|action_start|>",
|
| 12 |
"<|action_end|>",
|
|
|
|
| 6 |
|
| 7 |
# 💎 JiRack Tokenizer
|
| 8 |
- Compatible with Llama 3 • Optimized for Code
|
| 9 |
+
- A Llama 3-based tokenizer enhanced with FIM markers (
|
| 10 |
+
"<fim_prefix>",
|
| 11 |
+
"<fim_middle>",
|
| 12 |
+
"<fim_suffix>",
|
| 13 |
+
)
|
| 14 |
- Added Robotics & Embodiment tags (
|
| 15 |
"<|action_start|>",
|
| 16 |
"<|action_end|>",
|