Update README.md
Browse files
README.md
CHANGED
|
@@ -5,6 +5,11 @@ license_link: LICENSE
|
|
| 5 |
---
|
| 6 |
|
| 7 |
# 💎 JiRack Tokenizer
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
- Compatible with Llama 3 • Optimized for Code
|
| 9 |
- A Llama 3-based tokenizer enhanced with FIM markers from Microsoft datasets (
|
| 10 |
<|fim_prefix|>,
|
|
@@ -195,6 +200,7 @@ print("Vocab size:", len(tok))
|
|
| 195 |
print("pad_token_id:", tok.pad_token_id)
|
| 196 |
print("eos_token_id:", tok.eos_token_id)
|
| 197 |
'
|
| 198 |
-
|
| 199 |
-
|
|
|
|
| 200 |
eos_token_id: 128001
|
|
|
|
| 5 |
---
|
| 6 |
|
| 7 |
# 💎 JiRack Tokenizer
|
| 8 |
+
- Size {
|
| 9 |
+
Vocab size: 128365
|
| 10 |
+
pad_token_id: None
|
| 11 |
+
eos_token_id: 128001
|
| 12 |
+
}
|
| 13 |
- Compatible with Llama 3 • Optimized for Code
|
| 14 |
- A Llama 3-based tokenizer enhanced with FIM markers from Microsoft datasets (
|
| 15 |
<|fim_prefix|>,
|
|
|
|
| 200 |
print("pad_token_id:", tok.pad_token_id)
|
| 201 |
print("eos_token_id:", tok.eos_token_id)
|
| 202 |
'
|
| 203 |
+
|
| 204 |
+
Vocab size: 128365
|
| 205 |
+
pad_token_id: None
|
| 206 |
eos_token_id: 128001
|