kgrabko commited on
Commit
1eaffcc
·
verified ·
1 Parent(s): 23cd614

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +8 -2
README.md CHANGED
@@ -5,6 +5,11 @@ license_link: LICENSE
5
  ---
6
 
7
  # 💎 JiRack Tokenizer
 
 
 
 
 
8
  - Compatible with Llama 3 • Optimized for Code
9
  - A Llama 3-based tokenizer enhanced with FIM markers from Microsoft datasets (
10
  <|fim_prefix|>,
@@ -195,6 +200,7 @@ print("Vocab size:", len(tok))
195
  print("pad_token_id:", tok.pad_token_id)
196
  print("eos_token_id:", tok.eos_token_id)
197
  '
198
- Vocab size: 128259
199
- pad_token_id: 128001
 
200
  eos_token_id: 128001
 
5
  ---
6
 
7
  # 💎 JiRack Tokenizer
8
+ - Size {
9
+ Vocab size: 128365
10
+ pad_token_id: None
11
+ eos_token_id: 128001
12
+ }
13
  - Compatible with Llama 3 • Optimized for Code
14
  - A Llama 3-based tokenizer enhanced with FIM markers from Microsoft datasets (
15
  <|fim_prefix|>,
 
200
  print("pad_token_id:", tok.pad_token_id)
201
  print("eos_token_id:", tok.eos_token_id)
202
  '
203
+
204
+ Vocab size: 128365
205
+ pad_token_id: None
206
  eos_token_id: 128001