kgrabko commited on
Commit
23cd614
·
verified ·
1 Parent(s): 58e3084

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -6,7 +6,7 @@ license_link: LICENSE
6
 
7
  # 💎 JiRack Tokenizer
8
  - Compatible with Llama 3 • Optimized for Code
9
- - A Llama 3-based tokenizer enhanced with FIM markers (
10
  <|fim_prefix|>,
11
  <|fim_middle|>,
12
  <f|im_suffix|>
 
6
 
7
  # 💎 JiRack Tokenizer
8
  - Compatible with Llama 3 • Optimized for Code
9
+ - A Llama 3-based tokenizer enhanced with FIM markers from Microsoft datasets (
10
  <|fim_prefix|>,
11
  <|fim_middle|>,
12
  <f|im_suffix|>