Hugging Face's logo

allenai
/
dolma2-tokenizer

Transformers
Model card Files
xet
 Community
1
soldni commited on
Commit
dd6339e
·
verified ·
1 Parent(s): 58b74ae

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -4,5 +4,5 @@ tags: []
4
  ---
5
 
6
  Slightly modified version of `cl100k_base` that supports Dolma 1.x special tokens
7
- (`|||IP_ADDRESS|||`, `|||EMAIL_ADDRESS|||`, `|||IP_ADDRESS|||`) as well as adds
8
  extra tokens to fill gaps in tiktoken `cl100k_base` version.
 
4
  ---
5
 
6
  Slightly modified version of `cl100k_base` that supports Dolma 1.x special tokens
7
+ (`|||PHONE_NUMBER|||`, `|||EMAIL_ADDRESS|||`, `|||IP_ADDRESS|||`) as well as adds
8
  extra tokens to fill gaps in tiktoken `cl100k_base` version.