| library_name: transformers | |
| tags: [] | |
| Slightly modified version of `cl100k_base` that supports Dolma 1.x special tokens | |
| (`|||PHONE_NUMBER|||`, `|||EMAIL_ADDRESS|||`, `|||IP_ADDRESS|||`) as well as adds | |
| extra tokens to fill gaps in tiktoken `cl100k_base` version. |