Hugging Face's logo

allenai
/
dolma2-tokenizer

Transformers
Model card Files
xet
 Community
1
How to use from the
Use from the
Transformers library
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("allenai/dolma2-tokenizer", dtype="auto")
Quick Links

Slightly modified version of cl100k_base that supports Dolma 1.x special tokens (|||PHONE_NUMBER|||, |||EMAIL_ADDRESS|||, |||IP_ADDRESS|||) as well as adds extra tokens to fill gaps in tiktoken cl100k_base version.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support