Hugging Face
Models
Datasets
Spaces
Community
Docs
Enterprise
Pricing
Log In
Sign Up
toksuite
/
tiktoken-gpt-4o
like
0
Follow
TokSuite
7
Text Generation
Transformers
Safetensors
5 languages
llama
toksuite
tokenization
gpt-4o
multilingual
research
robustness
bpe
text-generation-inference
arxiv:
2512.20757
License:
mit
Model card
Files
Files and versions
xet
Community
Deploy
Use this model
main
tiktoken-gpt-4o
4.22 GB
2 contributors
History:
18 commits
gsaltintas
Update README.md
a4cf525
verified
about 2 months ago
.gitattributes
1.64 kB
Upload model-performance-comparison.png
2 months ago
README.md
8.06 kB
Update README.md
about 2 months ago
config.json
610 Bytes
Upload config
6 months ago
generation_config.json
150 Bytes
Upload model files
6 months ago
model-performance-comparison.png
279 kB
xet
Upload model-performance-comparison.png
2 months ago
model.safetensors
4.21 GB
xet
Upload model files
6 months ago
tiktoken--gpt-4o.yaml
55 Bytes
Upload tokenizer file tiktoken--gpt-4o.yaml - Upload model files
6 months ago
tiktoken--gpt-4o_super_mapping.json
3.46 MB
Upload tokenizer file tiktoken--gpt-4o_super_mapping.json - Upload model files
6 months ago
tokenizer.json
6.62 MB
Upload tokenizer file tiktoken--gpt-4o_vocab.json - Upload model files
6 months ago
tokenizer_config.json
63 Bytes
Upload tokenizer file tiktoken--gpt-4o_info.json - Upload model files
6 months ago
toksuite-logo.png
1 MB
xet
Upload toksuite-logo.png
2 months ago
