Hugging Face
Models
Datasets
Spaces
Community
Docs
Enterprise
Pricing
Log In
Sign Up
toksuite
/
tokenmonster-englishcode-32000-consistent-v1
like
0
Follow
TokSuite
7
Text Generation
Transformers
Safetensors
5 languages
llama
toksuite
tokenization
tokenmonster
subword
research
robustness
text-generation-inference
arxiv:
2512.20757
License:
mit
Model card
Files
Files and versions
xet
Community
Deploy
Use this model
main
tokenmonster-englishcode-32000-consistent-v1
2.83 GB
2 contributors
History:
16 commits
gsaltintas
Update README.md
e504c56
verified
about 1 month ago
.gitattributes
1.64 kB
Upload model-performance-comparison.png
about 1 month ago
README.md
8.19 kB
Update README.md
about 1 month ago
config.json
609 Bytes
Upload config
5 months ago
generation_config.json
150 Bytes
Upload model files
5 months ago
model-performance-comparison.png
279 kB
xet
Upload model-performance-comparison.png
about 1 month ago
model.safetensors
2.83 GB
xet
Upload model files
5 months ago
tokenizer.json
684 kB
Upload tokenizer file tokenmonster--englishcode-32000-consistent-v1_vocab.json - Upload model files
5 months ago
tokenizer_config.json
92 Bytes
Upload tokenizer file tokenmonster--englishcode-32000-consistent-v1_info.json - Upload model files
5 months ago
tokenmonster--englishcode-32000-consistent-v1.yaml
84 Bytes
Upload tokenizer file tokenmonster--englishcode-32000-consistent-v1.yaml - Upload model files
5 months ago
tokenmonster--englishcode-32000-consistent-v1_super_mapping.json
528 kB
Upload tokenizer file tokenmonster--englishcode-32000-consistent-v1_super_mapping.json - Upload model files
5 months ago
toksuite-logo.png
1 MB
xet
Upload toksuite-logo.png
about 2 months ago