Hugging Face's logo

ggunio
/
intelligent-tokenizer-v6

PyTorch
intelligent_tokenizer
tokenization
byte-level
neural-tokenizer
pattern-learning
vocabulary-free
Eval Results (legacy)
Model card Files
xet
Community
intelligent-tokenizer-v6 / src /core /byte_tokenizer_v6.py

Commit History

Upload src/core/byte_tokenizer_v6.py with huggingface_hub
0a81e70
verified

ggunio commited on