Hugging Face
Models
Datasets
Spaces
Community
Docs
Enterprise
Pricing
Log In
Sign Up
ggunio
/
B2NL-IntelligentTokenizer-v6.2.1
like
0
PyTorch
Safetensors
flores200
5 languages
b2nl_tokenizer
tokenizer
byte-to-nl
compression
embeddings
semantic-boundaries
Eval Results
License:
apache-2.0
Model card
Files
Files and versions
xet
Community
main
B2NL-IntelligentTokenizer-v6.2.1
/
core
126 kB
1 contributor
History:
1 commit
ggunio
Initial upload: B2NL-IntelligentTokenizer v6.2.1 (Autoregressive Mode)
ffbd655
verified
3 months ago
decoder.py
Safe
19.3 kB
Initial upload: B2NL-IntelligentTokenizer v6.2.1 (Autoregressive Mode)
3 months ago
encoder.py
Safe
24.1 kB
Initial upload: B2NL-IntelligentTokenizer v6.2.1 (Autoregressive Mode)
3 months ago
intelligent_loss.py
Safe
21.3 kB
Initial upload: B2NL-IntelligentTokenizer v6.2.1 (Autoregressive Mode)
3 months ago
scheduler.py
Safe
23.3 kB
Initial upload: B2NL-IntelligentTokenizer v6.2.1 (Autoregressive Mode)
3 months ago
tokenizer.py
Safe
16.8 kB
Initial upload: B2NL-IntelligentTokenizer v6.2.1 (Autoregressive Mode)
3 months ago
unified_model.py
Safe
21.3 kB
Initial upload: B2NL-IntelligentTokenizer v6.2.1 (Autoregressive Mode)
3 months ago
