Hugging Face
Models
Datasets
Spaces
Buckets
new
Docs
Enterprise
Pricing
Log In
Sign Up
almaghrabima
/
SARFTokenizer
like
1
Arabic
English
tokenizers
tokenizer
unigram
sarf
bilingual
arabic
english
math
code
sentencepiece-style
arxiv:
2512.18399
arxiv:
2508.08424
License:
apache-2.0
Model card
Files
Files and versions
xet
Community
9770614
SARFTokenizer
5.03 MB
Ctrl+K
Ctrl+K
1 contributor
History:
10 commits
almaghrabima
Upload benchmark_tiktoken_style.py with huggingface_hub
9770614
verified
3 months ago
.gitattributes
Safe
1.52 kB
initial commit
3 months ago
README.md
5.59 kB
Upload README.md with huggingface_hub
3 months ago
benchmark_pypi.py
11.6 kB
Upload benchmark_pypi.py with huggingface_hub
3 months ago
benchmark_tiktoken_style.py
Safe
8.13 kB
Upload benchmark_tiktoken_style.py with huggingface_hub
3 months ago
morf_map.bin
pickle
Pickle imports
No problematic imports detected
What is a pickle import?
115 kB
xet
Upload morf_map.bin with huggingface_hub
3 months ago
requirements.txt
Safe
144 Bytes
Upload requirements.txt with huggingface_hub
3 months ago
tokenizer.json
4.89 MB
Upload tokenizer.json with huggingface_hub
3 months ago
