almaghrabima
/

SARFTokenizer

Text Generation

Model card Files Files and versions

113 MB

1 contributor

History: 30 commits

almaghrabima's picture

Upload lexicons/lexicon_en.txt with huggingface_hub

eef9b5a verified 2 days ago

lexicons
Upload lexicons/lexicon_en.txt with huggingface_hub 2 days ago
morfessor_models
Upload morfessor_models/morf_map_reverse.json with huggingface_hub 2 days ago
.gitattributes

1.64 kB

Upload lexicons/lexicon_en.txt with huggingface_hub 2 days ago
BENCHMARK_REPORT.md

5.15 kB

Upload BENCHMARK_REPORT.md with huggingface_hub 6 days ago
README.md

6.99 kB

Update README.md 3 days ago
benchmark_parallel_results.json

5.38 kB

Upload benchmark_parallel_results.json with huggingface_hub 6 days ago
benchmark_pypi_full.py

11.6 kB

Upload benchmark_pypi_full.py with huggingface_hub 5 days ago
benchmark_tiktoken_style.py

8.13 kB

Upload benchmark_tiktoken_style.py with huggingface_hub 6 days ago
morf_map.bin

115 kB
xet

Upload morf_map.bin with huggingface_hub 6 days ago
requirements.txt

144 Bytes

Upload requirements.txt with huggingface_hub 6 days ago
special_tokens_map.json

630 Bytes

Upload special_tokens_map.json with huggingface_hub 5 days ago
test_comprehensive_million.py

32.6 kB

Upload test_comprehensive_million.py with huggingface_hub 6 days ago
test_comprehensive_results.json

2.54 kB

Upload test_comprehensive_results.json with huggingface_hub 6 days ago
tokenizer.json

4.9 MB

Upload tokenizer.json with huggingface_hub 5 days ago
tokenizer_config.json

4.9 kB

Upload tokenizer_config.json with huggingface_hub 5 days ago