almaghrabima
/

SARFTokenizer

sentencepiece-style

Model card Files Files and versions

11.8 MB

Ctrl+K

Ctrl+K

1 contributor

History: 27 commits

almaghrabima's picture

Upload morfessor_models/morf_map.json with huggingface_hub

5fee9e3 verified 3 months ago

morfessor_models
Upload morfessor_models/morf_map.json with huggingface_hub 3 months ago
.gitattributes

1.52 kB
initial commit 3 months ago
BENCHMARK_REPORT.md

5.15 kB
Upload BENCHMARK_REPORT.md with huggingface_hub 3 months ago
README.md

6.99 kB
Update README.md 3 months ago
benchmark_parallel_results.json

5.38 kB
Upload benchmark_parallel_results.json with huggingface_hub 3 months ago
benchmark_pypi_full.py

11.6 kB
Upload benchmark_pypi_full.py with huggingface_hub 3 months ago
benchmark_tiktoken_style.py

8.13 kB
Upload benchmark_tiktoken_style.py with huggingface_hub 3 months ago
morf_map.bin
Pickle imports
- No problematic imports detected
What is a pickle import?
115 kB
xet

Upload morf_map.bin with huggingface_hub 3 months ago
requirements.txt

144 Bytes
Upload requirements.txt with huggingface_hub 3 months ago
special_tokens_map.json

630 Bytes
Upload special_tokens_map.json with huggingface_hub 3 months ago
test_comprehensive_million.py

32.6 kB
Upload test_comprehensive_million.py with huggingface_hub 3 months ago
test_comprehensive_results.json

2.54 kB
Upload test_comprehensive_results.json with huggingface_hub 3 months ago
tokenizer.json

4.9 MB
Upload tokenizer.json with huggingface_hub 3 months ago
tokenizer_config.json

4.9 kB
Upload tokenizer_config.json with huggingface_hub 3 months ago