Upload lexicons/lexicon_en.txt with huggingface_hub

Upload lexicons/lexicon_ar.txt with huggingface_hub

Upload morfessor_models/morf_map_reverse.json with huggingface_hub

Upload morfessor_models/morf_map.json with huggingface_hub

Upload morfessor_models/morfessor_en.bin with huggingface_hub

Upload morfessor_models/morfessor_ar.bin with huggingface_hub

Update README.md

Upload benchmark_pypi_full.py with huggingface_hub

Delete benchmark_pypi.py with huggingface_hub

Upload tokenizer_config.json with huggingface_hub

Upload special_tokens_map.json with huggingface_hub

Upload tokenizer.json with huggingface_hub

Update README.md

Update README.md

Update README.md

Update benchmark results with new tokenizers (Falcon-H1, ALLaM, Hala, Mistral)

Upload benchmark_parallel_results.json with huggingface_hub

Upload test_comprehensive_results.json with huggingface_hub

Upload BENCHMARK_REPORT.md with huggingface_hub

Upload test_comprehensive_million.py with huggingface_hub

Upload benchmark_tiktoken_style.py with huggingface_hub

Upload README.md with huggingface_hub

Upload README.md with huggingface_hub

Upload README.md with huggingface_hub

Upload requirements.txt with huggingface_hub

Upload benchmark_pypi.py with huggingface_hub

Upload README.md with huggingface_hub

Upload morf_map.bin with huggingface_hub

Upload tokenizer.json with huggingface_hub

initial commit