Hugging Face
Models
Datasets
Spaces
Buckets
new
Docs
Enterprise
Pricing
Log In
Sign Up
Ethosoft
/
NedoTurkishTokenizer
like
1
Follow
Ethosoft
4
Model card
Files
Files and versions
xet
Community
47e9fd4
NedoTurkishTokenizer
31.7 MB
Ctrl+K
Ctrl+K
6 contributors
History:
6 commits
nmstech
Update model card with Use This Model section
47e9fd4
verified
about 1 month ago
turk_tokenizer
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92%
about 1 month ago
.gitattributes
Safe
1.59 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92%
about 1 month ago
README.md
Safe
6.89 kB
Update model card with Use This Model section
about 1 month ago
pyproject.toml
1.14 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92%
about 1 month ago
tokenization_turk.py
Safe
6.4 kB
Add AutoTokenizer support (trust_remote_code)
about 1 month ago
tokenizer_config.json
Safe
358 Bytes
Add AutoTokenizer support (trust_remote_code)
about 1 month ago
