Models
Datasets
Spaces
Buckets new
Docs
Enterprise
Pricing
Log In
Sign Up

Ethosoft
/

NedoTurkishTokenizer

Model card Files Files and versions

NedoTurkishTokenizer

32.4 MB

Ctrl+K

Ctrl+K

6 contributors

History: 13 commits

nmstech's picture

Upload turk_tokenizer/_preprocessor.py with huggingface_hub

8f794ec verified about 1 month ago

turk_tokenizer
Upload turk_tokenizer/_preprocessor.py with huggingface_hub about 1 month ago
.gitattributes

1.59 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92% about 1 month ago
README.md

6.89 kB
Update model card with Use This Model section about 1 month ago
pyproject.toml

1.13 kB
Fix build backend: setuptools.backends.legacy → setuptools.build_meta about 1 month ago
tokenization_turk.py

6.4 kB
Add AutoTokenizer support (trust_remote_code) about 1 month ago
tokenizer_config.json

358 Bytes
Add AutoTokenizer support (trust_remote_code) about 1 month ago