nmstech
/

turk-tokenizer

Model card Files Files and versions

turk-tokenizer / turk_tokenizer

31.7 MB

Ctrl+K

Ctrl+K

1 contributor

History: 1 commit

nmstech's picture

Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92%

ca41c16 verified 5 days ago

data
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92% 5 days ago
__init__.py

657 Bytes
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92% 5 days ago
_acronym_dict.py

4.31 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92% 5 days ago
_allomorph.py

2.23 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92% 5 days ago
_compound.py

2.95 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92% 5 days ago
_context_aware.py

2.09 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92% 5 days ago
_java_check.py

2.48 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92% 5 days ago
_medical_vocab.py

7.06 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92% 5 days ago
_normalizer.py

4.35 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92% 5 days ago
_preprocessor.py

5.54 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92% 5 days ago
_root_validator.py

6.66 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92% 5 days ago
_suffix_expander.py

8.39 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92% 5 days ago
_tdk_vocab.py

2.66 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92% 5 days ago
tokenizer.py

11.7 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92% 5 days ago