Rename project from TurkTokenizer to NedoTurkishTokenizer cfffd93 nmstech Claude Opus 4.6 commited on 1 day ago
Add smart ACRONYM detection: TDK-based disambiguation for uppercase tokens fcd513a verified nmstech commited on 3 days ago
Fix broken placeholder mechanism: replace with segment-based tokenization 58e6961 verified nmstech commited on 3 days ago
Upload turk_tokenizer/data/turkish_proper_nouns.txt with huggingface_hub 986e073 verified nmstech commited on 3 days ago
Upload turk_tokenizer/_preprocessor.py with huggingface_hub 8f794ec verified nmstech commited on 3 days ago
Replace hardcoded base lists with TDK vocab lookup in apostrophe split 183e656 verified nmstech commited on 3 days ago
Fix İ lowercase bug + apostrophe merge for BPE-split foreign words 63dbb3f verified nmstech commited on 3 days ago
Fix İ lowercase bug + apostrophe merge for BPE-split foreign words 6330193 verified nmstech commited on 3 days ago
Fix build backend: setuptools.backends.legacy → setuptools.build_meta 091c896 verified nmstech commited on 3 days ago