Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper

Resim eklendi

Readme daha sigma oldu

Add 64K vocab + transformers-compatible tokenizer (encode/decode/batch)

remove embedded hugging face tokens

merge github main before publish

use zemberek-python and add regression tests

Upload folder using huggingface_hub

Fix README example output and punctuation status

Upload folder using huggingface_hub

Update TR-MMLU benchmark score to 92.64%

Upload folder using huggingface_hub

Migrate to zemberek-python, remove JVM dependency and 31MB JAR, apply O(N^2) init fix

Update README.md

Zemberek dosyasini gercekten ekliyorum

Merge ve dosya düzenleme

Zemberek jar dosyası başarıyla eklendi

zemberek temizlik

Delete nedo_turkish_tokenizer/data/zemberek_yeni.jar

Delete .claude

Zemberek dosyası isim değiştirilerek eklendi

Zemberek LFS ile eklendi

Delete nedo_turkish_tokenizer/data/zemberek-full.jar

Delete .claude directory

Rename project from TurkTokenizer to NedoTurkishTokenizer

Update README.md

Update README.md

Add smart ACRONYM detection: TDK-based disambiguation for uppercase tokens

Fix broken placeholder mechanism: replace with segment-based tokenization

Upload turk_tokenizer/data/turkish_proper_nouns.txt with huggingface_hub

Upload turk_tokenizer/_preprocessor.py with huggingface_hub

Replace hardcoded base lists with TDK vocab lookup in apostrophe split

Fix İ lowercase bug + apostrophe merge for BPE-split foreign words

Fix İ lowercase bug + apostrophe merge for BPE-split foreign words

Load TDK words from HF repo, fallback to TDK API

Add TDK word list (64K words)

Fix build backend: setuptools.backends.legacy → setuptools.build_meta

Update model card with Use This Model section

Add AutoTokenizer support (trust_remote_code)

Add AutoTokenizer support (trust_remote_code)

Add AutoTokenizer support (trust_remote_code)

Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92%

initial commit