| language: | |
| - tr | |
| license: apache-2.0 | |
| tags: | |
| - tokenizer | |
| - turkish | |
| - morphology | |
| - rust | |
| - lossless | |
| library_name: nedotokenizer | |
| # NedoTokenizer | |
| > **Official Hugging Face mirror.** Source repository: `Ethosoft/NedoTokenizer` on GitHub. Interactive browser demo: `Ethosoft/NedoTokenizer-Demo` on Hugging Face Spaces. | |
| Byte-exact, morphology-aware tokenizer for Turkish and mixed Turkish/code text. | |
| ## Python | |
| ```bash | |
| pip install . | |
| ``` | |
| ```python | |
| from pathlib import Path | |
| from nedotokenizer import SurfaceTokenizer | |
| vocab = Path("assets/surface-vocab.bin").read_bytes() | |
| tokenizer = SurfaceTokenizer(vocab) | |
| text = "Evlerimizden çıktık.".encode("utf-8") | |
| ids = tokenizer.encode_ids(text) | |
| assert tokenizer.decode_ids(ids) == text | |
| print(ids) | |
| ``` | |
| ## Source | |
| The tokenizer core is implemented in Rust. The Python package calls the same native core through PyO3. | |
| The released 32K surface vocabulary is `assets/surface-vocab.bin`. The large precompiled surface-analysis lookup table is only a throughput accelerator and is not required for correct tokenization, so it is not included here. | |
| ## License | |
| Apache-2.0. See `THIRD_PARTY_LICENSES/ZEMBEREK_NOTICE.md` for the morphology resource notice. | |