File size: 1,197 Bytes
f00e721 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | ---
language:
- tr
license: apache-2.0
tags:
- tokenizer
- turkish
- morphology
- rust
- lossless
library_name: nedotokenizer
---
# NedoTokenizer
> **Official Hugging Face mirror.** Source repository: `Ethosoft/NedoTokenizer` on GitHub. Interactive browser demo: `Ethosoft/NedoTokenizer-Demo` on Hugging Face Spaces.
Byte-exact, morphology-aware tokenizer for Turkish and mixed Turkish/code text.
## Python
```bash
pip install .
```
```python
from pathlib import Path
from nedotokenizer import SurfaceTokenizer
vocab = Path("assets/surface-vocab.bin").read_bytes()
tokenizer = SurfaceTokenizer(vocab)
text = "Evlerimizden çıktık.".encode("utf-8")
ids = tokenizer.encode_ids(text)
assert tokenizer.decode_ids(ids) == text
print(ids)
```
## Source
The tokenizer core is implemented in Rust. The Python package calls the same native core through PyO3.
The released 32K surface vocabulary is `assets/surface-vocab.bin`. The large precompiled surface-analysis lookup table is only a throughput accelerator and is not required for correct tokenization, so it is not included here.
## License
Apache-2.0. See `THIRD_PARTY_LICENSES/ZEMBEREK_NOTICE.md` for the morphology resource notice.
|