Publish NedoTokenizer 32K tokenizer and source mirror

f00e721 verified 16 days ago

1.2 kB

language:
  - tr
license: apache-2.0
tags:
  - tokenizer
  - turkish
  - morphology
  - rust
  - lossless
library_name: nedotokenizer

NedoTokenizer

Official Hugging Face mirror. Source repository: Ethosoft/NedoTokenizer on GitHub. Interactive browser demo: Ethosoft/NedoTokenizer-Demo on Hugging Face Spaces.

Byte-exact, morphology-aware tokenizer for Turkish and mixed Turkish/code text.

Python

pip install .

from pathlib import Path
from nedotokenizer import SurfaceTokenizer

vocab = Path("assets/surface-vocab.bin").read_bytes()
tokenizer = SurfaceTokenizer(vocab)

text = "Evlerimizden çıktık.".encode("utf-8")
ids = tokenizer.encode_ids(text)
assert tokenizer.decode_ids(ids) == text
print(ids)

Source

The tokenizer core is implemented in Rust. The Python package calls the same native core through PyO3.

The released 32K surface vocabulary is assets/surface-vocab.bin. The large precompiled surface-analysis lookup table is only a throughput accelerator and is not required for correct tokenization, so it is not included here.

License

Apache-2.0. See THIRD_PARTY_LICENSES/ZEMBEREK_NOTICE.md for the morphology resource notice.