NedoTurkishTokenizer / nedo_turkish_tokenizer

774 kB

Ctrl+K

6 contributors

History: 11 commits

Ethosoft

Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper

edec8b7 about 1 month ago

data
use zemberek-python and add regression tests about 1 month ago
__init__.py

647 Bytes
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper about 1 month ago
_acronym_table.py

5.11 kB
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper about 1 month ago
_domain_vocab.py

7.56 kB
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper about 1 month ago
_suffix_table.py

16.4 kB
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper about 1 month ago
apostrophe.py

4.3 kB
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper about 1 month ago
engine.py

6.22 kB
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper about 1 month ago
morphology.py

6.99 kB
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper about 1 month ago
normalization.py

2.01 kB
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper about 1 month ago
resources.py

3.71 kB
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper about 1 month ago
segmentation.py

17.7 kB
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper about 1 month ago
special_spans.py

10.7 kB
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper about 1 month ago
tokenizer.py

5.81 kB
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper about 1 month ago
types.py

3.52 kB
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper about 1 month ago