---
language:
- tr
tags:
- tokenizer
- morphology
- turkish
- nlp
license: mit
library_name: turk-tokenizer
---

# TurkTokenizer

**Turkish morphological tokenizer — TR-MMLU world record 92%**

TurkTokenizer performs linguistically-aware tokenization of Turkish text using morphological rules. Unlike BPE-based tokenizers, it produces meaningful morphological units (roots and suffixes) aligned with Turkish grammar.

## Installation

```bash
pip install git+https://huggingface.co/Ethosoft/turk-tokenizer
```

**Java is required** (for Zemberek morphological analysis):

| OS | Command |
|---|---|
| Ubuntu / Debian | `sudo apt install default-jre` |
| Fedora / RHEL | `sudo dnf install java-latest-openjdk` |
| macOS | `brew install openjdk` |
| Windows | `winget install Microsoft.OpenJDK.21` |

## Quick Start

```python
from turk_tokenizer import TurkTokenizer

tok = TurkTokenizer()
tokens = tok("İstanbul'da meeting'e katılamadım")

for t in tokens:
    print(t["token"], t["token_type"], t["morph_pos"])
```

Output:
```
<uppercase_word>  ROOT    0
 istanbul         ROOT    0
da               SUFFIX  1
 meeting         FOREIGN 0
e                SUFFIX  1
 katılama        ROOT    0
dı               SUFFIX  1
m                SUFFIX  2
```

## Output Fields

Each token is a dict with the following guaranteed fields:

| Field | Type | Description |
|---|---|---|
| `token` | `str` | Token string (leading space = word-initial) |
| `token_type` | `str` | See types below |
| `morph_pos` | `int` | `0` = root/word-initial, `1` = first suffix, `2` = second… |

### Token Types

| Type | Description |
|---|---|
| `ROOT` | Turkish root word |
| `SUFFIX` | Turkish morphological suffix |
| `FOREIGN` | Foreign/loanword root (e.g. "meeting", "zoom") |
| `BPE` | Unknown subword (fallback) |
| `PUNCT` | Punctuation mark |
| `NUM` | Number |
| `DATE` | Date |
| `UNIT` | Measurement unit |
| `URL` | Web URL |
| `MENTION` | @username |
| `HASHTAG` | #topic |
| `EMOJI` | Emoji |

### Optional Metadata Fields

| Field | Description |
|---|---|
| `_canonical` | Canonical morpheme ID (e.g. `"PL"`, `"ACC"`, `"DAT"`) |
| `_suffix_label` | Detailed morphological label (e.g. `"-PL+ACC"`) |
| `_foreign` | `True` if foreign root |
| `_caps` | `True` if originally ALL CAPS |
| `_domain` | `True` if medical/sports/tourism domain |
| `_compound` | `True` if compound word |
| `_parts` | Compound word parts |
| `_expansion` | Acronym expansion (e.g. `"CMV"` → `"Sitomegalovirüs"`) |
| `_pos` | POS tag from Zemberek (Noun, Verb, Adj…) |
| `_lemma` | Lemma from Zemberek |
| `_disambiguated` | `True` if context disambiguation was applied |
| `_root_corrected` | `True` if root was corrected by Zemberek |

## Batch Tokenization

```python
texts = ["Ankara'da kar yağıyor.", "Meeting'e katılacak mısın?"]
results = tok.batch_tokenize(texts, workers=4)
```

## Statistics

```python
tokens = tok("Türk dili zengin bir morfolojiye sahiptir.")
s = tok.stats(tokens)
print(f"TR coverage: {s['tr_pct']}%")
```

## Morphological Fixes Applied

1. **ALL CAPS** — `"İSTANBUL"` → 2 tokens instead of 16
2. **Apostrophe splitting** — `"meeting'e"` → `[meeting:FOREIGN][e:SUFFIX]`
3. **BPE→SUFFIX** — 260+ suffix patterns reclassified
4. **Zemberek root validation** — phonetic root correction (`"gök"` → `"göğüs"`)
5. **Punctuation** — classified as PUNCT (counted in TR coverage)
6. **Domain vocabulary** — 500+ medical/sports/tourism roots
7. **TDK FOREIGN detection** — 76K+ Turkish words used as reference
8. **Special token normalization** — NUM, DATE, URL, MENTION, HASHTAG, EMOJI
9. **Allomorph canonicalization** — `"lar"/"ler"` → `PL`, `"dan"/"den"` → `ABL`
10. **Compound decomposition** — `"başbakan"` → `["baş", "bakan"]`
11. **Acronym expansion** — `"CMV"` → `"Sitomegalovirüs"`
12. **Context disambiguation** — Zemberek sentence-level POS selection

## Benchmark

| Benchmark | Score |
|---|---|
| TR-MMLU | **92%** (world record) |

## License

MIT