| --- |
| license: mit |
| language: |
| - az |
| library_name: aztext |
| tags: |
| - azerbaijani |
| - nlp |
| - text-processing |
| - deasciify |
| - tokenizer |
| - turkic |
| - low-resource |
| --- |
| |
| # aztext |
|
|
| Lightweight, **dependency-free** Azerbaijani text-processing toolkit — the small utilities every |
| Azerbaijani NLP project re-implements, done once and done correctly. Pure Python standard library, no |
| `numpy`/`torch`/`regex` required. |
|
|
| Built as part of an open Azerbaijani LLM stack (tokenizer → dataset → model → evals). |
|
|
| ## Why |
|
|
| Azerbaijani (Latin script) adds `ə ğ ı i ö ş ç ü` beyond ASCII, and the **dotted/dotless i** distinction |
| (`i`≠`ı`, `İ`≠`I`) is meaningful — but Python's `str.lower()/upper()` get it wrong, people type without |
| diacritics, and Azerbaijani is easily confused with Turkish. `aztext` handles these correctly. |
|
|
| ## Install |
|
|
| ```bash |
| pip install -e aztext # from this repo |
| ``` |
|
|
| ## Usage |
|
|
| ```python |
| import aztext |
| |
| # Correct Turkic-i-aware casing (Python's str.upper gets this wrong) |
| aztext.az_upper("işıq") # -> "İŞIQ" |
| aztext.az_lower("İSTİQLAL") # -> "istiqlal" |
| |
| # Restore diacritics to ASCII-typed text (best-effort, dictionary-based) |
| aztext.deasciify("ucun cixis") # -> "üçün çıxış" |
| aztext.ascii_fold("gözəl çıxış") # -> "gozel cixis" |
| |
| # Azerbaijani vs Turkish language ID (heuristic; ə is the key signal) |
| aztext.is_azerbaijani("Mən kitab oxuyuram.") # -> True |
| aztext.detect_language("Ben kitap okuyorum.") # -> ("tr", 0.87) |
| |
| # Numbers to Azerbaijani words |
| aztext.num_to_words(1234) # -> "min iki yüz otuz dörd" |
| aztext.num_to_words(-5) # -> "mənfi beş" |
| |
| # Normalization, tokenization, script detection |
| aztext.normalize("Gözəl şəhər — “Bakı”.") # NFC, quotes/dashes, whitespace |
| aztext.word_tokenize("Bakı, paytaxtdır.") # -> ["Bakı", "paytaxtdır"] |
| aztext.sent_tokenize("Bir. İki! Üç?") # -> ["Bir.", "İki!", "Üç?"] |
| aztext.is_latin_azerbaijani("Azərbaycan dili") # -> True |
| ``` |
|
|
| ## API |
|
|
| | function | does | |
| |---|---| |
| | `normalize(text)` | NFC, mojibake repair, zero-width strip, quote/dash + whitespace cleanup (preserves casing & Az letters; idempotent) | |
| | `deasciify(text)` | restore Azerbaijani diacritics on ASCII-typed text (dictionary best-effort) | |
| | `ascii_fold(text)` | strip diacritics (`ə→e`, `ş→s`, …) | |
| | `is_azerbaijani(text)` / `detect_language(text)` | Az-vs-Tr-vs-other heuristic ID | |
| | `num_to_words(n)` | Azerbaijani cardinal spelling (0 … < 10¹², negatives) | |
| | `word_tokenize` / `sent_tokenize` | explicit-alphabet word/sentence tokenizers | |
| | `is_latin_azerbaijani` / `script_ratios` | Latin-vs-Cyrillic/Arabic script detection | |
| | `az_lower` / `az_upper` | Turkic-i-aware case mapping | |
|
|
| ## Limitations (honest) |
|
|
| - **`deasciify` is dictionary-based best-effort.** It restores common words; unknown words pass through |
| unchanged, and genuinely ambiguous folds (e.g. `el` → *el* "people" vs *əl* "hand") resolve to a single |
| listed form. It is not a language model. |
| - **`detect_language` is a lightweight heuristic**, not a trained classifier — tuned for the Az/Tr split. |
| An optional `fasttext` model is used as a tiebreaker only if `AZTEXT_FASTTEXT_MODEL` points to one. |
| - Latin-script Modern (North) Azerbaijani only; Cyrillic/Perso-Arabic are detected but not transliterated. |
| |
| ## Tests |
| |
| ```bash |
| python -m pytest aztext/tests -q # or, dependency-free: |
| python aztext/tests/test_aztext.py |
| ``` |
| |
| ## License |
| |
| MIT. |
| |
