aztext

Lightweight, dependency-free Azerbaijani text-processing toolkit — the small utilities every Azerbaijani NLP project re-implements, done once and done correctly. Pure Python standard library, no numpy/torch/regex required.

Built as part of an open Azerbaijani LLM stack (tokenizer → dataset → model → evals).

Why

Azerbaijani (Latin script) adds ə ğ ı i ö ş ç ü beyond ASCII, and the dotted/dotless i distinction (i≠ı, İ≠I) is meaningful — but Python's str.lower()/upper() get it wrong, people type without diacritics, and Azerbaijani is easily confused with Turkish. aztext handles these correctly.

Install

pip install -e aztext          # from this repo

Usage

import aztext

# Correct Turkic-i-aware casing (Python's str.upper gets this wrong)
aztext.az_upper("işıq")            # -> "İŞIQ"
aztext.az_lower("İSTİQLAL")        # -> "istiqlal"

# Restore diacritics to ASCII-typed text (best-effort, dictionary-based)
aztext.deasciify("ucun cixis")     # -> "üçün çıxış"
aztext.ascii_fold("gözəl çıxış")   # -> "gozel cixis"

# Azerbaijani vs Turkish language ID (heuristic; ə is the key signal)
aztext.is_azerbaijani("Mən kitab oxuyuram.")             # -> True
aztext.detect_language("Ben kitap okuyorum.")            # -> ("tr", 0.87)

# Numbers to Azerbaijani words
aztext.num_to_words(1234)          # -> "min iki yüz otuz dörd"
aztext.num_to_words(-5)            # -> "mənfi beş"

# Normalization, tokenization, script detection
aztext.normalize("Gözəl   şəhər — “Bakı”.")   # NFC, quotes/dashes, whitespace
aztext.word_tokenize("Bakı, paytaxtdır.")      # -> ["Bakı", "paytaxtdır"]
aztext.sent_tokenize("Bir. İki! Üç?")          # -> ["Bir.", "İki!", "Üç?"]
aztext.is_latin_azerbaijani("Azərbaycan dili") # -> True

API

function	does
normalize(text)	NFC, mojibake repair, zero-width strip, quote/dash + whitespace cleanup (preserves casing & Az letters; idempotent)
deasciify(text)	restore Azerbaijani diacritics on ASCII-typed text (dictionary best-effort)
ascii_fold(text)	strip diacritics (ə→e, ş→s, …)
is_azerbaijani(text) / detect_language(text)	Az-vs-Tr-vs-other heuristic ID
num_to_words(n)	Azerbaijani cardinal spelling (0 … < 10¹², negatives)
word_tokenize / sent_tokenize	explicit-alphabet word/sentence tokenizers
is_latin_azerbaijani / script_ratios	Latin-vs-Cyrillic/Arabic script detection
az_lower / az_upper	Turkic-i-aware case mapping

Limitations (honest)

• deasciify is dictionary-based best-effort. It restores common words; unknown words pass through unchanged, and genuinely ambiguous folds (e.g. el → el "people" vs əl "hand") resolve to a single listed form. It is not a language model.
• detect_language is a lightweight heuristic, not a trained classifier — tuned for the Az/Tr split. An optional fasttext model is used as a tiebreaker only if AZTEXT_FASTTEXT_MODEL points to one.
• Latin-script Modern (North) Azerbaijani only; Cyrillic/Perso-Arabic are detected but not transliterated.

Tests

python -m pytest aztext/tests -q      # or, dependency-free:
python aztext/tests/test_aztext.py

License

MIT.