Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,80 @@
+# aztext
+
+Lightweight, **dependency-free** Azerbaijani text-processing toolkit — the small utilities every
+Azerbaijani NLP project re-implements, done once and done correctly. Pure Python standard library, no
+`numpy`/`torch`/`regex` required.
+
+Built as part of an open Azerbaijani LLM stack (tokenizer → dataset → model → evals).
+
+## Why
+
+Azerbaijani (Latin script) adds `ə ğ ı i ö ş ç ü` beyond ASCII, and the **dotted/dotless i** distinction
+(`i`≠`ı`, `İ`≠`I`) is meaningful — but Python's `str.lower()/upper()` get it wrong, people type without
+diacritics, and Azerbaijani is easily confused with Turkish. `aztext` handles these correctly.
+
+## Install
+
+```bash
+pip install -e aztext          # from this repo
+```
+
+## Usage
+
+```python
+import aztext
+
+# Correct Turkic-i-aware casing (Python's str.upper gets this wrong)
+aztext.az_upper("işıq")            # -> "İŞIQ"
+aztext.az_lower("İSTİQLAL")        # -> "istiqlal"
+
+# Restore diacritics to ASCII-typed text (best-effort, dictionary-based)
+aztext.deasciify("ucun cixis")     # -> "üçün çıxış"
+aztext.ascii_fold("gözəl çıxış")   # -> "gozel cixis"
+
+# Azerbaijani vs Turkish language ID (heuristic; ə is the key signal)
+aztext.is_azerbaijani("Mən kitab oxuyuram.")             # -> True
+aztext.detect_language("Ben kitap okuyorum.")            # -> ("tr", 0.87)
+
+# Numbers to Azerbaijani words
+aztext.num_to_words(1234)          # -> "min iki yüz otuz dörd"
+aztext.num_to_words(-5)            # -> "mənfi beş"
+
+# Normalization, tokenization, script detection
+aztext.normalize("Gözəl   şəhər — “Bakı”.")   # NFC, quotes/dashes, whitespace
+aztext.word_tokenize("Bakı, paytaxtdır.")      # -> ["Bakı", "paytaxtdır"]
+aztext.sent_tokenize("Bir. İki! Üç?")          # -> ["Bir.", "İki!", "Üç?"]
+aztext.is_latin_azerbaijani("Azərbaycan dili") # -> True
+```
+
+## API
+
+| function | does |
+|---|---|
+| `normalize(text)` | NFC, mojibake repair, zero-width strip, quote/dash + whitespace cleanup (preserves casing & Az letters; idempotent) |
+| `deasciify(text)` | restore Azerbaijani diacritics on ASCII-typed text (dictionary best-effort) |
+| `ascii_fold(text)` | strip diacritics (`ə→e`, `ş→s`, …) |
+| `is_azerbaijani(text)` / `detect_language(text)` | Az-vs-Tr-vs-other heuristic ID |
+| `num_to_words(n)` | Azerbaijani cardinal spelling (0 … < 10¹², negatives) |
+| `word_tokenize` / `sent_tokenize` | explicit-alphabet word/sentence tokenizers |
+| `is_latin_azerbaijani` / `script_ratios` | Latin-vs-Cyrillic/Arabic script detection |
+| `az_lower` / `az_upper` | Turkic-i-aware case mapping |
+
+## Limitations (honest)
+
+- **`deasciify` is dictionary-based best-effort.** It restores common words; unknown words pass through
+  unchanged, and genuinely ambiguous folds (e.g. `el` → *el* "people" vs *əl* "hand") resolve to a single
+  listed form. It is not a language model.
+- **`detect_language` is a lightweight heuristic**, not a trained classifier — tuned for the Az/Tr split.
+  An optional `fasttext` model is used as a tiebreaker only if `AZTEXT_FASTTEXT_MODEL` points to one.
+- Latin-script Modern (North) Azerbaijani only; Cyrillic/Perso-Arabic are detected but not transliterated.
+
+## Tests
+
+```bash
+python -m pytest aztext/tests -q      # or, dependency-free:
+python aztext/tests/test_aztext.py
+```
+
+## License
+
+MIT.