| # C0 Urdu Text Normalization — v1.0.0 |
|
|
| Deterministic surface-form normalization applied **identically to references |
| and hypotheses** before the normalized-track WER in `aggregate-results.json`. |
|
|
| Design principles: no ML, no drifting lexicons; surface-form only (no |
| morphological merging — کے/کی stay distinct); idempotent; never merges |
| distinctions that carry meaning in Urdu (aspirates like بھی survive intact). |
|
|
| Pipeline, in order: |
|
|
| 1. **Unicode NFC** composition. |
| 2. **Character identity mapping** (Arabic-script variants → Urdu canonical): |
| ي→ی · ى→ی · ك→ک · أ→ا · إ→ا · ٱ→ا · ة→ت · ۀ→ہ |
| 3. **Word-initial ھ → ہ** (nonstandard word-initially, e.g. "ھے"→"ہے"). |
| ھ after a consonant is the aspiration digraph (بھی، کھلا، گھر) and is |
| **preserved** — replacement happens only at start-of-string or after |
| whitespace, never post-consonant. |
| 4. **Digit unification**: Urdu ۰-۹ and Arabic ٠-٩ → ASCII 0-9. |
| 5. **Removals**: |
| - Arabic diacritics/tashkeel (U+064B–U+065F), superscript alef, |
| Quranic annotation marks (U+06D6–U+06ED) |
| - tatweel (ـ) and zero-width/bidi control characters |
| (ZWSP, ZWNJ, ZWJ, LRM, RLM, word joiner, BOM, bidi embeddings) |
| - punctuation, mirroring the frozen strict-track set: |
| `؛،؟۔.,!?;:"'()[]{}<>/\|@#$%^&*-_=+~`…–—«»‹›""''` and ASCII space-class |
| specials unified to a single space |
| 6. **Whitespace collapse**: runs of any whitespace → single space; strip ends. |
| 7. **Lowercase** (Latin-script tokens only — English/code-switch words; |
| Urdu script has no case). |
| |
| Tokenization for WER: whitespace split of the normalized string. |
|
|
| Reference implementation: `c0_normalizer.py` (NORMALIZER_VERSION = "1.0.0"), |
| sha256 `7b040047a92b211be6a3341149269463b65461b091216da3e3653155c5deaf81`, |
| with unit tests in the project's stage_c test module. Every evaluation run |
| record cites the normalizer version used. |
|
|
