# C0 Urdu Text Normalization — v1.0.0

Deterministic surface-form normalization applied **identically to references
and hypotheses** before the normalized-track WER in `aggregate-results.json`.

Design principles: no ML, no drifting lexicons; surface-form only (no
morphological merging — کے/کی stay distinct); idempotent; never merges
distinctions that carry meaning in Urdu (aspirates like بھی survive intact).

Pipeline, in order:

1. **Unicode NFC** composition.
2. **Character identity mapping** (Arabic-script variants → Urdu canonical):
   ي→ی · ى→ی · ك→ک · أ→ا · إ→ا · ٱ→ا · ة→ت · ۀ→ہ
3. **Word-initial ھ → ہ** (nonstandard word-initially, e.g. "ھے"→"ہے").
   ھ after a consonant is the aspiration digraph (بھی، کھلا، گھر) and is
   **preserved** — replacement happens only at start-of-string or after
   whitespace, never post-consonant.
4. **Digit unification**: Urdu ۰-۹ and Arabic ٠-٩ → ASCII 0-9.
5. **Removals**:
   - Arabic diacritics/tashkeel (U+064B–U+065F), superscript alef,
     Quranic annotation marks (U+06D6–U+06ED)
   - tatweel (ـ) and zero-width/bidi control characters
     (ZWSP, ZWNJ, ZWJ, LRM, RLM, word joiner, BOM, bidi embeddings)
   - punctuation, mirroring the frozen strict-track set:
     `؛،؟۔.,!?;:"'()[]{}<>/\|@#$%^&*-_=+~`…–—«»‹›""''` and ASCII space-class
     specials unified to a single space
6. **Whitespace collapse**: runs of any whitespace → single space; strip ends.
7. **Lowercase** (Latin-script tokens only — English/code-switch words;
   Urdu script has no case).

Tokenization for WER: whitespace split of the normalized string.

Reference implementation: `c0_normalizer.py` (NORMALIZER_VERSION = "1.0.0"),
sha256 `7b040047a92b211be6a3341149269463b65461b091216da3e3653155c5deaf81`,
with unit tests in the project's stage_c test module. Every evaluation run
record cites the normalizer version used.