C0 Urdu Text Normalization — v1.0.0

Deterministic surface-form normalization applied identically to references and hypotheses before the normalized-track WER in aggregate-results.json.

Design principles: no ML, no drifting lexicons; surface-form only (no morphological merging — کے/کی stay distinct); idempotent; never merges distinctions that carry meaning in Urdu (aspirates like بھی survive intact).

Pipeline, in order:

Unicode NFC composition.
Character identity mapping (Arabic-script variants → Urdu canonical): ي→ی · ى→ی · ك→ک · أ→ا · إ→ا · ٱ→ا · ة→ت · ۀ→ہ
Word-initial ھ → ہ (nonstandard word-initially, e.g. "ھے"→"ہے"). ھ after a consonant is the aspiration digraph (بھی، کھلا، گھر) and is preserved — replacement happens only at start-of-string or after whitespace, never post-consonant.
Digit unification: Urdu ۰-۹ and Arabic ٠-٩ → ASCII 0-9.
Removals:
- Arabic diacritics/tashkeel (U+064B–U+065F), superscript alef, Quranic annotation marks (U+06D6–U+06ED)
- tatweel (ـ) and zero-width/bidi control characters (ZWSP, ZWNJ, ZWJ, LRM, RLM, word joiner, BOM, bidi embeddings)
- punctuation, mirroring the frozen strict-track set: ؛،؟۔.,!?;:"'()[]{}<>/\|@#$%^&*-_=+~…–—«»‹›""''` and ASCII space-class specials unified to a single space
Whitespace collapse: runs of any whitespace → single space; strip ends.
Lowercase (Latin-script tokens only — English/code-switch words; Urdu script has no case).

Tokenization for WER: whitespace split of the normalized string.

Reference implementation: c0_normalizer.py (NORMALIZER_VERSION = "1.0.0"), sha256 7b040047a92b211be6a3341149269463b65461b091216da3e3653155c5deaf81, with unit tests in the project's stage_c test module. Every evaluation run record cites the normalizer version used.