File size: 1,962 Bytes
8f3666f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | # C0 Urdu Text Normalization — v1.0.0
Deterministic surface-form normalization applied **identically to references
and hypotheses** before the normalized-track WER in `aggregate-results.json`.
Design principles: no ML, no drifting lexicons; surface-form only (no
morphological merging — کے/کی stay distinct); idempotent; never merges
distinctions that carry meaning in Urdu (aspirates like بھی survive intact).
Pipeline, in order:
1. **Unicode NFC** composition.
2. **Character identity mapping** (Arabic-script variants → Urdu canonical):
ي→ی · ى→ی · ك→ک · أ→ا · إ→ا · ٱ→ا · ة→ت · ۀ→ہ
3. **Word-initial ھ → ہ** (nonstandard word-initially, e.g. "ھے"→"ہے").
ھ after a consonant is the aspiration digraph (بھی، کھلا، گھر) and is
**preserved** — replacement happens only at start-of-string or after
whitespace, never post-consonant.
4. **Digit unification**: Urdu ۰-۹ and Arabic ٠-٩ → ASCII 0-9.
5. **Removals**:
- Arabic diacritics/tashkeel (U+064B–U+065F), superscript alef,
Quranic annotation marks (U+06D6–U+06ED)
- tatweel (ـ) and zero-width/bidi control characters
(ZWSP, ZWNJ, ZWJ, LRM, RLM, word joiner, BOM, bidi embeddings)
- punctuation, mirroring the frozen strict-track set:
`؛،؟۔.,!?;:"'()[]{}<>/\|@#$%^&*-_=+~`…–—«»‹›""''` and ASCII space-class
specials unified to a single space
6. **Whitespace collapse**: runs of any whitespace → single space; strip ends.
7. **Lowercase** (Latin-script tokens only — English/code-switch words;
Urdu script has no case).
Tokenization for WER: whitespace split of the normalized string.
Reference implementation: `c0_normalizer.py` (NORMALIZER_VERSION = "1.0.0"),
sha256 `7b040047a92b211be6a3341149269463b65461b091216da3e3653155c5deaf81`,
with unit tests in the project's stage_c test module. Every evaluation run
record cites the normalizer version used.
|