# Limbu Unicode And Diacritic Stats Updated: 2026-06-24. This note records the Unicode facts, shaping test, corpus statistics, and corrective coverage data for Sirijonga Limbu OCR training. Use it before generating new Limbu synthetic rows. ## Primary Unicode Sources - Unicode Character Database latest ReadMe: . Checked on 2026-06-24; it resolves to Unicode 17.0.0 final data dated 2025-08-15. - Unicode Character Database latest `UnicodeData.txt`: . - Unicode Character Database latest `Blocks.txt`: . - Unicode Standard core spec, South and Central Asia chapter, Limbu section: . The local Python runtime reports `unicodedata.unidata_version=16.0.0`; the checked Unicode 17.0.0 UCD still has the same assigned Limbu block facts used here. ## Unicode Facts The Limbu block is `U+1900..U+194F`. There are `68` assigned Limbu codepoints: - `31` bases: `U+1900..U+191E` - `9` vowel signs: `U+1920..U+1928` - `3` subjoined letters: `U+1929..U+192B` - `9` finals/small letters: `U+1930..U+1938`, including `U+1932` Limbu small letter anusvara - `3` signs: `U+1939..U+193B` - `1` symbol: `U+1940` - `2` punctuation marks: `U+1944..U+1945` - `10` digits: `U+1946..U+194F` Most Limbu marks have canonical combining class `0`. Only the three signs have non-zero combining classes: - `U+193B` Limbu sign SA-I: `220` - `U+1939` Limbu sign MUKPHRENG: `222` - `U+193A` Limbu sign KEMPHRENG: `230` Consequence: `NFC`/`NFD` will not fix most wrong Limbu mark orders. Vowel signs, subjoined letters, and most final signs are class `0`, so generated text must put marks in sane order before rendering. ## Shaping Test Tool: ```bash uv run python tools/analyze_limbu_unicode_shaping.py \ --out outputs/limbu-unicode-shaping-analysis-v1-2026-06-24 \ --random-order-samples 5000 \ --random-canonical-samples 1000 \ --seed 240624 ``` Output: - `outputs/limbu-unicode-shaping-analysis-v1-2026-06-24/limbu-shaping-cases.csv` - `outputs/limbu-unicode-shaping-analysis-v1-2026-06-24/limbu-shaping-summary.json` - `outputs/limbu-unicode-shaping-analysis-v1-2026-06-24/limbu-shaping-summary.md` Result using upstream Namdhinggo Regular: ```text cases=21891 clean=21891 visible=21891 dotted-circle cases=0 .notdef cases=0 ``` Important interpretation: Namdhinggo is permissive. It renders even intentionally odd component orders without `.notdef` or dotted-circle glyphs. Font renderability is necessary, but it is not a grammar check. ## Corpus Stats Before Coverage Tool: ```bash uv run python tools/analyze_limbu_unicode_corpus_stats.py \ --manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-v1-2026-06-24/train.jsonl \ --manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-v1-2026-06-24/eval.jsonl \ --out outputs/limbu-unicode-corpus-stats-v1-2026-06-24 \ --top-n 80 ``` Result: ```text rows=58722 Limbu chars=2241565 covered codepoints=61/68 missing codepoints=7 NFC-changed rows=0 NFD-changed rows=0 ``` Missing before coverage: - `U+1909` Limbu letter JHA - `U+190A` Limbu letter YAN - `U+191A` Limbu letter SSA - `U+191D` Limbu letter GYAN - `U+191E` Limbu letter TRA - `U+1932` Limbu small letter anusvara - `U+1940` Limbu sign LOO Under `1000` occurrences before coverage: - missing seven above - `U+1919` SHA: `106` - `U+1924` vowel sign AI: `86` - `U+1926` vowel sign AU: `76` - `U+1928` vowel sign O: `63` - `U+192B` subjoined WA: `748` - `U+1944` Limbu exclamation mark: `22` - `U+194E` digit eight: `971` - `U+194F` digit nine: `919` Dominant cluster signatures before coverage: ```text base+vowel=410919 base+vowel+final=258854 base+vowel+sign=58492 base+final=44527 base=39648 base+vowel+sign+final=28238 base+sign=28184 base+vowel+sign+vowel+final=15420 base+sign+vowel+final=7377 base+subjoined+vowel=3372 ``` Existing-corpus warning counts: ```text has_duplicate_vowel=16757 orphan_mark=289 has_duplicate_final=82 has_duplicate_sign=11 has_mukphreng_after_final=16 ``` These warnings are inherited from existing text sources and older synthetic rows. They are not proof that every warned sequence is unusable, but they show why new coverage generation must not blindly emit random component permutations. ## Coverage Generation Fix Generator: ```bash uv run python tools/build_limbu_unicode_coverage_text.py \ --baseline-manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-v1-2026-06-24/train.jsonl \ --out-jsonl data/synthesis-text/limbu-sirijonga-unicode-coverage-v1-2026-06-24/coverage-text.jsonl \ --target-count 1000 \ --planned-render-variants 2 \ --targets-per-line 3 \ --seed 240624 ``` The generator was corrected after the stats pass: - subjoined letters are emitted before vowel signs: `base+subjoined+vowel` - long closed syllables use the observed high-frequency order `base+vowel+KEMPHRENG+final` - no final-plus-sign context is added by the generator - `U+1932` Limbu small letter anusvara is included in the finals list, not emitted as a standalone orphan mark Coverage text result: ```text rows=1936 warnings={} NFC-changed rows=0 NFD-changed rows=0 ``` Rendered fabrication: ```bash uv run python tools/build_limbu_targeted_fabrication_pack.py \ --text-jsonl data/synthesis-text/limbu-sirijonga-unicode-coverage-v1-2026-06-24/coverage-text.jsonl \ --width-profile-manifest outputs/limbu-gorkhapatra-pdf-backed-silver-2026-06-23-v3/splits/train.jsonl \ --out outputs/limbu-unicode-coverage-fabrication-v1-2026-06-24 \ --max-lines 0 \ --variants-per-line 2 \ --profile mixed \ --seed 240625 \ --force ``` Rendered result: ```text source rows=1936 rendered rows=3872 render rejects=0 dark-pixel min/p50/max=33/1880/49435 min-gray min/p50/max=0/16/124 ``` Contact sheet visually checked at `/tmp/limbu-unicode-coverage-fabrication-contact.png`; crops are nonblank. A few are intentionally faint from the mixed degradation profile, but only `7/3872` have fewer than `100` dark pixels. ## Combined Pack After Coverage Builder command: ```bash uv run python tools/build_limbu_combined_recognizer_pack.py \ --out outputs/limbu-combined-synth-real-gorkhapatra-targeted-unicode-v1-2026-06-24 \ --targeted-fabrication-train-manifest outputs/limbu-targeted-fabrication-v1-2026-06-24/manifests/train.jsonl \ --unicode-coverage-fabrication-train-manifest outputs/limbu-unicode-coverage-fabrication-v1-2026-06-24/manifests/train.jsonl \ --seed 240626 ``` Result: ```text train rows=62339 eval rows=255 unicode-coverage-fabrication-v1 rows=3872 exact eval-text overlap drops=49 train audit passed=True eval audit passed=True ``` Post-coverage Unicode stats: ```bash uv run python tools/analyze_limbu_unicode_corpus_stats.py \ --manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-unicode-v1-2026-06-24/train.jsonl \ --manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-unicode-v1-2026-06-24/eval.jsonl \ --out outputs/limbu-unicode-corpus-stats-after-coverage-v1-2026-06-24 \ --top-n 80 ``` Result: ```text rows=62594 Limbu chars=2307121 covered codepoints=68/68 missing codepoints=0 under 1000 occurrences=none NFC-changed rows=0 NFD-changed rows=0 ``` The old corpus warnings remain because the old rows remain. The new Unicode coverage text itself added no orphan marks or duplicate/order warnings. ## Policy The Unicode coverage pack is training-only synthetic data. It is not evaluation evidence and not claim evidence. Keep it as a separate training component named `unicode-coverage-fabrication-v1` so the next recognizer run can ablate it against held-out real/silver Gorkhapatra eval and separate CDC/phone-page checks.