|
Download ocr-tech/docs/LIMBU_UNICODE_DIACRITIC_STATS.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 7.87 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/LIMBU_UNICODE_DIACRITIC_STATS.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/ocr-tech/docs/LIMBU_UNICODE_DIACRITIC_STATS.md
-
curl -L -o LIMBU_UNICODE_DIACRITIC_STATS.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/LIMBU_UNICODE_DIACRITIC_STATS.md
7.87 kB
| # Limbu Unicode And Diacritic Stats | |
| Updated: 2026-06-24. | |
| This note records the Unicode facts, shaping test, corpus statistics, and corrective | |
| coverage data for Sirijonga Limbu OCR training. Use it before generating new Limbu | |
| synthetic rows. | |
| ## Primary Unicode Sources | |
| - Unicode Character Database latest ReadMe: | |
| <https://www.unicode.org/Public/UCD/latest/ucd/ReadMe.txt>. Checked on | |
| 2026-06-24; it resolves to Unicode 17.0.0 final data dated 2025-08-15. | |
| - Unicode Character Database latest `UnicodeData.txt`: | |
| <https://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt>. | |
| - Unicode Character Database latest `Blocks.txt`: | |
| <https://www.unicode.org/Public/UCD/latest/ucd/Blocks.txt>. | |
| - Unicode Standard core spec, South and Central Asia chapter, Limbu section: | |
| <https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-13/>. | |
| The local Python runtime reports `unicodedata.unidata_version=16.0.0`; the checked | |
| Unicode 17.0.0 UCD still has the same assigned Limbu block facts used here. | |
| ## Unicode Facts | |
| The Limbu block is `U+1900..U+194F`. There are `68` assigned Limbu codepoints: | |
| - `31` bases: `U+1900..U+191E` | |
| - `9` vowel signs: `U+1920..U+1928` | |
| - `3` subjoined letters: `U+1929..U+192B` | |
| - `9` finals/small letters: `U+1930..U+1938`, including `U+1932` Limbu small | |
| letter anusvara | |
| - `3` signs: `U+1939..U+193B` | |
| - `1` symbol: `U+1940` | |
| - `2` punctuation marks: `U+1944..U+1945` | |
| - `10` digits: `U+1946..U+194F` | |
| Most Limbu marks have canonical combining class `0`. Only the three signs have | |
| non-zero combining classes: | |
| - `U+193B` Limbu sign SA-I: `220` | |
| - `U+1939` Limbu sign MUKPHRENG: `222` | |
| - `U+193A` Limbu sign KEMPHRENG: `230` | |
| Consequence: `NFC`/`NFD` will not fix most wrong Limbu mark orders. Vowel signs, | |
| subjoined letters, and most final signs are class `0`, so generated text must put | |
| marks in sane order before rendering. | |
| ## Shaping Test | |
| Tool: | |
| ```bash | |
| uv run python tools/analyze_limbu_unicode_shaping.py \ | |
| --out outputs/limbu-unicode-shaping-analysis-v1-2026-06-24 \ | |
| --random-order-samples 5000 \ | |
| --random-canonical-samples 1000 \ | |
| --seed 240624 | |
| ``` | |
| Output: | |
| - `outputs/limbu-unicode-shaping-analysis-v1-2026-06-24/limbu-shaping-cases.csv` | |
| - `outputs/limbu-unicode-shaping-analysis-v1-2026-06-24/limbu-shaping-summary.json` | |
| - `outputs/limbu-unicode-shaping-analysis-v1-2026-06-24/limbu-shaping-summary.md` | |
| Result using upstream Namdhinggo Regular: | |
| ```text | |
| cases=21891 | |
| clean=21891 | |
| visible=21891 | |
| dotted-circle cases=0 | |
| .notdef cases=0 | |
| ``` | |
| Important interpretation: Namdhinggo is permissive. It renders even intentionally | |
| odd component orders without `.notdef` or dotted-circle glyphs. Font renderability | |
| is necessary, but it is not a grammar check. | |
| ## Corpus Stats Before Coverage | |
| Tool: | |
| ```bash | |
| uv run python tools/analyze_limbu_unicode_corpus_stats.py \ | |
| --manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-v1-2026-06-24/train.jsonl \ | |
| --manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-v1-2026-06-24/eval.jsonl \ | |
| --out outputs/limbu-unicode-corpus-stats-v1-2026-06-24 \ | |
| --top-n 80 | |
| ``` | |
| Result: | |
| ```text | |
| rows=58722 | |
| Limbu chars=2241565 | |
| covered codepoints=61/68 | |
| missing codepoints=7 | |
| NFC-changed rows=0 | |
| NFD-changed rows=0 | |
| ``` | |
| Missing before coverage: | |
| - `U+1909` Limbu letter JHA | |
| - `U+190A` Limbu letter YAN | |
| - `U+191A` Limbu letter SSA | |
| - `U+191D` Limbu letter GYAN | |
| - `U+191E` Limbu letter TRA | |
| - `U+1932` Limbu small letter anusvara | |
| - `U+1940` Limbu sign LOO | |
| Under `1000` occurrences before coverage: | |
| - missing seven above | |
| - `U+1919` SHA: `106` | |
| - `U+1924` vowel sign AI: `86` | |
| - `U+1926` vowel sign AU: `76` | |
| - `U+1928` vowel sign O: `63` | |
| - `U+192B` subjoined WA: `748` | |
| - `U+1944` Limbu exclamation mark: `22` | |
| - `U+194E` digit eight: `971` | |
| - `U+194F` digit nine: `919` | |
| Dominant cluster signatures before coverage: | |
| ```text | |
| base+vowel=410919 | |
| base+vowel+final=258854 | |
| base+vowel+sign=58492 | |
| base+final=44527 | |
| base=39648 | |
| base+vowel+sign+final=28238 | |
| base+sign=28184 | |
| base+vowel+sign+vowel+final=15420 | |
| base+sign+vowel+final=7377 | |
| base+subjoined+vowel=3372 | |
| ``` | |
| Existing-corpus warning counts: | |
| ```text | |
| has_duplicate_vowel=16757 | |
| orphan_mark=289 | |
| has_duplicate_final=82 | |
| has_duplicate_sign=11 | |
| has_mukphreng_after_final=16 | |
| ``` | |
| These warnings are inherited from existing text sources and older synthetic rows. | |
| They are not proof that every warned sequence is unusable, but they show why new | |
| coverage generation must not blindly emit random component permutations. | |
| ## Coverage Generation Fix | |
| Generator: | |
| ```bash | |
| uv run python tools/build_limbu_unicode_coverage_text.py \ | |
| --baseline-manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-v1-2026-06-24/train.jsonl \ | |
| --out-jsonl data/synthesis-text/limbu-sirijonga-unicode-coverage-v1-2026-06-24/coverage-text.jsonl \ | |
| --target-count 1000 \ | |
| --planned-render-variants 2 \ | |
| --targets-per-line 3 \ | |
| --seed 240624 | |
| ``` | |
| The generator was corrected after the stats pass: | |
| - subjoined letters are emitted before vowel signs: `base+subjoined+vowel` | |
| - long closed syllables use the observed high-frequency order | |
| `base+vowel+KEMPHRENG+final` | |
| - no final-plus-sign context is added by the generator | |
| - `U+1932` Limbu small letter anusvara is included in the finals list, not emitted | |
| as a standalone orphan mark | |
| Coverage text result: | |
| ```text | |
| rows=1936 | |
| warnings={} | |
| NFC-changed rows=0 | |
| NFD-changed rows=0 | |
| ``` | |
| Rendered fabrication: | |
| ```bash | |
| uv run python tools/build_limbu_targeted_fabrication_pack.py \ | |
| --text-jsonl data/synthesis-text/limbu-sirijonga-unicode-coverage-v1-2026-06-24/coverage-text.jsonl \ | |
| --width-profile-manifest outputs/limbu-gorkhapatra-pdf-backed-silver-2026-06-23-v3/splits/train.jsonl \ | |
| --out outputs/limbu-unicode-coverage-fabrication-v1-2026-06-24 \ | |
| --max-lines 0 \ | |
| --variants-per-line 2 \ | |
| --profile mixed \ | |
| --seed 240625 \ | |
| --force | |
| ``` | |
| Rendered result: | |
| ```text | |
| source rows=1936 | |
| rendered rows=3872 | |
| render rejects=0 | |
| dark-pixel min/p50/max=33/1880/49435 | |
| min-gray min/p50/max=0/16/124 | |
| ``` | |
| Contact sheet visually checked at | |
| `/tmp/limbu-unicode-coverage-fabrication-contact.png`; crops are nonblank. A few are | |
| intentionally faint from the mixed degradation profile, but only `7/3872` have fewer | |
| than `100` dark pixels. | |
| ## Combined Pack After Coverage | |
| Builder command: | |
| ```bash | |
| uv run python tools/build_limbu_combined_recognizer_pack.py \ | |
| --out outputs/limbu-combined-synth-real-gorkhapatra-targeted-unicode-v1-2026-06-24 \ | |
| --targeted-fabrication-train-manifest outputs/limbu-targeted-fabrication-v1-2026-06-24/manifests/train.jsonl \ | |
| --unicode-coverage-fabrication-train-manifest outputs/limbu-unicode-coverage-fabrication-v1-2026-06-24/manifests/train.jsonl \ | |
| --seed 240626 | |
| ``` | |
| Result: | |
| ```text | |
| train rows=62339 | |
| eval rows=255 | |
| unicode-coverage-fabrication-v1 rows=3872 | |
| exact eval-text overlap drops=49 | |
| train audit passed=True | |
| eval audit passed=True | |
| ``` | |
| Post-coverage Unicode stats: | |
| ```bash | |
| uv run python tools/analyze_limbu_unicode_corpus_stats.py \ | |
| --manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-unicode-v1-2026-06-24/train.jsonl \ | |
| --manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-unicode-v1-2026-06-24/eval.jsonl \ | |
| --out outputs/limbu-unicode-corpus-stats-after-coverage-v1-2026-06-24 \ | |
| --top-n 80 | |
| ``` | |
| Result: | |
| ```text | |
| rows=62594 | |
| Limbu chars=2307121 | |
| covered codepoints=68/68 | |
| missing codepoints=0 | |
| under 1000 occurrences=none | |
| NFC-changed rows=0 | |
| NFD-changed rows=0 | |
| ``` | |
| The old corpus warnings remain because the old rows remain. The new Unicode coverage | |
| text itself added no orphan marks or duplicate/order warnings. | |
| ## Policy | |
| The Unicode coverage pack is training-only synthetic data. It is not evaluation | |
| evidence and not claim evidence. Keep it as a separate training component named | |
| `unicode-coverage-fabrication-v1` so the next recognizer run can ablate it against | |
| held-out real/silver Gorkhapatra eval and separate CDC/phone-page checks. | |