ocr-projects-archive / ocr-tech /docs /LIMBU_UNICODE_DIACRITIC_STATS.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
7.87 kB
# Limbu Unicode And Diacritic Stats
Updated: 2026-06-24.
This note records the Unicode facts, shaping test, corpus statistics, and corrective
coverage data for Sirijonga Limbu OCR training. Use it before generating new Limbu
synthetic rows.
## Primary Unicode Sources
- Unicode Character Database latest ReadMe:
<https://www.unicode.org/Public/UCD/latest/ucd/ReadMe.txt>. Checked on
2026-06-24; it resolves to Unicode 17.0.0 final data dated 2025-08-15.
- Unicode Character Database latest `UnicodeData.txt`:
<https://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt>.
- Unicode Character Database latest `Blocks.txt`:
<https://www.unicode.org/Public/UCD/latest/ucd/Blocks.txt>.
- Unicode Standard core spec, South and Central Asia chapter, Limbu section:
<https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-13/>.
The local Python runtime reports `unicodedata.unidata_version=16.0.0`; the checked
Unicode 17.0.0 UCD still has the same assigned Limbu block facts used here.
## Unicode Facts
The Limbu block is `U+1900..U+194F`. There are `68` assigned Limbu codepoints:
- `31` bases: `U+1900..U+191E`
- `9` vowel signs: `U+1920..U+1928`
- `3` subjoined letters: `U+1929..U+192B`
- `9` finals/small letters: `U+1930..U+1938`, including `U+1932` Limbu small
letter anusvara
- `3` signs: `U+1939..U+193B`
- `1` symbol: `U+1940`
- `2` punctuation marks: `U+1944..U+1945`
- `10` digits: `U+1946..U+194F`
Most Limbu marks have canonical combining class `0`. Only the three signs have
non-zero combining classes:
- `U+193B` Limbu sign SA-I: `220`
- `U+1939` Limbu sign MUKPHRENG: `222`
- `U+193A` Limbu sign KEMPHRENG: `230`
Consequence: `NFC`/`NFD` will not fix most wrong Limbu mark orders. Vowel signs,
subjoined letters, and most final signs are class `0`, so generated text must put
marks in sane order before rendering.
## Shaping Test
Tool:
```bash
uv run python tools/analyze_limbu_unicode_shaping.py \
--out outputs/limbu-unicode-shaping-analysis-v1-2026-06-24 \
--random-order-samples 5000 \
--random-canonical-samples 1000 \
--seed 240624
```
Output:
- `outputs/limbu-unicode-shaping-analysis-v1-2026-06-24/limbu-shaping-cases.csv`
- `outputs/limbu-unicode-shaping-analysis-v1-2026-06-24/limbu-shaping-summary.json`
- `outputs/limbu-unicode-shaping-analysis-v1-2026-06-24/limbu-shaping-summary.md`
Result using upstream Namdhinggo Regular:
```text
cases=21891
clean=21891
visible=21891
dotted-circle cases=0
.notdef cases=0
```
Important interpretation: Namdhinggo is permissive. It renders even intentionally
odd component orders without `.notdef` or dotted-circle glyphs. Font renderability
is necessary, but it is not a grammar check.
## Corpus Stats Before Coverage
Tool:
```bash
uv run python tools/analyze_limbu_unicode_corpus_stats.py \
--manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-v1-2026-06-24/train.jsonl \
--manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-v1-2026-06-24/eval.jsonl \
--out outputs/limbu-unicode-corpus-stats-v1-2026-06-24 \
--top-n 80
```
Result:
```text
rows=58722
Limbu chars=2241565
covered codepoints=61/68
missing codepoints=7
NFC-changed rows=0
NFD-changed rows=0
```
Missing before coverage:
- `U+1909` Limbu letter JHA
- `U+190A` Limbu letter YAN
- `U+191A` Limbu letter SSA
- `U+191D` Limbu letter GYAN
- `U+191E` Limbu letter TRA
- `U+1932` Limbu small letter anusvara
- `U+1940` Limbu sign LOO
Under `1000` occurrences before coverage:
- missing seven above
- `U+1919` SHA: `106`
- `U+1924` vowel sign AI: `86`
- `U+1926` vowel sign AU: `76`
- `U+1928` vowel sign O: `63`
- `U+192B` subjoined WA: `748`
- `U+1944` Limbu exclamation mark: `22`
- `U+194E` digit eight: `971`
- `U+194F` digit nine: `919`
Dominant cluster signatures before coverage:
```text
base+vowel=410919
base+vowel+final=258854
base+vowel+sign=58492
base+final=44527
base=39648
base+vowel+sign+final=28238
base+sign=28184
base+vowel+sign+vowel+final=15420
base+sign+vowel+final=7377
base+subjoined+vowel=3372
```
Existing-corpus warning counts:
```text
has_duplicate_vowel=16757
orphan_mark=289
has_duplicate_final=82
has_duplicate_sign=11
has_mukphreng_after_final=16
```
These warnings are inherited from existing text sources and older synthetic rows.
They are not proof that every warned sequence is unusable, but they show why new
coverage generation must not blindly emit random component permutations.
## Coverage Generation Fix
Generator:
```bash
uv run python tools/build_limbu_unicode_coverage_text.py \
--baseline-manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-v1-2026-06-24/train.jsonl \
--out-jsonl data/synthesis-text/limbu-sirijonga-unicode-coverage-v1-2026-06-24/coverage-text.jsonl \
--target-count 1000 \
--planned-render-variants 2 \
--targets-per-line 3 \
--seed 240624
```
The generator was corrected after the stats pass:
- subjoined letters are emitted before vowel signs: `base+subjoined+vowel`
- long closed syllables use the observed high-frequency order
`base+vowel+KEMPHRENG+final`
- no final-plus-sign context is added by the generator
- `U+1932` Limbu small letter anusvara is included in the finals list, not emitted
as a standalone orphan mark
Coverage text result:
```text
rows=1936
warnings={}
NFC-changed rows=0
NFD-changed rows=0
```
Rendered fabrication:
```bash
uv run python tools/build_limbu_targeted_fabrication_pack.py \
--text-jsonl data/synthesis-text/limbu-sirijonga-unicode-coverage-v1-2026-06-24/coverage-text.jsonl \
--width-profile-manifest outputs/limbu-gorkhapatra-pdf-backed-silver-2026-06-23-v3/splits/train.jsonl \
--out outputs/limbu-unicode-coverage-fabrication-v1-2026-06-24 \
--max-lines 0 \
--variants-per-line 2 \
--profile mixed \
--seed 240625 \
--force
```
Rendered result:
```text
source rows=1936
rendered rows=3872
render rejects=0
dark-pixel min/p50/max=33/1880/49435
min-gray min/p50/max=0/16/124
```
Contact sheet visually checked at
`/tmp/limbu-unicode-coverage-fabrication-contact.png`; crops are nonblank. A few are
intentionally faint from the mixed degradation profile, but only `7/3872` have fewer
than `100` dark pixels.
## Combined Pack After Coverage
Builder command:
```bash
uv run python tools/build_limbu_combined_recognizer_pack.py \
--out outputs/limbu-combined-synth-real-gorkhapatra-targeted-unicode-v1-2026-06-24 \
--targeted-fabrication-train-manifest outputs/limbu-targeted-fabrication-v1-2026-06-24/manifests/train.jsonl \
--unicode-coverage-fabrication-train-manifest outputs/limbu-unicode-coverage-fabrication-v1-2026-06-24/manifests/train.jsonl \
--seed 240626
```
Result:
```text
train rows=62339
eval rows=255
unicode-coverage-fabrication-v1 rows=3872
exact eval-text overlap drops=49
train audit passed=True
eval audit passed=True
```
Post-coverage Unicode stats:
```bash
uv run python tools/analyze_limbu_unicode_corpus_stats.py \
--manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-unicode-v1-2026-06-24/train.jsonl \
--manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-unicode-v1-2026-06-24/eval.jsonl \
--out outputs/limbu-unicode-corpus-stats-after-coverage-v1-2026-06-24 \
--top-n 80
```
Result:
```text
rows=62594
Limbu chars=2307121
covered codepoints=68/68
missing codepoints=0
under 1000 occurrences=none
NFC-changed rows=0
NFD-changed rows=0
```
The old corpus warnings remain because the old rows remain. The new Unicode coverage
text itself added no orphan marks or duplicate/order warnings.
## Policy
The Unicode coverage pack is training-only synthetic data. It is not evaluation
evidence and not claim evidence. Keep it as a separate training component named
`unicode-coverage-fabrication-v1` so the next recognizer run can ablate it against
held-out real/silver Gorkhapatra eval and separate CDC/phone-page checks.