ocr-projects-archive / ocr-tech /docs /LIMBU_UNICODE_DIACRITIC_STATS.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
7.87 kB

Limbu Unicode And Diacritic Stats

Updated: 2026-06-24.

This note records the Unicode facts, shaping test, corpus statistics, and corrective coverage data for Sirijonga Limbu OCR training. Use it before generating new Limbu synthetic rows.

Primary Unicode Sources

The local Python runtime reports unicodedata.unidata_version=16.0.0; the checked Unicode 17.0.0 UCD still has the same assigned Limbu block facts used here.

Unicode Facts

The Limbu block is U+1900..U+194F. There are 68 assigned Limbu codepoints:

  • 31 bases: U+1900..U+191E
  • 9 vowel signs: U+1920..U+1928
  • 3 subjoined letters: U+1929..U+192B
  • 9 finals/small letters: U+1930..U+1938, including U+1932 Limbu small letter anusvara
  • 3 signs: U+1939..U+193B
  • 1 symbol: U+1940
  • 2 punctuation marks: U+1944..U+1945
  • 10 digits: U+1946..U+194F

Most Limbu marks have canonical combining class 0. Only the three signs have non-zero combining classes:

  • U+193B Limbu sign SA-I: 220
  • U+1939 Limbu sign MUKPHRENG: 222
  • U+193A Limbu sign KEMPHRENG: 230

Consequence: NFC/NFD will not fix most wrong Limbu mark orders. Vowel signs, subjoined letters, and most final signs are class 0, so generated text must put marks in sane order before rendering.

Shaping Test

Tool:

uv run python tools/analyze_limbu_unicode_shaping.py \
  --out outputs/limbu-unicode-shaping-analysis-v1-2026-06-24 \
  --random-order-samples 5000 \
  --random-canonical-samples 1000 \
  --seed 240624

Output:

  • outputs/limbu-unicode-shaping-analysis-v1-2026-06-24/limbu-shaping-cases.csv
  • outputs/limbu-unicode-shaping-analysis-v1-2026-06-24/limbu-shaping-summary.json
  • outputs/limbu-unicode-shaping-analysis-v1-2026-06-24/limbu-shaping-summary.md

Result using upstream Namdhinggo Regular:

cases=21891
clean=21891
visible=21891
dotted-circle cases=0
.notdef cases=0

Important interpretation: Namdhinggo is permissive. It renders even intentionally odd component orders without .notdef or dotted-circle glyphs. Font renderability is necessary, but it is not a grammar check.

Corpus Stats Before Coverage

Tool:

uv run python tools/analyze_limbu_unicode_corpus_stats.py \
  --manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-v1-2026-06-24/train.jsonl \
  --manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-v1-2026-06-24/eval.jsonl \
  --out outputs/limbu-unicode-corpus-stats-v1-2026-06-24 \
  --top-n 80

Result:

rows=58722
Limbu chars=2241565
covered codepoints=61/68
missing codepoints=7
NFC-changed rows=0
NFD-changed rows=0

Missing before coverage:

  • U+1909 Limbu letter JHA
  • U+190A Limbu letter YAN
  • U+191A Limbu letter SSA
  • U+191D Limbu letter GYAN
  • U+191E Limbu letter TRA
  • U+1932 Limbu small letter anusvara
  • U+1940 Limbu sign LOO

Under 1000 occurrences before coverage:

  • missing seven above
  • U+1919 SHA: 106
  • U+1924 vowel sign AI: 86
  • U+1926 vowel sign AU: 76
  • U+1928 vowel sign O: 63
  • U+192B subjoined WA: 748
  • U+1944 Limbu exclamation mark: 22
  • U+194E digit eight: 971
  • U+194F digit nine: 919

Dominant cluster signatures before coverage:

base+vowel=410919
base+vowel+final=258854
base+vowel+sign=58492
base+final=44527
base=39648
base+vowel+sign+final=28238
base+sign=28184
base+vowel+sign+vowel+final=15420
base+sign+vowel+final=7377
base+subjoined+vowel=3372

Existing-corpus warning counts:

has_duplicate_vowel=16757
orphan_mark=289
has_duplicate_final=82
has_duplicate_sign=11
has_mukphreng_after_final=16

These warnings are inherited from existing text sources and older synthetic rows. They are not proof that every warned sequence is unusable, but they show why new coverage generation must not blindly emit random component permutations.

Coverage Generation Fix

Generator:

uv run python tools/build_limbu_unicode_coverage_text.py \
  --baseline-manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-v1-2026-06-24/train.jsonl \
  --out-jsonl data/synthesis-text/limbu-sirijonga-unicode-coverage-v1-2026-06-24/coverage-text.jsonl \
  --target-count 1000 \
  --planned-render-variants 2 \
  --targets-per-line 3 \
  --seed 240624

The generator was corrected after the stats pass:

  • subjoined letters are emitted before vowel signs: base+subjoined+vowel
  • long closed syllables use the observed high-frequency order base+vowel+KEMPHRENG+final
  • no final-plus-sign context is added by the generator
  • U+1932 Limbu small letter anusvara is included in the finals list, not emitted as a standalone orphan mark

Coverage text result:

rows=1936
warnings={}
NFC-changed rows=0
NFD-changed rows=0

Rendered fabrication:

uv run python tools/build_limbu_targeted_fabrication_pack.py \
  --text-jsonl data/synthesis-text/limbu-sirijonga-unicode-coverage-v1-2026-06-24/coverage-text.jsonl \
  --width-profile-manifest outputs/limbu-gorkhapatra-pdf-backed-silver-2026-06-23-v3/splits/train.jsonl \
  --out outputs/limbu-unicode-coverage-fabrication-v1-2026-06-24 \
  --max-lines 0 \
  --variants-per-line 2 \
  --profile mixed \
  --seed 240625 \
  --force

Rendered result:

source rows=1936
rendered rows=3872
render rejects=0
dark-pixel min/p50/max=33/1880/49435
min-gray min/p50/max=0/16/124

Contact sheet visually checked at /tmp/limbu-unicode-coverage-fabrication-contact.png; crops are nonblank. A few are intentionally faint from the mixed degradation profile, but only 7/3872 have fewer than 100 dark pixels.

Combined Pack After Coverage

Builder command:

uv run python tools/build_limbu_combined_recognizer_pack.py \
  --out outputs/limbu-combined-synth-real-gorkhapatra-targeted-unicode-v1-2026-06-24 \
  --targeted-fabrication-train-manifest outputs/limbu-targeted-fabrication-v1-2026-06-24/manifests/train.jsonl \
  --unicode-coverage-fabrication-train-manifest outputs/limbu-unicode-coverage-fabrication-v1-2026-06-24/manifests/train.jsonl \
  --seed 240626

Result:

train rows=62339
eval rows=255
unicode-coverage-fabrication-v1 rows=3872
exact eval-text overlap drops=49
train audit passed=True
eval audit passed=True

Post-coverage Unicode stats:

uv run python tools/analyze_limbu_unicode_corpus_stats.py \
  --manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-unicode-v1-2026-06-24/train.jsonl \
  --manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-unicode-v1-2026-06-24/eval.jsonl \
  --out outputs/limbu-unicode-corpus-stats-after-coverage-v1-2026-06-24 \
  --top-n 80

Result:

rows=62594
Limbu chars=2307121
covered codepoints=68/68
missing codepoints=0
under 1000 occurrences=none
NFC-changed rows=0
NFD-changed rows=0

The old corpus warnings remain because the old rows remain. The new Unicode coverage text itself added no orphan marks or duplicate/order warnings.

Policy

The Unicode coverage pack is training-only synthetic data. It is not evaluation evidence and not claim evidence. Keep it as a separate training component named unicode-coverage-fabrication-v1 so the next recognizer run can ablate it against held-out real/silver Gorkhapatra eval and separate CDC/phone-page checks.