Download ocr-tech/docs/LIMBU_UNICODE_DIACRITIC_STATS.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 7.87 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/LIMBU_UNICODE_DIACRITIC_STATS.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/ocr-tech/docs/LIMBU_UNICODE_DIACRITIC_STATS.md
-
curl -L -o LIMBU_UNICODE_DIACRITIC_STATS.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/LIMBU_UNICODE_DIACRITIC_STATS.md
Limbu Unicode And Diacritic Stats
Updated: 2026-06-24.
This note records the Unicode facts, shaping test, corpus statistics, and corrective coverage data for Sirijonga Limbu OCR training. Use it before generating new Limbu synthetic rows.
Primary Unicode Sources
- Unicode Character Database latest ReadMe: https://www.unicode.org/Public/UCD/latest/ucd/ReadMe.txt. Checked on 2026-06-24; it resolves to Unicode 17.0.0 final data dated 2025-08-15.
- Unicode Character Database latest
UnicodeData.txt: https://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt. - Unicode Character Database latest
Blocks.txt: https://www.unicode.org/Public/UCD/latest/ucd/Blocks.txt. - Unicode Standard core spec, South and Central Asia chapter, Limbu section: https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-13/.
The local Python runtime reports unicodedata.unidata_version=16.0.0; the checked
Unicode 17.0.0 UCD still has the same assigned Limbu block facts used here.
Unicode Facts
The Limbu block is U+1900..U+194F. There are 68 assigned Limbu codepoints:
31bases:U+1900..U+191E9vowel signs:U+1920..U+19283subjoined letters:U+1929..U+192B9finals/small letters:U+1930..U+1938, includingU+1932Limbu small letter anusvara3signs:U+1939..U+193B1symbol:U+19402punctuation marks:U+1944..U+194510digits:U+1946..U+194F
Most Limbu marks have canonical combining class 0. Only the three signs have
non-zero combining classes:
U+193BLimbu sign SA-I:220U+1939Limbu sign MUKPHRENG:222U+193ALimbu sign KEMPHRENG:230
Consequence: NFC/NFD will not fix most wrong Limbu mark orders. Vowel signs,
subjoined letters, and most final signs are class 0, so generated text must put
marks in sane order before rendering.
Shaping Test
Tool:
uv run python tools/analyze_limbu_unicode_shaping.py \
--out outputs/limbu-unicode-shaping-analysis-v1-2026-06-24 \
--random-order-samples 5000 \
--random-canonical-samples 1000 \
--seed 240624
Output:
outputs/limbu-unicode-shaping-analysis-v1-2026-06-24/limbu-shaping-cases.csvoutputs/limbu-unicode-shaping-analysis-v1-2026-06-24/limbu-shaping-summary.jsonoutputs/limbu-unicode-shaping-analysis-v1-2026-06-24/limbu-shaping-summary.md
Result using upstream Namdhinggo Regular:
cases=21891
clean=21891
visible=21891
dotted-circle cases=0
.notdef cases=0
Important interpretation: Namdhinggo is permissive. It renders even intentionally
odd component orders without .notdef or dotted-circle glyphs. Font renderability
is necessary, but it is not a grammar check.
Corpus Stats Before Coverage
Tool:
uv run python tools/analyze_limbu_unicode_corpus_stats.py \
--manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-v1-2026-06-24/train.jsonl \
--manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-v1-2026-06-24/eval.jsonl \
--out outputs/limbu-unicode-corpus-stats-v1-2026-06-24 \
--top-n 80
Result:
rows=58722
Limbu chars=2241565
covered codepoints=61/68
missing codepoints=7
NFC-changed rows=0
NFD-changed rows=0
Missing before coverage:
U+1909Limbu letter JHAU+190ALimbu letter YANU+191ALimbu letter SSAU+191DLimbu letter GYANU+191ELimbu letter TRAU+1932Limbu small letter anusvaraU+1940Limbu sign LOO
Under 1000 occurrences before coverage:
- missing seven above
U+1919SHA:106U+1924vowel sign AI:86U+1926vowel sign AU:76U+1928vowel sign O:63U+192Bsubjoined WA:748U+1944Limbu exclamation mark:22U+194Edigit eight:971U+194Fdigit nine:919
Dominant cluster signatures before coverage:
base+vowel=410919
base+vowel+final=258854
base+vowel+sign=58492
base+final=44527
base=39648
base+vowel+sign+final=28238
base+sign=28184
base+vowel+sign+vowel+final=15420
base+sign+vowel+final=7377
base+subjoined+vowel=3372
Existing-corpus warning counts:
has_duplicate_vowel=16757
orphan_mark=289
has_duplicate_final=82
has_duplicate_sign=11
has_mukphreng_after_final=16
These warnings are inherited from existing text sources and older synthetic rows. They are not proof that every warned sequence is unusable, but they show why new coverage generation must not blindly emit random component permutations.
Coverage Generation Fix
Generator:
uv run python tools/build_limbu_unicode_coverage_text.py \
--baseline-manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-v1-2026-06-24/train.jsonl \
--out-jsonl data/synthesis-text/limbu-sirijonga-unicode-coverage-v1-2026-06-24/coverage-text.jsonl \
--target-count 1000 \
--planned-render-variants 2 \
--targets-per-line 3 \
--seed 240624
The generator was corrected after the stats pass:
- subjoined letters are emitted before vowel signs:
base+subjoined+vowel - long closed syllables use the observed high-frequency order
base+vowel+KEMPHRENG+final - no final-plus-sign context is added by the generator
U+1932Limbu small letter anusvara is included in the finals list, not emitted as a standalone orphan mark
Coverage text result:
rows=1936
warnings={}
NFC-changed rows=0
NFD-changed rows=0
Rendered fabrication:
uv run python tools/build_limbu_targeted_fabrication_pack.py \
--text-jsonl data/synthesis-text/limbu-sirijonga-unicode-coverage-v1-2026-06-24/coverage-text.jsonl \
--width-profile-manifest outputs/limbu-gorkhapatra-pdf-backed-silver-2026-06-23-v3/splits/train.jsonl \
--out outputs/limbu-unicode-coverage-fabrication-v1-2026-06-24 \
--max-lines 0 \
--variants-per-line 2 \
--profile mixed \
--seed 240625 \
--force
Rendered result:
source rows=1936
rendered rows=3872
render rejects=0
dark-pixel min/p50/max=33/1880/49435
min-gray min/p50/max=0/16/124
Contact sheet visually checked at
/tmp/limbu-unicode-coverage-fabrication-contact.png; crops are nonblank. A few are
intentionally faint from the mixed degradation profile, but only 7/3872 have fewer
than 100 dark pixels.
Combined Pack After Coverage
Builder command:
uv run python tools/build_limbu_combined_recognizer_pack.py \
--out outputs/limbu-combined-synth-real-gorkhapatra-targeted-unicode-v1-2026-06-24 \
--targeted-fabrication-train-manifest outputs/limbu-targeted-fabrication-v1-2026-06-24/manifests/train.jsonl \
--unicode-coverage-fabrication-train-manifest outputs/limbu-unicode-coverage-fabrication-v1-2026-06-24/manifests/train.jsonl \
--seed 240626
Result:
train rows=62339
eval rows=255
unicode-coverage-fabrication-v1 rows=3872
exact eval-text overlap drops=49
train audit passed=True
eval audit passed=True
Post-coverage Unicode stats:
uv run python tools/analyze_limbu_unicode_corpus_stats.py \
--manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-unicode-v1-2026-06-24/train.jsonl \
--manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-unicode-v1-2026-06-24/eval.jsonl \
--out outputs/limbu-unicode-corpus-stats-after-coverage-v1-2026-06-24 \
--top-n 80
Result:
rows=62594
Limbu chars=2307121
covered codepoints=68/68
missing codepoints=0
under 1000 occurrences=none
NFC-changed rows=0
NFD-changed rows=0
The old corpus warnings remain because the old rows remain. The new Unicode coverage text itself added no orphan marks or duplicate/order warnings.
Policy
The Unicode coverage pack is training-only synthetic data. It is not evaluation
evidence and not claim evidence. Keep it as a separate training component named
unicode-coverage-fabrication-v1 so the next recognizer run can ablate it against
held-out real/silver Gorkhapatra eval and separate CDC/phone-page checks.