File size: 7,871 Bytes
f9bc0c1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 | # Limbu Unicode And Diacritic Stats
Updated: 2026-06-24.
This note records the Unicode facts, shaping test, corpus statistics, and corrective
coverage data for Sirijonga Limbu OCR training. Use it before generating new Limbu
synthetic rows.
## Primary Unicode Sources
- Unicode Character Database latest ReadMe:
<https://www.unicode.org/Public/UCD/latest/ucd/ReadMe.txt>. Checked on
2026-06-24; it resolves to Unicode 17.0.0 final data dated 2025-08-15.
- Unicode Character Database latest `UnicodeData.txt`:
<https://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt>.
- Unicode Character Database latest `Blocks.txt`:
<https://www.unicode.org/Public/UCD/latest/ucd/Blocks.txt>.
- Unicode Standard core spec, South and Central Asia chapter, Limbu section:
<https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-13/>.
The local Python runtime reports `unicodedata.unidata_version=16.0.0`; the checked
Unicode 17.0.0 UCD still has the same assigned Limbu block facts used here.
## Unicode Facts
The Limbu block is `U+1900..U+194F`. There are `68` assigned Limbu codepoints:
- `31` bases: `U+1900..U+191E`
- `9` vowel signs: `U+1920..U+1928`
- `3` subjoined letters: `U+1929..U+192B`
- `9` finals/small letters: `U+1930..U+1938`, including `U+1932` Limbu small
letter anusvara
- `3` signs: `U+1939..U+193B`
- `1` symbol: `U+1940`
- `2` punctuation marks: `U+1944..U+1945`
- `10` digits: `U+1946..U+194F`
Most Limbu marks have canonical combining class `0`. Only the three signs have
non-zero combining classes:
- `U+193B` Limbu sign SA-I: `220`
- `U+1939` Limbu sign MUKPHRENG: `222`
- `U+193A` Limbu sign KEMPHRENG: `230`
Consequence: `NFC`/`NFD` will not fix most wrong Limbu mark orders. Vowel signs,
subjoined letters, and most final signs are class `0`, so generated text must put
marks in sane order before rendering.
## Shaping Test
Tool:
```bash
uv run python tools/analyze_limbu_unicode_shaping.py \
--out outputs/limbu-unicode-shaping-analysis-v1-2026-06-24 \
--random-order-samples 5000 \
--random-canonical-samples 1000 \
--seed 240624
```
Output:
- `outputs/limbu-unicode-shaping-analysis-v1-2026-06-24/limbu-shaping-cases.csv`
- `outputs/limbu-unicode-shaping-analysis-v1-2026-06-24/limbu-shaping-summary.json`
- `outputs/limbu-unicode-shaping-analysis-v1-2026-06-24/limbu-shaping-summary.md`
Result using upstream Namdhinggo Regular:
```text
cases=21891
clean=21891
visible=21891
dotted-circle cases=0
.notdef cases=0
```
Important interpretation: Namdhinggo is permissive. It renders even intentionally
odd component orders without `.notdef` or dotted-circle glyphs. Font renderability
is necessary, but it is not a grammar check.
## Corpus Stats Before Coverage
Tool:
```bash
uv run python tools/analyze_limbu_unicode_corpus_stats.py \
--manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-v1-2026-06-24/train.jsonl \
--manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-v1-2026-06-24/eval.jsonl \
--out outputs/limbu-unicode-corpus-stats-v1-2026-06-24 \
--top-n 80
```
Result:
```text
rows=58722
Limbu chars=2241565
covered codepoints=61/68
missing codepoints=7
NFC-changed rows=0
NFD-changed rows=0
```
Missing before coverage:
- `U+1909` Limbu letter JHA
- `U+190A` Limbu letter YAN
- `U+191A` Limbu letter SSA
- `U+191D` Limbu letter GYAN
- `U+191E` Limbu letter TRA
- `U+1932` Limbu small letter anusvara
- `U+1940` Limbu sign LOO
Under `1000` occurrences before coverage:
- missing seven above
- `U+1919` SHA: `106`
- `U+1924` vowel sign AI: `86`
- `U+1926` vowel sign AU: `76`
- `U+1928` vowel sign O: `63`
- `U+192B` subjoined WA: `748`
- `U+1944` Limbu exclamation mark: `22`
- `U+194E` digit eight: `971`
- `U+194F` digit nine: `919`
Dominant cluster signatures before coverage:
```text
base+vowel=410919
base+vowel+final=258854
base+vowel+sign=58492
base+final=44527
base=39648
base+vowel+sign+final=28238
base+sign=28184
base+vowel+sign+vowel+final=15420
base+sign+vowel+final=7377
base+subjoined+vowel=3372
```
Existing-corpus warning counts:
```text
has_duplicate_vowel=16757
orphan_mark=289
has_duplicate_final=82
has_duplicate_sign=11
has_mukphreng_after_final=16
```
These warnings are inherited from existing text sources and older synthetic rows.
They are not proof that every warned sequence is unusable, but they show why new
coverage generation must not blindly emit random component permutations.
## Coverage Generation Fix
Generator:
```bash
uv run python tools/build_limbu_unicode_coverage_text.py \
--baseline-manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-v1-2026-06-24/train.jsonl \
--out-jsonl data/synthesis-text/limbu-sirijonga-unicode-coverage-v1-2026-06-24/coverage-text.jsonl \
--target-count 1000 \
--planned-render-variants 2 \
--targets-per-line 3 \
--seed 240624
```
The generator was corrected after the stats pass:
- subjoined letters are emitted before vowel signs: `base+subjoined+vowel`
- long closed syllables use the observed high-frequency order
`base+vowel+KEMPHRENG+final`
- no final-plus-sign context is added by the generator
- `U+1932` Limbu small letter anusvara is included in the finals list, not emitted
as a standalone orphan mark
Coverage text result:
```text
rows=1936
warnings={}
NFC-changed rows=0
NFD-changed rows=0
```
Rendered fabrication:
```bash
uv run python tools/build_limbu_targeted_fabrication_pack.py \
--text-jsonl data/synthesis-text/limbu-sirijonga-unicode-coverage-v1-2026-06-24/coverage-text.jsonl \
--width-profile-manifest outputs/limbu-gorkhapatra-pdf-backed-silver-2026-06-23-v3/splits/train.jsonl \
--out outputs/limbu-unicode-coverage-fabrication-v1-2026-06-24 \
--max-lines 0 \
--variants-per-line 2 \
--profile mixed \
--seed 240625 \
--force
```
Rendered result:
```text
source rows=1936
rendered rows=3872
render rejects=0
dark-pixel min/p50/max=33/1880/49435
min-gray min/p50/max=0/16/124
```
Contact sheet visually checked at
`/tmp/limbu-unicode-coverage-fabrication-contact.png`; crops are nonblank. A few are
intentionally faint from the mixed degradation profile, but only `7/3872` have fewer
than `100` dark pixels.
## Combined Pack After Coverage
Builder command:
```bash
uv run python tools/build_limbu_combined_recognizer_pack.py \
--out outputs/limbu-combined-synth-real-gorkhapatra-targeted-unicode-v1-2026-06-24 \
--targeted-fabrication-train-manifest outputs/limbu-targeted-fabrication-v1-2026-06-24/manifests/train.jsonl \
--unicode-coverage-fabrication-train-manifest outputs/limbu-unicode-coverage-fabrication-v1-2026-06-24/manifests/train.jsonl \
--seed 240626
```
Result:
```text
train rows=62339
eval rows=255
unicode-coverage-fabrication-v1 rows=3872
exact eval-text overlap drops=49
train audit passed=True
eval audit passed=True
```
Post-coverage Unicode stats:
```bash
uv run python tools/analyze_limbu_unicode_corpus_stats.py \
--manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-unicode-v1-2026-06-24/train.jsonl \
--manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-unicode-v1-2026-06-24/eval.jsonl \
--out outputs/limbu-unicode-corpus-stats-after-coverage-v1-2026-06-24 \
--top-n 80
```
Result:
```text
rows=62594
Limbu chars=2307121
covered codepoints=68/68
missing codepoints=0
under 1000 occurrences=none
NFC-changed rows=0
NFD-changed rows=0
```
The old corpus warnings remain because the old rows remain. The new Unicode coverage
text itself added no orphan marks or duplicate/order warnings.
## Policy
The Unicode coverage pack is training-only synthetic data. It is not evaluation
evidence and not claim evidence. Keep it as a separate training component named
`unicode-coverage-fabrication-v1` so the next recognizer run can ablate it against
held-out real/silver Gorkhapatra eval and separate CDC/phone-page checks.
|