File size: 7,871 Bytes
f9bc0c1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
# Limbu Unicode And Diacritic Stats

Updated: 2026-06-24.

This note records the Unicode facts, shaping test, corpus statistics, and corrective
coverage data for Sirijonga Limbu OCR training. Use it before generating new Limbu
synthetic rows.

## Primary Unicode Sources

- Unicode Character Database latest ReadMe:
  <https://www.unicode.org/Public/UCD/latest/ucd/ReadMe.txt>. Checked on
  2026-06-24; it resolves to Unicode 17.0.0 final data dated 2025-08-15.
- Unicode Character Database latest `UnicodeData.txt`:
  <https://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt>.
- Unicode Character Database latest `Blocks.txt`:
  <https://www.unicode.org/Public/UCD/latest/ucd/Blocks.txt>.
- Unicode Standard core spec, South and Central Asia chapter, Limbu section:
  <https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-13/>.

The local Python runtime reports `unicodedata.unidata_version=16.0.0`; the checked
Unicode 17.0.0 UCD still has the same assigned Limbu block facts used here.

## Unicode Facts

The Limbu block is `U+1900..U+194F`. There are `68` assigned Limbu codepoints:

- `31` bases: `U+1900..U+191E`
- `9` vowel signs: `U+1920..U+1928`
- `3` subjoined letters: `U+1929..U+192B`
- `9` finals/small letters: `U+1930..U+1938`, including `U+1932` Limbu small
  letter anusvara
- `3` signs: `U+1939..U+193B`
- `1` symbol: `U+1940`
- `2` punctuation marks: `U+1944..U+1945`
- `10` digits: `U+1946..U+194F`

Most Limbu marks have canonical combining class `0`. Only the three signs have
non-zero combining classes:

- `U+193B` Limbu sign SA-I: `220`
- `U+1939` Limbu sign MUKPHRENG: `222`
- `U+193A` Limbu sign KEMPHRENG: `230`

Consequence: `NFC`/`NFD` will not fix most wrong Limbu mark orders. Vowel signs,
subjoined letters, and most final signs are class `0`, so generated text must put
marks in sane order before rendering.

## Shaping Test

Tool:

```bash
uv run python tools/analyze_limbu_unicode_shaping.py \
  --out outputs/limbu-unicode-shaping-analysis-v1-2026-06-24 \
  --random-order-samples 5000 \
  --random-canonical-samples 1000 \
  --seed 240624
```

Output:

- `outputs/limbu-unicode-shaping-analysis-v1-2026-06-24/limbu-shaping-cases.csv`
- `outputs/limbu-unicode-shaping-analysis-v1-2026-06-24/limbu-shaping-summary.json`
- `outputs/limbu-unicode-shaping-analysis-v1-2026-06-24/limbu-shaping-summary.md`

Result using upstream Namdhinggo Regular:

```text
cases=21891
clean=21891
visible=21891
dotted-circle cases=0
.notdef cases=0
```

Important interpretation: Namdhinggo is permissive. It renders even intentionally
odd component orders without `.notdef` or dotted-circle glyphs. Font renderability
is necessary, but it is not a grammar check.

## Corpus Stats Before Coverage

Tool:

```bash
uv run python tools/analyze_limbu_unicode_corpus_stats.py \
  --manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-v1-2026-06-24/train.jsonl \
  --manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-v1-2026-06-24/eval.jsonl \
  --out outputs/limbu-unicode-corpus-stats-v1-2026-06-24 \
  --top-n 80
```

Result:

```text
rows=58722
Limbu chars=2241565
covered codepoints=61/68
missing codepoints=7
NFC-changed rows=0
NFD-changed rows=0
```

Missing before coverage:

- `U+1909` Limbu letter JHA
- `U+190A` Limbu letter YAN
- `U+191A` Limbu letter SSA
- `U+191D` Limbu letter GYAN
- `U+191E` Limbu letter TRA
- `U+1932` Limbu small letter anusvara
- `U+1940` Limbu sign LOO

Under `1000` occurrences before coverage:

- missing seven above
- `U+1919` SHA: `106`
- `U+1924` vowel sign AI: `86`
- `U+1926` vowel sign AU: `76`
- `U+1928` vowel sign O: `63`
- `U+192B` subjoined WA: `748`
- `U+1944` Limbu exclamation mark: `22`
- `U+194E` digit eight: `971`
- `U+194F` digit nine: `919`

Dominant cluster signatures before coverage:

```text
base+vowel=410919
base+vowel+final=258854
base+vowel+sign=58492
base+final=44527
base=39648
base+vowel+sign+final=28238
base+sign=28184
base+vowel+sign+vowel+final=15420
base+sign+vowel+final=7377
base+subjoined+vowel=3372
```

Existing-corpus warning counts:

```text
has_duplicate_vowel=16757
orphan_mark=289
has_duplicate_final=82
has_duplicate_sign=11
has_mukphreng_after_final=16
```

These warnings are inherited from existing text sources and older synthetic rows.
They are not proof that every warned sequence is unusable, but they show why new
coverage generation must not blindly emit random component permutations.

## Coverage Generation Fix

Generator:

```bash
uv run python tools/build_limbu_unicode_coverage_text.py \
  --baseline-manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-v1-2026-06-24/train.jsonl \
  --out-jsonl data/synthesis-text/limbu-sirijonga-unicode-coverage-v1-2026-06-24/coverage-text.jsonl \
  --target-count 1000 \
  --planned-render-variants 2 \
  --targets-per-line 3 \
  --seed 240624
```

The generator was corrected after the stats pass:

- subjoined letters are emitted before vowel signs: `base+subjoined+vowel`
- long closed syllables use the observed high-frequency order
  `base+vowel+KEMPHRENG+final`
- no final-plus-sign context is added by the generator
- `U+1932` Limbu small letter anusvara is included in the finals list, not emitted
  as a standalone orphan mark

Coverage text result:

```text
rows=1936
warnings={}
NFC-changed rows=0
NFD-changed rows=0
```

Rendered fabrication:

```bash
uv run python tools/build_limbu_targeted_fabrication_pack.py \
  --text-jsonl data/synthesis-text/limbu-sirijonga-unicode-coverage-v1-2026-06-24/coverage-text.jsonl \
  --width-profile-manifest outputs/limbu-gorkhapatra-pdf-backed-silver-2026-06-23-v3/splits/train.jsonl \
  --out outputs/limbu-unicode-coverage-fabrication-v1-2026-06-24 \
  --max-lines 0 \
  --variants-per-line 2 \
  --profile mixed \
  --seed 240625 \
  --force
```

Rendered result:

```text
source rows=1936
rendered rows=3872
render rejects=0
dark-pixel min/p50/max=33/1880/49435
min-gray min/p50/max=0/16/124
```

Contact sheet visually checked at
`/tmp/limbu-unicode-coverage-fabrication-contact.png`; crops are nonblank. A few are
intentionally faint from the mixed degradation profile, but only `7/3872` have fewer
than `100` dark pixels.

## Combined Pack After Coverage

Builder command:

```bash
uv run python tools/build_limbu_combined_recognizer_pack.py \
  --out outputs/limbu-combined-synth-real-gorkhapatra-targeted-unicode-v1-2026-06-24 \
  --targeted-fabrication-train-manifest outputs/limbu-targeted-fabrication-v1-2026-06-24/manifests/train.jsonl \
  --unicode-coverage-fabrication-train-manifest outputs/limbu-unicode-coverage-fabrication-v1-2026-06-24/manifests/train.jsonl \
  --seed 240626
```

Result:

```text
train rows=62339
eval rows=255
unicode-coverage-fabrication-v1 rows=3872
exact eval-text overlap drops=49
train audit passed=True
eval audit passed=True
```

Post-coverage Unicode stats:

```bash
uv run python tools/analyze_limbu_unicode_corpus_stats.py \
  --manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-unicode-v1-2026-06-24/train.jsonl \
  --manifest outputs/limbu-combined-synth-real-gorkhapatra-targeted-unicode-v1-2026-06-24/eval.jsonl \
  --out outputs/limbu-unicode-corpus-stats-after-coverage-v1-2026-06-24 \
  --top-n 80
```

Result:

```text
rows=62594
Limbu chars=2307121
covered codepoints=68/68
missing codepoints=0
under 1000 occurrences=none
NFC-changed rows=0
NFD-changed rows=0
```

The old corpus warnings remain because the old rows remain. The new Unicode coverage
text itself added no orphan marks or duplicate/order warnings.

## Policy

The Unicode coverage pack is training-only synthetic data. It is not evaluation
evidence and not claim evidence. Keep it as a separate training component named
`unicode-coverage-fabrication-v1` so the next recognizer run can ablate it against
held-out real/silver Gorkhapatra eval and separate CDC/phone-page checks.