File size: 101,313 Bytes
f9bc0c1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
1492
1493
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
1505
1506
1507
1508
1509
1510
1511
1512
1513
1514
1515
1516
1517
1518
1519
1520
1521
1522
1523
1524
1525
1526
1527
1528
1529
1530
# Research State: Nepal Language Pack OCR

Updated: 2026-09-24

## Devanagari recognizer fine-tune on k2 (2026-09-24)

Goal (owner steer): make the Devanagari route of the live demo (`ampixa.thapa-ashish.com.np`,
k2 OCR worker) read citizenship-certificate-style and general documents properly. Resource
inventory: `docs/DEVANAGARI_RESOURCES_INVENTORY.md`.

- **Base**: official `devanagari_PP-OCRv5_mobile_rec` (568-entry dictionary incl. all Latin
  letters and digits); replaced our `deva-v2` on the demo the same morning.
- **Training path**: Paddle CPU training on k2 (Apple M2 Ultra) runs at ~2 img/s, so the same
  network is trained with the `transformers` PyTorch port on MPS (~70-90 img/s), exported to
  ONNX, and served through PaddleX `create_model(engine="onnxruntime")`. Code:
  `tools/deva_rec_torch/`. Preprocessing is bit-matched to PaddleX (98.8% line parity).
- **Data**: real Gorkhapatra print re-cut into PDF-aligned chunks
  (`tools/build_gorkhapatra_devanagari_chunks.py`, edition-disjoint: test 2082-03-24,
  2082-08-12, 2082-12-25; dev 2081-12-12, 2083-01-11); synthetic citizenship/office/real-text
  lines in 44 OFL font families (`tools/render_devanagari_form_synth.py`); NVIDIA
  OCR-Synthetic-Multilingual-v1 English (CC BY 4.0) plus Liberation/DejaVu/Lato English renders.
- **Converter bug found and fixed**: npttf2utf 0.3.7 mis-decodes the nayanepal alternate
  क्र ligature `j|m` (-> व्र, and प -> फ / stray `m`). 3,426 of 114,452 spans (3%) were affected
  (कार्यक्रम, प्रक्रिया, शुक्रबार, ...). Fixed in the chunk builder (`RAW_FIXES`); labels of all
  earlier Gorkhapatra-derived packs (incl. `data/gorkhapatra-devanagari-real`, the Newar and
  language-pack silver packs built from the same converter) carry the bug. **Upstream fix
  still needed in nepal-ttf2utf.**
- **Results, corrected labels, served path (PaddleX), codepoint CER / exact line**
  (`~/ocr-deva-ft/preds/final_comparison_v3.json` on k2):

  | Set | official | v2 | **v3 (deployed 13:07)** |
  |---|---|---|---|
  | E1v4 real chunks, held-out editions (n=3000) | 5.13% / 0.47 | 0.69% / 0.91 | 0.57% / 0.93 |
  | E7 real chunks containing क्र (n=530) | 5.52% / 0.33 | 4.28% / 0.18 | 0.86% / 0.89 |
  | E1b real full lines, independent labels (n=1500) | 7.69% / 0.13 | 2.62% / 0.53 | 2.57% / 0.54 |
  | E2 Newar test (n=1139) | 9.11% / 0.06 | 3.53% / 0.35 | 3.54% / 0.35 |
  | E3 synthetic citizenship-ne, unseen fonts (n=800) | 6.78% / 0.56 | 0.84% / 0.93 | 0.75% / 0.94 |
  | E3 synthetic office-ne, unseen fonts (n=300) | 7.77% / 0.54 | 0.62% / 0.94 | 0.92% / 0.92 |
  | E3 synthetic citizenship-en (n=300) | 1.94% / 0.83 | 0.53% / 0.96 | 0.64% / 0.95 |
  | E4 English, unseen Noto Sans (n=120) | 2.57% / 0.62 | 1.58% / 0.73 | 1.64% / 0.72 |
  | E4 mixed Nepali/English (n=120) | 6.78% / 0.36 | 2.88% / 0.68 | 2.67% / 0.70 |
  | E6 NVIDIA English, held-out pages (n=419) | 10.6% / 0.71 | 2.99% / 0.84 | 2.84% / 0.84 |
  | E5 handwriting probe (n=450) | 64.7% | 60.5% | 61.0% (unusable) |

  v2 learned the converter's क्र->व्र error (402 of 530 E7 predictions contained व्र); v3,
  a 6,198-step continuation on the corrected `chunks-v4`, removes it (2 of 530, same as the
  official model). Deployed as `~/ampixa-demo/models/ocr/space/models/devanagari-ft-v3`;
  v2 and the official model remain on disk for rollback. E3 is synthetic from the same
  generator as training (different seed, held-out fonts) and overstates real-certificate
  accuracy; no real citizenship certificate has been tested. Real labels are PDF-text silver,
  not human-reviewed. Page OCR at 144 dpi garbles small text; at 288 dpi it reads cleanly.
- **Next**: real citizenship samples (with consent) for a true eval; handwriting data (none
  licensed yet); upstream the `j|m` fix into nepal-ttf2utf and rebuild the other
  Gorkhapatra-derived packs; the demo's "Rows LTR" reading order interleaves newspaper columns.

### Page pipeline on real government letters (2026-09-24, smoke test)

Two public notices from ocr.gov.np (a scanned Lalitpur District Court letter tilted ~2°, and a
Department of Industry letter), rendered at 200 dpi; 43 printed lines transcribed by hand
(`tools/deva_rec_torch/page_probe/`: GT, probe scripts, results; PDFs on k2 `~/ocr-deva-ft/layout-probe/`; not a frozen pack, not claim-grade). Same detector config and v3 recognizer as
the k2 worker; micro CER via `tools/eval_metrics.py`:

| Crop of each detected line | cp CER | grapheme CER | exact |
|---|---|---|---|
| worker today: axis-aligned bbox of the quad + 12 px | 21.4% | 31.7% | 0.40 |
| rotated min-area quad (PaddleX `CropByPolys`), no pad | 3.6% | 5.5% | 0.49 |
| rotated quad + 6 px | **2.9%** | **4.2%** | **0.65** |
| rotated quad + 12 px | 8.1% | 13.3% | 0.42 |

- **The worker's crop, not the recognizer, is the dominant error on real letters.**
  `_box_from_polygon` flattens the detector quad to an upright box; on a 2° tilt a full-width
  line spans ~40 px vertically, so the crop swallows neighbouring lines (court-letter body
  lines read at confidence 0.24-0.57). Upstream PaddleOCR uses rotated crops; our port dropped
  them. Line-level evals (E1-E7) cannot see this. Not yet deployed: needs a page-level eval.
- **PP-DocLayoutV3 on the same pages**: QR codes found (image, 0.95/0.96); emblems found
  (header_image); **0 of 2 stamps labelled `seal`** (court seal -> header_image 0.65; the
  Industry stamp printed over the title not detected at all); signatures merged into the
  printed-name `text` block (0.52); no class for handwritten fill-ins; several printed lines
  (subject, dates, reference numbers) not boxed at the default threshold. Use layout to mask
  non-text regions, never to decide what gets OCR'd.
- Signature/stamp crops produce junk text at recognizer confidence 0.35-0.71; damaged but
  real lines (tick mark, strike-through) score 0.74-0.82, so confidence can flag but not drop.
- `train.py` now supports `--guard-dev` (print/synthetic/English dev sets may not regress more
  than `--guard-tol` over step 0); the plain dev mean let a 5-point handwriting gain hide a
  1-point print loss. `run_train_v4.sh` on k2 uses it.

## Paper-rewrite verification pass (2026-07-19): corrections to this document

While rewriting `paper/DRAFT.md` to cover the language pack, every number was
re-verified against on-disk artifacts. Four values recorded earlier in this
file do NOT match any surviving local artifact and must not be cited:

- **Newar "both-sides-normalized cp CER 0.0441"** — no local artifact; the
  omac rescore output was never mirrored. The surviving verified score is
  **0.04655** cp / 0.09569 grapheme / 0.2450 exact (artifact-bearing labels,
  `outputs/newar-ft-v1-verify-omac-2026-07-07/metrics.json`, n=1,139).
- **Ol Chiki "public headline cp CER 0.0961 / n=358"** — no local score
  artifact exists for the 358-row public-issue subset. Locally verified:
  v2 on the 772-line v3 pack scores **0.08264** cp / 0.2034 exact
  (`outputs/olchiki-v2-run-2026-07-07/score-real-print-v3pack/metrics.json`);
  the documented public-issue historical diagnostic remains 0.1018 (n=366)
  per `docs/DATA_LICENSING.md`. The v3 358-row candidate was never scored.
- **"299,611 per-language-attributed silver lines"** — refers to the lost
  pre-crash batches. The rebuilt batches on disk total **306,971**
  (122,882 + 184,089, `outputs/silver-lines-rebuild-b{1,2}-2026-07-05.log`).
- **Ol Chiki v3 FT payload "5,330 entries"** — the payload's own
  assembly manifest says **5,307** total images (242 real train + 5,000
  synth + 65 real dev).

Also noted: the 8-language Devanagari scorecard (grapheme CER 0.075–0.129,
2026-06-18) lost its backing artifacts in the crash; it is text-only history
now and the paper deliberately does not report it.

## Machine-death recovery (read this first, 2026-07-05)

The Mac died 2026-07-04; the repo now lives at
`/Volumes/TRANSCEND/recovery-backup-20260704-172213/github/llm/ocr-tech`
(symlinked from the old `/Users/cdjk/github/llm/ocr-tech` path). **`outputs/`
and `runs/` are lost**; code, `.git`, docs, `spaces/` model exports
(sha256-verified, champion included), and most of `data/` survived. Every
`outputs/…`/`runs/…` path below therefore refers to a lost artifact unless
`docs/RECOVERY_LEDGER.md` maps it to a rebuilt one — read that ledger for the
full survived/partial/lost inventory, recovery routes, and the rebuild policy
(rebuilds go to new dated dirs). Batch-1 re-harvest launched 2026-07-05.
Blocked on user credentials: AWS SSO (training checkpoints, newar-ft-v1
weights) and HF auth (Space cross-check).

## Overnight lane results (2026-07-16/17)

These lanes delivered mechanically filtered silver substrate, not claim-grade
training or evaluation data. The counts below establish file and archive
parity; they do not replace native label review, edition-isolated leakage
checks, or a real-domain score.

The Sikkim pack at `outputs/sikkim-silver-packs-2026-07-16/` contains **9,743
Sunuwar/Jenticha clean real-page silver lines** and **3,699 Kirat Rai clean
real-page silver lines**. In each case the `summary.json` clean-line count,
on-disk `labels.txt` line count, and non-AppleDouble `.tgz` image-member count
are identical. Sampled Sunuwar labels contain Jenticha characters in
U+11BC0..U+11BFF, and sampled Kirat Rai labels contain Kirat Rai characters in
U+16D40..U+16D7F, without non-ASCII characters outside the expected target
block. Each summary records the eval manifests it checked. None of those eval
editions occurs in the discovered source list, so the recorded
`excluded_editions: []` is consistent rather than evidence of an unperformed
check.

Lepcha takes a different route. The diagnosis at
`outputs/lepcha-zero-yield-diagnosis-2026-07-16/REPORT.md` shows that the new
edition bodies are raster content and that the live PDF text is ordinary
header/punctuation material, so there is no converter alias or byte-map fix
that can recover the body. The converter lane is therefore closed and Lepcha
moves to a synth-first recognizer route, retaining the **132-crop** real pack
recorded in `data/lepcha-real-eval-pack/README.json` as a real evaluation
anchor. That pack is still machine-labelled and human verification remains
pending, so it is diagnostic rather than claim-grade.

The Gorkhapatra full-depth backfill now covers every month from March 2024
through July 2025. The figures come from each month's
`gorkhapatra-language-pages.json`; `silver-kept` comes from the corresponding
silver summary and was independently matched to the `.tgz` member count.

| Month | Items | Aligned | Hits | Silver-kept |
| --- | ---: | ---: | ---: | ---: |
| 2024-03 | 1,776 | 62 | 120 | 27,668 |
| 2024-04 | 1,731 | 61 | 129 | 21,429 |
| 2024-05 | 1,666 | 62 | 128 | 24,612 |
| 2024-06 | 1,601 | 60 | 118 | 21,924 |
| 2024-07 | 1,538 | 61 | 120 | 28,301 |
| 2024-08 | 1,473 | 60 | 120 | 24,534 |
| 2024-09 | 1,408 | 60 | 122 | 32,804 |
| 2024-10 | 1,343 | 53 | 103 | 25,055 |
| 2024-11 | 1,278 | 52 | 106 | 30,788 |
| 2024-12 | 1,213 | 61 | 121 | 39,115 |
| 2025-01 | 1,148 | 62 | 124 | 40,526 |
| 2025-02 | 1,083 | 56 | 112 | 35,391 |
| 2025-03 | 1,018 | 59 | 117 | 35,482 |
| 2025-04 | 953 | 57 | 113 | 34,815 |
| 2025-05 | 888 | 59 | 118 | 36,878 |
| 2025-06 | 823 | 60 | 120 | 37,836 |
| 2025-07 | 798 | 32 | 62 | 18,428 |
| **Total** | **21,738** | **977** | **1,953** | **515,586** |

All monthly audit warning arrays are empty, all archive counts match their
silver summaries, and no source PDF remains under any monthly output. This is
a delivered backfill substrate, not a statement that its page-attributed
silver labels are language-verified.

Detector remediation remains active. The scale-normalized tile build is
staged/running on `omac`; the local evidence at
`outputs/det-remediation-2026-07-16/REPORT.md` ends at the completed code,
config, and handoff. The GPU gate remains closed until the returned build
summary and tile manifest pass the recorded re-audit checklist.

## Current Goal

The end goal is an OCR system covering the multiple languages of the Nepal pack,
built out of Paddle, **with Limbu and Newar (Nepal Bhasa) as the first two focus
languages** (user steer, 2026-07-04). The system should accept real book/page images
or PDFs for any language in the pack and output editable Unicode text with page,
line, crop, confidence, and provenance evidence preserved. Limbu stays the proven
template — it has the strongest recognizer and the deepest harvest. Newar is the
deliberate second: it has its own native script (Prachalit), real cultural demand,
and is currently the weakest trained recognizer, so it exercises every part of the
method that Limbu already proved. Every stage that was built Limbu-specific must
generalize: harvest, silver-label extraction, recognizer training, script routing,
page/book OCR, and eval. The product is the full pipeline: page
detection, script-aware routing to per-script recognizers, target-region and
furniture filtering, reading order, assembly, Unicode/JSON/Markdown export, and
review artifacts for hard lines. YOLO line detection stays a diagnostic/probe;
Paddle-first is the production story.

## The Language Pack

Two inventories exist and they are not the same thing:

- The registry `languages/nepal-v1.json` tracks 18 target languages (17 Nepal + English)
  with scripts, fonts, dictionaries, and corpus status.
- The Gorkhapatra Naya Nepal archive publishes far more: the 2026-07-04 harvest found
  **50 normalized languages (53 raw labels) in 400 articles over ~7 months** (see the
  harvest section). Labels include all registry languages plus Chamling Rai, Bahing,
  Jirel, Chepang, Baram, Tajpuriya, Kumal, Doteli, Bajhangi, Dandeldhureli, Hyolmo,
  Mugal, Marwadi, Urdu, Sanskrit, Awadhi, Kisan, Achhami, Bote, Hayu, Meche, Lhomi,
  Uraon, Bhote, Danuwar, Majhi, Magahi, Bajjika, Khash Jumli, and more.
  Label variants must be normalized before counting (थारु/थारू, याक्ख/याक्खा,
  ह्योलमो/ह्योल्मो, लिम्बु/लिम्बू, चाम्लिङ vs चाम्लिङ राई) —
  `tools/build_gorkhapatra_language_pack_inventory.py` owns that normalization and the
  label→registry mapping.

### Newari / Nepal Bhasa writing-system boundary (2026-07-15)

Newari support is explicitly four-profile, not a single generic "Newa" bucket:

| Profile | Source encoding | OCR transcription | Current capability |
| --- | --- | --- | --- |
| Newar / Prachalit | Unicode Newa, U+11400..U+1147F | Native Newa | Route, training candidate, diagnostic checkpoint, and synthetic runtime smoke registered; no claim-grade real-domain CER |
| Bhujimol / Bhujinmol | No separate Unicode script; configured as a Newa-codepoint style profile | Newa plus mandatory source-style metadata | Route, two acquired manuscript groups, and reviewer reference registered; zero qualified decisions, admitted labels, model, or CER |
| Ranjana | Unencoded in Unicode 17.0 | Newa transcription plus mandatory source image and Ranjana metadata | Image route, two acquired manuscript groups, and reference font registered; zero qualified decisions, admitted labels, model, or CER |
| Devanagari | Unicode Devanagari, U+0900..U+097F plus U+A8E0..U+A8FF | Native Devanagari | Route, diagnostic checkpoint/runtime smoke, and one permission-covered official Newa-edition acquisition candidate registered; source script and real-line quality remain unaudited |

Unicode's current [Newa chapter](https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-13/)
and [code chart](https://www.unicode.org/charts/PDF/U11400.pdf) define the
encoded Newa route. WG2 document
[N4555](https://www.unicode.org/wg2/docs/n4555.pdf) describes Bhujimol/Bhujinmol
as graphically distinct and discusses possible plain-text unification with Newa;
that does not justify visual-model reuse. Ranjana is still represented by a
[preliminary encoding proposal](https://www.unicode.org/L2/L2023/23028-ranjana.pdf),
so this project does not invent a Ranjana Unicode range.

The canonical runtime profiles live in `src/ocrtech/newari_scripts.py`. The
registry and book-OCR pipeline declare four separate recognizer references and
preserve the source encoding/output policy in run metadata. The frozen audit at
`outputs/newari-four-writing-system-support-audit-2026-07-15-r1/` passes all four
structural bindings while reporting `model_ready=0/4` and
`four_script_ocr_claim_ready=false`. "Supported" here means explicit input,
profile, routing, and output semantics; each profile still needs at least 200
license-clean, pixel-label-audited, leakage-controlled real lines and a frozen
real-domain evaluation before delivery or quality claims.

The r2 license-first discovery audit records three Bhujimol and three Ranjana
source candidates without admitting any pixels. The most promising Bhujimol
item is [Asa Archives item 1147](https://archive.org/details/1147_20250817):
its Internet Archive metadata has an uploader-applied CC0 URL, 63 scanned
pages, and 325 automatically detected hOCR line boxes. Those boxes are
acquisition-scale metadata, not labels. Uploader authority, institutional
accession, exact-file SHA-256, qualified Bhujimol review, pixel faithfulness,
and transcription mapping all remain open. For Ranjana, the Smithsonian's
[Karandavyuha Sutra record](https://asia-archive.si.edu/object/S1991.161/)
specifically marks image `FS-6314_06` CC0 and its IIIF metadata identifies
Ranjana, but the [manifest](https://ids.si.edu/ids/manifest/FS-6314_06) contains
only one canvas; the 77 physical folios are not 77 downloadable pages. British
Library [EAP790/21/121](https://searcharchives.bl.uk/catalog/040-003461535)
identifies 12 Bhujimol TIFFs but restricts access to research use, and
[EAP1296/2/217](https://eap.bl.uk/archive-file/EAP1296-2-217) exposes 13
Ranjana images under noncommercial terms. The [Nepalbhasa Manuscripts in
UK](https://www.nepalbhasamanuscript.org/) archive states that copies and
transcriptions are openly accessible but supplies no explicit reuse grant.
The frozen audit is `outputs/newari-source-candidate-audit-2026-07-15-r2/`;
its pass means the catalogue is fail-closed, not that any source, model, or OCR
claim is ready.

The r3 provenance crosswalk matches item 1147 to the uploader's Asa Kuthi
catalog spreadsheet at worksheet `aiStuidoOutput`, row 1170: DPN 1147,
*Bhāsvati*, NRN `ज्योति - २७१`, RN 2919, Sanskrit/Nepalbhasa, N.S. 528, and
33.3 × 4.1 cm. The spreadsheet SHA-256 is
`8484dbe95358b83d7d78d5d1752fafac5feeecfc007ccf5d1fe8b6ea3ce9bffd`.
This disambiguates a separate row whose RN, rather than DPN, is 1147. It does
not attest Bhujimol because the matched catalog script field is blank, and it
does not prove that the Internet Archive uploader was authorized to apply
CC0. The exact authorization and reuse questions are staged in
`docs/NEWARI_ASA_1147_RIGHTS_REQUEST.md` but have not been sent. The r3 frozen
audit therefore remains `PASS_CATALOG_INVENTORIED_ZERO_DATA_ADMITTED` with no
review manifest ready.

The r4 alternative-source sweep found a materially stronger Bhujimol route in
Wellcome Collection. The official [manuscript
record](https://wellcomecollection.org/works/jnsn9gjd) identifies `MS Indic
epsilon 1`, and Wellcome's [editorial account of the
object](https://wellcomecollection.org/stories/stories-of-asian-palm-leaf-manuscripts)
explicitly says it uses Bhujimol. Its [complete IIIF
manifest](https://iiif.wellcomecollection.org/presentation/v2/b14901560) has
217 unique canvases/image services, SHA-256
`596296750114948b39da2252b5a7b7c0738287dadcf7aed32939bfc29980c6d4`, and
a Public Domain Mark statement that expressly permits commercial copying,
modification, and distribution. Wellcome describes 208 leaves with only six
illustrated. Five 2000-pixel-wide probes at canvases 1, 50, 100, 150, and 210
decoded successfully; canvas 1 was a cover/label view, while the other four
showed dense multi-block palm-leaf text. This is not a script-expert or label
audit. Because the current host is an 18GB Mac, the repository freezes only a
metadata acquisition manifest and an omac handoff; it does not bulk-download
the 217 canvases. The whole manuscript must remain on one side of any split,
so a separate manuscript is still required for leakage-proof training.

The r5 Ranjana follow-up resolved Wellcome `MS Indic delta 33` without
inflating selected images into a complete manuscript. The official [physical
record](https://wellcomecollection.org/works/bp979bf3) is closed-store only and
has no digital item; the predictable [IIIF presentation
URL](https://iiif.wellcomecollection.org/presentation/v2/b14901468) returns
404. Four exact digital-image records identify Ranjana and carry CC BY 4.0,
but a prediction-free 1200-pixel probe shows one two-leaf overview, one
partial-leaf environmental photograph, and two close-up miniature crops whose
content overlaps the overview. The records therefore provide zero independent
complete pages, not four. Their titles say AD 1653 while one description says
transcribed in 1635; both values are retained. The frozen catalogues at
`outputs/newari-wellcome-ranjana-source-catalog-2026-07-15-r5/` and
`outputs/newari-source-candidate-audit-2026-07-15-r5/` admit zero pixels,
labels, rows, models, or metrics. A different complete Ranjana manuscript is
still required at the r5 boundary.

The r6 BDRC sweep resolves that complete-source discovery gap without claiming
a dataset. BDRC physical instance `MW1NPL132`, *Ārya nāmasaṁgīti*, is marked
only `ScriptRanj` with print method Manuscript; linked scan `W1NPL132` is
`AccessOpen` and `StatusReleased`, and image group `I1NPL132` exposes 21 unique
IIIF canvases/services. The manifest SHA-256 is
`a3c49a3fd68f30f0b4610a2f0c3d6432fd189caae5c371a0d482ae97fb02d11d`.
BDRC's official access policy classifies scanned original manuscripts as
public domain and permits sharing/adaptation for any purpose. The manifest
itself has no license field; the work-record CC0 marker applies to metadata and
is explicitly not treated as a pixel license. Five bounded 1200-pixel probes
decoded: canvases 1 and 21 are cover/note material, while 5, 10, and 15 show
dense visually Ranjana-like running text. Those observations are not qualified
script decisions or a line-count audit. The frozen catalog at
`outputs/newari-bdrc-ranjana-source-catalog-2026-07-15-r6/` records 21 exact
remote resources, zero admitted pixels, zero audited lines, zero labels, and a
prediction-free `omac` handoff. All pages remain one manuscript group; a
distinct manuscript is still required for a manuscript-disjoint train/eval
split.

The r7 remote-acquisition round executed that BDRC handoff on
`omacs-MacBook-Pro` (68,719,476,736 memory bytes), without OCR, training, or
scoring. All 21 full-resolution JPEGs were fetched, SHA-256 recorded, and
decoded at their exact manifest dimensions (all width 1200, heights
1676–1769); the capture totals 6,680,141 bytes and has 21 unique byte hashes.
The source pixels remain on `omac`; the orchestration Mac mirrors only the
metadata, hash list, dimension list, and rights/manifest evidence. A dynamic
WordPress detail was caught rather than waived: BDRC's policy HTML randomizes
CSS class-number suffixes across hosts, so raw page hashes differ while a
narrow normalization of only those suffixes yields the same SHA-256
`99977ba3f0cb00da828a3833e6649a39c89c9f8e928a7127eeac1c35cd48d60e`
and the required rights sentences remain present. The local adversarial audit
at `outputs/newari-bdrc-ranjana-remote-acquisition-audit-2026-07-15-r7/`
verifies the returned metadata bindings and creates a 21-row blank review
worksheet. It explicitly cannot recompute the remote image hashes without the
pixels. Qualified page decisions remain 0/21, audited lines remain zero, and
no label, admission, model, or metric exists.

The r9/r10 remote-acquisition round executed the independent Wellcome
Bhujimol handoff on the same approved host, again without OCR, training, or
scoring. All 217 requested review JPEGs were fetched, SHA-256 recorded, and
decoded; the capture totals 58,471,370 bytes and has 217 unique byte hashes.
Wellcome's `full/2000,/...` IIIF derivative decoded at width 2000 for 214
canvases and width 2001 for canvases 214, 216, and 217. The runner failed
closed on the first 2001-pixel result, then the replay narrowed the accepted
contract to exactly those observed widths and froze every exact dimension.
The source pixels remain on `omac`; the local mirror is metadata-only. The
adversarial audit at
`outputs/newari-wellcome-bhujimol-remote-acquisition-audit-2026-07-15-r10/`
verifies the returned bindings and creates a 217-row blank qualified-review
worksheet. Qualified page decisions remain 0/217, audited lines and labels
remain zero, and the entire manuscript is still one indivisible split group.
A second Bhujimol manuscript is required for manuscript-disjoint evidence.

The r12/r13 second-source round catalogs a distinct manuscript group for both
Bhujimol and Ranjana without admitting data. The exact Wikimedia Commons JPEG
of Cambridge `MS Add.2137` is photographer-authored and CC BY-SA 4.0. It
contains four folio photographs; the first three selected regions are visually
crosswalked to Cambridge canvases 48–50 (`28r`, `28v`, `29r`). Cambridge's
official manifest independently calls all three hands
hooked Nepālākṣarā (Bhujimol), but the complete 228-canvas institutional scan
is CC BY-NC 4.0 and remains pixel-excluded. The single licensed composite is a
separate manuscript group from Wellcome `MS Indic epsilon 1`, but cannot by
itself establish the 200-line gate and still needs exact-original SHA-256 and
qualified line review. BDRC `MW1NPL23` is separate from `MW1NPL132`; its
open/released scan has 25 unique IIIF canvases under BDRC's scanned-manuscript
public-domain policy. Its physical record assigns both `ScriptNewa` and
`ScriptRanj` without page-level attribution, so every text-bearing canvas must
be classified Ranjana/Newa/mixed/other by a qualified reviewer. The frozen r12
catalog stores no source pixels and reports zero qualified decisions, audited
lines, labels, OCR, training, or scoring. The r13 inventory audit passes with
11 candidates (5 Bhujimol, 6 Ranjana), 3 acquisition-ready manifests, 2
completed remote acquisitions, and zero admitted sources. Its tests reject
license, institutional-script, manifest-license, duplicate-service,
probe-hash, and line/admission drift. Evidence:
`outputs/newari-second-manuscript-source-catalog-2026-07-15-r12/` and
`outputs/newari-source-candidate-audit-2026-07-15-r13/`.

The r14/r15 acquisition round executed the second-source handoff on
`omacs-MacBook-Pro` (68,719,476,736 memory bytes) without OCR, training, or
scoring. The exact Commons original and all 25 BDRC canvases decoded at their
frozen dimensions; all 26 byte hashes are unique and total 12,738,550 bytes
(3,435,984 Bhujimol; 9,302,566 Ranjana). The Commons SHA-256 is
`938013af2c4c16ac84164a41f903cf34d131663b7a069e58df779bacfcd4258d` and
its upstream SHA-1 also matches. Separate profile checksum manifests bind one
Bhujimol image and 25 Ranjana images, while the orchestration host mirrors
metadata only. The local audit creates 28 blank qualified-review units: three
Bhujimol panels (`28r`, `28v`, `29r`) and 25 BDRC canvases. All decisions,
audited lines, labels, admissions, and claims remain zero. Evidence:
`outputs/newari-second-source-prediction-free-review-2026-07-15-r14/` and
`outputs/newari-second-source-remote-acquisition-audit-2026-07-15-r15/`.
The refreshed r16 central inventory now reports 11 candidates, 7 explicit
commercial-permission statements, 3 acquisition-ready manifests, 4 remotely
fetched-and-hash-recorded sources, and still zero admitted sources or support
claims (`outputs/newari-source-candidate-audit-2026-07-15-r16/`).

The r17 review-readiness round joins the four acquired manuscript groups
without copying pixels or inventing decisions. It binds 264 unique remote
source images to 266 blank review units: Bhujimol has 218 images/220 units
across Wellcome `b14901560` and Cambridge `MS Add.2137`; Ranjana has 46
images/units across BDRC `MW1NPL132` and mixed-script `MW1NPL23`. Both
profiles meet only the two-manuscript-group precondition. Qualified decisions,
estimated lines, audited lines, labels, admissions, OCR, training, scoring,
and claims remain zero, so the >=200-line gate is explicitly false for both.
Evidence:
`outputs/newari-acquired-source-review-readiness-audit-2026-07-15-r17/` and
`docs/handoffs/HANDOFF-newari-acquired-source-review.md`.

The r43-r45 correction separates manuscript identity from exact transcription.
The Wellcome story identifies `b14901560` / MS Indic epsilon 1 as the
*Aṣṭasāhasrikā Prajñāpāramitā Sūtra*, but its 217 IIIF canvas labels are all
`-`; no folio or line transcription accompanies the images. A GRETIL e-text
exists for the Vaidya 1960 edition, but its page says it is for reference only,
defers terms to the source file, and supplies no crosswalk to this manuscript.
It is therefore neither rights-cleared prefill nor ground truth. BDRC's frozen
physical record labels `MW1NPL99` “Unidentified” and supplies no e-text.
Cambridge's official TEI says the *Nāradasmṛti* begins at verse 5.8cd on `28r`,
but the manuscript interleaves that Sanskrit root with a Newari
commentary-cum-translation; the TEI explicitly states that the Newari legal
text and translation are unpublished and gives only short catalog excerpts.
Accordingly, the r45 fail-closed audit reports three routes checked, zero exact
line-aligned transcriptions, zero admissible prefills, and zero labels.

The same pass found that the frozen r15 worksheet named `panel-28r`,
`panel-28v`, and `panel-29r` but pointed all three rows at the same uncropped
2,332 x 1,800 four-folio Commons composite. The row identities and licensed
source hash remain valid, so no frozen review data was rewritten. The r43
profile dispatch and r44 combined UI instead render fixed source-bounded
regions `[0,0,2332,400]`, `[0,430,2332,420]`, and
`[0,890,2332,410]`, crosschecked against institutional canvases 48–50. They
fail closed to the full original if dimensions or bounds drift, preserve the
full-image link and SHA-256, contain no OCR text or labels, and keep the same
339-row assignment scope. The complete selected regions were visually checked
against the CC BY-SA Commons original. Tests cover rendering metadata,
source-dimension refusal, immutable review scopes, output checksums, and the
companion-text admission gate. Artifacts:
`outputs/newari-qualified-page-review-dispatch-2026-07-15-r43/`,
`outputs/newari-qualified-page-review-ui-2026-07-15-r44/`, and
`outputs/newari-bhujimol-companion-text-audit-2026-07-15-r45/`.

The r41 Devanagari-Newari source round found one exact official State Portal
record titled `SIKKIM HERALD-NEWA EDITION`, dated 13 April 2022, under the
permission-covered `sikkim.gov.in/uploads/SikkimHerald/` prefix. The existing
Sikkim source policy remains exact at SHA-256
`2247033ac1ce7de8d99cad22feb702f9fe48a11341a2fb2d20c1d6d48b06717a`.
An eight-byte HTTP Range probe confirms a PDF response with a total size of
3,975,535 bytes; no PDF body or page image was downloaded locally. The newer
IPR portal currently returns zero Newar records and was used only as negative
discovery evidence. The administration URL currently exposes only the official
Government of Sikkim site shell, so no substantive publication claim is derived
from it. The target writing-system route is Devanagari, but the edition's actual
source script, fonts, and third-party content remain explicitly unconfirmed
until all pages are rendered and reviewed on an approved remote host. The
catalog therefore admits zero pixels, decisions, labels, real lines, models,
or metrics. Evidence:
`outputs/newari-devanagari-sikkim-source-catalog-2026-07-15-r41/`.

Two tempting Commons catalog leads were rejected for production use in the same
round. `AN-PM-12.pdf` and `AN-PM-09-Dvadasa Tirtha Yagu Bakham.pdf` are described
on Commons as old Newari material in Devanagari with Public Domain Mark labels,
but the British Library EAP record for AN-PM-09 states that access is for
research purposes only. A downstream Commons rights label is not treated as
authority to erase that source restriction. Recent uploader-applied Internet
Archive CC0 metadata was likewise kept out when uploader authority could not be
established.

Per-language recognizer state (grapheme CER, honest micro metric, from the re-grounded
2026-06-18 scorecard in `docs/RECOGNIZER_GOAL_PROGRESS.md`):

- **Limbu/Sirijonga 0.041 real-print** (champion:
  `runs/limbu-real-heavy-gork-cdc-replay-v1-lr1e6-8ep-normedit-g6-l4-ppocrv5-2026-06-23`,
  cp CER 0.011 on the frozen 255-line Gorkhapatra silver eval).
- **8 Devanagari languages 0.075–0.129 real-print** (Majhi, Magar, Sunuwar-Deva, Magahi,
  Bhojpuri, Thami, Wambule, Darai) from one shared Devanagari recognizer.
- **Sunuwar (Jenticha) 0.121 real-crop** after the same-font retrain (0.342 synth-only).
- **Tirhuta 0.343 and Newa 0.613 synthetic-only** — weak, transliteration-fed, honest
  fails; Newa needs real printed-register Nepal Lipi, not more pixels.
- **Kirat Rai (70 real crops) and Lepcha (132 real crops)** blocked on legacy-font
  converter coverage outside their eval editions. **Gurung Khema now has official
  real-print source material:** two 2022 State Portal Herald editions yield 749
  mechanically convertible line candidates, but zero rows are admitted pending
  independent provenance review, native pixel-label review, and split/leakage gates.

The near-term pack claim should therefore be script-family shaped: one Sirijonga
recognizer, one shared Devanagari recognizer covering the many Devanagari-written
languages, plus per-script experts (Jenticha, Kirat Rai, Lepcha, Prachalit/Newa,
Tirhuta, Gurung Khema, Ol Chiki, Tibetan-script) admitted one by one as real eval
crops exist.

**Focus order (2026-07-04 steer): Limbu first, Newar second.** The 2026-07-15
Newari steer requires Newar/Prachalit, Bhujimol, Ranjana, and Devanagari as
separate writing-system profiles. Newar state: the
Prachalit recognizer is the weakest in the pack (grapheme CER 0.613 on its own
synthetic eval; transliteration-fed training failed; real printed-register Nepal
Lipi is the known lever, and only 2 Unicode Newa fonts exist). The Devanagari-Newar
route moved fast on 2026-07-04: 6,800 real silver lines harvested from 20
Gorkhapatra pages; `newar` added to the registry (Devanagari primary + Prachalit
secondary; audit passes); a frozen by-date pack built at
`outputs/newar-devanagari-silver-pack-v1/` (train 4,096 / dev 1,528 / test 1,139
lines; test dates 2082-07-25 and 2083-01-05; zero date/text overlap independently
verified; pack sha256
`1fa7b23b6c18701857aea686fd55f3075e104d224730896f6ee35ba25e7ccb85`); and the
**first real-print Devanagari-Newar baseline** was scored locally (CPU export
route, `tools/score_recognizer_pack.py`, per `docs/LOCAL_RECOGNIZER_SCORING.md`):
the shared deva-v2 recognizer, never trained on Newar, reads the frozen test at
**codepoint CER 0.0622, grapheme CER 0.1312, exact-line 0.158** (n=1,139 silver
lines — machine-converted labels, diagnostic, not claim-grade). The same-day
warm-start fine-tune (**newar-ft-v1**: deva-v2 base, LR 2e-5 cosine, 60 epochs,
batch 64, selection on the all-batch2 dev split, best_epoch 46; G6 L4 instance
`ashish-experiments`, ~68 min; artifacts + train log at
`outputs/aws-newar-deva-ft-v1/`) scores the same frozen test at **codepoint CER
0.0465, grapheme CER 0.0955, exact-line 0.245** — a 25/27% relative CER drop and
+55% exact-line over the deva-v2 baseline, scored through the identical local
CPU path. **Newar's grapheme CER now sits inside deva-v2's own-language range
(0.075–0.129): the second focus language reached Devanagari-pack parity from
zero in one day.** Same claim boundary: silver machine-converted labels,
diagnostic, not claim-grade until native review. Next Newar lever: native
Prachalit (real printed-register data), and page-level Newar eval for the
region-filtering track.

## Source Rule

For Gorkhapatra/Naya Nepal language pages, the Naya Nepal page is discovery metadata
only. Use `https://gorkhapatraonline.com/publications/naya-nepal` to find the language,
date, and article, but do not use the blog/card image for OCR, training, or evaluation
when the full paper exists. The authoritative source is the matching Gorkhapatra epaper
PDF from the date-range endpoint. The correct pair is the rendered PDF page image plus
the text layer extracted from that same PDF page, with PDF URL/hash, page number,
rendered image hash, extraction command/version, font evidence, and converter/map hash
recorded. For Limbu/Sirijonga the text layer is Namdhinggo-style legacy converted with
`src/ocrtech/limbu_legacy.py` + `data/mappings/limbu-legacy/Limbu.map`. The same
pattern — classify the text layer, then build an explicit converter only when the font
requires it — replicates to every other language. Manual transcription is the fallback
for no usable text layer, not the default.

## Getting More Gorkhapatra Pages (the specified harvest recipe)

All numbers below were verified live on 2026-07-04.

**Archive size.** The Naya Nepal publication archive paginates at
`https://gorkhapatraonline.com/publications/naya-nepal?page=N` with 5 articles per
page and runs to page ~352 before drying up: **~1,750 language-page articles total**
(about 2 per day going back years). The 60-page crawl already yielded 300 items across
53 labels spanning BS 2082-10-18 → 2083-03-19; deeper pages reach correspondingly
older dates.

**Epaper endpoint behavior (the critical constraint).** The date-range endpoint
`https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=A&pdfdaterange2=B&slug=gorkhapatra`
**caps results at 30 PDFs per query and ignores `page=N` pagination**. A wide window
silently drops most dates — the 2026-07-04 six-window discovery pass aligned only
111/300 items for exactly this reason, with unmatched dates interleaved among matched
ones. **Seven-day windows return complete daily coverage (7/7 verified).** Always
harvest with weekly windows.

**The recipe** (one command; add one `--epaper-url` per week to cover the publication
date span being harvested):

```bash
env PYTHONPATH=src uv run --python 3.11 --with pymupdf \
  ocrtech audit-gorkhapatra-language-pages \
    --out outputs/gorkhapatra-language-pages-full-<date> \
    --publication-url https://gorkhapatraonline.com/publications/naya-nepal \
    --publication-pages 80 \
    --epaper-url 'https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=2026-06-28&pdfdaterange2=2026-07-04&slug=gorkhapatra' \
    --epaper-url '...one per week, going back...' \
    --download-pdfs \
    --render-pages \
    --timeout-seconds 180
```

The command aligns every publication card (all languages at once — alignment is
language-agnostic) to its same-date PDF, downloads each PDF once (cached by URL),
searches pages for the language-page markers (`भाषा पृष्ठ` and its legacy-encoding
form `efiff k[i7`), and renders hit pages at 2×. The 2026-07-04 run with 23 weekly
windows (2026-01-25 → 2026-07-04, publication-pages 80) completed with
**400 publication items, 195 aligned dates, 394 rendered language-page hits, zero
warnings** at `outputs/gorkhapatra-language-pages-full-2026-07-04/`. Batch 2
(2025-08-03 → 2026-01-24, 25 weekly windows, publication-pages 160) added 398 more
items: **798 items, 294 aligned dates, 589 further page hits**, at
`outputs/gorkhapatra-language-pages-full-batch2-2026-07-04/`. The merged normalized
inventory (`tools/build_gorkhapatra_language_pack_inventory.py`) shows **56
languages, 983 rendered language-page hits, most languages with ~20–27 pages
spanning 14 months**, at `outputs/language-pack-inventory-merged-2026-07-04/`. For
comparison, the entire previous state of the project had 8 usable pages for one
language. Batches cost ~2–2.5 GB each; the remaining archive depth (back to at least
March 2024 — verified the epaper endpoint serves complete weeks there) is another
~2× and stays month-batchable with the same recipe.

**Scaling to the full archive.** Full depth is ~700+ unique dates ≈ 35 GB of PDFs.
Do not do it in one run: batch by month (4–5 weekly windows + the matching
publication-page depth), run `extract-gorkhapatra-language-page-pdf-text` per batch,
then keep rendered pages, extracted text, and hashes; PDFs can be pruned after
extraction because URL+SHA-256 make them re-fetchable. The remote 2 TB volume
(`cdjk@100.110.148.43:/Volumes/TRANSCEND`) is the right home for a full-depth mirror.

**Per-language downstream.** After harvest, run per language:

```bash
env PYTHONPATH=src uv run --python 3.11 --with pymupdf \
  ocrtech extract-gorkhapatra-language-page-pdf-text \
    <pack-manifest.jsonl> \
    --candidate-language <label> \
    --target-font <font> \
    --out outputs/<language>-pdf-text-<date>
```

then classify each page's text layer (native Unicode / legacy font / Devanagari
transcription / Latin / none) and only then decide converter vs manual transcription.

**Font/text-layer facts already established** (30-page, 15-language extraction at
`outputs/gorkhapatra-multilingual-language-pages-current-2026-06-24-pdf-text/`):

- The body font of non-Limbu language pages is **`nayanepal` — a legacy 8-bit
  Preeti-family encoding, not Unicode** (8,845 of 11,791 spans; the text layer reads
  `k[i7 ;+of]hs` where the page shows पृष्ठ संयोजक). **The converter already
  exists**: `nepal_ttf2utf.convert_devanagari(font="nayanepal")` in the sibling repo
  `/Users/cdjk/github/llm/nepal-ttf2utf` (Preeti base map + nayanepal extension
  glyphs + NFC, ~97–99% clean on real Gorkhapatra pages), already used by
  `tools/build_gorkhapatra_devanagari_real.py` for the 79k-row real Devanagari
  corpus. The remaining leverage is wiring it **per language**: run it over each
  harvested language page and emit per-language silver line packs with the same
  faithfulness gate (clean-Unicode allow-set, `conversion_gap` accounting, leakage
  checks) that the Devanagari pack builder already implements. The same sibling repo
  also hosts the limbu, kiratrai, lepcha, and sunuwar legacy converters.
- Tibetan-script fonts (`TibetanMachine`, `MonlamUniOuChan5`, `Qomolangma-*`) appear on
  Mugal and related pages, but **every such span has empty extracted text (verified
  2026-07-04 across all 2026-06-24 samples)** — Tibetan-script pages are visual-only,
  like the known Limbu `2082-11-12` date. They need OCR bootstrap or manual
  transcription; no converter is possible.
- `Gorkhapatra-*`, `Nagarik-*`, `TimesNewRoman*`, `MyriadPro-*` are masthead/furniture
  fonts — useful as non-target filters for target-region selection.
- `NamdhinggoSILL` appears only on Limbu dates (the existing Limbu path).

## What We Built (Limbu template, still true)

The Gorkhapatra alignment path: publication-page discovery, epaper-date matching, PDF
download/rendering, page-marker search, PDF-native span/font extraction, review
sheets, reference templates, and dashboard visualization. The expanded Limbu harvest
has 9 discovered dates, 8 usable PDF-text-layer pages, 2,441 line rows, 2,430 usable
silver labels (`outputs/limbu-gorkhapatra-pdf-backed-silver-2026-07-03-v1/`).

The Limbu recognizer training/eval path: 48,507-row combined pack; real-heavy
fine-tune champion at codepoint CER 0.011336 / grapheme 0.024736 / exact-line 0.729 on
the frozen 255-line silver eval. Synthetic hardening lessons: targeted+Unicode packs
regressed; v3 dictionary warm-start catastrophically failed (~0.98 CER); keep the
real-heavy v2 champion; future gains come from real/book-domain harvesting or a
deliberate rehead, not blind synth rows.

The page/book OCR loop: local + HF Space (`spaces/limbu-ocr/`, deployed private at
`https://huggingface.co/spaces/ampixa/limbu-ocr`) with line OCR, Paddle Page OCR,
Paddle Book OCR, image-line fallback, runtime checks, annotated pages, and crops.

Page-level diagnostics on the 8-page silver pack: Rows LTR + y/conf70 filter scores
page cp CER 0.414330, line F1 0.963009, matched-line cp CER 0.026039, order accuracy
0.993176. Recognition on matched lines is strong; **full-page CER is dominated by page
reconstruction, target-region selection, and furniture filtering** — that is the
pipeline problem to solve, and it is language-independent.

Supervised line-detector infrastructure from CDC PDF geometry (287 pages, 26,160
annotations, mAP50 ≈ 0.921 on PDF-rendered val) — evidence the detector path works,
not a solved product.

A claim-grade eval harness (versioned packs, hashes, provenance, normalization
profiles, cp/grapheme CER, CIs, paired comparisons, leakage manifests) — parked as a
guardrail; do not let it eat work blocks that should go to the pipeline.

## Current Working State

The strongest component is the Sirijonga line recognizer. The weakest component is
end-to-end page/book reconstruction: target-region selection, furniture filtering,
line preservation, and ordering. Of the three Limbu-locked code spots, two were
generalized on 2026-07-04 (Codex worker, tests green): `_script_ratio` in
`src/ocrtech/engines.py` is now driven by the shared `SCRIPT_CODEPOINT_RANGES` table
covering all pack scripts (devanagari, limbu/sirijonga, latin, sunuwar/jenticha,
lepcha, kirat_rai, newa/prachalit, tirhuta, gurung_khema, ol_chiki, tibetan, with
aliases), and the image-line filter in `src/ocrtech/image_lines.py` accepts any
script key from that same table. The third spot fell the same day: the
`src/ocrtech/limbu_runtime.py` preflight now takes `required_script` (default
`limbu`, validated against the shared table; CLI `--required-script`), so a router
card for any pack script can be preflighted. The engine classes themselves (`CandidateEngine`, `PaddleOcrEngine`,
`ScriptSelectCompositeEngine`, `build_image_line_document`, `structure.py` reading
order) are already parameterized — the rest of the generalization is registry-driven
wiring, not a rewrite.

## The Paddle Pipeline (whole pipeline, not just recognizers)

Target architecture, all stages Paddle-first and language-pack-parameterized by the
per-language pipeline config **`configs/nepal-language-pack-pipeline-v1.json`**
(50 languages: label, registry id, scripts, recognizer artifact, dictionary,
converter, text-layer class, harvest counts — generated 2026-07-04 from the
inventory):

1. **Capture/prep** — existing `prepare-limbu-capture` path, already
   language-agnostic in behavior; rename/rescope.
2. **Detection** — Paddle text detection as production default; YOLO line detector and
   morphology image-line boxes stay as diagnostics/rescue passes.
3. **Script identification & routing** — generalize `_script_ratio` to a
   Unicode-block-driven scorer over the pack's script list (Devanagari, Limbu,
   Jenticha/Sunuwar, Lepcha, Kirat Rai, Newa, Tirhuta, Gurung Khema, Ol Chiki,
   Tibetan, Latin), then route each line to the script's recognizer from the registry.
4. **Recognition** — per-script PP-OCRv5 recognizers (existing runs) with per-script
   dictionaries and input-width profiles (Sirijonga width-480 lesson).
5. **Target-region & furniture filtering** — non-oracle region selection using layout
   plus font/language signals; furniture fonts and masthead geometry as negative
   evidence. This is the main open research item; score against the Rows LTR +
   y/conf70 baseline.
6. **Reading order & assembly** — `auto_layout` selection (wide-column RTL vs rows)
   per page, conservative assembly stays diagnostic.
7. **Export** — Unicode text, JSON with provenance, Markdown; review queues +
   dashboards per page (existing Limbu artifact shapes, renamed pack-wide).
8. **Eval guardrail** — frozen real-page packs per language: page CER/WER, line
   P/R/F1, reading order, blank/extra/missed lines, review burden.

PP-StructureV3 and PaddleOCR-VL remain the documented fallback/teacher direction for
hard layouts, per `docs/NEPAL_MULTILINGUAL_OCR_GOAL.md`.

### Production book-ingestion plan (2026-07-04)

The production target: a book (scanned PDF, image folder, phone captures) in;
editable Unicode, provenance JSON, and Markdown out; a review queue covering only
lines that need a human; frozen regression packs gating every model/config change.
The production KPI is **review burden** (can an editor fix a book in hours), not
CER alone. Phases in leverage order:

1. **PP-StructureV3 spike — DONE 2026-07-05, verdict: stays fallback.** The
   2026-07-04 CPU run died producing nothing (~6 h on page 1/8); the re-run on
   the L4 GPU did all 16 rebuilt pages in ~4 s/page. Scored on the rebuilt
   16-page / 4,895-line Limbu silver pack, same champion recognizer and same
   `run_page_ocr_eval.py` flags in every arm (diagnostic, non-claim):

   | arm | page cp CER | det F1 | matched cp CER | order acc |
   |---|---|---|---|---|
   | PP-StructureV3 raw (its own reading order) | 0.797 | 0.904 | 0.0345 | 0.778 |
   | PP-StructureV3 boxes + Rows-LTR + y/conf70 | 0.421 | 0.934 | 0.0331 | 0.993 |
   | Current stack: Paddle det + Rows-LTR | 0.437 | 0.942 | 0.0266 | 0.993 |
   | **Current stack + y/conf70 filter (baseline)** | **0.413** | **0.966** | **0.0266** | 0.993 |

   Stock PP-StructureV3 loses on every axis even after grafting our reading
   order and filter onto its boxes: worse detection, worse crops for our
   recognizer (matched CER 0.033 vs 0.027), far worse native reading order,
   and it needs a GPU. The Paddle det + Rows-LTR + y/conf filter chain stays
   the production backbone; PP-StructureV3 remains the documented fallback /
   teacher for table- and figure-heavy layouts (not exercised by these
   text-heavy newspaper pages). Artifacts:
   `outputs/ppstructure-spike-rebuild-2026-07-05/` (stage-1 GPU JSONs,
   stage-2 documents, all score dirs);
   baseline `outputs/limbu-baseline-rebuild-2026-07-05/`.
2. **Detector fine-tune** — cash in the supervised det goldmine: 983 language
   pages with PDF-text-layer line geometry across 50+ languages plus 26k CDC
   annotations; fine-tune PP-OCRv5 det to fix the center-column miss pattern.
3. **One config-driven book runner** — generalize `run_limbu_book_ocr.py` into a
   language-pack `run_book_ocr` (input → prep → layout → det → script-route →
   recognize → assemble → export + review queue), resumable per page.
4. **Acceptance gates** — frozen page packs (Limbu 8-page, Newar pages, one real
   book) with thresholds on page CER, line F1, reading order, review-burden %;
   the parked eval harness serves as the gate runner, not a research project.
5. **Packaging/serving** — versioned per-script model bundles, Space/API on the
   pack pipeline, CPU eval path, GPU batch for book jobs.

## External Native-Script Real-Data Harvest (2026-07-06)

Rebuilt the lost `newa-zenodo-6967421-v1` asset and swept Zenodo/HuggingFace/
GitHub/Internet Archive for other real (non-synthetic) native-script OCR
material. Everything landed at
`data/external-language-resources/native-script-real-2026-07-06/<script>/`
with a `provenance.json` per source (license, byte size, md5, verify method).
Full details in that tree; headline results:

- **Newa/Pracalit (Zenodo 6967421, CC-BY-4.0)**: re-downloaded, md5-verified
  (`76d464a6b656482b7c0594c0cb37fb18`), extracted. **Correction to the record**:
  it is **441 page images + 441 PAGE-XML**, not the 683 pages this doc
  previously cited — 441 matches the record's own `metadata.xml` and the
  Zenodo description's "441 training pages"; the "242 validation pages" the
  same description mentions are not in this file. Do not carry 683 forward.
  Mirrored bit-for-bit at heiDATA (`10.11588/DATA/WI9184`, same title/size) —
  useful as a second-source integrity check, not new content.
  `tools/newa_zenodo_build.py` expects `data/newa-zenodo/extracted/`; this
  pull is at the new path above, so point the tool there (or symlink) before
  re-running the line-crop pipeline.
- **Newa/Pracalit, other leads that did NOT pan out**: a newer O'Neill paper
  (Zenodo 17268040, "Revisiting HTR for Nepalese Manuscripts in Pracalit
  Script," 2025) cites no new public GT beyond 6967421. A published
  "First-Ever Prachalit Newa Lipi Character Dataset" paper exists, but its
  data (via `sobit-nep/Prachalit-Script-Character-Recognition-using-Deep-CNN`
  on GitHub) is a **dead Google Drive link (404, not actually
  "anyone with the link")** — verified, not assumed. heiDATA's `FID4SA-GT`
  dataverse (the same group that hosts the Newa mirror) also has Devanagari,
  Malayalam, and Bengali printed GT sets, CC-BY/CC0 — none of them our target
  scripts, not pulled.
- **Ol Chiki (Santali)**: no image+GT dataset exists on Zenodo/HF/GitHub (only
  a Tesseract `.traineddata` blob and app scaffolding with no bundled data).
  Internet Archive had real wins instead: **`chiki-chemed`** (a Santali
  primer, real scanned pages, CC BY-NC-ND 4.0 — note the NC-ND clause before
  any commercial use) and **`aale-chhatka-pdf-e-magazine-{2023,september-2023}`**
  (a real monthly Ol-Chiki literary e-zine, self-published by its own editor's
  archive.org account; the 2023 item carries the Creative Commons Public
  Domain Mark 1.0, which is neither CC0 nor a license, while the September item
  has no license field despite the same publisher — don't assume terms
  transfer). All three downloaded, size-verified, script-confirmed by rendering
  a sample page.
- **Tirhuta/Mithilakshar — best new lead of this pass**: the Maithili e-zine
  *Videha* (world's first Maithili webzine) publishes a **parallel Tirhuta
  edition of every issue**, archived at Internet Archive under the `videha_*`
  family, **CC BY-SA 4.0**, issues 001–149 confirmed present (3 items,
  ~12 GB) plus larger combined items (`videha-262` ~23 GB, `videha-306`
  ~40 GB) referencing further issues — **not bulk-downloaded, catalogued
  only**. Pulled one sample PDF and confirmed by rendering it that the pages
  are genuine Tirhuta-shaped glyphs, but `pdftotext` on the embedded text
  layer yields **Devanagari-range codepoints**, not the true Tirhuta Unicode
  block (U+114xx). That means these are **legacy glyph-substitution fonts**
  — the same shape of problem this repo already solved for Gorkhapatra
  Devanagari (`nayanepal`/legacy Preeti-family). If a Videha-specific
  font map can be built, this is real running text at real periodical scale,
  the same playbook as the Limbu win. This is the single most promising
  unexploited source found this session.
- **Lepcha / Kirat Rai / Sunuwar / Limbu (Sikkim Herald)**: the four newer IPR
  listing endpoints (`?language=Rai|Mukhia|Lepcha|Limboo`) remain useful for
  discovery, and no Zenodo/HF/GitHub/IA periodical archive was found. This
  2026-07-06 note is superseded for admission by the 2026-07-15 rights audit:
  the express reproduction permission covers material hosted on the official
  State Portal, not automatically `ipr.sikkim.gov.in`. The IPR tool is now
  metadata-only by default; admit only a mirrored
  `sikkim.gov.in/uploads/SikkimHerald/` file or separately permitted IPR file.
- **Nepal Lipi periodicals**: the Internet Archive `nepalbhasatexts`
  collection (477 items) is almost entirely **Devanagari**-script Nepal
  Bhasa print media (Situ, Jhee, Thaunkanhe, Khonapau, etc. — already the
  domain of the existing Newar-Devanagari track, not this one). Checked
  several candidates that looked native-script by title: `kmc-notice-2023`
  is a single decorative Ranjana logo clipping, not running text.
  `NepalLipiSangrahaByHemrajShakyavamsha_201712` (no stated license) is a
  Devanagari-language book *about* Nepal Lipi that does contain real
  alphabet-plate specimens of historic native scripts (Kunmol, etc., with
  transliteration) — a glyph reference, not periodical running text.
  No genuine Nepal-Lipi-script periodical PDF archive was found.

**Recommendation, one line per script**: Newa — nothing left to pull, move to
building the line-crop pipeline on the corrected 441-page set. Ol Chiki — use
`aale-chhatka-pdf-e-magazine-2023` (publisher/uploader Public Domain Mark 1.0,
confirmed script) as the first real
training/eval seed. Tirhuta — decode the Videha legacy Tirhuta font (highest
leverage find this session: real, licensed, periodical-scale) before anything
else. Lepcha/Kirat Rai/Sunuwar/Limbu — use the IPR listing only to discover
editions, then catalog hash-pinned mirrors on the permission-covered State
Portal; do not bulk-download IPR PDF bodies without separate exact-prefix
permission evidence.

## Native-script wave 1 results (2026-07-06)

- **Ol Chiki v1 (first recognizer for the script here)**: PP-OCRv5 mobile
  from generic pretrained, 56,640 synth lines (Santali Wikipedia corpus ×
  3 Noto weights × 6 degradations), 25 epochs. Synth eval best exact-line
  0.546 / norm_edit_dis 0.800. **Real print (234 Aale Chhatka magazine
  lines via the new OLCKOptimum converter): cp CER 0.2245, exact-line
  0.184** — real-print performance ≈ synth-eval performance, so the
  synthesis recipe transfers; quality itself is the v1 limit. Known levers,
  in leverage order: (1) **same-font synth** — we hold the extracted
  OLCKOptimum TTFs; replay the Sunuwar same-font lesson (0.342→0.121);
  (2) resolve the 2 uncertain converter bytes (`n`,`T`) to unlock 653 more
  real lines for train/eval; (3) real-line fine-tune on an issue-split of
  those lines. Artifacts: `outputs/olchiki-v1-run-2026-07-06/`
  (checkpoint+export+scores), pack `data/olchiki-real-eval-pack/`,
  converter `nepal-ttf2utf@acab167`. Claim note: Santali OCR exists
  elsewhere (India); this is pack coverage, not a first-ever claim.
  **v2 (same-font, 2026-07-07)**: re-rendered the same corpus with the
  OLCKOptimum fonts extracted from the magazine PDFs (all 48/48 block chars
  covered by the embedded subsets) + one Noto weight: **real-print cp CER
  0.1911, exact-line 0.321** (same 234-line pack) vs v1 0.2245 / 0.184 —
  the same-font lever works on this script too, though less dramatically
  than Sunuwar's; remaining error is heading/display type, tight-leading
  crop bleed, and genuine model limits. Artifacts:
  `outputs/olchiki-v2-run-2026-07-07/`. A later asset-independence audit found
  that those fonts came from the exact evaluation edition, so v2 remains a
  useful diagnostic but cannot provide an edition-independent claim result.
- **Newa native v1 (manuscript domain): honest failure.** 150 epochs on the
  recovered Zenodo 441-page GT (2,391 train lines, by-manuscript split,
  800px width, PP-OCRv5 mobile from generic print pretrained): dev exact
  0.0016 / norm_edit 0.144, and crucially **train-side norm_edit is also
  0.146** — the model could not fit its own training set, so this is a
  recipe failure (print-pretrained mobile backbone vs dense 16th–19th-c.
  handwriting at 63–154 chars/line), not a hand-generalization failure.
  The reference point (O'Neill 2022/2025, Transkribus HTR on the same data)
  shows the task is learnable with HTR-specific training. v2 design:
  synth-first curriculum — render the TRAIN-split transcription text (plus
  the 36k-char web corpus; no test text, no transliteration) in the 2
  Unicode Newa fonts with degradations, pretrain, then manuscript
  fine-tune; consider the server rec backbone. Artifacts:
  `outputs/newa-native-v1-run-2026-07-07/`.

## Fan-out results (2026-07-13, Codex wave)

- **Ol Chiki uncertain bytes RESOLVED** (`nepal-ttf2utf@5822f30`): bytes
  `n`→ᱱ (EN U+1C71) and `T`→ᱛ (AT U+1C5B), each confirmed by three
  independent evidence lines (glyph stroke features, corpus context
  distributions vs the 204k-line Wikipedia corpus, lexicon hit-rates 0.89 vs
  0.57–0.60 runner-up); evidence at `outputs/olchiki-uncertain-bytes-2026-07-13/`.
  Clean real-print yield rose 234 → 895 lines; expanded pack at
  `data/olchiki-real-eval-pack-v2/`. **olchiki-v2 on the 895-line pack:
  cp CER 0.1275, exact-line 0.197** (scored on omac, 5m18s) — the new
  body-text-dominated benchmark reads meaningfully better than the
  display-type-heavy 234-line pack suggested (0.1911). Pack v2 was initially
  treated as the reference eval, with 895 candidate lines. The
  following audit superseded that interpretation before any claim use.
  **Corrections (2026-07-14, license audit + label audit)**: (a) public
  claims may use only the publisher/uploader public-domain-marked issue; the
  mark is not CC0 or a license; (b) the label audit found 67/895 crops blank
  (pack-builder bug, all in that issue). The resulting cp CER 0.1018 / exact
  0.191 (n=366) and combined clean 0.0826 (n=828, internal) are historical
  diagnostics, not claim headlines: surviving crop/label mismatches, ambiguous
  punctuation, incomplete provenance, and same-edition font leakage remain.
  Also found: the
  Newar-Devanagari packs carry systematic converter whitespace artifacts
  (spurious space after halant / before combining marks; ≥1 in ~45% of test
  lines) — word content is faithful, but a whitespace-normalization pass is
  required before those numbers are claim-grade. See docs/DATA_LICENSING.md
  and the verification-wave section.

## Newa root cause found (2026-07-15): CTC timestep cap, not data or height

The height-ablation prep (dry-runs on omac; artifacts at
`outputs/newa-height-ablation-prep-2026-07-14/`, VERDICT.md) found the
mechanical cause of the v1 training failure: PPLCNetV3's train-mode forward
ends in `adaptive_avg_pool2d([1,40])`, hard-capping the CTC sequence at
**T=40 timesteps at any input height or width** — and **95.7% of manuscript
train lines exceed 40 chars** (median 63, max 154). CTC cannot emit more
labels than timesteps: v1 could never fit its own training set, and a pure
height ablation is predicted to reproduce the failure. This coheres with
every other result: Limbu/Ol Chiki/Newar-Devanagari lines mostly fit ≤40
chars and their models trained fine. Also verified: heights 96/128 train as
pure config changes (3 finite loss steps each), but the EVAL path breaks
(`Im2Seq` asserts H==1) — a verified 1-line patch ships in the prep dir;
pretrained weights load identically at all heights; an earlier T=40 memory
estimate proposed L4 batches 16 (h96) / 12 (h128), but T=200 requires a fresh
approved-GPU measurement. The corrected Newa experiment therefore needs a single
train/eval pool width of T=200 (the exact CTC lower-bound maximum is 176 after
normalization), h96, and only the CC-BY-4.0 Zenodo manuscript source. A
backward-compatible pool patch has now been shape-tested on `omac`: defaults
remain train T=40 / eval T=100, while the opt-in h48 and h96 paths both produce
T=200 and a length-162 CTC loss is finite. The pre-existing zero-byte
`newa_real_000041.png` was not dropped: it was restored from the authoritative
local master and pinned at SHA-256
`c86f18e93b123a87e26682b973c5db8a2f8931434f11f6c88a63105161920360`.

## Native-script verification round (2026-07-15)

- **Newa CTC200 payload staged, not trained.** Labels are narrowly normalized
  (NFC, NBSP to space, whitespace runs collapsed, boundaries trimmed), and
  every referenced crop decodes with ink. Exact-label and 16-codepoint
  substring isolation removed 430 of 2,391 training rows, leaving 1,961.
  Hitopadesa is the 144-line checkpoint-selection dev manuscript; the 635-line
  Cambridge manuscript is sealed as `claim_eval` and must not enter training or
  model selection. Post-filter overlap is zero and all labels are feasible at
  T=200. A prediction-free all-row Cambridge review bundle has now been built
  on `omac` from the exact payload and original Zenodo export: 635/635 bundled
  PNGs are byte-identical to their payload crops, all 635 decision rows remain
  blank, and the archive is explicitly not claim evidence. The crop labels
  still need documented native pixel review; any corrections require a fresh
  post-review exact/16-codepoint leakage audit before a claim-eligible freeze.
  A fail-closed freezer now revalidates the exact payload/source/review
  archives, immutable fields, pixels, all 635 decisions, normalization, and
  the >=200-row dedup gate. It keeps non-attested outputs claim-false and
  requires exact CTC200, train/dev, post-review leakage, checkpoint, and
  training-history bindings for claim readiness. The real blank worksheet
  fails atomically; no reviewed pack exists. GPU training remains
  approval-gated. Artifacts:
  `outputs/newa-ctc200-prep-2026-07-15/`,
  `outputs/newa-cambridge-review-v1/`, and `docs/handoffs/HANDOFF-newa-ctc200.md`.
- **Newa balanced sampling is now a deterministic, disabled candidate.** The
  exact 1,961-row train-only capsule contains 146,153 Newa codepoint tokens
  across 69 types and 87,777 extended-grapheme tokens across 585 types. The
  schedule excludes the 2 codepoint and 71 grapheme singleton types, adds 490
  references under a three-occurrence row cap, and preserves exactly the same
  1,961 unique image/text pairs. Train/Hitopadesa exact-label and 16-codepoint
  overlap are zero before and after scheduling; no Hitopadesa or Cambridge row
  is copied. The six-file artifact reproduced byte-identically when rebuilt
  with the pinned `regex==2026.5.9` segmentation runtime. This is a sampling
  heuristic, not accuracy evidence, and no current payload or config enables
  it. Activation requires a newly hash-bound training path, approved GPU
  ablation, and the final corrected-Cambridge leakage audit. Artifact:
  `outputs/newa-balanced-sampling-v1/`.
- **Stock RepSVTR is a runnable CTC200 fallback, not a trained result.** A
  no-image, batch-1 CPU probe on `omac` used the clean PaddleOCR v3.7.0 checkout
  and the official hash-pinned RepSVTR checkpoint. At H96xW1600, the
  train-mode CTC branch and eval-mode full forward both emit T=200, leaving 24
  timesteps above the audited corpus maximum of 176; an alternating length-176
  synthetic CTC loss is finite. All 301 backbone tensors and 394/398 total
  tensors are pretrained-compatible, with only the four Newa-vocabulary-sized
  tensors reinitialized. A follow-up synthetic CPU probe then exercised the
  exact five-tensor loader contract through one full train-mode MultiHead
  forward, CTC+NRTR MultiLoss, backward, and scheduler-positioned AdamW update.
  It observed CTC `[1,200,108]`, NRTR `[1,177,112]`, finite component losses,
  279/279 finite nonzero gradient tensors, and finite positive updates in the
  backbone plus both heads; peak RSS was 1,646,444,544 bytes. No image, label
  file, dataset row, prediction, score, GPU, cloud, or network action occurred.
  The exact configured warmup starts at LR 0, so the probe explicitly advanced
  the scheduler 1,961 times without optimizer calls before its sole update at
  LR `4e-5`; this is mechanism evidence, not first-iteration chronology or
  training-loop history. Together the probes establish an independent
  backbone/temporal path from patched PPLCNetV3 and close the synthetic
  full-loss/backward/update gap, not the framework, real-data, GPU-memory,
  convergence, or accuracy gaps. Training remains approval-gated, and
  selecting this path requires the distinct stock-RepSVTR claim-attestation
  profile now implemented in `tools/audit_newa_cambridge_claim_readiness.py`
  and its freezer. That profile pins the exact config, feasibility report,
  fully clean PaddleOCR checkout, stock sources, official initialization,
  approval, run log, selection record, and a CPU-loaded final state dict with
  the exact 398 Newa model keys/shapes. The freezer independently requires the
  raw approval/run/log/selection records and the selected checkpoint. The
  legacy four-file PPLCNetV3 profile remains unchanged. Identity and execution
  authenticity are still an external custody/signature boundary; no real
  approval, training record, compatible final checkpoint, attestation, or
  score exists. Artifacts: `outputs/newa-repsvtr-ctc200-feasibility-v1/` and
  `outputs/newa-repsvtr-full-train-step-v1/`.
- **Ol Chiki v2 withdrawn from claim use; v3 is only a review candidate.** A
  full audit found blank/rule-only crops, unrelated or neighboring text,
  unresolved legacy punctuation, a stale converter-commit claim, and training
  fonts extracted from the evaluation edition. The public-source-only v3
  builder pins the PDF and resolved map, uses neighbor-clamped vertical crops,
  rejects blank renders, emits line-level hashes/provenance, and refuses to call
  its output frozen or claim-eligible. The remote candidate contains 358 rows
  after 75 blank renders were excluded. Every row still requires review by an
  Ol Chiki reader; no new score is permitted before freeze and independent
  exact/substring and asset-isolation audits. A fail-closed freezer validates
  the pinned candidate and all-row review archives, the source PDFs, every crop
  and provenance binding, and all 358 immutable review decisions; it refuses
  claim eligibility without separately bound leakage and asset-isolation
  attestations. No decisions or frozen pack exist. Artifacts:
  `outputs/olchiki-claim-eval-v3-2026-07-15/`.
- **Tirhuta now has an edition-disjoint train/eval candidate route, not a
  score.** PyMuPDF retained glyph IDs for all 3,306 U+FFFD characters in the
  152-page issue-001 sample. A fail-closed 164-entry functional map recovered
  every one; 30,396/30,581 Janaki trace fragments then converted cleanly
  (99.395049%). Issue 001 now supplies 400 nonblank real-line eval candidates
  across 116 pages and a prediction-free all-row review packet. The separately
  dated, CC BY-SA 4.0 April 2008 issue required 34 additional GID mappings.
  Exhaustive shaping with the hash-pinned Janaki 1.000 reference font
  reproduced all 34 plus the prior 164 with zero regressions. Its 2,000-row
  candidate has 2,000 unique image hashes and zero image-audit issues; removing
  one 16-codepoint cross-issue substring conflict leaves 1,999 training rows
  with zero exact or long-substring overlap. Thirty new GIDs have clean crop
  witnesses and four have shape evidence only. The 400 eval rows and 30 focused
  mapping rows remain native-review pending; no model or CER exists. Artifacts:
  `outputs/tirhuta-videha-gid-recovery-v1/`,
  `outputs/tirhuta-videha-line-candidate-v1/`, and
  `outputs/tirhuta-videha-april2008-train-candidate-v1/`. A separate
  fail-closed freezer now validates the two pinned issue-001 archives, source
  PDF, packaged audits, converter/runtime provenance, and the exact crop bytes
  shown to reviewers. It requires all 400 immutable decisions, at least 200
  nonblank unique accepted lines, and Tirhuta-bearing human labels. Claim
  eligibility remains false unless an independent attestation binds the
  reviewed label-file hash and the exact April train label/manifest hashes and
  passes exact-text, 16-codepoint substring, edition, and model-selection
  isolation. The decisions and attestation do not yet exist, so no pack was
  frozen and no score was run.
- **The other-script registry was downgraded to extant evidence.** The Kirat
  Rai pack has 70 rows / 53 unique labels, and all 53 occur in its real-render
  training inventory. The shared Devanagari harvest has 79,364 labels but only
  40,358 images, so it is a training inventory with 39,006 missing assets, not
  eval. Sunuwar's historical 429-row pack/checkpoint and Lepcha's source PDFs
  and checkpoint are absent. Historical metrics remain historical and none of
  these routes is a current script delivery. Registry evidence:
  `configs/nepal-language-pack-pipeline-v1.json`,
  `configs/recognizer-eval-pack-normalization-v1.json`, and
  `configs/eval-harness-recipes-v1.json`.
- **Sikkim Herald source permission was found and the three routes reopened.**
  The earlier Indian-government-work term analysis missed the Government of
  Sikkim State Portal's express copyright permission. Its official copyright
  page permits free reproduction in any format/media without specific
  permission when the material is reproduced accurately, is not used
  derogatorily or misleadingly, and the source is prominently acknowledged;
  explicitly identified third-party material is excluded. The same portal
  hosts Sikkim Herald, and the IPR Department identifies it as its official
  government paper. A live metadata-only enumeration of the newer IPR portal
  found 19 current Lepcha, 22 Mukhia/Sunuwar, and 5 Rai discovery candidates,
  while the State Portal exposes dated editions for all three. A bounded live
  State Portal catalog retained 29 unique canonical PDF URLs: 8 Lepcha, 11
  Mukhia/Sunuwar, and 10 Kirat Rai. It rejected one duplicate and one non-PDF
  path. The searches advertise 193/167/174 records, but `PageNo=1/2` returned
  identical URL lists, so the artifact is explicitly incomplete. It fetched
  no full PDF body and made only three eight-byte range probes against samples
  already full-hash-pinned by the prior policy. Every edition remains
  third-party-screening pending, train/eval blocked, and claim-grade false.
  Because the permission page is domain-specific, only `sikkim.gov.in`-hosted
  PDFs are catalog candidates; IPR-subdomain files remain discovery-only unless
  mirrored or separately permitted. This resolves the State Portal
  source-permission gate under a custom permission, not a public-domain claim.
  It does not restore the lost packs/models or cure the Kirat Rai pack's 100%
  text leakage. Next steps are full-hash acquisition and third-party screening
  of selected editions, followed by fresh edition-disjoint converter and native
  label audits. Evidence:
  `outputs/sikkim-herald-source-rights-2026-07-15/`,
  `outputs/sikkim-herald-state-portal-catalog-2026-07-15/`,
  `docs/DATA_LICENSING.md`, and `docs/ATTRIBUTIONS.md`.
- **Six Sikkim Herald editions were acquired, page-screened, and font-audited
  without creating labels.** The source contract selected two State
  Portal-hosted editions per script: Kirat Rai 2024-09-26/2025-01-13 (109
  days), Lepcha 2024-07-26/2024-10-04 (70 days), and Sunuwar/Jenticha
  2024-04-12/2025-01-26 (289 days). All six full PDF hashes and all 26 page
  renders were frozen after an `omac` run. Reviewer 1 inspected every page at
  original render resolution: 25 pages have no explicit third-party marker
  observed, while both complete press-photo rectangles on Lepcha 2024-10-04
  page 1 are bounded for exclusion because incidental storefront signage is
  visible. Both Lepcha PDFs have a structurally and visually blank page 5.
  This is a one-reviewer observation under the State Portal's custom
  permission, not a public-domain or government-authorship finding; reviewer 2
  and adjudication remain null.
- **The revised live-layout converter audit reaches 100% aggregate coverage on
  both Kirat Rai and both Sunuwar issues.** Kirat raw body-unit totals are
  13,101 and 7,920. Exact comparison to the `kirat rai font new` program at
  xref 536 of official Unicode proposal L2/22-043R resolves the newly observed
  Herald `T` through canonical `Z` to U+16D6C; the isolated extracted `C` has
  an empty outline and normalizes to space, and original-resolution review
  establishes that the custom glyph under `:` is a double quotation mark.
  The pair has 91 observed semantic source-code matches, zero mismatches, and
  zero unresolved units. Sunuwar raw totals are 9,697 and 14,676. Both font
  subsets match the exact `kirat1` program in the cited
  `sunuwar_01348c63.pdf` derivation source on every observed script code (76
  summed matches, zero mismatches); backslash is an empty glyph, and the one
  U+201D extraction value visibly prints as a colon. The official proposal PDF
  hash is `88b1d4dfedd8b36a687533f3fa1ef0a97c42965fd595b28678ca8d7d17cb8f17`
  and its embedded reference font hash is
  `bab945f8a5fe51f9401dc2a55a54ac99f3ba2b5071ac72324a05e7ff82cb4149`;
  the Sunuwar reference PDF/font hashes are
  `b3c001ff428de3b4e539c88d6d93566ae969a46b9ba947fe680fc1b68eb10456`
  and `484550543232f587fab3a5b3e900c922845c5ed44ef7bdd493cace4a5b6bce78`,
  tied to MIT-licensed `nepal-ttf2utf` 0.2.0 at
  `a158a8cbdbe230df6849afd9c9736bf31cb9c24b`. The two 2024 Lepcha bodies are
  raster-composed, with 640 and 2,212 image placements, so converter coverage
  is N/A rather than 0%.
- **The joined bundle passes adversarial metadata audit but remains
  `NOT_ADMITTED`.** Mutation tests reject any label-write claim, injected
  text/label field, unreconciled counter, font mismatch, or invented second
  reviewer. The frozen audit joins the exact source review, 26-page reviewer-1
  record, revised converter artifact, and reference config; it records four
  live editions at 100%, two raster N/A editions, zero labels, zero native
  reviewers, no GPU/cloud/training/scoring, and no stored source or converted
  text. Exact and substring leakage are explicitly N/A with zero rows, not
  passed. Before any line can exist, the route still needs independent
  provenance reviewer 2/adjudication and native-script pixel/label review;
  only then may it create an edition-isolated train/eval split, run exact plus
  substring leakage audit, train, and compute codepoint CER. Artifacts:
  `outputs/sikkim-herald-six-edition-source-review-2026-07-15-r11/`,
  `outputs/sikkim-herald-six-edition-manual-review-r1-2026-07-15-r11/`,
  `outputs/sikkim-herald-six-edition-converter-layout-audit-2026-07-15-r11b/`,
  and `outputs/sikkim-herald-r11b-adversarial-bundle-audit-2026-07-15-r11/`.
- **A blinded Sikkim Herald provenance-review handoff is frozen, but no second
  review is claimed.** The reviewer-2 packet links all 26 exact full-page
  renders at original resolution without copying images or exposing reviewer-1
  decisions. Its byte-bound worksheet requires a distinct reviewer 2 who did
  not inspect reviewer-1 decisions before completion, a third distinct
  adjudicator, exact six-edition/26-page coverage, and one resolution for every
  disagreement and none for an agreement. Mutation tests reject prefilled
  decisions, reused identities, missing/duplicate pages, out-of-page exclusion
  geometry, label/prediction payload keys, and incomplete adjudication. The
  independently frozen packet audit verifies all 26 HTML links and hashes and
  reports `PASS_READY_NOT_ADMITTED`; the real worksheet remains blank, so
  reviewer 2, adjudication, native line review, rows, leakage, a model, and CER
  remain absent. Artifacts:
  `outputs/sikkim-herald-six-edition-provenance-review-r2-packet-2026-07-15-r11/`
  and
  `outputs/sikkim-herald-six-edition-provenance-review-r2-packet-audit-2026-07-15-r11/`.
- **The recovered Newar-Devanagari whitespace rewrite is mechanically exact,
  but the underlying pack fails admission.** A 64 GiB `omac` run independently
  reproduced all 1,820/565 train, 634/353 dev, and 427/142 test removals for
  the two narrowly defined artifact classes, matching all 6,763 normalized
  labels and leaving zero targeted patterns. All referenced PNGs decode and
  contain at least 16 dark pixels; there are 6,757 unique image hashes, four
  intra-train duplicate groups (six rows beyond the first), zero conflicting
  labels within those groups, and zero cross-split image hashes. The honest
  split audit fails: train versus dev+test has 18 exact rows and 575 rows with
  a shared 16+-codepoint substring (including exact), while dev versus test has
  zero exact but 136 substring rows. The recovered labels also lack source URL,
  PDF hash, page, bbox, date, and edition fields, and Gorkhapatra rights remain
  `RISK`. A deterministic non-native geometry review saw ink in all seven
  risk-selected samples but obvious neighboring-line fragments in two of the
  three widest crops; it makes no population-rate or label-faithfulness claim.
  No filtering, training, prediction, score, or CER was run in this audit
  round. Artifacts:
  `outputs/newar-devanagari-normalized-claim-boundary-audit-2026-07-15-r12/`
  and
  `outputs/newar-devanagari-normalized-visual-sample-review-2026-07-15-r12/`.

## Gurung Khema official-print reopening (2026-07-15 r13)

- **The former “synthetic-by-nature” source conclusion is refuted.** Two
  official State Portal Sikkim Herald Gurung Editions contain running Gurung
  Khema print across eight pages and are 52 days apart. The 2022-03-01 PDF is
  SHA-256
  `e667bbd2eb4f6769209013fe569cdc37d3e1aff40a9f0121301b43a03afe5d14`;
  the 2022-04-22 PDF is
  `2db16c7b7abe678e02103037397c7a35e21fb2fd34d4e4413d1979d1cf7f7c93`.
  The pages contain 21,106 text-layer characters. April page 4 prints issue 4
  while pages 1–3 print issue 5; the frozen source record preserves this
  anomaly. Reviewer 1 inspected every original-resolution page and observed no
  explicit third-party marker, but that is not a government-authorship or
  license-cleanliness finding.
- **The legacy font has a fail-closed Unicode 16 conversion path.** Unicode
  proposal L2/22-096 states that the republished 2022 Herald uses the same
  legacy KHEMA font as the proposal and supplies the legacy key semantics. Both
  Herald PDFs use U+F0xx PUA re-encodings of those legacy bytes. Their 56 shared
  cmap codes have exact decomposed outlines and advances. For each edition, all
  53 compared observed mapped glyphs rank first against the expected proposal
  glyph; the minimum observed binary-mask IoU is 0.490751. The converter handles
  direct keys, the Herald C/N/X printed-shape aliases, O/V combining-mark
  unifications, adjacent pre-base T/E reordering into Unicode logical order,
  and paired `DD` as double danda. It rejects ambiguous input rather than
  guessing.
- **Mechanical eligibility is 749 of 759 candidate lines, not a dataset.** The
  March edition has 382 eligible and 6 rejected lines; April has 367 eligible
  and 4 rejected. Across both editions, the 10 rejections are four lines with
  no Gurung Khema character, five non-adjacent pre-base-medial sequences, and
  one isolated legacy danda unit. The audit serializes no source or converted
  strings and writes no crops or labels.
- **The route remains `NOT_ADMITTED`.** The separately frozen reviewer-2 packet
  links all eight hash-bound source renders, exposes no reviewer-1 decisions,
  and contains a byte-identical blank worksheet. Its adversarial audit reports
  `PASS_READY_NOT_ADMITTED`. An independent reviewer 2 and distinct
  adjudicator, then native pixel-label review, crop construction, an
  edition/manuscript-isolated split, exact plus substring leakage audit, and
  codepoint-CER scoring are still required. There are currently zero labels,
  predictions, scores, training rows, or CER values. The joined r13
  adversarial audit also pins every source/font/review/packet hash, rejects
  mutated counts, glyph ranks, blinding, payloads, and admission, and exercises
  the real freezer to prove that the blank worksheet fails before writing an
  output directory. Artifacts:
  `outputs/sikkim-herald-gurung-khema-two-edition-source-review-2026-07-15-r13/`,
  `outputs/sikkim-herald-gurung-khema-font-semantic-audit-2026-07-15-r13/`,
  `outputs/sikkim-herald-gurung-khema-manual-review-r1-2026-07-15-r13/`,
  `outputs/sikkim-herald-gurung-khema-provenance-review-r2-packet-2026-07-15-r13/`,
  and
  `outputs/sikkim-herald-gurung-khema-provenance-review-r2-packet-audit-2026-07-15-r13/`,
  plus the joined verdict at
  `outputs/sikkim-herald-gurung-khema-r13-adversarial-bundle-audit-2026-07-15-r13/`.

## Corrected claim numbers (2026-07-16, fixes executed on omac)

- **Ol Chiki eval pack v3** (blank-guard builder, mechanism: white-fill
  invisible text in the source PDF): 772 clean lines (123 blank/near-blank
  dropped — the <16-dark-px guard caught 56 more than the census's fully
  blank 67). olchiki-v2 scores: **public headline (CC0 issue) cp CER 0.0961
  / exact 0.193 (n=358)**; September subset 0.0704 (n=414); combined 0.0826
  / 0.2034 (n=772, internal until September license confirmed). Supersedes
  all prior Ol Chiki numbers.
- **Newar-Devanagari whitespace-normalized rescore** (newar-ft-v1, frozen
  1,139-line test, scored on omac): old artifact-bearing 0.0466; raw
  predictions vs normalized labels 0.0501 (the model reproduces training
  artifacts); **both-sides-normalized cp CER 0.0441 / exact 0.2888 — the
  claim-comparable number** (labels at
  outputs/newar-pack-v1-normalized-2026-07-15, utility
  src/ocrtech/devanagari_whitespace.py).

## Tirhuta real substrate delivered (2026-07-17)

First-ever real Tirhuta training text: `outputs/videha-tirhuta-silver-2026-07-17/`
— **4,148 clean converted lines** (labels.txt) from Videha issues (CC BY-SA
4.0, per-issue provenance), zero-unmapped-only policy. Extension-byte drop is the
dominant loss (e.g. first issue: 1,235 extension-dropped of 7,622 examined)
— building the conjunct extension-glyph map would multiply yield; per-issue
numbers in the pack summary.json. Also staged and VERIFIED: Ol Chiki v3 fine-tune payload
(`outputs/olchiki-v3-ft-payload-2026-07-17/`, 5,330 entries) — train is 242
real Public-Domain-Mark-issue lines + 5,000 synth replay, September 414
eval-only with a zero-intersection leakage audit on filename/line-id/label;
READY-PENDING-APPROVAL in the GPU dossier (see
outputs/overnight-consolidation-2026-07-16/gpu-batch-proposal.md, candidate 3
update). Operational note: the Codex quota briefly hit its limit on 2026-07-16
(reset message named 2026-07-22) but lifted the same night — fan-out resumed
with the Tirhuta extension-map and Lepcha zero-yield lanes.

## Lepcha zero-yield root cause (2026-07-17)

The Sikkim Herald Lepcha lane's 0 lines (vs Sunuwar 9,743 / Kirat Rai 3,699)
is a data reality, not a bug: in all 18 editions the Lepcha article body is
rasterized as image XObjects — there is no live Lepcha text layer (live text
is English masthead/punctuation in Times New Roman / anonymized CID fonts;
zero U+1C00-1C4F codepoints in sampled pages). The converter was never
invoked. Full evidence: outputs/lepcha-zero-yield-diagnosis-2026-07-17/REPORT.md.
Consequence: Lepcha real-line substrate requires a raster OCR/transcription
lane (or stays synth-first per the headline plan); the builder should record
`raster_body_requires_ocr` instead of `wrong_font` for such editions.

## Tirhuta extension map + v2 pack: yield 2.6x (2026-07-17)

Codex built the Videha Tirhuta extension-glyph map (149 new mappings, each
grounded in font glyph semantics + Pandey L2/11-175R; 2,131 converter tests
pass) — branch `agent/tirhuta-extension-map` in nepal-ttf2utf, commit
a5ba521. Rebuilt the pack on omac with the new converter:
`outputs/videha-tirhuta-silver-v2-2026-07-17/` — **10,696 clean lines**
(vs 4,148 in v1; 45,157 examined; extension-byte drops now 1, down from
thousands). Pack verified (archive files == labels == manifest == 10,696),
backed up to HF ampixa/ocr-tech-artifacts. Builder bug found during
finalize: `str.splitlines()` splits JSON-legal raw U+0085 in provenance
strings — fixed to split("\n") (patched builder stored in the pack dir).
Remaining converter work: merge the branch to main after review.

## Three FT payloads staged (2026-07-17)

Codex assembled per-script fine-tune payloads on the Ol Chiki v3 discipline
(issue/page-boundary dev splits, four-way zero-leakage audits, sha-pinned
manifests, training_launched=false):

- **Tirhuta READY**: outputs/tirhuta-v2-ft-payload-2026-07-17/ — 9,765 real
  train / 844 dev (whole issue videha_01_02_08 held out), 10,639 entries,
  sha 55c18998...13e4. CC BY-SA 4.0 with attribution recorded.
- **Sunuwar BLOCKED-PENDING-LEGAL**: outputs/sunuwar-v1-ft-payload-2026-07-17/
  (8,422/514) — Sikkim Herald State-Portal permission + India §17(d) question
  unresolved; assembled but not trainable yet.
- **Kirat Rai BLOCKED-PENDING-LEGAL**: outputs/kiratrai-v1-ft-payload-2026-07-17/
  (2,865/193) — same gate.

Verified: no synth replay pools or warm-start checkpoints exist for these
three scripts anywhere in outputs/ or HF artifacts (the native-script wave
built substrate, not models; only Ol Chiki trained). Consequence: the proven
synth-pretrain-then-real-FT recipe needs a Tirhuta synth pack before the
Tirhuta payload trains warm; alternatively from-scratch real-only must be an
explicit dossier decision.

## Tirhuta synth-v1 pretrain pack rendered (2026-07-19)

The missing pretrain stage for Tirhuta is now built:
`outputs/tirhuta-synth-v1-prep-2026-07-17/tirhuta-synth-v1-rendered.tgz`
(sha 2bf91b49...1327, 48,960 PNGs = 8,160 texts x 6 degradation profiles,
7,997 train / 163 eval texts, zero normalized collisions with the real FT
dev set, Noto Sans Tirhuta OFL only, RAQM shaping verified, dark-pixel
guard min 102). Rendered on omac; archive re-extracted and hash-verified;
backed up to HF. Engineering notes: macOS Pillow lacks RAQM — fixed via
Xcode CLT + brew libraqm + DYLD_FALLBACK_LIBRARY_PATH exported INSIDE the
script (SIP strips DYLD_* across /bin/zsh exec); audit had a /tmp vs
/private/tmp resolve() mismatch, patched in the pack's audit script.
Tirhuta training chain is now fully staged: synth-v1 pretrain ->
tirhuta-v2-ft real payload (dossier candidate 5); GPU run still
human-gated.

## Next Work

0. **Post-recovery next (2026-07-05)**: the spike verdict keeps the Paddle
   det + Rows-LTR + y/conf70 chain as the backbone. **Phase-2 pivot, measured
   2026-07-05**: decomposing the 0.4134 filtered-baseline page CER on the
   16-page pack showed the standing "region selection + furniture" assumption
   is wrong — deleting every unmatched prediction (perfect furniture/region
   filtering) changes CER by ~0 (0.4152), while reordering the matched lines
   into reference order alone drops it to **0.0378** (10.9×). **Reading order
   is 91% of page CER.** Geometry heuristics cannot recover it (Columns LTR
   0.799, PP-Structure-region banding 0.702, PP-Structure native order 0.797)
   because the reference order is the PDF's article-box emission order, not a
   row/column pattern. Three follow-up experiments (all 2026-07-05,
   artifacts under `outputs/reading-order-model-v1-2026-07-05/` and
   `outputs/line-merge-v1-2026-07-05/`) then closed the question:

   - **Learned reading-order model v1: refuted.** LightGBM pairwise
     comparator on the silver `span_bbox` order supervision regressed to
     0.799 — the Devanagari packs preserve PDF article-box emission order
     (not row-major) while the Limbu eval reference is a strict y-sort;
     wrong supervision for the target. Leakage-clean; kept as a negative
     result.
   - **Detector-fragmentation hypothesis: refuted.** Det boxes ≈ 1:1 with
     reference lines (mean best-IoU 0.68; the widest lines average 1.05
     fragments). Tuned merge (gap 0.6×h) merges ~3 boxes/page, CER flat.
     Naive merge (gap 1.0×h) bridges the 40 px column gutters and collapses
     det F1 to 0.71 — gutters are narrower than line height on these pages.
   - **Root cause found: the reference linearization is our own artifact.**
     `build_limbu_gorkhapatra_all_corpus.py:209` sorts spans strict
     y-then-x across a 6-column layout; det-vs-PDF y-jitter of a few px
     flips column interleaving constantly (pair accuracy 0.998, page CER
     still 0.30–0.41). Pure-y prediction sort gives a free win (0.4134 →
     0.3006), and **one shared deterministic column-major linearization
     applied to both reference and prediction yields page cp CER 0.0451** —
     at the content floor (~0.054: recognition 0.026 + 211 missed / 87
     extra lines). **Page reconstruction on this pack is solved up to
     linearization definition; the honest next metric is page-eval v2 with
     column-major linearization on both sides** (a documented metric
     change — not comparable to the old 0.41-scale numbers), and the
     production assembly should emit column-major order. Remaining real
     levers after v2: det recall/precision (211/87), recognizer (0.026),
     and native review for claim-grade.

   **Page-eval v2 + Column Major assembly shipped 2026-07-05**: the shared
   linearization lives in `src/ocrtech/linearization.py` (tests in
   `tests/test_linearization.py`); `tools/apply_page_reading_order_strategy.py`
   gained mode `"Column Major"`; `tools/run_page_ocr_eval.py` gained
   `--page-linearization {native,column-major}` (recorded in
   `page-eval-run.json`, per-sample fallback to native recorded when a
   reference lacks line geometry). Validated on the 16-page pack:
   Column-Major assembly + v2 scoring gives **page cp CER 0.043 / grapheme
   0.067** (`outputs/limbu-column-major-2026-07-05-score-v2/`); the native
   path is regression-identical (0.413/0.437). v1-scale and v2-scale numbers
   must never be compared directly.

   **Newar page-eval v2 (second-language proof, 2026-07-05/06)**: a 19-page
   Newar page pack built from the rebuilt harvest
   (`tools/build_newar_gorkhapatra_silver_page_eval.py`,
   `outputs/newar-page-eval-v2-2026-07-05/pack/`) and scored through the
   identical non-oracle chain (Paddle det → recognizer → Column Major →
   y/conf70 → page-eval): **newar-ft-v1 native 0.778 → v2 0.199 page cp CER**
   (grapheme 0.279; det F1 0.912; matched-line cp CER 0.0905) — the
   column-major linearization story replicates exactly on the second
   language; the Newar residual is recognition + det-crop domain gap, not
   reconstruction. Two follow-ups surfaced and **both were fixed
   2026-07-06**: (a) the deployed `limbu-devanagari-ppocrv5` export was
   unusable as a general Devanagari recognizer (409-byte dict missing
   anusvara, ii-matra, ṇa…; emitted garbage — its page numbers were a
   formality, not a model comparison); the recovered deva-v2 training
   checkpoint was re-exported with its correct 144-char `devanagari-v2.txt`
   and **verified to reproduce the recorded pre-crash Newar baseline
   exactly** (cp CER 0.062229 / grapheme 0.13120 / exact 0.158 on the
   recovered 1,139-line frozen test); the Space now serves it as
   `models/deva-v2` (export also at `outputs/deva-v2-reexport-2026-07-06/`
   and on the HF dataset). (b) The Newar pack builder's digest mismatch was
   a computation-path divergence from the eval harness's recomputation —
   builder fixed (9 tests green), rebuilt pack at
   `outputs/newar-page-eval-v2-2026-07-06-digestfix/` scores with zero
   digest warnings and identical CER (0.19918).

   Phase 3 done 2026-07-05: config-driven `tools/run_book_ocr.py` landed
   (det → script-route → per-script recognizers from
   `configs/nepal-language-pack-pipeline-v1.json` → Column Major default →
   filter → Unicode/MD export + review queue, resumable, preflight
   validation; smoke on 2 rebuilt Limbu pages: 606 lines kept / 25 queued).
   The rebuilt Limbu pack is 2× the lost one (16 pages / 4,872 labels).

   **ft-v2 retrain result (2026-07-06): new champion candidate.** 40 epochs
   on the L4 (~8.5 GPU-h), 46,753 rows (25,065 synth + 2,711 real ×8
   replay), PP-OCRv5 mobile generic base, selection on real dev (best epoch
   39). On the frozen 902-line test (dates 2082-03-24 / 2082-08-12 /
   2082-12-25, by-date disjoint, leakage-gated), scored through the
   identical local CPU path as the deployed champion:
   **ft-v2 cp CER 0.0047 / grapheme 0.0098 / exact-line 0.859** vs champion
   0.0138 / 0.0284 / 0.637 — ~3× lower CER, +22 pts exact-line. The gap is
   if anything understated: the old champion's training pack drew on the old
   harvest and may have seen these dates; ft-v2's test isolation is proven.
   We also now own a warm-startable training checkpoint again
   (`outputs/limbu-ft-v2-run-2026-07-06/checkpoints/best_accuracy.*`; export
   at `.../export/`). Silver labels, diagnostic, not claim-grade.
   **Acceptance gate PASSED (2026-07-06)**: ft-v2 through the full non-oracle
   16-page chain (same detector, Column Major, y/conf70 filter, page-eval
   v2): **page cp CER 0.0272 / grapheme 0.0347** vs champion 0.043 / 0.067;
   matched-line cp CER 0.0090 vs 0.0266 (−66%); det F1 unchanged (0.9650 vs
   0.9657, same detector). 3,725 of 4,699 matched lines exact. Gate
   artifacts: `outputs/limbu-ftv2-page-gate-2026-07-06/`. Real Gorkhapatra
   newspaper pages now read at ~2.7% codepoint error in human-readable
   column-major order, silver-diagnostic boundary unchanged. Remaining step:
   package ft-v2 (model card, hashes) and swap the Space bundle — a
   production-facing change, pending user confirmation.
1. ~~Finish the 2026-07-04 full harvest and inventory~~ **Done 2026-07-04**: 394
   language-page hits, 50 languages, inventory manifests at
   `outputs/language-pack-inventory-full-2026-07-04/`.
2. ~~Per-language silver packs via the existing `nayanepal` converter~~ **Done
   2026-07-04** (`tools/build_language_pack_silver_line_packs.py`, tests green): the
   batch-1 run produced **116,287 kept silver lines across 50 languages, zero
   errors**, with per-line provenance (PDF hash, page, bbox, raw legacy bytes,
   converted text, image hash) at `outputs/language-pack-silver-lines-2026-07-04/`;
   spot-checked conversions read as clean, language-plausible Devanagari. The
   batch-2 run over the older 589 page hits kept **183,324 more lines (49 languages,
   zero errors)** at `outputs/language-pack-silver-lines-batch2-2026-07-04/` —
   **299,611 per-language-attributed silver lines total** across the two batches
   (~2.4 GB of crops). Run the tool with
   `uv run --python 3.11 --with pymupdf --with npttf2utf` — `nepal_ttf2utf` needs the
   `npttf2utf` base-map package at runtime, and without it every span fails
   (cleanly, with per-span error records, but kept=0). **Honest caveat**: lines
   are page-attributed, not language-verified — a language page also carries Nepali
   furniture/boxes, and pages whose body is not nayanepal Devanagari (Urdu, the
   Tibetan-script pages) yield mostly furniture lines. A language-ID/lexicon filter
   before per-language recognizer training is the follow-up gate.
3. **Newar track (second focus language)**: add the Newar registry entry
   (Devanagari + Prachalit rows); build the Devanagari-written Newar recognizer
   eval/train split from the 6,800 harvested silver lines (by-date split, frozen
   test); keep native Prachalit as the separate longer arc — real printed-register
   Nepal Lipi harvesting, not more transliteration, is the lever.
4. **Classify Tibetan-script pages** (Hyolmo/Mugal/Bhote) and decide converter vs
   native Unicode.
5. ~~Generalize the pipeline wiring~~ **Done 2026-07-04**: all three Limbu-locked
   spots generalized (shared script table, filter keys, `--required-script`
   preflight), tests green; `configs/nepal-language-pack-pipeline-v1.json` holds the
   per-language wiring.
6. **Target-region/furniture filtering stage** for Paddle page/book OCR, scored
   against the current baseline on the 8-page Limbu pack and at least one
   Devanagari-language pack (Newar is the natural candidate given the focus order).
7. Only after page reconstruction improves, export high-quality real/silver line
   crops back into recognizer training, per language.
8. Month-batch the archive back-harvest toward the full ~1,750-article depth, storing
   the mirror on the remote 2 TB volume.

## Mistakes To Avoid

Do not use Naya Nepal blog images as training/eval evidence when the PDF exists. Do
not use wide epaper date windows — the endpoint silently caps at 30 PDFs per query;
weekly windows only. Do not treat unnormalized Gorkhapatra labels as distinct
languages (थारु vs थारू). Do not treat the `nayanepal` text layer as Unicode — it is
legacy Preeti-family and needs the converter before any silver label is real. Do not
report Paddle `norm_edit_dis` as CER; real CER comes from `tools/eval_metrics.py`
micro CER over canonical predictions. Do not treat synthetic data as claim evidence.
Do not call Gurung Khema synthetic-by-nature now that official Herald print is
frozen, and do not promote its 749 mechanically eligible candidates to rows before
independent provenance, native label, crop, split, and leakage review. Do not promote the
targeted+Unicode checkpoint or v3 dictionary warm-starts. Do not spend long blocks on
generic eval-harness work unless it directly improves the book/page OCR loop. Do not
interpret a syntactically valid approval/auditor identity as cryptographic proof;
retain independent custody or signed runner evidence for a production claim. Do not
extend the State Portal's Sikkim Herald permission to the separate IPR PDF origin;
use `ipr.sikkim.gov.in` as discovery-only unless a file is mirrored or separately
permitted. Do not frame video-to-page work as OCR scope; reduce video to stills
first. Do not present
page OCR as solved: matched-line recognition is strong, but full-page CER is still
dominated by reconstruction and region selection.