File size: 101,313 Bytes
f9bc0c1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 | # Research State: Nepal Language Pack OCR
Updated: 2026-09-24
## Devanagari recognizer fine-tune on k2 (2026-09-24)
Goal (owner steer): make the Devanagari route of the live demo (`ampixa.thapa-ashish.com.np`,
k2 OCR worker) read citizenship-certificate-style and general documents properly. Resource
inventory: `docs/DEVANAGARI_RESOURCES_INVENTORY.md`.
- **Base**: official `devanagari_PP-OCRv5_mobile_rec` (568-entry dictionary incl. all Latin
letters and digits); replaced our `deva-v2` on the demo the same morning.
- **Training path**: Paddle CPU training on k2 (Apple M2 Ultra) runs at ~2 img/s, so the same
network is trained with the `transformers` PyTorch port on MPS (~70-90 img/s), exported to
ONNX, and served through PaddleX `create_model(engine="onnxruntime")`. Code:
`tools/deva_rec_torch/`. Preprocessing is bit-matched to PaddleX (98.8% line parity).
- **Data**: real Gorkhapatra print re-cut into PDF-aligned chunks
(`tools/build_gorkhapatra_devanagari_chunks.py`, edition-disjoint: test 2082-03-24,
2082-08-12, 2082-12-25; dev 2081-12-12, 2083-01-11); synthetic citizenship/office/real-text
lines in 44 OFL font families (`tools/render_devanagari_form_synth.py`); NVIDIA
OCR-Synthetic-Multilingual-v1 English (CC BY 4.0) plus Liberation/DejaVu/Lato English renders.
- **Converter bug found and fixed**: npttf2utf 0.3.7 mis-decodes the nayanepal alternate
क्र ligature `j|m` (-> व्र, and प -> फ / stray `m`). 3,426 of 114,452 spans (3%) were affected
(कार्यक्रम, प्रक्रिया, शुक्रबार, ...). Fixed in the chunk builder (`RAW_FIXES`); labels of all
earlier Gorkhapatra-derived packs (incl. `data/gorkhapatra-devanagari-real`, the Newar and
language-pack silver packs built from the same converter) carry the bug. **Upstream fix
still needed in nepal-ttf2utf.**
- **Results, corrected labels, served path (PaddleX), codepoint CER / exact line**
(`~/ocr-deva-ft/preds/final_comparison_v3.json` on k2):
| Set | official | v2 | **v3 (deployed 13:07)** |
|---|---|---|---|
| E1v4 real chunks, held-out editions (n=3000) | 5.13% / 0.47 | 0.69% / 0.91 | 0.57% / 0.93 |
| E7 real chunks containing क्र (n=530) | 5.52% / 0.33 | 4.28% / 0.18 | 0.86% / 0.89 |
| E1b real full lines, independent labels (n=1500) | 7.69% / 0.13 | 2.62% / 0.53 | 2.57% / 0.54 |
| E2 Newar test (n=1139) | 9.11% / 0.06 | 3.53% / 0.35 | 3.54% / 0.35 |
| E3 synthetic citizenship-ne, unseen fonts (n=800) | 6.78% / 0.56 | 0.84% / 0.93 | 0.75% / 0.94 |
| E3 synthetic office-ne, unseen fonts (n=300) | 7.77% / 0.54 | 0.62% / 0.94 | 0.92% / 0.92 |
| E3 synthetic citizenship-en (n=300) | 1.94% / 0.83 | 0.53% / 0.96 | 0.64% / 0.95 |
| E4 English, unseen Noto Sans (n=120) | 2.57% / 0.62 | 1.58% / 0.73 | 1.64% / 0.72 |
| E4 mixed Nepali/English (n=120) | 6.78% / 0.36 | 2.88% / 0.68 | 2.67% / 0.70 |
| E6 NVIDIA English, held-out pages (n=419) | 10.6% / 0.71 | 2.99% / 0.84 | 2.84% / 0.84 |
| E5 handwriting probe (n=450) | 64.7% | 60.5% | 61.0% (unusable) |
v2 learned the converter's क्र->व्र error (402 of 530 E7 predictions contained व्र); v3,
a 6,198-step continuation on the corrected `chunks-v4`, removes it (2 of 530, same as the
official model). Deployed as `~/ampixa-demo/models/ocr/space/models/devanagari-ft-v3`;
v2 and the official model remain on disk for rollback. E3 is synthetic from the same
generator as training (different seed, held-out fonts) and overstates real-certificate
accuracy; no real citizenship certificate has been tested. Real labels are PDF-text silver,
not human-reviewed. Page OCR at 144 dpi garbles small text; at 288 dpi it reads cleanly.
- **Next**: real citizenship samples (with consent) for a true eval; handwriting data (none
licensed yet); upstream the `j|m` fix into nepal-ttf2utf and rebuild the other
Gorkhapatra-derived packs; the demo's "Rows LTR" reading order interleaves newspaper columns.
### Page pipeline on real government letters (2026-09-24, smoke test)
Two public notices from ocr.gov.np (a scanned Lalitpur District Court letter tilted ~2°, and a
Department of Industry letter), rendered at 200 dpi; 43 printed lines transcribed by hand
(`tools/deva_rec_torch/page_probe/`: GT, probe scripts, results; PDFs on k2 `~/ocr-deva-ft/layout-probe/`; not a frozen pack, not claim-grade). Same detector config and v3 recognizer as
the k2 worker; micro CER via `tools/eval_metrics.py`:
| Crop of each detected line | cp CER | grapheme CER | exact |
|---|---|---|---|
| worker today: axis-aligned bbox of the quad + 12 px | 21.4% | 31.7% | 0.40 |
| rotated min-area quad (PaddleX `CropByPolys`), no pad | 3.6% | 5.5% | 0.49 |
| rotated quad + 6 px | **2.9%** | **4.2%** | **0.65** |
| rotated quad + 12 px | 8.1% | 13.3% | 0.42 |
- **The worker's crop, not the recognizer, is the dominant error on real letters.**
`_box_from_polygon` flattens the detector quad to an upright box; on a 2° tilt a full-width
line spans ~40 px vertically, so the crop swallows neighbouring lines (court-letter body
lines read at confidence 0.24-0.57). Upstream PaddleOCR uses rotated crops; our port dropped
them. Line-level evals (E1-E7) cannot see this. Not yet deployed: needs a page-level eval.
- **PP-DocLayoutV3 on the same pages**: QR codes found (image, 0.95/0.96); emblems found
(header_image); **0 of 2 stamps labelled `seal`** (court seal -> header_image 0.65; the
Industry stamp printed over the title not detected at all); signatures merged into the
printed-name `text` block (0.52); no class for handwritten fill-ins; several printed lines
(subject, dates, reference numbers) not boxed at the default threshold. Use layout to mask
non-text regions, never to decide what gets OCR'd.
- Signature/stamp crops produce junk text at recognizer confidence 0.35-0.71; damaged but
real lines (tick mark, strike-through) score 0.74-0.82, so confidence can flag but not drop.
- `train.py` now supports `--guard-dev` (print/synthetic/English dev sets may not regress more
than `--guard-tol` over step 0); the plain dev mean let a 5-point handwriting gain hide a
1-point print loss. `run_train_v4.sh` on k2 uses it.
## Paper-rewrite verification pass (2026-07-19): corrections to this document
While rewriting `paper/DRAFT.md` to cover the language pack, every number was
re-verified against on-disk artifacts. Four values recorded earlier in this
file do NOT match any surviving local artifact and must not be cited:
- **Newar "both-sides-normalized cp CER 0.0441"** — no local artifact; the
omac rescore output was never mirrored. The surviving verified score is
**0.04655** cp / 0.09569 grapheme / 0.2450 exact (artifact-bearing labels,
`outputs/newar-ft-v1-verify-omac-2026-07-07/metrics.json`, n=1,139).
- **Ol Chiki "public headline cp CER 0.0961 / n=358"** — no local score
artifact exists for the 358-row public-issue subset. Locally verified:
v2 on the 772-line v3 pack scores **0.08264** cp / 0.2034 exact
(`outputs/olchiki-v2-run-2026-07-07/score-real-print-v3pack/metrics.json`);
the documented public-issue historical diagnostic remains 0.1018 (n=366)
per `docs/DATA_LICENSING.md`. The v3 358-row candidate was never scored.
- **"299,611 per-language-attributed silver lines"** — refers to the lost
pre-crash batches. The rebuilt batches on disk total **306,971**
(122,882 + 184,089, `outputs/silver-lines-rebuild-b{1,2}-2026-07-05.log`).
- **Ol Chiki v3 FT payload "5,330 entries"** — the payload's own
assembly manifest says **5,307** total images (242 real train + 5,000
synth + 65 real dev).
Also noted: the 8-language Devanagari scorecard (grapheme CER 0.075–0.129,
2026-06-18) lost its backing artifacts in the crash; it is text-only history
now and the paper deliberately does not report it.
## Machine-death recovery (read this first, 2026-07-05)
The Mac died 2026-07-04; the repo now lives at
`/Volumes/TRANSCEND/recovery-backup-20260704-172213/github/llm/ocr-tech`
(symlinked from the old `/Users/cdjk/github/llm/ocr-tech` path). **`outputs/`
and `runs/` are lost**; code, `.git`, docs, `spaces/` model exports
(sha256-verified, champion included), and most of `data/` survived. Every
`outputs/…`/`runs/…` path below therefore refers to a lost artifact unless
`docs/RECOVERY_LEDGER.md` maps it to a rebuilt one — read that ledger for the
full survived/partial/lost inventory, recovery routes, and the rebuild policy
(rebuilds go to new dated dirs). Batch-1 re-harvest launched 2026-07-05.
Blocked on user credentials: AWS SSO (training checkpoints, newar-ft-v1
weights) and HF auth (Space cross-check).
## Overnight lane results (2026-07-16/17)
These lanes delivered mechanically filtered silver substrate, not claim-grade
training or evaluation data. The counts below establish file and archive
parity; they do not replace native label review, edition-isolated leakage
checks, or a real-domain score.
The Sikkim pack at `outputs/sikkim-silver-packs-2026-07-16/` contains **9,743
Sunuwar/Jenticha clean real-page silver lines** and **3,699 Kirat Rai clean
real-page silver lines**. In each case the `summary.json` clean-line count,
on-disk `labels.txt` line count, and non-AppleDouble `.tgz` image-member count
are identical. Sampled Sunuwar labels contain Jenticha characters in
U+11BC0..U+11BFF, and sampled Kirat Rai labels contain Kirat Rai characters in
U+16D40..U+16D7F, without non-ASCII characters outside the expected target
block. Each summary records the eval manifests it checked. None of those eval
editions occurs in the discovered source list, so the recorded
`excluded_editions: []` is consistent rather than evidence of an unperformed
check.
Lepcha takes a different route. The diagnosis at
`outputs/lepcha-zero-yield-diagnosis-2026-07-16/REPORT.md` shows that the new
edition bodies are raster content and that the live PDF text is ordinary
header/punctuation material, so there is no converter alias or byte-map fix
that can recover the body. The converter lane is therefore closed and Lepcha
moves to a synth-first recognizer route, retaining the **132-crop** real pack
recorded in `data/lepcha-real-eval-pack/README.json` as a real evaluation
anchor. That pack is still machine-labelled and human verification remains
pending, so it is diagnostic rather than claim-grade.
The Gorkhapatra full-depth backfill now covers every month from March 2024
through July 2025. The figures come from each month's
`gorkhapatra-language-pages.json`; `silver-kept` comes from the corresponding
silver summary and was independently matched to the `.tgz` member count.
| Month | Items | Aligned | Hits | Silver-kept |
| --- | ---: | ---: | ---: | ---: |
| 2024-03 | 1,776 | 62 | 120 | 27,668 |
| 2024-04 | 1,731 | 61 | 129 | 21,429 |
| 2024-05 | 1,666 | 62 | 128 | 24,612 |
| 2024-06 | 1,601 | 60 | 118 | 21,924 |
| 2024-07 | 1,538 | 61 | 120 | 28,301 |
| 2024-08 | 1,473 | 60 | 120 | 24,534 |
| 2024-09 | 1,408 | 60 | 122 | 32,804 |
| 2024-10 | 1,343 | 53 | 103 | 25,055 |
| 2024-11 | 1,278 | 52 | 106 | 30,788 |
| 2024-12 | 1,213 | 61 | 121 | 39,115 |
| 2025-01 | 1,148 | 62 | 124 | 40,526 |
| 2025-02 | 1,083 | 56 | 112 | 35,391 |
| 2025-03 | 1,018 | 59 | 117 | 35,482 |
| 2025-04 | 953 | 57 | 113 | 34,815 |
| 2025-05 | 888 | 59 | 118 | 36,878 |
| 2025-06 | 823 | 60 | 120 | 37,836 |
| 2025-07 | 798 | 32 | 62 | 18,428 |
| **Total** | **21,738** | **977** | **1,953** | **515,586** |
All monthly audit warning arrays are empty, all archive counts match their
silver summaries, and no source PDF remains under any monthly output. This is
a delivered backfill substrate, not a statement that its page-attributed
silver labels are language-verified.
Detector remediation remains active. The scale-normalized tile build is
staged/running on `omac`; the local evidence at
`outputs/det-remediation-2026-07-16/REPORT.md` ends at the completed code,
config, and handoff. The GPU gate remains closed until the returned build
summary and tile manifest pass the recorded re-audit checklist.
## Current Goal
The end goal is an OCR system covering the multiple languages of the Nepal pack,
built out of Paddle, **with Limbu and Newar (Nepal Bhasa) as the first two focus
languages** (user steer, 2026-07-04). The system should accept real book/page images
or PDFs for any language in the pack and output editable Unicode text with page,
line, crop, confidence, and provenance evidence preserved. Limbu stays the proven
template — it has the strongest recognizer and the deepest harvest. Newar is the
deliberate second: it has its own native script (Prachalit), real cultural demand,
and is currently the weakest trained recognizer, so it exercises every part of the
method that Limbu already proved. Every stage that was built Limbu-specific must
generalize: harvest, silver-label extraction, recognizer training, script routing,
page/book OCR, and eval. The product is the full pipeline: page
detection, script-aware routing to per-script recognizers, target-region and
furniture filtering, reading order, assembly, Unicode/JSON/Markdown export, and
review artifacts for hard lines. YOLO line detection stays a diagnostic/probe;
Paddle-first is the production story.
## The Language Pack
Two inventories exist and they are not the same thing:
- The registry `languages/nepal-v1.json` tracks 18 target languages (17 Nepal + English)
with scripts, fonts, dictionaries, and corpus status.
- The Gorkhapatra Naya Nepal archive publishes far more: the 2026-07-04 harvest found
**50 normalized languages (53 raw labels) in 400 articles over ~7 months** (see the
harvest section). Labels include all registry languages plus Chamling Rai, Bahing,
Jirel, Chepang, Baram, Tajpuriya, Kumal, Doteli, Bajhangi, Dandeldhureli, Hyolmo,
Mugal, Marwadi, Urdu, Sanskrit, Awadhi, Kisan, Achhami, Bote, Hayu, Meche, Lhomi,
Uraon, Bhote, Danuwar, Majhi, Magahi, Bajjika, Khash Jumli, and more.
Label variants must be normalized before counting (थारु/थारू, याक्ख/याक्खा,
ह्योलमो/ह्योल्मो, लिम्बु/लिम्बू, चाम्लिङ vs चाम्लिङ राई) —
`tools/build_gorkhapatra_language_pack_inventory.py` owns that normalization and the
label→registry mapping.
### Newari / Nepal Bhasa writing-system boundary (2026-07-15)
Newari support is explicitly four-profile, not a single generic "Newa" bucket:
| Profile | Source encoding | OCR transcription | Current capability |
| --- | --- | --- | --- |
| Newar / Prachalit | Unicode Newa, U+11400..U+1147F | Native Newa | Route, training candidate, diagnostic checkpoint, and synthetic runtime smoke registered; no claim-grade real-domain CER |
| Bhujimol / Bhujinmol | No separate Unicode script; configured as a Newa-codepoint style profile | Newa plus mandatory source-style metadata | Route, two acquired manuscript groups, and reviewer reference registered; zero qualified decisions, admitted labels, model, or CER |
| Ranjana | Unencoded in Unicode 17.0 | Newa transcription plus mandatory source image and Ranjana metadata | Image route, two acquired manuscript groups, and reference font registered; zero qualified decisions, admitted labels, model, or CER |
| Devanagari | Unicode Devanagari, U+0900..U+097F plus U+A8E0..U+A8FF | Native Devanagari | Route, diagnostic checkpoint/runtime smoke, and one permission-covered official Newa-edition acquisition candidate registered; source script and real-line quality remain unaudited |
Unicode's current [Newa chapter](https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-13/)
and [code chart](https://www.unicode.org/charts/PDF/U11400.pdf) define the
encoded Newa route. WG2 document
[N4555](https://www.unicode.org/wg2/docs/n4555.pdf) describes Bhujimol/Bhujinmol
as graphically distinct and discusses possible plain-text unification with Newa;
that does not justify visual-model reuse. Ranjana is still represented by a
[preliminary encoding proposal](https://www.unicode.org/L2/L2023/23028-ranjana.pdf),
so this project does not invent a Ranjana Unicode range.
The canonical runtime profiles live in `src/ocrtech/newari_scripts.py`. The
registry and book-OCR pipeline declare four separate recognizer references and
preserve the source encoding/output policy in run metadata. The frozen audit at
`outputs/newari-four-writing-system-support-audit-2026-07-15-r1/` passes all four
structural bindings while reporting `model_ready=0/4` and
`four_script_ocr_claim_ready=false`. "Supported" here means explicit input,
profile, routing, and output semantics; each profile still needs at least 200
license-clean, pixel-label-audited, leakage-controlled real lines and a frozen
real-domain evaluation before delivery or quality claims.
The r2 license-first discovery audit records three Bhujimol and three Ranjana
source candidates without admitting any pixels. The most promising Bhujimol
item is [Asa Archives item 1147](https://archive.org/details/1147_20250817):
its Internet Archive metadata has an uploader-applied CC0 URL, 63 scanned
pages, and 325 automatically detected hOCR line boxes. Those boxes are
acquisition-scale metadata, not labels. Uploader authority, institutional
accession, exact-file SHA-256, qualified Bhujimol review, pixel faithfulness,
and transcription mapping all remain open. For Ranjana, the Smithsonian's
[Karandavyuha Sutra record](https://asia-archive.si.edu/object/S1991.161/)
specifically marks image `FS-6314_06` CC0 and its IIIF metadata identifies
Ranjana, but the [manifest](https://ids.si.edu/ids/manifest/FS-6314_06) contains
only one canvas; the 77 physical folios are not 77 downloadable pages. British
Library [EAP790/21/121](https://searcharchives.bl.uk/catalog/040-003461535)
identifies 12 Bhujimol TIFFs but restricts access to research use, and
[EAP1296/2/217](https://eap.bl.uk/archive-file/EAP1296-2-217) exposes 13
Ranjana images under noncommercial terms. The [Nepalbhasa Manuscripts in
UK](https://www.nepalbhasamanuscript.org/) archive states that copies and
transcriptions are openly accessible but supplies no explicit reuse grant.
The frozen audit is `outputs/newari-source-candidate-audit-2026-07-15-r2/`;
its pass means the catalogue is fail-closed, not that any source, model, or OCR
claim is ready.
The r3 provenance crosswalk matches item 1147 to the uploader's Asa Kuthi
catalog spreadsheet at worksheet `aiStuidoOutput`, row 1170: DPN 1147,
*Bhāsvati*, NRN `ज्योति - २७१`, RN 2919, Sanskrit/Nepalbhasa, N.S. 528, and
33.3 × 4.1 cm. The spreadsheet SHA-256 is
`8484dbe95358b83d7d78d5d1752fafac5feeecfc007ccf5d1fe8b6ea3ce9bffd`.
This disambiguates a separate row whose RN, rather than DPN, is 1147. It does
not attest Bhujimol because the matched catalog script field is blank, and it
does not prove that the Internet Archive uploader was authorized to apply
CC0. The exact authorization and reuse questions are staged in
`docs/NEWARI_ASA_1147_RIGHTS_REQUEST.md` but have not been sent. The r3 frozen
audit therefore remains `PASS_CATALOG_INVENTORIED_ZERO_DATA_ADMITTED` with no
review manifest ready.
The r4 alternative-source sweep found a materially stronger Bhujimol route in
Wellcome Collection. The official [manuscript
record](https://wellcomecollection.org/works/jnsn9gjd) identifies `MS Indic
epsilon 1`, and Wellcome's [editorial account of the
object](https://wellcomecollection.org/stories/stories-of-asian-palm-leaf-manuscripts)
explicitly says it uses Bhujimol. Its [complete IIIF
manifest](https://iiif.wellcomecollection.org/presentation/v2/b14901560) has
217 unique canvases/image services, SHA-256
`596296750114948b39da2252b5a7b7c0738287dadcf7aed32939bfc29980c6d4`, and
a Public Domain Mark statement that expressly permits commercial copying,
modification, and distribution. Wellcome describes 208 leaves with only six
illustrated. Five 2000-pixel-wide probes at canvases 1, 50, 100, 150, and 210
decoded successfully; canvas 1 was a cover/label view, while the other four
showed dense multi-block palm-leaf text. This is not a script-expert or label
audit. Because the current host is an 18GB Mac, the repository freezes only a
metadata acquisition manifest and an omac handoff; it does not bulk-download
the 217 canvases. The whole manuscript must remain on one side of any split,
so a separate manuscript is still required for leakage-proof training.
The r5 Ranjana follow-up resolved Wellcome `MS Indic delta 33` without
inflating selected images into a complete manuscript. The official [physical
record](https://wellcomecollection.org/works/bp979bf3) is closed-store only and
has no digital item; the predictable [IIIF presentation
URL](https://iiif.wellcomecollection.org/presentation/v2/b14901468) returns
404. Four exact digital-image records identify Ranjana and carry CC BY 4.0,
but a prediction-free 1200-pixel probe shows one two-leaf overview, one
partial-leaf environmental photograph, and two close-up miniature crops whose
content overlaps the overview. The records therefore provide zero independent
complete pages, not four. Their titles say AD 1653 while one description says
transcribed in 1635; both values are retained. The frozen catalogues at
`outputs/newari-wellcome-ranjana-source-catalog-2026-07-15-r5/` and
`outputs/newari-source-candidate-audit-2026-07-15-r5/` admit zero pixels,
labels, rows, models, or metrics. A different complete Ranjana manuscript is
still required at the r5 boundary.
The r6 BDRC sweep resolves that complete-source discovery gap without claiming
a dataset. BDRC physical instance `MW1NPL132`, *Ārya nāmasaṁgīti*, is marked
only `ScriptRanj` with print method Manuscript; linked scan `W1NPL132` is
`AccessOpen` and `StatusReleased`, and image group `I1NPL132` exposes 21 unique
IIIF canvases/services. The manifest SHA-256 is
`a3c49a3fd68f30f0b4610a2f0c3d6432fd189caae5c371a0d482ae97fb02d11d`.
BDRC's official access policy classifies scanned original manuscripts as
public domain and permits sharing/adaptation for any purpose. The manifest
itself has no license field; the work-record CC0 marker applies to metadata and
is explicitly not treated as a pixel license. Five bounded 1200-pixel probes
decoded: canvases 1 and 21 are cover/note material, while 5, 10, and 15 show
dense visually Ranjana-like running text. Those observations are not qualified
script decisions or a line-count audit. The frozen catalog at
`outputs/newari-bdrc-ranjana-source-catalog-2026-07-15-r6/` records 21 exact
remote resources, zero admitted pixels, zero audited lines, zero labels, and a
prediction-free `omac` handoff. All pages remain one manuscript group; a
distinct manuscript is still required for a manuscript-disjoint train/eval
split.
The r7 remote-acquisition round executed that BDRC handoff on
`omacs-MacBook-Pro` (68,719,476,736 memory bytes), without OCR, training, or
scoring. All 21 full-resolution JPEGs were fetched, SHA-256 recorded, and
decoded at their exact manifest dimensions (all width 1200, heights
1676–1769); the capture totals 6,680,141 bytes and has 21 unique byte hashes.
The source pixels remain on `omac`; the orchestration Mac mirrors only the
metadata, hash list, dimension list, and rights/manifest evidence. A dynamic
WordPress detail was caught rather than waived: BDRC's policy HTML randomizes
CSS class-number suffixes across hosts, so raw page hashes differ while a
narrow normalization of only those suffixes yields the same SHA-256
`99977ba3f0cb00da828a3833e6649a39c89c9f8e928a7127eeac1c35cd48d60e`
and the required rights sentences remain present. The local adversarial audit
at `outputs/newari-bdrc-ranjana-remote-acquisition-audit-2026-07-15-r7/`
verifies the returned metadata bindings and creates a 21-row blank review
worksheet. It explicitly cannot recompute the remote image hashes without the
pixels. Qualified page decisions remain 0/21, audited lines remain zero, and
no label, admission, model, or metric exists.
The r9/r10 remote-acquisition round executed the independent Wellcome
Bhujimol handoff on the same approved host, again without OCR, training, or
scoring. All 217 requested review JPEGs were fetched, SHA-256 recorded, and
decoded; the capture totals 58,471,370 bytes and has 217 unique byte hashes.
Wellcome's `full/2000,/...` IIIF derivative decoded at width 2000 for 214
canvases and width 2001 for canvases 214, 216, and 217. The runner failed
closed on the first 2001-pixel result, then the replay narrowed the accepted
contract to exactly those observed widths and froze every exact dimension.
The source pixels remain on `omac`; the local mirror is metadata-only. The
adversarial audit at
`outputs/newari-wellcome-bhujimol-remote-acquisition-audit-2026-07-15-r10/`
verifies the returned bindings and creates a 217-row blank qualified-review
worksheet. Qualified page decisions remain 0/217, audited lines and labels
remain zero, and the entire manuscript is still one indivisible split group.
A second Bhujimol manuscript is required for manuscript-disjoint evidence.
The r12/r13 second-source round catalogs a distinct manuscript group for both
Bhujimol and Ranjana without admitting data. The exact Wikimedia Commons JPEG
of Cambridge `MS Add.2137` is photographer-authored and CC BY-SA 4.0. It
contains four folio photographs; the first three selected regions are visually
crosswalked to Cambridge canvases 48–50 (`28r`, `28v`, `29r`). Cambridge's
official manifest independently calls all three hands
hooked Nepālākṣarā (Bhujimol), but the complete 228-canvas institutional scan
is CC BY-NC 4.0 and remains pixel-excluded. The single licensed composite is a
separate manuscript group from Wellcome `MS Indic epsilon 1`, but cannot by
itself establish the 200-line gate and still needs exact-original SHA-256 and
qualified line review. BDRC `MW1NPL23` is separate from `MW1NPL132`; its
open/released scan has 25 unique IIIF canvases under BDRC's scanned-manuscript
public-domain policy. Its physical record assigns both `ScriptNewa` and
`ScriptRanj` without page-level attribution, so every text-bearing canvas must
be classified Ranjana/Newa/mixed/other by a qualified reviewer. The frozen r12
catalog stores no source pixels and reports zero qualified decisions, audited
lines, labels, OCR, training, or scoring. The r13 inventory audit passes with
11 candidates (5 Bhujimol, 6 Ranjana), 3 acquisition-ready manifests, 2
completed remote acquisitions, and zero admitted sources. Its tests reject
license, institutional-script, manifest-license, duplicate-service,
probe-hash, and line/admission drift. Evidence:
`outputs/newari-second-manuscript-source-catalog-2026-07-15-r12/` and
`outputs/newari-source-candidate-audit-2026-07-15-r13/`.
The r14/r15 acquisition round executed the second-source handoff on
`omacs-MacBook-Pro` (68,719,476,736 memory bytes) without OCR, training, or
scoring. The exact Commons original and all 25 BDRC canvases decoded at their
frozen dimensions; all 26 byte hashes are unique and total 12,738,550 bytes
(3,435,984 Bhujimol; 9,302,566 Ranjana). The Commons SHA-256 is
`938013af2c4c16ac84164a41f903cf34d131663b7a069e58df779bacfcd4258d` and
its upstream SHA-1 also matches. Separate profile checksum manifests bind one
Bhujimol image and 25 Ranjana images, while the orchestration host mirrors
metadata only. The local audit creates 28 blank qualified-review units: three
Bhujimol panels (`28r`, `28v`, `29r`) and 25 BDRC canvases. All decisions,
audited lines, labels, admissions, and claims remain zero. Evidence:
`outputs/newari-second-source-prediction-free-review-2026-07-15-r14/` and
`outputs/newari-second-source-remote-acquisition-audit-2026-07-15-r15/`.
The refreshed r16 central inventory now reports 11 candidates, 7 explicit
commercial-permission statements, 3 acquisition-ready manifests, 4 remotely
fetched-and-hash-recorded sources, and still zero admitted sources or support
claims (`outputs/newari-source-candidate-audit-2026-07-15-r16/`).
The r17 review-readiness round joins the four acquired manuscript groups
without copying pixels or inventing decisions. It binds 264 unique remote
source images to 266 blank review units: Bhujimol has 218 images/220 units
across Wellcome `b14901560` and Cambridge `MS Add.2137`; Ranjana has 46
images/units across BDRC `MW1NPL132` and mixed-script `MW1NPL23`. Both
profiles meet only the two-manuscript-group precondition. Qualified decisions,
estimated lines, audited lines, labels, admissions, OCR, training, scoring,
and claims remain zero, so the >=200-line gate is explicitly false for both.
Evidence:
`outputs/newari-acquired-source-review-readiness-audit-2026-07-15-r17/` and
`docs/handoffs/HANDOFF-newari-acquired-source-review.md`.
The r43-r45 correction separates manuscript identity from exact transcription.
The Wellcome story identifies `b14901560` / MS Indic epsilon 1 as the
*Aṣṭasāhasrikā Prajñāpāramitā Sūtra*, but its 217 IIIF canvas labels are all
`-`; no folio or line transcription accompanies the images. A GRETIL e-text
exists for the Vaidya 1960 edition, but its page says it is for reference only,
defers terms to the source file, and supplies no crosswalk to this manuscript.
It is therefore neither rights-cleared prefill nor ground truth. BDRC's frozen
physical record labels `MW1NPL99` “Unidentified” and supplies no e-text.
Cambridge's official TEI says the *Nāradasmṛti* begins at verse 5.8cd on `28r`,
but the manuscript interleaves that Sanskrit root with a Newari
commentary-cum-translation; the TEI explicitly states that the Newari legal
text and translation are unpublished and gives only short catalog excerpts.
Accordingly, the r45 fail-closed audit reports three routes checked, zero exact
line-aligned transcriptions, zero admissible prefills, and zero labels.
The same pass found that the frozen r15 worksheet named `panel-28r`,
`panel-28v`, and `panel-29r` but pointed all three rows at the same uncropped
2,332 x 1,800 four-folio Commons composite. The row identities and licensed
source hash remain valid, so no frozen review data was rewritten. The r43
profile dispatch and r44 combined UI instead render fixed source-bounded
regions `[0,0,2332,400]`, `[0,430,2332,420]`, and
`[0,890,2332,410]`, crosschecked against institutional canvases 48–50. They
fail closed to the full original if dimensions or bounds drift, preserve the
full-image link and SHA-256, contain no OCR text or labels, and keep the same
339-row assignment scope. The complete selected regions were visually checked
against the CC BY-SA Commons original. Tests cover rendering metadata,
source-dimension refusal, immutable review scopes, output checksums, and the
companion-text admission gate. Artifacts:
`outputs/newari-qualified-page-review-dispatch-2026-07-15-r43/`,
`outputs/newari-qualified-page-review-ui-2026-07-15-r44/`, and
`outputs/newari-bhujimol-companion-text-audit-2026-07-15-r45/`.
The r41 Devanagari-Newari source round found one exact official State Portal
record titled `SIKKIM HERALD-NEWA EDITION`, dated 13 April 2022, under the
permission-covered `sikkim.gov.in/uploads/SikkimHerald/` prefix. The existing
Sikkim source policy remains exact at SHA-256
`2247033ac1ce7de8d99cad22feb702f9fe48a11341a2fb2d20c1d6d48b06717a`.
An eight-byte HTTP Range probe confirms a PDF response with a total size of
3,975,535 bytes; no PDF body or page image was downloaded locally. The newer
IPR portal currently returns zero Newar records and was used only as negative
discovery evidence. The administration URL currently exposes only the official
Government of Sikkim site shell, so no substantive publication claim is derived
from it. The target writing-system route is Devanagari, but the edition's actual
source script, fonts, and third-party content remain explicitly unconfirmed
until all pages are rendered and reviewed on an approved remote host. The
catalog therefore admits zero pixels, decisions, labels, real lines, models,
or metrics. Evidence:
`outputs/newari-devanagari-sikkim-source-catalog-2026-07-15-r41/`.
Two tempting Commons catalog leads were rejected for production use in the same
round. `AN-PM-12.pdf` and `AN-PM-09-Dvadasa Tirtha Yagu Bakham.pdf` are described
on Commons as old Newari material in Devanagari with Public Domain Mark labels,
but the British Library EAP record for AN-PM-09 states that access is for
research purposes only. A downstream Commons rights label is not treated as
authority to erase that source restriction. Recent uploader-applied Internet
Archive CC0 metadata was likewise kept out when uploader authority could not be
established.
Per-language recognizer state (grapheme CER, honest micro metric, from the re-grounded
2026-06-18 scorecard in `docs/RECOGNIZER_GOAL_PROGRESS.md`):
- **Limbu/Sirijonga 0.041 real-print** (champion:
`runs/limbu-real-heavy-gork-cdc-replay-v1-lr1e6-8ep-normedit-g6-l4-ppocrv5-2026-06-23`,
cp CER 0.011 on the frozen 255-line Gorkhapatra silver eval).
- **8 Devanagari languages 0.075–0.129 real-print** (Majhi, Magar, Sunuwar-Deva, Magahi,
Bhojpuri, Thami, Wambule, Darai) from one shared Devanagari recognizer.
- **Sunuwar (Jenticha) 0.121 real-crop** after the same-font retrain (0.342 synth-only).
- **Tirhuta 0.343 and Newa 0.613 synthetic-only** — weak, transliteration-fed, honest
fails; Newa needs real printed-register Nepal Lipi, not more pixels.
- **Kirat Rai (70 real crops) and Lepcha (132 real crops)** blocked on legacy-font
converter coverage outside their eval editions. **Gurung Khema now has official
real-print source material:** two 2022 State Portal Herald editions yield 749
mechanically convertible line candidates, but zero rows are admitted pending
independent provenance review, native pixel-label review, and split/leakage gates.
The near-term pack claim should therefore be script-family shaped: one Sirijonga
recognizer, one shared Devanagari recognizer covering the many Devanagari-written
languages, plus per-script experts (Jenticha, Kirat Rai, Lepcha, Prachalit/Newa,
Tirhuta, Gurung Khema, Ol Chiki, Tibetan-script) admitted one by one as real eval
crops exist.
**Focus order (2026-07-04 steer): Limbu first, Newar second.** The 2026-07-15
Newari steer requires Newar/Prachalit, Bhujimol, Ranjana, and Devanagari as
separate writing-system profiles. Newar state: the
Prachalit recognizer is the weakest in the pack (grapheme CER 0.613 on its own
synthetic eval; transliteration-fed training failed; real printed-register Nepal
Lipi is the known lever, and only 2 Unicode Newa fonts exist). The Devanagari-Newar
route moved fast on 2026-07-04: 6,800 real silver lines harvested from 20
Gorkhapatra pages; `newar` added to the registry (Devanagari primary + Prachalit
secondary; audit passes); a frozen by-date pack built at
`outputs/newar-devanagari-silver-pack-v1/` (train 4,096 / dev 1,528 / test 1,139
lines; test dates 2082-07-25 and 2083-01-05; zero date/text overlap independently
verified; pack sha256
`1fa7b23b6c18701857aea686fd55f3075e104d224730896f6ee35ba25e7ccb85`); and the
**first real-print Devanagari-Newar baseline** was scored locally (CPU export
route, `tools/score_recognizer_pack.py`, per `docs/LOCAL_RECOGNIZER_SCORING.md`):
the shared deva-v2 recognizer, never trained on Newar, reads the frozen test at
**codepoint CER 0.0622, grapheme CER 0.1312, exact-line 0.158** (n=1,139 silver
lines — machine-converted labels, diagnostic, not claim-grade). The same-day
warm-start fine-tune (**newar-ft-v1**: deva-v2 base, LR 2e-5 cosine, 60 epochs,
batch 64, selection on the all-batch2 dev split, best_epoch 46; G6 L4 instance
`ashish-experiments`, ~68 min; artifacts + train log at
`outputs/aws-newar-deva-ft-v1/`) scores the same frozen test at **codepoint CER
0.0465, grapheme CER 0.0955, exact-line 0.245** — a 25/27% relative CER drop and
+55% exact-line over the deva-v2 baseline, scored through the identical local
CPU path. **Newar's grapheme CER now sits inside deva-v2's own-language range
(0.075–0.129): the second focus language reached Devanagari-pack parity from
zero in one day.** Same claim boundary: silver machine-converted labels,
diagnostic, not claim-grade until native review. Next Newar lever: native
Prachalit (real printed-register data), and page-level Newar eval for the
region-filtering track.
## Source Rule
For Gorkhapatra/Naya Nepal language pages, the Naya Nepal page is discovery metadata
only. Use `https://gorkhapatraonline.com/publications/naya-nepal` to find the language,
date, and article, but do not use the blog/card image for OCR, training, or evaluation
when the full paper exists. The authoritative source is the matching Gorkhapatra epaper
PDF from the date-range endpoint. The correct pair is the rendered PDF page image plus
the text layer extracted from that same PDF page, with PDF URL/hash, page number,
rendered image hash, extraction command/version, font evidence, and converter/map hash
recorded. For Limbu/Sirijonga the text layer is Namdhinggo-style legacy converted with
`src/ocrtech/limbu_legacy.py` + `data/mappings/limbu-legacy/Limbu.map`. The same
pattern — classify the text layer, then build an explicit converter only when the font
requires it — replicates to every other language. Manual transcription is the fallback
for no usable text layer, not the default.
## Getting More Gorkhapatra Pages (the specified harvest recipe)
All numbers below were verified live on 2026-07-04.
**Archive size.** The Naya Nepal publication archive paginates at
`https://gorkhapatraonline.com/publications/naya-nepal?page=N` with 5 articles per
page and runs to page ~352 before drying up: **~1,750 language-page articles total**
(about 2 per day going back years). The 60-page crawl already yielded 300 items across
53 labels spanning BS 2082-10-18 → 2083-03-19; deeper pages reach correspondingly
older dates.
**Epaper endpoint behavior (the critical constraint).** The date-range endpoint
`https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=A&pdfdaterange2=B&slug=gorkhapatra`
**caps results at 30 PDFs per query and ignores `page=N` pagination**. A wide window
silently drops most dates — the 2026-07-04 six-window discovery pass aligned only
111/300 items for exactly this reason, with unmatched dates interleaved among matched
ones. **Seven-day windows return complete daily coverage (7/7 verified).** Always
harvest with weekly windows.
**The recipe** (one command; add one `--epaper-url` per week to cover the publication
date span being harvested):
```bash
env PYTHONPATH=src uv run --python 3.11 --with pymupdf \
ocrtech audit-gorkhapatra-language-pages \
--out outputs/gorkhapatra-language-pages-full-<date> \
--publication-url https://gorkhapatraonline.com/publications/naya-nepal \
--publication-pages 80 \
--epaper-url 'https://epaper.gorkhapatraonline.com/epaper-list?pdfdaterange1=2026-06-28&pdfdaterange2=2026-07-04&slug=gorkhapatra' \
--epaper-url '...one per week, going back...' \
--download-pdfs \
--render-pages \
--timeout-seconds 180
```
The command aligns every publication card (all languages at once — alignment is
language-agnostic) to its same-date PDF, downloads each PDF once (cached by URL),
searches pages for the language-page markers (`भाषा पृष्ठ` and its legacy-encoding
form `efiff k[i7`), and renders hit pages at 2×. The 2026-07-04 run with 23 weekly
windows (2026-01-25 → 2026-07-04, publication-pages 80) completed with
**400 publication items, 195 aligned dates, 394 rendered language-page hits, zero
warnings** at `outputs/gorkhapatra-language-pages-full-2026-07-04/`. Batch 2
(2025-08-03 → 2026-01-24, 25 weekly windows, publication-pages 160) added 398 more
items: **798 items, 294 aligned dates, 589 further page hits**, at
`outputs/gorkhapatra-language-pages-full-batch2-2026-07-04/`. The merged normalized
inventory (`tools/build_gorkhapatra_language_pack_inventory.py`) shows **56
languages, 983 rendered language-page hits, most languages with ~20–27 pages
spanning 14 months**, at `outputs/language-pack-inventory-merged-2026-07-04/`. For
comparison, the entire previous state of the project had 8 usable pages for one
language. Batches cost ~2–2.5 GB each; the remaining archive depth (back to at least
March 2024 — verified the epaper endpoint serves complete weeks there) is another
~2× and stays month-batchable with the same recipe.
**Scaling to the full archive.** Full depth is ~700+ unique dates ≈ 35 GB of PDFs.
Do not do it in one run: batch by month (4–5 weekly windows + the matching
publication-page depth), run `extract-gorkhapatra-language-page-pdf-text` per batch,
then keep rendered pages, extracted text, and hashes; PDFs can be pruned after
extraction because URL+SHA-256 make them re-fetchable. The remote 2 TB volume
(`cdjk@100.110.148.43:/Volumes/TRANSCEND`) is the right home for a full-depth mirror.
**Per-language downstream.** After harvest, run per language:
```bash
env PYTHONPATH=src uv run --python 3.11 --with pymupdf \
ocrtech extract-gorkhapatra-language-page-pdf-text \
<pack-manifest.jsonl> \
--candidate-language <label> \
--target-font <font> \
--out outputs/<language>-pdf-text-<date>
```
then classify each page's text layer (native Unicode / legacy font / Devanagari
transcription / Latin / none) and only then decide converter vs manual transcription.
**Font/text-layer facts already established** (30-page, 15-language extraction at
`outputs/gorkhapatra-multilingual-language-pages-current-2026-06-24-pdf-text/`):
- The body font of non-Limbu language pages is **`nayanepal` — a legacy 8-bit
Preeti-family encoding, not Unicode** (8,845 of 11,791 spans; the text layer reads
`k[i7 ;+of]hs` where the page shows पृष्ठ संयोजक). **The converter already
exists**: `nepal_ttf2utf.convert_devanagari(font="nayanepal")` in the sibling repo
`/Users/cdjk/github/llm/nepal-ttf2utf` (Preeti base map + nayanepal extension
glyphs + NFC, ~97–99% clean on real Gorkhapatra pages), already used by
`tools/build_gorkhapatra_devanagari_real.py` for the 79k-row real Devanagari
corpus. The remaining leverage is wiring it **per language**: run it over each
harvested language page and emit per-language silver line packs with the same
faithfulness gate (clean-Unicode allow-set, `conversion_gap` accounting, leakage
checks) that the Devanagari pack builder already implements. The same sibling repo
also hosts the limbu, kiratrai, lepcha, and sunuwar legacy converters.
- Tibetan-script fonts (`TibetanMachine`, `MonlamUniOuChan5`, `Qomolangma-*`) appear on
Mugal and related pages, but **every such span has empty extracted text (verified
2026-07-04 across all 2026-06-24 samples)** — Tibetan-script pages are visual-only,
like the known Limbu `2082-11-12` date. They need OCR bootstrap or manual
transcription; no converter is possible.
- `Gorkhapatra-*`, `Nagarik-*`, `TimesNewRoman*`, `MyriadPro-*` are masthead/furniture
fonts — useful as non-target filters for target-region selection.
- `NamdhinggoSILL` appears only on Limbu dates (the existing Limbu path).
## What We Built (Limbu template, still true)
The Gorkhapatra alignment path: publication-page discovery, epaper-date matching, PDF
download/rendering, page-marker search, PDF-native span/font extraction, review
sheets, reference templates, and dashboard visualization. The expanded Limbu harvest
has 9 discovered dates, 8 usable PDF-text-layer pages, 2,441 line rows, 2,430 usable
silver labels (`outputs/limbu-gorkhapatra-pdf-backed-silver-2026-07-03-v1/`).
The Limbu recognizer training/eval path: 48,507-row combined pack; real-heavy
fine-tune champion at codepoint CER 0.011336 / grapheme 0.024736 / exact-line 0.729 on
the frozen 255-line silver eval. Synthetic hardening lessons: targeted+Unicode packs
regressed; v3 dictionary warm-start catastrophically failed (~0.98 CER); keep the
real-heavy v2 champion; future gains come from real/book-domain harvesting or a
deliberate rehead, not blind synth rows.
The page/book OCR loop: local + HF Space (`spaces/limbu-ocr/`, deployed private at
`https://huggingface.co/spaces/ampixa/limbu-ocr`) with line OCR, Paddle Page OCR,
Paddle Book OCR, image-line fallback, runtime checks, annotated pages, and crops.
Page-level diagnostics on the 8-page silver pack: Rows LTR + y/conf70 filter scores
page cp CER 0.414330, line F1 0.963009, matched-line cp CER 0.026039, order accuracy
0.993176. Recognition on matched lines is strong; **full-page CER is dominated by page
reconstruction, target-region selection, and furniture filtering** — that is the
pipeline problem to solve, and it is language-independent.
Supervised line-detector infrastructure from CDC PDF geometry (287 pages, 26,160
annotations, mAP50 ≈ 0.921 on PDF-rendered val) — evidence the detector path works,
not a solved product.
A claim-grade eval harness (versioned packs, hashes, provenance, normalization
profiles, cp/grapheme CER, CIs, paired comparisons, leakage manifests) — parked as a
guardrail; do not let it eat work blocks that should go to the pipeline.
## Current Working State
The strongest component is the Sirijonga line recognizer. The weakest component is
end-to-end page/book reconstruction: target-region selection, furniture filtering,
line preservation, and ordering. Of the three Limbu-locked code spots, two were
generalized on 2026-07-04 (Codex worker, tests green): `_script_ratio` in
`src/ocrtech/engines.py` is now driven by the shared `SCRIPT_CODEPOINT_RANGES` table
covering all pack scripts (devanagari, limbu/sirijonga, latin, sunuwar/jenticha,
lepcha, kirat_rai, newa/prachalit, tirhuta, gurung_khema, ol_chiki, tibetan, with
aliases), and the image-line filter in `src/ocrtech/image_lines.py` accepts any
script key from that same table. The third spot fell the same day: the
`src/ocrtech/limbu_runtime.py` preflight now takes `required_script` (default
`limbu`, validated against the shared table; CLI `--required-script`), so a router
card for any pack script can be preflighted. The engine classes themselves (`CandidateEngine`, `PaddleOcrEngine`,
`ScriptSelectCompositeEngine`, `build_image_line_document`, `structure.py` reading
order) are already parameterized — the rest of the generalization is registry-driven
wiring, not a rewrite.
## The Paddle Pipeline (whole pipeline, not just recognizers)
Target architecture, all stages Paddle-first and language-pack-parameterized by the
per-language pipeline config **`configs/nepal-language-pack-pipeline-v1.json`**
(50 languages: label, registry id, scripts, recognizer artifact, dictionary,
converter, text-layer class, harvest counts — generated 2026-07-04 from the
inventory):
1. **Capture/prep** — existing `prepare-limbu-capture` path, already
language-agnostic in behavior; rename/rescope.
2. **Detection** — Paddle text detection as production default; YOLO line detector and
morphology image-line boxes stay as diagnostics/rescue passes.
3. **Script identification & routing** — generalize `_script_ratio` to a
Unicode-block-driven scorer over the pack's script list (Devanagari, Limbu,
Jenticha/Sunuwar, Lepcha, Kirat Rai, Newa, Tirhuta, Gurung Khema, Ol Chiki,
Tibetan, Latin), then route each line to the script's recognizer from the registry.
4. **Recognition** — per-script PP-OCRv5 recognizers (existing runs) with per-script
dictionaries and input-width profiles (Sirijonga width-480 lesson).
5. **Target-region & furniture filtering** — non-oracle region selection using layout
plus font/language signals; furniture fonts and masthead geometry as negative
evidence. This is the main open research item; score against the Rows LTR +
y/conf70 baseline.
6. **Reading order & assembly** — `auto_layout` selection (wide-column RTL vs rows)
per page, conservative assembly stays diagnostic.
7. **Export** — Unicode text, JSON with provenance, Markdown; review queues +
dashboards per page (existing Limbu artifact shapes, renamed pack-wide).
8. **Eval guardrail** — frozen real-page packs per language: page CER/WER, line
P/R/F1, reading order, blank/extra/missed lines, review burden.
PP-StructureV3 and PaddleOCR-VL remain the documented fallback/teacher direction for
hard layouts, per `docs/NEPAL_MULTILINGUAL_OCR_GOAL.md`.
### Production book-ingestion plan (2026-07-04)
The production target: a book (scanned PDF, image folder, phone captures) in;
editable Unicode, provenance JSON, and Markdown out; a review queue covering only
lines that need a human; frozen regression packs gating every model/config change.
The production KPI is **review burden** (can an editor fix a book in hours), not
CER alone. Phases in leverage order:
1. **PP-StructureV3 spike — DONE 2026-07-05, verdict: stays fallback.** The
2026-07-04 CPU run died producing nothing (~6 h on page 1/8); the re-run on
the L4 GPU did all 16 rebuilt pages in ~4 s/page. Scored on the rebuilt
16-page / 4,895-line Limbu silver pack, same champion recognizer and same
`run_page_ocr_eval.py` flags in every arm (diagnostic, non-claim):
| arm | page cp CER | det F1 | matched cp CER | order acc |
|---|---|---|---|---|
| PP-StructureV3 raw (its own reading order) | 0.797 | 0.904 | 0.0345 | 0.778 |
| PP-StructureV3 boxes + Rows-LTR + y/conf70 | 0.421 | 0.934 | 0.0331 | 0.993 |
| Current stack: Paddle det + Rows-LTR | 0.437 | 0.942 | 0.0266 | 0.993 |
| **Current stack + y/conf70 filter (baseline)** | **0.413** | **0.966** | **0.0266** | 0.993 |
Stock PP-StructureV3 loses on every axis even after grafting our reading
order and filter onto its boxes: worse detection, worse crops for our
recognizer (matched CER 0.033 vs 0.027), far worse native reading order,
and it needs a GPU. The Paddle det + Rows-LTR + y/conf filter chain stays
the production backbone; PP-StructureV3 remains the documented fallback /
teacher for table- and figure-heavy layouts (not exercised by these
text-heavy newspaper pages). Artifacts:
`outputs/ppstructure-spike-rebuild-2026-07-05/` (stage-1 GPU JSONs,
stage-2 documents, all score dirs);
baseline `outputs/limbu-baseline-rebuild-2026-07-05/`.
2. **Detector fine-tune** — cash in the supervised det goldmine: 983 language
pages with PDF-text-layer line geometry across 50+ languages plus 26k CDC
annotations; fine-tune PP-OCRv5 det to fix the center-column miss pattern.
3. **One config-driven book runner** — generalize `run_limbu_book_ocr.py` into a
language-pack `run_book_ocr` (input → prep → layout → det → script-route →
recognize → assemble → export + review queue), resumable per page.
4. **Acceptance gates** — frozen page packs (Limbu 8-page, Newar pages, one real
book) with thresholds on page CER, line F1, reading order, review-burden %;
the parked eval harness serves as the gate runner, not a research project.
5. **Packaging/serving** — versioned per-script model bundles, Space/API on the
pack pipeline, CPU eval path, GPU batch for book jobs.
## External Native-Script Real-Data Harvest (2026-07-06)
Rebuilt the lost `newa-zenodo-6967421-v1` asset and swept Zenodo/HuggingFace/
GitHub/Internet Archive for other real (non-synthetic) native-script OCR
material. Everything landed at
`data/external-language-resources/native-script-real-2026-07-06/<script>/`
with a `provenance.json` per source (license, byte size, md5, verify method).
Full details in that tree; headline results:
- **Newa/Pracalit (Zenodo 6967421, CC-BY-4.0)**: re-downloaded, md5-verified
(`76d464a6b656482b7c0594c0cb37fb18`), extracted. **Correction to the record**:
it is **441 page images + 441 PAGE-XML**, not the 683 pages this doc
previously cited — 441 matches the record's own `metadata.xml` and the
Zenodo description's "441 training pages"; the "242 validation pages" the
same description mentions are not in this file. Do not carry 683 forward.
Mirrored bit-for-bit at heiDATA (`10.11588/DATA/WI9184`, same title/size) —
useful as a second-source integrity check, not new content.
`tools/newa_zenodo_build.py` expects `data/newa-zenodo/extracted/`; this
pull is at the new path above, so point the tool there (or symlink) before
re-running the line-crop pipeline.
- **Newa/Pracalit, other leads that did NOT pan out**: a newer O'Neill paper
(Zenodo 17268040, "Revisiting HTR for Nepalese Manuscripts in Pracalit
Script," 2025) cites no new public GT beyond 6967421. A published
"First-Ever Prachalit Newa Lipi Character Dataset" paper exists, but its
data (via `sobit-nep/Prachalit-Script-Character-Recognition-using-Deep-CNN`
on GitHub) is a **dead Google Drive link (404, not actually
"anyone with the link")** — verified, not assumed. heiDATA's `FID4SA-GT`
dataverse (the same group that hosts the Newa mirror) also has Devanagari,
Malayalam, and Bengali printed GT sets, CC-BY/CC0 — none of them our target
scripts, not pulled.
- **Ol Chiki (Santali)**: no image+GT dataset exists on Zenodo/HF/GitHub (only
a Tesseract `.traineddata` blob and app scaffolding with no bundled data).
Internet Archive had real wins instead: **`chiki-chemed`** (a Santali
primer, real scanned pages, CC BY-NC-ND 4.0 — note the NC-ND clause before
any commercial use) and **`aale-chhatka-pdf-e-magazine-{2023,september-2023}`**
(a real monthly Ol-Chiki literary e-zine, self-published by its own editor's
archive.org account; the 2023 item carries the Creative Commons Public
Domain Mark 1.0, which is neither CC0 nor a license, while the September item
has no license field despite the same publisher — don't assume terms
transfer). All three downloaded, size-verified, script-confirmed by rendering
a sample page.
- **Tirhuta/Mithilakshar — best new lead of this pass**: the Maithili e-zine
*Videha* (world's first Maithili webzine) publishes a **parallel Tirhuta
edition of every issue**, archived at Internet Archive under the `videha_*`
family, **CC BY-SA 4.0**, issues 001–149 confirmed present (3 items,
~12 GB) plus larger combined items (`videha-262` ~23 GB, `videha-306`
~40 GB) referencing further issues — **not bulk-downloaded, catalogued
only**. Pulled one sample PDF and confirmed by rendering it that the pages
are genuine Tirhuta-shaped glyphs, but `pdftotext` on the embedded text
layer yields **Devanagari-range codepoints**, not the true Tirhuta Unicode
block (U+114xx). That means these are **legacy glyph-substitution fonts**
— the same shape of problem this repo already solved for Gorkhapatra
Devanagari (`nayanepal`/legacy Preeti-family). If a Videha-specific
font map can be built, this is real running text at real periodical scale,
the same playbook as the Limbu win. This is the single most promising
unexploited source found this session.
- **Lepcha / Kirat Rai / Sunuwar / Limbu (Sikkim Herald)**: the four newer IPR
listing endpoints (`?language=Rai|Mukhia|Lepcha|Limboo`) remain useful for
discovery, and no Zenodo/HF/GitHub/IA periodical archive was found. This
2026-07-06 note is superseded for admission by the 2026-07-15 rights audit:
the express reproduction permission covers material hosted on the official
State Portal, not automatically `ipr.sikkim.gov.in`. The IPR tool is now
metadata-only by default; admit only a mirrored
`sikkim.gov.in/uploads/SikkimHerald/` file or separately permitted IPR file.
- **Nepal Lipi periodicals**: the Internet Archive `nepalbhasatexts`
collection (477 items) is almost entirely **Devanagari**-script Nepal
Bhasa print media (Situ, Jhee, Thaunkanhe, Khonapau, etc. — already the
domain of the existing Newar-Devanagari track, not this one). Checked
several candidates that looked native-script by title: `kmc-notice-2023`
is a single decorative Ranjana logo clipping, not running text.
`NepalLipiSangrahaByHemrajShakyavamsha_201712` (no stated license) is a
Devanagari-language book *about* Nepal Lipi that does contain real
alphabet-plate specimens of historic native scripts (Kunmol, etc., with
transliteration) — a glyph reference, not periodical running text.
No genuine Nepal-Lipi-script periodical PDF archive was found.
**Recommendation, one line per script**: Newa — nothing left to pull, move to
building the line-crop pipeline on the corrected 441-page set. Ol Chiki — use
`aale-chhatka-pdf-e-magazine-2023` (publisher/uploader Public Domain Mark 1.0,
confirmed script) as the first real
training/eval seed. Tirhuta — decode the Videha legacy Tirhuta font (highest
leverage find this session: real, licensed, periodical-scale) before anything
else. Lepcha/Kirat Rai/Sunuwar/Limbu — use the IPR listing only to discover
editions, then catalog hash-pinned mirrors on the permission-covered State
Portal; do not bulk-download IPR PDF bodies without separate exact-prefix
permission evidence.
## Native-script wave 1 results (2026-07-06)
- **Ol Chiki v1 (first recognizer for the script here)**: PP-OCRv5 mobile
from generic pretrained, 56,640 synth lines (Santali Wikipedia corpus ×
3 Noto weights × 6 degradations), 25 epochs. Synth eval best exact-line
0.546 / norm_edit_dis 0.800. **Real print (234 Aale Chhatka magazine
lines via the new OLCKOptimum converter): cp CER 0.2245, exact-line
0.184** — real-print performance ≈ synth-eval performance, so the
synthesis recipe transfers; quality itself is the v1 limit. Known levers,
in leverage order: (1) **same-font synth** — we hold the extracted
OLCKOptimum TTFs; replay the Sunuwar same-font lesson (0.342→0.121);
(2) resolve the 2 uncertain converter bytes (`n`,`T`) to unlock 653 more
real lines for train/eval; (3) real-line fine-tune on an issue-split of
those lines. Artifacts: `outputs/olchiki-v1-run-2026-07-06/`
(checkpoint+export+scores), pack `data/olchiki-real-eval-pack/`,
converter `nepal-ttf2utf@acab167`. Claim note: Santali OCR exists
elsewhere (India); this is pack coverage, not a first-ever claim.
**v2 (same-font, 2026-07-07)**: re-rendered the same corpus with the
OLCKOptimum fonts extracted from the magazine PDFs (all 48/48 block chars
covered by the embedded subsets) + one Noto weight: **real-print cp CER
0.1911, exact-line 0.321** (same 234-line pack) vs v1 0.2245 / 0.184 —
the same-font lever works on this script too, though less dramatically
than Sunuwar's; remaining error is heading/display type, tight-leading
crop bleed, and genuine model limits. Artifacts:
`outputs/olchiki-v2-run-2026-07-07/`. A later asset-independence audit found
that those fonts came from the exact evaluation edition, so v2 remains a
useful diagnostic but cannot provide an edition-independent claim result.
- **Newa native v1 (manuscript domain): honest failure.** 150 epochs on the
recovered Zenodo 441-page GT (2,391 train lines, by-manuscript split,
800px width, PP-OCRv5 mobile from generic print pretrained): dev exact
0.0016 / norm_edit 0.144, and crucially **train-side norm_edit is also
0.146** — the model could not fit its own training set, so this is a
recipe failure (print-pretrained mobile backbone vs dense 16th–19th-c.
handwriting at 63–154 chars/line), not a hand-generalization failure.
The reference point (O'Neill 2022/2025, Transkribus HTR on the same data)
shows the task is learnable with HTR-specific training. v2 design:
synth-first curriculum — render the TRAIN-split transcription text (plus
the 36k-char web corpus; no test text, no transliteration) in the 2
Unicode Newa fonts with degradations, pretrain, then manuscript
fine-tune; consider the server rec backbone. Artifacts:
`outputs/newa-native-v1-run-2026-07-07/`.
## Fan-out results (2026-07-13, Codex wave)
- **Ol Chiki uncertain bytes RESOLVED** (`nepal-ttf2utf@5822f30`): bytes
`n`→ᱱ (EN U+1C71) and `T`→ᱛ (AT U+1C5B), each confirmed by three
independent evidence lines (glyph stroke features, corpus context
distributions vs the 204k-line Wikipedia corpus, lexicon hit-rates 0.89 vs
0.57–0.60 runner-up); evidence at `outputs/olchiki-uncertain-bytes-2026-07-13/`.
Clean real-print yield rose 234 → 895 lines; expanded pack at
`data/olchiki-real-eval-pack-v2/`. **olchiki-v2 on the 895-line pack:
cp CER 0.1275, exact-line 0.197** (scored on omac, 5m18s) — the new
body-text-dominated benchmark reads meaningfully better than the
display-type-heavy 234-line pack suggested (0.1911). Pack v2 was initially
treated as the reference eval, with 895 candidate lines. The
following audit superseded that interpretation before any claim use.
**Corrections (2026-07-14, license audit + label audit)**: (a) public
claims may use only the publisher/uploader public-domain-marked issue; the
mark is not CC0 or a license; (b) the label audit found 67/895 crops blank
(pack-builder bug, all in that issue). The resulting cp CER 0.1018 / exact
0.191 (n=366) and combined clean 0.0826 (n=828, internal) are historical
diagnostics, not claim headlines: surviving crop/label mismatches, ambiguous
punctuation, incomplete provenance, and same-edition font leakage remain.
Also found: the
Newar-Devanagari packs carry systematic converter whitespace artifacts
(spurious space after halant / before combining marks; ≥1 in ~45% of test
lines) — word content is faithful, but a whitespace-normalization pass is
required before those numbers are claim-grade. See docs/DATA_LICENSING.md
and the verification-wave section.
## Newa root cause found (2026-07-15): CTC timestep cap, not data or height
The height-ablation prep (dry-runs on omac; artifacts at
`outputs/newa-height-ablation-prep-2026-07-14/`, VERDICT.md) found the
mechanical cause of the v1 training failure: PPLCNetV3's train-mode forward
ends in `adaptive_avg_pool2d([1,40])`, hard-capping the CTC sequence at
**T=40 timesteps at any input height or width** — and **95.7% of manuscript
train lines exceed 40 chars** (median 63, max 154). CTC cannot emit more
labels than timesteps: v1 could never fit its own training set, and a pure
height ablation is predicted to reproduce the failure. This coheres with
every other result: Limbu/Ol Chiki/Newar-Devanagari lines mostly fit ≤40
chars and their models trained fine. Also verified: heights 96/128 train as
pure config changes (3 finite loss steps each), but the EVAL path breaks
(`Im2Seq` asserts H==1) — a verified 1-line patch ships in the prep dir;
pretrained weights load identically at all heights; an earlier T=40 memory
estimate proposed L4 batches 16 (h96) / 12 (h128), but T=200 requires a fresh
approved-GPU measurement. The corrected Newa experiment therefore needs a single
train/eval pool width of T=200 (the exact CTC lower-bound maximum is 176 after
normalization), h96, and only the CC-BY-4.0 Zenodo manuscript source. A
backward-compatible pool patch has now been shape-tested on `omac`: defaults
remain train T=40 / eval T=100, while the opt-in h48 and h96 paths both produce
T=200 and a length-162 CTC loss is finite. The pre-existing zero-byte
`newa_real_000041.png` was not dropped: it was restored from the authoritative
local master and pinned at SHA-256
`c86f18e93b123a87e26682b973c5db8a2f8931434f11f6c88a63105161920360`.
## Native-script verification round (2026-07-15)
- **Newa CTC200 payload staged, not trained.** Labels are narrowly normalized
(NFC, NBSP to space, whitespace runs collapsed, boundaries trimmed), and
every referenced crop decodes with ink. Exact-label and 16-codepoint
substring isolation removed 430 of 2,391 training rows, leaving 1,961.
Hitopadesa is the 144-line checkpoint-selection dev manuscript; the 635-line
Cambridge manuscript is sealed as `claim_eval` and must not enter training or
model selection. Post-filter overlap is zero and all labels are feasible at
T=200. A prediction-free all-row Cambridge review bundle has now been built
on `omac` from the exact payload and original Zenodo export: 635/635 bundled
PNGs are byte-identical to their payload crops, all 635 decision rows remain
blank, and the archive is explicitly not claim evidence. The crop labels
still need documented native pixel review; any corrections require a fresh
post-review exact/16-codepoint leakage audit before a claim-eligible freeze.
A fail-closed freezer now revalidates the exact payload/source/review
archives, immutable fields, pixels, all 635 decisions, normalization, and
the >=200-row dedup gate. It keeps non-attested outputs claim-false and
requires exact CTC200, train/dev, post-review leakage, checkpoint, and
training-history bindings for claim readiness. The real blank worksheet
fails atomically; no reviewed pack exists. GPU training remains
approval-gated. Artifacts:
`outputs/newa-ctc200-prep-2026-07-15/`,
`outputs/newa-cambridge-review-v1/`, and `docs/handoffs/HANDOFF-newa-ctc200.md`.
- **Newa balanced sampling is now a deterministic, disabled candidate.** The
exact 1,961-row train-only capsule contains 146,153 Newa codepoint tokens
across 69 types and 87,777 extended-grapheme tokens across 585 types. The
schedule excludes the 2 codepoint and 71 grapheme singleton types, adds 490
references under a three-occurrence row cap, and preserves exactly the same
1,961 unique image/text pairs. Train/Hitopadesa exact-label and 16-codepoint
overlap are zero before and after scheduling; no Hitopadesa or Cambridge row
is copied. The six-file artifact reproduced byte-identically when rebuilt
with the pinned `regex==2026.5.9` segmentation runtime. This is a sampling
heuristic, not accuracy evidence, and no current payload or config enables
it. Activation requires a newly hash-bound training path, approved GPU
ablation, and the final corrected-Cambridge leakage audit. Artifact:
`outputs/newa-balanced-sampling-v1/`.
- **Stock RepSVTR is a runnable CTC200 fallback, not a trained result.** A
no-image, batch-1 CPU probe on `omac` used the clean PaddleOCR v3.7.0 checkout
and the official hash-pinned RepSVTR checkpoint. At H96xW1600, the
train-mode CTC branch and eval-mode full forward both emit T=200, leaving 24
timesteps above the audited corpus maximum of 176; an alternating length-176
synthetic CTC loss is finite. All 301 backbone tensors and 394/398 total
tensors are pretrained-compatible, with only the four Newa-vocabulary-sized
tensors reinitialized. A follow-up synthetic CPU probe then exercised the
exact five-tensor loader contract through one full train-mode MultiHead
forward, CTC+NRTR MultiLoss, backward, and scheduler-positioned AdamW update.
It observed CTC `[1,200,108]`, NRTR `[1,177,112]`, finite component losses,
279/279 finite nonzero gradient tensors, and finite positive updates in the
backbone plus both heads; peak RSS was 1,646,444,544 bytes. No image, label
file, dataset row, prediction, score, GPU, cloud, or network action occurred.
The exact configured warmup starts at LR 0, so the probe explicitly advanced
the scheduler 1,961 times without optimizer calls before its sole update at
LR `4e-5`; this is mechanism evidence, not first-iteration chronology or
training-loop history. Together the probes establish an independent
backbone/temporal path from patched PPLCNetV3 and close the synthetic
full-loss/backward/update gap, not the framework, real-data, GPU-memory,
convergence, or accuracy gaps. Training remains approval-gated, and
selecting this path requires the distinct stock-RepSVTR claim-attestation
profile now implemented in `tools/audit_newa_cambridge_claim_readiness.py`
and its freezer. That profile pins the exact config, feasibility report,
fully clean PaddleOCR checkout, stock sources, official initialization,
approval, run log, selection record, and a CPU-loaded final state dict with
the exact 398 Newa model keys/shapes. The freezer independently requires the
raw approval/run/log/selection records and the selected checkpoint. The
legacy four-file PPLCNetV3 profile remains unchanged. Identity and execution
authenticity are still an external custody/signature boundary; no real
approval, training record, compatible final checkpoint, attestation, or
score exists. Artifacts: `outputs/newa-repsvtr-ctc200-feasibility-v1/` and
`outputs/newa-repsvtr-full-train-step-v1/`.
- **Ol Chiki v2 withdrawn from claim use; v3 is only a review candidate.** A
full audit found blank/rule-only crops, unrelated or neighboring text,
unresolved legacy punctuation, a stale converter-commit claim, and training
fonts extracted from the evaluation edition. The public-source-only v3
builder pins the PDF and resolved map, uses neighbor-clamped vertical crops,
rejects blank renders, emits line-level hashes/provenance, and refuses to call
its output frozen or claim-eligible. The remote candidate contains 358 rows
after 75 blank renders were excluded. Every row still requires review by an
Ol Chiki reader; no new score is permitted before freeze and independent
exact/substring and asset-isolation audits. A fail-closed freezer validates
the pinned candidate and all-row review archives, the source PDFs, every crop
and provenance binding, and all 358 immutable review decisions; it refuses
claim eligibility without separately bound leakage and asset-isolation
attestations. No decisions or frozen pack exist. Artifacts:
`outputs/olchiki-claim-eval-v3-2026-07-15/`.
- **Tirhuta now has an edition-disjoint train/eval candidate route, not a
score.** PyMuPDF retained glyph IDs for all 3,306 U+FFFD characters in the
152-page issue-001 sample. A fail-closed 164-entry functional map recovered
every one; 30,396/30,581 Janaki trace fragments then converted cleanly
(99.395049%). Issue 001 now supplies 400 nonblank real-line eval candidates
across 116 pages and a prediction-free all-row review packet. The separately
dated, CC BY-SA 4.0 April 2008 issue required 34 additional GID mappings.
Exhaustive shaping with the hash-pinned Janaki 1.000 reference font
reproduced all 34 plus the prior 164 with zero regressions. Its 2,000-row
candidate has 2,000 unique image hashes and zero image-audit issues; removing
one 16-codepoint cross-issue substring conflict leaves 1,999 training rows
with zero exact or long-substring overlap. Thirty new GIDs have clean crop
witnesses and four have shape evidence only. The 400 eval rows and 30 focused
mapping rows remain native-review pending; no model or CER exists. Artifacts:
`outputs/tirhuta-videha-gid-recovery-v1/`,
`outputs/tirhuta-videha-line-candidate-v1/`, and
`outputs/tirhuta-videha-april2008-train-candidate-v1/`. A separate
fail-closed freezer now validates the two pinned issue-001 archives, source
PDF, packaged audits, converter/runtime provenance, and the exact crop bytes
shown to reviewers. It requires all 400 immutable decisions, at least 200
nonblank unique accepted lines, and Tirhuta-bearing human labels. Claim
eligibility remains false unless an independent attestation binds the
reviewed label-file hash and the exact April train label/manifest hashes and
passes exact-text, 16-codepoint substring, edition, and model-selection
isolation. The decisions and attestation do not yet exist, so no pack was
frozen and no score was run.
- **The other-script registry was downgraded to extant evidence.** The Kirat
Rai pack has 70 rows / 53 unique labels, and all 53 occur in its real-render
training inventory. The shared Devanagari harvest has 79,364 labels but only
40,358 images, so it is a training inventory with 39,006 missing assets, not
eval. Sunuwar's historical 429-row pack/checkpoint and Lepcha's source PDFs
and checkpoint are absent. Historical metrics remain historical and none of
these routes is a current script delivery. Registry evidence:
`configs/nepal-language-pack-pipeline-v1.json`,
`configs/recognizer-eval-pack-normalization-v1.json`, and
`configs/eval-harness-recipes-v1.json`.
- **Sikkim Herald source permission was found and the three routes reopened.**
The earlier Indian-government-work term analysis missed the Government of
Sikkim State Portal's express copyright permission. Its official copyright
page permits free reproduction in any format/media without specific
permission when the material is reproduced accurately, is not used
derogatorily or misleadingly, and the source is prominently acknowledged;
explicitly identified third-party material is excluded. The same portal
hosts Sikkim Herald, and the IPR Department identifies it as its official
government paper. A live metadata-only enumeration of the newer IPR portal
found 19 current Lepcha, 22 Mukhia/Sunuwar, and 5 Rai discovery candidates,
while the State Portal exposes dated editions for all three. A bounded live
State Portal catalog retained 29 unique canonical PDF URLs: 8 Lepcha, 11
Mukhia/Sunuwar, and 10 Kirat Rai. It rejected one duplicate and one non-PDF
path. The searches advertise 193/167/174 records, but `PageNo=1/2` returned
identical URL lists, so the artifact is explicitly incomplete. It fetched
no full PDF body and made only three eight-byte range probes against samples
already full-hash-pinned by the prior policy. Every edition remains
third-party-screening pending, train/eval blocked, and claim-grade false.
Because the permission page is domain-specific, only `sikkim.gov.in`-hosted
PDFs are catalog candidates; IPR-subdomain files remain discovery-only unless
mirrored or separately permitted. This resolves the State Portal
source-permission gate under a custom permission, not a public-domain claim.
It does not restore the lost packs/models or cure the Kirat Rai pack's 100%
text leakage. Next steps are full-hash acquisition and third-party screening
of selected editions, followed by fresh edition-disjoint converter and native
label audits. Evidence:
`outputs/sikkim-herald-source-rights-2026-07-15/`,
`outputs/sikkim-herald-state-portal-catalog-2026-07-15/`,
`docs/DATA_LICENSING.md`, and `docs/ATTRIBUTIONS.md`.
- **Six Sikkim Herald editions were acquired, page-screened, and font-audited
without creating labels.** The source contract selected two State
Portal-hosted editions per script: Kirat Rai 2024-09-26/2025-01-13 (109
days), Lepcha 2024-07-26/2024-10-04 (70 days), and Sunuwar/Jenticha
2024-04-12/2025-01-26 (289 days). All six full PDF hashes and all 26 page
renders were frozen after an `omac` run. Reviewer 1 inspected every page at
original render resolution: 25 pages have no explicit third-party marker
observed, while both complete press-photo rectangles on Lepcha 2024-10-04
page 1 are bounded for exclusion because incidental storefront signage is
visible. Both Lepcha PDFs have a structurally and visually blank page 5.
This is a one-reviewer observation under the State Portal's custom
permission, not a public-domain or government-authorship finding; reviewer 2
and adjudication remain null.
- **The revised live-layout converter audit reaches 100% aggregate coverage on
both Kirat Rai and both Sunuwar issues.** Kirat raw body-unit totals are
13,101 and 7,920. Exact comparison to the `kirat rai font new` program at
xref 536 of official Unicode proposal L2/22-043R resolves the newly observed
Herald `T` through canonical `Z` to U+16D6C; the isolated extracted `C` has
an empty outline and normalizes to space, and original-resolution review
establishes that the custom glyph under `:` is a double quotation mark.
The pair has 91 observed semantic source-code matches, zero mismatches, and
zero unresolved units. Sunuwar raw totals are 9,697 and 14,676. Both font
subsets match the exact `kirat1` program in the cited
`sunuwar_01348c63.pdf` derivation source on every observed script code (76
summed matches, zero mismatches); backslash is an empty glyph, and the one
U+201D extraction value visibly prints as a colon. The official proposal PDF
hash is `88b1d4dfedd8b36a687533f3fa1ef0a97c42965fd595b28678ca8d7d17cb8f17`
and its embedded reference font hash is
`bab945f8a5fe51f9401dc2a55a54ac99f3ba2b5071ac72324a05e7ff82cb4149`;
the Sunuwar reference PDF/font hashes are
`b3c001ff428de3b4e539c88d6d93566ae969a46b9ba947fe680fc1b68eb10456`
and `484550543232f587fab3a5b3e900c922845c5ed44ef7bdd493cace4a5b6bce78`,
tied to MIT-licensed `nepal-ttf2utf` 0.2.0 at
`a158a8cbdbe230df6849afd9c9736bf31cb9c24b`. The two 2024 Lepcha bodies are
raster-composed, with 640 and 2,212 image placements, so converter coverage
is N/A rather than 0%.
- **The joined bundle passes adversarial metadata audit but remains
`NOT_ADMITTED`.** Mutation tests reject any label-write claim, injected
text/label field, unreconciled counter, font mismatch, or invented second
reviewer. The frozen audit joins the exact source review, 26-page reviewer-1
record, revised converter artifact, and reference config; it records four
live editions at 100%, two raster N/A editions, zero labels, zero native
reviewers, no GPU/cloud/training/scoring, and no stored source or converted
text. Exact and substring leakage are explicitly N/A with zero rows, not
passed. Before any line can exist, the route still needs independent
provenance reviewer 2/adjudication and native-script pixel/label review;
only then may it create an edition-isolated train/eval split, run exact plus
substring leakage audit, train, and compute codepoint CER. Artifacts:
`outputs/sikkim-herald-six-edition-source-review-2026-07-15-r11/`,
`outputs/sikkim-herald-six-edition-manual-review-r1-2026-07-15-r11/`,
`outputs/sikkim-herald-six-edition-converter-layout-audit-2026-07-15-r11b/`,
and `outputs/sikkim-herald-r11b-adversarial-bundle-audit-2026-07-15-r11/`.
- **A blinded Sikkim Herald provenance-review handoff is frozen, but no second
review is claimed.** The reviewer-2 packet links all 26 exact full-page
renders at original resolution without copying images or exposing reviewer-1
decisions. Its byte-bound worksheet requires a distinct reviewer 2 who did
not inspect reviewer-1 decisions before completion, a third distinct
adjudicator, exact six-edition/26-page coverage, and one resolution for every
disagreement and none for an agreement. Mutation tests reject prefilled
decisions, reused identities, missing/duplicate pages, out-of-page exclusion
geometry, label/prediction payload keys, and incomplete adjudication. The
independently frozen packet audit verifies all 26 HTML links and hashes and
reports `PASS_READY_NOT_ADMITTED`; the real worksheet remains blank, so
reviewer 2, adjudication, native line review, rows, leakage, a model, and CER
remain absent. Artifacts:
`outputs/sikkim-herald-six-edition-provenance-review-r2-packet-2026-07-15-r11/`
and
`outputs/sikkim-herald-six-edition-provenance-review-r2-packet-audit-2026-07-15-r11/`.
- **The recovered Newar-Devanagari whitespace rewrite is mechanically exact,
but the underlying pack fails admission.** A 64 GiB `omac` run independently
reproduced all 1,820/565 train, 634/353 dev, and 427/142 test removals for
the two narrowly defined artifact classes, matching all 6,763 normalized
labels and leaving zero targeted patterns. All referenced PNGs decode and
contain at least 16 dark pixels; there are 6,757 unique image hashes, four
intra-train duplicate groups (six rows beyond the first), zero conflicting
labels within those groups, and zero cross-split image hashes. The honest
split audit fails: train versus dev+test has 18 exact rows and 575 rows with
a shared 16+-codepoint substring (including exact), while dev versus test has
zero exact but 136 substring rows. The recovered labels also lack source URL,
PDF hash, page, bbox, date, and edition fields, and Gorkhapatra rights remain
`RISK`. A deterministic non-native geometry review saw ink in all seven
risk-selected samples but obvious neighboring-line fragments in two of the
three widest crops; it makes no population-rate or label-faithfulness claim.
No filtering, training, prediction, score, or CER was run in this audit
round. Artifacts:
`outputs/newar-devanagari-normalized-claim-boundary-audit-2026-07-15-r12/`
and
`outputs/newar-devanagari-normalized-visual-sample-review-2026-07-15-r12/`.
## Gurung Khema official-print reopening (2026-07-15 r13)
- **The former “synthetic-by-nature” source conclusion is refuted.** Two
official State Portal Sikkim Herald Gurung Editions contain running Gurung
Khema print across eight pages and are 52 days apart. The 2022-03-01 PDF is
SHA-256
`e667bbd2eb4f6769209013fe569cdc37d3e1aff40a9f0121301b43a03afe5d14`;
the 2022-04-22 PDF is
`2db16c7b7abe678e02103037397c7a35e21fb2fd34d4e4413d1979d1cf7f7c93`.
The pages contain 21,106 text-layer characters. April page 4 prints issue 4
while pages 1–3 print issue 5; the frozen source record preserves this
anomaly. Reviewer 1 inspected every original-resolution page and observed no
explicit third-party marker, but that is not a government-authorship or
license-cleanliness finding.
- **The legacy font has a fail-closed Unicode 16 conversion path.** Unicode
proposal L2/22-096 states that the republished 2022 Herald uses the same
legacy KHEMA font as the proposal and supplies the legacy key semantics. Both
Herald PDFs use U+F0xx PUA re-encodings of those legacy bytes. Their 56 shared
cmap codes have exact decomposed outlines and advances. For each edition, all
53 compared observed mapped glyphs rank first against the expected proposal
glyph; the minimum observed binary-mask IoU is 0.490751. The converter handles
direct keys, the Herald C/N/X printed-shape aliases, O/V combining-mark
unifications, adjacent pre-base T/E reordering into Unicode logical order,
and paired `DD` as double danda. It rejects ambiguous input rather than
guessing.
- **Mechanical eligibility is 749 of 759 candidate lines, not a dataset.** The
March edition has 382 eligible and 6 rejected lines; April has 367 eligible
and 4 rejected. Across both editions, the 10 rejections are four lines with
no Gurung Khema character, five non-adjacent pre-base-medial sequences, and
one isolated legacy danda unit. The audit serializes no source or converted
strings and writes no crops or labels.
- **The route remains `NOT_ADMITTED`.** The separately frozen reviewer-2 packet
links all eight hash-bound source renders, exposes no reviewer-1 decisions,
and contains a byte-identical blank worksheet. Its adversarial audit reports
`PASS_READY_NOT_ADMITTED`. An independent reviewer 2 and distinct
adjudicator, then native pixel-label review, crop construction, an
edition/manuscript-isolated split, exact plus substring leakage audit, and
codepoint-CER scoring are still required. There are currently zero labels,
predictions, scores, training rows, or CER values. The joined r13
adversarial audit also pins every source/font/review/packet hash, rejects
mutated counts, glyph ranks, blinding, payloads, and admission, and exercises
the real freezer to prove that the blank worksheet fails before writing an
output directory. Artifacts:
`outputs/sikkim-herald-gurung-khema-two-edition-source-review-2026-07-15-r13/`,
`outputs/sikkim-herald-gurung-khema-font-semantic-audit-2026-07-15-r13/`,
`outputs/sikkim-herald-gurung-khema-manual-review-r1-2026-07-15-r13/`,
`outputs/sikkim-herald-gurung-khema-provenance-review-r2-packet-2026-07-15-r13/`,
and
`outputs/sikkim-herald-gurung-khema-provenance-review-r2-packet-audit-2026-07-15-r13/`,
plus the joined verdict at
`outputs/sikkim-herald-gurung-khema-r13-adversarial-bundle-audit-2026-07-15-r13/`.
## Corrected claim numbers (2026-07-16, fixes executed on omac)
- **Ol Chiki eval pack v3** (blank-guard builder, mechanism: white-fill
invisible text in the source PDF): 772 clean lines (123 blank/near-blank
dropped — the <16-dark-px guard caught 56 more than the census's fully
blank 67). olchiki-v2 scores: **public headline (CC0 issue) cp CER 0.0961
/ exact 0.193 (n=358)**; September subset 0.0704 (n=414); combined 0.0826
/ 0.2034 (n=772, internal until September license confirmed). Supersedes
all prior Ol Chiki numbers.
- **Newar-Devanagari whitespace-normalized rescore** (newar-ft-v1, frozen
1,139-line test, scored on omac): old artifact-bearing 0.0466; raw
predictions vs normalized labels 0.0501 (the model reproduces training
artifacts); **both-sides-normalized cp CER 0.0441 / exact 0.2888 — the
claim-comparable number** (labels at
outputs/newar-pack-v1-normalized-2026-07-15, utility
src/ocrtech/devanagari_whitespace.py).
## Tirhuta real substrate delivered (2026-07-17)
First-ever real Tirhuta training text: `outputs/videha-tirhuta-silver-2026-07-17/`
— **4,148 clean converted lines** (labels.txt) from Videha issues (CC BY-SA
4.0, per-issue provenance), zero-unmapped-only policy. Extension-byte drop is the
dominant loss (e.g. first issue: 1,235 extension-dropped of 7,622 examined)
— building the conjunct extension-glyph map would multiply yield; per-issue
numbers in the pack summary.json. Also staged and VERIFIED: Ol Chiki v3 fine-tune payload
(`outputs/olchiki-v3-ft-payload-2026-07-17/`, 5,330 entries) — train is 242
real Public-Domain-Mark-issue lines + 5,000 synth replay, September 414
eval-only with a zero-intersection leakage audit on filename/line-id/label;
READY-PENDING-APPROVAL in the GPU dossier (see
outputs/overnight-consolidation-2026-07-16/gpu-batch-proposal.md, candidate 3
update). Operational note: the Codex quota briefly hit its limit on 2026-07-16
(reset message named 2026-07-22) but lifted the same night — fan-out resumed
with the Tirhuta extension-map and Lepcha zero-yield lanes.
## Lepcha zero-yield root cause (2026-07-17)
The Sikkim Herald Lepcha lane's 0 lines (vs Sunuwar 9,743 / Kirat Rai 3,699)
is a data reality, not a bug: in all 18 editions the Lepcha article body is
rasterized as image XObjects — there is no live Lepcha text layer (live text
is English masthead/punctuation in Times New Roman / anonymized CID fonts;
zero U+1C00-1C4F codepoints in sampled pages). The converter was never
invoked. Full evidence: outputs/lepcha-zero-yield-diagnosis-2026-07-17/REPORT.md.
Consequence: Lepcha real-line substrate requires a raster OCR/transcription
lane (or stays synth-first per the headline plan); the builder should record
`raster_body_requires_ocr` instead of `wrong_font` for such editions.
## Tirhuta extension map + v2 pack: yield 2.6x (2026-07-17)
Codex built the Videha Tirhuta extension-glyph map (149 new mappings, each
grounded in font glyph semantics + Pandey L2/11-175R; 2,131 converter tests
pass) — branch `agent/tirhuta-extension-map` in nepal-ttf2utf, commit
a5ba521. Rebuilt the pack on omac with the new converter:
`outputs/videha-tirhuta-silver-v2-2026-07-17/` — **10,696 clean lines**
(vs 4,148 in v1; 45,157 examined; extension-byte drops now 1, down from
thousands). Pack verified (archive files == labels == manifest == 10,696),
backed up to HF ampixa/ocr-tech-artifacts. Builder bug found during
finalize: `str.splitlines()` splits JSON-legal raw U+0085 in provenance
strings — fixed to split("\n") (patched builder stored in the pack dir).
Remaining converter work: merge the branch to main after review.
## Three FT payloads staged (2026-07-17)
Codex assembled per-script fine-tune payloads on the Ol Chiki v3 discipline
(issue/page-boundary dev splits, four-way zero-leakage audits, sha-pinned
manifests, training_launched=false):
- **Tirhuta READY**: outputs/tirhuta-v2-ft-payload-2026-07-17/ — 9,765 real
train / 844 dev (whole issue videha_01_02_08 held out), 10,639 entries,
sha 55c18998...13e4. CC BY-SA 4.0 with attribution recorded.
- **Sunuwar BLOCKED-PENDING-LEGAL**: outputs/sunuwar-v1-ft-payload-2026-07-17/
(8,422/514) — Sikkim Herald State-Portal permission + India §17(d) question
unresolved; assembled but not trainable yet.
- **Kirat Rai BLOCKED-PENDING-LEGAL**: outputs/kiratrai-v1-ft-payload-2026-07-17/
(2,865/193) — same gate.
Verified: no synth replay pools or warm-start checkpoints exist for these
three scripts anywhere in outputs/ or HF artifacts (the native-script wave
built substrate, not models; only Ol Chiki trained). Consequence: the proven
synth-pretrain-then-real-FT recipe needs a Tirhuta synth pack before the
Tirhuta payload trains warm; alternatively from-scratch real-only must be an
explicit dossier decision.
## Tirhuta synth-v1 pretrain pack rendered (2026-07-19)
The missing pretrain stage for Tirhuta is now built:
`outputs/tirhuta-synth-v1-prep-2026-07-17/tirhuta-synth-v1-rendered.tgz`
(sha 2bf91b49...1327, 48,960 PNGs = 8,160 texts x 6 degradation profiles,
7,997 train / 163 eval texts, zero normalized collisions with the real FT
dev set, Noto Sans Tirhuta OFL only, RAQM shaping verified, dark-pixel
guard min 102). Rendered on omac; archive re-extracted and hash-verified;
backed up to HF. Engineering notes: macOS Pillow lacks RAQM — fixed via
Xcode CLT + brew libraqm + DYLD_FALLBACK_LIBRARY_PATH exported INSIDE the
script (SIP strips DYLD_* across /bin/zsh exec); audit had a /tmp vs
/private/tmp resolve() mismatch, patched in the pack's audit script.
Tirhuta training chain is now fully staged: synth-v1 pretrain ->
tirhuta-v2-ft real payload (dossier candidate 5); GPU run still
human-gated.
## Next Work
0. **Post-recovery next (2026-07-05)**: the spike verdict keeps the Paddle
det + Rows-LTR + y/conf70 chain as the backbone. **Phase-2 pivot, measured
2026-07-05**: decomposing the 0.4134 filtered-baseline page CER on the
16-page pack showed the standing "region selection + furniture" assumption
is wrong — deleting every unmatched prediction (perfect furniture/region
filtering) changes CER by ~0 (0.4152), while reordering the matched lines
into reference order alone drops it to **0.0378** (10.9×). **Reading order
is 91% of page CER.** Geometry heuristics cannot recover it (Columns LTR
0.799, PP-Structure-region banding 0.702, PP-Structure native order 0.797)
because the reference order is the PDF's article-box emission order, not a
row/column pattern. Three follow-up experiments (all 2026-07-05,
artifacts under `outputs/reading-order-model-v1-2026-07-05/` and
`outputs/line-merge-v1-2026-07-05/`) then closed the question:
- **Learned reading-order model v1: refuted.** LightGBM pairwise
comparator on the silver `span_bbox` order supervision regressed to
0.799 — the Devanagari packs preserve PDF article-box emission order
(not row-major) while the Limbu eval reference is a strict y-sort;
wrong supervision for the target. Leakage-clean; kept as a negative
result.
- **Detector-fragmentation hypothesis: refuted.** Det boxes ≈ 1:1 with
reference lines (mean best-IoU 0.68; the widest lines average 1.05
fragments). Tuned merge (gap 0.6×h) merges ~3 boxes/page, CER flat.
Naive merge (gap 1.0×h) bridges the 40 px column gutters and collapses
det F1 to 0.71 — gutters are narrower than line height on these pages.
- **Root cause found: the reference linearization is our own artifact.**
`build_limbu_gorkhapatra_all_corpus.py:209` sorts spans strict
y-then-x across a 6-column layout; det-vs-PDF y-jitter of a few px
flips column interleaving constantly (pair accuracy 0.998, page CER
still 0.30–0.41). Pure-y prediction sort gives a free win (0.4134 →
0.3006), and **one shared deterministic column-major linearization
applied to both reference and prediction yields page cp CER 0.0451** —
at the content floor (~0.054: recognition 0.026 + 211 missed / 87
extra lines). **Page reconstruction on this pack is solved up to
linearization definition; the honest next metric is page-eval v2 with
column-major linearization on both sides** (a documented metric
change — not comparable to the old 0.41-scale numbers), and the
production assembly should emit column-major order. Remaining real
levers after v2: det recall/precision (211/87), recognizer (0.026),
and native review for claim-grade.
**Page-eval v2 + Column Major assembly shipped 2026-07-05**: the shared
linearization lives in `src/ocrtech/linearization.py` (tests in
`tests/test_linearization.py`); `tools/apply_page_reading_order_strategy.py`
gained mode `"Column Major"`; `tools/run_page_ocr_eval.py` gained
`--page-linearization {native,column-major}` (recorded in
`page-eval-run.json`, per-sample fallback to native recorded when a
reference lacks line geometry). Validated on the 16-page pack:
Column-Major assembly + v2 scoring gives **page cp CER 0.043 / grapheme
0.067** (`outputs/limbu-column-major-2026-07-05-score-v2/`); the native
path is regression-identical (0.413/0.437). v1-scale and v2-scale numbers
must never be compared directly.
**Newar page-eval v2 (second-language proof, 2026-07-05/06)**: a 19-page
Newar page pack built from the rebuilt harvest
(`tools/build_newar_gorkhapatra_silver_page_eval.py`,
`outputs/newar-page-eval-v2-2026-07-05/pack/`) and scored through the
identical non-oracle chain (Paddle det → recognizer → Column Major →
y/conf70 → page-eval): **newar-ft-v1 native 0.778 → v2 0.199 page cp CER**
(grapheme 0.279; det F1 0.912; matched-line cp CER 0.0905) — the
column-major linearization story replicates exactly on the second
language; the Newar residual is recognition + det-crop domain gap, not
reconstruction. Two follow-ups surfaced and **both were fixed
2026-07-06**: (a) the deployed `limbu-devanagari-ppocrv5` export was
unusable as a general Devanagari recognizer (409-byte dict missing
anusvara, ii-matra, ṇa…; emitted garbage — its page numbers were a
formality, not a model comparison); the recovered deva-v2 training
checkpoint was re-exported with its correct 144-char `devanagari-v2.txt`
and **verified to reproduce the recorded pre-crash Newar baseline
exactly** (cp CER 0.062229 / grapheme 0.13120 / exact 0.158 on the
recovered 1,139-line frozen test); the Space now serves it as
`models/deva-v2` (export also at `outputs/deva-v2-reexport-2026-07-06/`
and on the HF dataset). (b) The Newar pack builder's digest mismatch was
a computation-path divergence from the eval harness's recomputation —
builder fixed (9 tests green), rebuilt pack at
`outputs/newar-page-eval-v2-2026-07-06-digestfix/` scores with zero
digest warnings and identical CER (0.19918).
Phase 3 done 2026-07-05: config-driven `tools/run_book_ocr.py` landed
(det → script-route → per-script recognizers from
`configs/nepal-language-pack-pipeline-v1.json` → Column Major default →
filter → Unicode/MD export + review queue, resumable, preflight
validation; smoke on 2 rebuilt Limbu pages: 606 lines kept / 25 queued).
The rebuilt Limbu pack is 2× the lost one (16 pages / 4,872 labels).
**ft-v2 retrain result (2026-07-06): new champion candidate.** 40 epochs
on the L4 (~8.5 GPU-h), 46,753 rows (25,065 synth + 2,711 real ×8
replay), PP-OCRv5 mobile generic base, selection on real dev (best epoch
39). On the frozen 902-line test (dates 2082-03-24 / 2082-08-12 /
2082-12-25, by-date disjoint, leakage-gated), scored through the
identical local CPU path as the deployed champion:
**ft-v2 cp CER 0.0047 / grapheme 0.0098 / exact-line 0.859** vs champion
0.0138 / 0.0284 / 0.637 — ~3× lower CER, +22 pts exact-line. The gap is
if anything understated: the old champion's training pack drew on the old
harvest and may have seen these dates; ft-v2's test isolation is proven.
We also now own a warm-startable training checkpoint again
(`outputs/limbu-ft-v2-run-2026-07-06/checkpoints/best_accuracy.*`; export
at `.../export/`). Silver labels, diagnostic, not claim-grade.
**Acceptance gate PASSED (2026-07-06)**: ft-v2 through the full non-oracle
16-page chain (same detector, Column Major, y/conf70 filter, page-eval
v2): **page cp CER 0.0272 / grapheme 0.0347** vs champion 0.043 / 0.067;
matched-line cp CER 0.0090 vs 0.0266 (−66%); det F1 unchanged (0.9650 vs
0.9657, same detector). 3,725 of 4,699 matched lines exact. Gate
artifacts: `outputs/limbu-ftv2-page-gate-2026-07-06/`. Real Gorkhapatra
newspaper pages now read at ~2.7% codepoint error in human-readable
column-major order, silver-diagnostic boundary unchanged. Remaining step:
package ft-v2 (model card, hashes) and swap the Space bundle — a
production-facing change, pending user confirmation.
1. ~~Finish the 2026-07-04 full harvest and inventory~~ **Done 2026-07-04**: 394
language-page hits, 50 languages, inventory manifests at
`outputs/language-pack-inventory-full-2026-07-04/`.
2. ~~Per-language silver packs via the existing `nayanepal` converter~~ **Done
2026-07-04** (`tools/build_language_pack_silver_line_packs.py`, tests green): the
batch-1 run produced **116,287 kept silver lines across 50 languages, zero
errors**, with per-line provenance (PDF hash, page, bbox, raw legacy bytes,
converted text, image hash) at `outputs/language-pack-silver-lines-2026-07-04/`;
spot-checked conversions read as clean, language-plausible Devanagari. The
batch-2 run over the older 589 page hits kept **183,324 more lines (49 languages,
zero errors)** at `outputs/language-pack-silver-lines-batch2-2026-07-04/` —
**299,611 per-language-attributed silver lines total** across the two batches
(~2.4 GB of crops). Run the tool with
`uv run --python 3.11 --with pymupdf --with npttf2utf` — `nepal_ttf2utf` needs the
`npttf2utf` base-map package at runtime, and without it every span fails
(cleanly, with per-span error records, but kept=0). **Honest caveat**: lines
are page-attributed, not language-verified — a language page also carries Nepali
furniture/boxes, and pages whose body is not nayanepal Devanagari (Urdu, the
Tibetan-script pages) yield mostly furniture lines. A language-ID/lexicon filter
before per-language recognizer training is the follow-up gate.
3. **Newar track (second focus language)**: add the Newar registry entry
(Devanagari + Prachalit rows); build the Devanagari-written Newar recognizer
eval/train split from the 6,800 harvested silver lines (by-date split, frozen
test); keep native Prachalit as the separate longer arc — real printed-register
Nepal Lipi harvesting, not more transliteration, is the lever.
4. **Classify Tibetan-script pages** (Hyolmo/Mugal/Bhote) and decide converter vs
native Unicode.
5. ~~Generalize the pipeline wiring~~ **Done 2026-07-04**: all three Limbu-locked
spots generalized (shared script table, filter keys, `--required-script`
preflight), tests green; `configs/nepal-language-pack-pipeline-v1.json` holds the
per-language wiring.
6. **Target-region/furniture filtering stage** for Paddle page/book OCR, scored
against the current baseline on the 8-page Limbu pack and at least one
Devanagari-language pack (Newar is the natural candidate given the focus order).
7. Only after page reconstruction improves, export high-quality real/silver line
crops back into recognizer training, per language.
8. Month-batch the archive back-harvest toward the full ~1,750-article depth, storing
the mirror on the remote 2 TB volume.
## Mistakes To Avoid
Do not use Naya Nepal blog images as training/eval evidence when the PDF exists. Do
not use wide epaper date windows — the endpoint silently caps at 30 PDFs per query;
weekly windows only. Do not treat unnormalized Gorkhapatra labels as distinct
languages (थारु vs थारू). Do not treat the `nayanepal` text layer as Unicode — it is
legacy Preeti-family and needs the converter before any silver label is real. Do not
report Paddle `norm_edit_dis` as CER; real CER comes from `tools/eval_metrics.py`
micro CER over canonical predictions. Do not treat synthetic data as claim evidence.
Do not call Gurung Khema synthetic-by-nature now that official Herald print is
frozen, and do not promote its 749 mechanically eligible candidates to rows before
independent provenance, native label, crop, split, and leakage review. Do not promote the
targeted+Unicode checkpoint or v3 dictionary warm-starts. Do not spend long blocks on
generic eval-harness work unless it directly improves the book/page OCR loop. Do not
interpret a syntactically valid approval/auditor identity as cryptographic proof;
retain independent custody or signed runner evidence for a production claim. Do not
extend the State Portal's Sikkim Herald permission to the separate IPR PDF origin;
use `ipr.sikkim.gov.in` as discovery-only unless a file is mirrored or separately
permitted. Do not frame video-to-page work as OCR scope; reduce video to stills
first. Do not present
page OCR as solved: matched-line recognition is strong, but full-page CER is still
dominated by reconstruction and region selection.
|