ocr-projects-archive / ocr-tech /docs /SUNUWAR_CONVERTER_STATUS.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
6.94 kB

Sikkim Herald Sunuwar legacy converter status

Status: converter mapping complete for the proven koits / kirat1 layout; the selected 2024-04-12 and 2025-01-26 editions pass exact font proof.

The earlier conclusion that no faithful converter existed is obsolete. The project derived the legacy-byte map from exact embedded-font evidence and resolved the final previously uncertain byte, |, as U+11BC5 SUNUWAR LETTER UTTHI. The correction is implemented in src/ocrtech/sunuwar_legacy.py and is packaged by commit 68528a5. Commit 6a89292 adds the two extraction-level corrections established by the 2024/2025 audit: empty backslash normalizes to space, and the one U+201D value whose printed glyph is a colon normalizes to ASCII colon.

This does not automatically admit every PDF whose resource happens to be named F2, koits, or kirat1. PDF font names are hints, not identities. Every new edition must bind the converter route to its full PDF hash and an embedded font program or glyph-outline/cmap fingerprint before any converted text can be used as a label.

Proven mapping

The Mukhia editions investigated here store visible Sunuwar as a legacy Latin byte stream. The koits and kirat1 font programs were shown to use the same byte-to-glyph layout. The map contains:

  • 10 digits, 0 through 9, mapped in order to U+11BF0..U+11BF9;
  • all observed Sunuwar letter bytes, including the eight second-pass letters v q x r u g h j;
  • | → U+11BC5 UTTHI;
  • empty \\ → U+0020 normalization;
  • the hash- and bbox-bound 2025 extraction correction U+201D → U+003A;
  • an explicit literal-punctuation set; and
  • fail-closed reporting for every other byte.

SUNUWAR_LETTERS_UNCERTAIN remains as an empty compatibility constant. apply_uncertain=True is therefore a compatibility no-op. The strict API rejects any unknown byte, including unknown ASCII, instead of allowing a valid-looking mixed-script string to pass.

Final UTTHI evidence

The legacy | glyph is the flowing open-2 Sikkim form of UTTHI. The evidence record in the released converter repository reports:

  • 2,748 occurrences across eight PDFs;
  • a 600-dpi corpus crop compared against separately labelled Sikkim UTTHI and SHYELE forms;
  • normalized largest-component IoU 0.7395 against Sikkim UTTHI and 0.3681 against Sikkim SHYELE; and
  • the independently mapped corpus form t|v|, which reads m-u-r-u under t → MA, | → UTTHI, and v → REU.

Primary references:

The local derivation artifacts remain under outputs/sunuwar-map-derivation/. They include the embedded-font comparisons, printed-crop checks, and round-trip panels used before the final regional-form adjudication.

Selected-edition attestation (2026-07-15 r11b)

The full-hash 2024-04-12 and 2025-01-26 Mukhia editions are 289 days apart. Their embedded kirat1 programs have SHA-256 a56a09338742c2563bcc355b3c701cd96093d813c3fa05582e131331e0783f42 and 45af39726717d8b6cd211da6a1966e706d85dd60408d4ef3c36e0dad2db3c22d. Every observed mapped script code matches the program extracted from the released converter's cited sunuwar_01348c63.pdf derivation source by exact decomposed outline commands and advance width: 38/38 per edition, 76 summed, with no mismatch. The reference PDF/font hashes are b3c001ff428de3b4e539c88d6d93566ae969a46b9ba947fe680fc1b68eb10456 and 484550543232f587fab3a5b3e900c922845c5ed44ef7bdd493cace4a5b6bce78. The semantic map provenance is pinned to MIT-licensed nepal-ttf2utf 0.2.0 commit a158a8cbdbe230df6849afd9c9736bf31cb9c24b.

Both editions reconcile at 100% aggregate converter coverage. This attests font-layout compatibility only. It stores no converted strings and supplies no native transcription, line label, training row, evaluation row, or CER.

Required per-edition gate

Before applying this converter to a newly acquired PDF, record all of the following:

  1. Canonical source URL, retrieval time, complete PDF SHA-256, byte count, page count, and printed issue identity.
  2. Embedded font stream SHA-256 and the PDF object/xref that supplies every target-bearing span.
  3. A normalized code/CID-to-outline-command, advance-width, and cmap/ToUnicode fingerprint. Generic names such as F2 are insufficient.
  4. Exact compatibility of every observed source code with the proven layout. A partial visual similarity or block-valid output is not sufficient.
  5. Raw and shadow-deduplicated source-unit counts reconciled as mapped script, verified literal, verified blank/whitespace, or unresolved.
  6. An aggregate coverage report that stores counts and hashes only. Do not persist converted strings, crops, labels, or evaluation rows at this stage.

If any target-bearing font object fails the layout proof, classify it as unsupported_layout. Do not call that a converter failure and do not try a different converter merely because the output looks plausible.

Unicode validity boundary

The local Python runtime uses Unicode 15.1 and treats the Unicode 16 Sunuwar characters as unassigned. Consequently, unicodedata.name, broad block membership, NFC, and successful rendering are not acceptance evidence by themselves. A coverage audit must constrain output to the exact pinned map target set, reject replacement/private-use/control/residual-Latin values, and require nonzero mapped Sunuwar content.

These checks establish encoding compatibility, not semantic transcription. Native pixel/label review remains mandatory before any row can enter a frozen train or evaluation set.

Current source status

The bounded 2026-07-15 State Portal catalog contains 11 Mukhia/Sunuwar URLs. The selected 2024-04-12 and 2025-01-26 editions now have full hashes, printed identity records, all-page reviewer-1 provenance decisions, and embedded-font layout proof. They remain unadmitted because independent provenance reviewer 2, adjudication, and native-script pixel/label review do not exist. They contribute zero train rows, zero evaluation rows, and no claim evidence.

The old raw-span extractor and corpus builders still contain historical prose and generic font-name routing. They must not be used for admission until they are wired to the fingerprint gate and the completed converter. Their old RAW_ONLY_NO_FAITHFUL_CONVERTER wording describes an earlier investigation, not the present mapping status.