|
Download ocr-tech/docs/SUNUWAR_CONVERTER_STATUS.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 6.94 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/SUNUWAR_CONVERTER_STATUS.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/ocr-tech/docs/SUNUWAR_CONVERTER_STATUS.md
-
curl -L -o SUNUWAR_CONVERTER_STATUS.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/SUNUWAR_CONVERTER_STATUS.md
6.94 kB
| # Sikkim Herald Sunuwar legacy converter status | |
| Status: **converter mapping complete for the proven `koits` / `kirat1` | |
| layout; the selected 2024-04-12 and 2025-01-26 editions pass exact font proof.** | |
| The earlier conclusion that no faithful converter existed is obsolete. The | |
| project derived the legacy-byte map from exact embedded-font evidence and | |
| resolved the final previously uncertain byte, `|`, as U+11BC5 SUNUWAR LETTER | |
| UTTHI. The correction is implemented in `src/ocrtech/sunuwar_legacy.py` and is | |
| packaged by commit `68528a5`. Commit `6a89292` adds the two extraction-level | |
| corrections established by the 2024/2025 audit: empty backslash normalizes to | |
| space, and the one U+201D value whose printed glyph is a colon normalizes to | |
| ASCII colon. | |
| This does not automatically admit every PDF whose resource happens to be named | |
| `F2`, `koits`, or `kirat1`. PDF font names are hints, not identities. Every new | |
| edition must bind the converter route to its full PDF hash and an embedded font | |
| program or glyph-outline/cmap fingerprint before any converted text can be used | |
| as a label. | |
| ## Proven mapping | |
| The Mukhia editions investigated here store visible Sunuwar as a legacy Latin | |
| byte stream. The `koits` and `kirat1` font programs were shown to use the same | |
| byte-to-glyph layout. The map contains: | |
| - 10 digits, `0` through `9`, mapped in order to U+11BF0..U+11BF9; | |
| - all observed Sunuwar letter bytes, including the eight second-pass letters | |
| `v q x r u g h j`; | |
| - `| → U+11BC5` UTTHI; | |
| - empty `\\ → U+0020` normalization; | |
| - the hash- and bbox-bound 2025 extraction correction `U+201D → U+003A`; | |
| - an explicit literal-punctuation set; and | |
| - fail-closed reporting for every other byte. | |
| `SUNUWAR_LETTERS_UNCERTAIN` remains as an empty compatibility constant. | |
| `apply_uncertain=True` is therefore a compatibility no-op. The strict API | |
| rejects any unknown byte, including unknown ASCII, instead of allowing a | |
| valid-looking mixed-script string to pass. | |
| ## Final UTTHI evidence | |
| The legacy `|` glyph is the flowing open-2 Sikkim form of UTTHI. The evidence | |
| record in the released converter repository reports: | |
| - 2,748 occurrences across eight PDFs; | |
| - a 600-dpi corpus crop compared against separately labelled Sikkim UTTHI and | |
| SHYELE forms; | |
| - normalized largest-component IoU 0.7395 against Sikkim UTTHI and 0.3681 | |
| against Sikkim SHYELE; and | |
| - the independently mapped corpus form `t|v|`, which reads `m-u-r-u` under | |
| `t → MA`, `| → UTTHI`, and `v → REU`. | |
| Primary references: | |
| - Richard Ishida's reviewed [Sunuwar orthography notes](https://r12a.github.io/scripts/sunu/suz.html#writingstyles), including labelled Sikkim forms. | |
| - Unicode proposal [L2/21-157R](https://www.unicode.org/L2/L2021/21157r-sunuwar.pdf), which identifies UTTHI `/u/`. | |
| - Unicode font comparison [L2/24-022](https://www.unicode.org/L2/L2024/24022-sunuwar-font-comp.pdf), which documents regional glyph differences. | |
| - The released [`nepal-ttf2utf` evidence record](https://github.com/Ampixa/nepal-ttf2utf/blob/main/docs/EVIDENCE.md) and converter implementation. | |
| The local derivation artifacts remain under | |
| `outputs/sunuwar-map-derivation/`. They include the embedded-font comparisons, | |
| printed-crop checks, and round-trip panels used before the final regional-form | |
| adjudication. | |
| ## Selected-edition attestation (2026-07-15 r11b) | |
| The full-hash 2024-04-12 and 2025-01-26 Mukhia editions are 289 days apart. | |
| Their embedded `kirat1` programs have SHA-256 | |
| `a56a09338742c2563bcc355b3c701cd96093d813c3fa05582e131331e0783f42` | |
| and `45af39726717d8b6cd211da6a1966e706d85dd60408d4ef3c36e0dad2db3c22d`. | |
| Every observed mapped script code matches the program extracted from the | |
| released converter's cited `sunuwar_01348c63.pdf` derivation source by exact | |
| decomposed outline commands and advance width: 38/38 per edition, 76 summed, | |
| with no mismatch. The reference PDF/font hashes are | |
| `b3c001ff428de3b4e539c88d6d93566ae969a46b9ba947fe680fc1b68eb10456` | |
| and `484550543232f587fab3a5b3e900c922845c5ed44ef7bdd493cace4a5b6bce78`. | |
| The semantic map provenance is pinned to MIT-licensed `nepal-ttf2utf` 0.2.0 | |
| commit `a158a8cbdbe230df6849afd9c9736bf31cb9c24b`. | |
| Both editions reconcile at 100% aggregate converter coverage. This attests | |
| font-layout compatibility only. It stores no converted strings and supplies no | |
| native transcription, line label, training row, evaluation row, or CER. | |
| ## Required per-edition gate | |
| Before applying this converter to a newly acquired PDF, record all of the | |
| following: | |
| 1. Canonical source URL, retrieval time, complete PDF SHA-256, byte count, page | |
| count, and printed issue identity. | |
| 2. Embedded font stream SHA-256 and the PDF object/xref that supplies every | |
| target-bearing span. | |
| 3. A normalized code/CID-to-outline-command, advance-width, and cmap/ToUnicode | |
| fingerprint. Generic names such as `F2` are insufficient. | |
| 4. Exact compatibility of every observed source code with the proven layout. | |
| A partial visual similarity or block-valid output is not sufficient. | |
| 5. Raw and shadow-deduplicated source-unit counts reconciled as mapped script, | |
| verified literal, verified blank/whitespace, or unresolved. | |
| 6. An aggregate coverage report that stores counts and hashes only. Do not | |
| persist converted strings, crops, labels, or evaluation rows at this stage. | |
| If any target-bearing font object fails the layout proof, classify it as | |
| `unsupported_layout`. Do not call that a converter failure and do not try a | |
| different converter merely because the output looks plausible. | |
| ## Unicode validity boundary | |
| The local Python runtime uses Unicode 15.1 and treats the Unicode 16 Sunuwar | |
| characters as unassigned. Consequently, `unicodedata.name`, broad block | |
| membership, NFC, and successful rendering are not acceptance evidence by | |
| themselves. A coverage audit must constrain output to the exact pinned map | |
| target set, reject replacement/private-use/control/residual-Latin values, and | |
| require nonzero mapped Sunuwar content. | |
| These checks establish encoding compatibility, not semantic transcription. | |
| Native pixel/label review remains mandatory before any row can enter a frozen | |
| train or evaluation set. | |
| ## Current source status | |
| The bounded 2026-07-15 State Portal catalog contains 11 Mukhia/Sunuwar URLs. | |
| The selected 2024-04-12 and 2025-01-26 editions now have full hashes, printed | |
| identity records, all-page reviewer-1 provenance decisions, and embedded-font | |
| layout proof. They remain unadmitted because independent provenance reviewer 2, | |
| adjudication, and native-script pixel/label review do not exist. They contribute | |
| zero train rows, zero evaluation rows, and no claim evidence. | |
| The old raw-span extractor and corpus builders still contain historical prose | |
| and generic font-name routing. They must not be used for admission until they | |
| are wired to the fingerprint gate and the completed converter. Their old | |
| `RAW_ONLY_NO_FAITHFUL_CONVERTER` wording describes an earlier investigation, | |
| not the present mapping status. | |