ocr-projects-archive / ocr-tech /docs /SUNUWAR_CONVERTER_STATUS.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
6.94 kB
# Sikkim Herald Sunuwar legacy converter status
Status: **converter mapping complete for the proven `koits` / `kirat1`
layout; the selected 2024-04-12 and 2025-01-26 editions pass exact font proof.**
The earlier conclusion that no faithful converter existed is obsolete. The
project derived the legacy-byte map from exact embedded-font evidence and
resolved the final previously uncertain byte, `|`, as U+11BC5 SUNUWAR LETTER
UTTHI. The correction is implemented in `src/ocrtech/sunuwar_legacy.py` and is
packaged by commit `68528a5`. Commit `6a89292` adds the two extraction-level
corrections established by the 2024/2025 audit: empty backslash normalizes to
space, and the one U+201D value whose printed glyph is a colon normalizes to
ASCII colon.
This does not automatically admit every PDF whose resource happens to be named
`F2`, `koits`, or `kirat1`. PDF font names are hints, not identities. Every new
edition must bind the converter route to its full PDF hash and an embedded font
program or glyph-outline/cmap fingerprint before any converted text can be used
as a label.
## Proven mapping
The Mukhia editions investigated here store visible Sunuwar as a legacy Latin
byte stream. The `koits` and `kirat1` font programs were shown to use the same
byte-to-glyph layout. The map contains:
- 10 digits, `0` through `9`, mapped in order to U+11BF0..U+11BF9;
- all observed Sunuwar letter bytes, including the eight second-pass letters
`v q x r u g h j`;
- `| → U+11BC5` UTTHI;
- empty `\\ → U+0020` normalization;
- the hash- and bbox-bound 2025 extraction correction `U+201D → U+003A`;
- an explicit literal-punctuation set; and
- fail-closed reporting for every other byte.
`SUNUWAR_LETTERS_UNCERTAIN` remains as an empty compatibility constant.
`apply_uncertain=True` is therefore a compatibility no-op. The strict API
rejects any unknown byte, including unknown ASCII, instead of allowing a
valid-looking mixed-script string to pass.
## Final UTTHI evidence
The legacy `|` glyph is the flowing open-2 Sikkim form of UTTHI. The evidence
record in the released converter repository reports:
- 2,748 occurrences across eight PDFs;
- a 600-dpi corpus crop compared against separately labelled Sikkim UTTHI and
SHYELE forms;
- normalized largest-component IoU 0.7395 against Sikkim UTTHI and 0.3681
against Sikkim SHYELE; and
- the independently mapped corpus form `t|v|`, which reads `m-u-r-u` under
`t → MA`, `| → UTTHI`, and `v → REU`.
Primary references:
- Richard Ishida's reviewed [Sunuwar orthography notes](https://r12a.github.io/scripts/sunu/suz.html#writingstyles), including labelled Sikkim forms.
- Unicode proposal [L2/21-157R](https://www.unicode.org/L2/L2021/21157r-sunuwar.pdf), which identifies UTTHI `/u/`.
- Unicode font comparison [L2/24-022](https://www.unicode.org/L2/L2024/24022-sunuwar-font-comp.pdf), which documents regional glyph differences.
- The released [`nepal-ttf2utf` evidence record](https://github.com/Ampixa/nepal-ttf2utf/blob/main/docs/EVIDENCE.md) and converter implementation.
The local derivation artifacts remain under
`outputs/sunuwar-map-derivation/`. They include the embedded-font comparisons,
printed-crop checks, and round-trip panels used before the final regional-form
adjudication.
## Selected-edition attestation (2026-07-15 r11b)
The full-hash 2024-04-12 and 2025-01-26 Mukhia editions are 289 days apart.
Their embedded `kirat1` programs have SHA-256
`a56a09338742c2563bcc355b3c701cd96093d813c3fa05582e131331e0783f42`
and `45af39726717d8b6cd211da6a1966e706d85dd60408d4ef3c36e0dad2db3c22d`.
Every observed mapped script code matches the program extracted from the
released converter's cited `sunuwar_01348c63.pdf` derivation source by exact
decomposed outline commands and advance width: 38/38 per edition, 76 summed,
with no mismatch. The reference PDF/font hashes are
`b3c001ff428de3b4e539c88d6d93566ae969a46b9ba947fe680fc1b68eb10456`
and `484550543232f587fab3a5b3e900c922845c5ed44ef7bdd493cace4a5b6bce78`.
The semantic map provenance is pinned to MIT-licensed `nepal-ttf2utf` 0.2.0
commit `a158a8cbdbe230df6849afd9c9736bf31cb9c24b`.
Both editions reconcile at 100% aggregate converter coverage. This attests
font-layout compatibility only. It stores no converted strings and supplies no
native transcription, line label, training row, evaluation row, or CER.
## Required per-edition gate
Before applying this converter to a newly acquired PDF, record all of the
following:
1. Canonical source URL, retrieval time, complete PDF SHA-256, byte count, page
count, and printed issue identity.
2. Embedded font stream SHA-256 and the PDF object/xref that supplies every
target-bearing span.
3. A normalized code/CID-to-outline-command, advance-width, and cmap/ToUnicode
fingerprint. Generic names such as `F2` are insufficient.
4. Exact compatibility of every observed source code with the proven layout.
A partial visual similarity or block-valid output is not sufficient.
5. Raw and shadow-deduplicated source-unit counts reconciled as mapped script,
verified literal, verified blank/whitespace, or unresolved.
6. An aggregate coverage report that stores counts and hashes only. Do not
persist converted strings, crops, labels, or evaluation rows at this stage.
If any target-bearing font object fails the layout proof, classify it as
`unsupported_layout`. Do not call that a converter failure and do not try a
different converter merely because the output looks plausible.
## Unicode validity boundary
The local Python runtime uses Unicode 15.1 and treats the Unicode 16 Sunuwar
characters as unassigned. Consequently, `unicodedata.name`, broad block
membership, NFC, and successful rendering are not acceptance evidence by
themselves. A coverage audit must constrain output to the exact pinned map
target set, reject replacement/private-use/control/residual-Latin values, and
require nonzero mapped Sunuwar content.
These checks establish encoding compatibility, not semantic transcription.
Native pixel/label review remains mandatory before any row can enter a frozen
train or evaluation set.
## Current source status
The bounded 2026-07-15 State Portal catalog contains 11 Mukhia/Sunuwar URLs.
The selected 2024-04-12 and 2025-01-26 editions now have full hashes, printed
identity records, all-page reviewer-1 provenance decisions, and embedded-font
layout proof. They remain unadmitted because independent provenance reviewer 2,
adjudication, and native-script pixel/label review do not exist. They contribute
zero train rows, zero evaluation rows, and no claim evidence.
The old raw-span extractor and corpus builders still contain historical prose
and generic font-name routing. They must not be used for admission until they
are wired to the fingerprint gate and the completed converter. Their old
`RAW_ONLY_NO_FAITHFUL_CONVERTER` wording describes an earlier investigation,
not the present mapping status.