Download ocr-tech/docs/SUNUWAR_CONVERTER_STATUS.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 6.94 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/SUNUWAR_CONVERTER_STATUS.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/ocr-tech/docs/SUNUWAR_CONVERTER_STATUS.md
-
curl -L -o SUNUWAR_CONVERTER_STATUS.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/SUNUWAR_CONVERTER_STATUS.md
Sikkim Herald Sunuwar legacy converter status
Status: converter mapping complete for the proven koits / kirat1
layout; the selected 2024-04-12 and 2025-01-26 editions pass exact font proof.
The earlier conclusion that no faithful converter existed is obsolete. The
project derived the legacy-byte map from exact embedded-font evidence and
resolved the final previously uncertain byte, |, as U+11BC5 SUNUWAR LETTER
UTTHI. The correction is implemented in src/ocrtech/sunuwar_legacy.py and is
packaged by commit 68528a5. Commit 6a89292 adds the two extraction-level
corrections established by the 2024/2025 audit: empty backslash normalizes to
space, and the one U+201D value whose printed glyph is a colon normalizes to
ASCII colon.
This does not automatically admit every PDF whose resource happens to be named
F2, koits, or kirat1. PDF font names are hints, not identities. Every new
edition must bind the converter route to its full PDF hash and an embedded font
program or glyph-outline/cmap fingerprint before any converted text can be used
as a label.
Proven mapping
The Mukhia editions investigated here store visible Sunuwar as a legacy Latin
byte stream. The koits and kirat1 font programs were shown to use the same
byte-to-glyph layout. The map contains:
- 10 digits,
0through9, mapped in order to U+11BF0..U+11BF9; - all observed Sunuwar letter bytes, including the eight second-pass letters
v q x r u g h j; | → U+11BC5UTTHI;- empty
\\ → U+0020normalization; - the hash- and bbox-bound 2025 extraction correction
U+201D → U+003A; - an explicit literal-punctuation set; and
- fail-closed reporting for every other byte.
SUNUWAR_LETTERS_UNCERTAIN remains as an empty compatibility constant.
apply_uncertain=True is therefore a compatibility no-op. The strict API
rejects any unknown byte, including unknown ASCII, instead of allowing a
valid-looking mixed-script string to pass.
Final UTTHI evidence
The legacy | glyph is the flowing open-2 Sikkim form of UTTHI. The evidence
record in the released converter repository reports:
- 2,748 occurrences across eight PDFs;
- a 600-dpi corpus crop compared against separately labelled Sikkim UTTHI and SHYELE forms;
- normalized largest-component IoU 0.7395 against Sikkim UTTHI and 0.3681 against Sikkim SHYELE; and
- the independently mapped corpus form
t|v|, which readsm-u-r-uundert → MA,| → UTTHI, andv → REU.
Primary references:
- Richard Ishida's reviewed Sunuwar orthography notes, including labelled Sikkim forms.
- Unicode proposal L2/21-157R, which identifies UTTHI
/u/. - Unicode font comparison L2/24-022, which documents regional glyph differences.
- The released
nepal-ttf2utfevidence record and converter implementation.
The local derivation artifacts remain under
outputs/sunuwar-map-derivation/. They include the embedded-font comparisons,
printed-crop checks, and round-trip panels used before the final regional-form
adjudication.
Selected-edition attestation (2026-07-15 r11b)
The full-hash 2024-04-12 and 2025-01-26 Mukhia editions are 289 days apart.
Their embedded kirat1 programs have SHA-256
a56a09338742c2563bcc355b3c701cd96093d813c3fa05582e131331e0783f42
and 45af39726717d8b6cd211da6a1966e706d85dd60408d4ef3c36e0dad2db3c22d.
Every observed mapped script code matches the program extracted from the
released converter's cited sunuwar_01348c63.pdf derivation source by exact
decomposed outline commands and advance width: 38/38 per edition, 76 summed,
with no mismatch. The reference PDF/font hashes are
b3c001ff428de3b4e539c88d6d93566ae969a46b9ba947fe680fc1b68eb10456
and 484550543232f587fab3a5b3e900c922845c5ed44ef7bdd493cace4a5b6bce78.
The semantic map provenance is pinned to MIT-licensed nepal-ttf2utf 0.2.0
commit a158a8cbdbe230df6849afd9c9736bf31cb9c24b.
Both editions reconcile at 100% aggregate converter coverage. This attests font-layout compatibility only. It stores no converted strings and supplies no native transcription, line label, training row, evaluation row, or CER.
Required per-edition gate
Before applying this converter to a newly acquired PDF, record all of the following:
- Canonical source URL, retrieval time, complete PDF SHA-256, byte count, page count, and printed issue identity.
- Embedded font stream SHA-256 and the PDF object/xref that supplies every target-bearing span.
- A normalized code/CID-to-outline-command, advance-width, and cmap/ToUnicode
fingerprint. Generic names such as
F2are insufficient. - Exact compatibility of every observed source code with the proven layout. A partial visual similarity or block-valid output is not sufficient.
- Raw and shadow-deduplicated source-unit counts reconciled as mapped script, verified literal, verified blank/whitespace, or unresolved.
- An aggregate coverage report that stores counts and hashes only. Do not persist converted strings, crops, labels, or evaluation rows at this stage.
If any target-bearing font object fails the layout proof, classify it as
unsupported_layout. Do not call that a converter failure and do not try a
different converter merely because the output looks plausible.
Unicode validity boundary
The local Python runtime uses Unicode 15.1 and treats the Unicode 16 Sunuwar
characters as unassigned. Consequently, unicodedata.name, broad block
membership, NFC, and successful rendering are not acceptance evidence by
themselves. A coverage audit must constrain output to the exact pinned map
target set, reject replacement/private-use/control/residual-Latin values, and
require nonzero mapped Sunuwar content.
These checks establish encoding compatibility, not semantic transcription. Native pixel/label review remains mandatory before any row can enter a frozen train or evaluation set.
Current source status
The bounded 2026-07-15 State Portal catalog contains 11 Mukhia/Sunuwar URLs. The selected 2024-04-12 and 2025-01-26 editions now have full hashes, printed identity records, all-page reviewer-1 provenance decisions, and embedded-font layout proof. They remain unadmitted because independent provenance reviewer 2, adjudication, and native-script pixel/label review do not exist. They contribute zero train rows, zero evaluation rows, and no claim evidence.
The old raw-span extractor and corpus builders still contain historical prose
and generic font-name routing. They must not be used for admission until they
are wired to the fingerprint gate and the completed converter. Their old
RAW_ONLY_NO_FAITHFUL_CONVERTER wording describes an earlier investigation,
not the present mapping status.