prosperious's picture
Add 1000 files (ocr-tech/tests/test_audit_limbu_book_missed_line_review_coverage.py ...)
d82db76 verified
|
Raw History Blame Contribute Delete
12.6 kB

Font Assets

The language registry points to repo-relative runtime font assets under data/fonts/. The data/ tree is ignored because training and validation data can be large, so these files must be provisioned on each runtime host before synthetic rendering or font coverage claims.

Current required assets:

File Purpose Current source used in audit
data/fonts/namdhinggo/Namdhinggo-3.100/Namdhinggo-Regular.ttf Primary Limbu/Sirijonga OCR synthesis and coverage configs/namdhinggo-font-assets-v1.json; SIL Namdhinggo 3.100 release ZIP from https://github.com/silnrsi/font-namdhinggo/releases/download/v3.100/Namdhinggo-3.100.zip; OFL licensed; package SHA-256 073369fbaf064f9075bfbc65e1a76776f449a280c4c367f3cb560bebf70b3915; Regular TTF SHA-256 1b46b16277e4b6784bd19306a9474281888b656e9c57233b4e7dc63bd1229d6c
data/fonts/NotoSansDevanagari-VariableFont_wdth,wght.ttf Devanagari OCR synthesis and coverage for Nepali, Tamang, Magar, Bhojpuri, Maithili, and related Devanagari targets Local Noto Sans Devanagari download on cdjk; copied to k2 runtime data
data/fonts/NotoSansOlChiki-Regular.ttf Santhali/Ol Chiki coverage macOS system font /System/Library/Fonts/Supplemental/NotoSansOlChiki-Regular.ttf; copied to runtime data

Host-local system fonts currently used directly:

File Purpose
/System/Library/Fonts/Supplemental/NotoSansLimbu-Regular.ttf Limbu/Sirijonga fallback coverage
/Library/Fonts/Arial Unicode.ttf or /System/Library/Fonts/Helvetica.ttc Latin/English regression coverage

Namdhinggo notes:

  • SIL describes Namdhinggo as a Unicode Limbu font covering the full Limbu range U+1900..U+194F plus basic Latin.
  • The 3.100 package includes Regular, Medium, SemiBold, Bold, ExtraBold, WOFF, and WOFF2 files.
  • The committed Namdhinggo manifest provisions the five desktop TTF weights and key upstream metadata/license files from the upstream ZIP by archive member path, verifying both package and extracted-member SHA-256 values.
  • SIL also documents legacy Namdhinggo SIL L and Devanagari-to-Limbu TECkit conversion maps. Treat those as data-cleanup leads only; any converted text still needs source, schema, and normalization audits before OCR synthesis.

Namdhinggo reproducible provisioning:

uv run ocrtech prepare-font-assets \
  configs/namdhinggo-font-assets-v1.json \
  --out outputs/font-assets-namdhinggo-2026-06-16

Result on the current runtime:

passed=True
assets=10
downloaded=0
reused=10
failed=0

Namdhinggo readiness gate:

uv run ocrtech audit-font-asset-readiness \
  configs/namdhinggo-font-assets-v1.json \
  outputs/font-assets-namdhinggo-2026-06-16/font-assets-preparation.json \
  outputs/font-inventory-all-runtime-fonts-2026-06-16/font-inventory.json \
  outputs/font-renderability-all-scripts-google-fonts-2026-06-16/font-renderability-audit.json \
  --out outputs/font-asset-readiness-namdhinggo-2026-06-16
passed=True
assets=10
checked=10
issues=0

Validation command:

uv run ocrtech audit-font-renderability languages/nepal-v1.json --out outputs/font-renderability-audit

This audit is cmap coverage only unless --render-samples is used in an environment with Pillow installed.

Inventory command:

uv run ocrtech inventory-fonts \
  languages/nepal-v1.json \
  --root data/fonts \
  --out outputs/font-inventory-current-2026-06-16

Current inventory result:

passed=True
fonts=17
readable=7
archives=0
duplicates=0
warnings=0

The inventory includes desktop fonts and web font files. Only .ttf, .otf, and .ttc files are currently cmap-scored; .woff and .woff2 files are hashed and recorded but not treated as renderability evidence.

LTK archive inventory:

uv run ocrtech inventory-fonts \
  languages/nepal-v1.json \
  --root /Users/cdjk/github/llm/ltk/fonts.zip \
  --out outputs/font-inventory-ltk-fonts-2026-06-16

Result:

passed=True
fonts=2
readable=2
archives=1
duplicates=0
warnings=0

The archive contains janaki.ttf and madan.ttf. Both are cmap-readable, but neither matched the registry Unicode ranges at the 0.95 coverage threshold. Treat them as legacy-font/conversion research leads, not Unicode OCR rendering fonts.

Remote cdjk@100.110.148.43 probe:

/Users/cdjk/github/llm/ltk/fonts.zip
/Users/cdjk/github/llm/g2p/site/assets/8-bit-devanagari.otf
/Users/cdjk/github/llm/twitter-threads/references/fonts/DejaVuSansMNerdFontMono-Regular.ttf
/Users/cdjk/github/llm/twitter-threads/references/fonts/NotoSansDevanagari.ttf
/Users/cdjk/github/llm/twitter-threads/references/fonts/3270NerdFontMono-Regular.ttf

Copied the four direct font files into ignored local runtime data:

data/fonts/remote-cdjk-2026-06-16/

Inventory command:

uv run ocrtech inventory-fonts \
  languages/nepal-v1.json \
  --root data/fonts/remote-cdjk-2026-06-16 \
  --out outputs/font-inventory-remote-cdjk-2026-06-16

Result:

passed=True
fonts=4
readable=4
archives=0
duplicates=0
warnings=0

The remote NotoSansDevanagari.ttf has the same SHA prefix observed for the current runtime Devanagari font and covers all registered Devanagari script targets. 3270NerdFontMono-Regular.ttf and DejaVuSansMNerdFontMono-Regular.ttf match only Latin/English. 8-bit-devanagari.otf is readable but does not match the Unicode Devanagari registry range; keep it in legacy conversion research.

Google Fonts expansion batch:

Reproducible provisioning command:

uv run ocrtech prepare-font-assets \
  configs/nepal-font-assets-v1.json \
  --out outputs/font-assets-google-fonts-2026-06-16

Result on the current runtime:

passed=True
assets=13
downloaded=0
reused=13
failed=0

The manifest includes five font binaries and eight OFL/upstream metadata files. It is the authoritative way to recreate this batch under data/fonts/.

data/fonts/google-fonts-2026-06-16/kanchenjunga/Kanchenjunga-Regular.ttf
data/fonts/google-fonts-2026-06-16/kanchenjunga/Kanchenjunga-Bold.ttf
data/fonts/google-fonts-2026-06-16/notosanstirhuta/NotoSansTirhuta-Regular.ttf
data/fonts/google-fonts-2026-06-16/notoserifdevanagari/NotoSerifDevanagari-VariableFont_wdth,wght.ttf
data/fonts/google-fonts-2026-06-16/notosansolchiki/NotoSansOlChiki-VariableFont_wght.ttf

The batch was downloaded from the upstream Google Fonts repository under ofl/kanchenjunga, ofl/notosanstirhuta, ofl/notoserifdevanagari, and ofl/notosansolchiki, with each source directory's OFL.txt and upstream_info.md copied into ignored runtime data.

Font SHA-256 values:

45609f8cc90d4733d3d1665346d359afc1a659def340d0f369121af034322ef9  Kanchenjunga-Regular.ttf
67a9a29cd49a17832a5eee8d2747fb9a66ee32e49dc52f5dd43887ff24eb0935  Kanchenjunga-Bold.ttf
ad7123ee63118b83ed2f723591e5e861baad8dd157508b8339362850c6036efe  NotoSansTirhuta-Regular.ttf
1191e07bfeb062d80e252eb85b0eafdfbda1e350707a2a60628668e8f677dbbb  NotoSerifDevanagari-VariableFont_wdth,wght.ttf
c9c31988656f49eccec9588825ab3b5045099c2f850ef98f356f976e8a596b4d  NotoSansOlChiki-VariableFont_wght.ttf

Inventory command:

uv run ocrtech inventory-fonts \
  languages/nepal-v1.json \
  --root data/fonts/google-fonts-2026-06-16 \
  --out outputs/font-inventory-google-fonts-2026-06-16

Result:

passed=True
fonts=5
readable=5
archives=0
duplicates=0
warnings=0

Coverage result:

  • Kanchenjunga Regular/Bold: 100% Bantawa/Kirat Rai range coverage.
  • Noto Sans Tirhuta: 100% Maithili/Tirhuta range coverage.
  • Noto Serif Devanagari: 100% coverage for every registered Devanagari script target.
  • Noto Sans Ol Chiki Variable: 100% Santhali/Ol Chiki coverage.

All-script renderability gate after registry promotion:

uv run ocrtech audit-font-renderability \
  languages/nepal-v1.json \
  --all-scripts \
  --out outputs/font-renderability-all-scripts-google-fonts-2026-06-16
passed=True
scripts=22
candidates=58
rendered=0

Font asset readiness gate:

uv run ocrtech audit-font-asset-readiness \
  configs/nepal-font-assets-v1.json \
  outputs/font-assets-google-fonts-2026-06-16/font-assets-preparation.json \
  outputs/font-inventory-all-runtime-fonts-2026-06-16/font-inventory.json \
  outputs/font-renderability-all-scripts-google-fonts-2026-06-16/font-renderability-audit.json \
  --out outputs/font-asset-readiness-google-fonts-2026-06-16
passed=True
assets=13
checked=13
issues=0

This gate ties acquisition, local files, inventory, and renderability together. It SHA-checks every manifest asset and requires inventory membership for font files. License/upstream metadata files are SHA-checked but not required in the font inventory because the inventory intentionally scans font suffixes only.

Training-bound rendered synthesis audits can now require this evidence:

uv run ocrtech audit-rendered-synthesis-lines \
  data/rendered-line-split/eval/manifests/rendered-line-manifest.jsonl \
  --label-file data/rendered-line-split/eval/labels/eval.txt \
  --require-font \
  --font-readiness-report outputs/font-asset-readiness-google-fonts-2026-06-16/font-asset-readiness.json \
  --out outputs/rendered-line-manifest-audit

render-synthesis-text-split also accepts --font-readiness-report; when it is supplied, both train and eval rendered-line audits require every rendered row's metadata.font_sha256 to appear in the passed readiness manifest. This prevents training-bound synthetic OCR bundles from silently using unprovisioned or unaudited fonts.

Current Namdhinggo-rendered Limbu/Sirijonga split passed this gate:

uv run ocrtech audit-rendered-synthesis-lines \
  outputs/sirijonga-limbu-rendered-split-namdhinggo-2026-06-16/train/manifests/rendered-line-manifest.jsonl \
  --label-file outputs/sirijonga-limbu-rendered-split-namdhinggo-2026-06-16/train/labels/train.txt \
  --require-font \
  --font-readiness-report outputs/font-asset-readiness-namdhinggo-2026-06-16/font-asset-readiness.json \
  --out outputs/sirijonga-limbu-rendered-split-namdhinggo-2026-06-16/train-rendered-audit-font-ready

uv run ocrtech audit-rendered-synthesis-lines \
  outputs/sirijonga-limbu-rendered-split-namdhinggo-2026-06-16/eval/manifests/rendered-line-manifest.jsonl \
  --label-file outputs/sirijonga-limbu-rendered-split-namdhinggo-2026-06-16/eval/labels/eval.txt \
  --require-font \
  --font-readiness-report outputs/font-asset-readiness-namdhinggo-2026-06-16/font-asset-readiness.json \
  --out outputs/sirijonga-limbu-rendered-split-namdhinggo-2026-06-16/eval-rendered-audit-font-ready
train: passed=True samples=150 claim_eligible=0 issues=0 warnings=0
eval:  passed=True samples=27 claim_eligible=0 issues=0 warnings=0

Collection policy:

  • Every collected font source needs file path, original source URL or local source path, license, SHA-256, and cmap/script coverage.
  • Keep font binaries under ignored data/fonts/; commit only the registry, source notes, hashes, and audit commands/results.
  • Treat all legacy/non-Unicode fonts as conversion leads only. They cannot be used for OCR labels until the encoding mapping is verified against real text.
  • Use multiple font weights/styles for synthesis diversity only after the primary regular face passes renderability and label audits.

Next font collection targets:

  • Limbu/Sirijonga: Namdhinggo full family is present; inspect legacy Namdhinggo SIL L conversion assets separately.
  • Devanagari: collect more families used in Nepali newspapers and textbooks, not just Noto Sans Devanagari.
  • Ol Chiki/Santhali: add Noto Serif or other print-style alternatives.
  • Kirat Rai: locate Unicode 16.0-compatible fonts before treating the script as trainable. Kanchenjunga now passes cmap coverage, but Kirat Rai still needs real text, labels, and validation before training claims.
  • Tibetan/Tamyig/Sherpa-related paths: audit actual Unicode usage and fonts before adding synthesis labels.
  • Local LTK font archive: inventoried. Keep janaki.ttf and madan.ttf out of Unicode synthesis until their encoding/conversion behavior is verified.
  • Remote cdjk direct font files: inventoried. Promote only the duplicate Noto Sans Devanagari if needed for provisioning; keep the 8-bit Devanagari font as legacy/conversion evidence and the Nerd fonts as Latin-only utilities.
  • Google Fonts expansion: inventoried and promoted as font candidates for Kirat Rai, Tirhuta, print-style Devanagari, and Ol Chiki diversity.