Add 1000 files (ocr-tech/tests/test_audit_limbu_book_missed_line_review_coverage.py ...)
d82db76 verified |
Download ocr-tech-mps-integration/docs/FONT_ASSETS.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 12.6 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech-mps-integration/docs/FONT_ASSETS.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/ocr-tech-mps-integration/docs/FONT_ASSETS.md
-
curl -L -o FONT_ASSETS.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech-mps-integration/docs/FONT_ASSETS.md
12.6 kB
| # Font Assets | |
| The language registry points to repo-relative runtime font assets under | |
| `data/fonts/`. The `data/` tree is ignored because training and validation data | |
| can be large, so these files must be provisioned on each runtime host before | |
| synthetic rendering or font coverage claims. | |
| Current required assets: | |
| | File | Purpose | Current source used in audit | | |
| | --- | --- | --- | | |
| | `data/fonts/namdhinggo/Namdhinggo-3.100/Namdhinggo-Regular.ttf` | Primary Limbu/Sirijonga OCR synthesis and coverage | `configs/namdhinggo-font-assets-v1.json`; SIL Namdhinggo 3.100 release ZIP from <https://github.com/silnrsi/font-namdhinggo/releases/download/v3.100/Namdhinggo-3.100.zip>; OFL licensed; package SHA-256 `073369fbaf064f9075bfbc65e1a76776f449a280c4c367f3cb560bebf70b3915`; Regular TTF SHA-256 `1b46b16277e4b6784bd19306a9474281888b656e9c57233b4e7dc63bd1229d6c` | | |
| | `data/fonts/NotoSansDevanagari-VariableFont_wdth,wght.ttf` | Devanagari OCR synthesis and coverage for Nepali, Tamang, Magar, Bhojpuri, Maithili, and related Devanagari targets | Local Noto Sans Devanagari download on `cdjk`; copied to k2 runtime data | | |
| | `data/fonts/NotoSansOlChiki-Regular.ttf` | Santhali/Ol Chiki coverage | macOS system font `/System/Library/Fonts/Supplemental/NotoSansOlChiki-Regular.ttf`; copied to runtime data | | |
| Host-local system fonts currently used directly: | |
| | File | Purpose | | |
| | --- | --- | | |
| | `/System/Library/Fonts/Supplemental/NotoSansLimbu-Regular.ttf` | Limbu/Sirijonga fallback coverage | | |
| | `/Library/Fonts/Arial Unicode.ttf` or `/System/Library/Fonts/Helvetica.ttc` | Latin/English regression coverage | | |
| Namdhinggo notes: | |
| - SIL describes Namdhinggo as a Unicode Limbu font covering the full Limbu | |
| range `U+1900..U+194F` plus basic Latin. | |
| - The 3.100 package includes Regular, Medium, SemiBold, Bold, ExtraBold, WOFF, | |
| and WOFF2 files. | |
| - The committed Namdhinggo manifest provisions the five desktop TTF weights and | |
| key upstream metadata/license files from the upstream ZIP by archive member | |
| path, verifying both package and extracted-member SHA-256 values. | |
| - SIL also documents legacy Namdhinggo SIL L and Devanagari-to-Limbu TECkit | |
| conversion maps. Treat those as data-cleanup leads only; any converted text | |
| still needs source, schema, and normalization audits before OCR synthesis. | |
| Namdhinggo reproducible provisioning: | |
| ```bash | |
| uv run ocrtech prepare-font-assets \ | |
| configs/namdhinggo-font-assets-v1.json \ | |
| --out outputs/font-assets-namdhinggo-2026-06-16 | |
| ``` | |
| Result on the current runtime: | |
| ```text | |
| passed=True | |
| assets=10 | |
| downloaded=0 | |
| reused=10 | |
| failed=0 | |
| ``` | |
| Namdhinggo readiness gate: | |
| ```bash | |
| uv run ocrtech audit-font-asset-readiness \ | |
| configs/namdhinggo-font-assets-v1.json \ | |
| outputs/font-assets-namdhinggo-2026-06-16/font-assets-preparation.json \ | |
| outputs/font-inventory-all-runtime-fonts-2026-06-16/font-inventory.json \ | |
| outputs/font-renderability-all-scripts-google-fonts-2026-06-16/font-renderability-audit.json \ | |
| --out outputs/font-asset-readiness-namdhinggo-2026-06-16 | |
| ``` | |
| ```text | |
| passed=True | |
| assets=10 | |
| checked=10 | |
| issues=0 | |
| ``` | |
| Validation command: | |
| ```bash | |
| uv run ocrtech audit-font-renderability languages/nepal-v1.json --out outputs/font-renderability-audit | |
| ``` | |
| This audit is cmap coverage only unless `--render-samples` is used in an | |
| environment with Pillow installed. | |
| Inventory command: | |
| ```bash | |
| uv run ocrtech inventory-fonts \ | |
| languages/nepal-v1.json \ | |
| --root data/fonts \ | |
| --out outputs/font-inventory-current-2026-06-16 | |
| ``` | |
| Current inventory result: | |
| ```text | |
| passed=True | |
| fonts=17 | |
| readable=7 | |
| archives=0 | |
| duplicates=0 | |
| warnings=0 | |
| ``` | |
| The inventory includes desktop fonts and web font files. Only `.ttf`, `.otf`, | |
| and `.ttc` files are currently cmap-scored; `.woff` and `.woff2` files are | |
| hashed and recorded but not treated as renderability evidence. | |
| LTK archive inventory: | |
| ```bash | |
| uv run ocrtech inventory-fonts \ | |
| languages/nepal-v1.json \ | |
| --root /Users/cdjk/github/llm/ltk/fonts.zip \ | |
| --out outputs/font-inventory-ltk-fonts-2026-06-16 | |
| ``` | |
| Result: | |
| ```text | |
| passed=True | |
| fonts=2 | |
| readable=2 | |
| archives=1 | |
| duplicates=0 | |
| warnings=0 | |
| ``` | |
| The archive contains `janaki.ttf` and `madan.ttf`. Both are cmap-readable, but | |
| neither matched the registry Unicode ranges at the `0.95` coverage threshold. | |
| Treat them as legacy-font/conversion research leads, not Unicode OCR rendering | |
| fonts. | |
| Remote `cdjk@100.110.148.43` probe: | |
| ```text | |
| /Users/cdjk/github/llm/ltk/fonts.zip | |
| /Users/cdjk/github/llm/g2p/site/assets/8-bit-devanagari.otf | |
| /Users/cdjk/github/llm/twitter-threads/references/fonts/DejaVuSansMNerdFontMono-Regular.ttf | |
| /Users/cdjk/github/llm/twitter-threads/references/fonts/NotoSansDevanagari.ttf | |
| /Users/cdjk/github/llm/twitter-threads/references/fonts/3270NerdFontMono-Regular.ttf | |
| ``` | |
| Copied the four direct font files into ignored local runtime data: | |
| ```text | |
| data/fonts/remote-cdjk-2026-06-16/ | |
| ``` | |
| Inventory command: | |
| ```bash | |
| uv run ocrtech inventory-fonts \ | |
| languages/nepal-v1.json \ | |
| --root data/fonts/remote-cdjk-2026-06-16 \ | |
| --out outputs/font-inventory-remote-cdjk-2026-06-16 | |
| ``` | |
| Result: | |
| ```text | |
| passed=True | |
| fonts=4 | |
| readable=4 | |
| archives=0 | |
| duplicates=0 | |
| warnings=0 | |
| ``` | |
| The remote `NotoSansDevanagari.ttf` has the same SHA prefix observed for the | |
| current runtime Devanagari font and covers all registered Devanagari script | |
| targets. `3270NerdFontMono-Regular.ttf` and `DejaVuSansMNerdFontMono-Regular.ttf` | |
| match only Latin/English. `8-bit-devanagari.otf` is readable but does not match | |
| the Unicode Devanagari registry range; keep it in legacy conversion research. | |
| Google Fonts expansion batch: | |
| Reproducible provisioning command: | |
| ```bash | |
| uv run ocrtech prepare-font-assets \ | |
| configs/nepal-font-assets-v1.json \ | |
| --out outputs/font-assets-google-fonts-2026-06-16 | |
| ``` | |
| Result on the current runtime: | |
| ```text | |
| passed=True | |
| assets=13 | |
| downloaded=0 | |
| reused=13 | |
| failed=0 | |
| ``` | |
| The manifest includes five font binaries and eight OFL/upstream metadata files. | |
| It is the authoritative way to recreate this batch under `data/fonts/`. | |
| ```text | |
| data/fonts/google-fonts-2026-06-16/kanchenjunga/Kanchenjunga-Regular.ttf | |
| data/fonts/google-fonts-2026-06-16/kanchenjunga/Kanchenjunga-Bold.ttf | |
| data/fonts/google-fonts-2026-06-16/notosanstirhuta/NotoSansTirhuta-Regular.ttf | |
| data/fonts/google-fonts-2026-06-16/notoserifdevanagari/NotoSerifDevanagari-VariableFont_wdth,wght.ttf | |
| data/fonts/google-fonts-2026-06-16/notosansolchiki/NotoSansOlChiki-VariableFont_wght.ttf | |
| ``` | |
| The batch was downloaded from the upstream Google Fonts repository under | |
| `ofl/kanchenjunga`, `ofl/notosanstirhuta`, `ofl/notoserifdevanagari`, and | |
| `ofl/notosansolchiki`, with each source directory's `OFL.txt` and | |
| `upstream_info.md` copied into ignored runtime data. | |
| Font SHA-256 values: | |
| ```text | |
| 45609f8cc90d4733d3d1665346d359afc1a659def340d0f369121af034322ef9 Kanchenjunga-Regular.ttf | |
| 67a9a29cd49a17832a5eee8d2747fb9a66ee32e49dc52f5dd43887ff24eb0935 Kanchenjunga-Bold.ttf | |
| ad7123ee63118b83ed2f723591e5e861baad8dd157508b8339362850c6036efe NotoSansTirhuta-Regular.ttf | |
| 1191e07bfeb062d80e252eb85b0eafdfbda1e350707a2a60628668e8f677dbbb NotoSerifDevanagari-VariableFont_wdth,wght.ttf | |
| c9c31988656f49eccec9588825ab3b5045099c2f850ef98f356f976e8a596b4d NotoSansOlChiki-VariableFont_wght.ttf | |
| ``` | |
| Inventory command: | |
| ```bash | |
| uv run ocrtech inventory-fonts \ | |
| languages/nepal-v1.json \ | |
| --root data/fonts/google-fonts-2026-06-16 \ | |
| --out outputs/font-inventory-google-fonts-2026-06-16 | |
| ``` | |
| Result: | |
| ```text | |
| passed=True | |
| fonts=5 | |
| readable=5 | |
| archives=0 | |
| duplicates=0 | |
| warnings=0 | |
| ``` | |
| Coverage result: | |
| - Kanchenjunga Regular/Bold: `100%` Bantawa/Kirat Rai range coverage. | |
| - Noto Sans Tirhuta: `100%` Maithili/Tirhuta range coverage. | |
| - Noto Serif Devanagari: `100%` coverage for every registered Devanagari | |
| script target. | |
| - Noto Sans Ol Chiki Variable: `100%` Santhali/Ol Chiki coverage. | |
| All-script renderability gate after registry promotion: | |
| ```bash | |
| uv run ocrtech audit-font-renderability \ | |
| languages/nepal-v1.json \ | |
| --all-scripts \ | |
| --out outputs/font-renderability-all-scripts-google-fonts-2026-06-16 | |
| ``` | |
| ```text | |
| passed=True | |
| scripts=22 | |
| candidates=58 | |
| rendered=0 | |
| ``` | |
| Font asset readiness gate: | |
| ```bash | |
| uv run ocrtech audit-font-asset-readiness \ | |
| configs/nepal-font-assets-v1.json \ | |
| outputs/font-assets-google-fonts-2026-06-16/font-assets-preparation.json \ | |
| outputs/font-inventory-all-runtime-fonts-2026-06-16/font-inventory.json \ | |
| outputs/font-renderability-all-scripts-google-fonts-2026-06-16/font-renderability-audit.json \ | |
| --out outputs/font-asset-readiness-google-fonts-2026-06-16 | |
| ``` | |
| ```text | |
| passed=True | |
| assets=13 | |
| checked=13 | |
| issues=0 | |
| ``` | |
| This gate ties acquisition, local files, inventory, and renderability together. | |
| It SHA-checks every manifest asset and requires inventory membership for font | |
| files. License/upstream metadata files are SHA-checked but not required in the | |
| font inventory because the inventory intentionally scans font suffixes only. | |
| Training-bound rendered synthesis audits can now require this evidence: | |
| ```bash | |
| uv run ocrtech audit-rendered-synthesis-lines \ | |
| data/rendered-line-split/eval/manifests/rendered-line-manifest.jsonl \ | |
| --label-file data/rendered-line-split/eval/labels/eval.txt \ | |
| --require-font \ | |
| --font-readiness-report outputs/font-asset-readiness-google-fonts-2026-06-16/font-asset-readiness.json \ | |
| --out outputs/rendered-line-manifest-audit | |
| ``` | |
| `render-synthesis-text-split` also accepts `--font-readiness-report`; when it is | |
| supplied, both train and eval rendered-line audits require every rendered row's | |
| `metadata.font_sha256` to appear in the passed readiness manifest. This prevents | |
| training-bound synthetic OCR bundles from silently using unprovisioned or | |
| unaudited fonts. | |
| Current Namdhinggo-rendered Limbu/Sirijonga split passed this gate: | |
| ```bash | |
| uv run ocrtech audit-rendered-synthesis-lines \ | |
| outputs/sirijonga-limbu-rendered-split-namdhinggo-2026-06-16/train/manifests/rendered-line-manifest.jsonl \ | |
| --label-file outputs/sirijonga-limbu-rendered-split-namdhinggo-2026-06-16/train/labels/train.txt \ | |
| --require-font \ | |
| --font-readiness-report outputs/font-asset-readiness-namdhinggo-2026-06-16/font-asset-readiness.json \ | |
| --out outputs/sirijonga-limbu-rendered-split-namdhinggo-2026-06-16/train-rendered-audit-font-ready | |
| uv run ocrtech audit-rendered-synthesis-lines \ | |
| outputs/sirijonga-limbu-rendered-split-namdhinggo-2026-06-16/eval/manifests/rendered-line-manifest.jsonl \ | |
| --label-file outputs/sirijonga-limbu-rendered-split-namdhinggo-2026-06-16/eval/labels/eval.txt \ | |
| --require-font \ | |
| --font-readiness-report outputs/font-asset-readiness-namdhinggo-2026-06-16/font-asset-readiness.json \ | |
| --out outputs/sirijonga-limbu-rendered-split-namdhinggo-2026-06-16/eval-rendered-audit-font-ready | |
| ``` | |
| ```text | |
| train: passed=True samples=150 claim_eligible=0 issues=0 warnings=0 | |
| eval: passed=True samples=27 claim_eligible=0 issues=0 warnings=0 | |
| ``` | |
| Collection policy: | |
| - Every collected font source needs file path, original source URL or local | |
| source path, license, SHA-256, and cmap/script coverage. | |
| - Keep font binaries under ignored `data/fonts/`; commit only the registry, | |
| source notes, hashes, and audit commands/results. | |
| - Treat all legacy/non-Unicode fonts as conversion leads only. They cannot be | |
| used for OCR labels until the encoding mapping is verified against real text. | |
| - Use multiple font weights/styles for synthesis diversity only after the | |
| primary regular face passes renderability and label audits. | |
| Next font collection targets: | |
| - Limbu/Sirijonga: Namdhinggo full family is present; inspect legacy | |
| Namdhinggo SIL L conversion assets separately. | |
| - Devanagari: collect more families used in Nepali newspapers and textbooks, | |
| not just Noto Sans Devanagari. | |
| - Ol Chiki/Santhali: add Noto Serif or other print-style alternatives. | |
| - Kirat Rai: locate Unicode 16.0-compatible fonts before treating the script as | |
| trainable. Kanchenjunga now passes cmap coverage, but Kirat Rai still needs | |
| real text, labels, and validation before training claims. | |
| - Tibetan/Tamyig/Sherpa-related paths: audit actual Unicode usage and fonts | |
| before adding synthesis labels. | |
| - Local LTK font archive: inventoried. Keep `janaki.ttf` and `madan.ttf` out of | |
| Unicode synthesis until their encoding/conversion behavior is verified. | |
| - Remote `cdjk` direct font files: inventoried. Promote only the duplicate | |
| Noto Sans Devanagari if needed for provisioning; keep the 8-bit Devanagari | |
| font as legacy/conversion evidence and the Nerd fonts as Latin-only utilities. | |
| - Google Fonts expansion: inventoried and promoted as font candidates for | |
| Kirat Rai, Tirhuta, print-style Devanagari, and Ol Chiki diversity. | |