File size: 12,630 Bytes
f9bc0c1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 | # Font Assets
The language registry points to repo-relative runtime font assets under
`data/fonts/`. The `data/` tree is ignored because training and validation data
can be large, so these files must be provisioned on each runtime host before
synthetic rendering or font coverage claims.
Current required assets:
| File | Purpose | Current source used in audit |
| --- | --- | --- |
| `data/fonts/namdhinggo/Namdhinggo-3.100/Namdhinggo-Regular.ttf` | Primary Limbu/Sirijonga OCR synthesis and coverage | `configs/namdhinggo-font-assets-v1.json`; SIL Namdhinggo 3.100 release ZIP from <https://github.com/silnrsi/font-namdhinggo/releases/download/v3.100/Namdhinggo-3.100.zip>; OFL licensed; package SHA-256 `073369fbaf064f9075bfbc65e1a76776f449a280c4c367f3cb560bebf70b3915`; Regular TTF SHA-256 `1b46b16277e4b6784bd19306a9474281888b656e9c57233b4e7dc63bd1229d6c` |
| `data/fonts/NotoSansDevanagari-VariableFont_wdth,wght.ttf` | Devanagari OCR synthesis and coverage for Nepali, Tamang, Magar, Bhojpuri, Maithili, and related Devanagari targets | Local Noto Sans Devanagari download on `cdjk`; copied to k2 runtime data |
| `data/fonts/NotoSansOlChiki-Regular.ttf` | Santhali/Ol Chiki coverage | macOS system font `/System/Library/Fonts/Supplemental/NotoSansOlChiki-Regular.ttf`; copied to runtime data |
Host-local system fonts currently used directly:
| File | Purpose |
| --- | --- |
| `/System/Library/Fonts/Supplemental/NotoSansLimbu-Regular.ttf` | Limbu/Sirijonga fallback coverage |
| `/Library/Fonts/Arial Unicode.ttf` or `/System/Library/Fonts/Helvetica.ttc` | Latin/English regression coverage |
Namdhinggo notes:
- SIL describes Namdhinggo as a Unicode Limbu font covering the full Limbu
range `U+1900..U+194F` plus basic Latin.
- The 3.100 package includes Regular, Medium, SemiBold, Bold, ExtraBold, WOFF,
and WOFF2 files.
- The committed Namdhinggo manifest provisions the five desktop TTF weights and
key upstream metadata/license files from the upstream ZIP by archive member
path, verifying both package and extracted-member SHA-256 values.
- SIL also documents legacy Namdhinggo SIL L and Devanagari-to-Limbu TECkit
conversion maps. Treat those as data-cleanup leads only; any converted text
still needs source, schema, and normalization audits before OCR synthesis.
Namdhinggo reproducible provisioning:
```bash
uv run ocrtech prepare-font-assets \
configs/namdhinggo-font-assets-v1.json \
--out outputs/font-assets-namdhinggo-2026-06-16
```
Result on the current runtime:
```text
passed=True
assets=10
downloaded=0
reused=10
failed=0
```
Namdhinggo readiness gate:
```bash
uv run ocrtech audit-font-asset-readiness \
configs/namdhinggo-font-assets-v1.json \
outputs/font-assets-namdhinggo-2026-06-16/font-assets-preparation.json \
outputs/font-inventory-all-runtime-fonts-2026-06-16/font-inventory.json \
outputs/font-renderability-all-scripts-google-fonts-2026-06-16/font-renderability-audit.json \
--out outputs/font-asset-readiness-namdhinggo-2026-06-16
```
```text
passed=True
assets=10
checked=10
issues=0
```
Validation command:
```bash
uv run ocrtech audit-font-renderability languages/nepal-v1.json --out outputs/font-renderability-audit
```
This audit is cmap coverage only unless `--render-samples` is used in an
environment with Pillow installed.
Inventory command:
```bash
uv run ocrtech inventory-fonts \
languages/nepal-v1.json \
--root data/fonts \
--out outputs/font-inventory-current-2026-06-16
```
Current inventory result:
```text
passed=True
fonts=17
readable=7
archives=0
duplicates=0
warnings=0
```
The inventory includes desktop fonts and web font files. Only `.ttf`, `.otf`,
and `.ttc` files are currently cmap-scored; `.woff` and `.woff2` files are
hashed and recorded but not treated as renderability evidence.
LTK archive inventory:
```bash
uv run ocrtech inventory-fonts \
languages/nepal-v1.json \
--root /Users/cdjk/github/llm/ltk/fonts.zip \
--out outputs/font-inventory-ltk-fonts-2026-06-16
```
Result:
```text
passed=True
fonts=2
readable=2
archives=1
duplicates=0
warnings=0
```
The archive contains `janaki.ttf` and `madan.ttf`. Both are cmap-readable, but
neither matched the registry Unicode ranges at the `0.95` coverage threshold.
Treat them as legacy-font/conversion research leads, not Unicode OCR rendering
fonts.
Remote `cdjk@100.110.148.43` probe:
```text
/Users/cdjk/github/llm/ltk/fonts.zip
/Users/cdjk/github/llm/g2p/site/assets/8-bit-devanagari.otf
/Users/cdjk/github/llm/twitter-threads/references/fonts/DejaVuSansMNerdFontMono-Regular.ttf
/Users/cdjk/github/llm/twitter-threads/references/fonts/NotoSansDevanagari.ttf
/Users/cdjk/github/llm/twitter-threads/references/fonts/3270NerdFontMono-Regular.ttf
```
Copied the four direct font files into ignored local runtime data:
```text
data/fonts/remote-cdjk-2026-06-16/
```
Inventory command:
```bash
uv run ocrtech inventory-fonts \
languages/nepal-v1.json \
--root data/fonts/remote-cdjk-2026-06-16 \
--out outputs/font-inventory-remote-cdjk-2026-06-16
```
Result:
```text
passed=True
fonts=4
readable=4
archives=0
duplicates=0
warnings=0
```
The remote `NotoSansDevanagari.ttf` has the same SHA prefix observed for the
current runtime Devanagari font and covers all registered Devanagari script
targets. `3270NerdFontMono-Regular.ttf` and `DejaVuSansMNerdFontMono-Regular.ttf`
match only Latin/English. `8-bit-devanagari.otf` is readable but does not match
the Unicode Devanagari registry range; keep it in legacy conversion research.
Google Fonts expansion batch:
Reproducible provisioning command:
```bash
uv run ocrtech prepare-font-assets \
configs/nepal-font-assets-v1.json \
--out outputs/font-assets-google-fonts-2026-06-16
```
Result on the current runtime:
```text
passed=True
assets=13
downloaded=0
reused=13
failed=0
```
The manifest includes five font binaries and eight OFL/upstream metadata files.
It is the authoritative way to recreate this batch under `data/fonts/`.
```text
data/fonts/google-fonts-2026-06-16/kanchenjunga/Kanchenjunga-Regular.ttf
data/fonts/google-fonts-2026-06-16/kanchenjunga/Kanchenjunga-Bold.ttf
data/fonts/google-fonts-2026-06-16/notosanstirhuta/NotoSansTirhuta-Regular.ttf
data/fonts/google-fonts-2026-06-16/notoserifdevanagari/NotoSerifDevanagari-VariableFont_wdth,wght.ttf
data/fonts/google-fonts-2026-06-16/notosansolchiki/NotoSansOlChiki-VariableFont_wght.ttf
```
The batch was downloaded from the upstream Google Fonts repository under
`ofl/kanchenjunga`, `ofl/notosanstirhuta`, `ofl/notoserifdevanagari`, and
`ofl/notosansolchiki`, with each source directory's `OFL.txt` and
`upstream_info.md` copied into ignored runtime data.
Font SHA-256 values:
```text
45609f8cc90d4733d3d1665346d359afc1a659def340d0f369121af034322ef9 Kanchenjunga-Regular.ttf
67a9a29cd49a17832a5eee8d2747fb9a66ee32e49dc52f5dd43887ff24eb0935 Kanchenjunga-Bold.ttf
ad7123ee63118b83ed2f723591e5e861baad8dd157508b8339362850c6036efe NotoSansTirhuta-Regular.ttf
1191e07bfeb062d80e252eb85b0eafdfbda1e350707a2a60628668e8f677dbbb NotoSerifDevanagari-VariableFont_wdth,wght.ttf
c9c31988656f49eccec9588825ab3b5045099c2f850ef98f356f976e8a596b4d NotoSansOlChiki-VariableFont_wght.ttf
```
Inventory command:
```bash
uv run ocrtech inventory-fonts \
languages/nepal-v1.json \
--root data/fonts/google-fonts-2026-06-16 \
--out outputs/font-inventory-google-fonts-2026-06-16
```
Result:
```text
passed=True
fonts=5
readable=5
archives=0
duplicates=0
warnings=0
```
Coverage result:
- Kanchenjunga Regular/Bold: `100%` Bantawa/Kirat Rai range coverage.
- Noto Sans Tirhuta: `100%` Maithili/Tirhuta range coverage.
- Noto Serif Devanagari: `100%` coverage for every registered Devanagari
script target.
- Noto Sans Ol Chiki Variable: `100%` Santhali/Ol Chiki coverage.
All-script renderability gate after registry promotion:
```bash
uv run ocrtech audit-font-renderability \
languages/nepal-v1.json \
--all-scripts \
--out outputs/font-renderability-all-scripts-google-fonts-2026-06-16
```
```text
passed=True
scripts=22
candidates=58
rendered=0
```
Font asset readiness gate:
```bash
uv run ocrtech audit-font-asset-readiness \
configs/nepal-font-assets-v1.json \
outputs/font-assets-google-fonts-2026-06-16/font-assets-preparation.json \
outputs/font-inventory-all-runtime-fonts-2026-06-16/font-inventory.json \
outputs/font-renderability-all-scripts-google-fonts-2026-06-16/font-renderability-audit.json \
--out outputs/font-asset-readiness-google-fonts-2026-06-16
```
```text
passed=True
assets=13
checked=13
issues=0
```
This gate ties acquisition, local files, inventory, and renderability together.
It SHA-checks every manifest asset and requires inventory membership for font
files. License/upstream metadata files are SHA-checked but not required in the
font inventory because the inventory intentionally scans font suffixes only.
Training-bound rendered synthesis audits can now require this evidence:
```bash
uv run ocrtech audit-rendered-synthesis-lines \
data/rendered-line-split/eval/manifests/rendered-line-manifest.jsonl \
--label-file data/rendered-line-split/eval/labels/eval.txt \
--require-font \
--font-readiness-report outputs/font-asset-readiness-google-fonts-2026-06-16/font-asset-readiness.json \
--out outputs/rendered-line-manifest-audit
```
`render-synthesis-text-split` also accepts `--font-readiness-report`; when it is
supplied, both train and eval rendered-line audits require every rendered row's
`metadata.font_sha256` to appear in the passed readiness manifest. This prevents
training-bound synthetic OCR bundles from silently using unprovisioned or
unaudited fonts.
Current Namdhinggo-rendered Limbu/Sirijonga split passed this gate:
```bash
uv run ocrtech audit-rendered-synthesis-lines \
outputs/sirijonga-limbu-rendered-split-namdhinggo-2026-06-16/train/manifests/rendered-line-manifest.jsonl \
--label-file outputs/sirijonga-limbu-rendered-split-namdhinggo-2026-06-16/train/labels/train.txt \
--require-font \
--font-readiness-report outputs/font-asset-readiness-namdhinggo-2026-06-16/font-asset-readiness.json \
--out outputs/sirijonga-limbu-rendered-split-namdhinggo-2026-06-16/train-rendered-audit-font-ready
uv run ocrtech audit-rendered-synthesis-lines \
outputs/sirijonga-limbu-rendered-split-namdhinggo-2026-06-16/eval/manifests/rendered-line-manifest.jsonl \
--label-file outputs/sirijonga-limbu-rendered-split-namdhinggo-2026-06-16/eval/labels/eval.txt \
--require-font \
--font-readiness-report outputs/font-asset-readiness-namdhinggo-2026-06-16/font-asset-readiness.json \
--out outputs/sirijonga-limbu-rendered-split-namdhinggo-2026-06-16/eval-rendered-audit-font-ready
```
```text
train: passed=True samples=150 claim_eligible=0 issues=0 warnings=0
eval: passed=True samples=27 claim_eligible=0 issues=0 warnings=0
```
Collection policy:
- Every collected font source needs file path, original source URL or local
source path, license, SHA-256, and cmap/script coverage.
- Keep font binaries under ignored `data/fonts/`; commit only the registry,
source notes, hashes, and audit commands/results.
- Treat all legacy/non-Unicode fonts as conversion leads only. They cannot be
used for OCR labels until the encoding mapping is verified against real text.
- Use multiple font weights/styles for synthesis diversity only after the
primary regular face passes renderability and label audits.
Next font collection targets:
- Limbu/Sirijonga: Namdhinggo full family is present; inspect legacy
Namdhinggo SIL L conversion assets separately.
- Devanagari: collect more families used in Nepali newspapers and textbooks,
not just Noto Sans Devanagari.
- Ol Chiki/Santhali: add Noto Serif or other print-style alternatives.
- Kirat Rai: locate Unicode 16.0-compatible fonts before treating the script as
trainable. Kanchenjunga now passes cmap coverage, but Kirat Rai still needs
real text, labels, and validation before training claims.
- Tibetan/Tamyig/Sherpa-related paths: audit actual Unicode usage and fonts
before adding synthesis labels.
- Local LTK font archive: inventoried. Keep `janaki.ttf` and `madan.ttf` out of
Unicode synthesis until their encoding/conversion behavior is verified.
- Remote `cdjk` direct font files: inventoried. Promote only the duplicate
Noto Sans Devanagari if needed for provisioning; keep the 8-bit Devanagari
font as legacy/conversion evidence and the Nerd fonts as Latin-only utilities.
- Google Fonts expansion: inventoried and promoted as font candidates for
Kirat Rai, Tirhuta, print-style Devanagari, and Ol Chiki diversity.
|