|
Download ocr-tech/docs/RECOVERY_LEDGER.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 9.26 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/RECOVERY_LEDGER.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/ocr-tech/docs/RECOVERY_LEDGER.md
-
curl -L -o RECOVERY_LEDGER.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech/docs/RECOVERY_LEDGER.md
9.26 kB
| # Recovery Ledger β 2026-07-05 | |
| The Mac died on 2026-07-04. A last-chance rescue (`manifest.txt` at the backup | |
| root) copied code, `.git`, docs, configs, `spaces/`, and most of `data/` to | |
| `/Volumes/TRANSCEND/recovery-backup-20260704-172213/github/llm/ocr-tech`, which | |
| is now the **only** live repo (symlinked from `/Users/cdjk/github/llm/ocr-tech`; | |
| same for the sibling `nepal-ttf2utf`). The internal SSD has ~6.5 GB free, so the | |
| repo stays on TRANSCEND. `outputs/` and `runs/` were never copied and are gone. | |
| ## Repo repair done 2026-07-05 | |
| - Deleted exFAT AppleDouble `._*` sidecars inside `.git` (they caused | |
| `non-monotonic index` errors); `git fsck` clean afterwards. | |
| - `git config core.filemode false` (exFAT churns modes; 469 fake diffs silenced). | |
| - `._*` and `.claude/` added to `.git/info/exclude`; repo-tree `._*` files deleted. | |
| They will regenerate as macOS touches files on exFAT β harmless, excluded. | |
| - Restored deleted tracked files from git: `data/mappings/limbu-legacy/{Limbu.map,README.txt}`, | |
| `examples/sample-page.txt`, `languages/nepal-v1.json` (the registry β the rescue | |
| pass had skipped those dirs). | |
| - Working tree now matches the pre-crash snapshot | |
| (`../../git-snapshots/ocr-tech/status.txt`): only `RESEARCH.md` modified | |
| (the uncommitted production book-ingestion plan) + 3 untracked scratch files. | |
| ## Survived (verified 2026-07-05) | |
| - **Champion Sirijonga recognizer (inference export)** at | |
| `spaces/limbu-ocr/models/limbu-sirijonga-real-heavy/` β | |
| `limbu-real-heavy-gork-cdc-replay-v1-lr1e6-8ep-best`; all 4 artifacts | |
| **sha256-verified against the model card**; loads and reads Sirijonga through | |
| the pinned CPU path (`docs/LOCAL_RECOGNIZER_SCORING.md` env rebuilt, smoke OK). | |
| - **Devanagari recognizer export** at `spaces/limbu-ocr/models/limbu-devanagari-ppocrv5/` | |
| β sha256-verified. **YOLO line detector** `spaces/limbu-ocr/models/line-detector/best.pt` present. | |
| - `data/` 155G: dictionaries (v1 set; `devanagari-v2.txt` lost), fonts (512M), | |
| `gorkhapatra-limbu-editions/` (24 source PDFs + listings), kiratrai/lepcha real | |
| eval packs (70/132 crops), leipzig corpora, external toolkits (4.3G), | |
| `gorkhapatra-devanagari-real/labels.txt` (79,364 lines, intact). | |
| - Paddle caches at backup root: `.paddleocr` (PP-OCRv5_mobile_rec pretrained), | |
| `.paddlex` (det/orientation/UVDoc official models). | |
| - Sibling `nepal-ttf2utf` (backup copy, git-functional; `nayanepal` conversion | |
| verified: `k[i7 ;+of]hs` β `ΰ€ͺΰ₯ΰ€·ΰ₯ΰ€ ΰ€Έΰ€ΰ€―ΰ₯ΰ€ΰ€`, clean). | |
| - Test suite: **passes** except 5 `validate_*` tests that assert on | |
| lost-`outputs/` eval packs (`limbu-gorkhapatra-real-v1` etc.) β environment | |
| losses, not code regressions. | |
| ## Partial | |
| - `data/gorkhapatra-devanagari-real/images/`: **40,359 of 79,364** images | |
| (~51%). Labels intact; any sequential consumer will hit missing files. | |
| Rebuildable from re-fetched PDFs via `tools/build_gorkhapatra_devanagari_real.py`. | |
| - `data/limbu-aug-train/images/`: 26,248 of 28,155 (~93%). Synthetic | |
| (Namdhinggo, seed 13) β regenerate rather than hunt the missing 1,907. | |
| - `data/crops-crop-smoke-v1/` and `data/limbu-gorkhapatra-extra/images/`: empty shells. | |
| ## Lost, and the recovery route for each | |
| | Lost | What it was | Route back | | |
| |---|---|---| | |
| | `outputs/` (~330 dirs) | every harvest, inventory, eval pack, spike artifact | re-harvest (recipe in RESEARCH.md); rebuild running, see below | | |
| | `runs/` (~44 dirs) | training checkpoints incl. Limbu champion training run | inference exports survived in `spaces/`; training checkpoints may exist on AWS (`ashish-experiments`) β **blocked on `aws sso login`** | | |
| | `outputs/aws-newar-deva-ft-v1/` (newar-ft-v1 weights) | first Newar recognizer (cp CER 0.0465) | AWS instance/S3 β blocked on AWS creds | | |
| | Newar frozen pack `newar-devanagari-silver-pack-v1` | train 4,096 / dev 1,528 / test 1,139; pack sha256 recorded in RESEARCH.md | rebuild from re-harvest; recorded sha256 lets us verify the rebuild | | |
| | Limbu 8-page silver pack + 255-line frozen eval | champion's claim eval | rebuild from re-harvest manifests (`extract-...-pdf-text` β silver builder); some source PDFs already local in `data/gorkhapatra-limbu-editions/` | | |
| | `ppstructure-spike-2026-07-04` | was running when the machine died | re-run after the Limbu 8-page pack is rebuilt | | |
| | HF Space cross-check | deployed `ampixa/limbu-ocr` bundle | **blocked on `hf auth login`** (local copy is hash-verified anyway) | | |
| ## Rebuild policy | |
| Rebuilt artifacts go to **new dated dirs** (e.g. | |
| `outputs/gorkhapatra-language-pages-full-rebuild-b1-2026-07-05/`), never the old | |
| names β docs describing the originals stay true, and this ledger maps old β new. | |
| Where an original sha256 was recorded (Newar pack), verify the rebuild against it | |
| and note match/mismatch here. | |
| ## Recovered 2026-07-05 | |
| - **AWS pull complete** (`outputs/aws-newar-recovery-2026-07-05/`, ~1.4G, from | |
| `ashish-experiments` i-09a2252533fe828b4, started β rsynced β stopped): | |
| `newar-ft-v1-artifacts.tgz` (the lost `outputs/aws-newar-deva-ft-v1` bundle: | |
| `export/newar-ft-v1` inference artifacts + best checkpoint + train log + | |
| config), full `runs/newar-deva-ft-v1/checkpoints/` (best + epochs 20/40/60), | |
| `deva-v2/best_accuracy.pdparams` (the shared Devanagari recognizer training | |
| checkpoint, also lost from `runs/`), `newar-ft-payload.tgz` (the uploaded | |
| train/dev/test splits β the frozen Newar pack content), `devanagari-v2.txt` | |
| (lost dictionary), setup/train logs. No S3 was ever used; no Limbu training | |
| runs on this instance (provisioned fresh 2026-07-04) β the Limbu champion | |
| **training** checkpoint is unrecoverable, but its inference export is safe | |
| locally and byte-identical on the HF Space. | |
| - **HF Space cross-check passed**: deployed `ampixa/limbu-ocr` | |
| `inference.pdiparams` sha256 matches the local copy exactly. | |
| - **PP-StructureV3 spike (old session mining)**: produced no numbers before the | |
| crash β ~6h CPU-bound on page 1/8 (PP-DocLayout_plus-L + PP-OCRv5_server | |
| det/rec on ~4000Γ6000 px pages). Re-run must go to the L4 GPU instance. | |
| - **Newar pack rebuild recipe recovered**: `tools/build_newar_devanagari_silver_pack.py`, | |
| deterministic by-date split (every 5th date from index 2 β test, index 0 β | |
| dev), inputs = the per-language silver line dirs (label `ΰ€¨ΰ₯ΰ€ͺΰ€Ύΰ€²`) from both | |
| harvest batches; original run: train 4,096 / dev 1,528 / test 1,139, | |
| `cross_split_leaked_dropped: 37`. | |
| Rebuild runs launched 2026-07-05: | |
| - Batch-1 re-harvest (23 weekly windows 2026-01-25 β 2026-07-04, | |
| publication-pages 80) β `outputs/gorkhapatra-language-pages-full-rebuild-b1-2026-07-05/` | |
| β **done: 400 items, 197 aligned, 398 hits, warnings=0** (original: | |
| 400/195/394/0). 9 Limbu alignments, 4 with PDFs in this window; the other 5 | |
| dates fall in batch 2. Note: the first three attempts died on WAF connection | |
| resets β the epaper server intermittently blocks the tool UA | |
| `ocrtech-source-audit/0.1` (browser UA passes); fix in flight in | |
| `src/ocrtech/source_audit.py` (Codex: UA constant + retry/backoff + on-disk | |
| PDF resume). | |
| - Batch-2 re-harvest (25 weekly windows 2025-08-03 β 2026-01-24, | |
| publication-pages 160) β `outputs/gorkhapatra-language-pages-full-rebuild-b2-2026-07-05/`, | |
| running. Original result: 798 items, 294 aligned, 589 hits. | |
| - Batch-1 silver line packs β | |
| `outputs/language-pack-silver-lines-rebuild-b1-2026-07-05/` β **done: | |
| 122,882 kept, 50 languages, zero errors** (original 116,287). | |
| - Batch-2 silver line packs β | |
| `outputs/language-pack-silver-lines-rebuild-b2-2026-07-05/` β **done: | |
| 184,089 kept, 49 languages, zero errors** (original 183,324). | |
| - **Limbu silver corpus rebuilt at 2Γ the original**: | |
| `outputs/limbu-gorkhapatra-pdf-backed-silver-rebuild-2026-07-05-v1/` β 17 | |
| discovered dates (vs 9), **16 usable pages / 4,872 usable silver labels** | |
| (vs 8 / 2,430), only the known visual-only `2082-11-12` missing; page eval | |
| pack at `outputs/limbu-page-eval-rebuild-2026-07-05-v1/` (16 pages, 4,895 | |
| lines). | |
| - **Newar frozen pack recovered AND rebuilt, split determinism proven**: | |
| - Recovered original (from the AWS payload, all 6,763 crops + labels): | |
| `outputs/newar-devanagari-silver-pack-v1-recovered/` β use this for | |
| score-comparability with the recorded newar-ft-v1 numbers. | |
| - Rebuild from re-harvest: `outputs/newar-devanagari-silver-pack-rebuild-2026-07-05/` | |
| (sha `5908d682β¦` β original `1fa7b23bβ¦` for a verified benign reason: | |
| train and dev are **set-identical** to the original 4,096/1,528; test is a | |
| strict superset β all 1,139 original lines + 7 rows (3 unique new texts) | |
| the richer re-harvest found on the same test dates `2082-07-25`/`2083-01-05`). | |
| - **PP-StructureV3 spike completed end-to-end** (stage 1 on the L4 GPU, 16/16 | |
| pages at ~4 s/page; stage 2 champion-through-its-boxes scored vs the rebuilt | |
| Rows-LTR baseline). Verdict in RESEARCH.md: PP-StructureV3 **stays | |
| fallback** β the current Paddle det + Rows-LTR + y/conf70 chain wins every | |
| metric (page cp CER 0.413 vs 0.421, det F1 0.966 vs 0.934, matched CER | |
| 0.0266 vs 0.0331, order 0.993 vs 0.778 native). Baseline reproduction on the | |
| new pack also validated the lost baseline numbers (0.4134 vs old 0.4228). | |