ocr-projects-archive / ocr-tech /docs /RECOVERY_LEDGER.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
9.26 kB
# Recovery Ledger β€” 2026-07-05
The Mac died on 2026-07-04. A last-chance rescue (`manifest.txt` at the backup
root) copied code, `.git`, docs, configs, `spaces/`, and most of `data/` to
`/Volumes/TRANSCEND/recovery-backup-20260704-172213/github/llm/ocr-tech`, which
is now the **only** live repo (symlinked from `/Users/cdjk/github/llm/ocr-tech`;
same for the sibling `nepal-ttf2utf`). The internal SSD has ~6.5 GB free, so the
repo stays on TRANSCEND. `outputs/` and `runs/` were never copied and are gone.
## Repo repair done 2026-07-05
- Deleted exFAT AppleDouble `._*` sidecars inside `.git` (they caused
`non-monotonic index` errors); `git fsck` clean afterwards.
- `git config core.filemode false` (exFAT churns modes; 469 fake diffs silenced).
- `._*` and `.claude/` added to `.git/info/exclude`; repo-tree `._*` files deleted.
They will regenerate as macOS touches files on exFAT β€” harmless, excluded.
- Restored deleted tracked files from git: `data/mappings/limbu-legacy/{Limbu.map,README.txt}`,
`examples/sample-page.txt`, `languages/nepal-v1.json` (the registry β€” the rescue
pass had skipped those dirs).
- Working tree now matches the pre-crash snapshot
(`../../git-snapshots/ocr-tech/status.txt`): only `RESEARCH.md` modified
(the uncommitted production book-ingestion plan) + 3 untracked scratch files.
## Survived (verified 2026-07-05)
- **Champion Sirijonga recognizer (inference export)** at
`spaces/limbu-ocr/models/limbu-sirijonga-real-heavy/` β€”
`limbu-real-heavy-gork-cdc-replay-v1-lr1e6-8ep-best`; all 4 artifacts
**sha256-verified against the model card**; loads and reads Sirijonga through
the pinned CPU path (`docs/LOCAL_RECOGNIZER_SCORING.md` env rebuilt, smoke OK).
- **Devanagari recognizer export** at `spaces/limbu-ocr/models/limbu-devanagari-ppocrv5/`
β€” sha256-verified. **YOLO line detector** `spaces/limbu-ocr/models/line-detector/best.pt` present.
- `data/` 155G: dictionaries (v1 set; `devanagari-v2.txt` lost), fonts (512M),
`gorkhapatra-limbu-editions/` (24 source PDFs + listings), kiratrai/lepcha real
eval packs (70/132 crops), leipzig corpora, external toolkits (4.3G),
`gorkhapatra-devanagari-real/labels.txt` (79,364 lines, intact).
- Paddle caches at backup root: `.paddleocr` (PP-OCRv5_mobile_rec pretrained),
`.paddlex` (det/orientation/UVDoc official models).
- Sibling `nepal-ttf2utf` (backup copy, git-functional; `nayanepal` conversion
verified: `k[i7 ;+of]hs` β†’ `ΰ€ͺΰ₯ƒΰ€·ΰ₯ΰ€  ΰ€Έΰ€‚ΰ€―ΰ₯‹ΰ€œΰ€•`, clean).
- Test suite: **passes** except 5 `validate_*` tests that assert on
lost-`outputs/` eval packs (`limbu-gorkhapatra-real-v1` etc.) β€” environment
losses, not code regressions.
## Partial
- `data/gorkhapatra-devanagari-real/images/`: **40,359 of 79,364** images
(~51%). Labels intact; any sequential consumer will hit missing files.
Rebuildable from re-fetched PDFs via `tools/build_gorkhapatra_devanagari_real.py`.
- `data/limbu-aug-train/images/`: 26,248 of 28,155 (~93%). Synthetic
(Namdhinggo, seed 13) β€” regenerate rather than hunt the missing 1,907.
- `data/crops-crop-smoke-v1/` and `data/limbu-gorkhapatra-extra/images/`: empty shells.
## Lost, and the recovery route for each
| Lost | What it was | Route back |
|---|---|---|
| `outputs/` (~330 dirs) | every harvest, inventory, eval pack, spike artifact | re-harvest (recipe in RESEARCH.md); rebuild running, see below |
| `runs/` (~44 dirs) | training checkpoints incl. Limbu champion training run | inference exports survived in `spaces/`; training checkpoints may exist on AWS (`ashish-experiments`) β€” **blocked on `aws sso login`** |
| `outputs/aws-newar-deva-ft-v1/` (newar-ft-v1 weights) | first Newar recognizer (cp CER 0.0465) | AWS instance/S3 β€” blocked on AWS creds |
| Newar frozen pack `newar-devanagari-silver-pack-v1` | train 4,096 / dev 1,528 / test 1,139; pack sha256 recorded in RESEARCH.md | rebuild from re-harvest; recorded sha256 lets us verify the rebuild |
| Limbu 8-page silver pack + 255-line frozen eval | champion's claim eval | rebuild from re-harvest manifests (`extract-...-pdf-text` β†’ silver builder); some source PDFs already local in `data/gorkhapatra-limbu-editions/` |
| `ppstructure-spike-2026-07-04` | was running when the machine died | re-run after the Limbu 8-page pack is rebuilt |
| HF Space cross-check | deployed `ampixa/limbu-ocr` bundle | **blocked on `hf auth login`** (local copy is hash-verified anyway) |
## Rebuild policy
Rebuilt artifacts go to **new dated dirs** (e.g.
`outputs/gorkhapatra-language-pages-full-rebuild-b1-2026-07-05/`), never the old
names β€” docs describing the originals stay true, and this ledger maps old β†’ new.
Where an original sha256 was recorded (Newar pack), verify the rebuild against it
and note match/mismatch here.
## Recovered 2026-07-05
- **AWS pull complete** (`outputs/aws-newar-recovery-2026-07-05/`, ~1.4G, from
`ashish-experiments` i-09a2252533fe828b4, started β†’ rsynced β†’ stopped):
`newar-ft-v1-artifacts.tgz` (the lost `outputs/aws-newar-deva-ft-v1` bundle:
`export/newar-ft-v1` inference artifacts + best checkpoint + train log +
config), full `runs/newar-deva-ft-v1/checkpoints/` (best + epochs 20/40/60),
`deva-v2/best_accuracy.pdparams` (the shared Devanagari recognizer training
checkpoint, also lost from `runs/`), `newar-ft-payload.tgz` (the uploaded
train/dev/test splits β€” the frozen Newar pack content), `devanagari-v2.txt`
(lost dictionary), setup/train logs. No S3 was ever used; no Limbu training
runs on this instance (provisioned fresh 2026-07-04) β€” the Limbu champion
**training** checkpoint is unrecoverable, but its inference export is safe
locally and byte-identical on the HF Space.
- **HF Space cross-check passed**: deployed `ampixa/limbu-ocr`
`inference.pdiparams` sha256 matches the local copy exactly.
- **PP-StructureV3 spike (old session mining)**: produced no numbers before the
crash β€” ~6h CPU-bound on page 1/8 (PP-DocLayout_plus-L + PP-OCRv5_server
det/rec on ~4000Γ—6000 px pages). Re-run must go to the L4 GPU instance.
- **Newar pack rebuild recipe recovered**: `tools/build_newar_devanagari_silver_pack.py`,
deterministic by-date split (every 5th date from index 2 β†’ test, index 0 β†’
dev), inputs = the per-language silver line dirs (label `ΰ€¨ΰ₯‡ΰ€ͺΰ€Ύΰ€²`) from both
harvest batches; original run: train 4,096 / dev 1,528 / test 1,139,
`cross_split_leaked_dropped: 37`.
Rebuild runs launched 2026-07-05:
- Batch-1 re-harvest (23 weekly windows 2026-01-25 β†’ 2026-07-04,
publication-pages 80) β†’ `outputs/gorkhapatra-language-pages-full-rebuild-b1-2026-07-05/`
β€” **done: 400 items, 197 aligned, 398 hits, warnings=0** (original:
400/195/394/0). 9 Limbu alignments, 4 with PDFs in this window; the other 5
dates fall in batch 2. Note: the first three attempts died on WAF connection
resets β€” the epaper server intermittently blocks the tool UA
`ocrtech-source-audit/0.1` (browser UA passes); fix in flight in
`src/ocrtech/source_audit.py` (Codex: UA constant + retry/backoff + on-disk
PDF resume).
- Batch-2 re-harvest (25 weekly windows 2025-08-03 β†’ 2026-01-24,
publication-pages 160) β†’ `outputs/gorkhapatra-language-pages-full-rebuild-b2-2026-07-05/`,
running. Original result: 798 items, 294 aligned, 589 hits.
- Batch-1 silver line packs β†’
`outputs/language-pack-silver-lines-rebuild-b1-2026-07-05/` β€” **done:
122,882 kept, 50 languages, zero errors** (original 116,287).
- Batch-2 silver line packs β†’
`outputs/language-pack-silver-lines-rebuild-b2-2026-07-05/` β€” **done:
184,089 kept, 49 languages, zero errors** (original 183,324).
- **Limbu silver corpus rebuilt at 2Γ— the original**:
`outputs/limbu-gorkhapatra-pdf-backed-silver-rebuild-2026-07-05-v1/` β€” 17
discovered dates (vs 9), **16 usable pages / 4,872 usable silver labels**
(vs 8 / 2,430), only the known visual-only `2082-11-12` missing; page eval
pack at `outputs/limbu-page-eval-rebuild-2026-07-05-v1/` (16 pages, 4,895
lines).
- **Newar frozen pack recovered AND rebuilt, split determinism proven**:
- Recovered original (from the AWS payload, all 6,763 crops + labels):
`outputs/newar-devanagari-silver-pack-v1-recovered/` β€” use this for
score-comparability with the recorded newar-ft-v1 numbers.
- Rebuild from re-harvest: `outputs/newar-devanagari-silver-pack-rebuild-2026-07-05/`
(sha `5908d682…` β‰  original `1fa7b23b…` for a verified benign reason:
train and dev are **set-identical** to the original 4,096/1,528; test is a
strict superset β€” all 1,139 original lines + 7 rows (3 unique new texts)
the richer re-harvest found on the same test dates `2082-07-25`/`2083-01-05`).
- **PP-StructureV3 spike completed end-to-end** (stage 1 on the L4 GPU, 16/16
pages at ~4 s/page; stage 2 champion-through-its-boxes scored vs the rebuilt
Rows-LTR baseline). Verdict in RESEARCH.md: PP-StructureV3 **stays
fallback** β€” the current Paddle det + Rows-LTR + y/conf70 chain wins every
metric (page cp CER 0.413 vs 0.421, det F1 0.966 vs 0.934, matched CER
0.0266 vs 0.0331, order 0.993 vs 0.778 native). Baseline reproduction on the
new pack also validated the lost baseline numbers (0.4134 vs old 0.4228).