ocr-projects-archive / ocr-tech /docs /RECOVERY_LEDGER.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
9.26 kB

Recovery Ledger β€” 2026-07-05

The Mac died on 2026-07-04. A last-chance rescue (manifest.txt at the backup root) copied code, .git, docs, configs, spaces/, and most of data/ to /Volumes/TRANSCEND/recovery-backup-20260704-172213/github/llm/ocr-tech, which is now the only live repo (symlinked from /Users/cdjk/github/llm/ocr-tech; same for the sibling nepal-ttf2utf). The internal SSD has ~6.5 GB free, so the repo stays on TRANSCEND. outputs/ and runs/ were never copied and are gone.

Repo repair done 2026-07-05

  • Deleted exFAT AppleDouble ._* sidecars inside .git (they caused non-monotonic index errors); git fsck clean afterwards.
  • git config core.filemode false (exFAT churns modes; 469 fake diffs silenced).
  • ._* and .claude/ added to .git/info/exclude; repo-tree ._* files deleted. They will regenerate as macOS touches files on exFAT β€” harmless, excluded.
  • Restored deleted tracked files from git: data/mappings/limbu-legacy/{Limbu.map,README.txt}, examples/sample-page.txt, languages/nepal-v1.json (the registry β€” the rescue pass had skipped those dirs).
  • Working tree now matches the pre-crash snapshot (../../git-snapshots/ocr-tech/status.txt): only RESEARCH.md modified (the uncommitted production book-ingestion plan) + 3 untracked scratch files.

Survived (verified 2026-07-05)

  • Champion Sirijonga recognizer (inference export) at spaces/limbu-ocr/models/limbu-sirijonga-real-heavy/ β€” limbu-real-heavy-gork-cdc-replay-v1-lr1e6-8ep-best; all 4 artifacts sha256-verified against the model card; loads and reads Sirijonga through the pinned CPU path (docs/LOCAL_RECOGNIZER_SCORING.md env rebuilt, smoke OK).
  • Devanagari recognizer export at spaces/limbu-ocr/models/limbu-devanagari-ppocrv5/ β€” sha256-verified. YOLO line detector spaces/limbu-ocr/models/line-detector/best.pt present.
  • data/ 155G: dictionaries (v1 set; devanagari-v2.txt lost), fonts (512M), gorkhapatra-limbu-editions/ (24 source PDFs + listings), kiratrai/lepcha real eval packs (70/132 crops), leipzig corpora, external toolkits (4.3G), gorkhapatra-devanagari-real/labels.txt (79,364 lines, intact).
  • Paddle caches at backup root: .paddleocr (PP-OCRv5_mobile_rec pretrained), .paddlex (det/orientation/UVDoc official models).
  • Sibling nepal-ttf2utf (backup copy, git-functional; nayanepal conversion verified: k[i7 ;+of]hs β†’ ΰ€ͺΰ₯ƒΰ€·ΰ₯ΰ€  ΰ€Έΰ€‚ΰ€―ΰ₯‹ΰ€œΰ€•, clean).
  • Test suite: passes except 5 validate_* tests that assert on lost-outputs/ eval packs (limbu-gorkhapatra-real-v1 etc.) β€” environment losses, not code regressions.

Partial

  • data/gorkhapatra-devanagari-real/images/: 40,359 of 79,364 images (~51%). Labels intact; any sequential consumer will hit missing files. Rebuildable from re-fetched PDFs via tools/build_gorkhapatra_devanagari_real.py.
  • data/limbu-aug-train/images/: 26,248 of 28,155 (~93%). Synthetic (Namdhinggo, seed 13) β€” regenerate rather than hunt the missing 1,907.
  • data/crops-crop-smoke-v1/ and data/limbu-gorkhapatra-extra/images/: empty shells.

Lost, and the recovery route for each

Lost What it was Route back
outputs/ (~330 dirs) every harvest, inventory, eval pack, spike artifact re-harvest (recipe in RESEARCH.md); rebuild running, see below
runs/ (~44 dirs) training checkpoints incl. Limbu champion training run inference exports survived in spaces/; training checkpoints may exist on AWS (ashish-experiments) β€” blocked on aws sso login
outputs/aws-newar-deva-ft-v1/ (newar-ft-v1 weights) first Newar recognizer (cp CER 0.0465) AWS instance/S3 β€” blocked on AWS creds
Newar frozen pack newar-devanagari-silver-pack-v1 train 4,096 / dev 1,528 / test 1,139; pack sha256 recorded in RESEARCH.md rebuild from re-harvest; recorded sha256 lets us verify the rebuild
Limbu 8-page silver pack + 255-line frozen eval champion's claim eval rebuild from re-harvest manifests (extract-...-pdf-text β†’ silver builder); some source PDFs already local in data/gorkhapatra-limbu-editions/
ppstructure-spike-2026-07-04 was running when the machine died re-run after the Limbu 8-page pack is rebuilt
HF Space cross-check deployed ampixa/limbu-ocr bundle blocked on hf auth login (local copy is hash-verified anyway)

Rebuild policy

Rebuilt artifacts go to new dated dirs (e.g. outputs/gorkhapatra-language-pages-full-rebuild-b1-2026-07-05/), never the old names β€” docs describing the originals stay true, and this ledger maps old β†’ new. Where an original sha256 was recorded (Newar pack), verify the rebuild against it and note match/mismatch here.

Recovered 2026-07-05

  • AWS pull complete (outputs/aws-newar-recovery-2026-07-05/, ~1.4G, from ashish-experiments i-09a2252533fe828b4, started β†’ rsynced β†’ stopped): newar-ft-v1-artifacts.tgz (the lost outputs/aws-newar-deva-ft-v1 bundle: export/newar-ft-v1 inference artifacts + best checkpoint + train log + config), full runs/newar-deva-ft-v1/checkpoints/ (best + epochs 20/40/60), deva-v2/best_accuracy.pdparams (the shared Devanagari recognizer training checkpoint, also lost from runs/), newar-ft-payload.tgz (the uploaded train/dev/test splits β€” the frozen Newar pack content), devanagari-v2.txt (lost dictionary), setup/train logs. No S3 was ever used; no Limbu training runs on this instance (provisioned fresh 2026-07-04) β€” the Limbu champion training checkpoint is unrecoverable, but its inference export is safe locally and byte-identical on the HF Space.
  • HF Space cross-check passed: deployed ampixa/limbu-ocr inference.pdiparams sha256 matches the local copy exactly.
  • PP-StructureV3 spike (old session mining): produced no numbers before the crash β€” ~6h CPU-bound on page 1/8 (PP-DocLayout_plus-L + PP-OCRv5_server det/rec on ~4000Γ—6000 px pages). Re-run must go to the L4 GPU instance.
  • Newar pack rebuild recipe recovered: tools/build_newar_devanagari_silver_pack.py, deterministic by-date split (every 5th date from index 2 β†’ test, index 0 β†’ dev), inputs = the per-language silver line dirs (label ΰ€¨ΰ₯‡ΰ€ͺΰ€Ύΰ€²) from both harvest batches; original run: train 4,096 / dev 1,528 / test 1,139, cross_split_leaked_dropped: 37.

Rebuild runs launched 2026-07-05:

  • Batch-1 re-harvest (23 weekly windows 2026-01-25 β†’ 2026-07-04, publication-pages 80) β†’ outputs/gorkhapatra-language-pages-full-rebuild-b1-2026-07-05/ β€” done: 400 items, 197 aligned, 398 hits, warnings=0 (original: 400/195/394/0). 9 Limbu alignments, 4 with PDFs in this window; the other 5 dates fall in batch 2. Note: the first three attempts died on WAF connection resets β€” the epaper server intermittently blocks the tool UA ocrtech-source-audit/0.1 (browser UA passes); fix in flight in src/ocrtech/source_audit.py (Codex: UA constant + retry/backoff + on-disk PDF resume).
  • Batch-2 re-harvest (25 weekly windows 2025-08-03 β†’ 2026-01-24, publication-pages 160) β†’ outputs/gorkhapatra-language-pages-full-rebuild-b2-2026-07-05/, running. Original result: 798 items, 294 aligned, 589 hits.
  • Batch-1 silver line packs β†’ outputs/language-pack-silver-lines-rebuild-b1-2026-07-05/ β€” done: 122,882 kept, 50 languages, zero errors (original 116,287).
  • Batch-2 silver line packs β†’ outputs/language-pack-silver-lines-rebuild-b2-2026-07-05/ β€” done: 184,089 kept, 49 languages, zero errors (original 183,324).
  • Limbu silver corpus rebuilt at 2Γ— the original: outputs/limbu-gorkhapatra-pdf-backed-silver-rebuild-2026-07-05-v1/ β€” 17 discovered dates (vs 9), 16 usable pages / 4,872 usable silver labels (vs 8 / 2,430), only the known visual-only 2082-11-12 missing; page eval pack at outputs/limbu-page-eval-rebuild-2026-07-05-v1/ (16 pages, 4,895 lines).
  • Newar frozen pack recovered AND rebuilt, split determinism proven:
    • Recovered original (from the AWS payload, all 6,763 crops + labels): outputs/newar-devanagari-silver-pack-v1-recovered/ β€” use this for score-comparability with the recorded newar-ft-v1 numbers.
    • Rebuild from re-harvest: outputs/newar-devanagari-silver-pack-rebuild-2026-07-05/ (sha 5908d682… β‰  original 1fa7b23b… for a verified benign reason: train and dev are set-identical to the original 4,096/1,528; test is a strict superset β€” all 1,139 original lines + 7 rows (3 unique new texts) the richer re-harvest found on the same test dates 2082-07-25/2083-01-05).
  • PP-StructureV3 spike completed end-to-end (stage 1 on the L4 GPU, 16/16 pages at ~4 s/page; stage 2 champion-through-its-boxes scored vs the rebuilt Rows-LTR baseline). Verdict in RESEARCH.md: PP-StructureV3 stays fallback β€” the current Paddle det + Rows-LTR + y/conf70 chain wins every metric (page cp CER 0.413 vs 0.421, det F1 0.966 vs 0.934, matched CER 0.0266 vs 0.0331, order 0.993 vs 0.778 native). Baseline reproduction on the new pack also validated the lost baseline numbers (0.4134 vs old 0.4228).