smoke_signal/README.md

Totem: Smoke Signal v0.1

Picture-book OCR & Codex extraction pipeline.

Folder Structure

smoke_signal/
├── source_pdfs/       # Immutable — never overwrite source PDFs
├── manifest/          # source_manifest.csv, run logs
├── renders/           # Page PNG/TIFF renders from PDF
├── ocr_raw/           # Raw OCR outputs per page/region
├── regions/           # Region detection JSON, bounding boxes
├── review/            # Human review queue CSVs and corrections
├── exports/           # Codex-ready JSONL/CSV/Markdown exports
├── logs/              # Per-run timestamped logs
├── configs/           # Frozen config files per run
├── schemas/           # JSON schemas for all output types
├── reports/           # Validation and batch reports
└── scripts/           # Pipeline scripts

Quickstart

# 1. Install dependencies
pip install -r requirements.txt

# 2. Add PDFs to source_pdfs/ folder

# 3. Register and hash all sources
python scripts/01_register_sources.py

# 4. Review manifest/source_manifest.csv and set rights_class for each book

# 5. Run PDF profiler (Stage 3)
# python scripts/02_profile_pdfs.py  ← coming in next stage

Control Principles

Rule	Description
No silent acquisition	Every source file must be logged with rights/provenance/hash before extraction
No OCR soup in Codex	Low-confidence pages must be flagged before influencing fingerprints
No extractor logic drift	Changes to scoring/labels/thresholds require a versioned issue
Page-level traceability	Every exported line maps back to book ID, page, region, and extraction method

Naming Convention

• Files: BOOKID_page_REGIONID.ext
• Batches: SS-BATCH-001, SS-BATCH-002, ...
• Configs: ss_ocr_config_v0.1.json
• Schemas: schema_book_manifest_v1.json

push test