smoke_signal/README.md

# Totem: Smoke Signal v0.1

Picture-book OCR & Codex extraction pipeline.

## Folder Structure

```
smoke_signal/
├── source_pdfs/       # Immutable — never overwrite source PDFs
├── manifest/          # source_manifest.csv, run logs
├── renders/           # Page PNG/TIFF renders from PDF
├── ocr_raw/           # Raw OCR outputs per page/region
├── regions/           # Region detection JSON, bounding boxes
├── review/            # Human review queue CSVs and corrections
├── exports/           # Codex-ready JSONL/CSV/Markdown exports
├── logs/              # Per-run timestamped logs
├── configs/           # Frozen config files per run
├── schemas/           # JSON schemas for all output types
├── reports/           # Validation and batch reports
└── scripts/           # Pipeline scripts
```

## Quickstart

```bash
# 1. Install dependencies
pip install -r requirements.txt

# 2. Add PDFs to source_pdfs/ folder

# 3. Register and hash all sources
python scripts/01_register_sources.py

# 4. Review manifest/source_manifest.csv and set rights_class for each book

# 5. Run PDF profiler (Stage 3)
# python scripts/02_profile_pdfs.py  ← coming in next stage
```

## Control Principles

| Rule | Description |
|------|-------------|
| No silent acquisition | Every source file must be logged with rights/provenance/hash before extraction |
| No OCR soup in Codex | Low-confidence pages must be flagged before influencing fingerprints |
| No extractor logic drift | Changes to scoring/labels/thresholds require a versioned issue |
| Page-level traceability | Every exported line maps back to book ID, page, region, and extraction method |

## Naming Convention

- Files: `BOOKID_page_REGIONID.ext`
- Batches: `SS-BATCH-001`, `SS-BATCH-002`, ...
- Configs: `ss_ocr_config_v0.1.json`
- Schemas: `schema_book_manifest_v1.json`
# push test