Upload 13 files

smoke_signal/.gitignore

@@ -0,0 +1,34 @@
+# Source PDFs — never commit these (large files + rights issues)
+source_pdfs/
+
+# Derived processing outputs — large, regenerable
+renders/
+ocr_raw/
+regions/
+
+# Keep review decisions and exports (small, important)
+# review/  ← intentionally tracked
+# exports/ ← intentionally tracked
+
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.venv/
+venv/
+dist/
+build/
+
+# Environment
+.env
+.env.local
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Logs (keep manifests and reports, not raw logs)
+logs/
+
+# HF cache
+.cache/

smoke_signal/README.md

@@ -0,0 +1,54 @@
+# Totem: Smoke Signal v0.1
+
+Picture-book OCR & Codex extraction pipeline.
+
+## Folder Structure
+
+```
+smoke_signal/
+├── source_pdfs/       # Immutable — never overwrite source PDFs
+├── manifest/          # source_manifest.csv, run logs
+├── renders/           # Page PNG/TIFF renders from PDF
+├── ocr_raw/           # Raw OCR outputs per page/region
+├── regions/           # Region detection JSON, bounding boxes
+├── review/            # Human review queue CSVs and corrections
+├── exports/           # Codex-ready JSONL/CSV/Markdown exports
+├── logs/              # Per-run timestamped logs
+├── configs/           # Frozen config files per run
+├── schemas/           # JSON schemas for all output types
+├── reports/           # Validation and batch reports
+└── scripts/           # Pipeline scripts
+```
+
+## Quickstart
+
+```bash
+# 1. Install dependencies
+pip install -r requirements.txt
+
+# 2. Add PDFs to source_pdfs/ folder
+
+# 3. Register and hash all sources
+python scripts/01_register_sources.py
+
+# 4. Review manifest/source_manifest.csv and set rights_class for each book
+
+# 5. Run PDF profiler (Stage 3)
+# python scripts/02_profile_pdfs.py  ← coming in next stage
+```
+
+## Control Principles
+
+| Rule | Description |
+|------|-------------|
+| No silent acquisition | Every source file must be logged with rights/provenance/hash before extraction |
+| No OCR soup in Codex | Low-confidence pages must be flagged before influencing fingerprints |
+| No extractor logic drift | Changes to scoring/labels/thresholds require a versioned issue |
+| Page-level traceability | Every exported line maps back to book ID, page, region, and extraction method |
+
+## Naming Convention
+
+- Files: `BOOKID_page_REGIONID.ext`
+- Batches: `SS-BATCH-001`, `SS-BATCH-002`, ...
+- Configs: `ss_ocr_config_v0.1.json`
+- Schemas: `schema_book_manifest_v1.json`

smoke_signal/README_HF.md

@@ -0,0 +1,15 @@
+---
+title: Smoke Signal
+emoji: 🔭
+colorFrom: gray
+colorTo: slate
+sdk: gradio
+sdk_version: "4.0.0"
+app_file: app.py
+pinned: false
+license: mit
+---
+
+# Totem: Smoke Signal
+
+Picture-book OCR & Codex extraction pipeline with human review workbench.

smoke_signal/app.py

@@ -0,0 +1,45 @@
+"""
+Smoke Signal — Hugging Face Spaces entry point
+===============================================
+This file is the Gradio app for the human review workbench (Stage 8).
+Currently a placeholder — will be built after Stage 1-7 pipeline is working.
+
+To run locally:
+    pip install gradio
+    python app.py
+
+To deploy to HF Spaces:
+    Push this repo to a Hugging Face Space with SDK: gradio
+"""
+
+import gradio as gr
+
+def placeholder(message):
+    return f"Smoke Signal review workbench — coming in Stage 8. You said: {message}"
+
+with gr.Blocks(title="Smoke Signal — Review Workbench") as demo:
+    gr.Markdown("""
+    # 🔭 Totem: Smoke Signal
+    ### Picture-Book OCR & Codex Extraction — Human Review Workbench
+
+    **Status:** Pipeline under construction (Stage 0–1 complete)
+
+    Current stages:
+    - ✅ Stage 0: Project lock & scaffold
+    - ✅ Stage 1: Source registry & hashing
+    - 🔲 Stage 2: Calibration corpus selection
+    - 🔲 Stage 3: PDF profiling & rendering
+    - 🔲 Stage 4: OCR bake-off (Surya baseline)
+    - 🔲 Stage 5: Region detection
+    - 🔲 Stage 6: Reading order resolver
+    - 🔲 Stage 7: LLM normalisation
+    - 🔲 Stage 8: This review workbench
+    """)
+
+    with gr.Row():
+        msg = gr.Textbox(label="Test input")
+        out = gr.Textbox(label="Output")
+    msg.submit(placeholder, inputs=msg, outputs=out)
+
+if __name__ == "__main__":
+    demo.launch()

smoke_signal/manifest/PROJECT_CHARTER.md

@@ -0,0 +1,82 @@
+# Smoke Signal v0.1 — Project Charter
+
+**Project name:** Totem: Smoke Signal  
+**Version:** 0.1 Discovery  
+**Date:** 2026-05-16  
+**Status:** Active
+
+---
+
+## Purpose
+
+Smoke Signal is a narrow, reliable extraction subsystem.  
+Its job: convert messy picture-book PDFs into structured, traceable, reviewable source text and extraction metadata for the wider TOTEM/Codex environment.
+
+It is **not** a general writing assistant.  
+It is **not** a Codex fingerprint scorer.  
+It is **not** a content generator.
+
+---
+
+## Inputs
+
+- 150+ picture-book PDF source files (various scan quality, layouts, rights classes)
+- Source manifest with rights classification per book
+- Frozen pipeline configs per run
+
+---
+
+## Outputs
+
+- `codex_export.jsonl` — Codex-ready structured text with full provenance
+- `page_profile.json` — Per-book page routing metadata
+- `review_queue.csv` — Human review queue for low-confidence pages
+- `batch_report.md` — Per-batch validation summary
+- `source_manifest.csv` — Immutable source registry
+
+---
+
+## Exclusions
+
+- Does **not** tune Codex fingerprint scoring logic
+- Does **not** acquire new source material autonomously
+- Does **not** export unreviewed low-confidence text as clean source
+- Does **not** mix rights classes in the same validation batch
+
+---
+
+## Risk Rules
+
+| Risk | Control |
+|------|---------|
+| OCR noise entering Codex | Confidence thresholds + human review gate |
+| Rights violation | Rights class separation; unknown = excluded from export |
+| Silent logic drift | All config/schema/threshold changes require versioned issue |
+| Hallucinated text | LLM outputs must be strict JSON; uncertain words flagged explicitly |
+| Data loss | Source PDFs immutable; all processing to derived folders only |
+
+---
+
+## Approval Gates
+
+| Gate | Condition |
+|------|-----------|
+| A | Source manifest complete before extraction begins |
+| B | Calibration corpus approved before OCR bake-off |
+| C | Baseline config approved before validation run |
+| D | Validation report accepted before full-library batching |
+| E | Batch QA accepted before Codex export |
+| F | Closure pack accepted before declaring v1.0 |
+
+---
+
+## Roles
+
+| Role | Responsibility |
+|------|---------------|
+| Human operator | Rights classification, review queue decisions, gate approvals |
+| AI operator | Plans, schemas, scripts, reports — within approved configs only |
+
+---
+
+*This charter was created as part of Smoke Signal Stage 0 (Project Lock & Scope).*

smoke_signal/manifest/decision_log.csv

@@ -0,0 +1 @@
+date,decision,reason,approver,affected_files_configs,notes

smoke_signal/manifest/issue_log.csv

@@ -0,0 +1 @@
+issue_id,date_raised,raised_by,type,description,affected_component,status,resolution,date_resolved,approver

smoke_signal/manifest/run_log.csv

@@ -0,0 +1 @@
+run_id,date,operator,config_version,schema_version,source_batch,pages_processed,errors,cost_usd,output_path,notes

smoke_signal/manifest/source_manifest.csv

@@ -0,0 +1 @@
+book_id,source_id,filename,sha256,file_size_bytes,page_count,rights_class,source_location,acquisition_date,status,allowed_use,notes

smoke_signal/requirements.txt

@@ -0,0 +1,35 @@
+# Smoke Signal v0.1 — Core dependencies
+
+# PDF handling
+pdfplumber>=0.10.0
+pymupdf>=1.23.0          # fitz — fast rendering and text layer detection
+pypdf>=3.0.0
+
+# OCR — Surya (HF-native, primary engine)
+surya-ocr>=0.4.0
+
+# Image processing
+Pillow>=10.0.0
+numpy>=1.24.0
+
+# Data handling
+pandas>=2.0.0
+jsonschema>=4.20.0
+
+# Hugging Face
+transformers>=4.40.0
+huggingface_hub>=0.22.0
+datasets>=2.18.0
+
+# Utilities
+tqdm>=4.66.0
+rich>=13.0.0             # CLI output formatting
+click>=8.1.0             # CLI argument parsing
+python-dotenv>=1.0.0
+
+# Hashing / manifest
+hashlib                  # stdlib — no install needed
+
+# Optional: Tesseract fallback
+# pytesseract>=0.3.10
+# Install tesseract system package: sudo apt-get install tesseract-ocr

smoke_signal/schemas/schema_book_manifest_v1.json

@@ -0,0 +1,68 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "$id": "schema_book_manifest_v1.json",
+  "schema_version": "1.0",
+  "title": "Smoke Signal — Source Manifest Record",
+  "type": "object",
+  "required": [
+    "book_id", "source_id", "filename", "sha256",
+    "file_size_bytes", "page_count", "rights_class",
+    "source_location", "acquisition_date", "status"
+  ],
+  "properties": {
+    "book_id": {
+      "type": "string",
+      "pattern": "^SS-BOOK-[0-9]{4}$",
+      "description": "Stable unique book identifier, e.g. SS-BOOK-0001"
+    },
+    "source_id": {
+      "type": "string",
+      "description": "Optional secondary source tracking ID"
+    },
+    "filename": {
+      "type": "string",
+      "description": "Original filename as received"
+    },
+    "sha256": {
+      "type": "string",
+      "pattern": "^[a-f0-9]{64}$",
+      "description": "SHA-256 hash of the source PDF file"
+    },
+    "file_size_bytes": {
+      "type": "integer",
+      "minimum": 0
+    },
+    "page_count": {
+      "type": ["integer", "null"],
+      "minimum": 0,
+      "description": "null if not yet profiled"
+    },
+    "rights_class": {
+      "type": "string",
+      "enum": ["public-domain", "licensed-owned", "controlled-internal", "unknown", "excluded"],
+      "description": "Rights classification — unknown blocks training/export use"
+    },
+    "source_location": {
+      "type": "string",
+      "description": "Where the file was acquired from"
+    },
+    "acquisition_date": {
+      "type": "string",
+      "format": "date",
+      "description": "ISO 8601 date: YYYY-MM-DD"
+    },
+    "status": {
+      "type": "string",
+      "enum": ["pending", "profiled", "rendered", "ocred", "reviewed", "exported", "quarantined"],
+      "description": "Current processing status"
+    },
+    "allowed_use": {
+      "type": ["string", "null"],
+      "description": "What this source may be used for: extraction, training, both, none"
+    },
+    "notes": {
+      "type": ["string", "null"]
+    }
+  },
+  "additionalProperties": false
+}

smoke_signal/schemas/schema_codex_export_v1.json

@@ -0,0 +1,52 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "$id": "schema_codex_export_v1.json",
+  "schema_version": "1.0",
+  "title": "Smoke Signal — Codex Export Record",
+  "type": "object",
+  "required": [
+    "book_id", "source_hash", "page_number", "region_id",
+    "page_class", "region_class", "text_final", "confidence",
+    "review_status", "extraction_method", "run_id", "schema_version"
+  ],
+  "properties": {
+    "book_id":           { "type": "string" },
+    "source_id":         { "type": ["string", "null"] },
+    "source_hash":       { "type": "string", "pattern": "^[a-f0-9]{64}$" },
+    "page_number":       { "type": "integer", "minimum": 1 },
+    "region_id":         { "type": "string", "description": "BOOKID_page_REGIONID format" },
+    "page_class": {
+      "type": "string",
+      "enum": ["cover", "title", "copyright", "story-page", "illustration-only",
+               "endpaper", "table-list", "advertisement-appendix", "unknown"]
+    },
+    "region_class": {
+      "type": "string",
+      "enum": ["narration", "dialogue-speech-bubble", "title", "subtitle",
+               "caption", "sign-label", "page-number", "copyright-legal",
+               "publisher-imprint", "decorative-uncertain"]
+    },
+    "text_raw":          { "type": ["string", "null"], "description": "Raw OCR output before cleanup" },
+    "text_clean":        { "type": ["string", "null"], "description": "After LLM normalisation" },
+    "text_final":        { "type": "string", "description": "Approved final text for Codex" },
+    "uncertain_words":   { "type": "array", "items": { "type": "string" } },
+    "confidence": {
+      "type": "number", "minimum": 0.0, "maximum": 1.0,
+      "description": "Composite confidence score"
+    },
+    "review_status": {
+      "type": "string",
+      "enum": ["auto-accepted", "human-accepted", "human-edited", "quarantined", "excluded"]
+    },
+    "exclusion_reason":  { "type": ["string", "null"] },
+    "extraction_method": {
+      "type": "string",
+      "enum": ["embedded-text", "ocr", "ocr+llm", "manual", "excluded"]
+    },
+    "config_version":    { "type": ["string", "null"] },
+    "schema_version":    { "type": "string" },
+    "run_id":            { "type": "string" },
+    "prompt_version":    { "type": ["string", "null"] }
+  },
+  "additionalProperties": false
+}

smoke_signal/scripts/01_register_sources.py

@@ -0,0 +1,258 @@
+#!/usr/bin/env python3
+"""
+Smoke Signal — Stage 1: Source Registry
+========================================
+Scans /source_pdfs, hashes every PDF, and writes/updates source_manifest.csv.
+
+Usage:
+    python scripts/01_register_sources.py
+    python scripts/01_register_sources.py --source-dir /path/to/pdfs
+    python scripts/01_register_sources.py --validate-only
+"""
+
+import hashlib
+import csv
+import json
+import os
+import sys
+import argparse
+from datetime import date
+from pathlib import Path
+from typing import Optional
+
+try:
+    from rich.console import Console
+    from rich.table import Table
+    from rich.progress import track
+    RICH = True
+except ImportError:
+    RICH = False
+
+# ── Paths ──────────────────────────────────────────────────────────────────────
+ROOT         = Path(__file__).resolve().parents[1]
+SOURCE_DIR   = ROOT / "source_pdfs"
+MANIFEST_CSV = ROOT / "manifest" / "source_manifest.csv"
+SCHEMA_FILE  = ROOT / "schemas" / "schema_book_manifest_v1.json"
+
+console = Console() if RICH else None
+
+MANIFEST_FIELDS = [
+    "book_id", "source_id", "filename", "sha256", "file_size_bytes",
+    "page_count", "rights_class", "source_location", "acquisition_date",
+    "status", "allowed_use", "notes"
+]
+
+RIGHTS_CLASSES = {"public-domain", "licensed-owned", "controlled-internal", "unknown", "excluded"}
+
+
+# ── Hashing ────────────────────────────────────────────────────────────────────
+def sha256_file(path: Path, chunk: int = 1 << 20) -> str:
+    """Return hex SHA-256 of a file, reading in chunks."""
+    h = hashlib.sha256()
+    with open(path, "rb") as f:
+        for block in iter(lambda: f.read(chunk), b""):
+            h.update(block)
+    return h.hexdigest()
+
+
+# ── Page count (best-effort, no hard dependency on PDF libs) ──────────────────
+def get_page_count(path: Path) -> Optional[int]:
+    """Try to count pages without a hard crash if libs are missing."""
+    try:
+        import fitz  # pymupdf
+        doc = fitz.open(str(path))
+        count = doc.page_count
+        doc.close()
+        return count
+    except Exception:
+        pass
+    try:
+        import pdfplumber
+        with pdfplumber.open(str(path)) as pdf:
+            return len(pdf.pages)
+    except Exception:
+        return None
+
+
+# ── Book ID generation ─────────────────────────────────────────────────────────
+def next_book_id(existing: dict) -> str:
+    """Generate the next SS-BOOK-NNNN ID not already in the manifest."""
+    used = {v["book_id"] for v in existing.values()}
+    for i in range(1, 10_000):
+        candidate = f"SS-BOOK-{i:04d}"
+        if candidate not in used:
+            return candidate
+    raise RuntimeError("Ran out of book IDs — this shouldn't happen.")
+
+
+# ── Manifest I/O ───────────────────────────────────────────────────────────────
+def load_manifest() -> dict:
+    """Load manifest CSV keyed by sha256."""
+    records = {}
+    if not MANIFEST_CSV.exists():
+        return records
+    with open(MANIFEST_CSV, newline="", encoding="utf-8") as f:
+        reader = csv.DictReader(f)
+        for row in reader:
+            if row.get("sha256"):
+                records[row["sha256"]] = row
+    return records
+
+
+def save_manifest(records: dict) -> None:
+    """Write manifest back to CSV, sorted by book_id."""
+    rows = sorted(records.values(), key=lambda r: r.get("book_id", ""))
+    with open(MANIFEST_CSV, "w", newline="", encoding="utf-8") as f:
+        writer = csv.DictWriter(f, fieldnames=MANIFEST_FIELDS)
+        writer.writeheader()
+        writer.writerows(rows)
+
+
+# ── Main ───────────────────────────────────────────────────────────────────────
+def register_sources(source_dir: Path, validate_only: bool = False) -> None:
+    pdf_files = sorted(source_dir.glob("*.pdf"))
+
+    if not pdf_files:
+        print(f"[warn] No PDFs found in {source_dir}")
+        print("       Add your source PDFs to source_pdfs/ and re-run.")
+        return
+
+    print(f"\n{'='*60}")
+    print(f"  Smoke Signal — Stage 1: Source Registry")
+    print(f"{'='*60}")
+    print(f"  Found {len(pdf_files)} PDF(s) in {source_dir}\n")
+
+    existing = load_manifest()
+    new_count = 0
+    updated_count = 0
+    duplicate_count = 0
+
+    iterable = track(pdf_files, description="Hashing PDFs...") if RICH else pdf_files
+
+    for pdf_path in iterable:
+        if not RICH:
+            print(f"  Processing: {pdf_path.name}")
+
+        file_hash = sha256_file(pdf_path)
+        file_size = pdf_path.stat().st_size
+
+        if file_hash in existing:
+            rec = existing[file_hash]
+            # Update file size if it changed (shouldn't, but track it)
+            if rec["filename"] != pdf_path.name:
+                print(f"  [dup] {pdf_path.name} → same content as {rec['filename']} ({rec['book_id']})")
+                duplicate_count += 1
+            continue
+
+        # New source
+        page_count = get_page_count(pdf_path)
+        book_id    = next_book_id(existing)
+
+        record = {
+            "book_id":          book_id,
+            "source_id":        "",
+            "filename":         pdf_path.name,
+            "sha256":           file_hash,
+            "file_size_bytes":  file_size,
+            "page_count":       page_count if page_count is not None else "",
+            "rights_class":     "unknown",        # MUST be set manually
+            "source_location":  str(source_dir),
+            "acquisition_date": date.today().isoformat(),
+            "status":           "pending",
+            "allowed_use":      "",
+            "notes":            ""
+        }
+        existing[file_hash] = record
+        new_count += 1
+
+        status_icon = "✓" if page_count else "?"
+        print(f"  [{status_icon}] Registered {book_id} — {pdf_path.name} ({page_count or '?'} pages)")
+
+    if not validate_only:
+        save_manifest(existing)
+        print(f"\n  Manifest saved → {MANIFEST_CSV}")
+
+    # ── Summary ───────────────────────────────────────────────────────────────
+    total = len(existing)
+    unknown_rights = sum(1 for r in existing.values() if r["rights_class"] == "unknown")
+
+    print(f"\n{'─'*60}")
+    print(f"  Total sources registered : {total}")
+    print(f"  New this run             : {new_count}")
+    print(f"  Duplicates skipped       : {duplicate_count}")
+    print(f"  Rights class = unknown   : {unknown_rights}  ← ACTION REQUIRED")
+    print(f"{'─'*60}\n")
+
+    if unknown_rights > 0:
+        print("  ⚠️  ACTION: Open manifest/source_manifest.csv and set")
+        print("     rights_class for each book before extraction.")
+        print("     Valid values: public-domain | licensed-owned |")
+        print("                   controlled-internal | unknown | excluded\n")
+
+    # ── Rights breakdown table ────────────────────────────────────────────────
+    rights_counts = {}
+    for r in existing.values():
+        rc = r.get("rights_class", "unknown")
+        rights_counts[rc] = rights_counts.get(rc, 0) + 1
+
+    print("  Rights class breakdown:")
+    for rc, count in sorted(rights_counts.items()):
+        flag = "  ← EXCLUDED FROM PROCESSING" if rc in ("unknown", "excluded") else ""
+        print(f"    {rc:<25} {count}{flag}")
+    print()
+
+
+def validate_manifest() -> bool:
+    """Validate all manifest records against JSON schema."""
+    try:
+        import jsonschema
+    except ImportError:
+        print("[skip] jsonschema not installed — skipping schema validation")
+        return True
+
+    with open(SCHEMA_FILE) as f:
+        schema = json.load(f)
+
+    records = load_manifest()
+    errors = []
+    for sha, rec in records.items():
+        # Convert numeric strings for validation
+        test_rec = dict(rec)
+        if test_rec.get("file_size_bytes"):
+            test_rec["file_size_bytes"] = int(test_rec["file_size_bytes"])
+        if test_rec.get("page_count"):
+            try:
+                test_rec["page_count"] = int(test_rec["page_count"])
+            except (ValueError, TypeError):
+                test_rec["page_count"] = None
+
+        try:
+            jsonschema.validate(test_rec, schema)
+        except jsonschema.ValidationError as e:
+            errors.append(f"  {rec.get('book_id', sha[:8])}: {e.message}")
+
+    if errors:
+        print(f"\n  ✗ Schema validation FAILED ({len(errors)} errors):")
+        for err in errors:
+            print(err)
+        return False
+    else:
+        print(f"\n  ✓ All {len(records)} manifest records pass schema validation.")
+        return True
+
+
+# ── CLI ────────────────────────────────────────────────────────────────────────
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description="Smoke Signal — Stage 1: Source Registry")
+    parser.add_argument("--source-dir",    type=Path, default=SOURCE_DIR,
+                        help="Directory containing source PDFs")
+    parser.add_argument("--validate-only", action="store_true",
+                        help="Only validate existing manifest, don't scan for new files")
+    args = parser.parse_args()
+
+    if args.validate_only:
+        ok = validate_manifest()
+        sys.exit(0 if ok else 1)
+    else:
+        register_sources(args.source_dir)
+        validate_manifest()