seed: specs + loop scaffolding

.claude/agents/verifier.md

@@ -0,0 +1,33 @@
+---
+name: verifier
+description: Independent reviewer for completed build tasks. Re-runs the test
+  suite and checks finished work against the specs and the acceptance criteria,
+  reporting discrepancies. Use as a final adversarial pass after a batch of
+  tasks, or whenever you want a second opinion that did not write the code.
+tools: Read, Bash, Grep, Glob
+---
+
+You are a verifier. You did not write this code, and your job is to find where
+it falls short — not to be agreeable. The maker is too generous grading its own
+work; you are the check.
+
+When invoked, do the following and report concisely:
+
+1. Run the full suite and linter yourself: `uv run pytest -q` and
+   `uv run ruff check .`. Paste the real output. Do not trust a prior claim that
+   they passed — run them.
+2. For each task marked complete in `PROGRESS.md` (TONIGHT section), open the
+   corresponding code and confirm it actually satisfies that task's acceptance
+   criterion in `docs/05_build_plan.md` and the relevant spec in `docs/`. Look
+   specifically for the hard part being skipped: a function that returns a
+   placeholder, a test that asserts nothing meaningful, a rule that is declared
+   but never applied, an arithmetic check that does not actually reconcile.
+3. Confirm scope was respected: no task under **TOMORROW** was started, and no
+   dependency outside task **N1**'s night set was added (check `pyproject.toml`
+   / `uv.lock`).
+4. Confirm the core stays decoupled: `core.py` must not import from `ingest/`,
+   `web/`, or `store/`, and no provider SDK is called outside a backend adapter.
+
+Report as: what genuinely passes, what is incomplete or wrong (with file and
+line), and anything out of scope. If everything holds, say so plainly. If not,
+list the specific gaps so the maker can fix them. Do not edit code yourself.

CLAUDE.md

@@ -0,0 +1,109 @@
+# CLAUDE.md — Working Conventions for This Project
+
+## What this project is
+
+An autonomous document-extraction agent for invoices/receipts. A reusable core
+pipeline (`process_document`) turns a document into a validated, structured
+record and decides whether to **auto-accept** it or route it to **review**. The
+core is invoked by two thin entry points (a folder watcher and a Gradio web
+demo) and depends on a swappable model backend (Gemini free tier or local
+Ollama). Full detail in `docs/`.
+
+Read `docs/01_requirements.md`, `docs/02_architecture.md`,
+`docs/03_data_and_extraction_spec.md`, and `docs/04_project_setup.md` before
+making design decisions. Follow `docs/05_build_plan.md` for task order.
+
+## Tech stack
+
+Python **3.11** (pinned via `.python-version`; `requires-python = ">=3.11"`),
+**uv** (package/venv management — `uv add`, `uv sync`, `uv run`; commit
+`uv.lock` and `.python-version`), Docling (parsing), Pydantic v2
+(contract/validation), google-genai (Gemini) + local Ollama, Gradio (demo),
+SQLite + CSV (storage), watchdog (watcher), pytest (tests).
+
+## Architectural rules (do not violate)
+
+1. **Core stays decoupled.** `core.py` must not import from `ingest/`, `web/`,
+   or `store/`. Entry points depend on the core, never the reverse. The core
+   returns a result object; it performs no file moves and no DB writes.
+2. **Backends sit behind the interface.** All model access goes through
+   `ExtractionBackend`. No entry point or core code calls a provider SDK
+   directly. Adding a backend = implement the interface + register in the
+   factory; nothing else changes.
+3. **Model identifiers are config, never literals.** Free model catalogs change
+   without notice — a hardcoded model name is a latent outage. Read names from
+   config; treat a missing/renamed model as a recoverable config error with a
+   clear message.
+4. **Structured output is enforced, not parsed.** Use schema/grammar-constrained
+   output (Pydantic schema for Gemini, JSON-schema/grammar for Ollama). Never
+   regex JSON out of free-form text.
+5. **Validation gates auto-accept.** A hard-rule failure (especially an
+   arithmetic cross-check) forces `review` regardless of model confidence. The
+   model is treated as fallible by design.
+6. **The loop never dies on one document.** Every document is processed in
+   isolation with try/except; failures log full context and route to review,
+   then processing continues.
+7. **Pure functions stay pure.** `validation/rules.py` and `routing/score.py`
+   do no I/O and are fully unit-tested.
+
+## Precision posture (important)
+
+Optimize **precision on the auto-accepted path** for the critical fields
+`total`, `tax`, `invoice_number`. A confidently-wrong number is the costly
+error because it is written and propagates silently; a missing field is caught
+by review. When in doubt, route to review. Recall is measured and traded
+against precision via the single `CONFIDENCE_THRESHOLD`, set empirically in
+evaluation. Arithmetic cross-checks are the mechanism that keeps precision high
+without destroying recall.
+
+## Coding conventions
+
+- Type-hint everything; keep functions small and single-purpose.
+- Pydantic models are the single source of truth for the data contract.
+- Fail fast on misconfiguration at startup with actionable messages.
+- Structured logging (one record per document: inputs, backend, decision,
+  validation failures, timings). No secrets in logs.
+- Bounded retries + timeouts on all network/model calls.
+- No secrets in code or git. Config via `.env` / Space secrets only.
+- Manage dependencies with uv; commit `uv.lock` so installs are reproducible.
+  Add deps via `uv add`, never by hand-editing pins. Run commands via `uv run`.
+
+## Testing & definition of done
+
+- A task is done when its build-plan **[AC]** is met and tests pass.
+- Pure logic (schema, validation, routing) must have unit tests before the
+  pipeline is wired together.
+- `test_core_smoke.py` runs the pipeline end-to-end with a stub backend (no
+  network) so the core is testable offline.
+- The watcher must survive a deliberately corrupt file (routes to review,
+  loop continues).
+
+## Privacy & cost guardrails
+
+- Development and demo must remain **free**: local Ollama (no quota) or Gemini
+  free tier; Hugging Face Spaces free CPU tier for hosting.
+- The public demo is **stateless** and must show a "synthetic/public documents
+  only" notice. Free hosted backends may train on inputs — never send real
+  financial data through the demo or a free API. Sensitive data is handled only
+  via the local Ollama backend.
+
+## Autonomous / overnight runs
+
+- The authoritative task ledger is `PROGRESS.md`. Do the next unchecked task in
+  its **TONIGHT** section; **never** start a task under **TOMORROW**; **never**
+  add a dependency outside the night set in `PROGRESS.md` task **N1**.
+- Commit cadence: **one commit per completed task**, using the message given in
+  `PROGRESS.md`. Never commit failing tests. Tick the task's box in the same or
+  an immediately following commit.
+- Prove completion in the transcript: run the task's acceptance check and paste
+  the output — an unattended loop's evaluator only sees what you print.
+- If blocked, record a one-line note under **BLOCKED** in `PROGRESS.md` and move
+  to the next task rather than halting the whole run.
+- The specs in `docs/` are inputs, not work products — do not edit them.
+
+## When unsure
+
+Prefer the choice that (a) keeps the core independent of entry points and
+backends, (b) routes uncertain results to review rather than auto-accepting,
+and (c) keeps the project runnable for free. Surface assumptions in code
+comments and the README rather than silently deciding.

LOOP.md

@@ -0,0 +1,103 @@
+# LOOP.md — How to Run This Overnight
+
+This runs the **TONIGHT** scope in `PROGRESS.md` unattended and stops at the
+`⛔ STOP` boundary. It builds the deterministic core (schema, validation,
+routing, backend interface + offline stub, stub-backed core pipeline). It needs
+**no Gemini key and no Ollama** — those are tomorrow.
+
+## Before you start
+
+- Be on a throwaway branch or an isolated worktree so the night's work is
+  contained and easy to review:
+  `git switch -c overnight-build` (or `git worktree add ../overnight overnight-build`).
+- Have these in the repo root: `CLAUDE.md`, `PROGRESS.md`, `LOOP.md`,
+  `.claude/agents/verifier.md`, and the specs in `docs/`.
+- Be logged into Claude Code as usual. The loop uses your existing auth; on a
+  subscription it counts against your normal weekly rate limit, on an API key it
+  bills pay-as-you-go (the `--max-budget-usd` guard below applies in that case).
+
+## Recommended: the driver loop (one task per iteration, commit each)
+
+A fresh context per iteration means no context-window drift over a long night,
+each task is its own commit, and a single crashed iteration just resumes from
+the ledger next loop.
+
+```bash
+#!/usr/bin/env bash
+# run-overnight.sh — stops itself at the PROGRESS.md ⛔ STOP boundary
+set -u
+for i in $(seq 1 20); do
+  echo "=== iteration $i ==="
+  claude -p "Read PROGRESS.md, CLAUDE.md, and the specs in docs/. Do the NEXT
+    unchecked task in the TONIGHT section only — exactly one. Implement it per
+    docs/05_build_plan.md. Run that task's Check command and paste the output.
+    If it passes: commit just that task's changes with its Commit message, then
+    tick its box in PROGRESS.md and commit that. If it fails after reasonable
+    attempts: add a one-line note under BLOCKED in PROGRESS.md, commit, and move
+    on. NEVER start a TOMORROW task. NEVER add a dependency outside task N1's
+    night set. NEVER edit files in docs/. When every TONIGHT box is checked, run
+    'uv run pytest -q' and 'uv run ruff check .'; if both are clean, print
+    exactly DONE_ALL." \
+    --allowedTools "Read,Edit,Write,Bash" \
+    --permission-mode acceptEdits \
+    --max-turns 25 \
+    --max-budget-usd 0.75 \
+    --output-format json | tee "run_$i.json"
+  if grep -q "DONE_ALL" "run_$i.json"; then
+    echo "All TONIGHT tasks complete."; break
+  fi
+done
+```
+
+Launch it and walk away:
+
+```bash
+chmod +x run-overnight.sh
+nohup ./run-overnight.sh > overnight.log 2>&1 &
+```
+
+(`nohup ... &` keeps it running if the terminal closes. For a laptop that
+sleeps, run it in `tmux` on a machine that stays awake.)
+
+## Alternative: a single `/goal` session
+
+Simpler, one process. `/goal` keeps the session going until a small evaluator
+model confirms the condition from the transcript, so the condition ends by
+running the proof commands. It has no built-in budget, hence the turn cap.
+
+```bash
+claude -p "/goal Every task in the TONIGHT section of PROGRESS.md is checked
+  off, and 'uv run pytest -q' exits 0, and 'uv run ruff check .' reports no
+  errors. Work one TONIGHT task at a time, top to bottom; after each task's
+  Check passes, commit it with its PROGRESS.md message and tick its box. Never
+  start a TOMORROW task. Never add a dependency outside task N1's night set.
+  Never edit docs/. Prove completion by running pytest and ruff and printing
+  'cat PROGRESS.md' at the end. Stop after 60 turns regardless." \
+  --allowedTools "Read,Edit,Write,Bash" \
+  --permission-mode acceptEdits
+```
+
+The driver loop is the safer choice for a long unattended night; `/goal` is
+fine if you prefer one process and will glance at it.
+
+## Guardrails (why this is safe to leave)
+
+- **Hard scope:** both the ledger and the prompt forbid starting TOMORROW work
+  or adding model dependencies, so it cannot wander into Gemini/Ollama.
+- **Isolation:** running on a branch/worktree means the morning review is a
+  clean diff and nothing touched `main`.
+- **Caps:** `--max-turns` (and `--max-budget-usd` on API billing) stop a runaway
+  iteration; the loop caps total iterations.
+- **Broad Bash is granted** so it can run `uv`, `pytest`, and `git` — which is
+  exactly why you keep it on an isolated branch.
+
+## In the morning
+
+1. `git log --oneline` — you should see roughly one commit per N-task.
+2. **Read the diffs.** Agents are good at *looking* done; the test suite is the
+   real gate, but skim the code. `uv run pytest -q` yourself.
+3. Check `PROGRESS.md`: which TONIGHT boxes are ticked, and read any **BLOCKED**
+   entries — those are your first fixes.
+4. Then continue with TOMORROW: set up the Gemini key (and/or Ollama), implement
+   the Docling/OCR parsing and the two backends, wire the real `acquire` into
+   `core.py`, then persistence, the watcher, the web demo, and the eval harness.

PROGRESS.md

@@ -0,0 +1,152 @@
+# PROGRESS.md — Autonomous Build Ledger
+
+This file is the memory for the overnight run. It outlives any single agent
+session. The loop reads it at the start of every iteration, does the **next
+unchecked task in the TONIGHT section only**, proves the task's acceptance
+check, commits, and ticks the box.
+
+## Protocol (read every iteration)
+
+1. Do the next unchecked task under **TONIGHT**, exactly one per iteration.
+2. Implement it per `docs/05_build_plan.md` and the other specs in `docs/`.
+3. Run the task's **Check** command and paste the output (the loop's evaluator
+   only sees what is printed).
+4. If it passes: commit just that task's changes with the listed **Commit**
+   message; tick the box; commit the ledger update.
+5. If it fails after reasonable attempts: add a one-line note under **BLOCKED**,
+   commit, and move to the next task. Do not halt the whole run.
+6. **Never** start a task under **TOMORROW**.
+7. **Never** add a dependency outside the night set in task **N1**.
+8. The specs already exist in `docs/` — do not regenerate or edit them.
+
+When every TONIGHT box is checked, run `uv run pytest -q` and
+`uv run ruff check .`; if both are clean, print `DONE_ALL` and stop.
+
+---
+
+## Night dependency set (the ONLY packages to add tonight)
+
+- Runtime: `pydantic`, `pydantic-settings`
+- Dev: `pytest`, `ruff`
+
+Everything else (docling, paddleocr/pytesseract, google-genai, ollama, gradio,
+watchdog) belongs to TOMORROW and must NOT be added tonight.
+
+---
+
+## TONIGHT
+
+- [ ] **N1 — Project scaffold** (build plan 0.1)
+  Run `uv init`; `uv python pin 3.11`; set `requires-python = ">=3.11"`. Create
+  the `src/doc_agent/` package tree and `tests/` from the setup-doc layout (the
+  `docs/` specs are already present — leave them). Add `.gitignore` (ignore
+  `data/`, `.env`, `.venv/`, caches; **commit** `uv.lock` and `.python-version`)
+  and an empty `README.md`. Add night deps: `uv add pydantic pydantic-settings`
+  and `uv add --dev pytest ruff`.
+  Check: `uv sync && uv run python -c "import sys; assert sys.version_info[:2]==(3,11)" && cat .python-version`
+  Commit: `phase 0.1: project scaffold (uv, py3.11, package layout)`
+
+- [ ] **N2 — Config loader** (0.2)
+  `src/doc_agent/config.py` with pydantic-settings: load env, validate combos
+  (gemini requires key; `vision_direct` requires a multimodal backend), fail
+  fast with clear messages. Add `tests/test_config.py`.
+  Check: `uv run pytest tests/test_config.py -q`
+  Commit: `phase 0.2: config loader with validation`
+
+- [ ] **N3 — Document schema** (1.1)
+  `src/doc_agent/schema/models.py`: `Document` and `LineItem` per
+  `docs/03_data_and_extraction_spec.md`, with money→float and date→ISO
+  normalizers. Add `tests/test_schema.py`.
+  Check: `uv run pytest tests/test_schema.py -q`
+  Commit: `phase 1.1: pydantic document schema + normalizers`
+
+- [ ] **N4 — Validation rules** (1.2)
+  `src/doc_agent/validation/rules.py`: hard rules H1–H4, soft rules S1–S4,
+  monetary epsilon, returning a structured report. Pure functions, no I/O. Add
+  `tests/test_validation.py` (reconciling totals pass H2/H3; mismatches fail;
+  soft failures recorded without forcing review).
+  Check: `uv run pytest tests/test_validation.py -q`
+  Commit: `phase 1.2: validation rules (hard/soft + arithmetic checks)`
+
+- [ ] **N5 — Confidence & routing** (1.3)
+  `src/doc_agent/routing/score.py`: pure `score(data, report, model_signal)` and
+  `route(score, report)` with hard-failure short-circuit. Add
+  `tests/test_routing.py` (hard fail ⇒ review regardless of score; threshold
+  boundary; missing required fields lower score).
+  Check: `uv run pytest tests/test_routing.py -q`
+  Commit: `phase 1.3: confidence scoring + routing decision`
+
+- [ ] **N6 — Modality detection** (2.1)
+  `src/doc_agent/parsing/detect.py`: map a file to `native_pdf | image` by
+  extension/MIME. Add `tests/test_detect.py`.
+  Check: `uv run pytest tests/test_detect.py -q`
+  Commit: `phase 2.1: modality detection`
+
+- [ ] **N7 — Backend interface + offline stub** (2.4 + stub)
+  `src/doc_agent/backends/base.py`: the `ExtractionBackend` protocol,
+  `BackendResult`, and a factory built from config.
+  `src/doc_agent/backends/stub.py`: a `StubBackend` returning deterministic,
+  schema-valid `Document` data with no network (this is the stub the smoke test
+  and CLAUDE.md reference). **Do NOT implement `gemini.py` or `ollama.py`
+  tonight.** Add `tests/test_backends.py` (factory returns configured backend;
+  unknown backend ⇒ clear error; stub returns schema-valid data).
+  Check: `uv run pytest tests/test_backends.py -q`
+  Commit: `phase 2.4: backend interface + factory + offline stub backend`
+
+- [ ] **N8 — Core pipeline orchestration** (3.1, stub-backed)
+  `src/doc_agent/core.py`: `process_document(path) -> ExtractionResult` chaining
+  detect → acquire → `backend.extract` → validate → score → route. Pure of
+  side-effects (no file moves, no DB). For tonight, define the `acquire`
+  interface and a minimal injectable path so the smoke test can supply a fake
+  payload — **real Docling/OCR `acquire` is a TOMORROW task**. Define the
+  `ExtractionResult` type here. Add `tests/test_core_smoke.py` running
+  end-to-end with `StubBackend` + an injected payload (no network), asserting a
+  populated result with a decision.
+  Check: `uv run pytest tests/test_core_smoke.py -q`
+  Commit: `phase 3.1: core pipeline orchestration (stub-backed smoke test)`
+
+- [ ] **N9 — Idempotency helper** (3.2)
+  A content-hash helper so the same file is not reprocessed across runs. Add
+  `tests/test_hash.py` (same file → same hash).
+  Check: `uv run pytest tests/test_hash.py -q`
+  Commit: `phase 3.2: content-hash idempotency helper`
+
+- [ ] **N-FINAL — Full green gate**
+  Run the whole suite and linter; fix anything red until clean.
+  Check: `uv run pytest -q && uv run ruff check .`
+  Commit: `chore: full suite + lint green` (only if any fixes were needed)
+
+---
+
+## ⛔ STOP — END OF AUTONOMOUS SCOPE
+
+Do not proceed past this line. Everything below requires API keys, a local
+model server, large/finicky native dependencies, or live datasets — all for the
+morning, supervised.
+
+---
+
+## TOMORROW (DO NOT START — supervised, needs setup)
+
+- [ ] Add deferred deps: `docling`, (`paddleocr` or `pytesseract`),
+  `google-genai`, `ollama`, `gradio`, `watchdog`.
+- [ ] 2.2 Docling parser (downloads layout models on first run).
+- [ ] 2.3 OCR path (PaddleOCR/Tesseract — watch the Paddle/3.11 wheel risk;
+  fall back to pytesseract if it won't resolve).
+- [ ] Wire the real Docling/OCR `acquire` into `core.py`.
+- [ ] 2.5 Gemini backend — **needs `GEMINI_API_KEY`**.
+- [ ] 2.6 Ollama backend — **needs a local Ollama server + pulled model**.
+- [ ] 4.1 persistence (SQLite + CSV); 4.2 watcher; 4.3 Gradio web demo.
+- [ ] 5 evaluation harness — needs datasets + a real backend; **counts against
+  the Gemini free-tier daily limit**, so run deliberately.
+- [ ] 6 deploy to Hugging Face Spaces.
+
+---
+
+## BLOCKED
+
+(none yet — the loop appends one-line entries here)
+
+## RUN LOG
+
+(the loop may append short per-iteration notes here)

docs/01_requirements.md

@@ -0,0 +1,145 @@
+# Requirements Specification — Document Extraction Agent
+
+## 1. Overview
+
+An autonomous agent that ingests invoices, receipts, and similar
+semi-structured financial documents, extracts their key fields into a
+validated structured record, and routes anything it is not confident about
+to a human review queue. The agent runs unattended over a stream of incoming
+documents and is also exposed through a small public web demo.
+
+The reasoning step (turning document text/images into structured fields) is
+performed by a swappable LLM backend. Everything around it — triggering,
+parsing, validation, confidence scoring, routing, persistence, logging — is
+application code. The engineering value of this project is the system around
+the model, not the model itself.
+
+## 2. Problem statement
+
+Manually keying fields off invoices and receipts is slow and error-prone, and
+the documents arrive in inconsistent formats and qualities (clean PDFs,
+flatbed scans, phone photos). We want a pipeline that processes them
+automatically, is confident only when it should be, and surfaces the rest for
+a human — with measurable accuracy.
+
+## 3. Goals
+
+- Ingest documents from a watched location with no human trigger per document.
+- Support three input modalities: native-text PDFs, scanned images, and phone
+  photos.
+- Extract a defined set of fields into a strict JSON schema.
+- Validate extracted values, including arithmetic consistency checks.
+- Assign a confidence to each document and route low-confidence documents to
+  review rather than auto-accepting them.
+- Persist accepted records and export them to CSV.
+- Provide a public demo URL where a single uploaded document is processed and
+  its result shown.
+- Run entirely on free infrastructure and free model access.
+- Be measurable against ground-truth datasets (precision, recall, F1).
+
+## 4. Non-goals (explicitly out of scope for v1)
+
+- Fine-tuning or training a model. Off-the-shelf models only.
+- A full review *application* with auth, multi-user workflows, or audit trails.
+  The review queue is a directory plus a CSV, not a product.
+- Persistent multi-tenant storage in the cloud demo. The demo is
+  presentation-only and stateless.
+- Handling non-financial document types (contracts, IDs, medical records).
+- Real-time / low-latency guarantees. This is a background batch system.
+- Production hardening (SLAs, horizontal scale, queue infrastructure).
+
+## 5. Users and usage modes
+
+1. **Autonomous batch mode (primary).** Operator drops files into an `inbox/`
+   directory (local or mounted). The agent processes each, writes accepted
+   records to storage, and moves uncertain ones to `review/`. No interaction
+   per document.
+2. **Demo mode (secondary).** A visitor uploads one document to the public web
+   UI and sees the extracted fields, per-field confidence, validation results,
+   and the accept/review decision. Nothing is persisted.
+
+Both modes call the same core pipeline.
+
+## 6. Functional requirements
+
+- **FR-1 Ingestion.** Detect new files in `inbox/` (file-watcher or poll) and
+  enqueue them for processing. Supported types: `.pdf`, `.png`, `.jpg`,
+  `.jpeg`, `.webp`, `.tif/.tiff`.
+- **FR-2 Parsing / text acquisition.** For native-text PDFs, extract text and
+  layout. For scans/photos, obtain content either via OCR or via a multimodal
+  model that reads the image directly. The chosen path is backend-dependent
+  (see architecture).
+- **FR-3 Field extraction.** Produce a JSON object conforming to the schema in
+  the data spec, using the active model backend with structured-output
+  enforcement.
+- **FR-4 Validation.** Apply type/format checks and arithmetic cross-checks.
+  Each field carries a validation status.
+- **FR-5 Confidence + routing.** Compute a document-level confidence from
+  model signal, validation results, and required-field completeness. If it
+  clears the threshold, auto-accept; otherwise route to review.
+- **FR-6 Persistence.** Append accepted records to a local SQLite database and
+  export to CSV. Move source files to `processed/` or `review/` accordingly.
+- **FR-7 Logging.** Emit structured logs for every document: inputs, backend
+  used, decision, validation failures, and timings. Never crash the loop on a
+  single bad document — isolate, log, and continue.
+- **FR-8 Web demo.** Accept one uploaded document, run the core pipeline, and
+  render fields, confidence, validation, and decision. Stateless.
+- **FR-9 Backend selection.** The model backend is chosen by configuration at
+  startup with no code change (Gemini free tier or local Ollama).
+- **FR-10 Evaluation.** A harness runs the pipeline over a labelled dataset and
+  reports field-level precision, recall, and F1, plus document-level routing
+  statistics.
+
+## 7. Non-functional requirements
+
+- **NFR-1 Cost.** Zero spend for development and demo. Local model = no quota;
+  hosted model = free tier only.
+- **NFR-2 Privacy.** Free hosted backends may use inputs for training; the
+  public demo must process only synthetic/public documents. This must be
+  stated in the demo UI. Sensitive data is handled only via the local backend.
+- **NFR-3 Swappability.** Adding or replacing a backend requires implementing
+  one interface and changing config — nothing else.
+- **NFR-4 Robustness.** A malformed or unreadable document produces a logged
+  failure and a review routing, never a crash.
+- **NFR-5 Reproducibility.** Pinned dependencies; deterministic config;
+  documented setup that runs from a clean checkout.
+- **NFR-6 Portability.** The core pipeline is independent of both entry points
+  and of any specific host.
+
+## 8. Success criteria
+
+The project is successful when:
+
+- The agent processes a mixed batch (native PDFs + scans + phone photos)
+  end-to-end with no per-document intervention, persisting accepted records and
+  correctly diverting uncertain ones to review.
+- On a held-out labelled set, **auto-accept precision on the critical fields
+  (`total`, `tax`, `invoice_number`) is ≥ 0.98**, with recall reported at that
+  operating point. (Rationale and method in the data spec.)
+- A public demo URL processes an uploaded document of each modality and
+  displays a correct, validated result.
+- Swapping between the Gemini and Ollama backends requires only a config
+  change.
+
+## 9. Key assumptions
+
+- Documents are predominantly English. Multilingual handling is best-effort.
+- Volume during development is low (tens to low hundreds of documents), well
+  within free-tier limits.
+- The operator's local machine or chosen free host can run lightweight Python
+  continuously; the model itself runs locally (Ollama) or via free API.
+- Free-tier quotas and free hosting behaviour (idle sleep, CPU-only) are
+  acceptable for a portfolio demo.
+
+## 10. Glossary
+
+- **Auto-accept:** a document whose confidence clears the threshold and whose
+  record is persisted without human review.
+- **Review:** a document routed to a human because confidence is below
+  threshold or a hard validation rule failed.
+- **Critical fields:** fields where a confidently-wrong value is most costly —
+  `total`, `tax`, `invoice_number`.
+- **Backend:** an implementation of the model interface that turns a document
+  into structured fields.
+- **Core pipeline:** the host- and entry-point-independent function that takes
+  a document and returns an extraction result.

docs/02_architecture.md

@@ -0,0 +1,177 @@
+# Technical Architecture — Document Extraction Agent
+
+## 1. Guiding principle
+
+One reusable core; thin replaceable edges. The core is a pure-ish function:
+
+```
+process_document(path) -> ExtractionResult
+```
+
+Everything that triggers it (folder watcher, web upload) and everything it
+depends on (the model backend) sits behind interfaces so the core never knows
+which entry point invoked it or which model produced the fields. This is what
+keeps the project portable across local and cloud, and it is the main thing
+that makes the design read as engineering rather than prompting.
+
+## 2. System overview
+
+```
+                         ┌─────────────────────────────┐
+   inbox/ (watcher) ─────▶                             │
+                         │        CORE PIPELINE         │────▶ SQLite + CSV
+   web upload (demo) ────▶   parse → extract → validate │      (batch mode)
+                         │   → score → route            │
+                         │                             │────▶ review/ (batch)
+                         └───────────────┬─────────────┘
+                                         │
+                                  Model Backend (interface)
+                                  ├── GeminiBackend  (free API, multimodal)
+                                  └── OllamaBackend   (local, offline)
+```
+
+In batch mode the pipeline persists and moves files. In demo mode the same
+pipeline returns a result that is rendered and then discarded.
+
+## 3. Core pipeline stages
+
+1. **Intake & type detection.** Identify modality from extension/MIME:
+   `native_pdf`, `image` (scan or photo). This determines the parse path.
+2. **Text/representation acquisition.**
+   - `native_pdf`: parse with Docling to get text + layout. Optionally also
+     keep the page image for backends that prefer vision.
+   - `image`: two supported strategies, backend-dependent —
+     (a) *vision-direct*: pass the image to a multimodal backend (Gemini),
+     no separate OCR; (b) *ocr-then-text*: run OCR (e.g. PaddleOCR/Tesseract)
+     to text, then pass text to a text-only backend (Ollama small models).
+3. **Extraction.** Call `backend.extract(document_payload, schema)` and get
+   back a structured object plus whatever confidence signal the backend
+   exposes. Structured output is enforced (JSON schema / grammar), not parsed
+   out of free text.
+4. **Validation.** Run the rule set (types, formats, required fields,
+   arithmetic cross-checks). Produce a per-field validation status and a list
+   of failures.
+5. **Confidence scoring.** Combine model signal, validation outcome, and
+   required-field completeness into a single document-level score.
+6. **Routing.** Compare score to threshold and apply hard-fail overrides
+   (a failed critical cross-check forces review regardless of score). Emit an
+   `ExtractionResult` with `decision ∈ {accept, review}`.
+
+## 4. Components and responsibilities
+
+- **Watcher / runner** (`ingest/`): detects new files, calls the core,
+  performs file moves and persistence side-effects in batch mode. Owns retry
+  and isolation so one document never halts the loop.
+- **Parser** (`parsing/`): modality detection and text/layout acquisition.
+  Wraps Docling and the OCR option behind a single `acquire(path) -> Payload`.
+- **Backend interface** (`backends/base.py`): defines `extract()`. Concrete
+  adapters: `gemini.py`, `ollama.py`. Selected by config via a factory.
+- **Schema & models** (`schema/`): the Pydantic models that define the output
+  contract and are used to enforce structured output and to validate types.
+- **Validation** (`validation/`): pure functions over the parsed object →
+  validation report. No I/O.
+- **Confidence & routing** (`routing/`): pure functions → score and decision.
+- **Persistence** (`store/`): SQLite writer + CSV exporter. Batch mode only.
+- **Web demo** (`web/`): a Gradio app that uploads one file, calls the core,
+  and renders the result. No persistence.
+- **Eval** (`eval/`): runs the core over a labelled dataset and computes
+  metrics.
+
+## 5. Model backend abstraction
+
+```python
+class ExtractionBackend(Protocol):
+    name: str
+    def extract(self, payload: DocumentPayload, schema: type[BaseModel]) -> BackendResult: ...
+
+# BackendResult: { data: dict, field_confidence: dict | None, raw: Any }
+```
+
+- **GeminiBackend.** Calls the Gemini free tier. Multimodal: accepts the page
+  image directly for scans/photos and text for native PDFs. Requests
+  schema-constrained JSON output. Used by the cloud demo (CPU host can't run a
+  local model) and available locally.
+- **OllamaBackend.** Calls a local Ollama server (e.g. a 3B–7B model). Text-in
+  only, so images must go through the OCR path first. Uses grammar/JSON-schema
+  constrained decoding for reliable structure. Used for offline, private, and
+  no-quota local runs.
+
+**Backend rule:** never hardcode a single remote model *name* in logic — model
+identifiers are config, because free catalogs change without notice. Treat a
+missing/renamed model as a recoverable config error.
+
+## 6. Dual entry points
+
+- **Watcher (autonomous).** A long-running process using a filesystem watcher
+  (or a poll loop for portability). On a new file: `process_document()`, then
+  persist + move. This is the "runs on its own" capability.
+- **Web demo (URL).** A Gradio interface with a single upload control. On
+  upload: `process_document()`, render fields + confidence + validation +
+  decision, then discard. Carries an explicit "synthetic/public documents
+  only" notice (see NFR-2).
+
+Both are ~50–100 lines of glue. All real logic lives in the core.
+
+## 7. Data flow for one document
+
+```
+file ─▶ detect modality ─▶ acquire payload (Docling | OCR | raw image)
+     ─▶ backend.extract(payload, schema) ─▶ raw structured data
+     ─▶ validate(data) ─▶ validation report
+     ─▶ score(data, report, model_signal) ─▶ confidence
+     ─▶ route(confidence, report) ─▶ {accept | review}
+     ─▶ [batch] persist + move file   /   [demo] render + discard
+```
+
+## 8. Confidence and routing logic
+
+Document confidence blends three inputs:
+
+- **Model signal** — backend-exposed token/field confidence where available;
+  otherwise treated as neutral.
+- **Validation** — a hard-failed critical cross-check (e.g. totals don't
+  reconcile) forces `review` regardless of score. Soft failures reduce score.
+- **Completeness** — missing required fields reduce score.
+
+Routing:
+
+```
+if any(critical_hard_failures): decision = review
+elif confidence >= THRESHOLD:    decision = accept
+else:                            decision = review
+```
+
+`THRESHOLD` is a single tunable constant. The evaluation harness exists
+precisely to set it (see data spec — optimize auto-accept precision on
+critical fields, accept the resulting recall, and report both).
+
+## 9. Error handling
+
+- Per-document `try/except` in the runner; failures log full context and route
+  to `review/` with a reason. The loop continues.
+- Backend calls are wrapped with bounded retries and timeouts; exhausted
+  retries route to review, they do not crash.
+- Idempotency: a content hash prevents reprocessing the same file twice across
+  restarts.
+
+## 10. Technology choices (and why)
+
+- **Python** — ecosystem fit for parsing/ML.
+- **Docling** — open-source, free, structured PDF/scan parsing with layout and
+  tables, agent-framework friendly.
+- **Pydantic** — the output contract doubles as a validation and
+  structured-output schema.
+- **Gradio** — minimal code to a public demo UI; native to Hugging Face Spaces.
+- **SQLite + CSV** — zero-infra local persistence and a portable export.
+- **Gemini free tier + local Ollama** — two free inference paths covering
+  cloud-demo and offline/private use behind one interface.
+
+## 11. Deployment topology
+
+- **Local / batch:** watcher + core + (Gemini or Ollama) + SQLite/CSV on the
+  operator's machine. Fully free; Ollama gives no quotas and full privacy.
+- **Cloud / demo:** Gradio app + core + Gemini backend on Hugging Face Spaces
+  (free, CPU-only, sleeps when idle). No local model in the cloud; no
+  persistence. Public URL for the portfolio.
+
+See the setup doc for concrete structure, configuration, and deployment steps.

docs/03_data_and_extraction_spec.md

@@ -0,0 +1,164 @@
+# Data & Extraction Specification
+
+## 1. Datasets
+
+All free, all on Hugging Face and/or Kaggle. data.gov.sg is **not** a source —
+it publishes statistical open data, not document images. The set below is
+chosen to cover every required input modality *and* to provide ground-truth
+labels, which the evaluation harness needs.
+
+| Dataset | What it is | Modality covered | Labels | Where |
+|---|---|---|---|---|
+| **SROIE (ICDAR 2019)** | ~1,000 real scanned receipts from shops/restaurants; variable print & scan quality | Scans | company, address, date, total | HF: `Voxel51/scanned_receipts`; also `priyank-m/SROIE_2019_text_recognition` |
+| **CORD** | ~11,000 Indonesian receipts captured in the wild; noisy, low quality; multi-level labels (store/menu/subtotal/total + subclasses) | Scans / in-the-wild | rich key-value + line items | HF: search `CORD` (naver-clova-ix) |
+| **MC-OCR** | 2,436 receipts captured on mobile devices | Phone photos | quality + key fields | Search "MC-OCR RIVF 2021" (HF mirrors / challenge page) |
+| **High Quality Invoice Images for OCR** | ~6,700 synthetic but realistic invoices | Native-style invoices | invoice fields | Kaggle: `osamahosamabdellatif/high-quality-invoice-images-for-ocr`; HF: `Voxel51/high-quality-invoice-images-for-ocr` |
+| **invoices-and-receipts_ocr_v1** | Mixed invoices + receipts with OCR/structure | Mixed | structured | HF: `mychen76/invoices-and-receipts_ocr_v1` |
+| **invoice-ocr-json** | Invoices with structured JSON ground truth | Native invoices | JSON key-values | HF: `GokulRajaR/invoice-ocr-json` |
+| **FUNSD** | 199 noisy scanned forms | Messy scans/forms | entities + relations | HF: search `funsd` |
+| **DocILE** (optional, large) | Large invoice benchmark for key-info localization & extraction | Native invoices | KILE/LIR | DocILE project (HF/registration) |
+
+### Recommended working split
+
+- **Messy real receipts:** SROIE (baseline real) + CORD (noisy in-the-wild).
+- **Phone photos:** MC-OCR.
+- **Native invoices:** High Quality Invoice Images + `invoice-ocr-json` (has
+  clean JSON labels, convenient for the eval harness).
+- **Messy forms (stretch):** FUNSD.
+
+Hold out a fixed evaluation slice per source and never tune against it.
+
+### Licensing note
+
+These are research/benchmark datasets with their own terms; several invoice
+sets are synthetic. Use them for development, evaluation, and demo only, and
+keep real/sensitive documents off the public demo and off free hosted
+backends (see NFR-2). Confirm each dataset's license before any redistribution.
+
+## 2. Output schema
+
+A single unified schema spans receipts and invoices; fields not present in a
+given document are `null`. Define with Pydantic so the same model enforces
+structured output, validates types, and serializes to storage.
+
+```python
+class LineItem(BaseModel):
+    description: str | None
+    quantity: float | None
+    unit_price: float | None
+    amount: float | None
+
+class Document(BaseModel):
+    doc_type: Literal["receipt", "invoice", "other"]
+    vendor_name: str | None
+    vendor_address: str | None
+    invoice_number: str | None          # critical
+    document_date: date | None          # ISO 8601
+    due_date: date | None
+    currency: str | None                # ISO 4217 where detectable
+    line_items: list[LineItem]
+    subtotal: float | None
+    tax: float | None                   # critical
+    total: float | None                 # critical
+    # populated by the pipeline, not the model:
+    field_confidence: dict[str, float] = {}
+    validation: dict = {}
+    decision: Literal["accept", "review"] | None = None
+```
+
+**Field requirements**
+
+- Always attempt: `doc_type`, `vendor_name`, `document_date`, `total`.
+- Critical (precision-prioritised): `total`, `tax`, `invoice_number`.
+- Monetary fields are numbers (no currency symbols/thousands separators);
+  normalize during extraction.
+- Dates are ISO 8601 (`YYYY-MM-DD`); store raw string alongside if parsing is
+  ambiguous.
+
+## 3. Validation rules
+
+Validation is pure functions over the parsed `Document` → a report. Two
+classes of rule:
+
+**Hard rules (a failure forces `review`):**
+
+- `H1` All critical fields parse to the correct type when present.
+- `H2` Arithmetic reconciliation, when the inputs exist:
+  `subtotal + tax ≈ total` within a small epsilon (rounding tolerance).
+- `H3` Line-item reconciliation, when line items exist:
+  `sum(line_items.amount) ≈ subtotal` (or `total` if no subtotal).
+- `H4` `total` is present and non-negative.
+
+**Soft rules (reduce confidence, do not force review):**
+
+- `S1` `document_date` present and plausible (not in the far future).
+- `S2` `currency` resolves to a known code.
+- `S3` `vendor_name` non-empty.
+- `S4` Per-line arithmetic: `quantity * unit_price ≈ amount`.
+
+Epsilon for monetary comparisons accommodates rounding (e.g. ±0.02 absolute or
+a small relative tolerance, whichever is larger).
+
+## 4. Confidence scoring
+
+Document confidence ∈ [0, 1] blends:
+
+- **Model signal** (weighted) — backend field/token confidence where exposed;
+  neutral (0.5) when unavailable.
+- **Validation** — start from model signal; subtract penalties for each soft
+  failure; any hard failure short-circuits to `review`.
+- **Completeness** — penalty proportional to missing required fields.
+
+Exact weights live in config and are set empirically via the eval harness. Keep
+the function pure and unit-tested with hand-built cases.
+
+## 5. Routing
+
+```
+decision = review  if any hard rule fails
+         = accept  if confidence >= THRESHOLD
+         = review  otherwise
+```
+
+`THRESHOLD` is one constant, tuned in evaluation.
+
+## 6. Evaluation methodology
+
+This is what turns "seems to work" into evidence, and it is how the
+precision/recall question is answered concretely.
+
+**Definitions (field level, against ground truth):**
+
+- *Precision* = correct extracted values / all values the system produced
+  (and auto-accepted).
+- *Recall* = correct extracted values / all values present in ground truth.
+- *F1* = harmonic mean.
+
+**Procedure:**
+
+1. Run the core over each held-out dataset slice.
+2. Normalize predicted and gold values (numbers, dates, casing/whitespace)
+   before comparison.
+3. Compute precision, recall, F1 **per field** and **per critical field**.
+4. Compute document-level routing stats: % auto-accepted, % to review, and —
+   crucially — **precision on the auto-accepted subset** for critical fields.
+5. Sweep `THRESHOLD` and report the precision/recall trade-off curve.
+
+**Target / operating point:**
+
+- Optimize so **auto-accept precision on `total`, `tax`, `invoice_number` ≥
+  0.98**, then report recall at that point. Recall "lost" to the threshold is
+  simply review-queue volume — acceptable, because the asymmetric cost favours
+  not writing wrong numbers. Arithmetic cross-checks (H2/H3) are the lever that
+  raises precision without collapsing recall, since they let confident,
+  internally-consistent values through while catching the inconsistent ones.
+
+Report a small table per dataset (precision/recall/F1 per field, plus routing
+stats) in the project README — this is the portfolio's evidence of rigor.
+
+## 7. Modality handling summary
+
+- **Native PDF:** Docling → text/layout → backend (text or vision).
+- **Scan:** vision-direct (Gemini reads the image) **or** OCR → text → backend.
+- **Phone photo:** same as scan; vision-direct is more robust to skew/lighting,
+  which is why the Gemini path is preferred for the demo.

docs/04_project_setup.md

@@ -0,0 +1,190 @@
+# Project Setup, Stack & Deployment
+
+## 1. Repository layout
+
+```
+doc-extraction-agent/
+├── CLAUDE.md                     # conventions & guardrails for the coding agent
+├── README.md                     # quickstart + results table (eval evidence)
+├── pyproject.toml                # project + dependency declarations (managed by uv)
+├── uv.lock                       # resolved, pinned dependency lock (committed)
+├── .python-version               # uv interpreter pin: 3.11 (committed)
+├── .env.example                  # config template (no secrets committed)
+├── docs/
+│   ├── 01_requirements.md
+│   ├── 02_architecture.md
+│   ├── 03_data_and_extraction_spec.md
+│   └── 05_build_plan.md
+├── src/doc_agent/
+│   ├── __init__.py
+│   ├── config.py                 # loads env/config; selects backend
+│   ├── core.py                   # process_document(): the reusable pipeline
+│   ├── schema/
+│   │   └── models.py             # Pydantic Document, LineItem
+│   ├── parsing/
+│   │   ├── detect.py             # modality detection
+│   │   ├── docling_parser.py     # native PDF → text/layout
+│   │   └── ocr.py                # image → text (optional path)
+│   ├── backends/
+│   │   ├── base.py               # ExtractionBackend protocol + factory
+│   │   ├── gemini.py             # free-tier multimodal adapter
+│   │   └── ollama.py             # local model adapter
+│   ├── validation/
+│   │   └── rules.py              # hard/soft rules → report
+│   ├── routing/
+│   │   └── score.py              # confidence + decision (pure)
+│   ├── store/
+│   │   ├── db.py                 # SQLite writer
+│   │   └── export.py             # CSV export
+│   ├── ingest/
+│   │   └── watcher.py            # folder watcher / poll loop (batch entry)
+│   └── web/
+│       └── app.py                # Gradio demo (URL entry)
+├── eval/
+│   ├── run_eval.py               # metrics over labelled datasets
+│   └── datasets/                 # download scripts / loaders (no data in git)
+├── data/                         # gitignored: inbox/ processed/ review/ exports/
+│   ├── inbox/
+│   ├── processed/
+│   ├── review/
+│   └── exports/
+└── tests/
+    ├── test_validation.py
+    ├── test_routing.py
+    ├── test_schema.py
+    └── test_core_smoke.py
+```
+
+## 2. Stack
+
+- **Runtime:** Python **3.11**, pinned via `.python-version` (`uv python pin
+  3.11`). Chosen over 3.12 for broadest wheel coverage across the Torch-based
+  Docling stack and PaddleOCR/PaddlePaddle, which lags newest Pythons.
+  Declared range: `requires-python = ">=3.11"`.
+- **Package manager:** `uv` (manages the venv, resolves and locks deps via
+  `uv.lock`; add deps with `uv add`, run with `uv run`).
+- **Parsing:** `docling` (native PDF/scan structure). Optional OCR:
+  `paddleocr` or `pytesseract` + system Tesseract.
+- **Modeling:** `google-genai` (Gemini free tier) and a local `ollama` server
+  (e.g. `qwen2.5:7b` or a 3B variant) reached over HTTP.
+- **Contract/validation:** `pydantic` v2.
+- **Web demo:** `gradio`.
+- **Storage:** stdlib `sqlite3` + `csv`.
+- **Watcher:** `watchdog` (or a stdlib poll loop for max portability).
+- **Config:** `pydantic-settings` / `python-dotenv`.
+- **Testing:** `pytest`.
+
+Dependencies are declared in `pyproject.toml` and pinned via the committed
+`uv.lock` (`uv sync` installs exactly that lock). Do not float the model
+identifier in code — it is config (see guardrails).
+
+## 3. Configuration (`.env.example`)
+
+```
+# Backend selection: "gemini" | "ollama"
+EXTRACTION_BACKEND=gemini
+
+# Gemini (free tier via Google AI Studio key; no card required)
+GEMINI_API_KEY=
+GEMINI_MODEL=gemini-flash-latest        # identifier is config, not hardcoded
+
+# Ollama (local)
+OLLAMA_HOST=http://localhost:11434
+OLLAMA_MODEL=qwen2.5:7b
+
+# Image handling: "vision_direct" | "ocr_then_text"
+IMAGE_STRATEGY=vision_direct            # vision_direct requires a multimodal backend
+
+# Routing
+CONFIDENCE_THRESHOLD=0.85               # tuned via eval
+
+# Paths (batch mode)
+INBOX_DIR=./data/inbox
+PROCESSED_DIR=./data/processed
+REVIEW_DIR=./data/review
+EXPORT_DIR=./data/exports
+DB_PATH=./data/agent.db
+```
+
+`config.py` validates these at startup and fails fast with a clear message if,
+e.g., `gemini` is selected with no key, or `vision_direct` is selected with a
+text-only backend.
+
+## 4. Local setup
+
+```bash
+# 1. Pin the interpreter to 3.11 (writes .python-version; uv fetches it if absent)
+uv python pin 3.11
+
+# 2. Install (uv creates the venv on 3.11 and installs from pyproject.toml + uv.lock)
+uv sync
+
+# 3a. Gemini path: get a free AI Studio key, put it in .env
+#     (free tier, no credit card; quota resets daily)
+
+# 3b. Ollama path (offline/private):
+#     install Ollama, then:
+ollama pull qwen2.5:7b
+#     set EXTRACTION_BACKEND=ollama and IMAGE_STRATEGY=ocr_then_text
+
+# 4. Create working dirs
+mkdir -p data/{inbox,processed,review,exports}
+```
+
+## 5. Running
+
+**Autonomous batch mode:**
+
+```bash
+uv run python -m doc_agent.ingest.watcher
+# drop files into data/inbox/ — accepted records land in SQLite + data/exports/,
+# uncertain ones move to data/review/
+```
+
+**Web demo (local):**
+
+```bash
+uv run python -m doc_agent.web.app
+# opens a Gradio URL; upload one document to see fields + confidence + decision
+```
+
+**Evaluation:**
+
+```bash
+uv run python eval/run_eval.py --dataset sroie --split holdout
+# prints per-field precision/recall/F1 and auto-accept precision on critical fields
+```
+
+## 6. Deployment to Hugging Face Spaces (free public demo URL)
+
+1. Create a new **Space** → SDK: **Gradio** (free CPU tier). Set the Space's
+   Python to **3.11** (the `python_version: "3.11"` field in the Space README
+   metadata) so the deployed runtime matches the pinned local interpreter.
+2. Add `app.py` at the Space root that imports and launches
+   `doc_agent.web.app` (or copy the web entry there), plus a `requirements.txt`
+   the Gradio builder can read — generate it from the uv-managed project rather
+   than hand-maintaining it: `uv export --no-hashes --no-dev -o requirements.txt`.
+3. Set **Repository secrets** in the Space: `GEMINI_API_KEY`,
+   `EXTRACTION_BACKEND=gemini`, `IMAGE_STRATEGY=vision_direct`,
+   `GEMINI_MODEL=gemini-flash-latest`.
+4. Push; the Space builds and serves a public URL.
+
+**Free-tier realities to design around (and to note in the UI):**
+
+- CPU-only and the Space **sleeps when idle** → first request after idle has a
+  cold start. This is why the cloud demo uses the **Gemini API** for inference
+  rather than a local model, and why `vision_direct` (no heavy OCR in the
+  Space) is the demo's image path.
+- **Stateless:** no persistent DB in the demo. Show the result; don't store it.
+- **Privacy:** the free Gemini tier may use inputs for training, so the demo
+  must display a "synthetic/public documents only" notice and must not be used
+  for real financial data.
+
+## 7. What stays free
+
+- **Inference:** local Ollama (no quota, private) or Gemini free tier
+  (~1,500 req/day, resets daily, no card) — far above dev volume.
+- **Hosting:** Hugging Face Spaces free CPU tier for the public demo.
+- **Storage:** local SQLite/CSV; nothing paid.
+
+No component requires a credit card or paid plan for development or demo.

docs/05_build_plan.md

@@ -0,0 +1,160 @@
+# Build Plan — Ordered Tasks for Claude Code
+
+Execute phases in order. Each task lists what to build and its acceptance
+criteria. Every phase ends in something runnable or testable. Keep the core
+decoupled from entry points and backends throughout (see CLAUDE.md).
+
+Legend: **[AC]** = acceptance criteria.
+
+---
+
+## Phase 0 — Scaffolding
+
+**0.1 Project skeleton.** Create the repo layout from the setup doc and
+initialize the project with **uv** (`uv init`). Pin the interpreter to 3.11
+(`uv python pin 3.11`) and set `requires-python = ">=3.11"` in `pyproject.toml`.
+Add dependencies via `uv add` (committing `uv.lock`); add `.env.example` and
+`.gitignore` (ignore `data/`, `.env`, `.venv/`, caches — but **commit**
+`uv.lock` and `.python-version`); add an empty `README.md`.
+**[AC]** `.python-version` reads `3.11`; `uv sync` succeeds and
+`uv run python -c "import sys, doc_agent; assert sys.version_info[:2]==(3,11)"`
+passes.
+
+**0.2 Config loader.** Implement `config.py` using pydantic-settings: load env,
+validate combinations (gemini requires key; vision_direct requires multimodal
+backend), fail fast with clear messages.
+**[AC]** Unit test: invalid combos raise; valid config parses.
+
+---
+
+## Phase 1 — Contract & validation (pure, no model yet)
+
+**1.1 Schema.** Implement `schema/models.py` (`Document`, `LineItem`) per the
+data spec, with normalization helpers (money → float, date → ISO).
+**[AC]** `test_schema.py`: valid dicts parse; malformed money/date handled.
+
+**1.2 Validation rules.** Implement `validation/rules.py`: hard rules H1–H4,
+soft rules S1–S4, monetary epsilon, returning a structured report.
+**[AC]** `test_validation.py`: reconciling totals pass H2/H3; mismatches fail;
+soft failures recorded without forcing review.
+
+**1.3 Confidence & routing.** Implement `routing/score.py`: pure
+`score(data, report, model_signal) -> float` and
+`route(score, report) -> decision`, with hard-failure short-circuit.
+**[AC]** `test_routing.py`: hard failure ⇒ review regardless of score;
+threshold boundary behaves; missing required fields lower score.
+
+These three modules are fully testable before any model exists.
+
+---
+
+## Phase 2 — Parsing & backends
+
+**2.1 Modality detection.** `parsing/detect.py`: map file → `native_pdf` |
+`image` by extension/MIME.
+**[AC]** Correctly classifies the supported extensions.
+
+**2.2 Docling parser.** `parsing/docling_parser.py`: native PDF → text/layout
+payload; optionally retain page image.
+**[AC]** A sample text PDF yields non-empty structured text.
+
+**2.3 OCR path (optional strategy).** `parsing/ocr.py`: image → text via
+PaddleOCR/Tesseract, behind the same payload interface.
+**[AC]** A sample receipt image yields text; absence of the OCR engine degrades
+gracefully with a clear error.
+
+**2.4 Backend interface + factory.** `backends/base.py`: the
+`ExtractionBackend` protocol, `BackendResult`, and a factory that builds the
+backend from config.
+**[AC]** Factory returns the configured backend; unknown backend ⇒ clear error.
+
+**2.5 Gemini backend.** `backends/gemini.py`: multimodal call to the free tier;
+text for native PDFs, image for scans/photos; schema-constrained JSON output;
+bounded retries + timeout; model id from config.
+**[AC]** Given a sample document, returns schema-valid JSON; transient errors
+retry then route to review (don't crash).
+
+**2.6 Ollama backend.** `backends/ollama.py`: local call with JSON-schema/grammar
+constrained decoding; text-in (pairs with OCR path).
+**[AC]** With a local model present, returns schema-valid JSON for a text
+payload. Skipped/marked if no local server (documented).
+
+---
+
+## Phase 3 — Core pipeline
+
+**3.1 Assemble core.** `core.py`: `process_document(path) -> ExtractionResult`
+chaining detect → acquire → extract → validate → score → route. Pure of
+side-effects (no file moves, no DB) — returns a result object.
+**[AC]** `test_core_smoke.py`: runs end-to-end on a couple of sample files with
+a stub backend; returns a populated result with a decision.
+
+**3.2 Idempotency.** Content-hash helper so the same file isn't reprocessed.
+**[AC]** Same file hashed identically across runs.
+
+---
+
+## Phase 4 — Entry points
+
+**4.1 Persistence.** `store/db.py` (SQLite append of accepted records) and
+`store/export.py` (CSV export).
+**[AC]** Accepted record persists and appears in CSV; schema columns match.
+
+**4.2 Watcher / batch runner.** `ingest/watcher.py`: watch (or poll) `inbox/`,
+call core, persist accepted, move source to `processed/` or `review/`,
+per-document try/except with structured logging.
+**[AC]** Dropping a batch (mixed PDFs/scans/photos) processes all; one
+deliberately corrupt file routes to review with a logged reason and does not
+stop the loop.
+
+**4.3 Web demo.** `web/app.py`: Gradio single-upload UI rendering fields,
+per-field confidence, validation report, and decision; explicit
+"synthetic/public only" notice; stateless.
+**[AC]** Local Gradio URL processes one upload of each modality and displays a
+correct, validated result.
+
+---
+
+## Phase 5 — Evaluation
+
+**5.1 Dataset loaders.** `eval/datasets/`: scripts to fetch a held-out slice of
+SROIE and the labelled invoice JSON set; map gold labels to the schema. No data
+committed to git.
+**[AC]** Loader yields (document, gold) pairs for a slice.
+
+**5.2 Metrics harness.** `eval/run_eval.py`: run core over a slice; normalize;
+compute per-field and per-critical-field precision/recall/F1; compute
+auto-accept precision on critical fields; sweep threshold.
+**[AC]** Prints a metrics table; produces the precision/recall trade-off across
+thresholds; recommends a `CONFIDENCE_THRESHOLD`.
+
+**5.3 Tune & record.** Set `CONFIDENCE_THRESHOLD` so auto-accept precision on
+critical fields ≥ 0.98; record the resulting recall.
+**[AC]** README contains a results table per dataset (precision/recall/F1 +
+routing stats) — the portfolio evidence.
+
+---
+
+## Phase 6 — Deploy & document
+
+**6.1 Hugging Face Space.** Add Space `app.py` and a `requirements.txt` for the
+Gradio builder, generated from uv (`uv export --no-hashes --no-dev -o
+requirements.txt`); set secrets (`GEMINI_API_KEY`, backend=gemini,
+vision_direct). Deploy.
+**[AC]** Public URL processes an uploaded document of each modality.
+
+**6.2 README.** Quickstart (both modes), the swappable-backend explanation, the
+results table, the demo URL, and the free-tier/privacy caveats.
+**[AC]** A reader can run locally from scratch and understands the design and
+the evidence.
+
+---
+
+## Build order rationale
+
+Pure logic first (Phases 1) so the hard-to-test parts (validation, routing) are
+locked down before any model variance enters. Parsing and backends next
+(Phase 2), then the core that composes them (Phase 3). Entry points (Phase 4)
+are thin glue added only once the core is proven. Evaluation (Phase 5) sets the
+one tunable threshold with evidence. Deployment (Phase 6) is last and small
+because the core was kept host-independent from the start.