# PROGRESS_TOMORROW.md — Supervised Build Ledger The second half of the build: real models, real parsing, the entry points, evaluation, and deployment. Everything past the `⛔ STOP` line in `PROGRESS.md`. **This half is interactive, not looped.** You drive it in a normal `claude` session, one task at a time, and verify each task on a real document before moving on. There is no `DONE_ALL` and no `run-overnight.ps1` here — these tasks call real models (non-deterministic), need API keys and a local server, and have decisions only you should make. Leaving them to an unattended loop would burn quota on work you can't blindly trust. ## Protocol (per task) 1. Prompt Claude Code for the **next unchecked task**, one at a time. 2. Let it implement, then **review the diff** and **eyeball the result on a real sample document** — not just that tests pass. Model output that looks plausible can still be wrong; you are the check. 3. When satisfied, commit (one commit per task) and tick the box. 4. Add dependencies **per task, not all at once** (see policy below), so an install failure is isolated to the task that needs it. 5. Default to **Sonnet** (`/model sonnet`). These are routine integration tasks; Opus is not needed and costs ~3x. ## Setup before T1 - **Gemini key:** get a free API key from Google AI Studio. Put it in `.env` as `GEMINI_API_KEY=...` — this is the key for *the app's* calls to Gemini, and is completely separate from your Claude Code login. Do **not** set `ANTHROPIC_API_KEY` (that would silently bill your Claude Code usage to the API). - **Sample documents:** have one real **receipt photo**, one **scan**, and one **native-text PDF invoice** on hand. You'll use them to eyeball real extractions (synthetic/public docs only — the free tier may train on inputs). - **Branch:** lock in last night's milestone and start fresh (see the chat instructions accompanying this file). ## Dependency policy Add only what each task needs, when it needs it: `google-genai` (T2) · `docling` (T3) · `paddleocr` **or** `pytesseract` (T4) · `ollama` (T6) · `watchdog` (T8) · `gradio` (T9). This isolates the Paddle-on-3.11 risk to T4 instead of breaking everything at once. --- ## TASKS - [x] **T1 — Gemini config + key** (prep) Add `uv add google-genai`. In `.env` set `GEMINI_API_KEY`, `EXTRACTION_BACKEND=gemini`, `IMAGE_STRATEGY=vision_direct`, and `GEMINI_MODEL` per `docs/04_project_setup.md`. No new code beyond confirming `config.py` accepts it. Check: `uv run python -c "from doc_agent.config import load_config; c=load_config(); print(c.extraction_backend, bool(c.gemini_api_key))"` → `gemini True`. - [x] **T2 — Gemini backend + real image acquire** (build plan 2.5) ⭐ milestone Implement `src/doc_agent/backends/gemini.py` per architecture §5: multimodal call, schema-constrained JSON output, bounded retries + timeout, model id from config; register its builder in the factory. Wire a minimal real `acquire` for images in `vision_direct` mode (load image bytes into the payload) so `process_document` runs end-to-end on a photo/scan with no Docling/OCR yet. Tests: `tests/test_gemini.py` with a **mocked** Gemini response (deterministic, no network/quota) covering schema-valid parsing, retry, and timeout. Check (auto): `uv run pytest tests/test_gemini.py -q`. Check (manual): run `process_document` on your real **receipt photo** and confirm `vendor_name`, `total`, `document_date` are actually correct. *This is where the project stops being scaffolding.* - [x] **T3 — Docling parser** (2.2) `uv add docling` (first run downloads layout models — needs internet). `src/doc_agent/parsing/docling_parser.py`: native PDF → text/layout payload; wire into `acquire` for `native_pdf`. Check (manual): `process_document` on your **PDF invoice** via Gemini returns correct fields. - [ ] **T4 — OCR path** (2.3) — *needed for Ollama / `ocr_then_text`; deferrable* `src/doc_agent/parsing/ocr.py`: image → text behind the payload interface; wire into `acquire` for `IMAGE_STRATEGY=ocr_then_text`. **DECISION:** try `uv add paddleocr`; if it won't resolve on 3.11, fall back to `uv add pytesseract` (and install the Tesseract binary). Check: a sample image yields text; `process_document` works in `ocr_then_text` mode. *With `vision_direct` + Gemini you don't strictly need this for the demo — you can skip it now and return before adding Ollama.* - [x] **T5 — Consolidate real `acquire` into core** Make `core.py`'s default `acquire` do the real thing by modality + strategy (`native_pdf`→Docling; `image`+`vision_direct`→bytes; `image`+`ocr_then_text`→OCR), so `process_document(path)` works on a real path for every input without injection. Keep the injectable seam for tests. Check (auto): `uv run pytest -q` (full suite still green; smoke tests still use the stub). Check (manual): one of **each** input type returns correct fields. - [ ] **T6 — Ollama backend** (2.6) — *OPTIONAL: offline/private; skip to ship faster* `uv add` the ollama client; needs a local Ollama server + a pulled model (e.g. `ollama pull qwen2.5:7b`). `src/doc_agent/backends/ollama.py`: local call with JSON-schema/grammar-constrained decoding; text-in (pairs with the OCR path). Tests: mocked unit tests + a manual smoke against the local server. Check: with `EXTRACTION_BACKEND=ollama` + `IMAGE_STRATEGY=ocr_then_text`, `process_document` returns schema-valid data. - [x] **T7 — Persistence** (4.1) `src/doc_agent/store/db.py` (SQLite append of accepted records) and `store/export.py` (CSV export). Add `tests/test_store.py`. Check: `uv run pytest tests/test_store.py -q`; an accepted record persists and appears in the CSV. - [x] **T8 — Watcher / batch runner** (4.2) `uv add watchdog`. `src/doc_agent/ingest/watcher.py`: watch (or poll) `inbox/`, call core, persist accepted, move source to `processed/` or `review/`, per-document try/except + structured logging. Check: drop a **mixed batch** (PDF + scan + photo + one deliberately corrupt file) into `data/inbox/`; all valid ones process, the corrupt one routes to `review/` with a logged reason, and the loop does not stop. - [x] **T9 — Web demo** (4.3) `uv add gradio`. `src/doc_agent/web/app.py`: single-upload UI rendering fields, per-field confidence, the validation report, and the decision; a "synthetic/public documents only" notice; stateless. Check: `uv run python -m doc_agent.web.app`; upload one of each modality and confirm a correct, validated result is displayed. - [x] **T10 — Evaluation harness** (5) `eval/datasets/` loaders for a held-out **SROIE** slice + the labelled invoice JSON set, mapping gold labels to the schema. `eval/run_eval.py`: run core over a slice, normalize, compute per-field and per-critical-field precision/recall/F1, auto-accept precision on critical fields, and sweep the threshold. **DECISION:** *you* pick `CONFIDENCE_THRESHOLD` so auto-accept precision on `total`/`tax`/`invoice_number` ≥ 0.98; record the resulting recall. **COST:** this calls the model over many documents and counts against the Gemini free-tier daily limit — run a **small slice first**, deliberately. Check: `run_eval` prints the metrics table; you set the threshold; the results table goes into the README. - [x] **T11 — Deploy** (6) Create a Hugging Face **Space** (Gradio SDK, free), `python_version: "3.11"`, `requirements.txt` via `uv export --no-hashes --no-dev -o requirements.txt`, secrets (`GEMINI_API_KEY`, `EXTRACTION_BACKEND=gemini`, `IMAGE_STRATEGY=vision_direct`). Write the README: quickstart (both modes), the swappable-backend note, the results table, the demo URL, and the free-tier/privacy caveats. Check: the public URL processes an uploaded document of each modality. --- ## Phase done when A real document of each input type flows end-to-end through a live model into a validated record; the watcher processes a mixed batch unattended and routes exceptions to review; the eval harness has produced a precision/recall table and set the threshold; and the demo is live at a public URL. T4 and T6 are optional and can be revisited after the demo ships.