--- title: Document Extraction Agent short_description: Validate and auto-accept or review invoice/receipt fields colorFrom: indigo colorTo: blue sdk: gradio sdk_version: 6.19.0 python_version: "3.11" app_file: app.py pinned: false --- # Document Extraction Agent Drop invoices and receipts into a folder. The agent extracts structured fields with an LLM, cross-checks the arithmetic, auto-accepts only what it can verify, and routes everything else to human review. It runs unattended, on free infrastructure, and treats the model as fallible by design. **[Live Demo](https://huggingface.co/spaces/knzychw/document-extract-agent)** (upload one PDF, scan, or photo) | **[Specs and architecture](docs/)** | **[Backlog](PROGRESS_FUTURE.md)** **Headline result:** on a 100-document held-out SROIE slice, every auto-accepted `total` was correct -- **100% precision (18/18)**. The one wrong total in the slice failed a line-item arithmetic check and went to review instead of being written. ## Architecture One reusable core, two thin entry points. The core computes and returns; the entry points own all side effects. ``` +----------------------------------+ inbox/ (watcher) --->| CORE PIPELINE |---> SQLite (accepted) | detect -> acquire -> extract -> | upload (web demo) -->| validate -> score -> route |---> review/ +-----------------+----------------+ | Model backend (interface) |-- Gemini (multimodal API) -- implemented |-- Ollama (local, offline) -- planned ``` - **The watcher is the agent.** It wakes on new files, decides accept-or-escalate per document with no human in the loop, persists what it trusts, and quarantines the rest. One corrupt file never halts the run: every document runs inside its own try/except, and failures log and route to review. - **The web demo is a window onto the same core.** One upload, one pass, result rendered, nothing stored. - **The LLM is a replaceable part.** All model access goes through one `ExtractionBackend` interface, and the model name is config, not code. Gemini powers the hosted demo today; the interface is built for a local Ollama backend (offline, private), which is scaffolded in config but not yet implemented. - **Validation gates acceptance.** Hard rules -- including `subtotal + tax ~= total` and line-item reconciliation -- force review on any failure, regardless of what the model says. ### Where's the agent? Per document there is no agent -- there is a pipeline. Six stages run in a fixed order, and five of them are plain code. The LLM appears in exactly one: a single API call that fills in the fields of a fixed schema. It cannot call tools, loop, skip a stage, or affect what happens next -- its output is data, never control flow. The accept-or-review decision is made after it, by arithmetic checks and a three-line routing rule. The autonomy lives one level up. The watcher is a long-running process that notices new files, runs the pipeline, and acts on the result -- persisting accepted records, quarantining the rest -- with no human in the loop. Call it an agent in the classic sense, or just a daemon: the point is that the model was deliberately given no decision authority, because a probabilistic model should not decide when to trust itself. ## Why I built this I wanted a project that demonstrates the engineering *around* a model rather than the model itself -- the trust decision, not the extraction -- and a testbed for building software agentically against a spec package (more on that below). Document extraction was the right problem because the failure mode is so concrete: a confidently-wrong total written to the books propagates silently, while a document sitting in a review queue is visible and recoverable. So the system optimizes precision on the auto-accepted path and pays for it in review volume -- an explicit, measured trade (18% auto-accepted at 100% critical-field precision on SROIE). What that took: - **Arithmetic cross-checks as the acceptance gate.** The model's own confidence turned out to be unusable (see Challenges), so reconciliation rules carry the precision instead. - **A decoupled core.** `core.py` imports nothing from the watcher, web, or storage layers, performs no I/O side effects, and is fully testable offline with a stub backend. - **Crash-proof, idempotent ingestion.** Per-document isolation plus a content-hash UNIQUE constraint in SQLite: a crash-and-restart never double-writes a record and never loses a file. - **A two-phase eval harness.** Inference runs once and is cached; every metric and the full threshold sweep replay offline from the cache. Re-tuning costs zero API calls. - **Schema-enforced model output.** The extraction schema is a Pydantic model that doubles as the API's constrained-output schema. No JSON is ever regexed out of free text. ## Tech stack | Layer | Choice | Why | |---|---|---| | Contract and validation | Python 3.11, Pydantic v2 | One schema defines the data contract, validates model output, and constrains the API's JSON generation. | | Model access | google-genai (Gemini, vision-direct) | Multimodal free tier reads receipt photos directly; no OCR stage needed for the demo path. | | PDF parsing | Docling (OCR disabled) | Native PDFs carry embedded text; layout-aware parsing without the OCR model stack. | | Entry points | watchdog (folder agent), Gradio (demo) | Filesystem events for autonomy; a stateless UI for inspection. | | Storage | stdlib sqlite3 + csv | Append-only records with an idempotency constraint; no server, no ORM. | | Tooling | uv, pytest, ruff | Locked reproducible installs; 218 offline tests; lint kept at zero. | ## Technical challenges **Model confidence is a mirage on the free tier.** The eval exposed that Gemini's free tier returns no per-field confidence, so the pipeline's confidence score falls back to a neutral 0.5 prior and can only be penalized downward -- every clean document scores exactly 0.50 and the accept threshold becomes a binary switch. The fix was architectural, not numeric: arithmetic cross-checks (H2: `subtotal + tax ~= total`, H3: line items sum to the subtotal) gate acceptance instead. On the eval slice, 74 of 100 documents failed a hard rule; of the 77 that scored 0.50, the 59 hard-failures were forced to review, and all 18 documents that survived both gates had correct totals. **An adversarial review caught a metric-inflating bug before the eval ran.** The eval's money comparison originally reused the pipeline's reconciliation tolerance, which includes a 0.5% relative term -- meaning a $2-wrong total on a $500 receipt would have scored as "correct" and silently inflated the headline precision. Independent review agents flagged it; the comparison is now cent-exact with a named regression test (`test_money_rejects_relative_tolerance_error`). **Hugging Face Spaces created a dependency deadlock.** The Space build force-installs `gradio[oauth,mcp]`, whose `mcp` extra caps `pydantic<=2.12.5`, while `google-genai` requires `>=2.12.5`. Exactly one version satisfies both. Resolving the platform's full install set locally with `uv pip compile` found it; `requirements.txt` pins `pydantic==2.12.5` with the reasoning documented inline. ## How it was built Half of this project was written by an autonomous loop overnight; the split was by risk, not convenience. A spec package came first: requirements, architecture, data spec, and a phased build plan with per-task acceptance criteria (`docs/`), plus a `CLAUDE.md` encoding the architectural rules the code must not break. The specs are kept frozen as the original design inputs; where the build diverged from them (the tuned threshold, the dataset mirror, the eval CLI), the code and this README are authoritative. The deterministic core -- schema, validation, routing, backend interface, stub pipeline -- landed as 19 commits in one unattended overnight run: a driver script ran Claude Code headless against a task ledger ([`PROGRESS.md`](PROGRESS.md), harness in `run-overnight.ps1`), one task per fresh-context iteration, each proven by its acceptance check before commit, with a hard scope boundary the loop could not cross. The model-touching half was built interactively ([`PROGRESS_TOMORROW.md`](PROGRESS_TOMORROW.md)), with every extraction verified on real documents -- because a plausible-looking wrong total passes a smoke test. The ledgers and the loop harness are committed as part of the repo's history. ## Evaluation Two-phase harness in `eval/`: `predict` runs the pipeline over a held-out slice and caches every result (the only phase that touches the model); `score` computes all metrics and the threshold sweep offline from that cache. ### SROIE (ICDAR 2019), 100-document held-out test slice Gemini `gemini-2.5-flash`, vision-direct. Predicted and gold values are normalized before comparison: money cent-exact, dates on ISO equality, text case- and whitespace-insensitive. SROIE labels four schema fields; of the three critical fields (`total`, `tax`, `invoice_number`) it labels only `total`. | Field | Precision | Recall | F1 | |---|---|---|---| | `total` (critical) | **99.0%** | 99.0% | 99.0% | | `vendor_name` | 86.0% | 86.0% | 86.0% | | `document_date` | 81.0% | 98.8% | 89.0% | | `vendor_address` | 53.0% | 53.0% | 53.0% | Routing at `CONFIDENCE_THRESHOLD = 0.50`: | Metric | Value | |---|---| | Auto-accepted | 18 / 100 (18%) | | Auto-accept precision on `total` | **100% (18 / 18)** | | Routed to review | 82 / 100 | Reading these numbers: - The precision comes from arithmetic, not model self-assessment. The one wrong total in the slice (99/100 correct overall) failed line-item reconciliation and went to review; it never reached the accepted set. - Low `vendor_address` accuracy is expected and absorbed by design: noisy free-form fields surface in the review queue rather than being trusted. - This slice labels `total` only. The CORD adapter (`tax`, line items) and an invoice-JSON adapter (`invoice_number`) are scaffolded next steps -- tracked in [`PROGRESS_FUTURE.md`](PROGRESS_FUTURE.md). - Honest limit: arithmetic consistency correlates with correctness but does not prove it. A document whose numbers are all wrong yet mutually consistent would pass. On this slice the filter yielded 18/18. Reproduce: ```bash # Phase 1 -- runs the model over a slice (spends free-tier quota); idempotent. uv run python -m eval.run_eval predict --dataset sroie --limit 100 # Phase 2 -- offline; recomputes metrics and the threshold sweep from the cache. uv run python -m eval.run_eval score --dataset sroie ``` ## Getting started Requires Python 3.11 and [uv](https://docs.astral.sh/uv/). ```bash uv sync # create the venv and install from uv.lock cp .env.example .env # add your Gemini key (free, from Google AI Studio) ``` Run the web demo (single upload, result rendered, nothing stored): ```bash uv run python -m doc_agent.web.app ``` Run the autonomous watcher (drop files into `data/inbox/`; accepted records land in SQLite, accepted files move to `data/processed/`, everything else to `data/review/`; CSV export is a separate step over the accumulated records): ```bash uv run python -m doc_agent.ingest.watcher ``` Or call the core directly -- it has no side effects and no dependency on either entry point: ```python from doc_agent.config import load_config from doc_agent.core import process_document result = process_document("receipt.jpg", settings=load_config()) print(result.decision) # "accept" | "review" print(result.confidence) # document-level confidence ``` Run the tests (218 tests, fully offline -- no API key needed): ```bash uv run pytest -q ``` The implemented backend is Gemini (`EXTRACTION_BACKEND=gemini`). A local Ollama backend with an OCR path -- for fully offline, private runs -- is scaffolded in config but not yet built ([`PROGRESS_FUTURE.md`](PROGRESS_FUTURE.md)). ## Caveats - The hosted demo runs on the Gemini free tier, which may use inputs for training. **Synthetic or public documents only** -- never real financial data. Fully private local processing is planned, not yet implemented. - The free Space sleeps when idle; the first request after a quiet period is a cold start, and the first PDF triggers a one-time parser model download. ## License MIT -- see [LICENSE](LICENSE). The benchmark datasets used for evaluation (SROIE and others) carry their own research licenses and are not redistributed in this repository.