LEARN.md

# LEARN — a working knowledge of what you built

> If a hiring manager pulls this project up in a screen and asks you to walk
> them through it, this doc is the map you narrate from. Every design
> decision, every trade-off, every "why not the other thing" is captured
> here. Read it end to end once. Reread the section for whichever thing they
> poke at.

---

## Table of contents

1. [The one-liner (what it is, why it exists)](#1-the-one-liner)
2. [The end-to-end request flow](#2-the-end-to-end-request-flow)
3. [Every layer, one at a time](#3-every-layer-one-at-a-time)
   - 3.1 Pydantic schemas + the strict-mode constraint
   - 3.2 The extractor + the envelope pattern
   - 3.3 The document loader
   - 3.4 The OpenAI client (why `temperature=None`, why `reasoning_effort="minimal"`)
   - 3.5 Section chunker for 10-Ks
   - 3.6 FastAPI backend + dependency injection for tests
   - 3.7 Evaluation harness — the P/R/F1 machinery
   - 3.8 UI (Paper & Ink, React + Motion + R3F)
   - 3.9 Docker + HF Spaces single-container deploy
   - 3.10 GitHub Actions CI
4. [The 14 design decisions you should be able to defend](#4-the-14-design-decisions-you-should-be-able-to-defend)
5. [The roadmap explained — v1 v2 v3 v4](#5-the-roadmap-explained)
6. [Interview cheat sheet](#6-interview-cheat-sheet)
7. [File map — what lives where](#7-file-map)

---

## 1. The one-liner

**A production-grade LLM service that turns unstructured business documents
(invoices, receipts, SEC 10-K filings) into schema-validated JSON with
per-field confidence, cost accounting, and a reproducible evaluation harness.**

The point of the project isn't the extraction. Every LLM demo does that. The
point is everything *around* it — schemas, evaluation, cost tracking, multi-
model benchmarking, section chunking for long documents, tests, CI, a real
frontend, Docker, and a live public URL. That's the shape of shipped LLM
software, and it's what hiring managers screen for.

**Who would use this**: a fintech that gets 10,000 vendor invoices a month and
needs to auto-populate their AP system. A hedge fund analyst who wants 10-K
financials in Snowflake. A small business owner who wants receipts categorized
into QuickBooks. All three use the same API — different `doc_type`, different
schema, same envelope.

---

## 2. The end-to-end request flow

Here's what happens when a user drops a PDF into the UI. Follow this sequence
carefully — every layer earns its keep.

```
┌────────────┐   1. multipart POST   ┌──────────────┐   2. dispatch on doc_type    ┌───────────────┐
│  React UI  │──────────────────────►│   FastAPI    │─────────────────────────────►│  Extractor    │
│ (Dropzone) │◄──────────────────────│  /extract    │◄─── 6. ExtractionResult ─────│ orchestrator  │
└────────────┘                       └──────────────┘                              └───────┬───────┘
                                                                                          │
                                                                                          │ 3. load
                                                                                          ▼
                                                                                  ┌───────────────┐
                                                                                  │ document      │
                                                                                  │ loader        │
                                                                                  │ (pdf/img/txt) │
                                                                                  └───────┬───────┘
                                                                                          │
                                                                                          │ text + images
                                                                                          │
                                    ┌─────────────────────────────────────────────────────┤
                                    │  4. filing? → chunk into cover + Item 1A + Item 8   │
                                    │     invoice/receipt? → send whole doc               │
                                    ▼                                                     ▼
                             ┌─────────────┐                                    ┌──────────────────┐
                             │ system      │                                    │  OpenAI          │
                             │ prompt      │──── messages ─────────────────────►│ chat.completions │
                             │ (per type)  │                                    │      .parse      │
                             └─────────────┘                                    └────────┬─────────┘
                                                                                         │
                                              5. envelope validated ◄──────── strict JSON│
                                              by Pydantic v2                             │
                                              (extra=forbid)                             ▼
                                                                                ┌──────────────────┐
                                                                                │ ExtractionResult │
                                                                                │ + metrics        │
                                                                                └──────────────────┘
```

**Steps 1-6 in words:**

1. **UI POSTs `file + doc_type + optional model` to `/extract`.** The React
   Dropzone builds the multipart request. `doc_type` is one of
   `receipt | invoice | filing`. The optional `model` flag is what powers the
   multi-model benchmark.

2. **FastAPI receives, dispatches to `DocumentExtractor.extract()`.** The
   router (`src/api/routers/extract.py`) validates the file size (413 error
   over 10 MB), the content type (415), the doc_type (400), and injects the
   extractor via `Depends(get_extractor)`. That dependency injection is
   important — it lets tests swap in a fake extractor.

3. **Document loader parses the bytes.** `document_loader.py` sniffs the
   filename and dispatches. PDFs go through pdfplumber for text; if
   text-density is under 100 chars per page it falls back to PyMuPDF page
   rendering → PNG → vision. Images pass straight through as base64. `.txt`
   / `.md` decode as UTF-8. Returns a `LoadedDocument(text, images_b64,
   source_type)`.

4. **Dispatch on `doc_type`.** For invoices/receipts, one system prompt + the
   full document text + images → one OpenAI call. For filings, the section
   chunker runs first, slicing the doc into cover + Item 8 + Item 1A blocks,
   and the message builder stitches those into three labeled sections. This is
   the trick that makes 10-Ks affordable (~$0.06/doc instead of ~$0.60/doc if
   you ship the whole 150K-token document).

5. **OpenAI `chat.completions.parse` with a Pydantic response format.** This
   is the game-changer. Instead of asking for JSON in a prompt and hoping,
   we hand OpenAI the *envelope schema* — Pydantic v2 → JSON Schema —
   and OpenAI's strict mode guarantees the return matches. No JSON parsing
   errors, no missing fields, no extra fields. If the model can't comply,
   the API returns 400 instead of garbage.

6. **Response gets wrapped in `ExtractionResult`.** The extractor computes
   `overall_confidence` from the per-field scores, attaches the raw text
   snippet for debugging, and returns `(result, metrics)`. Metrics include
   tokens, cost, latency, and the model used — every request is fully
   accounted for.

---

## 3. Every layer, one at a time

### 3.1 Pydantic schemas + the strict-mode constraint

`src/schemas/{base,common,invoice,receipt,filing,registry}.py`.

**The base**: `StrictModel` has `extra="forbid"`. This one config choice
cascades through everything. It means:

- The model cannot invent fields. If it hallucinates `internal_note`, Pydantic
  rejects the whole response.
- OpenAI's structured-outputs strict mode requires `additionalProperties:
  false` on every object in the schema — which is exactly what `extra="forbid"`
  produces.
- Adding a new field means adding it to the schema. There's no other place it
  can come from.

**The registry pattern**: `src/schemas/registry.py` maps `"invoice" | "receipt"
| "filing"` to schema classes. Adding a new domain (e.g. purchase orders)
is a one-line change here plus a new schema file. Downstream code — the API,
the eval CLI, the extractor, the envelope factory — all look up via
`get_schema(doc_type)`. No hardcoded branches anywhere.

**Money as `float`, not `Decimal`**. Read this carefully because it's a
question you *will* be asked. OpenAI structured outputs represent `Decimal`
as `string` in JSON Schema — which pushes the numeric parsing back onto the
model as a string task and reduces precision. `float` is `number` in JSON
Schema, which the model handles natively. We round to 2 decimals on assignment
via a validator. If this were an accounting system-of-record, we'd use
`Decimal` at the *application* layer *after* extraction — but at the LLM
boundary, `float` is correct.

### 3.2 The extractor + the envelope pattern

`src/extractors/extractor.py` orchestrates. `src/extractors/envelope.py`
handles the Pydantic gymnastics.

The **envelope pattern** answers a subtle question: what does the LLM
return? Not `ExtractionResult` directly, because some of that (like
`overall_confidence`, computed as mean-of-scores) is our code's job, not the
model's. Instead we generate — dynamically per doc type — an envelope class:

```python
Envelope = create_model(
    "InvoiceEnvelope",
    data=(Invoice, ...),
    field_confidences=(list[FieldConfidence], ...),
    warnings=(list[ExtractionWarning], ...),
)
```

The model outputs the envelope, our code wraps it in `ExtractionResult` and
adds the computed pieces. Envelopes are LRU-cached so they're created once
per doc type.

### 3.3 The document loader

`src/extractors/document_loader.py`. Dispatches on filename + magic bytes:

- `.pdf` → try pdfplumber first. If text density is under 100 chars/page,
  fall back to PyMuPDF page rendering → PNG → send images to vision.
- `.png / .jpg / .jpeg / .webp / .tiff / .bmp` → base64 encode, ship as
  vision input.
- `.txt / .md / .log / .text` → UTF-8 decode, ship as text.
- Anything else → `source_type="empty"`, extractor raises.

The text/image branch matters because a scanned receipt with no OCR layer
looks empty to pdfplumber. The rendering fallback lets vision do the work.

### 3.4 The OpenAI client

`src/extractors/openai_client.py`. Two non-obvious details:

**`temperature=None` (not 0.0)**. The gpt-5 family only accepts the default
temperature; passing `0.0` returns a 400. We use a nullable field and only
send it if the caller explicitly requests one. This bit us early — the
original code hardcoded `temperature=0.0` and every gpt-5-nano call failed.

**`reasoning_effort="minimal"`**. gpt-5 models have an internal chain-of-
thought that's billed to you as reasoning tokens. On our schema, the default
effort costs $0.042/doc and takes 30 seconds. `minimal` costs $0.012/doc and
takes 7 seconds with no measured quality loss. The flag is exposed all the
way through the CLI so the benchmark can compare.

**Tenacity retry**. The client retries on `RateLimitError` and `APIError` with
exponential backoff. Nothing exotic — just a decorator.

### 3.5 Section chunker for 10-Ks

`src/extractors/section_chunker.py`. This one's pretty:

A 10-K is 50-250 pages. Even at gpt-5's 400K token context, feeding the whole
thing costs ~$0.60/doc. But 10-Ks have structured Items — `Item 1A. Risk
Factors`, `Item 7. MD&A`, `Item 8. Financial Statements` — that the chunker
finds via regex.

The regex tolerates every real-world variant: `ITEM 1A.`, `Item 1A —`,
`item 1a:`. It runs across the whole document and returns every hit, then
**keeps only the last occurrence of each item ID** — because every 10-K
mentions each Item at least twice (Table of Contents + real section), and the
last one is always the real body.

Cover slice = first 4KB before the first Item. Section bodies = from just
after the heading to the start of the next heading. Result: a `ChunkedFiling`
with `.get("1A")`, `.has("8")`, `.get_text("1A", default="…")`.

Downstream, the extractor stitches cover + Item 8 + Item 1A into three
labeled blocks in one prompt. Prompt tokens drop from ~150K to ~30-40K,
cost drops from ~$0.60/doc to ~$0.06/doc, and the model has less distractor
text to sift through.

### 3.6 FastAPI backend

`src/api/main.py`, `routers/`, `middleware.py`, `errors.py`.

- **Endpoints**: `GET /` (banner), `GET /health` (probe for Docker),
  `GET /schemas` (list doc types), `GET /schemas/{doc_type}` (JSON Schema),
  `POST /extract` (the money endpoint).
- **Dependency injection**: `get_extractor` returns a singleton
  `DocumentExtractor` via `lru_cache`. In tests, `app.dependency_overrides`
  swaps it for a fake — so the whole test suite runs without an OpenAI key.
- **Middleware**: `RequestIDMiddleware` attaches an `X-Request-ID` header;
  `AccessLogMiddleware` emits one structured JSON log line per request.
- **Error envelope**: every 4xx/5xx returns `{"error": {code, message,
  request_id, details}}`. Consistent shape for the frontend to render.
- **CORS**: wide open. Fine for a portfolio; you'd lock this down in prod.

### 3.7 Evaluation harness

`src/eval/{flatten,comparators,metrics,runner,report,cli}.py`.

This is the piece I'm proudest of. Given a JSONL of ground-truth records
and an extractor function, it produces per-record CSV, JSON summary, and
resume-worthy markdown — all reproducible from `python scripts/run_eval.py`.

**Flattener** walks a Pydantic model + dicts recursively and produces
`{"total": 12.99, "line_items[0].description": "Coffee", ...}`. So nested
schemas score the same way flat ones do.

**Type classifier** looks at each field's Python type and JSON value:
- `float` fields with `money` in the name → money comparator (0.01 absolute
  OR 0.5% relative tolerance — either passes)
- other `float` / `int` → number comparator (exact)
- `str` with currency/SKU/phone/ID hints → exact comparator
- other `str` → fuzzy comparator (rapidfuzz `token_set_ratio ≥ 85`)
- `date` / `time` → ISO equality

**Metrics** compute per-field TP/FP/FN/TN, then **micro F1** (weighted by
field support), **macro F1** (mean of per-field F1 across every field with
non-zero support), and **doc-exact-match** (fraction of docs where every
field is right).

**Runner** takes an *injected* extractor function (`ExtractorFn` protocol),
so `--mode selfcheck` uses a mock that returns ground truth verbatim
(validates the eval pipeline itself always hits F1=1.0), and `--mode live`
uses `DocumentExtractor`. Same code path.

The CLI accepts `--model` and `--reasoning-effort` — which is what makes the
multi-model benchmark a one-command sweep.

### 3.8 UI — Paper & Ink

`ui/src/`. React + Vite + TypeScript + Tailwind + Motion + React Three Fiber.

Deliberate choice: **do not look like a generic AI-SaaS demo**. Editorial
aesthetic. Cream paper + deep ink navy in light mode; warm charcoal +
parchment in dark. Instrument Serif italic for the display face. Grain
overlay via inline SVG turbulence.

**Hero**: a 3D paper sheet with mouse parallax (React Three Fiber), procedural
canvas texture (no glTF file), that folds a corner and turns green when the
extraction succeeds. Reduced-motion respected.

**Dropzone**: drag-drop or browse, sample buttons for a real receipt PNG and
invoice PDF that ship in `ui/public/samples/`. Doc-type picker with `Receipt
| Invoice | 10-K`. Confidence rendered as an ink well that fills from the
bottom; cost as a wax-stamped number.

**API contract** in `ui/src/lib/api.ts`. Types mirror the FastAPI response
shape 1:1.

### 3.9 Docker + HF Spaces single-container deploy

Root `Dockerfile` is three-stage:

1. `node:20-alpine` builds the UI → `dist/`.
2. `python:3.11-slim` installs the Python venv.
3. Runtime: nginx + uvicorn + tini as PID 1. `docker/nginx.hf.conf` listens
   on 7860 (HF's required port), serves `dist/` at `/`, proxies `/api/*` to
   `127.0.0.1:8000`. Entrypoint launches uvicorn in the background, polls
   `/health` for up to 30s so nginx never serves 502s during boot, then
   exec's nginx in the foreground with a SIGTERM trap.

`docker/api.Dockerfile` + `docker/ui.Dockerfile` + `docker-compose.yml` are
the two-container local-dev variant. Same code, different packaging.

HF Space YAML frontmatter (`sdk: docker`, `app_port: 7860`) tells HF to build
the root `Dockerfile`. `OPENAI_API_KEY` is a Space secret.

### 3.10 GitHub Actions CI

`.github/workflows/ci.yml`. Three parallel jobs on every push:

1. **python-lint-and-test** — ruff + pytest across 126 unit tests (no OpenAI
   key needed, thanks to the injected extractor).
2. **ui-typecheck-and-build** — tsc + vite build.
3. **docker-build** — buildx multi-arch build with GHA cache, gated on both
   lint jobs.

---

## 4. The 14 design decisions you should be able to defend

If asked "why did you do X instead of Y," here's the honest answer for each.

| # | Decision | Alternative | Why we chose ours |
|---|----------|-------------|-------------------|
| 1 | Pydantic v2 with `extra="forbid"` | dataclasses, Marshmallow, JSON Schema alone | OpenAI's structured-outputs strict mode maps directly onto Pydantic's config. Free validation + JSON Schema generation. |
| 2 | Money as `float` with rounder validator | `Decimal` | OpenAI structured outputs render `Decimal` as `string` in JSON Schema — model has to parse. `float` is native `number`. Round on assignment. Convert to `Decimal` at the app layer if needed. |
| 3 | Envelope pattern (dynamically generated) | Model outputs full `ExtractionResult` | Some fields (`overall_confidence`, `raw_text_snippet`) are our code's job. Envelope is the LLM contract; ExtractionResult is the app contract. |
| 4 | Registry for doc types | `if/elif` on doc_type | One place to add a new domain. Downstream code is generic. |
| 5 | `client.beta.chat.completions.parse` | Prompt-then-JSON.loads | Native structured outputs eliminate a whole class of parsing errors and hallucinated schemas. |
| 6 | `reasoning_effort="minimal"` on gpt-5 | Default effort | 3.5× cheaper, 4× faster, no measurable quality drop on structured extraction. Verified with the eval harness. |
| 7 | `temperature=None` (opt-in) | `temperature=0.0` hardcoded | gpt-5 family rejects `temperature=0.0`. Had to be discovered by breakage. |
| 8 | Section chunking for 10-Ks | Whole-doc single call | ~10× cost reduction ($0.60→$0.06) with equal-or-better signal ratio. |
| 9 | Regex chunker, "keep last occurrence" | Full parser (unstructured, llama_index) | Real 10-Ks mention each Item at least twice (TOC + body). Keep-last is deterministic and works. |
| 10 | Injected extractor in eval + API | Real extractor everywhere | Test suite runs without OpenAI. CI stays deterministic and free. |
| 11 | Rapidfuzz `token_set_ratio ≥ 85` for text | Exact string match | Merchant names print as "SANYU STATIONERY" vs `"Sanyu Stationery Shop"`; those are the same merchant. |
| 12 | Money tolerance: 0.01 abs OR 0.5% rel | Exact match | Real receipts round differently. Both prevent silly-false-negatives. |
| 13 | React + Motion + R3F for UI | Streamlit / Gradio | Portfolio project. Recruiters can tell in 5 seconds whether the UI was thrown together. Editorial aesthetic on purpose. |
| 14 | Single Docker container on HF Spaces | HF SDK Gradio, or Vercel + Railway | HF Spaces is the AI-community-recognized host, single container = one URL, one deploy, one health check. |

---

## 5. The roadmap explained

The README roadmap has four checkboxes. Here's what each one actually means
in concrete engineering terms.

### v1 — Invoices & Receipts pipeline + multi-model benchmark ✅

**Shipped 2026-07-04 → 2026-07-05.** Full stack: schemas, extractor,
FastAPI, React UI, Docker, HF Spaces, 96 tests, CI green. Live eval on
gpt-5-nano hit **0.94 F1 on receipts at $0.012/doc, 6s latency**. Multi-model
benchmark swept nano vs mini vs full at reasoning-effort=minimal on the same
10 records; **nano was Pareto-optimal (0.896 micro F1 at $0.0116/doc)** —
larger tiers led on macro F1 only.

### v2 — SEC 10-K pipeline ✅

**Shipped 2026-07-05 → 2026-07-06.** Adds the `filing` doc type end-to-end:
`FilingCover` + `FilingFinancials` + `RiskFactor` Pydantic schemas, regex-based
section chunker for Items 1A/8, filing-specific system prompt with unit-of-
measure normalization rules, EDGAR downloader script that respects SEC's
10 req/s + User-Agent policy, XBRL-driven auto-ground-truth builder. First
live eval: **0.56 micro F1 at $0.06/doc**. v2.2 attempted three targeted
fixes (prompt worked examples, FIN_MAP expansion for banks, cover backfill)
— per-field improvements on revenue/total_equity/operating_income, aggregate
stayed flat because filing_date regressed and total_debt didn't move. Full
diagnose→try→measure narrative in the README. **30 new tests, 126 total.**

### v3 — Streaming extraction + async batch API ✅

**Shipped 2026-07-06.** Two extra endpoints landed alongside `POST /extract`:

**Streaming (`POST /extract/stream`).** Returns Server-Sent Events. Each
frame is `event: <name>\ndata: <json>\n\n`. The stages we emit are
`starting`, `loading`, `model_call`, `validated`, then `result` with the
full validated `ExtractionResult`, then `done`. Errors are in-band
`error` events (the HTTP 200 has already gone out by the time the
extractor raises).

Why *progress events* instead of *partial JSON deltas*: OpenAI's
`chat.completions.parse()` with `stream=True` yields raw token chunks,
not partial validated Pydantic objects. To emit field-by-field you'd
have to run your own streaming JSON parser (`ijson` / `json-stream`)
that tolerates mid-value boundaries, then re-validate the accumulated
object against the envelope schema — that gets fragile on nested arrays
(line items, risk factors). Shipping progress events is the honest UX
without the JSON-stream footguns. If OpenAI adds object-level streaming
for structured outputs later, we extend this endpoint with `partial`
events on top of the existing progress ones — the SSE format welcomes
both.

**Batch (`POST /extract/batch` + `GET /extract/batch/{job_id}`).**
Multipart with N files. Returns 202 with a `job_id` immediately;
extraction runs in FastAPI `BackgroundTasks`. `GET` returns a snapshot:
overall status, per-item status, per-item result/metrics/error as they
land. Poll every 1-3s.

Concurrency architecture (`src/api/batch_store.py`):

- One in-memory `dict` guarded by an `asyncio.Lock` = the job store.
  Restart wipes it — that's the accepted tradeoff at this scale. Prod
  swaps Redis + RQ in without touching the API.
- One `asyncio.Semaphore(5)` initialized on first use, shared across
  every in-flight job. OpenAI rate limits are per-org, not per-job, so
  job-scoped limits wouldn't protect us.
- Extraction runs via `asyncio.to_thread(extractor.extract, …)` because
  the openai-python SDK is synchronous under `.parse()`. Threading keeps
  the event loop free to service `GET /extract/batch/{id}` polls and to
  fan work out to the next job.

**Tests: 12 new (in `tests/unit/test_api_v3.py`), 138 total.** All use the
injected fake extractor — no OpenAI key required in CI.

**Estimated work: shipped in 1 session, no API cost.** Same extractor,
different wrappers.

### v4 — Fine-tuning experiment vs. base GPT-5 nano ✅

**Shipped 2026-07-06 (code + tests + docs).** Real-scale training run pending
on your side.

**Three scripts, one loop:**

- `scripts/prep_ft_dataset.py` — reads a receipt/invoice JSONL (must have
  `text` + `ground_truth`), converts each row to OpenAI's chat-completions
  fine-tuning format. The system message is the *exact* production prompt
  (`get_prompt(doc_type)`); the user message is the *exact* production shape
  (`---BEGIN DOCUMENT TEXT---` block); the assistant message is the
  envelope (`{data, field_confidences, warnings}`) with the ground truth as
  `data`. If any of those diverge from what `DocumentExtractor` sends at
  inference, the fine-tune won't transfer — hence the shape-matching tests.
  80/20 train/val split, deterministic per `--seed`.

- `scripts/launch_finetune.py` — uploads train + val files via
  `client.files.create(purpose="fine-tune")`, kicks off the job via
  `client.fine_tuning.jobs.create()`, then polls status every 30 s until
  the job finishes. `Ctrl-C` detaches cleanly (the job keeps running on
  OpenAI's side). Default target is `gpt-4o-mini-2024-07-18` — the
  cheapest reliable fine-tune base at ~$3 / 1M training tokens.

- `scripts/compare_finetune.py` — takes the resulting `ft-model` id and
  runs the same 10-record smoke eval (SROIE + CORD) against both the base
  gpt-5-nano @ minimal and the fine-tune, emits a side-by-side comparison
  table (`evaluation/finetuning/<timestamp>/comparison.{md,csv,json}`) —
  same shape as the multi-model benchmark.

**9 new tests** in `tests/unit/test_prep_ft_dataset.py` — pure JSON
transformation tests, no network. Confirms the fine-tuning JSONL is
valid OpenAI chat format, the assistant content is a valid envelope, the
system prompt matches production, the user shape matches
`DocumentExtractor._build_messages()`, the 80/20 split is deterministic,
malformed rows are skipped without crashing, and OpenAI's ≥10-example
minimum is warned about explicitly.

**The interesting resume story** isn't "F1 went up." It's the tradeoff:
gpt-5-nano @ minimal already hits 0.94 F1 on receipts at $0.012/doc with
no schema lock-in. A fine-tuned gpt-4o-mini will probably beat that on
F1 (say 0.96+) at ~2× per-token cost, and only for *this schema* — a
schema change means retraining. Whether you ship it is a
business-tradeoff question, not a technical one. **Senior answer**:
"I built the pipeline, ran it, and the numbers didn't justify shipping
the fine-tune for a schema this small and this well-served by
prompting." **Junior answer**: "I fine-tuned and F1 went up." Pick your
audience.

**To run the real experiment**, you need > 10 training examples — the
smoke datasets have 5 each. Pull the full SROIE / CORD splits with
`python scripts/prep_datasets.py all` first, then point
`prep_ft_dataset.py --input data/processed/sroie.jsonl` at the fuller
data. Realistic cost: **$0.50-$2 to train, $0.30-$0.80 for the
comparison eval** on gpt-4o-mini.

---

## 6. Interview cheat sheet

### "Walk me through your project."

Take 90 seconds. Don't start with the stack. Start with the story:

> I built a service that turns three kinds of business documents — invoices,
> receipts, SEC 10-Ks — into schema-validated JSON. The interesting part
> isn't the extraction itself; it's everything around it. Pydantic v2 with
> `extra="forbid"` gives OpenAI's structured-outputs strict mode a schema
> to enforce, so the model can't hallucinate fields. There's an evaluation
> harness that computes per-field precision, recall, and F1 against public
> ground truth. I used that harness to benchmark three GPT-5 tiers and
> proved that nano is Pareto-optimal on this workload. For 10-Ks I built a
> regex-based section chunker so we only ship Item 8 and Item 1A to the
> model — that drops cost from $0.60/doc to $0.06/doc. Deployed on Hugging
> Face Spaces via a single Docker container. Full stack: FastAPI backend,
> React + Motion + R3F frontend, GitHub Actions CI running 126 tests.

### "Why Pydantic?"

Runtime validation + JSON Schema generation for free. And its `extra="forbid"`
config maps 1:1 onto OpenAI's strict mode requirement of
`additionalProperties: false`. If I hadn't used Pydantic I'd have written
that JSON Schema by hand and validated the response manually.

### "How do you handle model errors or hallucinations?"

Three layers. First, `extra="forbid"` means the model *cannot* invent fields —
OpenAI's strict mode rejects those before we ever see them. Second, per-field
confidence scores let us surface uncertainty to the UI. Third, the eval
harness measures F1 against ground truth so we know *empirically* which
fields the model gets wrong — you can't catch what you don't measure.

### "How do you evaluate quality?"

Per-field precision, recall, F1 with type-appropriate comparators — money
uses a 0.01 absolute or 0.5% relative tolerance; text uses rapidfuzz's
token-set ratio at 85; dates use ISO equality. Then micro F1 for overall,
macro F1 for how rare fields do, and doc-level exact-match as the strictest
metric.

### "What was the hardest part?"

Two things. **One**: 10-K unit-of-measure — "in millions" means multiply
by a million and the model kept forgetting. I did a diagnose→try→measure
loop where I added worked examples to the prompt. Revenue precision doubled;
some other fields regressed. Real ML/eng work. **Two**: the streaming
consideration — I punted it to v3 because getting partial validated Pydantic
objects out of a token stream is a whole architectural rethink; the current
`.parse()` API only gives you the object at completion.

### "What would you do differently?"

Two-pass extract-then-verify for money fields. First call extracts as we do
now; second call is prompted with "here's what you just returned — verify
each dollar figure against the scale header in the source." Empirically that
kind of self-review pattern is where the next 15-20 F1 points on 10-K
financials live.

### "How did you deploy?"

Single-container Docker on Hugging Face Spaces. Multi-stage build: node
builds the UI to `dist/`, python installs the venv, runtime image has nginx
+ uvicorn + tini. Entrypoint launches uvicorn in the background, polls
`/health` for up to 30 seconds so nginx never serves 502s while boot is
in progress, then exec's nginx as PID 1. Space secret carries the OpenAI
key.

### "Why not just use LangChain?"

For this use case it would add complexity without helping. LangChain's
value is when you need agents, tool use, RAG. Structured extraction with
a Pydantic response format is one API call — LangChain would be a wrapper
over what I already have. If v3 grows an agent-style tool-use pattern
(e.g. "look up this vendor in our CRM") I'd reconsider.

### "How much did all this cost you to build?"

Under $2 in OpenAI API cost total. Multi-model benchmark was $0.36. Two
10-K eval runs were $0.30 each. Everything else is free — HF Spaces free
tier, GitHub free-tier CI, the sandbox for dev.

---

## 7. File map

```
04-structured-data-extraction/
├── src/                          # Backend Python
│   ├── schemas/                  # Pydantic schemas + registry
│   │   ├── base.py               # StrictModel, ExtractionResult, FieldConfidence, ExtractionWarning
│   │   ├── common.py             # Address, Party, MoneyMixin, currency helpers
│   │   ├── invoice.py            # Invoice, LineItem
│   │   ├── receipt.py            # Receipt, ReceiptLineItem
│   │   ├── filing.py             # Filing, FilingCover, FilingFinancials, RiskFactor  ← v2
│   │   └── registry.py           # doc_type → schema class
│   ├── extractors/
│   │   ├── document_loader.py    # PDF/image/txt → LoadedDocument
│   │   ├── prompts.py            # SYSTEM_PROMPT_{INVOICE,RECEIPT,FILING} + common rules
│   │   ├── envelope.py           # Dynamic envelope Pydantic model, per-schema LRU cached
│   │   ├── openai_client.py      # OpenAI wrapper, retry, reasoning_effort, temperature=None
│   │   ├── section_chunker.py    # Regex-based 10-K Item splitter, TOC dedup            ← v2
│   │   └── extractor.py          # Top-level DocumentExtractor, dispatches on doc_type
│   ├── api/
│   │   ├── main.py               # FastAPI app + CORS + middleware wiring
│   │   ├── deps.py               # get_extractor dependency (LRU-cached singleton)
│   │   ├── errors.py             # APIError subclasses, exception handler
│   │   ├── middleware.py         # RequestID + AccessLog middleware
│   │   └── routers/
│   │       ├── health.py         # GET /, /health
│   │       ├── schemas.py        # GET /schemas, /schemas/{doc_type}
│   │       └── extract.py        # POST /extract (multipart)
│   ├── data_prep/                # SROIE + CORD normalizers, JSONL writer
│   ├── eval/
│   │   ├── flatten.py            # Nested Pydantic → flat dotted-path dict
│   │   ├── comparators.py        # money / number / text / exact / date comparators
│   │   ├── metrics.py            # TP/FP/FN/TN, micro F1, macro F1, doc-exact
│   │   ├── runner.py             # run_eval(records, extractor_fn, doc_type) → EvalReport
│   │   ├── report.py             # write CSV, JSON summary, markdown
│   │   └── cli.py                # argparse entry point (--mode, --model, --reasoning-effort)
│   └── utils/                    # config (dotenv), cost_tracker, logging (loguru)
├── ui/                           # React + Vite + TS + Tailwind + Motion + R3F
│   ├── src/
│   │   ├── App.tsx               # Page composition
│   │   ├── main.tsx              # Vite entry
│   │   ├── components/
│   │   │   ├── Hero.tsx          # Kinetic-headline + 3D paper
│   │   │   ├── PaperScene.tsx    # R3F Canvas, procedural paper texture
│   │   │   ├── ExtractSection.tsx# Workbench (Dropzone + ResultsPanel)
│   │   │   ├── Dropzone.tsx      # File upload + doc-type picker + samples
│   │   │   ├── ResultsPanel.tsx  # JSON view + confidence + cost
│   │   │   ├── ConfidenceInkwell.tsx, JsonView.tsx, MetricsStrip.tsx, WarningsList.tsx
│   │   │   ├── HowItWorks.tsx, Numbers.tsx, TopNav.tsx, Footer.tsx
│   │   │   ├── ThemeToggle.tsx   # dark/light one-attribute flip
│   │   │   └── CustomCursor.tsx  # ink-dot cursor with spring lag
│   │   ├── hooks/                # useExtract, useTheme
│   │   ├── lib/                  # api.ts (fetch client), samples.ts (Coffee/Software)
│   │   ├── styles/               # theme.css (tokens), globals.css (grain, cursor)
│   │   └── types.ts              # DocType, ExtractResponse mirror the API
│   ├── public/samples/           # coffee_receipt.png + software_invoice.pdf (real files!)
│   ├── package.json, tsconfig.json, vite.config.ts, tailwind.config.js
├── scripts/                      # CLI tools
│   ├── prep_datasets.py          # Download + normalize SROIE/CORD
│   ├── run_eval.py               # Thin wrapper over src/eval/cli.py
│   ├── run_multimodel_benchmark.py  # Matrix sweep → comparison.{md,csv,json}
│   ├── download_edgar.py         # 5-issuer 10-K download from SEC EDGAR      ← v2
│   └── build_filings_gt.py       # XBRL companyfacts → ground truth JSONL     ← v2
├── tests/
│   └── unit/                     # 126 tests, no OpenAI key required
│       ├── test_schemas.py       # Invoice/Receipt schema
│       ├── test_filing_schema.py # Filing schema — 20 tests                    ← v2
│       ├── test_section_chunker.py # Section chunker — 10 tests                ← v2
│       ├── test_extractor.py     # DocumentExtractor with FakeClient
│       ├── test_eval.py          # Flatten, comparators, metrics
│       ├── test_data_prep.py     # SROIE + CORD normalizers
│       └── test_api.py           # FastAPI TestClient + dependency override
├── data/
│   ├── samples/                  # Committed 10 hand-crafted samples
│   └── raw/10k/                  # Downloaded 10-Ks (gitignored by default)   ← v2
├── evaluation/
│   ├── smoke_sroie_sample.jsonl  # 5 receipts (SROIE, live eval)
│   ├── smoke_cord_sample.jsonl   # 5 receipts (CORD, live eval)
│   ├── smoke_filings_sample.jsonl # 5 10-Ks (live eval)                        ← v2
│   ├── reports/<timestamp>/      # Per-run reports
│   └── benchmarks/<timestamp>/   # Multi-model comparison rollups
├── docker/
│   ├── api.Dockerfile            # Two-container local dev
│   ├── ui.Dockerfile             # Two-container local dev
│   ├── nginx.conf                # Two-container proxy config
│   ├── nginx.hf.conf             # HF Space single-container config (port 7860)
│   └── hf-entrypoint.sh          # tini + uvicorn + nginx orchestration
├── Dockerfile                    # Single-container HF build (3-stage)
├── docker-compose.yml            # Two-container local dev
├── .github/workflows/ci.yml      # 3 parallel jobs
├── requirements.txt              # Python deps
├── pyproject.toml                # ruff + black + pytest config
├── .env.example                  # OPENAI_API_KEY placeholder
├── .gitignore, .gitattributes
├── README.md                     # The narrative + numbers
├── SHIPPING.md                   # Resume bullet + LinkedIn draft
└── LEARN.md                      # This document
```

---

**Read this once. Skim it before any interview. Every design decision has an
answer here — no hand-waving needed.**