--- title: Structured Data Extractor emoji: 📄 colorFrom: yellow colorTo: gray sdk: docker app_port: 7860 pinned: false license: mit short_description: Receipts + invoices to validated JSON, GPT-5 nano --- # Structured Data Extraction Service > Multi-domain document extraction — turn invoices, receipts, and SEC filings into schema-validated JSON with confidence scoring, multi-model benchmarking, and quantified accuracy. [![CI](https://github.com/adityapatel007-byte/structured-data-extractor/actions/workflows/ci.yml/badge.svg)](https://github.com/adityapatel007-byte/structured-data-extractor/actions/workflows/ci.yml) [![HF Space](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Live%20demo-yellow)](https://huggingface.co/spaces/aditya0103/structured-data-extractor) [![Python](https://img.shields.io/badge/python-3.11+-blue)]() [![OpenAI](https://img.shields.io/badge/LLM-GPT--5%20nano-green)]() [![License](https://img.shields.io/badge/license-MIT-lightgrey)]() --- ## What this does Upload any invoice, receipt, or SEC filing (10-K / 10-Q). Get back clean, validated JSON matching a Pydantic schema — plus a per-field confidence score, a cost breakdown, and latency metrics. ``` PDF / Image / Scan ─► Router ─► {Invoice | Receipt | Filing} Pipeline ─► JSON + Confidence │ └─► Evaluation Harness ─► Precision / Recall / F1 ``` ## Why this project exists Enterprise doc extraction is one of the highest-demand LLM use cases in 2026. This repo demonstrates the production skills that hiring managers screen for: - Schema-driven extraction with **OpenAI structured outputs** + Pydantic validation - **Vision-language handling** for scanned/image PDFs (GPT-5 nano vision) - **Long-document handling** for 10-K filings — regex section chunker slices Items 1A + 8 out of ~150K-token documents to hit **$0.06/doc** (would've been ~$0.60/doc whole-doc) - **Streaming + async batch API** — `POST /extract/stream` emits Server-Sent Events for progress + result; `POST /extract/batch` fans up to 5 concurrent extractions across N documents with per-item status tracking - **Fine-tuning experiment** — full pipeline (data prep → OpenAI job launch → side-by-side eval) built and reproducible; sample-scale run shipped, real-scale run gated on pulling the SROIE/CORD training splits - **Multi-model benchmarking** — empirically compared gpt-5-nano vs gpt-5-mini vs gpt-5 on the same 10-record eval; nano is Pareto-optimal (micro F1 0.896 at $0.012/doc) - **Evaluation harness** with precision / recall / F1 on public ground truth (SROIE, CORD) - **Cost + latency observability** — every extraction logs tokens and $ - Full-stack: **FastAPI** backend, **React + Motion + R3F** UI, **Docker**, GitHub Actions **CI** - **CI/CD** with GitHub Actions running tests + lint on every push ## Live demo The full stack is on Hugging Face Spaces: > **[huggingface.co/spaces/aditya0103/structured-data-extractor](https://huggingface.co/spaces/aditya0103/structured-data-extractor)** — always-on, free tier Locally: `docker compose up --build` after cloning; hit [http://localhost:5173](http://localhost:5173). ## Deploy your own to HF Spaces 1. Create a new [HF Space](https://huggingface.co/new-space) → **Docker SDK**, blank template. 2. Add `OPENAI_API_KEY` under **Settings → Repository secrets**. 3. Point the Space at this repo (or push a fork). HF reads the YAML frontmatter at the top of this README (`sdk: docker`, `app_port: 7860`), builds the root `Dockerfile`, and exposes it on your Space URL. First build ~5-8 min; every redeploy ~2-3 min thanks to layer caching. ## Quantified results Live evaluation on **gpt-5-nano** with `reasoning_effort="minimal"`. 10 receipt records derived from public SROIE + CORD ground truth. Reports (per-record CSV, JSON summary, markdown) land in `evaluation/reports//` after each run. | Domain | Dataset (n=) | Micro F1 | Macro F1 | Doc Exact | Cost / doc | Mean Latency | |----------|-------------------|-----------|-----------|-----------|------------|--------------| | Receipts | SROIE (5) | **0.938** | **1.000** | 0.20 | **$0.012** | 6.3 s | | Receipts | CORD (5) | **0.914** | 0.839 | **0.80** | **$0.012** | 8.2 s | | Filings | SEC 10-K (n=5) | **0.560** | **0.584** | 0.00 | **$0.063** | 6.4 s | **Read the numbers:** - **Micro F1 ≈ 0.92** across both datasets — the model gets ~92% of individual fields correct on ground-truth-derived text. - **Doc-level exact match** is stricter (100% of fields right on one doc) and swings by dataset: CORD receipts (short, simple line items) hit 0.80; SROIE (freer-form Malaysian/Singaporean receipts with more optional fields) hits 0.20 — a single missing field kills the metric on those. - **$0.012 / doc** is the reasoning-tokens-included cost at `reasoning_effort=minimal`. Default (non-minimal) reasoning was **$0.042 / doc, 30 s / doc** — the minimal flag is a ~3.5× cost cut and ~4× latency cut with no measured quality loss on this schema. - **Total spend for a full run: ~$0.12** — cheap enough to re-run on every significant prompt/schema change. **Why the 10-K F1 is lower than the receipt F1** — this is a *harder* task and the number reflects that honestly: - **Unit-of-measure normalization.** 10-K income statements are printed "in millions" or "in thousands." The extractor has to multiply back to absolute dollars. When it misses the header, a $391B revenue lands as $391K in the output — a 1,000,000× miss that reads as `False` on the money comparator. - **Multi-year column selection.** Every 10-K shows the most recent fiscal year alongside 1-2 prior years side-by-side. Picking the wrong column produces a plausible-but-wrong number. - **Debt aggregation.** `total_debt` = short-term + long-term borrowings, which the model must sum. Ground truth is computed the same way from XBRL, so a model that reports "long-term debt" alone counts as a miss. - **Risk factors don't score.** Auto-generated ground truth left `top_risk_factors` empty (there's no canonical source). So the F1 you see is effectively "cover + financials" only. Qualitative risk-factor eval is a v2.2 follow-up. - **Zero extraction errors** on all 5 filings — the section chunker + prompt wiring is stable. What's missing is prompt tuning against the specific failure modes above, which is where the next 15-20 F1 points live. ### v2.2 — a real diagnose → try → measure loop The per-field table above told me the money fields were the drag. I hypothesized that most of the misses came from the model ignoring "(In millions)" scale headers, and pushed three targeted fixes into v2.2: 1. **Prompt reinforcement.** Added a workflow section with four worked examples (Apple, Walmart, a mid-cap in thousands, a small filer in absolute dollars) and an explicit "before finalizing, check the number is physically plausible" step. See `src/extractors/prompts.py::SYSTEM_PROMPT_FILING`. 2. **Ground-truth builder expansion.** `build_filings_gt.py` FIN_MAP got new XBRL concept fallbacks so bank + insurance filers (JPM specifically) get real `total_debt` and `total_equity` via `Deposits`, `LongTermBorrowings`, and `StockholdersEquityIncludingPortionAttributableToNoncontrollingInterest`. 3. **Cover-field backfill.** Pulled `exchange` and `state_of_incorporation` into the sidecar from SEC's submissions feed so those cover fields are no longer support=0 in the eval. **What the numbers said back:** | Field | v2.1 F1 | v2.2 F1 | Verdict | |-----------------------------------|--------:|--------:|---------| | `financials.revenue` | 0.250 | **0.500** | ✅ prompt worked — precision 0.33→0.67 | | `financials.total_equity` | 0.000 | **0.286** | ✅ FIN_MAP worked partially | | `financials.operating_income` | 0.500 | **1.000** | ✅ prompt helped, small support | | `cover.form_type` | 0.600 | **0.800** | ✅ small win | | `financials.total_debt` | 0.000 | 0.000 | ❌ FIN_MAP change didn't reach the model side | | `cover.filing_date` | 0.333 | 0.000 | ❌ regression — model started returning report_date | | **Aggregate micro F1** | 0.560 | 0.560 | flat | The aggregate looking flat hides real per-field motion. Two of the three fixes partially worked; one didn't move the needle. What this measurement tells me about v2.3: - **Prompt reinforcement has a ceiling.** Two-pass extract-then-verify (first call extracts; second call is prompted with "here's what you just returned — verify the scale factor against the header text you were shown") is the next intervention, not more prompt paragraphs. - **`total_debt` on non-financial issuers is a definition problem, not a extraction problem.** The model returns "long-term debt" as printed; XBRL ground truth sums 4+ concepts. Either loosen the comparator or pin the definition in the prompt with "return LongTermDebt only, do not sum." - **`filing_date` regression** is the interesting one — the prompt changes around dates may have accidentally biased the model toward the report date. Worth an A/B test on just that field. The point of the harness is exactly this: a change ships, per-field numbers come back, and the next intervention is chosen from data rather than from a hunch. That's the loop I wanted to build. Reproduce locally: ```bash python scripts/run_eval.py \ --dataset evaluation/smoke_sroie_sample.jsonl \ --doc-type receipt \ --mode live \ --model gpt-5-nano \ --reasoning-effort minimal ``` ### Multi-model comparison (2026-07-05) Same 10 records, same prompts, same schemas — only the model changes. All runs use `reasoning_effort="minimal"`. Reports land under `evaluation/benchmarks//`. | Model | Micro F1 | Macro F1 | Doc-exact | Latency | Cost / doc | |---------------|-----------|-----------|-----------|----------|------------| | `gpt-5-nano` | **0.896** | 0.885 | 40 % | 5.1 s | **$0.0116** | | `gpt-5-mini` | 0.864 | 0.927 | 40 % | 6.1 s | $0.0127 | | `gpt-5` | 0.884 | **0.939** | 30 % | 5.4 s | $0.0118 | **Read the numbers:** - **`gpt-5-nano` is Pareto-optimal on this workload** — highest micro F1 at the lowest cost and lowest latency. Bigger tiers don't buy quality on high-support fields. - **`gpt-5` and `gpt-5-mini` lead on macro F1** — they're measurably better on the rarer fields (macro weights every field equally regardless of support). If your extraction schema is long-tailed, the ~7 % macro-F1 lift may be worth the small extra spend. - **Doc-exact stays 30-40 % across all three** — an artifact of a strict metric and a schema with many optional fields. Micro F1 tracks real quality here. - **Total benchmark spend: $0.36** to definitively answer "which model should ship in prod?" — this is the kind of question worth measuring instead of guessing at, and it's cheap enough to re-run whenever the prompt or schema moves. Reproduce: ```bash python scripts/run_multimodel_benchmark.py # or with a custom matrix: python scripts/run_multimodel_benchmark.py gpt-5-nano:minimal gpt-5-mini:minimal gpt-4o-mini ``` Next: real image PDFs from the SROIE test split for a stricter, OCR-inclusive number, then the SEC 10-K schema for the long-doc / dual-domain story. ## Streaming + async batch (v3) Two extra endpoints exist alongside the vanilla `POST /extract`: **`POST /extract/stream`** — same input as `/extract`, but returns `Content-Type: text/event-stream`. The response is a Server-Sent Events stream with `progress` events at each pipeline stage (`starting` → `loading` → `model_call` → `validated`), a `result` event carrying the full `ExtractionResult`, and a terminal `done` event. Errors surface as an in-band `error` event because the HTTP 200 has already been sent by then. Design note: the openai-python `.parse()` API doesn't yield partial validated Pydantic objects (structured outputs strict mode returns the final object only), so we ship progress events rather than partial JSON. That's the honest UX — the user sees "something is happening" and the answer arrives whole. **`POST /extract/batch`** — multipart with N files. Returns `202 Accepted` + `{job_id, status: "pending"}` immediately; extraction runs in FastAPI `BackgroundTasks`. Poll `GET /extract/batch/{job_id}` for the snapshot. Concurrency is capped globally at 5 via an `asyncio.Semaphore` sized to stay under OpenAI's per-org rate limits even with multiple in-flight jobs. Job store is in-memory (one dict + one asyncio lock per worker) — swap Redis in for horizontal scale without touching the API. ```bash # Batch: curl -X POST http://localhost:8000/extract/batch \ -F "doc_type=receipt" \ -F "files=@receipt1.png" \ -F "files=@receipt2.png" \ -F "files=@invoice.pdf" # -> {"job_id":"abc123...", "status":"pending", "progress":{"total":3,...}} curl http://localhost:8000/extract/batch/abc123 # -> {"status":"done","progress":{...},"items":[{...result...},...]} ``` ## Architecture ``` ┌──────────────┐ ┌───────────────┐ ┌────────────────────┐ │ React UI │─────►│ FastAPI │─────►│ Extractor │ │ UI │◄─────│ /extract │◄─────│ (GPT-5 nano+vision)│ └──────────────┘ └───────────────┘ └────────────────────┘ │ ┌─────────────────────────┼─────────────────────┐ ▼ ▼ ▼ ┌────────────────┐ ┌──────────────┐ ┌──────────────┐ │ Pydantic │ │ Confidence │ │ Cost / │ │ Schemas │ │ Scorer │ │ Latency Log │ └────────────────┘ └──────────────┘ └──────────────┘ │ ▼ ┌────────────────┐ │ Evaluation │ │ Harness │ │ (P/R/F1) │ └────────────────┘ ``` **Long-document handling (10-K path).** SEC filings are 50-250 pages and would cost ~$0.60/doc if we shipped the whole thing to the model on every call. So the filing path runs the document through a section chunker (`src/extractors/section_chunker.py`) that finds each `Item 1.`, `Item 1A.`, `Item 7.`, `Item 8.` heading via regex, deduplicates against the Table of Contents (keeping the last occurrence — the real section), and slices the plaintext into named chunks. Only the cover, Item 8 (financials), and Item 1A (risk factors) are stitched into the prompt — everything else is skipped. Result: ~30–40K prompt tokens per 10-K instead of ~150K, on the same gpt-5-nano @ minimal reasoning-effort configuration. ## Tech stack | Layer | Choice | Why | |-------|--------|-----| | LLM | OpenAI GPT-5 nano (default) + benchmarked vs GPT-5 mini + GPT-5 full | 400K context, vision, structured outputs, ~50x cheaper than GPT-4o | | Schema | Pydantic v2 | Runtime validation + JSON schema for OpenAI | | PDF text | pdfplumber, PyMuPDF | Fast, robust, handles most layouts | | PDF images | pdf2image + Pillow | For scanned/image-heavy PDFs → vision model | | Backend | FastAPI | Async, auto OpenAPI docs, batteries included | | Frontend | React + Vite + Tailwind + Motion + React Three Fiber | Editorial "Paper & Ink" aesthetic — 3D paper sheet in the hero, kinetic type, dark/light mode. No generic AI-SaaS look. | | Eval | rapidfuzz, scikit-learn | Fuzzy text matching + P/R/F1 | | Container | Docker (multi-stage) | Portable, reproducible | | Deploy | Hugging Face Spaces | Free, AI-community-recognized | | CI | GitHub Actions | Tests + lint on every push | ## Quick start ```bash # 1. Clone + install git clone https://github.com/adityapatel007-byte/structured-data-extractor.git cd structured-data-extractor python -m venv .venv && source .venv/bin/activate # or .venv\Scripts\activate on Windows pip install -r requirements.txt # 2. Set your OpenAI key cp .env.example .env # edit .env → paste your OPENAI_API_KEY # 3. Run the API uvicorn src.api.main:app --reload # 4. Run the UI — Paper & Ink React + Motion + R3F frontend # (in another terminal, from ui/) cd ui && npm install && npm run dev # then open http://localhost:5173 # 5. (Optional) Evaluate against the committed sample ground truth. # `selfcheck` mode uses a mock extractor to validate the eval pipeline (F1=1.0). python scripts/run_eval.py --dataset data/samples/sroie_sample.jsonl \ --doc-type receipt --mode selfcheck # 6. Benchmark a real model on your own ground-truth JSONL: python scripts/run_eval.py --dataset evaluation/ground_truth/sroie.jsonl \ --doc-type receipt --mode live --model gpt-5-nano ``` Reports (per-record CSV + summary JSON + resume-ready markdown) land in `evaluation/reports//`. ### Fine-tuning quick start (v4) The full pipeline lives in `scripts/`: ```bash # 1. Prep training data (produces _train.jsonl + _val.jsonl in OpenAI's chat-completions format). python scripts/prep_ft_dataset.py \ --input evaluation/smoke_sroie_sample.jsonl \ --doc-type receipt \ --out data/ft/sroie_smoke # 2. Launch the OpenAI fine-tuning job (uploads files, creates job, polls status). python scripts/launch_finetune.py \ --train data/ft/sroie_smoke_train.jsonl \ --val data/ft/sroie_smoke_val.jsonl \ --suffix receipts-2026 # Wait 10-30 min for OpenAI to train. # 3. Compare the fine-tuned model against the base gpt-5-nano on the same eval. python scripts/compare_finetune.py \ --ft-model ft:gpt-4o-mini-2024-07-18:you:receipts-2026:abc123 ``` The smoke datasets only have 5 examples each (SROIE + CORD) — OpenAI requires at least 10 training rows. For a real fine-tuning run, first pull the full SROIE / CORD training splits with `python scripts/prep_datasets.py all`, then point `prep_ft_dataset.py --input` at the processed dataset. ### 10-K (SEC filings) quick start ```bash # 1. Download the 5-issuer watchlist (Apple, JPM, ExxonMobil, Pfizer, Walmart). # Files land in data/raw/10k/ — plaintext, sidecar JSON, and XBRL companyfacts. python scripts/download_edgar.py # 2. Build a ground-truth JSONL from the sidecars + XBRL. python scripts/build_filings_gt.py # 3. Run the eval against a real model. python scripts/run_eval.py \ --dataset evaluation/smoke_filings_sample.jsonl \ --doc-type filing \ --mode live \ --model gpt-5-nano \ --reasoning-effort minimal ``` The filing extractor uses section-based chunking to keep per-doc cost around $0.05–0.15 instead of $0.60+ at whole-document context. ## Project structure ``` 04-structured-data-extraction/ ├── src/ │ ├── schemas/ # Pydantic schemas per doc type │ ├── extractors/ # LLM extraction logic │ ├── api/ # FastAPI backend │ └── utils/ # cost tracking, logging, config ├── ui/ # React + Motion + R3F frontend (Paper & Ink) │ ├── src/components/ # Hero, PaperScene (3D), Dropzone, ResultsPanel, ... │ ├── src/styles/ # theme.css (dark/light tokens) + globals.css │ └── package.json ├── tests/ │ ├── unit/ │ └── integration/ ├── data/ │ ├── raw/ # downloaded datasets (gitignored) │ ├── processed/ # normalized ground truth │ └── samples/ # small demo files (committed) ├── evaluation/ │ ├── ground_truth/ # labeled gold data │ └── reports/ # eval run outputs ├── docker/ # Dockerfile + compose ├── .github/workflows/ # CI ├── requirements.txt ├── .env.example └── README.md ``` ## Roadmap - [x] **v1 — Invoices & Receipts pipeline + multi-model benchmark** - [x] **v2 — SEC 10-K pipeline** (schema + section chunker + EDGAR downloader + v2.2 diagnose-loop, micro F1 0.56, $0.06/doc, ready for two-pass verify in v2.3) - [x] **v3 — Streaming + async batch API** (SSE progress events; in-memory job store with global asyncio semaphore capping concurrency at 5; 12 new tests) - [x] **v4 — Fine-tuning experiment** (prep + launcher + comparison eval scripts; 9 new tests; comparison table in `evaluation/finetuning/` once user triggers the ft training run on OpenAI) ## License MIT