Spaces:

aditya0103
/

structured-data-extractor

Sleeping

App Files Files Community

structured-data-extractor / README.md

aditya0103

v4: fine-tuning pipeline complete + bug fixes

d95323c 19 days ago

preview code

Raw

History Blame Contribute Delete

21.6 kB

metadata

title: Structured Data Extractor
emoji: 📄
colorFrom: yellow
colorTo: gray
sdk: docker
app_port: 7860
pinned: false
license: mit
short_description: Receipts + invoices to validated JSON, GPT-5 nano

Structured Data Extraction Service

Multi-domain document extraction — turn invoices, receipts, and SEC filings into schema-validated JSON with confidence scoring, multi-model benchmarking, and quantified accuracy.

What this does

Upload any invoice, receipt, or SEC filing (10-K / 10-Q). Get back clean, validated JSON matching a Pydantic schema — plus a per-field confidence score, a cost breakdown, and latency metrics.

PDF / Image / Scan  ─►  Router  ─►  {Invoice | Receipt | Filing} Pipeline  ─►  JSON + Confidence
                                        │
                                        └─►  Evaluation Harness  ─►  Precision / Recall / F1

Why this project exists

Enterprise doc extraction is one of the highest-demand LLM use cases in 2026. This repo demonstrates the production skills that hiring managers screen for:

Schema-driven extraction with OpenAI structured outputs + Pydantic validation
Vision-language handling for scanned/image PDFs (GPT-5 nano vision)
Long-document handling for 10-K filings — regex section chunker slices Items 1A + 8 out of ~150K-token documents to hit $0.06/doc (would've been ~$0.60/doc whole-doc)
Streaming + async batch API — POST /extract/stream emits Server-Sent Events for progress + result; POST /extract/batch fans up to 5 concurrent extractions across N documents with per-item status tracking
Fine-tuning experiment — full pipeline (data prep → OpenAI job launch → side-by-side eval) built and reproducible; sample-scale run shipped, real-scale run gated on pulling the SROIE/CORD training splits
Multi-model benchmarking — empirically compared gpt-5-nano vs gpt-5-mini vs gpt-5 on the same 10-record eval; nano is Pareto-optimal (micro F1 0.896 at $0.012/doc)
Evaluation harness with precision / recall / F1 on public ground truth (SROIE, CORD)
Cost + latency observability — every extraction logs tokens and $
Full-stack: FastAPI backend, React + Motion + R3F UI, Docker, GitHub Actions CI
CI/CD with GitHub Actions running tests + lint on every push

Live demo

The full stack is on Hugging Face Spaces:

huggingface.co/spaces/aditya0103/structured-data-extractor — always-on, free tier

Locally: docker compose up --build after cloning; hit http://localhost:5173.

Deploy your own to HF Spaces

Create a new HF Space → Docker SDK, blank template.
Add OPENAI_API_KEY under Settings → Repository secrets.
Point the Space at this repo (or push a fork). HF reads the YAML frontmatter at the top of this README (sdk: docker, app_port: 7860), builds the root Dockerfile, and exposes it on your Space URL. First build ~5-8 min; every redeploy ~2-3 min thanks to layer caching.

Quantified results

Live evaluation on gpt-5-nano with reasoning_effort="minimal". 10 receipt records derived from public SROIE + CORD ground truth. Reports (per-record CSV, JSON summary, markdown) land in evaluation/reports/<timestamp>/ after each run.

Domain	Dataset (n=)	Micro F1	Macro F1	Doc Exact	Cost / doc	Mean Latency
Receipts	SROIE (5)	0.938	1.000	0.20	$0.012	6.3 s
Receipts	CORD (5)	0.914	0.839	0.80	$0.012	8.2 s
Filings	SEC 10-K (n=5)	0.560	0.584	0.00	$0.063	6.4 s

Read the numbers:

Micro F1 ≈ 0.92 across both datasets — the model gets ~92% of individual fields correct on ground-truth-derived text.
Doc-level exact match is stricter (100% of fields right on one doc) and swings by dataset: CORD receipts (short, simple line items) hit 0.80; SROIE (freer-form Malaysian/Singaporean receipts with more optional fields) hits 0.20 — a single missing field kills the metric on those.
$0.012 / doc is the reasoning-tokens-included cost at reasoning_effort=minimal. Default (non-minimal) reasoning was $0.042 / doc, 30 s / doc — the minimal flag is a ~3.5× cost cut and ~4× latency cut with no measured quality loss on this schema.
Total spend for a full run: ~$0.12 — cheap enough to re-run on every significant prompt/schema change.

Why the 10-K F1 is lower than the receipt F1 — this is a harder task and the number reflects that honestly:

Unit-of-measure normalization. 10-K income statements are printed "in millions" or "in thousands." The extractor has to multiply back to absolute dollars. When it misses the header, a $391B revenue lands as $391K in the output — a 1,000,000× miss that reads as False on the money comparator.
Multi-year column selection. Every 10-K shows the most recent fiscal year alongside 1-2 prior years side-by-side. Picking the wrong column produces a plausible-but-wrong number.
Debt aggregation. total_debt = short-term + long-term borrowings, which the model must sum. Ground truth is computed the same way from XBRL, so a model that reports "long-term debt" alone counts as a miss.
Risk factors don't score. Auto-generated ground truth left top_risk_factors empty (there's no canonical source). So the F1 you see is effectively "cover + financials" only. Qualitative risk-factor eval is a v2.2 follow-up.
Zero extraction errors on all 5 filings — the section chunker + prompt wiring is stable. What's missing is prompt tuning against the specific failure modes above, which is where the next 15-20 F1 points live.

v2.2 — a real diagnose → try → measure loop

The per-field table above told me the money fields were the drag. I hypothesized that most of the misses came from the model ignoring "(In millions)" scale headers, and pushed three targeted fixes into v2.2:

Prompt reinforcement. Added a workflow section with four worked examples (Apple, Walmart, a mid-cap in thousands, a small filer in absolute dollars) and an explicit "before finalizing, check the number is physically plausible" step. See src/extractors/prompts.py::SYSTEM_PROMPT_FILING.
Ground-truth builder expansion. build_filings_gt.py FIN_MAP got new XBRL concept fallbacks so bank + insurance filers (JPM specifically) get real total_debt and total_equity via Deposits, LongTermBorrowings, and StockholdersEquityIncludingPortionAttributableToNoncontrollingInterest.
Cover-field backfill. Pulled exchange and state_of_incorporation into the sidecar from SEC's submissions feed so those cover fields are no longer support=0 in the eval.

What the numbers said back:

Field	v2.1 F1	v2.2 F1	Verdict
`financials.revenue`	0.250	0.500	✅ prompt worked — precision 0.33→0.67
`financials.total_equity`	0.000	0.286	✅ FIN_MAP worked partially
`financials.operating_income`	0.500	1.000	✅ prompt helped, small support
`cover.form_type`	0.600	0.800	✅ small win
`financials.total_debt`	0.000	0.000	❌ FIN_MAP change didn't reach the model side
`cover.filing_date`	0.333	0.000	❌ regression — model started returning report_date
Aggregate micro F1	0.560	0.560	flat

The aggregate looking flat hides real per-field motion. Two of the three fixes partially worked; one didn't move the needle. What this measurement tells me about v2.3:

Prompt reinforcement has a ceiling. Two-pass extract-then-verify (first call extracts; second call is prompted with "here's what you just returned — verify the scale factor against the header text you were shown") is the next intervention, not more prompt paragraphs.
total_debt on non-financial issuers is a definition problem, not a extraction problem. The model returns "long-term debt" as printed; XBRL ground truth sums 4+ concepts. Either loosen the comparator or pin the definition in the prompt with "return LongTermDebt only, do not sum."
filing_date regression is the interesting one — the prompt changes around dates may have accidentally biased the model toward the report date. Worth an A/B test on just that field.

The point of the harness is exactly this: a change ships, per-field numbers come back, and the next intervention is chosen from data rather than from a hunch. That's the loop I wanted to build.

Reproduce locally:

python scripts/run_eval.py \
  --dataset evaluation/smoke_sroie_sample.jsonl \
  --doc-type receipt \
  --mode live \
  --model gpt-5-nano \
  --reasoning-effort minimal

Multi-model comparison (2026-07-05)

Same 10 records, same prompts, same schemas — only the model changes. All runs use reasoning_effort="minimal". Reports land under evaluation/benchmarks/<timestamp>/.

Model	Micro F1	Macro F1	Doc-exact	Latency	Cost / doc
`gpt-5-nano`	0.896	0.885	40 %	5.1 s	$0.0116
`gpt-5-mini`	0.864	0.927	40 %	6.1 s	$0.0127
`gpt-5`	0.884	0.939	30 %	5.4 s	$0.0118

Read the numbers:

gpt-5-nano is Pareto-optimal on this workload — highest micro F1 at the lowest cost and lowest latency. Bigger tiers don't buy quality on high-support fields.
gpt-5 and gpt-5-mini lead on macro F1 — they're measurably better on the rarer fields (macro weights every field equally regardless of support). If your extraction schema is long-tailed, the ~7 % macro-F1 lift may be worth the small extra spend.
Doc-exact stays 30-40 % across all three — an artifact of a strict metric and a schema with many optional fields. Micro F1 tracks real quality here.
Total benchmark spend: $0.36 to definitively answer "which model should ship in prod?" — this is the kind of question worth measuring instead of guessing at, and it's cheap enough to re-run whenever the prompt or schema moves.

Reproduce:

python scripts/run_multimodel_benchmark.py
# or with a custom matrix:
python scripts/run_multimodel_benchmark.py gpt-5-nano:minimal gpt-5-mini:minimal gpt-4o-mini

Next: real image PDFs from the SROIE test split for a stricter, OCR-inclusive number, then the SEC 10-K schema for the long-doc / dual-domain story.

Streaming + async batch (v3)

Two extra endpoints exist alongside the vanilla POST /extract:

POST /extract/stream — same input as /extract, but returns Content-Type: text/event-stream. The response is a Server-Sent Events stream with progress events at each pipeline stage (starting → loading → model_call → validated), a result event carrying the full ExtractionResult, and a terminal done event. Errors surface as an in-band error event because the HTTP 200 has already been sent by then. Design note: the openai-python .parse() API doesn't yield partial validated Pydantic objects (structured outputs strict mode returns the final object only), so we ship progress events rather than partial JSON. That's the honest UX — the user sees "something is happening" and the answer arrives whole.

POST /extract/batch — multipart with N files. Returns 202 Accepted

{job_id, status: "pending"} immediately; extraction runs in FastAPI BackgroundTasks. Poll GET /extract/batch/{job_id} for the snapshot. Concurrency is capped globally at 5 via an asyncio.Semaphore sized to stay under OpenAI's per-org rate limits even with multiple in-flight jobs. Job store is in-memory (one dict + one asyncio lock per worker) — swap Redis in for horizontal scale without touching the API.

# Batch:
curl -X POST http://localhost:8000/extract/batch \
  -F "doc_type=receipt" \
  -F "files=@receipt1.png" \
  -F "files=@receipt2.png" \
  -F "files=@invoice.pdf"
# -> {"job_id":"abc123...", "status":"pending", "progress":{"total":3,...}}

curl http://localhost:8000/extract/batch/abc123
# -> {"status":"done","progress":{...},"items":[{...result...},...]}

Architecture

┌──────────────┐      ┌───────────────┐      ┌────────────────────┐
│  React UI    │─────►│    FastAPI    │─────►│   Extractor        │
│      UI      │◄─────│   /extract    │◄─────│  (GPT-5 nano+vision)│
└──────────────┘      └───────────────┘      └────────────────────┘
                                                       │
                             ┌─────────────────────────┼─────────────────────┐
                             ▼                         ▼                     ▼
                    ┌────────────────┐        ┌──────────────┐      ┌──────────────┐
                    │ Pydantic       │        │ Confidence   │      │  Cost /      │
                    │ Schemas        │        │ Scorer       │      │  Latency Log │
                    └────────────────┘        └──────────────┘      └──────────────┘
                             │
                             ▼
                    ┌────────────────┐
                    │  Evaluation    │
                    │  Harness       │
                    │  (P/R/F1)      │
                    └────────────────┘

Long-document handling (10-K path). SEC filings are 50-250 pages and would cost ~$0.60/doc if we shipped the whole thing to the model on every call. So the filing path runs the document through a section chunker (src/extractors/section_chunker.py) that finds each Item 1., Item 1A., Item 7., Item 8. heading via regex, deduplicates against the Table of Contents (keeping the last occurrence — the real section), and slices the plaintext into named chunks. Only the cover, Item 8 (financials), and Item 1A (risk factors) are stitched into the prompt — everything else is skipped. Result: ~30–40K prompt tokens per 10-K instead of ~150K, on the same gpt-5-nano @ minimal reasoning-effort configuration.

Tech stack

Layer	Choice	Why
LLM	OpenAI GPT-5 nano (default) + benchmarked vs GPT-5 mini + GPT-5 full	400K context, vision, structured outputs, ~50x cheaper than GPT-4o
Schema	Pydantic v2	Runtime validation + JSON schema for OpenAI
PDF text	pdfplumber, PyMuPDF	Fast, robust, handles most layouts
PDF images	pdf2image + Pillow	For scanned/image-heavy PDFs → vision model
Backend	FastAPI	Async, auto OpenAPI docs, batteries included
Frontend	React + Vite + Tailwind + Motion + React Three Fiber	Editorial "Paper & Ink" aesthetic — 3D paper sheet in the hero, kinetic type, dark/light mode. No generic AI-SaaS look.
Eval	rapidfuzz, scikit-learn	Fuzzy text matching + P/R/F1
Container	Docker (multi-stage)	Portable, reproducible
Deploy	Hugging Face Spaces	Free, AI-community-recognized
CI	GitHub Actions	Tests + lint on every push

Quick start

# 1. Clone + install
git clone https://github.com/adityapatel007-byte/structured-data-extractor.git
cd structured-data-extractor
python -m venv .venv && source .venv/bin/activate    # or .venv\Scripts\activate on Windows
pip install -r requirements.txt

# 2. Set your OpenAI key
cp .env.example .env
# edit .env → paste your OPENAI_API_KEY

# 3. Run the API
uvicorn src.api.main:app --reload

# 4. Run the UI — Paper & Ink React + Motion + R3F frontend
#    (in another terminal, from ui/)
cd ui && npm install && npm run dev
# then open http://localhost:5173

# 5. (Optional) Evaluate against the committed sample ground truth.
#    `selfcheck` mode uses a mock extractor to validate the eval pipeline (F1=1.0).
python scripts/run_eval.py --dataset data/samples/sroie_sample.jsonl \
    --doc-type receipt --mode selfcheck

# 6. Benchmark a real model on your own ground-truth JSONL:
python scripts/run_eval.py --dataset evaluation/ground_truth/sroie.jsonl \
    --doc-type receipt --mode live --model gpt-5-nano

Reports (per-record CSV + summary JSON + resume-ready markdown) land in evaluation/reports/<UTC-timestamp>/.

Fine-tuning quick start (v4)

The full pipeline lives in scripts/:

# 1. Prep training data (produces <name>_train.jsonl + <name>_val.jsonl in OpenAI's chat-completions format).
python scripts/prep_ft_dataset.py \
    --input evaluation/smoke_sroie_sample.jsonl \
    --doc-type receipt \
    --out data/ft/sroie_smoke

# 2. Launch the OpenAI fine-tuning job (uploads files, creates job, polls status).
python scripts/launch_finetune.py \
    --train data/ft/sroie_smoke_train.jsonl \
    --val   data/ft/sroie_smoke_val.jsonl \
    --suffix receipts-2026
# Wait 10-30 min for OpenAI to train.

# 3. Compare the fine-tuned model against the base gpt-5-nano on the same eval.
python scripts/compare_finetune.py \
    --ft-model ft:gpt-4o-mini-2024-07-18:you:receipts-2026:abc123

The smoke datasets only have 5 examples each (SROIE + CORD) — OpenAI requires at least 10 training rows. For a real fine-tuning run, first pull the full SROIE / CORD training splits with python scripts/prep_datasets.py all, then point prep_ft_dataset.py --input at the processed dataset.

10-K (SEC filings) quick start

# 1. Download the 5-issuer watchlist (Apple, JPM, ExxonMobil, Pfizer, Walmart).
#    Files land in data/raw/10k/ — plaintext, sidecar JSON, and XBRL companyfacts.
python scripts/download_edgar.py

# 2. Build a ground-truth JSONL from the sidecars + XBRL.
python scripts/build_filings_gt.py

# 3. Run the eval against a real model.
python scripts/run_eval.py \
    --dataset evaluation/smoke_filings_sample.jsonl \
    --doc-type filing \
    --mode live \
    --model gpt-5-nano \
    --reasoning-effort minimal

The filing extractor uses section-based chunking to keep per-doc cost around $0.05–0.15 instead of $0.60+ at whole-document context.

Project structure

04-structured-data-extraction/
├── src/
│   ├── schemas/         # Pydantic schemas per doc type
│   ├── extractors/      # LLM extraction logic
│   ├── api/             # FastAPI backend
│   └── utils/           # cost tracking, logging, config
├── ui/                  # React + Motion + R3F frontend (Paper & Ink)
│   ├── src/components/  # Hero, PaperScene (3D), Dropzone, ResultsPanel, ...
│   ├── src/styles/      # theme.css (dark/light tokens) + globals.css
│   └── package.json
├── tests/
│   ├── unit/
│   └── integration/
├── data/
│   ├── raw/             # downloaded datasets (gitignored)
│   ├── processed/       # normalized ground truth
│   └── samples/         # small demo files (committed)
├── evaluation/
│   ├── ground_truth/    # labeled gold data
│   └── reports/         # eval run outputs
├── docker/              # Dockerfile + compose
├── .github/workflows/   # CI
├── requirements.txt
├── .env.example
└── README.md

Roadmap

v1 — Invoices & Receipts pipeline + multi-model benchmark
v2 — SEC 10-K pipeline (schema + section chunker + EDGAR downloader + v2.2 diagnose-loop, micro F1 0.56, $0.06/doc, ready for two-pass verify in v2.3)
v3 — Streaming + async batch API (SSE progress events; in-memory job store with global asyncio semaphore capping concurrency at 5; 12 new tests)
v4 — Fine-tuning experiment (prep + launcher + comparison eval scripts; 9 new tests; comparison table in evaluation/finetuning/ once user triggers the ft training run on OpenAI)

License

MIT