README.md

metadata

title: Multimodal Document Extraction Pipeline
emoji: 📄
colorFrom: blue
colorTo: blue
sdk: gradio
sdk_version: 5.9.1
app_file: app.py
pinned: false
license: mit
short_description: GPT-4o structured JSON extraction with validation layer
python_version: '3.10'

📄 Multimodal Document Extraction Pipeline

GPT-4o vision extracts structured JSON from any document image. Schema-based validation + cross-field consistency checks + automatic human review routing for low-confidence outputs.

Architecture

Document Image
    ↓
GPT-4o Vision Extraction (schema-driven, json_object mode, per-field confidence)
    ↓
JSON Schema Validation (type coercion, cross-field consistency, required fields)
    ↓
Confidence Scoring (extraction confidence - validation penalty)
    ↓
Human Review Queue (auto-approve if ≥70% AND no errors, else queue)
    ↓
Structured JSON Output

Supported Document Types

Type	Key Fields
Invoice	Vendor, invoice #, line items, totals, tax, payment terms
ID Document	Name, DOB, document #, expiry, issuing authority
Medical Record	Patient info, diagnoses, medications, follow-up
Business Card	Name, title, company, email, phone, website
Form	All form fields as key-value pairs, signature

Key Engineering Decisions

Schema-first extraction: Providing GPT-4o the exact expected JSON schema reduces hallucination and forces consistent field naming. The model returns null for fields it cannot read (vs. omitting them), which distinguishes "field not found" from "field not present."

Per-field confidence: The extraction prompt explicitly requests confidence scores (0-1) per field. These are used downstream for validation penalty calculation.

Cross-field consistency: Invoice totals are cross-validated: subtotal + tax ≈ total_amount within 2% tolerance. Violations trigger a confidence penalty and human review.

Validation penalty: final_confidence = extraction_confidence - validation_penalty where penalty accumulates per error type (type mismatch: -0.10, consistency error: -0.15, missing required: -0.15).

Vision vs OCR Baseline

Capability	GPT-4o	Tesseract + Regex
Tables / line items	✅ Full structure	❌ Loses structure
Handwritten text	✅ Good	❌ Very poor
Multi-language	✅ Built-in	⚠️ Needs config
Speed	~3 sec	~0.3 sec
Cost	~$0.02/doc	Free

Running Locally

git clone https://github.com/data-geek-astronomy/multimodal-doc-extraction
cd multimodal-doc-extraction
pip install -r requirements.txt
OPENAI_API_KEY=sk-... python app.py

License

MIT