| --- |
| title: Multimodal Document Extraction Pipeline |
| emoji: 📄 |
| colorFrom: blue |
| colorTo: blue |
| sdk: gradio |
| sdk_version: 5.9.1 |
| app_file: app.py |
| pinned: false |
| license: mit |
| short_description: GPT-4o structured JSON extraction with validation layer |
| python_version: "3.10" |
| --- |
| |
| # 📄 Multimodal Document Extraction Pipeline |
|
|
| > GPT-4o vision extracts structured JSON from any document image. Schema-based validation + cross-field consistency checks + automatic human review routing for low-confidence outputs. |
|
|
| ## Architecture |
|
|
| ``` |
| Document Image |
| ↓ |
| GPT-4o Vision Extraction (schema-driven, json_object mode, per-field confidence) |
| ↓ |
| JSON Schema Validation (type coercion, cross-field consistency, required fields) |
| ↓ |
| Confidence Scoring (extraction confidence - validation penalty) |
| ↓ |
| Human Review Queue (auto-approve if ≥70% AND no errors, else queue) |
| ↓ |
| Structured JSON Output |
| ``` |
|
|
| ## Supported Document Types |
|
|
| | Type | Key Fields | |
| |---|---| |
| | Invoice | Vendor, invoice #, line items, totals, tax, payment terms | |
| | ID Document | Name, DOB, document #, expiry, issuing authority | |
| | Medical Record | Patient info, diagnoses, medications, follow-up | |
| | Business Card | Name, title, company, email, phone, website | |
| | Form | All form fields as key-value pairs, signature | |
|
|
| ## Key Engineering Decisions |
|
|
| **Schema-first extraction**: Providing GPT-4o the exact expected JSON schema reduces hallucination and forces consistent field naming. The model returns `null` for fields it cannot read (vs. omitting them), which distinguishes "field not found" from "field not present." |
|
|
| **Per-field confidence**: The extraction prompt explicitly requests confidence scores (0-1) per field. These are used downstream for validation penalty calculation. |
|
|
| **Cross-field consistency**: Invoice totals are cross-validated: `subtotal + tax ≈ total_amount` within 2% tolerance. Violations trigger a confidence penalty and human review. |
|
|
| **Validation penalty**: `final_confidence = extraction_confidence - validation_penalty` where penalty accumulates per error type (type mismatch: -0.10, consistency error: -0.15, missing required: -0.15). |
|
|
| ## Vision vs OCR Baseline |
|
|
| | Capability | GPT-4o | Tesseract + Regex | |
| |---|---|---| |
| | Tables / line items | ✅ Full structure | ❌ Loses structure | |
| | Handwritten text | ✅ Good | ❌ Very poor | |
| | Multi-language | ✅ Built-in | ⚠️ Needs config | |
| | Speed | ~3 sec | ~0.3 sec | |
| | Cost | ~$0.02/doc | Free | |
|
|
| ## Running Locally |
|
|
| ```bash |
| git clone https://github.com/data-geek-astronomy/multimodal-doc-extraction |
| cd multimodal-doc-extraction |
| pip install -r requirements.txt |
| OPENAI_API_KEY=sk-... python app.py |
| ``` |
|
|
| ## License |
|
|
| MIT |
|
|
