Spaces:
Sleeping
DocAgent — Project Overview, Research, Design & How It Works
Project: Document Processing AI Agent (chat-based, 40/60 UI) Org: Monkhub Innovations One-line: Upload any document → it parses, classifies, extracts structured data, summarizes, answers grounded questions, flags anomalies, and exports clean data.
This single document brings together what we researched, what we chose and why,
the comparison, the feature set, and exactly how the agent works end-to-end.
For the full deep-dive research see RESEARCH.md; for the layered
architecture record see ARCHITECTURE.md; to deploy see
DEPLOY.md.
1. Research — what we evaluated
We needed a foundation for a custom, chat-based, production document-processing agent with a 40/60 split UI and a free-LLM-first model strategy. We evaluated the five leading systems, one per relevant archetype:
| Archetype | System | Why it represents the category |
|---|---|---|
| Full open-source app | RAGFlow | A complete chat-over-documents product with deep understanding, citations, and an agent layer — the closest existing thing to our target. |
| LLM app platform | Dify | Reference for clean platform architecture, visual orchestration, observability, polished UI/UX. |
| Extraction toolkit | Docling (IBM / LF AI & Data) | Best-in-class open document parsing — layout, tables, OCR, VLM. The extraction backbone. |
| Workflow automation | n8n | The "glue" archetype: trigger → OCR → LLM extract → validate → route, assembled visually. |
| Agentic framework | LlamaIndex ADW | Reference for stateful, multi-step document agents and schema-based structured extraction. |
Method: sequential, single-researcher web research; each system and source read
in full. Full notes and sources in RESEARCH.md.
2. Comparison table & scoring
Rubric (1–5; 5 = best) — scored for our specific goal, not in the abstract:
| Code | Parameter |
|---|---|
| A | Feature completeness (OCR, layout, tables, KV, entities, classify, summarize, Q&A, export) |
| B | Architecture quality & extensibility |
| C | Ease of customisation to our UI/flows |
| D | Production-readiness (auth, scaling, observability, deploy) |
| E | UI/UX quality for document work |
| F | Cost / open-source friendliness (license, no per-page cost, self-host) |
| G | Community & documentation |
| System | A | B | C | D | E | F | G | Total /35 |
|---|---|---|---|---|---|---|---|---|
| Dify | 3 | 5 | 5 | 5 | 5 | 4 | 5 | 32 |
| RAGFlow | 5 | 4 | 3 | 4 | 4 | 5 | 5 | 30 |
| Docling | 5 | 5 | 4 | 4 | 1 | 5 | 5 | 29 |
| LlamaIndex ADW | 4 | 5 | 4 | 4 | 2 | 3 | 5 | 27 |
| n8n workflows | 3 | 3 | 5 | 4 | 2 | 3 | 5 | 25 |
How to read it. Dify wins on raw total, but its lead comes from platform polish, not document intelligence (its weakest axis, A=3). RAGFlow and Docling lead on the axis this project lives or dies on — document depth (A=5). The honest conclusion is that no single system is simultaneously the deepest extractor, the cleanest platform, and a ready-made 40/60 chat UI. So we composed.
3. What we chose & why
Decision: a composed custom stack — not a fork of any one system.
| Borrowed from | What we took |
|---|---|
| RAGFlow | The end-to-end pipeline blueprint (deep-understanding-at-ingestion → grounded, citable answers) and citation UX. |
| Docling | The actual extraction engine — MIT, local, layout + TableFormer tables + OCR, zero per-page cost. |
| LlamaIndex ADW | The agent loop pattern: parse → maintain state → retrieve → reason → surface for validation. |
| Dify | Separation-of-concerns discipline — thin routes, services, provider abstraction. |
| Our own | FastAPI backend + Next.js/React 40/60 UI tailored to the brief. |
Why composed instead of forking RAGFlow (the top product): its infra (MinIO + Elasticsearch + MySQL + Redis) and opinionated UI are too heavy to host our bespoke 40/60 experience, and the footprint (≥4 cores / ≥16 GB / ≥50 GB) is unjustified at our stage. Composing gives us RAGFlow's proven flow, Docling's best-in-class and free/local extraction, LlamaIndex's stateful agent patterns, and Dify's architectural cleanliness — with none of the licensing, cost, or lock-in downsides.
Chosen stack at a glance
| Layer | Choice | Why |
|---|---|---|
| Extraction | Docling (layout, TableFormer, EasyOCR) | Best open accuracy; MIT; local; $0 per page; offline-capable. |
| LLM (default) | Gemini Flash | Generous free tier; ~1M-token context for long docs; native vision for scans. |
| LLM (fallback) | GPT-4o-mini | Cheap, reliable tool-calling; switchable per-request behind a provider interface. |
| Backend | FastAPI (async, Pydantic v2) | Clean route→service layering; SSE streaming; production-ready. |
| Frontend | Next.js (App Router) + React + TS + Tailwind | 40/60 layout, document viewer with bounding-box overlays, streaming chat. |
| Vector store | Pure-Python numpy cosine (.npz/.json on disk) |
Dependency-light, no native build/service; chunks carry page+bbox for citations. Swappable for pgvector/Chroma. |
| Metadata store | SQLite (JSON blobs) | Zero external services; access funnelled through one module → easy Postgres swap. |
Local-first & ~zero marginal cost: the whole app runs with only a free Gemini key.
Nothing imports a vendor SDK except llm/*, so providers swap by env, per request, or
from the UI.
4. Features — the 7 core functions
| # | Function | What it does | Agent tool |
|---|---|---|---|
| 1 | Ingest | Parse PDF, DOCX, PPTX, XLSX, HTML, images with Docling (layout, tables, OCR) → unified Markdown + page images + provenance-carrying chunks. | (pipeline) |
| 2 | Extract | Adaptive, template-free structured data: fields / records / tables / entities, each with a confidence score and best-effort citation. | get_extracted_data |
| 3 | Classify | Zero-shot document-type detection with confidence + rationale. | classify_document |
| 4 | Summarize | Map-reduce summary (TL;DR + key points) over chunks. | summarize_document |
| 5 | Q&A | Conversational, agentic-RAG answers grounded with citations you can click to highlight on the page. | query_document |
| 6 | Flag anomalies | Missing fields, low-confidence values, generic arithmetic reconciliation, multi-record sanity, LLM consistency review. | flag_anomalies |
| 7 | Export | Serialize extracted data to JSON / CSV / Excel (records become a table). | (API) |
The headline feature: adaptive, domain-agnostic extraction
Extraction does not rely on per-domain templates. For any document — any layout, any label convention, known type or not — it produces a stable, consistently-keyed shape by:
- Layout detection — the model decides whether the doc is one entity
(
single_record, e.g. an invoice), many similar entities (multi_record, e.g. an employee roster or transaction list), or a header + repeated sub-list (mixed, e.g. invoice + line items). - Schema induction — fields are derived from the actual content (a known type only suggests fields; the model may add/drop to fit reality).
- Canonical keys — inconsistent source labels ("Staff Number" / "Personnel ID" / "ID")
are normalised to one
snake_casekey, identical across every record. - Grounding — values come only from the document body (never file metadata); dates and amounts are normalised (ISO dates, currency); citations are attached by matching values back to provenance-carrying chunks.
5. How it works — architecture
┌──────────────────────────────────────────────────────────────┐
│ FRONTEND (Next.js + React + TypeScript + Tailwind) │
│ 40% Chat (streaming agent) │ 60% Workspace │
│ │ Tabs: Viewer · Fields · │
│ │ Classify · Summary · Export │
│ │ Viewer w/ bounding-box cites │
└─────┬────────────────────────────────────────────────────────┘
│ REST + SSE (token streaming)
┌─────▼────────────────────────────────────────────────────────┐
│ BACKEND (FastAPI, async) │
│ api/ documents · chat(SSE) · actions · meta(health) │
│ agent/ orchestrator (tool-calling loop) + prompts │
│ llm/ provider abstraction: Gemini(default) ⇄ OpenAI │
│ services/ ingestion(Docling) · extraction · classification │
│ · summary · qa(RAG) · anomaly · export · │
│ vectorstore · storage · pipeline │
│ schemas/ Pydantic contracts (documents, optional hints) │
└─────┬───────────────────────────┬────────────────────────────┘
│ │
┌─────▼─────┐ ┌─────────▼──────────┐
│ Docling │ │ Storage (/data) │
│ (local │ │ • files on disk │
│ extract) │ │ • SQLite metadata │
│ models │ │ • numpy vectors │
│ baked in │ │ (.npz/.json) │
└───────────┘ │ • rendered pages │
└────────────────────┘
Discipline (borrowed from Dify): routes are thin → call services; the agent
orchestrates services as tools; LLM access only through llm/; storage only through
services/storage + services/vectorstore; schemas are shared.
6. Working — the two flows
Flow A — Ingestion pipeline (runs once per upload, in the background)
Implemented in services/pipeline.py. State is written back at every milestone so the
UI can poll status and render results progressively.
upload → [Stage 1: INGEST]──fail?──► status=failed (hard fail; only this stage can)
│ Docling parse → Markdown + page images + chunks(page,bbox)
▼
[index] embed chunks → numpy vector store ┐
[classify] LLM zero-shot → type + confidence │ Stages 2–6 are
[extract] adaptive induction → fields/records/... │ best-effort: a
[summarize] map-reduce → TL;DR + key points │ failure (e.g. no
[anomalies] rule + LLM checks (needs extraction) │ LLM key) degrades
▼ ┘ gracefully
status=ready (viewer always works, even if AI steps were skipped)
Key robustness property: only ingestion can hard-fail a document. Everything else is
wrapped so a missing API key or a flaky step still yields a ready document with a
working viewer and a note about what was skipped.
Flow B — Chat agent (per user message)
Implemented in agent/orchestrator.py — a bounded tool-calling loop, then a
streamed final answer.
user message + history + doc context
│
▼
Phase 1: TOOL ROUNDS (up to MAX_TOOL_ROUNDS = 4, non-streamed reasoning)
llm.complete_tools(messages, TOOL_SPECS)
│
├── model returns tool_calls? ──yes──► run each tool ─► append results ─► loop
│ tools: query_document · classify_document ·
│ get_extracted_data · summarize_document · flag_anomalies
│
└── no tool_calls (ready to answer) ─► exit loop
▼
Phase 2: FINAL ANSWER (streamed token-by-token over SSE, temperature 0.4)
│
▼
client renders answer + citation chips → click a chip → highlight the
page region (numpy store returned page+bbox with each retrieved chunk)
query_document runs agentic RAG: retrieve the most similar chunks, answer from
them, and return grounding passages whose page+bbox the UI turns into clickable
highlights. The orchestrator collects citations in its ToolContext so the API can
emit them alongside the streamed answer.
7. End-to-end walkthrough (a user's view)
- Drag a document into the left rail (or click New document).
- The pipeline runs: pages render in the viewer first, then classification, fields, summary, and anomaly flags stream into the workspace tabs as each stage finishes.
- Chat on the left — e.g. "Summarize this", "What's the total?", "Any missing fields?" The agent calls the right tools, then streams a grounded answer. Click any citation chip to highlight its exact source on the page.
- Review & edit low-confidence fields inline (human-in-the-loop) before exporting.
- Export from the Export tab as JSON / CSV / Excel.
8. Why this design holds up
- Document depth where it matters — Docling gives best-in-class layout/table/OCR locally and for free, the axis on which generic platforms (Dify, n8n) are weakest.
- Truly document-agnostic — extraction induces its own schema, so a new document type needs no code or template changes.
- Grounded & trustworthy — every chunk carries page + bbox, so answers and fields point at their exact source; the single most trust-building UX feature.
- Cheap and portable — runs on a free Gemini key with no external services; SQLite and the numpy store are single-file and swappable for Postgres/pgvector at scale via one module each.
- Graceful degradation — the app stays useful even when AI steps fail.
- Clean seams for growth — provider abstraction, storage/vectorstore interfaces, and a single-orchestrator agent that can grow more tools without a rewrite.