docs/OVERVIEW.md

# DocAgent — Project Overview, Research, Design & How It Works

**Project:** Document Processing AI Agent (chat-based, 40/60 UI)
**Org:** Monkhub Innovations
**One-line:** Upload any document → it parses, classifies, extracts structured
data, summarizes, answers grounded questions, flags anomalies, and exports clean data.

This single document brings together **what we researched, what we chose and why,
the comparison, the feature set, and exactly how the agent works end-to-end.**
For the full deep-dive research see [`RESEARCH.md`](RESEARCH.md); for the layered
architecture record see [`ARCHITECTURE.md`](ARCHITECTURE.md); to deploy see
[`DEPLOY.md`](DEPLOY.md).

---

## 1. Research — what we evaluated

We needed a foundation for a **custom, chat-based, production document-processing
agent** with a 40/60 split UI and a **free-LLM-first** model strategy. We evaluated
the five leading systems, one per relevant archetype:

| Archetype | System | Why it represents the category |
|---|---|---|
| Full open-source app | **RAGFlow** | A complete chat-over-documents product with deep understanding, citations, and an agent layer — the closest existing thing to our target. |
| LLM app platform | **Dify** | Reference for clean platform architecture, visual orchestration, observability, polished UI/UX. |
| Extraction toolkit | **Docling** (IBM / LF AI & Data) | Best-in-class open document parsing — layout, tables, OCR, VLM. The extraction *backbone*. |
| Workflow automation | **n8n** | The "glue" archetype: trigger → OCR → LLM extract → validate → route, assembled visually. |
| Agentic framework | **LlamaIndex ADW** | Reference for stateful, multi-step *document agents* and schema-based structured extraction. |

**Method:** sequential, single-researcher web research; each system and source read
in full. Full notes and sources in [`RESEARCH.md`](RESEARCH.md).

---

## 2. Comparison table & scoring

**Rubric (1–5; 5 = best)** — scored for *our* specific goal, not in the abstract:

| Code | Parameter |
|---|---|
| A | Feature completeness (OCR, layout, tables, KV, entities, classify, summarize, Q&A, export) |
| B | Architecture quality & extensibility |
| C | Ease of customisation to our UI/flows |
| D | Production-readiness (auth, scaling, observability, deploy) |
| E | UI/UX quality for document work |
| F | Cost / open-source friendliness (license, no per-page cost, self-host) |
| G | Community & documentation |

| System | A | B | C | D | E | F | G | **Total /35** |
|---|---|---|---|---|---|---|---|---|
| **Dify** | 3 | 5 | 5 | 5 | 5 | 4 | 5 | **32** |
| **RAGFlow** | 5 | 4 | 3 | 4 | 4 | 5 | 5 | **30** |
| **Docling** | 5 | 5 | 4 | 4 | 1 | 5 | 5 | **29** |
| **LlamaIndex ADW** | 4 | 5 | 4 | 4 | 2 | 3 | 5 | **27** |
| **n8n workflows** | 3 | 3 | 5 | 4 | 2 | 3 | 5 | **25** |

**How to read it.** Dify wins on raw total, but its lead comes from *platform polish*,
not *document intelligence* (its weakest axis, A=3). RAGFlow and Docling lead on the
axis this project lives or dies on — document depth (A=5). The honest conclusion is
that **no single system is simultaneously the deepest extractor, the cleanest platform,
and a ready-made 40/60 chat UI.** So we composed.

---

## 3. What we chose & why

> **Decision: a composed custom stack — not a fork of any one system.**

| Borrowed from | What we took |
|---|---|
| **RAGFlow** | The end-to-end **pipeline blueprint** (deep-understanding-at-ingestion → grounded, citable answers) and citation UX. |
| **Docling** | The **actual extraction engine** — MIT, local, layout + TableFormer tables + OCR, zero per-page cost. |
| **LlamaIndex ADW** | The **agent loop pattern**: *parse → maintain state → retrieve → reason → surface for validation.* |
| **Dify** | **Separation-of-concerns discipline** — thin routes, services, provider abstraction. |
| **Our own** | **FastAPI** backend + **Next.js/React** 40/60 UI tailored to the brief. |

**Why composed instead of forking RAGFlow** (the top *product*): its infra (MinIO +
Elasticsearch + MySQL + Redis) and opinionated UI are too heavy to host our bespoke
40/60 experience, and the footprint (≥4 cores / ≥16 GB / ≥50 GB) is unjustified at our
stage. Composing gives us RAGFlow's proven flow, Docling's best-in-class *and free/local*
extraction, LlamaIndex's stateful agent patterns, and Dify's architectural cleanliness —
with none of the licensing, cost, or lock-in downsides.

### Chosen stack at a glance

| Layer | Choice | Why |
|---|---|---|
| **Extraction** | **Docling** (layout, TableFormer, EasyOCR) | Best open accuracy; MIT; local; $0 per page; offline-capable. |
| **LLM (default)** | **Gemini Flash** | Generous free tier; ~1M-token context for long docs; native vision for scans. |
| **LLM (fallback)** | **GPT-4o-mini** | Cheap, reliable tool-calling; switchable per-request behind a provider interface. |
| **Backend** | **FastAPI** (async, Pydantic v2) | Clean route→service layering; SSE streaming; production-ready. |
| **Frontend** | **Next.js (App Router) + React + TS + Tailwind** | 40/60 layout, document viewer with bounding-box overlays, streaming chat. |
| **Vector store** | **Pure-Python numpy cosine** (`.npz`/`.json` on disk) | Dependency-light, no native build/service; chunks carry page+bbox for citations. Swappable for pgvector/Chroma. |
| **Metadata store** | **SQLite** (JSON blobs) | Zero external services; access funnelled through one module → easy Postgres swap. |

**Local-first & ~zero marginal cost:** the whole app runs with only a free Gemini key.
Nothing imports a vendor SDK except `llm/*`, so providers swap by env, per request, or
from the UI.

---

## 4. Features — the 7 core functions

| # | Function | What it does | Agent tool |
|---|---|---|---|
| 1 | **Ingest** | Parse PDF, DOCX, PPTX, XLSX, HTML, images with Docling (layout, tables, OCR) → unified Markdown + page images + provenance-carrying chunks. | *(pipeline)* |
| 2 | **Extract** | **Adaptive, template-free** structured data: fields / records / tables / entities, each with a confidence score and best-effort citation. | `get_extracted_data` |
| 3 | **Classify** | Zero-shot document-type detection with confidence + rationale. | `classify_document` |
| 4 | **Summarize** | Map-reduce summary (TL;DR + key points) over chunks. | `summarize_document` |
| 5 | **Q&A** | Conversational, **agentic-RAG** answers **grounded with citations** you can click to highlight on the page. | `query_document` |
| 6 | **Flag anomalies** | Missing fields, low-confidence values, generic arithmetic reconciliation, multi-record sanity, LLM consistency review. | `flag_anomalies` |
| 7 | **Export** | Serialize extracted data to **JSON / CSV / Excel** (records become a table). | *(API)* |

### The headline feature: adaptive, domain-agnostic extraction

Extraction does **not** rely on per-domain templates. For *any* document — any layout,
any label convention, known type or not — it produces a stable, consistently-keyed shape by:

- **Layout detection** — the model decides whether the doc is one entity
  (`single_record`, e.g. an invoice), many similar entities (`multi_record`, e.g. an
  employee roster or transaction list), or a header + repeated sub-list
  (`mixed`, e.g. invoice + line items).
- **Schema induction** — fields are derived from the actual content (a known type only
  *suggests* fields; the model may add/drop to fit reality).
- **Canonical keys** — inconsistent source labels ("Staff Number" / "Personnel ID" / "ID")
  are normalised to one `snake_case` key, identical across every record.
- **Grounding** — values come only from the document body (never file metadata); dates and
  amounts are normalised (ISO dates, currency); citations are attached by matching values
  back to provenance-carrying chunks.

---

## 5. How it works — architecture

```
┌──────────────────────────────────────────────────────────────┐
│  FRONTEND  (Next.js + React + TypeScript + Tailwind)          │
│  40% Chat (streaming agent)  │  60% Workspace                 │
│                              │  Tabs: Viewer · Fields ·       │
│                              │  Classify · Summary · Export   │
│                              │  Viewer w/ bounding-box cites   │
└─────┬────────────────────────────────────────────────────────┘
      │  REST + SSE (token streaming)
┌─────▼────────────────────────────────────────────────────────┐
│  BACKEND  (FastAPI, async)                                    │
│  api/        documents · chat(SSE) · actions · meta(health)   │
│  agent/      orchestrator (tool-calling loop) + prompts       │
│  llm/        provider abstraction: Gemini(default) ⇄ OpenAI   │
│  services/   ingestion(Docling) · extraction · classification │
│              · summary · qa(RAG) · anomaly · export ·          │
│              vectorstore · storage · pipeline                  │
│  schemas/    Pydantic contracts (documents, optional hints)   │
└─────┬───────────────────────────┬────────────────────────────┘
      │                           │
┌─────▼─────┐            ┌─────────▼──────────┐
│ Docling   │            │ Storage (/data)    │
│ (local    │            │ • files on disk    │
│  extract) │            │ • SQLite metadata  │
│ models    │            │ • numpy vectors    │
│ baked in  │            │   (.npz/.json)     │
└───────────┘            │ • rendered pages   │
                         └────────────────────┘
```

**Discipline (borrowed from Dify):** routes are thin → call services; the agent
orchestrates services as tools; LLM access only through `llm/`; storage only through
`services/storage` + `services/vectorstore`; schemas are shared.

---

## 6. Working — the two flows

### Flow A — Ingestion pipeline (runs once per upload, in the background)

Implemented in `services/pipeline.py`. State is written back at every milestone so the
UI can poll status and render results progressively.

```
upload → [Stage 1: INGEST]──fail?──► status=failed (hard fail; only this stage can)
            │ Docling parse → Markdown + page images + chunks(page,bbox)
            ▼
        [index]      embed chunks → numpy vector store        ┐
        [classify]   LLM zero-shot → type + confidence        │ Stages 2–6 are
        [extract]    adaptive induction → fields/records/...   │ best-effort: a
        [summarize]  map-reduce → TL;DR + key points           │ failure (e.g. no
        [anomalies]  rule + LLM checks (needs extraction)      │ LLM key) degrades
            ▼                                                  ┘ gracefully
        status=ready  (viewer always works, even if AI steps were skipped)
```

Key robustness property: **only ingestion can hard-fail a document.** Everything else is
wrapped so a missing API key or a flaky step still yields a `ready` document with a
working viewer and a note about what was skipped.

### Flow B — Chat agent (per user message)

Implemented in `agent/orchestrator.py` — a bounded **tool-calling loop**, then a
**streamed** final answer.

```
user message + history + doc context
        │
        ▼
  Phase 1: TOOL ROUNDS  (up to MAX_TOOL_ROUNDS = 4, non-streamed reasoning)
     llm.complete_tools(messages, TOOL_SPECS)
        │
        ├── model returns tool_calls?  ──yes──► run each tool ─► append results ─► loop
        │        tools: query_document · classify_document ·
        │               get_extracted_data · summarize_document · flag_anomalies
        │
        └── no tool_calls (ready to answer) ─► exit loop
        ▼
  Phase 2: FINAL ANSWER  (streamed token-by-token over SSE, temperature 0.4)
        │
        ▼
  client renders answer + citation chips → click a chip → highlight the
  page region (numpy store returned page+bbox with each retrieved chunk)
```

`query_document` runs **agentic RAG**: retrieve the most similar chunks, answer from
them, and return grounding passages whose page+bbox the UI turns into clickable
highlights. The orchestrator collects citations in its `ToolContext` so the API can
emit them alongside the streamed answer.

---

## 7. End-to-end walkthrough (a user's view)

1. **Drag a document** into the left rail (or click *New document*).
2. The pipeline runs: pages render in the viewer first, then **classification, fields,
   summary, and anomaly flags** stream into the workspace tabs as each stage finishes.
3. **Chat** on the left — e.g. *"Summarize this"*, *"What's the total?"*, *"Any missing
   fields?"* The agent calls the right tools, then streams a grounded answer. Click any
   **citation chip** to highlight its exact source on the page.
4. **Review & edit** low-confidence fields inline (human-in-the-loop) before exporting.
5. **Export** from the Export tab as **JSON / CSV / Excel**.

---

## 8. Why this design holds up

- **Document depth where it matters** — Docling gives best-in-class layout/table/OCR
  locally and for free, the axis on which generic platforms (Dify, n8n) are weakest.
- **Truly document-agnostic** — extraction induces its own schema, so a new document
  type needs no code or template changes.
- **Grounded & trustworthy** — every chunk carries page + bbox, so answers and fields
  point at their exact source; the single most trust-building UX feature.
- **Cheap and portable** — runs on a free Gemini key with no external services; SQLite
  and the numpy store are single-file and swappable for Postgres/pgvector at scale via
  one module each.
- **Graceful degradation** — the app stays useful even when AI steps fail.
- **Clean seams for growth** — provider abstraction, storage/vectorstore interfaces,
  and a single-orchestrator agent that can grow more tools without a rewrite.