docs/RESEARCH.md

# Document Processing AI Agent — Deep Research Report

**Project:** Production-ready, chat-based Document Processing AI Agent
**Org:** Monkhub Innovations
**Date:** 2026-06-24
**Method:** Sequential, single-researcher web research (no sub-agents). Each system and source read in full, one at a time.

---

## 0. Executive Summary

We evaluated five leading systems for document-processing automation, spanning the four relevant archetypes:

| Archetype | System | Why it represents the category |
|---|---|---|
| Full open-source app | **RAGFlow** (infiniflow) | A complete chat-over-documents product with deep document understanding, citations, and an agent layer — the closest existing thing to what we are building. |
| LLM app platform | **Dify** (langgenius) | The reference for clean platform architecture, visual orchestration, observability, and polished UI/UX. |
| Extraction toolkit | **Docling** (IBM / LF AI & Data) | Best-in-class open document parsing (layout, tables, OCR, VLM) — the extraction *backbone*. |
| Workflow automation | **n8n** document workflows | The "glue" archetype: trigger → OCR → LLM extract → validate → route, assembled visually. |
| Agentic framework | **LlamaIndex** (Agentic Document Workflows + LlamaParse/Extract) | The reference for stateful, multi-step *document agents* and schema-based structured extraction. |

**Winner / primary reference foundation: RAGFlow**, with the explicit recommendation that we **do not fork it wholesale**. For our build we adopt **RAGFlow's pipeline design as the reference architecture**, use **Docling as the actual extraction engine** (MIT, local, zero per-page cost), borrow **Dify's clean separation-of-concerns and observability**, and structure the agent loop using **LlamaIndex-style agentic-workflow patterns**. Details in §3–§6.

---

## 1. The Five Systems — Deep Analysis

### 1.1 RAGFlow (infiniflow) — *Apache 2.0, ~83.5k★*

**What it does / core purpose.** Open-source RAG engine fused with agent capabilities that turns unstructured documents into a production "context layer" for LLMs. Philosophy: *"quality in, quality out"* — invest heavily in deep document understanding at ingestion so retrieval and answers are grounded and citable.

**End-to-end pipeline.**
1. **Ingest** — Word, PPT, Excel, PDF, images, scans, structured data, web pages.
2. **Deep document understanding** — the **DeepDoc** module performs layout analysis and structure extraction; can also delegate to MinerU or Docling parsers.
3. **Template-based chunking** — intelligent, configurable, *visualized* chunking with human review/intervention.
4. **Embed + index** — into Elasticsearch (default) or Infinity (their own engine).
5. **Multi-recall retrieval + fused re-ranking.**
6. **Grounded answer with citations** — traceable references to reduce hallucination.

**Architecture.** Frontend (web) → REST API → backend services → DeepDoc / embedding / LLM interface, over a storage tier of **MinIO** (objects), **Elasticsearch or Infinity** (vector + full-text), **MySQL** (metadata), **Redis** (cache). Agent framework adds workflow automation, MCP support, code executor (Python/JS, gVisor sandbox), and agent memory.

**Tech stack / models.** Python + Go backend, TypeScript frontend. Pluggable LLM and embedding providers (OpenAI, DeepSeek, Gemini, local). Docker / Docker Compose / Kubernetes (Helm).

**Key features.** Explainable template chunking; grounded citations with visualization; broad format support; agent templates + MCP; memory; many chat-channel and data-source connectors (Confluence, S3, Notion, Google Drive, Discord, etc.); multi-language UI.

**Strengths.** Deepest end-to-end document pipeline of the five; citation grounding; fully permissive license; mature, very active, well-documented; the most *directly comparable* product to our target.

**Weaknesses.** Heavy footprint (≥4 cores, ≥16 GB RAM, ≥50 GB disk; x86-first); operational learning curve; opinionated stack that is hard to bend into a *custom* 40/60 UI without significant surgery.

**Relevance.** ★★★★★ as a **reference** for the pipeline, citation UX, and chunking visualization. Lower as a thing-to-fork because its UI and infra are opinionated.

---

### 1.2 Dify (langgenius) — *Apache-2.0-based (with extra conditions), ~146k★*

**What it does / core purpose.** Open-source **LLM app development platform** — bridges prototype to production with a low-code visual canvas plus developer APIs (Backend-as-a-Service).

**End-to-end flow.** Design on a visual canvas → pick model (50+ providers) → ingest documents into a managed RAG pipeline → define agents (function-calling or ReAct) with 50+ built-in tools → test in a prompt IDE → monitor (logs/metrics) → deploy via API or hosted UI.

**Architecture.** Python backend (BaaS) + Next.js/React/TypeScript frontend; **PostgreSQL**; a canvas **workflow engine**; a unified **model-abstraction layer**; out-of-the-box **RAG pipeline**; first-class **observability** (Langfuse, Opik, Arize Phoenix).

**Tech stack / models.** TS (≈53%) + Python (≈43%). 50+ LLM providers incl. OpenAI, Anthropic, Google, Mistral, local. Docker Compose / K8s / Terraform / CDK. Min 2 cores / 4 GB.

**Key features.** Visual workflow builder; agent framework with tools; managed RAG; prompt IDE with side-by-side model comparison; production observability; complete API surface.

**Strengths.** Cleanest, most extensible *platform* architecture; very polished UI/UX; superb model-provider abstraction (directly relevant to our "free Gemini + GPT-4o-mini fallback" requirement); strong production/observability story; huge community.

**Weaknesses.** Document processing is **generic** — basic ingest/chunk, *no* specialized OCR / table-structure / layout intelligence. License has extra conditions beyond pure Apache 2.0. As a platform it can feel heavy if you only need an app.

**Relevance.** ★★★★☆ — the best **architectural and UI/UX reference**, and a candidate **model-routing layer**, but its document intelligence is too shallow to be our extraction engine.

---

### 1.3 Docling (IBM Research → LF AI & Data) — *MIT, ~62k★*

**What it does / core purpose.** Open document-processing library that parses diverse formats (with advanced PDF understanding) into a unified, AI-ready representation. It is the **extraction backbone**, not an agent.

**Pipeline.** Ingest (path/URL) → format detect → **layout analysis** (structure + reading order) → content extraction (text/tables/images) → **model inference** for hard elements → unified **DoclingDocument** → export (Markdown / JSON / HTML / DocTags).

**Architecture / models.** Layout analysis; **TableFormer** for table structure; OCR engines for scans; pluggable **VLMs** (built-in **GraniteDocling 258M**); ASR for audio; chart-understanding module. Python, Pydantic v2.

**Formats.** **In:** PDF, DOCX, PPTX, XLSX, HTML, EPUB, images (PNG/TIFF/JPEG), audio, email, LaTeX, Markdown, ODF, XBRL, more. **Out:** Markdown, HTML, lossless JSON, DocTags, domain XML.

**Key features.** Best-in-class layout + table extraction; semantic **chunking**; native adapters for **LangChain, LlamaIndex, CrewAI, Haystack, MCP**; **local / air-gapped** execution (no per-page API cost); CLI + Python.

**Strengths.** Best open extraction accuracy; widest format coverage; permissive MIT; local-first (privacy + zero marginal cost); OpenSSF badge; peer-reviewed; trivially embeddable into any backend.

**Weaknesses.** Not an application — no UI, chat, classification, or Q&A on its own (by design). VLM inference adds latency/compute; scaling docs sparse; some domains need fine-tuning.

**Relevance.** ★★★★★ as the **component we should actually use** for ingest/extraction. Not a competitor to the whole app — a building block.

---

### 1.4 n8n Document Workflows — *Sustainable Use License (fair-code), source-available*

**What it does / core purpose.** Visual workflow-automation platform; "document processing" emerges from chaining nodes. Hundreds of templates exist (e.g. *Mistral OCR + GPT-4o-mini invoice processing*, *Gemini OCR + Google Sheets*).

**Typical pipeline (3 layers).** **Ingestion** (webhook / email / Drive / upload trigger → download file) → **Intelligence** (OCR node — Mistral / Gemini / Google Vision — then an **AI Agent / LLM node** with a **structured-output parser** to emit JSON; classification + field extraction) → **Orchestration** (validation against business rules / POs, enrichment, routing to Sheets / SQL / CRM / ERP / Telegram / Gmail). Core design principle: **decouple extraction from delivery** via modular, reusable sub-workflows.

**Strengths.** Fastest way to prototype an extract→validate→route pipeline; 400+ integrations; visual + maintainable; built-in retries/logging; self-hostable.

**Weaknesses.** Not an end-user product — **no document-viewer or chat UI** for our use case (you'd build that separately). Logic-in-canvas sprawls at scale; n8n can become an orchestration bottleneck; **fair-code** license (not OSI-open) carries commercial-use restrictions.

**Relevance.** ★★★☆☆ — excellent **reference for the orchestration pattern** (and a possible internal automation tool), but not the foundation for a polished chat app.

---

### 1.5 LlamaIndex — Agentic Document Workflows (ADW) + LlamaParse/Extract — *Core MIT; Parse/Extract/Cloud are SaaS with free tier*

**What it does / core purpose.** A leading framework for **document agents**. ADW goes a step beyond both IDP and RAG: a **document agent** that parses, **maintains state across multi-step processes**, retrieves reference material, applies business logic, and produces recommendations (human-in-the-loop by design).

**Architecture / pattern.** Document agent orchestrates: **LlamaParse** (VLM-powered parsing) → **LlamaExtract** (Pydantic-schema structured extraction *with confidence scores*) → indexes/RAG for reference retrieval → **Workflow engine** (event-driven; fan-out/fan-in, self-reflection, human-in-the-loop). Inverts classic RAG: *parse → maintain state → retrieve contextually → reason → surface for validation*.

**Strengths.** Best **agent-orchestration** abstractions; schema-first extraction with confidence; excellent for conversational, stateful Q&A over documents; huge community and docs; MIT core.

**Weaknesses.** The strongest parsing/extraction (LlamaParse/Extract/Cloud) is **paid SaaS** (free tier limited); framework is code-first (no end-user UI provided); you assemble the product yourself.

**Relevance.** ★★★★☆ — the best **mental model and orchestration reference** for our agent loop; we can mirror ADW patterns using our own free-LLM + Docling stack instead of the paid services.

---

## 2. Comparison Table & Scoring

**Rubric (1–5; 5 = best).** Scored for the specific goal: *foundation/reference for a custom, chat-based, production document-processing agent with a 40/60 UI and free-LLM-first model strategy.*

| # | Parameter | What a 5 looks like |
|---|---|---|
| A | **Feature completeness (doc processing)** | Native OCR, layout, tables, KV, entities, classification, summarization, Q&A, export |
| B | **Architecture quality & extensibility** | Clean layers, pluggable models/stores, easy to add tools/formats |
| C | **Ease of customisation** | Fast to bend to our UI/flows without fighting the framework |
| D | **Production-readiness** | Auth, scaling, observability, error handling, deploy story |
| E | **UI/UX quality** | Polished, relevant end-user experience for document work |
| F | **Cost / open-source friendliness** | Permissive license, no forced per-page/API cost, self-hostable |
| G | **Community & documentation** | Stars, activity, docs depth |

| System | A | B | C | D | E | F | G | **Total /35** |
|---|---|---|---|---|---|---|---|---|
| **RAGFlow** | 5 | 4 | 3 | 4 | 4 | 5 | 5 | **30** |
| **Dify** | 3 | 5 | 5 | 5 | 5 | 4 | 5 | **32** |
| **Docling** | 5 | 5 | 4 | 4 | 1 | 5 | 5 | **29** |
| **n8n workflows** | 3 | 3 | 5 | 4 | 2 | 3 | 5 | **25** |
| **LlamaIndex ADW** | 4 | 5 | 4 | 4 | 2 | 3 | 5 | **27** |

**Reading the table.** By raw total, **Dify (32)** edges **RAGFlow (30)** — but Dify's lead comes from platform polish, *not* document intelligence (its weakest axis, A=3). When we weight by what this project actually needs — **document-processing depth (A), architecture (B), and customisation (C)** — the picture is:

- **Document depth (A):** RAGFlow & Docling lead (5).
- **Architecture (B):** Dify, Docling, LlamaIndex lead (5).
- **The honest conclusion is a *combination*, not a single fork.** No one system is simultaneously the deepest extractor *and* the cleanest platform *and* a ready-made 40/60 chat UI.

---

## 3. Winner & Why

**Primary reference foundation: RAGFlow.** It is the only system that already *is* the product we're describing — chat over documents, deep understanding, grounded citations, agentic layer — and it is fully Apache-2.0. It proves the end-to-end pipeline and gives us a battle-tested blueprint for chunking visualization and citation UX.

**But the senior-level recommendation is a composed stack, not a fork**, because RAGFlow's infra and UI are too opinionated to host our bespoke 40/60 experience, and its heaviness is unjustified at our stage:

> **Recommended build = RAGFlow's pipeline blueprint  +  Docling as the extraction engine  +  LlamaIndex-style agentic-workflow orchestration  +  Dify-grade separation-of-concerns & observability  +  our own FastAPI backend and React 40/60 UI.**

This gives us RAGFlow's proven flow, Docling's best-in-class *and free/local* extraction, LlamaIndex's stateful agent patterns, Dify's clean architecture discipline — with none of the licensing, cost, or lock-in downsides.

---

## 4. State-of-the-Art Capabilities (what our agent should do)

Modern IDP = **OCR + ICR + NLP + VLMs + agentic AI**. Expected capabilities:

- **Ingest:** PDF (digital + scanned), DOCX, PPTX, XLSX, images (PNG/JPG/TIFF), email, HTML.
- **Parse/extract:** layout & reading order, **tables** (structure-aware), **key–value pairs**, **named entities**, line items, multi-page spanning content.
- **Classify:** document type/category (invoice, contract, form, receipt, report…), zero-shot via LLM.
- **Summarize:** whole-doc and section-level.
- **Conversational Q&A:** grounded, **with citations** back to the source span.
- **Validate / flag:** missing required fields, inconsistencies, anomalies, confidence scoring, **human-in-the-loop** review.
- **Export:** **JSON** (schema-conformant), CSV/Excel (tables), and structured records to downstream systems.

**Field direction:** extraction is shifting from a *spatial* problem to a *semantic reasoning* problem — VLMs/LLMs read layout + meaning together and emit JSON directly, which is more robust to layout drift than legacy OCR. Best practice = **hybrid**: deterministic parser (Docling) for structure + LLM/VLM for semantic extraction, classification, and Q&A.

---

## 5. Ideal UI/UX (informs the 40/60 build)

Validated patterns from current document-AI and AI-chat UX research:

- **Split-screen** (chat left / work right) is the established pattern when the AI produces artifacts — see Claude Artifacts, Perplexity. Our **40% chat / 60% workspace** split is exactly this, correctly proportioned.
- **Document viewer with bounding-box highlights:** extracted fields and citations should **highlight the exact source region** in the rendered document (each cell/field carries a bounding polygon + confidence). This is the single most trust-building feature.
- **Citations:** numbered inline references linking to expandable source cards / highlighted spans; visually distinguish quoted evidence from synthesis.
- **Workspace panels (the 60%)** — multiple pages/tabs encouraged: **Document Viewer**, **Extraction/Fields** (editable table with confidence + flags), **Classification** panel, **Summary**, **Export** controls.
- **Side-panel, no context-switch:** keep the document in view while the AI works.
- **Streaming + accessibility:** stream responses; `aria-live="polite"` on response containers.
- **Human-in-the-loop:** low-confidence fields visibly flagged and inline-editable before export.

---

## 6. Recommended Architecture, LLM & Toolchain Strategy

**Agent design patterns (2025–26).** The field has moved past static RAG toward **reasoning/agentic RAG** and multi-agent patterns. For us: an **orchestrator agent** + **tool-calling** + **agentic RAG** (the agent decides *when/what/how* to retrieve), with structured-output tools and human-in-the-loop. Keep it a single well-orchestrated agent with tools (upload, parse, classify, extract, summarize, query, validate, export) before reaching for multi-agent complexity.

**LLM strategy (matches the brief).**
- **Default (free):** **Google Gemini Flash** family — generous free tier, ~1M-token context (great for long documents), native multimodal/vision (can read document images directly).
- **Fallback (switchable):** **GPT-4o-mini** — cheap, 128k context, reliable tool-calling.
- **Provider-abstraction layer** so models are swappable per-task (cheap model for classification, stronger for reasoning) — mirroring Dify's model layer.

**Toolchain / OCR / orchestration.**
- **Extraction:** **Docling** (layout, TableFormer tables, OCR for scans, DoclingDocument → JSON/Markdown) — local, MIT, zero per-page cost. Optional VLM path (Gemini vision) for hard scans.
- **Structured extraction:** **Pydantic schemas + confidence scores** (LlamaExtract pattern), enforced via LLM structured output / tool-calling.
- **Orchestration:** lightweight agent loop in our own backend (LlamaIndex-style workflow patterns; optionally the LlamaIndex Workflows library) — avoids the heavy multi-service footprint of RAGFlow.
- **RAG:** chunk (Docling) → embed → vector store (pgvector / Qdrant) → agentic retrieval with citations.

**Proposed stack for the build.**
- **Backend:** **FastAPI** (Python) — async, clean separation (routes / services / agent / tools / schemas), production API.
- **Frontend:** **React + TypeScript** (Next.js or Vite) — 40/60 layout, document viewer with bounding-box overlays, streaming chat.
- **Storage:** Postgres (+ pgvector) for metadata + vectors; object storage (local/MinIO/S3) for files.
- **Models:** Gemini Flash (default) ⇄ GPT-4o-mini (fallback) behind a provider interface.
- **Extraction:** Docling.

---

## 7. Sources

- https://www.firecrawl.dev/blog/best-open-source-agent-frameworks
- https://github.com/infiniflow/ragflow
- https://github.com/langgenius/dify
- https://github.com/docling-project/docling
- https://github.com/opendatalab/mineru
- https://procycons.com/en/blogs/pdf-data-extraction-benchmark/
- https://n8n.io/workflows/4933-document-parsing-and-data-extraction-with-mistral-ocr-ai-processing-and-multi-channel-delivery/
- https://data-sleek.com/blog/document-workflow-automation-seamless-data-stack-n8n-ai/
- https://www.llamaindex.ai/blog/introducing-agentic-document-workflows
- https://www.llamaindex.ai/workflows
- https://www.everestgrp.com/report/egr-2025-38-r-7283/
- https://scalehub.com/2025-idp-guide/
- https://uxdesign.cc/where-should-ai-sit-in-your-ui-1710a258390e
- https://thefrontkit.com/blogs/ai-chat-ui-best-practices
- https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/prebuilt/layout
- https://www.azilen.com/blog/agentic-ai-design-patterns/
- https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/ai-agent-design-patterns
- https://modal.com/blog/8-top-open-source-ocr-models-compared
- https://www.llamaindex.ai/insights/best-vision-language-models