Spaces:

jai291103
/

document_agent

Sleeping

App Files Files Community

document_agent / docs /RESEARCH.md

Jai-rathore29

Deploy: DocAgent backend (deterministic date-anomaly fix)

f65e025 about 1 month ago

preview code

Raw

History Blame Contribute Delete

19.8 kB

Document Processing AI Agent — Deep Research Report

Project: Production-ready, chat-based Document Processing AI Agent Org: Monkhub Innovations Date: 2026-06-24 Method: Sequential, single-researcher web research (no sub-agents). Each system and source read in full, one at a time.

0. Executive Summary

We evaluated five leading systems for document-processing automation, spanning the four relevant archetypes:

Archetype	System	Why it represents the category
Full open-source app	RAGFlow (infiniflow)	A complete chat-over-documents product with deep document understanding, citations, and an agent layer — the closest existing thing to what we are building.
LLM app platform	Dify (langgenius)	The reference for clean platform architecture, visual orchestration, observability, and polished UI/UX.
Extraction toolkit	Docling (IBM / LF AI & Data)	Best-in-class open document parsing (layout, tables, OCR, VLM) — the extraction backbone.
Workflow automation	n8n document workflows	The "glue" archetype: trigger → OCR → LLM extract → validate → route, assembled visually.
Agentic framework	LlamaIndex (Agentic Document Workflows + LlamaParse/Extract)	The reference for stateful, multi-step document agents and schema-based structured extraction.

Winner / primary reference foundation: RAGFlow, with the explicit recommendation that we do not fork it wholesale. For our build we adopt RAGFlow's pipeline design as the reference architecture, use Docling as the actual extraction engine (MIT, local, zero per-page cost), borrow Dify's clean separation-of-concerns and observability, and structure the agent loop using LlamaIndex-style agentic-workflow patterns. Details in §3–§6.

1. The Five Systems — Deep Analysis

1.1 RAGFlow (infiniflow) — Apache 2.0, ~83.5k★

What it does / core purpose. Open-source RAG engine fused with agent capabilities that turns unstructured documents into a production "context layer" for LLMs. Philosophy: "quality in, quality out" — invest heavily in deep document understanding at ingestion so retrieval and answers are grounded and citable.

End-to-end pipeline.

Ingest — Word, PPT, Excel, PDF, images, scans, structured data, web pages.
Deep document understanding — the DeepDoc module performs layout analysis and structure extraction; can also delegate to MinerU or Docling parsers.
Template-based chunking — intelligent, configurable, visualized chunking with human review/intervention.
Embed + index — into Elasticsearch (default) or Infinity (their own engine).
Multi-recall retrieval + fused re-ranking.
Grounded answer with citations — traceable references to reduce hallucination.

Architecture. Frontend (web) → REST API → backend services → DeepDoc / embedding / LLM interface, over a storage tier of MinIO (objects), Elasticsearch or Infinity (vector + full-text), MySQL (metadata), Redis (cache). Agent framework adds workflow automation, MCP support, code executor (Python/JS, gVisor sandbox), and agent memory.

Tech stack / models. Python + Go backend, TypeScript frontend. Pluggable LLM and embedding providers (OpenAI, DeepSeek, Gemini, local). Docker / Docker Compose / Kubernetes (Helm).

Key features. Explainable template chunking; grounded citations with visualization; broad format support; agent templates + MCP; memory; many chat-channel and data-source connectors (Confluence, S3, Notion, Google Drive, Discord, etc.); multi-language UI.

Strengths. Deepest end-to-end document pipeline of the five; citation grounding; fully permissive license; mature, very active, well-documented; the most directly comparable product to our target.

Weaknesses. Heavy footprint (≥4 cores, ≥16 GB RAM, ≥50 GB disk; x86-first); operational learning curve; opinionated stack that is hard to bend into a custom 40/60 UI without significant surgery.

Relevance. ★★★★★ as a reference for the pipeline, citation UX, and chunking visualization. Lower as a thing-to-fork because its UI and infra are opinionated.

1.2 Dify (langgenius) — Apache-2.0-based (with extra conditions), ~146k★

What it does / core purpose. Open-source LLM app development platform — bridges prototype to production with a low-code visual canvas plus developer APIs (Backend-as-a-Service).

End-to-end flow. Design on a visual canvas → pick model (50+ providers) → ingest documents into a managed RAG pipeline → define agents (function-calling or ReAct) with 50+ built-in tools → test in a prompt IDE → monitor (logs/metrics) → deploy via API or hosted UI.

Architecture. Python backend (BaaS) + Next.js/React/TypeScript frontend; PostgreSQL; a canvas workflow engine; a unified model-abstraction layer; out-of-the-box RAG pipeline; first-class observability (Langfuse, Opik, Arize Phoenix).

Tech stack / models. TS (≈53%) + Python (≈43%). 50+ LLM providers incl. OpenAI, Anthropic, Google, Mistral, local. Docker Compose / K8s / Terraform / CDK. Min 2 cores / 4 GB.

Key features. Visual workflow builder; agent framework with tools; managed RAG; prompt IDE with side-by-side model comparison; production observability; complete API surface.

Strengths. Cleanest, most extensible platform architecture; very polished UI/UX; superb model-provider abstraction (directly relevant to our "free Gemini + GPT-4o-mini fallback" requirement); strong production/observability story; huge community.

Weaknesses. Document processing is generic — basic ingest/chunk, no specialized OCR / table-structure / layout intelligence. License has extra conditions beyond pure Apache 2.0. As a platform it can feel heavy if you only need an app.

Relevance. ★★★★☆ — the best architectural and UI/UX reference, and a candidate model-routing layer, but its document intelligence is too shallow to be our extraction engine.

1.3 Docling (IBM Research → LF AI & Data) — MIT, ~62k★

What it does / core purpose. Open document-processing library that parses diverse formats (with advanced PDF understanding) into a unified, AI-ready representation. It is the extraction backbone, not an agent.

Pipeline. Ingest (path/URL) → format detect → layout analysis (structure + reading order) → content extraction (text/tables/images) → model inference for hard elements → unified DoclingDocument → export (Markdown / JSON / HTML / DocTags).

Architecture / models. Layout analysis; TableFormer for table structure; OCR engines for scans; pluggable VLMs (built-in GraniteDocling 258M); ASR for audio; chart-understanding module. Python, Pydantic v2.

Formats. In: PDF, DOCX, PPTX, XLSX, HTML, EPUB, images (PNG/TIFF/JPEG), audio, email, LaTeX, Markdown, ODF, XBRL, more. Out: Markdown, HTML, lossless JSON, DocTags, domain XML.

Key features. Best-in-class layout + table extraction; semantic chunking; native adapters for LangChain, LlamaIndex, CrewAI, Haystack, MCP; local / air-gapped execution (no per-page API cost); CLI + Python.

Strengths. Best open extraction accuracy; widest format coverage; permissive MIT; local-first (privacy + zero marginal cost); OpenSSF badge; peer-reviewed; trivially embeddable into any backend.

Weaknesses. Not an application — no UI, chat, classification, or Q&A on its own (by design). VLM inference adds latency/compute; scaling docs sparse; some domains need fine-tuning.

Relevance. ★★★★★ as the component we should actually use for ingest/extraction. Not a competitor to the whole app — a building block.

1.4 n8n Document Workflows — Sustainable Use License (fair-code), source-available

What it does / core purpose. Visual workflow-automation platform; "document processing" emerges from chaining nodes. Hundreds of templates exist (e.g. Mistral OCR + GPT-4o-mini invoice processing, Gemini OCR + Google Sheets).

Typical pipeline (3 layers). Ingestion (webhook / email / Drive / upload trigger → download file) → Intelligence (OCR node — Mistral / Gemini / Google Vision — then an AI Agent / LLM node with a structured-output parser to emit JSON; classification + field extraction) → Orchestration (validation against business rules / POs, enrichment, routing to Sheets / SQL / CRM / ERP / Telegram / Gmail). Core design principle: decouple extraction from delivery via modular, reusable sub-workflows.

Strengths. Fastest way to prototype an extract→validate→route pipeline; 400+ integrations; visual + maintainable; built-in retries/logging; self-hostable.

Weaknesses. Not an end-user product — no document-viewer or chat UI for our use case (you'd build that separately). Logic-in-canvas sprawls at scale; n8n can become an orchestration bottleneck; fair-code license (not OSI-open) carries commercial-use restrictions.

Relevance. ★★★☆☆ — excellent reference for the orchestration pattern (and a possible internal automation tool), but not the foundation for a polished chat app.

1.5 LlamaIndex — Agentic Document Workflows (ADW) + LlamaParse/Extract — Core MIT; Parse/Extract/Cloud are SaaS with free tier

What it does / core purpose. A leading framework for document agents. ADW goes a step beyond both IDP and RAG: a document agent that parses, maintains state across multi-step processes, retrieves reference material, applies business logic, and produces recommendations (human-in-the-loop by design).

Architecture / pattern. Document agent orchestrates: LlamaParse (VLM-powered parsing) → LlamaExtract (Pydantic-schema structured extraction with confidence scores) → indexes/RAG for reference retrieval → Workflow engine (event-driven; fan-out/fan-in, self-reflection, human-in-the-loop). Inverts classic RAG: parse → maintain state → retrieve contextually → reason → surface for validation.

Strengths. Best agent-orchestration abstractions; schema-first extraction with confidence; excellent for conversational, stateful Q&A over documents; huge community and docs; MIT core.

Weaknesses. The strongest parsing/extraction (LlamaParse/Extract/Cloud) is paid SaaS (free tier limited); framework is code-first (no end-user UI provided); you assemble the product yourself.

Relevance. ★★★★☆ — the best mental model and orchestration reference for our agent loop; we can mirror ADW patterns using our own free-LLM + Docling stack instead of the paid services.

2. Comparison Table & Scoring

Rubric (1–5; 5 = best). Scored for the specific goal: foundation/reference for a custom, chat-based, production document-processing agent with a 40/60 UI and free-LLM-first model strategy.

#	Parameter	What a 5 looks like
A	Feature completeness (doc processing)	Native OCR, layout, tables, KV, entities, classification, summarization, Q&A, export
B	Architecture quality & extensibility	Clean layers, pluggable models/stores, easy to add tools/formats
C	Ease of customisation	Fast to bend to our UI/flows without fighting the framework
D	Production-readiness	Auth, scaling, observability, error handling, deploy story
E	UI/UX quality	Polished, relevant end-user experience for document work
F	Cost / open-source friendliness	Permissive license, no forced per-page/API cost, self-hostable
G	Community & documentation	Stars, activity, docs depth

System	A	B	C	D	E	F	G	Total /35
RAGFlow	5	4	3	4	4	5	5	30
Dify	3	5	5	5	5	4	5	32
Docling	5	5	4	4	1	5	5	29
n8n workflows	3	3	5	4	2	3	5	25
LlamaIndex ADW	4	5	4	4	2	3	5	27

Reading the table. By raw total, Dify (32) edges RAGFlow (30) — but Dify's lead comes from platform polish, not document intelligence (its weakest axis, A=3). When we weight by what this project actually needs — document-processing depth (A), architecture (B), and customisation (C) — the picture is:

Document depth (A): RAGFlow & Docling lead (5).
Architecture (B): Dify, Docling, LlamaIndex lead (5).
The honest conclusion is a combination, not a single fork. No one system is simultaneously the deepest extractor and the cleanest platform and a ready-made 40/60 chat UI.

3. Winner & Why

Primary reference foundation: RAGFlow. It is the only system that already is the product we're describing — chat over documents, deep understanding, grounded citations, agentic layer — and it is fully Apache-2.0. It proves the end-to-end pipeline and gives us a battle-tested blueprint for chunking visualization and citation UX.

But the senior-level recommendation is a composed stack, not a fork, because RAGFlow's infra and UI are too opinionated to host our bespoke 40/60 experience, and its heaviness is unjustified at our stage:

Recommended build = RAGFlow's pipeline blueprint + Docling as the extraction engine + LlamaIndex-style agentic-workflow orchestration + Dify-grade separation-of-concerns & observability + our own FastAPI backend and React 40/60 UI.

This gives us RAGFlow's proven flow, Docling's best-in-class and free/local extraction, LlamaIndex's stateful agent patterns, Dify's clean architecture discipline — with none of the licensing, cost, or lock-in downsides.

4. State-of-the-Art Capabilities (what our agent should do)

Modern IDP = OCR + ICR + NLP + VLMs + agentic AI. Expected capabilities:

Ingest: PDF (digital + scanned), DOCX, PPTX, XLSX, images (PNG/JPG/TIFF), email, HTML.
Parse/extract: layout & reading order, tables (structure-aware), key–value pairs, named entities, line items, multi-page spanning content.
Classify: document type/category (invoice, contract, form, receipt, report…), zero-shot via LLM.
Summarize: whole-doc and section-level.
Conversational Q&A: grounded, with citations back to the source span.
Validate / flag: missing required fields, inconsistencies, anomalies, confidence scoring, human-in-the-loop review.
Export: JSON (schema-conformant), CSV/Excel (tables), and structured records to downstream systems.

Field direction: extraction is shifting from a spatial problem to a semantic reasoning problem — VLMs/LLMs read layout + meaning together and emit JSON directly, which is more robust to layout drift than legacy OCR. Best practice = hybrid: deterministic parser (Docling) for structure + LLM/VLM for semantic extraction, classification, and Q&A.

5. Ideal UI/UX (informs the 40/60 build)

Validated patterns from current document-AI and AI-chat UX research:

Split-screen (chat left / work right) is the established pattern when the AI produces artifacts — see Claude Artifacts, Perplexity. Our 40% chat / 60% workspace split is exactly this, correctly proportioned.
Document viewer with bounding-box highlights: extracted fields and citations should highlight the exact source region in the rendered document (each cell/field carries a bounding polygon + confidence). This is the single most trust-building feature.
Citations: numbered inline references linking to expandable source cards / highlighted spans; visually distinguish quoted evidence from synthesis.
Workspace panels (the 60%) — multiple pages/tabs encouraged: Document Viewer, Extraction/Fields (editable table with confidence + flags), Classification panel, Summary, Export controls.
Side-panel, no context-switch: keep the document in view while the AI works.
Streaming + accessibility: stream responses; aria-live="polite" on response containers.
Human-in-the-loop: low-confidence fields visibly flagged and inline-editable before export.

6. Recommended Architecture, LLM & Toolchain Strategy

Agent design patterns (2025–26). The field has moved past static RAG toward reasoning/agentic RAG and multi-agent patterns. For us: an orchestrator agent + tool-calling + agentic RAG (the agent decides when/what/how to retrieve), with structured-output tools and human-in-the-loop. Keep it a single well-orchestrated agent with tools (upload, parse, classify, extract, summarize, query, validate, export) before reaching for multi-agent complexity.

LLM strategy (matches the brief).

Default (free): Google Gemini Flash family — generous free tier, ~1M-token context (great for long documents), native multimodal/vision (can read document images directly).
Fallback (switchable): GPT-4o-mini — cheap, 128k context, reliable tool-calling.
Provider-abstraction layer so models are swappable per-task (cheap model for classification, stronger for reasoning) — mirroring Dify's model layer.

Toolchain / OCR / orchestration.

Extraction: Docling (layout, TableFormer tables, OCR for scans, DoclingDocument → JSON/Markdown) — local, MIT, zero per-page cost. Optional VLM path (Gemini vision) for hard scans.
Structured extraction: Pydantic schemas + confidence scores (LlamaExtract pattern), enforced via LLM structured output / tool-calling.
Orchestration: lightweight agent loop in our own backend (LlamaIndex-style workflow patterns; optionally the LlamaIndex Workflows library) — avoids the heavy multi-service footprint of RAGFlow.
RAG: chunk (Docling) → embed → vector store (pgvector / Qdrant) → agentic retrieval with citations.

Proposed stack for the build.

Backend: FastAPI (Python) — async, clean separation (routes / services / agent / tools / schemas), production API.
Frontend: React + TypeScript (Next.js or Vite) — 40/60 layout, document viewer with bounding-box overlays, streaming chat.
Storage: Postgres (+ pgvector) for metadata + vectors; object storage (local/MinIO/S3) for files.
Models: Gemini Flash (default) ⇄ GPT-4o-mini (fallback) behind a provider interface.
Extraction: Docling.