docs/ARCHITECTURE.md

Architecture — Document Processing AI Agent

Decision record from research (see RESEARCH.md). Stack chosen: custom composed stack.

High-level

┌──────────────────────────────────────────────────────────────┐
│  FRONTEND  (Next.js + React + TypeScript + Tailwind)          │
│  40% Chat  │  60% Workspace                                   │
│  ────────  │  ── Tabs: Viewer · Fields · Classify · Summary · │
│  streaming │     Export ──                                    │
│  agent     │  Document viewer w/ bounding-box highlights      │
└─────┬────────────────────────────────────────────────────────┘
      │  REST + SSE (streaming)
┌─────▼────────────────────────────────────────────────────────┐
│  BACKEND  (FastAPI, async)                                    │
│                                                              │
│  api/        routes: documents, chat(SSE), extract,          │
│              classify, summary, export, health               │
│  agent/      orchestrator (tool-calling loop) + prompts      │
│  llm/        provider abstraction: Gemini(default)⇄OpenAI     │
│  services/   ingestion(Docling) · extraction · classification│
│              · summary · qa(RAG) · anomaly · export ·         │
│              vectorstore · storage                           │
│  schemas/    Pydantic models (incl. invoice/contract)        │
│  core/       config · logging · deps                         │
└─────┬───────────────────────────┬────────────────────────────┘
      │                           │
┌─────▼─────┐            ┌─────────▼──────────┐
│ Docling   │            │ Storage            │
│ (local    │            │  • files (disk/S3) │
│  extract) │            │  • SQLite/Postgres │
└───────────┘            │  • vectors (chroma/│
                         │    pgvector)       │
                         └────────────────────┘

The 7 core functions → implementation

Function	How	Tool exposed to agent
1. Upload & ingest	POST /documents → Docling parse → DoclingDocument JSON + page images	ingest_document
2. Parse & extract structured data	Docling structure + LLM structured-output against Pydantic schema (fields, tables, KV, entities) + confidence	extract_fields
3. Classify	LLM zero-shot classification → type + confidence	classify_document
4. Summarize	Map-reduce summarization over chunks	summarize_document
5. Conversational Q&A	Agentic RAG: retrieve chunks → answer with citations (page + bbox)	query_document
6. Flag anomalies	Rule + LLM checks: missing required fields, inconsistencies, low-confidence	flag_anomalies
7. Export	Serialize extracted data → JSON / CSV / Excel	export_data

Agent design

Single orchestrator agent with the tools above (tool-calling loop), agentic-RAG retrieval, structured outputs (Pydantic), human-in-the-loop on low-confidence fields. Pattern mirrors LlamaIndex ADW: parse → maintain state → retrieve → reason → surface for validation.

LLM strategy

• Default (free): Gemini Flash (1M context, native vision for scans).
• Fallback (switchable): GPT-4o-mini.
• llm/base.py defines a LLMProvider interface; runtime switch via config/env or per-request header. Cheap model for classify, stronger for reasoning.

Tech choices (pragmatic defaults for a runnable, impressive build)

• Backend: FastAPI, Pydantic v2, Docling, google-generativeai, openai, ChromaDB (embedded vector store — no external service), SQLite (dev) with a path to Postgres/pgvector.
• Frontend: Next.js (App Router) + React + TypeScript + Tailwind + a PDF/image viewer with overlay layer for bounding boxes; SSE for streaming chat.
• Local-first: runs with only a free Gemini API key. No paid services required.

Separation of concerns

Routes are thin → call services. Agent orchestrates services as tools. LLM access only via llm/. Storage only via services/storage + services/vectorstore. Schemas shared. This is the Dify-grade discipline we borrowed.