docs/ARCHITECTURE.md

Architecture

Overview

This project is a Retrieval-Augmented Generation (RAG) system. Given a research question, it (1) retrieves relevant evidence from a corpus of ArXiv abstracts and web search results, (2) reasons over the evidence with an LLM, and (3) returns an answer with inline citations.

The novel parts compared to a tutorial RAG are:

1. Hybrid retrieval — dense embeddings + BM25 in parallel, merged and reranked.
2. LangGraph workflow — query rewriting, evidence grading, generation, and a reflection loop, modeled as a state machine.
3. Pluggable LLM backend — one factory, four providers.

Component map

Layer	Module	Responsibility
Config	src/config.py	One Pydantic settings object; all env vars validated at startup
Ingestion	src/ingestion/	Load from ArXiv & DuckDuckGo, chunk with recursive splitting
Retrieval	src/retrieval/	Chroma (dense) + BM25 (sparse) + cross-encoder rerank
LLM	src/llm/	Provider factory + prompt templates
Orchestration	src/graph/workflow.py	LangGraph state machine
Pipeline	src/pipeline.py	High-level ingest() / ask() API
Interfaces	api.py, app.py	FastAPI service, Gradio UI
Evaluation	src/evaluation/	RAGAS metrics runner

Data flow

                       User
                        │ question
                        ▼
                ┌───────────────────┐
                │ Gradio / FastAPI  │
                └─────────┬─────────┘
                          │
                          ▼
                ┌───────────────────┐
                │ ResearchPipeline  │  ◀── singleton, holds the Chroma store
                └─────────┬─────────┘
                          │
                          ▼
                ┌───────────────────┐
                │ LangGraph         │
                └─────────┬─────────┘
                          │
   ┌──────────────────────┼──────────────────────┐
   ▼                      ▼                      ▼
rewrite_query  →   retrieve   →   grade   →   generate   →   reflect
                          ▲                                       │
                          └───────────────────────────────────────┘
                                       (one loop allowed)

LangGraph nodes — what each one does

Node	Purpose	Why it exists
rewrite_query	LLM rewrites the user's question into 2–3 search queries	Improves recall — academic search benefits from keyword variation that users don't naturally produce
retrieve	Hybrid retrieval over the corpus	The core retrieval step
grade	Per-document LLM call: is this relevant?	Filters out near-misses that the retriever still ranks high; cheap because we use the small/free LLM here
generate	LLM produces the answer with inline [N] citations	The synthesis step
reflect	LLM reads its own draft, decides "ok" or "go fetch X"	Lets the system self-correct on questions where the first retrieval missed key evidence

The reflection loop is capped at one iteration (MAX_REFLECTION_LOOPS = 1) to bound latency and cost — important on free-tier inference where each call can take seconds.

Retrieval strategy

Why hybrid? Pure dense retrieval misses rare-keyword matches (e.g., specific algorithm names, model identifiers). Pure BM25 misses semantic paraphrases. Combining them and deduplicating consistently produces a higher-recall candidate set, which the cross-encoder then reorders for precision.

Pipeline:

1. Embed the query and the corpus chunks with all-MiniLM-L6-v2 (384-dim, CPU-friendly).
2. Dense top-10 from Chroma (cosine similarity).
3. Sparse top-10 from BM25 over the same chunks.
4. Merge, dedupe by the first 200 chars of page_content.
5. Cross-encoder (ms-marco-MiniLM-L-6-v2) scores each (query, chunk) pair.
6. Return top-5.

Why pluggable LLM providers?

In development we use the Hugging Face Inference API (free tier, no credit card). For higher quality or production load we'd switch to Groq (fast Llama 3 inference, generous free tier), Anthropic (Claude Haiku for cost-effective quality), or OpenAI. The factory in src/llm/providers.py is the only place that knows about provider SDKs — everything downstream depends on LangChain's BaseChatModel interface.

This is a deliberate design choice: it means swapping providers is a single env var change, not a refactor.

Trade-offs and limitations

Trade-off	Choice made	Why	What I'd change at scale
Vector store	Chroma (embedded)	Zero ops, persists to disk, works on HF Spaces	Qdrant or Weaviate for multi-tenant, sharded deployments
Embedding dim	384 (MiniLM)	Fits on free CPU	bge-large (1024) on GPU for higher recall
Chunk size	800 chars	Good middle ground for abstracts	Adaptive sizing per source type
BM25 in memory	In-process, rebuilt on ingest	Simple, no extra service	OpenSearch / Elasticsearch for persistence and scale
Reflection loops	Capped at 1	Latency budget	Adaptive: stop when answer "stable"
Evaluation	Offline RAGAS	Standard, reproducible	Online A/B + LangSmith feedback loop

What runs where

• Local dev: python app.py (Gradio on :7860) or uvicorn api:app (FastAPI on :8000)
• Docker: docker compose up — both services with persisted Chroma + HF cache volumes
• HF Spaces: bash scripts/deploy_hf_space.sh <user> <space> — Docker SDK Space, app on :7860