README.md

metadata

title: AgentBench
emoji: ⚡
colorFrom: blue
colorTo: indigo
sdk: streamlit
sdk_version: 1.41.0
app_file: app.py
pinned: false

⚡ AgentBench

Multi-Agent Research Evaluation System

A production-structured LangGraph pipeline that benchmarks multi-agent vs single-agent LLM responses across 50 queries on 5 real evaluation metrics

---

What Is This?

Most AI demos show you an output and say "looks good." AgentBench actually measures it.

It runs two approaches on the same query — a single ReAct agent and a 5-node multi-agent pipeline — then scores both using a real LLM-as-judge evaluator on 5 metrics. Every result is computed, not hardcoded.

The core question it answers: Does splitting reasoning, research, analysis, and writing across specialised agents produce more trustworthy responses than one agent doing everything?

Answer: Yes — 6× lower hallucination. Here are the numbers.

---

Benchmark Results

Evaluated on 50 queries across GenAI, ML fundamentals, agentic AI, retrieval, and applied AI. Judged by llama-3.1-8b-instant as an independent LLM evaluator.

Metric	Single Agent	Multi-Agent	Winner
Avg Relevance	0.877	0.850	Single
Avg Coherence	0.912	0.900	Single
Avg Completeness	0.798	0.720	Single
Avg Depth	0.658	0.590	Single
Hallucination Rate	24%	4%	Multi ✓
Success Rate (rel ≥ 0.70)	78%	78%	Tied
Avg Latency	5.5s	307s	Single

Key Finding

The multi-agent pipeline is 6× more factually reliable (4% vs 24% hallucination). The Planner → Researcher → Analyst chain forces grounding before the Writer ever produces output — no claim goes unverified.

Single agent wins on speed and structural quality metrics for straightforward queries. The real tradeoff is latency vs trust, not latency vs quality.

---

Architecture

User Query
    │
    ├── Single Agent (ReAct Loop)
    │       └── llama-3.3-70b  ──►  Tavily Search  ──►  FAISS Retrieval
    │                                                        └── Response
    │
    └── Multi-Agent Pipeline (LangGraph StateGraph)
            │
            ├── 1. Planner      → Pydantic-structured subtasks + search queries
            │                     (llama-3.3-70b + structured output schema)
            │
            ├── 2. Researcher   → Parallel web search + FAISS vector retrieval
            │                     (llama-3.1-8b + Tavily + all-MiniLM-L6-v2)
            │
            ├── 3. Analyst      → Synthesises findings, scores confidence
            │                     (insight extraction + optional Python REPL)
            │
            ├── 4. Writer       → Structured markdown report with citations
            │                     (title, body, word count, sources)
            │
            └── 5. Memory       → Persists session to SQLite (SQLiteDict)
                                  (cross-session recall)

Both outputs evaluated by:
    LLM-as-Judge (llama-3.1-8b-instant)
        └── Relevance · Hallucination · Coherence · Completeness · Depth

---

Dashboard

Two panels in one animated Streamlit app:

Live Query — type any question, both agents respond in real time with a live pipeline progress bar. LLM-as-judge scores the responses on submission.

Benchmarks — 7 interactive Plotly charts built from real evaluated data:

Chart	What it shows
5-Metric Radar	All metrics for both agents on one chart
Category Breakdown	Avg relevance by topic (10 categories × 2 agents)
Relevance Trend	Query-by-query score progression across all 50
Latency vs Relevance	Scatter — every query as a dot, log-scale x-axis
Win Distribution	Donut — who scored higher per query
Hallucination Gauges	Dual dials showing 4% vs 24%
Latency Distribution	Bucketed bar chart of response times

---

Evaluation Metrics

The LLM-as-judge evaluates every (query, response) pair on:

Metric	Scale	Measures
Relevance	0 – 1	Does it directly answer the question?
Hallucination	No / Possible / Yes	Are all claims grounded in sources?
Coherence	0 – 1	Is it logically structured and readable?
Completeness	0 – 1	Does it cover all key sub-topics?
Depth	0 – 1	Does it explain how and why, not just what?

---

Project Structure

AgentBench/
│
├── app.py                  # Streamlit dashboard (Live Query + Benchmarks)
├── graph.py                # LangGraph StateGraph pipeline definition
├── evaluator.py            # LLM-as-judge (5-metric scoring)
├── bench_runner.py         # Resumable benchmark runner (50 queries)
├── bench_results.json      # Real evaluated results
│
├── agents/
│   ├── planner.py          # Query decomposition (Pydantic + llama-3.3-70b)
│   ├── researcher.py       # Web search + FAISS retrieval (llama-3.1-8b)
│   ├── analyst.py          # Research synthesis + confidence scoring
│   ├── writer.py           # Structured report generation
│   └── single_agent.py     # ReAct baseline agent
│
├── tools/
│   ├── web_search.py       # Tavily search wrapper
│   ├── vector_store.py     # FAISS store (all-MiniLM-L6-v2 embeddings)
│   └── python_repl.py      # Sandboxed Python executor
│
├── memory/
│   └── store.py            # SQLiteDict session memory
│
├── requirements.txt
└── PROBLEMS_FACED.md       # 15 real engineering problems + fixes

---

Setup

# 1. Clone the repo
git clone https://github.com/Adityax-07/AgentBench.git
cd AgentBench

# 2. Install dependencies
pip install -r requirements.txt

# 3. Add API keys
# Create a .env file with:
GROQ_API_KEY=your_groq_api_key
TAVILY_API_KEY=your_tavily_api_key

# 4. Run the dashboard
streamlit run app.py

# 5. (Optional) Run the full benchmark
python bench_runner.py

Note: The benchmark runner is resumable — if Groq's rate limit cuts it off, re-run and it picks up from where it stopped.

---

Tech Stack

Layer	Technology
Agent orchestration	LangGraph StateGraph
LLM provider	Groq (llama-3.3-70b-versatile, llama-3.1-8b-instant)
Web search	Tavily Search API
Vector retrieval	FAISS + HuggingFace all-MiniLM-L6-v2
Structured outputs	Pydantic + LangChain .with_structured_output()
Session memory	SQLiteDict
Dashboard	Streamlit + Plotly
Language	Python 3.10+

---

Numbers at a Glance

50 queries evaluated       5 metrics per response      2 agents compared
6× lower hallucination     3,500+ lines of code        15 files, 7 modules
7 interactive charts       8 animated metric cards     100 LLM-judged responses

---

Engineering Notes

PROBLEMS_FACED.md documents 15 real engineering problems hit during development — Groq rate limit handling, LangGraph state serialization, Streamlit session resets, stream_mode format differences, and more. Each entry has the root cause, the fix, and an interview talking point.