README.md

---
title: AgentBench
emoji: ⚡
colorFrom: blue
colorTo: indigo
sdk: streamlit
sdk_version: 1.41.0
app_file: app.py
pinned: false
---

<div align="center">

# ⚡ AgentBench

### Multi-Agent Research Evaluation System

*A production-structured LangGraph pipeline that benchmarks multi-agent vs single-agent LLM responses across 50 queries on 5 real evaluation metrics*

<br/>

[![Python](https://img.shields.io/badge/Python-3.10+-3776AB?style=for-the-badge&logo=python&logoColor=white)](https://python.org)
[![LangGraph](https://img.shields.io/badge/LangGraph-0.2+-1C3C3C?style=for-the-badge&logo=langchain&logoColor=white)](https://langchain-ai.github.io/langgraph)
[![Streamlit](https://img.shields.io/badge/Streamlit-1.x-FF4B4B?style=for-the-badge&logo=streamlit&logoColor=white)](https://streamlit.io)
[![Groq](https://img.shields.io/badge/Groq-llama--3.3--70b-F55036?style=for-the-badge)](https://groq.com)
[![FAISS](https://img.shields.io/badge/FAISS-Vector_Store-0467DF?style=for-the-badge)](https://faiss.ai)

<br/>

</div>

---

## What Is This?

Most AI demos show you an output and say "looks good." AgentBench actually **measures** it.

It runs two approaches on the same query — a single ReAct agent and a 5-node multi-agent pipeline — then scores both using a real **LLM-as-judge** evaluator on 5 metrics. Every result is computed, not hardcoded.

The core question it answers: *Does splitting reasoning, research, analysis, and writing across specialised agents produce more trustworthy responses than one agent doing everything?*

**Answer: Yes — 6× lower hallucination. Here are the numbers.**

---

## Benchmark Results

> Evaluated on **50 queries** across GenAI, ML fundamentals, agentic AI, retrieval, and applied AI.
> Judged by `llama-3.1-8b-instant` as an independent LLM evaluator.

| Metric | Single Agent | Multi-Agent | Winner |
|:---|:---:|:---:|:---:|
| Avg Relevance | 0.877 | 0.850 | Single |
| Avg Coherence | **0.912** | 0.900 | Single |
| Avg Completeness | **0.798** | 0.720 | Single |
| Avg Depth | **0.658** | 0.590 | Single |
| Hallucination Rate | 24% | **4%** | **Multi ✓** |
| Success Rate (rel ≥ 0.70) | 78% | 78% | Tied |
| Avg Latency | **5.5s** | 307s | Single |

### Key Finding

The multi-agent pipeline is **6× more factually reliable** (4% vs 24% hallucination). The Planner → Researcher → Analyst chain forces grounding before the Writer ever produces output — no claim goes unverified.

Single agent wins on speed and structural quality metrics for straightforward queries. The real tradeoff is **latency vs trust**, not latency vs quality.

---

## Architecture

```
User Query
    │
    ├── Single Agent (ReAct Loop)
    │       └── llama-3.3-70b  ──►  Tavily Search  ──►  FAISS Retrieval
    │                                                        └── Response
    │
    └── Multi-Agent Pipeline (LangGraph StateGraph)
            │
            ├── 1. Planner      → Pydantic-structured subtasks + search queries
            │                     (llama-3.3-70b + structured output schema)
            │
            ├── 2. Researcher   → Parallel web search + FAISS vector retrieval
            │                     (llama-3.1-8b + Tavily + all-MiniLM-L6-v2)
            │
            ├── 3. Analyst      → Synthesises findings, scores confidence
            │                     (insight extraction + optional Python REPL)
            │
            ├── 4. Writer       → Structured markdown report with citations
            │                     (title, body, word count, sources)
            │
            └── 5. Memory       → Persists session to SQLite (SQLiteDict)
                                  (cross-session recall)

Both outputs evaluated by:
    LLM-as-Judge (llama-3.1-8b-instant)
        └── Relevance · Hallucination · Coherence · Completeness · Depth
```

---

## Dashboard

Two panels in one animated Streamlit app:

**Live Query** — type any question, both agents respond in real time with a live pipeline progress bar. LLM-as-judge scores the responses on submission.

**Benchmarks** — 7 interactive Plotly charts built from real evaluated data:

| Chart | What it shows |
|:---|:---|
| 5-Metric Radar | All metrics for both agents on one chart |
| Category Breakdown | Avg relevance by topic (10 categories × 2 agents) |
| Relevance Trend | Query-by-query score progression across all 50 |
| Latency vs Relevance | Scatter — every query as a dot, log-scale x-axis |
| Win Distribution | Donut — who scored higher per query |
| Hallucination Gauges | Dual dials showing 4% vs 24% |
| Latency Distribution | Bucketed bar chart of response times |

---

## Evaluation Metrics

The LLM-as-judge evaluates every (query, response) pair on:

| Metric | Scale | Measures |
|:---|:---:|:---|
| **Relevance** | 0 – 1 | Does it directly answer the question? |
| **Hallucination** | No / Possible / Yes | Are all claims grounded in sources? |
| **Coherence** | 0 – 1 | Is it logically structured and readable? |
| **Completeness** | 0 – 1 | Does it cover all key sub-topics? |
| **Depth** | 0 – 1 | Does it explain *how* and *why*, not just *what*? |

---

## Project Structure

```
AgentBench/
│
├── app.py                  # Streamlit dashboard (Live Query + Benchmarks)
├── graph.py                # LangGraph StateGraph pipeline definition
├── evaluator.py            # LLM-as-judge (5-metric scoring)
├── bench_runner.py         # Resumable benchmark runner (50 queries)
├── bench_results.json      # Real evaluated results
│
├── agents/
│   ├── planner.py          # Query decomposition (Pydantic + llama-3.3-70b)
│   ├── researcher.py       # Web search + FAISS retrieval (llama-3.1-8b)
│   ├── analyst.py          # Research synthesis + confidence scoring
│   ├── writer.py           # Structured report generation
│   └── single_agent.py     # ReAct baseline agent
│
├── tools/
│   ├── web_search.py       # Tavily search wrapper
│   ├── vector_store.py     # FAISS store (all-MiniLM-L6-v2 embeddings)
│   └── python_repl.py      # Sandboxed Python executor
│
├── memory/
│   └── store.py            # SQLiteDict session memory
│
├── requirements.txt
└── PROBLEMS_FACED.md       # 15 real engineering problems + fixes
```

---

## Setup

```bash
# 1. Clone the repo
git clone https://github.com/Adityax-07/AgentBench.git
cd AgentBench

# 2. Install dependencies
pip install -r requirements.txt

# 3. Add API keys
# Create a .env file with:
GROQ_API_KEY=your_groq_api_key
TAVILY_API_KEY=your_tavily_api_key

# 4. Run the dashboard
streamlit run app.py

# 5. (Optional) Run the full benchmark
python bench_runner.py
```

> **Note:** The benchmark runner is resumable — if Groq's rate limit cuts it off, re-run and it picks up from where it stopped.

---

## Tech Stack

| Layer | Technology |
|:---|:---|
| Agent orchestration | LangGraph `StateGraph` |
| LLM provider | Groq (`llama-3.3-70b-versatile`, `llama-3.1-8b-instant`) |
| Web search | Tavily Search API |
| Vector retrieval | FAISS + HuggingFace `all-MiniLM-L6-v2` |
| Structured outputs | Pydantic + LangChain `.with_structured_output()` |
| Session memory | SQLiteDict |
| Dashboard | Streamlit + Plotly |
| Language | Python 3.10+ |

---

## Numbers at a Glance

```
50 queries evaluated       5 metrics per response      2 agents compared
6× lower hallucination     3,500+ lines of code        15 files, 7 modules
7 interactive charts       8 animated metric cards     100 LLM-judged responses
```

---

## Engineering Notes

[PROBLEMS_FACED.md](PROBLEMS_FACED.md) documents **15 real engineering problems** hit during development — Groq rate limit handling, LangGraph state serialization, Streamlit session resets, `stream_mode` format differences, and more. Each entry has the root cause, the fix, and an interview talking point.