--- title: AgentBench emoji: ⚡ colorFrom: blue colorTo: indigo sdk: streamlit sdk_version: 1.41.0 app_file: app.py pinned: false ---
# ⚡ AgentBench ### Multi-Agent Research Evaluation System *A production-structured LangGraph pipeline that benchmarks multi-agent vs single-agent LLM responses across 50 queries on 5 real evaluation metrics*
[![Python](https://img.shields.io/badge/Python-3.10+-3776AB?style=for-the-badge&logo=python&logoColor=white)](https://python.org) [![LangGraph](https://img.shields.io/badge/LangGraph-0.2+-1C3C3C?style=for-the-badge&logo=langchain&logoColor=white)](https://langchain-ai.github.io/langgraph) [![Streamlit](https://img.shields.io/badge/Streamlit-1.x-FF4B4B?style=for-the-badge&logo=streamlit&logoColor=white)](https://streamlit.io) [![Groq](https://img.shields.io/badge/Groq-llama--3.3--70b-F55036?style=for-the-badge)](https://groq.com) [![FAISS](https://img.shields.io/badge/FAISS-Vector_Store-0467DF?style=for-the-badge)](https://faiss.ai)
--- ## What Is This? Most AI demos show you an output and say "looks good." AgentBench actually **measures** it. It runs two approaches on the same query — a single ReAct agent and a 5-node multi-agent pipeline — then scores both using a real **LLM-as-judge** evaluator on 5 metrics. Every result is computed, not hardcoded. The core question it answers: *Does splitting reasoning, research, analysis, and writing across specialised agents produce more trustworthy responses than one agent doing everything?* **Answer: Yes — 6× lower hallucination. Here are the numbers.** --- ## Benchmark Results > Evaluated on **50 queries** across GenAI, ML fundamentals, agentic AI, retrieval, and applied AI. > Judged by `llama-3.1-8b-instant` as an independent LLM evaluator. | Metric | Single Agent | Multi-Agent | Winner | |:---|:---:|:---:|:---:| | Avg Relevance | 0.877 | 0.850 | Single | | Avg Coherence | **0.912** | 0.900 | Single | | Avg Completeness | **0.798** | 0.720 | Single | | Avg Depth | **0.658** | 0.590 | Single | | Hallucination Rate | 24% | **4%** | **Multi ✓** | | Success Rate (rel ≥ 0.70) | 78% | 78% | Tied | | Avg Latency | **5.5s** | 307s | Single | ### Key Finding The multi-agent pipeline is **6× more factually reliable** (4% vs 24% hallucination). The Planner → Researcher → Analyst chain forces grounding before the Writer ever produces output — no claim goes unverified. Single agent wins on speed and structural quality metrics for straightforward queries. The real tradeoff is **latency vs trust**, not latency vs quality. --- ## Architecture ``` User Query │ ├── Single Agent (ReAct Loop) │ └── llama-3.3-70b ──► Tavily Search ──► FAISS Retrieval │ └── Response │ └── Multi-Agent Pipeline (LangGraph StateGraph) │ ├── 1. Planner → Pydantic-structured subtasks + search queries │ (llama-3.3-70b + structured output schema) │ ├── 2. Researcher → Parallel web search + FAISS vector retrieval │ (llama-3.1-8b + Tavily + all-MiniLM-L6-v2) │ ├── 3. Analyst → Synthesises findings, scores confidence │ (insight extraction + optional Python REPL) │ ├── 4. Writer → Structured markdown report with citations │ (title, body, word count, sources) │ └── 5. Memory → Persists session to SQLite (SQLiteDict) (cross-session recall) Both outputs evaluated by: LLM-as-Judge (llama-3.1-8b-instant) └── Relevance · Hallucination · Coherence · Completeness · Depth ``` --- ## Dashboard Two panels in one animated Streamlit app: **Live Query** — type any question, both agents respond in real time with a live pipeline progress bar. LLM-as-judge scores the responses on submission. **Benchmarks** — 7 interactive Plotly charts built from real evaluated data: | Chart | What it shows | |:---|:---| | 5-Metric Radar | All metrics for both agents on one chart | | Category Breakdown | Avg relevance by topic (10 categories × 2 agents) | | Relevance Trend | Query-by-query score progression across all 50 | | Latency vs Relevance | Scatter — every query as a dot, log-scale x-axis | | Win Distribution | Donut — who scored higher per query | | Hallucination Gauges | Dual dials showing 4% vs 24% | | Latency Distribution | Bucketed bar chart of response times | --- ## Evaluation Metrics The LLM-as-judge evaluates every (query, response) pair on: | Metric | Scale | Measures | |:---|:---:|:---| | **Relevance** | 0 – 1 | Does it directly answer the question? | | **Hallucination** | No / Possible / Yes | Are all claims grounded in sources? | | **Coherence** | 0 – 1 | Is it logically structured and readable? | | **Completeness** | 0 – 1 | Does it cover all key sub-topics? | | **Depth** | 0 – 1 | Does it explain *how* and *why*, not just *what*? | --- ## Project Structure ``` AgentBench/ │ ├── app.py # Streamlit dashboard (Live Query + Benchmarks) ├── graph.py # LangGraph StateGraph pipeline definition ├── evaluator.py # LLM-as-judge (5-metric scoring) ├── bench_runner.py # Resumable benchmark runner (50 queries) ├── bench_results.json # Real evaluated results │ ├── agents/ │ ├── planner.py # Query decomposition (Pydantic + llama-3.3-70b) │ ├── researcher.py # Web search + FAISS retrieval (llama-3.1-8b) │ ├── analyst.py # Research synthesis + confidence scoring │ ├── writer.py # Structured report generation │ └── single_agent.py # ReAct baseline agent │ ├── tools/ │ ├── web_search.py # Tavily search wrapper │ ├── vector_store.py # FAISS store (all-MiniLM-L6-v2 embeddings) │ └── python_repl.py # Sandboxed Python executor │ ├── memory/ │ └── store.py # SQLiteDict session memory │ ├── requirements.txt └── PROBLEMS_FACED.md # 15 real engineering problems + fixes ``` --- ## Setup ```bash # 1. Clone the repo git clone https://github.com/Adityax-07/AgentBench.git cd AgentBench # 2. Install dependencies pip install -r requirements.txt # 3. Add API keys # Create a .env file with: GROQ_API_KEY=your_groq_api_key TAVILY_API_KEY=your_tavily_api_key # 4. Run the dashboard streamlit run app.py # 5. (Optional) Run the full benchmark python bench_runner.py ``` > **Note:** The benchmark runner is resumable — if Groq's rate limit cuts it off, re-run and it picks up from where it stopped. --- ## Tech Stack | Layer | Technology | |:---|:---| | Agent orchestration | LangGraph `StateGraph` | | LLM provider | Groq (`llama-3.3-70b-versatile`, `llama-3.1-8b-instant`) | | Web search | Tavily Search API | | Vector retrieval | FAISS + HuggingFace `all-MiniLM-L6-v2` | | Structured outputs | Pydantic + LangChain `.with_structured_output()` | | Session memory | SQLiteDict | | Dashboard | Streamlit + Plotly | | Language | Python 3.10+ | --- ## Numbers at a Glance ``` 50 queries evaluated 5 metrics per response 2 agents compared 6× lower hallucination 3,500+ lines of code 15 files, 7 modules 7 interactive charts 8 animated metric cards 100 LLM-judged responses ``` --- ## Engineering Notes [PROBLEMS_FACED.md](PROBLEMS_FACED.md) documents **15 real engineering problems** hit during development — Groq rate limit handling, LangGraph state serialization, Streamlit session resets, `stream_mode` format differences, and more. Each entry has the root cause, the fix, and an interview talking point.