README.md

---
title: Agentbee | GAIA Project | HuggingFace Course
emoji: 🕵🏻‍♂️
colorFrom: indigo
colorTo: indigo
sdk: gradio
sdk_version: 6.3.0
app_file: app.py
pinned: false
hf_oauth: true
hf_oauth_expiration_minutes: 480
---

Check out the configuration reference at <https://huggingface.co/docs/hub/spaces-config-reference>

## Project Overview

**Project Name:** Final_Assignment_Template

**Purpose:** Course assignment template for building an AI agent that passes the GAIA benchmark (General AI Assistants). This project serves as a learning-focused workspace to support iterative agent development and experimentation.

**Target Users:** Students learning agent development through hands-on implementation

**Key Objectives:**

- Build production-ready code that passes GAIA test requirements
- Learn agent development through discovery-based implementation
- Develop systematic approach to complex AI task solving
- Document learning process and key decisions

## Project Architecture

**Technology Stack:**

- **Platform:** Hugging Face Spaces with OAuth integration
- **UI Framework:** Gradio 5.x with OAuth support
- **Agent Framework:** LangGraph (state graph orchestration)
- **LLM Providers (4-tier fallback):**
  - Google Gemini 2.0 Flash (free tier)
  - HuggingFace Inference API (free tier)
  - Groq (Llama 3.1 70B / Qwen 3 32B, free tier)
  - Anthropic Claude Sonnet 4.5 (paid tier)
- **Tools:**
  - Web Search: Tavily API / Exa API
  - File Parser: PyPDF2, openpyxl, python-docx, pillow
  - Calculator: Safe expression evaluator
  - Vision: Multimodal LLM (Gemini/Claude)
- **Language:** Python 3.12+
- **Package Manager:** uv

**Project Structure:**

```
Final_Assignment_Template/
├── archive/             # Reference materials, previous solutions, static resources
├── input/               # Input files, configuration, raw data
├── output/              # Generated files, results, processed data
├── test/                # Testing files, test scripts (99 tests)
├── dev/                 # Development records (permanent knowledge packages)
├── src/                 # Source code
│   ├── agent/           # Agent orchestration
│   │   ├── graph.py     # LangGraph state machine
│   │   └── llm_client.py # Multi-provider LLM integration with retry logic
│   └── tools/           # Agent tools
│       ├── __init__.py  # Tool registry
│       ├── web_search.py    # Tavily/Exa web search
│       ├── file_parser.py   # Multi-format file reader
│       ├── calculator.py    # Safe math evaluator
│       └── vision.py        # Multimodal image/video analysis
├── app.py               # Gradio UI with OAuth, LLM provider selection
├── pyproject.toml       # uv package management
├── requirements.txt     # Python dependencies (generated from pyproject.toml)
├── .env                 # Local environment variables (API keys, config)
├── README.md            # Project overview, architecture, workflow, specification
├── CLAUDE.md            # Project-specific AI instructions
├── PLAN.md              # Active implementation plan (temporary workspace)
├── TODO.md              # Active task tracking (temporary workspace)
└── CHANGELOG.md         # Session changelog (temporary workspace)
```

**Core Components:**

- **GAIAAgent class** (src/agent/graph.py): LangGraph-based agent with state machine orchestration
  - Planning node: Analyze question and generate execution plan
  - Tool selection node: LLM function calling for dynamic tool selection
  - Tool execution node: Execute selected tools with timeout and error handling
  - Answer synthesis node: Generate factoid answer from evidence
- **LLM Client** (src/agent/llm_client.py): Multi-provider LLM integration
  - 4-tier fallback chain: Gemini → HuggingFace → Groq → Claude
  - Exponential backoff retry logic (3 attempts per provider)
  - Runtime config for UI-based provider selection
  - Few-shot prompting for improved tool selection
- **Tool System** (src/tools/):
  - Web Search: Tavily/Exa API with query optimization
  - File Parser: Multi-format support (PDF, Excel, Word, CSV, images)
  - Calculator: Safe expression evaluator with graceful error handling
  - Vision: Multimodal analysis for images/videos
- **Gradio UI** (app.py):
  - Test & Debug tab: Single question testing with LLM provider dropdown
  - Full Evaluation tab: Run all GAIA questions with provider selection
  - Results export: JSON file download for analysis
  - OAuth integration for submission
- **Evaluation Infrastructure**: Pre-built orchestration (question fetching, submission, scoring)

**System Architecture Diagram:**

```mermaid
---
config:
  layout: elk
---
graph TB
    subgraph "UI Layer"
        GradioUI[Gradio UI<br/>LLM Provider Selection<br/>Test & Full Evaluation]
        OAuth[HF OAuth<br/>User authentication]
    end

    subgraph "Agent Orchestration (LangGraph)"
        GAIAAgent[GAIAAgent<br/>State Machine]
        PlanNode[Planning Node<br/>Analyze question]
        ToolSelectNode[Tool Selection Node<br/>LLM function calling]
        ToolExecNode[Tool Execution Node<br/>Run selected tools]
        SynthesizeNode[Answer Synthesis Node<br/>Generate factoid]
    end

    subgraph "LLM Layer (4-Tier Fallback)"
        LLMClient[LLM Client<br/>Retry + Fallback]
        Gemini[Gemini 2.0 Flash<br/>Free Tier 1]
        HF[HuggingFace API<br/>Free Tier 2]
        Groq[Groq Llama/Qwen<br/>Free Tier 3]
        Claude[Claude Sonnet 4.5<br/>Paid Tier 4]
    end

    subgraph "Tool Layer"
        WebSearch[Web Search<br/>Tavily/Exa]
        FileParser[File Parser<br/>PDF/Excel/Word]
        Calculator[Calculator<br/>Safe eval]
        Vision[Vision<br/>Multimodal LLM]
    end

    subgraph "External Services"
        API[GAIA Scoring API]
        QEndpoint["/questions endpoint"]
        SEndpoint["/submit endpoint"]
    end

    GradioUI --> OAuth
    OAuth -->|Authenticated| GAIAAgent
    GAIAAgent --> PlanNode
    PlanNode --> ToolSelectNode
    ToolSelectNode --> ToolExecNode
    ToolExecNode --> SynthesizeNode

    PlanNode --> LLMClient
    ToolSelectNode --> LLMClient
    SynthesizeNode --> LLMClient

    LLMClient -->|Try 1| Gemini
    LLMClient -->|Fallback 2| HF
    LLMClient -->|Fallback 3| Groq
    LLMClient -->|Fallback 4| Claude

    ToolExecNode --> WebSearch
    ToolExecNode --> FileParser
    ToolExecNode --> Calculator
    ToolExecNode --> Vision

    GAIAAgent -->|Answers| API
    API --> QEndpoint
    API --> SEndpoint
    SEndpoint -->|Score| GradioUI

    style GAIAAgent fill:#ffcccc
    style LLMClient fill:#fff4cc
    style GradioUI fill:#cce5ff
    style API fill:#d9f2d9
```

## Project Specification

**Project Context:**

This is a course assignment template for building an AI agent that passes the GAIA benchmark (General AI Assistants). The project was recently started as a learning-focused workspace to support iterative agent development and experimentation.

**Current State:**

- **Status:** Stage 5 Complete - Performance Optimization
- **Development Progress:**
  - Stage 1-2: Basic infrastructure and LangGraph setup ✅
  - Stage 3: Multi-provider LLM integration ✅
  - Stage 4: Tool system and MVP (10% GAIA score: 2/20 questions) ✅
  - Stage 5: Performance optimization (retry logic, Groq integration, improved prompts) ✅
- **Current Performance:** Testing in progress (target: 25% accuracy, 5/20 questions)
- **Next Stage:** Stage 6 - Advanced optimizations based on Stage 5 results

**Data & Workflows:**

- **Input Data:** GAIA test questions fetched from external scoring API (`agents-course-unit4-scoring.hf.space`)
- **Processing:** BasicAgent class processes questions and generates answers
- **Output:** Agent responses submitted to scoring endpoint for evaluation
- **Development Workflow:**
  1. Local development and testing
  2. Deploy to Hugging Face Space
  3. Submit via integrated evaluation UI

**User Workflow Diagram:**

```mermaid
---
config:
  layout: fixed
---
flowchart TB
    Start(["Student starts assignment"]) --> Clone["Clone HF Space template"]
    Clone --> LocalDev["Local development:<br>Implement BasicAgent logic"]
    LocalDev --> LocalTest{"Test locally?"}
    LocalTest -- Yes --> RunLocal["Run app locally"]
    RunLocal --> Debug{"Works?"}
    Debug -- No --> LocalDev
    Debug -- Yes --> Deploy["Deploy to HF Space"]
    LocalTest -- Skip --> Deploy
    Deploy --> Login["Login with HF OAuth"]
    Login --> RunEval@{ label: "Click 'Run Evaluation'<br>button in UI" }
    RunEval --> FetchQ["System fetches GAIA<br>questions from API"]
    FetchQ --> RunAgent["Agent processes<br>each question"]
    RunAgent --> Submit["Submit answers<br>to scoring API"]
    Submit --> Display["Display score<br>and results"]
    Display --> Iterate{"Satisfied with<br>score?"}
    Iterate -- "No - improve agent" --> LocalDev
    Iterate -- Yes --> Complete(["Assignment complete"])

    RunEval@{ shape: rect}
    style Start fill:#e1f5e1
    style LocalDev fill:#fff4e1
    style Deploy fill:#e1f0ff
    style RunAgent fill:#ffe1f0
    style Complete fill:#e1f5e1
```

**Technical Architecture:**

- **Platform:** Hugging Face Spaces with OAuth integration
- **Framework:** Gradio for UI, Requests for API communication
- **Core Component:** BasicAgent class (student-customizable template)
- **Evaluation Infrastructure:** Pre-built orchestration (question fetching, submission, scoring display)
- **Deployment:** HF Space with environment variables (SPACE_ID, SPACE_HOST)

**Requirements & Constraints:**

- **Constraint Type:** Minimal at current stage
- **Infrastructure:** Must run on Hugging Face Spaces platform
- **Integration:** Fixed scoring API endpoints (cannot modify evaluation system)
- **Flexibility:** Students have full freedom to design agent capabilities

**Integration Points:**

- **External API:** `https://agents-course-unit4-scoring.hf.space`
  - `/questions` endpoint: Fetch GAIA test questions
  - `/submit` endpoint: Submit answers and receive scores
- **Authentication:** Hugging Face OAuth for student identification
- **Deployment:** HF Space runtime environment variables

**Development Goals:**

- **Primary:** Achieve competitive GAIA benchmark performance through systematic optimization
- **Focus:** Multi-tier LLM architecture with free-tier prioritization to minimize costs
- **Key Features:**
  - 4-tier LLM fallback for quota resilience (Gemini → HF → Groq → Claude)
  - Exponential backoff retry logic for quota/rate limit errors
  - UI-based LLM provider selection for easy A/B testing in cloud
  - Comprehensive tool system (web search, file parsing, calculator, vision)
  - Graceful error handling and degradation
  - Extensive test coverage (99 tests)
- **Documentation:** Full dev record workflow tracking all decisions and changes

## Key Features

### LLM Provider Selection (UI-Based)

**Local Testing (.env configuration):**

```bash
LLM_PROVIDER=gemini          # Options: gemini, huggingface, groq, claude
ENABLE_LLM_FALLBACK=false    # Disable fallback for debugging single provider
```

**Cloud Testing (HuggingFace Spaces):**

- Use UI dropdowns in Test & Debug tab or Full Evaluation tab
- Select from: Gemini, HuggingFace, Groq, Claude
- Toggle fallback behavior with checkbox
- No environment variable changes needed, instant provider switching

**Benefits:**

- Easy A/B testing between providers
- Clear visibility which LLM is used
- Isolated testing for debugging
- Production safety with fallback enabled

### Retry Logic

- **Exponential backoff:** 3 attempts with 1s, 2s, 4s delays
- **Error detection:** 429 status, quota errors, rate limits
- **Scope:** All LLM calls (planning, tool selection, synthesis)

### Tool System

**Web Search (Tavily/Exa):**

- Factual information, current events, statistics
- Wikipedia, company info, people

**File Parser:**

- PDF, Excel, Word, CSV, Text, Images
- Handles uploaded files and local paths

**Calculator:**

- Safe expression evaluation
- Arithmetic, algebra, trigonometry, logarithms
- Functions: sqrt, sin, cos, log, abs, etc.

**Vision:**

- Multimodal image/video analysis
- Describe content, identify objects, read text
- YouTube video understanding

### Performance Optimizations (Stage 5)

- Few-shot prompting for improved tool selection
- Graceful vision question skip when quota exhausted
- Relaxed calculator validation (returns error dicts instead of crashes)
- Improved tool descriptions with "Use when..." guidance
- Config-based provider debugging

## GAIA Benchmark Results

**Baseline (Stage 4):** 10% accuracy (2/20 questions correct)

**Stage 5 Target:** 25% accuracy (5/20 questions correct)

- Status: Testing in progress
- Expected improvements from retry logic, Groq integration, improved prompts

**Test Coverage:** 99 passing tests (~2min 40sec runtime)

> **Note:** This project implements the **Course Leaderboard** (20 questions, 30% target). See [GAIA Submission Guide](../agentbee/docs/gaia_submission_guide.md) for distinction between Course and Official GAIA leaderboards.

## Workflow

### Dev Record Workflow

**Philosophy:** Dev records are the single source of truth. CHANGELOG/PLAN/TODO are temporary workspace files.

**Dev Record Types:**

- 🐞 **Issue:** Problem-solving, bug fixes, error resolution
- 🔨 **Development:** Feature development, enhancements, new functionality

### Session Start Workflow

#### Phase 1: Planning (Explicit)

1. **Create or identify dev record:** `dev/dev_YYMMDD_##_concise_title.md`
   - Choose type: 🐞 Issue or 🔨 Development
2. **Create PLAN.md ONLY:** Use `/plan` command or write directly
   - Document implementation approach, steps, files to modify
   - DO NOT create TODO.md or CHANGELOG.md yet

#### Phase 2: Development (Automatic)

3. **Create TODO.md:** Automatically populate as you start implementing
   - Track tasks in real-time using TodoWrite tool
   - Mark in_progress/completed as you work
4. **Create CHANGELOG.md:** Automatically populate as you make changes
   - Record file modifications/creations/deletions as they happen
5. **Work on solution:** Update all three files during development

### Session End Workflow

#### Phase 3: Completion (Manual)

After AI completes all work and updates PLAN/TODO/CHANGELOG:

- AI stops and waits for user review (Checkpoint 3)
- User reviews PLAN.md, TODO.md, and CHANGELOG.md
- User manually runs `/update-dev dev_YYMMDD_##` when satisfied

When /update-dev runs:

1. Distills PLAN decisions → dev record "Key Decisions" section
2. Distills TODO deliverables → dev record "Outcome" section
3. Distills CHANGELOG changes → dev record "Changelog" section
4. Empties PLAN.md, TODO.md, CHANGELOG.md back to templates
5. Marks dev record status as ✅ Resolved

### AI Context Loading Protocol

**MANDATORY - Execute in exact order. NO delegating to sub-agents for initial context.**

**Phase 1: Current State (What's happening NOW)**

1. **Read workspace files:**

   - `CHANGELOG.md` - Active session changes (reverse chronological, newest first)
   - `PLAN.md` - Current implementation plan (if exists)
   - `TODO.md` - Active task tracking (if exists)

2. **Read actual outputs (CRITICAL - verify claims, don't trust summaries):**
   - Latest files in `output/` folder (sorted by timestamp, newest first)
   - For GAIA projects: Read latest `output/gaia_results_*.json` completely
     - Check `metadata.score_percent` and `metadata.correct_count`
     - Read ALL `results[].submitted_answer` to understand failure patterns
     - Identify error categories (vision failures, tool errors, wrong answers)
   - For test projects: Read latest test output logs
   - **Purpose:** Ground truth of what ACTUALLY happened, not what was claimed

**Phase 2: Recent History (What was done recently)**

3. **Read last 3 dev records from `dev/` folder:**
   - Sort by filename (newest `dev_YYMMDD_##_title.md` first)
   - Read: Problem Description, Key Decisions, Outcome, Changelog
   - **Cross-verify:** Compare dev record claims with actual output files
   - **Red flag:** If dev record says "25% accuracy" but latest JSON shows "0%", prioritize JSON truth

**Phase 3: Project Structure (How it works)**

4. **Read README.md sections in order:**

   - Section 1: Overview (purpose, objectives)
   - Section 2: Architecture (tech stack, components, diagrams)
   - Section 3: Specification (current state, workflows, requirements)
   - Section 4: Workflow (this protocol)

5. **Read CLAUDE.md:**
   - Project-specific coding standards
   - Usually empty (inherits from global ~/.claude/CLAUDE.md)

**Phase 4: Code Structure (Critical files)**

6. **Identify critical files from README.md Architecture section:**
   - Note main entry points (e.g., `app.py`)
   - Note core logic files (e.g., `src/agent/graph.py`, `src/agent/llm_client.py`)
   - Note tool implementations (e.g., `src/tools/*.py`)
   - **DO NOT read these yet** - only note their locations for later reference

**Verification Checklist (Before claiming "I have context"):**

- [ ] I personally read CHANGELOG.md, PLAN.md, TODO.md (not delegated)
- [ ] I personally read latest output files (JSON results, test logs, etc.)
- [ ] I know the ACTUAL current accuracy/status from output files
- [ ] I read last 3 dev records and cross-verified claims with output data
- [ ] I read README.md sections 1-4 completely
- [ ] I can answer: "What is the current status and why?"
- [ ] I can answer: "What were the last 3 major changes and their outcomes?"
- [ ] I can answer: "What specific problems exist based on latest outputs?"

**Anti-Patterns (NEVER do these):**

- ❌ Delegate initial context loading to Explore/Task agents
- ❌ Trust dev record claims without verifying against output files
- ❌ Skip reading actual output data (JSON results, logs, test outputs)
- ❌ Claim "I have context" after only reading summaries
- ❌ Read code files before understanding current state from outputs

**Context Priority:** Latest Outputs (ground truth) > CHANGELOG (active work) > Dev Records (history) > README (structure)