refactor: implement 3-tier folder naming convention

Implement clear 3-tier naming convention to distinguish user-only, runtime, and application folders.

3-Tier Convention:
1. User-only (user_* prefix): Manual use, not app runtime
- user_input/ - Testing files, manual data input
- user_output/ - Downloaded results, exports
- user_dev/ - Dev records, problem documentation
- user_archive/ - Archived code/reference materials

2. Runtime/Internal (_ prefix): App creates, temporary
- _cache/ - Runtime cache, served via app download
- _log/ - Runtime logs, debugging traces

3. Application (no prefix): Permanent code
- src/, test/, docs/ - Application code and tests

Folders renamed:
- _input/ → user_input/
- _output/ → user_output/
- dev/ → user_dev/
- archive/ → user_archive/

Updated files:
- test/test_phase0_hf_vision_api.py - Path("_output") → Path("user_output")
- .gitignore - Updated folder references and added comments
- CHANGELOG.md - Added 3-tier convention entry
- ~/.claude/CLAUDE.md - Updated Project Structure Standard with 3-tier naming

Rationale: Clear separation between user-managed folders (user_*), runtime folders (_), and application code (no prefix).

Co-Authored-By: Claude <noreply@anthropic.com>

.gitignore

@@ -27,26 +27,22 @@ uv.lock
 .DS_Store
 Thumbs.db
 
-# Input documents (PDFs not allowed in HF Spaces)
-_input/*.pdf
-_input/
+# User-only folders (manual use, not app runtime)
+user_input/
+user_output/
+user_dev/
+user_archive/
 
-# Downloaded GAIA question files (user testing)
-_input/*
-
-# Runtime cache (not in git, served via app download)
+# Runtime cache (app creates, temporary, served via app download)
 _cache/
 
-# Runtime logs (not in git, for debugging and analysis)
+# Runtime logs (app creates, temporary, for debugging)
 _log/
 
-# Runtime results (not in git, for user download)
-_output/
-
-# Reference materials (not in git, static copies)
+# Reference materials (static copies, not in git)
 ref/
 
-# Documentation folder (not in git, external storage)
+# Documentation folder (external storage)
 docs/
 
 # Testing

CHANGELOG.md

@@ -1,5 +1,48 @@
 # Session Changelog
 
+## [2026-01-13] [Infrastructure] [COMPLETED] 3-Tier Folder Naming Convention
+
+**Problem:** Previous rename used `_` prefix for both runtime folders AND user-only folders, creating ambiguity.
+
+**Solution:** Implemented 3-tier naming convention to clearly distinguish folder purposes.
+
+**3-Tier Convention:**
+1. **User-only** (`user_*` prefix) - Manual use, not app runtime:
+   - `user_input/` - User testing files, not app input
+   - `user_output/` - User downloads, not app output
+   - `user_dev/` - Dev records (manual documentation)
+   - `user_archive/` - Archived code/reference materials
+
+2. **Runtime/Internal** (`_` prefix) - App creates, temporary:
+   - `_cache/` - Runtime cache, served via app download
+   - `_log/` - Runtime logs, debugging
+
+3. **Application** (no prefix) - Permanent code:
+   - `src/`, `test/`, `docs/`, `ref/` - Application folders
+
+**Folders Renamed:**
+- `_input/` → `user_input/` (user testing files)
+- `_output/` → `user_output/` (user downloads)
+- `dev/` → `user_dev/` (dev records)
+- `archive/` → `user_archive/` (archived materials)
+
+**Folders Unchanged (correct tier):**
+- `_cache/`, `_log/` - Runtime ✓
+- `src/`, `test/`, `docs/`, `ref/` - Application ✓
+
+**Updated Files:**
+- **test/test_phase0_hf_vision_api.py** - `Path("_output")` → `Path("user_output")`
+- **.gitignore** - Updated folder references and comments
+
+**Git Status:**
+- Old folders removed from git tracking
+- New folders excluded by .gitignore
+- Existing files become untracked
+
+**Result:** Clear 3-tier structure: user_*, _*, and no prefix
+
+---
+
 ## [2026-01-13] [Infrastructure] [COMPLETED] Runtime Folder Naming Convention - Underscore Prefix
 
 **Problem:** Folders `log/`, `output/`, and `input/` didn't clearly indicate they were runtime-only storage, making it unclear which folders are internal vs permanent.

dev/dev_251222_01_api_integration_guide.md

@@ -1,766 +0,0 @@
-# [dev_251222_01] API Integration Guide
-
-**Date:** 2025-12-22
-**Type:** 🔨 Development
-**Status:** 🔄 In Progress
-**Related Dev:** N/A (Initial documentation)
-
-## Problem Description
-
-As a beginner learning API integration, needed comprehensive documentation of the GAIA scoring API to understand how to properly interact with all endpoints. The existing code only uses 2 of 4 available endpoints, missing critical file download functionality that many GAIA questions require.
-
----
-
-## API Overview
-
-**Base URL:** `https://agents-course-unit4-scoring.hf.space`
-
-**Purpose:** GAIA benchmark evaluation system that provides test questions, accepts agent answers, calculates scores, and maintains leaderboards.
-
-**Documentation Format:** FastAPI with Swagger UI (OpenAPI specification)
-
-**Authentication:** None required (public API)
-
-## Complete Endpoint Reference
-
-### Endpoint 1: GET /questions
-
-**Purpose:** Retrieve complete list of all GAIA test questions
-
-**Request:**
-
-```python
-import requests
-
-api_url = "https://agents-course-unit4-scoring.hf.space"
-response = requests.get(f"{api_url}/questions", timeout=15)
-questions = response.json()
-```
-
-**Parameters:** None
-
-**Response Format:**
-
-```json
-[
-  {
-    "task_id": "string",
-    "question": "string",
-    "level": "integer (1-3)",
-    "file_name": "string or null",
-    "file_path": "string or null",
-    ...additional metadata...
-  }
-]
-```
-
-**Response Codes:**
-
-- 200: Success - Returns array of question objects
-- 500: Server error
-
-**Key Fields:**
-
-- `task_id`: Unique identifier for each question (required for submission)
-- `question`: The actual question text your agent needs to answer
-- `level`: Difficulty level (1=easy, 2=medium, 3=hard)
-- `file_name`: Name of attached file if question includes one (null if no file)
-- `file_path`: Path to file on server (null if no file)
-
-**Current Implementation:** ✅ Already implemented in app.py:41-73
-
-**Usage in Your Code:**
-
-```python
-# Existing code location: app.py:54-66
-response = requests.get(questions_url, timeout=15)
-response.raise_for_status()
-questions_data = response.json()
-```
-
----
-
-### Endpoint 2: GET /random-question
-
-**Purpose:** Get single random question for testing/debugging
-
-**Request:**
-
-```python
-import requests
-
-api_url = "https://agents-course-unit4-scoring.hf.space"
-response = requests.get(f"{api_url}/random-question", timeout=15)
-question = response.json()
-```
-
-**Parameters:** None
-
-**Response Format:**
-
-```json
-{
-  "task_id": "string",
-  "question": "string",
-  "level": "integer",
-  "file_name": "string or null",
-  "file_path": "string or null"
-}
-```
-
-**Response Codes:**
-
-- 200: Success - Returns single question object
-- 404: No questions available
-- 500: Server error
-
-**Current Implementation:** ❌ Not implemented
-
-**Use Cases:**
-
-- Quick testing during agent development
-- Debugging specific question types
-- Iterative development without processing all questions
-
-**Example Implementation:**
-
-```python
-def test_agent_on_random_question(agent):
-    """Test agent on a single random question"""
-    api_url = "https://agents-course-unit4-scoring.hf.space"
-    response = requests.get(f"{api_url}/random-question", timeout=15)
-
-    if response.status_code == 404:
-        return "No questions available"
-
-    response.raise_for_status()
-    question_data = response.json()
-
-    task_id = question_data.get("task_id")
-    question_text = question_data.get("question")
-
-    answer = agent(question_text)
-    print(f"Task: {task_id}")
-    print(f"Question: {question_text}")
-    print(f"Agent Answer: {answer}")
-
-    return answer
-```
-
----
-
-### Endpoint 3: POST /submit
-
-**Purpose:** Submit all agent answers for evaluation and receive score
-
-**Request:**
-
-```python
-import requests
-
-api_url = "https://agents-course-unit4-scoring.hf.space"
-submission_data = {
-    "username": "your-hf-username",
-    "agent_code": "https://huggingface.co/spaces/your-space/tree/main",
-    "answers": [
-        {"task_id": "task_001", "submitted_answer": "42"},
-        {"task_id": "task_002", "submitted_answer": "Paris"}
-    ]
-}
-
-response = requests.post(
-    f"{api_url}/submit",
-    json=submission_data,
-    timeout=60
-)
-result = response.json()
-```
-
-**Request Body Schema:**
-
-```json
-{
-  "username": "string (required)",
-  "agent_code": "string (min 10 chars, required)",
-  "answers": [
-    {
-      "task_id": "string (required)",
-      "submitted_answer": "string | number | integer (required)"
-    }
-  ]
-}
-```
-
-**Field Requirements:**
-
-- `username`: Your Hugging Face username (obtained from OAuth profile)
-- `agent_code`: URL to your agent's source code (typically HF Space repo URL)
-- `answers`: Array of answer objects, one per question
-  - `task_id`: Must match task_id from /questions endpoint
-  - `submitted_answer`: Can be string, integer, or number depending on question
-
-**Response Format:**
-
-```json
-{
-  "username": "string",
-  "score": 85.5,
-  "correct_count": 17,
-  "total_attempted": 20,
-  "message": "Submission successful!",
-  "timestamp": "2025-12-22T10:30:00.123Z"
-}
-```
-
-**Response Codes:**
-
-- 200: Success - Returns score and statistics
-- 400: Invalid input (missing fields, wrong format)
-- 404: One or more task_ids not found
-- 500: Server error
-
-**Current Implementation:** ✅ Already implemented in app.py:112-161
-
-**Usage in Your Code:**
-
-```python
-# Existing code location: app.py:112-135
-submission_data = {
-    "username": username.strip(),
-    "agent_code": agent_code,
-    "answers": answers_payload,
-}
-response = requests.post(submit_url, json=submission_data, timeout=60)
-response.raise_for_status()
-result_data = response.json()
-```
-
-**Important Notes:**
-
-- Timeout set to 60 seconds (longer than /questions because scoring takes time)
-- All answers must be submitted together in single request
-- Score is calculated immediately and returned in response
-- Results also update the public leaderboard
-
----
-
-### Endpoint 4: GET /files/{task_id}
-
-**Purpose:** Download files attached to questions (images, PDFs, data files, etc.)
-
-**Request:**
-
-```python
-import requests
-
-api_url = "https://agents-course-unit4-scoring.hf.space"
-task_id = "task_001"
-response = requests.get(f"{api_url}/files/{task_id}", timeout=30)
-
-# Save file to disk
-with open(f"downloaded_{task_id}.file", "wb") as f:
-    f.write(response.content)
-```
-
-**Parameters:**
-
-- `task_id` (string, required, path parameter): The task_id of the question
-
-**Response Format:**
-
-- Binary file content (could be image, PDF, CSV, JSON, etc.)
-- Content-Type header indicates file type
-
-**Response Codes:**
-
-- 200: Success - Returns file content
-- 403: Access denied (path traversal attempt blocked)
-- 404: Task not found OR task has no associated file
-- 500: Server error
-
-**Current Implementation:** ❌ Not implemented - THIS IS CRITICAL GAP
-
-**Why This Matters:**
-Many GAIA questions include attached files that contain essential information for answering the question. Without downloading these files, your agent cannot answer those questions correctly.
-
-**Detection Logic:**
-
-```python
-# Check if question has an attached file
-question_data = {
-    "task_id": "task_001",
-    "question": "What is shown in the image?",
-    "file_name": "image.png",      # Not null = file exists
-    "file_path": "/files/task_001" # Path to file
-}
-
-has_file = question_data.get("file_name") is not None
-```
-
-**Example Implementation:**
-
-```python
-def download_task_file(task_id, save_dir="input/"):
-    """Download file associated with a task_id"""
-    api_url = "https://agents-course-unit4-scoring.hf.space"
-    file_url = f"{api_url}/files/{task_id}"
-
-    try:
-        response = requests.get(file_url, timeout=30)
-        response.raise_for_status()
-
-        # Determine file extension from Content-Type or use generic
-        content_type = response.headers.get('Content-Type', '')
-        extension_map = {
-            'image/png': '.png',
-            'image/jpeg': '.jpg',
-            'application/pdf': '.pdf',
-            'text/csv': '.csv',
-            'application/json': '.json',
-        }
-        extension = extension_map.get(content_type, '.file')
-
-        # Save file
-        file_path = f"{save_dir}{task_id}{extension}"
-        with open(file_path, 'wb') as f:
-            f.write(response.content)
-
-        print(f"Downloaded file for {task_id}: {file_path}")
-        return file_path
-
-    except requests.exceptions.HTTPError as e:
-        if e.response.status_code == 404:
-            print(f"No file found for task {task_id}")
-            return None
-        raise
-```
-
-**Integration Example:**
-
-```python
-# Enhanced agent workflow
-for item in questions_data:
-    task_id = item.get("task_id")
-    question_text = item.get("question")
-    file_name = item.get("file_name")
-
-    # Download file if question has one
-    file_path = None
-    if file_name:
-        file_path = download_task_file(task_id)
-
-    # Pass both question and file to agent
-    answer = agent(question_text, file_path=file_path)
-```
-
----
-
-## API Request Flow Diagram
-
-```
-Student Agent Workflow:
-┌─────────────────────────────────────────────────────────────┐
-│ 1. Fetch Questions                                          │
-│    GET /questions                                           │
-│    → Receive list of all questions with metadata           │
-└────────────────────┬────────────────────────────────────────┘
-                     ↓
-┌─────────────────────────────────────────────────────────────┐
-│ 2. Process Each Question                                    │
-│    For each question:                                       │
-│    a) Check if file_name exists                            │
-│    b) If yes: GET /files/{task_id}                         │
-│       → Download and save file                             │
-│    c) Pass question + file to agent                        │
-│    d) Agent generates answer                               │
-└────────────────────┬────────────────────────────────────────┘
-                     ↓
-┌─────────────────────────────────────────────────────────────┐
-│ 3. Submit All Answers                                       │
-│    POST /submit                                             │
-│    → Send username, agent_code, and all answers            │
-│    → Receive score and statistics                          │
-└─────────────────────────────────────────────────────────────┘
-```
-
-## Error Handling Best Practices
-
-### Connection Errors
-
-```python
-try:
-    response = requests.get(url, timeout=15)
-    response.raise_for_status()
-except requests.exceptions.Timeout:
-    print("Request timed out")
-except requests.exceptions.ConnectionError:
-    print("Network connection error")
-except requests.exceptions.HTTPError as e:
-    print(f"HTTP error: {e.response.status_code}")
-```
-
-### Response Validation
-
-```python
-# Always validate response format
-response = requests.get(questions_url)
-response.raise_for_status()
-
-try:
-    data = response.json()
-except requests.exceptions.JSONDecodeError:
-    print("Invalid JSON response")
-    print(f"Response text: {response.text[:500]}")
-```
-
-### Timeout Recommendations
-
-- GET /questions: 15 seconds (fetching list)
-- GET /random-question: 15 seconds (single question)
-- GET /files/{task_id}: 30 seconds (file download may be larger)
-- POST /submit: 60 seconds (scoring all answers takes time)
-
-## Current Implementation Status
-
-### ✅ Implemented Endpoints
-
-1. **GET /questions** - Fully implemented in app.py:41-73
-2. **POST /submit** - Fully implemented in app.py:112-161
-
-### ❌ Missing Endpoints
-
-1. **GET /random-question** - Not implemented (useful for testing)
-2. **GET /files/{task_id}** - Not implemented (CRITICAL - many questions need files)
-
-### 🚨 Critical Gap Analysis
-
-**Impact of Missing /files Endpoint:**
-
-- Questions with attached files cannot be answered correctly
-- Agent will only see question text, not the actual content to analyze
-- Significantly reduces potential score on GAIA benchmark
-
-**Example Questions That Need Files:**
-
-- "What is shown in this image?" → Needs image file
-- "What is the total in column B?" → Needs spreadsheet file
-- "Summarize this document" → Needs PDF/text file
-- "What patterns do you see in this data?" → Needs CSV/JSON file
-
-**Estimated Impact:**
-
-- GAIA benchmark: ~30-40% of questions include files
-- Without file handling: Maximum achievable score ~60-70%
-- With file handling: Can potentially achieve 100%
-
-## Next Steps for Implementation
-
-### Priority 1: Add File Download Support
-
-1. Detect questions with files (check `file_name` field)
-2. Download files using GET /files/{task_id}
-3. Save files to input/ directory
-4. Modify BasicAgent to accept file_path parameter
-5. Update agent logic to process files
-
-### Priority 2: Add Testing Endpoint
-
-1. Implement GET /random-question for quick testing
-2. Create test script in test/ directory
-3. Enable iterative development without full evaluation runs
-
-### Priority 3: Enhanced Error Handling
-
-1. Add retry logic for network failures
-2. Validate file downloads (check file size, type)
-3. Handle partial failures gracefully
-
-## How to Read FastAPI Swagger Documentation
-
-### Understanding the Swagger UI
-
-FastAPI APIs use Swagger UI for interactive documentation. Here's how to read it systematically:
-
-### Main UI Components
-
-#### 1. Header Section
-
-```
-Agent Evaluation API  [0.1.0] [OAS 3.1]
-/openapi.json
-```
-
-**What you learn:**
-
-- **API Name:** Service identification
-- **Version:** `0.1.0` - API version (important for tracking changes)
-- **OAS 3.1:** OpenAPI Specification standard version
-- **Link:** `/openapi.json` - raw machine-readable specification
-
-#### 2. API Description
-
-High-level summary of what the service provides
-
-#### 3. Endpoints Section (Expandable List)
-
-**HTTP Method Colors:**
-
-- **Blue "GET"** = Retrieve/fetch data (read-only, safe to call multiple times)
-- **Green "POST"** = Submit/create data (writes data, may change state)
-- **Orange "PUT"** = Update existing data
-- **Red "DELETE"** = Remove data
-
-**Each endpoint shows:**
-
-- Path (URL structure)
-- Short description
-- Click to expand for details
-
-#### 4. Expanded Endpoint Details
-
-When you click an endpoint, you get:
-
-**Section A: Description**
-
-- Detailed explanation of functionality
-- Use cases and purpose
-
-**Section B: Parameters**
-
-- **Path Parameters:** Variables in URL like `/files/{task_id}`
-- **Query Parameters:** Key-value pairs after `?` like `?level=1&limit=10`
-- Each parameter shows:
-  - Name
-  - Type (string, integer, boolean, etc.)
-  - Required vs Optional
-  - Description
-  - Example values
-
-**Section C: Request Body** (POST/PUT only)
-
-- JSON structure to send
-- Field names and types
-- Required vs optional fields
-- Example payload
-- Schema button shows structure
-
-**Section D: Responses**
-
-- Status codes (200, 400, 404, 500)
-- Response structure for each code
-- Example responses
-- What each status means
-
-**Section E: Try It Out Button**
-
-- Test API directly in browser
-- Fill parameters and send real requests
-- See actual responses
-
-#### 5. Schemas Section (Bottom)
-
-Reusable data structures used across endpoints:
-
-```
-Schemas
-  ├─ AnswerItem
-  ├─ ErrorResponse
-  ├─ ScoreResponse
-  └─ Submission
-```
-
-Click each to see:
-
-- All fields in the object
-- Field types and constraints
-- Required vs optional
-- Descriptions
-
-### Step-by-Step: Reading One Endpoint
-
-**Example: POST /submit**
-
-**Step 1:** Click the endpoint to expand
-
-**Step 2:** Read description
-*"Submit agent answers, calculate scores, and update leaderboard"*
-
-**Step 3:** Check Parameters
-
-- Path parameters? None (URL is just `/submit`)
-- Query parameters? None
-
-**Step 4:** Check Request Body
-
-```json
-{
-  "username": "string (required)",
-  "agent_code": "string, min 10 chars (required)",
-  "answers": [
-    {
-      "task_id": "string (required)",
-      "submitted_answer": "string | number | integer (required)"
-    }
-  ]
-}
-```
-
-**Step 5:** Check Responses
-
-**200 Success:**
-
-```json
-{
-  "username": "string",
-  "score": 85.5,
-  "correct_count": 15,
-  "total_attempted": 20,
-  "message": "Success!"
-}
-```
-
-**Other codes:**
-
-- 400: Invalid input
-- 404: Task ID not found
-- 500: Server error
-
-**Step 6:** Write Python code
-
-```python
-url = "https://agents-course-unit4-scoring.hf.space/submit"
-payload = {
-    "username": "your-username",
-    "agent_code": "https://...",
-    "answers": [
-        {"task_id": "task_001", "submitted_answer": "42"}
-    ]
-}
-response = requests.post(url, json=payload, timeout=60)
-result = response.json()
-```
-
-### Information Extraction Checklist
-
-For each endpoint, extract:
-
-**Basic Info:**
-
-- HTTP method (GET, POST, PUT, DELETE)
-- Endpoint path (URL)
-- One-line description
-
-**Request Details:**
-
-- Path parameters (variables in URL)
-- Query parameters (after ? in URL)
-- Request body structure (POST/PUT)
-- Required vs optional fields
-- Data types and constraints
-
-**Response Details:**
-
-- Success response structure (200)
-- Success response example
-- All possible status codes
-- Error response structures
-- What each status code means
-
-**Additional Info:**
-
-- Authentication requirements
-- Rate limits
-- Example requests
-- Related schemas
-
-### Pro Tips
-
-**Tip 1: Start with GET endpoints**
-Simpler (no request body) and safe to test
-
-**Tip 2: Use "Try it out" button**
-Best way to learn - send real requests and see responses
-
-**Tip 3: Check Schemas section**
-Understanding schemas helps decode complex structures
-
-**Tip 4: Copy examples**
-Most Swagger UIs have example values - use them!
-
-**Tip 5: Required vs Optional**
-Required fields cause 400 error if missing
-
-**Tip 6: Read error responses**
-They tell you what went wrong and how to fix it
-
-### Practice Exercise
-
-**Try reading GET /files/{task_id}:**
-
-1. What HTTP method? → GET
-2. What's the path parameter? → `task_id` (string, required)
-3. What does it return? → File content (binary)
-4. What status codes? → 200, 403, 404, 500
-5. Python code? → `requests.get(f"{api_url}/files/{task_id}")`
-
-## Learning Resources
-
-**Understanding REST APIs:**
-
-- REST = Representational State Transfer
-- APIs communicate using HTTP methods: GET (retrieve), POST (submit), PUT (update), DELETE (remove)
-- Data typically exchanged in JSON format
-
-**Key Concepts:**
-
-- **Endpoint:** Specific URL path that performs one action (/questions, /submit)
-- **Request:** Data you send to the API (parameters, body)
-- **Response:** Data the API sends back (JSON, files, status codes)
-- **Status Codes:**
-  - 200 = Success
-  - 400 = Bad request (your input was wrong)
-  - 404 = Not found
-  - 500 = Server error
-
-**Python Requests Library:**
-
-```python
-# GET request - retrieve data
-response = requests.get(url, params={...}, timeout=15)
-
-# POST request - submit data
-response = requests.post(url, json={...}, timeout=60)
-
-# Always check status
-response.raise_for_status()  # Raises error if status >= 400
-
-# Parse JSON response
-data = response.json()
-```
-
----
-
-## Key Decisions
-
-- **Documentation Structure:** Organized by endpoint with complete examples for each
-- **Learning Approach:** Beginner-friendly explanations with code examples
-- **Priority Focus:** Highlighted critical missing functionality (file downloads)
-- **Practical Examples:** Included copy-paste ready code snippets
-
-## Outcome
-
-Created comprehensive API integration guide documenting all 4 endpoints of the GAIA scoring API, identified critical gap in current implementation (missing file download support), and provided actionable examples for enhancement.
-
-**Deliverables:**
-
-- `dev/dev_251222_01_api_integration_guide.md` - Complete API reference documentation
-
-## Changelog
-
-**What was changed:**
-
-- Created new documentation file: dev_251222_01_api_integration_guide.md
-- Documented all 4 API endpoints with request/response formats
-- Added code examples for each endpoint
-- Identified critical missing functionality (file downloads)
-- Provided implementation roadmap for enhancements

dev/dev_260101_02_level1_strategic_foundation.md

@@ -1,68 +0,0 @@
-# [dev_260101_02] Level 1 Strategic Foundation Decisions
-
-**Date:** 2026-01-01
-**Type:** Development
-**Status:** Resolved
-**Related Dev:** dev_251222_01
-
-## Problem Description
-
-Applied AI Agent System Design Framework (8-level decision model) to GAIA benchmark agent project. Level 1 establishes strategic foundation by defining business problem scope, value alignment, and organizational readiness before architectural decisions.
-
----
-
-## Key Decisions
-
-**Parameter 1: Business Problem Scope → Single workflow**
-
-- **Reasoning:** GAIA tests ONE unified meta-skill (multi-step reasoning + tool use) applied across diverse content domains (science, personal tasks, general knowledge)
-- **Critical distinction:** Content diversity ≠ workflow diversity. Same question-answering process across all 466 questions
-- **Evidence:** GAIA_TuyenPham_Analysis.pdf Benchmark Contents section confirms "GAIA focuses more on the types of capabilities required rather than academic subject coverage"
-
-**Parameter 2: Value Alignment → Capability enhancement**
-
-- **Reasoning:** Learning-focused project with benchmark score as measurable success metric
-- **Stakeholder:** Student learning + course evaluation system
-- **Success measure:** Performance improvement on GAIA leaderboard
-
-**Parameter 3: Organizational Readiness → High (experimental)**
-
-- **Reasoning:** Learning environment, fixed dataset (466 questions), rapid iteration possible
-- **Constraints:** Zero-shot evaluation (no training on GAIA), factoid answer format
-- **Risk tolerance:** High - experimental learning context allows failure
-
-**Rejected alternatives:**
-
-- Multi-workflow approach: Would incorrectly treat content domains as separate business processes
-- Production-level readiness: Inappropriate for learning/benchmark context
-
-## Outcome
-
-Established strategic foundation for GAIA agent architecture. Confirmed single-workflow approach enables unified agent design rather than multi-agent orchestration.
-
-**Deliverables:**
-
-- `dev/dev_260101_02_level1_strategic_foundation.md` - Level 1 decision documentation
-
-**Critical Outputs:**
-
-- **Use Case:** Build AI agent that answers GAIA benchmark questions
-- **Baseline Target:** >60% on Level 1 (text-only questions)
-- **Intermediate Target:** >40% overall (with file handling)
-- **Stretch Target:** >80% overall (full multi-modal + reasoning)
-- **Stakeholder:** Student learning + course evaluation system
-
-## Learnings and Insights
-
-**Pattern discovered:** Content domain diversity does NOT imply workflow diversity. A single unified process can handle multiple knowledge domains if the meta-skill (reasoning + tool use) remains constant.
-
-**What worked well:** Reading GAIA_TuyenPham_Analysis.pdf twice (after Benchmark Contents update) prevented premature architectural decisions.
-
-**Framework application:** Level 1 Strategic Foundation successfully scoped the project before diving into technical architecture.
-
-## Changelog
-
-**What was changed:**
-
-- Created `dev/dev_260101_02_level1_strategic_foundation.md` - Level 1 strategic decisions
-- Referenced analysis files: GAIA_TuyenPham_Analysis.pdf, GAIA_Article_2023.pdf, AI Agent System Design Framework (2026-01-01).pdf

dev/dev_260101_03_level2_system_architecture.md

@@ -1,58 +0,0 @@
-# [dev_260101_03] Level 2 System Architecture Decisions
-
-**Date:** 2026-01-01
-**Type:** Development
-**Status:** Resolved
-**Related Dev:** dev_260101_02
-
-## Problem Description
-
-Applied Level 2 System Architecture parameters from AI Agent System Design Framework to determine agent ecosystem structure, orchestration strategy, and human-in-loop positioning for GAIA benchmark agent.
-
----
-
-## Key Decisions
-
-**Parameter 1: Agent Ecosystem Type → Single agent**
-- **Reasoning:** Task decomposition complexity is LOW for GAIA
-- **Evidence:** Each GAIA question is self-contained factoid task requiring multi-step reasoning + tool use, not collaborative multi-agent workflows
-- **Implication:** One agent orchestrates tools directly without delegation hierarchy
-
-**Parameter 2: Orchestration Strategy → N/A (single agent)**
-- **Reasoning:** With single agent decision, orchestration strategy (Hierarchical/Event-driven/Hybrid) doesn't apply
-- **Implication:** The single agent controls its own tool execution flow sequentially
-
-**Parameter 3: Human-in-Loop Position → Full autonomy**
-- **Reasoning:** GAIA benchmark is zero-shot automated evaluation with 6-17 min time constraints
-- **Evidence:** Human intervention (approval gates/feedback loops) would invalidate benchmark scores
-- **Implication:** Agent must answer all 466 questions independently without human assistance
-
-**Rejected alternatives:**
-- Multi-agent collaborative: Would add unnecessary coordination overhead for independent question-answering tasks
-- Hierarchical delegation: Inappropriate for self-contained factoid questions without complex sub-task decomposition
-- Human approval gates: Violates benchmark zero-shot evaluation requirements
-
-## Outcome
-
-Confirmed single-agent architecture with full autonomy. Agent will directly orchestrate tools (web browser, code interpreter, file reader, multi-modal processor) without multi-agent coordination or human intervention.
-
-**Deliverables:**
-- `dev/dev_260101_03_level2_system_architecture.md` - Level 2 architectural decisions
-
-**Architectural Constraints:**
-- Single ReasoningAgent class design
-- Direct tool orchestration without delegation
-- No human-in-loop mechanisms
-- Stateless execution per question (from Level 1 single workflow)
-
-## Learnings and Insights
-
-**Pattern discovered:** Single agent with tool orchestration is appropriate when tasks are self-contained and don't require collaborative decomposition across multiple reasoning entities.
-
-**Critical distinction:** Agent ecosystem type (single vs multi-agent) should be determined by task decomposition complexity, not tool diversity. GAIA requires multiple tool types but single reasoning entity.
-
-## Changelog
-
-**What was changed:**
-- Created `dev/dev_260101_03_level2_system_architecture.md` - Level 2 system architecture decisions
-- Referenced AI Agent System Design Framework (2026-01-01).pdf Level 2 parameters

dev/dev_260101_04_level3_task_workflow_design.md

@@ -1,77 +0,0 @@
-# [dev_260101_04] Level 3 Task & Workflow Design Decisions
-
-**Date:** 2026-01-01
-**Type:** Development
-**Status:** Resolved
-**Related Dev:** dev_260101_03
-
-## Problem Description
-
-Applied Level 3 Task & Workflow Design parameters from AI Agent System Design Framework to define task decomposition strategy and workflow execution pattern for GAIA benchmark agent MVP.
-
----
-
-## Key Decisions
-
-**Parameter 1: Task Decomposition → Dynamic planning**
-
-- **Reasoning:** GAIA questions vary widely in complexity and required tool combinations
-- **Evidence:** Cannot use static pipeline - each question requires analyzing intent, then planning multi-step approach dynamically
-- **Implication:** Agent must generate execution plan per question based on question analysis
-
-**Parameter 2: Workflow Pattern → Sequential**
-
-- **Reasoning:** Agent follows linear reasoning chain with dependencies between steps
-- **Execution flow:** (1) Parse question → (2) Plan approach → (3) Execute tool calls → (4) Synthesize factoid answer
-- **Evidence:** Each step depends on previous step's output - no parallel execution needed
-- **Implication:** Sequential workflow pattern fits question-answering nature (vs routing/orchestrator-worker for multi-agent)
-
-**Parameter 3: Task Prioritization → N/A (single task processing)**
-
-- **Reasoning:** GAIA benchmark processes one question at a time in zero-shot evaluation
-- **Evidence:** No multi-task scheduling required - agent answers one question per invocation
-- **Implication:** No task queue, priority system, or LLM-based scheduling needed
-- **Alignment:** Matches zero-shot stateless design (Level 1, Level 5)
-
-**Rejected alternatives:**
-
-- Static pipeline: Cannot handle diverse GAIA question types requiring different tool combinations
-- Reactive decomposition: Less efficient than planning upfront for factoid question-answering
-- Parallel workflow: GAIA reasoning chains have linear dependencies
-- Routing pattern: Inappropriate for single-agent architecture (Level 2 decision)
-
-**Future experimentation:**
-
-- **Reflection pattern:** Self-critique and refinement loops for improved answer quality
-- **ReAct pattern:** Reasoning-Action interleaving for more adaptive execution
-- **Current MVP:** Sequential + Dynamic planning for baseline performance
-
-## Outcome
-
-Established MVP workflow architecture: Dynamic planning with sequential execution. Agent analyzes each question, generates step-by-step plan, executes tools sequentially, synthesizes factoid answer.
-
-**Deliverables:**
-
-- `dev/dev_260101_04_level3_task_workflow_design.md` - Level 3 workflow design decisions
-
-**Workflow Specifications:**
-
-- **Task Decomposition:** Dynamic planning per question
-- **Execution Pattern:** Sequential reasoning chain
-- **Future Enhancement:** Reflection/ReAct patterns for advanced iterations
-
-## Learnings and Insights
-
-**Pattern discovered:** MVP approach favors simplicity (Sequential + Dynamic) before complexity (Reflection/ReAct). Baseline performance measurement enables informed optimization decisions.
-
-**Design philosophy:** Start with linear workflow, measure performance, then add complexity (self-reflection, adaptive reasoning) only if needed.
-
-**Critical connection:** Level 3 workflow patterns will be implemented in Level 6 using specific framework capabilities (LangGraph/AutoGen/CrewAI).
-
-## Changelog
-
-**What was changed:**
-
-- Created `dev/dev_260101_04_level3_task_workflow_design.md` - Level 3 task & workflow design decisions
-- Referenced AI Agent System Design Framework (2026-01-01).pdf Level 3 parameters
-- Documented future experimentation plans (Reflection/ReAct patterns)

dev/dev_260101_05_level4_agent_level_design.md

@@ -1,77 +0,0 @@
-# [dev_260101_05] Level 4 Agent-Level Design Decisions
-
-**Date:** 2026-01-01
-**Type:** Development
-**Status:** Resolved
-**Related Dev:** dev_260101_04
-
-## Problem Description
-
-Applied Level 4 Agent-Level Design parameters from AI Agent System Design Framework to define agent granularity, decision-making capability, responsibility scope, and communication protocol for GAIA benchmark agent.
-
----
-
-## Key Decisions
-
-**Parameter 1: Agent Granularity → Coarse-grained generalist**
-- **Reasoning:** Single agent architecture (Level 2) requires one generalist agent
-- **Evidence:** GAIA covers diverse content domains (science, personal tasks, general knowledge) - agent must handle all types with dynamic tool selection
-- **Implication:** One agent with broad capabilities rather than fine-grained specialists per domain
-- **Alignment:** Prevents coordination overhead, matches single-agent architecture decision
-
-**Parameter 2: Agent Type per Role → Goal-Based**
-- **Reasoning:** Agent must achieve specific goal (produce factoid answer) using multi-step planning and tool use
-- **Decision-making level:** More sophisticated than Model-Based (reactive state-based), less complex than Utility-Based (optimization across multiple objectives)
-- **Capability:** Goal-directed reasoning - maintains end goal while planning intermediate steps
-- **Implication:** Agent requires goal-tracking and means-end reasoning capabilities
-
-**Parameter 3: Agent Responsibility → Multi-task within domain**
-- **Reasoning:** Single agent handles diverse task types within question-answering domain
-- **Task diversity:** Web search, code execution, file reading, multi-modal processing
-- **Domain boundary:** All tasks serve question-answering goal (single domain)
-- **Implication:** Agent must select appropriate tool combinations based on question requirements
-
-**Parameter 4: Inter-Agent Protocol → N/A (single agent)**
-- **Reasoning:** Single-agent architecture eliminates need for inter-agent communication
-- **Implication:** No message passing, shared state, or event-driven protocols required
-
-**Parameter 5: Termination Logic → Fixed steps (3-node workflow)**
-- **Reasoning:** Sequential workflow (Level 3) defines clear termination point after answer_node
-- **Execution flow:** plan_node → execute_node → answer_node → END
-- **Evidence:** 3-node LangGraph workflow terminates after final answer synthesis
-- **Implication:** No LLM-based completion detection needed - workflow structure defines termination
-- **Alignment:** Matches sequential workflow pattern (Level 3)
-
-**Rejected alternatives:**
-- Fine-grained specialists: Would require multi-agent architecture, rejected in Level 2
-- Simple Reflex agent: Insufficient reasoning capability for multi-step GAIA questions
-- Utility-Based agent: Over-engineered for factoid question-answering (no multi-objective optimization needed)
-- Learning agent: GAIA is zero-shot evaluation, no learning across questions permitted
-
-## Outcome
-
-Defined agent as coarse-grained generalist with goal-based reasoning capability. Agent maintains question-answering goal, plans multi-step execution, handles diverse tools within single domain, operates autonomously without inter-agent communication.
-
-**Deliverables:**
-- `dev/dev_260101_05_level4_agent_level_design.md` - Level 4 agent-level design decisions
-
-**Agent Specifications:**
-- **Granularity:** Coarse-grained generalist (single agent, all tasks)
-- **Decision-Making:** Goal-Based reasoning (maintains goal, plans steps)
-- **Responsibility:** Multi-task within question-answering domain
-- **Communication:** None (single-agent architecture)
-
-## Learnings and Insights
-
-**Pattern discovered:** Agent Type selection (Goal-Based) directly correlates with task complexity. GAIA requires planning and tool orchestration, not simple stimulus-response (Reflex) or multi-objective optimization (Utility-Based).
-
-**Design constraint:** Agent granularity is determined by Level 2 ecosystem type decision. Single-agent architecture → coarse-grained generalist is the only viable option.
-
-**Critical connection:** Goal-Based agent type requires planning capabilities to be implemented in Level 6 framework selection (e.g., LangGraph planning nodes).
-
-## Changelog
-
-**What was changed:**
-- Created `dev/dev_260101_05_level4_agent_level_design.md` - Level 4 agent-level design decisions
-- Referenced AI Agent System Design Framework (2026-01-01).pdf Level 4 parameters
-- Established Goal-Based reasoning requirement for framework implementation

dev/dev_260101_06_level5_component_selection.md

@@ -1,118 +0,0 @@
-# [dev_260101_06] Level 5 Component Selection Decisions
-
-**Date:** 2026-01-01
-**Type:** Development
-**Status:** Resolved
-**Related Dev:** dev_260101_05
-
-## Problem Description
-
-Applied Level 5 Component Selection parameters from AI Agent System Design Framework to select LLM model, tool suite, memory architecture, and guardrails for GAIA benchmark agent MVP.
-
----
-
-## Key Decisions
-
-**Parameter 1: LLM Model → Claude Sonnet 4.5 (primary) + Free API baseline options**
-- **Primary choice:** Claude Sonnet 4.5
-  - **Reasoning:** Framework best practice - "Start with most capable model to baseline performance, then optimize downward for cost"
-  - **Capability match:** Sonnet 4.5 provides strong reasoning + tool use capabilities required for GAIA
-  - **Budget alignment:** Learning project allows premium model for baseline measurement
-- **Free API baseline alternatives:**
-  - **Google Gemini 2.0 Flash** (via AI Studio free tier)
-    - Function calling support, multi-modal, good reasoning
-    - Free quota: 1500 requests/day, suitable for GAIA evaluation
-  - **Qwen 2.5 72B** (via HuggingFace Inference API)
-    - Open source, function calling via HF API
-    - Free tier available, strong reasoning performance
-  - **Meta Llama 3.3 70B** (via HuggingFace Inference API)
-    - Open source, good tool use capability
-    - Free tier for experimentation
-- **Optimization path:** Start with free baseline (Gemini Flash), compare with Claude if budget allows
-- **Implication:** Dual-track approach - free API for experimentation, premium model for performance ceiling
-
-**Parameter 2: Tool Suite → Web browser / Python interpreter / File reader / Multi-modal processor**
-- **Evidence-based selection:** GAIA requirements breakdown:
-  - Web browsing: 76% of questions
-  - Code execution: 33% of questions
-  - File reading: 28% of questions (diverse formats)
-  - Multi-modal (vision): 30% of questions
-- **Specific tools:**
-  - Web search: Exa or Tavily API
-  - Code execution: Python interpreter (sandboxed)
-  - File reader: Multi-format parser (PDF, CSV, Excel, images)
-  - Vision: Multi-modal LLM capability for image analysis
-- **Coverage:** 4 core tools address primary GAIA capability requirements for MVP
-
-**Parameter 3: Memory Architecture → Short-term context only**
-- **Reasoning:** GAIA questions are independent and stateless (Level 1 decision)
-- **Evidence:** Zero-shot evaluation requires each question answered in isolation
-- **Implication:** No vector stores/RAG/semantic memory/episodic memory needed
-- **Memory scope:** Only maintain context within single question execution
-- **Alignment:** Matches Level 1 stateless design, prevents cross-question contamination
-
-**Parameter 4: Guardrails → Output validation + Tool restrictions**
-- **Output validation:** Enforce factoid answer format (numbers/few words/comma-separated lists)
-- **Tool restrictions:** Execution timeouts (prevent infinite loops), resource limits
-- **Minimal constraints:** No heavy content filtering for MVP (learning context)
-- **Safety focus:** Format compliance and execution safety, not content policy enforcement
-
-**Parameter 5: Answer Synthesis → LLM-generated (Stage 3 implementation)**
-- **Reasoning:** GAIA requires extracting factoid answers from multi-source evidence
-- **Evidence:** Answers must synthesize information from web searches, code outputs, file contents
-- **Implication:** LLM must reason about evidence and generate final answer (not template-based)
-- **Stage alignment:** Core logic implementation in Stage 3 (beyond MVP tool integration)
-- **Capability requirement:** LLM must distill complex evidence into concise factoid format
-
-**Parameter 6: Conflict Resolution → LLM-based reasoning (Stage 3 implementation)**
-- **Reasoning:** Multi-source evidence may contain conflicting information requiring judgment
-- **Example scenarios:** Conflicting search results, outdated vs current information, contradictory sources
-- **Implication:** LLM must evaluate source credibility and recency to resolve conflicts
-- **Stage alignment:** Decision logic in Stage 3 (not needed for Stage 2 tool integration)
-- **Alternative rejected:** Latest wins / Source priority too simplistic for GAIA evidence evaluation
-
-**Rejected alternatives:**
-
-- Vector stores/RAG: Unnecessary for stateless question-answering
-- Semantic/episodic memory: Violates GAIA zero-shot evaluation requirements
-- Heavy prompt constraints: Over-engineering for learning/benchmark context
-- Procedural caches: No repeated procedures to cache in stateless design
-
-**Future optimization:**
-
-- Model selection: A/B test free baselines (Gemini Flash, Qwen, Llama) vs premium (Claude, GPT-4)
-- Tool expansion: Add specialized tools based on failure analysis
-- Memory: Consider episodic memory for self-improvement experiments (non-benchmark mode)
-
-## Outcome
-
-Selected component stack optimized for GAIA MVP: Claude Sonnet 4.5 for reasoning, 4 core tools (web/code/file/vision) for capability coverage, short-term context for stateless execution, minimal guardrails for format validation and safety.
-
-**Deliverables:**
-- `dev/dev_260101_06_level5_component_selection.md` - Level 5 component selection decisions
-
-**Component Specifications:**
-
-- **LLM:** Claude Sonnet 4.5 (primary) with free baseline alternatives (Gemini 2.0 Flash, Qwen 2.5 72B, Llama 3.3 70B)
-- **Tools:** Web (Exa/Tavily) + Python interpreter + File reader + Vision
-- **Memory:** Short-term context only (stateless)
-- **Guardrails:** Output format validation + execution timeouts
-
-## Learnings and Insights
-
-**Pattern discovered:** Component selection driven by evidence-based requirements (GAIA capability analysis: 76% web, 33% code, 28% file, 30% multi-modal) rather than speculative "might need this" additions.
-
-**Best practice application:** "Start with most capable model to baseline performance" prevents premature optimization. Measure first, optimize second.
-
-**Memory architecture principle:** Stateless design enforced by benchmark requirements creates clean separation - no cross-question context leakage.
-
-**Critical connection:** Tool suite selection directly impacts Level 6 framework choice (framework must support function calling for tool integration).
-
-## Changelog
-
-**What was changed:**
-- Created `dev/dev_260101_06_level5_component_selection.md` - Level 5 component selection decisions
-- Referenced AI Agent System Design Framework (2026-01-01).pdf Level 5 parameters
-- Referenced GAIA_TuyenPham_Analysis.pdf capability requirements (76% web, 33% code, 28% file, 30% multi-modal)
-- Established Claude Sonnet 4.5 as primary LLM with free baseline alternatives (Gemini 2.0 Flash, Qwen 2.5 72B, Llama 3.3 70B)
-- Added dual-track optimization path: free API for experimentation, premium model for performance ceiling

dev/dev_260101_07_level6_implementation_framework.md

@@ -1,116 +0,0 @@
-# [dev_260101_07] Level 6 Implementation Framework Decisions
-
-**Date:** 2026-01-01
-**Type:** Development
-**Status:** Resolved
-**Related Dev:** dev_260101_06
-
-## Problem Description
-
-Applied Level 6 Implementation Framework parameters from AI Agent System Design Framework to select concrete framework, state management strategy, error handling approach, and tool interface standards for GAIA benchmark agent implementation.
-
----
-
-## Key Decisions
-
-**Parameter 1: Framework Choice → LangGraph**
-- **Reasoning:** Best fit for goal-based agent (Level 4) with sequential workflow (Level 3)
-- **Capability alignment:**
-  - StateGraph for workflow orchestration
-  - Planning nodes for dynamic task decomposition
-  - Tool nodes for execution
-  - Sequential routing matches Level 3 workflow pattern
-- **Alternative analysis:**
-  - CrewAI: Too high-level for single agent, designed for multi-agent teams
-  - AutoGen: Overkill for non-collaborative scenarios, adds complexity
-  - Custom framework: Unnecessary complexity for MVP, reinventing solved problems
-- **Implication:** Use LangGraph StateGraph as implementation foundation
-
-**Parameter 2: State Management → In-memory**
-- **Reasoning:** Stateless per question design (Levels 1, 5) eliminates persistence needs
-- **State scope:** Maintain state only during single question execution, clear after answer submission
-- **Implementation:** Python dict/dataclass for state tracking within question
-- **No database needed:** No PostgreSQL, Redis, or distributed cache required
-- **Alignment:** Matches zero-shot evaluation requirement (no cross-question state)
-
-**Parameter 3: Error Handling → Retry logic with timeout fallback**
-- **Constraint:** Full autonomy (Level 2) eliminates human escalation option
-- **Retry strategy:**
-  - Retry tool calls on transient failures (API timeouts, rate limits)
-  - Exponential backoff pattern
-  - Max 3 retries per tool call
-  - Overall question timeout (6-17 min GAIA limit)
-- **Fallback behavior:** Return "Unable to answer" if max retries exceeded or timeout reached
-- **No fallback agents:** Single agent architecture prevents agent delegation
-
-**Parameter 4: Tool Interface Standard → Function calling + MCP protocol**
-- **Primary interface:** Claude native function calling for tool integration
-- **Standardization:** MCP (Model Context Protocol) for tool definitions
-- **Benefits:**
-  - Flexible tool addition without agent code changes
-  - Standardized tool schemas
-  - Easy testing and tool swapping
-- **Implementation:** MCP server for tools (web/code/file/vision) + function calling interface
-
-**Parameter 5: Tool Selection Mechanism → LLM function calling (Stage 3 implementation)**
-- **Reasoning:** Dynamic tool selection required for diverse GAIA question types
-- **Evidence:** Questions require different tool combinations - LLM must reason about which tools to invoke
-- **Implementation:** Claude function calling enables LLM to select appropriate tools based on question analysis
-- **Stage alignment:** Core decision logic in Stage 3 (beyond MVP tool integration)
-- **Alternative rejected:** Static routing insufficient - cannot predetermine tool sequences for all GAIA questions
-
-**Parameter 6: Parameter Extraction → LLM-based parsing (Stage 3 implementation)**
-- **Reasoning:** Tool parameters must be extracted from natural language questions
-- **Example:** Question "What's the population of Tokyo?" → extract "Tokyo" as location parameter for search tool
-- **Implementation:** LLM interprets question and generates appropriate tool parameters
-- **Stage alignment:** Decision logic in Stage 3 (LLM reasoning about parameter values)
-- **Alternative rejected:** Structured input not applicable - GAIA provides natural language questions, not structured data
-
-**Rejected alternatives:**
-- Database-backed state: Violates stateless design, adds complexity
-- Distributed cache: Unnecessary for single-instance deployment
-- Human escalation: Violates GAIA full autonomy requirement
-- Fallback agents: Impossible with single-agent architecture
-- Custom tool schemas: MCP provides standardization
-- REST APIs only: Function calling more efficient than HTTP calls
-
-**Critical connection:** Level 3 workflow patterns (Sequential, Dynamic planning) get implemented using LangGraph StateGraph with planning and tool nodes.
-
-## Outcome
-
-Selected LangGraph as implementation framework with in-memory state management, retry-based error handling, and MCP/function-calling tool interface. Architecture supports goal-based reasoning with dynamic planning and sequential execution.
-
-**Deliverables:**
-- `dev/dev_260101_07_level6_implementation_framework.md` - Level 6 implementation framework decisions
-
-**Implementation Specifications:**
-- **Framework:** LangGraph StateGraph
-- **State:** In-memory (Python dict/dataclass)
-- **Error Handling:** Retry logic (max 3 retries, exponential backoff) + timeout fallback
-- **Tool Interface:** Function calling + MCP protocol
-
-**Technical Stack:**
-- LangGraph for workflow orchestration
-- Claude function calling for tool execution
-- MCP servers for tool standardization
-- Python dataclass for state tracking
-
-## Learnings and Insights
-
-**Pattern discovered:** Framework selection driven by architectural decisions from earlier levels. Goal-based agent (L4) + sequential workflow (L3) + single agent (L2) → LangGraph is natural fit.
-
-**Framework alignment:** LangGraph StateGraph maps directly to sequential workflow pattern. Planning nodes implement dynamic decomposition, tool nodes execute capabilities.
-
-**Error handling constraint:** Full autonomy requirement forces retry-based approach. No human-in-loop means agent must handle all failures autonomously within time constraints.
-
-**Tool standardization:** MCP protocol prevents tool interface fragmentation, enables future tool additions without core agent changes.
-
-**Critical insight:** In-memory state management is sufficient when Level 1 establishes stateless design. Database overhead unnecessary for MVP.
-
-## Changelog
-
-**What was changed:**
-- Created `dev/dev_260101_07_level6_implementation_framework.md` - Level 6 implementation framework decisions
-- Referenced AI Agent System Design Framework (2026-01-01).pdf Level 6 parameters
-- Established LangGraph + MCP as technical foundation
-- Defined retry logic specification (max 3 retries, exponential backoff, timeout fallback)

dev/dev_260101_08_level7_infrastructure_deployment.md

@@ -1,99 +0,0 @@
-# [dev_260101_08] Level 7 Infrastructure & Deployment Decisions
-
-**Date:** 2026-01-01
-**Type:** Development
-**Status:** Resolved
-**Related Dev:** dev_260101_07
-
-## Problem Description
-
-Applied Level 7 Infrastructure & Deployment parameters from AI Agent System Design Framework to select hosting strategy, scalability model, security controls, and observability stack for GAIA benchmark agent deployment.
-
----
-
-## Key Decisions
-
-**Parameter 1: Hosting Strategy → Cloud serverless (Hugging Face Spaces)**
-- **Reasoning:** Project already deployed on HF Spaces, no migration needed
-- **Benefits:**
-  - Serverless fits learning context (no infrastructure management)
-  - Gradio UI already implemented
-  - OAuth integration already working
-  - GPU available for multi-modal processing if needed
-- **Alignment:** Existing deployment target, minimal infrastructure overhead
-
-**Parameter 2: Scalability Model → Vertical scaling (single instance)**
-- **Reasoning:** GAIA is fixed 466 questions, no concurrent user load requirements
-- **Evidence:** Benchmark evaluation is sequential question processing, single-user context
-- **Implication:** No horizontal scaling, agent pools, or autoscaling needed
-- **Cost efficiency:** Single instance sufficient for benchmark evaluation
-
-**Parameter 3: Security Controls → API key management + OAuth authentication**
-- **API key management:** Environment variables via HF Secrets for tool APIs (Exa, Anthropic, Tavily)
-- **Authentication:** HF OAuth for user authentication (already implemented in app.py)
-- **Data sensitivity:** No encryption needed - GAIA is public benchmark dataset
-- **Access controls:** HF Space visibility settings (public/private toggle)
-- **Minimal security:** Standard API key protection, no sensitive data handling required
-
-**Parameter 4: Observability Stack → Logging + basic metrics**
-- **Logging:** stdout/stderr with print statements (already in app.py)
-- **Execution trace:** Question processing time, tool call success/failure, reasoning steps
-- **Metrics tracking:**
-  - Task success rate (correct answers / total questions)
-  - Per-question latency
-  - Tool usage statistics
-  - Final accuracy score
-- **UI metrics:** Gradio provides basic interface metrics
-- **Simplicity:** No complex tracing/debugging tools for MVP (APM, distributed tracing not needed)
-
-**Rejected alternatives:**
-- Containerized microservices: Over-engineering for single-agent, single-user benchmark
-- On-premise deployment: Unnecessary infrastructure management
-- Horizontal scaling: No concurrent load to justify
-- Autoscaling: Fixed dataset, predictable compute requirements
-- Data encryption: GAIA is public dataset
-- Complex observability: APM/distributed tracing overkill for MVP
-
-**Infrastructure constraints:**
-- HF Spaces limitations: Ephemeral storage, compute quotas
-- GPU availability: Optional for multi-modal processing
-- No database required: Stateless design (Level 5)
-
-## Outcome
-
-Confirmed cloud serverless deployment on existing HF Spaces infrastructure. Single instance with vertical scaling, minimal security controls (API keys + OAuth), simple observability (logs + basic metrics).
-
-**Deliverables:**
-- `dev/dev_260101_08_level7_infrastructure_deployment.md` - Level 7 infrastructure & deployment decisions
-
-**Infrastructure Specifications:**
-- **Hosting:** HF Spaces (serverless, existing deployment)
-- **Scalability:** Single instance, vertical scaling
-- **Security:** HF Secrets (API keys) + OAuth (authentication)
-- **Observability:** Print logging + success rate tracking
-
-**Deployment Context:**
-- No migration required (already on HF Spaces)
-- Gradio UI + OAuth already implemented
-- Environment variables for tool API keys
-- Public benchmark data (no encryption needed)
-
-## Learnings and Insights
-
-**Pattern discovered:** Infrastructure decisions heavily influenced by deployment context. Existing HF Spaces deployment eliminates migration complexity.
-
-**Right-sizing principle:** Single instance sufficient when workload is sequential, fixed dataset, single-user evaluation. No premature scaling architecture.
-
-**Security alignment:** Security controls match data sensitivity. Public benchmark requires standard API key protection, not enterprise encryption.
-
-**Observability philosophy:** Start simple (logs + metrics), add complexity only when debugging requires it. MVP doesn't need distributed tracing.
-
-**Critical constraint:** HF Spaces serverless architecture aligns with stateless design (Level 5) - ephemeral storage acceptable when no persistence needed.
-
-## Changelog
-
-**What was changed:**
-- Created `dev/dev_260101_08_level7_infrastructure_deployment.md` - Level 7 infrastructure & deployment decisions
-- Referenced AI Agent System Design Framework (2026-01-01).pdf Level 7 parameters
-- Confirmed existing HF Spaces deployment as hosting strategy
-- Established single-instance architecture with basic observability

dev/dev_260101_09_level8_evaluation_governance.md

@@ -1,110 +0,0 @@
-# [dev_260101_09] Level 8 Evaluation & Governance Decisions
-
-**Date:** 2026-01-01
-**Type:** Development
-**Status:** Resolved
-**Related Dev:** dev_260101_08
-
-## Problem Description
-
-Applied Level 8 Evaluation & Governance parameters from AI Agent System Design Framework to define evaluation metrics, testing strategy, governance model, and feedback loops for GAIA benchmark agent performance measurement and improvement.
-
----
-
-## Key Decisions
-
-**Parameter 1: Evaluation Metrics → Performance + Explainability**
-- **Performance metrics (primary):**
-  - **Task success rate:** % correct answers on GAIA benchmark (primary metric)
-    - Baseline target: >60% on Level 1 questions (text-only)
-    - Intermediate target: >40% overall (with file handling)
-    - Stretch target: >80% overall (full multi-modal + reasoning)
-  - **Cost per task:** API call costs (LLM + tools) per question
-  - **Latency per question:** Execution time within GAIA constraint (6-17 min)
-- **Explainability metrics:**
-  - **Chain-of-thought clarity:** Reasoning trace readability for debugging
-  - **Decision traceability:** Tool selection rationale, step-by-step logic
-- **Excluded metrics:**
-  - Safety: Not applicable (no harmful content risk in factoid question-answering)
-  - Compliance: Not applicable (public benchmark, learning context)
-  - Hallucination rate: Covered by task success rate (wrong answer = failure)
-
-**Parameter 2: Testing Strategy → End-to-end scenarios**
-- **Primary testing:** GAIA validation split before full submission
-- **Test approach:** Execute agent on validation questions, measure success rate
-- **No unit tests:** MVP favors rapid iteration over test coverage
-- **Integration testing:** Actual question execution tests entire pipeline (LLM + tools + reasoning)
-- **Focus:** End-to-end accuracy validation, not component-level testing
-
-**Parameter 3: Governance Model → Audit trails**
-- **Logging:** All question executions, tool calls, reasoning steps logged for debugging
-- **No centralized approval:** Full autonomy (Level 2) eliminates human oversight
-- **No automated guardrails:** Beyond output validation (Level 5 decision)
-- **Transparency:** Execution logs provide complete audit trail for failure analysis
-- **Lightweight governance:** Learning context doesn't require enterprise compliance
-
-**Parameter 4: Feedback Loops → Manual review of failures**
-- **Failure analysis:** Manually review failed questions, identify capability gaps
-- **Iteration cycle:** Failure patterns → capability enhancement → retest
-- **No automated retraining:** GAIA zero-shot constraint prevents learning across questions
-- **A/B testing:** Compare model performance (Gemini vs Claude), tool effectiveness (Exa vs Tavily)
-- **Improvement path:** Manual debugging → targeted improvements → measure impact
-
-**Rejected alternatives:**
-- Unit tests: Too slow for MVP iteration speed
-- Automated retraining: Violates zero-shot evaluation requirement
-- Safety metrics: Not applicable to factoid question-answering
-- Compliance tracking: Over-engineering for learning context
-- Centralized approval: Violates full autonomy architecture (Level 2)
-
-**Evaluation framework alignment:**
-- GAIA provides ground truth answers → automated success rate calculation
-- Benchmark leaderboard provides external validation
-- Reasoning traces enable root cause analysis
-
-## Outcome
-
-Established evaluation framework centered on GAIA task success rate (primary metric) with cost and latency tracking. End-to-end testing on validation split, audit trail logging for debugging, manual failure analysis for iterative improvement.
-
-**Deliverables:**
-- `dev/dev_260101_09_level8_evaluation_governance.md` - Level 8 evaluation & governance decisions
-
-**Evaluation Specifications:**
-- **Primary Metric:** Task success rate (% correct on GAIA)
-  - Baseline: >60% Level 1
-  - Intermediate: >40% overall
-  - Stretch: >80% overall
-- **Secondary Metrics:** Cost per task, Latency per question
-- **Explainability:** Chain-of-thought traces, decision traceability
-- **Testing:** End-to-end validation before submission
-- **Governance:** Audit trail logs, manual failure review
-- **Improvement:** A/B testing, failure pattern analysis
-
-**Success Criteria:**
-- Measurable improvement over baseline (fixed "This is a default answer")
-- Cost-effective API usage (track spend vs accuracy trade-offs)
-- Explainable failures (reasoning trace enables debugging)
-- Reproducible results (logged executions)
-
-## Learnings and Insights
-
-**Pattern discovered:** Evaluation metrics must align with benchmark requirements. GAIA provides ground truth → task success rate is objective primary metric.
-
-**Testing philosophy:** End-to-end testing more valuable than unit tests for agent systems. Integration points (LLM + tools + reasoning) tested together in realistic scenarios.
-
-**Governance simplification:** Full autonomy + learning context → minimal governance overhead. Audit trails sufficient for debugging without enterprise compliance.
-
-**Feedback loop design:** Manual failure analysis enables targeted capability improvements. Zero-shot constraint prevents automated learning, requires human-in-loop debugging.
-
-**Critical insight:** Explainability metrics (chain-of-thought, decision traceability) are debugging tools, not performance metrics. Enable failure analysis but don't measure agent quality directly.
-
-**Framework completion:** Level 8 completes 8-level decision framework. All architectural decisions documented from strategic foundation (L1) through evaluation (L8).
-
-## Changelog
-
-**What was changed:**
-- Created `dev/dev_260101_09_level8_evaluation_governance.md` - Level 8 evaluation & governance decisions
-- Referenced AI Agent System Design Framework (2026-01-01).pdf Level 8 parameters
-- Established task success rate as primary metric with baseline/intermediate/stretch targets
-- Defined end-to-end testing strategy on GAIA validation split
-- Completed all 8 levels of AI Agent System Design Framework application

dev/dev_260101_10_implementation_process_design.md

@@ -1,243 +0,0 @@
-# [dev_260101_10] Implementation Process Design
-
-**Date:** 2026-01-01
-**Type:** Development
-**Status:** Resolved
-**Related Dev:** dev_260101_09
-
-## Problem Description
-
-Designed implementation process for GAIA benchmark agent based on completed 8-level architectural decisions. Determined optimal execution sequence that differs from top-down design framework order.
-
----
-
-## Key Decisions
-
-**Critical Distinction: Design vs Implementation Order**
-
-- **Design Framework (Levels 1-8):** Top-down strategic planning (business problem → components)
-- **Implementation Process:** Bottom-up execution (components → working system)
-- **Reasoning:** Cannot code high-level decisions (L1 "single workflow") without low-level infrastructure (L6 LangGraph setup, L5 tools)
-
-**Implementation Strategy → 5-Stage Bottom-Up Approach**
-
-**Stage 1: Foundation Setup (Infrastructure First)**
-
-- **Build from:** Level 7 (Infrastructure) & Level 6 (Framework) decisions
-- **Deliverables:**
-  - HuggingFace Space environment configured
-  - LangGraph + dependencies installed
-  - API keys configured (HF Secrets)
-  - Basic project structure created
-- **Milestone:** Empty LangGraph agent runs successfully
-- **Estimated effort:** 1-2 days
-
-**Stage 2: Tool Development (Components Before Integration)**
-
-- **Build from:** Level 5 (Component Selection) decisions
-- **Deliverables:**
-  - 4 core tools as MCP servers:
-    1. Web search (Exa/Tavily API)
-    2. Python interpreter (sandboxed execution)
-    3. File reader (multi-format parser)
-    4. Multi-modal processor (vision)
-  - Independent test cases for each tool
-- **Milestone:** Each tool works independently with test validation
-- **Estimated effort:** 3-5 days
-
-**Stage 3: Agent Core (Reasoning Logic)**
-
-- **Build from:** Level 3 (Workflow) & Level 4 (Agent Design) decisions
-- **Deliverables:**
-  - LangGraph StateGraph structure
-  - Planning node (dynamic task decomposition)
-  - Tool selection logic (goal-based reasoning)
-  - Sequential execution flow
-- **Milestone:** Agent can plan and execute simple single-tool questions
-- **Estimated effort:** 3-4 days
-
-**Stage 4: Integration & Robustness**
-
-- **Build from:** Level 6 (Implementation Framework) decisions
-- **Deliverables:**
-  - All 4 tools connected to agent
-  - Retry logic + error handling (max 3 retries, exponential backoff)
-  - Execution timeouts (6-17 min GAIA constraint)
-  - Output validation (factoid format)
-- **Milestone:** Agent handles multi-tool questions with error recovery
-- **Estimated effort:** 2-3 days
-
-**Stage 5: Evaluation & Iteration**
-
-- **Build from:** Level 8 (Evaluation & Governance) decisions
-- **Deliverables:**
-  - GAIA validation split evaluation pipeline
-  - Task success rate measurement
-  - Failure analysis (reasoning traces)
-  - Capability gap identification
-  - Iterative improvements
-- **Milestone:** Meet baseline target (>60% Level 1 or >40% overall)
-- **Estimated effort:** Ongoing iteration
-
-**Why NOT Sequential L1→L8 Implementation?**
-
-| Design Level | Problem for Direct Implementation |
-|--------------|-----------------------------------|
-| L1: Strategic Foundation | Can't code "single workflow" - it's a decision, not code |
-| L2: System Architecture | Can't code "single agent" without tools/framework first |
-| L3: Workflow Design | Can't implement "sequential pattern" without StateGraph setup |
-| L4: Agent-Level Design | Can't implement "goal-based reasoning" without planning infrastructure |
-| L5 before L6 | Can't select components (tools) before framework installed |
-
-**Iteration Strategy → Build-Measure-Learn Cycles**
-
-**Cycle 1: MVP (Weeks 1-2)**
-
-- Stages 1-3 → Simple agent with 1-2 tools
-- Test on easiest GAIA questions (Level 1, text-only)
-- Measure baseline success rate
-- **Goal:** Prove architecture works end-to-end
-
-**Cycle 2: Enhancement (Weeks 3-4)**
-
-- Stage 4 → Add remaining tools + robustness
-- Test on validation split (mixed difficulty)
-- Analyze failure patterns by question type
-- **Goal:** Reach intermediate target (>40% overall)
-
-**Cycle 3: Optimization (Weeks 5+)**
-
-- Stage 5 → Iterate based on data
-- A/B test LLMs: Gemini Flash (free) vs Claude (premium)
-- Enhance tools based on failure analysis
-- Experiment with Reflection pattern (future)
-- **Goal:** Approach stretch target (>80% overall)
-
-**Rejected alternatives:**
-
-- Sequential L1→L8 implementation: Impossible to code high-level strategic decisions first
-- Big-bang integration: Too risky without incremental validation
-- Tool-first without framework: Cannot test tools without agent orchestration
-- Framework-first without tools: Agent has nothing to execute
-
-## Outcome
-
-Established 5-stage bottom-up implementation process aligned with architectural decisions. Each stage builds on previous infrastructure, enabling incremental validation and risk reduction.
-
-**Deliverables:**
-
-- `dev/dev_260101_10_implementation_process_design.md` - Implementation process documentation
-- `PLAN.md` - Detailed Stage 1 implementation plan (next step)
-
-**Implementation Roadmap:**
-
-- **Stage 1:** Foundation Setup (L6, L7) - Infrastructure ready
-- **Stage 2:** Tool Development (L5) - Components ready
-- **Stage 3:** Agent Core (L3, L4) - Reasoning ready
-- **Stage 4:** Integration (L6) - Robustness ready
-- **Stage 5:** Evaluation (L8) - Performance optimization
-
-**Critical Dependencies:**
-
-- Stage 2 depends on Stage 1 (need framework to test tools)
-- Stage 3 depends on Stage 2 (need tools to orchestrate)
-- Stage 4 depends on Stage 3 (need core logic to make robust)
-- Stage 5 depends on Stage 4 (need working system to evaluate)
-
-## Learnings and Insights
-
-**Pattern discovered:** Design framework order (top-down strategic) is inverse of implementation order (bottom-up tactical). Strategic planning flows from business to components, but execution flows from components to business value.
-
-**Critical insight:** Each design level informs specific implementation stage, but NOT in sequential order:
-
-- L7 → Stage 1 (infrastructure)
-- L6 → Stage 1 (framework) & Stage 4 (error handling)
-- L5 → Stage 2 (tools)
-- L3, L4 → Stage 3 (agent core)
-- L8 → Stage 5 (evaluation)
-
-**Build-Measure-Learn philosophy:** Incremental delivery with validation gates reduces risk. Each stage produces testable milestone before proceeding.
-
-**Anti-pattern avoided:** Attempting to implement strategic decisions (L1-L2) first leads to abstract code without concrete functionality. Bottom-up ensures each layer is executable and testable.
-
-## Standard Template for Future Projects
-
-**Purpose:** Convert top-down design framework into bottom-up executable implementation process.
-
-**Core Principle:** Design flows strategically (business → components), Implementation flows tactically (components → business value).
-
-### Implementation Process Template
-
-**Stage 1: Foundation Setup**
-
-- **Build From:** Infrastructure + Framework selection levels
-- **Deliverables:** Environment configured / Core dependencies installed / Basic structure runs
-- **Milestone:** Empty system executes successfully
-- **Dependencies:** None
-
-**Stage 2: Component Development**
-
-- **Build From:** Component selection level
-- **Deliverables:** Individual components as isolated units / Independent test cases per component
-- **Milestone:** Each component works standalone with validation
-- **Dependencies:** Stage 1 (need framework to test components)
-
-**Stage 3: Core Logic Implementation**
-
-- **Build From:** Workflow + Agent/System design levels
-- **Deliverables:** Orchestration structure / Decision logic / Execution flow
-- **Milestone:** System executes simple single-component tasks
-- **Dependencies:** Stage 2 (need components to orchestrate)
-
-**Stage 4: Integration & Robustness**
-
-- **Build From:** Framework implementation level (error handling)
-- **Deliverables:** All components connected / Error handling / Edge case management
-- **Milestone:** System handles multi-component tasks with recovery
-- **Dependencies:** Stage 3 (need core logic to make robust)
-
-**Stage 5: Evaluation & Iteration**
-
-- **Build From:** Evaluation level
-- **Deliverables:** Validation pipeline / Performance metrics / Failure analysis / Improvements
-- **Milestone:** Meet baseline performance target
-- **Dependencies:** Stage 4 (need working system to evaluate)
-
-### Iteration Strategy Template
-
-**Cycle Structure:**
-
-```
-Cycle N:
-  Scope: [Subset of functionality]
-  Test: [Validation criteria]
-  Measure: [Performance metric]
-  Goal: [Target threshold]
-```
-
-**Application Pattern:**
-
-- **Cycle 1:** MVP (minimal components, simplest tests)
-- **Cycle 2:** Enhancement (all components, mixed complexity)
-- **Cycle 3:** Optimization (refinement based on data)
-
-### Validation Checklist
-
-| Criterion                                                  | Pass/Fail     | Notes                            |
-|------------------------------------------------------------|---------------|----------------------------------|
-| Can Stage N be executed without Stage N-1 outputs?         | Should be NO  | Validates dependency chain       |
-| Does each stage produce testable artifacts?                | Should be YES | Ensures incremental validation   |
-| Can design level X be directly coded without lower levels? | Should be NO  | Validates bottom-up necessity    |
-| Are there circular dependencies?                           | Should be NO  | Ensures linear progression       |
-| Does each milestone have binary pass/fail?                 | Should be YES | Prevents ambiguous progress      |
-
-## Changelog
-
-**What was changed:**
-
-- Created `dev/dev_260101_10_implementation_process_design.md` - Implementation process design
-- Defined 5-stage bottom-up implementation approach
-- Mapped design framework levels to implementation stages
-- Established Build-Measure-Learn iteration cycles
-- Added "Standard Template for Future Projects" section with reusable 5-stage process, iteration strategy, and validation checklist
-- Created detailed PLAN.md for Stage 1 execution

dev/dev_260101_11_stage1_completion.md

@@ -1,105 +0,0 @@
-# [dev_260101_11] Stage 1: Foundation Setup - Completion
-
-**Date:** 2026-01-01
-**Type:** Development
-**Status:** Resolved
-**Related Dev:** dev_260101_10 (Implementation Process Design), dev_260101_12 (Isolated Environment Setup)
-
-## Problem Description
-
-Execute Stage 1 of the 5-stage bottom-up implementation process: Foundation Setup. Establish project infrastructure, dependency management, basic agent skeleton, and validation framework to prepare for tool development in Stage 2.
-
----
-
-## Key Decisions
-
-- **Dependency Management:** Used `uv` package manager with isolated project environment (102 packages total) including LangGraph, Anthropic, Google Genai, Exa, Tavily, and file parsers
-- **Environment Isolation:** Created project-specific `pyproject.toml` and `.venv/` separate from parent workspace to prevent package conflicts
-- **LangGraph StateGraph Structure:** Implemented 3-node sequential workflow (plan → execute → answer) with typed state dictionary
-- **Placeholder Implementation:** All nodes return placeholder responses to validate graph compilation and execution flow
-- **Test Organization:** Separated unit tests (test_agent_basic.py) from integration verification (test_stage1.py) in tests/ folder
-- **Configuration Validation:** Added `get_search_api_key()` method to Settings class for search tool API key retrieval
-- **Free Tier Optimization:** Set Tavily as default search tool (1000 free requests/month) vs Exa (paid tier)
-- **Gradio Integration:** Updated app.py to use GAIAAgent with logging for deployment readiness
-
-## Outcome
-
-Stage 1 Foundation Setup completed successfully. All validation checkpoints passed:
-- ✓ Isolated environment created (102 packages in local `.venv/`)
-- ✓ Project structure established (src/config, src/agent, src/tools, tests/)
-- ✓ StateGraph compiles without errors
-- ✓ Agent initialization works
-- ✓ Basic question processing returns placeholder answers
-- ✓ Configuration loading validates API keys
-- ✓ Gradio UI integration ready
-- ✓ Test suite organized and passing
-- ✓ Security setup complete (.env protected, .gitignore configured)
-
-**Deliverables:**
-
-Environment Setup:
-- `pyproject.toml` - UV project configuration (102 dependencies, dev-dependencies, hatchling build)
-- `.venv/` - Local isolated virtual environment (all packages installed here)
-- `uv.lock` - Dependency lock file for reproducible installs
-- `.gitignore` - Protection for `.env`, `.venv/`, `uv.lock`, Python artifacts
-
-Core Implementation:
-- `requirements.txt` - 102 dependencies for HF Spaces compatibility
-- `src/config/settings.py` - Configuration management with `get_search_api_key()` method, Tavily default
-- `src/agent/graph.py` - LangGraph StateGraph with AgentState TypedDict and 3 placeholder nodes
-- `src/agent/__init__.py` - GAIAAgent export
-- `src/tools/__init__.py` - Placeholder for Stage 2 tool integration
-- `app.py` - Updated with GAIAAgent integration and logging
-
-Configuration:
-- `.env.example` - Template with placeholders (safe to commit)
-- `.env` - Real API keys for local testing (gitignored)
-
-Testing:
-- `tests/__init__.py` - Test package initialization
-- `tests/test_agent_basic.py` - Unit tests (initialization, settings, basic execution, graph structure)
-- `tests/test_stage1.py` - Integration verification (configuration, agent init, end-to-end processing)
-- `tests/README.md` - Test organization documentation
-
-## Learnings and Insights
-
-- **Environment Isolation:** Creating project-specific uv environment prevents package conflicts and provides clear dependency boundaries
-- **Dual Configuration:** Maintaining both `pyproject.toml` (local dev) and `requirements.txt` (HF Spaces) ensures compatibility across environments
-- **Validation Strategy:** Separating unit tests from integration verification provides clearer validation checkpoints
-- **Configuration Pattern:** Adding tool-specific API key getters (get_llm_api_key, get_search_api_key) simplifies tool initialization logic
-- **Test Organization:** Moving test files to tests/ folder with README documentation improves project structure clarity
-- **Free Tier Priority:** Defaulting to free tier services (Gemini, Tavily) enables immediate testing without API costs
-- **Placeholder Pattern:** Using placeholder nodes in Stage 1 validates graph structure before implementing complex logic
-- **Security Best Practice:** Proper `.env` handling with `.gitignore` prevents accidental secret commits
-
-## Changelog
-
-**Created:**
-- `pyproject.toml` - UV project configuration (name="gaia-agent", 102 dependencies)
-- `.venv/` - Local isolated virtual environment
-- `uv.lock` - Auto-generated dependency lock file
-- `.gitignore` - Git ignore rules for secrets and build artifacts
-- `src/agent/graph.py` - StateGraph skeleton with 3 nodes
-- `src/agent/__init__.py` - GAIAAgent export
-- `src/tools/__init__.py` - Placeholder
-- `tests/__init__.py` - Test package
-- `tests/README.md` - Test documentation
-- `.env.example` - Configuration template with placeholders
-- `.env` - Real API keys for local testing (gitignored)
-
-**Modified:**
-- `requirements.txt` - Updated to 102 packages for isolated environment
-- `src/config/settings.py` - Added DEFAULT_SEARCH_TOOL, get_search_api_key() method
-- `app.py` - Replaced BasicAgent with GAIAAgent, added logging
-
-**Moved:**
-- `test_stage1.py` → `tests/test_stage1.py` - Organized test files
-
-**Installation Commands:**
-```bash
-uv venv                           # Created isolated .venv
-uv sync                          # Installed 102 packages from pyproject.toml
-uv run python tests/test_stage1.py  # Validated with isolated environment
-```
-
-**Next Stage:** Stage 2: Tool Development - Implement web search (Tavily/Exa), file parsing (PDF/Excel/images), calculator tools with retry logic and error handling.

dev/dev_260101_12_isolated_environment_setup.md

@@ -1,188 +0,0 @@
-# [dev_260101_12] Isolated Environment Setup
-
-**Date:** 2026-01-01
-**Type:** Issue
-**Status:** Resolved
-**Related Dev:** dev_260101_11 (Stage 1 Completion)
-
-## Problem Description
-
-Environment confusion arose during Stage 1 validation. The HF project existed as a subdirectory within parent `/Users/mangobee/Documents/Python` (uv-managed workspace), but had only `requirements.txt` without project-specific environment configuration.
-
-**Core Issues:**
-
-1. Unclear where `uv pip install` installs packages (parent's `.venv` vs project-specific location)
-2. Package installation incomplete - some packages (google-genai, tavily-python) not found in parent environment
-3. Mixing parent's pyproject.toml dependencies with HF project dependencies causes potential conflicts
-4. `.env` vs `.env.example` confusion - user accidentally put real API keys in template file
-5. No `.gitignore` file - risk of committing secrets to git
-
-**Root Cause:** HF project treated as subdirectory without isolated environment, creating dependency confusion and security risks.
-
----
-
-## Key Decisions
-
-- **Isolated uv Environment:** Create project-specific `.venv/` within HF project directory, managed by its own `pyproject.toml`
-- **Dual Configuration Strategy:** Maintain both `pyproject.toml` (local development) and `requirements.txt` (HF Spaces compatibility)
-- **Environment Separation:** Complete isolation from parent's `.venv/` to prevent package conflicts
-- **Security Setup:** Proper `.env` file handling with `.gitignore` protection
-- **Package Source:** Install all 102 packages directly into project's `.venv/lib/python3.12/site-packages`
-
-**Rejected Alternatives:**
-
-- Using parent's shared `.venv/` - rejected due to package conflict risks and unclear dependency boundaries
-- HF Spaces-only testing without local environment - rejected due to slow iteration cycles
-- Manual virtual environment (python -m venv) - rejected in favor of uv's superior dependency management
-
-## Outcome
-
-Successfully established isolated uv environment for HF project with complete dependency isolation from parent workspace.
-
-**Validation Results:**
-
-- ✓ All 102 packages installed in local `.venv/` (tavily-python, google-genai, anthropic, langgraph, etc.)
-- ✓ Configuration loads correctly (LLM=gemini, Search=tavily)
-- ✓ All Stage 1 tests passing with isolated environment
-- ✓ Security setup complete (.env protected, .gitignore configured)
-- ✓ Imports working: `from src.agent import GAIAAgent; from src.config import Settings`
-
-**Deliverables:**
-
-Environment Configuration:
-
-- `pyproject.toml` - UV project configuration with 102 dependencies, dev-dependencies (pytest, pytest-asyncio), hatchling build backend
-- `.venv/` - Local isolated virtual environment (gitignored)
-- `uv.lock` - Auto-generated lock file for reproducible installs (gitignored)
-- `.gitignore` - Protection for `.env`, `.venv/`, `uv.lock`, Python artifacts
-
-Security Setup:
-
-- `.env.example` - Template with placeholders (safe to commit)
-- `.env` - Real API keys for local testing (gitignored)
-- API keys verified: ANTHROPIC_API_KEY, GOOGLE_API_KEY, TAVILY_API_KEY, EXA_API_KEY
-- SPACE_ID configured: mangoobee/Final_Assignment_Template
-
-## Learnings and Insights
-
-- **uv Workspace Behavior:** When running `uv pip install` from subdirectory without local `pyproject.toml`, uv searches upward and uses parent's `.venv/`, creating hidden dependencies
-- **Dual Configuration Pattern:** Maintaining both `pyproject.toml` (uv local dev) and `requirements.txt` (HF Spaces deployment) ensures compatibility across environments
-- **Security Best Practice:** Never put real API keys in `.env.example` - it's a template file that gets committed to git
-- **Hatchling Requirement:** When using hatchling build backend, must specify `packages = ["src"]` in `[tool.hatch.build.targets.wheel]` to avoid build errors
-- **Package Location Verification:** Always verify package installation location with `uv pip show <package>` to confirm expected environment isolation
-- **uv sync vs uv pip install:** `uv sync` reads from `pyproject.toml` and creates lockfile; `uv pip install` is lower-level and doesn't modify project configuration
-
-## Changelog
-
-**Created:**
-
-- `pyproject.toml` - UV project configuration (name="gaia-agent", 102 dependencies)
-- `.venv/` - Local isolated virtual environment
-- `uv.lock` - Auto-generated dependency lock file
-- `.gitignore` - Git ignore rules for secrets and build artifacts
-- `.env` - Local API keys (real secrets, gitignored)
-
-**Modified:**
-
-- `.env.example` - Restored placeholders (removed accidentally committed real API keys)
-
-**Commands Executed:**
-
-```bash
-uv venv                           # Create isolated .venv
-uv sync                          # Install all dependencies from pyproject.toml
-uv pip show tavily-python        # Verify package location
-uv run python tests/test_stage1.py  # Validate with isolated environment
-```
-
-**Validation Evidence:**
-
-```
-tavily-python: .../Final_Assignment_Template/.venv/lib/python3.12/site-packages
-google-genai: .../Final_Assignment_Template/.venv/lib/python3.12/site-packages
-✓ All Stage 1 tests passing
-✓ Configuration loaded correctly
-```
-
-**Next Steps:** Environment setup complete - ready to proceed with Stage 2: Tool Development or deploy to HF Spaces for integration testing.
-
----
-
-## Reference: Environment Management Guide
-
-**Strategy:** This HF project has its own isolated virtual environment managed by uv, separate from the parent `Python` folder.
-
-### Project Structure
-
-```
-16_HuggingFace/Final_Assignment_Template/
-├── .venv/              # LOCAL isolated virtual environment
-├── pyproject.toml      # UV project configuration
-├── uv.lock            # Lock file (auto-generated, gitignored)
-├── requirements.txt    # For HF Spaces compatibility
-└── .env               # Local API keys (gitignored)
-```
-
-### How It Works
-
-**Local Development:**
-
-- Uses local `.venv/` with uv-managed packages
-- All 102 packages installed in isolation
-- No interference with parent `/Users/mangobee/Documents/Python/.venv`
-
-**HuggingFace Spaces Deployment:**
-
-- Reads `requirements.txt` (not pyproject.toml)
-- Creates its own cloud environment
-- Reads API keys from HF Secrets (not .env)
-
-### Common Commands
-
-**Run Python code:**
-
-```bash
-uv run python app.py
-uv run python tests/test_stage1.py
-```
-
-**Add new package:**
-
-```bash
-uv add package-name          # Adds to pyproject.toml + installs
-```
-
-**Install dependencies:**
-
-```bash
-uv sync                      # Install from pyproject.toml
-```
-
-**Update requirements.txt for HF Spaces:**
-
-```bash
-uv pip freeze > requirements.txt  # Export current packages
-```
-
-### Package Locations Verified
-
-All packages installed in LOCAL `.venv/`:
-
-```
-tavily-python: .../Final_Assignment_Template/.venv/lib/python3.12/site-packages
-google-genai: .../Final_Assignment_Template/.venv/lib/python3.12/site-packages
-```
-
-NOT in parent's `.venv/`:
-
-```
-Parent: /Users/mangobee/Documents/Python/.venv  (isolated)
-HF:     /Users/mangobee/.../Final_Assignment_Template/.venv  (isolated)
-```
-
-### Key Benefits
-
-✓ **Isolation:** No package conflicts between projects
-✓ **Clean:** Each project manages its own dependencies
-✓ **Compatible:** Still works with HF Spaces via requirements.txt
-✓ **Reproducible:** uv.lock ensures consistent installs

dev/dev_260102_13_stage2_tool_development.md

@@ -1,305 +0,0 @@
-# [dev_260102_13] Stage 2: Tool Development Complete
-
-**Date:** 2026-01-02
-**Type:** Development
-**Status:** Resolved
-**Related Dev:** dev_260101_11 (Stage 1 Foundation Setup)
-
-## Problem Description
-
-Stage 1 established the LangGraph StateGraph skeleton with placeholder nodes. Stage 2 needed to implement the actual tools that the agent would use to answer GAIA benchmark questions, including web search, file parsing, mathematical computation, and multimodal image analysis.
-
-**Root cause:** GAIA questions require external tool use (web search, file reading, calculations, image analysis). Stage 1 had no actual tool implementations - just placeholders.
-
----
-
-## Key Decisions
-
-### Decision 1: Direct API Implementation vs MCP Servers
-
-**Chosen:** Direct Python function implementations for all tools
-
-**Why:**
-
-- HuggingFace Spaces doesn't support running MCP servers (requires separate processes)
-- Direct API approach is simpler and more reliable for deployment
-- Full control over retry logic, error handling, and timeouts
-- MCP servers are external dependencies with additional failure points
-
-**Rejected alternative:** Using MCP protocol servers for Tavily/Exa
-
-- Would require complex Docker configuration on HF Spaces
-- Additional process management overhead
-- Not necessary for MVP stage
-
-### Decision 2: Retry Logic with Tenacity
-
-**Chosen:** Use `tenacity` library with exponential backoff, max 3 retries
-
-**Why:**
-
-- Industry-standard retry library with clean decorator syntax
-- Exponential backoff prevents API rate limit issues
-- Configurable retry conditions (only retry on connection errors, not on validation errors)
-- Easy to test with mocking
-
-**Configuration:**
-
-- Max retries: 3
-- Min wait: 1 second
-- Max wait: 10 seconds
-- Retry only on: ConnectionError, TimeoutError, IOError (for file operations)
-
-### Decision 3: Tool Architecture - Unified Functions with Fallback
-
-**Pattern applied to all tools:**
-
-- Primary implementation (e.g., `tavily_search`)
-- Fallback implementation (e.g., `exa_search`)
-- Unified function with automatic fallback (e.g., `search`)
-
-**Example:**
-
-```python
-def search(query):
-    if default_tool == "tavily":
-        try:
-            return tavily_search(query)
-        except:
-            return exa_search(query)  # Fallback
-```
-
-**Why:** Maximizes reliability - if primary service fails, automatic fallback ensures tool still works
-
-### Decision 4: Calculator Security - AST-based Evaluation
-
-**Chosen:** Custom AST visitor with whitelisted operations only
-
-**Why:**
-
-- Python's `eval()` is dangerous (arbitrary code execution)
-- `ast.literal_eval()` is too restrictive (doesn't support math operations)
-- Custom AST visitor allows precise control over allowed operations
-- Timeout protection prevents infinite loops
-- Whitelist approach: only allow known-safe operations (add, multiply, sin, cos, etc.)
-
-**Rejected alternatives:**
-
-- Using `eval()`: Major security vulnerability
-- Using `sympify()` from sympy: Too complex, allows too much
-
-**Security layers:**
-
-1. AST whitelist (only allow specific node types)
-2. Expression length limit (500 chars)
-3. Number size limit (prevent huge calculations)
-4. Timeout protection (2 seconds max)
-5. No attribute access, no imports, no exec/eval
-
-### Decision 5: File Parser - Generic Dispatcher Pattern
-
-**Chosen:** Single `parse_file()` function that dispatches based on extension
-
-```python
-def parse_file(file_path):
-    extension = Path(file_path).suffix.lower()
-    if extension == '.pdf':
-        return parse_pdf(file_path)
-    elif extension in ['.xlsx', '.xls']:
-        return parse_excel(file_path)
-    # ... etc
-```
-
-**Why:**
-
-- Simple interface for users (one function for all file types)
-- Easy to add new file types (just add new parser and update dispatcher)
-- Each parser can have format-specific logic
-- Fallback to specific parsers still available for advanced use
-
-### Decision 6: Vision Tool - Gemini as Default with Claude Fallback
-
-**Chosen:** Gemini 2.0 Flash as primary, Claude Sonnet 4.5 as fallback
-
-**Why:**
-
-- Gemini 2.0 Flash: Free tier (1500 req/day), fast, good quality
-- Claude Sonnet 4.5: Paid but highest quality, automatic fallback if Gemini fails
-- Same pattern as web search (primary + fallback = reliability)
-
-**Image handling:**
-
-- Load file, encode as base64
-- Check file size (max 10MB)
-- Support common formats (JPG, PNG, GIF, WEBP, BMP)
-- Return structured answer with model metadata
-
-## Outcome
-
-Successfully implemented 4 production-ready tools with comprehensive error handling and test coverage.
-
-**Deliverables:**
-
-1. **Web Search Tool** ([src/tools/web_search.py](./../src/tools/web_search.pyeb_search.py))
-
-   - Tavily API integration (primary, free tier)
-   - Exa API integration (fallback, paid)
-   - Automatic fallback if primary fails./../test/test_web_search.py
-   - 10 passing tests in [test/test_web_search.py](../../agentbee/test/test_web_search.py)
-     ./../src/tools/file_parser.py
-
-2. **File Parser Tool** ([src/tools/file_parser.py](../../agentbee/src/tools/file_parser.py))
-
-   - PDF parsing (PyPDF2)
-   - Excel parsing (openpyxl)
-   - Word parsing (python-docx)
-   - Text/CSV parsing (built-in open)./../test/test_file_parser.py
-   - Generic `parse_file()` dispatcher
-   - 19 passing tests in [test/test_file_parser.py./../src/tools/calculator.py_file_parser.py)
-
-3. **Calculator Tool** ([src/tools/calculator.py](../../agentbee/src/tools/calculator.py))
-
-   - Safe AST-based expression evaluation
-   - Whitelisted operations only (no code execution./../test/test_calculator.py
-   - Mathematical functions (sin, cos, sqrt, factorial, etc.)
-   - Security hardened (timeout, complexit./../src/tools/vision.py
-   - 41 passing tests in [test/test_calculator.py](../../agentbee/test/test_calculator.py)
-
-4. **Vision Tool** ([src/tools/vision.py](../../agentbee/src/tools/vision.py))
-
-   - Multimodal image analysis using LLMs./../test/test_vision.py
-   - Gemini 2.0 Flash (primary, free)
-   - Claude Sonnet 4.5 (fallback, paid)./../src/tools/**init**.py
-   - Image loading and base64 encoding
-   - 15 passing tests in [test/test_vision.py](../../agentbee/test/test_vision.py)
-
-5. **Tool Registry** ([src/tools/**init**.py](../../agentbee/src/tools/__init__.py))
-   ./../src/agent/graph.py
-
-   - Exports all 4 main tools: `search`, `parse_file`, `safe_eval`, `analyze_image`
-   - TOOLS dict with metadata (description, parameters, category)
-   - Ready for Stage 3 dynamic tool selection
-
-6. **StateGraph Integration** ([src/agent/graph.py](../../agentbee/src/agent/graph.py))
-   - Updated `execute_node` to load tool registry
-   - Stage 2: Reports tool availability
-   - Stage 3: Will add dynamic tool selection and execution
-
-**Test Coverage:**
-
-- 85 tool tests passing (web_search: 10, file_parser: 19, calculator: 41, vision: 15)
-- 6 existing agent tests still passing
-- 91 total tests passing
-- No regressions from Stage 1
-
-**Deployment:**
-
-- All changes committed and pushed to HuggingFace Spaces
-- Build succeeded
-- Agent now reports: "Stage 2 complete: 4 tools ready for execution in Stage 3"
-
-## Learnings and Insights
-
-### Pattern: Unified Function with Fallback
-
-This pattern worked extremely well for both web search and vision tools:
-
-```python
-def tool_name(args):
-    # Try primary service
-    try:
-        return primary_implementation(args)
-    except Exception as e:
-        logger.warning(f"Primary failed: {e}")
-        # Fallback to secondary
-        try:
-            return fallback_implementation(args)
-        except Exception as fallback_error:
-            raise Exception(f"Both failed")
-```
-
-**Why it works:**
-
-- Maximizes reliability (2 chances to succeed)
-- Transparent to users (single function call)
-- Preserves cost optimization (use free tier first, paid only as fallback)
-
-**Recommendation:** Use this pattern for any tool with multiple service providers.
-
-### Pattern: Test Fixtures for File Parsers
-
-Creating real test fixtures (sample.pdf, sample.xlsx, etc.) was critical for file parser testing:
-
-**What worked:**
-
-- Tests are realistic (test actual file parsing, not just mocks)
-- Easy to add new test cases (just add new fixture files)
-- Catches edge cases that mocks miss
-
-**Created fixtures:**
-
-- `test/fixtures/sample.txt` - Plain text
-- `test/fixtures/sample.csv` - CSV data
-- `test/fixtures/sample.xlsx` - Excel spreadsheet
-- `test/fixtures/sample.docx` - Word document
-- `test/fixtures/test_image.jpg` - Test image (red square)
-- `test/fixtures/generate_fixtures.py` - Script to regenerate fixtures
-
-**Recommendation:** For any file processing tool, create comprehensive fixture library.
-
-### What Worked Well: Mock Path for Import Testing
-
-Initially had issues with mock paths like `src.tools.vision.genai.Client`. The fix:
-
-```python
-# WRONG: src.tools.vision.genai.Client
-# RIGHT: google.genai.Client
-with patch('google.genai.Client') as mock_client:
-    # Mock the original import, not the re-export
-```
-
-**Lesson:** Always mock the original module path, not where it's imported into your code.
-
-### What to Avoid: Premature Integration Testing
-
-Initially planned to create `tests/test_tools_integration.py` for cross-tool testing. **Decision:** Skip for Stage 2.
-
-**Why:**
-
-- Tools work independently (don't need to interact yet)
-- Integration testing makes sense in Stage 3 when tools are orchestrated
-- Unit tests provide sufficient coverage for Stage 2
-
-**Recommendation:** Only write integration tests when components actually integrate. Don't test imaginary integration.
-
-## Changelog
-
-**What was created:**
-
-- `src/tools/web_search.py` - Tavily/Exa web search with retry logic
-- `src/tools/file_parser.py` - PDF/Excel/Word/Text parsing with retry logic
-- `src/tools/calculator.py` - Safe AST-based math evaluation
-- `src/tools/vision.py` - Multimodal image analysis (Gemini/Claude)
-- `test/test_web_search.py` - 10 tests for web search tool
-- `test/test_file_parser.py` - 19 tests for file parser
-- `test/test_calculator.py` - 41 tests for calculator (including security)
-- `test/test_vision.py` - 15 tests for vision tool
-- `test/fixtures/sample.txt` - Test text file
-- `test/fixtures/sample.csv` - Test CSV file
-- `test/fixtures/sample.xlsx` - Test Excel file
-- `test/fixtures/sample.docx` - Test Word document
-- `test/fixtures/test_image.jpg` - Test image
-- `test/fixtures/generate_fixtures.py` - Fixture generation script
-
-**What was modified:**
-
-- `src/tools/__init__.py` - Added tool exports and TOOLS registry
-- `src/agent/graph.py` - Updated execute_node to load tool registry
-- `requirements.txt` - Added `tenacity>=8.2.0` for retry logic
-- `pyproject.toml` - Installed tenacity, fpdf2, defusedxml packages
-- `PLAN.md` - Emptied for next stage
-- `TODO.md` - Emptied for next stage
-
-**What was deleted:**
-
-- None (Stage 2 was purely additive)

dev/dev_260102_14_stage3_core_logic.md

@@ -1,207 +0,0 @@
-# [dev_260102_14] Stage 3: Core Logic Implementation with Multi-Provider LLM
-
-**Date:** 2026-01-02
-**Type:** Development
-**Status:** Resolved
-**Related Dev:** dev_260102_13
-
-## Problem Description
-
-Implemented Stage 3 core agent logic with LLM-based decision making for planning, tool selection, and answer synthesis. Fixed inconsistency where Stage 2 used Gemini primary + Claude fallback pattern, but initial Stage 3 implementation only used Claude.
-
----
-
-## Key Decisions
-
-**Decision 1: Multi-Provider LLM Architecture → Gemini primary, Claude fallback**
-
-- **Reasoning:** Match Stage 2 tool pattern for codebase consistency
-- **Evidence:** Stage 2 vision tool uses `analyze_image_gemini()` with `analyze_image_claude()` fallback
-- **Pattern applied:** Free tier first (Gemini 2.0 Flash, 1500 req/day), paid fallback (Claude Sonnet 4.5)
-- **Implication:** Cost optimization while maintaining reliability through automatic fallback
-- **Consistency:** All LLM operations now follow same pattern as tools
-
-**Decision 2: LLM-Based Planning → Dynamic question analysis**
-
-- **Implementation:** `plan_question()` calls LLM to analyze question and generate step-by-step plan
-- **Reasoning:** GAIA questions vary widely - cannot use static planning
-- **LLM determines:** Which tools needed, execution order, parameter extraction strategy
-- **Framework alignment:** Level 3 decision (Dynamic planning)
-
-**Decision 3: Tool Selection → LLM function calling**
-
-- **Implementation:** `select_tools_with_function_calling()` uses native function calling API
-- **Claude:** `tools` parameter with `tool_use` response parsing
-- **Gemini:** `genai.protos.Tool` with `function_call` response parsing
-- **Reasoning:** LLM extracts tool names and parameters from natural language questions
-- **Framework alignment:** Level 6 decision (LLM function calling for tool selection)
-
-**Decision 4: Answer Synthesis → LLM-generated factoid answers**
-
-- **Implementation:** `synthesize_answer()` calls LLM to extract factoid from evidence
-- **Reasoning:** Evidence from multiple tools needs intelligent synthesis
-- **Prompt engineering:** Explicit factoid format requirements (number, few words, comma-separated list)
-- **Conflict resolution:** Integrated into synthesis prompt (evaluate credibility and recency)
-- **Framework alignment:** Level 5 decision (LLM-generated answer synthesis)
-
-**Decision 5: State Schema Expansion → Evidence tracking**
-
-- **Added fields:** `file_paths`, `tool_results`, `evidence`
-- **Reasoning:** Need to track evidence flow from tools to answer synthesis
-- **Evidence format:** `"[tool_name] result_text"` for clear source attribution
-- **Usage:** Answer synthesis node uses evidence list, not raw tool results
-
-**Rejected alternatives:**
-
-- Claude-only implementation: Inconsistent with Stage 2, no free tier option
-- Template-based answer synthesis: Insufficient for diverse GAIA questions requiring reasoning
-- Static tool routing: Cannot handle dynamic GAIA question requirements
-- Separate conflict resolution step: Adds complexity, integrated into synthesis instead
-
-## Outcome
-
-Successfully implemented Stage 3 with multi-provider LLM support. Agent now performs end-to-end question answering: planning → tool execution → answer synthesis, using Gemini as primary LLM (free tier) with Claude as fallback (paid).
-
-**Deliverables:**
-
-1. **LLM Client Module** ([src/agent/llm_client.py](./../src/agent/llm_client.pylm_client.py))
-
-   - Gemini implementation: 3 functions (planning, tool selection, answer synthesis)
-   - Claude implementation: 3 functions (same)
-   - Unified API with automatic fallback
-   - 624 lines of code
-     ./../src/agent/graph.py
-
-2. **Updated Agent Graph** ([src/agent/graph.py](../../agentbee/src/agent/graph.py))
-
-   - plan_node: Calls `plan_question()` for LLM-based planning
-   - execute_node: Calls `select_tools_with_function_calling()` + executes tools + collects evidence
-   - answer_node: Calls `synthesize_answer()` for factoid generation
-   - Updated AgentState with new fields./../test/test_llm_integration.py
-
-3. **LLM Integration Tests** ([test/test_llm_integration.py](../../agentbee/test/test_llm_integration.py))
-
-   - 8 tests covering all 3 LLM functions
-   - Tests use mocked LLM responses (provider-agno./../test/test_stage3_e2e.py
-   - Full workflow test: planning → tool selection → answer synthesis
-
-4. **E2E Test Script** ([test/test_stage3_e2e.py](../../agentbee/test/test_stage3_e2e.py))
-   - Manual test script for real API testing
-   - Requires ANTHROPIC_API_KEY or GOOGLE_API_KEY
-   - Tests simple math and factual questions
-
-**Test Coverage:**
-
-- All 99 tests passing (Stage 1: 6, Stage 2: 85, Stage 3: 8)
-- No regressions from previous stages
-- Multi-provider architecture tested with mocks
-
-**Deployment:**
-
-- Committed and pushed to HuggingFace Spaces
-- Build successful
-- Agent now supports both Gemini (free) and Claude (paid) LLMs
-
-## Learnings and Insights
-
-### Pattern: Free Primary + Paid Fallback
-
-**Discovered:** Consistent pattern across all external services maximizes cost efficiency
-
-**Evidence:**
-
-- Vision tool: Gemini → Claude
-- Web search: Tavily → Exa
-- LLM operations: Gemini → Claude
-
-**Recommendation:** Apply this pattern to all dual-provider integrations. Free tier first, premium fallback.
-
-### Pattern: Provider-Specific API Differences
-
-**Challenge:** Gemini and Claude have different function calling APIs
-
-**Gemini:**
-
-```python
-genai.protos.Tool(
-    function_declarations=[...]
-)
-response.parts[0].function_call
-```
-
-**Claude:**
-
-```python
-tools=[{"name": ..., "input_schema": ...}]
-response.content[0].tool_use
-```
-
-**Solution:** Separate implementation functions, unified API wrapper. Abstraction handles provider differences.
-
-### Anti-Pattern: Hardcoded Provider Selection
-
-**Initial mistake:** Hardcoded Claude client creation in all functions
-
-**Problem:** Forces paid tier usage even when free tier available
-
-**Fix:** Try-except fallback pattern allows graceful degradation
-
-**Lesson:** Never hardcode provider selection when multiple providers available. Always implement fallback chain.
-
-### What Worked Well: Evidence-Based State Design
-
-**Decision:** Add `evidence` field separate from `tool_results`
-
-**Why it worked:**
-
-- Clean separation: raw results vs. formatted evidence
-- Answer synthesis only needs evidence strings, not full tool metadata
-- Format: `"[tool_name] result"` provides source attribution
-
-**Recommendation:** Design state schema based on actual usage patterns, not just data storage.
-
-### What to Avoid: Mixing Planning and Execution
-
-**Temptation:** Let tool selection node also execute tools
-
-**Why avoided:**
-
-- Clean separation of concerns (planning vs execution)
-- Matches sequential workflow (Level 3 decision)
-- Easier to debug and test each node independently
-
-**Lesson:** Keep node responsibilities focused. One node = one responsibility.
-
-## Changelog
-
-**What was created:**
-
-- `src/agent/llm_client.py` - Multi-provider LLM client (624 lines)
-  - Gemini implementation: plan_question_gemini, select_tools_gemini, synthesize_answer_gemini
-  - Claude implementation: plan_question_claude, select_tools_claude, synthesize_answer_claude
-  - Unified API: plan_question, select_tools_with_function_calling, synthesize_answer
-- `test/test_llm_integration.py` - 8 LLM integration tests
-- `test/test_stage3_e2e.py` - Manual E2E test script
-
-**What was modified:**
-
-- `src/agent/graph.py` - Updated all three nodes with Stage 3 logic
-  - plan_node: LLM-based planning (lines 51-84)
-  - execute_node: LLM function calling + tool execution (lines 87-177)
-  - answer_node: LLM-based answer synthesis (lines 179-218)
-  - AgentState: Added file_paths, tool_results, evidence fields (lines 31-44)
-- `requirements.txt` - Already included anthropic>=0.39.0 and google-genai>=0.2.0
-- `PLAN.md` - Created Stage 3 implementation plan
-- `TODO.md` - Tracked Stage 3 tasks
-- `CHANGELOG.md` - Documented Stage 3 changes
-
-**Dependencies added:**
-
-- `google-generativeai>=0.8.6` - Gemini SDK (installed via uv)
-
-**Framework alignment verified:**
-
-- ✅ Level 3: Dynamic planning with sequential execution
-- ✅ Level 4: Goal-based reasoning, fixed-step termination (plan → execute → answer → END)
-- ✅ Level 5: LLM-generated answer synthesis, LLM-based conflict resolution
-- ✅ Level 6: LLM function calling for tool selection, LLM-based parameter extraction

dev/dev_260102_15_stage4_mvp_real_integration.md

@@ -1,409 +0,0 @@
-# [dev_260102_15] Stage 4: MVP - Real Integration
-
-**Date:** 2026-01-02 to 2026-01-03
-**Type:** Development
-**Status:** Resolved
-**Related Dev:** dev_260102_14_stage3_core_logic.md, dev_260103_16_huggingface_llm_integration.md
-
-## Problem Description
-
-**Context:** After Stage 3 core logic implementation, agent was deployed to HuggingFace Spaces for real GAIA testing. Result: 0/20 questions correct with all answers = "Unable to answer: No evidence collected".
-
-**Root Causes:**
-
-1. **Silent LLM Failures:** Function calling errors swallowed, no diagnostic visibility
-2. **Tool Execution Broken:** Evidence collection failing but continuing silently
-3. **No Error Visibility:** User sees "Unable to answer" with zero debug info
-4. **API Integration Issues:** Environment variables, network errors, quota limits not handled
-
-**Objective:** Fix integration issues to achieve MVP. Target: 0/20 → 5/20 questions answered (quality doesn't matter, just prove APIs work).
-
----
-
-## Key Decisions
-
-### **Decision 1: Comprehensive Debug Logging Over Silent Failures**
-
-**Why chosen:**
-
-- ✅ Visibility into where integration breaks (LLM? Tools? Network?)
-- ✅ Each node logs inputs, outputs, errors with full context
-- ✅ State transitions tracked for debugging flow issues
-- ✅ Production-ready logging infrastructure for future stages
-
-**Implementation:**
-
-- Added detailed logging in `plan_node`, `execute_node`, `answer_node`
-- Log LLM provider used, tool calls made, evidence collected
-- Full error stack traces with context
-
-**Result:** Can now diagnose failures from HuggingFace Space logs
-
-### **Decision 2: Actionable Error Messages Over Generic Failures**
-
-**Previous:** `"Unable to answer: No evidence collected"`
-**New:** `"ERROR: No evidence. Errors: Gemini 429 quota exceeded, Claude 400 credit low, Tavily timeout"`
-
-**Why chosen:**
-
-- ✅ Users understand WHY it failed (API key missing? Quota? Network?)
-- ✅ Developers can fix root cause without re-running
-- ✅ Gradio UI shows diagnostics instead of hiding failures
-
-**Trade-offs:**
-
-- **Pro:** Debugging 10x faster with actionable feedback
-- **Con:** Longer error messages (acceptable for MVP)
-
-### **Decision 3: API Key Validation at Startup Over First-Use Failures**
-
-**Why chosen:**
-
-- ✅ Fail fast with clear message listing missing keys
-- ✅ Prevents wasting time on runs that will fail anyway
-- ✅ Non-blocking warnings (continues anyway for partial API availability)
-
-**Implementation:**
-
-```python
-def validate_environment() -> List[str]:
-    """Check API keys at startup."""
-    missing = []
-    for key in ["GOOGLE_API_KEY", "HF_TOKEN", "ANTHROPIC_API_KEY", "TAVILY_API_KEY"]:
-        if not os.getenv(key):
-            missing.append(key)
-    if missing:
-        logger.warning(f"⚠️ Missing API keys: {', '.join(missing)}")
-    return missing
-```
-
-**Result:** Immediate feedback on configuration issues
-
-### **Decision 4: Graceful LLM Fallback Chain Over Single Provider Dependency**
-
-**Final Architecture:**
-
-1. **Gemini 2.0 Flash** (free, 1,500 req/day) - Primary
-2. **HuggingFace Qwen 2.5 72B** (free, rate limited) - Middle tier (added later)
-3. **Claude Sonnet 4.5** (paid, credits) - Expensive fallback
-4. **Keyword matching** (deterministic) - Last resort
-
-**Why 3-tier free-first:**
-
-- ✅ Maximizes free tier usage before burning paid credits
-- ✅ Different quota models (daily vs rate-limited) provide resilience
-- ✅ Guarantees agent never completely fails (keyword fallback)
-
-**Trade-offs:**
-
-- **Pro:** 4 layers of resilience, cost-optimized
-- **Con:** Slightly higher latency on fallback traversal (acceptable)
-
-### **Decision 5: Tool Execution Fallback Over Hard Failures**
-
-**Problem:** If LLM function calling returns empty tool_calls, execution would continue silently
-
-**Solution:**
-
-```python
-tool_calls = select_tools_with_function_calling(...)
-
-if not tool_calls:
-    logger.warning("LLM function calling failed, using keyword fallback")
-    # Simple heuristics: "search" in question → use web_search
-    tool_calls = fallback_tool_selection(question)
-```
-
-**Why chosen:**
-
-- ✅ MVP priority: Get SOMETHING working even if LLM fails
-- ✅ Keyword matching better than no tools at all
-- ✅ Temporary hack acceptable for MVP validation
-
-**Result:** Agent can still collect evidence when LLM function calling broken
-
-### **Decision 6: Gradio Diagnostics Display Over Answer-Only UI**
-
-**Why chosen:**
-
-- ✅ Users see plan, tools selected, evidence, errors in real-time
-- ✅ Debugging possible without checking logs
-- ✅ Test & Debug tab shows API key status
-- ✅ Transparency builds user trust
-
-**Implementation:**
-
-- `format_diagnostics()` function formats state for display
-- Test & Debug tab shows: API keys, plan, tools, evidence, errors, final answer
-
-**Result:** Self-service debugging for users
-
-### **Decision 7: TOOLS Schema Fix - Dict Format Over List Format (CRITICAL)**
-
-**Problem Discovered:** `src/tools/__init__.py` had parameters as list `["query"]` but LLM client expected dict `{"query": {"type": "string", "description": "..."}}`.
-
-**Impact:** Gemini function calling completely broken - `'list' object has no attribute 'items'` error.
-
-**Fix:** Updated all tool definitions to proper schema:
-
-```python
-"parameters": {
-    "query": {
-        "description": "Search query string",
-        "type": "string"
-    },
-    "max_results": {
-        "description": "Maximum number of search results",
-        "type": "integer"
-    }
-},
-"required_params": ["query"]
-```
-
-**Result:** Gemini function calling now working correctly (verified in tests)
-
----
-
-## Outcome
-
-Successfully achieved MVP: Agent operational with real API integration, 10% GAIA score (2/20 correct), proving APIs connected and evidence collection working.
-
-**Deliverables:**
-
-### 1. src/agent/graph.py (~100 lines added/modified)
-
-- Added `validate_environment()` - API key validation at startup
-- Updated `plan_node` - Comprehensive logging, error context
-- Updated `execute_node` - Fallback tool selection when LLM fails
-- Updated `answer_node` - Actionable error messages with error summary
-- Added state inspection logging throughout execution flow
-
-### 2. src/agent/llm_client.py (~200 lines added - includes HF integration)
-
-- Improved exception handling with specific error types
-- Distinguished: API key missing, rate limit, network error, API error
-- Added `create_hf_client()` - HuggingFace InferenceClient initialization
-- Added `plan_question_hf()`, `select_tools_hf()`, `synthesize_answer_hf()`
-- Updated unified functions to use 3-tier fallback (Gemini → HF → Claude)
-- Log which provider failed and why
-
-### 3. app.py (~100 lines added/modified)
-
-- Added `format_diagnostics()` - Format agent state for display
-- Updated Test & Debug tab - Shows API key status, plan, tools, evidence, errors
-- Added `check_api_keys()` - Display all API key statuses (GOOGLE, HF, ANTHROPIC, TAVILY, EXA)
-- Updated UI to show diagnostics alongside answers
-- Added export functionality (later enhanced to JSON in dev_260104_17)
-
-### 4. src/tools/__init__.py
-
-- Fixed TOOLS schema bug - Changed parameters from list to dict format
-- Added type/description for each parameter
-- Added `"required_params"` field
-- Fixed Gemini function calling compatibility
-
-**GAIA Validation Results:**
-
-- **Score:** 10.0% (2/20 correct)
-- **Improvement:** 0/20 → 2/20 (MVP validated!)
-- **Success Cases:**
-  - Question 3: Reverse text reasoning → "right" ✅
-  - Question 5: Wikipedia search → "FunkMonk" ✅
-
-**Test Results:**
-
-```bash
-uv run pytest test/ -q
-99 passed, 11 warnings in 51.99s ✅
-```
-
----
-
-## Learnings and Insights
-
-### **Pattern: Free-First Fallback Architecture**
-
-**What worked well:**
-
-- Prioritizing free tiers (Gemini → HuggingFace) before paid tier (Claude) maximizes cost efficiency
-- Multiple free alternatives with different quota models (daily vs rate-limited) provide better resilience than single free tier
-- Keyword fallback ensures agent never completely fails even when all LLMs unavailable
-
-**Reusable pattern:**
-
-```python
-def unified_llm_function(...):
-    """3-tier fallback with comprehensive error capture."""
-    errors = []
-
-    try:
-        return free_tier_1(...)  # Gemini - daily quota
-    except Exception as e1:
-        errors.append(f"Tier 1: {e1}")
-        try:
-            return free_tier_2(...)  # HuggingFace - rate limited
-        except Exception as e2:
-            errors.append(f"Tier 2: {e2}")
-            try:
-                return paid_tier(...)  # Claude - credits
-            except Exception as e3:
-                errors.append(f"Tier 3: {e3}")
-                # Deterministic fallback as last resort
-                return keyword_fallback(...)
-```
-
-### **Pattern: Function Calling Schema Compatibility**
-
-**Critical insight:** Different LLM providers require different function calling schemas.
-
-1. **Gemini:** `genai.protos.Tool` with `function_declarations`
-2. **HuggingFace:** OpenAI-compatible tools array format
-3. **Claude:** Anthropic native format with `input_schema`
-
-**Best practice:** Maintain single source of truth in `src/tools/__init__.py` with rich schema (dict format with type/description), then transform to provider-specific format in LLM client functions.
-
-### **Pattern: Environment Validation at Startup**
-
-**What worked well:**
-
-- Validating all API keys at agent initialization (not at first use) provides immediate feedback
-- Clear warnings listing missing keys help users diagnose setup issues
-- Non-blocking warnings (continue anyway) allow testing with partial configuration
-
-**Implementation:**
-
-```python
-def validate_environment() -> List[str]:
-    """Check API keys at startup, return list of missing keys."""
-    missing = []
-    for key_name in ["GOOGLE_API_KEY", "HF_TOKEN", "ANTHROPIC_API_KEY", "TAVILY_API_KEY"]:
-        if not os.getenv(key_name):
-            missing.append(key_name)
-
-    if missing:
-        logger.warning(f"⚠️ Missing API keys: {', '.join(missing)}")
-    else:
-        logger.info("✓ All API keys configured")
-
-    return missing
-```
-
-### **What to avoid:**
-
-**Anti-pattern: List-based parameter schemas**
-
-```python
-# WRONG - breaks LLM function calling
-"parameters": ["query", "max_results"]
-
-# CORRECT - works with all providers
-"parameters": {
-    "query": {"type": "string", "description": "..."},
-    "max_results": {"type": "integer", "description": "..."}
-}
-```
-
-**Why it breaks:** LLM clients iterate over `parameters.items()` to extract type/description metadata. List has no `.items()` method.
-
-### **Critical Issues Discovered for Stage 5:**
-
-**P0 - Critical: LLM Quota Exhaustion (15/20 failed - 75%)**
-
-- Gemini: 429 quota exceeded (daily limit)
-- HuggingFace: 402 payment required (novita free limit)
-- Claude: 400 credit balance too low
-- **Impact:** 75% of failures not due to logic, but infrastructure
-
-**P1 - High: Vision Tool Failures (3/20 failed)**
-
-- All image/video questions auto-fail
-- "Vision analysis failed - Gemini and Claude both failed"
-- Vision depends on quota-limited multimodal LLMs
-
-**P1 - High: Tool Selection Errors (2/20 failed)**
-
-- Fallback to keyword matching in some cases
-- Calculator tool validation too strict (empty expression errors)
-
----
-
-## Changelog
-
-**Session Date:** 2026-01-02 to 2026-01-03
-
-### Stage 4 Tasks Completed (10/10)
-
-1. ✅ **Comprehensive Debug Logging** - All nodes log inputs, LLM details, tool execution, state transitions
-2. ✅ **Improved Error Messages** - answer_node shows specific failure reasons and suggestions
-3. ✅ **API Key Validation** - Agent startup checks GOOGLE_API_KEY, HF_TOKEN, ANTHROPIC_API_KEY, TAVILY_API_KEY
-4. ✅ **Tool Execution Error Handling** - execute_node validates tool_calls, handles exceptions gracefully
-5. ✅ **Fallback Tool Execution** - Keyword matching when LLM function calling fails
-6. ✅ **LLM Exception Handling** - 3-tier fallback with comprehensive error capture
-7. ✅ **Diagnostics Display** - Test & Debug tab shows API status, plan, tools, evidence, errors, answer
-8. ✅ **Documentation** - Dev log created (this file + dev_260103_16_huggingface_integration.md)
-9. ✅ **Tool Name Consistency Fix** - Fixed web_search, calculator, vision tool naming (commit d94eeec)
-10. ✅ **Deploy to HF Space and Run GAIA Validation** - 10% score achieved (2/20 correct)
-
-### Modified Files
-
-1. **src/agent/graph.py**
-   - Added `validate_environment()` function
-   - Updated `plan_node` with comprehensive logging
-   - Updated `execute_node` with fallback tool selection
-   - Updated `answer_node` with actionable error messages
-
-2. **src/agent/llm_client.py**
-   - Improved exception handling across all LLM functions
-   - Added HuggingFace integration (see dev_260103_16)
-   - Updated unified functions for 3-tier fallback
-
-3. **app.py**
-   - Added `format_diagnostics()` function
-   - Updated Test & Debug tab UI
-   - Added `check_api_keys()` display
-   - Added export functionality
-
-4. **src/tools/__init__.py**
-   - Fixed TOOLS schema bug (list → dict)
-   - Updated all tool parameter definitions
-
-### Test Results
-
-All tests passing with new fallback architecture:
-
-```bash
-uv run pytest test/ -q
-======================== 99 passed, 11 warnings in 51.99s ========================
-```
-
-### Deployment Results
-
-**HuggingFace Space:** Deployed and operational
-**GAIA Validation:** 10.0% (2/20 correct)
-**Status:** MVP achieved - APIs connected, evidence collection working
-
----
-
-## Stage 4 Complete ✅
-
-**Final Status:** MVP validated with 10% GAIA score
-
-**What Worked:**
-
-- ✅ Real API integration operational (Gemini, HuggingFace, Claude, Tavily)
-- ✅ Evidence collection working (not empty anymore)
-- ✅ Diagnostic visibility enables debugging
-- ✅ Fallback chains provide resilience
-- ✅ Agent functional and deployed to production
-
-**Critical Issues for Stage 5:**
-
-1. **LLM Quota Management** (P0) - 75% of failures due to quota exhaustion
-2. **Vision Tool Failures** (P1) - All image questions auto-fail
-3. **Tool Selection Accuracy** (P1) - Keyword fallback too simplistic
-
-**Ready for Stage 5:** Performance Optimization
-
-- **Target:** 10% → 25% accuracy (5/20 questions)
-- **Priority:** Fix quota management, improve tool selection, fix vision tool
-- **Infrastructure:** Debugging tools ready, JSON export system in place

dev/dev_260103_16_huggingface_llm_integration.md

@@ -1,358 +0,0 @@
-# [dev_260103_16] HuggingFace LLM API Integration
-
-**Date:** 2026-01-03
-**Type:** Development
-**Status:** Resolved
-**Related Dev:** dev_260102_15_stage4_mvp_real_integration.md
-
-## Problem Description
-
-**Context:** Stage 4 implementation was 7/10 complete with comprehensive diagnostics and error handling. However, testing revealed critical LLM availability issues:
-
-1. **Gemini 2.0 Flash** - Quota exceeded (1,500 requests/day free tier limit exhausted from testing)
-2. **Claude Sonnet 4.5** - Credit balance too low (paid tier, user's balance depleted)
-
-**Root Cause:** Agent relied on only 2 LLM tiers (free Gemini → paid Claude), with no middle fallback when free tier exhausted. This caused complete LLM failure, falling back to keyword-based tool selection (Stage 4 fallback mechanism).
-
-**User Request:** Add completely free LLM alternative that works in HuggingFace Spaces environment without requiring local GPU resources.
-
-**Requirements:**
-
-- Must be completely free (no credits, reasonable rate limits)
-- Must support function calling (critical for tool selection)
-- Must work in HuggingFace Spaces (cloud-based, no local GPU)
-- Must integrate into existing 3-tier fallback architecture
-
----
-
-## Key Decisions
-
-### **Decision 1: HuggingFace LLM API over Ollama (local LLMs)**
-
-**Why chosen:**
-
-- ✅ Works in HuggingFace Spaces (cloud-based API)
-- ✅ Free tier with rate limits (~60 req/min vs Gemini's 1,500 req/day)
-- ✅ Function calling support via OpenAI-compatible API
-- ✅ No GPU requirements (serverless inference)
-- ✅ Already deployed to HF Spaces - logical integration
-
-**Rejected alternative: Ollama + Llama 3.1 70B (local)**
-
-- ❌ Requires local GPU or high-end CPU
-- ❌ Won't work in HuggingFace Free Spaces (CPU-only, 16GB RAM limit)
-- ❌ Would need GPU Spaces upgrade (not free)
-- ❌ Complex setup for user's deployment environment
-
-### **Decision 2: Qwen 2.5 72B Instruct as HuggingFace Model**
-
-**Why chosen:**
-
-- ✅ Excellent function calling capabilities (OpenAI-compatible tools format)
-- ✅ Strong reasoning performance (competitive with GPT-4 on benchmarks)
-- ✅ Free on HuggingFace LLM API
-- ✅ 72B parameters - sufficient intelligence for GAIA tasks
-
-**Considered alternatives:**
-
-- `meta-llama/Llama-3.1-70B-Instruct` - Good but slightly worse function calling
-- `NousResearch/Hermes-3-Llama-3.1-70B` - Excellent but less tested for tool use
-
-### **Decision 3: 3-Tier Fallback Architecture**
-
-**Final chain:**
-
-1. **Gemini 2.0 Flash** (free, 1,500 req/day) - Primary
-2. **HuggingFace Qwen 2.5 72B** (free, rate limited) - NEW Middle Tier
-3. **Claude Sonnet 4.5** (paid) - Expensive fallback
-4. **Keyword matching** (deterministic) - Last resort
-
-**Trade-offs:**
-
-- **Pro:** 4 layers of resilience ensure agent always produces output
-- **Pro:** Maximizes free tier usage before burning paid credits
-- **Con:** Slightly higher latency on fallback chain traversal
-- **Con:** More API keys to manage (but HF_TOKEN already required for Space)
-
-### **Decision 4: TOOLS Schema Bug Fix (Critical)**
-
-**Problem discovered:** `src/tools/__init__.py` had parameters as list `["query"]` but LLM client expected dict `{"query": {...}}` with type/description.
-
-**Impact:** Gemini function calling was completely broken - caused `'list' object has no attribute 'items'` error.
-
-**Fix:** Updated all tool definitions to proper schema:
-
-```python
-"parameters": {
-    "query": {
-        "description": "Search query string",
-        "type": "string"
-    },
-    "max_results": {
-        "description": "Maximum number of search results to return",
-        "type": "integer"
-    }
-},
-"required_params": ["query"]
-```
-
-**Result:** Gemini function calling now working correctly (verified in tests).
-
----
-
-## Outcome
-
-Successfully integrated HuggingFace LLM API as free LLM fallback tier, completing Stage 4 MVP with robust multi-tier resilience.
-
-**Deliverables:**
-
-1. **src/agent/llm_client.py** - Added ~150 lines of HuggingFace integration
-   - `create_hf_client()` - Initialize InferenceClient with HF_TOKEN
-   - `plan_question_hf()` - Planning using Qwen 2.5 72B
-   - `select_tools_hf()` - Function calling with OpenAI-compatible tools format
-   - `synthesize_answer_hf()` - Answer synthesis from evidence
-   - Updated unified functions: `plan_question()`, `select_tools_with_function_calling()`, `synthesize_answer()` to use 3-tier fallback
-
-2. **src/agent/graph.py** - Added HF_TOKEN validation
-   - Updated `validate_environment()` to check HF_TOKEN at agent startup
-   - Shows ⚠️ WARNING if HF_TOKEN missing
-
-3. **app.py** - Updated UI and added JSON export functionality
-   - Added HF_TOKEN to `check_api_keys()` display in Test & Debug tab
-   - Added `export_results_to_json()` - Exports evaluation results as clean JSON
-     - Local: ~/Downloads/gaia_results_TIMESTAMP.json
-     - HF Spaces: ./exports/gaia_results_TIMESTAMP.json (environment-aware)
-     - Full error messages preserved (no truncation), easy code processing
-   - Updated `run_and_submit_all()` - ALL return paths now export results
-   - Added gr.File download button - Direct download instead of text display
-
-4. **src/tools/__init__.py** - Fixed TOOLS schema bug (earlier in session)
-   - Changed parameters from list to dict format
-   - Added type/description for each parameter
-   - Fixed Gemini function calling compatibility
-
-**Test Results:**
-
-```bash
-uv run pytest test/ -q
-99 passed, 11 warnings in 51.99s  ✅
-```
-
-All tests passing with new 3-tier fallback architecture.
-
-**Stage 4 Progress: 10/10 tasks completed** ✅
-
-- ✅ Comprehensive debug logging
-- ✅ Improved error messages
-- ✅ API key validation (including HF_TOKEN)
-- ✅ Tool execution error handling
-- ✅ Fallback tool execution (keyword matching)
-- ✅ LLM exception handling (3-tier fallback)
-- ✅ Diagnostics display in Gradio UI
-- ✅ Documentation in dev log (this file)
-- ✅ Tool name consistency fix (web_search, calculator, vision)
-- ✅ Deploy to HF Space and run GAIA validation
-
-**GAIA Validation Results (Real Test):**
-
-- **Score:** 10.0% (2/20 correct)
-- **Improvement:** 0/20 → 2/20 (MVP validated!)
-- **Status:** Agent is functional and operational
-
-**What worked:**
-
-- ✅ Question 1: "How many studio albums were published by Mercedes Sosa between 2000 and 2009?" → Answer: "3" (CORRECT)
-- ✅ HuggingFace LLM (Qwen 2.5 72B) successfully used for planning and tool selection
-- ✅ Web search tool executed successfully
-- ✅ Evidence collection and answer synthesis working
-
-**What failed:**
-
-- ❌ Question 2: YouTube video analysis (vision tool) - "Tool vision failed: Exception: Vision analysis failed - Gemini and Claude both failed"
-- Issue: Vision tool requires multimodal LLM access (quota-limited or needs configuration)
-
-**Next Stage:** Stage 5 - Performance Optimization (target: 5/20 questions)
-
----
-
-## Learnings and Insights
-
-### **Pattern: Free-First Fallback Architecture**
-
-**What worked well:**
-
-- Prioritizing free tiers (Gemini → HuggingFace) before paid tier (Claude) maximizes cost efficiency
-- Multiple free alternatives with different quota models (daily vs rate-limited) provide better resilience than single free tier
-- Keyword fallback ensures agent never completely fails even when all LLMs unavailable
-
-**Reusable pattern:**
-
-```python
-def unified_llm_function(...):
-    """3-tier fallback with comprehensive error capture"""
-    errors = []
-
-    try:
-        return free_tier_1(...)  # Gemini - daily quota
-    except Exception as e1:
-        errors.append(f"Tier 1: {e1}")
-        try:
-            return free_tier_2(...)  # HuggingFace - rate limited
-        except Exception as e2:
-            errors.append(f"Tier 2: {e2}")
-            try:
-                return paid_tier(...)  # Claude - credits
-            except Exception as e3:
-                errors.append(f"Tier 3: {e3}")
-                # Deterministic fallback as last resort
-                return keyword_fallback(...)
-```
-
-### **Pattern: Function Calling Schema Compatibility**
-
-**Critical insight:** Different LLM providers require different function calling schemas:
-
-1. **Gemini** - `genai.protos.Tool` with `function_declarations`:
-
-   ```python
-   Tool(function_declarations=[
-       FunctionDeclaration(
-           name="search_web",
-           description="...",
-           parameters={
-               "type": "object",
-               "properties": {"query": {"type": "string", "description": "..."}},
-               "required": ["query"]
-           }
-       )
-   ])
-   ```
-
-2. **HuggingFace** - OpenAI-compatible tools array:
-
-   ```python
-   tools = [{
-       "type": "function",
-       "function": {
-           "name": "search_web",
-           "description": "...",
-           "parameters": {
-               "type": "object",
-               "properties": {"query": {"type": "string", "description": "..."}},
-               "required": ["query"]
-           }
-       }
-   }]
-   ```
-
-3. **Claude** - Anthropic native format (simplified):
-
-   ```python
-   tools = [{
-       "name": "search_web",
-       "description": "...",
-       "input_schema": {
-           "type": "object",
-           "properties": {"query": {"type": "string", "description": "..."}},
-           "required": ["query"]
-       }
-   }]
-   ```
-
-**Best practice:** Maintain single source of truth in `src/tools/__init__.py` with rich schema (dict format with type/description), then transform to provider-specific format in LLM client functions.
-
-### **Pattern: Environment Validation at Startup**
-
-**What worked well:**
-
-- Validating all API keys at agent initialization (not at first use) provides immediate feedback
-- Clear warnings listing missing keys help users diagnose setup issues
-- Non-blocking warnings (continue anyway) allow testing with partial configuration
-
-**Implementation:**
-
-```python
-def validate_environment() -> List[str]:
-    """Check API keys at startup, return list of missing keys"""
-    missing = []
-    for key_name in ["GOOGLE_API_KEY", "HF_TOKEN", "ANTHROPIC_API_KEY", "TAVILY_API_KEY"]:
-        if not os.getenv(key_name):
-            missing.append(key_name)
-
-    if missing:
-        logger.warning(f"⚠️  Missing API keys: {', '.join(missing)}")
-    else:
-        logger.info("✓ All API keys configured")
-
-    return missing
-```
-
-### **What to avoid:**
-
-**Anti-pattern: List-based parameter schemas**
-
-```python
-# WRONG - breaks LLM function calling
-"parameters": ["query", "max_results"]
-
-# CORRECT - works with all providers
-"parameters": {
-    "query": {"type": "string", "description": "..."},
-    "max_results": {"type": "integer", "description": "..."}
-}
-```
-
-**Why it breaks:** LLM clients iterate over `parameters.items()` to extract type/description metadata. List has no `.items()` method.
-
----
-
-## Changelog
-
-**Session Date:** 2026-01-03
-
-### Modified Files
-
-1. **src/agent/llm_client.py** (~150 lines added)
-   - Added `create_hf_client()` - Initialize HuggingFace InferenceClient with HF_TOKEN
-   - Added `plan_question_hf(question, available_tools, file_paths)` - Planning with Qwen 2.5 72B
-   - Added `select_tools_hf(question, plan, available_tools)` - Function calling with OpenAI-compatible tools format
-   - Added `synthesize_answer_hf(question, evidence)` - Answer synthesis from evidence
-   - Updated `plan_question()` - Added HuggingFace as middle fallback tier (Gemini → HF → Claude)
-   - Updated `select_tools_with_function_calling()` - Added HuggingFace as middle fallback tier
-   - Updated `synthesize_answer()` - Added HuggingFace as middle fallback tier
-   - Added CONFIG constant: `HF_MODEL = "Qwen/Qwen2.5-72B-Instruct"`
-   - Added import: `from huggingface_hub import InferenceClient`
-
-2. **src/agent/graph.py**
-   - Updated `validate_environment()` - Added HF_TOKEN to API key validation check
-   - Updated startup logging - Shows ⚠️ WARNING if HF_TOKEN missing
-
-3. **app.py**
-   - Updated `check_api_keys()` - Added HF_TOKEN status display in Test & Debug tab
-   - UI now shows: "HF_TOKEN (HuggingFace): ✓ SET" or "✗ MISSING"
-   - Added `export_results_to_json(results_log, submission_status)` - Export evaluation results as JSON
-     - Local: ~/Downloads/gaia_results_TIMESTAMP.json
-     - HF Spaces: ./exports/gaia_results_TIMESTAMP.json
-     - Pretty formatted (indent=2), full error messages, easy code processing
-   - Updated `run_and_submit_all()` - ALL return paths now export results
-   - Added gr.File download button - Direct download of JSON file
-   - Updated run_button click handler - Outputs 3 values (status, table, export_path)
-
-4. **src/tools/__init__.py** (Fixed earlier in session)
-   - Fixed TOOLS schema bug - Changed parameters from list to dict format
-   - Updated all tool definitions to include type/description for each parameter
-   - Added `"required_params"` field to specify required parameters
-   - Fixed Gemini function calling compatibility
-
-### Dependencies
-
-**No changes to requirements.txt** - `huggingface-hub>=0.26.0` already present from initial setup.
-
-### Test Results
-
-All tests passing with new 3-tier fallback architecture:
-
-```bash
-uv run pytest test/ -q
-======================== 99 passed, 11 warnings in 51.99s ========================
-```

dev/dev_260104_01_ui_control_question_limit.md

@@ -1,44 +0,0 @@
-# [dev_260104_01] UI Control for Question Limit
-
-**Date:** 2026-01-04
-**Type:** Feature
-**Status:** Resolved
-**Stage:** [Stage 6: Async Processing & Ground Truth Integration]
-
-## Problem Description
-
-DEBUG_QUESTION_LIMIT in .env requires file editing to change. In HF Spaces cloud, users can't easily modify .env for testing different question counts.
-
----
-
-## Key Decisions
-
-- **UI over config files:** Add number input directly in Gradio interface
-- **Zero = all:** Default 0 means process all questions
-- **Priority override:** UI value takes precedence over .env value
-- **Production safe:** Default behavior unchanged (process all)
-
----
-
-## Outcome
-
-Users can now change question limit directly in HF Spaces UI without file editing or rebuild.
-
-**Deliverables:**
-- `app.py` - Added eval_question_limit number input in Full Evaluation tab
-
-## Changelog
-
-**What was changed:**
-- **app.py** (~15 lines modified)
-  - Added `eval_question_limit` number input in Full Evaluation tab (lines 608-615)
-    - Range: 0-165 (0 = process all)
-    - Default: 0 (process all)
-    - Info: "Limit questions for testing (0 = process all)"
-  - Updated `run_and_submit_all()` function signature (line 285)
-    - Added `question_limit: int = 0` parameter
-    - Added docstring documenting parameter
-  - Updated `run_button.click()` to pass UI value (line 629)
-  - Updated question limiting logic (lines 345-351)
-    - Priority: UI value > .env value
-    - Falls back to .env if UI value is 0

dev/dev_260104_04_gaia_evaluation_limitation_correctness.md

@@ -1,65 +0,0 @@
-# [dev_260104_04] GAIA Evaluation Limitation - Per-Question Correctness Unavailable
-
-**Date:** 2026-01-04
-**Type:** Issue
-**Status:** Resolved
-**Stage:** [Stage 6: Async Processing & Ground Truth Integration]
-
-## Problem Description
-
-User reported "Correct?" column showing "null" in JSON export and missing from UI table. Investigation revealed GAIA evaluation submission doesn't provide per-question correctness data.
-
-**Root Cause:** GAIA evaluation API response structure only includes summary stats:
-
-```json
-{
-  "username": "...",
-  "score": 5.0,
-  "correct_count": 1,
-  "total_attempted": 3,
-  "message": "...",
-  "timestamp": "..."
-}
-```
-
-No "results" array exists with per-question correctness. Evaluation API tells us "1/3 correct" but NOT which specific questions are correct.
-
----
-
-## Key Decisions
-
-- **Accept evaluation limitation:** Can't get per-question correctness from submission endpoint
-- **Clean removal:** Remove useless extraction logic entirely
-- **Document clearly:** Add comments explaining evaluation API limitation
-- **Summary only:** Show score stats in submission status message
-- **Local solution:** Use local validation dataset for per-question correctness (separate feature)
-
----
-
-## Outcome
-
-Code cleaned up, evaluation limitation documented clearly. Per-question correctness handled by local validation dataset feature.
-
-**Deliverables:**
-- `.env` - Added DEBUG_QUESTION_LIMIT for faster testing
-- `app.py` - Removed useless extraction logic, documented evaluation API limitation
-
-## Changelog
-
-**What was changed:**
-- **.env** (~2 lines added)
-  - Added `DEBUG_QUESTION_LIMIT=3` - Limit questions for faster evaluation API response debugging (0 = process all)
-
-- **app.py** (~40 lines modified)
-  - Removed useless `correct_task_ids` extraction logic (lines 452-457 deleted)
-  - Removed useless "Correct?" column addition logic (lines 460-465 deleted)
-  - Added clear comment documenting evaluation API limitation (lines 444-447)
-  - Updated `export_results_to_json()` - Removed extraction logic (lines 78-84 deleted)
-  - Simplified JSON export - Hardcoded `"correct": None` with explanatory comment (lines 106-107)
-  - Added `DEBUG_QUESTION_LIMIT` support for faster testing (lines 320-324)
-
-**Solution:**
-- UI table: No "Correct?" column (cleanly omitted, not showing useless data)
-- JSON export: `"correct": null` for all questions (evaluation API doesn't provide this data)
-- Metadata: Includes summary stats (`score_percent`, `correct_count`, `total_attempted`)
-- User sees score summary in submission status message: "5.0% (1/3 correct)"

dev/dev_260104_05_evaluation_metadata_tracking.md

@@ -1,51 +0,0 @@
-# [dev_260104_05] Evaluation Metadata Tracking - Execution Time and Correct Answers
-
-**Date:** 2026-01-04
-**Type:** Feature
-**Status:** Resolved
-**Stage:** [Stage 6: Async Processing & Ground Truth Integration]
-
-## Problem Description
-
-No execution time tracking to verify async performance improvement. JSON export doesn't show which questions were answered correctly, making error analysis difficult.
-
----
-
-## Key Decisions
-
-- **Time tracking:** Add execution_time parameter to export function, track in run_and_submit_all()
-- **API response parsing:** Extract correct task IDs from submission response if available
-- **Visual indicators:** Use ✅/❌ in UI table for clear correctness display
-- **Metadata enrichment:** Add execution_time_formatted, score_percent, correct_count to JSON export
-
----
-
-## Outcome
-
-Performance now trackable (expect 60-80s vs previous 240s for async). Error analysis easier with correct answer identification.
-
-**Deliverables:**
-- `app.py` - Added execution time tracking, correct answer display, metadata enrichment
-
-## Changelog
-
-**What was changed:**
-- **app.py** (~60 lines added/modified)
-  - Added `import time` (line 8) - For execution timing
-  - Updated `export_results_to_json()` function signature (lines 38-113)
-    - Added `execution_time` parameter (optional float)
-    - Added `submission_response` parameter (optional dict with GAIA API response)
-    - Extracts correct task_ids from `submission_response["results"]` if available
-    - Adds execution time to metadata: `execution_time_seconds` and `execution_time_formatted` (Xm Ys)
-    - Adds score info to metadata: `score_percent`, `correct_count`, `total_attempted`
-    - Adds `"correct": true/false/null` flag to each result entry
-  - Updated `run_and_submit_all()` timing tracking (lines 274-435)
-    - Added `start_time = time.time()` at function start (line 275)
-    - Added `execution_time = time.time() - start_time` before all returns
-    - Logs execution time: "Total execution time: X.XX seconds (Xm Ys)" (line 397)
-    - Updated all 6 `export_results_to_json()` calls to pass `execution_time`
-    - Successful submission: passes both `execution_time` and `result_data` (line 417)
-  - Added correct answer column to results display (lines 399-413)
-    - Extracts correct task_ids from `result_data["results"]` if available
-    - Adds "Correct?" column to `results_log` with "✅ Yes" or "❌ No"
-    - Falls back to summary message if per-question data unavailable

dev/dev_260104_08_ui_selection_runtime_config.md

@@ -1,43 +0,0 @@
-# [dev_260104_08] UI Selection Not Applied - Runtime Config Reading
-
-**Date:** 2026-01-04
-**Type:** Bugfix
-**Status:** Resolved
-**Stage:** [Stage 5: Performance Optimization]
-
-## Problem Description
-
-UI dropdown selections weren't being applied. Selected "HuggingFace" but system still used "Gemini". Root cause: LLM_PROVIDER and ENABLE_LLM_FALLBACK were read at module import time, before UI could set environment variables.
-
----
-
-## Key Decisions
-
-- **Runtime reading:** Read config on every function call, not at module import
-- **Remove constants:** Delete module-level LLM_PROVIDER and ENABLE_LLM_FALLBACK constants
-- **Use os.getenv directly:** Call `os.getenv("LLM_PROVIDER", "gemini")` in _call_with_fallback()
-- **Immediate effect:** Changes take effect without module reload
-
----
-
-## Outcome
-
-UI selections now work correctly. Config read at runtime when function is called.
-
-**Deliverables:**
-- `src/agent/llm_client.py` - Removed module-level constants, updated to read config at runtime
-
-## Changelog
-
-**What was changed:**
-- **src/agent/llm_client.py** (~5 lines modified)
-  - Removed module-level constants `LLM_PROVIDER` and `ENABLE_LLM_FALLBACK` (line 48-50)
-  - Updated `_call_with_fallback()` to read config at runtime (lines 173-175)
-    - Now calls `os.getenv("LLM_PROVIDER", "gemini")` on every function call
-    - Now calls `os.getenv("ENABLE_LLM_FALLBACK", "false")` on every function call
-  - Changed variable references from constants to local variables
-
-**Solution:**
-- Config now read at runtime when function is called, not at module import
-- UI can set environment variables before function execution
-- Changes take effect immediately without module reload

dev/dev_260104_09_ui_based_llm_selection.md

@@ -1,47 +0,0 @@
-# [dev_260104_09] Cloud Testing UX - UI-Based LLM Selection
-
-**Date:** 2026-01-04
-**Type:** Feature
-**Status:** Resolved
-**Stage:** [Stage 5: Performance Optimization]
-
-## Problem Description
-
-Testing different LLM providers in HF Spaces cloud requires manually changing environment variables in Space settings, then waiting for rebuild. Slow iteration, poor UX.
-
----
-
-## Key Decisions
-
-- **UI dropdowns:** Add provider selection in both Test & Debug and Full Evaluation tabs
-- **Environment override:** Set os.environ directly from UI selection (overrides .env and HF Space env vars)
-- **Toggle fallback:** Checkbox to enable/disable fallback behavior
-- **Default strategy:** Groq for testing, fallback enabled for production
-
----
-
-## Outcome
-
-Cloud testing now much faster - test all 4 providers directly from HF Space UI without rebuild.
-
-**Deliverables:**
-- `app.py` - Added UI dropdowns and checkboxes for LLM provider selection in both tabs
-
-## Changelog
-
-**What was changed:**
-- **app.py** (~30 lines added/modified)
-  - Updated `test_single_question()` function signature - Added `llm_provider` and `enable_fallback` parameters
-    - Sets `os.environ["LLM_PROVIDER"]` from UI selection (overrides .env and HF Space env vars)
-    - Sets `os.environ["ENABLE_LLM_FALLBACK"]` from UI checkbox
-    - Adds provider info to diagnostics output
-  - Updated `run_and_submit_all()` function signature - Added `llm_provider` and `enable_fallback` parameters
-    - Reordered params: UI inputs first, profile last (optional)
-    - Sets environment variables before agent initialization
-  - Added UI components in "Test & Debug" tab:
-    - `llm_provider_dropdown` - Select from: Gemini, HuggingFace, Groq, Claude (default: Groq)
-    - `enable_fallback_checkbox` - Toggle fallback behavior (default: false for testing)
-  - Added UI components in "Full Evaluation" tab:
-    - `eval_llm_provider_dropdown` - Select LLM for all questions (default: Groq)
-    - `eval_enable_fallback_checkbox` - Toggle fallback (default: true for production)
-  - Updated button click handlers to pass new UI inputs to functions

dev/dev_260104_10_config_based_llm_selection.md

@@ -1,51 +0,0 @@
-# [dev_260104_10] LLM Provider Debugging - Config-Based Selection
-
-**Date:** 2026-01-04
-**Type:** Feature
-**Status:** Resolved
-**Stage:** [Stage 5: Performance Optimization]
-
-## Problem Description
-
-Hard to debug which LLM provider is handling each step with 4-tier fallback chain. Cannot isolate provider performance for improvement.
-
----
-
-## Key Decisions
-
-- **Env config:** Add LLM_PROVIDER and ENABLE_LLM_FALLBACK to .env
-- **Routing function:** Create _call_with_fallback() to centralize provider selection logic
-- **Provider mapping:** _get_provider_function() maps function names to implementations
-- **Clear logging:** Info logs show exactly which provider is used
-- **Fallback control:** ENABLE_LLM_FALLBACK=false for isolated testing
-
----
-
-## Outcome
-
-Easy debugging: change LLM_PROVIDER in .env or UI to test specific provider. Clear logs show which LLM handled each step.
-
-**Deliverables:**
-- `.env` - Added LLM_PROVIDER and ENABLE_LLM_FALLBACK config
-- `src/agent/llm_client.py` - Added config-based selection with routing function
-
-## Changelog
-
-**What was changed:**
-- **.env** (~5 lines added)
-  - Added `LLM_PROVIDER=gemini` - Select single provider: "gemini", "huggingface", "groq", or "claude"
-  - Added `ENABLE_LLM_FALLBACK=false` - Toggle fallback behavior (true/false)
-  - Removed deprecated `DEFAULT_LLM_MODEL` config
-
-- **src/agent/llm_client.py** (~150 lines added/modified)
-  - Added `LLM_PROVIDER` config variable (line 49) - Reads from environment
-  - Added `ENABLE_LLM_FALLBACK` config variable (line 50) - Reads from environment
-  - Added `_get_provider_function()` helper (lines 114-158) - Maps function names to provider implementations
-  - Added `_call_with_fallback()` routing function (lines 161-212)
-    - Primary provider: Uses LLM_PROVIDER config
-    - Fallback behavior: Controlled by ENABLE_LLM_FALLBACK
-    - Logging: Clear info logs showing which provider is used
-    - Error handling: Specific error messages when fallback disabled
-  - Updated `plan_question()` - Now uses `_call_with_fallback()` (simplified from 40 lines to 1 line)
-  - Updated `select_tools_with_function_calling()` - Now uses `_call_with_fallback()` (simplified from 40 lines to 1 line)
-  - Updated `synthesize_answer()` - Now uses `_call_with_fallback()` (simplified from 40 lines to 1 line)

dev/dev_260104_11_calculator_test_updates.md

@@ -1,40 +0,0 @@
-# [dev_260104_11] Calculator Tool Crashes - Test Updates
-
-**Date:** 2026-01-04
-**Type:** Feature
-**Status:** Resolved
-**Stage:** [Stage 5: Performance Optimization]
-
-## Problem Description
-
-Calculator validation changed to return error dict instead of raising ValueError. Tests need to match new behavior.
-
----
-
-## Key Decisions
-
-- **Update test expectations:** Check for error dict instead of ValueError exception
-- **Verify structure:** Test that result["success"] == False, error message present, result is None
-- **Maintain coverage:** Ensure all validation scenarios still tested
-
----
-
-## Outcome
-
-All 99 tests passing. Tests now match new calculator behavior (error dict instead of exception).
-
-**Deliverables:**
-- `test/test_calculator.py` - Updated tests to check error dict instead of ValueError
-
-## Changelog
-
-**What was changed:**
-- **test/test_calculator.py** (~15 lines modified)
-  - Updated `test_empty_expression()` - Changed from expecting ValueError to checking error dict
-  - Updated `test_too_long_expression()` - Changed from expecting ValueError to checking error dict
-  - Tests now verify: result["success"] == False, error message present, result is None
-
-**Test Results:**
-- ✅ All 99 tests passing (0 failures)
-- ✅ No regressions introduced by Stage 5 changes
-- ✅ Test suite run time: ~2min 40sec

dev/dev_260104_17_json_export_system.md

@@ -1,240 +0,0 @@
-# [dev_260104_17] JSON Export System for GAIA Results
-
-**Date:** 2026-01-04
-**Type:** Development
-**Status:** Resolved
-**Related Dev:** dev_260103_16_huggingface_llm_integration.md
-
-## Problem Description
-
-**Context:** After Stage 4 completion and GAIA validation run, the markdown table export format had critical issues that prevented effective Stage 5 debugging:
-
-1. **Truncation Issues:** Error messages truncated at 100 characters, losing critical failure details
-2. **Special Character Escaping:** Pipe characters (`|`) and special chars in error logs broke markdown table formatting
-3. **Manual Processing Difficulty:** Markdown format unsuitable for programmatic analysis of 20 question results
-
-**User Feedback:** "you see it need some improvement, since as you see, the Error log getting truncated" and "i dont think the markdown table will handle because there will be special char in log"
-
-**Root Cause:** Markdown tables are presentation-focused, not data-focused. They require escaping and truncation to maintain formatting, which destroys debugging value.
-
----
-
-## Key Decisions
-
-### **Decision 1: JSON Export over Markdown Table**
-
-**Why chosen:**
-
-- ✅ No special character escaping required
-- ✅ Full error messages preserved (no truncation)
-- ✅ Easy programmatic processing for Stage 5 analysis
-- ✅ Clean data structure with metadata
-- ✅ Universal format for both human and machine reading
-
-**Rejected alternative: Fixed markdown table**
-
-- ❌ Still requires escaping pipes, quotes, newlines
-- ❌ Still needs truncation to maintain readable width
-- ❌ Hard to parse programmatically
-- ❌ Not suitable for error logs with technical details
-
-### **Decision 2: Unified Output Folder**
-
-**Why chosen:**
-
-- ✅ All environments: Save to `./output/` (consistent location)
-- ✅ Gradio serves from any folder via `gr.File(type="filepath")`
-- ✅ No environment detection needed
-- ✅ Matches project structure expectations
-
-**Trade-offs:**
-
-- **Pro:** Single code path for local and HF Spaces
-- **Pro:** No confusion about file locations
-- **Pro:** Simpler code, easier maintenance
-
-### **Decision 3: gr.File Download Button over Textbox Display**
-
-**Why chosen:**
-
-- ✅ Better UX - direct download instead of copy-paste
-- ✅ Preserves formatting (JSON indentation, Unicode characters)
-- ✅ Gradio natively handles file serving in HF Spaces
-- ✅ Cleaner UI without large text blocks
-
-**Previous approach:** gr.Textbox with markdown table string
-**New approach:** gr.File with filepath return value
-
----
-
-## Outcome
-
-Successfully implemented production-ready JSON export system for GAIA evaluation results, enabling Stage 5 debugging with full error details.
-
-**Deliverables:**
-
-1. **app.py - `export_results_to_json()` function**
-   - Environment detection: `SPACE_ID` check for HF Spaces vs local
-   - Path logic: `~/Downloads` (local) vs `./exports` (HF Spaces)
-   - JSON structure: metadata + submission_status + results array
-   - Pretty formatting: `indent=2`, `ensure_ascii=False` for readability
-   - Full error preservation: No truncation, no escaping issues
-
-2. **app.py - UI updates**
-   - Changed `export_output` from `gr.Textbox` to `gr.File`
-   - Updated `run_and_submit_all()` to call `export_results_to_json()` in ALL return paths
-   - Updated button click handler to output 3 values: `(status, table, export_path)`
-
-**Test Results:**
-
-- ✅ All tests passing (99/99)
-- ✅ JSON export verified with real GAIA validation results
-- ✅ File: `output/gaia_results_20260104_011001.json` (20 questions, full error details)
-
----
-
-## Learnings and Insights
-
-### **Pattern: Data Format Selection Based on Use Case**
-
-**What worked well:**
-
-- Choosing JSON for machine-readable debugging data over human-readable presentation formats
-- Environment-aware paths avoid deployment issues between local and cloud
-- File download UI pattern better than inline text display for large data
-
-**Reusable pattern:**
-
-```python
-def export_to_appropriate_format(data: dict, use_case: str) -> str:
-    """Choose export format based on use case, not habit."""
-    if use_case == "debugging" or use_case == "programmatic":
-        return export_as_json(data)  # Machine-readable
-    elif use_case == "reporting":
-        return export_as_markdown(data)  # Human-readable
-    elif use_case == "data_analysis":
-        return export_as_csv(data)  # Tabular analysis
-```
-
-### **Pattern: Environment-Aware File Paths**
-
-**Critical insight:** Cloud deployments have different filesystem constraints than local development.
-
-**Best practice:**
-
-```python
-def get_export_path(filename: str) -> str:
-    """Return appropriate export path based on environment."""
-    if os.getenv("SPACE_ID"):  # HuggingFace Spaces
-        export_dir = os.path.join(os.getcwd(), "exports")
-        os.makedirs(export_dir, exist_ok=True)
-        return os.path.join(export_dir, filename)
-    else:  # Local development
-        downloads_dir = os.path.expanduser("~/Downloads")
-        return os.path.join(downloads_dir, filename)
-```
-
-### **What to avoid:**
-
-**Anti-pattern: Using presentation formats for data storage**
-
-```python
-# WRONG - Markdown tables for error logs
-results_md = "| Task ID | Question | Error |\n"
-results_md += f"| {id} | {q[:50]} | {err[:100]} |"  # Truncation loses data
-
-# CORRECT - JSON for structured data with full details
-results_json = {
-    "task_id": id,
-    "question": q,  # Full text, no truncation
-    "error": err    # Full error message, no escaping
-}
-```
-
-**Why it breaks:** Presentation formats prioritize visual formatting over data integrity. Truncation and escaping destroy debugging value.
-
----
-
-## Changelog
-
-**Session Date:** 2026-01-04
-
-### Modified Files
-
-1. **app.py** (~50 lines added/modified)
-   - Added `export_results_to_json(results_log, submission_status)` function
-     - Environment detection via `SPACE_ID` check
-     - Local: `~/Downloads/gaia_results_TIMESTAMP.json`
-     - HF Spaces: `./exports/gaia_results_TIMESTAMP.json`
-     - JSON structure: metadata, submission_status, results array
-     - Pretty formatting: indent=2, ensure_ascii=False
-   - Updated `run_and_submit_all()` - Added `export_results_to_json()` call in ALL return paths (7 locations)
-   - Changed `export_output` from `gr.Textbox` to `gr.File` in Gradio UI
-   - Updated `run_button.click()` handler - Now outputs 3 values: (status, table, export_path)
-   - Added `check_api_keys()` update - Shows EXA_API_KEY status (discovered during session)
-
-### Created Files
-
-- **output/gaia_results_20260104_011001.json** - Real GAIA validation results export
-  - 20 questions with full error details
-  - Metadata: generated timestamp, total_questions count
-  - No truncation, no special char issues
-  - Ready for Stage 5 analysis
-
-### Dependencies
-
-**No changes to requirements.txt** - All JSON functionality uses Python standard library.
-
-### Implementation Details
-
-**JSON Export Function:**
-
-```python
-def export_results_to_json(results_log: list, submission_status: str) -> str:
-    """Export evaluation results to JSON file for easy processing.
-
-    - Local: Saves to ~/Downloads/gaia_results_TIMESTAMP.json
-    - HF Spaces: Saves to ./exports/gaia_results_TIMESTAMP.json
-    - Format: Clean JSON with full error messages, no truncation
-    """
-    from datetime import datetime
-
-    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
-    filename = f"gaia_results_{timestamp}.json"
-
-    # Detect environment: HF Spaces or local
-    if os.getenv("SPACE_ID"):
-        export_dir = os.path.join(os.getcwd(), "exports")
-        os.makedirs(export_dir, exist_ok=True)
-        filepath = os.path.join(export_dir, filename)
-    else:
-        downloads_dir = os.path.expanduser("~/Downloads")
-        filepath = os.path.join(downloads_dir, filename)
-
-    # Build JSON structure
-    export_data = {
-        "metadata": {
-            "generated": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
-            "timestamp": timestamp,
-            "total_questions": len(results_log)
-        },
-        "submission_status": submission_status,
-        "results": [
-            {
-                "task_id": result.get("Task ID", "N/A"),
-                "question": result.get("Question", "N/A"),
-                "submitted_answer": result.get("Submitted Answer", "N/A")
-            }
-            for result in results_log
-        ]
-    }
-
-    # Write JSON file with pretty formatting
-    with open(filepath, 'w', encoding='utf-8') as f:
-        json.dump(export_data, f, indent=2, ensure_ascii=False)
-
-    logger.info(f"Results exported to: {filepath}")
-    return filepath
-```
-
-**Result:** Production-ready export system enabling Stage 5 error analysis with full debugging details.

dev/dev_260104_18_stage5_performance_optimization.md

@@ -1,93 +0,0 @@
-# [dev_260104_18] Stage 5: Performance Optimization
-
-**Date:** 2026-01-04
-**Type:** Development
-**Status:** Resolved
-**Stage:** [Stage 5: Performance Optimization]
-
-## Problem Description
-
-GAIA agent performance at 10% (2/20) accuracy. 75% of failures caused by LLM quota exhaustion across all 3 tiers (Gemini, HuggingFace, Claude). Additional issues: vision tool crashes, poor tool selection accuracy.
-
----
-
-## Key Decisions
-
-- **4-tier LLM fallback:** Gemini → HF → Groq → Claude ensures at least one tier always available
-- **Retry logic:** Exponential backoff (1s, 2s, 4s) handles transient quota errors
-- **Few-shot learning:** Concrete examples in prompts improve tool selection accuracy
-- **Graceful degradation:** Vision questions fail gracefully when quota exhausted
-- **Config-based testing:** Environment variables enable isolated provider testing
-
----
-
-## Outcome
-
-**Test Results:**
-
-- ✅ All 99 tests passing (0 failures)
-- ✅ Target achieved: 25% accuracy (5/20 correct)
-- ✅ No regressions introduced
-- ✅ Test suite run time: ~2min 40sec
-
-**Implementation Summary:**
-
-- ✅ Step 1: Retry logic with exponential backoff
-- ✅ Step 2: Groq integration (Llama 3.1 70B, 30 req/min free tier)
-- ✅ Step 3: Few-shot examples in all tool selection prompts
-- ✅ Step 4: Graceful vision question skip
-- ✅ Step 5: Calculator validation relaxed (error dict instead of exception)
-- ✅ Step 6: Tool descriptions improved with "Use when..." guidance
-
-**Deliverables:**
-
-- `src/agent/llm_client.py` - Retry logic, Groq integration, few-shot prompts, config-based selection
-- `src/agent/graph.py` - Graceful vision skip
-- `src/tools/calculator.py` - Relaxed validation
-- `src/tools/__init__.py` - Improved tool descriptions
-- `test/test_calculator.py` - Updated tests
-- `requirements.txt` - Added groq>=0.4.0
-- `.env` - Added LLM_PROVIDER, ENABLE_LLM_FALLBACK configs
-
-## Changelog
-
-**Step 1: Retry Logic (P0 - Critical)**
-
-- Added `retry_with_backoff()` function - Exponential backoff: 1s, 2s, 4s
-- Detects 429, quota, rate limit errors
-- Max 3 retries per provider
-- Wrapped all LLM calls in plan_question(), select_tools_with_function_calling(), synthesize_answer()
-
-**Step 2: Groq Integration (P0 - Critical)**
-
-- Added `create_groq_client()`, `plan_question_groq()`, `select_tools_groq()`, `synthesize_answer_groq()`
-- New fallback chain: Gemini → HF → **Groq** → Claude (4-tier)
-- Groq model: llama-3.1-70b-versatile
-- Free tier: 30 requests/minute
-
-**Step 3: Few-Shot Examples (P1 - High Impact)**
-
-- Updated all 4 provider prompts: Claude, Gemini, HF, Groq
-- Added examples: web_search, calculator, vision, parse_file
-- Changed tone from "agent" to "expert"
-- Added explicit instruction: "Use exact parameter names from tool schemas"
-
-**Step 4: Graceful Vision Skip (P1 - High Impact)**
-
-- Added `is_vision_question()` helper - Detects: image, video, youtube, photo, picture, watch, screenshot, visual
-- Two checkpoints: tool selection and tool execution
-- Context-aware error: "Vision analysis failed: LLM quota exhausted"
-
-**Step 5: Calculator Validation (P1 - High Impact)**
-
-- Changed from raising ValueError to returning error dict
-- Handles empty, whitespace-only, oversized expressions gracefully
-- All validation errors now non-fatal
-
-**Step 6: Improved Tool Descriptions (P1 - High Impact)**
-
-- web_search: "factual information, current events, Wikipedia, statistics, people, companies"
-- calculator: Lists arithmetic, algebra, trig, logarithms; functions: sqrt, sin, cos, log, abs
-- parse_file: Mentions "the file", "uploaded document", "attachment" triggers
-- vision: "describe content, identify objects, read text"; triggers: images, photos, videos, YouTube
-- All descriptions now have explicit "Use when..." guidance

dev/dev_260104_19_stage6_async_ground_truth.md

@@ -1,84 +0,0 @@
-# [dev_260104_19] Stage 6: Async Processing & Ground Truth Integration
-
-**Date:** 2026-01-04
-**Type:** Development
-**Status:** Resolved
-**Stage:** [Stage 6: Async Processing & Ground Truth Integration]
-
-## Problem Description
-
-Two major issues: (1) Sequential processing takes 4-5 minutes for 20 questions, poor UX. (2) GAIA API doesn't provide per-question correctness, making debugging impossible without local ground truth comparison.
-
----
-
-## Key Decisions
-
-- **Async processing:** ThreadPoolExecutor with configurable workers (default: 5) for 60-70% speedup
-- **Local validation dataset:** Download GAIA validation set from HuggingFace for local correctness checking
-- **Metadata tracking:** Add execution time and correct answer tracking to verify performance improvements
-- **UI controls:** Add question limit input for flexible cloud testing
-- **Single source architecture:** results_log as source of truth for both UI and JSON
-
----
-
-## Outcome
-
-**Performance Improvement:**
-
-- 4-5 min → 1-2 min (60-70% reduction in processing time)
-- Real-time progress logging during execution
-- Individual question errors don't block others
-
-**Debugging Capabilities:**
-
-- Local correctness checking without API dependency
-- See which specific questions are correct/incorrect
-- Execution time metadata for performance tracking
-- Error analysis with ground truth answers and solving steps
-
-**Deliverables:**
-
-- `src/utils/ground_truth.py` (NEW) - GAIAGroundTruth class for validation dataset
-- `src/utils/__init__.py` (NEW) - Package initialization
-- `app.py` - Async processing, ground truth integration, metadata tracking, UI controls
-- `requirements.txt` - Added datasets>=4.4.2, huggingface-hub
-- `.env` - Added MAX_CONCURRENT_WORKERS, DEBUG_QUESTION_LIMIT
-
-## Changelog
-
-**Async Processing:**
-
-- Added `process_single_question()` worker function - Processes single question with error handling
-- Replaced sequential loop with ThreadPoolExecutor
-- Configurable max_workers from environment (default: 5)
-- Progress logging: "[X/Y] Processing task_id..." and "Progress: X/Y questions processed"
-- Balances speed (5× faster) with API rate limits (Tavily: 1000/month, Groq: 30-60 req/min)
-
-**Ground Truth Integration:**
-
-- Created `GAIAGroundTruth` class with singleton pattern
-- `load_validation_set()` - Downloads GAIA validation set (2023_all split)
-- `get_answer(task_id)` - Returns ground truth answer
-- `compare_answer(task_id, submitted_answer)` - Exact match comparison
-- Caches dataset to `~/.cache/gaia_dataset` for fast reload
-- Graceful fallback if dataset unavailable
-
-**Results Collection:**
-
-- Added "Correct?" column with "✅ Yes" or "❌ No" indicators
-- Added "Ground Truth Answer" column showing correct answer
-- Added "Annotator Metadata" column with solving steps
-- All columns display in both UI table and JSON export (same source: results_log)
-
-**Metadata Tracking:**
-
-- Execution time: `execution_time_seconds` and `execution_time_formatted` (Xm Ys)
-- Score info: `score_percent`, `correct_count`, `total_attempted`
-- Per-question `"correct": true/false/null` in JSON export
-- Logging: "Total execution time: X.XX seconds (Xm Ys)"
-
-**UI Controls:**
-
-- Question limit number input (0-165, default 0 = all)
-- Priority: UI value > .env value
-- Enables flexible testing in HF Spaces without file editing

dev/dev_260105_02_remove_annotator_metadata_raw_ui.md

@@ -1,49 +0,0 @@
-# [dev_260105_02] Remove colum "annotator_metadata_raw" from UI Table
-
-**Date:** 2026-01-05
-**Type:** Development
-**Status:** Resolved
-**Stage:** [Stage 6: Async Processing & Ground Truth Integration]
-
-## Problem Description
-
-Internal `annotator_metadata_raw` field showing in UI table as a confusing column.
-
----
-
-## Key Decisions
-
-- **Pass ground_truth to export:** Export function fetches metadata directly from ground_truth object
-- **Remove from results_log:** Internal fields shouldn't appear in UI
-- **Clean UI display:** Table shows only user-facing columns
-
----
-
-## Outcome
-
-UI table cleaned up, JSON export still includes annotator_metadata (fetched from ground_truth object).
-
-**Deliverables:**
-
-- `app.py` - Removed `annotator_metadata_raw` from results_entry, updated export to use ground_truth parameter
-
-## Changelog
-
-**What was changed:**
-
-- **app.py** (~20 lines modified)
-  - Removed `annotator_metadata_raw` from result_entry (line 426 removed)
-  - Removed unused local variables: metadata_item, annotator_metadata (lines 411-412 removed)
-  - Updated `export_results_to_json()` signature (line 52)
-    - Added `ground_truth = None` parameter
-  - Updated JSON export logic (lines 120-126)
-    - Fetch annotator_metadata from ground_truth.metadata during export
-    - No longer relies on result.get("annotator_metadata_raw")
-  - Updated all 6 calls to export_results_to_json (lines 453, 493, 507, 516, 525, 534)
-    - Added ground_truth as final parameter
-
-**Result:**
-
-- UI table: Clean - no internal/hidden fields
-- JSON export: Still includes annotator_metadata (fetched from ground_truth object)
-- Better separation of concerns: UI uses results_log, export uses ground_truth object

dev/dev_260105_03_ground_truth_single_source.md

@@ -1,54 +0,0 @@
-# [dev_260105_03] Ground Truth Single Source Architecture
-
-**Date:** 2026-01-05
-**Type:** Development
-**Status:** Resolved
-**Stage:** [Stage 6: Async Processing & Ground Truth Integration]
-
-## Problem Description
-
-Ground truth data (answers, metadata) needed for both UI table display and JSON export. Previous iterations had complex dual-storage approaches and double access patterns.
-
----
-
-## Key Decisions
-
-- **Single source of truth:** Store all data once in results_log, both formats read from it
-- **Remove ground_truth parameter:** Export function no longer needs ground_truth object
-- **Accept UI limitation:** Dict displays as "[object Object]" in pandas table - acceptable tradeoff
-- **JSON export primary:** Metadata most useful in JSON format for analysis
-
----
-
-## Outcome
-
-Clean single-source architecture: results_log contains all data, export function simplified, no double work.
-
-**Architecture:**
-
-- One object (results_log) → Two formats (UI table + JSON)
-- Both identical, no filtering, no double access
-- Export function uses `result.get("annotator_metadata")` directly from stored data
-
-**Deliverables:**
-
-- `app.py` - Removed ground_truth parameter, simplified data flow, single storage approach
-
-## Changelog
-
-**What was changed:**
-
-- **app.py** (~10 lines modified)
-  - Removed `ground_truth` parameter from `export_results_to_json()` function signature
-  - Removed double work: no longer access `ground_truth.metadata` in export function
-  - Changed `_annotator_metadata` to `annotator_metadata` (removed underscore prefix)
-  - Updated all 6 function calls to remove `ground_truth` parameter
-  - Simplified JSON export: `result.get("annotator_metadata")` from stored data
-  - Updated docstring: "Single source: Both UI and JSON use identical results_log data"
-
-**Current Behavior:**
-
-- results_log contains: `{"annotator_metadata": {...dict...}}`
-- UI table: Shows "[object Object]" for dict values (pandas limitation, acceptable)
-- JSON export: Includes full `annotator_metadata` object
-- Both formats read from same source, no filtering

docs/gaia_submission_guide.md

@@ -1,120 +0,0 @@
-# GAIA Submission Guide
-
-## Two Different Leaderboards
-
-### 1. Course Leaderboard (CURRENT - Course Assignment)
-
-**API Endpoint:** `https://agents-course-unit4-scoring.hf.space`
-
-**Purpose:** Hugging Face Agents Course Unit 4 assignment
-
-**Dataset:** 20 questions from GAIA validation set (level 1), filtered by tools/steps complexity
-
-**Target Score:** 30% = **6/20 correct**
-
-**API Routes:**
-- `GET /questions` - Retrieve full list of evaluation questions
-- `GET /random-question` - Fetch single random question
-- `GET /files/{task_id}` - Download file associated with task
-- `POST /submit` - Submit answers for scoring
-
-**Submission Format:**
-```json
-{
-  "username": "your-hf-username",
-  "agent_code": "https://huggingface.co/spaces/your-username/your-space/tree/main",
-  "answers": [
-    {"task_id": "...", "submitted_answer": "..."}
-  ]
-}
-```
-
-**Scoring:** EXACT MATCH with ground truth
-- Answer should be plain text, NO "FINAL ANSWER:" prefix
-- Answer should be precise and well-formatted
-
-**Debugging Features (Course-Specific):**
-- ✅ "Target Task IDs" - Run specific questions for debugging
-- ✅ "Question Limit" - Run first N questions for testing
-- ✅ Course API is forgiving for development iteration
-
-**Leaderboard:** https://huggingface.co/spaces/gaia-benchmark/gaia-leaderboard
-
----
-
-### 2. Official GAIA Leaderboard (FUTURE - Not Yet Implemented)
-
-**Space:** https://huggingface.co/spaces/gaia-benchmark/leaderboard
-
-**Purpose:** Official GAIA benchmark for AI research community
-
-**Dataset:** Full GAIA benchmark (450+ questions across 3 levels)
-
-**Submission Format:** File upload (JSON) with model metadata
-- Model name, family, parameters
-- Complete answers for ALL questions
-- Different evaluation process
-
-**Status:** ⚠️ **FUTURE DEVELOPMENT** - Not implemented in this template
-
-**Differences from Course:**
-| Aspect | Course | Official GAIA |
-|--------|--------|--------------|
-| Dataset Size | 20 questions | 450+ questions |
-| Submission Method | API POST | File upload |
-| Question Filtering | Allowed for debugging | Must submit ALL |
-| Scoring | Exact match | TBC (likely more flexible) |
-
-**Documentation:** https://huggingface.co/datasets/gaia-benchmark/GAIA
-
----
-
-## Implementation Notes
-
-### Current Implementation Status
-
-**✅ Implemented:**
-- Course API integration (`/questions`, `/submit`, `/files/{task_id}`)
-- Agent execution with LangGraph StateGraph
-- OAuth login integration
-- Debug features (Target Task IDs, Question Limit)
-- Results export (JSON format)
-
-**⚠️ Course Constraints:**
-- Only 20 level 1 questions
-- Exact match scoring (strict)
-- Agent code must be public
-
-**🔮 Future Work (Official GAIA):**
-- File-based submission format
-- Full 450+ question support
-- Leaderboard-specific metadata
-- Official evaluation pipeline
-
----
-
-## Development Workflow
-
-### For Course Assignment:
-
-1. **Develop:** Use "Target Task IDs" to test specific questions
-2. **Debug:** Use "Question Limit" for quick iteration
-3. **Test:** Run full evaluation on all 20 questions
-4. **Submit:** Course API evaluates exact match score
-5. **Iterate:** Improve prompts, tools, reasoning
-
-### For Official GAIA (Future):
-
-1. **Generate:** Create submission JSON with all 450+ answers
-2. **Format:** Follow official GAIA format requirements
-3. **Upload:** Submit via gaia-benchmark/leaderboard Space
-4. **Evaluate:** Official benchmark evaluation
-
----
-
-## References
-
-- **Course Documentation:** https://huggingface.co/learn/agents-course/en/unit4/hands-on
-- **Course Leaderboard:** https://huggingface.co/spaces/gaia-benchmark/gaia-leaderboard
-- **Official GAIA Dataset:** https://huggingface.co/datasets/gaia-benchmark/GAIA
-- **Official GAIA Leaderboard:** https://huggingface.co/spaces/gaia-benchmark/leaderboard

test/test_phase0_hf_vision_api.py

@@ -364,7 +364,7 @@ if __name__ == "__main__":
         from datetime import datetime
 
         timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
-        output_dir = Path("_output")
+        output_dir = Path("user_output")
         output_dir.mkdir(exist_ok=True)
 
         output_file = output_dir / f"phase0_vision_validation_{timestamp}.json"