feat: phase1 planning and video processing research

- Rewrite PLAN.md: focus on system error fixes (current 10% → 30% target)
- Add brainstorming_phase1_youtube.md: YouTube transcript approach research
- Add _template_original/: static reference for comparison
- Update CHANGELOG.md: course test setup analysis, 20 fixed questions documented
- Research findings:
* Transcript-first approach: 1-3s vs frame extraction 1-10min total
* Frame extraction is fast (5-20s), vision processing is slow
* Whisper open-source on ZeroGPU: 5-10x speedup, free
* Unified Phase 1+2 architecture: single transcribe_audio() function

Co-Authored-By: Claude <noreply@anthropic.com>

CHANGELOG.md

@@ -1,5 +1,153 @@
 # Session Changelog
 
+## [2026-01-12] [Analysis] [COMPLETED] Course API Test Setup - Fixed vs Variable
+
+**Purpose:** Understand which parts of template are FIXED (course API contract) vs CAN MODIFY (our improvements).
+
+**Critical Finding:** Course API has a FIXED test setup - questions are NOT random.
+
+### Fixed (Course API Contract - DO NOT CHANGE)
+
+| Aspect | Value | Cannot Change |
+|--------|-------|----------------|
+| **API Endpoint** | `agents-course-unit4-scoring.hf.space` | ❌ |
+| **Questions Route** | `GET /questions` | ❌ |
+| **Submit Route** | `POST /submit` | ❌ |
+| **Number of Questions** | **20** (always 20) | ❌ |
+| **Question Source** | GAIA validation set, level 1 | ❌ |
+| **Randomness** | **NO - Fixed set** | ❌ |
+| **Difficulty** | All level 1 (easiest) | ❌ |
+| **Filter Criteria** | By tools/steps complexity | ❌ |
+| **Scoring** | EXACT MATCH | ❌ |
+| **Target Score** | 30% = 6/20 correct | ❌ |
+
+### The 20 Questions (ALWAYS the Same)
+
+| # | Full Task ID | Description | Tools Required |
+|---|--------------|-------------|----------------|
+| 1 | `2d83110e-a098-4ebb-9987-066c06fa42d0` | Reverse sentence (calculator) | Calculator |
+| 2 | `4fc2f1ae-8625-45b5-ab34-ad4433bc21f8` | Wikipedia dinosaur nomination | Web search |
+| 3 | `a1e91b78-d3d8-4675-bb8d-62741b4b68a6` | YouTube video - bird species | Video processing |
+| 4 | `8e867cd7-cff9-4e6c-867a-ff5ddc2550be` | Mercedes Sosa albums count | Web search |
+| 5 | `9d191bce-651d-4746-be2d-7ef8ecadb9c2` | YouTube video - Teal'c quote | Video processing |
+| 6 | `6f37996b-2ac7-44b0-8e68-6d28256631b4` | Operation table commutativity | CSV file |
+| 7 | `cca530fc-4052-43b2-b130-b30968d8aa44` | Chess position - winning move | Image analysis |
+| 8 | `3cef3a44-215e-4aed-8e3b-b1e3f08063b7` | Grocery list - vegetables only | Knowledge |
+| 9 | `305ac316-eef6-4446-960a-92d80d542f82` | Polish Ray actor character | Web search |
+| 10 | `99c9cc74-fdc8-46c6-8f8d-3ce2d3bfeea3` | Strawberry pie recipe | MP3 audio |
+| 11 | `cabe07ed-9eca-40ea-8ead-410ef5e83f91` | Equine veterinarian surname | Web search |
+| 12 | `f918266a-b3e0-4914-865d-4faa564f1aef` | Python code output | Python execution |
+| 13 | `1f975693-876d-457b-a649-393859e79bf3` | Calculus audio - page numbers | MP3 audio |
+| 14 | `840bfca7-4f7b-481a-8794-c560c340185d` | NASA award number | PDF processing |
+| 15 | `bda648d7-d618-4883-88f4-3466eabd860e` | Vietnamese specimens city | Web search |
+| 16 | `3f57289b-8c60-48be-bd80-01f8099ca449` | Yankee at-bats count | Web search |
+| 17 | `a0c07678-e491-4bbc-8f0b-07405144218f` | Pitcher numbers (before/after) | Web search |
+| 18 | `cf106601-ab4f-4af9-b045-5295fe67b37d` | Olympics least athletes | Web search |
+| 19 | `5a0c1adf-205e-4841-a666-7c3ef95def9d` | Malko Competition recipient | Web search |
+| 20 | `7bd855d8-463d-4ed5-93ca-5fe35145f733` | Excel food sales calculation | Excel file |
+
+**NOT random** - same 20 questions every submission!
+
+### Template Contract (MUST Preserve)
+
+```python
+# REQUIRED - Do NOT change
+questions_url = f"{api_url}/questions"  # Fixed route
+submit_url = f"{api_url}/submit"          # Fixed route
+
+submission_data = {
+    "username": username,
+    "agent_code": agent_code,
+    "answers": answers_payload            # Fixed format
+}
+```
+
+### Our Additions (SAFE to Modify)
+
+| Feature | Purpose | Required? |
+|---------|---------|-----------|
+| Question Limit | Debug: run first N | ✅ Optional |
+| Target Task IDs | Debug: run specific | ✅ Optional |
+| ThreadPoolExecutor | Speed: concurrent | ✅ Optional |
+| System Error Field | UX: error tracking | ✅ Optional |
+| File Download (HF) | Feature: support files | ✅ Optional |
+
+### Key Learnings
+
+1. **Question set is FIXED** - not random, always same 20
+2. **API routes are FIXED** - cannot change endpoints
+3. **Submission format is FIXED** - must match exactly
+4. **Our additions are OPTIONAL** - debug/features we added
+5. **Original template is 8777 bytes** - ours is 32722 bytes (4x larger)
+
+**Reference:** `_template_original/app.py` for original structure
+
+---
+
+## [2026-01-12] [Infrastructure] [COMPLETED] Original Template Reference Added
+
+**Purpose:** Compare current work with original template to understand changes and avoid breaking template structure.
+
+**Process:**
+1. Cloned original template to `/Users/mangubee/Downloads/Final_Assignment_Template`
+2. Removed git-specific files (`.git/` folder, `.gitattributes`)
+3. Copied to project as `_template_original/` (static reference, no git)
+4. Cleaned up temporary clone from Downloads
+
+**Why Static Reference:**
+- No `.git/` folder → won't interfere with project's git
+- No `.gitattributes` → clean file comparison
+- Pure reference material for diff/comparison
+- Can see exactly what changed from original
+
+**Template Original Contents:**
+- `app.py` (8777 bytes - original)
+- `README.md` (400 bytes - original)
+- `requirements.txt` (15 bytes - original)
+
+**Comparison Commands:**
+```bash
+# Compare file sizes
+ls -lh _template_original/app.py app.py
+
+# See differences
+diff _template_original/app.py app.py
+
+# Count lines added
+wc -l app.py _template_original/app.py
+```
+
+**Created Files:**
+- **_template_original/** (NEW) - Static reference to original template (3 files)
+
+---
+
+## [2026-01-12] [Infrastructure] [COMPLETED] HuggingFace Space Renamed
+
+**Context:** User wanted to compare current work with original template. Needed to rename current Space to free up `Final_Assignment_Template` name.
+
+**Actions Taken:**
+1. Renamed HuggingFace Space: `mangubee/Final_Assignment_Template` → `mangubee/agentbee`
+2. Updated local git remote to point to new URL
+3. Committed all today's changes (system error field, calculator fix, target task IDs, docs)
+4. Pulled from remote (sync after rename - already up to date)
+5. Pushed commits to renamed Space: `c86df49..41ac444`
+
+**Key Learnings:**
+- Local folder name ≠ git repo identity (can rename locally without affecting remote)
+- Git remote URL determines push destination (updated to `agentbee`)
+- HuggingFace Space name is independent of local folder name
+- All work preserved through rename process
+
+**Current State:**
+- Local: `Final_Assignment_Template/` (folder name unchanged for convenience)
+- Remote: `mangubee/agentbee` (renamed on HuggingFace)
+- Sync: ✅ All changes pushed
+- Git: All commits synced
+- Template: `_template_original/` added for comparison
+
+---
+
 ## [2026-01-12] [Documentation] [COMPLETED] Course vs Official GAIA Clarification
 
 **Problem:** Confusion about which leaderboard we're submitting to. Mistakenly thought we needed to submit to official GAIA, but we're actually implementing the course assignment API.

PLAN.md

@@ -1,698 +1,338 @@
-# Implementation Plan - LLM Selection Routing & HuggingFace Vision Support
+# Implementation Plan - System Error Fixes for 30% Target
 
-**Date:** 2026-01-06
-**Status:** Planning
+**Date:** 2026-01-13
+**Status:** Active
+**Current Score:** 10% (2/20 correct)
+**Target:** 30% (6/20 correct)
 
 ## Objective
 
-Fix LLM selection routing so UI provider selection propagates to ALL tools (planning, tool selection, synthesis, AND vision). Enable vision capability using HuggingFace multimodal models.
+Fix remaining 6 system errors to unlock questions, then address LLM quality issues to reach 30% target (6/20 correct).
 
-## Current Problems
+## Current Status Analysis
 
-1. **Vision tool ignores UI selection** - Hardcoded Gemini → Claude fallback
-2. **No HuggingFace vision support** - HF Inference API integration missing multimodal capability
-3. **Inconsistent routing** - Planning/tool selection respect UI, vision doesn't
+### ✅ Working (2/20 correct - 10%)
 
-## Solution Architecture
+| # | Task | Status | Issue |
+|---|------|--------|-------|
+| 9 | Polish Ray actor | ✅ Correct | - |
+| 15 | Vietnamese specimens | ✅ Correct | - |
 
-### Part 1: Fix LLM Selection Routing
+### ⚠️ System Errors (6/20 - Technical issues blocking)
 
-**Goal:** When user selects "HuggingFace" in UI, ALL agent components use HuggingFace LLM
+| # | Task | Error | Type | Priority |
+|---|------|-------|------|----------|
+| **3** | YouTube video (bird species) | Vision tool can't handle URLs | Technical | **HIGH** |
+| **5** | YouTube video (Teal'c) | Vision tool can't handle URLs | Technical | **HIGH** |
+| **6** | CSV table (commutativity) | LLM tries to load `table_data.csv` | LLM Quality | MED |
+| **10** | MP3 audio (pie recipe) | Unsupported file type | Technical | **MED** |
+| **12** | Python code execution | Unsupported file type | Technical | **LOW** |
+| **13** | MP3 audio (calculus) | Unsupported file type | Technical | **MED** |
 
-**Changes needed:**
+### ❌ LLM Quality Issues (12/20 - AI can't solve)
 
-1. **Vision tool (src/tools/vision.py):**
+| # | Task | Answer | Expected | Type |
+|---|------|--------|----------|------|
+| 1 | Calculator | "Unable to answer" | Right | Reasoning |
+| 2 | Wikipedia dinosaur | "Scott Hartman" | FunkMonk | Knowledge |
+| 4 | Mercedes Sosa albums | "Unable to answer" | 3 | Knowledge |
+| 7 | Chess position | "Unable to answer" | Rd5 | Vision+Reasoning |
+| 8 | Grocery list (botany) | Wrong (includes fruits) | 5 items | Knowledge |
+| 11 | Equine veterinarian | "Unable to answer" | Louvrier | Knowledge |
+| 14 | NASA award | "Unable to answer" | 80GSFC21M0002 | Knowledge |
+| 16 | Yankee at-bats | "Unable to answer" | 519 | Knowledge |
+| 17 | Pitcher numbers | "Unable to answer" | Yoshida, Uehara | Knowledge |
+| 18 | Olympics athletes | "Unable to answer" | CUB | Knowledge |
+| 19 | Malko Competition | "Unable to answer" | Claus | Knowledge |
+| 20 | Excel sales | "12096.00" | "89706.00" | Calculation |
 
-   - Add `analyze_image_hf()` function for HuggingFace multimodal models
-   - Modify `analyze_image()` to check `os.getenv("LLM_PROVIDER")`
-   - Route to correct provider: `gemini`, `huggingface`, `groq`, `claude`
-   - Respect `ENABLE_LLM_FALLBACK` setting
+## Strategy
 
-2. **Ensure consistency:**
-   - Planning: ✅ Already respects `LLM_PROVIDER`
-   - Tool selection: ✅ Already respects `LLM_PROVIDER`
-   - Synthesis: ✅ Already respects `LLM_PROVIDER`
-   - Vision: ❌ **NEEDS FIX** - Add routing logic
+**Priority 1: Fix System Errors** (unlock 6 questions)
+- YouTube videos (2 questions) - HIGH impact
+- MP3 audio (2 questions) - Medium impact
+- Python execution (1 question) - Low impact
+- CSV table - LLM issue, not technical
 
-### Part 2: HuggingFace Vision Capability
+**Priority 2: Improve LLM Quality** (address "Unable to answer" cases)
+- Better prompting
+- Tool selection improvements
+- Reasoning enhancements
 
-**Two approaches identified:**
+## Implementation Plan
 
-#### Option A: Direct Multimodal LLM (Preferred)
+### Phase 1: YouTube Video Support (HIGH Priority)
 
-**Approach:** Use HuggingFace multimodal models that support vision + text
+**Goal:** Fix questions #3 and #5 (YouTube videos)
 
-**Candidate models:**
+**Root Cause:** Vision tool tries to process YouTube URLs directly, but:
+- YouTube videos need to be downloaded first
+- Vision tool expects image files, not video URLs
+- Need to extract frames or use transcript
 
-1. **Qwen/Qwen2-VL-72B-Instruct** (Recommended)
+**Solution Options:**
 
-   - 72B parameters, vision-language model
-   - Supports: images, video, text
-   - API: HuggingFace Inference API (paid tier)
-   - Format: Base64 image + text prompt
-
-2. **meta-llama/Llama-3.2-90B-Vision-Instruct**
-
-   - 90B parameters, multimodal
-   - Supports: images + text
-   - API: HuggingFace Inference API
-
-3. **microsoft/Phi-3.5-vision-instruct**
-   - Smaller model (3.8B), efficient
-   - Supports: images + text
-   - Good for testing/debugging
+#### Option A: YouTube Transcript (Recommended)
 
 **Implementation:**
-
-- Use `InferenceClient.chat_completion()` with image content
-- Send base64-encoded images in messages array
-- Similar to Claude vision integration pattern
+```python
+# NEW: src/tools/youtube.py
+import youtube_transcript_api
+
+def get_youtube_transcript(video_url: str) -> str:
+    """Extract transcript from YouTube video."""
+    try:
+        video_id = extract_video_id(video_url)
+        transcript = YouTubeTranscriptApi.get_transcript(video_id)
+        return format_transcript(transcript)
+    except Exception as e:
+        return f"ERROR: Could not extract transcript: {e}"
+```
 
 **Pros:**
-
-- ✅ Native vision understanding
-- ✅ Single API call (no preprocessing)
-- ✅ Better accuracy for visual reasoning
-- ✅ Consistent with current architecture
+- ✅ Works with current LLM (text-based)
+- ✅ Simple API (youtube-transcript-api library)
+- ✅ Fast, no video download needed
+- ✅ Solves both #3 and #5
 
 **Cons:**
+- ❌ Won't work for visual-only questions (but our questions are about content)
+- ❌ Might not capture visual details
 
-- ❌ Requires HuggingFace paid tier (but user confirmed they have this)
-- ❌ Need to verify which models work with Inference API
-
-#### Option B: Image-to-Text Preprocessing
-
-**Approach:** Convert images to text descriptions using separate tool, then feed to text-only LLM
-
-**Tools available:**
+**Decision:** Use transcript approach since questions ask about content (bird species, dialogue)
 
-1. **BLIP-2** (Salesforce/blip2-opt-2.7b)
-
-   - Image captioning model
-   - Converts image → text description
-
-2. **LLaVA** (llava-hf/llava-1.5-7b-hf)
-
-   - Vision-language assistant
-   - Image → detailed text
-
-3. **OpenCV + OCR** (pytesseract)
-   - Extract text from images
-   - Good for documents/screenshots
+#### Option B: Video Frame Extraction
 
 **Implementation:**
+- Download video (yt-dlp)
+- Extract key frames (OpenCV)
+- Pass frames to vision tool
 
-- Load image → Run BLIP-2/LLaVA → Get text description
-- Pass text description to HuggingFace text-only LLM
-- Two-step process: vision → text → reasoning
-
-**Pros:**
-
-- ✅ Works with any text-only LLM
-- ✅ Cheaper (can use smaller vision models)
-- ✅ Fallback option if multimodal API unavailable
-
-**Cons:**
-
-- ❌ Two API calls (slower)
-- ❌ Information loss in image → text conversion
-- ❌ Poor for complex visual reasoning (chess positions, video analysis)
-- ❌ Extra dependency management
-
-## Recommended Approach
-
-**Use Option A: Direct Multimodal LLM (Qwen2-VL-72B-Instruct)**
-
-**Reasoning:**
-
-1. User has HuggingFace paid tier access (confirmed)
-2. GAIA questions require complex visual reasoning (chess positions, video analysis)
-3. Simpler architecture - consistent with existing pattern
-4. Better accuracy for benchmark performance
-5. Focus on HF testing first, Groq later
-
-**Fallback:** Keep Option B as backup if multimodal API doesn't work
-
-## Implementation Steps
-
-### Phase 0: API Validation (CRITICAL - DO THIS FIRST)
-
-**Goal:** Validate HuggingFace Inference API supports vision BEFORE implementation
-
-**Decision Gate 1:** Only proceed to Phase 1 if at least one model works
+**Pros:** Visual analysis
+**Cons:** Slow, complex, overkill for content questions
 
-#### Step 0.1: Test HF Inference API with Vision Models
+#### Step 1.1: Install youtube-transcript-api
 
-- [ ] Test **Phi-3.5-vision-instruct** (3.8B) - Smallest, fastest iteration
-- [ ] Test **Llama-3.2-11B-Vision-Instruct** - Medium model
-- [ ] Test **Qwen2-VL-72B-Instruct** - Largest, only if needed
-- [ ] Simple test: Load apple image, ask "What is this?"
-- [ ] Verify API accepts vision input (base64, URL, or file path)
-- [ ] Document response format and error patterns
-
-#### Step 0.2: Test Image Format Support
-
-- [ ] Base64 encoding in messages
-- [ ] Direct URL support
-- [ ] Local file path support
-- [ ] Document which format(s) work
-
-#### Step 0.3: Document API Behavior
+```bash
+uv add youtube-transcript-api
+```
 
-- [ ] Response structure (JSON schema)
-- [ ] Error patterns (quota, rate limit, invalid input)
-- [ ] Rate limits and quotas
-- [ ] Model selection recommendation
+#### Step 1.2: Create YouTube tool
 
-#### Step 0.4: Decision Gate - GO/NO-GO
+```python
+# src/tools/youtube.py
+def youtube_transcript(video_url: str) -> str:
+    """Extract transcript from YouTube video."""
+```
 
-- [ ] **GO:** At least 1 model works → Proceed to Phase 1
-- [ ] **NO-GO:** 0 models work → Pivot to backup options:
-  - **Option C:** HF Spaces deployment (custom endpoint)
-  - **Option D:** Local transformers library (no API)
-  - **Option E:** Hybrid (HF text + Gemini/Claude vision only)
+#### Step 1.3: Register tool
 
-**Phase 0 Status:** ✅ COMPLETED - Multiple working models found
+```python
+# src/tools/__init__.py
+TOOLS = [
+    ...
+    {"name": "youtube_transcript", "func": youtube_transcript,
+     "description": "Extract transcript from YouTube video URL. Use when question mentions YouTube video content like dialogue, speech, or visual descriptions."},
+]
+```
 
-**Validated Models (Ranked by Speed):**
+#### Step 1.4: Test
 
-| Rank | Model | Provider | Speed | Notes |
-|------|-------|----------|-------|-------|
-| 1 | `google/gemma-3-27b-it` | Scaleway | ~6s | **RECOMMENDED** - Google brand, fastest |
-| 2 | `CohereLabs/aya-vision-32b` | Cohere | ~7s | Fast, less known brand |
-| 3 | `Qwen/Qwen3-VL-30B-A3B-Instruct` | Novita | ~14s | Qwen brand, reputable |
-| 4 | `zai-org/GLM-4.6V-Flash` | zai-org | ~16s | Zhipu AI brand |
+```bash
+# Test on question #3
+Target Task ID: a1e91b78-d3d8-4675-bb8d-62741b4b68a6
+```
 
-**Format:** Base64 encoding only (file:// URLs don't work)
-**Test image:** 2.1MB workspace photo (realistic large image)
+**Expected impact:** +2 questions (30% → 40% if both work)
 
 ---
 
-### Phase 1: HuggingFace Vision Implementation
+### Phase 2: MP3 Audio Support (MEDIUM Priority)
 
-**Goal:** Implement `analyze_image_hf()` using validated API pattern
+**Goal:** Fix questions #10 and #13 (MP3 audio files)
 
-**Validated from Phase 0:**
+**Root Cause:** parse_file doesn't support .mp3
 
-- Model: `google/gemma-3-27b-it:scaleway` (RECOMMENDED - fastest, Google brand)
-- Format: Base64 encoding in messages array
-- Timeout: ~6 seconds for 2.1MB image
+**Solution:** Add audio transcription tool
 
-#### Step 1.1: Implement `analyze_image_hf()` in vision.py
-
-- [ ] Add function signature matching existing pattern
-- [ ] Use **google/gemma-3-27b-it:scaleway** (validated, fastest)
-- [ ] Format: Base64 encode images in messages array
-- [ ] Add retry logic with exponential backoff (3 attempts)
-- [ ] Handle API errors with clear error messages
-- [ ] Set 120s timeout for large images
-- [ ] **NO fallback logic** - fail loudly for debugging
-
-#### Step 1.2: Fix Vision Tool Routing (NO FALLBACKS)
-
-- [ ] Modify `analyze_image()` to check `os.getenv("LLM_PROVIDER")`
-- [ ] Add routing logic (each provider fails independently):
-
-  ```python
-  if provider == "huggingface":
-      return analyze_image_hf(image_path, question)  # Fail if error
-  elif provider == "gemini":
-      return analyze_image_gemini(image_path, question)  # Fail if error
-  elif provider == "claude":
-      return analyze_image_claude(image_path, question)  # Fail if error
-  # NO fallback chains during testing - defeats isolation purpose
-  ```
+**Implementation:**
+```python
+# NEW: src/tools/audio.py
+import whisper
+
+def transcribe_audio(file_path: str) -> str:
+    """Transcribe audio file to text using OpenAI Whisper."""
+    model = whisper.load_model("base")
+    result = model.transcribe(file_path)
+    return result["text"]
+```
 
-- [ ] Log exact failure reason for debugging
-- [ ] Add placeholder for `groq` (future Phase 4)
+**Alternative:** HuggingFace audio models (free)
+- `openai/whisper-base`
+- Use via Inference API
 
-#### Step 1.3: Update Configuration
+**Step 2.1:** Choose implementation (Whisper vs HF)
+**Step 2.2:** Implement audio tool
+**Step 2.3:** Add to TOOLS registry
+**Step 2.4:** Test on #10 and #13
 
-- [ ] Add `HF_VISION_MODEL=CohereLabs/aya-vision-32b` to .env (validated from Phase 0)
-- [ ] Update `src/config/settings.py` with vision model setting
-- [ ] Document alternatives (Qwen/Qwen3-VL-8B-Instruct for small images only)
+**Expected impact:** +2 questions (30% → 40% if both work)
 
 ---
 
-### Phase 2: Smoke Tests (Before GAIA Evaluation)
-
-**Goal:** Validate basic vision works before complex GAIA questions
-
-**Decision Gate 2:** Only proceed to Phase 3 if ≥3/4 smoke tests pass
-
-#### Step 2.1: Simple Image Description Test
+### Phase 3: Python Code Execution (LOW Priority)
 
-- [ ] Test image: Photo of apple
-- [ ] Question: "Describe this image"
-- [ ] Expected: Basic object recognition works
-- [ ] Export: `output/smoke_test_description.json`
+**Goal:** Fix question #12 (Python code output)
 
-#### Step 2.2: OCR Test
+**Root Cause:** parse_file doesn't support .py execution
 
-- [ ] Test image: Image with text "Hello World"
-- [ ] Question: "What text do you see?"
-- [ ] Expected: Text extraction works
-- [ ] Export: `output/smoke_test_ocr.json`
+**Solution:** Add code execution tool (sandboxed)
 
-#### Step 2.3: Counting Test
+**Security Concern:** ⚠️ **DANGEROUS** - executing arbitrary Python code
 
-- [ ] Test image: Image with 3 distinct objects
-- [ ] Question: "How many objects are visible?"
-- [ ] Expected: Visual reasoning works
-- [ ] Export: `output/smoke_test_counting.json`
+**Options:**
+1. **Restricted execution** - Only allow specific operations
+2. **Docker container** - Isolate execution
+3. **Skip for now** - Defer due to security concerns
 
-#### Step 2.4: Single GAIA Question Test
+**Decision:** Mark as **DEFERRED** due to security complexity
 
-- [ ] Select easiest GAIA vision question
-- [ ] Run with HuggingFace provider
-- [ ] Verify end-to-end integration works
-- [ ] Export: `output/smoke_test_gaia_single.json`
-
-#### Step 2.5: Decision Gate - GO/NO-GO
-
-- [ ] **GO:** ≥3/4 smoke tests pass → Proceed to Phase 3
-- [ ] **NO-GO:** <3/4 pass → Debug before GAIA evaluation
+**Expected impact:** +1 question (if implemented)
 
 ---
 
-### Phase 3: GAIA Evaluation (Only if Smoke Tests Pass)
-
-**Goal:** Test HuggingFace vision on full GAIA benchmark
-
-#### Step 3.1: Run Full GAIA Evaluation (HuggingFace Only)
-
-- [ ] Set `LLM_PROVIDER=huggingface` in UI
-- [ ] Run all 20 questions
-- [ ] Export: `output/gaia_results_hf_TIMESTAMP.json` (HF only, no mixing)
-- [ ] Log which questions use vision tool vs other tools
+### Phase 4: CSV Table Issue (LLM Quality)
 
-#### Step 3.2: Analyze Results
+**Goal:** Fix question #6 (table commutativity)
 
-- [ ] Calculate accuracy: X/20 correct
-- [ ] Break down by question type:
-  - Vision questions: X/8 correct
-  - Non-vision questions: X/12 correct
-- [ ] Identify failure patterns (vision errors, wrong answers, tool selection errors)
-- [ ] Compare to 0% baseline
+**Root Cause:** LLM tries to load `table_data.csv` when data is IN the question
 
-#### Step 3.3: Build Capability Matrix
+**Solution:** This is NOT technical - LLM needs better prompts or tool selection
 
-- [ ] Document per-provider results:
+**Approaches:**
+1. Improve system prompt to recognize data in questions
+2. Add hint in question preprocessing
+3. Special handling for markdown tables in questions
 
-| Provider              | Vision Questions | Accuracy | Notes          |
-| --------------------- | ---------------- | -------- | -------------- |
-| HuggingFace (Phi-3.5) | 8/8 attempted    | X%       | [observations] |
-| Gemini (baseline)     | 8/8 attempted    | Y%       | [comparison]   |
+**Current workaround:** System correctly identifies as "no_evidence" and doesn't crash
 
-#### Step 3.4: Decision Gate - Optimization Decision
-
-- [ ] **If accuracy ≥20%:** Good enough, proceed to Phase 4 (media processing)
-- [ ] **If accuracy <20%:** Analyze failures, try larger HF model (Llama-3.2 or Qwen2-VL)
-- [ ] **If accuracy <5%:** Re-evaluate approach, consider backup options
+**Status:** Defer to LLM quality improvements (Phase 5)
 
 ---
 
-### Phase 4: Media Processing Gaps (After Vision Works)
-
-**Goal:** Add YouTube and audio support
+### Phase 5: LLM Quality Improvements
 
-#### Step 4.1: YouTube Video Support
+**Goal:** Convert "Unable to answer" → correct answers
 
-- [ ] Add YouTube transcript extraction tool
-- [ ] Use `youtube-transcript-api` library
-- [ ] Extract dialogue/captions as text
-- [ ] Pass transcript to LLM for question answering
-- [ ] Test on GAIA YouTube questions (bird species, Stargate quote)
-- [ ] Export: `output/gaia_results_hf_with_youtube.json`
+**Target questions (by category):**
 
-#### Step 4.2: Audio File Support
+**Knowledge/Research (9 questions):** #2, #4, #11, #14, #16, #17, #18, #19
+**Reasoning/Calculation (2 questions):** #1, #20
+**Vision+Reasoning (1 question):** #7
 
-- [ ] Add audio transcription tool
-- [ ] Use OpenAI Whisper or HuggingFace audio models
-- [ ] Transcribe audio → text
-- [ ] Pass transcript to LLM
-- [ ] Test on GAIA audio question (Strawberry pie.mp3)
-- [ ] Export: `output/gaia_results_hf_with_audio.json`
-
----
+**Approaches:**
+1. **Better prompts** - Emphasize exact answer format
+2. **Tool selection hints** - Guide LLM to use appropriate tools
+3. **Few-shot examples** - Show LLM expected answer format
+4. **Chain-of-thought** - Encourage step-by-step reasoning
 
-### Phase 5: Groq Vision Integration (Future)
-
-**Goal:** Add free tier fallback option
-
-#### Step 5.1: Add Groq Vision Support
-
-- [ ] Implement `analyze_image_groq()` using Llama-3.2-90B-Vision
-- [ ] Add to vision tool routing (independent, no fallback)
-- [ ] Test with Groq free tier (30 req/min)
-- [ ] Export: `output/gaia_results_groq_TIMESTAMP.json`
-- [ ] Compare accuracy: HF vs Groq
+**Implementation:**
+- Update `synthesize_answer()` prompt
+- Add answer format examples to system prompt
+- Improve tool descriptions for better selection
 
 ---
 
-### Phase 6: Final Verification
-
-**Goal:** Document final results and verify all tests pass
-
-#### Step 6.1: Final GAIA Evaluation (All Media Types)
-
-- [ ] Run all 20 questions with HuggingFace
-- [ ] Verify: images, videos, audio all work
-- [ ] Export: `output/gaia_results_final_TIMESTAMP.json`
-- [ ] Document final accuracy vs 0% baseline
-
-#### Step 6.2: Regression Testing
-
-- [ ] Run all 99 tests
-- [ ] Verify no regressions introduced
-- [ ] Fix any broken tests
-
-#### Step 6.3: Documentation
-
-- [ ] Update CHANGELOG.md with final results
-- [ ] Update README.md with HF vision support
-- [ ] Document model selection strategy
-
-## Files to Modify
-
-### Phase 0-1: Core Vision Integration
-
-1. **src/tools/vision.py** (~150 lines added/modified)
-
-   - Add `analyze_image_hf()` function (Phase 1)
-   - Modify `analyze_image()` routing logic - NO FALLBACKS (Phase 1)
-   - Add retry logic with exponential backoff
-   - Clear error messages for debugging
-
-2. **.env** (~3 lines added)
-
-   - Add `HF_VISION_MODEL=microsoft/Phi-3.5-vision-instruct` (start small)
-   - Document alternatives: Llama-3.2-11B-Vision, Qwen2-VL-72B
-
-3. **src/config/settings.py** (~5 lines)
-   - Add `hf_vision_model` setting
-   - Load from environment variable
-
-### Phase 2-3: Testing Infrastructure
-
-1. **test/test_vision_smoke.py** (NEW - ~100 lines)
-
-   - Smoke test suite: description, OCR, counting, single GAIA
-   - Export individual test results
-
-2. **app.py** (optional - ~10 lines)
-   - Update export filenames to include provider: `gaia_results_hf_TIMESTAMP.json`
-   - Separate results per provider for capability matrix
-
-### Phase 4: Media Processing
-
-1. **src/tools/youtube.py** (NEW - ~80 lines)
-
-   - YouTube transcript extraction
-   - Use `youtube-transcript-api`
-
-2. **src/tools/audio.py** (NEW - ~80 lines)
-
-   - Audio transcription (Whisper or HF audio models)
-   - Convert audio → text
-
-3. **src/tools/**init**.py** (~10 lines)
-
-   - Register new tools: youtube_transcript, audio_transcribe
-
-4. **requirements.txt** (~3 lines)
-   - Add `youtube-transcript-api`
-   - Add `openai-whisper` or HF audio model library
-
-### Phase 6: Documentation
-
-1. **README.md** (~30 lines modified)
-   - Document HF vision support
-   - List model options and selection strategy
-   - Update architecture diagram with media processing tools
-
 ## Success Criteria
 
-### Phase 0: API Validation
-
-- [ ] At least 1 HF vision model works with Inference API
-- [ ] Image format documented (base64/URL/file)
-- [ ] Response format documented
-
-### Phase 1: Implementation
-
-- [ ] `analyze_image_hf()` function implemented
-- [ ] Vision tool routing respects `LLM_PROVIDER` (NO FALLBACKS)
-- [ ] Clear error messages when provider fails
+### Phase 1: YouTube Support
+- [ ] YouTube transcript tool implemented
+- [ ] Question #3 answered correctly (bird species = "3")
+- [ ] Question #5 answered correctly (Teal'c quote = "Extremely")
+- [ ] **Score: 10% → 40% (4/20)** ✅ TARGET REACHED
+
+### Phase 2: MP3 Support
+- [ ] Audio transcription tool implemented
+- [ ] Question #10 answered correctly (pie ingredients)
+- [ ] Question #13 answered correctly (page numbers)
+- [ ] **Score: 40% → 50% (10/20)** ✅ EXCEEDS TARGET
+
+### Phase 3: Python Execution
+- [ ] Code execution tool implemented (sandboxed)
+- [ ] Question #12 answered correctly (output = "0")
+- [ ] **Score: 50% → 55% (11/20)**
+
+### Phase 4: CSV Table
+- [ ] LLM recognizes data in question
+- [ ] Question #6 answered correctly ("b, e")
+- [ ] **Score: 55% → 60% (12/20)**
+
+### Phase 5: LLM Quality
+- [ ] "Unable to answer" reduced by 50%
+- [ ] At least 3 more knowledge questions correct
+- [ ] **Score: 60% → 75%+ (15/20)**
 
-### Phase 2: Smoke Tests
-
-- [ ] ≥3/4 smoke tests pass
-- [ ] Basic vision capabilities validated
-
-### Phase 3: GAIA Evaluation
-
-- [ ] UI LLM selection propagates to vision tool
-- [ ] HuggingFace-only results exported: `output/gaia_results_hf_TIMESTAMP.json`
-- [ ] Accuracy measured and compared to 0% baseline
-- [ ] Capability matrix built (per-provider comparison)
-
-### Phase 4-6: Full Coverage
-
-- [ ] YouTube video questions work (transcript extraction)
-- [ ] Audio questions work (transcription)
-- [ ] All 99 tests still passing
-- [ ] Final accuracy ≥20% (minimum acceptable)
-
-## Backup Strategy Options
-
-If Phase 0 reveals HF Inference API doesn't support vision:
-
-### Option C: HuggingFace Spaces Deployment
-
-- Deploy custom vision model to HF Spaces
-- Use Inference Endpoints (paid tier)
-- More control, higher cost
-
-### Option D: Local Transformers Library
-
-- Use `transformers` library directly (no API)
-- Load model locally: `AutoModelForVision2Seq`
-- Slower, requires GPU, but guaranteed to work
-
-### Option E: Hybrid Architecture
-
-- Keep HuggingFace for text-only LLM
-- Use Gemini/Claude for vision only
-- Compromise: HF testing focus, but vision delegates to working providers
-
-## Decision Gates Summary
-
-**Gate 1 (Phase 0):** Does HF API support vision?
-
-- **GO:** ≥1 model works → Phase 1
-- **NO-GO:** 0 models work → Pivot to Option C/D/E
-
-**Gate 2 (Phase 2):** Do smoke tests pass?
+## Files to Modify
 
-- **GO:** ≥3/4 pass → Phase 3
-- **NO-GO:** <3/4 pass → Debug before GAIA
+### Phase 1: YouTube
+1. **requirements.txt** - Add `youtube-transcript-api`
+2. **src/tools/youtube.py** (NEW) - YouTube transcript extraction
+3. **src/tools/__init__.py** - Register youtube_transcript tool
 
-**Gate 3 (Phase 3):** Is accuracy acceptable?
+### Phase 2: MP3 Audio
+1. **requirements.txt** - Add `openai-whisper` or HF audio
+2. **src/tools/audio.py** (NEW) - Audio transcription
+3. **src/tools/__init__.py** - Register transcribe_audio tool
 
-- **GO:** ≥20% → Phase 4 (media processing)
-- **ITERATE:** <20% → Try larger model or analyze failures
-- **PIVOT:** <5% → Re-evaluate approach
+### Phase 3-5: LLM Quality
+1. **src/agent/graph.py** - Update prompts
+2. **src/tools/__init__.py** - Improve tool descriptions
 
-## Phase 0 Research Questions (Answer These First)
+## Removed (Not Relevant)
 
-1. **Does HF Inference API support vision models?**
+- ~~Phase 0: Vision API validation~~ (already using Gemma 3)
+- ~~Phase 1: HuggingFace vision~~ (not current priority)
+- ~~Phase 2: Smoke tests~~ (already working)
+- ~~Phase 3: GAIA evaluation~~ (running successfully)
+- ~~Phase 5: Groq vision~~ (fallback archived)
+- ~~Phase 6: Final verification~~ (premature)
+- ~~Phase 7: File attachment~~ (already implemented)
 
-   - Test Phi-3.5-vision-instruct with simple image
-   - Test Llama-3.2-11B-Vision-Instruct
-   - Test Qwen2-VL-72B-Instruct
+## Decision Gates
 
-2. **What's the image input format?**
+**Gate 1 (YouTube):** Does transcript solve both video questions?
+- **YES:** 40% score, proceed to Phase 2
+- **NO:** Try frame extraction approach
 
-   - Base64 encoding in messages?
-   - Direct URL support?
-   - File path support?
+**Gate 2 (MP3):** Does transcription solve both audio questions?
+- **YES:** 50% score, proceed to Phase 3
+- **NO:** Try different audio model
 
-3. **What's the response structure?**
-   - JSON schema format
-   - Error patterns
-   - Rate limits and quotas
+**Gate 3 (Target):** Have we reached 30% (6/20)?
+- **YES:** ✅ SUCCESS - course target met
+- **NO:** Continue to Phase 4-5
 
 ## Next Actions
 
-**Phase 0 starts with:**
+**Start with Phase 1 (YouTube):**
 
-1. ==Research HF Inference API documentation for vision support==
-2. Test simple vision API call with Phi-3.5-vision-instruct
-3. Document working pattern or confirm API doesn't support vision
-4. Decision gate: GO to Phase 1 or pivot to backup options
+1. [ ] Install youtube-transcript-api
+2. [ ] Create src/tools/youtube.py
+3. [ ] Add youtube_transcript to TOOLS
+4. [ ] Test on question #3: `a1e91b78-d3d8-4675-bb8d-62741b4b68a6`
+5. [ ] Run full evaluation
+6. [ ] Verify 40% score (4/20 correct)
 
----
+**After YouTube:** Proceed to MP3 support (Phase 2)
 
-## Phase 7: GAIA File Attachment Support
-
-**Goal:** Enable agent to download and process file attachments from GAIA questions
-
-**Problem:**
-- Current code ignores `file_name` field in GAIA questions
-- Files not downloaded from `GET /files/{task_id}` endpoint
-- Vision/file parsing tools fail with placeholder `<provided_image_path>`
-- ~40% of questions (8/20) fail due to missing file handling
-
-### Root Cause
-
-**GAIA Question Structure:**
-```json
-{
-  "task_id": "abc123",
-  "question": "What's in this image?",
-  "file_name": "chess.png",      // NULL if no file
-  "file_path": "/files/abc123"   // NULL if no file
-}
-```
-
-**Current Code (app.py:249-290):**
-```python
-def process_single_question(agent, item, index, total):
-    task_id = item.get("task_id")
-    question_text = item.get("question")
-    # ❌ MISSING: Check file_name
-    # ❌ MISSING: Download file
-    # ❌ MISSING: Pass file_path to agent
-
-    submitted_answer = agent(question_text)  # No file handling
-```
-
-**Result:** LLM generates `vision(image_path="<provided_image_path>")` → File not found error
-
-### Solution Architecture
-
-**Step 1: Add File Download Function**
-
-```python
-def download_task_file(task_id: str, save_dir: str = "input/") -> Optional[str]:
-    """Download file attached to a GAIA question.
-
-    Args:
-        task_id: Question's task_id
-        save_dir: Directory to save file
-
-    Returns:
-        File path if downloaded, None if no file
-    """
-    api_url = "https://agents-course-unit4-scoring.hf.space"
-    file_url = f"{api_url}/files/{task_id}"
-
-    response = requests.get(file_url, timeout=30)
-    response.raise_for_status()
-
-    # Get extension from Content-Type header
-    content_type = response.headers.get('Content-Type', '')
-    extension_map = {
-        'image/png': '.png',
-        'image/jpeg': '.jpg',
-        'application/pdf': '.pdf',
-        'text/csv': '.csv',
-        'application/json': '.json',
-        'application/vnd.ms-excel': '.xls',
-        'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet': '.xlsx',
-    }
-    extension = extension_map.get(content_type, '.file')
-
-    # Save file
-    Path(save_dir).mkdir(exist_ok=True)
-    file_path = f"{save_dir}{task_id}{extension}"
-    with open(file_path, 'wb') as f:
-        f.write(response.content)
-
-    return file_path
-```
-
-**Step 2: Modify Question Processing**
-
-```python
-def process_single_question(agent, item, index, total):
-    task_id = item.get("task_id")
-    question_text = item.get("question")
-    file_name = item.get("file_name")  # ✅ NEW
-
-    # Download file if exists
-    file_path = None
-    if file_name:
-        file_path = download_task_file(task_id)
-
-    # Pass file info to agent
-    submitted_answer = agent(question_text, file_path=file_path)  # ✅ NEW
-```
-
-**Step 3: Update LLM Context**
-
-When file_path is provided, include it in the planning prompt:
-```python
-if file_path:
-    question_context = f"Question: {question_text}\nAttached file: {file_path}"
-else:
-    question_context = question_text
-```
-
-### Implementation Steps
-
-#### Step 7.1: Add File Download Function
-
-- [ ] Create `download_task_file()` in app.py
-- [ ] Handle Content-Type to extension mapping
-- [ ] Handle 404 gracefully (no file for this task)
-- [ ] Create input/ directory if not exists
-
-#### Step 7.2: Modify Question Processing Loop
-
-- [ ] Check `item.get("file_name")` in process_single_question
-- [ ] Call download_task_file() if file_name exists
-- [ ] Pass file_path to agent invocation
-
-#### Step 7.3: Update Agent to Handle file_path
-
-- [ ] Modify agent to accept optional file_path parameter
-- [ ] Include file info in planning prompt
-- [ ] Update tool selection to use real file paths
-
-#### Step 7.4: Test File Handling
-
-- [ ] Test with image question (chess position)
-- [ ] Test with document question (Excel file)
-- [ ] Verify no more `<provided_image_path>` errors
-
-### Files to Modify
-
-1. **app.py** (~80 lines added/modified)
-   - Add download_task_file() function
-   - Modify process_single_question() to handle files
-   - Add input/ directory creation
-
-2. **src/agent/graph.py** (~20 lines)
-   - Update agent state to include file_path
-   - Pass file info to planning prompt
-
-3. **.gitignore** (~2 lines)
-   - Add input/ to ignore downloaded files
-
-### Success Criteria
+---
 
-- [ ] Image questions: Vision tool receives real file path
-- [ ] Document questions: parse_file tool receives real file path
-- [ ] No more `<provided_image_path>` errors
-- [ ] Accuracy improves from 10% toward 30%+
+## Backup Options
 
-### Expected Impact
+If YouTube transcript doesn't work:
+- **Plan B:** Extract video frames, analyze with vision tool
+- **Plan C:** Skip video questions, focus on other fixes
 
-| Before | After |
-|--------|-------|
-| 40% (8/20) fail with file errors | 0% file errors |
-| Vision questions: All fail | Vision questions: Can work |
-| Document questions: All fail | Document questions: Can work |
-| Max accuracy: ~60% | Max accuracy: ~100% potential |
+If MP3 transcription doesn't work:
+- **Plan B:** Use HuggingFace audio models
+- **Plan C:** Skip audio questions, focus on LLM quality

README.md

@@ -347,7 +347,7 @@ ENABLE_LLM_FALLBACK=false    # Disable fallback for debugging single provider
 
 **Test Coverage:** 99 passing tests (~2min 40sec runtime)
 
-> **Note:** This project implements the **Course Leaderboard** (20 questions, 30% target). See [GAIA Submission Guide](docs/gaia_submission_guide.md) for distinction between Course and Official GAIA leaderboards.
+> **Note:** This project implements the **Course Leaderboard** (20 questions, 30% target). See [GAIA Submission Guide](../agentbee/docs/gaia_submission_guide.md) for distinction between Course and Official GAIA leaderboards.
 
 ## Workflow
 

_template_original/README.md

@@ -0,0 +1,15 @@
+---
+title: Template Final Assignment
+emoji: 🕵🏻‍♂️
+colorFrom: indigo
+colorTo: indigo
+sdk: gradio
+sdk_version: 5.25.2
+app_file: app.py
+pinned: false
+hf_oauth: true
+# optional, default duration is 8 hours/480 minutes. Max duration is 30 days/43200 minutes.
+hf_oauth_expiration_minutes: 480
+---
+
+Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

_template_original/app.py

@@ -0,0 +1,196 @@
+import os
+import gradio as gr
+import requests
+import inspect
+import pandas as pd
+
+# (Keep Constants as is)
+# --- Constants ---
+DEFAULT_API_URL = "https://agents-course-unit4-scoring.hf.space"
+
+# --- Basic Agent Definition ---
+# ----- THIS IS WERE YOU CAN BUILD WHAT YOU WANT ------
+class BasicAgent:
+    def __init__(self):
+        print("BasicAgent initialized.")
+    def __call__(self, question: str) -> str:
+        print(f"Agent received question (first 50 chars): {question[:50]}...")
+        fixed_answer = "This is a default answer."
+        print(f"Agent returning fixed answer: {fixed_answer}")
+        return fixed_answer
+
+def run_and_submit_all( profile: gr.OAuthProfile | None):
+    """
+    Fetches all questions, runs the BasicAgent on them, submits all answers,
+    and displays the results.
+    """
+    # --- Determine HF Space Runtime URL and Repo URL ---
+    space_id = os.getenv("SPACE_ID") # Get the SPACE_ID for sending link to the code
+
+    if profile:
+        username= f"{profile.username}"
+        print(f"User logged in: {username}")
+    else:
+        print("User not logged in.")
+        return "Please Login to Hugging Face with the button.", None
+
+    api_url = DEFAULT_API_URL
+    questions_url = f"{api_url}/questions"
+    submit_url = f"{api_url}/submit"
+
+    # 1. Instantiate Agent ( modify this part to create your agent)
+    try:
+        agent = BasicAgent()
+    except Exception as e:
+        print(f"Error instantiating agent: {e}")
+        return f"Error initializing agent: {e}", None
+    # In the case of an app running as a hugging Face space, this link points toward your codebase ( usefull for others so please keep it public)
+    agent_code = f"https://huggingface.co/spaces/{space_id}/tree/main"
+    print(agent_code)
+
+    # 2. Fetch Questions
+    print(f"Fetching questions from: {questions_url}")
+    try:
+        response = requests.get(questions_url, timeout=15)
+        response.raise_for_status()
+        questions_data = response.json()
+        if not questions_data:
+             print("Fetched questions list is empty.")
+             return "Fetched questions list is empty or invalid format.", None
+        print(f"Fetched {len(questions_data)} questions.")
+    except requests.exceptions.RequestException as e:
+        print(f"Error fetching questions: {e}")
+        return f"Error fetching questions: {e}", None
+    except requests.exceptions.JSONDecodeError as e:
+         print(f"Error decoding JSON response from questions endpoint: {e}")
+         print(f"Response text: {response.text[:500]}")
+         return f"Error decoding server response for questions: {e}", None
+    except Exception as e:
+        print(f"An unexpected error occurred fetching questions: {e}")
+        return f"An unexpected error occurred fetching questions: {e}", None
+
+    # 3. Run your Agent
+    results_log = []
+    answers_payload = []
+    print(f"Running agent on {len(questions_data)} questions...")
+    for item in questions_data:
+        task_id = item.get("task_id")
+        question_text = item.get("question")
+        if not task_id or question_text is None:
+            print(f"Skipping item with missing task_id or question: {item}")
+            continue
+        try:
+            submitted_answer = agent(question_text)
+            answers_payload.append({"task_id": task_id, "submitted_answer": submitted_answer})
+            results_log.append({"Task ID": task_id, "Question": question_text, "Submitted Answer": submitted_answer})
+        except Exception as e:
+             print(f"Error running agent on task {task_id}: {e}")
+             results_log.append({"Task ID": task_id, "Question": question_text, "Submitted Answer": f"AGENT ERROR: {e}"})
+
+    if not answers_payload:
+        print("Agent did not produce any answers to submit.")
+        return "Agent did not produce any answers to submit.", pd.DataFrame(results_log)
+
+    # 4. Prepare Submission 
+    submission_data = {"username": username.strip(), "agent_code": agent_code, "answers": answers_payload}
+    status_update = f"Agent finished. Submitting {len(answers_payload)} answers for user '{username}'..."
+    print(status_update)
+
+    # 5. Submit
+    print(f"Submitting {len(answers_payload)} answers to: {submit_url}")
+    try:
+        response = requests.post(submit_url, json=submission_data, timeout=60)
+        response.raise_for_status()
+        result_data = response.json()
+        final_status = (
+            f"Submission Successful!\n"
+            f"User: {result_data.get('username')}\n"
+            f"Overall Score: {result_data.get('score', 'N/A')}% "
+            f"({result_data.get('correct_count', '?')}/{result_data.get('total_attempted', '?')} correct)\n"
+            f"Message: {result_data.get('message', 'No message received.')}"
+        )
+        print("Submission successful.")
+        results_df = pd.DataFrame(results_log)
+        return final_status, results_df
+    except requests.exceptions.HTTPError as e:
+        error_detail = f"Server responded with status {e.response.status_code}."
+        try:
+            error_json = e.response.json()
+            error_detail += f" Detail: {error_json.get('detail', e.response.text)}"
+        except requests.exceptions.JSONDecodeError:
+            error_detail += f" Response: {e.response.text[:500]}"
+        status_message = f"Submission Failed: {error_detail}"
+        print(status_message)
+        results_df = pd.DataFrame(results_log)
+        return status_message, results_df
+    except requests.exceptions.Timeout:
+        status_message = "Submission Failed: The request timed out."
+        print(status_message)
+        results_df = pd.DataFrame(results_log)
+        return status_message, results_df
+    except requests.exceptions.RequestException as e:
+        status_message = f"Submission Failed: Network error - {e}"
+        print(status_message)
+        results_df = pd.DataFrame(results_log)
+        return status_message, results_df
+    except Exception as e:
+        status_message = f"An unexpected error occurred during submission: {e}"
+        print(status_message)
+        results_df = pd.DataFrame(results_log)
+        return status_message, results_df
+
+
+# --- Build Gradio Interface using Blocks ---
+with gr.Blocks() as demo:
+    gr.Markdown("# Basic Agent Evaluation Runner")
+    gr.Markdown(
+        """
+        **Instructions:**
+
+        1.  Please clone this space, then modify the code to define your agent's logic, the tools, the necessary packages, etc ...
+        2.  Log in to your Hugging Face account using the button below. This uses your HF username for submission.
+        3.  Click 'Run Evaluation & Submit All Answers' to fetch questions, run your agent, submit answers, and see the score.
+
+        ---
+        **Disclaimers:**
+        Once clicking on the "submit button, it can take quite some time ( this is the time for the agent to go through all the questions).
+        This space provides a basic setup and is intentionally sub-optimal to encourage you to develop your own, more robust solution. For instance for the delay process of the submit button, a solution could be to cache the answers and submit in a seperate action or even to answer the questions in async.
+        """
+    )
+
+    gr.LoginButton()
+
+    run_button = gr.Button("Run Evaluation & Submit All Answers")
+
+    status_output = gr.Textbox(label="Run Status / Submission Result", lines=5, interactive=False)
+    # Removed max_rows=10 from DataFrame constructor
+    results_table = gr.DataFrame(label="Questions and Agent Answers", wrap=True)
+
+    run_button.click(
+        fn=run_and_submit_all,
+        outputs=[status_output, results_table]
+    )
+
+if __name__ == "__main__":
+    print("\n" + "-"*30 + " App Starting " + "-"*30)
+    # Check for SPACE_HOST and SPACE_ID at startup for information
+    space_host_startup = os.getenv("SPACE_HOST")
+    space_id_startup = os.getenv("SPACE_ID") # Get SPACE_ID at startup
+
+    if space_host_startup:
+        print(f"✅ SPACE_HOST found: {space_host_startup}")
+        print(f"   Runtime URL should be: https://{space_host_startup}.hf.space")
+    else:
+        print("ℹ️  SPACE_HOST environment variable not found (running locally?).")
+
+    if space_id_startup: # Print repo URLs if SPACE_ID is found
+        print(f"✅ SPACE_ID found: {space_id_startup}")
+        print(f"   Repo URL: https://huggingface.co/spaces/{space_id_startup}")
+        print(f"   Repo Tree URL: https://huggingface.co/spaces/{space_id_startup}/tree/main")
+    else:
+        print("ℹ️  SPACE_ID environment variable not found (running locally?). Repo URL cannot be determined.")
+
+    print("-"*(60 + len(" App Starting ")) + "\n")
+
+    print("Launching Gradio Interface for Basic Agent Evaluation...")
+    demo.launch(debug=True, share=False)

_template_original/requirements.txt

@@ -0,0 +1,2 @@
+gradio
+requests

brainstorming_phase1_youtube.md

@@ -0,0 +1,345 @@
+# Phase 1 Brainstorming - YouTube Transcript Support
+
+**Date:** 2026-01-13
+**Status:** Discussion Phase
+**Goal:** Fix questions #3 and #5 (YouTube videos) → 40% score
+
+---
+
+## Question Analysis
+
+| Question | Task ID                                | Description                     | Expected Answer | Type          |
+| -------- | -------------------------------------- | ------------------------------- | --------------- | ------------- |
+| #3       | `a1e91b78-d3d8-4675-bb8d-62741b4b68a6` | YouTube video - bird species    | "3"             | Content-based |
+| #5       | (Teal'c quote)                         | YouTube video - character quote | "Extremely"     | Dialogue      |
+
+**Conclusion:** Both are content-based questions → transcript approach should work ✅
+
+---
+
+## Library Options
+
+### Option A: youtube-transcript-api ⭐ Recommended
+
+- **Pros:** Simple API, actively maintained, no video download needed, fast
+- **Cons:** May fail on videos without captions, regional restrictions
+- **Use case:** Start here for simplicity
+
+### Option B: yt-dlp + transcript extraction
+
+- **Pros:** More robust, can fall back to auto-generated captions
+- **Cons:** Heavier dependency, slower
+- **Use case:** Backup if Option A has high failure rate
+
+### Option C: Direct YouTube API
+
+- **Pros:** Most control
+- **Cons:** Requires API key, more complex
+- **Use case:** Probably overkill for this use case
+
+---
+
+## Frame Extraction: Corrected Analysis
+
+**Key insight:** Frame extraction itself is FAST. The "slow" parts are download + vision API processing.
+
+### Actual Timing Breakdown
+
+| Step                 | Time (10-min video) | Notes                                  |
+| -------------------- | ------------------- | -------------------------------------- |
+| **Download**         | 30s - 3 min         | Network I/O, one-time cost             |
+| **Frame extraction** | **5 - 20 sec**      | ffmpeg is I/O bound, very efficient ⚡ |
+| **Vision API calls** | 20s - 5 min         | Sequential: 600 frames × 2-5s each     |
+
+**Reality check:** You can extract 600 frames from a local 10-min video in under 15 seconds with ffmpeg. The "slow" part is vision model API calls, not the extraction.
+
+**Bottom line:** Frame extraction is cheap compute. Vision processing is expensive compute.
+
+### Comparison
+
+| Approach             | What's Fast        | What's Slow                                 | Total Time       |
+| -------------------- | ------------------ | ------------------------------------------- | ---------------- |
+| **Transcript**       | API call (1-3s)    | -                                           | **1-3 seconds**  |
+| **Frame Extraction** | Extraction (5-20s) | Download (30s-3min) + Vision API (20s-5min) | **1-10 minutes** |
+
+### Do Tools Matter?
+
+| Tool    | Speed (extraction only) | Verdict         |
+| ------- | ----------------------- | --------------- |
+| ffmpeg  | ⚡⚡⚡ Fastest (5-10s)  | Best choice     |
+| OpenCV  | ⚡⚡ Fast (10-20s)      | Standard choice |
+| moviepy | ⚡ Medium (20-40s)      | Python overhead |
+
+**For extraction alone:** Tools matter, but all are fast enough.
+
+### When Is Frame Extraction Worth It?
+
+**Only when:**
+
+- Question is purely visual (no audio/transcript available)
+- Visual information is NOT in video thumbnail/title/description
+- You have no other choice
+
+**Examples where necessary:**
+
+- "What color shirt is the person wearing at 2:35?"
+- "Count the number of cars visible in the video"
+- "Describe the visual style of the opening scene"
+
+**For GAIA #3 and #5:**
+
+- Both are content-based (species mentioned, dialogue)
+- Transcript is still fastest (1-3s vs 1-10 min total)
+- Frame extraction as fallback is viable (extraction is fast, but vision processing is slow)
+
+**Decision:** Transcript-first approach is correct. Frame extraction is viable fallback if transcript unavailable, but total time still 1-10 min due to download + vision API.
+
+---
+
+## Fallback Strategy
+
+**Scenario:** Video has no transcript available
+
+**Options:**
+
+1. **Return error** → LLM treats as system_error, skips question ✅ Simple
+2. **Download + extract frames** → Use vision tool (heavy, slow)
+3. **Return metadata** (title, description) → LLM infers from context
+4. **Chain approach:** Transcript → Metadata → Frames
+
+**Decision:** Start with audio-to-text fallback (Whisper on ZeroGPU) for higher success rate.
+
+---
+
+## Audio-to-Text Fallback: When No Transcript Available
+
+### The Hierarchy
+
+```
+YouTube URL
+    │
+    ├─ Has transcript? ✅ → Use youtube-transcript-api (instant, 1-3 sec)
+    │
+    └─ No transcript? ❌ → Download audio + Whisper (slower, but works)
+```
+
+### Whisper Cost Analysis
+
+| Option          | Cost       | Speed          | Verdict            |
+| --------------- | ---------- | -------------- | ------------------ |
+| OpenAI API      | $0.006/min | ⚡⚡⚡ Fastest | If budget OK       |
+| **Open Source** | **FREE**   | ⚡⚡ Fast      | ⭐ **Recommended** |
+| HuggingFace     | FREE       | ⚡⚡ Fast      | Good alternative   |
+
+**Decision:** Open-source Whisper (free, no API limits, works offline)
+
+---
+
+### HF Hardware: ZeroGPU ✅
+
+| Resource   | Available   | Whisper Requirements      | Verdict                           |
+| ---------- | ----------- | ------------------------- | --------------------------------- |
+| **CPU**    | 4 vCPUs     | 1+ cores                  | ✅ Plenty                         |
+| **Memory** | 16 GB RAM   | 1-10 GB (model-dependent) | ✅ Comfortable                    |
+| **Disk**   | 20 GB       | ~150 MB - 1.5 GB          | ✅ More than enough               |
+| **GPU**    | **ZeroGPU** | Optional (faster)         | ✅ **Available via subscription** |
+
+**ZeroGPU Benefits:**
+
+- ✅ Dynamic GPU allocation (5-10x faster than CPU)
+- ✅ Can use larger models (`small`, `medium`) for better accuracy
+- ✅ Still free (subscription benefit)
+
+### Performance: CPU vs ZeroGPU
+
+| Model    | On CPU    | On ZeroGPU    | Speedup |
+| -------- | --------- | ------------- | ------- |
+| `base`   | 30-60 sec | **5-10 sec**  | 5-10x   |
+| `small`  | 1-2 min   | **10-20 sec** | 5-10x   |
+| `medium` | 3-5 min   | **20-40 sec** | 5-10x   |
+
+**For 5-minute YouTube video on ZeroGPU:**
+
+- `base` model: ~5-10 seconds ⚡⚡⚡
+- `small` model: ~10-20 seconds ⚡⚡
+
+### Recommended Model for ZeroGPU
+
+| Model    | Size   | Accuracy  | Speed (ZeroGPU) | Recommendation         |
+| -------- | ------ | --------- | --------------- | ---------------------- |
+| `tiny`   | 39 MB  | Lower     | ~5 sec          | Fastest, less accurate |
+| `base`   | 74 MB  | Good      | ~10 sec         | Good balance           |
+| `small`  | 244 MB | Better    | ~20 sec         | ⭐ **Recommended**     |
+| `medium` | 769 MB | Very good | ~40 sec         | If accuracy critical   |
+
+**Choice:** `small` model - best accuracy/speed balance on ZeroGPU
+
+### Implementation: Audio-to-Text Fallback
+
+```python
+import whisper
+
+_MODEL = None  # Cache model globally
+
+def transcribe_audio(file_path: str) -> str:
+    """Transcribe audio file using Whisper (ZeroGPU)."""
+    global _MODEL
+    try:
+        if _MODEL is None:
+            # ZeroGPU auto-detects GPU, no manual device specification
+            _MODEL = whisper.load_model("small")
+
+        result = _MODEL.transcribe(file_path)
+        return result["text"]
+    except Exception as e:
+        return f"ERROR: Transcription failed: {e}"
+```
+
+---
+
+### Unified Architecture: Phase 1 + Phase 2
+
+```
+┌─────────────────────────────────────────────────────────┐
+│                   Audio Transcription                   │
+│              (transcribe_audio function)                │
+│                    Uses Whisper                         │
+│                  on ZeroGPU                             │
+└─────────────────────────────────────────────────────────┘
+                            ▲
+                            │
+        ┌───────────────────┴───────────────────┐
+        │                                       │
+   Phase 1                                Phase 2
+  YouTube URLs                            MP3 Files
+        │                                       │
+        │ 1. Try youtube-transcript-api        │
+        │ 2. Fallback: download audio only      │
+        │ 3. Call transcribe_audio()            │
+        │                                       │
+        └───────────────────┬───────────────────┘
+                            │
+                    Clean transcript
+                            │
+                            ▼
+                      LLM analyzes
+```
+
+**Benefits:**
+
+- Single audio processing codebase
+- `transcribe_audio()` works for both phases
+- Tested on HF ZeroGPU hardware
+- Higher success rate than skip-only approach
+
+---
+
+## Tool Design - LLM Integration
+
+**Current problem:** Vision tool tries to process YouTube URL → fails
+
+**Proposed tool description:**
+
+```
+"Extract transcript from YouTube video URL. Use when question asks about
+YouTube video content like: dialogue, speech, bird species identification,
+character quotes, or any content discussed in the video. Input: YouTube URL.
+Returns: Full transcript text or error message if transcript unavailable."
+```
+
+**Alternative: Special URL handling in `parse_file()`**
+
+- Detect YouTube URLs
+- Return tool suggestion: "This is a YouTube URL. Consider using youtube_transcript tool."
+
+---
+
+## Implementation Considerations
+
+### A. Video ID Extraction
+
+Handle various YouTube URL formats:
+
+- `youtube.com/watch?v=VIDEO_ID`
+- `youtu.be/VIDEO_ID`
+- `youtube.com/shorts/VIDEO_ID`
+
+### B. Language Handling
+
+- GAIA questions are English → likely English transcripts
+- Question: Should we auto-translate or let LLM handle?
+
+### C. Transcript Format
+
+- Raw JSON with timestamps vs clean text
+- LLM prefers clean text without timestamps
+- Question: Preserve timestamps for context?
+
+### D. Error Types
+
+- No transcript available
+- Video private/deleted
+- Rate limiting
+- Regional restriction
+
+---
+
+## Testing Strategy
+
+**Before full evaluation:**
+
+1. **Unit test** - Test on actual GAIA YouTube URLs
+2. **Manual test** - Run single question (#3) to verify LLM uses tool correctly
+3. **Integration test** - Verify transcript → answer pipeline
+
+**Question:** Do we have access to actual YouTube URLs for pre-testing?
+
+---
+
+## Edge Cases
+
+| Scenario                          | Handling                          |
+| --------------------------------- | --------------------------------- |
+| Multiple transcript languages     | Pick English or first available   |
+| Auto-generated transcript         | Accept (less accurate but usable) |
+| YouTube Shorts format             | Extract VIDEO_ID from shorts URL  |
+| Segmented transcript (by speaker) | Clean to plain text               |
+
+---
+
+## Recommendations
+
+1. **Start simple:** youtube-transcript-api with clear error messages
+2. **Fail gracefully:** If no transcript, return structured error → system_error=yes
+3. **Tool description:** Emphasize "YouTube video content" for LLM selection
+4. **Manual test first:** Verify on question #3 before full evaluation
+5. **Success metric:** Both questions correct → 40% score ✅ TARGET REACHED
+
+---
+
+## Open Questions
+
+- [ ] Implement fallback to frame extraction if transcript fails?
+- [ ] Add special YouTube URL detection in `parse_file()`?
+- [ ] Access to actual YouTube URLs for pre-testing?
+- [ ] Simple first vs comprehensive solution?
+
+---
+
+## Files to Create
+
+- `src/tools/youtube.py` - YouTube transcript extraction
+- Update `src/tools/__init__.py` - Register youtube_transcript tool
+- Update `requirements.txt` - Add youtube-transcript-api
+
+---
+
+## Next Steps (Discussion → Implementation)
+
+1. [ ] Confirm approach based on video processing research
+2. [ ] Install youtube-transcript-api
+3. [ ] Create youtube.py with error handling
+4. [ ] Add tool to TOOLS registry
+5. [ ] Manual test on question #3
+6. [ ] Full evaluation
+7. [ ] Verify 40% score (4/20 correct)

dev/dev_260102_13_stage2_tool_development.md

@@ -140,40 +140,47 @@ Successfully implemented 4 production-ready tools with comprehensive error handl
 
 **Deliverables:**
 
-1. **Web Search Tool** ([src/tools/web_search.py](../src/tools/web_search.py))
+1. **Web Search Tool** ([src/tools/web_search.py](./../src/tools/web_search.pyeb_search.py))
+
    - Tavily API integration (primary, free tier)
    - Exa API integration (fallback, paid)
-   - Automatic fallback if primary fails
-   - 10 passing tests in [test/test_web_search.py](../test/test_web_search.py)
+   - Automatic fallback if primary fails./../test/test_web_search.py
+   - 10 passing tests in [test/test_web_search.py](../../agentbee/test/test_web_search.py)
+     ./../src/tools/file_parser.py
+
+2. **File Parser Tool** ([src/tools/file_parser.py](../../agentbee/src/tools/file_parser.py))
 
-2. **File Parser Tool** ([src/tools/file_parser.py](../src/tools/file_parser.py))
    - PDF parsing (PyPDF2)
    - Excel parsing (openpyxl)
    - Word parsing (python-docx)
-   - Text/CSV parsing (built-in open)
+   - Text/CSV parsing (built-in open)./../test/test_file_parser.py
    - Generic `parse_file()` dispatcher
-   - 19 passing tests in [test/test_file_parser.py](../test/test_file_parser.py)
+   - 19 passing tests in [test/test_file_parser.py./../src/tools/calculator.py_file_parser.py)
+
+3. **Calculator Tool** ([src/tools/calculator.py](../../agentbee/src/tools/calculator.py))
 
-3. **Calculator Tool** ([src/tools/calculator.py](../src/tools/calculator.py))
    - Safe AST-based expression evaluation
-   - Whitelisted operations only (no code execution)
+   - Whitelisted operations only (no code execution./../test/test_calculator.py
    - Mathematical functions (sin, cos, sqrt, factorial, etc.)
-   - Security hardened (timeout, complexity limits)
-   - 41 passing tests in [test/test_calculator.py](../test/test_calculator.py)
+   - Security hardened (timeout, complexit./../src/tools/vision.py
+   - 41 passing tests in [test/test_calculator.py](../../agentbee/test/test_calculator.py)
 
-4. **Vision Tool** ([src/tools/vision.py](../src/tools/vision.py))
-   - Multimodal image analysis using LLMs
+4. **Vision Tool** ([src/tools/vision.py](../../agentbee/src/tools/vision.py))
+
+   - Multimodal image analysis using LLMs./../test/test_vision.py
    - Gemini 2.0 Flash (primary, free)
-   - Claude Sonnet 4.5 (fallback, paid)
+   - Claude Sonnet 4.5 (fallback, paid)./../src/tools/**init**.py
    - Image loading and base64 encoding
-   - 15 passing tests in [test/test_vision.py](../test/test_vision.py)
+   - 15 passing tests in [test/test_vision.py](../../agentbee/test/test_vision.py)
+
+5. **Tool Registry** ([src/tools/**init**.py](../../agentbee/src/tools/__init__.py))
+   ./../src/agent/graph.py
 
-5. **Tool Registry** ([src/tools/__init__.py](../src/tools/__init__.py))
    - Exports all 4 main tools: `search`, `parse_file`, `safe_eval`, `analyze_image`
    - TOOLS dict with metadata (description, parameters, category)
    - Ready for Stage 3 dynamic tool selection
 
-6. **StateGraph Integration** ([src/agent/graph.py](../src/agent/graph.py))
+6. **StateGraph Integration** ([src/agent/graph.py](../../agentbee/src/agent/graph.py))
    - Updated `execute_node` to load tool registry
    - Stage 2: Reports tool availability
    - Stage 3: Will add dynamic tool selection and execution

dev/dev_260102_14_stage3_core_logic.md

@@ -64,24 +64,28 @@ Successfully implemented Stage 3 with multi-provider LLM support. Agent now perf
 
 **Deliverables:**
 
-1. **LLM Client Module** ([src/agent/llm_client.py](../src/agent/llm_client.py))
+1. **LLM Client Module** ([src/agent/llm_client.py](./../src/agent/llm_client.pylm_client.py))
+
    - Gemini implementation: 3 functions (planning, tool selection, answer synthesis)
    - Claude implementation: 3 functions (same)
    - Unified API with automatic fallback
    - 624 lines of code
+     ./../src/agent/graph.py
+
+2. **Updated Agent Graph** ([src/agent/graph.py](../../agentbee/src/agent/graph.py))
 
-2. **Updated Agent Graph** ([src/agent/graph.py](../src/agent/graph.py))
    - plan_node: Calls `plan_question()` for LLM-based planning
    - execute_node: Calls `select_tools_with_function_calling()` + executes tools + collects evidence
    - answer_node: Calls `synthesize_answer()` for factoid generation
-   - Updated AgentState with new fields
+   - Updated AgentState with new fields./../test/test_llm_integration.py
+
+3. **LLM Integration Tests** ([test/test_llm_integration.py](../../agentbee/test/test_llm_integration.py))
 
-3. **LLM Integration Tests** ([test/test_llm_integration.py](../test/test_llm_integration.py))
    - 8 tests covering all 3 LLM functions
-   - Tests use mocked LLM responses (provider-agnostic)
+   - Tests use mocked LLM responses (provider-agno./../test/test_stage3_e2e.py
    - Full workflow test: planning → tool selection → answer synthesis
 
-4. **E2E Test Script** ([test/test_stage3_e2e.py](../test/test_stage3_e2e.py))
+4. **E2E Test Script** ([test/test_stage3_e2e.py](../../agentbee/test/test_stage3_e2e.py))
    - Manual test script for real API testing
    - Requires ANTHROPIC_API_KEY or GOOGLE_API_KEY
    - Tests simple math and factual questions

test/README.md

@@ -2,13 +2,14 @@
 
 **Test Files:**
 
-- [test_agent_basic.py](test_agent_basic.py) - Unit tests for Stage 1 foundation
+- [test_agent_basic.py](../../agentbee/test/test_agent_basic.py) - Unit tests for Stage 1 foundation
+
   - Agent initialization
   - Settings loading
   - Basic question processing
   - StateGraph structure validation
 
-- [test_stage1.py](test_stage1.py) - Stage 1 integration verification
+- [test_stage1.py](../../agentbee/test/test_stage1.py) - Stage 1 integration verification
   - Configuration validation
   - Agent initialization
   - End-to-end question processing