Updated agent

README.md

@@ -1,15 +1,266 @@
----
-title: Template Final Assignment
-emoji: 🕵🏻‍♂️
-colorFrom: indigo
-colorTo: indigo
-sdk: gradio
-sdk_version: 5.25.2
-app_file: app.py
-pinned: false
-hf_oauth: true
-# optional, default duration is 8 hours/480 minutes. Max duration is 30 days/43200 minutes.
-hf_oauth_expiration_minutes: 480
+# 🎯 GAIA Benchmark Agent - Course Final Project
+
+A comprehensive AI agent that demonstrates course learning while achieving 30%+ score on GAIA benchmark to earn your course certificate.
+
+## 🌟 What This Agent Demonstrates
+
+This project combines all major concepts from the course:
+
+### 📚 **Course Learning Applied**
+- **🔧 Tools Integration**: Multiple tool types working together
+- **📖 RAG Implementation**: Persona database with 5K diverse individuals using vector embeddings  
+- **🤖 Agent Workflows**: LlamaIndex agent orchestration
+- **🧠 LLM Integration**: Fallback options for accessibility
+- **📁 Modular Architecture**: Clean separation of concerns
+
+### 🎯 **GAIA Benchmark Optimized**
+- **🔍 Web Search**: For current information and facts
+- **🧮 Calculator**: For mathematical accuracy (critical for GAIA)
+- **📊 File Analysis**: For data processing questions
+- **💬 Conversational**: Natural language interaction
+
+## 🗂️ Project Structure
+
+```
+your-space/
+├── app.py              # Main application with Gradio interface
+├── tools.py            # All agent tools (web search, calculator, etc.)
+├── retriever.py        # RAG implementation with guest database
+├── requirements.txt    # Python dependencies
+└── README.md          # This file
+```
+
+### 📁 **File Explanations**
+
+**`app.py`** - Main Application
+- Gradio interface for GAIA evaluation
+- Agent initialization with error handling
+- Question processing and answer submission
+- Results display and certificate status
+
+**`tools.py`** - Agent Tools
+- **Web Search Tool**: DuckDuckGo integration for current info
+- **Calculator Tool**: Safe mathematical expression evaluation
+- **File Analysis Tool**: Process CSV, text, and data files
+- All tools have detailed documentation and error handling
+
+**`retriever.py`** - Advanced RAG System
+- Persona database with 5K diverse individuals from HuggingFace
+- Vector embeddings with ChromaDB for semantic search
+- IngestionPipeline for document processing
+- Demonstrates state-of-the-art RAG concepts
+
+## 🚀 Quick Setup Guide
+
+### 1. **Clone or Duplicate This Space**
+```bash
+# If cloning locally
+git clone https://huggingface.co/spaces/your-username/your-space
+cd your-space
+
+# Or duplicate this space to your HF account
+```
+
+### 2. **Set API Keys** ⚡ **CRITICAL STEP**
+
+In your HuggingFace Space:
+1. Go to **Settings** → **Repository secrets**
+2. Add **at least one** of these:
+
+**Option A: OpenAI (Recommended)**
+- Name: `OPENAI_API_KEY`
+- Value: `sk-...` (your OpenAI API key)
+- **Why**: Better performance on GAIA benchmark
+
+**Option B: HuggingFace (Free Alternative)**
+- Name: `HF_TOKEN`  
+- Value: `hf_...` (your HF token)
+- **Why**: Free alternative, works without OpenAI credits
+
+**Get API Keys:**
+- **OpenAI**: https://platform.openai.com/api-keys
+- **HuggingFace**: https://huggingface.co/settings/tokens
+
+### 3. **Ensure Public Space**
+- Your space must be **public** for leaderboard verification
+- Go to Settings → Change from Private to Public
+
+### 4. **Run Evaluation**
+1. Click the HuggingFace login button
+2. Click "Run GAIA Evaluation & Submit Results"  
+3. Wait 5-10 minutes for completion
+4. Check your score - need 30%+ to pass! 🏆
+
+## 🔧 Why Each Tool Matters for GAIA
+
+### 🌐 **Web Search Tool**
+```python
+# Example GAIA questions this helps with:
+"Who is the current president of France?"
+"What was Tesla's stock price yesterday?"
+"Recent developments in AI research"
+```
+**Why needed**: GAIA questions often require current information beyond LLM training data.
+
+### 🧮 **Calculator Tool**  
+```python
+# Example GAIA questions this helps with:
+"What is 15% of 847?"
+"Calculate the area of a circle with radius 23.7m"
+"If I invest $5000 at 3.2% annual interest for 7 years..."
+```
+**Why needed**: LLMs can make arithmetic errors. GAIA requires exact numerical accuracy.
+
+### 📊 **File Analysis Tool**
+```python
+# Example GAIA questions this helps with:
+"Analyze this CSV file and tell me the average..."
+"What is the most common value in column 3?"
+"Process this data file and extract..."
+```
+**Why needed**: Some GAIA questions include file attachments requiring analysis.
+
+### 📚 **Persona RAG Tool**
+```python
+# Example questions this demonstrates:
+"Find writers and authors"
+"Who are the scientists?"
+"People interested in travel"
+"Creative professionals at the event"
+```
+**Why included**: Demonstrates advanced RAG with 5K real personas, vector embeddings, and semantic search.
+
+## 📖 Course Concepts Demonstrated
+
+### 🔧 **Components** (From Course Unit 2)
+- **LLM Integration**: OpenAI + HuggingFace fallback
+- **Document Processing**: Text chunking and metadata
+- **Response Synthesis**: Clean answer formatting
+
+### 🛠️ **Tools** (From Course Unit 3)  
+- **FunctionTool Creation**: Multiple tool types
+- **Tool Descriptions**: Proper LLM guidance
+- **Error Handling**: Graceful tool failures
+
+### 🤖 **Agents** (From Course Unit 4)
+- **AgentWorkflow**: Multi-tool orchestration
+- **System Prompts**: GAIA-optimized instructions
+- **Async Processing**: Efficient question handling
+
+### 📖 **RAG Implementation** (From Course Unit 5)
+- **Dataset Integration**: 5K personas from HuggingFace
+- **Vector Embeddings**: Semantic search with BAAI/bge-small-en-v1.5
+- **ChromaDB Storage**: Persistent vector database
+- **Ingestion Pipeline**: Document processing and chunking
+
+### 🏗️ **Workflows** (From Course Unit 6)
+- **Event-Driven**: Tool selection and execution
+- **State Management**: Context preservation
+- **Error Recovery**: Robust failure handling
+
+## 🎓 Why This Approach Works for GAIA
+
+### ✅ **Accuracy First**
+- Calculator prevents math errors
+- Web search provides current facts
+- Low temperature LLM settings for consistency
+
+### ✅ **Comprehensive Coverage**
+- Factual questions → Web search
+- Mathematical questions → Calculator  
+- Data questions → File analysis
+- Knowledge questions → RAG system
+
+### ✅ **Robust Error Handling**
+- Graceful API failures
+- Tool availability checking
+- Fallback responses
+
+### ✅ **GAIA-Specific Optimizations**
+- Direct, concise answers
+- Exact match optimization
+- Minimal extra text
+
+## 🔧 Troubleshooting
+
+### ❌ **"No LLM available" Error**
+**Problem**: No API keys set
+**Solution**: Add `OPENAI_API_KEY` or `HF_TOKEN` to Space secrets
+
+### ❌ **Import Errors**
+**Problem**: Dependencies not installed
+**Solution**: Check requirements.txt is in root directory, restart Space
+
+### ❌ **Low GAIA Score**
+**Problem**: Agent giving wrong answers
+**Solutions**:
+- Check API key is working (OpenAI generally performs better)
+- Review agent logs for tool usage
+- Ensure web search and calculator are working
+
+### ❌ **"Could not submit" Error**
+**Problem**: Network or authentication issue
+**Solution**: 
+- Ensure logged in to HuggingFace
+- Check space is public
+- Try again (temporary network issues)
+
+### ❌ **Tools Not Working**
+**Problem**: Missing dependencies or API issues
+**Solution**: Check Space logs, verify all packages installed
+
+## 📊 Expected Performance
+
+### 🎯 **Target Scores**
+- **Minimum for Certificate**: 30%
+- **Good Performance**: 40-50%
+- **Excellent Performance**: 60%+
+
+### 📈 **Performance Factors**
+- **API Choice**: OpenAI typically scores higher than HuggingFace
+- **Tool Usage**: Questions requiring tools score better when tools work
+- **Answer Format**: Direct answers score better than verbose responses
+
+## 🚀 Getting Better Scores
+
+### 💡 **Optimization Tips**
+1. **Use OpenAI**: Generally more accurate than HuggingFace for GAIA
+2. **Check Tool Functionality**: Test web search and calculator work
+3. **Review Failed Questions**: Look at specific errors in results table
+4. **Adjust System Prompt**: Fine-tune for your specific weak areas
+
+### 🔄 **Iterative Improvement**
+1. Run evaluation and check results
+2. Identify patterns in failed questions
+3. Adjust tools or prompts accordingly
+4. Re-run evaluation
+
+## 🏆 Certificate Achievement
+
+**To earn your course certificate:**
+1. ✅ Score 30% or higher on GAIA evaluation
+2. ✅ Keep your space public for verification
+3. ✅ Submit through the official interface
+
+**When you pass:**
+- You'll see "✅ PASSED - Certificate Earned!" in results
+- Your score will appear on the student leaderboard
+- You can download your official certificate
+
+## 🤝 Getting Help
+
+**If you're stuck:**
+1. Check the troubleshooting section above
+2. Review Space logs for specific errors
+3. Test individual components (tools.py, retriever.py)
+4. Ask in the course Discord for community help
+
+## 🎉 Good Luck!
+
+This agent represents everything you've learned in the course. The modular design makes it easy to understand, debug, and improve. Focus on getting those API keys set up correctly, and you'll be well on your way to earning your certificate!
+
+**Remember**: The goal isn't just to pass the benchmark, but to demonstrate your understanding of modern AI agent development. This codebase serves as a portfolio piece showing your skills in RAG, tool integration, and agent orchestration.
+
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+*Built with ❤️ using LlamaIndex and course concepts*

app.py

@@ -1,196 +1,685 @@
+"""
+app.py - GAIA Benchmark Agent Application
+
+This is the main application file that brings together:
+1. Tools from tools.py (web search, calculator, file analysis)
+2. RAG system from retriever.py (guest database)
+3. LLM integration with fallback options
+4. Agent workflow for handling GAIA questions
+5. Gradio interface for submission to the GAIA benchmark
+
+The goal is to achieve 30%+ score on GAIA benchmark questions to earn the course certificate.
+
+How it works:
+1. User logs in with HuggingFace account
+2. System fetches GAIA questions from the evaluation API
+3. Our agent processes each question using its tools
+4. Answers are submitted and scored
+5. Results are displayed with pass/fail status
+
+Key design decisions:
+- Modular architecture: tools and retriever in separate files
+- Robust error handling: graceful failures with logging
+- API key flexibility: OpenAI (best) or HuggingFace (fallback)
+- GAIA-optimized: focused on accuracy over speed
+"""
+
 import os
 import gradio as gr
 import requests
-import inspect
 import pandas as pd
+import asyncio
+import logging
+from typing import List, Dict, Any, Optional
 
-# (Keep Constants as is)
-# --- Constants ---
+# Setup comprehensive logging
+logging.basicConfig(
+    level=logging.INFO,
+    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
+)
+logger = logging.getLogger(__name__)
+
+# ============================================================================
+# CONSTANTS AND CONFIGURATION
+# ============================================================================
+
+# GAIA evaluation API endpoint
 DEFAULT_API_URL = "https://agents-course-unit4-scoring.hf.space"
 
-# --- Basic Agent Definition ---
-# ----- THIS IS WERE YOU CAN BUILD WHAT YOU WANT ------
-class BasicAgent:
-    def __init__(self):
-        print("BasicAgent initialized.")
-    def __call__(self, question: str) -> str:
-        print(f"Agent received question (first 50 chars): {question[:50]}...")
-        fixed_answer = "This is a default answer."
-        print(f"Agent returning fixed answer: {fixed_answer}")
-        return fixed_answer
+# Required score to pass the course
+PASSING_SCORE = 30  # 30% minimum to earn certificate
+
+# ============================================================================
+# LLM SETUP WITH FALLBACK OPTIONS
+# ============================================================================
 
-def run_and_submit_all( profile: gr.OAuthProfile | None):
+def create_llm():
     """
-    Fetches all questions, runs the BasicAgent on them, submits all answers,
-    and displays the results.
+    Create an LLM (Large Language Model) with fallback options.
+    
+    Priority order:
+    1. OpenAI GPT-4 (best performance for GAIA)
+    2. HuggingFace Qwen model (free alternative)
+    
+    Why this order:
+    - OpenAI models generally perform better on GAIA benchmark
+    - HuggingFace provides free alternative for those without OpenAI credits
+    - Fallback ensures the agent works regardless of available keys
+    
+    API Keys Setup:
+    - Go to your HuggingFace Space settings
+    - Add "Repository secrets"
+    - Set OPENAI_API_KEY (recommended) and/or HF_TOKEN
+    
+    Returns:
+        LLM: Configured language model ready for use
+        
+    Raises:
+        RuntimeError: If no API keys are available
     """
-    # --- Determine HF Space Runtime URL and Repo URL ---
-    space_id = os.getenv("SPACE_ID") # Get the SPACE_ID for sending link to the code
-
-    if profile:
-        username= f"{profile.username}"
-        print(f"User logged in: {username}")
+    logger.info("Initializing LLM with fallback options...")
+    
+    # Try OpenAI first (recommended for GAIA performance)
+    openai_key = os.getenv("OPENAI_API_KEY")
+    if openai_key:
+        try:
+            from llama_index.llms.openai import OpenAI
+            
+            llm = OpenAI(
+                api_key=openai_key,
+                model="gpt-4o-mini",  # Good balance of cost and performance
+                max_tokens=1024,      # Reasonable limit for GAIA answers
+                temperature=0.1       # Low temperature for more consistent, factual responses
+            )
+            
+            logger.info("✅ Successfully initialized OpenAI LLM")
+            return llm
+            
+        except ImportError:
+            logger.warning("❌ OpenAI library not available, trying HuggingFace...")
+        except Exception as e:
+            logger.warning(f"❌ OpenAI initialization failed: {e}, trying HuggingFace...")
     else:
-        print("User not logged in.")
-        return "Please Login to Hugging Face with the button.", None
+        logger.info("ℹ️ No OPENAI_API_KEY found, trying HuggingFace...")
+    
+    # Fallback to HuggingFace
+    hf_token = os.getenv("HF_TOKEN")
+    if hf_token:
+        try:
+            from llama_index.llms.huggingface_api import HuggingFaceInferenceAPI
+            
+            llm = HuggingFaceInferenceAPI(
+                model_name="Qwen/Qwen2.5-Coder-32B-Instruct",  # Good open-source model
+                token=hf_token,
+                max_new_tokens=512,    # Limit for response length
+                temperature=0.1,       # Low temperature for consistency
+                context_window=8192    # Context window size
+            )
+            
+            logger.info("✅ Successfully initialized HuggingFace LLM")
+            return llm
+            
+        except ImportError:
+            logger.error("❌ HuggingFace library not available")
+        except Exception as e:
+            logger.error(f"❌ HuggingFace initialization failed: {e}")
+    else:
+        logger.info("ℹ️ No HF_TOKEN found")
+    
+    # If we get here, no LLM could be initialized
+    error_msg = (
+        "No LLM could be initialized. Please set either:\n"
+        "- OPENAI_API_KEY (recommended for better GAIA performance)\n"
+        "- HF_TOKEN (free alternative)\n"
+        "In your HuggingFace Space settings → Repository secrets"
+    )
+    logger.error(error_msg)
+    raise RuntimeError(error_msg)
+
+
+# ============================================================================
+# GAIA AGENT CLASS - Main Agent Implementation
+# ============================================================================
+
+class GAIAAgent:
+    """
+    GAIA Benchmark Agent that combines course learning with benchmark capabilities.
+    
+    This agent demonstrates:
+    1. Multi-tool usage (web search, calculator, file analysis)
+    2. RAG implementation (guest database from course)
+    3. LLM integration with robust error handling
+    4. GAIA-optimized prompting for accurate answers
+    
+    The agent is designed to handle various types of GAIA questions:
+    - Factual questions requiring web search
+    - Mathematical problems requiring calculations
+    - Data analysis questions requiring file processing
+    - Questions about the guest database (demonstrating RAG)
+    """
+    
+    def __init__(self):
+        """
+        Initialize the GAIA agent with LLM and tools.
+        
+        This sets up:
+        1. The language model (with fallback options)
+        2. All available tools (web search, calculator, etc.)
+        3. The agent workflow that orchestrates everything
+        """
+        logger.info("🚀 Initializing GAIA Agent...")
+        
+        # Step 1: Initialize the LLM
+        try:
+            self.llm = create_llm()
+            logger.info("✅ LLM initialized successfully")
+        except Exception as e:
+            logger.error(f"❌ Failed to initialize LLM: {e}")
+            raise
+        
+        # Step 2: Import and create tools
+        tools = []
+        
+        # Import tools from our tools.py file
+        try:
+            from tools import create_all_tools
+            tool_list = create_all_tools()
+            tools.extend(tool_list)
+            logger.info(f"✅ Loaded {len(tool_list)} tools from tools.py")
+        except ImportError as e:
+            logger.error(f"❌ Could not import tools.py: {e}")
+        except Exception as e:
+            logger.warning(f"⚠️ Error loading tools from tools.py: {e}")
+        
+        # Import RAG tool from our retriever.py file
+        try:
+            from retriever import create_persona_tool
+            persona_tool = create_persona_tool()
+            if persona_tool:
+                tools.append(persona_tool)
+                logger.info("✅ Loaded persona RAG tool from retriever.py")
+        except ImportError as e:
+            logger.error(f"❌ Could not import retriever.py: {e}")
+        except Exception as e:
+            logger.warning(f"⚠️ Error loading RAG tool from retriever.py: {e}")
+        
+        # Check if we have any tools
+        if not tools:
+            error_msg = "❌ No tools available! Check tools.py and retriever.py"
+            logger.error(error_msg)
+            raise RuntimeError(error_msg)
+        
+        logger.info(f"✅ Total tools available: {len(tools)}")
+        for tool in tools:
+            logger.info(f"   - {tool.metadata.name}: {tool.metadata.description[:50]}...")
+        
+        # Step 3: Create the agent workflow
+        try:
+            from llama_index.core.agent.workflow import AgentWorkflow
+            
+            # Create the agent with a GAIA-optimized system prompt
+            self.agent = AgentWorkflow.from_tools_or_functions(
+                tools_or_functions=tools,
+                llm=self.llm,
+                system_prompt=self._create_system_prompt()
+            )
+            
+            logger.info("✅ Agent workflow created successfully")
+            
+        except ImportError as e:
+            error_msg = f"❌ Could not import AgentWorkflow: {e}"
+            logger.error(error_msg)
+            raise RuntimeError(error_msg)
+        except Exception as e:
+            error_msg = f"❌ Failed to create agent workflow: {e}"
+            logger.error(error_msg)
+            raise RuntimeError(error_msg)
+        
+        logger.info("🎉 GAIA Agent initialization complete!")
+    
+    def _create_system_prompt(self) -> str:
+        """
+        Create a system prompt optimized for GAIA benchmark performance.
+        
+        The prompt is designed to:
+        1. Encourage accuracy over creativity
+        2. Guide proper tool usage
+        3. Ensure concise, direct answers
+        4. Handle various question types
+        
+        Returns:
+            str: Optimized system prompt for GAIA questions
+        """
+        return """You are a helpful AI assistant specialized in answering questions accurately and concisely.
+
+            IMPORTANT - GAIA BENCHMARK GUIDELINES:
+            - Provide direct, factual answers without extra explanations
+            - Use your tools when you need specific information or calculations
+            - Be precise and accurate - exact matches are required for scoring
+            - If you're not certain about an answer, use available tools to verify
+
+            AVAILABLE TOOLS AND WHEN TO USE THEM:
+            1. web_search: Use for current information, recent events, facts not in your training data
+            2. calculator: Use for ANY mathematical calculations to ensure accuracy
+            3. file_analyzer: Use when questions involve analyzing data files or documents
+            4. persona_database: Use for questions about people, characteristics, interests, professions
+            (Database contains 5000 diverse personas with various backgrounds and interests)
+
+            RESPONSE GUIDELINES:
+            - Give direct answers without phrases like "Based on my search..." or "According to..."
+            - For numerical answers, provide just the number or value
+            - For factual questions, provide just the fact
+            - For yes/no questions, answer yes or no clearly
+            - Always use tools for calculations rather than doing math in your head
 
+            EXAMPLES:
+            Question: "What is 15% of 847?"
+            Good: Use calculator tool, then respond with just the number
+            Bad: Try to calculate mentally and risk errors
+
+            Question: "Who is the current president of France?"
+            Good: Use web search to get current information
+            Bad: Guess based on training data that might be outdated
+
+            Remember: Accuracy is more important than speed. Use your tools to ensure correct answers."""
+    
+    def __call__(self, question: str) -> str:
+        """
+        Process a GAIA question and return an answer.
+        
+        This is the main method that the evaluation system calls.
+        It handles the entire question-answering pipeline:
+        1. Logs the incoming question
+        2. Runs the agent workflow asynchronously  
+        3. Extracts and cleans the response
+        4. Returns a properly formatted answer
+        
+        Args:
+            question (str): The GAIA question to answer
+            
+        Returns:
+            str: The agent's answer to the question
+        """
+        logger.info(f"📝 Processing GAIA question: {question[:100]}...")
+        
+        try:
+            # Run the agent asynchronously
+            # GAIA questions can be complex and may require multiple tool calls
+            loop = asyncio.new_event_loop()
+            asyncio.set_event_loop(loop)
+            
+            try:
+                # Execute the agent workflow
+                result = loop.run_until_complete(
+                    self.agent.run(user_msg=question)
+                )
+                
+                # Extract the response from the result object
+                answer = self._extract_response(result)
+                
+                # Clean and format the answer for GAIA submission
+                cleaned_answer = self._clean_answer(answer)
+                
+                logger.info(f"✅ Generated answer: {cleaned_answer[:100]}...")
+                return cleaned_answer
+                
+            finally:
+                # Always close the event loop to prevent memory leaks
+                loop.close()
+                
+        except Exception as e:
+            # If anything goes wrong, return a helpful error message
+            error_msg = f"I encountered an error processing this question: {str(e)}"
+            logger.error(f"❌ Error processing question: {e}")
+            return error_msg
+    
+    def _extract_response(self, result: Any) -> str:
+        """
+        Extract the text response from the agent workflow result.
+        
+        Agent workflows can return different types of objects.
+        This method handles various result formats robustly.
+        
+        Args:
+            result: The result object from the agent workflow
+            
+        Returns:
+            str: Extracted response text
+        """
+        # Try different ways to extract the response
+        if hasattr(result, 'response'):
+            return str(result.response)
+        elif hasattr(result, 'content'):
+            return str(result.content)
+        elif hasattr(result, 'message'):
+            if hasattr(result.message, 'content'):
+                return str(result.message.content)
+            else:
+                return str(result.message)
+        else:
+            # Fallback: convert whatever we got to string
+            return str(result)
+    
+    def _clean_answer(self, answer: str) -> str:
+        """
+        Clean and format the answer for GAIA submission.
+        
+        GAIA requires exact matches, so we need to:
+        1. Remove common prefixes that agents add
+        2. Strip whitespace
+        3. Ensure clean, direct responses
+        
+        Args:
+            answer (str): Raw answer from the agent
+            
+        Returns:
+            str: Cleaned answer ready for submission
+        """
+        # Remove common agent response prefixes
+        prefixes_to_remove = [
+            "assistant:",
+            "Assistant:",
+            "Based on my search,",
+            "According to the search results,",
+            "The answer is:",
+            "Answer:"
+        ]
+        
+        cleaned = answer.strip()
+        
+        for prefix in prefixes_to_remove:
+            if cleaned.startswith(prefix):
+                cleaned = cleaned[len(prefix):].strip()
+        
+        return cleaned
+
+
+# ============================================================================
+# EVALUATION AND SUBMISSION LOGIC
+# ============================================================================
+
+def run_and_submit_all(profile: gr.OAuthProfile | None) -> tuple[str, pd.DataFrame]:
+    """
+    Main function that handles the entire GAIA evaluation process.
+    
+    This function:
+    1. Validates user authentication
+    2. Fetches questions from GAIA API
+    3. Runs the agent on all questions
+    4. Submits answers for scoring
+    5. Returns results and status
+    
+    Args:
+        profile: Gradio OAuth profile (None if not logged in)
+        
+    Returns:
+        tuple: (status_message, results_dataframe)
+    """
+    # Step 1: Check authentication
+    if not profile:
+        logger.warning("❌ User not logged in")
+        return "Please log in to HuggingFace using the button above.", None
+    
+    username = profile.username
+    logger.info(f"👤 User logged in: {username}")
+    
+    # Step 2: Get space information for code link
+    space_id = os.getenv("SPACE_ID")
+    agent_code = f"https://huggingface.co/spaces/{space_id}/tree/main" if space_id else "No space ID available"
+    
+    # Step 3: Set up API endpoints
     api_url = DEFAULT_API_URL
     questions_url = f"{api_url}/questions"
     submit_url = f"{api_url}/submit"
-
-    # 1. Instantiate Agent ( modify this part to create your agent)
+    
+    # Step 4: Initialize the agent
+    logger.info("🤖 Initializing GAIA Agent...")
     try:
-        agent = BasicAgent()
+        agent = GAIAAgent()
+        logger.info("✅ GAIA Agent ready for evaluation")
     except Exception as e:
-        print(f"Error instantiating agent: {e}")
-        return f"Error initializing agent: {e}", None
-    # In the case of an app running as a hugging Face space, this link points toward your codebase ( usefull for others so please keep it public)
-    agent_code = f"https://huggingface.co/spaces/{space_id}/tree/main"
-    print(agent_code)
-
-    # 2. Fetch Questions
-    print(f"Fetching questions from: {questions_url}")
+        error_msg = f"❌ Failed to initialize agent: {str(e)}"
+        logger.error(error_msg)
+        return error_msg, None
+    
+    # Step 5: Fetch GAIA questions
+    logger.info(f"📥 Fetching questions from: {questions_url}")
     try:
         response = requests.get(questions_url, timeout=15)
         response.raise_for_status()
         questions_data = response.json()
+        
         if not questions_data:
-             print("Fetched questions list is empty.")
-             return "Fetched questions list is empty or invalid format.", None
-        print(f"Fetched {len(questions_data)} questions.")
+            return "❌ No questions received from GAIA API", None
+        
+        logger.info(f"✅ Fetched {len(questions_data)} GAIA questions")
+        
     except requests.exceptions.RequestException as e:
-        print(f"Error fetching questions: {e}")
-        return f"Error fetching questions: {e}", None
-    except requests.exceptions.JSONDecodeError as e:
-         print(f"Error decoding JSON response from questions endpoint: {e}")
-         print(f"Response text: {response.text[:500]}")
-         return f"Error decoding server response for questions: {e}", None
+        error_msg = f"❌ Network error fetching questions: {str(e)}"
+        logger.error(error_msg)
+        return error_msg, None
     except Exception as e:
-        print(f"An unexpected error occurred fetching questions: {e}")
-        return f"An unexpected error occurred fetching questions: {e}", None
-
-    # 3. Run your Agent
+        error_msg = f"❌ Error processing questions: {str(e)}"
+        logger.error(error_msg)
+        return error_msg, None
+    
+    # Step 6: Process all questions
+    logger.info(f"🧠 Running agent on {len(questions_data)} questions...")
     results_log = []
     answers_payload = []
-    print(f"Running agent on {len(questions_data)} questions...")
-    for item in questions_data:
+    
+    for i, item in enumerate(questions_data, 1):
         task_id = item.get("task_id")
         question_text = item.get("question")
+        
         if not task_id or question_text is None:
-            print(f"Skipping item with missing task_id or question: {item}")
+            logger.warning(f"⚠️ Skipping invalid question item: {item}")
             continue
+        
+        logger.info(f"📝 Processing question {i}/{len(questions_data)}: {task_id}")
+        
         try:
+            # Run the agent on this question
             submitted_answer = agent(question_text)
-            answers_payload.append({"task_id": task_id, "submitted_answer": submitted_answer})
-            results_log.append({"Task ID": task_id, "Question": question_text, "Submitted Answer": submitted_answer})
+            
+            # Store for submission
+            answers_payload.append({
+                "task_id": task_id,
+                "submitted_answer": submitted_answer
+            })
+            
+            # Store for display (truncated for readability)
+            results_log.append({
+                "Task ID": task_id,
+                "Question": question_text[:100] + "..." if len(question_text) > 100 else question_text,
+                "Answer": submitted_answer[:150] + "..." if len(submitted_answer) > 150 else submitted_answer
+            })
+            
+            logger.info(f"✅ Question {i} completed")
+            
         except Exception as e:
-             print(f"Error running agent on task {task_id}: {e}")
-             results_log.append({"Task ID": task_id, "Question": question_text, "Submitted Answer": f"AGENT ERROR: {e}"})
-
+            error_answer = f"ERROR: {str(e)}"
+            logger.error(f"❌ Error on question {i}: {e}")
+            
+            answers_payload.append({
+                "task_id": task_id,
+                "submitted_answer": error_answer
+            })
+            
+            results_log.append({
+                "Task ID": task_id,
+                "Question": question_text[:100] + "..." if len(question_text) > 100 else question_text,
+                "Answer": error_answer
+            })
+    
     if not answers_payload:
-        print("Agent did not produce any answers to submit.")
-        return "Agent did not produce any answers to submit.", pd.DataFrame(results_log)
-
-    # 4. Prepare Submission 
-    submission_data = {"username": username.strip(), "agent_code": agent_code, "answers": answers_payload}
-    status_update = f"Agent finished. Submitting {len(answers_payload)} answers for user '{username}'..."
-    print(status_update)
-
-    # 5. Submit
-    print(f"Submitting {len(answers_payload)} answers to: {submit_url}")
+        return "❌ No answers generated for submission", pd.DataFrame(results_log)
+    
+    # Step 7: Submit answers to GAIA API
+    logger.info(f"📤 Submitting {len(answers_payload)} answers...")
+    submission_data = {
+        "username": username.strip(),
+        "agent_code": agent_code,
+        "answers": answers_payload
+    }
+    
     try:
         response = requests.post(submit_url, json=submission_data, timeout=60)
         response.raise_for_status()
         result_data = response.json()
+        
+        # Extract results
+        score = result_data.get('score', 0)
+        correct_count = result_data.get('correct_count', 0)
+        total_attempted = result_data.get('total_attempted', len(answers_payload))
+        
+        # Determine pass/fail status
+        passed = score >= PASSING_SCORE
+        status_emoji = "🎉" if passed else "📊"
+        
+        # Create status message
         final_status = (
-            f"Submission Successful!\n"
-            f"User: {result_data.get('username')}\n"
-            f"Overall Score: {result_data.get('score', 'N/A')}% "
-            f"({result_data.get('correct_count', '?')}/{result_data.get('total_attempted', '?')} correct)\n"
-            f"Message: {result_data.get('message', 'No message received.')}"
+            f"{status_emoji} GAIA Evaluation Results\n"
+            f"User: {username}\n"
+            f"Score: {score}% ({correct_count}/{total_attempted} correct)\n"
+            f"Required: {PASSING_SCORE}% to pass\n"
+            f"Status: {'✅ PASSED - Certificate Earned!' if passed else '❌ Not passed - Try again!'}\n"
+            f"Message: {result_data.get('message', 'Evaluation completed')}"
         )
-        print("Submission successful.")
-        results_df = pd.DataFrame(results_log)
-        return final_status, results_df
-    except requests.exceptions.HTTPError as e:
-        error_detail = f"Server responded with status {e.response.status_code}."
-        try:
-            error_json = e.response.json()
-            error_detail += f" Detail: {error_json.get('detail', e.response.text)}"
-        except requests.exceptions.JSONDecodeError:
-            error_detail += f" Response: {e.response.text[:500]}"
-        status_message = f"Submission Failed: {error_detail}"
-        print(status_message)
-        results_df = pd.DataFrame(results_log)
-        return status_message, results_df
-    except requests.exceptions.Timeout:
-        status_message = "Submission Failed: The request timed out."
-        print(status_message)
-        results_df = pd.DataFrame(results_log)
-        return status_message, results_df
+        
+        logger.info(f"✅ Submission successful - Score: {score}%")
+        return final_status, pd.DataFrame(results_log)
+        
     except requests.exceptions.RequestException as e:
-        status_message = f"Submission Failed: Network error - {e}"
-        print(status_message)
-        results_df = pd.DataFrame(results_log)
-        return status_message, results_df
+        error_msg = f"❌ Submission failed: {str(e)}"
+        logger.error(error_msg)
+        return error_msg, pd.DataFrame(results_log)
     except Exception as e:
-        status_message = f"An unexpected error occurred during submission: {e}"
-        print(status_message)
-        results_df = pd.DataFrame(results_log)
-        return status_message, results_df
-
-
-# --- Build Gradio Interface using Blocks ---
-with gr.Blocks() as demo:
-    gr.Markdown("# Basic Agent Evaluation Runner")
-    gr.Markdown(
-        """
-        **Instructions:**
+        error_msg = f"❌ Unexpected error during submission: {str(e)}"
+        logger.error(error_msg)
+        return error_msg, pd.DataFrame(results_log)
 
-        1.  Please clone this space, then modify the code to define your agent's logic, the tools, the necessary packages, etc ...
-        2.  Log in to your Hugging Face account using the button below. This uses your HF username for submission.
-        3.  Click 'Run Evaluation & Submit All Answers' to fetch questions, run your agent, submit answers, and see the score.
 
-        ---
-        **Disclaimers:**
-        Once clicking on the "submit button, it can take quite some time ( this is the time for the agent to go through all the questions).
-        This space provides a basic setup and is intentionally sub-optimal to encourage you to develop your own, more robust solution. For instance for the delay process of the submit button, a solution could be to cache the answers and submit in a seperate action or even to answer the questions in async.
-        """
-    )
+# ============================================================================
+# GRADIO INTERFACE
+# ============================================================================
 
+# Create the Gradio interface
+with gr.Blocks(title="GAIA Benchmark Agent") as demo:
+    # Header and instructions
+    gr.Markdown("# 🎯 GAIA Benchmark Agent - Course Final Project")
+    
+    gr.Markdown("""
+    ## 🚀 Welcome to Your Final Challenge!
+    
+    This agent combines everything you've learned in the course:
+    - **🔧 Multi-Tool Integration**: Web search, calculator, file analysis
+    - **📚 RAG Implementation**: Persona database with 5K diverse individuals
+    - **🤖 Agent Workflows**: LlamaIndex agent orchestration
+    - **🎯 GAIA Optimization**: Designed for benchmark performance
+    
+    ### 📋 Setup Checklist:
+    1. **🔑 API Keys**: Set `OPENAI_API_KEY` or `HF_TOKEN` in Space secrets
+    2. **🔓 Public Space**: Keep your space public for verification
+    3. **👤 Login**: Use the HuggingFace login button below
+    4. **▶️ Run**: Click the evaluation button and wait for results
+    
+    ### 🏆 Goal: Score 30%+ to earn your certificate!
+    
+    ---
+    """)
+    
+    # Login section
+    gr.Markdown("### Step 1: Login to HuggingFace")
     gr.LoginButton()
-
-    run_button = gr.Button("Run Evaluation & Submit All Answers")
-
-    status_output = gr.Textbox(label="Run Status / Submission Result", lines=5, interactive=False)
-    # Removed max_rows=10 from DataFrame constructor
-    results_table = gr.DataFrame(label="Questions and Agent Answers", wrap=True)
-
+    
+    # Evaluation section
+    gr.Markdown("### Step 2: Run GAIA Evaluation")
+    gr.Markdown("⚠️ **Note**: This may take 5-10 minutes to complete all questions. Please be patient!")
+    
+    run_button = gr.Button(
+        "🚀 Run GAIA Evaluation & Submit Results",
+        variant="primary",
+        size="lg"
+    )
+    
+    # Results section
+    gr.Markdown("### Step 3: View Results")
+    
+    status_output = gr.Textbox(
+        label="📊 Evaluation Status & Results",
+        lines=8,
+        interactive=False,
+        placeholder="Results will appear here after evaluation..."
+    )
+    
+    results_table = gr.DataFrame(
+        label="📝 Question-by-Question Results",
+        wrap=True
+    )
+    
+    # Wire up the interface
     run_button.click(
         fn=run_and_submit_all,
         outputs=[status_output, results_table]
     )
+    
+    # Footer
+    gr.Markdown("""
+    ---
+    ### 🔧 Troubleshooting:
+    - **No API Key Error**: Add `OPENAI_API_KEY` or `HF_TOKEN` to your Space secrets
+    - **Import Errors**: Check that all dependencies are installed
+    - **Low Score**: GAIA requires exact answers - the agent uses tools for accuracy
+    
+    ### 🏅 Good luck earning your certificate!
+    """)
 
-if __name__ == "__main__":
-    print("\n" + "-"*30 + " App Starting " + "-"*30)
-    # Check for SPACE_HOST and SPACE_ID at startup for information
-    space_host_startup = os.getenv("SPACE_HOST")
-    space_id_startup = os.getenv("SPACE_ID") # Get SPACE_ID at startup
-
-    if space_host_startup:
-        print(f"✅ SPACE_HOST found: {space_host_startup}")
-        print(f"   Runtime URL should be: https://{space_host_startup}.hf.space")
-    else:
-        print("ℹ️  SPACE_HOST environment variable not found (running locally?).")
-
-    if space_id_startup: # Print repo URLs if SPACE_ID is found
-        print(f"✅ SPACE_ID found: {space_id_startup}")
-        print(f"   Repo URL: https://huggingface.co/spaces/{space_id_startup}")
-        print(f"   Repo Tree URL: https://huggingface.co/spaces/{space_id_startup}/tree/main")
-    else:
-        print("ℹ️  SPACE_ID environment variable not found (running locally?). Repo URL cannot be determined.")
 
-    print("-"*(60 + len(" App Starting ")) + "\n")
+# ============================================================================
+# MAIN EXECUTION
+# ============================================================================
 
-    print("Launching Gradio Interface for Basic Agent Evaluation...")
-    demo.launch(debug=True, share=False)
+if __name__ == "__main__":
+    print("\n" + "="*60)
+    print("🎯 GAIA BENCHMARK AGENT - Course Final Project")
+    print("="*60)
+    
+    # Check environment setup
+    print("\n🔍 Environment Check:")
+    
+    space_host = os.getenv("SPACE_HOST")
+    space_id = os.getenv("SPACE_ID")
+    openai_key = os.getenv("OPENAI_API_KEY")
+    hf_token = os.getenv("HF_TOKEN")
+    
+    if space_host:
+        print(f"✅ SPACE_HOST: {space_host}")
+    if space_id:
+        print(f"✅ SPACE_ID: {space_id}")
+    if openai_key:
+        print("✅ OPENAI_API_KEY: Set")
+    if hf_token:
+        print("✅ HF_TOKEN: Set")
+    
+    if not openai_key and not hf_token:
+        print("⚠️  WARNING: No API keys found!")
+        print("   Please set OPENAI_API_KEY or HF_TOKEN in Space secrets")
+    
+    print(f"\n🎯 Target Score: {PASSING_SCORE}% (to earn certificate)")
+    print("🚀 Agent Features:")
+    print("   - Web Search (DuckDuckGo)")
+    print("   - Calculator (Math operations)")
+    print("   - Guest Database RAG (Course demo)")
+    print("   - File Analysis (Data processing)")
+    
+    print("\n" + "="*60)
+    print("🌐 Launching Gradio Interface...")
+    print("="*60 + "\n")
+    
+    # Launch the Gradio app
+    demo.launch(
+        debug=True,
+        share=False,
+        show_error=True
+    )

requirements.txt

@@ -1,2 +1,157 @@
-gradio
-requests
+# ============================================================================
+# GAIA Benchmark Agent - Requirements
+# ============================================================================
+# This file lists all the Python packages needed for the GAIA agent to work.
+# Each section explains what the packages are used for.
+
+# ============================================================================
+# CORE INTERFACE AND API DEPENDENCIES
+# ============================================================================
+# These are essential for the app to run and communicate with GAIA API
+
+gradio>=4.0.0
+# Web interface for the agent - provides the UI where users interact
+# Includes login functionality and result display
+
+requests>=2.28.0
+# For HTTP requests to the GAIA evaluation API
+# Used to fetch questions and submit answers
+
+pandas>=1.5.0
+# Data manipulation and display of results in tables
+# Used to show question-answer pairs in a nice format
+
+# ============================================================================
+# LLAMAINDEX CORE - The Foundation
+# ============================================================================
+# LlamaIndex is the main framework from the course
+
+llama-index-core>=0.10.0
+# Core LlamaIndex functionality - documents, nodes, retrievers, etc.
+# This is the foundation that everything else builds on
+
+# ============================================================================
+# LLM (Language Model) INTEGRATIONS
+# ============================================================================
+# These allow us to use different LLMs with fallback options
+
+llama-index-llms-openai
+# OpenAI integration (GPT-4, GPT-3.5) - recommended for best GAIA performance
+# Requires OPENAI_API_KEY in your Space secrets
+
+llama-index-llms-huggingface-api
+# HuggingFace Inference API integration - free alternative
+# Uses models like Qwen/Qwen2.5-Coder-32B-Instruct
+# Requires HF_TOKEN in your Space secrets
+
+# ============================================================================
+# AGENT WORKFLOW SYSTEM
+# ============================================================================
+# This enables the agent functionality from the course
+
+llama-index-agent-workflow
+# Agent workflow system - allows creating agents that can use multiple tools
+# This is what orchestrates the web search, calculator, and RAG tools
+
+# ============================================================================
+# RETRIEVAL SYSTEMS (RAG) - Enhanced with Vector Embeddings
+# ============================================================================
+# These are for the advanced RAG (Retrieval-Augmented Generation) functionality
+
+llama-index-retrievers-bm25
+# BM25 retriever for keyword-based search (still useful as fallback)
+# Great for finding exact matches and proper nouns
+
+llama-index-embeddings-huggingface
+# HuggingFace embedding models for semantic search
+# Converts text to vectors that capture meaning and context
+# Used with BAAI/bge-small-en-v1.5 model
+
+llama-index-vector-stores-chroma
+# ChromaDB vector store integration
+# Provides persistent storage for vector embeddings
+# Fast similarity search for semantic retrieval
+
+chromadb>=0.4.0
+# ChromaDB database for vector storage
+# Self-contained vector database with no external dependencies
+# Stores embeddings locally for fast retrieval
+
+datasets>=2.0.0
+# HuggingFace datasets library
+# Used to load the finepersonas dataset
+# Provides easy access to thousands of datasets
+
+# ============================================================================
+# TOOLS AND EXTERNAL SERVICES
+# ============================================================================
+# These packages enable the agent's tools
+
+duckduckgo-search>=6.0.0
+# Web search functionality using DuckDuckGo
+# Essential for GAIA questions requiring current information
+# Free alternative to Google Search API
+
+# ============================================================================
+# UTILITIES AND ENVIRONMENT
+# ============================================================================
+# Supporting packages for configuration and development
+
+python-dotenv
+# For loading environment variables from .env files
+# Useful for local development and testing
+
+nest-asyncio
+# Allows running async code in environments that already have an event loop
+# Required for running LlamaIndex query engines in Jupyter/Gradio
+# Fixes "RuntimeError: This event loop is already running" errors
+
+# ============================================================================
+# OPTIONAL: ADDITIONAL USEFUL PACKAGES
+# ============================================================================
+# These might be helpful for specific GAIA questions but aren't required
+
+# numpy
+# For numerical computations if needed for advanced math questions
+
+# matplotlib
+# For creating charts/graphs if GAIA questions require visualizations
+
+# beautifulsoup4
+# For parsing HTML if web search results need detailed extraction
+
+# ============================================================================
+# DEVELOPMENT AND DEBUGGING (Optional)
+# ============================================================================
+# Uncomment these if you want enhanced debugging capabilities
+
+# jupyter
+# For interactive development and testing
+
+# ipywidgets
+# For enhanced Jupyter notebook widgets
+
+# rich
+# For beautiful terminal output and better logging
+
+# ============================================================================
+# INSTALLATION NOTES
+# ============================================================================
+# 
+# To install all dependencies:
+#   pip install -r requirements.txt
+#
+# For HuggingFace Spaces:
+#   - This file should be in your Space root directory
+#   - Dependencies are automatically installed when you deploy
+#   - No manual installation needed
+#
+# API Keys Setup:
+#   - Go to your Space Settings → Repository secrets
+#   - Add OPENAI_API_KEY (for OpenAI) or HF_TOKEN (for HuggingFace)
+#   - At least one is required for the agent to work
+#
+# Troubleshooting:
+#   - If imports fail, check that all packages installed correctly
+#   - Some packages may require specific versions for compatibility
+#   - Check the Space logs for detailed error messages

retriever.py

@@ -0,0 +1,526 @@
+"""
+retriever.py - Advanced RAG Implementation with Personas Database
+
+This file implements an advanced RAG system using:
+1. Real dataset from HuggingFace (dvilasuero/finepersonas-v0.1-tiny)
+2. Vector embeddings for semantic search
+3. ChromaDB for persistent vector storage
+4. LlamaIndex IngestionPipeline for processing
+
+This demonstrates advanced course concepts:
+- Dataset integration from HuggingFace
+- Vector embeddings vs keyword search
+- Persistent storage with ChromaDB
+- Ingestion pipelines for data processing
+
+Why this approach:
+- 5K personas provide rich, diverse data
+- Vector embeddings capture semantic meaning
+- ChromaDB provides fast, persistent storage
+- More realistic than simple guest database
+
+download_and_prepare_personas()   # Download 5K personas
+load_persona_documents()          # Load into documents  
+create_persona_index()            # Create vector index
+get_persona_query_engine()        # For tools.py to use
+"""
+
+import logging
+import os
+from typing import List, Dict, Any
+from pathlib import Path
+
+# LlamaIndex core components
+from llama_index.core.schema import Document
+from llama_index.core.tools import FunctionTool, QueryEngineTool
+from llama_index.core import SimpleDirectoryReader, VectorStoreIndex
+from llama_index.core.node_parser import SentenceSplitter
+from llama_index.core.ingestion import IngestionPipeline
+
+# Embeddings and vector store
+from llama_index.embeddings.huggingface import HuggingFaceEmbedding
+from llama_index.vector_stores.chroma import ChromaVectorStore
+
+# External libraries
+from datasets import load_dataset
+import chromadb
+
+# Setup logging
+logger = logging.getLogger(__name__)
+
+# ============================================================================
+# CONFIGURATION AND CONSTANTS
+# ============================================================================
+
+# Dataset configuration
+DATASET_NAME = "dvilasuero/finepersonas-v0.1-tiny"
+DATA_DIR = Path("data")
+CHROMA_DB_PATH = "./alfred_chroma_db"
+COLLECTION_NAME = "alfred"
+
+# Embedding model - good balance of performance and speed
+EMBEDDING_MODEL = "BAAI/bge-small-en-v1.5"
+
+# Chunk size for text splitting - optimal for personas
+CHUNK_SIZE = 1024
+CHUNK_OVERLAP = 20
+
+# ============================================================================
+# DATA PREPARATION - Loading Personas from HuggingFace
+# ============================================================================
+
+def download_and_prepare_personas() -> int:
+    """
+    Download personas from HuggingFace and save as individual text files.
+    
+    This approach demonstrates:
+    1. Dataset integration from HuggingFace Hub
+    2. Local file preparation for SimpleDirectoryReader
+    3. Data persistence for repeated runs
+    
+    Why save as files:
+    - SimpleDirectoryReader expects file-based input
+    - Allows for easy inspection and debugging
+    - Caches data locally to avoid repeated downloads
+    - Mimics real-world scenario where you have document files
+    
+    Returns:
+        int: Number of persona files created
+    """
+    logger.info(f"Starting persona data preparation...")
+    
+    # Create data directory if it doesn't exist
+    DATA_DIR.mkdir(parents=True, exist_ok=True)
+    
+    # Check if we already have data (avoid re-downloading)
+    existing_files = list(DATA_DIR.glob("persona_*.txt"))
+    if existing_files:
+        logger.info(f"Found {len(existing_files)} existing persona files, skipping download")
+        return len(existing_files)
+    
+    try:
+        # Load the dataset from HuggingFace
+        logger.info(f"Loading dataset: {DATASET_NAME}")
+        dataset = load_dataset(path=DATASET_NAME, split="train")
+        logger.info(f"Dataset loaded successfully with {len(dataset)} personas")
+        
+        # Save each persona as a separate text file
+        personas_created = 0
+        for i, persona_data in enumerate(dataset):
+            persona_file = DATA_DIR / f"persona_{i}.txt"
+            
+            # Extract the persona text
+            persona_text = persona_data["persona"]
+            
+            # Add some metadata to make the persona more searchable
+            enhanced_text = f"Persona {i}:\n{persona_text}"
+            
+            # Write to file
+            with open(persona_file, "w", encoding="utf-8") as f:
+                f.write(enhanced_text)
+            
+            personas_created += 1
+            
+            # Log progress for large datasets
+            if personas_created % 1000 == 0:
+                logger.info(f"Created {personas_created} persona files...")
+        
+        logger.info(f"✅ Successfully created {personas_created} persona files")
+        return personas_created
+        
+    except Exception as e:
+        logger.error(f"❌ Error downloading personas: {e}")
+        raise RuntimeError(f"Failed to download personas: {e}")
+
+
+# ============================================================================
+# DOCUMENT LOADING - Converting Files to LlamaIndex Documents
+# ============================================================================
+
+def load_persona_documents() -> List[Document]:
+    """
+    Load persona files into LlamaIndex Document objects.
+    
+    This demonstrates:
+    1. SimpleDirectoryReader usage for file loading
+    2. Document object creation and metadata handling
+    3. Error handling for file operations
+    
+    Why SimpleDirectoryReader:
+    - Handles multiple file formats automatically
+    - Preserves file metadata (filename, path, etc.)
+    - Integrates seamlessly with LlamaIndex pipeline
+    - Scales well for large document collections
+    
+    Returns:
+        List[Document]: List of loaded persona documents
+    """
+    logger.info("Loading persona documents...")
+    
+    # Ensure we have persona data
+    if not DATA_DIR.exists() or not list(DATA_DIR.glob("persona_*.txt")):
+        logger.info("No persona files found, downloading...")
+        download_and_prepare_personas()
+    
+    try:
+        # Use SimpleDirectoryReader to load all text files
+        reader = SimpleDirectoryReader(input_dir=str(DATA_DIR))
+        documents = reader.load_data()
+        
+        logger.info(f"✅ Loaded {len(documents)} persona documents")
+        
+        # Log some statistics about the documents
+        if documents:
+            total_chars = sum(len(doc.text) for doc in documents)
+            avg_chars = total_chars / len(documents)
+            logger.info(f"Average document length: {avg_chars:.0f} characters")
+        
+        return documents
+        
+    except Exception as e:
+        logger.error(f"❌ Error loading documents: {e}")
+        raise RuntimeError(f"Failed to load persona documents: {e}")
+
+
+# ============================================================================
+# VECTOR STORE SETUP - ChromaDB Configuration
+# ============================================================================
+
+def setup_chroma_vector_store():
+    """
+    Set up ChromaDB vector store for persistent storage.
+    
+    This demonstrates:
+    1. Persistent vector database configuration
+    2. Collection management
+    3. Integration with LlamaIndex vector stores
+    
+    Why ChromaDB:
+    - Persistent storage (survives application restarts)
+    - Fast vector similarity search
+    - Easy integration with LlamaIndex
+    - Good for development and production
+    - No external dependencies (self-contained)
+    
+    Returns:
+        ChromaVectorStore: Configured vector store ready for use
+    """
+    logger.info("Setting up ChromaDB vector store...")
+    
+    try:
+        # Create persistent ChromaDB client
+        # This creates a local database that persists between runs
+        db = chromadb.PersistentClient(path=CHROMA_DB_PATH)
+        logger.info(f"ChromaDB client created at: {CHROMA_DB_PATH}")
+        
+        # Get or create collection for our personas
+        # Collections are like tables in a traditional database
+        chroma_collection = db.get_or_create_collection(name=COLLECTION_NAME)
+        logger.info(f"Using collection: {COLLECTION_NAME}")
+        
+        # Wrap ChromaDB collection in LlamaIndex vector store
+        vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
+        
+        logger.info("✅ ChromaDB vector store configured successfully")
+        return vector_store
+        
+    except Exception as e:
+        logger.error(f"❌ Error setting up ChromaDB: {e}")
+        raise RuntimeError(f"Failed to setup ChromaDB: {e}")
+
+
+# ============================================================================
+# INGESTION PIPELINE - Document Processing with Embeddings
+# ============================================================================
+
+def create_ingestion_pipeline(vector_store) -> IngestionPipeline:
+    """
+    Create an ingestion pipeline for processing persona documents.
+    
+    This demonstrates:
+    1. Text chunking with SentenceSplitter
+    2. Embedding generation with HuggingFace models
+    3. Pipeline composition for complex processing
+    
+    The pipeline does:
+    1. Split documents into smaller chunks (better for retrieval)
+    2. Generate vector embeddings for each chunk
+    3. Store embeddings in the vector database
+    
+    Why this approach:
+    - Chunking improves retrieval precision
+    - Embeddings capture semantic meaning
+    - Pipeline caches results for efficiency
+    - Modular design allows easy modification
+    
+    Args:
+        vector_store: ChromaDB vector store for persistence
+        
+    Returns:
+        IngestionPipeline: Configured pipeline ready for document processing
+    """
+    logger.info("Creating ingestion pipeline...")
+    
+    try:
+        # Create text splitter
+        # SentenceSplitter respects sentence boundaries for better coherence
+        text_splitter = SentenceSplitter(
+            chunk_size=CHUNK_SIZE,     # Max characters per chunk
+            chunk_overlap=CHUNK_OVERLAP  # Overlap to maintain context
+        )
+        logger.info(f"Text splitter configured: {CHUNK_SIZE} chars, {CHUNK_OVERLAP} overlap")
+        
+        # Create embedding model
+        # This model converts text to numerical vectors that capture meaning
+        embed_model = HuggingFaceEmbedding(model_name=EMBEDDING_MODEL)
+        logger.info(f"Embedding model configured: {EMBEDDING_MODEL}")
+        
+        # Create the ingestion pipeline
+        # This processes documents through the transformations in order
+        pipeline = IngestionPipeline(
+            transformations=[
+                text_splitter,  # First: split into chunks
+                embed_model,    # Second: create embeddings
+            ],
+            vector_store=vector_store  # Third: store in database
+        )
+        
+        logger.info("✅ Ingestion pipeline created successfully")
+        return pipeline
+        
+    except Exception as e:
+        logger.error(f"❌ Error creating ingestion pipeline: {e}")
+        raise RuntimeError(f"Failed to create ingestion pipeline: {e}")
+
+
+# ============================================================================
+# INDEX CREATION - Vector Search Index
+# ============================================================================
+
+def create_persona_index():
+    """
+    Create or load the persona vector index.
+    
+    This is the main function that orchestrates the entire RAG setup:
+    1. Load documents from files
+    2. Set up vector storage
+    3. Process documents through pipeline
+    4. Create searchable index
+    
+    The index enables semantic search where:
+    - Similar meanings are found even with different words
+    - Context and relationships are preserved
+    - Fast retrieval from thousands of personas
+    
+    Returns:
+        VectorStoreIndex: Ready-to-use search index
+    """
+    logger.info("Creating persona search index...")
+    
+    try:
+        # Step 1: Load persona documents
+        documents = load_persona_documents()
+        if not documents:
+            raise RuntimeError("No documents loaded")
+        
+        # Step 2: Set up vector store
+        vector_store = setup_chroma_vector_store()
+        
+        # Step 3: Check if we already have processed data
+        # This saves time on repeated runs
+        try:
+            # Try to create index from existing vector store
+            embed_model = HuggingFaceEmbedding(model_name=EMBEDDING_MODEL)
+            existing_index = VectorStoreIndex.from_vector_store(
+                vector_store=vector_store,
+                embed_model=embed_model
+            )
+            
+            # Test if the index has data
+            test_retriever = existing_index.as_retriever(similarity_top_k=1)
+            test_results = test_retriever.retrieve("test query")
+            
+            if test_results:
+                logger.info("✅ Found existing persona index with data")
+                return existing_index
+            else:
+                logger.info("Existing index is empty, rebuilding...")
+                
+        except Exception:
+            logger.info("No existing index found, creating new one...")
+        
+        # Step 4: Process documents through ingestion pipeline
+        pipeline = create_ingestion_pipeline(vector_store)
+        
+        logger.info(f"Processing {len(documents)} documents through pipeline...")
+        # This may take a while for large datasets as it generates embeddings
+        nodes = pipeline.run(documents=documents)
+        logger.info(f"✅ Processed {len(nodes)} document chunks")
+        
+        # Step 5: Create the final index
+        embed_model = HuggingFaceEmbedding(model_name=EMBEDDING_MODEL)
+        index = VectorStoreIndex.from_vector_store(
+            vector_store=vector_store,
+            embed_model=embed_model
+        )
+        
+        logger.info("✅ Persona index created successfully")
+        return index
+        
+    except Exception as e:
+        logger.error(f"❌ Error creating persona index: {e}")
+        raise RuntimeError(f"Failed to create persona index: {e}")
+
+
+# ============================================================================
+# MAIN FUNCTIONS USED BY TOOLS.PY
+# ============================================================================
+# These are the core functions that tools.py uses to access the persona database.
+# Tool creation is handled in tools.py following the course structure.
+
+def get_persona_index():
+    """
+    Get the persona index for use by tools.py.
+    
+    This is a simple wrapper function that tools.py can import and use.
+    It ensures the index is created and ready for use.
+    
+    Returns:
+        VectorStoreIndex: The persona database index
+    """
+    return create_persona_index()
+
+
+def get_persona_query_engine():
+    """
+    Get a configured query engine for the persona database.
+    
+    This creates a query engine ready for use in QueryEngineTool.
+    Tools.py can import this to create the persona database tool.
+    
+    Returns:
+        QueryEngine: Configured query engine for persona database
+    """
+    try:
+        # Get the index
+        index = create_persona_index()
+        
+        # Configure embedding model (same as indexing)
+        embed_model = HuggingFaceEmbedding(model_name=EMBEDDING_MODEL)
+        
+        # Create query engine with optimal settings
+        query_engine = index.as_query_engine(
+            response_mode="tree_summarize",  # Good for combining multiple sources
+            similarity_top_k=5,  # Retrieve top 5 most relevant personas
+            streaming=False  # Disable streaming for stability
+        )
+        
+        logger.info("✅ Persona query engine ready for tools.py")
+        return query_engine
+        
+    except Exception as e:
+        logger.error(f"❌ Error creating query engine for tools.py: {e}")
+        raise
+
+
+# ============================================================================
+# TESTING AND DEBUGGING FUNCTIONS
+# ============================================================================
+
+def test_persona_system():
+    """
+    Test the persona system components available in retriever.py.
+    This helps verify that the database setup is working correctly.
+    
+    Note: Tool creation testing is now in tools.py since that's where tools are created.
+    """
+    print("\n=== Testing Persona Database System ===")
+    
+    # Test data preparation
+    print("\n--- Testing Data Preparation ---")
+    try:
+        count = download_and_prepare_personas()
+        print(f"✅ Data preparation successful: {count} personas")
+    except Exception as e:
+        print(f"❌ Data preparation failed: {e}")
+        return
+    
+    # Test document loading
+    print("\n--- Testing Document Loading ---")
+    try:
+        docs = load_persona_documents()
+        print(f"✅ Document loading successful: {len(docs)} documents")
+    except Exception as e:
+        print(f"❌ Document loading failed: {e}")
+        return
+    
+    # Test index creation
+    print("\n--- Testing Index Creation ---")
+    try:
+        index = create_persona_index()
+        print("✅ Index creation successful")
+    except Exception as e:
+        print(f"❌ Index creation failed: {e}")
+        return
+    
+    # Test basic retrieval (without tool wrapper)
+    print("\n--- Testing Basic Retrieval ---")
+    test_queries = [
+        "writers and authors",
+        "people interested in travel", 
+        "scientists and researchers"
+    ]
+    
+    try:
+        retriever = index.as_retriever(similarity_top_k=2)
+        
+        for query in test_queries:
+            print(f"\nQuery: {query}")
+            try:
+                results = retriever.retrieve(query)
+                if results:
+                    print(f"✅ Found {len(results)} results")
+                    print(f"Sample: {results[0].text[:100]}...")
+                else:
+                    print("No results found")
+            except Exception as e:
+                print(f"❌ Query failed: {e}")
+                
+    except Exception as e:
+        print(f"❌ Retriever creation failed: {e}")
+    
+    # Test query engine creation (for tools.py)
+    print("\n--- Testing Query Engine Creation ---")
+    try:
+        query_engine = get_persona_query_engine()
+        print("✅ Query engine creation successful")
+        print("   (This query engine can be used by tools.py)")
+    except Exception as e:
+        print(f"❌ Query engine creation failed: {e}")
+    
+    print("\n=== Database System Testing Complete ===")
+    print("\nNote: For tool testing, run tools.py or usage_example.py")
+
+
+# ============================================================================
+# MAIN EXECUTION
+# ============================================================================
+
+if __name__ == "__main__":
+    # If this file is run directly, run tests
+    print("Persona Database System Testing")
+    print("=" * 50)
+    
+    # Set up logging for testing
+    logging.basicConfig(level=logging.INFO)
+    
+    # Run database system tests
+    test_persona_system()
+    
+    print("\n" + "=" * 50)
+    print("Database testing complete!")
+    print("\nFor tool testing, run:")
+    print("  python tools.py")
+    print("  python usage_example.py")
+    print("\nFor full agent testing, run:")
+    print("  python app.py")

tools.py

@@ -0,0 +1,656 @@
+"""
+tools.py - Agent Tools for GAIA Benchmark (Course Didactic Structure)
+
+This file follows the course approach of separating:
+1. Raw functions (the actual functionality)
+2. Tool wrappers (FunctionTool and QueryEngineTool creation)
+
+This makes it easier to understand and debug each component separately.
+Each tool addresses specific GAIA benchmark needs while demonstrating course concepts.
+
+create_persona_database_tool()    # QueryEngineTool creation
+get_all_tools()                   # All tools collection
+"""
+
+import logging
+import math
+import os
+import random
+from typing import List
+import chromadb
+
+# LlamaIndex imports
+from llama_index.core.tools import FunctionTool, QueryEngineTool
+from llama_index.core import VectorStoreIndex
+from llama_index.embeddings.huggingface import HuggingFaceEmbedding
+from llama_index.vector_stores.chroma import ChromaVectorStore
+from llama_index.llms.huggingface_api import HuggingFaceInferenceAPI
+
+# Setup logging
+logger = logging.getLogger(__name__)
+
+# ============================================================================
+# PART 1: RAW FUNCTIONS (The actual functionality)
+# ============================================================================
+# These are the core functions that do the actual work.
+# They can be tested independently and are easy to understand.
+
+def web_search(query: str) -> str:
+    """
+    Search the web for information using DuckDuckGo.
+    
+    This function handles the actual web searching logic.
+    Critical for GAIA questions requiring current information.
+    
+    Args:
+        query (str): The search query/question
+        
+    Returns:
+        str: Formatted search results with titles, content, and URLs
+        
+    Why this is essential for GAIA:
+    - Many GAIA questions need current information (news, prices, events)
+    - LLMs have knowledge cutoffs and may not know recent facts
+    - Web search provides access to the latest information
+    """
+    logger.info(f"🔍 Web search requested: {query}")
+    
+    try:
+        # Import DuckDuckGo search - free search API
+        from duckduckgo_search import DDGS
+        
+        # Perform the search with a reasonable limit
+        with DDGS() as ddgs:
+            # Get top 3 results to avoid overwhelming the LLM
+            results = list(ddgs.text(query, max_results=3))
+            
+            if not results:
+                logger.warning("No search results found")
+                return "No search results found for this query."
+            
+            # Format results in a clean, readable way
+            formatted_results = []
+            for i, result in enumerate(results, 1):
+                formatted_result = (
+                    f"Result {i}:\n"
+                    f"Title: {result['title']}\n"
+                    f"Content: {result['body']}\n"
+                    f"URL: {result['href']}\n"
+                )
+                formatted_results.append(formatted_result)
+            
+            final_result = "\n".join(formatted_results)
+            logger.info(f"✅ Web search completed: {len(results)} results found")
+            return final_result
+            
+    except ImportError:
+        error_msg = "DuckDuckGo search library not available. Please install duckduckgo-search."
+        logger.error(error_msg)
+        return error_msg
+    except Exception as e:
+        error_msg = f"Search error: {str(e)}"
+        logger.error(f"Web search failed: {e}")
+        return error_msg
+
+
+def calculate(expression: str) -> str:
+    """
+    Safely evaluate mathematical expressions.
+    
+    This function handles mathematical calculations with safety measures.
+    CRITICAL for GAIA because many questions involve precise calculations.
+    
+    Args:
+        expression (str): Mathematical expression (e.g., "2 + 2", "sqrt(16)", "sin(pi/2)")
+        
+    Returns:
+        str: The result of the calculation or an error message
+        
+    Why this is essential for GAIA:
+    - GAIA has many mathematical questions (percentages, conversions, etc.)
+    - LLMs can make arithmetic errors, especially with complex math
+    - Exact numerical accuracy is required (GAIA uses exact match scoring)
+    
+    Examples:
+        calculate("2 + 2") → "4"
+        calculate("15% of 847") → calculate("0.15 * 847") → "127.05"
+        calculate("sqrt(16)") → "4.0"
+    """
+    logger.info(f"🧮 Calculation requested: {expression}")
+    
+    try:
+        # Create a safe environment for evaluation
+        # Only allow mathematical functions, no dangerous operations
+        allowed_names = {
+            # Include all math module functions (sin, cos, sqrt, log, etc.)
+            k: v for k, v in math.__dict__.items() if not k.startswith("__")
+        }
+        
+        # Add safe Python functions
+        allowed_names.update({
+            "abs": abs,      # Absolute value
+            "round": round,  # Rounding
+            "min": min,      # Minimum
+            "max": max,      # Maximum
+            "sum": sum,      # Sum of iterables
+            "pow": pow,      # Power function
+        })
+        
+        # Add mathematical constants
+        allowed_names.update({
+            "pi": math.pi,   # π
+            "e": math.e,     # Euler's number
+        })
+        
+        # Evaluate the expression safely
+        # __builtins__ = {} prevents dangerous functions like open(), exec()
+        result = eval(expression, {"__builtins__": {}}, allowed_names)
+        
+        result_str = str(result)
+        logger.info(f"✅ Calculation result: {expression} = {result_str}")
+        return result_str
+        
+    except ZeroDivisionError:
+        error_msg = "Error: Division by zero"
+        logger.error(error_msg)
+        return error_msg
+    except ValueError as e:
+        error_msg = f"Error: Invalid mathematical operation - {str(e)}"
+        logger.error(error_msg)
+        return error_msg
+    except SyntaxError:
+        error_msg = "Error: Invalid mathematical expression syntax"
+        logger.error(error_msg)
+        return error_msg
+    except Exception as e:
+        error_msg = f"Calculation error: {str(e)}"
+        logger.error(f"Unexpected calculation error: {e}")
+        return error_msg
+
+
+def analyze_file(file_content: str, file_type: str = "text") -> str:
+    """
+    Analyze file content and extract relevant information.
+    
+    This function processes different file types for analysis.
+    Useful for GAIA questions that include file attachments.
+    
+    Args:
+        file_content (str): The content of the file
+        file_type (str): Type of file ("text", "csv", "json", etc.)
+        
+    Returns:
+        str: Analysis results or extracted information
+        
+    Why this helps with GAIA:
+    - Some GAIA questions include data files to analyze
+    - Questions might ask for statistics, summaries, or specific data extraction
+    - File processing shows practical data analysis skills
+    """
+    logger.info(f"📊 File analysis requested for {file_type} file")
+    
+    try:
+        if file_type.lower() == "csv":
+            # For CSV files, provide basic statistics
+            lines = file_content.strip().split('\n')
+            if not lines:
+                return "Empty file"
+            
+            # Count rows and columns (assuming first row is header)
+            num_rows = len(lines) - 1  # Subtract header
+            if lines:
+                num_cols = len(lines[0].split(','))
+                analysis = (
+                    f"CSV Analysis:\n"
+                    f"- Rows: {num_rows}\n"
+                    f"- Columns: {num_cols}\n"
+                    f"- Headers: {lines[0]}"
+                )
+                if num_rows > 0:
+                    analysis += f"\n- First data row: {lines[1] if len(lines) > 1 else 'None'}"
+                return analysis
+                
+        elif file_type.lower() in ["txt", "text"]:
+            # For text files, provide basic statistics
+            lines = file_content.split('\n')
+            words = file_content.split()
+            chars = len(file_content)
+            
+            return (
+                f"Text Analysis:\n"
+                f"- Lines: {len(lines)}\n"
+                f"- Words: {len(words)}\n"
+                f"- Characters: {chars}"
+            )
+            
+        else:
+            # For other file types, return content with basic info
+            preview = file_content[:1000] + '...' if len(file_content) > 1000 else file_content
+            return f"File content ({file_type}):\n{preview}"
+            
+    except Exception as e:
+        error_msg = f"File analysis error: {str(e)}"
+        logger.error(error_msg)
+        return error_msg
+
+
+def get_weather(location: str) -> str:
+    """
+    Get dummy weather information for a location.
+    
+    This is a simplified weather function for demonstration.
+    In a real implementation, you'd connect to a weather API like OpenWeatherMap.
+    
+    Args:
+        location (str): City or location name
+        
+    Returns:
+        str: Weather description with temperature
+        
+    Note: This is a dummy implementation for course purposes.
+    Real weather data would require an API key and actual weather service.
+    """
+    logger.info(f"🌤️ Weather requested for: {location}")
+    
+    # Dummy weather data for demonstration
+    weather_conditions = [
+        {"condition": "Sunny", "temp_c": 25, "humidity": 60},
+        {"condition": "Cloudy", "temp_c": 20, "humidity": 70},
+        {"condition": "Rainy", "temp_c": 15, "humidity": 85},
+        {"condition": "Windy", "temp_c": 22, "humidity": 55},
+        {"condition": "Clear", "temp_c": 28, "humidity": 45}
+    ]
+    
+    # Randomly select weather (in real implementation, this would be API call)
+    weather = random.choice(weather_conditions)
+    
+    result = (
+        f"Weather in {location.title()}:\n"
+        f"Condition: {weather['condition']}\n"
+        f"Temperature: {weather['temp_c']}°C\n"
+        f"Humidity: {weather['humidity']}%"
+    )
+    
+    logger.info(f"✅ Weather result: {weather['condition']}, {weather['temp_c']}°C")
+    return result
+
+
+# ============================================================================
+# PART 2: PERSONA DATABASE SETUP (QueryEngine creation)
+# ============================================================================
+# This sets up the persona database query engine following the course pattern.
+
+def create_persona_query_engine():
+    """
+    Create a query engine for the persona database following course pattern.
+    
+    This demonstrates the exact approach from the course:
+    1. Connect to existing ChromaDB database
+    2. Create VectorStoreIndex from the stored vectors
+    3. Configure LLM for response generation
+    4. Create QueryEngine with specific settings
+    
+    Returns:
+        QueryEngine: Ready-to-use query engine for persona database
+        
+    Why QueryEngine vs simple retrieval:
+    - QueryEngine combines retrieval + LLM generation
+    - Provides natural, conversational responses
+    - Can synthesize information from multiple personas
+    - Better for complex questions requiring reasoning
+    """
+    logger.info("🏗️ Creating persona database query engine...")
+    
+    try:
+        # Step 1: Connect to existing ChromaDB (created by retriever.py)
+        db = chromadb.PersistentClient(path="./alfred_chroma_db")
+        chroma_collection = db.get_or_create_collection("alfred")
+        vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
+        logger.info("✅ Connected to ChromaDB")
+        
+        # Step 2: Set up embedding model (same as used during indexing)
+        embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")
+        logger.info("✅ Embedding model configured")
+        
+        # Step 3: Create VectorStoreIndex from existing data
+        index = VectorStoreIndex.from_vector_store(
+            vector_store=vector_store, 
+            embed_model=embed_model
+        )
+        logger.info("✅ Vector index created")
+        
+        # Step 4: Configure LLM for response generation
+        # Try to get LLM from settings first, then fallback
+        try:
+            from llama_index.core import Settings
+            llm = Settings.llm
+            
+            if llm is None:
+                # Fallback to HuggingFace LLM
+                hf_token = os.getenv("HF_TOKEN")
+                if hf_token:
+                    llm = HuggingFaceInferenceAPI(
+                        model_name="Qwen/Qwen2.5-Coder-32B-Instruct",
+                        token=hf_token,
+                        max_new_tokens=512,
+                        temperature=0.1
+                    )
+                    logger.info("✅ Using HuggingFace LLM")
+                else:
+                    logger.warning("⚠️ No LLM available, query engine will use default")
+                    llm = None
+        except Exception:
+            logger.warning("⚠️ Could not configure LLM, using default")
+            llm = None
+        
+        # Step 5: Create QueryEngine with optimized settings
+        query_engine = index.as_query_engine(
+            llm=llm,
+            response_mode="tree_summarize",  # Good for combining multiple sources
+            similarity_top_k=5,  # Retrieve top 5 most relevant personas
+            streaming=False  # Disable streaming for stability
+        )
+        
+        logger.info("✅ Persona query engine created successfully")
+        return query_engine
+        
+    except Exception as e:
+        logger.error(f"❌ Error creating persona query engine: {e}")
+        raise RuntimeError(f"Failed to create persona query engine: {e}")
+
+
+# ============================================================================
+# PART 3: TOOL WRAPPERS (Converting functions to tools)
+# ============================================================================
+# This section creates the actual tools that the agent can use.
+# Each tool wraps a function with metadata for the LLM to understand.
+
+# Web Search Tool
+web_search_tool = FunctionTool.from_defaults(
+    fn=web_search,
+    name="web_search",
+    description=(
+        "Search the web for current information, recent events, statistics, "
+        "facts, or any information not in the LLM's training data. "
+        "Use this when you need up-to-date or specific factual information. "
+        "Essential for GAIA questions about current events, prices, or recent developments."
+    )
+)
+
+# Calculator Tool  
+calculator_tool = FunctionTool.from_defaults(
+    fn=calculate,
+    name="calculator", 
+    description=(
+        "Perform mathematical calculations and evaluate mathematical expressions. "
+        "Supports basic arithmetic (+, -, *, /), advanced math functions (sqrt, sin, cos, log), "
+        "and mathematical constants (pi, e). Use this for any numerical computations, "
+        "percentage calculations, unit conversions, or statistical operations. "
+        "CRITICAL for GAIA mathematical questions to ensure accuracy."
+    )
+)
+
+# File Analysis Tool
+file_analysis_tool = FunctionTool.from_defaults(
+    fn=analyze_file,
+    name="file_analyzer",
+    description=(
+        "Analyze file contents including CSV files, text files, and other data files. "
+        "Can extract statistics, summarize content, and process structured data. "
+        "Use this when GAIA questions involve analyzing attached files or datasets."
+    )
+)
+
+# Weather Tool (demonstration)
+weather_tool = FunctionTool.from_defaults(
+    fn=get_weather,
+    name="weather_tool",
+    description=(
+        "Get weather information for a specific location. "
+        "Note: This is a demo implementation with dummy data. "
+        "Use when questions ask about weather conditions."
+    )
+)
+
+# Persona Database Query Engine Tool
+def create_persona_database_tool():
+    """
+    Create the persona database tool using QueryEngineTool.
+    
+    This follows the exact course pattern for creating QueryEngineTool.
+    The tool combines retrieval with LLM generation for natural responses.
+    
+    Returns:
+        QueryEngineTool: Tool for querying the persona database
+    """
+    logger.info("🛠️ Creating persona database tool...")
+    
+    try:
+        # First ensure we have the persona data (this will create it if needed)
+        try:
+            from retriever import create_persona_index
+            # This creates the index if it doesn't exist
+            create_persona_index()
+            logger.info("✅ Persona index ready")
+        except Exception as e:
+            logger.warning(f"⚠️ Could not ensure persona index: {e}")
+        
+        # Create the query engine
+        query_engine = create_persona_query_engine()
+        
+        # Create the QueryEngineTool following course pattern
+        persona_tool = QueryEngineTool.from_defaults(
+            query_engine=query_engine,
+            name="persona_database",
+            description=(
+                "Search and query a database of 5000 diverse personas with various backgrounds, "
+                "interests, and professions. Use this to find people with specific characteristics, "
+                "skills, or interests. Can answer questions like 'find writers', 'who likes travel', "
+                "'scientists in the group', 'creative professionals', or 'people interested in technology'. "
+                "Returns detailed information about matching personas with their backgrounds and interests."
+            )
+        )
+        
+        logger.info("✅ Persona database tool created successfully")
+        return persona_tool
+        
+    except Exception as e:
+        logger.error(f"❌ Error creating persona database tool: {e}")
+        # Return None so the agent can still work without this tool
+        return None
+
+
+# ============================================================================
+# PART 4: TOOL COLLECTION (Getting all tools together)
+# ============================================================================
+
+def get_all_tools() -> List:
+    """
+    Get all available tools for the GAIA agent.
+    
+    This function collects all tools and handles any creation errors gracefully.
+    The agent will work with whatever tools are successfully created.
+    
+    Returns:
+        List: All successfully created tools
+    """
+    logger.info("🔧 Collecting all tools...")
+    
+    tools = []
+    
+    # Add function-based tools (these should always work)
+    try:
+        tools.extend([
+            web_search_tool,
+            calculator_tool, 
+            file_analysis_tool,
+            weather_tool
+        ])
+        logger.info(f"✅ Added {len(tools)} function-based tools")
+    except Exception as e:
+        logger.error(f"❌ Error adding function tools: {e}")
+    
+    # Add persona database tool (this might fail if database isn't ready)
+    try:
+        persona_tool = create_persona_database_tool()
+        if persona_tool:
+            tools.append(persona_tool)
+            logger.info("✅ Added persona database tool")
+        else:
+            logger.warning("⚠️ Persona database tool not available")
+    except Exception as e:
+        logger.warning(f"⚠️ Could not create persona database tool: {e}")
+    
+    logger.info(f"🎯 Total tools available: {len(tools)}")
+    for tool in tools:
+        tool_name = getattr(tool.metadata, 'name', 'Unknown')
+        logger.info(f"   - {tool_name}")
+    
+    return tools
+
+
+# ============================================================================
+# PART 5: TESTING FUNCTIONS (For development and debugging)
+# ============================================================================
+
+def test_individual_functions():
+    """
+    Test each function individually to make sure they work.
+    This helps with debugging and understanding what each function does.
+    """
+    print("\n=== Testing Individual Functions ===")
+    
+    # Test web search
+    print("\n--- Testing Web Search Function ---")
+    try:
+        result = web_search("current year")
+        print(f"Web search result: {result[:150]}...")
+        print("✅ Web search function works")
+    except Exception as e:
+        print(f"❌ Web search failed: {e}")
+    
+    # Test calculator
+    print("\n--- Testing Calculator Function ---")
+    try:
+        result = calculate("2 + 2 * 3")
+        print(f"Calculator result (2 + 2 * 3): {result}")
+        result = calculate("sqrt(16)")
+        print(f"Calculator result (sqrt(16)): {result}")
+        print("✅ Calculator function works")
+    except Exception as e:
+        print(f"❌ Calculator failed: {e}")
+    
+    # Test file analyzer
+    print("\n--- Testing File Analysis Function ---")
+    try:
+        sample_csv = "name,age,city\nJohn,25,NYC\nJane,30,LA\nBob,35,SF"
+        result = analyze_file(sample_csv, "csv")
+        print(f"File analysis result: {result}")
+        print("✅ File analysis function works")
+    except Exception as e:
+        print(f"❌ File analysis failed: {e}")
+    
+    # Test weather
+    print("\n--- Testing Weather Function ---")
+    try:
+        result = get_weather("Paris")
+        print(f"Weather result: {result}")
+        print("✅ Weather function works")
+    except Exception as e:
+        print(f"❌ Weather failed: {e}")
+
+
+def test_tool_creation():
+    """
+    Test that all tools can be created successfully.
+    """
+    print("\n=== Testing Tool Creation ===")
+    
+    try:
+        tools = get_all_tools()
+        print(f"✅ Successfully created {len(tools)} tools")
+        
+        for tool in tools:
+            tool_name = getattr(tool.metadata, 'name', 'Unknown')
+            tool_desc = getattr(tool.metadata, 'description', 'No description')[:100]
+            print(f"   - {tool_name}: {tool_desc}...")
+            
+    except Exception as e:
+        print(f"❌ Tool creation failed: {e}")
+
+
+def test_tool_functionality():
+    """
+    Test that tools can actually be called and return results.
+    """
+    print("\n=== Testing Tool Functionality ===")
+    
+    tools = get_all_tools()
+    
+    for tool in tools:
+        tool_name = getattr(tool.metadata, 'name', 'Unknown')
+        print(f"\n--- Testing {tool_name} ---")
+        
+        try:
+            if tool_name == "calculator":
+                # Test calculator tool
+                result = tool.func("5 * 8")
+                print(f"Calculator test (5 * 8): {result}")
+                
+            elif tool_name == "web_search":
+                # Test web search (might be slow)
+                print("Testing web search (this might take a moment)...")
+                result = tool.func("Python programming")
+                print(f"Web search test: {result[:100]}...")
+                
+            elif tool_name == "file_analyzer":
+                # Test file analyzer
+                test_data = "col1,col2\nval1,val2\nval3,val4"
+                result = tool.func(test_data, "csv")
+                print(f"File analyzer test: {result}")
+                
+            elif tool_name == "weather_tool":
+                # Test weather tool
+                result = tool.func("London")
+                print(f"Weather test: {result}")
+                
+            elif tool_name == "persona_database":
+                # Test persona database (might be slow on first run)
+                print("Testing persona database (this might take a moment)...")
+                # This would be an async call in real usage
+                print("Persona database test skipped (requires async)")
+                
+            print(f"✅ {tool_name} test completed")
+            
+        except Exception as e:
+            print(f"❌ {tool_name} test failed: {e}")
+
+
+# ============================================================================
+# MAIN EXECUTION (For testing when file is run directly)
+# ============================================================================
+
+if __name__ == "__main__":
+    print("GAIA Agent Tools Testing")
+    print("=" * 50)
+    
+    # Set up logging for testing
+    logging.basicConfig(level=logging.INFO)
+    
+    # Test individual functions first
+    test_individual_functions()
+    
+    # Test tool creation
+    test_tool_creation()
+    
+    # Test tool functionality (optional - can be slow)
+    response = input("\nRun tool functionality tests? (y/n): ")
+    if response.lower() == 'y':
+        test_tool_functionality()
+    else:
+        print("Skipping functionality tests")
+    
+    print("\n=== Tools Testing Complete ===")
+    print("\nTo use these tools in your agent:")
+    print("from tools import get_all_tools")
+    print("tools = get_all_tools()")