Deploy to HuggingFace Space

.clinerules

@@ -0,0 +1,98 @@
+# Python Coding Conventions for GAIA Benchmark Agent
+
+## Naming Conventions
+- Private methods and functions MUST start with underscore (_)
+- Internal helper methods that are only called within the same class/module are private
+- Public API methods should NOT start with underscore
+- Class names use PascalCase
+- Functions and methods use snake_case
+- Constants use UPPER_SNAKE_CASE
+
+## Function Privacy Rules
+These functions should be private (prefixed with _):
+- Helper functions used only within the same class
+- Internal implementation details not part of public API
+- Functions only called by other functions in the same module
+
+These functions should be public (no underscore):
+- Entry points (like main())
+- Functions called from other modules
+- API endpoints
+- Functions passed as callbacks to external libraries
+
+## Function Organization
+- Helper functions used only internally should be marked private
+- Public functions should have comprehensive docstrings with Args, Returns, Raises
+- Private functions should have brief docstrings explaining their purpose
+- Group related functions together
+
+## Import Organization
+- Standard library imports first
+- Third-party imports second
+- Local application imports last
+- Separate groups with blank lines
+- Use absolute imports for clarity
+
+## Documentation
+- All public functions must have Google-style docstrings
+- Include Args, Returns, and Raises sections where applicable
+- Private functions should have brief one-line docstrings
+- Avoid redundant comments that just repeat what the code does
+
+## Code Structure
+- Prefer composition over inheritance
+- Use wrapper classes for extensibility (like MyGAIAAgents)
+- Keep functions focused on single responsibility
+- Extract complex logic into private helper methods
+- Classes should have clear, single responsibilities
+
+## Error Handling
+- Use specific exception types, not bare except clauses
+- Validate inputs at API boundaries using validators module
+- Log errors with context information
+- Use custom exception classes where appropriate (like ValidationError)
+
+## Testing Philosophy
+- Public API should be easily testable
+- Private methods don't need direct tests (tested via public API)
+- Test behavior, not implementation
+
+## Project-Specific Architecture Rules
+- Agent implementations should be in separate files (e.g., langgraphagent.py)
+- All agent classes must implement __call__(question, file_name) method
+- Configuration should be centralized in config.py
+- Use ResultFormatter for all output formatting
+- Use QuestionLoader for all question fetching
+- Use AgentRunner for agent execution orchestration
+
+## Type Hints
+- Use type hints for all function signatures
+- Import types from typing module
+- Use Optional[] for nullable parameters
+- Use List[], Dict[], Tuple[] for collections
+
+## Async/Concurrency
+- Not currently used in this project
+- If added, use async/await consistently
+- Document async functions clearly
+
+## File Organization
+- One class per file for major components
+- Related utility functions can share a module
+- Keep files under 500 lines when possible
+
+## Comments
+- Use # for single-line comments
+- Use """docstrings""" for function/class documentation
+- Avoid obvious comments like "# increment counter"
+- Explain WHY, not WHAT (code shows what, comments explain why)
+
+## Git Commit Messages
+- Use conventional commit format
+- Include "Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>" for AI-assisted code
+- Explain both what changed and why
+
+## External Libraries
+- scorer.py is copied from official GAIA - do NOT modify function names
+- Prefer using existing utilities over creating new ones
+- Document external dependencies clearly

.github/workflows/sync-to-hf.yml

@@ -0,0 +1,36 @@
+name: Sync to Hugging Face
+
+on:
+  push:
+    branches:
+      - main
+    paths-ignore:
+      - 'README.md'
+      - 'docs/**'
+      - '**.md'
+      - 'LICENSE'
+
+jobs:
+  sync:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+          lfs: true
+          
+      - name: Push to Hugging Face Space
+        env:
+          HF_SYNC_TOKEN: ${{ secrets.HF_SYNC_TOKEN }}
+        run: |
+          git config --global user.email "github-actions[bot]@users.noreply.github.com"
+          git config --global user.name "github-actions[bot]"
+          git remote add hf https://hemantvirmani:$HF_SYNC_TOKEN@huggingface.co/spaces/hemantvirmani/Final_Assignment_Template
+          # Push only the current file state as a single orphan commit.
+          # This prevents HF from seeing old commits that contained binary files.
+          git checkout --orphan hf-deploy
+          git add -A
+          git commit -m "Deploy to HuggingFace Space"
+          git push hf hf-deploy:main --force
+          

.gitignore

@@ -0,0 +1,8 @@
+__pycache__/*.pyc
+.venv/
+
+# GAIA question attachment files — downloaded at runtime from HuggingFace dataset
+# Keep only questions.json and metadata.jsonl; ignore everything else in files/
+files/*
+!files/questions.json
+!files/metadata.jsonl

README.md

@@ -0,0 +1,310 @@
+---
+title: GAIA Benchmark Agent
+emoji: 🕵🏻‍♂️
+colorFrom: indigo
+colorTo: indigo
+sdk: gradio
+sdk_version: 6.2.0
+app_file: app.py
+pinned: false
+hf_oauth: true
+hf_oauth_expiration_minutes: 480
+---
+
+Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+
+
+# GAIA Benchmark Agent
+
+A LangGraph-based AI agent designed to solve questions from the GAIA (General AI Assistants) benchmark. This agent uses Google's Gemini model with custom tools for web search, file processing, and multimodal analysis to answer complex questions requiring reasoning and information gathering.
+
+## Features
+
+- **LangGraph Architecture**: Implements a state-graph agent workflow with tool calling capabilities
+- **Multimodal Capabilities**:
+  - Image analysis (PNG, JPG, JPEG, GIF, WebP, BMP)
+  - YouTube video analysis and transcript extraction
+  - Audio transcription (MP3)
+  - PDF and Excel file processing
+- **Web Research Tools**:
+  - DuckDuckGo web search
+  - Wikipedia integration
+  - ArXiv academic paper search
+  - Web page content extraction
+- **Mathematical Operations**: Basic arithmetic and modulus operations
+- **Gradio Interface**: User-friendly web UI for testing and evaluation
+- **Automated Evaluation**: Fetches questions from API, processes them, and submits answers
+- **Observability**: Built-in integration with Langfuse for tracking traces and metrics
+
+## Project Structure
+
+```
+GAIA_Benchmark_Agent/
+├── app.py              # Main application entry point
+├── agents.py           # LangGraph agent implementation
+├── custom_tools.py     # Tool definitions for web search, files, etc.
+├── system_prompt.py    # Agent system prompt and instructions
+├── gradioapp.py        # Gradio UI components
+├── requirements.txt    # Python dependencies
+└── files/
+    └── metadata.jsonl  # Ground truth data for local testing
+```
+
+## Installation
+
+1. Clone the repository:
+```bash
+git clone https://github.com/yourusername/GAIA_Benchmark_Agent.git
+cd GAIA_Benchmark_Agent
+```
+
+2. Install dependencies:
+```bash
+pip install -r requirements.txt
+```
+
+3. Set up environment variables:
+```bash
+export GOOGLE_API_KEY="your_google_api_key"
+export HUGGINGFACEHUB_API_TOKEN="your_hf_token"  # Optional.  not yet used
+
+# Langfuse Observability (Optional)
+export LANGFUSE_PUBLIC_KEY="pk-lf-..."
+export LANGFUSE_SECRET_KEY="sk-lf-..."
+export LANGFUSE_HOST="https://cloud.langfuse.com" # Optional
+```
+
+## Requirements
+
+- Python 3.8+
+- Google API Key (for Gemini model)
+- ffmpeg (optional, for audio processing)
+
+### Key Dependencies
+
+- `langchain-core`, `langgraph` - Agent framework
+- `langchain-google-genai` - Google Gemini integration
+- `gradio` - Web UI
+- `requests`, `beautifulsoup4` - Web scraping
+- `pypdf`, `pandas` - File processing
+- `youtube-transcript-api` - YouTube integration
+- `ddgs` - DuckDuckGo search
+
+## Usage
+
+### Running the Gradio Interface
+
+Launch the web interface for interactive testing:
+
+```bash
+python app.py
+```
+
+This will start a Gradio app where you can:
+- Log in with your Hugging Face account
+- Run evaluation on all questions
+- Test individual questions
+- View results and scores
+
+### Running Local Tests
+
+Test the agent on specific questions without the web interface:
+
+```bash
+python app.py --test
+```
+
+Edit the question indices in [app.py:196](app.py#L196) to customize which questions to test.
+
+### Using the Agent Programmatically
+
+```python
+from agents import MyGAIAAgents
+
+# Initialize agent (automatically uses ACTIVE_AGENT from config)
+agent = MyGAIAAgents()
+
+# Ask a question
+answer = agent("What is the capital of France?")
+print(answer)
+
+# Ask a question with a file reference
+answer = agent(
+    "What data is in this spreadsheet?",
+    file_name="data.xlsx"
+)
+print(answer)
+```
+
+## How It Works
+
+### Agent Architecture
+
+The agent is built using LangGraph with the following workflow:
+
+1. **Initialize**: Loads the question and system prompt
+2. **Assistant Node**: Calls the LLM (Gemini) to decide on tool usage
+3. **Tool Node**: Executes requested tools (search, file reading, etc.)
+4. **Iteration**: Loops between assistant and tools until answer is found
+5. **Termination**: Returns final answer or hits step limit (25 steps max)
+
+### Available Tools
+
+**Search & Research:**
+- `websearch` - DuckDuckGo web search
+- `wiki_search` - Wikipedia articles
+- `arvix_search` - Academic papers
+- `get_webpage_content` - Extract webpage text
+- `get_youtube_transcript` - YouTube video transcripts
+- `analyze_youtube_video` - AI analysis of YouTube videos
+
+**File Processing:**
+- `read_excel_file` - Read Excel spreadsheets
+- `read_python_script` - Read Python source code
+- `parse_audio_file` - Transcribe MP3 files
+- `analyze_image` - AI vision analysis of images
+
+**Utilities:**
+- Math operations: `add`, `subtract`, `multiply`, `divide`, `power`, `modulus`
+- `string_reverse` - Reverse encoded/gibberish text
+- `get_current_time_in_timezone` - Get time in any timezone
+
+### System Prompt
+
+The agent follows strict output formatting rules defined in [system_prompt.py](system_prompt.py):
+- Returns only the final answer (no conversational filler)
+- No markdown formatting or JSON structures
+- Uses tools instead of guessing
+- Handles encoded/reversed text
+- Verifies answers before output
+
+## Configuration
+
+### Change Agent Type
+
+Edit the `ACTIVE_AGENT` variable in [config.py:32](config.py#L32):
+
+```python
+# Valid values: "LangGraph", "ReActLangGraph", "LLamaIndex", "SMOL"
+ACTIVE_AGENT = "LangGraph"  # Currently only LangGraph is implemented
+```
+
+The `MyGAIAAgents` wrapper class will automatically instantiate the correct agent based on this configuration.
+
+### Adjust Step Limits
+
+Modify the maximum iteration count in [agents.py:169](agents.py#L169):
+
+```python
+if step_count >= 25:  # Change this value
+    # ...
+```
+
+### Customize Tools
+
+Add or modify tools in [custom_tools.py](custom_tools.py) using the `@tool` decorator:
+
+```python
+from langchain_core.tools import tool
+
+@tool
+def my_custom_tool(param: str) -> str:
+    """Tool description for the LLM."""
+    # Implementation
+    return result
+```
+
+## API Integration
+
+The agent integrates with the GAIA benchmark API:
+
+- **Questions Endpoint**: `https://agents-course-unit4-scoring.hf.space/questions`
+- **Submit Endpoint**: `https://agents-course-unit4-scoring.hf.space/submit`
+
+Questions may include file references which are automatically fetched from:
+- Local `files/` directory (if available)
+- Remote API endpoint (fallback)
+
+## Testing
+
+### Local Ground Truth Verification
+
+The app includes local verification against ground truth data in `files/metadata.jsonl`. This allows you to test your agent's performance before submitting to the evaluation server.
+
+### Test Mode
+
+Run specific questions in test mode by modifying [app.py:196](app.py#L196):
+
+```python
+my_questions = [
+    {
+        "question": my_questions_data[i]["question"],
+        "file_name": my_questions_data[i].get("file_name")
+    }
+    for i in (0, 5, 17) if i < len(my_questions_data)  # Customize indices
+]
+```
+
+## Performance Considerations
+
+- **Timeout**: Agent has 180-second timeout per question
+- **Step Limit**: Maximum 25 reasoning steps to prevent infinite loops
+- **Tool Timeouts**: Individual tools have their own timeout settings
+- **Cost**: Uses Google Gemini API (gemini-2.5-flash model)
+
+## Deployment
+
+### Hugging Face Spaces
+
+This project is designed to run on Hugging Face Spaces:
+
+1. Create a new Space on Hugging Face
+2. Set SDK to Gradio (version 6.2.0+)
+3. Add environment variables: `GOOGLE_API_KEY`, `SPACE_ID`, `SPACE_HOST`
+4. Enable OAuth authentication
+
+The app will automatically detect the Hugging Face environment and configure URLs accordingly.
+
+### Local Deployment
+
+Simply run `python app.py` locally. The app will detect it's not in a Hugging Face Space and adjust behavior accordingly.
+
+## Troubleshooting
+
+### Common Issues
+
+**"GOOGLE_API_KEY not found"**
+- Set the environment variable: `export GOOGLE_API_KEY="your_key"`
+
+**Audio parsing fails**
+- Install ffmpeg: `apt-get install ffmpeg` (Linux) or `brew install ffmpeg` (macOS)
+
+**Tool timeouts**
+- Adjust timeout values in respective tool functions in [custom_tools.py](custom_tools.py)
+
+**Agent exceeds step limit**
+- Increase limit in [agents.py:169](agents.py#L169) or optimize tool usage in system prompt
+
+## Contributing
+
+Contributions are welcome! Areas for improvement:
+- Add more tools (database access, code execution, etc.)
+- Move the Benchmark from 50% to 100%
+- Improve error handling and retry logic
+- Try with smaller LLMs
+- Make it work with Ollama
+
+## License
+
+This project is open-source and available under the MIT License.
+
+## Acknowledgments
+
+- Built for the GAIA (General AI Assistants) benchmark
+- Uses Google's Gemini model via LangChain
+- LangGraph framework by LangChain
+- Gradio for web interface
+
+## Contact
+
+For questions, issues, or suggestions, please open an issue on GitHub.

agent_runner.py

@@ -0,0 +1,64 @@
+"""Agent execution functionality for running questions through the GAIA agent."""
+
+from typing import Optional, Tuple, List, Dict
+from colorama import Fore, Style
+from agents import MyGAIAAgents
+import config
+
+class AgentRunner:
+    """Handles agent execution and question processing.
+    """
+
+    def __init__(self, active_agent: str = None):
+        """Initialize the AgentRunner.
+
+        Args:
+            active_agent: The agent type to use. If None, uses config.ACTIVE_AGENT.
+        """
+        self.agent = None
+        self.active_agent = active_agent
+
+    def _initialize_agent(self) -> bool:
+        """Initialize the agent. Returns True if successful."""
+        try:
+            self.agent = MyGAIAAgents(active_agent=self.active_agent)
+            return True
+        except Exception as e:
+            print(f"{Fore.RED}Error instantiating agent: {e}{Style.RESET_ALL}")
+            return False
+
+    def run_on_questions(self, questions_data: List[Dict]) -> Optional[List[Tuple]]:
+        """Run agent on a list of questions and return results."""
+        if not self._initialize_agent():
+            return None
+
+        results = []
+        total = len(questions_data)
+        print(f"{Fore.CYAN}Running agent on {total} questions...{Style.RESET_ALL}")
+
+        for idx, item in enumerate(questions_data, 1):
+            task_id = item.get("task_id")
+            question_text = item.get("question")
+            file_name = item.get("file_name")
+
+            if not task_id or question_text is None:
+                print(f"\n{Fore.YELLOW}Skipping item with missing task_id or question: {item}{Style.RESET_ALL}\n")
+                continue
+
+            print(f"\n{'#' * config.SEPARATOR_WIDTH}")
+            print(f"{Fore.CYAN}Processing Question {idx}/{total} - Task ID: {task_id}{Style.RESET_ALL}")
+            print(f"{'#' * config.SEPARATOR_WIDTH}")
+
+            try:
+                answer = self.agent(question_text, file_name=file_name)
+
+                print(f"\n{Fore.GREEN}[RESULT] Task ID: {task_id}{Style.RESET_ALL}")
+                print(f"Question: {question_text[:config.QUESTION_PREVIEW_LENGTH]}{'...' if len(question_text) > config.QUESTION_PREVIEW_LENGTH else ''}")
+                print(f"Answer: {answer}")
+                results.append((task_id, question_text, answer))
+            except Exception as e:
+                print(f"{Fore.RED}[ERROR] Exception running agent on task {task_id}: {e}{Style.RESET_ALL}")
+                error_msg = f"AGENT ERROR: {str(e)[:config.ERROR_MESSAGE_LENGTH]}"
+                results.append((task_id, question_text, error_msg))
+
+        return results

agents.py

@@ -0,0 +1,51 @@
+"""Agent wrapper module for GAIA Benchmark."""
+
+import config
+
+# All agents are imported lazily to avoid loading unnecessary dependencies
+# and suppress warnings from unused agent implementations
+
+
+class MyGAIAAgents:
+    """Wrapper class to manage multiple agent implementations.
+
+    This class provides a unified interface for different agent types.
+    The active agent is determined by the ACTIVE_AGENT configuration or constructor parameter.
+    """
+
+    def __init__(self, active_agent: str = None):
+        """Initialize the wrapper with the active agent.
+
+        Args:
+            active_agent: The agent type to use. If None, uses config.ACTIVE_AGENT.
+                         Valid values: config.AGENT_LANGGRAPH, config.AGENT_REACT_LANGGRAPH
+        """
+        if active_agent is None:
+            active_agent = config.ACTIVE_AGENT
+
+        if active_agent == config.AGENT_LANGGRAPH:
+            from langgraphagent import LangGraphAgent
+            self.agent = LangGraphAgent()
+        elif active_agent == config.AGENT_REACT_LANGGRAPH:
+            from reactlanggraphagent import ReActLangGraphAgent
+            self.agent = ReActLangGraphAgent()
+        elif active_agent == config.AGENT_LLAMAINDEX:
+            from llamaindexagent import LlamaIndexAgent
+            self.agent = LlamaIndexAgent()
+        else:
+            # Default to LangGraph if unknown agent type
+            print(f"[WARNING] Unknown agent type '{active_agent}', defaulting to {config.AGENT_LANGGRAPH}")
+            from langgraphagent import LangGraphAgent
+            self.agent = LangGraphAgent()
+
+    def __call__(self, question: str, file_name: str = None) -> str:
+        """Invoke the active agent with the given question.
+
+        Args:
+            question: The question to answer
+            file_name: Optional file name if the question references a file
+
+        Returns:
+            The agent's answer as a string
+        """
+        return self.agent(question, file_name)

app.py

@@ -0,0 +1,383 @@
+import os
+import argparse
+import requests
+import pandas as pd
+import json
+import time
+import warnings
+import logging
+from enum import Enum
+from colorama import init
+
+# Initialize colorama for Windows compatibility
+init(autoreset=True)
+
+# Suppress asyncio event loop cleanup warnings (common on HF Spaces)
+warnings.filterwarnings('ignore', message='.*Invalid file descriptor.*')
+logging.getLogger('asyncio').setLevel(logging.ERROR)
+
+# Import configuration
+import config
+
+# Agent-related code is imported via agent_runner module
+# Import Gradio UI creation function
+from gradioapp import create_ui
+# Import scoring function for answer verification
+from scorer import question_scorer
+
+# Import new utilities
+from question_loader import QuestionLoader
+from result_formatter import ResultFormatter
+from agent_runner import AgentRunner
+from validators import InputValidator, ValidationError
+from utils import retry_with_backoff
+
+# --- Run Modes ---
+class RunMode(Enum):
+    UI = "ui"   # Gradio UI mode
+    CLI = "cli" # Command-line test mode
+
+
+@retry_with_backoff(max_retries=3, initial_delay=2.0)
+def _submit_to_server(submit_url: str, submission_data: dict) -> dict:
+    """Internal function to submit to server (with retries)."""
+    response = requests.post(submit_url, json=submission_data, timeout=config.SUBMIT_TIMEOUT)
+    response.raise_for_status()
+    return response.json()
+
+def submit_and_score(username: str, results: list) -> str:
+    """
+    Submit answers to the GAIA scoring server and return status message.
+
+    Args:
+        username: Hugging Face username for submission
+        results: List of tuples (task_id, question_text, answer)
+
+    Returns:
+        str: Status message (success or error details)
+    """
+    # Validate username
+    try:
+        username = InputValidator.validate_username(username)
+    except ValidationError as e:
+        error_msg = f"Invalid username: {e}"
+        print(error_msg)
+        return error_msg
+
+    # Format results for API submission
+    answers_payload = ResultFormatter.format_for_api(results)
+
+    if not answers_payload:
+        error_msg = "No answers to submit."
+        print(error_msg)
+        return error_msg
+
+    space_id = config.SPACE_ID
+    submit_url = f"{config.DEFAULT_API_URL}/submit"
+    agent_code = f"https://huggingface.co/spaces/{space_id}/tree/main"
+
+    # Prepare submission data
+    submission_data = {
+        "username": username,
+        "agent_code": agent_code,
+        "answers": answers_payload
+    }
+
+    print(f"\n{'=' * config.SEPARATOR_WIDTH}")
+    print(f"Submitting {len(answers_payload)} answers for user '{username}'...")
+    print(f"{'=' * config.SEPARATOR_WIDTH}\n")
+
+    # Submit to server
+    print(f"Submitting to: {submit_url}")
+    try:
+        result_data = _submit_to_server(submit_url, submission_data)
+
+        final_status = (
+            f"Submission Successful!\n"
+            f"User: {result_data.get('username')}\n"
+            f"Overall Score: {result_data.get('score', 'N/A')}% "
+            f"({result_data.get('correct_count', '?')}/{result_data.get('total_attempted', '?')} correct)\n"
+            f"Message: {result_data.get('message', 'No message received.')}"
+        )
+        print("Submission successful.")
+        return final_status
+
+    except requests.exceptions.HTTPError as e:
+        error_detail = f"Server responded with status {e.response.status_code}."
+        try:
+            error_json = e.response.json()
+            error_detail += f" Detail: {error_json.get('detail', e.response.text)}"
+        except requests.exceptions.JSONDecodeError:
+            error_detail += f" Response: {e.response.text[:500]}"
+        status_message = f"Submission Failed: {error_detail}"
+        print(status_message)
+        return status_message
+
+    except requests.exceptions.Timeout:
+        status_message = "Submission Failed: The request timed out."
+        print(status_message)
+        return status_message
+
+    except requests.exceptions.RequestException as e:
+        status_message = f"Submission Failed: Network error - {e}"
+        print(status_message)
+        return status_message
+
+    except Exception as e:
+        status_message = f"An unexpected error occurred during submission: {e}"
+        print(status_message)
+        return status_message
+
+
+def run_and_submit_all(username: str, active_agent: str = None) -> tuple:
+    """
+    Fetches all questions, runs the GAIA agent on them, submits all answers,
+    and displays the results.
+
+    Args:
+        username: Hugging Face username for submission
+        active_agent: The agent type to use (default: config.AGENT_LANGGRAPH)
+
+    Returns:
+        tuple: (status_message: str, results_df: pd.DataFrame)
+    """
+    # Fetch questions from API (always online for submission)
+    try:
+        questions_data = QuestionLoader().get_questions(test_mode=False)
+    except Exception as e:
+        return f"Error loading questions: {e}", None
+
+    # Validate questions data
+    try:
+        questions_data = InputValidator.validate_questions_data(questions_data)
+    except ValidationError as e:
+        return f"Invalid questions data: {e}", None
+
+    results = AgentRunner(active_agent=active_agent).run_on_questions(questions_data)
+
+    if results is None:
+        return "Error initializing agent.", None
+
+    # Submit answers and get score (formatting happens inside submit_and_score)
+    status_message = submit_and_score(username, results)
+
+    # Format results for UI display
+    results_for_display = ResultFormatter.format_for_display(results)
+    results_df = pd.DataFrame(results_for_display)
+    return status_message, results_df
+
+def _load_ground_truth(file_path: str = config.METADATA_FILE) -> dict:
+    """Load ground truth data indexed by task_id.
+
+    Args:
+        file_path: Path to the metadata file
+
+    Returns:
+        dict: Mapping of task_id -> {"question": str, "answer": str}
+    """
+    truth_mapping = {}
+    try:
+        with open(file_path, 'r', encoding='utf-8') as f:
+            for line in f:
+                data = json.loads(line)
+                task_id = data.get("task_id")
+                question = data.get("Question")
+                answer = data.get("Final answer")
+                if task_id and answer:
+                    truth_mapping[task_id] = {
+                        "question": question,
+                        "answer": answer
+                    }
+    except Exception as e:
+        print(f"Error loading ground truth: {e}")
+    return truth_mapping
+
+def _verify_answers(results: list, log_output: list, runtime: tuple = None) -> None:
+    """Verify answers against ground truth using the official GAIA scorer.
+
+    Args:
+        results: List of tuples (task_id, question_text, answer)
+        log_output: List to append verification results to
+        runtime: Optional tuple of (minutes, seconds) for total runtime
+    """
+    ground_truth = _load_ground_truth()
+    log_output.append("\n=== Verification Results ===")
+
+    correct_count = 0
+    total_count = 0
+
+    for task_id, question_text, answer in results:
+        if task_id in ground_truth:
+            truth_data = ground_truth[task_id]
+            correct_answer = truth_data["answer"]
+
+            # Use the official GAIA question_scorer for comparison
+            # This handles numbers, lists, and strings with proper normalization
+            is_correct = question_scorer(str(answer), str(correct_answer))
+
+            if is_correct:
+                correct_count += 1
+            total_count += 1
+
+            log_output.append(f"Task ID: {task_id}")
+            log_output.append(f"Question: {question_text[:config.ERROR_MESSAGE_LENGTH]}...")
+            log_output.append(f"Expected: {correct_answer}")
+            log_output.append(f"Got: {answer}")
+            log_output.append(f"Match: {'✓ Correct' if is_correct else '✗ Incorrect'}\n")
+        else:
+            log_output.append(f"Task ID: {task_id}")
+            log_output.append(f"Question: {question_text[:config.ERROR_MESSAGE_LENGTH]}...")
+            log_output.append(f"No ground truth found.\n")
+
+    # Add summary statistics
+    if total_count > 0:
+        accuracy = (correct_count / total_count) * 100
+        log_output.append("=" * config.SEPARATOR_WIDTH)
+        log_output.append(f"SUMMARY: {correct_count}/{total_count} correct ({accuracy:.1f}%)")
+        if runtime:
+            minutes, seconds = runtime
+            log_output.append(f"Runtime: {minutes}m {seconds}s")
+        log_output.append("=" * config.SEPARATOR_WIDTH)
+
+def run_test_code(filter=None, active_agent=None) -> pd.DataFrame:
+    """Run test code on selected questions.
+
+    Args:
+        filter: Optional tuple/list of question indices to test (e.g., (4, 7, 15)).
+                If None, processes all questions.
+        active_agent: Optional agent type to use (e.g., "LangGraph", "ReActLangGraph", "LLamaIndex").
+                      If None, uses config.ACTIVE_AGENT.
+
+    Returns:
+        pd.DataFrame: Results and verification output
+    """
+    start_time = time.time()
+    logs_for_display = []
+    logs_for_display.append("=== Processing Example Questions One by One ===")
+
+    # Fetch questions (OFFLINE for testing)
+    try:
+        questions_data = QuestionLoader().get_questions(test_mode=True)
+    except Exception as e:
+        return pd.DataFrame([f"Error loading questions: {e}"])
+
+    # Validate questions data
+    try:
+        questions_data = InputValidator.validate_questions_data(questions_data)
+    except ValidationError as e:
+        return pd.DataFrame([f"Invalid questions data: {e}"])
+
+    # Validate and apply filter
+    try:
+        filter = InputValidator.validate_filter_indices(filter, len(questions_data))
+    except ValidationError as e:
+        return pd.DataFrame([f"Invalid filter: {e}"])
+
+    # Apply filter or use all questions
+    if filter is not None:
+        questions_to_process = [questions_data[i] for i in filter]
+        logs_for_display.append(f"Testing {len(questions_to_process)} selected questions (indices: {filter})")
+    else:
+        questions_to_process = questions_data
+        logs_for_display.append(f"Testing all {len(questions_to_process)} questions")
+
+    results = AgentRunner(active_agent=active_agent).run_on_questions(questions_to_process)
+
+    if results is None:
+        return pd.DataFrame(["Error initializing agent."])
+
+    logs_for_display.append("\n=== Completed Example Questions ===")
+
+    # Calculate runtime
+    elapsed_time = time.time() - start_time
+    minutes = int(elapsed_time // 60)
+    seconds = int(elapsed_time % 60)
+
+    _verify_answers(results, logs_for_display, runtime=(minutes, seconds))
+    return pd.DataFrame(logs_for_display)
+
+
+def main() -> None:
+    """Main entry point for the application."""
+    parser = argparse.ArgumentParser(description="Run the agent application.")
+    parser.add_argument("--test", type=str, nargs='?', const='default', help="Run local tests on selected questions and exit. Optionally provide comma-separated question indices (e.g., --test 2,4,6). If no indices provided, uses default test questions.")
+    parser.add_argument("--testall", action="store_true", help="Run local tests on all questions and exit.")
+    parser.add_argument("--agent", type=str, choices=['langgraph', 'reactlangg', 'llamaindex'], help="Agent to use in CLI mode (case-insensitive). Options: langgraph, react langgraph, llamaindex. Default: uses config.ACTIVE_AGENT")
+    args = parser.parse_args()
+
+    # Map agent name to config constant (case-insensitive)
+    agent_mapping = {
+        'langgraph': config.AGENT_LANGGRAPH,
+        'reactlangg': config.AGENT_REACT_LANGGRAPH,
+        'llamaindex': config.AGENT_LLAMAINDEX,
+    }
+
+    active_agent = None
+    if args.agent:
+        agent_key = args.agent.lower()
+        active_agent = agent_mapping.get(agent_key)
+        if not active_agent:
+            print(f"Error: Unknown agent '{args.agent}'. Valid options: langgraph, react, llamaindex")
+            return
+        print(f"[CLI] Using agent: {active_agent}")
+
+    print(f"\n{'-' * 30} App Starting {'-' * 30}")
+
+    # Determine run mode
+    run_mode = RunMode.CLI if (args.test or args.testall) else RunMode.UI
+
+    # Print environment info only in UI mode
+    if run_mode == RunMode.UI:
+        space_host = config.SPACE_HOST
+        space_id = config.SPACE_ID
+
+        if space_host:
+            print(f"[OK] SPACE_HOST found: {space_host}")
+            print(f"   Runtime URL should be: https://{space_host}")
+        else:
+            print("[INFO] SPACE_HOST environment variable not found (running locally?).")
+
+        if space_id:
+            print(f"[OK] SPACE_ID found: {space_id}")
+            print(f"   Repo URL: https://huggingface.co/spaces/{space_id}")
+            print(f"   Repo Tree URL: https://huggingface.co/spaces/{space_id}/tree/main")
+        else:
+            print("[INFO] SPACE_ID environment variable not found (running locally?). Repo URL cannot be determined.")
+
+    print(f"{'-' * (60 + len(' App Starting '))}\n")
+
+    # Execute based on run mode
+    if run_mode == RunMode.UI:
+        print("Launching Gradio Interface for Basic Agent Evaluation...")
+        grTestApp = create_ui(run_and_submit_all, run_test_code)
+        grTestApp.launch()
+
+    else:  # RunMode.CLI
+        # Determine test filter based on which CLI flag was used
+        if args.test:
+            # Check if custom indices were provided
+            if args.test == 'default':
+                # No indices provided, use default
+                test_filter = config.DEFAULT_TEST_FILTER
+            else:
+                # Parse comma-separated indices
+                try:
+                    test_filter = tuple(int(idx.strip()) for idx in args.test.split(','))
+                except ValueError:
+                    print(f"Error: Invalid test indices '{args.test}'. Must be comma-separated integers (e.g., 2,4,6)")
+                    return
+        else:  # args.testall
+            test_filter = None  # Test all questions
+
+        print(f"Running test code on {len(test_filter) if test_filter else 'ALL'} questions (CLI mode)...")
+        result = run_test_code(filter=test_filter, active_agent=active_agent)
+
+        # Print results
+        if isinstance(result, pd.DataFrame):
+            ResultFormatter.print_dataframe(result)
+        else:
+            print(result)
+
+
+if __name__ == "__main__":
+    main()

config.py

@@ -0,0 +1,63 @@
+"""Configuration settings for GAIA Benchmark Agent."""
+
+import os
+
+# API Configuration
+DEFAULT_API_URL = "https://agents-course-unit4-scoring.hf.space"
+AGENT_TIMEOUT_SECONDS = 180  # 3 minutes max per question
+
+# File Paths
+QUESTIONS_FILE = "files/questions.json"
+METADATA_FILE = "files/metadata.jsonl"
+FILES_DIR = "files"
+
+# API Timeouts (in seconds)
+FETCH_TIMEOUT = 15
+SUBMIT_TIMEOUT = 60
+WEBPAGE_TIMEOUT = 30
+
+# Test Configuration
+DEFAULT_TEST_FILTER = (4, 7, 15)  # Q2, Q5, Q8, Q16
+
+# Display Configuration
+QUESTION_PREVIEW_LENGTH = 200  # Characters to show in question preview
+ERROR_MESSAGE_LENGTH = 100  # Characters to show in error messages
+SEPARATOR_WIDTH = 60  # Width of separator lines
+
+# Environment Variables
+SPACE_HOST = os.getenv("SPACE_HOST")
+SPACE_ID = os.getenv("SPACE_ID")
+GOOGLE_API_KEY = os.getenv("GOOGLE_DESKGENIE_KEY")
+
+# Agent Type Constants
+AGENT_LANGGRAPH = "LangGraph"
+AGENT_REACT_LANGGRAPH = "ReActLangGraph"
+AGENT_LLAMAINDEX = "LLamaIndex"
+AGENT_SMOL = "SMOL"
+
+ACTIVE_AGENT = AGENT_LANGGRAPH  # Active agent to use by default
+
+# Model Configuration
+GEMINI_MODEL = "gemini-3.5-flash"
+GEMINI_TEMPERATURE = 0
+GEMINI_MAX_TOKENS = 1024
+
+ACTIVE_AGENT_LLM_MODEL = GEMINI_MODEL
+
+# Agent Step Limits
+# AGENT_STEP_LIMIT is the single source of truth — the max number of assistant
+# iterations (LLM + tool call) per question before the graph is force-terminated.
+# The agent forces a final bare-answer call one step BEFORE this limit.
+# AGENT_RECURSION_LIMIT is DERIVED so the invariant always holds: LangGraph's
+# recursion_limit must exceed 2x the step limit (each step ~= 2 graph nodes:
+# assistant + tools), plus a safety buffer.
+AGENT_STEP_LIMIT = 60
+AGENT_RECURSION_LIMIT = AGENT_STEP_LIMIT * 2 + 20
+
+# ArXiv timeout
+ARXIV_TIMEOUT_SECONDS = 30
+
+# Retry Configuration for 504 DEADLINE_EXCEEDED errors
+MAX_RETRIES = 3
+INITIAL_RETRY_DELAY = 2.0  # seconds
+RETRY_BACKOFF_FACTOR = 2.0

custom_tools.py

@@ -0,0 +1,827 @@
+import concurrent.futures
+from ddgs import DDGS
+from bs4 import BeautifulSoup
+import requests
+import re
+import io
+import os
+import subprocess
+import sys
+from google import genai
+from google.genai import types
+import config
+
+from langchain_community.document_loaders import WikipediaLoader
+from langchain_community.document_loaders import ArxivLoader
+from youtube_transcript_api import YouTubeTranscriptApi
+from pytube import extract
+from langchain_core.tools import tool
+
+import pandas as pd
+import speech_recognition as sr
+from pydub import AudioSegment
+from pypdf import PdfReader
+from io import BytesIO
+from markdownify import markdownify as md
+
+# ============================================================================
+# Shared HTTP headers
+# ============================================================================
+# Many sites (notably Wikipedia) return 403 to the default python-requests
+# User-Agent. Send a browser-like UA for all outbound page/file fetches.
+_HTTP_HEADERS = {
+    "User-Agent": (
+        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
+        "AppleWebKit/537.36 (KHTML, like Gecko) "
+        "Chrome/120.0.0.0 Safari/537.36 GAIA-Agent/1.0"
+    )
+}
+
+# ============================================================================
+# Per-question tool call counters (reset at start of each question)
+# ============================================================================
+_analyze_image_call_count = 0
+MAX_ANALYZE_IMAGE_CALLS = 2
+
+# Maps normalized websearch query -> its result string, for the current question.
+# Lets websearch detect and short-circuit verbatim repeated queries (loop guard).
+_websearch_seen_queries = {}
+
+
+def reset_tool_counters():
+    """Reset per-question tool counters. Call at the start of each new question."""
+    global _analyze_image_call_count
+    _analyze_image_call_count = 0
+    _websearch_seen_queries.clear()
+
+
+# ============================================================================
+# Helper Functions (must be defined before tools that use them)
+# ============================================================================
+
+def _sanitize_file_path(file_name: str) -> tuple:
+    """
+    Sanitize file name to prevent path traversal attacks.
+
+    Args:
+        file_name: The file name to sanitize
+
+    Returns:
+        tuple: (is_valid: bool, sanitized_name_or_error: str)
+    """
+    # Check for path traversal attempts
+    if '..' in file_name or file_name.startswith('/') or file_name.startswith('\\'):
+        return False, "Invalid file name: path traversal not allowed"
+
+    # Check for absolute paths (Windows and Unix)
+    if os.path.isabs(file_name):
+        return False, "Invalid file name: absolute paths not allowed"
+
+    # Normalize the path and ensure it doesn't escape the files directory
+    normalized = os.path.normpath(file_name)
+    if normalized.startswith('..') or os.path.isabs(normalized):
+        return False, "Invalid file name: path traversal detected"
+
+    return True, normalized
+
+def _get_file_content(file_name: str, mode: str = 'binary'):
+    """
+    Helper function to get file content from local filesystem or remote URL.
+
+    Args:
+        file_name: The file name (without 'files/' prefix)
+        mode: 'binary' for bytes, 'text' for string
+
+    Returns:
+        tuple: (success: bool, data: bytes/str or error_message: str)
+
+    NOTE — File source for GAIA benchmark question attachments:
+    The question files (.png, .mp3, .py, .xlsx, etc.) are NOT served by the
+    scoring API at agents-course-unit4-scoring.hf.space. A previous version of
+    this code defaulted to that URL, which caused silent 404 failures for any
+    question that referenced a file attachment.
+
+    The correct source is the HuggingFace dataset:
+        repo: gaia-benchmark/GAIA  (type: dataset)
+        path: 2023/validation/<file_name>
+
+    This function now tries sources in order:
+        1. Local files/ directory (cache)
+        2. HuggingFace dataset download (saves to files/ for future runs)
+        3. SPACE_HOST env var (only when deployed on HF Spaces)
+
+    To pre-download all question files manually, run:
+        python -c "
+        import json, os, shutil
+        from huggingface_hub import hf_hub_download
+        questions = json.load(open('files/questions.json', encoding='utf-8'))
+        for q in questions:
+            fn = q.get('file_name', '')
+            if fn and not os.path.exists(f'files/{fn}'):
+                src = hf_hub_download('gaia-benchmark/GAIA', f'2023/validation/{fn}', repo_type='dataset')
+                shutil.copy(src, f'files/{fn}')
+                print('Downloaded', fn)
+        "
+    """
+    # Sanitize file name first
+    is_valid, result = _sanitize_file_path(file_name)
+    if not is_valid:
+        return False, result
+
+    file_name = result  # Use sanitized name
+    file_path = f"files/{file_name}"
+
+    def _read(path: str):
+        if mode == 'binary':
+            with open(path, 'rb') as f:
+                return True, f.read()
+        else:
+            with open(path, 'r', encoding='utf-8') as f:
+                return True, f.read()
+
+    # 1. Local cache
+    if os.path.exists(file_path):
+        try:
+            return _read(file_path)
+        except Exception as e:
+            return False, f"Error reading local file: {e}"
+
+    # 2. HuggingFace GAIA dataset — downloads and caches locally
+    try:
+        import shutil
+        from huggingface_hub import hf_hub_download
+        print(f"[INFO] Downloading {file_name} from HuggingFace GAIA dataset...")
+        hf_local = hf_hub_download(
+            repo_id='gaia-benchmark/GAIA',
+            filename=f'2023/validation/{file_name}',
+            repo_type='dataset',
+        )
+        os.makedirs('files', exist_ok=True)
+        shutil.copy(hf_local, file_path)
+        print(f"[INFO] Cached to {file_path}")
+        return _read(file_path)
+    except Exception as e:
+        print(f"[WARNING] HuggingFace download failed for {file_name}: {e}")
+
+    # 3. SPACE_HOST fallback (only when explicitly deployed on a HF Space that serves files)
+    space_host = os.getenv("SPACE_HOST")
+    if space_host:
+        try:
+            if not space_host.startswith("http"):
+                file_url = f"https://{space_host}/files/{file_name}"
+            else:
+                file_url = f"{space_host}/files/{file_name}"
+            print(f"[INFO] Fetching {file_name} from {file_url}")
+            response = requests.get(file_url, timeout=30)
+            response.raise_for_status()
+            if mode == 'binary':
+                return True, response.content
+            else:
+                return True, response.text
+        except Exception as e:
+            print(f"[WARNING] SPACE_HOST fetch failed for {file_name}: {e}")
+
+    return False, f"Could not retrieve file '{file_name}' from any source."
+
+def _get_mime_type(file_name: str) -> str:
+    """Helper function to determine MIME type from file extension."""
+    ext = file_name.lower().split('.')[-1]
+    mime_types = {
+        'png': 'image/png',
+        'jpg': 'image/jpeg',
+        'jpeg': 'image/jpeg',
+        'gif': 'image/gif',
+        'webp': 'image/webp',
+        'bmp': 'image/bmp'
+    }
+    return mime_types.get(ext, 'image/png')
+
+# ============================================================================
+# Tools
+# ============================================================================
+
+@tool
+def calculate(operation: str, a: float, b: float) -> str:
+    """Perform a basic arithmetic operation on two numbers.
+
+    Args:
+        operation (str): One of 'add', 'subtract', 'multiply', 'divide', 'power', 'modulus'.
+        a (float): First number.
+        b (float): Second number.
+    """
+    op = (operation or "").strip().lower()
+    if op == "add":
+        return str(a + b)
+    elif op == "subtract":
+        return str(a - b)
+    elif op == "multiply":
+        return str(a * b)
+    elif op == "divide":
+        if b == 0:
+            return "Cannot divide by zero"
+        return str(a / b)
+    elif op == "power":
+        return str(a ** b)
+    elif op == "modulus":
+        return str(int(a) % int(b))
+    else:
+        return f"Unsupported operation '{operation}'. Use: add, subtract, multiply, divide, power, modulus."
+
+@tool
+def string_reverse(input_string: str) -> str:
+    """
+    Reverses the input string. Useful whenever a string seems to be non-sensical or
+    contains a lot of gibberish. This function can be used to reverse the string
+    and check if it makes more sense when reversed.
+
+    Args:
+        input_string (str): The string to reverse.
+
+    Returns:
+        str: The reversed string.
+    """
+    return input_string[::-1]
+
+
+@tool
+def websearch(query: str) -> str:
+    """This tool will search the web using DuckDuckGo.
+
+    Args:
+        query: The search query.
+    """
+
+    try:
+        print(f"websearch called: {query}")
+
+        # Loop guard: if this exact query (normalized) was already run for the
+        # current question, don't re-run it — repeating it returns nothing new.
+        # Return the prior results plus a nudge to change strategy or answer.
+        norm_query = " ".join(query.lower().split())
+        if norm_query in _websearch_seen_queries:
+            print("[WEBSEARCH] Duplicate query detected — returning cached result with nudge")
+            return (
+                "DUPLICATE SEARCH: You already ran this exact query earlier for this question, "
+                "so it returns no new information. Do NOT repeat it. Instead: try a substantially "
+                "different query, call get_webpage_content on a promising URL from earlier results, "
+                "or give your best answer now based on what you already have.\n\n"
+                f"Previous results for this query:\n{_websearch_seen_queries[norm_query]}"
+            )
+
+        with DDGS() as ddgs:
+            results = ddgs.text(query, max_results=5, timelimit='y')  # Limit to past year for faster results
+            if results:
+                print(f"websearch results: {len(results)}")
+                output = "\n\n".join([f"Title: {r['title']}\nURL: {r['href']}\nSnippet: {r['body']}" for r in results])
+            else:
+                output = "No results found. Try search with a different query."
+        _websearch_seen_queries[norm_query] = output
+        return output
+    except Exception as e:
+        return f"Search error (try again): {str(e)}"
+
+@tool
+def wiki_search(query: str) -> str:
+    """Search Wikipedia for a query and return maximum 3 results.
+
+    Args:
+        query: The search query."""
+    try:
+        print(f"wiki_search called: {query}")
+
+        search_docs = WikipediaLoader(query=query, load_max_docs=3).load()
+        formatted_search_docs = "\n\n---\n\n".join(
+            [
+                f'<Document source="{doc.metadata["source"]}" page="{doc.metadata.get("page", "")}"/>\n{doc.page_content}\n</Document>'
+                for doc in search_docs
+            ])
+        print(f"wiki_results: {len(formatted_search_docs)} characters")
+        return {"wiki_results": formatted_search_docs}
+    except Exception as e:
+        return f"Error performing wikipedia search: {e}. try again."
+
+@tool
+def arvix_search(query: str) -> str:
+    """Search Arxiv for a query and return maximum 3 result.
+
+    Args:
+        query: The search query."""
+    try:
+        print(f"arvix_search called: {query}")
+
+        with concurrent.futures.ThreadPoolExecutor(max_workers=1) as executor:
+            future = executor.submit(lambda: ArxivLoader(query=query, load_max_docs=3).load())
+            search_docs = future.result(timeout=config.ARXIV_TIMEOUT_SECONDS)
+
+        formatted_search_docs = "\n\n---\n\n".join(
+            [
+                f'<Document source="{doc.metadata["source"]}" page="{doc.metadata.get("page", "")}"/>\n{doc.page_content[:1000]}\n</Document>'
+                for doc in search_docs
+            ])
+
+        print(f"arvix_results: {len(formatted_search_docs)} characters")
+        return {"arvix_results": formatted_search_docs}
+    except concurrent.futures.TimeoutError:
+        return f"ArXiv timed out after {config.ARXIV_TIMEOUT_SECONDS}s — try websearch instead"
+    except Exception as e:
+        return f"Error performing arxiv search: {e}. try again."
+
+@tool
+def youtube_tool(youtube_url: str, question: str = "") -> str:
+    """Get the transcript of a YouTube video, or analyze it with AI to answer a question.
+
+    If question is provided, uses a multimodal AI model to analyze the video (handles visual
+    or audio content beyond just transcript). If question is empty, returns the raw transcript.
+
+    Args:
+        youtube_url (str): Full HTTPS URL of the YouTube video.
+        question (str): Optional question to answer about the video. If empty, returns raw transcript.
+    """
+    print(f"youtube_tool called: {youtube_url} question={question!r}")
+
+    if not question:
+        # Transcript-only path — no API key needed
+        try:
+            video_id = extract.video_id(youtube_url)
+            ytt_api = YouTubeTranscriptApi()
+            transcript = ytt_api.fetch(video_id)
+            txt = '\n'.join([s.text for s in transcript.snippets])
+            print(f"youtube_transcript: {len(txt)} characters")
+            return txt
+        except Exception as e:
+            msg = f"youtube_tool (transcript) failed: {e}"
+            print(msg)
+            return msg
+
+    # AI analysis path
+    try:
+        api_key = config.GOOGLE_API_KEY
+        if not api_key:
+            return "Error: GOOGLE_API_KEY environment variable not set"
+
+        client = genai.Client(api_key=api_key)
+        response = client.models.generate_content(
+            model=config.GEMINI_MODEL,
+            contents=[types.Content(
+                parts=[
+                    types.Part(file_data=types.FileData(file_uri=youtube_url)),
+                    types.Part(text=question)
+                ]
+            )],
+            config=types.GenerateContentConfig(
+                temperature=config.GEMINI_TEMPERATURE,
+                max_output_tokens=config.GEMINI_MAX_TOKENS,
+            )
+        )
+        return response.text or "(no response from model)"
+    except Exception as e:
+        error_msg = f"youtube_tool (AI analysis) failed: {str(e)[:config.QUESTION_PREVIEW_LENGTH]}"
+        print(error_msg)
+        return error_msg
+
+@tool
+def get_webpage_content(page_url: str) -> str:
+    """Load a web page and return it as markdown if possible
+
+    Args:
+        page_url (str): the URL of web page to get
+
+    Returns:
+        str: The content of the page(s).
+   """
+
+    try:
+        print(f"get_web_page_content called: with url {page_url}")
+        r = requests.get(page_url, timeout=30, headers=_HTTP_HEADERS)  # Add 30s timeout
+        r.raise_for_status()
+        text = ""
+        # special case if page is a PDF file
+        if r.headers.get('Content-Type', '') == 'application/pdf':
+            pdf_file = BytesIO(r.content)
+            reader = PdfReader(pdf_file)
+            for page in reader.pages:
+                text += page.extract_text()
+        else:
+            soup = BeautifulSoup((r.text), 'html.parser')
+            if soup.body:
+                # convert to markdown
+                text = md(str(soup.body))
+            else:
+                # return the raw content
+                text = r.text
+        print(f"webpage_content: {len(text)} characters")
+        return text
+    except Exception as e:
+        return f"get_webpage_content failed: {e}"
+
+@tool
+def read_file(file_name: str, sheet_name: str = "") -> str:
+    """Read a file from the files directory and return its content.
+
+    Supported formats:
+    - .xlsx / .csv  → returned as a Markdown table
+    - .py / .txt / .md / .json / .jsonl → returned as raw text
+
+    Args:
+        file_name (str): Name of the file (e.g., 'data.xlsx'). Do not include 'files/' prefix.
+        sheet_name (str): For Excel files, the sheet name to read. Leave empty to read the first sheet.
+    """
+    print(f"read_file called: {file_name}")
+    ext = file_name.rsplit(".", 1)[-1].lower() if "." in file_name else ""
+
+    if ext in ("xlsx", "xls"):
+        success, data = _get_file_content(file_name, mode='binary')
+        if not success:
+            return f"Error: {data}"
+        assert isinstance(data, bytes)
+        try:
+            df = pd.read_excel(BytesIO(data), sheet_name=sheet_name or 0)
+            return df.to_markdown(index=False)
+        except Exception as e:
+            return f"Error reading Excel file: {e}"
+
+    if ext == "csv":
+        success, data = _get_file_content(file_name, mode='binary')
+        if not success:
+            return f"Error: {data}"
+        assert isinstance(data, bytes)
+        try:
+            df = pd.read_csv(BytesIO(data))
+            return df.to_markdown(index=False)
+        except Exception as e:
+            return f"Error reading CSV file: {e}"
+
+    # Text-based formats
+    if ext in ("py", "txt", "md", "json", "jsonl", ""):
+        success, data = _get_file_content(file_name, mode='text')
+        if not success:
+            return f"Error: {data}"
+        return data
+
+    return f"Unsupported file type '.{ext}'. Supported: xlsx, xls, csv, py, txt, md, json, jsonl."
+
+@tool
+def parse_audio_file(file_name: str) -> str:
+    """
+    Transcribes audio from an MP3 file into text.
+    Use this tool to extract speech/text from audio files.
+
+    Args:
+        file_name (str): The name of the MP3 file (e.g., 'audio.mp3'). Do not include the 'files/' prefix.
+
+    Returns:
+        str: The transcribed text.
+    """
+
+    try:
+        print(f"parse_audio_file called: with file {file_name}")
+
+        # Get file content using helper function
+        success, data = _get_file_content(file_name, mode='binary')
+        if not success:
+            return f"Error: Failed to read audio file. {data}"
+
+        # Load audio from bytes
+        audio = AudioSegment.from_file(io.BytesIO(data), format="mp3")
+        # SpeechRecognition works best with WAV data so we to WAV format in memory
+        wav_data = io.BytesIO()
+        audio.export(wav_data, format="wav")
+        wav_data.seek(0)  # Rewind the buffer to the beginning
+
+        # Now we directly process the WAV data
+        recognizer = sr.Recognizer()
+        with sr.AudioFile(wav_data) as source:
+            audio_data = recognizer.record(source)
+        text = recognizer.recognize_google(audio_data)
+        return text
+
+    except sr.RequestError as e:
+        return f"Error: Could not request results from Google Web Speech API; {e}"
+    except Exception as e:
+        if "ffmpeg" in str(e).lower() or "avlib" in str(e).lower():
+            return f"Error: Failed to process audio. Reason: {e}. Ensure ffmpeg is installed and in your system's PATH."
+        return f"Error: Failed to parse the audio file. Reason: {e}"
+
+@tool
+def analyze_image(question: str, file_name: str) -> str:
+    """
+    Analyzes an image file and answers a specific question about it using AI vision.
+    Use this tool when you need to understand image content (e.g., chess positions, diagrams, photos).
+
+    Args:
+        question (str): The question you want answered about the image.
+        file_name (str): The name of the image file (e.g., 'image.png'). Do not include the 'files/' prefix.
+
+    Returns:
+        str: The answer to the question based on the image analysis.
+    """
+
+    global _analyze_image_call_count
+    _analyze_image_call_count += 1
+    print(f"analyze_image called: {file_name} with question: {question}")
+    if _analyze_image_call_count > MAX_ANALYZE_IMAGE_CALLS:
+        return (
+            f"ERROR: analyze_image has already been called {_analyze_image_call_count - 1} times. "
+            f"MAXIMUM is {MAX_ANALYZE_IMAGE_CALLS}. "
+            "Do NOT call analyze_image again. Commit to the chess position already described and use "
+            "execute_python with the chess library to find the winning move."
+        )
+
+    try:
+        api_key = config.GOOGLE_API_KEY
+        if not api_key:
+            return "Error: GOOGLE_API_KEY environment variable not set"
+
+        # Get file content using helper function
+        success, image_data = _get_file_content(file_name, mode='binary')
+        if not success:
+            return f"Error: Failed to read image file. {image_data}"
+
+        client = genai.Client(api_key=api_key)
+
+        # Use Gemini vision model with image data
+        response = client.models.generate_content(
+            model=config.GEMINI_MODEL,
+            contents=[types.Content(
+                parts=[
+                    types.Part(inline_data=types.Blob(
+                        mime_type=_get_mime_type(file_name),
+                        data=image_data
+                    )),
+                    types.Part(text=question)
+                ]
+            )],
+            config=types.GenerateContentConfig(
+                temperature=config.GEMINI_TEMPERATURE,
+                max_output_tokens=config.GEMINI_MAX_TOKENS,
+            )
+        )
+        return response.text
+
+    except Exception as e:
+        error_msg = f"Error analyzing image: {str(e)[:config.QUESTION_PREVIEW_LENGTH]}"
+        print(error_msg)
+        return error_msg
+
+
+@tool
+def classical_cipher(cipher_type: str, mode: str, text: str, keyword: str = "", period: int = 5) -> str:
+    """Encrypt or decrypt common classical ciphers.
+
+    Supported ciphers: playfair, bifid.
+
+    Args:
+        cipher_type (str): Cipher family: 'playfair' or 'bifid'.
+        mode (str): 'encrypt' or 'decrypt'.
+        text (str): Input text (letters only; j is mapped to i).
+        keyword (str): Key phrase used to build the 5x5 square.
+        period (int): Bifid period (ignored for Playfair). Default is 5.
+    """
+    ctype = (cipher_type or "").strip().lower()
+    op = (mode or "").strip().lower()
+    if ctype not in {"playfair", "bifid"}:
+        return "Unsupported cipher_type. Use 'playfair' or 'bifid'."
+    if op not in {"encrypt", "decrypt"}:
+        return "Unsupported mode. Use 'encrypt' or 'decrypt'."
+    if period <= 0:
+        return "Invalid period. Use a positive integer."
+
+    alphabet = "abcdefghiklmnopqrstuvwxyz"
+
+    def _normalize(s: str) -> str:
+        return re.sub(r"[^a-z]", "", (s or "").lower().replace("j", "i"))
+
+    def _build_square(key: str):
+        seen = []
+        for c in _normalize(key) + alphabet:
+            if c not in seen:
+                seen.append(c)
+        sq = [seen[i * 5:(i + 1) * 5] for i in range(5)]
+        pos = {c: (r, cidx) for r, row in enumerate(sq) for cidx, c in enumerate(row)}
+        inv = {(r, cidx): ch for r, row in enumerate(sq) for cidx, ch in enumerate(row)}
+        return sq, pos, inv
+
+    sq, pos, inv = _build_square(keyword)
+    normalized = _normalize(text)
+    if not normalized:
+        return ""
+
+    if ctype == "playfair":
+        if len(normalized) % 2 != 0:
+            normalized = normalized + "x"
+        d = -1 if op == "decrypt" else 1
+        out = []
+        for i in range(0, len(normalized), 2):
+            a, b = normalized[i], normalized[i + 1]
+            ra, ca = pos[a]
+            rb, cb = pos[b]
+            if ra == rb:
+                out.append(sq[ra][(ca + d) % 5])
+                out.append(sq[rb][(cb + d) % 5])
+            elif ca == cb:
+                out.append(sq[(ra + d) % 5][ca])
+                out.append(sq[(rb + d) % 5][cb])
+            else:
+                out.append(sq[ra][cb])
+                out.append(sq[rb][ca])
+        return "".join(out)
+
+    # bifid
+    if op == "encrypt":
+        out = []
+        for i in range(0, len(normalized), period):
+            block = normalized[i:i + period]
+            rows, cols = [], []
+            for ch in block:
+                r, c = pos[ch]
+                rows.append(r + 1)
+                cols.append(c + 1)
+            nums = rows + cols
+            for j in range(0, len(nums), 2):
+                out.append(inv[(nums[j] - 1, nums[j + 1] - 1)])
+        return "".join(out)
+
+    out = []
+    for i in range(0, len(normalized), period):
+        block = normalized[i:i + period]
+        nums = []
+        for ch in block:
+            r, c = pos[ch]
+            nums.extend([r + 1, c + 1])
+        half = len(block)
+        rows, cols = nums[:half], nums[half:]
+        for rr, cc in zip(rows, cols):
+            out.append(inv[(rr - 1, cc - 1)])
+    return "".join(out)
+
+
+@tool
+def execute_python(code: str) -> str:
+    """Execute a Python code snippet and return its stdout output.
+
+    Use this for precise computations the LLM cannot do reliably:
+    counting characters, implementing algorithms (ciphers, prime sieves),
+    math calculations, data transformations, etc.
+
+    Args:
+        code (str): Valid Python 3 code. Use print() to produce output.
+                    Do not read/write files or make network calls from within the code.
+    """
+    timeout = 30
+    try:
+        result = subprocess.run(
+            [sys.executable, "-c", code],
+            capture_output=True,
+            text=True,
+            timeout=timeout,
+        )
+        if result.returncode == 0:
+            return result.stdout.strip() or "(no output)"
+        return f"Exit {result.returncode}:\n{result.stderr.strip()}"
+    except subprocess.TimeoutExpired:
+        return f"Execution timed out after {timeout}s"
+    except Exception as e:
+        return f"execute_python failed: {e}"
+
+
+@tool
+def http_request(method: str, url: str, headers_json: str = "{}", body_json: str = "{}") -> str:
+    """Make an HTTP request with a custom method, headers, and JSON body.
+
+    Use this for POST, DELETE, or authenticated GET requests that require
+    custom headers (e.g. Authorization: Bearer ...) or a request body.
+
+    Args:
+        method (str): HTTP method — 'GET', 'POST', or 'DELETE'.
+        url (str): The full URL to call.
+        headers_json (str): JSON object of request headers, e.g. '{"Authorization": "Bearer TOKEN"}'.
+        body_json (str): JSON object for the request body (POST only). Use '{}' for empty body.
+
+    Returns:
+        str: Response body as text, prefixed with the HTTP status code.
+    """
+    import json
+    method = method.upper()
+    try:
+        headers = json.loads(headers_json)
+    except Exception as e:
+        return f"Invalid headers_json: {e}"
+    try:
+        body = json.loads(body_json)
+    except Exception as e:
+        return f"Invalid body_json: {e}"
+
+    try:
+        if method == "GET":
+            r = requests.get(url, headers=headers, timeout=30)
+        elif method == "POST":
+            r = requests.post(url, headers=headers, json=body, timeout=30)
+        elif method == "DELETE":
+            r = requests.delete(url, headers=headers, timeout=30)
+        else:
+            return f"Unsupported method '{method}'. Use GET, POST, or DELETE."
+        try:
+            content = json.dumps(r.json(), ensure_ascii=False)
+        except ValueError:
+            content = r.text
+        return f"HTTP {r.status_code}\n{content}"
+    except Exception as e:
+        return f"http_request failed ({method} {url}): {e}"
+
+
+@tool
+def download_file(url: str, file_name: str) -> str:
+    """Download a binary file from a URL and save it to the files directory.
+
+    Use this before calling read_file, parse_audio_file,
+    or analyze_image on files fetched from an API.
+    After downloading, call the appropriate tool with the same file_name.
+
+    Args:
+        url (str): The full URL of the file to download.
+        file_name (str): Local file name to save as (e.g. 'data.xlsx', 'audio.mp3').
+                         Must not contain path separators or '..'.
+    """
+    if "/" in file_name or "\\" in file_name or ".." in file_name:
+        return "Invalid file_name: path separators and '..' are not allowed."
+
+    try:
+        r = requests.get(url, timeout=60, headers=_HTTP_HEADERS)
+        r.raise_for_status()
+    except Exception as e:
+        return f"download_file failed (fetch): {e}"
+
+    os.makedirs(config.FILES_DIR, exist_ok=True)
+    dest = os.path.join(config.FILES_DIR, file_name)
+    try:
+        with open(dest, "wb") as f:
+            f.write(r.content)
+        return f"Downloaded {len(r.content)} bytes → {dest}"
+    except Exception as e:
+        return f"download_file failed (write): {e}"
+
+
+@tool
+def ask_advisor(question: str) -> str:
+    """Consult a more powerful AI model when you are stuck or uncertain after 2+ failed attempts.
+
+    Describe what you are trying to solve and what you have already tried.
+    The advisor returns a concise recommendation (2-3 sentences) to guide your next step.
+    Use sparingly — only for genuinely hard reasoning or planning problems, not for tool failures.
+
+    Args:
+        question (str): A clear description of the problem and what approaches you have already tried.
+    """
+    try:
+        api_key = config.GOOGLE_API_KEY
+        if not api_key:
+            return "Error: GOOGLE_API_KEY not configured"
+        client = genai.Client(api_key=api_key)
+        response = client.models.generate_content(
+            model=config.GEMINI_MODEL,
+            contents=question,
+            config=types.GenerateContentConfig(
+                system_instruction=(
+                    "You are an expert advisor for an AI agent that is stuck on a search or reasoning problem. "
+                    "Give a concise, actionable recommendation in 2-3 sentences about what to search for or how to reason. "
+                    "Do NOT suggest installing Python packages or software. "
+                    "Do NOT suggest writing code. "
+                    "Only give search strategy or reasoning guidance."
+                ),
+                temperature=0,
+            )
+        )
+        return response.text or "Advisor returned no response."
+    except Exception as e:
+        return f"Advisor unavailable: {e}"
+
+
+# ============================================================================
+# Tools List
+# ============================================================================
+
+
+def get_custom_tools_list() -> list:
+    """Get list of all custom tools for the agent.
+
+    Returns:
+        list: List of tool functions
+    """
+    tools = [
+        calculate,
+        string_reverse,
+        websearch,
+        wiki_search,
+        arvix_search,
+        youtube_tool,
+        get_webpage_content,
+        read_file,
+        parse_audio_file,
+        analyze_image,
+        classical_cipher,
+        execute_python,
+        ask_advisor,
+        http_request,
+        download_file,
+    ]
+    return tools

files/questions.json

@@ -0,0 +1 @@
+[{"task_id":"8e867cd7-cff9-4e6c-867a-ff5ddc2550be","question":"How many studio albums were published by Mercedes Sosa between 2000 and 2009 (included)? You can use the latest 2022 version of english wikipedia.","Level":"1","file_name":""},{"task_id":"a1e91b78-d3d8-4675-bb8d-62741b4b68a6","question":"In the video https://www.youtube.com/watch?v=L1vXCYZAYYM, what is the highest number of bird species to be on camera simultaneously?","Level":"1","file_name":""},{"task_id":"2d83110e-a098-4ebb-9987-066c06fa42d0","question":".rewsna eht sa \"tfel\" drow eht fo etisoppo eht etirw ,ecnetnes siht dnatsrednu uoy fI","Level":"1","file_name":""},{"task_id":"cca530fc-4052-43b2-b130-b30968d8aa44","question":"Review the chess position provided in the image. It is black's turn. Provide the correct next move for black which guarantees a win. Please provide your response in algebraic notation.","Level":"1","file_name":"cca530fc-4052-43b2-b130-b30968d8aa44.png"},{"task_id":"4fc2f1ae-8625-45b5-ab34-ad4433bc21f8","question":"Who nominated the only Featured Article on English Wikipedia about a dinosaur that was promoted in November 2016?","Level":"1","file_name":""},{"task_id":"6f37996b-2ac7-44b0-8e68-6d28256631b4","question":"Given this table defining * on the set S = {a, b, c, d, e}\n\n|*|a|b|c|d|e|\n|---|---|---|---|---|---|\n|a|a|b|c|b|d|\n|b|b|c|a|e|c|\n|c|c|a|b|b|a|\n|d|b|e|b|e|d|\n|e|d|b|a|d|c|\n\nprovide the subset of S involved in any possible counter-examples that prove * is not commutative. Provide your answer as a comma separated list of the elements in the set in alphabetical order.","Level":"1","file_name":""},{"task_id":"9d191bce-651d-4746-be2d-7ef8ecadb9c2","question":"Examine the video at https://www.youtube.com/watch?v=1htKBjuUWec.\n\nWhat does Teal'c say in response to the question \"Isn't that hot?\"","Level":"1","file_name":""},{"task_id":"cabe07ed-9eca-40ea-8ead-410ef5e83f91","question":"What is the surname of the equine veterinarian mentioned in 1.E Exercises from the chemistry materials licensed by Marisa Alviar-Agnew & Henry Agnew under the CK-12 license in LibreText's Introductory Chemistry materials as compiled 08/21/2023?","Level":"1","file_name":""},{"task_id":"3cef3a44-215e-4aed-8e3b-b1e3f08063b7","question":"I'm making a grocery list for my mom, but she's a professor of botany and she's a real stickler when it comes to categorizing things. I need to add different foods to different categories on the grocery list, but if I make a mistake, she won't buy anything inserted in the wrong category. Here's the list I have so far:\n\nmilk, eggs, flour, whole bean coffee, Oreos, sweet potatoes, fresh basil, plums, green beans, rice, corn, bell pepper, whole allspice, acorns, broccoli, celery, zucchini, lettuce, peanuts\n\nI need to make headings for the fruits and vegetables. Could you please create a list of just the vegetables from my list? If you could do that, then I can figure out how to categorize the rest of the list into the appropriate categories. But remember that my mom is a real stickler, so make sure that no botanical fruits end up on the vegetable list, or she won't get them when she's at the store. Please alphabetize the list of vegetables, and place each item in a comma separated list.","Level":"1","file_name":""},{"task_id":"99c9cc74-fdc8-46c6-8f8d-3ce2d3bfeea3","question":"Hi, I'm making a pie but I could use some help with my shopping list. I have everything I need for the crust, but I'm not sure about the filling. I got the recipe from my friend Aditi, but she left it as a voice memo and the speaker on my phone is buzzing so I can't quite make out what she's saying. Could you please listen to the recipe and list all of the ingredients that my friend described? I only want the ingredients for the filling, as I have everything I need to make my favorite pie crust. I've attached the recipe as Strawberry pie.mp3.\n\nIn your response, please only list the ingredients, not any measurements. So if the recipe calls for \"a pinch of salt\" or \"two cups of ripe strawberries\" the ingredients on the list would be \"salt\" and \"ripe strawberries\".\n\nPlease format your response as a comma separated list of ingredients. Also, please alphabetize the ingredients.","Level":"1","file_name":"99c9cc74-fdc8-46c6-8f8d-3ce2d3bfeea3.mp3"},{"task_id":"305ac316-eef6-4446-960a-92d80d542f82","question":"Who did the actor who played Ray in the Polish-language version of Everybody Loves Raymond play in Magda M.? Give only the first name.","Level":"1","file_name":""},{"task_id":"f918266a-b3e0-4914-865d-4faa564f1aef","question":"What is the final numeric output from the attached Python code?","Level":"1","file_name":"f918266a-b3e0-4914-865d-4faa564f1aef.py"},{"task_id":"3f57289b-8c60-48be-bd80-01f8099ca449","question":"How many at bats did the Yankee with the most walks in the 1977 regular season have that same season?","Level":"1","file_name":""},{"task_id":"1f975693-876d-457b-a649-393859e79bf3","question":"Hi, I was out sick from my classes on Friday, so I'm trying to figure out what I need to study for my Calculus mid-term next week. My friend from class sent me an audio recording of Professor Willowbrook giving out the recommended reading for the test, but my headphones are broken :(\n\nCould you please listen to the recording for me and tell me the page numbers I'm supposed to go over? I've attached a file called Homework.mp3 that has the recording. Please provide just the page numbers as a comma-delimited list. And please provide the list in ascending order.","Level":"1","file_name":"1f975693-876d-457b-a649-393859e79bf3.mp3"},{"task_id":"840bfca7-4f7b-481a-8794-c560c340185d","question":"On June 6, 2023, an article by Carolyn Collins Petersen was published in Universe Today. This article mentions a team that produced a paper about their observations, linked at the bottom of the article. Find this paper. Under what NASA award number was the work performed by R. G. Arendt supported by?","Level":"1","file_name":""},{"task_id":"bda648d7-d618-4883-88f4-3466eabd860e","question":"Where were the Vietnamese specimens described by Kuznetzov in Nedoshivina's 2010 paper eventually deposited? Just give me the city name without abbreviations.","Level":"1","file_name":""},{"task_id":"cf106601-ab4f-4af9-b045-5295fe67b37d","question":"What country had the least number of athletes at the 1928 Summer Olympics? If there's a tie for a number of athletes, return the first in alphabetical order. Give the IOC country code as your answer.","Level":"1","file_name":""},{"task_id":"a0c07678-e491-4bbc-8f0b-07405144218f","question":"Who are the pitchers with the number before and after Taishō Tamai's number as of July 2023? Give them to me in the form Pitcher Before, Pitcher After, use their last names only, in Roman characters.","Level":"1","file_name":""},{"task_id":"7bd855d8-463d-4ed5-93ca-5fe35145f733","question":"The attached Excel file contains the sales of menu items for a local fast-food chain. What were the total sales that the chain made from food (not including drinks)? Express your answer in USD with two decimal places.","Level":"1","file_name":"7bd855d8-463d-4ed5-93ca-5fe35145f733.xlsx"},{"task_id":"5a0c1adf-205e-4841-a666-7c3ef95def9d","question":"What is the first name of the only Malko Competition recipient from the 20th Century (after 1977) whose nationality on record is a country that no longer exists?","Level":"1","file_name":""}]

gradioapp.py

@@ -0,0 +1,126 @@
+import gradio as gr
+import config
+
+# --- Build Gradio Interface without Blocks Context ---
+
+run_and_submit_all_callback = None  # Placeholder for the actual function
+
+def _run_and_submit_all_local(profile: gr.OAuthProfile | None = None, active_agent: str = None):
+    """Run and submit with specified agent type."""
+    username = None
+
+    if profile is not None:
+        username = f"{profile.username}"
+        print(f"User logged in: {username}")
+    else:
+        print("User not logged in.")
+        return "Please Login to Hugging Face with the button.", None
+
+    return run_and_submit_all_callback(username, active_agent)
+
+def _run_and_submit_langgraph(profile: gr.OAuthProfile | None = None):
+    """Run and submit with LangGraph agent."""
+    return _run_and_submit_all_local(profile, active_agent=config.AGENT_LANGGRAPH)
+
+def _run_and_submit_react(profile: gr.OAuthProfile | None = None):
+    """Run and submit with ReActLangGraph agent."""
+    return _run_and_submit_all_local(profile, active_agent=config.AGENT_REACT_LANGGRAPH)
+
+def _run_and_submit_llamaindex(profile: gr.OAuthProfile | None = None):
+    """Run and submit with LlamaIndex agent."""
+    return _run_and_submit_all_local(profile, active_agent=config.AGENT_LLAMAINDEX)
+
+
+def _parse_filter_indices(filter_text: str):
+    """Parse comma-separated filter indices from text input.
+
+    Args:
+        filter_text: Comma-separated indices (e.g., "4, 7, 15") or empty for all questions
+
+    Returns:
+        tuple of indices or None if empty/invalid
+    """
+    if not filter_text or not filter_text.strip():
+        return None  # Run all questions
+
+    try:
+        indices = tuple(int(idx.strip()) for idx in filter_text.split(',') if idx.strip())
+        return indices if indices else None
+    except ValueError:
+        return None  # Invalid input, run all questions
+
+
+def create_ui(run_and_submit_all, run_test_code):
+    """Create the Main App with custom layout to include LoginButton"""
+
+    global run_and_submit_all_callback
+    run_and_submit_all_callback = run_and_submit_all
+
+    def _run_test_with_filter(filter_text: str):
+        """Wrapper to run test code with parsed filter indices."""
+        filter_indices = _parse_filter_indices(filter_text)
+        return run_test_code(filter=filter_indices)
+
+    # --- Build Gradio Interface using Blocks ---
+    with gr.Blocks() as demoApp:
+        gr.Markdown("# Basic Agent Evaluation Runner")
+        gr.Markdown(
+            """
+            **Instructions:**
+            1.  Please clone this space, then modify the code to define your agent's logic, the tools, the necessary packages, etc ...
+            2.  Log in to your Hugging Face account using the button below. This uses your HF username for submission.
+            3.  Click 'Run Evaluation & Submit All Answers' to fetch questions, run your agent, submit answers, and see the score.
+            ---
+            **Disclaimers:**
+            Once clicking on the "submit button, it can take quite some time ( this is the time for the agent to go through all the questions).
+            This space provides a basic setup and is intentionally sub-optimal to encourage you to develop your own, more robust solution. For instance for the delay process of the submit button, a solution could be to cache the answers and submit in a seperate action or even to answer the questions in async.
+            """
+        )
+
+        gr.LoginButton()
+
+        gr.Markdown("### Run Evaluation with Different Agents")
+
+        with gr.Row():
+            run_button_langgraph = gr.Button("Run with LangGraph Agent", variant="primary")
+            run_button_react = gr.Button("Run with ReAct Agent", variant="secondary")
+            run_button_llamaindex = gr.Button("Run with LlamaIndex Agent", variant="secondary")
+
+        status_output = gr.Textbox(label="Run Status / Submission Result", lines=5, interactive=False)
+        # Removed max_rows=10 from DataFrame constructor
+        results_table = gr.DataFrame(label="Questions and Agent Answers", wrap=True)
+
+        run_button_langgraph.click(
+            fn=_run_and_submit_langgraph,
+            outputs=[status_output, results_table]
+        )
+
+        run_button_react.click(
+            fn=_run_and_submit_react,
+            outputs=[status_output, results_table]
+        )
+
+        run_button_llamaindex.click(
+            fn=_run_and_submit_llamaindex,
+            outputs=[status_output, results_table]
+        )
+
+        gr.Markdown("---")
+        gr.Markdown("### Test Mode")
+        gr.Markdown("Run agent on specific questions for testing. Leave empty to run all questions.")
+
+        test_filter_input = gr.Textbox(
+            label="Question Indices (comma-separated)",
+            placeholder="e.g., 4, 7, 15 (leave empty for all questions)",
+            value="",
+            interactive=True
+        )
+        test_button = gr.Button("Run Test Examples")
+        test_results_table = gr.DataFrame(label="Test Answers from Agent", wrap=True)
+        test_button.click(
+            fn=_run_test_with_filter,
+            inputs=[test_filter_input],
+            outputs=[test_results_table]
+        )
+
+    return demoApp

langgraphagent.py

@@ -0,0 +1,348 @@
+import os
+import logging
+import warnings
+import re
+import time
+
+# Suppress TensorFlow/Keras warnings
+os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
+logging.getLogger('tensorflow').setLevel(logging.ERROR)
+warnings.filterwarnings('ignore', module='tensorflow')
+warnings.filterwarnings('ignore', module='tf_keras')
+
+from typing import TypedDict, Optional, List, Annotated
+from langchain_core.messages import HumanMessage, SystemMessage
+from langgraph.graph import MessagesState, StateGraph, START, END
+from langgraph.graph.message import add_messages
+from langgraph.prebuilt import tools_condition
+from langgraph.prebuilt import ToolNode
+from langchain_google_genai import ChatGoogleGenerativeAI
+from langchain_huggingface import ChatHuggingFace, HuggingFaceEndpoint
+
+from custom_tools import get_custom_tools_list, reset_tool_counters
+from system_prompt import SYSTEM_PROMPT
+from utils import cleanup_answer, extract_text_from_content
+import config
+
+# Suppress BeautifulSoup GuessedAtParserWarning
+try:
+    from bs4 import GuessedAtParserWarning
+    warnings.filterwarnings('ignore', category=GuessedAtParserWarning)
+except ImportError:
+    pass
+
+
+class AgentState(TypedDict):
+    question: str
+    messages: Annotated[list , add_messages]   # for LangGraph
+    answer: str
+    step_count: int  # Track number of iterations to prevent infinite loops
+    file_name: str  # Optional file name for questions that reference files
+
+
+class LangGraphAgent:
+
+    def __init__(self):
+        # Validate API keys
+        if not config.GOOGLE_API_KEY:
+            print("WARNING: GOOGLE_API_KEY not found - analyze_youtube_video will fail")
+
+        self.tools = get_custom_tools_list()
+        self.llm_client_with_tools = self._create_llm_client()
+        self.graph = self._build_graph()
+
+    def _create_llm_client(self, model_provider: str = "google"):
+        """Create and return the LLM client with tools bound based on the model provider."""
+
+        if model_provider == "google":
+            apikey = config.GOOGLE_API_KEY
+
+            return ChatGoogleGenerativeAI(
+                model=config.ACTIVE_AGENT_LLM_MODEL,
+                temperature=0,
+                api_key=apikey,
+                thinking_budget=0,
+                timeout=120
+                ).bind_tools(self.tools)
+
+        elif model_provider == "huggingface":
+            LLM_MODEL = "meta-llama/Llama-3.1-8B-Instruct"
+            apikey = os.getenv("HUGGINGFACEHUB_API_TOKEN")
+
+            llmObject = HuggingFaceEndpoint(
+                repo_id=LLM_MODEL,
+                task="text-generation",
+                max_new_tokens=512,
+                temperature=0.7,
+                do_sample=False,
+                repetition_penalty=1.03,
+                huggingfacehub_api_token=apikey
+            )
+            return ChatHuggingFace(llm=llmObject).bind_tools(self.tools)
+
+    # Nodes
+    def _init_questions(self, state: AgentState):
+        """Initialize the messages in the state with system prompt and user question."""
+
+        # Reset per-question tool counters (e.g., analyze_image call limit)
+        reset_tool_counters()
+
+        # Build the question message, including file name if available
+        question_content = state["question"]
+        if state.get("file_name"):
+            question_content += f'\n\nNote: This question references a file: {state["file_name"]}'
+
+        return {
+            "messages": [
+                    SystemMessage(content=SYSTEM_PROMPT),
+                    HumanMessage(content=question_content)
+                    ],
+            "step_count": 0  # Initialize step counter
+                }
+
+    def _assistant(self, state: AgentState):
+        """Assistant node which calls the LLM with tools"""
+
+        # Track and log current step
+        current_step = state.get("step_count", 0) + 1
+        print(f"[STEP {current_step}] Calling assistant with {len(state['messages'])} messages")
+
+        # Force termination at step limit — _should_continue cannot persist state changes
+        # so we detect the near-limit here and force a final LLM call without tool binding
+        if current_step >= config.AGENT_STEP_LIMIT - 1:  # force a final bare-answer call one step before the hard limit
+            existing = state.get("answer")
+            if existing:
+                return {"messages": [], "answer": existing, "step_count": current_step}
+            print(f"[WARNING] Near step limit at step {current_step} with no answer — forcing bare LLM call")
+            from langchain_core.messages import SystemMessage as SM
+            forced_suffix = SM(content="STOP ALL TOOL CALLS. Based only on information gathered so far, output ONLY the bare answer value — one word, number, or short phrase. No explanation.")
+
+            def _extract_content(resp_content):
+                if not resp_content:
+                    return ""
+                if isinstance(resp_content, str):
+                    return resp_content.strip()
+                if isinstance(resp_content, list):
+                    parts = [item['text'] if isinstance(item, dict) and 'text' in item else str(item) for item in resp_content]
+                    return " ".join(parts).strip()
+                return str(resp_content).strip()
+
+            llm_client = self.llm_client_with_tools
+            if llm_client is None:
+                return {"messages": [], "answer": "Error: Step limit reached", "step_count": current_step}
+
+            # Attempt 1: full context
+            try:
+                forced_messages = list(state["messages"]) + [forced_suffix]
+                forced_resp = llm_client.invoke(forced_messages)
+                content = _extract_content(forced_resp.content)
+                if content:
+                    print(f"[FORCED FINAL] {content[:100]}")
+                    return {"messages": [], "answer": content, "step_count": current_step}
+                print("[FORCED FINAL] Empty content on attempt 1, retrying with reduced context")
+            except Exception as fe:
+                print(f"[WARNING] Forced final call attempt 1 failed: {fe}")
+
+            # Attempt 2: reduced context (first 2 messages + last 10 messages) to avoid token overload
+            try:
+                msgs = state["messages"]
+                reduced = msgs[:2] + (msgs[-10:] if len(msgs) > 12 else msgs[2:])
+                reduced_messages = reduced + [forced_suffix]
+                forced_resp2 = llm_client.invoke(reduced_messages)
+                content2 = _extract_content(forced_resp2.content)
+                if content2:
+                    print(f"[FORCED FINAL REDUCED] {content2[:100]}")
+                    return {"messages": [], "answer": content2, "step_count": current_step}
+                print("[FORCED FINAL] Empty content on attempt 2 as well")
+            except Exception as fe2:
+                print(f"[WARNING] Forced final call attempt 2 failed: {fe2}")
+
+            return {"messages": [], "answer": "Error: Step limit reached", "step_count": current_step}
+
+        # Invoke LLM with tools enabled, with retry logic for 504 errors
+        max_retries = config.MAX_RETRIES
+        delay = config.INITIAL_RETRY_DELAY
+
+        for attempt in range(max_retries + 1):
+            try:
+                response = self.llm_client_with_tools.invoke(state["messages"])
+                # Success - break out of retry loop
+                break
+            except Exception as e:
+                error_msg = str(e)
+
+                # Check if this is a 504 DEADLINE_EXCEEDED error
+                if "504" in error_msg and "DEADLINE_EXCEEDED" in error_msg:
+                    if attempt < max_retries:
+                        print(f"[RETRY] Attempt {attempt + 1}/{max_retries} failed with 504 DEADLINE_EXCEEDED")
+                        print(f"[RETRY] Retrying in {delay:.1f} seconds...")
+                        time.sleep(delay)
+                        delay *= config.RETRY_BACKOFF_FACTOR
+                        continue
+                    else:
+                        print(f"[RETRY] All {max_retries} retries exhausted for 504 error")
+                        print(f"[ERROR] LLM invocation failed after retries: {e}")
+                        return {
+                            "messages": [],
+                            "answer": f"Error: LLM failed after {max_retries} retries - {str(e)[:100]}",
+                            "step_count": current_step
+                        }
+                else:
+                    # Not a 504 error - fail immediately without retry
+                    print(f"[ERROR] LLM invocation failed: {e}")
+                    return {
+                        "messages": [],
+                        "answer": f"Error: LLM failed - {str(e)[:100]}",
+                        "step_count": current_step
+                    }
+
+        # If no tool calls, set the final answer
+        if not response.tool_calls:
+            content = response.content
+            print(f"[FINAL ANSWER] Agent produced answer (no tool calls)")
+
+            # Handle case where content is a list (e.g. mixed content from Gemini)
+            if isinstance(content, list):
+                # Extract text from list of content parts
+                text_parts = []
+                for item in content:
+                    if isinstance(item, dict) and 'text' in item:
+                        text_parts.append(item['text'])
+                    elif hasattr(item, 'text'):
+                        text_parts.append(item.text)
+                    else:
+                        text_parts.append(str(item))
+                content = " ".join(text_parts)
+            elif isinstance(content, dict) and 'text' in content:
+                # Handle single dict with 'text' field
+                content = content['text']
+            elif hasattr(content, 'text'):
+                # Handle object with text attribute
+                content = content.text
+            else:
+                # Fallback to string conversion
+                content = str(content)
+
+            # Clean up any remaining noise
+            content = content.strip()
+            print(f"[EXTRACTED TEXT] {content[:100]}{'...' if len(content) > 100 else ''}")
+
+            # If content is empty (transient Gemini API issue), retry up to 3 times
+            retry_num = 0
+            while not content and retry_num < 3:
+                retry_num += 1
+                print(f"[WARNING] Empty response from LLM at step {current_step} — retry {retry_num}/3")
+                try:
+                    import time as _time
+                    _time.sleep(retry_num * 2)  # back off: 2s, 4s, 6s
+                    retry_resp = self.llm_client_with_tools.invoke(state["messages"])  # type: ignore[union-attr]
+                    retry_content = retry_resp.content
+                    if isinstance(retry_content, str):
+                        content = retry_content.strip()
+                    elif isinstance(retry_content, list):
+                        parts = [item['text'] if isinstance(item, dict) and 'text' in item else str(item) for item in retry_content]
+                        content = " ".join(parts).strip()
+                    if content:
+                        print(f"[RETRY SUCCESS] Got content on retry {retry_num}: {content[:80]}")
+                except Exception as re_err:
+                    print(f"[WARNING] Retry {retry_num} failed: {re_err}")
+
+            return {
+                "messages": [response],
+                "answer": content,
+                "step_count": current_step
+            }
+
+        # Has tool calls, log them
+        print(f"[TOOL CALLS] Agent requesting {len(response.tool_calls)} tool(s):")
+        for tc in response.tool_calls:
+            print(f"  - {tc['name']}")
+
+        return {
+            "messages": [response],
+            "step_count": current_step
+        }
+
+
+    def _should_continue(self, state: AgentState):
+        """Check if we should continue or stop based on step count and other conditions."""
+
+        step_count = state.get("step_count", 0)
+
+        # Stop if we've exceeded maximum steps
+        if step_count >= config.AGENT_STEP_LIMIT:  # Backstop; recursion_limit is derived to exceed 2x this
+            print(f"[WARNING] Max steps ({config.AGENT_STEP_LIMIT}) reached, forcing termination")
+            # Force a final answer if we don't have one
+            if not state.get("answer"):
+                state["answer"] = "Error: Maximum iteration limit reached"
+            return END
+
+        # Otherwise use the default tools_condition
+        return tools_condition(state)
+
+
+    def _build_graph(self):
+        """Build and return the Compiled Graph for the agent."""
+
+        graph = StateGraph(AgentState)
+
+        # Build graph
+        graph.add_node("init", self._init_questions)
+        graph.add_node("assistant", self._assistant)
+        graph.add_node("tools", ToolNode(self.tools))
+        graph.add_edge(START, "init")
+        graph.add_edge("init", "assistant")
+        graph.add_conditional_edges(
+            "assistant",
+            # Use custom should_continue instead of tools_condition
+            self._should_continue,
+        )
+        graph.add_edge("tools", "assistant")
+        # Compile graph
+        return graph.compile()
+
+    def __call__(self, question: str, file_name: str = None) -> str:
+        """Invoke the agent graph with the given question and return the final answer.
+
+        Args:
+            question: The question to answer
+            file_name: Optional file name if the question references a file
+        """
+
+        print(f"\n{'='*60}")
+        print(f"[LANGGRAPH AGENT START] Question: {question}")
+        if file_name:
+            print(f"[FILE] {file_name}")
+        print(f"{'='*60}")
+
+        start_time = time.time()
+
+        try:
+            response = self.graph.invoke(
+                {"question": question, "messages": [], "answer": None, "step_count": 0, "file_name": file_name or ""},
+                config={"recursion_limit": config.AGENT_RECURSION_LIMIT}  # Derived in config: > 2x step limit
+            )
+
+            elapsed_time = time.time() - start_time
+            print(f"[LANGGRAPH AGENT COMPLETE] Time: {elapsed_time:.2f}s")
+            print(f"{'='*60}\n")
+
+            answer = response.get("answer")
+            if not answer or answer is None:
+                print("[WARNING] Agent completed but returned None as answer")
+                return "Error: No answer generated"
+
+            # Use utility function to extract text from various content formats
+            answer = extract_text_from_content(answer)
+
+            # Clean up the answer using utility function (includes stripping)
+            answer = cleanup_answer(answer)
+
+            print(f"[FINAL ANSWER] {answer}")
+            return answer
+
+        except Exception as e:
+            elapsed_time = time.time() - start_time
+            print(f"[LANGGRAPH AGENT ERROR] Failed after {elapsed_time:.2f}s: {e}")
+            print(f"{'='*60}\n")
+            return f"Error: {str(e)[:100]}"

llamaindexagent.py

@@ -0,0 +1,216 @@
+import os
+import logging
+import warnings
+import time
+import asyncio
+import nest_asyncio
+
+# Apply nest_asyncio to allow nested event loops
+nest_asyncio.apply()
+
+# Suppress TensorFlow/Keras warnings
+os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
+logging.getLogger('tensorflow').setLevel(logging.ERROR)
+warnings.filterwarnings('ignore', module='tensorflow')
+warnings.filterwarnings('ignore', module='tf_keras')
+
+# Suppress google.generativeai deprecation warning from llama_index
+warnings.filterwarnings('ignore', message='.*google.generativeai.*deprecated.*', category=FutureWarning)
+warnings.filterwarnings('ignore', module='google.generativeai')
+
+# Suppress asyncio selector warnings that occur during event loop cleanup on some platforms
+warnings.filterwarnings('ignore', message='.*Invalid file descriptor.*')
+logging.getLogger('asyncio').setLevel(logging.ERROR)
+
+from llama_index.core.agent import ReActAgent
+from llama_index.llms.gemini import Gemini
+from llama_index.core.tools import FunctionTool
+
+from custom_tools import get_custom_tools_list
+from system_prompt import SYSTEM_PROMPT
+from utils import cleanup_answer, extract_text_from_content
+import config
+
+# Suppress BeautifulSoup GuessedAtParserWarning
+try:
+    from bs4 import GuessedAtParserWarning
+    warnings.filterwarnings('ignore', category=GuessedAtParserWarning)
+except ImportError:
+    pass
+
+
+class LlamaIndexAgent:
+    """
+    LlamaIndex agent implementation using ReActAgent.
+
+    This agent uses LlamaIndex's ReAct agent pattern which integrates
+    with various LLM providers and tools. It provides an alternative
+    implementation to LangGraph-based agents.
+    """
+
+    def __init__(self):
+        # Validate API keys
+        if not config.GOOGLE_API_KEY:
+            print("WARNING: GOOGLE_API_KEY not found - analyze_youtube_video will fail")
+
+        self.langchain_tools = get_custom_tools_list()
+        self.llm = self._create_llm_client()
+        self.tools = self._convert_tools_to_llamaindex()
+        self.agent = self._build_agent()
+
+    def _create_llm_client(self):
+        """Create and return the LLM client for LlamaIndex."""
+        api_key = config.GOOGLE_API_KEY
+
+        # Create Gemini LLM for LlamaIndex
+        llm = Gemini(
+            model=config.ACTIVE_AGENT_LLM_MODEL,
+            api_key=api_key,
+            temperature=config.GEMINI_TEMPERATURE,
+            max_tokens=config.GEMINI_MAX_TOKENS,
+        )
+
+        return llm
+
+    def _convert_tools_to_llamaindex(self) -> list[FunctionTool]:
+        """Convert LangChain tools to LlamaIndex FunctionTool format."""
+        llamaindex_tools = []
+
+        for langchain_tool in self.langchain_tools:
+            # Extract the function from LangChain tool
+            tool_func = langchain_tool.func if hasattr(langchain_tool, 'func') else langchain_tool
+
+            # Create LlamaIndex FunctionTool
+            llamaindex_tool = FunctionTool.from_defaults(
+                fn=tool_func,
+                name=langchain_tool.name,
+                description=langchain_tool.description,
+            )
+
+            llamaindex_tools.append(llamaindex_tool)
+
+        return llamaindex_tools
+
+    def _build_agent(self) -> ReActAgent:
+        """Build and return the LlamaIndex ReAct agent."""
+
+        # Create ReAct agent with tools and LLM
+        agent = ReActAgent(
+            tools=self.tools,
+            llm=self.llm,
+            verbose=True,
+            max_iterations=40,  # Match the step limit from other agents
+            system_prompt=SYSTEM_PROMPT,
+        )
+
+        return agent
+
+    def __call__(self, question: str, file_name: str = None) -> str:
+        """
+        Invoke the LlamaIndex agent with the given question and return the final answer.
+
+        Args:
+            question: The question to answer
+            file_name: Optional file name if the question references a file
+
+        Returns:
+            The agent's answer as a string
+        """
+        print(f"\n{'='*60}")
+        print(f"[LLAMAINDEX AGENT START] Question: {question}")
+        if file_name:
+            print(f"[FILE] {file_name}")
+        print(f"{'='*60}")
+
+        start_time = time.time()
+
+        try:
+            # Build the question with file name if provided
+            question_content = question
+            if file_name:
+                question_content += f'\n\nNote: This question references a file: {file_name}'
+
+            # Invoke the agent with retry logic for 504 errors
+            max_retries = config.MAX_RETRIES
+            delay = config.INITIAL_RETRY_DELAY
+
+            for attempt in range(max_retries + 1):
+                try:
+                    # Create a dedicated async function to run the agent
+                    async def run_agent_async():
+                        # Pass max_iterations as a runtime parameter to the workflow
+                        return await self.agent.run(question_content, max_iterations=40)
+
+                    # Try different approaches to run the async function
+                    try:
+                        # Check if a loop is already running
+                        loop = asyncio.get_running_loop()
+                        # If we reach here, a loop is already running
+                        # Use nest_asyncio's patched loop to run coroutine
+                        response = loop.run_until_complete(run_agent_async())
+                    except RuntimeError:
+                        # No running loop, we can use asyncio.run directly
+                        # But wrap in try-except to suppress cleanup errors
+                        try:
+                            response = asyncio.run(run_agent_async())
+                        except ValueError as ve:
+                            # Suppress "Invalid file descriptor" errors during cleanup
+                            if "Invalid file descriptor" not in str(ve):
+                                raise
+
+                    # Success - break out of retry loop
+                    break
+                except Exception as e:
+                    error_msg = str(e)
+
+                    # Check if this is a 504 DEADLINE_EXCEEDED error
+                    if "504" in error_msg and "DEADLINE_EXCEEDED" in error_msg:
+                        if attempt < max_retries:
+                            print(f"[RETRY] Attempt {attempt + 1}/{max_retries} failed with 504 DEADLINE_EXCEEDED")
+                            print(f"[RETRY] Retrying in {delay:.1f} seconds...")
+                            time.sleep(delay)
+                            delay *= config.RETRY_BACKOFF_FACTOR
+                            continue
+                        else:
+                            print(f"[RETRY] All {max_retries} retries exhausted for 504 error")
+                            print(f"[ERROR] Agent invocation failed after retries: {e}")
+                            return f"Error: Agent failed after {max_retries} retries - {str(e)[:100]}"
+                    else:
+                        # Not a 504 error - fail immediately without retry
+                        print(f"[ERROR] Agent invocation failed: {e}")
+                        return f"Error: Agent failed - {str(e)[:100]}"
+
+            elapsed_time = time.time() - start_time
+            print(f"[LLAMAINDEX AGENT COMPLETE] Time: {elapsed_time:.2f}s")
+            print(f"{'='*60}\n")
+
+            # Extract the answer from the response using utility function
+            # This handles ChatMessage objects, dicts, lists, and strings
+            answer = extract_text_from_content(response)
+
+            if not answer or answer is None:
+                print("[WARNING] Agent completed but returned Empty answer")
+                return "Error: No answer generated"
+
+            # LlamaIndex ReActAgent may wrap answers in verbose format
+            # Check if the response starts with common verbose patterns and extract the core answer
+            import re
+
+            # Pattern 1: "Answer: X" or "Final Answer: X" from ReAct format
+            react_answer_match = re.search(r'(?:Final\s+)?Answer:\s*(.+)', answer, re.IGNORECASE | re.DOTALL)
+            if react_answer_match:
+                extracted = react_answer_match.group(1).strip()
+                print(f"[LLAMAINDEX] Extracted answer from ReAct format: '{extracted[:100]}...'")
+                answer = extracted
+
+            # Clean up the answer using utility function (includes stripping)
+            answer = cleanup_answer(answer)
+
+            print(f"[FINAL ANSWER] {answer}")
+            return answer
+
+        except Exception as e:
+            elapsed_time = time.time() - start_time
+            print(f"[LLAMAINDEX AGENT ERROR] Failed after {elapsed_time:.2f}s: {e}")
+            print(f"{'='*60}\n")
+            return f"Error: {str(e)[:100]}"

question_loader.py

@@ -0,0 +1,47 @@
+"""Question loading and fetching functionality."""
+
+import json
+import requests
+from typing import List, Dict
+import config
+from utils import retry_with_backoff
+
+
+class QuestionLoader:
+    """Handles loading questions from various sources."""
+
+    def __init__(self, api_url: str = config.DEFAULT_API_URL):
+        self.api_url = api_url
+
+    @retry_with_backoff(max_retries=3, initial_delay=1.0, backoff_factor=2.0)
+    def _fetch_from_api(self) -> List[Dict]:
+        """Fetch questions from the API with retry logic."""
+        questions_url = f"{self.api_url}/questions"
+        print(f"Fetching questions from: {questions_url}")
+
+        response = requests.get(questions_url, timeout=config.FETCH_TIMEOUT)
+        response.raise_for_status()
+        questions_data = response.json()
+
+        if not questions_data:
+            raise ValueError("Fetched questions list is empty.")
+
+        print(f"Fetched {len(questions_data)} questions.")
+        return questions_data
+
+    def _load_from_file(self, file_path: str = config.QUESTIONS_FILE) -> List[Dict]:
+        """Load questions from local file."""
+        with open(file_path, 'r', encoding='utf-8') as f:
+            questions = json.load(f)
+            print(f"[INFO] Loaded {len(questions)} questions from {file_path}")
+            return questions
+
+    def get_questions(self, test_mode: bool = False) -> List[Dict]:
+        """Get questions from local file (test) or API (production)."""
+        if test_mode:
+            try:
+                return self._load_from_file()
+            except Exception as e:
+                print(f"[WARNING] Offline loading failed: {e}, falling back to API")
+
+        return self._fetch_from_api()

reactlanggraphagent.py

@@ -0,0 +1,168 @@
+import os
+import logging
+import warnings
+import time
+
+# Suppress TensorFlow/Keras warnings
+os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
+logging.getLogger('tensorflow').setLevel(logging.ERROR)
+warnings.filterwarnings('ignore', module='tensorflow')
+warnings.filterwarnings('ignore', module='tf_keras')
+
+from langgraph.prebuilt import create_react_agent
+from langchain_google_genai import ChatGoogleGenerativeAI
+from langchain_core.messages import HumanMessage
+
+from custom_tools import get_custom_tools_list
+from system_prompt import SYSTEM_PROMPT
+from utils import cleanup_answer, extract_text_from_content
+import config
+
+# Suppress BeautifulSoup GuessedAtParserWarning
+try:
+    from bs4 import GuessedAtParserWarning
+    warnings.filterwarnings('ignore', category=GuessedAtParserWarning)
+except ImportError:
+    pass
+
+
+class ReActLangGraphAgent:
+    """
+    ReAct agent implementation using LangGraph's create_react_agent function.
+
+    This agent uses the ReAct (Reasoning + Acting) pattern where the agent
+    reasons about what to do and then acts by calling tools iteratively.
+    Built on top of LangGraph's prebuilt ReAct agent.
+    """
+
+    def __init__(self):
+        # Validate API keys
+        if not config.GOOGLE_API_KEY:
+            print("WARNING: GOOGLE_API_KEY not found - analyze_youtube_video will fail")
+
+        self.tools = get_custom_tools_list()
+        self.llm = self._create_llm_client()
+        self.agent_graph = self._build_agent()
+
+    def _create_llm_client(self):
+        """Create and return the LLM client."""
+        apikey = config.GOOGLE_API_KEY
+
+        return ChatGoogleGenerativeAI(
+            model=config.ACTIVE_AGENT_LLM_MODEL,
+            temperature=config.GEMINI_TEMPERATURE,
+            api_key=apikey,
+            thinking_budget=0,
+            timeout=120
+        )
+
+    def _build_agent(self):
+        """Build and return the ReAct agent graph using LangGraph's create_react_agent."""
+
+        # LangGraph's create_react_agent returns a compiled graph
+        # It automatically handles the ReAct loop with tools
+        agent_graph = create_react_agent(
+            model=self.llm,
+            tools=self.tools,
+            prompt=SYSTEM_PROMPT  # System prompt is added via the prompt parameter
+        )
+
+        return agent_graph
+
+    def __call__(self, question: str, file_name: str = None) -> str:
+        """
+        Invoke the ReAct agent with the given question and return the final answer.
+
+        Args:
+            question: The question to answer
+            file_name: Optional file name if the question references a file
+
+        Returns:
+            The agent's answer as a string
+        """
+        print(f"\n{'='*60}")
+        print(f"[REACT AGENT START] Question: {question}")
+        if file_name:
+            print(f"[FILE] {file_name}")
+        print(f"{'='*60}")
+
+        start_time = time.time()
+
+        try:
+            # Build the question with file name if provided
+            question_content = question
+            if file_name:
+                question_content += f'\n\nNote: This question references a file: {file_name}'
+
+            # Invoke the agent graph with retry logic for 504 errors
+            max_retries = config.MAX_RETRIES
+            delay = config.INITIAL_RETRY_DELAY
+
+            for attempt in range(max_retries + 1):
+                try:
+                    # LangGraph's create_react_agent expects messages as input
+                    response = self.agent_graph.invoke(
+                        {"messages": [HumanMessage(content=question_content)]},
+                        config={"recursion_limit": config.AGENT_RECURSION_LIMIT}  # Shared with LangGraphAgent via config
+                    )
+                    # Success - break out of retry loop
+                    break
+                except Exception as e:
+                    error_msg = str(e)
+
+                    # Check if this is a 504 DEADLINE_EXCEEDED error
+                    if "504" in error_msg and "DEADLINE_EXCEEDED" in error_msg:
+                        if attempt < max_retries:
+                            print(f"[RETRY] Attempt {attempt + 1}/{max_retries} failed with 504 DEADLINE_EXCEEDED")
+                            print(f"[RETRY] Retrying in {delay:.1f} seconds...")
+                            time.sleep(delay)
+                            delay *= config.RETRY_BACKOFF_FACTOR
+                            continue
+                        else:
+                            print(f"[RETRY] All {max_retries} retries exhausted for 504 error")
+                            print(f"[ERROR] Agent invocation failed after retries: {e}")
+                            return f"Error: Agent failed after {max_retries} retries - {str(e)[:100]}"
+                    else:
+                        # Not a 504 error - fail immediately without retry
+                        print(f"[ERROR] Agent invocation failed: {e}")
+                        return f"Error: Agent failed - {str(e)[:100]}"
+
+            elapsed_time = time.time() - start_time
+            print(f"[REACT AGENT COMPLETE] Time: {elapsed_time:.2f}s")
+            print(f"{'='*60}\n")
+
+            # Extract the answer from the response
+            # LangGraph's create_react_agent returns the last message in the messages list
+            messages = response.get("messages", [])
+
+            if not messages:
+                print("[WARNING] Agent completed but returned no messages")
+                return "Error: No answer generated"
+
+            # Get the last message (the agent's final response)
+            last_message = messages[-1]
+
+            # Extract content from the message
+            if hasattr(last_message, 'content'):
+                content = last_message.content
+            else:
+                content = str(last_message)
+
+            # Use utility function to extract text from various content formats
+            answer = extract_text_from_content(content)
+
+            if not answer or answer is None:
+                print("[WARNING] Agent completed but returned None as answer")
+                return "Error: No answer generated"
+
+            # Clean up the answer using utility function
+            answer = cleanup_answer(answer)
+
+            print(f"[FINAL ANSWER] {answer}")
+            return answer
+
+        except Exception as e:
+            elapsed_time = time.time() - start_time
+            print(f"[REACT AGENT ERROR] Failed after {elapsed_time:.2f}s: {e}")
+            print(f"{'='*60}\n")
+            return f"Error: {str(e)[:100]}"

requirements.txt

@@ -0,0 +1,29 @@
+gradio
+requests
+huggingface_hub
+pillow
+ddgs
+pytz
+wikipedia
+arxiv
+langchain
+langgraph
+langchain-core
+langchain-google-genai
+langchain-huggingface
+langchain-community
+llama-index
+llama-index-llms-gemini
+llama-index-core
+pypdf
+youtube-transcript-api
+pytube
+pymupdf
+nest_asyncio
+speechrecognition
+pydub
+markdownify
+numpy
+pandas
+colorama
+gradio[oauth]

result_formatter.py

@@ -0,0 +1,61 @@
+"""Result formatting for different output types."""
+
+import pandas as pd
+from typing import List, Tuple, Dict
+from colorama import Fore, Style
+
+
+class ResultFormatter:
+    """Formats results for different output targets."""
+
+    @staticmethod
+    def format_for_api(results: List[Tuple[str, str, str]]) -> List[Dict]:
+        """Format results for API submission."""
+        return [
+            {"task_id": task_id, "submitted_answer": answer}
+            for task_id, _, answer in results
+        ]
+
+    @staticmethod
+    def format_for_display(results: List[Tuple[str, str, str]]) -> List[Dict]:
+        """Format results for UI display."""
+        return [
+            {
+                "Task ID": task_id,
+                "Question": question_text,
+                "Submitted Answer": answer
+            }
+            for task_id, question_text, answer in results
+        ]
+
+    @staticmethod
+    def format_for_verification(results: List[Tuple[str, str, str]]) -> List[str]:
+        """Format results for test verification output."""
+        output = []
+        for task_id, question_text, answer in results:
+            output.append(f"\nTask ID: {task_id}")
+            output.append(f"Question: {question_text}")
+            output.append(f"Answer: {answer}")
+        return output
+
+    @staticmethod
+    def print_dataframe(df: pd.DataFrame) -> None:
+        """Print DataFrame with full content (no truncation) with colored output."""
+        pd.set_option('display.max_colwidth', None)
+        pd.set_option('display.max_rows', None)
+        for col in df.columns:
+            for val in df[col]:
+                val_str = str(val)
+                # Color based on content
+                if '✓ Correct' in val_str:
+                    print(f"{Fore.GREEN}{val}{Style.RESET_ALL}", flush=True)
+                elif '✗ Incorrect' in val_str:
+                    print(f"{Fore.RED}{val}{Style.RESET_ALL}", flush=True)
+                elif val_str.startswith('===') or val_str.startswith('SUMMARY'):
+                    print(f"{Fore.CYAN}{val}{Style.RESET_ALL}", flush=True)
+                elif 'ERROR' in val_str:
+                    print(f"{Fore.RED}{val}{Style.RESET_ALL}", flush=True)
+                elif val_str.startswith('Expected:') or val_str.startswith('Got:'):
+                    print(f"{Fore.YELLOW}{val}{Style.RESET_ALL}", flush=True)
+                else:
+                    print(val, flush=True)

scorer.py

@@ -0,0 +1,107 @@
+#Official GAIA Scorer Module from HF. Copied from https://huggingface.co/spaces/gaia-benchmark/leaderboard/blob/main/scorer.py for offline Use. Hoping there are no licensing issues as it is intended for learning purposes only.
+#Thanks, Hemant Virmani
+
+import json
+import re
+import string
+import warnings
+
+import numpy as np
+
+
+def normalize_number_str(number_str: str) -> float:
+    # we replace these common units and commas to allow
+    # conversion to float
+    for char in ["$", "%", ","]:
+        number_str = number_str.replace(char, "")
+    try:
+        return float(number_str)
+    except ValueError:
+        print(f"String {number_str} cannot be normalized to number str.")
+        return float("inf")
+
+
+def split_string(
+    s: str,
+    char_list: list[str] = [",", ";"],
+) -> list[str]:
+    pattern = f"[{''.join(char_list)}]"
+    return re.split(pattern, s)
+
+
+def question_scorer(
+    model_answer: str,
+    ground_truth: str,
+) -> bool:
+    def is_float(element: any) -> bool:
+        try:
+            float(element)
+            return True
+        except ValueError:
+            return False
+
+    if model_answer is None:
+        model_answer = "None"
+
+    # if gt is a number
+    if is_float(ground_truth):
+        print(f"Evaluating {model_answer} as a number.")
+        normalized_answer = normalize_number_str(model_answer)
+        return normalized_answer == float(ground_truth)
+
+    # if gt is a list
+    elif any(char in ground_truth for char in [",", ";"]):
+        print(f"Evaluating {model_answer} as a comma separated list.")
+        # question with the fish: normalization removes punct
+
+        gt_elems = split_string(ground_truth)
+        ma_elems = split_string(model_answer)
+
+        # check length is the same
+        if len(gt_elems) != len(ma_elems):
+            warnings.warn(
+                "Answer lists have different lengths, returning False.", UserWarning
+            )
+            return False
+
+        # compare each element as float or str
+        comparisons = []
+        for ma_elem, gt_elem in zip(ma_elems, gt_elems):
+            if is_float(gt_elem):
+                normalized_ma_elem = normalize_number_str(ma_elem)
+                comparisons.append(normalized_ma_elem == float(gt_elem))
+            else:
+                # we do not remove punct since comparisons can include punct
+                comparisons.append(
+                    normalize_str(ma_elem, remove_punct=False)
+                    == normalize_str(gt_elem, remove_punct=False)
+                )
+        return all(comparisons)
+
+    # if gt is a str
+    else:
+        print(f"Evaluating {model_answer} as a string.")
+        return normalize_str(model_answer) == normalize_str(ground_truth)
+
+
+def normalize_str(input_str, remove_punct=True) -> str:
+    """
+    Normalize a string by:
+    - Removing all white spaces
+    - Optionally removing punctuation (if remove_punct is True)
+    - Converting to lowercase
+    Parameters:
+    - input_str: str, the string to normalize
+    - remove_punct: bool, whether to remove punctuation (default: True)
+    Returns:
+    - str, the normalized string
+    """
+    # Remove all white spaces. Required e.g for seagull vs. sea gull
+    no_spaces = re.sub(r"\s", "", input_str)
+
+    # Remove punctuation, if specified.
+    if remove_punct:
+        translator = str.maketrans("", "", string.punctuation)
+        return no_spaces.lower().translate(translator)
+    else:
+        return no_spaces.lower()

system_prompt.py

@@ -0,0 +1,167 @@
+SYSTEM_PROMPT = """You are an expert, precise and disciplined AI assistant who can solve any task.
+To do so, you have been given access to a list of external tools that you MUST use to find information.
+
+CRITICAL: When you need to use a tool, you MUST call it using the tool calling mechanism. DO NOT write pseudo-code or descriptions of tools. ACTUALLY CALL THE TOOL.
+
+Your task is to answer the user's question using the available tools and provide the answer in a STRICT format.
+
+### AVAILABLE TOOLS
+
+You have access to the following categories of tools:
+
+**Mathematical Operations:**
+- calculate (operation, a, b): Perform arithmetic — add, subtract, multiply, divide, power, modulus
+
+**String & Encoding:**
+- string_reverse: Reverse a string (useful for gibberish or backwards-encoded text)
+- classical_cipher (cipher_type, mode, text, keyword): Encrypt or decrypt Playfair and Bifid classical ciphers
+
+**Computation:**
+- execute_python (code): Execute Python 3 code and return stdout. Use for precise counting, algorithms, or math the LLM cannot do reliably. Use print() for output. IMPORTANT: execute_python runs in a subprocess in the project directory and CAN read files from the files/ directory using pandas (e.g., `import pandas as pd; df = pd.read_excel('files/filename.xlsx'); print(df)`). However, it has NO access to data from previous tool calls as Python variables — to process data returned by a previous tool, embed that data as a string literal in your code. If execute_python fails 3 times, stop and use a different approach.
+
+**Time & Date:**
+- get_current_time_in_timezone: Get current time in any timezone
+
+**Web & Information Search:**
+- websearch: Search the web using DuckDuckGo (returns 5 results with titles, URLs, snippets)
+- wiki_search: Search Wikipedia (returns up to 3 detailed articles)
+- arvix_search: Search academic papers on Arxiv (returns up to 3 papers)
+- get_webpage_content: Load and parse any webpage as markdown (handles PDFs too)
+- youtube_tool (youtube_url, question=""): Pass question="" to get raw transcript; pass a question string to analyze the video with AI (handles visual/audio content)
+
+**File Operations:**
+- read_file (file_name): Read files from the files directory — Excel/CSV → markdown table; .py/.txt/.md/.json → raw text
+- parse_audio_file (file_name): Transcribe MP3 audio files to text
+- analyze_image (question, file_name): Analyze image files (.png, .jpg, .jpeg, etc.) using AI vision
+- download_file (url, file_name): Download a file from a URL and save it to the files directory before reading it
+
+**HTTP:**
+- http_request (method, url, headers_json, body_json): Make GET/POST/DELETE requests with custom headers or body
+
+**Meta / Planning:**
+- ask_advisor (question): Consult a more capable AI when you are completely stuck on HOW TO SEARCH for something, after 2+ failed search attempts with no useful results. NEVER call ask_advisor if any tool (websearch, wiki_search, get_webpage_content, read_file, execute_python, parse_audio_file, analyze_image) has already returned data — work with the data you have. NEVER call it for calculation help, code execution problems, or when you have partial results. At most 1 call per question.
+
+**IMPORTANT:** If the question mentions a file or you see "Note: This question references a file: filename.ext" in the question, use the appropriate file reading tool with that filename:
+- For images (.png, .jpg, .jpeg, .gif, .webp, .bmp): Use analyze_image with your question and the filename
+- For Excel files (.xlsx) or CSV (.csv): Use read_file
+- For Python files (.py) or text files (.txt, .md, .json): Use read_file
+- For audio files (.mp3): Use parse_audio_file
+
+### WORKFLOW
+
+1. **Analyze the Question**: Break down what information you need and what steps are required
+2. **Use Tools Strategically and Efficiently**:
+   - PRIORITY ORDER: Use specific domain tools first, then general search
+     1. For academic/scientific: Try arvix_search first
+     2. For general knowledge: Try wiki_search first
+     3. For current events/specific facts: Use websearch
+     4. For detailed investigation: Use get_webpage_content on promising URLs
+   - QUERY OPTIMIZATION: If first search fails, try 2-3 different query phrasings before switching tools
+   - AVOID REDUNDANCY: Don't repeat the same search with the same tool
+   - Chain calculations using math tools in sequence rather than separate calls
+3. **Process Tool Results**: Extract relevant information from tool outputs
+4. **Calculate/Reason**: If multiple steps are needed, use tools sequentially
+5. **Verify**: Double-check your answer makes sense given the question
+6. **Output**: Provide ONLY the final answer in the exact format required
+
+### CRITICAL OUTPUT RULES (ZERO TOLERANCE)
+
+1. **SINGLE LINE / SINGLE WORD OUTPUT**: Output ONLY the answer value — a single word, short phrase, or number. NO multi-line responses. NO paragraphs. NO explanations.
+2. **NO CONVERSATIONAL FILLER**: Do not use phrases like "I found", "The answer is", "Here are the results", "Based on the search", "According to", "After checking", "Looking at", "The X was Y", etc.
+3. **NO PREAMBLE OR POSTSCRIPT**: Do NOT include "FINAL ANSWER:", "Result:", "Answer:", or any other prefix/suffix
+4. **NO MARKDOWN/TAGS**: Do not wrap the answer in markdown code blocks, JSON, or XML tags
+5. **NO STRUCTURED DATA**: Do NOT output dictionaries, JSON objects, or any structured format - ONLY a single value
+6. **NO TOOL CODE IN OUTPUT**: Never output raw Python code or tool calls (like `tool_code`, `print()`, `default_api.websearch()`)
+7. **EXACT MATCH SCORING**: The grading system checks for an exact string match. Any extra character will cause failure
+8. **ALWAYS USE TOOLS**: If you do not know the answer, use the available tools. Do NOT hallucinate or guess
+9. **TRY MULTIPLE APPROACHES**: If one search doesn't work, try different queries or different tools
+10. **FOR NUMERICAL ANSWERS**:
+    - NO comma separators (use "17000" not "17,000")
+    - NO units unless explicitly requested (use "17" not "17 hours" or "17 thousand")
+    - NO text forms (use "17" not "seventeen")
+    - Follow rounding instructions exactly as specified in the question
+    - If question asks for "thousands", provide the actual thousand value (e.g., "17" for 17,000)
+
+### CRITICAL: SINGLE VALUE ONLY
+Your response must be a single line of plain text — just the answer with NO additional text. Examples of WRONG outputs:
+- ❌ {'type': 'text', 'text': 'answer'}
+- ❌ {"answer": "value"}
+- ❌ `answer`
+- ❌ **answer**
+- ❌ The answer is: answer
+- ❌ The nominator was JohnDoe   (WRONG - has preamble)
+- ❌ The featured article "SomeTopic" was promoted... (WRONG - full sentence)
+
+Examples of CORRECT outputs:
+- ✅ 7
+- ✅ 1995
+- ✅ blue
+- ✅ Harrison
+- ✅ Nf3
+- ✅ Tanaka, Yamamoto
+- ✅ Erik
+- ✅ semicolon
+- ✅ 23000
+
+CRITICAL: Even after long multi-step reasoning, your final output is ONLY the bare answer. Do NOT include the reasoning. Examples of WRONG outputs that contain the correct answer but will still fail:
+- ❌ The only recipient whose country no longer exists is John Smith... His first name is John   (WRONG — contains reasoning)
+- ❌ Player A's number is 12. The pitcher with number 18 is Garcia and number 20 is Martinez   (WRONG — contains reasoning)
+- ❌ The answer is John   (WRONG — has preamble)
+- ❌ Alex Brown led the team in walks with 80. In that same season, he had 412 at-bats   (WRONG — answer 412 is buried at end of sentence)
+- ❌ The specimens described in the 2005 paper were eventually deposited in Berlin   (WRONG — answer is buried at end of sentence)
+- ❌ The work was supported under grant number ABC123456   (WRONG — answer is buried at end of sentence)
+- ❌ The countries with the fewest athletes are Brazil (BRA) and Chile (CHI), both with 1. Alphabetically, Brazil comes first   (WRONG — answer is BRA)
+- ❌ The competition records show John Smith as a 1983 recipient with Westland as his nationality. Westland no longer exists   (WRONG — answer is John)
+- ❌ Player A's number is 12. The pitcher with number 18 is Garcia and number 20 is Martinez   (WRONG — answer must be just: Garcia, Martinez)
+
+For each of the above, the CORRECT output would be just: 412 / Berlin / ABC123456 / BRA / John / Garcia, Martinez
+
+### IMPORTANT NOTES
+
+- **Reversed/Encoded Text**: If text looks like gibberish, use string_reverse tool to decode it
+- **Multiple Search Results**: If websearch returns multiple results, you may need to use get_webpage_content on relevant URLs to find the exact answer
+- **Calculations**: Break down complex math problems and use the math tools step by step
+- **File References**: When questions mention files, use the appropriate read tool based on file extension
+- **Image Analysis**: For visual questions with image files (.png, .jpg, etc.), use analyze_image with the question and filename
+- **YouTube Content**: Use youtube_tool with question="" for raw transcript; pass a non-empty question to analyze the video visually/audio with AI
+- **Audio Transcription**: When listing ingredients, items, or any content from audio, use the EXACT phrasing heard — do NOT simplify or paraphrase. "freshly squeezed lemon juice" ≠ "lemon juice". Every modifier matters. If the question asks to alphabetize the result, sort the items alphabetically AFTER transcribing — the order heard in the audio does not matter, only the words.
+- **List Ordering**: When a question asks for a list of ingredients, grocery items, or similar unordered items and no explicit ordering is specified, output the items sorted in alphabetical order. When the question EXPLICITLY asks to alphabetize, always sort alphabetically regardless of the order encountered during research. CRITICAL: Alphabetize by the ENTIRE item string exactly as written, starting from the first character of the first word — NOT by the "main" noun or any internal keyword. A multi-word item with a leading modifier sorts by that modifier (e.g., "ground black pepper" sorts under G, not under B or P), not by a later noun in the phrase.
+- **Verification**: After finding an answer, verify it matches what the question is asking for
+- **Location Names**: Always expand abbreviated location names to their full form
+  - "St." → "Saint" (e.g., "Saint Petersburg", "Saint Paul", "Saint Louis")
+  - "Mt." → "Mount" (e.g., "Mount Everest", "Mount Rushmore")
+  - "Ft." → "Fort" (e.g., "Fort Worth", "Fort Lauderdale")
+  - Use the canonical/official name when multiple forms exist
+
+### PRECISION AND VERIFICATION
+
+- **Category Distinctions**: Pay careful attention to category qualifiers in questions (e.g., a subset qualifier vs. the whole set, or a part of a name vs. the full name). Filter results precisely to match the exact category requested, and answer the exact entity the question asks for rather than a related one.
+- **Time-Sensitive Data**: When questions specify a date or time period (e.g., "as of July 2023", "compiled 08/21/2023"), you MUST use data from that exact timeframe. **MANDATORY WAYBACK MACHINE RULE**: For ANY question containing date phrases like "as of [date]", "compiled [date]", "as of [month year]" — you MUST fetch the archived Wayback Machine version of relevant webpages. Use this URL format: https://web.archive.org/web/YYYYMMDD000000/[original_URL] where you replace YYYY, MM, DD with the question's date. Example: question says "compiled 08/21/2023" → fetch https://web.archive.org/web/20230821000000/[URL]. Example: question says "as of July 2023" → fetch https://web.archive.org/web/20230701000000/[URL]. Do NOT use current data when a historical date is specified — current pages may differ significantly. If the Wayback Machine page does not contain the expected information, try these variations: (1) simplify the URL path (e.g., remove parenthetical or trailing path segments), (2) try a snapshot a day or two before/after the target date, (3) try the current page as a fallback.
+- **Cross-Verification**: For factual questions, try to verify answers from multiple independent sources when possible. If sources conflict, prefer official/primary sources (Wikipedia, official websites) over secondary sources.
+- **Unique Constraints**: When questions use words like "only", "unique", or "single", verify that exactly one item matches the criteria. If multiple items match, re-examine the constraints.
+- **Sequential/Ordered Data**: For questions about sequences, rankings, or ordered lists (jersey numbers, chronological order, etc.), carefully verify the exact position or order from authoritative sources.
+
+### ERROR HANDLING
+
+- If a tool fails, try again with a different query or approach
+- If multiple sources give conflicting information, use the most authoritative source
+- If websearch returns results but you need more detail, use get_webpage_content on the most relevant URL
+- If you cannot find the answer after exhausting all tools and approaches, output: Unable to determine [brief reason]
+
+### REMEMBER
+
+Your intermediate reasoning and tool usage are separate from your final output. Think through the problem, use tools as needed, but when you output your final answer, it must be ONLY the answer value with NO additional text.
+
+### ABSOLUTE FINAL RULE
+
+After all reasoning and tool calls, your LAST message must be the BARE ANSWER ONLY — one word, one number, or a short comma-separated list. No sentence. No explanation. No prefix. If you find yourself writing a sentence as your final output, STOP, DELETE it, and output only the answer value.
+
+WRONG: "The answer based on my research is Jane Smith"
+RIGHT: Jane
+
+WRONG: "In that same season, Alex Brown had 412 at-bats"
+RIGHT: 412
+
+WRONG: "Brazil comes first alphabetically, so the answer is BRA"
+RIGHT: BRA
+"""

utils.py

@@ -0,0 +1,195 @@
+"""Utility functions for GAIA Benchmark Agent including retry logic and answer cleanup."""
+
+import re
+import time
+import requests
+from typing import Callable, Any
+from functools import wraps
+import config
+
+
+def retry_with_backoff(
+    max_retries: int = config.MAX_RETRIES,
+    initial_delay: float = config.INITIAL_RETRY_DELAY,
+    backoff_factor: float = config.RETRY_BACKOFF_FACTOR,
+    exceptions: tuple = (requests.RequestException,)
+):
+    """
+    Decorator to retry a function with exponential backoff.
+
+    Args:
+        max_retries: Maximum number of retry attempts
+        initial_delay: Initial delay in seconds before first retry
+        backoff_factor: Multiplier for delay after each retry
+        exceptions: Tuple of exception types to catch and retry
+    """
+    def decorator(func: Callable) -> Callable:
+        @wraps(func)
+        def wrapper(*args, **kwargs) -> Any:
+            delay = initial_delay
+            last_exception = None
+
+            for attempt in range(max_retries + 1):
+                try:
+                    return func(*args, **kwargs)
+                except exceptions as e:
+                    last_exception = e
+                    if attempt < max_retries:
+                        print(f"[RETRY] Attempt {attempt + 1}/{max_retries} failed: {e}")
+                        print(f"[RETRY] Retrying in {delay:.1f} seconds...")
+                        time.sleep(delay)
+                        delay *= backoff_factor
+                    else:
+                        print(f"[RETRY] All {max_retries} retries exhausted")
+
+            # Re-raise the last exception if all retries failed
+            raise last_exception
+
+        return wrapper
+    return decorator
+
+
+def extract_text_from_content(content: Any) -> str:
+    """
+    Extract plain text from various content formats returned by LLM agents.
+
+    This function handles multiple content formats:
+    - AgentOutput objects (LlamaIndex): Extracts the response attribute
+    - Message objects with 'content' attribute: Extracts the content attribute
+      (works for LlamaIndex ChatMessage, LangChain AIMessage, etc.)
+    - String: Returns as-is
+    - Dict with 'text' field: Extracts the text value
+    - List of content blocks: Extracts text from all blocks with type='text'
+    - Other types: Converts to string
+
+    Args:
+        content: The content object from an LLM response (can be str, dict, list, etc.)
+
+    Returns:
+        str: Extracted plain text content
+    """
+    # Handle LlamaIndex AgentOutput objects (has 'response' attribute)
+    if hasattr(content, 'response') and not isinstance(content, (str, dict, list)):
+        # Extract the response attribute from AgentOutput
+        response = content.response
+        # The response might itself be a message object with 'content'
+        if hasattr(response, 'content'):
+            return str(response.content)
+        elif hasattr(response, 'message') and hasattr(response.message, 'content'):
+            return str(response.message.content)
+        else:
+            return str(response)
+
+    # Handle message objects with 'content' attribute (e.g., ChatMessage from various frameworks)
+    # This works for LlamaIndex ChatMessage, LangChain AIMessage, etc.
+    if hasattr(content, 'content') and not isinstance(content, (str, dict, list)):
+        # Extract the content attribute (works for any message object)
+        return str(content.content)
+
+    # Handle dict format (e.g., {'text': 'answer'})
+    if isinstance(content, dict):
+        if 'text' in content:
+            return str(content['text'])
+        else:
+            print(f"[WARNING] Content was dict without 'text' field, converting to string")
+            return str(content)
+
+    # Handle list format (e.g., [{'type': 'text', 'text': 'answer'}])
+    elif isinstance(content, list):
+        text_parts = []
+        for item in content:
+            if isinstance(item, dict):
+                # Look for items with type='text' and extract the 'text' field
+                if item.get('type') == 'text':
+                    text_parts.append(str(item.get('text', '')))
+                # Fallback: if there's a 'text' field but no type, use it
+                elif 'text' in item:
+                    text_parts.append(str(item['text']))
+            elif isinstance(item, str):
+                text_parts.append(item)
+            else:
+                text_parts.append(str(item))
+
+        result = ' '.join(text_parts)
+        if len(content) > 1 or (len(content) == 1 and isinstance(content[0], dict)):
+            print(f"[INFO] Extracted text from list with {len(content)} item(s)")
+        return result
+
+    # Handle string format (already plain text)
+    elif isinstance(content, str):
+        return content
+
+    # Fallback for other types
+    else:
+        print(f"[WARNING] Content was {type(content)}, converting to string")
+        return str(content)
+
+
+def cleanup_answer(answer: Any) -> str:
+    """
+    Clean up the agent answer to ensure it's in plain text format.
+
+    This function:
+    - Converts answer to string
+    - Handles multi-line answers (extracts last meaningful non-debug line)
+    - Normalizes whitespace
+    - Strips trailing punctuation
+    - Logs warnings for verbose or malformatted answers
+
+    Args:
+        answer: The raw answer from the agent (can be str, dict, list, etc.)
+
+    Returns:
+        str: Cleaned up answer as plain text
+    """
+    answer = str(answer).strip()
+
+    if not answer:
+        return answer
+
+    # Handle multi-line: take the last line that isn't a debug/log prefix
+    lines = [l.strip() for l in answer.split('\n') if l.strip()]
+    if len(lines) > 1:
+        debug_prefixes = ('[info', '[warning', '[error', '[retry', '[step', '[tool', '[final')
+        for l in reversed(lines):
+            if not l.lower().startswith(debug_prefixes):
+                answer = l
+                break
+        else:
+            answer = lines[-1]
+        print(f"[CLEANUP] Extracted last meaningful line from {len(lines)}-line answer: '{answer[:80]}'")
+
+    # NOTE: Do NOT strip commas here. The GAIA scorer's normalize_number_str already
+    # strips commas from numeric answers, and split_string uses commas to split list
+    # answers. Stripping here would corrupt comma-separated lists (e.g., "132,133,134"
+    # becomes the invalid number string "132133134").
+
+    # Normalize whitespace and strip trailing punctuation
+    answer = ' '.join(answer.split()).strip().rstrip('.')
+
+    # Sentence.NUMBER suffix — the model echoed its final answer as a bare number
+    # appended directly after its reasoning, e.g. "...published 3 albums (included).3".
+    # Match a NON-DIGIT char before the period (covers letters, ')', etc.) and require
+    # whitespace earlier in the string so genuine bare decimals like "89706.00" or
+    # "3.14" (no spaces) are never altered.
+    if ' ' in answer and re.search(r'[^\d\s]\s*\.\d+$', answer):
+        extracted = re.search(r'\.(\d+)$', answer).group(1)
+        print(f"[CLEANUP] Extracted appended number from verbose answer: '{extracted}'")
+        answer = extracted
+
+    # NOTE: We deliberately do NOT regex-extract a bare answer out of a verbose
+    # sentence. The model is instructed to emit only the bare answer, and earlier
+    # extraction patterns here were reverse-engineered from specific GAIA questions —
+    # an integrity and maintenance hazard. If the model returns prose, the right fix
+    # is the prompt/model, not question-tuned post-processing.
+
+    # Log if answer looks verbose (agent not following instructions)
+    if len(answer) > 100:
+        print(f"[WARNING] Answer appears verbose ({len(answer)} chars). Agent may not be following SYSTEM_PROMPT instructions.")
+        print(f"[WARNING] First 150 chars: {answer[:150]}...")
+
+    # Log if answer contains suspicious formatting characters
+    if any(char in answer for char in ['{', '}', '[', ']', '`', '*', '#']):
+        print(f"[WARNING] Answer contains suspicious formatting characters: {answer[:100]}")
+
+    return answer

validators.py

@@ -0,0 +1,115 @@
+"""Input validation utilities."""
+
+import re
+from typing import List, Optional, Tuple
+
+
+class ValidationError(Exception):
+    """Custom exception for validation errors."""
+    pass
+
+
+class InputValidator:
+    """Validates user inputs."""
+
+    @staticmethod
+    def validate_username(username: str) -> str:
+        """
+        Validate username for submission.
+
+        Args:
+            username: The username to validate
+
+        Returns:
+            Cleaned username
+
+        Raises:
+            ValidationError: If username is invalid
+        """
+        if not username or not username.strip():
+            raise ValidationError("Username cannot be empty")
+
+        cleaned = username.strip()
+
+        if len(cleaned) < 3:
+            raise ValidationError("Username must be at least 3 characters")
+
+        if len(cleaned) > 50:
+            raise ValidationError("Username must be less than 50 characters")
+
+        # Allow alphanumeric, underscore, hyphen
+        if not re.match(r'^[a-zA-Z0-9_-]+$', cleaned):
+            raise ValidationError("Username can only contain letters, numbers, underscore, and hyphen")
+
+        return cleaned
+
+    @staticmethod
+    def validate_filter_indices(filter_list: Optional[Tuple], max_index: int) -> Optional[List[int]]:
+        """
+        Validate filter indices for test questions.
+
+        Args:
+            filter_list: Tuple/list of indices or None
+            max_index: Maximum valid index (exclusive)
+
+        Returns:
+            Validated list of indices or None
+
+        Raises:
+            ValidationError: If indices are invalid
+        """
+        if filter_list is None:
+            return None
+
+        if not isinstance(filter_list, (list, tuple)):
+            raise ValidationError("Filter must be a list or tuple")
+
+        if not filter_list:
+            raise ValidationError("Filter cannot be empty (use None for all questions)")
+
+        validated = []
+        for idx in filter_list:
+            if not isinstance(idx, int):
+                raise ValidationError(f"Filter index must be integer, got {type(idx)}")
+
+            if idx < 0:
+                raise ValidationError(f"Filter index cannot be negative: {idx}")
+
+            if idx >= max_index:
+                raise ValidationError(f"Filter index {idx} out of range (max: {max_index - 1})")
+
+            validated.append(idx)
+
+        return validated
+
+    @staticmethod
+    def validate_questions_data(questions_data: any) -> List[dict]:
+        """
+        Validate questions data structure.
+
+        Args:
+            questions_data: Data to validate
+
+        Returns:
+            Validated questions list
+
+        Raises:
+            ValidationError: If data is invalid
+        """
+        if not isinstance(questions_data, list):
+            raise ValidationError(f"Questions data must be a list, got {type(questions_data)}")
+
+        if not questions_data:
+            raise ValidationError("Questions list is empty")
+
+        for idx, item in enumerate(questions_data):
+            if not isinstance(item, dict):
+                raise ValidationError(f"Question {idx} must be a dict, got {type(item)}")
+
+            if "task_id" not in item:
+                raise ValidationError(f"Question {idx} missing 'task_id'")
+
+            if "question" not in item:
+                raise ValidationError(f"Question {idx} missing 'question'")
+
+        return questions_data