docs/devplan.md

# GAIA Agent Development Plan
This document outlines a structured approach to developing an agent that can successfully solve a subset of the GAIA benchmark, focusing on understanding the task, designing the agent architecture, and planning the development process.

**I. Understanding the Task & Data:**

1.  **Analyze common_questions.json:**
    *   **Structure:** Each entry has `task_id`, `Question`, `Level`, `Final answer`, and sometimes `file_name`.
    *   **Question Types:** Identify patterns:
        *   Direct information retrieval (e.g., "How many studio albums...").
        *   Web search required (e.g., "On June 6, 2023, an article...").
        *   File-based questions (audio, images, code - indicated by `file_name`).
        *   Logic/reasoning puzzles (e.g., the table-based commutativity question, reversed sentence).
        *   Multistep questions.
    *   **Answer Format:** Observe the format of `Final answer` for each type. Note the guidance in `docs/submission_instructions.md` regarding formatting (numbers, few words, comma-separated lists).
    *   **File Dependencies:** List all unique `file_name` extensions to understand what file processing capabilities are needed (e.g., `.mp3`, `.png`, `.py`, `.xlsx`).

2.  **Review Project Context:**
    *   **Agent Interface:** The agent will need to fit into the `BasicAgent` structure in `app.py` (i.e., an `__init__` and a `__call__(self, question: str) -> str` method).
    *   **Evaluation:** Keep `docs/testing_recipe.md` and the `normalize` function in mind for how answers will be compared.
    *   **Model:** The agent will use an LLM (like the Llama 3 model mentioned in `docs/log.md`).

**II. Agent Architecture Design (Conceptual):**

1.  **Core Agent Loop (`MyAgent.answer` or `MyAgent.__call__`):**
    *   **Input:** Question string (and `task_id`/`file_name` if passed separately or parsed from a richer input object).
    *   **Step 1: Question Analysis & Planning:**
        *   Use the LLM to understand the question.
        *   Determine the type of question (web search, file processing, direct knowledge, etc.).
        *   Identify if any tools are needed.
        *   Formulate a high-level plan (e.g., "Search web for X, then extract Y from the page").
    *   **Step 2: Tool Selection & Execution (if needed):**
        *   Based on the plan, select and invoke appropriate tools.
        *   Pass necessary parameters to tools (e.g., search query, file path).
        *   Collect tool outputs.
    *   **Step 3: Information Synthesis & Answer Generation:**
        *   Use the LLM to process tool outputs and any retrieved information.
        *   Generate the final answer string.
    *   **Step 4: Answer Formatting:**
        *   Ensure the answer conforms to the expected format (using guidance from common_questions.json examples and `docs/submission_instructions.md`). This might involve specific post-processing rules or prompting the LLM for a specific format.
    *   **Output:** Return the formatted answer string.

2.  **Key Modules/Components:**

    *   **LLM Interaction Module:**
        *   Handles communication with the chosen LLM (e.g., GPT4All Llama 3).
        *   Manages prompt construction (system prompts, user prompts, few-shot examples if useful).
        *   Parses LLM responses.
    *   **Tool Library:** A collection of functions/classes that the agent can call.
        *   `WebSearchTool`:
            *   Input: Search query.
            *   Action: Uses a search engine API (or simulates browsing if necessary, though direct API is better).
            *   Output: List of search results (titles, snippets, URLs) or page content.
        *   `FileReaderTool`:
            *   Input: File path (derived from `file_name` and `task_id` to locate/fetch the file).
            *   Action: Reads content based on file type.
                *   Text files (`.py`): Read as string.
                *   Spreadsheets (`.xlsx`): Parse relevant data (requires a library like `pandas` or `openpyxl`).
                *   Audio files (`.mp3`): Transcribe to text (requires a speech-to-text model/API).
                *   Image files (`.png`): Describe image content or extract text (requires a vision model/API or OCR).
            *   Output: Processed content (text, structured data).
        *   `CodeInterpreterTool` (for `.py` files like in task `f918266a-b3e0-4914-865d-4faa564f1aef`):
            *   Input: Python code string.
            *   Action: Executes the code in a sandboxed environment.
            *   Output: Captured stdout/stderr or final expression value.
        *   *(Potentially)* `KnowledgeBaseTool`: If there's a way to pre-process or index relevant documents/FAQs for faster lookups (though most GAIA questions imply dynamic information retrieval).
    *   **File Management/Access:**
        *   Mechanism to locate/download files associated with `task_id` and `file_name`. The API endpoint `GET /files/{task_id}` from `docs/API.md` is relevant here. For local testing with common_questions.json, ensure these files are available locally.
    *   **Prompt Engineering Strategy:**
        *   Develop a set of system prompts to guide the agent's behavior (e.g., "You are a helpful AI assistant designed to answer questions from the GAIA benchmark...").
        *   Develop task-specific prompts or prompt templates for different question types or tool usage.
        *   Incorporate answer formatting instructions into prompts.

**III. Development & Testing Strategy:**

1.  **Environment Setup:**
    *   Install necessary Python libraries for LLM interaction, web requests, file processing (e.g., `requests`, `beautifulsoup4` (for web scraping if needed), `pandas`, `Pillow` (for images), speech recognition libraries, etc.).
2.  **Iterative Implementation:**
    *   **Phase 1: Basic LLM Agent:** Start with an agent that only uses the LLM for direct-answer questions (no tools).
    *   **Phase 2: Web Search Integration:** Implement the `WebSearchTool` and integrate it for questions requiring web lookups.
    *   **Phase 3: File Handling:**
        *   Implement `FileReaderTool` for one file type at a time (e.g., start with `.txt` or `.py`, then `.mp3`, `.png`, `.xlsx`).
        *   Implement `CodeInterpreterTool`.
    *   **Phase 4: Complex Reasoning & Multi-step:** Refine the planning and synthesis capabilities of the LLM to handle more complex, multi-step questions that might involve multiple tool uses.
3.  **Testing:**
    *   Use `common_questions.json` as the primary test set.
    *   Adapt the script from `docs/testing_recipe.md` to run your agent against these questions and compare outputs.
    *   Focus on one question type or `task_id` at a time for debugging.
    *   Log agent's internal "thoughts" (plan, tool calls, tool outputs) for easier debugging.

**IV. Pre-computation/Pre-analysis (before coding):**

1.  **Map Question Types to Tools:** For each question in common_questions.json, manually note which tool(s) would ideally be used. This helps prioritize tool development.
    *   Example:
        *   `8e867cd7-cff9-4e6c-867a-ff5ddc2550be` (Mercedes Sosa albums): WebSearchTool
        *   `cca530fc-4052-43b2-b130-b30968d8aa44` (Chess): FileReaderTool (image) + Vision/Chess Engine Tool (or very advanced LLM vision)
        *   `99c9cc74-fdc8-46c6-8f8d-3ce2d3bfeea3` (Pie ingredients): FileReaderTool (audio) + SpeechToText
        *   `f918266a-b3e0-4914-865d-4faa564f1aef` (Python output): FileReaderTool (code) + CodeInterpreterTool
2.  **Define Tool Interfaces:** Specify the exact input/output signature for each planned tool.