Refactoring and started filling in the README file

README.md

@@ -12,4 +12,94 @@ hf_oauth: true
 hf_oauth_expiration_minutes: 480
 ---
 
+# General AI Assistant
+
+## Background
+Created as a final project for the HuggingFace Agents course ( https://huggingface.co/learn/agents-course).
+Aims to answer Level 1 questions from the **GAIA** validation set. It was tested on 20 such questions with a success rate of 65%.
+### GAIA
+
+GAIA is a benchmark for AI assistants evaluation on real-world tasks that require a combination of capabilities—such
+as reasoning, multimodal understanding, web browsing, and proficient tool use (see https://huggingface.co/learn/agents-course/unit4/what-is-gaia).
+
+GAIA was introduced in the paper [”GAIA: A Benchmark for General AI Assistants”](https://arxiv.org/abs/2311.12983).
+
+
+The questions challenge AI systems in several ways:
+
+- Involve multimodal reasoning (e.g., analyzing images, audio, documents)
+- Demand multi-hop retrieval of interdependent facts
+- Involve running python code
+- Require a structured response format
+
+
+## Implementation Highlights
+
+ 
+
+**The agent** is implemented using the LangGraph framework.
+
+**Nodes**:
+
+
+**Tools**
+
+🔎 **Web Search**: uses `tavily` search and extract tools. 
+
+- **Chunking**: The content returned by the exact might be too large to be further analyzed at once by a model (depending on the chosen model context window size or on the rate limitation),
+so if its size exceeds a pre-configured threshold, it will be chunked and only the most relevant chunks will be analyzed.
+    - **Text Splitting**: First by markdown (used Langchain's `MarkdownHeaderTextSplitter`) and then further by size with a sliding window (used LangChain's `RecursiveCharacterTextSplitter`). 
+    - **Embeddings**:  langchain_community.embeddings.OpenAIEmbeddings
+    - **Vector DB**: FAISS vector db. 
+    - **Retrieval**: FAISS similarity search
+
+  Updated the original extract tool response message content only with the relevant chunks content.
+
+🔉 **Audio**: uses `gpt-4o-audio-preview` to analyze the input
+
+🧮 **Math problems**: this is a subagent that uses `gpt-5` equipped with the following tools:
+
+- **Pyhton code executor**: executes a snipped of python code provided as input
+- **Think tool**: used for strategic reflection on the progress of the solving process
+
+**The Math Agent States:**
+
+
+🧩 **Python code**
+This tool can run either a snippet of python code or a python file. The python file is executed in a sub-process.
+
+📊 **Spreadsheets**
+In order to analyze `excel` files this tool uses the pandas dataframe agent 
+    `langchain_experimental.agents import create_pandas_dataframe_agent`
+It uses `gpt-4.1` model.
+
+♟️ **Chess**
+Given a chess board and the active color, this tool is able to suggest the best move to be performed by the active color.
+
+- **Picture analysis**: the tool must detect    
+
+🎥 **Videos**
+
+
+
+## Challenges 
+
+
+
+## Future improvements
+#### 1. Evaluation
+#### 2. Chunking
+#### 3. Audio Analysis
+#### 3. Video Analysis
+#### 4. Chessboard Images analysis
+
+## References:
+https://github.com/langchain-ai/open_deep_research
+
+
+
+
+
+
+
 Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

core/messages.py

@@ -40,28 +40,11 @@ class AttachmentHandler:
     def __init__(self, supported_types: list):
         self.supported_types = supported_types
 
-    def get_attachment_representation(self, attachment: Attachment) -> dict:
-        if attachment.type not in self.supported_types:
-            raise Exception(f"Invalid attachment type{attachment.type}")
-
-        if attachment.type == "image":
-            return {"type": "image_url",
-                    "image_url": {"url": f"data:{attachment.mime_type};base64," + attachment.get_encoded_content_b64()}}
-
-        if attachment.type == "audio":
-            return {"type": "text",
-                    "text": attachment.get_encoded_content_b64()}
-        if attachment.type == "text":
-            return {"type": attachment.type, "data": attachment.content, "mime_type": attachment.mime_type}
-
-        # The remaining types are image, file, audio
-        return {"type": attachment.type, "source": "base64", "data": attachment.get_encoded_content_b64(),
-                "mime_type": attachment.mime_type}
-
     def get_representation(self, type: str, content: bytes, format: str, mime_type) -> dict:
-        base64_content = base64.b64encode(content).decode("utf-8")
         if type not in self.supported_types:
             raise Exception(f"Invalid attachment type{type}")
+        base64_content = base64.b64encode(content).decode("utf-8")
+
         if type == "audio":
             return {"type": "input_audio",
                     "input_audio": {"data": base64_content, "format": format}}
@@ -72,18 +55,12 @@ class AttachmentHandler:
         raise Exception(f"Cannot extract a representation for type {type}")
 
     def fetch_file_from_reference(self, file_reference: str) -> bytes:
-        """Fetches file bytes from a reference (e.g., S3, local path, URL)."""
-
+        """Fetches file bytes from a reference """
         #  It's a local file path
         file = Path(file_reference)
         if file_reference.startswith("/") or file_reference.startswith("./") or file.exists():
             return file.read_bytes()
-
-
-        # Example 3: It's an ID in your database (pseudocode)
         else:
-            # file_bytes = database.lookup_file_bytes(file_reference)
-            # return file_bytes
             raise ValueError(
                 f"Could not resolve file reference: {file_reference}. Implement 'fetch_file_from_reference' for your "
                 f"storage system.")

nodes/nodes.py

@@ -66,7 +66,7 @@ def assistant(state: State):
         response = model.invoke([sys_msg] + state["messages"])
     except Exception as e:
         if "429" in str(e):
-            time.sleep(5)
+            time.sleep(20)
             print("Retrying after receiving 429 error")
             response = model.invoke([sys_msg] + state["messages"])
             return {"messages": [response]}