| --- |
| title: Template Final Assignment |
| emoji: 🕵🏻♂️ |
| colorFrom: indigo |
| colorTo: indigo |
| sdk: gradio |
| sdk_version: 5.25.2 |
| app_file: app.py |
| pinned: false |
| hf_oauth: true |
| |
| hf_oauth_expiration_minutes: 480 |
| --- |
| |
| # Local-Model GAIA Benchmark Agent |
|
|
| This project is my final assignment agent for the Hugging Face agents course benchmark. The goal was to see how close I could get to a perfect benchmark score while using a model that I could run locally instead of relying on a hosted frontier model. |
|
|
| The model used for the successful test runs was: |
|
|
| ```text |
| gemma-4-31b-it |
| ``` |
|
|
| It was served through a local OpenAI-compatible vLLM endpoint. |
|
|
| ## Important Caveat |
|
|
| This project is intentionally benchmark-maxed. It includes targeted tools for task patterns found in the benchmark, such as historical MLB stats, NPB roster lookup, Wikipedia table extraction, chess-board solving, Olympic participant tables, and Malko Competition lookup. |
|
|
| That was done as an educational exercise to understand where a local model fails, how tool design changes agent behavior, and how structured retrieval can outperform broad web search. This should not be read as a general-purpose agent architecture. A true general agent would need broader tool selection, better uncertainty handling, source validation, and less benchmark-specific routing. |
|
|
| ## Result |
|
|
| Using a clean answer cache, the agent generated and submitted all 20 answers and received: |
|
|
| ```text |
| 20/20 correct |
| 100.0% |
| ``` |
|
|
| ## What Made It Work |
|
|
| The local model was good enough for many tasks, but several benchmark items needed structured tools instead of free-form search or vision-only reasoning: |
|
|
| - Chess image task: `chess-fen-detector` reads the board, then Stockfish chooses the move. |
| - MLB stats task: official MLB Stats API avoids misleading search snippets. |
| - NPB roster task: official NPB registered roster pages avoid current-roster drift. |
| - Wikipedia table tasks: `pandas.read_html` extracts the exact table and revision instead of asking the model to reason over noisy page text. |
| - Olympics task: structured country/athlete table parsing avoids guessed Wikipedia URLs. |
| - Malko task: direct table extraction avoids open-ended web search. |
| - Audio tasks: `faster-whisper` transcribes attachments before the model sees them. |
| - Video tasks: YouTube videos are downloaded, sampled into frames, and sent as image inputs. |
|
|
| ## Non-Sensitive Configuration |
|
|
| The successful 20/20 run used these relevant settings. Real tokens, private keys, and machine-specific network details are intentionally omitted. |
|
|
| ```env |
| LOCAL_LLM_BASE_URL=http://<local-vllm-host>:8000/v1 |
| LOCAL_LLM_MODEL=gemma-4-31b-it |
| LOCAL_LLM_TIMEOUT=300 |
| LOCAL_LLM_API_KEY=<dummy-or-local-api-key> |
| |
| WEB_SEARCH_MAX_RESULTS=500 |
| WEB_FETCH_TIMEOUT=300 |
| WEB_FETCH_USER_AGENT=<browser-like-user-agent> |
| |
| AGENT_MAX_WORKERS=4 |
| AGENT_REQUEST_LIMIT=20 |
| AGENT_TOOL_CALLS_LIMIT=20 |
| AGENT_ANSWER_CACHE_PATH=.agent_cache/answers.json |
| |
| YOUTUBE_DOWNLOAD_DIR=.agent_cache/youtube |
| YOUTUBE_FORMAT=bestvideo[height<=480]+bestaudio/best[height<=480]/best |
| |
| VIDEO_DOWNLOAD_DIR=.agent_cache/videos |
| VIDEO_FRAME_CACHE_DIR=.agent_cache/video_frames |
| VIDEO_FRAME_FPS=1 |
| VIDEO_FRAME_MAX_FRAMES=30 |
| |
| AUDIO_TRANSCRIPT_CACHE_DIR=.agent_cache/transcripts |
| ASR_MODEL_SIZE=base |
| ASR_DEVICE=auto |
| ASR_COMPUTE_TYPE=default |
| |
| STOCKFISH_PATH=/path/to/stockfish |
| CHESS_ENGINE_TIME_LIMIT=2 |
| CHESS_BOARD_ORIENTATION=black |
| CHESS_DEFAULT_TURN=black |
| ``` |
|
|
| Do not commit: |
|
|
| - Hugging Face tokens |
| - Private API keys |
| - Private hostnames or IP addresses |
| - Local cache contents |
|
|
| ## Dependencies |
|
|
| The core additions beyond the template include: |
|
|
| - `pydantic-ai` |
| - `pydantic-ai-slim[duckduckgo,web-fetch]` |
| - `faster-whisper` |
| - `yt-dlp` |
| - `chess` |
| - `chess-fen-detector` |
| - `beautifulsoup4` |
| - `pandas` |
| - `openpyxl` |
| - `pypdf` |
|
|
| Stockfish must also be installed locally and referenced with `STOCKFISH_PATH`. |
|
|
| ## Running |
|
|
| Create a `.env` file with the sanitized configuration above adapted to your machine, install requirements, start your local vLLM server, then run: |
|
|
| ```bash |
| python app.py |
| ``` |
|
|
| The Gradio UI can fetch the benchmark questions, run the agent, cache answers, and submit the result. |
|
|
