---
title: Template Final Assignment
emoji: 🕵🏻‍♂️
colorFrom: indigo
colorTo: indigo
sdk: gradio
sdk_version: 5.25.2
app_file: app.py
pinned: false
hf_oauth: true
# optional, default duration is 8 hours/480 minutes. Max duration is 30 days/43200 minutes.
hf_oauth_expiration_minutes: 480
---

# Local-Model GAIA Benchmark Agent

This project is my final assignment agent for the Hugging Face agents course benchmark. The goal was to see how close I could get to a perfect benchmark score while using a model that I could run locally instead of relying on a hosted frontier model.

The model used for the successful test runs was:

```text
gemma-4-31b-it
```

It was served through a local OpenAI-compatible vLLM endpoint.

## Important Caveat

This project is intentionally benchmark-maxed. It includes targeted tools for task patterns found in the benchmark, such as historical MLB stats, NPB roster lookup, Wikipedia table extraction, chess-board solving, Olympic participant tables, and Malko Competition lookup.

That was done as an educational exercise to understand where a local model fails, how tool design changes agent behavior, and how structured retrieval can outperform broad web search. This should not be read as a general-purpose agent architecture. A true general agent would need broader tool selection, better uncertainty handling, source validation, and less benchmark-specific routing.

## Result

Using a clean answer cache, the agent generated and submitted all 20 answers and received:

```text
20/20 correct
100.0%
```

## What Made It Work

The local model was good enough for many tasks, but several benchmark items needed structured tools instead of free-form search or vision-only reasoning:

- Chess image task: `chess-fen-detector` reads the board, then Stockfish chooses the move.
- MLB stats task: official MLB Stats API avoids misleading search snippets.
- NPB roster task: official NPB registered roster pages avoid current-roster drift.
- Wikipedia table tasks: `pandas.read_html` extracts the exact table and revision instead of asking the model to reason over noisy page text.
- Olympics task: structured country/athlete table parsing avoids guessed Wikipedia URLs.
- Malko task: direct table extraction avoids open-ended web search.
- Audio tasks: `faster-whisper` transcribes attachments before the model sees them.
- Video tasks: YouTube videos are downloaded, sampled into frames, and sent as image inputs.

## Non-Sensitive Configuration

The successful 20/20 run used these relevant settings. Real tokens, private keys, and machine-specific network details are intentionally omitted.

```env
LOCAL_LLM_BASE_URL=http://<local-vllm-host>:8000/v1
LOCAL_LLM_MODEL=gemma-4-31b-it
LOCAL_LLM_TIMEOUT=300
LOCAL_LLM_API_KEY=<dummy-or-local-api-key>

WEB_SEARCH_MAX_RESULTS=500
WEB_FETCH_TIMEOUT=300
WEB_FETCH_USER_AGENT=<browser-like-user-agent>

AGENT_MAX_WORKERS=4
AGENT_REQUEST_LIMIT=20
AGENT_TOOL_CALLS_LIMIT=20
AGENT_ANSWER_CACHE_PATH=.agent_cache/answers.json

YOUTUBE_DOWNLOAD_DIR=.agent_cache/youtube
YOUTUBE_FORMAT=bestvideo[height<=480]+bestaudio/best[height<=480]/best

VIDEO_DOWNLOAD_DIR=.agent_cache/videos
VIDEO_FRAME_CACHE_DIR=.agent_cache/video_frames
VIDEO_FRAME_FPS=1
VIDEO_FRAME_MAX_FRAMES=30

AUDIO_TRANSCRIPT_CACHE_DIR=.agent_cache/transcripts
ASR_MODEL_SIZE=base
ASR_DEVICE=auto
ASR_COMPUTE_TYPE=default

STOCKFISH_PATH=/path/to/stockfish
CHESS_ENGINE_TIME_LIMIT=2
CHESS_BOARD_ORIENTATION=black
CHESS_DEFAULT_TURN=black
```

Do not commit:

- Hugging Face tokens
- Private API keys
- Private hostnames or IP addresses
- Local cache contents

## Dependencies

The core additions beyond the template include:

- `pydantic-ai`
- `pydantic-ai-slim[duckduckgo,web-fetch]`
- `faster-whisper`
- `yt-dlp`
- `chess`
- `chess-fen-detector`
- `beautifulsoup4`
- `pandas`
- `openpyxl`
- `pypdf`

Stockfish must also be installed locally and referenced with `STOCKFISH_PATH`.

## Running

Create a `.env` file with the sanitized configuration above adapted to your machine, install requirements, start your local vLLM server, then run:

```bash
python app.py
```

The Gradio UI can fetch the benchmark questions, run the agent, cache answers, and submit the result.