--- title: GAIA Agent - Agents Course Unit 4 emoji: 🤗 colorFrom: indigo colorTo: blue sdk: gradio sdk_version: 6.17.3 app_file: app.py pinned: false hf_oauth: true hf_oauth_expiration_minutes: 480 --- ### How the submitted answers were produced The 20 answers were generated by running this exact code locally with Gemini on Vertex AI, using Application Default Credentials. This Space only hosts the public code, so the agent can't run here without Google Cloud credentials. The Unit 4 scorer returned 20/20. # GAIA Agent — Hugging Face Agents Course Unit 4 A ground-up `smolagents` CodeAgent for the Unit 4 GAIA evaluation. ## What this project does The agent combines ideas from the earlier Agents Course units: - Unit 1: ReAct-style **Think → Act → Observe**, tools, and final-answer discipline. - Unit 2.1: `smolagents` **CodeAgent**, custom tools, web search, webpage reading, and Python execution. - Unit 3: an agentic retrieval pattern where the agent decides which retrieval/tool flow to use. - Unit 4: the official 20-question GAIA subset, exact-match scoring, and Hugging Face submission. The implementation is intentionally modular: ```text app.py # Gradio UI + evaluation/submission flow (resumes from progress.json) agent.py # Agent construction, question-type playbook, answer normalization gaia_tools.py # All tools (see below) gemini_limits.py # Per-endpoint rate limiting, cooldowns, failover rules vertex_auth.py # Vertex AI ADC and service-account authentication providers.py # Reads your API keys, builds the ordered provider/model list progress.py # Saves solved answers so a run can resume after a quota failure check_key.py # Shows which models your key can use right now requirements.txt # Python packages packages.txt # System packages for a Space (Stockfish for the chess question) ``` ### Tools | Tool | Used for | |---|---| | `wikipedia_page` | Full Wikipedia article text incl. tables, optionally as of a year (`year=2022`) or one `section` | | `wikipedia_search` | Find exact titles (articles and `Wikipedia:` project pages) | | `web_search` | DuckDuckGo then Bing | | `google_search` | Google-grounded answer with sources via Vertex Gemini (hard multi-hop lookups) | | `visit_webpage` | Open any URL as markdown | | `inspect_attachment` | Images (vision), audio (transcript), xlsx, csv, pdf, docx, text | | `run_attached_python` | Runs an attached `.py` (60 s limit, API keys hidden; disable with `ALLOW_RUN_ATTACHED_PYTHON=0`) | | `chess_best_move` | FEN to best move with Stockfish (mate search fallback) | | `youtube_transcript` / `analyze_youtube_video` | Spoken words / what is shown or heard | | `python_interpreter` | Calculations, pandas/numpy on spreadsheets | If the course API returns `404` for an attachment, `inspect_attachment` falls back to the matching file in GAIA's public validation dataset. To enable that fallback, accept the dataset's access terms at https://huggingface.co/datasets/gaia-benchmark/GAIA and add a read token as the `GAIA_HF_TOKEN` Space secret (or `HF_TOKEN`). The official course endpoint is always tried first. The fallback reads only task IDs and attachment paths from the dataset metadata; it does not load or use benchmark answers. ### Unit 4 rules this project follows - 20 Level-1 GAIA validation questions; scoring is **exact match**, so only the bare answer is submitted. - You need **more than 30 %** (at least 7 of 20) for the certificate. - Submission = username + public Space code URL (`.../tree/main`) + `{"task_id", "submitted_answer"}` list. - Keep the Space **public** to stay on the leaderboard. ## API keys (set the ones you have) **Locally:** copy `.env.example` to `.env` and fill it in (loaded automatically). **On a Hugging Face Space:** add them under Settings -> Secrets. Do **not** upload `.env` to a public Space: anyone could read your keys. (Real environment variables always override `.env`.) The agent "brain" can use several providers and **fails over between them in the middle of a question** when one runs out of quota. Set any of these (Space secrets or environment variables): ```text NVIDIA_API_KEY=nvapi-... # NVIDIA NIM; default openai/gpt-oss-20b (currently free endpoint) GROQ_API_KEY=... # openai/gpt-oss-120b (active; pricing/access depends on account tier) OPENROUTER_API_KEY=... # openrouter/free (dynamic free-model router; 50 requests/day on free accounts) GOOGLE_CLOUD_PROJECT=... # Vertex AI project; also required for Gemini image/audio/video/search tools GOOGLE_CLOUD_LOCATION=global GOOGLE_CLOUD_CREDENTIALS_JSON=... # Optional service-account JSON; otherwise use ADC GAIA_HF_TOKEN=hf_... # Optional: required only for GAIA attachment fallback after a scorer 404 ``` Order of use: `custom, nvidia, groq, openrouter, gemini` (change with `PROVIDER_ORDER`). Vertex Gemini is last so its quota is kept for the multimodal tools. Groq is skipped automatically for big prompts (its per-minute token budget is tiny). Provider limits and free-model availability change. The defaults use currently documented model IDs, while the local RPM values are conservative scheduler settings rather than provider guarantees. Override models with `NVIDIA_MODELS` (default `openai/gpt-oss-20b`), `GROQ_MODELS`, `OPENROUTER_MODELS` (default `openrouter/free`, comma-separated) and budgets with `NVIDIA_RPM`, `GROQ_RPM`, `VERTEX_GEMINI_RPM`, ... Vertex limits are enforced against the Google Cloud project and model, not an API key. Any other OpenAI-compatible API: ```text LLM_API_BASE=https://router.huggingface.co/v1 LLM_API_KEY=hf_... # or HF_TOKEN MODEL_ID=Qwen/Qwen3-235B-A22B-Instruct-2507 # comma-separate for fallbacks ``` Run `python check_key.py` to see which providers work right now. It sends one tiny request per provider; `python check_key.py --all` tests every configured model, and `--provider gemini` limits it to Vertex Gemini. If every endpoint is out of quota for longer than `MAX_WAIT_SECONDS` (default 180) the question fails fast (`AGENT_ERROR: QuotaExhausted`) instead of hanging; re-run later and solved questions are skipped. ## Local setup ```bash python -m venv .venv # Windows: # .venv\Scripts\activate # macOS/Linux: # source .venv/bin/activate pip install -r requirements.txt gcloud auth application-default login # Fill GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION in .env. # Add provider API keys there too if you use other providers. python app.py ``` Local runs can use `AGENT_CODE_URL` for the public Space code URL, but the final Unit 4 submission should be run from a public Hugging Face Space so `SPACE_ID` is available automatically. ## Hugging Face Space deployment Create a **public Gradio Space**, then upload these files. The public code URL used by the scorer should look like: ```text https://huggingface.co/spaces///tree/main ``` Enable the Vertex AI API and billing for your project. In Space Settings → Secrets, set `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_LOCATION`. For authentication, set `GOOGLE_CLOUD_CREDENTIALS_JSON` to the full service-account JSON for a service account with Vertex AI permissions. Do not commit this JSON to the Space. The service account's usage is billed to the Google Cloud project, where any trial credits apply. `packages.txt` makes the Space install Stockfish automatically. Then: 1. Open the Space. 2. Sign in with the Hugging Face login button. 3. Click **Run 20-question evaluation**. 4. Review the generated answers in the table. 5. Click **Submit cached answers**. The app blocks submission if any question failed or if the cached run does not contain all 20 answers. Run the evaluation again after resolving quota or tool errors; successfully solved answers are reused from `progress.json`. The official Unit 4 scorer exposes: ```text GET https://agents-course-unit4-scoring.hf.space/questions GET https://agents-course-unit4-scoring.hf.space/files/{task_id} POST https://agents-course-unit4-scoring.hf.space/submit ``` The scorer compares submitted answers with exact-match logic, so the final answer should contain only the answer text. ## Important This project does **not** hard-code the benchmark's ground-truth answers. It actually retrieves the current questions and solves them with tools. That makes the Space a genuine agent implementation rather than an answer key. ## Running out of quota mid-run Every solved question is saved to `progress.json` immediately. If the quota dies on question 14, just run again later: the 13 solved answers are reused and only failed ones are retried (the table's `Source` column shows `saved` / `solved` / `FAILED`). Use **Clear saved progress** to start over; this also clears any old cached submission. ## Useful debugging The app also has a **Test one task** box. Paste a task ID from the evaluation table and run it before spending inference calls on the complete evaluation. ## Official course references - https://huggingface.co/learn/agents-course/unit1/introduction - https://huggingface.co/learn/agents-course/unit2/smolagents/introduction - https://huggingface.co/learn/agents-course/unit3/agentic-rag/agentic-rag - https://huggingface.co/learn/agents-course/unit4/introduction - https://huggingface.co/learn/agents-course/unit4/hands-on