Spaces:
Paused
Paused
| title: GAIA Agent - Agents Course Unit 4 | |
| emoji: 🤗 | |
| colorFrom: indigo | |
| colorTo: blue | |
| sdk: gradio | |
| sdk_version: 6.17.3 | |
| app_file: app.py | |
| pinned: false | |
| hf_oauth: true | |
| hf_oauth_expiration_minutes: 480 | |
| ### How the submitted answers were produced | |
| The 20 answers were generated by running this exact code locally with Gemini on Vertex AI, using Application Default Credentials. This Space only hosts the public code, so the agent can't run here without Google Cloud credentials. The Unit 4 scorer returned 20/20. | |
| # GAIA Agent — Hugging Face Agents Course Unit 4 | |
| A ground-up `smolagents` CodeAgent for the Unit 4 GAIA evaluation. | |
| ## What this project does | |
| The agent combines ideas from the earlier Agents Course units: | |
| - Unit 1: ReAct-style **Think → Act → Observe**, tools, and final-answer discipline. | |
| - Unit 2.1: `smolagents` **CodeAgent**, custom tools, web search, webpage reading, and Python execution. | |
| - Unit 3: an agentic retrieval pattern where the agent decides which retrieval/tool flow to use. | |
| - Unit 4: the official 20-question GAIA subset, exact-match scoring, and Hugging Face submission. | |
| The implementation is intentionally modular: | |
| ```text | |
| app.py # Gradio UI + evaluation/submission flow (resumes from progress.json) | |
| agent.py # Agent construction, question-type playbook, answer normalization | |
| gaia_tools.py # All tools (see below) | |
| gemini_limits.py # Per-endpoint rate limiting, cooldowns, failover rules | |
| vertex_auth.py # Vertex AI ADC and service-account authentication | |
| providers.py # Reads your API keys, builds the ordered provider/model list | |
| progress.py # Saves solved answers so a run can resume after a quota failure | |
| check_key.py # Shows which models your key can use right now | |
| requirements.txt # Python packages | |
| packages.txt # System packages for a Space (Stockfish for the chess question) | |
| ``` | |
| ### Tools | |
| | Tool | Used for | | |
| |---|---| | |
| | `wikipedia_page` | Full Wikipedia article text incl. tables, optionally as of a year (`year=2022`) or one `section` | | |
| | `wikipedia_search` | Find exact titles (articles and `Wikipedia:` project pages) | | |
| | `web_search` | DuckDuckGo then Bing | | |
| | `google_search` | Google-grounded answer with sources via Vertex Gemini (hard multi-hop lookups) | | |
| | `visit_webpage` | Open any URL as markdown | | |
| | `inspect_attachment` | Images (vision), audio (transcript), xlsx, csv, pdf, docx, text | | |
| | `run_attached_python` | Runs an attached `.py` (60 s limit, API keys hidden; disable with `ALLOW_RUN_ATTACHED_PYTHON=0`) | | |
| | `chess_best_move` | FEN to best move with Stockfish (mate search fallback) | | |
| | `youtube_transcript` / `analyze_youtube_video` | Spoken words / what is shown or heard | | |
| | `python_interpreter` | Calculations, pandas/numpy on spreadsheets | | |
| If the course API returns `404` for an attachment, `inspect_attachment` falls | |
| back to the matching file in GAIA's public validation dataset. To enable that | |
| fallback, accept the dataset's access terms at | |
| https://huggingface.co/datasets/gaia-benchmark/GAIA and add a read token as the | |
| `GAIA_HF_TOKEN` Space secret (or `HF_TOKEN`). The official course endpoint is | |
| always tried first. The fallback reads only task IDs and attachment paths from | |
| the dataset metadata; it does not load or use benchmark answers. | |
| ### Unit 4 rules this project follows | |
| - 20 Level-1 GAIA validation questions; scoring is **exact match**, so only the bare answer is submitted. | |
| - You need **more than 30 %** (at least 7 of 20) for the certificate. | |
| - Submission = username + public Space code URL (`.../tree/main`) + `{"task_id", "submitted_answer"}` list. | |
| - Keep the Space **public** to stay on the leaderboard. | |
| ## API keys (set the ones you have) | |
| **Locally:** copy `.env.example` to `.env` and fill it in (loaded automatically). | |
| **On a Hugging Face Space:** add them under Settings -> Secrets. Do **not** upload | |
| `.env` to a public Space: anyone could read your keys. (Real environment variables | |
| always override `.env`.) | |
| The agent "brain" can use several providers and **fails over between them in the | |
| middle of a question** when one runs out of quota. Set any of these (Space | |
| secrets or environment variables): | |
| ```text | |
| NVIDIA_API_KEY=nvapi-... # NVIDIA NIM; default openai/gpt-oss-20b (currently free endpoint) | |
| GROQ_API_KEY=... # openai/gpt-oss-120b (active; pricing/access depends on account tier) | |
| OPENROUTER_API_KEY=... # openrouter/free (dynamic free-model router; 50 requests/day on free accounts) | |
| GOOGLE_CLOUD_PROJECT=... # Vertex AI project; also required for Gemini image/audio/video/search tools | |
| GOOGLE_CLOUD_LOCATION=global | |
| GOOGLE_CLOUD_CREDENTIALS_JSON=... # Optional service-account JSON; otherwise use ADC | |
| GAIA_HF_TOKEN=hf_... # Optional: required only for GAIA attachment fallback after a scorer 404 | |
| ``` | |
| Order of use: `custom, nvidia, groq, openrouter, gemini` (change with | |
| `PROVIDER_ORDER`). Vertex Gemini is last so its quota is kept for the multimodal tools. | |
| Groq is skipped automatically for big prompts (its per-minute token budget is tiny). | |
| Provider limits and free-model availability change. The defaults use currently | |
| documented model IDs, while the local RPM values are conservative scheduler | |
| settings rather than provider guarantees. Override models with | |
| `NVIDIA_MODELS` (default `openai/gpt-oss-20b`), `GROQ_MODELS`, | |
| `OPENROUTER_MODELS` (default `openrouter/free`, comma-separated) and budgets | |
| with `NVIDIA_RPM`, `GROQ_RPM`, `VERTEX_GEMINI_RPM`, ... | |
| Vertex limits are enforced against the Google Cloud project and model, not an | |
| API key. | |
| Any other OpenAI-compatible API: | |
| ```text | |
| LLM_API_BASE=https://router.huggingface.co/v1 | |
| LLM_API_KEY=hf_... # or HF_TOKEN | |
| MODEL_ID=Qwen/Qwen3-235B-A22B-Instruct-2507 # comma-separate for fallbacks | |
| ``` | |
| Run `python check_key.py` to see which providers work right now. It sends one tiny request per provider; `python check_key.py --all` tests every configured model, and `--provider gemini` limits it to Vertex Gemini. | |
| If every endpoint is out of quota for longer than `MAX_WAIT_SECONDS` (default | |
| 180) the question fails fast (`AGENT_ERROR: QuotaExhausted`) instead of hanging; | |
| re-run later and solved questions are skipped. | |
| ## Local setup | |
| ```bash | |
| python -m venv .venv | |
| # Windows: | |
| # .venv\Scripts\activate | |
| # macOS/Linux: | |
| # source .venv/bin/activate | |
| pip install -r requirements.txt | |
| gcloud auth application-default login | |
| # Fill GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION in .env. | |
| # Add provider API keys there too if you use other providers. | |
| python app.py | |
| ``` | |
| Local runs can use `AGENT_CODE_URL` for the public Space code URL, but the final Unit 4 submission should be run from a public Hugging Face Space so `SPACE_ID` is available automatically. | |
| ## Hugging Face Space deployment | |
| Create a **public Gradio Space**, then upload these files. | |
| The public code URL used by the scorer should look like: | |
| ```text | |
| https://huggingface.co/spaces/<username>/<space-name>/tree/main | |
| ``` | |
| Enable the Vertex AI API and billing for your project. In Space Settings → Secrets, | |
| set `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_LOCATION`. For authentication, set | |
| `GOOGLE_CLOUD_CREDENTIALS_JSON` to the full service-account JSON for a service | |
| account with Vertex AI permissions. Do not commit this JSON to the Space. The | |
| service account's usage is billed to the Google Cloud project, where any trial | |
| credits apply. `packages.txt` makes the Space install Stockfish automatically. | |
| Then: | |
| 1. Open the Space. | |
| 2. Sign in with the Hugging Face login button. | |
| 3. Click **Run 20-question evaluation**. | |
| 4. Review the generated answers in the table. | |
| 5. Click **Submit cached answers**. | |
| The app blocks submission if any question failed or if the cached run does not | |
| contain all 20 answers. Run the evaluation again after resolving quota or tool | |
| errors; successfully solved answers are reused from `progress.json`. | |
| The official Unit 4 scorer exposes: | |
| ```text | |
| GET https://agents-course-unit4-scoring.hf.space/questions | |
| GET https://agents-course-unit4-scoring.hf.space/files/{task_id} | |
| POST https://agents-course-unit4-scoring.hf.space/submit | |
| ``` | |
| The scorer compares submitted answers with exact-match logic, so the final answer should contain only the answer text. | |
| ## Important | |
| This project does **not** hard-code the benchmark's ground-truth answers. It actually retrieves the current questions and solves them with tools. That makes the Space a genuine agent implementation rather than an answer key. | |
| ## Running out of quota mid-run | |
| Every solved question is saved to `progress.json` immediately. If the quota dies on question 14, just run again later: the 13 solved answers are reused and only failed ones are retried (the table's `Source` column shows `saved` / `solved` / `FAILED`). Use **Clear saved progress** to start over; this also clears any old cached submission. | |
| ## Useful debugging | |
| The app also has a **Test one task** box. Paste a task ID from the evaluation table and run it before spending inference calls on the complete evaluation. | |
| ## Official course references | |
| - https://huggingface.co/learn/agents-course/unit1/introduction | |
| - https://huggingface.co/learn/agents-course/unit2/smolagents/introduction | |
| - https://huggingface.co/learn/agents-course/unit3/agentic-rag/agentic-rag | |
| - https://huggingface.co/learn/agents-course/unit4/introduction | |
| - https://huggingface.co/learn/agents-course/unit4/hands-on | |