Spaces:

Effina-byte
/

Agent-test

Paused

App Files Files Community

Agent-test / README.md

Effina-byte

Update README.md

62e7346 verified 26 days ago

preview code

Raw History Blame Contribute Delete

9.34 kB

A newer version of the Gradio SDK is available: 6.31.0

Upgrade

metadata

title: GAIA Agent - Agents Course Unit 4
emoji: 🤗
colorFrom: indigo
colorTo: blue
sdk: gradio
sdk_version: 6.17.3
app_file: app.py
pinned: false
hf_oauth: true
hf_oauth_expiration_minutes: 480

How the submitted answers were produced

The 20 answers were generated by running this exact code locally with Gemini on Vertex AI, using Application Default Credentials. This Space only hosts the public code, so the agent can't run here without Google Cloud credentials. The Unit 4 scorer returned 20/20.

GAIA Agent — Hugging Face Agents Course Unit 4

A ground-up smolagents CodeAgent for the Unit 4 GAIA evaluation.

What this project does

The agent combines ideas from the earlier Agents Course units:

Unit 1: ReAct-style Think → Act → Observe, tools, and final-answer discipline.
Unit 2.1: smolagents CodeAgent, custom tools, web search, webpage reading, and Python execution.
Unit 3: an agentic retrieval pattern where the agent decides which retrieval/tool flow to use.
Unit 4: the official 20-question GAIA subset, exact-match scoring, and Hugging Face submission.

The implementation is intentionally modular:

app.py            # Gradio UI + evaluation/submission flow (resumes from progress.json)
agent.py          # Agent construction, question-type playbook, answer normalization
gaia_tools.py     # All tools (see below)
gemini_limits.py  # Per-endpoint rate limiting, cooldowns, failover rules
vertex_auth.py    # Vertex AI ADC and service-account authentication
providers.py      # Reads your API keys, builds the ordered provider/model list
progress.py       # Saves solved answers so a run can resume after a quota failure
check_key.py      # Shows which models your key can use right now
requirements.txt  # Python packages
packages.txt      # System packages for a Space (Stockfish for the chess question)

Tools

Tool	Used for
`wikipedia_page`	Full Wikipedia article text incl. tables, optionally as of a year (`year=2022`) or one `section`
`wikipedia_search`	Find exact titles (articles and `Wikipedia:` project pages)
`web_search`	DuckDuckGo then Bing
`google_search`	Google-grounded answer with sources via Vertex Gemini (hard multi-hop lookups)
`visit_webpage`	Open any URL as markdown
`inspect_attachment`	Images (vision), audio (transcript), xlsx, csv, pdf, docx, text
`run_attached_python`	Runs an attached `.py` (60 s limit, API keys hidden; disable with `ALLOW_RUN_ATTACHED_PYTHON=0`)
`chess_best_move`	FEN to best move with Stockfish (mate search fallback)
`youtube_transcript` / `analyze_youtube_video`	Spoken words / what is shown or heard
`python_interpreter`	Calculations, pandas/numpy on spreadsheets

If the course API returns 404 for an attachment, inspect_attachment falls back to the matching file in GAIA's public validation dataset. To enable that fallback, accept the dataset's access terms at https://huggingface.co/datasets/gaia-benchmark/GAIA and add a read token as the GAIA_HF_TOKEN Space secret (or HF_TOKEN). The official course endpoint is always tried first. The fallback reads only task IDs and attachment paths from the dataset metadata; it does not load or use benchmark answers.

Unit 4 rules this project follows

20 Level-1 GAIA validation questions; scoring is exact match, so only the bare answer is submitted.
You need more than 30 % (at least 7 of 20) for the certificate.
Submission = username + public Space code URL (.../tree/main) + {"task_id", "submitted_answer"} list.
Keep the Space public to stay on the leaderboard.

API keys (set the ones you have)

Locally: copy .env.example to .env and fill it in (loaded automatically). On a Hugging Face Space: add them under Settings -> Secrets. Do not upload .env to a public Space: anyone could read your keys. (Real environment variables always override .env.)

The agent "brain" can use several providers and fails over between them in the middle of a question when one runs out of quota. Set any of these (Space secrets or environment variables):

NVIDIA_API_KEY=nvapi-... # NVIDIA NIM; default openai/gpt-oss-20b (currently free endpoint)
GROQ_API_KEY=...         # openai/gpt-oss-120b (active; pricing/access depends on account tier)
OPENROUTER_API_KEY=...   # openrouter/free (dynamic free-model router; 50 requests/day on free accounts)
GOOGLE_CLOUD_PROJECT=... # Vertex AI project; also required for Gemini image/audio/video/search tools
GOOGLE_CLOUD_LOCATION=global
GOOGLE_CLOUD_CREDENTIALS_JSON=... # Optional service-account JSON; otherwise use ADC
GAIA_HF_TOKEN=hf_...     # Optional: required only for GAIA attachment fallback after a scorer 404

Order of use: custom, nvidia, groq, openrouter, gemini (change with PROVIDER_ORDER). Vertex Gemini is last so its quota is kept for the multimodal tools. Groq is skipped automatically for big prompts (its per-minute token budget is tiny).

Provider limits and free-model availability change. The defaults use currently documented model IDs, while the local RPM values are conservative scheduler settings rather than provider guarantees. Override models with NVIDIA_MODELS (default openai/gpt-oss-20b), GROQ_MODELS, OPENROUTER_MODELS (default openrouter/free, comma-separated) and budgets with NVIDIA_RPM, GROQ_RPM, VERTEX_GEMINI_RPM, ... Vertex limits are enforced against the Google Cloud project and model, not an API key.

Any other OpenAI-compatible API:

LLM_API_BASE=https://router.huggingface.co/v1
LLM_API_KEY=hf_...        # or HF_TOKEN
MODEL_ID=Qwen/Qwen3-235B-A22B-Instruct-2507   # comma-separate for fallbacks

Run python check_key.py to see which providers work right now. It sends one tiny request per provider; python check_key.py --all tests every configured model, and --provider gemini limits it to Vertex Gemini. If every endpoint is out of quota for longer than MAX_WAIT_SECONDS (default 180) the question fails fast (AGENT_ERROR: QuotaExhausted) instead of hanging; re-run later and solved questions are skipped.

Local setup

python -m venv .venv
# Windows:
# .venv\Scripts\activate
# macOS/Linux:
# source .venv/bin/activate

pip install -r requirements.txt
gcloud auth application-default login
# Fill GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION in .env.
# Add provider API keys there too if you use other providers.
python app.py

Local runs can use AGENT_CODE_URL for the public Space code URL, but the final Unit 4 submission should be run from a public Hugging Face Space so SPACE_ID is available automatically.

Hugging Face Space deployment

Create a public Gradio Space, then upload these files.

The public code URL used by the scorer should look like:

https://huggingface.co/spaces/<username>/<space-name>/tree/main

Enable the Vertex AI API and billing for your project. In Space Settings → Secrets, set GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION. For authentication, set GOOGLE_CLOUD_CREDENTIALS_JSON to the full service-account JSON for a service account with Vertex AI permissions. Do not commit this JSON to the Space. The service account's usage is billed to the Google Cloud project, where any trial credits apply. packages.txt makes the Space install Stockfish automatically.

Then:

Open the Space.
Sign in with the Hugging Face login button.
Click Run 20-question evaluation.
Review the generated answers in the table.
Click Submit cached answers.

The app blocks submission if any question failed or if the cached run does not contain all 20 answers. Run the evaluation again after resolving quota or tool errors; successfully solved answers are reused from progress.json.

The official Unit 4 scorer exposes:

GET  https://agents-course-unit4-scoring.hf.space/questions
GET  https://agents-course-unit4-scoring.hf.space/files/{task_id}
POST https://agents-course-unit4-scoring.hf.space/submit

The scorer compares submitted answers with exact-match logic, so the final answer should contain only the answer text.

Important

This project does not hard-code the benchmark's ground-truth answers. It actually retrieves the current questions and solves them with tools. That makes the Space a genuine agent implementation rather than an answer key.

Running out of quota mid-run

Every solved question is saved to progress.json immediately. If the quota dies on question 14, just run again later: the 13 solved answers are reused and only failed ones are retried (the table's Source column shows saved / solved / FAILED). Use Clear saved progress to start over; this also clears any old cached submission.

Useful debugging

The app also has a Test one task box. Paste a task ID from the evaluation table and run it before spending inference calls on the complete evaluation.