Spaces:
Sleeping
Sleeping
| title: GAIA Agent - GPT-6 Sol Precision | |
| emoji: 🧠 | |
| colorFrom: blue | |
| colorTo: purple | |
| sdk: gradio | |
| sdk_version: 5.49.1 | |
| python_version: 3.10 | |
| app_file: app.py | |
| pinned: false | |
| hf_oauth: true | |
| hf_oauth_expiration_minutes: 480 | |
| # GAIA course — controlled-cost OpenAI agent | |
| This is an **original student agent**, not an official solution. It uses **GPT-6 Sol**, a limited sequence of model calls, public web search, local document extraction, a calculator, and a separate Hugging Face scoring submission button. **There is no guarantee of a particular score.** | |
| ## Installation in your EXISTING ZeroGPU Space | |
| 1. On Hugging Face, open your already duplicated `Final_Assignment_Template` Space (keep it PUBLIC and on ZeroGPU). | |
| 2. Open **Files**, and replace/add **`app.py`**, **`agent.py`**, **`requirements.txt`**, and **`README.md`** at the repository root. Upload the contents of this ZIP, **not the ZIP itself**. You can remove old Gemini agent files if any, but keep Hugging Face metadata. | |
| 3. Go to **Settings → Variables and secrets → Secrets → New secret**. Name `OPENAI_API_KEY` and paste your **OpenAI Platform API key**. Do **NOT** put it in a plain variable or in public code. | |
| 4. Allow the Space to rebuild. In **App**, log in with Hugging Face. | |
| 5. Click **1. Check setup (FREE)**. This checks only the GAIA scoring API and presence of your key; it does not make an OpenAI model call. | |
| 6. Click **2. Test ONE question**. Check the cached answer and the cost estimate, then check *actual* API spending on https://platform.openai.com/usage . | |
| 7. **BEFORE UPLOADING THIS UPDATE, download your existing answer backup** from the currently running Space. If it restarts, its /tmp answer cache can vanish. After deploying, restore the saved `.json` in the Backup section and click Show saved answers to verify your earlier answers survived. | |
| 8. To answer more, tick the paid-request consent checkbox. In **Maximum supported questions to solve in one batch**, choose **3, 5, 10 or 20**; choose **20** only after checking actual API usage/balance. Press **3. Run selected batch (OpenAI API)**. Previously cached and previously skipped questions are not repeated. A missing GAIA attachment (`/files/{task_id}` HTTP 404/410) is marked skipped **without OpenAI charges**, and the agent continues with the next available question. Audio/video attachments are skipped because transcription is disabled. | |
| 9. After the batch, click **Download answer backup** again. The batch stops on OpenAI API errors and the best-effort local budget guard. It does NOT submit to the leaderboard automatically. | |
| 10. Once you have as many answers as possible, press **Submit answers to GAIA scoring**. You can explicitly opt to submit blank answers for remaining unsupported questions. Scoring occurs only upon this click, and blank answers count as wrong. The Unit 4 target is 30% (6 correct out of 20). | |
| **Only one model is enabled:** `gpt-6-sol`, list prices checked October 28, 2026: **$2.00/M input tokens and $10.00/M output tokens**. The agent uses the **Responses API** with `reasoning.effort=medium` for function calling; built-in OpenAI tools are disabled (public web search is performed locally). The OpenAI API key is NOT the same as a ChatGPT subscription. | |
| ## Budget controls and safety | |
| - Default `GAIA_BUDGET_USD=2.00` local estimated TOTAL, `GAIA_QUESTION_BUDGET_USD=0.55` local estimated per question. You can change these in **Space Settings → Variables** later if needed; starting low is intentional. Never put secrets in Variables. | |
| - Maximum **4 paid model requests/question**, 5,000 output tokens/request (including reasoning), **8 locally run tools/question**. No implicit retries; OpenAI SDK retries disabled. | |
| - Before each request, the app temporarily reserves **$0.12** against the estimated budget; after a successful call, it reconciles using the model's reported input/output token usage. Failed calls keep their reservation to avoid undercounting. This is a conservative approximation, not an absolute per-call billing limit. | |
| - This guard **cannot enforce a hard limit on your OpenAI account**. OpenAI bills based on actual requests; actual costs could differ because of multimodal pricing, missing usage metadata, failures, deployment restarts, other apps/keys or changes in price. Set a project budget/alerts if available on the OpenAI dashboard and monitor its actual usage. Disable auto-recharge if you don't want it. | |
| - The application **can** run up to 20 eligible questions in one explicitly authorized batch, but defaults to 3 and does not automatically buy or renew credits. | |
| - Answer and local estimated spend files use **`/data` only when persistence is available**, otherwise temporary `/tmp` storage. On a Space restart, the latter may reset! Use **Download answer backup** and restore it if required. The budget estimate also resets in /tmp, so always check provider usage separately. | |
| - On purpose, **audio/video file transcription is disabled** to avoid extra, untracked transcription charges. Text, PDF, DOCX, spreadsheet and image tasks are partially supported. This means some GAIA tasks are unsupported until you choose to enable additional services. | |
| - For web search, the app uses `ddgs` via the Space itself, **not OpenAI paid web-search tools**. Search quality, page availability, YouTube captions, rate limits and exact-match accuracy are not guaranteed. | |
| ## Important troubleshooting | |
| - `429` / insufficient quota: look at your OpenAI project limits. No automatic retries. | |
| - `401` / wrong API key: verify that `OPENAI_API_KEY` is in **Secrets**. | |
| - `503`: temporary provider overload. Stop and decide whether a manual retry is worthwhile. | |
| - `404` from the official scoring `GET /files/{task_id}`: known server-side bug (https://github.com/huggingface/agents-course/issues/624). The app skips that task before sending an OpenAI request, and continues. Do not invent an answer to the missing attachment. | |
| - `Maximum model calls reached`: question is too complex for the conservative configuration. It is deliberately **not cached**. Decide whether to increase limits later. | |
| - Existing cached answers are **not re-run** accidentally; to retry one, select its question number, tick **I confirm clearing**, and click **Clear selected Q# answer**. This deletes the cached response without making any OpenAI API call. The next test will be billed normally. | |
| - Certificate and submission details: https://huggingface.co/learn/agents-course/unit4/hands-on | |
| The current version was syntax-checked and offline-tested for 404 skip, cache preservation, continuation past unavailable tasks, no model calls on missing attachments, and the scoring payload. It has **not** been executed against your OpenAI key or deployed live on your Space. | |
| ## Data-sharing token incentive | |
| If the organization is eligible **and enrolled**, GPT-6 Sol is in the **250K daily shared complimentary-token group for Build tier**, NOT the 2.5M group. The free quota is not guaranteed. Tokens exceeding the quota and separately priced API tools are billable. In particular, this app uses only local web functions; it does not enable the OpenAI built-in web-search tool. See https://help.openai.com/en/articles/10306912-sharing-feedback-evaluation-and-fine-tuning-data-and-api-inputs-and-outputs-with-openai . Monitor real billing in OpenAI Platform. | |
| Keep the previous answer backup before redeploying. The in-memory and /tmp caches may be reset; use the built-in **Restore answers** button after you deploy. Previous wrong answers will NOT automatically be regenerated. Choose **Clear selected Q#** to retry selected questions after creating a backup. | |
| GPT-6 Sol uses Responses API with medium reasoning and strict JSON final answers. If the model outputs no answer after 5,000 tokens, this app deliberately does not save a blank answer; the call can still consume tokens. | |
| ## Precision upgrade vs the Luna version | |
| - New evidence-first system prompt: check time periods, precise entity matches, calculation conditions and tie-break rules; cross-check ambiguous claims with primary sources where available. | |
| - Responses API `text.format` enforces one `final_answer` JSON string. Only that string is cached or submitted, never the JSON wrapper, web queries, debug traces, or narration. | |
| - Conservative output validation rejects explanations, tool traces and unsupported guesses without automatically retrying expensive calls. It strips *outer* quotation marks and unnecessary units on numerical count answers. | |
| - Sol is **20x the standard token price of Luna**: keep the default $2 local budget and run one question first. The $0.12 per-call reservation is an estimate, **not a billing cap**; provider charges can exceed it. | |
| - The original GAIA attachment-404 problem is **not fixed** here: missing attachments still get skipped. Some tasks cannot be solved accurately without them. | |
| - If a previous answer backup is restored, old answers remain cached. Use Clear selected Q# intentionally and re-run only selected questions. No hidden answer keys or prefilled benchmark solutions. | |
| ### Model configuration | |
| `agent.py` defines `MODEL = "gpt-6-sol"`, `REASONING_EFFORT = "medium"`, and the updated token prices. The exact updated prompt is in `SYSTEM_PROMPT`. | |
| ## v3 correction: empty answers no longer stop the batch | |
| The previous prompt made the model return an empty `final_answer` whenever | |
| it could not fully verify a source. This version instead requests a | |
| best-supported answer when the facts permit one; it does **not** fabricate | |
| sources or answers. For an inherently unanswerable question, a blank answer | |
| is marked **Unresolved/Skipped**, not cached, and the batch proceeds. A | |
| failed tool is reported in the skip reason, without automatic paid retries. | |
| Search first uses DDGS and falls back to Bing RSS if needed. These external | |
| web requests are not OpenAI billed tool calls (but network reliability | |
| remains outside our control). Existing answers are not overwritten. | |
| **Test Q1 on its own first, check the model response, and then run a small | |
| batch.** The $2 estimated guard is not a hard provider billing limit. | |