404 problem

README.md

@@ -1,5 +1,5 @@
 ---
-title: GAIA Agent - OpenAI Safe Edition
+title: GAIA Agent - OpenAI 404-safe Batch Edition
 emoji: 🧠
 colorFrom: blue
 colorTo: purple
@@ -24,8 +24,10 @@ This is an **original student agent**, not an official solution. It uses **GPT-5
 4. Allow the Space to rebuild. In **App**, log in with Hugging Face.
 5. Click **1. Check setup (FREE)**. This checks only the GAIA scoring API and presence of your key; it does not make an OpenAI model call.
 6. Click **2. Test ONE question**. Check the cached answer and the cost estimate, then check *actual* API spending on https://platform.openai.com/usage .
-7. Only after the first test looks good, tick the confirmation checkbox and use **3. Run next 3 unanswered questions**. Audio/video attachment questions are skipped by this batch because they are not yet supported. Back up answers with the download button after each few questions.
-8. Once you have as many answers as possible, press **Submit answers to GAIA scoring**. You can explicitly opt to submit blank answers for remaining unsupported questions. Scoring occurs only upon this click, and blank answers count as wrong. The Unit 4 target is 30% (6 correct out of 20).
+7. **BEFORE UPLOADING THIS UPDATE, download your existing answer backup** from the currently running Space. If it restarts, its /tmp answer cache can vanish. After deploying, restore the saved `.json` in the Backup section and click Show saved answers to verify your earlier answers survived.
+8. To answer more, tick the paid-request consent checkbox. In **Maximum supported questions to solve in one batch**, choose **3, 5, 10 or 20**; choose **20** only after checking actual API usage/balance. Press **3. Run selected batch (OpenAI API)**. Previously cached and previously skipped questions are not repeated. A missing GAIA attachment (`/files/{task_id}` HTTP 404/410) is marked skipped **without OpenAI charges**, and the agent continues with the next available question. Audio/video attachments are skipped because transcription is disabled.
+9. After the batch, click **Download answer backup** again. The batch stops on OpenAI API errors and the best-effort local budget guard. It does NOT submit to the leaderboard automatically.
+10. Once you have as many answers as possible, press **Submit answers to GAIA scoring**. You can explicitly opt to submit blank answers for remaining unsupported questions. Scoring occurs only upon this click, and blank answers count as wrong. The Unit 4 target is 30% (6 correct out of 20).
 
 **Only one model is enabled:** `gpt-5.4-mini`, list prices checked October 28, 2026: **$0.75/M input tokens and $4.50/M output tokens**. The OpenAI API key is NOT the same as a ChatGPT subscription.
 
@@ -35,7 +37,7 @@ This is an **original student agent**, not an official solution. It uses **GPT-5
 - Maximum **5 paid model requests/question**, 2,200 completion tokens/request, **8 locally run tools/question**. No implicit retries; OpenAI SDK retries disabled.
 - Before each request, the app temporarily reserves **$0.07** against the estimated budget; after a successful call, it reconciles using the model's reported input/output token usage. Failed calls keep their reservation to avoid undercounting. This is a conservative approximation, not an absolute per-call billing limit.
 - This guard **cannot enforce a hard limit on your OpenAI account**. OpenAI bills based on actual requests; actual costs could differ because of multimodal pricing, missing usage metadata, failures, deployment restarts, other apps/keys or changes in price. Set a project budget/alerts if available on the OpenAI dashboard and monitor its actual usage. Disable auto-recharge if you don't want it.
-- The application does **not** automatically run all 20 questions and does not automatically buy or renew credits.
+- The application **can** run up to 20 eligible questions in one explicitly authorized batch, but defaults to 3 and does not automatically buy or renew credits.
 - Answer and local estimated spend files use **`/data` only when persistence is available**, otherwise temporary `/tmp` storage. On a Space restart, the latter may reset! Use **Download answer backup** and restore it if required. The budget estimate also resets in /tmp, so always check provider usage separately.
 - On purpose, **audio/video file transcription is disabled** to avoid extra, untracked transcription charges. Text, PDF, DOCX, spreadsheet and image tasks are partially supported. This means some GAIA tasks are unsupported until you choose to enable additional services.
 - For web search, the app uses `ddgs` via the Space itself, **not OpenAI paid web-search tools**. Search quality, page availability, YouTube captions, rate limits and exact-match accuracy are not guaranteed.
@@ -45,8 +47,9 @@ This is an **original student agent**, not an official solution. It uses **GPT-5
 - `429` / insufficient quota: look at your OpenAI project limits. No automatic retries.
 - `401` / wrong API key: verify that `OPENAI_API_KEY` is in **Secrets**.
 - `503`: temporary provider overload. Stop and decide whether a manual retry is worthwhile.
+- `404` from the official scoring `GET /files/{task_id}`: known server-side bug (https://github.com/huggingface/agents-course/issues/624). The app skips that task before sending an OpenAI request, and continues. Do not invent an answer to the missing attachment.
 - `Maximum model calls reached`: question is too complex for the conservative configuration. It is deliberately **not cached**. Decide whether to increase limits later.
 - Existing cached answers are **not re-run** accidentally; to retry one, select its question number, tick **I confirm clearing**, and click **Clear selected Q# answer**. This deletes the cached response without making any OpenAI API call. The next test will be billed normally.
 - Certificate and submission details: https://huggingface.co/learn/agents-course/unit4/hands-on
 
-This was syntax-checked and unit-tested using mocked (not billable) model responses; it has **not** been executed against your OpenAI API key or the live Hugging Face Space.
+The current version was syntax-checked and offline-tested for 404 skip, cache preservation, continuation past unavailable tasks, no model calls on missing attachments, and the scoring payload. It has **not** been executed against your OpenAI key or deployed live on your Space.

agent.py

@@ -61,6 +61,16 @@ class BudgetError(RuntimeError):
     pass
 
 
+class AttachmentUnavailable(RuntimeError):
+    """An attachment could not be retrieved from the course scoring server."""
+    pass
+
+
+class UnsupportedMedia(RuntimeError):
+    """Audio/video needs a separate non-free transcription implementation."""
+    pass
+
+
 class BudgetLedger:
     """Best-effort estimated budget, not a provider-side hard billing limit.
 
@@ -310,14 +320,19 @@ class GaiaAgent:
         ledger = BudgetLedger()
         context = ""
         if file_name:
+            # Detect unsupported media BEFORE downloading or invoking the model.
+            mime = mimetypes.guess_type(file_name)[0] or ""
+            if mime.startswith(("audio/", "video/")) or file_name.lower().endswith((".m4a", ".mp3", ".wav", ".mp4", ".webm")):
+                raise UnsupportedMedia("Audio/video attachment requires separate transcription; skipped without any OpenAI call")
             r = requests.get(f"{SCORING_API}/files/{task_id}", timeout=25)
+            if r.status_code in (404, 410):
+                # Known HF unit 4 bug: /files/{task_id} returns 404 despite file_name.
+                # No OpenAI model call has been made for this question.
+                raise AttachmentUnavailable(f"GAIA attachment unavailable (HTTP {r.status_code}) for task {task_id}; skipped without any OpenAI call")
             r.raise_for_status()
             content = r.content
             if len(content) > MAX_ATTACHMENT_BYTES:
-                raise RuntimeError("Attachment too large: no model calls were made")
-            mime = mimetypes.guess_type(file_name)[0] or ""
-            if mime.startswith("audio/") or file_name.lower().endswith((".m4a", ".mp3", ".wav", ".mp4", ".webm")):
-                raise RuntimeError("Audio/video attachment: transcription is disabled to avoid untracked API charges. Skip for now.")
+                raise AttachmentUnavailable("Attachment too large to process safely; skipped without any OpenAI call")
             if mime.startswith("image/"):
                 context = "IMAGE DESCRIPTION: " + self._describe_image(ledger, question, file_name, content)
             else:

app.py

@@ -12,7 +12,7 @@ import pandas as pd
 import requests
 import spaces
 
-from agent import (BudgetError, BudgetLedger, GaiaAgent, MODEL, SCORING_API,
+from agent import (AttachmentUnavailable, UnsupportedMedia, BudgetError, BudgetLedger, GaiaAgent, MODEL, SCORING_API,
                    STORAGE, MAX_ESTIMATED_TOTAL_USD, MAX_ESTIMATED_QUESTION_USD)
 
 ANSWERS_DIR = STORAGE / "answers"
@@ -23,6 +23,28 @@ FILES_LOCK = threading.RLock()
 COLS = ["#", "Question", "Cached answer", "Status"]
 
 
+def _skips_path(username: str) -> Path:
+    return ANSWERS_DIR / (username + "_skips.json")
+
+
+def _load_skips(username: str) -> dict:
+    with FILES_LOCK:
+        try:
+            data = json.loads(_skips_path(username).read_text())
+            return data if isinstance(data, dict) else {}
+        except (FileNotFoundError, ValueError, OSError):
+            return {}
+
+
+def _save_skips(username: str, data: dict) -> None:
+    with FILES_LOCK:
+        dest = _skips_path(username)
+        tmp = dest.with_suffix(".tmp")
+        tmp.write_text(json.dumps(data, ensure_ascii=False, indent=2))
+        tmp.replace(dest)
+
+
+
 def _user(profile) -> str:
     username = getattr(profile, "username", "") if profile else ""
     if not username:
@@ -60,10 +82,13 @@ def _questions() -> list:
     return data
 
 
-def _table(questions: list, answers: dict) -> pd.DataFrame:
+def _table(questions: list, answers: dict, skips: dict | None = None) -> pd.DataFrame:
+    skips = skips or {}
     return pd.DataFrame([{"#":i, "Question":q["question"],
                           "Cached answer":answers.get(q["task_id"], ""),
-                          "Status":"Cached" if q["task_id"] in answers else "Not run"}
+                          "Status": ("Cached" if q["task_id"] in answers
+                                     else "Skipped: " + skips[q["task_id"]] if q["task_id"] in skips
+                                     else "Not run")}
                          for i,q in enumerate(questions,1)], columns=COLS)
 
 
@@ -82,7 +107,7 @@ def check_setup(profile: gr.OAuthProfile | None):
         questions = _questions()
         return (f"Setup OK: {len(questions)} GAIA questions found, OpenAI key is configured, "
                 f"HF username: {user}. NO paid OpenAI call was made.\n{_usage()}",
-                _table(questions, _load(user)))
+                _table(questions, _load(user), _load_skips(user)))
     except Exception as e:
         return f"Setup check: {e} (no paid OpenAI call made)", pd.DataFrame(columns=COLS)
 
@@ -98,58 +123,92 @@ def solve_one(n, profile: gr.OAuthProfile | None):
             raise ValueError(f"Question number must be between 1 and {len(questions)}")
         answers = _load(user)
         q = questions[num-1]
-        if q["task_id"] in answers:
-            yield f"Question {num} is already cached. To retry, clear its answer first (see README).\n{_usage()}", _table(questions,answers)
+        if q["task_id"] in answers or q["task_id"] in _load_skips(user):
+            yield f"Question {num} is cached or marked skipped. Use Clear selected Q# to retry.\n{_usage()}", _table(questions,answers, _load_skips(user))
             return
         if not RUN_LOCK.acquire(blocking=False):
-            yield "Another evaluation is running. No OpenAI request was sent.", _table(questions,answers)
+            yield "Another evaluation is running. No OpenAI request was sent.", _table(questions,answers, _load_skips(user))
             return
         try:
-            yield f"Solving question {num} with {MODEL} (maximum 5 model calls). Not submitting yet...", _table(questions,answers)
-            answer, calls = GaiaAgent().run(q["question"], q["task_id"], q.get("file_name") or "")
+            yield f"Solving question {num} with {MODEL} (maximum 5 model calls). Not submitting yet...", _table(questions,answers, _load_skips(user))
+            try:
+                answer, calls = GaiaAgent().run(q["question"], q["task_id"], q.get("file_name") or "")
+            except (AttachmentUnavailable, UnsupportedMedia) as exc:
+                skips = _load_skips(user)
+                skips[q["task_id"]] = str(exc)[:200]
+                _save_skips(user, skips)
+                yield f"Skipped Q{num}: {exc}\n{_usage()}", _table(questions,answers,skips)
+                return
             answers[q["task_id"]] = answer
             _save(user,answers)
             yield (f"Question {num} saved using {calls} OpenAI calls. "
-                   f"{len(answers)}/{len(questions)} cached.\n{_usage()}"), _table(questions,answers)
+                   f"{len(answers)}/{len(questions)} cached.\n{_usage()}"), _table(questions,answers, _load_skips(user))
         finally:
             RUN_LOCK.release()
     except Exception as exc:
         yield f"Stopped: {type(exc).__name__}: {str(exc)[:450]}\n{_usage()}", pd.DataFrame(columns=COLS)
 
 
-def next_three(approved: bool, profile: gr.OAuthProfile | None):
+def run_batch(approved: bool, max_questions: int, profile: gr.OAuthProfile | None):
+    """Run up to N billable questions, skip missing attachments without stalling.
+
+    No automatic model retries; all successful answers are saved immediately.
+    """
     if not approved:
-        yield "Check the consent box before running a paid batch. No paid requests made.", pd.DataFrame(columns=COLS)
+        yield "Tick the OpenAI API consent box first. No paid requests made.", pd.DataFrame(columns=COLS)
         return
     try:
         user = _user(profile)
         if not os.environ.get("OPENAI_API_KEY"):
             raise RuntimeError("OPENAI_API_KEY secret missing")
+        limit = int(max_questions)
+        if limit not in (3, 5, 10, 20):
+            raise ValueError("Choose 3, 5, 10 or 20")
         questions = _questions()
-        answers = _load(user)
-        todo = [(i,q) for i,q in enumerate(questions,1)
-                if q["task_id"] not in answers and not (q.get("file_name") or "").lower().endswith(
-                    (".m4a", ".mp3", ".wav", ".mp4", ".webm"))][:3]
-        if not todo:
-            yield "No remaining text/image-supported questions. Check audio/video tasks separately before submitting.", _table(questions,answers)
-            return
+        answers, skips = _load(user), _load_skips(user)
         if not RUN_LOCK.acquire(blocking=False):
-            yield "Another evaluation is running; no paid requests made.", _table(questions,answers)
+            yield "Another evaluation is running; no paid requests made.", _table(questions,answers,skips)
             return
         try:
-            for i,q in todo:
-                yield f"Solving Q{i} of {len(questions)}. Up to 3 questions in this batch.\n{_usage()}",_table(questions,answers)
+            paid_questions = 0
+            newly_skipped = 0
+            for i,q in enumerate(questions,1):
+                task_id = q["task_id"]
+                if task_id in answers or task_id in skips:
+                    continue
+                if paid_questions >= limit:
+                    break
+                file_name = (q.get("file_name") or "").lower()
+                if file_name.endswith((".m4a", ".mp3", ".wav", ".mp4", ".webm")):
+                    skips[task_id] = "Audio/video attachment unsupported (no OpenAI charge)"
+                    _save_skips(user, skips)
+                    newly_skipped += 1
+                    yield f"Skipped unsupported audio/video Q{i}.\n{_usage()}", _table(questions,answers,skips)
+                    continue
+                yield (f"Starting Q{i}. {paid_questions}/{limit} supported questions processed in this batch. "
+                       f"No answers are submitted automatically.\n{_usage()}"), _table(questions,answers,skips)
                 try:
-                    answer,calls = GaiaAgent().run(q["question"], q["task_id"], q.get("file_name") or "")
-                    answers[q["task_id"]] = answer
-                    _save(user,answers)
+                    answer,calls = GaiaAgent().run(q["question"], task_id, q.get("file_name") or "")
+                except (AttachmentUnavailable, UnsupportedMedia) as exc:
+                    skips[task_id] = str(exc)[:200]
+                    _save_skips(user, skips)
+                    newly_skipped += 1
+                    yield f"Skipped Q{i}: {exc}\n{_usage()}", _table(questions,answers,skips)
+                    continue
                 except BudgetError as exc:
-                    yield f"Stopped to protect your budget: {exc}\n{_usage()}", _table(questions,answers)
+                    yield f"Budget guard stopped the batch before Q{i}: {exc}\n{_usage()}", _table(questions,answers,skips)
                     return
                 except Exception as exc:
-                    yield f"Stopped at Q{i}: {type(exc).__name__}: {str(exc)[:300]}. Fix before proceeding.\n{_usage()}", _table(questions,answers)
+                    yield (f"Stopped at Q{i}: {type(exc).__name__}: {str(exc)[:300]}. "
+                           f"Already saved answers remain intact.\n{_usage()}"), _table(questions,answers,skips)
                     return
-            yield f"Batch complete. {len(answers)}/{len(questions)} cached.\n{_usage()}", _table(questions,answers)
+                answers[task_id] = answer
+                _save(user,answers)
+                paid_questions += 1
+                yield (f"Saved Q{i} ({calls} model calls). {len(answers)}/{len(questions)} answers saved; "
+                       f"{len(skips)} questions skipped.\n{_usage()}"), _table(questions,answers,skips)
+            yield (f"Batch finished. Saved {len(answers)}/{len(questions)}; skipped {len(skips)} "
+                   f"({newly_skipped} new skips in this batch). Download a backup now.\n{_usage()}"), _table(questions,answers,skips)
         finally:
             RUN_LOCK.release()
     except Exception as exc:
@@ -161,7 +220,7 @@ def view_cached(profile: gr.OAuthProfile | None):
         user = _user(profile)
         questions = _questions()
         answers = _load(user)
-        return f"{len(answers)}/{len(questions)} saved.\n{_usage()}",_table(questions,answers)
+        return f"{len(answers)}/{len(questions)} saved.\n{_usage()}",_table(questions,answers, _load_skips(user))
     except Exception as exc:
         return f"Error: {exc}", pd.DataFrame(columns=COLS)
 
@@ -172,15 +231,20 @@ def clear_one(n, confirmed: bool, profile: gr.OAuthProfile | None):
         user = _user(profile)
         questions = _questions()
         if not confirmed:
-            return "Check confirmation before clearing an answer. No paid requests made.", _table(questions,_load(user))
+            return "Check confirmation before clearing an answer. No paid requests made.", _table(questions,_load(user), _load_skips(user))
         num = int(n)
         if not (1 <= num <= len(questions)):
             raise ValueError("Question number out of range")
         answers = _load(user)
-        removed = answers.pop(questions[num-1]["task_id"], None)
+        task_id = questions[num-1]["task_id"]
+        removed = answers.pop(task_id, None)
+        skips = _load_skips(user)
+        skipped = skips.pop(task_id, None)
         _save(user, answers)
-        return (f"Cleared Q{num}; you can test it again. No OpenAI request made." if removed is not None
-                else f"Q{num} was not cached. No changes."), _table(questions,answers)
+        _save_skips(user, skips)
+        return (f"Cleared Q{num} cached answer/skip. Now eligible to retry. No API charge."
+                if removed is not None or skipped is not None
+                else f"Q{num} was neither cached nor skipped. No changes."), _table(questions,answers,skips)
     except Exception as exc:
         return f"Clear failed: {exc}", pd.DataFrame(columns=COLS)
 
@@ -216,7 +280,7 @@ def restore(uploaded_path, profile: gr.OAuthProfile | None):
         for k,v in cleaned.items():
             answers.setdefault(k,v) # Never overwrite an existing answer silently
         _save(user,answers)
-        return f"Restored {len(cleaned)} candidate answers; {len(answers)}/{len(questions)} cached. No API charges.", _table(questions,answers)
+        return f"Restored {len(cleaned)} candidate answers; {len(answers)}/{len(questions)} cached. No API charges.", _table(questions,answers, _load_skips(user))
     except Exception as exc:
         return f"Restore failed: {exc}", pd.DataFrame(columns=COLS)
 
@@ -230,7 +294,7 @@ def submit(allow_blanks: bool, profile: gr.OAuthProfile | None):
         missing = [i for i,q in enumerate(questions,1) if q["task_id"] not in answers]
         if missing and not allow_blanks:
             return (f"Not submitted: missing Q{missing}. To submit now with empty answers "
-                    "for unsupported questions, tick the confirmation checkbox. No API spend."), _table(questions,answers)
+                    "for unsupported questions, tick the confirmation checkbox. No API spend."), _table(questions,answers, _load_skips(user))
         space_id = os.environ.get("SPACE_ID")
         if not space_id:
             raise RuntimeError("SPACE_ID not set. Must submit from your public Hugging Face Space")
@@ -243,7 +307,7 @@ def submit(allow_blanks: bool, profile: gr.OAuthProfile | None):
         return (f"Official submission: score {score.get('score', 'n/a')}%, "
                 f"{score.get('correct_count','?')}/{score.get('total_attempted','?')} correct "
                 f"({len(missing)} unanswered submitted as blank). "
-                f"{str(score.get('message',''))[:400]}"), _table(questions,answers)
+                f"{str(score.get('message',''))[:400]}"), _table(questions,answers, _load_skips(user))
     except Exception as exc:
         return f"Submission failed: {exc}", pd.DataFrame(columns=COLS)
 
@@ -272,7 +336,8 @@ with gr.Blocks(title="GAIA Agent - budget-safe OpenAI") as demo:
     clear_confirm = gr.Checkbox(label="I confirm clearing the selected cached answer (free)", value=False)
     clear_button = gr.Button("Clear selected Q# answer to allow a retry (free)")
     consent = gr.Checkbox(label="I understand this makes paid OpenAI API requests", value=False)
-    batch = gr.Button("3. Run next 3 unanswered questions (paid)")
+    batch_size = gr.Dropdown([3, 5, 10, 20], value=3, label="Maximum supported questions to solve in one batch", interactive=True)
+    batch = gr.Button("3. Run selected batch (OpenAI API)")
     with gr.Row():
         see = gr.Button("Show saved answers")
         send = gr.Button("Submit answers to GAIA scoring (free)")
@@ -287,13 +352,16 @@ with gr.Blocks(title="GAIA Agent - budget-safe OpenAI") as demo:
     with gr.Row():
         uploaded = gr.File(label="Upload an earlier answer backup (.json)", file_types=[".json"], type="filepath")
         restore_button = gr.Button("Restore answers (free)")
-    gr.Markdown("Your HF Space remains PUBLIC, but your API key must be stored only as a **Secret**. "
+    gr.Markdown("**Before updating your Space or running a large batch:** click Download answer backup. "
+                "Unavailable GAIA attachments (HTTP 404) are marked Skipped, not answered blindly. "
+                "The batch stops at any model/API error or local budget limit. "
+                "Your HF Space remains PUBLIC, but your API key must be stored only as a **Secret**. "
                 "Audio transcription is disabled by default to avoid untracked charges. "
                 "The answer cache and cost ledger persist only if /data persistent storage is mounted; otherwise export backups regularly.")
     setup.click(check_setup, outputs=[status,table])
     test.click(solve_one,inputs=[number],outputs=[status,table])
     clear_button.click(clear_one,inputs=[number,clear_confirm],outputs=[status,table])
-    batch.click(next_three,inputs=[consent],outputs=[status,table])
+    batch.click(run_batch,inputs=[consent,batch_size],outputs=[status,table])
     see.click(view_cached,outputs=[status,table])
     send.click(submit,inputs=[allow_blanks],outputs=[status,table])
     export.click(backup,outputs=[exported,backup_status])