Day 4: 42-question eval harness; fix max file size (routing.py was unindexed)

data/eval_cache.json

@@ -0,0 +1,24 @@
+{
+  "f04d2a75d7d12ce4": "You set it on the path‑operation decorator itself (or by returning a response object with its own code).\n\n```python\nfrom fastapi import FastAPI, status\nfrom fastapi.responses import JSONResponse\n\napp = FastAPI()\n\n@app.post(\"/items/\", status_code=status.HTTP_201_CREATED)   # ← custom code\ndef create_item(item: Item):\n    # normal return → FastAPI will wrap it in a JSONResponse with 201\n    return item\n\n# or, if you need to decide at runtime:\ndef update_item(item_id: int):\n    if item_exists(item_id):\n        return {\"msg\": \"updated\"}                     # default 200\n    return JSONResponse(content={\"msg\": \"created\"},\n                        status_code=status.HTTP_201_CREATED)  # ← custom code\n```\n\n* The `status_code` argument on the decorator defines the default response code and is added to the OpenAPI schema [5][2].  \n* You can also return a `Response`/`JSONResponse` directly and set its `status_code` [1].",
+  "2768894883ca09f0": "{\"faithful\": true, \"correct\": true}",
+  "07679a24f3a9254a": "You declare a parameter of type **`UploadFile`** (or `bytes`) and mark it with **`File`** in the path‑operation function. FastAPI will then read the multipart/form‑data request and give you the uploaded file.\n\n```python\nfrom fastapi import FastAPI, File, UploadFile\n\napp = FastAPI()\n\n@app.post(\"/upload\")\nasync def upload(file: UploadFile = File(...)):\n    # `file` is an UploadFile instance with attributes like .filename, .content_type, etc.\n    contents = await file.read()          # read the file if needed\n    return {\"filename\": file.filename}\n```\n\n* `UploadFile` is imported from `fastapi` and provides methods such as `read`, `write`, `seek`, etc. [2]  \n* Declaring the parameter with `File` creates a file‑upload field in the request body. [5]  \n* Using `File`, `bytes`, or `UploadFile` lets FastAPI receive files sent as form data. [3]",
+  "fa4b51b532933b37": "{\"faithful\": true, \"correct\": true}",
+  "d98a54f321065d50": "You run a task after the response is sent by injecting a `BackgroundTasks` object into the path‑operation function, adding the callable with `add_task`, and then returning the normal response.\n\n```python\nfrom fastapi import FastAPI, BackgroundTasks\n\napp = FastAPI()\n\ndef write_log(message: str):\n    with open(\"log.txt\", \"a\") as f:\n        f.write(message + \"\\n\")\n\n@app.post(\"/process\")\nasync def process(background_tasks: BackgroundTasks):\n    # schedule the function to run after the response is sent\n    background_tasks.add_task(write_log, \"processing finished\")\n    return {\"status\": \"accepted\"}          # 2xx response → background runs\n```\n\n* Declare a parameter of type `BackgroundTasks` in the endpoint (or a dependency) [2].  \n* Use `background_tasks.add_task(func, *args, **kwargs)` to queue the function [4].  \n* The task is executed only when the response has a successful (2xx) status [1].  \n\nIf you raise an exception (e.g., `HTTPException`) the background tasks are **not** run, because they are only triggered for successful responses [3].",
+  "23c4601a08b6b227": "{\"faithful\": true, \"correct\": true}",
+  "3b284540067f764b": "Add the **CORSMiddleware** to your FastAPI app and list the origins that are allowed to call it.\n\n```python\nfrom fastapi import FastAPI\nfrom fastapi.middleware.cors import CORSMiddleware   # or starlette.middleware.cors\n\napp = FastAPI()\n\norigins = [\n    \"http://localhost:8080\",   # your frontend URL\n]\n\napp.add_middleware(\n    CORSMiddleware,\n    allow_origins=origins,          # <‑‑ origins that may call the API\n    allow_methods=[\"*\"],           # or a list like [\"GET\", \"POST\"]\n    allow_headers=[\"*\"],           # or a list of allowed request headers\n    allow_credentials=True,        # if you need cookies / auth headers\n)\n```\n\nThe middleware will handle CORS pre‑flight (`OPTIONS`) requests and add the appropriate `Access‑Control‑*` headers to normal responses, allowing the frontend at `http://localhost:8080` to call the API [2].",
+  "9bbcac71807abdb4": "{\"faithful\": true, \"correct\": true}",
+  "862d19b2979409bb": "Declare the query parameter in the endpoint function with a default value.  \nAny default (including `None`) makes the parameter optional:\n\n```python\nfrom fastapi import FastAPI\n\napp = FastAPI()\n\n@app.get(\"/items/\")\ndef read_items(\n    q: str = \"fixedquery\",      # optional, default value\n    skip: int = 0,              # optional, default value\n    limit: int | None = None    # optional, default None\n):\n    return {\"q\": q, \"skip\": skip, \"limit\": limit}\n```\n\n- Setting the default to `None` (e.g., `q: str | None = None`) makes the query parameter optional [1].  \n- Any other default value (e.g., `\"fixedquery\"`) also makes the parameter optional [2].",
+  "68f7e2127104a8d6": "{\"faithful\": true, \"correct\": true}",
+  "f5f87ebcadd556c4": "FastAPI lets a *path operation function* declare the things it needs by adding parameters wrapped with `Depends`.  \nWhen a request matches a route, FastAPI **automatically**:\n\n1. Calls each declared dependency (or any other callable that can be used as a dependency) with the same request‑derived parameters that normal path functions receive.  \n2. Takes the return value of that call.  \n3. Injects that value into the corresponding parameter of the original path operation function.\n\nThus the developer never calls the dependency directly; FastAPI resolves and “injects” it for every request, enabling shared logic such as database connections, security checks, etc. [2][3][5]",
+  "c08b0f6a49c19444": "{\"faithful\": true, \"correct\": true}",
+  "ba3259db2b841766": "`jsonable_encoder` is a FastAPI utility that walks through **any** Python object and returns a structure that the standard `json.dumps()` can handle (e.g., `dict`, `list`, `str`, `int`, `float`, `None`).  \nIt knows how to:\n\n* Convert Pydantic models to plain dictionaries (`model_dump(mode=\"json\")`)【3†L19-L31】.  \n* Turn dataclasses into dicts, Enums into their values, `Path` objects into strings, etc.【3†L33-L45】.  \n* Apply optional `include` / `exclude` filters, `by_alias`, `exclude_unset`, `exclude_none`, custom encoders, and SQL‑Alchemy‑safe handling【3†L9-L57】.  \n\n**When you need it**\n\n* **Before persisting data** in a store that only accepts JSON‑compatible values (e.g., a database that can’t store `datetime` objects). The docs show converting a Pydantic model with a `datetime` field into a `dict` where the date becomes an ISO‑format string【2†L7-L15】.  \n* **When building a response manually** (e.g., passing data to `JSONResponse` or a custom `Response`). FastAPI does not alter a `Response` you return, so you must ensure its content is JSON‑compatible first【4†L1-L7】.  \n* **Whenever FastAPI itself needs to serialize return values** that are not already a response model; FastAPI calls `jsonable_encoder` internally to prepare the data for `JSONResponse`【5†L7-L12】.  \n\nIn short, `jsonable_encoder` turns complex Python objects (Pydantic models, dataclasses, enums, paths, etc.) into plain JSON‑compatible data structures, and you use it whenever you have to serialize such objects yourself—e.g., for database storage or custom response handling.",
+  "38c440a1302f1bd5": "{\"faithful\": false, \"correct\": true}",
+  "5932c4a434926300": "The **`APIRouter`** class is defined in the **`fastapi.routing`** module and is re‑exported at the top‑level of the FastAPI package, so it can be imported directly with:\n\n```python\nfrom fastapi import APIRouter\n```\n\n[1]",
+  "d460c3c3e191d680": "{\"faithful\": false, \"correct\": true}",
+  "59d00e411e264fdd": "The `OAuth2PasswordBearer` class is defined in the **`fastapi/security/oauth2.py`** module. 【1】",
+  "ce3328a57476a69f": "{\"faithful\": true, \"correct\": true}",
+  "d26701006986fce5": "You can add a WebSocket endpoint to a FastAPI app by:\n\n1. **Importing the `WebSocket` class** (it’s re‑exported by FastAPI from Starlette)【4】.  \n2. **Declaring a function decorated with `@app.websocket(...)`** that receives a `WebSocket` parameter.  \n3. **Accepting the connection** with `await websocket.accept()`.  \n4. **Reading and writing** using the `receive_*` and `send_*` methods (e.g., `receive_text`, `send_text`).  \n5. **Running the app** (e.g., `fastapi dev`) and connecting from a browser or client【2】.\n\n```python\nfrom fastapi import FastAPI, WebSocket, WebSocketException, status\n\napp = FastAPI()\n\n@app.websocket(\"/items/{item_id}/ws\")\nasync def websocket_endpoint(\n    *,\n    websocket: WebSocket,\n    item_id: str,\n):\n    # Example of raising a client‑error exception\n    # raise WebSocketException(code=status.WS_1008_POLICY_VIOLATION)\n\n    await websocket.accept()                     # accept the connection\n    while True:\n        data = await websocket.receive_text()    # read a message\n        await websocket.send_text(\n            f\"Message text was: {data}, for item ID: {item_id}\"\n        )                                        # send a response\n```\n\n* The `WebSocketException` class can be raised to close the connection with a specific WebSocket close code (e.g., `status.WS_1008_POLICY_VIOLATION`)【5】.  \n\nRun the app:\n\n```bash\n$ fastapi dev\nINFO: Uvicorn running on http://127.0.0.1:8000\n```\n\nThen open `http://127.0.0.1:8000` in a browser, type messages, and see the server’s responses【2】.",
+  "63e74d7941780851": "{\"faithful\": true, \"correct\": true}",
+  "f82568e4b9ac6687": "I couldn't find this in the indexed codebase.",
+  "d427347a9f2028cb": "I couldn't find this in the indexed codebase."
+}

data/eval_set.jsonl

@@ -0,0 +1,42 @@
+{"question": "How do I set a custom status code for an endpoint?", "category": "api_usage", "answerable": true, "gold": ["response-status-code.md", "additional-status-codes.md"]}
+{"question": "How do I receive an uploaded file in an endpoint?", "category": "api_usage", "answerable": true, "gold": ["request-files.md"]}
+{"question": "How do I run a task in the background after returning a response?", "category": "api_usage", "answerable": true, "gold": ["background-tasks.md", "background.py"]}
+{"question": "How do I enable CORS so my frontend can call the API?", "category": "api_usage", "answerable": true, "gold": ["cors.md"]}
+{"question": "How do I declare optional query parameters with default values?", "category": "api_usage", "answerable": true, "gold": ["query-params.md"]}
+{"question": "How do I declare a path parameter with a type like int?", "category": "api_usage", "answerable": true, "gold": ["path-params.md"]}
+{"question": "How do I receive a JSON request body using a Pydantic model?", "category": "api_usage", "answerable": true, "gold": ["tutorial/body.md"]}
+{"question": "How do I receive form data instead of JSON?", "category": "api_usage", "answerable": true, "gold": ["request-forms.md"]}
+{"question": "How do I return a custom error response with a specific status code and message?", "category": "api_usage", "answerable": true, "gold": ["handling-errors.md"]}
+{"question": "How do I use response_model to control which fields are returned?", "category": "api_usage", "answerable": true, "gold": ["response-model.md"]}
+{"question": "How do I serve static files like images or CSS?", "category": "api_usage", "answerable": true, "gold": ["static-files.md", "staticfiles.py"]}
+{"question": "How do I write tests for my FastAPI endpoints?", "category": "api_usage", "answerable": true, "gold": ["tutorial/testing.md", "testclient.py"]}
+{"question": "How do I use WebSockets in FastAPI?", "category": "api_usage", "answerable": true, "gold": ["websockets.md"]}
+{"question": "How do I set a cookie in a response?", "category": "api_usage", "answerable": true, "gold": ["response-cookies.md"]}
+{"question": "How do I add custom headers to a response?", "category": "api_usage", "answerable": true, "gold": ["response-headers.md"]}
+{"question": "How do I implement JWT token authentication?", "category": "api_usage", "answerable": true, "gold": ["oauth2-jwt.md"]}
+{"question": "How does FastAPI's dependency injection system work?", "category": "behavior", "answerable": true, "gold": ["dependencies/index.md"]}
+{"question": "What does jsonable_encoder do and when do I need it?", "category": "behavior", "answerable": true, "gold": ["encoder.md", "encoders.py"]}
+{"question": "What is the difference between async def and def endpoints?", "category": "behavior", "answerable": true, "gold": ["async.md"]}
+{"question": "How can a dependency run cleanup code after the response is sent?", "category": "behavior", "answerable": true, "gold": ["dependencies-with-yield.md"]}
+{"question": "How can I customize or extend the generated OpenAPI schema?", "category": "behavior", "answerable": true, "gold": ["extending-openapi.md"]}
+{"question": "What error does FastAPI return when request validation fails, and how do I customize it?", "category": "behavior", "answerable": true, "gold": ["handling-errors.md", "exceptions.py"]}
+{"question": "How does the OAuth2 password flow work in FastAPI?", "category": "behavior", "answerable": true, "gold": ["security/first-steps.md", "simple-oauth2.md"]}
+{"question": "How do I run code on application startup and shutdown?", "category": "behavior", "answerable": true, "gold": ["events.md"]}
+{"question": "Can I mount another application under a path prefix?", "category": "behavior", "answerable": true, "gold": ["sub-applications.md"]}
+{"question": "How does FastAPI run regular def endpoints without blocking the event loop?", "category": "behavior", "answerable": true, "gold": ["async.md", "concurrency.py"]}
+{"question": "How do I manage configuration and settings with environment variables?", "category": "behavior", "answerable": true, "gold": ["settings.md"]}
+{"question": "Where is the APIRouter class defined?", "category": "location", "answerable": true, "gold": ["fastapi/routing.py"]}
+{"question": "Where is the OAuth2PasswordBearer class implemented?", "category": "location", "answerable": true, "gold": ["fastapi/security/oauth2.py"]}
+{"question": "Where is the jsonable_encoder function implemented?", "category": "location", "answerable": true, "gold": ["fastapi/encoders.py"]}
+{"question": "Where is the BackgroundTasks class defined?", "category": "location", "answerable": true, "gold": ["fastapi/background.py"]}
+{"question": "Where is FastAPI's HTTPException defined?", "category": "location", "answerable": true, "gold": ["fastapi/exceptions.py"]}
+{"question": "Where is the main FastAPI application class defined?", "category": "location", "answerable": true, "gold": ["fastapi/applications.py"]}
+{"question": "Where are the Query, Path, and Body parameter functions defined?", "category": "location", "answerable": true, "gold": ["fastapi/param_functions.py"]}
+{"question": "Where is the TestClient that FastAPI provides for testing?", "category": "location", "answerable": true, "gold": ["fastapi/testclient.py"]}
+{"question": "How do I connect FastAPI to MongoDB?", "category": "unanswerable", "answerable": false, "gold": []}
+{"question": "Does FastAPI have built-in rate limiting?", "category": "unanswerable", "answerable": false, "gold": []}
+{"question": "How do I integrate Celery task queues with FastAPI?", "category": "unanswerable", "answerable": false, "gold": []}
+{"question": "How do I use FastAPI with the Django ORM?", "category": "unanswerable", "answerable": false, "gold": []}
+{"question": "Does FastAPI include a built-in admin dashboard?", "category": "unanswerable", "answerable": false, "gold": []}
+{"question": "How do I schedule recurring cron jobs inside FastAPI?", "category": "unanswerable", "answerable": false, "gold": []}
+{"question": "Does FastAPI have built-in database migration support?", "category": "unanswerable", "answerable": false, "gold": []}

data/manifest.json

@@ -1,6 +1,6 @@
 {
   "repo": "fastapi/fastapi",
-  "ingested_at": "2026-07-03T07:56:26.690336+00:00",
+  "ingested_at": "2026-07-03T15:24:59.452448+00:00",
   "files": [
     {
       "source_type": "doc",
@@ -832,6 +832,11 @@
       "path": "fastapi/_compat/v2.py",
       "size_bytes": 17601
     },
+    {
+      "source_type": "code",
+      "path": "fastapi/applications.py",
+      "size_bytes": 183452
+    },
     {
       "source_type": "code",
       "path": "fastapi/background.py",
@@ -957,6 +962,11 @@
       "path": "fastapi/responses.py",
       "size_bytes": 4144
     },
+    {
+      "source_type": "code",
+      "path": "fastapi/routing.py",
+      "size_bytes": 253241
+    },
     {
       "source_type": "code",
       "path": "fastapi/security/__init__.py",
@@ -1028,881 +1038,5 @@
       "size_bytes": 222
     }
   ],
-  "issues": [
-    {
-      "source_type": "issue",
-      "path": "issues/617.json",
-      "url": "https://github.com/fastapi/fastapi/issues/617"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/142.json",
-      "url": "https://github.com/fastapi/fastapi/issues/142"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/10857.json",
-      "url": "https://github.com/fastapi/fastapi/issues/10857"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/11143.json",
-      "url": "https://github.com/fastapi/fastapi/issues/11143"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12133.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12133"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/754.json",
-      "url": "https://github.com/fastapi/fastapi/issues/754"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/11573.json",
-      "url": "https://github.com/fastapi/fastapi/issues/11573"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/13399.json",
-      "url": "https://github.com/fastapi/fastapi/issues/13399"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12965.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12965"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12554.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12554"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/1773.json",
-      "url": "https://github.com/fastapi/fastapi/issues/1773"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/13533.json",
-      "url": "https://github.com/fastapi/fastapi/issues/13533"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/10007.json",
-      "url": "https://github.com/fastapi/fastapi/issues/10007"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/1273.json",
-      "url": "https://github.com/fastapi/fastapi/issues/1273"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/190.json",
-      "url": "https://github.com/fastapi/fastapi/issues/190"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/10180.json",
-      "url": "https://github.com/fastapi/fastapi/issues/10180"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/4939.json",
-      "url": "https://github.com/fastapi/fastapi/issues/4939"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12402.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12402"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/13056.json",
-      "url": "https://github.com/fastapi/fastapi/issues/13056"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/3920.json",
-      "url": "https://github.com/fastapi/fastapi/issues/3920"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/13400.json",
-      "url": "https://github.com/fastapi/fastapi/issues/13400"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12313.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12313"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/10360.json",
-      "url": "https://github.com/fastapi/fastapi/issues/10360"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/3163.json",
-      "url": "https://github.com/fastapi/fastapi/issues/3163"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12419.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12419"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/2008.json",
-      "url": "https://github.com/fastapi/fastapi/issues/2008"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/9424.json",
-      "url": "https://github.com/fastapi/fastapi/issues/9424"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/1476.json",
-      "url": "https://github.com/fastapi/fastapi/issues/1476"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14128.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14128"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/10177.json",
-      "url": "https://github.com/fastapi/fastapi/issues/10177"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/13880.json",
-      "url": "https://github.com/fastapi/fastapi/issues/13880"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12245.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12245"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/11037.json",
-      "url": "https://github.com/fastapi/fastapi/issues/11037"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/608.json",
-      "url": "https://github.com/fastapi/fastapi/issues/608"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/501.json",
-      "url": "https://github.com/fastapi/fastapi/issues/501"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/10999.json",
-      "url": "https://github.com/fastapi/fastapi/issues/10999"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/10286.json",
-      "url": "https://github.com/fastapi/fastapi/issues/10286"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/10259.json",
-      "url": "https://github.com/fastapi/fastapi/issues/10259"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12198.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12198"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/10424.json",
-      "url": "https://github.com/fastapi/fastapi/issues/10424"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/3500.json",
-      "url": "https://github.com/fastapi/fastapi/issues/3500"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/639.json",
-      "url": "https://github.com/fastapi/fastapi/issues/639"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/11134.json",
-      "url": "https://github.com/fastapi/fastapi/issues/11134"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/10719.json",
-      "url": "https://github.com/fastapi/fastapi/issues/10719"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/10236.json",
-      "url": "https://github.com/fastapi/fastapi/issues/10236"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15612.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15612"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15401.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15401"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/13471.json",
-      "url": "https://github.com/fastapi/fastapi/issues/13471"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/13067.json",
-      "url": "https://github.com/fastapi/fastapi/issues/13067"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/11215.json",
-      "url": "https://github.com/fastapi/fastapi/issues/11215"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/10787.json",
-      "url": "https://github.com/fastapi/fastapi/issues/10787"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/10322.json",
-      "url": "https://github.com/fastapi/fastapi/issues/10322"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/3317.json",
-      "url": "https://github.com/fastapi/fastapi/issues/3317"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/13175.json",
-      "url": "https://github.com/fastapi/fastapi/issues/13175"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/13150.json",
-      "url": "https://github.com/fastapi/fastapi/issues/13150"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12459.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12459"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/5060.json",
-      "url": "https://github.com/fastapi/fastapi/issues/5060"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14221.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14221"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12426.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12426"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12246.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12246"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/11251.json",
-      "url": "https://github.com/fastapi/fastapi/issues/11251"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/10998.json",
-      "url": "https://github.com/fastapi/fastapi/issues/10998"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/10997.json",
-      "url": "https://github.com/fastapi/fastapi/issues/10997"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/10720.json",
-      "url": "https://github.com/fastapi/fastapi/issues/10720"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/10127.json",
-      "url": "https://github.com/fastapi/fastapi/issues/10127"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/1204.json",
-      "url": "https://github.com/fastapi/fastapi/issues/1204"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14680.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14680"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12240.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12240"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12239.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12239"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/54.json",
-      "url": "https://github.com/fastapi/fastapi/issues/54"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15845.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15845"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15503.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15503"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14454.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14454"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/13019.json",
-      "url": "https://github.com/fastapi/fastapi/issues/13019"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/11892.json",
-      "url": "https://github.com/fastapi/fastapi/issues/11892"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/5642.json",
-      "url": "https://github.com/fastapi/fastapi/issues/5642"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/1131.json",
-      "url": "https://github.com/fastapi/fastapi/issues/1131"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15448.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15448"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15188.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15188"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14888.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14888"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14344.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14344"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/13606.json",
-      "url": "https://github.com/fastapi/fastapi/issues/13606"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/13440.json",
-      "url": "https://github.com/fastapi/fastapi/issues/13440"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/13116.json",
-      "url": "https://github.com/fastapi/fastapi/issues/13116"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/13022.json",
-      "url": "https://github.com/fastapi/fastapi/issues/13022"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12780.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12780"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12055.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12055"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/11580.json",
-      "url": "https://github.com/fastapi/fastapi/issues/11580"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15764.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15764"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15762.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15762"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15716.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15716"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15715.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15715"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15714.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15714"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15713.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15713"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15712.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15712"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15268.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15268"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15118.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15118"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14810.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14810"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14508.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14508"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14496.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14496"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14484.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14484"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14483.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14483"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14467.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14467"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14466.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14466"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14465.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14465"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14444.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14444"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14431.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14431"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14316.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14316"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14271.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14271"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14247.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14247"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/13111.json",
-      "url": "https://github.com/fastapi/fastapi/issues/13111"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12901.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12901"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12323.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12323"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12290.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12290"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12017.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12017"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/11993.json",
-      "url": "https://github.com/fastapi/fastapi/issues/11993"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/11873.json",
-      "url": "https://github.com/fastapi/fastapi/issues/11873"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15855.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15855"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15844.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15844"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15738.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15738"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15680.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15680"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15578.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15578"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15538.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15538"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15535.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15535"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15466.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15466"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15325.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15325"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15324.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15324"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15322.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15322"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15238.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15238"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15237.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15237"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15236.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15236"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15197.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15197"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15138.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15138"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15111.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15111"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15085.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15085"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15049.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15049"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15002.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15002"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/15000.json",
-      "url": "https://github.com/fastapi/fastapi/issues/15000"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14989.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14989"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14988.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14988"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14787.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14787"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14503.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14503"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14502.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14502"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14501.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14501"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14500.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14500"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14499.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14499"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14498.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14498"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14497.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14497"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14494.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14494"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14493.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14493"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14312.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14312"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14225.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14225"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14199.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14199"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14114.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14114"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14078.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14078"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/14010.json",
-      "url": "https://github.com/fastapi/fastapi/issues/14010"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/13839.json",
-      "url": "https://github.com/fastapi/fastapi/issues/13839"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/13715.json",
-      "url": "https://github.com/fastapi/fastapi/issues/13715"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/13316.json",
-      "url": "https://github.com/fastapi/fastapi/issues/13316"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/13119.json",
-      "url": "https://github.com/fastapi/fastapi/issues/13119"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/13023.json",
-      "url": "https://github.com/fastapi/fastapi/issues/13023"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12987.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12987"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12963.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12963"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12936.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12936"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12924.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12924"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12765.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12765"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12497.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12497"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12425.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12425"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12382.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12382"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12111.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12111"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12039.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12039"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/12010.json",
-      "url": "https://github.com/fastapi/fastapi/issues/12010"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/11989.json",
-      "url": "https://github.com/fastapi/fastapi/issues/11989"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/11941.json",
-      "url": "https://github.com/fastapi/fastapi/issues/11941"
-    },
-    {
-      "source_type": "issue",
-      "path": "issues/11812.json",
-      "url": "https://github.com/fastapi/fastapi/issues/11812"
-    }
-  ]
+  "issues": []
 }

src/eval.py

@@ -0,0 +1,181 @@
+"""Evaluate the RAG pipeline against a hand-labeled question set.
+
+Metrics:
+    recall@k   - did any gold source appear in the top-k retrieved chunks?
+    MRR        - 1/rank of the first gold hit (higher = ranked better)
+    refusal    - (with --answers) did unanswerable questions get a refusal?
+    faithful/correct - (with --answers) LLM-as-judge on generated answers
+
+Gold labels are substrings matched (case-insensitive) against each retrieved
+chunk's id/path/symbol, so you label "where the answer lives", not exact ids.
+
+LLM answers and judgments are cached in data/eval_cache.json so re-runs
+are free and fast (important on Groq's free-tier rate limits).
+
+Usage:
+    python src/eval.py                 # retrieval metrics only (no LLM, fast)
+    python src/eval.py --answers      # + generation, refusal, judge metrics
+"""
+
+import argparse
+import hashlib
+import json
+import time
+from pathlib import Path
+
+import chromadb
+from sentence_transformers import SentenceTransformer
+
+DATA_DIR = Path("data")
+EVAL_SET = DATA_DIR / "eval_set.jsonl"
+CACHE_FILE = DATA_DIR / "eval_cache.json"
+EMBED_MODEL = "BAAI/bge-small-en-v1.5"
+QUERY_PREFIX = "Represent this sentence for searching relevant passages: "
+K = 5
+REFUSAL_TEXT = "I couldn't find this in the indexed codebase"
+SLEEP_BETWEEN_LLM_CALLS = 2  # stay under free-tier rate limits
+
+JUDGE_PROMPT = """\
+You are grading a RAG system's answer. Given the question, the context the
+system retrieved, and its answer, output ONLY a JSON object:
+{{"faithful": true/false, "correct": true/false}}
+
+faithful = every claim in the answer is supported by the context
+correct  = the answer actually answers the question accurately
+
+Question: {question}
+
+Context:
+{context}
+
+Answer:
+{answer}"""
+
+
+def load_cache() -> dict:
+    if CACHE_FILE.exists():
+        return json.loads(CACHE_FILE.read_text())
+    return {}
+
+
+def save_cache(cache: dict) -> None:
+    CACHE_FILE.write_text(json.dumps(cache, ensure_ascii=False, indent=2))
+
+
+def cache_key(*parts: str) -> str:
+    return hashlib.sha256("||".join(parts).encode()).hexdigest()[:16]
+
+
+def is_gold_hit(chunk_id: str, meta: dict, gold: list[str]) -> bool:
+    haystack = f"{chunk_id} {meta.get('path', '')} {meta.get('symbol', '')}".lower()
+    return any(g.lower() in haystack for g in gold)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--answers", action="store_true",
+                        help="also generate answers and run the LLM judge")
+    parser.add_argument("--k", type=int, default=K)
+    args = parser.parse_args()
+
+    items = [json.loads(line)
+             for line in EVAL_SET.read_text().splitlines() if line.strip()]
+    print(f"[eval] {len(items)} questions "
+          f"({sum(i['answerable'] for i in items)} answerable)")
+
+    model = SentenceTransformer(EMBED_MODEL)
+    collection = chromadb.PersistentClient(
+        path=str(DATA_DIR / "chroma")).get_collection("chunks")
+    cache = load_cache()
+
+    # -------- retrieval metrics --------
+    recalls, mrrs = [], []
+    retrieved_per_q = []  # keep for the answer phase
+    for item in items:
+        emb = model.encode(QUERY_PREFIX + item["question"],
+                           normalize_embeddings=True)
+        res = collection.query(query_embeddings=[emb.tolist()],
+                               n_results=args.k)
+        hits = list(zip(res["ids"][0], res["metadatas"][0],
+                        res["documents"][0]))
+        retrieved_per_q.append(hits)
+
+        if not item["answerable"]:
+            continue
+        rank = next((r for r, (cid, meta, _) in enumerate(hits, 1)
+                     if is_gold_hit(cid, meta, item["gold"])), None)
+        recalls.append(1.0 if rank else 0.0)
+        mrrs.append(1.0 / rank if rank else 0.0)
+        if not rank:
+            print(f"  [miss] {item['question']}")
+
+    print(f"\n=== Retrieval (k={args.k}) ===")
+    print(f"recall@{args.k}: {sum(recalls)/len(recalls):.2f}  "
+          f"({int(sum(recalls))}/{len(recalls)})")
+    print(f"MRR:       {sum(mrrs)/len(mrrs):.2f}")
+
+    if not args.answers:
+        print("\n(retrieval-only run; add --answers for generation metrics)")
+        return
+
+    # -------- generation + judge metrics --------
+    from ask import SYSTEM_PROMPT, build_prompt  # reuse the real pipeline
+    import os
+    from groq import Groq
+    client = Groq(api_key=os.environ["GROQ_API_KEY"])
+    llm_model = os.environ.get("GROQ_MODEL", "openai/gpt-oss-120b")
+
+    def llm(prompt: str, system: str | None = None) -> str:
+        key = cache_key(llm_model, system or "", prompt)
+        if key in cache:
+            return cache[key]
+        messages = ([{"role": "system", "content": system}] if system else [])
+        messages.append({"role": "user", "content": prompt})
+        out = client.chat.completions.create(
+            model=llm_model, messages=messages,
+            temperature=0.1).choices[0].message.content
+        cache[key] = out
+        save_cache(cache)
+        time.sleep(SLEEP_BETWEEN_LLM_CALLS)
+        return out
+
+    refusal_ok, faithful, correct = [], [], []
+    for item, hits in zip(items, retrieved_per_q):
+        hit_dicts = [{"text": doc, "meta": meta}
+                     for cid, meta, doc in hits]
+        ans = llm(build_prompt(item["question"], hit_dicts),
+                  system=SYSTEM_PROMPT)
+
+        if not item["answerable"]:
+            ok = REFUSAL_TEXT.lower() in ans.lower()
+            refusal_ok.append(1.0 if ok else 0.0)
+            if not ok:
+                print(f"  [no refusal] {item['question']}")
+            continue
+
+        context = "\n\n".join(h["text"][:1500] for h in hit_dicts)
+        verdict_raw = llm(JUDGE_PROMPT.format(
+            question=item["question"], context=context, answer=ans))
+        try:
+            start = verdict_raw.index("{")
+            end = verdict_raw.rindex("}") + 1
+            verdict = json.loads(verdict_raw[start:end])
+        except (ValueError, json.JSONDecodeError):
+            print(f"  [judge parse fail] {item['question']}")
+            continue
+        faithful.append(1.0 if verdict.get("faithful") else 0.0)
+        correct.append(1.0 if verdict.get("correct") else 0.0)
+        if not verdict.get("correct"):
+            print(f"  [incorrect] {item['question']}")
+
+    print("\n=== Generation ===")
+    if faithful:
+        print(f"faithful:  {sum(faithful)/len(faithful):.2f}")
+        print(f"correct:   {sum(correct)/len(correct):.2f}")
+    if refusal_ok:
+        print(f"refusal:   {sum(refusal_ok)/len(refusal_ok):.2f}  "
+              f"({int(sum(refusal_ok))}/{len(refusal_ok)})")
+
+
+if __name__ == "__main__":
+    main()

src/ingest.py

@@ -16,7 +16,7 @@ SKIP_DIRS = {".git", ".github", "__pycache__", "node_modules", "tests", "test",
              "scripts"}
 CODE_EXTS = {".py"}
 DOC_EXTS = {".md"}
-MAX_FILE_BYTES = 100_000   # skip generated/vendored monsters
+MAX_FILE_BYTES = 400_000   # skip generated/vendored monsters
 ISSUE_LOOKBACK_DAYS = 730  # ~2 years