Day 5 ablations: hybrid recall@20=0.97, bge-reranker-base hurts MRR (0.71->0.63)

data/eval_cache.json

@@ -112,5 +112,82 @@
   "9b0871cd909b0135": "I couldn't find this in the indexed codebase.",
   "5f4a45bd0a897d76": "I couldn't find this in the indexed codebase.",
   "95b2ceba4706cca8": "I couldn't find this in the indexed codebase.",
-  "615a9cbcb33f39c5": "I couldn't find this in the indexed codebase."
+  "615a9cbcb33f39c5": "I couldn't find this in the indexed codebase.",
+  "28bebeb9258a2ad9": "To set a custom status code for an endpoint in FastAPI, you can declare a parameter of type `Response` in your path operation function and set the `status_code` in that temporary response object. Here's an example [5]:\n```python\nfrom fastapi import FastAPI, Response\n\napp = FastAPI()\n\n@app.get(\"/\")\nasync def endpoint():\n    return Response(content=\"Custom response\", status_code=201)\n```\nAlternatively, you can use the `status_code` parameter when defining the route [2]:\n```python\n@app.post(\"/\", status_code=201)\nasync def endpoint():\n    return {\"message\": \"Success\"}\n```\nIf you need to dynamically set the status code, you can use a dependency to check the API key and raise an `HTTPException` with a custom status code [1]:\n```python\nfrom fastapi import Depends, HTTPException\nfrom fastapi.security import APIKeyHeader\n\nX_API_KEY = APIKeyHeader(name='X-API-Key')\n\ndef check_authentication_header(x_api_key: str = Depends(X_API_KEY)):\n    if x_api_key != \"1234567890\":\n        raise HTTPException(status_code=401, detail=\"Invalid API Key\")\n    return {\"id\": 1234567890}\n```",
+  "025e758768958c60": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "a1e4644f5631e3a4": "To receive an uploaded file in an endpoint, you can define a parameter of type `UploadFile` in your endpoint function. Here's an example [4]:\n```python\nfrom fastapi import FastAPI, UploadFile, File\n\napp = FastAPI()\n\n@app.post(\"/upload/\")\nasync def create_upload_file(file: UploadFile = File(...)):\n    return {\"filename\": file.filename}\n```\nMake sure to install `python-multipart` first, as uploaded files are sent as \"form data\" [4]:\n```console\n$ pip install python-multipart\n```\nYou can also define multiple files and form fields at the same time using `File` and `Form` [5]:\n```python\nfrom fastapi import FastAPI, File, Form, UploadFile\n\napp = FastAPI()\n\n@app.post(\"/upload/\")\nasync def create_upload_file(\n    file: UploadFile = File(...), \n    form_data: str = Form(...)\n):\n    return {\"filename\": file.filename, \"form_data\": form_data}\n```",
+  "c93032d71d708e33": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "34b74e6aa3aed093": "You can define background tasks to be run after returning a response using the `BackgroundTasks` class from FastAPI. This is useful for operations that need to happen after a request, but that the client doesn't really have to be waiting for the operation to complete before receiving the response [1].\n\nHere's an example of how to use it:\n```python\nfrom fastapi import BackgroundTasks, FastAPI\n\napp = FastAPI()\n\ndef write_notification(email: str, message=\"\"):\n    with open(\"log.txt\", mode=\"w\") as email_file:\n        content = f\"notification for {email}: {message}\"\n        email_file.write(content)\n\n@app.post(\"/send-notification/{email}\")\nasync def send_notification(email: str, background_tasks: BackgroundTasks):\n    background_tasks.add_task(write_notification, email, message=\"some notification\")\n    return {\"message\": \"Notification sent in the background\"}\n```\nIn this example, the `write_notification` function is run in the background after the response is sent [2].\n\nYou can also declare a parameter in a path operation function or dependency function with the type `BackgroundTasks`, and then use it to schedule the execution of background tasks after the response is sent [3].\n\nNote that if you return a `Response` object that already has its own `background` attribute set, the injected background tasks will be silently discarded. To avoid this, you can add a warning by emitting a `UserWarning` [5].",
+  "f295cb3f49e9f9eb": "```json\n{\n  \"faithful\": false,\n  \"correct\": true\n}\n```",
+  "dd0cb2c9debbdef4": "To enable CORS in your FastAPI application, you can use the `CORSMiddleware`. Here's how you can do it:\n\n* Import `CORSMiddleware`.\n* Create a list of allowed origins (as strings).\n* Add it as a \"middleware\" to your FastAPI application.\n\nYou can configure the `CORSMiddleware` with the following parameters:\n* `allow_origins`: A list of origins that should be permitted to make cross-origin requests.\n* `allow_methods`: A list of HTTP methods that should be allowed for cross-origin requests. Defaults to `['GET']`.\n* `allow_headers`: A list of HTTP request headers that should be supported for cross-origin requests. Defaults to `[]`.\n* `allow_credentials`: Indicate that cookies should be supported for cross-origin requests. Defaults to `False`.\n\nHere's an example:\n```python\nfrom fastapi import FastAPI\nfrom fastapi.middleware.cors import CORSMiddleware\n\napp = FastAPI()\n\norigins = [\n    \"http://localhost:8000\",\n    \"http://localhost:8001\",\n]\n\napp.add_middleware(\n    CORSMiddleware,\n    allow_origins=origins,\n    allow_credentials=True,\n    allow_methods=[\"*\"],\n    allow_headers=[\"*\"],\n)\n```\nThis way, you can enable CORS for your frontend to call the API [1].",
+  "3a005c0235f0e466": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "839285ca94ecb7e6": "You can declare optional query parameters with default values by setting their default to a specific value, such as `None`, or other default values. \n\nFor example, to declare an optional query parameter `q` with a default value of `None`, you can use:\n```python\nq: str | None = None\n```\nAlternatively, you can use `Query` to declare it explicitly as a query parameter:\n```python\nq: str | None = Query(default=None)\n```\nYou can also use other default values, such as strings, like `\"fixedquery\"`, and add validations, like `min_length`:\n```python\nq: str = \"fixedquery\"\n```\nOr \n```python\nq: Annotated[str, Query(min_length=3)] = \"fixedquery\"\n``` \n [2][1][3]",
+  "f842416650139f03": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "2cd8377170776646": "You can declare the type of a path parameter in the function using standard Python type annotations. For example, to declare a path parameter `item_id` as an `int`, you can do [1]:\n```python\n# item_id is declared to be an int\n```\nThis will give you editor support inside of your function, with error checks, completion, etc. \n\nHere's an example:\n```python\nfrom fastapi import FastAPI\n\napp = FastAPI()\n\n@app.get(\"/items/{item_id}\")\nasync def read_item(item_id: int):\n    return {\"item_id\": item_id}\n```",
+  "2d4a0180a291cd61": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "b83a0f2b3b85924d": "To receive a JSON request body using a Pydantic model, you first need to import `BaseModel` from `pydantic`. Then, define your Pydantic model by inheriting from `BaseModel` and including the fields you expect in the JSON request body. Here's a basic example:\n\n```python\nfrom pydantic import BaseModel\n\nclass Item(BaseModel):\n    name: str\n    description: str\n    price: float\n    tax: float\n```\n\nYou can then use this `Item` model as a parameter in your route functions. FastAPI will automatically validate the incoming JSON request body against your Pydantic model [3].\n\nFor instance:\n\n```python\nfrom fastapi import FastAPI\nfrom pydantic import BaseModel\n\napp = FastAPI()\n\nclass Item(BaseModel):\n    name: str\n    description: str\n    price: float\n    tax: float\n\n@app.post(\"/items/\")\nasync def create_item(item: Item):\n    return item\n```\n\nIn this example, FastAPI will expect a JSON request body like:\n\n```json\n{\n    \"name\": \"Foo\",\n    \"description\": \"The pretender\",\n    \"price\": 42.0,\n    \"tax\": 3.2\n}\n```\n\nAnd it will validate it against the `Item` model. If the JSON is valid, it will be available as the `item` parameter in your route function. If not, FastAPI will return a validation error. \n\nYou can also declare a body as a `dict` with specific key and value types, which allows receiving keys that you don't already know beforehand [1]. \n\nAdditionally, Pydantic supports more complex types and validation, including nested models, lists, and optional fields. For example:\n\n```python\nfrom pydantic import BaseModel\nfrom typing import List, Optional\n\nclass Detail(BaseModel):\n    some_data: Optional[List[str]]\n\nclass Item(BaseModel):\n    name: str\n    description: str\n    price: float\n    tax: float\n    details: Optional[Detail]\n```",
+  "b055668cea673be6": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "912194ce87a09163": "To receive form data instead of JSON, you can use `Form` in your FastAPI application. However, to use forms, you first need to install `python-multipart`. You can install it using pip:\n\n```console\n$ pip install python-multipart\n```\n\nThen, you can declare form fields using `Form` parameters in your path operations. Note that when using `Form` parameters, you can't also declare `Body` fields that you expect to receive as JSON, as the request will have the body encoded using `application/x-www-form-urlencoded` instead of `application/json` [1][2].\n\nHere's an example of how to define form parameters:\n\n```python\nfrom fastapi import FastAPI, Form\n\napp = FastAPI()\n\n@app.post(\"/login/\")\ndef login(username: str = Form(...), password: str = Form(...)):\n    return {\"username\": username}\n```\n\nIn this example, `username` and `password` are form fields that will be received from the request [1].",
+  "0650db67e0d2f9db": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "fc10baea436a99b2": "To return a custom error response with a specific status code and message in FastAPI, you can create a custom `Response` object. For example, you can use `JSONResponse` to return a JSON response with a specific status code and message.\n\nHere's an example using `JSONResponse` directly [1]:\n```python\nfrom fastapi import FastAPI\nfrom fastapi.responses import JSONResponse\n\napp = FastAPI()\n\n@app.get(\"/items/\")\nasync def read_items():\n    return JSONResponse(status_code=422, content={\"detail\": \"Custom error message\"})\n```\nAlternatively, you can use exception handlers to catch specific exceptions and return custom error responses. For example, you can use the `request_validation_exception_handler` in `fastapi/exception_handlers.py` as a reference [5]:\n```python\nfrom fastapi import FastAPI, Request\nfrom fastapi.exceptions import RequestValidationError\nfrom fastapi.responses import JSONResponse\nfrom fastapi import status\n\napp = FastAPI()\n\n@app.exception_handler(RequestValidationError)\nasync def validation_exception_handler(request: Request, exc: RequestValidationError):\n    return JSONResponse(\n        status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,\n        content={\"detail\": \"Custom validation error message\"},\n    )\n```\nYou can also use the `response_class` parameter in your path operation decorator to specify a custom response class, and then return a custom response object [3]:\n```python\nfrom fastapi import FastAPI\nfrom fastapi.responses import HTMLResponse\n\napp = FastAPI()\n\n@app.get(\"/items/\", response_class=HTMLResponse)\nasync def read_items():\n    return HTMLResponse(content=\"<html>Custom error message</html>\", status_code=422)\n```",
+  "4be371a4509f7b0f": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "31258a5bd2aabad3": "You can use the `response_model` parameter in FastAPI to control which fields are returned in the response. This is particularly useful when you want to filter data to expose only public fields and avoid exposing sensitive information.\n\nHere's an example:\n```python\nfrom fastapi import FastAPI\nfrom pydantic import BaseModel\n\napp = FastAPI()\n\nclass InternalItem(BaseModel):\n    name: str\n    description: str | None = None\n    secret_key: str\n\nclass Item(BaseModel):\n    name: str\n    description: str | None = None\n\n@app.get(\"/items/me\", response_model=Item)\nasync def get_item() -> Any:\n    item = InternalItem(\n        name=\"Foo\", description=\"A very nice Item\", secret_key=\"supersecret\"\n    )\n    return item\n```\nIn this example, even though the `get_item` function returns an `InternalItem` object, which has a `secret_key` attribute, the `response_model` parameter is set to `Item`, which does not include the `secret_key` field. As a result, the JSON response sent to the client will not include the `secret_key` field [2].\n\nYou can use `response_model` to define response models and ensure private data is filtered out. Additionally, you can use `response_model_exclude_unset` to return only the values explicitly set [3].",
+  "90b935eea78b3d8e": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "747a7a8dd0005c6b": "You can serve static files like images or CSS using the `StaticFiles` class in FastAPI. To do this, you need to import `StaticFiles` from `fastapi.staticfiles` and then mount a `StaticFiles()` instance in a specific path [1].\n\nHere's an example of how to serve static files:\n```python\nfrom fastapi.staticfiles import StaticFiles\n\n# Mount the StaticFiles instance\napp.mount(\"/static\", StaticFiles(directory=\"static\"), name=\"static\")\n```\nIn this example, the `StaticFiles` instance is mounted at the `/static` path and serves files from the `static` directory.\n\nYou can then access your static files by visiting `http://127.0.0.1:8000/static/filename.css` (or `.js`, `.jpg`, etc.) [4].\n\nFor example, if you have a CSS file `styles.css` in the `static` directory, you can link to it in your HTML template using `url_for()`:\n```jinja\n<link rel=\"stylesheet\" href=\"{{ url_for('static', path='styles.css') }}\">\n```\nThis way, FastAPI will serve the `styles.css` file automatically at the URL `/static/styles.css` [3].",
+  "5ca0d3bce8d33fdc": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "4505811c313090f9": "To write tests for your FastAPI endpoints, you can use the `TestClient` provided by FastAPI. Here's a step-by-step guide:\n\n1. Install `httpx` using pip: `pip install httpx`.\n2. Import `TestClient` and create a `TestClient` instance by passing your FastAPI application to it.\n3. Write test functions with names starting with `test_`.\n4. Use the `TestClient` object to send requests to your FastAPI application.\n5. Write simple `assert` statements to check the responses.\n\nExample:\n```python\nfrom fastapi.testclient import TestClient\nfrom myapp import app\n\nclient = TestClient(app)\n\ndef test_read_main():\n    response = client.get(\"/\")\n    assert response.status_code == 200\n    assert response.json() == {\"message\": \"Hello World\"}\n```\nNote that you can use `pytest` directly with FastAPI.\n\nIf you need to test asynchronous functions, you can use `async def` test functions and `await` the `TestClient` methods [2].\n\nFor more information, check out the [Testing](docs/en/docs/tutorial/testing.md) and [Async Tests](docs/en/docs/advanced/async-tests.md) sections in the FastAPI documentation [1][4].",
+  "300e299e5af00fb6": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "18929192fee244d9": "To use WebSockets in FastAPI, you first need to create a WebSocket endpoint using the `@app.websocket()` decorator. Here's a basic example [5]:\n```python\nfrom fastapi import FastAPI, WebSocket\n\napp = FastAPI()\n\n@app.websocket(\"/ws\")\nasync def websocket_endpoint(websocket: WebSocket):\n    await websocket.accept()\n    while True:\n        data = await websocket.receive_text()\n        await websocket.send_text(f\"Message text was: {data}\")\n```\nThis code sets up a WebSocket endpoint at `/ws` that accepts WebSocket connections, receives text messages, and sends responses back to the client.\n\nYou can run your FastAPI application using `fastapi dev` and access it at `http://127.0.0.1:8000` [1].\n\nFor more information on handling WebSocket connections, including disconnections and multiple clients, refer to the FastAPI documentation [3][4].\n\nThe `WebSocket` class is provided directly by Starlette, but you can import it from `fastapi` for convenience [2]:\n```python\nfrom fastapi import WebSocket\n```",
+  "f6d46715b83ee69b": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "0906e322d446b450": "You can set a cookie in a response by declaring a parameter of type `Response` in your path operation function and then setting the cookie in that temporary response object [1].\n\nHere's an example of how to set a cookie:\n```python\nfrom fastapi import FastAPI, Response\n\napp = FastAPI()\n\n@app.get(\"/\")\ndef read_root(response: Response):\n    response.set_cookie(\"cookie_name\", \"cookie_value\")\n    return {\"message\": \"Hello World\"}\n```\nAlternatively, you can also create a response directly and set cookies in it [2]:\n```python\nfrom fastapi import FastAPI, JSONResponse\n\napp = FastAPI()\n\n@app.get(\"/\")\ndef read_root():\n    response = JSONResponse(content={\"message\": \"Hello World\"}, media_type=\"application/json\")\n    response.set_cookie(\"cookie_name\", \"cookie_value\")\n    return response\n```\nYou can import the `Response` class directly from `fastapi` [3]:\n```python\nfrom fastapi import Response\n```",
+  "43e79d44a1cf2a87": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "33b387b4db40638d": "You can add custom headers to a response in several ways. \n\n1. **Using the `X-` prefix**: You can add custom proprietary headers using the `X-` prefix. However, if you want a client in a browser to see these headers, you need to add them to your CORS configurations using the `expose_headers` parameter [1][2].\n\n2. **Return a `Response` directly**: You can create a response and pass the headers as an additional parameter [3].\n\n```python\nfrom fastapi import FastAPI, Response\n\napp = FastAPI()\n\n@app.get(\"/\")\ndef read_root():\n    headers = {\"X-Custom-Header\": \"Custom Value\"}\n    return Response(content=\"Hello, World!\", media_type=\"text/plain\", headers=headers)\n```\n\n3. **Use a `Response` parameter**: You can declare a parameter of type `Response` in your path operation function and set headers in that temporary response object [5].\n\n```python\nfrom fastapi import FastAPI, Response\n\napp = FastAPI()\n\n@app.get(\"/\")\ndef read_root(response: Response):\n    response.headers[\"X-Custom-Header\"] = \"Custom Value\"\n    return {\"message\": \"Hello, World!\"}\n```\n\n4. **Middleware**: You can also add custom headers using middleware. You can create a middleware function that receives the request and a function `call_next` that will receive the request as a parameter, and then modify the response before returning it [2].\n\n```python\nfrom fastapi import FastAPI, Request\n\napp = FastAPI()\n\n@app.middleware(\"http\")\nasync def add_custom_header(request: Request, call_next):\n    response = await call_next(request)\n    response.headers[\"X-Custom-Header\"] = \"Custom Value\"\n    return response\n```",
+  "e7df91dc36af9185": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "d336527362c38e67": "To implement JWT token authentication, you'll need to handle JWT tokens and update the `/token` path operation. \n\nFirst, import the necessary modules, create a random secret key, and define the algorithm and expiration time for the token. Then, define a Pydantic model for the token endpoint response and create a utility function to generate a new access token [2].\n\nNext, update the `/token` path operation to create a `timedelta` with the expiration time and return a real JWT access token. Use the `sub` key to store the user's identification, ensuring it's unique and a string [3].\n\nHere's a high-level example of the steps:\n```python\nimport secrets\nimport jwt\nfrom datetime import datetime, timedelta\n\n# Define the secret key, algorithm, and expiration time\nSECRET_KEY = secrets.token_urlsafe(32)\nALGORITHM = \"HS256\"\nACCESS_TOKEN_EXPIRE_MINUTES = 30\n\n# Define the Pydantic model for the token endpoint response\nclass Token(BaseModel):\n    access_token: str\n    token_type: str\n\n# Create a utility function to generate a new access token\ndef create_access_token(data: dict, expires_delta: timedelta | None = None):\n    to_encode = data.copy()\n    if expires_delta:\n        expire = datetime.utcnow() + expires_delta\n    else:\n        expire = datetime.utcnow() + timedelta(minutes=15)\n    to_encode.update({\"exp\": expire})\n    encoded_jwt = jwt.encode(to_encode, SECRET_KEY, algorithm=ALGORITHM)\n    return encoded_jwt\n\n# Update the /token path operation to return a JWT access token\n@app.post(\"/token\", response_model=Token)\nasync def login_for_access_token(form_data: OAuth2PasswordRequestForm = Depends()):\n    # Authenticate the user and return a JWT access token\n    access_token_expires = timedelta(minutes=ACCESS_TOKEN_EXPIRE_MINUTES)\n    access_token = create_access_token(\n        data={\"sub\": form_data.username}, expires_delta=access_token_expires\n    )\n    return {\"access_token\": access_token, \"token_type\": \"bearer\"}\n```",
+  "fb49c2bfc14f427f": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "4b5e793ae7400937": "FastAPI's dependency injection system allows you to declare dependencies for your path operation functions, which are then executed and injected with the results by FastAPI. These dependencies are functions that can take the same parameters as path operation functions and return any value. You declare a dependency using the `Depends` function, passing the dependency function as an argument, and FastAPI will take care of executing it and injecting the results [1][3].\n\nDependencies can be used for various purposes, such as [2]:\n* Handling logic that can't be declared in Pydantic validation and requires additional logic\n* Managing external resources or blocking operations\n* Creating sub-dependencies\n* Sharing logic across multiple endpoints\n* Handling cleanup with `yield`, such as database sessions or file handles\n\nYou can apply shared dependencies at the router level using `dependencies=[Depends(...)]`. The dependency injection system also supports more advanced patterns, including `yield` with `scope` and class dependencies [4].\n\nThe dependency injection system integrates well with other FastAPI features, such as background tasks, allowing you to declare `BackgroundTasks` parameters in path operation functions, dependencies, or sub-dependencies [5].",
+  "fa4678ce5f5b85e9": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "6d5309decb05a061": "`jsonable_encoder` is a function in FastAPI that converts an object, such as a Pydantic model, to a JSON-compatible version. This is necessary when you need to store or return data that isn't natively compatible with JSON, such as `datetime` objects or NumPy types.\n\nIt takes an object and returns a Python standard data structure (e.g., a `dict`) with values and sub-values that are all compatible with JSON. For example, it can convert a `datetime` object to a string in ISO format and a Pydantic model to a `dict`.\n\nYou need to use `jsonable_encoder` in scenarios like [1][2][4]:\n* Storing data in a database that only accepts JSON-compatible data.\n* Returning Pydantic models or other non-JSON-compatible data directly in a response.\n* Handling data with non-JSON-compatible types, such as NumPy types from Pandas DataFrames.\n\nHere's an example of using `jsonable_encoder`:\n```python\nfrom fastapi.encoders import jsonable_encoder\nfrom datetime import datetime\n\nclass Item:\n    def __init__(self, name: str, created_at: datetime):\n        self.name = name\n        self.created_at = created_at\n\nitem = Item(\"Example Item\", datetime.now())\njson_compatible_item = jsonable_encoder(item)\n```\nIn this case, `jsonable_encoder` would convert the `item` object to a `dict` with a JSON-compatible representation of the `datetime` object. \n\nFastAPI internally uses `jsonable_encoder` to convert data, but it's also useful in other scenarios [1]. \n\nYou can also use it to handle Pandas DataFrames, which can contain non-JSON-compatible NumPy types [5]:\n```python\nimport pandas as pd\nfrom fastapi.encoders import jsonable_encoder\n\ndf = pd.DataFrame({\n    \"count\": [1, 2, 3],\n    \"score\": [1.5, float(\"nan\"), 3.0],\n})\n\njson_compatible_df = jsonable_encoder(df.to_dict(orient=\"records\"))\n```",
+  "6fb559b5ce1e6285": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "369742ae24b4bc16": "To address your question, I need to understand what you're asking. You haven't specified a particular question about the FastAPI codebase, but I can try to infer and provide information based on the context provided.\n\nIf you're asking about the difference between `async def` and `def` endpoints in FastAPI, here [5] is some relevant documentation:\n\nIn FastAPI, you can define path operation functions using either `async def` or `def`. The main difference lies in how they handle asynchronous operations.\n\n- Use `async def` for functions that are asynchronous and need to use `await` with third-party libraries that support asynchronous operations.\n- Use `def` for synchronous functions or when working with third-party libraries that do not support asynchronous operations. FastAPI will run these functions in a thread pool.\n\nHere's a brief example:\n\n```python\nfrom fastapi import FastAPI\n\napp = FastAPI()\n\n# Asynchronous function\n@app.get('/async')\nasync def read_async():\n    # Assume some_async_library is an async library\n    data = await some_async_library.fetch_data()\n    return data\n\n# Synchronous function\n@app.get('/sync')\ndef read_sync():\n    # Assume some_sync_library is a sync library\n    data = some_sync_library.fetch_data()\n    return data\n```\n\nFor more detailed information, you can refer to the [documentation](docs/en/docs/async.md) and [skills documentation](fastapi/.agents/skills/fastapi/SKILL.md) provided.\n\nIf your question was something else, please provide more details. \n\nI couldn't find this in the indexed codebase for anything else.",
+  "f8e01cc97294352b": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "7c2093facbe24099": "To run cleanup code after the response is sent, a dependency can use `yield` with the default scope `\"request\"`. This ensures that the exit code after `yield` is executed after the response is sent back to the client [1].\n\nHere's an example:\n```python\nfrom typing import Annotated\n\nfrom fastapi import Depends, FastAPI\n\napp = FastAPI()\n\ndef get_db():\n    db = DBSession()\n    try:\n        yield db\n    finally:\n        db.close()\n\nDBDep = Annotated[DBSession, Depends(get_db)]\n\n@app.get(\"/items/\")\nasync def read_items(db: DBDep):\n    return db.query(Item).all()\n```\nIn this case, the `db.close()` statement will be executed after the response is sent [1].\n\nAlternatively, you can explicitly specify the scope as `\"request\"`:\n```python\nUserDep = Annotated[DBSession, Depends(get_db, scope=\"request\")]\n```\nThis achieves the same result [1][4].",
+  "37abbb385dc1c1fc": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "2ea7cf2fa8800910": "You can customize or extend the generated OpenAPI schema in several ways.\n\n## Using `openapi_extra`\n\nYou can extend the OpenAPI schema for a path operation using the `openapi_extra` parameter. This is useful for declaring OpenAPI extensions [1].\n\n## Modifying the OpenAPI Schema\n\nYou can also generate the OpenAPI schema and override each part that you need using the utility function `get_openapi`. This allows you to add custom extensions, such as a custom logo for ReDoc [4].\n\n## Extra JSON Schema Data in Pydantic Models\n\nAdditionally, you can declare extra JSON schema data in Pydantic models using the `model_config` attribute and setting `\"json_schema_extra\"` with a `dict` containing any additional data [3].\n\n## Overriding the Defaults\n\nTo extend OpenAPI, you can override the defaults by generating the OpenAPI schema, modifying it, and caching it. This involves creating a custom `openapi()` function and replacing the original one [4].\n\n## Example Code\n\nHere is an example of using `openapi_extra`:\n```python\nfrom fastapi import FastAPI\nfrom fastapi.openapi.models import OpenAPI\n\napp = FastAPI()\n\n@app.get(\"/items/\", openapi_extra={\n    \"x-aperture-labs-portal\": \"stargate\"\n})\nasync def read_items():\n    return [{\"item\": \"Item 1\"}]\n```\nAnd here is an example of generating and modifying the OpenAPI schema:\n```python\nfrom fastapi import FastAPI\nfrom fastapi.openapi.utils import get_openapi\n\napp = FastAPI()\n\ndef custom_openapi():\n    if app.openapi_schema:\n        return app.openapi_schema\n    openapi_schema = get_openapi(\n        title=app.title,\n        version=app.version,\n        description=app.description,\n        routes=app.routes,\n    )\n    openapi_schema[\"info\"][\"x-logo\"] = {\"url\": \"https://example.com/logo.png\"}\n    app.openapi_schema = openapi_schema\n    return openapi_schema\n\napp.openapi = custom_openapi\n```",
+  "1c7b5a769de3e356": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "9461901effdb2eba": "When request validation fails in FastAPI, it returns a `ValidationError` with a 422 status code. The error response includes details about the validation errors, such as the location and message of the error.\n\nThe default error response for request validation errors in FastAPI is:\n```json\n{\n    \"detail\": [\n        {\n            \"loc\": [\n                \"path\",\n                \"item_id\"\n            ],\n            \"msg\": \"value is not a valid integer\",\n            \"type\": \"type_error.integer\"\n        }\n    ]\n}\n```\nTo customize this error response, you can override the default exception handler for `RequestValidationError` using the `@app.exception_handler(RequestValidationError)` decorator [3].\n\nFor example, you can return a plain text response instead of JSON:\n```python\nfrom fastapi import FastAPI, RequestValidationError\nfrom fastapi.responses import PlainTextResponse\n\napp = FastAPI()\n\n@app.exception_handler(RequestValidationError)\nasync def validation_error_handler(request, exc):\n    return PlainTextResponse(\n        content=\"Validation errors: \" + str(exc),\n        status_code=422,\n    )\n```\nAlternatively, you can modify the OpenAPI entry for `ValidationError` and `HTTPValidationError` using the `openapi_schema` attribute of the FastAPI app [4].\n\nFastAPI uses Pydantic models to validate request data, and the `ValidationError` exception is raised when the data is invalid [1]. You can use Pydantic models to define the expected structure and content of request bodies.\n\nFastAPI also provides a `jsonable_encoder` function to encode the error details in a JSON-compatible format [5].",
+  "f8aadfea4bd68d7b": "```json\n{\n    \"faithful\": true,\n    \"correct\": true\n}\n```",
+  "4d260caf05bd6b11": "The OAuth2 password flow in FastAPI works as follows:\n\n* The user types their `username` and `password` in the frontend and hits `Enter`.\n* The frontend sends the `username` and `password` to a specific URL in the API (declared with `tokenUrl=\"token\"`).\n* The API checks the `username` and `password` and responds with a \"token\".\n* The frontend stores the token temporarily.\n* When the frontend needs to access an authenticated endpoint, it sends an `Authorization` header with the token.\n\nIn FastAPI, this flow is implemented using the `OAuth2PasswordRequestForm` and `OAuth2PasswordBearer` classes. \n\nThe `OAuth2PasswordRequestForm` class is used to collect the `username` and `password` as form data, while the `OAuth2PasswordBearer` class is used to handle the token authentication.\n\nHere's an example of how to use these classes:\n```python\nfrom fastapi import Depends, FastAPI\nfrom fastapi.security import OAuth2PasswordRequestForm, OAuth2PasswordBearer\n\napp = FastAPI()\n\noauth2_scheme = OAuth2PasswordBearer(tokenUrl=\"token\")\n\n@app.post(\"/login\")\ndef login(form_data: OAuth2PasswordRequestForm = Depends()):\n    # Check username and password\n    # Return token\n    pass\n\n@app.get(\"/protected\")\ndef protected(token: str = Depends(oauth2_scheme)):\n    # Use token to authenticate\n    pass\n```\nThe `OAuth2PasswordRequestForm` class ensures that the `username` and `password` are collected using form data, while the `OAuth2PasswordBearer` class handles the token authentication [2][3].",
+  "a46ee8fe3b121777": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "5dee9d2d67fcfdd8": "You can run code on application startup and shutdown using event handlers or the `lifespan` parameter.\n\n## Using Event Handlers\n\nYou can define event handlers using `@app.on_event(\"startup\")` and `@app.on_event(\"shutdown\")` decorators. Here's an example [1]:\n```python\n@app.on_event(\"startup\")\nasync def startup_event():\n    # Code to run on startup\n    pass\n\n@app.on_event(\"shutdown\")\nasync def shutdown_event():\n    # Code to run on shutdown\n    pass\n```\n## Using Lifespan Parameter\n\nAlternatively, you can use the `lifespan` parameter when creating the FastAPI app [2]:\n```python\nfrom contextlib import asynccontextmanager\n\n@asynccontextmanager\nasync def lifespan(app: FastAPI):\n    # Code to run on startup\n    try:\n        yield\n    finally:\n        # Code to run on shutdown\n        pass\n\napp = FastAPI(lifespan=lifespan)\n```\nIn both cases, you can execute code before the application starts receiving requests and after it finishes handling requests [3].",
+  "b8457d6b5cb39455": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "01c7e4c1799ac5e9": "Yes, you can mount another application under a path prefix. This is achieved by using the `mount` method on a FastAPI instance. \n\nFor example, if you have a sub-application `subapi` and you want to mount it under the path prefix `/api/v1/sub`, you can do it like this:\n```python\nfrom fastapi import FastAPI\n\nsubapi = FastAPI()\n\n# ... define routes for subapi ...\n\napp = FastAPI()\n\napp.mount(\"/api/v1/sub\", subapi)\n```\nFastAPI will internally use the `root_path` smartly, so it will just work [1].\n\nHowever, when using `APIRouter`, it's a bit more complex. `APIRouter` has a `mount` method, but it's not optimized to work with sub-applications. The best practice is to mount your sub-application directly to the main FastAPI instance using the full path prefix [3].\n\nIt's also worth noting that when you mount a sub-application, the sub-application will have its `root_path` correctly set [1]. \n\nYou can also use `root_path` to configure your application when you have a proxy with a stripped path prefix [2]. \n\nFor example, if you have a proxy that adds a path prefix to your application, you can use `root_path` to handle this case. \n\nBut if you are trying to mount a sub-application under an `APIRouter`, it is a known limitation and you might need to reconsider your design [3]. \n\nYou can find more information in the documentation about [mounting sub-applications](https://fastapi.tiangolo.com/advanced/sub-applications/) and [using a proxy with a stripped path prefix](https://fastapi.tiangolo.com/advanced/behind-a-proxy/).",
+  "c1bd7ef1f9271cdf": "```json\n{\n  \"faithful\": false,\n  \"correct\": false\n}\n```",
+  "d0d0425aad0bc303": "FastAPI runs regular `def` endpoints without blocking the event loop by running them in a threadpool using `await run_in_threadpool(dependant.call, **values)` in the `run_endpoint_function` [4].\n\nThis allows the event loop to continue running other tasks while the endpoint function is executing, preventing blocking and ensuring that the application remains responsive.\n\nAdditionally, when needing to mix blocking and async code, it's recommended to use Asyncer, which provides a way to run blocking code inside async functions and vice versa using `asyncify()` and `syncify()` [5].",
+  "9ba358b72728a34d": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "e54c5e2d5de4f8ce": "You can manage configuration and settings with environment variables using Pydantic Settings. This allows you to handle settings or configurations for your application with the power of Pydantic models. You can use a dependency to simplify testing, utilize `.env` files, and apply `@lru_cache` to avoid reading the dotenv file repeatedly while allowing overrides during testing [1].\n\nFirst, install the required package:\n```console\n$ pip install pydantic-settings\n```\nor \n```console\n$ pip install \"fastapi[all]\"\n```\nThen, create a `Settings` object by importing `BaseSettings` from Pydantic and creating a sub-class. Declare class attributes with type annotations and possible default values. Pydantic will read environment variables in a case-insensitive way, convert and validate the data [4].\n\nFor example, you can define a `config.py` file with:\n```python\nfrom pydantic import BaseSettings\n\nclass Settings(BaseSettings):\n    admin_email: str\n    app_name: str\n```\nAnd use it in your `main.py` file:\n```python\nfrom config import Settings\n\nsettings = Settings()\nprint(settings.admin_email)\nprint(settings.app_name)\n```\nYou can also use `.env` files to store environment variables. Create a `.env` file with:\n```bash\nADMIN_EMAIL=\"deadpool@example.com\"\nAPP_NAME=\"ChimichangApp\"\n```\nThen, update your `config.py` file to read from the `.env` file:\n```python\nfrom pydantic import BaseSettings\n\nclass Settings(BaseSettings):\n    env_file = \".env\"\n    admin_email: str\n    app_name: str\n```\nTo avoid reading the `.env` file for each request, use the `@lru_cache` decorator:\n```python\nfrom functools import lru_cache\n\n@lru_cache()\ndef get_settings():\n    return Settings()\n``` [2]\n\nYou can run the server with environment variables:\n```console\n$ ADMIN_EMAIL=\"deadpool@example.com\" APP_NAME=\"ChimichangApp\" fastapi run main.py\n``` [4]",
+  "a947beae5eed76b6": "```\n{\n  \"faithful\": false,\n  \"correct\": true\n}\n```",
+  "32500c98ea5e65c1": "The `APIRouter` class is defined in the `fastapi/routing.py` file [3][5]. You can also import it directly from `fastapi` [2]:\n```python\nfrom fastapi import APIRouter\n```",
+  "80c01ebf828e21ed": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "042a557c2e4a4e71": "The `OAuth2PasswordBearer` class is implemented in the `fastapi/security/oauth2.py` file [1].",
+  "3c11ed85940cf1e7": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "2325f5f6b0381148": "The `jsonable_encoder` function is implemented in the `fastapi/encoders.py` file [3].",
+  "8aa97952f2b8a102": "```json\n{\n  \"faithful\": true,\n  \"correct\": false\n}\n```",
+  "a336a2afff2a7f5a": "The `BackgroundTasks` class is defined in `fastapi/background.py`. It comes directly from [`starlette.background`](https://www.starlette.dev/background/), but is imported into FastAPI so you can import it from `fastapi` [2][1].",
+  "62e9b40118ab6542": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "befaa7e5b9d09975": "FastAPI's `HTTPException` is defined in `fastapi/exceptions.py` [1].",
+  "30cc83d413377b42": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "67b28e6202c7fc35": "The main FastAPI application class is defined in `fastapi/applications.py` as `FastAPI` [2].\n\nHere's an example of creating a FastAPI instance:\n```python\nfrom fastapi import FastAPI\n\napp = FastAPI()\n```",
+  "971c1829df5a8414": "```json\n{\n  \"faithful\": false,\n  \"correct\": false\n}\n```",
+  "a19e25729146e8f6": "These parameter functions, including `Query`, `Path`, and `Body`, are defined as functions that return special classes, and they can be imported directly from `fastapi` [3]. \n\nThey are actually functions that return objects of subclasses of a common `Param` class, which is itself a subclass of Pydantic's `FieldInfo` class [2].",
+  "81712b5cb22198eb": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "dc681003c4ebc90e": "The `TestClient` that FastAPI provides for testing is located in `fastapi.testclient`. You can import it directly as:\n```python\nfrom fastapi.testclient import TestClient\n```\nIt is also available as `starlette.testclient.TestClient`, as FastAPI provides the same `starlette.testclient` as `fastapi.testclient` for convenience [3][4].",
+  "1ca643277d8b258f": "```json\n{\n  \"faithful\": true,\n  \"correct\": true\n}\n```",
+  "2e4133ebbed89f60": "To connect FastAPI to MongoDB, you would typically use a MongoDB client library for Python, such as `pymongo` or an ORM like `mongoengine` or `sqlmodel` doesn't support MongoDB but `mongoengine` does.\n\nFirst, you need to install the required library. For example, with `pymongo`:\n\n```bash\npip install pymongo\n```\n\nThen, you can use the library to connect to your MongoDB instance.\n\nHere is a basic example of how you might connect to MongoDB and use it with FastAPI:\n\n```python\nfrom fastapi import FastAPI\nfrom pymongo import MongoClient\n\n# Connect to MongoDB\nclient = MongoClient('mongodb://localhost:27017/')\ndb = client[\"your_database_name\"]\ncollection = db[\"your_collection_name\"]\n\napp = FastAPI()\n\n# Now you can use the collection to perform operations\n@app.get(\"/items/\")\ndef read_items():\n    items = collection.find()\n    return list(items)\n```\n\nHowever, for a real application, you'd likely want to handle connections and potential errors more robustly, possibly using dependency injection for database access.\n\nIn a containerized environment, you would also need to ensure MongoDB is running as a separate container and that your FastAPI application container can connect to it [1].\n\nFor more complex scenarios, consider using an async MongoDB client like `motor` which is compatible with async/await syntax used in FastAPI:\n\n```bash\npip install motor\n```\n\n```python\nfrom fastapi import FastAPI\nfrom motor.motor_asyncio import AsyncIOMotorClient\n\nclient = AsyncIOMotorClient('mongodb://localhost:27017/')\ndb = client[\"your_database_name\"]\ncollection = db[\"your_collection_name\"]\n\napp = FastAPI()\n\n@app.get(\"/items/\")\nasync def read_items():\n    items = await collection.find().to_list(length=100)\n    return items\n```",
+  "04b609bd6f97a903": "I couldn't find this in the indexed codebase.",
+  "e3a18d09f0cc9b25": "To integrate Celery task queues with FastAPI, you can use Celery as a tool for running background tasks that don't require sharing memory or variables with the same FastAPI app. This approach is suitable for heavy background computations where you don't need to access FastAPI app variables.\n\nHere's a basic example of how you might structure this:\n\nFirst, ensure you have Celery and a message broker like Redis or RabbitMQ installed. For this example, let's assume you're using Redis.\n\nThen, you can set up Celery in your FastAPI application like so:\n\n```python\nfrom fastapi import FastAPI\nfrom celery import Celery\n\napp = FastAPI()\n\n# Configure Celery\ncelery = Celery(\"tasks\", broker=\"redis://localhost:6379/0\")\n\n@celery.task\ndef my_background_task():\n    # Your background task code here\n    print(\"Running in the background\")\n\n@app.post(\"/start-task/\")\ndef start_task():\n    my_background_task.apply_async()\n    return {\"message\": \"Task started in the background\"}\n```\n\nIn this example, `my_background_task` is a Celery task that runs in the background. When you POST to `/start-task/`, it triggers `my_background_task` to run asynchronously.\n\nFor more complex use cases, including detailed configurations and monitoring, refer to the Celery documentation and [FastAPI's background tasks documentation][1].\n\n[1]: https://fastapi.tiangolo.com/tutorial/background-tasks/",
+  "d2137628f12fc5e5": "You can use FastAPI with the Django ORM by running your FastAPI application with a WSGI server that supports Django, or by using a third-party library that provides integration between FastAPI and Django.\n\nHowever, I couldn't find an official guide or a simple example in the provided context chunks. But I can tell you that it is possible to integrate FastAPI with Django.\n\nFastAPI is an ASGI framework, while Django is a WSGI framework. You can use a library like [Daphne](https://github.com/django/daphne) which is an ASGI server built for Django Channels, or [Hypercorn](https://hypercorn.readthedocs.io/) which is an ASGI server compatible with HTTP/2 and Trio.\n\nTo integrate FastAPI with the Django ORM, you would typically:\n- Run your Django application with the Django development server or a WSGI server like Gunicorn.\n- Run your FastAPI application with an ASGI server like Uvicorn.\n- Use a database like PostgreSQL that both Django and FastAPI can connect to.\n\nHere is an example of how you might structure your project:\n```python\n# myproject/settings.py (Django)\nDATABASES = {\n    'default': {\n        'ENGINE': 'django.db.backends.postgresql',\n        'NAME': 'mydatabase',\n        'USER': 'mydatabaseuser',\n        'PASSWORD': 'mypassword',\n        'HOST': '127.0.0.1',\n        'PORT': '5432',\n    }\n}\n\n# myapp/main.py (FastAPI)\nfrom fastapi import FastAPI\nfrom django.core.management import call_command\nimport os\n\n# Configure Django\nos.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings')\ncall_command('makemigrations')\ncall_command('migrate')\n\nfrom myapp.models import MyModel\n\napp = FastAPI()\n\n@app.get(\"/items/\")\ndef read_items():\n    items = MyModel.objects.all()\n    return [{\"id\": item.id, \"name\": item.name} for item in items]\n```\nKeep in mind that this is a simplified example and does not cover error handling, database connections and other important aspects.\n\nFor more information, you can check [docs/en/docs/deployment/manually.md][1] and [docs/en/docs/advanced/wsgi.md][4].",
+  "7d309358c1c04b1d": "I couldn't find this in the indexed codebase.",
+  "9b649898daa89d96": "I couldn't find this in the indexed codebase.",
+  "5451166579a6804a": "I couldn't find this in the indexed codebase."
 }

src/ask.py

@@ -1,30 +1,24 @@
-"""Ask a question about the codebase — the full RAG pipeline, v0 (baseline).
+"""Ask a question about the codebase — the full RAG pipeline.
 
 Flow:
-    1. Embed your question with the same model used for the chunks
-    2. Ask ChromaDB for the 5 most similar chunks (dense retrieval only, for now)
-    3. Hand those chunks to an LLM and have it answer USING ONLY THEM
-    4. Print the answer plus links to the sources
+    1. Retrieve the most relevant chunks (dense, hybrid, or hybrid+rerank —
+       see retrieval.py for how each mode works)
+    2. Hand those chunks to an LLM and have it answer USING ONLY THEM
+    3. Print the answer plus links to the sources
 
 Usage:
     export GROQ_API_KEY=gsk_...          # free key from console.groq.com
     python src/ask.py "How do I return a custom status code?"
-    python src/ask.py --show-chunks "..."   # also print retrieved chunks
+    python src/ask.py --mode dense "..."     # baseline retrieval
+    python src/ask.py --show-chunks "..."    # also print retrieved chunks
 """
 
 import argparse
 import os
 import sys
-from pathlib import Path
 
-import chromadb
-from groq import Groq
-from sentence_transformers import SentenceTransformer
+from retrieval import retrieve as retrieve_chunks
 
-DATA_DIR = Path("data")
-EMBED_MODEL = "BAAI/bge-small-en-v1.5"
-# BGE models retrieve better when queries carry this instruction prefix
-QUERY_PREFIX = "Represent this sentence for searching relevant passages: "
 LLM_MODEL = os.environ.get("GROQ_MODEL", "openai/gpt-oss-120b")
 TOP_K = 5
 
@@ -38,27 +32,11 @@ Answer ONLY from the numbered context chunks provided. Rules:
 - Be concise."""
 
 
-def retrieve(question: str, k: int) -> list[dict]:
-    model = SentenceTransformer(EMBED_MODEL)
-    query_emb = model.encode(QUERY_PREFIX + question,
-                             normalize_embeddings=True)
-    client = chromadb.PersistentClient(path=str(DATA_DIR / "chroma"))
-    collection = client.get_collection("chunks")
-    res = collection.query(query_embeddings=[query_emb.tolist()], n_results=k)
-    return [{
-        "text": doc,
-        "meta": meta,
-        "distance": dist,
-    } for doc, meta, dist in zip(res["documents"][0],
-                                 res["metadatas"][0],
-                                 res["distances"][0])]
-
-
 def build_prompt(question: str, hits: list[dict]) -> str:
     parts = []
     for i, h in enumerate(hits, 1):
         parts.append(f"[{i}] ({h['meta']['source_type']}: "
-                            f"{h['meta']['path']})\n{h['text'][:2500]}")
+                     f"{h['meta']['path']})\n{h['text'][:2500]}")
     context = "\n\n---\n\n".join(parts)
     return f"Context chunks:\n\n{context}\n\nQuestion: {question}"
 
@@ -67,6 +45,7 @@ def answer(question: str, hits: list[dict]) -> str:
     api_key = os.environ.get("GROQ_API_KEY")
     if not api_key:
         sys.exit("Set GROQ_API_KEY first (free key at console.groq.com).")
+    from groq import Groq
     client = Groq(api_key=api_key)
     response = client.chat.completions.create(
         model=LLM_MODEL,
@@ -83,16 +62,18 @@ def main() -> None:
     parser = argparse.ArgumentParser()
     parser.add_argument("question")
     parser.add_argument("--k", type=int, default=TOP_K)
+    parser.add_argument("--mode", default="hybrid_rerank",
+                        choices=["dense", "hybrid", "hybrid_rerank", "dense_rerank"])
     parser.add_argument("--show-chunks", action="store_true",
                         help="print retrieved chunks (debugging/learning)")
     args = parser.parse_args()
 
-    hits = retrieve(args.question, args.k)
+    hits = retrieve_chunks(args.question, k=args.k, mode=args.mode)
 
     if args.show_chunks:
         for i, h in enumerate(hits, 1):
-            print(f"\n=== [{i}] dist={h['distance']:.3f} "
-                  f"{h['meta']['path']} :: {h['meta']['symbol']} ===")
+            print(f"\n=== [{i}] {h['meta']['path']} :: "
+                  f"{h['meta']['symbol']} ===")
             print(h["text"][:500])
         print("\n" + "=" * 60)
 

src/eval.py

@@ -1,20 +1,19 @@
 """Evaluate the RAG pipeline against a hand-labeled question set.
 
+Now supports retrieval modes for ablation runs:
+    python src/eval.py --mode dense
+    python src/eval.py --mode hybrid
+    python src/eval.py --mode hybrid_rerank
+    python src/eval.py --mode hybrid_rerank --answers
+
 Metrics:
     recall@k   - did any gold source appear in the top-k retrieved chunks?
     MRR        - 1/rank of the first gold hit (higher = ranked better)
     refusal    - (with --answers) did unanswerable questions get a refusal?
     faithful/correct - (with --answers) LLM-as-judge on generated answers
 
-Gold labels are substrings matched (case-insensitive) against each retrieved
-chunk's id/path/symbol, so you label "where the answer lives", not exact ids.
-
-LLM answers and judgments are cached in data/eval_cache.json so re-runs
-are free and fast (important on Groq's free-tier rate limits).
-
-Usage:
-    python src/eval.py                 # retrieval metrics only (no LLM, fast)
-    python src/eval.py --answers      # + generation, refusal, judge metrics
+LLM answers/judgments are cached in data/eval_cache.json (keyed by model
+and prompt), so re-runs only pay for what changed.
 """
 
 import argparse
@@ -23,17 +22,14 @@ import json
 import time
 from pathlib import Path
 
-import chromadb
-from sentence_transformers import SentenceTransformer
+from retrieval import retrieve
 
 DATA_DIR = Path("data")
 EVAL_SET = DATA_DIR / "eval_set.jsonl"
 CACHE_FILE = DATA_DIR / "eval_cache.json"
-EMBED_MODEL = "BAAI/bge-small-en-v1.5"
-QUERY_PREFIX = "Represent this sentence for searching relevant passages: "
 K = 5
 REFUSAL_TEXT = "I couldn't find this in the indexed codebase"
-SLEEP_BETWEEN_LLM_CALLS = 5  # stay under free-tier rate limits
+SLEEP_BETWEEN_LLM_CALLS = 5
 
 JUDGE_PROMPT = """\
 You are grading a RAG system's answer. Given the question, the context the
@@ -66,13 +62,16 @@ def cache_key(*parts: str) -> str:
     return hashlib.sha256("||".join(parts).encode()).hexdigest()[:16]
 
 
-def is_gold_hit(chunk_id: str, meta: dict, gold: list[str]) -> bool:
-    haystack = f"{chunk_id} {meta.get('path', '')} {meta.get('symbol', '')}".lower()
+def is_gold_hit(hit: dict, gold: list[str]) -> bool:
+    meta = hit["meta"]
+    haystack = f"{hit['id']} {meta.get('path', '')} {meta.get('symbol', '')}".lower()
     return any(g.lower() in haystack for g in gold)
 
 
 def main() -> None:
     parser = argparse.ArgumentParser()
+    parser.add_argument("--mode", default="dense",
+                        choices=["dense", "hybrid", "hybrid_rerank", "dense_rerank"])
     parser.add_argument("--answers", action="store_true",
                         help="also generate answers and run the LLM judge")
     parser.add_argument("--k", type=int, default=K)
@@ -80,36 +79,27 @@ def main() -> None:
 
     items = [json.loads(line)
              for line in EVAL_SET.read_text().splitlines() if line.strip()]
-    print(f"[eval] {len(items)} questions "
+    print(f"[eval] mode={args.mode}, {len(items)} questions "
           f"({sum(i['answerable'] for i in items)} answerable)")
 
-    model = SentenceTransformer(EMBED_MODEL)
-    collection = chromadb.PersistentClient(
-        path=str(DATA_DIR / "chroma")).get_collection("chunks")
     cache = load_cache()
 
     # -------- retrieval metrics --------
     recalls, mrrs = [], []
-    retrieved_per_q = []  # keep for the answer phase
+    retrieved_per_q = []
     for item in items:
-        emb = model.encode(QUERY_PREFIX + item["question"],
-                           normalize_embeddings=True)
-        res = collection.query(query_embeddings=[emb.tolist()],
-                               n_results=args.k)
-        hits = list(zip(res["ids"][0], res["metadatas"][0],
-                        res["documents"][0]))
+        hits = retrieve(item["question"], k=args.k, mode=args.mode)
         retrieved_per_q.append(hits)
-
         if not item["answerable"]:
             continue
-        rank = next((r for r, (cid, meta, _) in enumerate(hits, 1)
-                     if is_gold_hit(cid, meta, item["gold"])), None)
+        rank = next((r for r, h in enumerate(hits, 1)
+                     if is_gold_hit(h, item["gold"])), None)
         recalls.append(1.0 if rank else 0.0)
         mrrs.append(1.0 / rank if rank else 0.0)
         if not rank:
             print(f"  [miss] {item['question']}")
 
-    print(f"\n=== Retrieval (k={args.k}) ===")
+    print(f"\n=== Retrieval (mode={args.mode}, k={args.k}) ===")
     print(f"recall@{args.k}: {sum(recalls)/len(recalls):.2f}  "
           f"({int(sum(recalls))}/{len(recalls)})")
     print(f"MRR:       {sum(mrrs)/len(mrrs):.2f}")
@@ -119,40 +109,40 @@ def main() -> None:
         return
 
     # -------- generation + judge metrics --------
-    from ask import SYSTEM_PROMPT, build_prompt  # reuse the real pipeline
     import os
+
+    from ask import SYSTEM_PROMPT, build_prompt
     from groq import Groq
     client = Groq(api_key=os.environ["GROQ_API_KEY"])
     llm_model = os.environ.get("GROQ_MODEL", "openai/gpt-oss-120b")
+    print(f"[eval] llm={llm_model}")
 
     def llm(prompt: str, system: str | None = None) -> str:
-            key = cache_key(llm_model, system or "", prompt)
-            if key in cache:
-                return cache[key]
-            messages = ([{"role": "system", "content": system}] if system else [])
-            messages.append({"role": "user", "content": prompt})
-            out = None
-            for attempt in range(4):
-                try:
-                    out = client.chat.completions.create(
-                        model=llm_model, messages=messages,
-                        temperature=0.1).choices[0].message.content
-                    break
-                except Exception as e:
-                    print(f"  [retry {attempt + 1}/4] {str(e)[:200]}; waiting 30s")
-                    time.sleep(30)
-            if out is None:
-                raise RuntimeError("LLM call failed 4 times; try again later")
-            cache[key] = out
-            save_cache(cache)
-            time.sleep(SLEEP_BETWEEN_LLM_CALLS)
-            return out
+        key = cache_key(llm_model, system or "", prompt)
+        if key in cache:
+            return cache[key]
+        messages = ([{"role": "system", "content": system}] if system else [])
+        messages.append({"role": "user", "content": prompt})
+        out = None
+        for attempt in range(4):
+            try:
+                out = client.chat.completions.create(
+                    model=llm_model, messages=messages,
+                    temperature=0.1).choices[0].message.content
+                break
+            except Exception as e:
+                print(f"  [retry {attempt + 1}/4] {str(e)[:160]}")
+                time.sleep(30)
+        if out is None:
+            raise RuntimeError("LLM call failed 4 times; try again later")
+        cache[key] = out
+        save_cache(cache)
+        time.sleep(SLEEP_BETWEEN_LLM_CALLS)
+        return out
 
     refusal_ok, faithful, correct = [], [], []
     for item, hits in zip(items, retrieved_per_q):
-        hit_dicts = [{"text": doc, "meta": meta}
-                     for cid, meta, doc in hits]
-        ans = llm(build_prompt(item["question"], hit_dicts),
+        ans = llm(build_prompt(item["question"], hits),
                   system=SYSTEM_PROMPT)
 
         if not item["answerable"]:
@@ -162,7 +152,7 @@ def main() -> None:
                 print(f"  [no refusal] {item['question']}")
             continue
 
-        context = "\n\n".join(h["text"][:1500] for h in hit_dicts)
+        context = "\n\n".join(h["text"][:1200] for h in hits)
         verdict_raw = llm(JUDGE_PROMPT.format(
             question=item["question"], context=context, answer=ans))
         try:
@@ -177,7 +167,7 @@ def main() -> None:
         if not verdict.get("correct"):
             print(f"  [incorrect] {item['question']}")
 
-    print("\n=== Generation ===")
+    print(f"\n=== Generation (mode={args.mode}) ===")
     if faithful:
         print(f"faithful:  {sum(faithful)/len(faithful):.2f}")
         print(f"correct:   {sum(correct)/len(correct):.2f}")

src/retrieval.py

@@ -0,0 +1,126 @@
+"""Retrieval backends: dense, hybrid (dense + BM25), and hybrid + reranker.
+
+Modes:
+    dense         - embedding similarity only (the v0 baseline)
+    hybrid        - dense + BM25 keyword search, fused with Reciprocal Rank
+                    Fusion (RRF)
+    hybrid_rerank - hybrid to get ~20 candidates, then a cross-encoder
+                    re-scores each (question, chunk) pair and keeps the best
+
+Why BM25 helps this corpus: code questions contain exact identifiers
+("APIRouter", "jsonable_encoder"). Embeddings blur those into meaning;
+BM25 matches them literally. The two are complementary, and RRF merges
+their rankings without needing to calibrate scores against each other.
+
+Why the reranker helps: the embedding compares question and chunk as two
+separate vectors. A cross-encoder reads them TOGETHER, token by token, so
+it is much better at judging true relevance - but too slow to run on all
+1300 chunks, which is why it only re-scores the top candidates.
+"""
+
+import json
+import re
+from pathlib import Path
+
+import chromadb
+from rank_bm25 import BM25Okapi
+from sentence_transformers import CrossEncoder, SentenceTransformer
+
+DATA_DIR = Path("data")
+EMBED_MODEL = "BAAI/bge-small-en-v1.5"
+RERANK_MODEL = "BAAI/bge-reranker-v2-m3"
+QUERY_PREFIX = "Represent this sentence for searching relevant passages: "
+CANDIDATES = 20   # how many candidates hybrid gathers before final cut
+RRF_K = 60        # standard RRF constant
+
+# Lazy singletons so models/indexes load once per process, not per query
+_embedder = None
+_reranker = None
+_collection = None
+_bm25 = None
+_chunk_ids: list[str] = []
+_chunk_by_id: dict[str, dict] = {}
+
+
+def _tokenize(text: str) -> list[str]:
+    return re.findall(r"[a-z0-9_]+", text.lower())
+
+
+def _load() -> None:
+    global _embedder, _collection, _bm25, _chunk_ids, _chunk_by_id
+    if _embedder is not None:
+        return
+    _embedder = SentenceTransformer(EMBED_MODEL)
+    _collection = chromadb.PersistentClient(
+        path=str(DATA_DIR / "chroma")).get_collection("chunks")
+    chunks = [json.loads(line) for line in
+              (DATA_DIR / "chunks.jsonl").read_text().splitlines()]
+    _chunk_ids = [c["id"] for c in chunks]
+    _chunk_by_id = {c["id"]: c for c in chunks}
+    _bm25 = BM25Okapi([_tokenize(c["text"]) for c in chunks])
+
+
+def _get_reranker() -> CrossEncoder:
+    global _reranker
+    if _reranker is None:
+        _reranker = CrossEncoder(RERANK_MODEL)  # first run downloads ~1GB
+    return _reranker
+
+
+def _dense_ids(question: str, n: int) -> list[str]:
+    emb = _embedder.encode(QUERY_PREFIX + question, normalize_embeddings=True)
+    res = _collection.query(query_embeddings=[emb.tolist()], n_results=n)
+    return res["ids"][0]
+
+
+def _bm25_ids(question: str, n: int) -> list[str]:
+    scores = _bm25.get_scores(_tokenize(question))
+    ranked = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)
+    return [_chunk_ids[i] for i in ranked[:n]]
+
+
+def _rrf_fuse(rankings: list[list[str]]) -> list[str]:
+    """Merge multiple ranked lists: score(id) = sum over lists of
+    1/(RRF_K + rank). Ids high in ANY list surface; high in BOTH win."""
+    scores: dict[str, float] = {}
+    for ranking in rankings:
+        for rank, cid in enumerate(ranking, 1):
+            scores[cid] = scores.get(cid, 0.0) + 1.0 / (RRF_K + rank)
+    return sorted(scores, key=scores.get, reverse=True)
+
+
+def _to_hit(cid: str) -> dict:
+    c = _chunk_by_id[cid]
+    return {
+        "id": cid,
+        "text": c["text"],
+        "meta": {
+            "source_type": c["source_type"],
+            "path": c["path"],
+            "symbol": c["symbol"] or "",
+            "url": c["url"],
+        },
+    }
+
+def _rerank(question: str, candidates: list[str]) -> list[str]:
+    pairs = [(question, _chunk_by_id[cid]["text"][:2000])
+             for cid in candidates]
+    scores = _get_reranker().predict(pairs)
+    return [cid for _, cid in sorted(zip(scores, candidates), reverse=True)]
+
+def retrieve(question: str, k: int = 5, mode: str = "dense") -> list[dict]:
+    _load()
+    if mode == "dense":
+        ids = _dense_ids(question, k)
+    elif mode == "dense_rerank":
+        ids = _rerank(question, _dense_ids(question, CANDIDATES))[:k]
+    elif mode in ("hybrid", "hybrid_rerank"):
+        fused = _rrf_fuse([_dense_ids(question, CANDIDATES),
+                           _bm25_ids(question, CANDIDATES)])
+        candidates = fused[:CANDIDATES]
+        if mode == "hybrid_rerank":
+            candidates = _rerank(question, candidates)
+        ids = candidates[:k]
+    else:
+        raise ValueError(f"unknown mode: {mode}")
+    return [_to_hit(cid) for cid in ids]