Fix silent 500: add global exception handler and NaN safety

Two issues caused 500 responses with no application-level log output:

1. Unhandled exception types
Any exception that is not AEMOFetchError or DataProcessingError
(e.g. polars.exceptions.PolarsError, KeyError, unexpected ValueError)
bypassed both except clauses in every API endpoint and was caught
only by Starlette's ServerErrorMiddleware, which logs to the
'uvicorn.error' logger rather than our application logger. The
result was a 500 with no visible traceback.

Fix: add a global @app .exception_handler(Exception) in main.py that
logs the full traceback via our own logger.error() call before
returning a JSON 500 response. This surfaces the real root cause in
the application log for every future unhandled exception.

2. float('nan') in JSON serialization
Polars cast(pl.Float64, strict=False) can produce a real float NaN
(distinct from null/None) when the source CSV contains the literal
string "nan" or certain edge-case numeric strings. float('nan') is
not JSON-serializable; FastAPI's JSONResponse silently raises a
ValueError during response serialisation, producing a 500 with no
application-level log entry.

Fix: call fill_nan(None) on MEASURED_MW in both to_json_records()
(before to_dicts()) and compute_summary() (before drop_nulls()).
This converts any float NaN to a Polars null, which serialises to
JSON null cleanly.

https://claude.ai/code/session_01TrCrzZmWbQscJiAQaW7Rg8

app/main.py

@@ -1,16 +1,18 @@
 import logging
+import traceback
 from contextlib import asynccontextmanager
 from pathlib import Path
 
-from fastapi import FastAPI
+from fastapi import FastAPI, Request
 from fastapi.middleware.cors import CORSMiddleware
-from fastapi.responses import FileResponse
+from fastapi.responses import FileResponse, JSONResponse
 from fastapi.staticfiles import StaticFiles
 
 from app.routers.api import router as api_router
 from app.services.analytics import init_db
 
 logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
 
 
 @asynccontextmanager
@@ -35,6 +37,25 @@ app.add_middleware(
 
 app.include_router(api_router)
 
+
+@app.exception_handler(Exception)
+async def unhandled_exception_handler(request: Request, exc: Exception) -> JSONResponse:
+    """
+    Catch-all for any exception not already handled by a specific handler or
+    HTTPException.  Logs the full traceback so silent 500s become visible in
+    the application log, then returns a JSON 500 response.
+    """
+    logger.error(
+        "Unhandled exception on %s %s\n%s",
+        request.method,
+        request.url,
+        traceback.format_exc(),
+    )
+    return JSONResponse(
+        status_code=500,
+        content={"detail": "An internal server error occurred. Please try again."},
+    )
+
 STATIC_DIR = Path(__file__).parent / "static"
 app.mount("/static", StaticFiles(directory=str(STATIC_DIR)), name="static")
 

app/services/data_processor.py

@@ -173,7 +173,9 @@ def filter_and_process(
 
 def compute_summary(df: pl.DataFrame) -> dict:
     """Compute summary statistics for the filtered data."""
-    mw = df["MEASURED_MW"].drop_nulls()
+    # fill_nan converts float NaN → null before drop_nulls so that NaN values
+    # (which Polars treats as distinct from null) don't silently corrupt stats.
+    mw = df["MEASURED_MW"].fill_nan(None).drop_nulls()
     flags = df["MW_QUALITY_FLAG"].value_counts().sort("MW_QUALITY_FLAG")
     total = len(df)
 
@@ -225,5 +227,7 @@ def to_json_records(df: pl.DataFrame, max_rows: int = 25_000) -> list[dict]:
     display_df = df.head(max_rows).with_columns([
         pl.col("INTERVAL_DATETIME").dt.strftime("%Y-%m-%d %H:%M:%S"),
         pl.col("MEASUREMENT_DATETIME").dt.strftime("%Y-%m-%d %H:%M:%S"),
+        # Polars float NaN is not JSON-serializable; normalise to null (→ JSON null).
+        pl.col("MEASURED_MW").fill_nan(None),
     ])
     return display_df.to_dicts()