Fix docstring and remove wasteful D+1 fetch

Now that fetch_csv_for_date downloads ALL files matching the settlement
date (both the _YYYYMMDD_ first-half and _2_YYYYMMDDHHMMSS_ second-half
ZIPs), the D+1 fetch in _fetch_day_pair was downloading two extra ZIP
files that the boundary filter immediately threw away, making every
request unnecessarily slow.

- Remove _fetch_day_pair; replace with _fetch_day_csv (single call)
- Update all three call sites in api.py accordingly
- Remove now-unused timedelta import from api.py
- Correct module docstring to document the actual two-file-per-day
naming convention: FPPMW_YYYYMMDD_<seq> (first half) and
FPPMW_2_YYYYMMDDHHMMSS_<seq+1> (second half), both sharing the
same settlement date string

https://claude.ai/code/session_01Vm5dTYdRf99LZ9MfB86vjx

app/routers/api.py

@@ -3,7 +3,7 @@ REST API endpoints for BESS SCADA data.
 """
 import json
 import logging
-from datetime import date, datetime, timedelta
+from datetime import date, datetime
 from pathlib import Path
 
 logger = logging.getLogger(__name__)
@@ -57,28 +57,16 @@ def get_quality_flags():
     return json.loads(flags_file.read_text())
 
 
-async def _fetch_day_pair(target_date: date, duid: str) -> tuple[bytes, bytes | None]:
+async def _fetch_day_csv(target_date: date, duid: str) -> bytes:
     """
-    Fetch CSV bytes for target_date and, if available, for target_date + 1 day.
-
-    Each FPPMW daily ZIP may contain two CSV halves covering the full NEM
-    market day (04:00–16:00 UTC and 16:00–04:00 UTC next day); both are
-    concatenated inside fetch_csv_for_date.  The next-day file is also
-    fetched as a safety net for older single-CSV files and boundary coverage.
-    It is fetched with skip_future_check=True (it may be today's date) and
-    any error is silently swallowed — partial data is acceptable.
-    """
-    import asyncio as _asyncio
-    csv_bytes = await fetch_csv_for_date(target_date, duid)
-
-    next_date = target_date + timedelta(days=1)
-    csv_next: bytes | None = None
-    try:
-        csv_next = await fetch_csv_for_date(next_date, duid, skip_future_check=True)
-    except Exception as exc:
-        logger.debug("Next-day CSV not available for %s (%s): %s", duid, next_date, exc)
+    Fetch and concatenate all FPPMW ZIP files for target_date.
 
-    return csv_bytes, csv_next
+    fetch_csv_for_date already finds ALL files whose name contains the
+    settlement date (both the first and second 12-hour halves) and
+    concatenates their CSV bytes.  The data_processor then applies the
+    [04:00 AEST D, 04:00 AEST D+1) boundary filter.
+    """
+    return await fetch_csv_for_date(target_date, duid)
 
 
 @router.get("/data")
@@ -95,8 +83,8 @@ async def get_data(
     ip = _get_ip(request)
 
     try:
-        csv_bytes, csv_next = await _fetch_day_pair(target_date, duid)
-        df = filter_and_process(csv_bytes, duid, target_date, csv_next)
+        csv_bytes = await _fetch_day_csv(target_date, duid)
+        df = filter_and_process(csv_bytes, duid, target_date)
         summary = compute_summary(df)
         records = to_json_records(df)
         log_request(ip, duid, date, "view")
@@ -125,8 +113,8 @@ async def download_csv(
     ip = _get_ip(request)
 
     try:
-        csv_bytes, csv_next = await _fetch_day_pair(target_date, duid)
-        df = filter_and_process(csv_bytes, duid, target_date, csv_next)
+        csv_bytes = await _fetch_day_csv(target_date, duid)
+        df = filter_and_process(csv_bytes, duid, target_date)
         output = to_csv_bytes(df)
         log_request(ip, duid, date, "download_csv")
         filename = f"BESS_SCADA_{duid}_{date}.csv"
@@ -152,8 +140,8 @@ async def download_parquet(
     ip = _get_ip(request)
 
     try:
-        csv_bytes, csv_next = await _fetch_day_pair(target_date, duid)
-        df = filter_and_process(csv_bytes, duid, target_date, csv_next)
+        csv_bytes = await _fetch_day_csv(target_date, duid)
+        df = filter_and_process(csv_bytes, duid, target_date)
         output = to_parquet_bytes(df)
         log_request(ip, duid, date, "download_parquet")
         filename = f"BESS_SCADA_{duid}_{date}.parquet"

app/services/aemo_fetcher.py

@@ -7,18 +7,31 @@ File naming conventions on NEMWEB:
     PUBLIC_NEXT_DAY_FPP_YYYYMMDD[_<suffix>].zip
     ZIP contains the daily CSV directly.
 
-  From 11 Jan 2026 — FPPMW format (nested ZIP):
-    PUBLIC_NEXT_DAY_FPPMW_YYYYMMDD[_<suffix>].zip
-    Outer ZIP → one or more inner ZIPs → CSV(s).
-    The NEM market day (04:00–04:00 AEST) is split into two 12-hour halves.
-    The second half (16:00 AEST → 04:00 AEST next calendar day) is typically
-    published as a separate inner ZIP whose filename carries the next calendar
-    date.  All inner ZIPs must be opened and concatenated; the data_processor
-    applies the authoritative [04:00 AEST D, 04:00 AEST D+1) boundary filter.
-    All AEMO NEMWEB timestamps are in AEST (UTC+10), no daylight saving.
+  From 11 Jan 2026 — FPPMW format (two ZIPs per market day):
+    Each NEM market day (04:00–04:00 AEST) is split into two 12-hour
+    halves, each published as a separate outer ZIP.  Both files carry
+    the same settlement date (YYYYMMDD) in their name:
+
+      First half  (04:00–16:00 AEST):
+        PUBLIC_NEXT_DAY_FPPMW_YYYYMMDD_<seq>.zip
+
+      Second half (16:00–04:00 AEST next calendar day):
+        PUBLIC_NEXT_DAY_FPPMW_2_YYYYMMDDHHMMSS_<seq+1>.zip
+
+    Both are identified by searching for "FPPMW" AND "YYYYMMDD" in the
+    directory listing.  _find_files_in returns ALL matches; fetch_csv_for_date
+    downloads them all and concatenates their CSV bytes.
+    The data_processor applies the authoritative [04:00 AEST D, 04:00 AEST D+1)
+    boundary filter.  All AEMO NEMWEB timestamps are in AEST (UTC+10),
+    no daylight saving.
+
+    Inner ZIP structure (each outer ZIP):
+      PUBLIC_NEXT_DAY_FPPMW_*.zip   ← outer ZIP (~MB)
+        PUBLIC_NEXT_DAY_FPPMW_*.ZIP ← inner ZIP
+          PUBLIC_NEXT_DAY_FPPMW_*.CSV
 
 Current directory (https://www.nemweb.com.au/REPORTS/Current/FPPDAILY/):
-  Individual daily files (FPP or FPPMW naming), one per market day.
+  Two files per market day (FPPMW format), or one per day (FPP format).
   Keeps a long rolling history (180+ files observed).
 
 Archive directory (https://nemweb.com.au/Reports/Archive/FPPDAILY/):