CLAUDE.md

CLAUDE.md — BESS-SCADA-Data project guide

A FastAPI app that streams BESS SCADA (4-second power) and dispatch (5-minute energy) data live from AEMO NEMWEB. Deployed as a Docker Space on HuggingFace. Nothing is stored server-side; every request fetches fresh from AEMO.

Where things live

app/
  main.py                       FastAPI app, static mounting, cache-busted index
  config.py                     AEMO URLs, start dates, timeouts, column lists
  routers/api.py                All HTTP endpoints
  services/
    aemo_fetcher.py             FPPDAILY ZIP download (Current + Archive)
    dispatch_fetcher.py         Next_Day_Dispatch ZIP download
    data_processor.py           Polars: parse + filter SCADA CSV
    dispatch_processor.py       Polars: parse + filter dispatch CSV
    gen_info_fetcher.py         AEMO XLSX → BESS list (live → bundled snapshot)
    analytics.py                SQLite request logging (/tmp/analytics.db)
  data/
    bess_list.json              Snapshot, refreshed daily by CI workflow
    bess_list_meta.json         Snapshot timestamp + provenance
    quality_flags.json          MW_QUALITY_FLAG descriptions
  static/                       index.html, app.js, style.css (UI)
scripts/refresh_bess_list.py    Standalone refresher invoked by CI
.github/workflows/
  sync_to_hf.yml                Master push → HF Space deploy (filters binaries)
  refresh_bess_list.yml         Daily cron → regenerates bess_list.json on master

Deploy flow

1. Push to master → sync_to_hf.yml runs.
2. The workflow rewrites local history (in the runner only) to strip app/data/aemo_gen_info.xlsx from every commit. HuggingFace rejects any pushed commit that contains a non-Xet binary file, including historical commits — this filter step exists because of that policy.
3. Force-pushes the filtered history to the HF Space's main branch.
4. HF rebuilds the Docker image and swaps the container (~30–60 s).

Origin master is never rewritten by the deploy. Only HF's main is.

Cache-busting

app/main.py reads app/static/index.html once at startup, substitutes __VERSION__ with int(time.time()), and serves the result from memory. Every container restart = new version stamp = browsers refetch JS/CSS. Keep the __VERSION__ placeholders on the app.js and style.css <script> / <link> tags in index.html — removing them silently reintroduces the cache-staleness bug.

AEMO data quirks

• Three FPPMW format eras — see aemo_fetcher.py docstring.
  • Era 1 (Dec 2024 – 28 Apr 2025): single daily file, settlement date in filename.
  • Era 2 (29 Apr – 10 Sep 2025): single 12-hour file (04:00–16:00 only), publication date in filename.
  • Era 3 (11 Sep 2025+): two files per day (FPPMW first half, FPPMW_2 second half). Era logic lives in _era() and _inner_zip_date_str().
• Rolling archive — AEMO removes old monthly bundles. DATA_START_DATE and DISPATCH_START_DATE must be updated when bundles roll off. As of 2026-05: SCADA from 29 Apr 2025, dispatch from 1 Apr 2025. Verify against the actual NEMWEB Archive directory listings, not memory.
• AEMO WAF blocks datacentre IPs on www.aemo.com.au (returns 403 to the HF egress range). The Generation Information XLSX fetch is wrapped in browser-style headers AND backed by the daily-refreshed bess_list.json snapshot. The snapshot's CI runner is on GitHub-hosted infrastructure, which AEMO doesn't block.
• NEM market day boundary is 04:00 AEST, not midnight. data_processor.py filters strictly to [04:00 AEST D, 04:00 AEST D+1). Era 1 callers must request both D and D+1 calendar files to cover one market day.

Critical API contracts

• /api/bess returns {states, source, fetched_at, warnings} — callers must read .states for the unit list. Breaking this shape breaks the frontend dropdown (and any external scripts). If you must change it, update app.js:init() in lockstep.
• /api/data and /api/energy-data return {duid, date, total_rows, summary, data, warnings?}.
• All endpoints raise HTTPException(404) for "no data for this DUID/date" and HTTPException(503) for upstream AEMO failures. Don't change these status codes — the frontend distinguishes them.

Running locally

pip install -r requirements.txt
uvicorn app.main:app --reload --port 8000
# then open http://localhost:8000

Things that have bitten us before

• bess_list.json once contained fictional DUIDs (e.g. NESBESS1, HORNSDALE_PWR1) that don't exist in real SCADA data. The daily workflow guarantees this can't happen now, but verify any manual edit against scripts/refresh_bess_list.py output.
• Browser cache served old app.js with new /api/bess shape → nonsense state-dropdown options. The __VERSION__ cache-bust prevents recurrence; do not remove it.
• HuggingFace rejects pushed commits with binary files anywhere in history, not just current tree. Don't reintroduce binary commits even if you plan to delete them in the next commit.

---

CLAUDE.md skill to append to any coding project

CLAUDE.md — 12-rule template

These rules apply to every task in this project unless explicitly overridden. Bias: caution over speed on non-trivial work. Use judgment on trivial tasks.

Rule 1 — Think Before Coding

State assumptions explicitly. If uncertain, ask rather than guess. Present multiple interpretations when ambiguity exists. Push back when a simpler approach exists. Stop when confused. Name what's unclear.

Rule 2 — Simplicity First

Minimum code that solves the problem. Nothing speculative. No features beyond what was asked. No abstractions for single-use code. Test: would a senior engineer say this is overcomplicated? If yes, simplify.

Rule 3 — Surgical Changes

Touch only what you must. Clean up only your own mess. Don't "improve" adjacent code, comments, or formatting. Don't refactor what isn't broken. Match existing style.

Rule 4 — Goal-Driven Execution

Define success criteria. Loop until verified. Don't follow steps. Define success and iterate. Strong success criteria let you loop independently.

Rule 5 — Use the model only for judgment calls

Use me for: classification, drafting, summarization, extraction. Do NOT use me for: routing, retries, deterministic transforms. If code can answer, code answers.

Rule 6 — Token budgets are not advisory

Per-task: 4,000 tokens. Per-session: 30,000 tokens. If approaching budget, summarize and start fresh. Surface the breach. Do not silently overrun.

Rule 7 — Surface conflicts, don't average them

If two patterns contradict, pick one (more recent / more tested). Explain why. Flag the other for cleanup. Don't blend conflicting patterns.

Rule 8 — Read before you write

Before adding code, read exports, immediate callers, shared utilities. "Looks orthogonal" is dangerous. If unsure why code is structured a way, ask.

Rule 9 — Tests verify intent, not just behavior

Tests must encode WHY behavior matters, not just WHAT it does. A test that can't fail when business logic changes is wrong.

Rule 10 — Checkpoint after every significant step

Summarize what was done, what's verified, what's left. Don't continue from a state you can't describe back. If you lose track, stop and restate.

Rule 11 — Match the codebase's conventions, even if you disagree

Conformance > taste inside the codebase. If you genuinely think a convention is harmful, surface it. Don't fork silently.

Rule 12 — Fail loud

"Completed" is wrong if anything was skipped silently. "Tests pass" is wrong if any were skipped. Default to surfacing uncertainty, not hiding it.