Spaces:
Sleeping
Sleeping
| # CLAUDE.md — BESS-SCADA-Data project guide | |
| A FastAPI app that streams BESS SCADA (4-second power) and dispatch (5-minute | |
| energy) data live from AEMO NEMWEB. Deployed as a Docker Space on HuggingFace. | |
| Nothing is stored server-side; every request fetches fresh from AEMO. | |
| ## Where things live | |
| ``` | |
| app/ | |
| main.py FastAPI app, static mounting, cache-busted index | |
| config.py AEMO URLs, start dates, timeouts, column lists | |
| routers/api.py All HTTP endpoints | |
| services/ | |
| aemo_fetcher.py FPPDAILY ZIP download (Current + Archive) | |
| dispatch_fetcher.py Next_Day_Dispatch ZIP download | |
| data_processor.py Polars: parse + filter SCADA CSV | |
| dispatch_processor.py Polars: parse + filter dispatch CSV | |
| gen_info_fetcher.py AEMO XLSX → BESS list (live → bundled snapshot) | |
| analytics.py SQLite request logging (/tmp/analytics.db) | |
| data/ | |
| bess_list.json Snapshot, refreshed daily by CI workflow | |
| bess_list_meta.json Snapshot timestamp + provenance | |
| quality_flags.json MW_QUALITY_FLAG descriptions | |
| static/ index.html, app.js, style.css (UI) | |
| scripts/refresh_bess_list.py Standalone refresher invoked by CI | |
| .github/workflows/ | |
| sync_to_hf.yml Master push → HF Space deploy (filters binaries) | |
| refresh_bess_list.yml Daily cron → regenerates bess_list.json on master | |
| ``` | |
| ## Deploy flow | |
| 1. Push to `master` → `sync_to_hf.yml` runs. | |
| 2. The workflow rewrites local history (in the runner only) to strip | |
| `app/data/aemo_gen_info.xlsx` from every commit. **HuggingFace rejects | |
| any pushed commit that contains a non-Xet binary file**, including | |
| historical commits — this filter step exists because of that policy. | |
| 3. Force-pushes the filtered history to the HF Space's `main` branch. | |
| 4. HF rebuilds the Docker image and swaps the container (~30–60 s). | |
| Origin `master` is never rewritten by the deploy. Only HF's `main` is. | |
| ## Cache-busting | |
| `app/main.py` reads `app/static/index.html` once at startup, substitutes | |
| `__VERSION__` with `int(time.time())`, and serves the result from memory. | |
| Every container restart = new version stamp = browsers refetch JS/CSS. | |
| **Keep the `__VERSION__` placeholders on the `app.js` and `style.css` | |
| `<script>` / `` tags in index.html** — removing them silently | |
| reintroduces the cache-staleness bug. | |
| ## AEMO data quirks | |
| - **Three FPPMW format eras** — see `aemo_fetcher.py` docstring. | |
| - Era 1 (Dec 2024 – 28 Apr 2025): single daily file, settlement date in filename. | |
| - Era 2 (29 Apr – 10 Sep 2025): single 12-hour file (04:00–16:00 only), publication date in filename. | |
| - Era 3 (11 Sep 2025+): two files per day (FPPMW first half, FPPMW_2 second half). | |
| Era logic lives in `_era()` and `_inner_zip_date_str()`. | |
| - **Rolling archive** — AEMO removes old monthly bundles. `DATA_START_DATE` | |
| and `DISPATCH_START_DATE` must be updated when bundles roll off. As of | |
| 2026-05: SCADA from 29 Apr 2025, dispatch from 1 Apr 2025. Verify against | |
| the actual NEMWEB Archive directory listings, not memory. | |
| - **AEMO WAF blocks datacentre IPs** on `www.aemo.com.au` (returns 403 to | |
| the HF egress range). The Generation Information XLSX fetch is wrapped | |
| in browser-style headers AND backed by the daily-refreshed | |
| `bess_list.json` snapshot. The snapshot's CI runner is on GitHub-hosted | |
| infrastructure, which AEMO doesn't block. | |
| - **NEM market day boundary is 04:00 AEST**, not midnight. `data_processor.py` | |
| filters strictly to `[04:00 AEST D, 04:00 AEST D+1)`. Era 1 callers must | |
| request both D and D+1 calendar files to cover one market day. | |
| ## Critical API contracts | |
| - `/api/bess` returns `{states, source, fetched_at, warnings}` — | |
| callers must read `.states` for the unit list. **Breaking this shape | |
| breaks the frontend dropdown** (and any external scripts). If you must | |
| change it, update `app.js:init()` in lockstep. | |
| - `/api/data` and `/api/energy-data` return `{duid, date, total_rows, summary, data, warnings?}`. | |
| - All endpoints raise `HTTPException(404)` for "no data for this DUID/date" | |
| and `HTTPException(503)` for upstream AEMO failures. Don't change these | |
| status codes — the frontend distinguishes them. | |
| ## Running locally | |
| ```bash | |
| pip install -r requirements.txt | |
| uvicorn app.main:app --reload --port 8000 | |
| # then open http://localhost:8000 | |
| ``` | |
| ## Things that have bitten us before | |
| - `bess_list.json` once contained fictional DUIDs (e.g. `NESBESS1`, | |
| `HORNSDALE_PWR1`) that don't exist in real SCADA data. The daily | |
| workflow guarantees this can't happen now, but verify any manual edit | |
| against `scripts/refresh_bess_list.py` output. | |
| - Browser cache served old `app.js` with new `/api/bess` shape → | |
| nonsense state-dropdown options. The `__VERSION__` cache-bust prevents | |
| recurrence; do not remove it. | |
| - HuggingFace rejects pushed commits with binary files anywhere in | |
| history, not just current tree. Don't reintroduce binary commits even | |
| if you plan to delete them in the next commit. | |
| --- | |
| CLAUDE.md skill to append to any coding project | |
| # CLAUDE.md — 12-rule template | |
| These rules apply to every task in this project unless explicitly overridden. | |
| Bias: caution over speed on non-trivial work. Use judgment on trivial tasks. | |
| ## Rule 1 — Think Before Coding | |
| State assumptions explicitly. If uncertain, ask rather than guess. | |
| Present multiple interpretations when ambiguity exists. | |
| Push back when a simpler approach exists. | |
| Stop when confused. Name what's unclear. | |
| ## Rule 2 — Simplicity First | |
| Minimum code that solves the problem. Nothing speculative. | |
| No features beyond what was asked. No abstractions for single-use code. | |
| Test: would a senior engineer say this is overcomplicated? If yes, simplify. | |
| ## Rule 3 — Surgical Changes | |
| Touch only what you must. Clean up only your own mess. | |
| Don't "improve" adjacent code, comments, or formatting. | |
| Don't refactor what isn't broken. Match existing style. | |
| ## Rule 4 — Goal-Driven Execution | |
| Define success criteria. Loop until verified. | |
| Don't follow steps. Define success and iterate. | |
| Strong success criteria let you loop independently. | |
| ## Rule 5 — Use the model only for judgment calls | |
| Use me for: classification, drafting, summarization, extraction. | |
| Do NOT use me for: routing, retries, deterministic transforms. | |
| If code can answer, code answers. | |
| ## Rule 6 — Token budgets are not advisory | |
| Per-task: 4,000 tokens. Per-session: 30,000 tokens. | |
| If approaching budget, summarize and start fresh. | |
| Surface the breach. Do not silently overrun. | |
| ## Rule 7 — Surface conflicts, don't average them | |
| If two patterns contradict, pick one (more recent / more tested). | |
| Explain why. Flag the other for cleanup. | |
| Don't blend conflicting patterns. | |
| ## Rule 8 — Read before you write | |
| Before adding code, read exports, immediate callers, shared utilities. | |
| "Looks orthogonal" is dangerous. If unsure why code is structured a way, ask. | |
| ## Rule 9 — Tests verify intent, not just behavior | |
| Tests must encode WHY behavior matters, not just WHAT it does. | |
| A test that can't fail when business logic changes is wrong. | |
| ## Rule 10 — Checkpoint after every significant step | |
| Summarize what was done, what's verified, what's left. | |
| Don't continue from a state you can't describe back. | |
| If you lose track, stop and restate. | |
| ## Rule 11 — Match the codebase's conventions, even if you disagree | |
| Conformance > taste inside the codebase. | |
| If you genuinely think a convention is harmful, surface it. Don't fork silently. | |
| ## Rule 12 — Fail loud | |
| "Completed" is wrong if anything was skipped silently. | |
| "Tests pass" is wrong if any were skipped. | |
| Default to surfacing uncertainty, not hiding it. | |