Spaces:
Running
Running
| # BESS SCADA Data Web App — Implementation Plan | |
| ## Architecture Overview | |
| **Hosting:** HuggingFace Spaces (Docker SDK) | |
| **Backend:** FastAPI (Python) | |
| **Frontend:** HTML/CSS/JS served by FastAPI (lightweight, no React build step needed) | |
| **Data Processing:** Polars (memory-efficient, faster than pandas for large CSVs) | |
| **Charts:** Plotly.js (interactive, zoom/pan, rendered client-side) | |
| **Analytics:** Lightweight SQLite logging (ephemeral but rebuilt; optionally push to free Supabase) | |
| **BESS List:** Static JSON file in repo, manually updated monthly | |
| ## Data Flow | |
| ``` | |
| Student selects: State → BESS → Date | |
| ↓ | |
| Frontend sends request to FastAPI: GET /api/data?duid=XXXX&date=2026-03-01 | |
| ↓ | |
| Backend constructs AEMO URL: | |
| - Current: https://www.nemweb.com.au/REPORTS/Current/FPPDAILY/ | |
| - Archive: https://nemweb.com.au/Reports/Archive/FPPDAILY/ | |
| ↓ | |
| Backend downloads ZIP from AEMO (~20-50 MB compressed) | |
| ↓ | |
| Backend extracts CSV in-memory (streaming, never full file to disk) | |
| ↓ | |
| Backend filters rows: FPP_UNITID == selected DUID | |
| Result: ~21,600 rows (~1-2 MB) for one BESS for one day | |
| ↓ | |
| Backend returns JSON (for chart/table display) or CSV/Parquet (for download) | |
| ↓ | |
| Frontend renders interactive chart + paginated table | |
| ``` | |
| ## Project Structure | |
| ``` | |
| BESS-SCADA-Data/ | |
| ├── Dockerfile # HuggingFace Spaces Docker config | |
| ├── requirements.txt # Python dependencies | |
| ├── README.md # HuggingFace Spaces metadata | |
| ├── app/ | |
| │ ├── main.py # FastAPI application entry point | |
| │ ├── config.py # Constants, URLs, settings | |
| │ ├── data/ | |
| │ │ ├── bess_list.json # Static BESS list by state (updated monthly) | |
| │ │ └── quality_flags.json # MW_QUALITY_FLAG descriptions | |
| │ ├── services/ | |
| │ │ ├── aemo_fetcher.py # Download & extract ZIP from AEMO | |
| │ │ ├── data_processor.py # Filter, transform CSV data using Polars | |
| │ │ └── analytics.py # Log requests (IP, timestamp, BESS, date) | |
| │ ├── routers/ | |
| │ │ ├── api.py # REST API endpoints | |
| │ │ └── pages.py # Serve HTML pages | |
| │ └── static/ | |
| │ ├── index.html # Main single-page app | |
| │ ├── css/ | |
| │ │ └── style.css # Styling | |
| │ └── js/ | |
| │ └── app.js # Frontend logic (fetch, render chart/table) | |
| ``` | |
| ## Implementation Steps | |
| ### Step 1: Project Setup & Docker Configuration | |
| - Initialize project structure | |
| - Create `Dockerfile` for HuggingFace Spaces (Python 3.11 slim, expose port 7860) | |
| - Create `requirements.txt`: fastapi, uvicorn, polars, httpx, pyarrow | |
| - Create HuggingFace Spaces `README.md` with metadata (sdk: docker) | |
| ### Step 2: Static BESS List | |
| - Parse the AEMO NEM Generation Information Excel manually | |
| - Create `bess_list.json` with structure: | |
| ```json | |
| { | |
| "NSW": [ | |
| {"duid": "BESS1", "name": "Battery Name", "capacity_mw": 100, "region": "NSW1"} | |
| ], | |
| "VIC": [...], | |
| "QLD": [...], | |
| "SA": [...], | |
| "TAS": [...] | |
| } | |
| ``` | |
| - Create `quality_flags.json`: | |
| ```json | |
| { | |
| "0": {"label": "Good", "description": "Good quality data", "color": "green"}, | |
| "1": {"label": "Bad", "description": "Sustained communication failure or manual override", "color": "red"}, | |
| "-1": {"label": "N/A", "description": "Not applicable", "color": "gray"} | |
| } | |
| ``` | |
| ### Step 3: AEMO Data Fetcher Service | |
| - `aemo_fetcher.py`: | |
| - Given a date, construct the correct AEMO URL (Current vs Archive) | |
| - Download ZIP using `httpx` with 30-second timeout | |
| - Extract CSV from ZIP in-memory using `zipfile` module | |
| - Return raw CSV bytes/stream | |
| - Error handling: timeout → retry once → user-friendly error | |
| - Need to investigate the exact filename pattern in the ZIP to construct URLs correctly | |
| ### Step 4: Data Processor Service | |
| - `data_processor.py`: | |
| - Read CSV bytes with Polars (lazy mode for memory efficiency) | |
| - Filter by `FPP_UNITID == selected_duid` | |
| - Select only relevant columns: INTERVAL_DATETIME, MEASUREMENT_DATETIME, FPP_UNITID, MEASURED_MW, MW_QUALITY_FLAG | |
| - Return as: | |
| - Polars DataFrame (for JSON serialization to frontend) | |
| - CSV bytes (for CSV download) | |
| - Parquet bytes (for Parquet download) | |
| - Compute summary statistics: min, max, mean MW, % good quality, % bad quality | |
| ### Step 5: Analytics Service | |
| - `analytics.py`: | |
| - On each data request, log to SQLite (`/tmp/analytics.db`): | |
| - Timestamp | |
| - IP address (from request headers, respecting X-Forwarded-For) | |
| - Selected BESS (DUID) | |
| - Selected date | |
| - Download format (view/csv/parquet) | |
| - Provide endpoint to view analytics (protected by simple query param token) | |
| - Note: SQLite on ephemeral storage will be lost on restart — acceptable for basic tracking. | |
| Could optionally push to free Supabase (500 MB free) for persistence. | |
| ### Step 6: FastAPI Backend Endpoints | |
| - `GET /` — Serve main HTML page | |
| - `GET /api/bess` — Return BESS list grouped by state | |
| - `GET /api/data?duid={duid}&date={YYYY-MM-DD}` — Return filtered SCADA data as JSON | |
| - Response includes: data rows, summary stats, quality flag breakdown | |
| - `GET /api/download/csv?duid={duid}&date={YYYY-MM-DD}` — Download as CSV | |
| - `GET /api/download/parquet?duid={duid}&date={YYYY-MM-DD}` — Download as Parquet | |
| - `GET /api/analytics?token={secret}` — View usage analytics (admin only) | |
| ### Step 7: Frontend (Single Page App) | |
| - **State selector:** Dropdown for Australian state (NSW, VIC, QLD, SA, TAS) | |
| - **BESS selector:** Populated dynamically based on selected state | |
| - **Date picker:** Single date (max 1 day), with validation (not future dates, not before data availability) | |
| - **Load button:** Fetches data from backend | |
| - **Loading state:** Spinner with message "Fetching data from AEMO... this may take 10-15 seconds" | |
| - **Chart:** Plotly.js time-series chart of MEASURED_MW vs MEASUREMENT_DATETIME | |
| - Color-coded by MW_QUALITY_FLAG (green=good, red=bad, gray=N/A) | |
| - Interactive zoom, pan, hover tooltips | |
| - **Data quality banner:** | |
| - "X% of measurements are good quality, Y% flagged as bad quality (communication failure or manual override)" | |
| - **Data table:** Paginated table showing all rows (virtual scrolling for performance) | |
| - **Download buttons:** CSV and Parquet download buttons | |
| - **Summary stats panel:** Min, Max, Mean, Std Dev of MEASURED_MW | |
| ### Step 8: Error Handling & UX | |
| - AEMO server unavailable → "AEMO data source is temporarily unavailable. Please try again in a few minutes." | |
| - No data for selected BESS/date → "No SCADA data found for [BESS name] on [date]. This unit may not have been operational on this date." | |
| - Large response handling → Stream response, show progress | |
| - Rate limiting → Simple in-memory rate limit (10 requests/minute per IP) to prevent abuse | |
| ### Step 9: Deployment to HuggingFace Spaces | |
| - Create HuggingFace Space with Docker SDK | |
| - Push code to HuggingFace repo | |
| - Configure Space settings | |
| - Test with sample BESS queries | |
| ### Step 10: Documentation & BESS List Maintenance | |
| - Document how to update `bess_list.json` when AEMO publishes new generation info | |
| - Provide a helper script to parse the AEMO Excel and generate the JSON | |
| ## Key Technical Decisions | |
| | Decision | Choice | Rationale | | |
| |----------|--------|-----------| | |
| | Polars over Pandas | Polars | 2-10x faster, much lower memory usage, lazy evaluation | | |
| | FastAPI over Flask | FastAPI | Async support (critical for downloading from AEMO), auto-docs, modern | | |
| | Plotly.js over server-side charts | Plotly.js | Client-side rendering reduces server load, interactive | | |
| | Single HTML page over React | Vanilla JS | No build step, simpler deployment, fewer dependencies | | |
| | Parquet for download | PyArrow | Parquet is 5-10x smaller than CSV, native to Python data science workflows | | |
| | httpx over requests | httpx | Async support, connection pooling, timeout handling | | |
| ## Risks & Mitigations | |
| | Risk | Mitigation | | |
| |------|------------| | |
| | AEMO changes URL structure | Configurable URLs in `config.py`, easy to update | | |
| | HF Space sleeps after inactivity | Acceptable for academic use; wakes in ~30 seconds | | |
| | 16 GB RAM exceeded | On-demand fetch + Polars lazy mode keeps memory under 500 MB per request | | |
| | AEMO rate-limits our requests | Cache recently fetched data in `/tmp` (ephemeral but helps during active sessions) | | |
| | BESS list becomes stale | Monthly manual update process documented; helper script provided | | |