Sync from GitHub 1e278fe

README.md

@@ -17,14 +17,14 @@ benchmarked against the **EIA's own published day-ahead forecast**.
 
 | Model | MAPE | vs EIA |
 |---|---|---|
-| **LightGBM hybrid** (+ EIA forecast as input) | **2.771%** | **+25.0%** |
-| **LightGBM** (global, from first principles) | 3.676% | +0.6% |
-| _EIA official forecast_ | 3.696% | benchmark |
-| Seasonal naive (24h) | 5.677% | -53.6% |
-| Weekly naive (168h) | 9.117% | -146.7% |
+| **LightGBM hybrid** (+ EIA forecast as input) | **2.797%** | **+24.1%** |
+| **LightGBM** (global, weather and calendar only) | 3.683% | +0.1% |
+| _EIA official forecast_ | 3.686% | benchmark |
+| Seasonal naive (24h) | 5.657% | -53.5% |
+| Weekly naive (168h) | 9.328% | -153.1% |
 
-Measured over 25,927 out-of-sample hours across 12 balancing authorities, using a
-strictly chronological split.
+Measured over 25,925 out-of-sample hours across 12 balancing authorities, split by
+date so the model never sees the test window.
 
 ## What you can do here
 

app.py

@@ -645,20 +645,10 @@ with tabs[6]:
 
     st.markdown(
         """
-### The problem
-
-Electricity cannot really be stored at grid scale, so the companies that run the
-grid have to decide today how much power to generate tomorrow. Guess too high and
-they burn fuel making electricity nobody uses. Guess too low and they have to buy
-the shortfall at emergency prices, or cut power to customers. On a large grid,
-being off by one percent costs millions of dollars a year.
-
-### What I compare against
-
-The EIA publishes each region's own day-ahead forecast next to what actually
-happened. So instead of making up an easy baseline, every model here is scored
-against the forecast grid operators really published and really used, which means
-anyone can check whether these results hold up.
+Grid operators decide today how much power to generate tomorrow, so a day-ahead
+demand forecast has real money attached to it. The EIA publishes each region's own
+day-ahead forecast next to what actually happened, so every model here is scored
+against that instead of a baseline I made up.
 
 ### The pipeline
 """
@@ -695,28 +685,15 @@ Open-Meteo      --+       |
 
     st.markdown(
         """
-### Why I built it this way
-
-Why DuckDB and not Postgres or Spark. It handles hundreds of millions of rows
-inside a single file with no server to run. I built this on a laptop with 8 GB of
-RAM, and that was the difference between a pipeline that finishes and one that runs
-out of memory. The SQL would still run on Snowflake or BigQuery unchanged.
-
-Why one model for all 12 regions. The regions behave similarly, so training
-together lets the bigger ones help the smaller ones. It also means one model file
-to deploy and monitor instead of twelve.
-
-Why I flag bad data instead of deleting it. A meter stuck on the same value is
-proof that the meter broke. Quietly dropping that row also drops the only record
-that anything went wrong.
-
-Why I only split by date. If you split time-series data randomly, rows from the
-future sit next to rows from the past and the model effectively sees answers it
-should not have. The scores look great and mean nothing.
-
-Why using tomorrow's weather is not cheating. A real grid operator also has
-tomorrow's weather forecast and knows what day of the week it is. Leaving it out
-would mean solving a harder problem than the real one.
+### Some choices
+
+- One LightGBM model across all 12 regions, with the region code as a categorical
+  feature, so the bigger regions help the smaller ones and there is one model file
+  to deploy instead of twelve.
+- Bad readings are flagged, not deleted, so a broken meter leaves evidence behind.
+- Every split is by date. Splitting time series randomly puts future rows next to
+  past ones and the scores stop meaning anything.
+- The models use tomorrow's weather forecast, which a real grid operator also has.
 
 ### What it is built with
 

deploy/README_SPACE.md

@@ -17,14 +17,14 @@ benchmarked against the **EIA's own published day-ahead forecast**.
 
 | Model | MAPE | vs EIA |
 |---|---|---|
-| **LightGBM hybrid** (+ EIA forecast as input) | **2.771%** | **+25.0%** |
-| **LightGBM** (global, from first principles) | 3.676% | +0.6% |
-| _EIA official forecast_ | 3.696% | benchmark |
-| Seasonal naive (24h) | 5.677% | -53.6% |
-| Weekly naive (168h) | 9.117% | -146.7% |
+| **LightGBM hybrid** (+ EIA forecast as input) | **2.797%** | **+24.1%** |
+| **LightGBM** (global, weather and calendar only) | 3.683% | +0.1% |
+| _EIA official forecast_ | 3.686% | benchmark |
+| Seasonal naive (24h) | 5.657% | -53.5% |
+| Weekly naive (168h) | 9.328% | -153.1% |
 
-Measured over 25,927 out-of-sample hours across 12 balancing authorities, using a
-strictly chronological split.
+Measured over 25,925 out-of-sample hours across 12 balancing authorities, split by
+date so the model never sees the test window.
 
 ## What you can do here
 

scripts/update_readme.py

@@ -18,7 +18,7 @@ PRETTY = {
     "lstm": "**LSTM** encoder",
     "transformer": "**Transformer** encoder",
     "ensemble": "**Ensemble** (GBM + LSTM)",
-    "eia_official": "_EIA official forecast_ ",
+    "eia_official": "_EIA official forecast_",
     "seasonal_naive": "Seasonal naive (24h)",
     "weekly_naive": "Weekly naive (168h)",
     "drift_naive": "Drift naive",
@@ -73,9 +73,8 @@ def main() -> int:
                 f"**{head['test_observations']:,}** out-of-sample hours across 12 balancing "
                 f"authorities.\n"
                 f">\n"
-                f"> Trained without ever seeing the test window. The EIA benchmark is the "
-                f"forecast the US government actually published and grid operators actually "
-                f"operated against.\n\n" + table
+                f"> The model never sees the test window during training, and the EIA "
+                f"benchmark is the forecast the US government actually published.\n\n" + table
             )
 
     table += (

src/gridpulse/features/build.py

@@ -44,9 +44,6 @@ def load_modelling_frame(ba_codes: list[str] | None = None) -> pd.DataFrame:
 
     frame = query(f"""
         SELECT period_utc, ba_code, date_local, hour_local,
-               -- The warehouse is the single source of truth for what counts as a
-               -- trustworthy reading. Modelling consumes demand_clean_mwh so the
-               -- cleaning rules live in one place rather than being reimplemented here.
                demand_clean_mwh AS demand_mwh, demand_forecast_mwh,
                temperature_2m, apparent_temperature, relative_humidity_2m,
                dew_point_2m, cloud_cover, wind_speed_10m, shortwave_radiation,

src/gridpulse/warehouse/build.py

@@ -173,7 +173,6 @@ def build_warehouse(rebuild: bool = False) -> dict[str, int]:
             e.total_interchange_mwh,
             {weather_select},
 
-            -- Data quality flags. Diagnosis, not deletion.
             (e.demand_mwh IS NULL)                                   AS flag_missing_demand,
             (e.demand_mwh IS NOT NULL
              AND e.demand_mwh < {MIN_PLAUSIBLE_MWH})                 AS flag_nonpositive_demand,
@@ -239,8 +238,6 @@ def build_warehouse(rebuild: bool = False) -> dict[str, int]:
             s.total_interchange_mwh,
             {", ".join("s." + v for v in WEATHER_VARIABLES)},
 
-            -- Interpolated demand: a modelling-ready series with short gaps bridged.
-            -- The raw column is retained untouched alongside it.
             CASE WHEN s.demand_mwh IS NOT NULL
                       AND NOT s.flag_nonpositive_demand
                       AND s.demand_mwh BETWEEN b.lower_bound AND b.upper_bound
@@ -258,13 +255,10 @@ def build_warehouse(rebuild: bool = False) -> dict[str, int]:
                 / nullif(lag(s.demand_mwh) OVER w, 0) * 100
                 > {MAX_HOURLY_RAMP_PCT})                                  AS flag_extreme_ramp,
 
-            -- Physically impossible magnitude. Flagged, never deleted: the reading
-            -- is the evidence that upstream telemetry failed.
             (s.demand_mwh IS NOT NULL
              AND (s.demand_mwh < b.lower_bound
                   OR s.demand_mwh > b.upper_bound))                       AS flag_implausible_magnitude,
 
-            -- Isolated excursion: departs sharply from the local 5-hour median.
             (s.demand_mwh IS NOT NULL
              AND n.local_median IS NOT NULL
              AND abs(s.demand_mwh - n.local_median)