Update logbook: Repro - U-Cast: A Surprisingly Simple and Efficient Frontier Probabilistic AI Weather Forecaster

logbook.json

@@ -10,7 +10,7 @@
     "icml2026-repro",
     "paper-XF0wkyEbuM"
   ],
-  "updated_at": "2026-07-16T06:34:48+00:00",
+  "updated_at": "2026-07-16T06:57:42+00:00",
   "root": {
     "slug": "index",
     "title": "Repro - U-Cast: A Surprisingly Simple and Efficient Frontier Probabilistic AI Weather Forecaster",
@@ -33,9 +33,15 @@
         "title": "Conclusion",
         "file": "pages/conclusion/page.md",
         "children": []
+      },
+      {
+        "slug": "claim-3-real-era5-forecasting-performance",
+        "title": "Claim 3: Real ERA5 forecasting performance",
+        "file": "pages/claim-3-real-era5-forecasting-performance/page.md",
+        "children": []
       }
     ]
   },
-  "agent_view_tokens": 1266,
-  "revision": "1784183688646745000"
+  "agent_view_tokens": 1836,
+  "revision": "1784185062598282000"
 }

pages/claim-3-real-era5-forecasting-performance/page.md

@@ -0,0 +1,21 @@
+# Claim 3: Real ERA5 forecasting performance
+
+
+---
+<!-- trackio-cell
+{"type": "markdown", "id": "cell_a9b8833dd482", "created_at": "2026-07-16T06:57:40+00:00", "title": "This addresses the paper's actual performance claim, not just the architecture.…"}
+-->
+This addresses the paper's actual performance claim, not just the architecture. I ran U-Cast's own standalone inference script on a Modal GPU with the released checkpoint (salv47/u-cast/ucast.ckpt), streaming real ERA5 from the public WeatherBench2 store, doing autoregressive 5-member ensemble rollouts from a 2020-01-01 initial condition, and scoring with the paper's area-weighted CRPS and ensemble-mean RMSE.
+
+| lead time | CRPS | RMSE |
+|---|---|---|
+| 12h | 0.292 | 0.618 |
+| 24h | 0.355 | 0.765 |
+| 48h | 0.475 | 1.053 |
+| 72h | 0.615 | 1.391 |
+| 96h | 0.788 | 1.893 |
+| 120h | 1.029 | 2.570 |
+
+The model runs end to end and produces real probabilistic forecasts. Both scores grow smoothly and monotonically with lead time, from CRPS 0.29 at 12h to 1.03 at 5 days, which is the physically expected behavior of a skillful forecaster (near-term forecasts are sharp, uncertainty widens with horizon). So the released U-Cast checkpoint genuinely works as a probabilistic weather forecaster.
+
+Honest scope: these are area-weighted scores in the model's normalized variable space, for a single initial-condition date with a 5-member ensemble. Establishing the full 'frontier' claim would mean matching the paper's exact WeatherBench2 protocol and comparing head-to-head against GenCast and IFS across many dates and variables. I verified that the model forecasts with sensible, monotonic skill; I did not run the competing systems, so I am not asserting the exact frontier ranking. Ran on Modal; raw output in ucast_inference_results.json in the bundle.

pages/index.md

@@ -7,3 +7,4 @@
 | [Claim 1: Surprisingly simple architecture (stock diffusion UNet)](#/claim-1-surprisingly-simple-architecture-stock-diffusion-unet) |
 | [Claim 2: Model size from the released checkpoint](#/claim-2-model-size-from-the-released-checkpoint) |
 | [Conclusion](#/conclusion) |
+| [Claim 3: Real ERA5 forecasting performance](#/claim-3-real-era5-forecasting-performance) |