Update integration handoff and CONUS retraining notes
Browse files- docs/hugh_handoff_status.md +49 -0
docs/hugh_handoff_status.md
ADDED
|
@@ -0,0 +1,49 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Integration Handoff Status
|
| 2 |
+
|
| 3 |
+
This note answers the implementation questions raised during web/inference
|
| 4 |
+
integration and points to the public files that should be used by downstream
|
| 5 |
+
clients. It is intentionally scoped to the released FireWx-FM repository.
|
| 6 |
+
|
| 7 |
+
## What is fully specified now
|
| 8 |
+
|
| 9 |
+
| Question | Status | Where to look |
|
| 10 |
+
|---|---|---|
|
| 11 |
+
| Which input channel maps to which dataset? | Specified in human-readable and machine-readable form. Channel order matters. | `data_sources/DATA_SOURCES.md`, `models/wildfire_fm/input_channels.json` |
|
| 12 |
+
| Is value normalization applied? | Released California checkpoints use native source units with no mean/std or min/max normalization. The training pipeline now supports optional train-split z-score normalization for retraining. | `models/wildfire_fm/input_channels.json`, `training/train_cold_tiled_mainline.py` |
|
| 13 |
+
| Which CAPE variable is used? | HRRR surface instantaneous CAPE, selected with `typeOfLevel=surface` and `stepType=instant`; not most-unstable or layer CAPE. | `models/wildfire_fm/input_channels.json`, `data_sources/DATA_SOURCES.md` |
|
| 14 |
+
| What do the validity masks mean? | `firewx_valid` is a cache-level dynamic input-presence mask and is 1.0 everywhere in the released California cache. `static_valid` is the fraction of four static layers valid after reprojection. | `models/wildfire_fm/README.md`, `models/wildfire_fm/input_channels.json` |
|
| 15 |
+
| How are static 30 m rasters resampled to the 5 km grid? | LANDFIRE fuel and canopy use nearest-neighbor resampling; WRC housing density and LandScan population use bilinear resampling. | `training/configs/stage1_cache_regional_hrrr_ca_5km_l12_template.json`, `training/build_phase1_cache_regional_hrrr.py` |
|
| 16 |
+
| Can outputs be shown at county or sub-county granularity? | Yes, by aggregating native 5 km grid probabilities after inference. This does not require changing the checkpoint. | `spatial_serving/README.md`, `spatial_serving/grid_to_polygons.py` |
|
| 17 |
+
| Can HRRR be downloaded automatically? | Yes. A direct NOAA HRRR public-archive downloader is included and has been smoke-tested with an `.idx` sidecar download. | `data_downloader/hrrr_downloader.py`, `data_downloader/README.md` |
|
| 18 |
+
| Is the released checkpoint nationwide-trained? | No. The public checkpoint is California regional. CONUS cache and training templates are included, but no CONUS checkpoint is released yet. | `models/wildfire_fm/training_scope.json`, `training/NATIONWIDE_RETRAINING.md` |
|
| 19 |
+
|
| 20 |
+
## Current model-serving boundary
|
| 21 |
+
|
| 22 |
+
The released checkpoints are the paper-aligned California regional checkpoints.
|
| 23 |
+
They are valid for reproducing the reported regional model behavior and for
|
| 24 |
+
inspecting the tensor contract. They should not be described as nationwide
|
| 25 |
+
weights.
|
| 26 |
+
|
| 27 |
+
For gridded inference, use full-map inference when possible. If tiled inference
|
| 28 |
+
is required, use overlapping tiles with halo cropping or blending. Independent
|
| 29 |
+
non-overlapping 32-by-32 tiles are not the intended serving mode because they can
|
| 30 |
+
introduce tile-position and edge artifacts. The helper
|
| 31 |
+
`models/wildfire_fm/tiled_inference.py` implements the overlap/halo pattern.
|
| 32 |
+
|
| 33 |
+
For CONUS or client-facing deployment, the current path is:
|
| 34 |
+
|
| 35 |
+
1. Build a local CONUS cache from provider-hosted HRRR, FIRMS, LANDFIRE, WRC,
|
| 36 |
+
and LandScan inputs.
|
| 37 |
+
2. Retrain using the CONUS template with random-containing positive tile
|
| 38 |
+
placement and optional continuous-channel z-score normalization.
|
| 39 |
+
3. Serve native-grid probabilities with full-map or overlap/halo inference.
|
| 40 |
+
4. Aggregate native-grid probabilities to county or sub-county polygons using
|
| 41 |
+
the spatial serving adapter.
|
| 42 |
+
|
| 43 |
+
## What is not being uploaded as solved
|
| 44 |
+
|
| 45 |
+
A preliminary CONUS normalized/jittered retraining run completed, but seed-level
|
| 46 |
+
behavior was not stable enough to release as a corrected nationwide checkpoint.
|
| 47 |
+
For that reason, the repository exposes the corrected training and serving
|
| 48 |
+
hooks, but it does not publish a new CONUS checkpoint or claim that the CONUS
|
| 49 |
+
serving behavior is fully resolved.
|