| # Split Design For FireWx-FM Refinement |
|
|
| This document defines the split logic needed to make FireWx-FM harder to |
| criticize as a regional or autocorrelated supervised model. |
|
|
| ## Required Split Families |
|
|
| ### 1. Chronological CONUS Split |
|
|
| Purpose: test future-season performance across the Lower 48. |
|
|
| Recommended default: |
|
|
| - Train: 2021-2024 fire seasons. |
| - Validation: 2025 fire season. |
| - Test: 2026 fire season or latest available held-out year. |
|
|
| Config support: |
|
|
| - Use `split_years` in `stage1_cache_conus_multiyear_template.json`. |
|
|
| ### 2. Region-Disjoint Split |
|
|
| Purpose: test whether the model transfers beyond the regions it saw during |
| training. |
|
|
| Recommended holdout regions: |
|
|
| - California / Pacific Coast. |
| - Pacific Northwest. |
| - Rockies / Intermountain West. |
| - Southeast including Florida. |
| - Great Plains / Central. |
|
|
| Implementation note: |
|
|
| - Masked evaluation can be done with trained full-CONUS checkpoints. |
| - Strict region-disjoint training requires region masks inside the training tile |
| sampler so positive and negative crops from the held-out region are excluded. |
|
|
| ### 3. Spatial Block Split |
|
|
| Purpose: reduce local grid-cell autocorrelation. |
|
|
| Recommended design: |
|
|
| - Divide the Lower 48 grid into large projected blocks. |
| - Hold out non-adjacent blocks for testing. |
| - Add buffer blocks between train and test when data volume permits. |
|
|
| ### 4. Seasonal Robustness Split |
|
|
| Purpose: test whether the model only fits one seasonal window. |
|
|
| Recommended designs: |
|
|
| - Train early season and test late season. |
| - Train excluding October-like months and test October-like months. |
| - Report monthly positive rate and metric curves. |
|
|
| ## Reporting Requirements |
|
|
| Every experiment should save: |
|
|
| - Config path and SHA-256. |
| - Cache manifest path and SHA-256. |
| - Split rule name. |
| - Number of samples per split. |
| - Number of positive cells per split. |
| - Total FIRMS fire points per split. |
| - Date range per split. |
| - Region mask used, if any. |
|
|
| ## Claim Boundary |
|
|
| Do not claim broad foundation-model generalization unless both cross-year and |
| region-disjoint tests are reported. A chronological split alone is not enough. |
|
|
