File size: 2,099 Bytes
939693f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 | # Split Design For FireWx-FM Refinement
This document defines the split logic needed to make FireWx-FM harder to
criticize as a regional or autocorrelated supervised model.
## Required Split Families
### 1. Chronological CONUS Split
Purpose: test future-season performance across the Lower 48.
Recommended default:
- Train: 2021-2024 fire seasons.
- Validation: 2025 fire season.
- Test: 2026 fire season or latest available held-out year.
Config support:
- Use `split_years` in `stage1_cache_conus_multiyear_template.json`.
### 2. Region-Disjoint Split
Purpose: test whether the model transfers beyond the regions it saw during
training.
Recommended holdout regions:
- California / Pacific Coast.
- Pacific Northwest.
- Rockies / Intermountain West.
- Southeast including Florida.
- Great Plains / Central.
Implementation note:
- Masked evaluation can be done with trained full-CONUS checkpoints.
- Strict region-disjoint training requires region masks inside the training tile
sampler so positive and negative crops from the held-out region are excluded.
### 3. Spatial Block Split
Purpose: reduce local grid-cell autocorrelation.
Recommended design:
- Divide the Lower 48 grid into large projected blocks.
- Hold out non-adjacent blocks for testing.
- Add buffer blocks between train and test when data volume permits.
### 4. Seasonal Robustness Split
Purpose: test whether the model only fits one seasonal window.
Recommended designs:
- Train early season and test late season.
- Train excluding October-like months and test October-like months.
- Report monthly positive rate and metric curves.
## Reporting Requirements
Every experiment should save:
- Config path and SHA-256.
- Cache manifest path and SHA-256.
- Split rule name.
- Number of samples per split.
- Number of positive cells per split.
- Total FIRMS fire points per split.
- Date range per split.
- Region mask used, if any.
## Claim Boundary
Do not claim broad foundation-model generalization unless both cross-year and
region-disjoint tests are reported. A chronological split alone is not enough.
|