Split Design For FireWx-FM Refinement

This document defines the split logic needed to make FireWx-FM harder to criticize as a regional or autocorrelated supervised model.

Required Split Families

1. Chronological CONUS Split

Purpose: test future-season performance across the Lower 48.

Recommended default:

Train: 2021-2024 fire seasons.
Validation: 2025 fire season.
Test: 2026 fire season or latest available held-out year.

Config support:

Use split_years in stage1_cache_conus_multiyear_template.json.

2. Region-Disjoint Split

Purpose: test whether the model transfers beyond the regions it saw during training.

Recommended holdout regions:

California / Pacific Coast.
Pacific Northwest.
Rockies / Intermountain West.
Southeast including Florida.
Great Plains / Central.

Implementation note:

Masked evaluation can be done with trained full-CONUS checkpoints.
Strict region-disjoint training requires region masks inside the training tile sampler so positive and negative crops from the held-out region are excluded.

3. Spatial Block Split

Purpose: reduce local grid-cell autocorrelation.

Recommended design:

Divide the Lower 48 grid into large projected blocks.
Hold out non-adjacent blocks for testing.
Add buffer blocks between train and test when data volume permits.

4. Seasonal Robustness Split

Purpose: test whether the model only fits one seasonal window.

Recommended designs:

Train early season and test late season.
Train excluding October-like months and test October-like months.
Report monthly positive rate and metric curves.

Reporting Requirements

Every experiment should save:

Config path and SHA-256.
Cache manifest path and SHA-256.
Split rule name.
Number of samples per split.
Number of positive cells per split.
Total FIRMS fire points per split.
Date range per split.
Region mask used, if any.

Claim Boundary

Do not claim broad foundation-model generalization unless both cross-year and region-disjoint tests are reported. A chronological split alone is not enough.