ablation.md

Rock-Glacier Segmentation — Full Ablation & Development Report (Exhaustive)

Task: Binary semantic segmentation of rock glaciers, Sikkim Himalaya (Eastern Himalaya, India) Input: 10 m multi-modal satellite features (Sentinel-2 MSI + Copernicus GLO-30 DEM + Sentinel-1 SAR) Output: Per-pixel binary mask (1 = rock glacier, 0 = background) Reference inventory: 626 active rock glaciers, mapped by SBAS-InSAR time-series (2022–2024) with STL seasonal filtering and Google Earth Pro optical refinement — Kumar, Malviya & Singh (2026), iScience 29, 115983. Study extent: ≈ 88.07°–88.95° E, 27.32°–28.08° N; rock glaciers predominantly above 4,500 m. Final model: 3-seed Swin-UNet ensemble with 8-way test-time augmentation (TTA). Headline test result (held-out, 101 patches): Dice 0.668, IoU 0.501, Precision 0.609, Recall 0.740, operating threshold 0.30. Cross-validated result (5-fold): Dice 0.583 ± 0.014.

---

Table of Contents

1. Executive summary
2. Background & the 30 m baseline
3. The old 10 m dataset & its data-health diagnosis
4. The four training-pipeline bugs (and fixes)
5. Dead-end experiments on the old data
6. New data-collection pipeline (the decisive fix)
7. Final dataset (patches_v2)
8. Band selection (Cohen's d + 13 experiments + modality ablation)
9. Model architectures
10. Training configuration
11. First training on clean data
12. Cumulative model/training improvements
13. Architecture benchmark — validation vs honest test
14. Ensemble study & final model selection
15. 5-fold cross-validation
16. Why test (0.67) ≠ CV (0.58): a measurement decomposition
17. Negative results (everything that failed)
18. Full-journey table
19. Reproduction
20. Key learnings

---

1. Executive Summary

Migrating from a 30 m dataset to a new 10 m dataset caused a large, alarming performance regression (validation Dice collapsed from an apparent 0.76 to ~0.43–0.53). The instinct was that the model or resolution was wrong. Systematic diagnosis proved otherwise: the problem was the data and the training pipeline, not the architecture.

The path to recovery had four decisive moves, in order of impact:

1. Rebuild the data collection — compositing only cloud-free post-monsoon (Oct–Nov) Sentinel-2 with cloud-probability masking eliminated the ~20 % NaN contamination of the old all-season composites. This was the single largest accuracy gain in the entire project.
2. Fix four training-pipeline bugs — metric inflation, inverted loss weights, biased NaN handling, and a premature learning-rate scheduler.
3. Select the right features — a Cohen's d ranking plus 13 retraining experiments identified an optimal 20-band subset and revealed that terrain morphology carries ≈ 2× the discriminative power of optical reflectance.
4. Choose the right model — a controlled benchmark of four architectures against a classical Random-Forest baseline established the Swin-UNet as best, with a 3-seed ensemble + TTA as the final predictor.

The final honest held-out test Dice is 0.668 (IoU 0.501), cross-validated at 0.583 ± 0.014. The remaining ceiling is set by dataset size (671 patches; CV std of only 0.014 shows near-zero overfitting), not by the model — only more labelled data would push it higher.

---

2. Background & the 30 m Baseline

Before this work, a 30 m model existed with reported metrics val IoU 0.5825, val Dice 0.7618, Precision 0.6233, Recall 0.8990, F1 0.7362. This number was later found to be inflated by the same metric bug described in §4 (empty background-only batches scoring a spurious 1.0). Its corrected value is closer to ~0.65–0.68, meaning the true target the 10 m work had to match was lower than it first appeared. This is why the honest 10 m test Dice of 0.67 is competitive with — not far below — the old baseline.

---

3. The Old 10 m Dataset & Its Data-Health Diagnosis

The first 10 m dataset (patches_10m, 28 bands, produced by an earlier collection script) performed far worse than the 30 m baseline. A temporary data-health script (written, run, then deleted per the task brief) revealed exactly why the data was unusable:

NaN / no-data contamination (train split, 3,074 patches, 50.36 M pixels):

• 267,127,234 NaN values in total.
• Per-band NaN rate ≈ 20–21 % for almost every band (e.g. B2 20.93 %, B11 21.33 %).
• Only 2 of 28 bands were clean (bands 18 and 27 had 0 % NaN).
• 2,263 of 3,074 patches (73.6 %) contained at least one NaN pixel.
• Cause: at 4,000–5,500 m in the monsoon-affected Himalaya, an all-season 3-year median leaves many optical pixels with no clear-sky observation across the entire window.

Class imbalance:

• Glacier pixels: 2,427,198 (4.82 %); background: 47,937,218 (95.18 %).
• Background : glacier = 19.8 : 1 across the full dataset (8.9 : 1 within glacier-only patches).

Mask sparsity:

• 52.4 % of patches were pure background (all-zero masks) — half of every training batch carried no glacier gradient signal.

Additionally, a fix was applied to data_generator.py: NaN pixels were being filled with 0 before z-score normalisation, so they became (0 − μ)/σ = −μ/σ — a biased, non-zero phantom value on ~20 % of pixels. This was corrected to fill NaN → 0 after normalisation (= the band mean, a neutral value).

The check_separability.py (Cohen's d) script was also reviewed and three bugs fixed: (1) the pooled standard-deviation formula used (n−1)·pop_var instead of n·pop_var; (2) the class-imbalance count was read from band-1's valid-pixel-filtered count rather than the raw mask; (3) the conclusion text hard-coded "607 pairs" and "4.5:1" instead of computing them.

---

4. The Four Training-Pipeline Bugs (and Fixes)

Diagnosis of the nine early band-selection runs (all poor, best val Dice ≈ 0.49) uncovered four independent bugs. These — not the model — were the cause of the regression.

#	Bug	Detailed mechanism	Effect	Fix
1	Metric inflation	Every metric used additive 1e-6 smoothing: dice = (2·i + 1e-6)/(sum + 1e-6). When a validation batch had zero glacier pixels (52 % of patches were empty) and the model predicted zero glacier, this evaluated to 1e-6/1e-6 = 1.0. Keras averaged these 1.0s into the epoch metric.	val Dice inflated by roughly +0.35–0.45; ModelCheckpoint(monitor='val_dice') therefore saved the most memorising epoch, not the best-generalising one. Example: run_14_best reported val Dice 0.9006 at epoch 2 — a freshly-initialised model that happened to see an all-background batch.	Each metric rewritten with tf.cond(denom < 0.5, lambda: 0.0, lambda: real_value) — empty batches return 0.0, not 1.0.
2	Inverted Focal-Tversky weights	α_FP = 0.6 > β_FN = 0.4 penalised false positives 1.5× harder than false negatives. At 19.8:1 imbalance this taught the model "when unsure, predict background."	train Dice plateaued ~0.55; recall stuck at 0.60–0.70; systematic under-detection of glacier.	Flipped to α = 0.3 / β = 0.7 (recall-biased); later balanced to α = 0.4 / β = 0.6 for the clean dataset.
3	Biased NaN normalisation	NaN → 0 before normalisation → −μ/σ per band on ~20 % of pixels.	non-zero phantom signal misled the encoder.	fill NaN → 0 after normalisation (= band mean).
4	LR killed too early	ReduceLROnPlateau(factor=0.5, patience=5) halved the LR every 5 stagnant epochs. But the stagnation was caused by bug #2, not real convergence. By epoch 45 the LR had collapsed to 2.5e-5; by epoch 65 to 1.25e-5.	the optimiser was throttled before it could learn.	replaced by cosine annealing (§10); early-stopping patience raised.

Supporting changes: BATCH_SIZE 8 → 32 (batch 8 used only ~200 MB of a 49 GB A6000; batch 32 gave ~4× faster epochs with no accuracy loss); ModelCheckpoint monitor changed from the inflated val_dice_coef to val_loss.

Result: on the old noisy data, run_20_FIXED reached val Dice 0.5304 (up from ~0.43), proving the model was sound and the remaining gap was a data-quality problem.

4.1 The nine original band-selection runs (old noisy data, buggy code — for the record)

These are the pre-fix numbers that triggered the investigation (Attention U-Net, val Dice):

Run	Bands	Val Dice		Run	Bands	Val Dice
run_12_native	12	0.4321		run_22_best	22	0.4601
run_14_best	14	0.4465		run_24_best	24	0.3793
run_16_best	16	0.4870		run_26_best	26	0.4442
run_18_best	18	0.4472		run_28_all	28	0.4228
run_20_best	20	0.4769

All poor and erratic — the fingerprint of a broken pipeline, not a band-selection signal.

---

5. Dead-End Experiments on the Old Data (before rebuilding)

Before deciding to rebuild the data, several fixes were tried on the noisy patches_10m. Most failed:

Attempt	Result	Verdict & reason
require_glacier=True (drop the 1,611 pure-bg patches)	imbalance 19.8:1 → 8.9:1; steps/epoch ↑	kept — every gradient step now carries glacier signal
min_glacier_pct = 5 % (drop 208 boundary-sliver patches)	mixed	situational
NaN-pixel exclusion from loss (valid = 0 on NaN)	val Dice −0.036	❌ reverted — removing the FP penalty on cloud pixels made the model over-predict glacier; recall +0.07 but precision −0.09
Hard-negative mining (bg_keep_n = 300)	val Dice −0.11 (0.55 → 0.42)	❌ reverted — Focal-Tversky returns loss = 1.0 for any FP in an all-bg patch, globally suppressing glacier prediction
256×256 patches (re-extract)	underperformed	❌ confounded by the same NaN/season issues

Conclusion: the old data had a hard ceiling (20 % NaN, mixed seasons, snow). No training trick could fix it. Decision: rebuild the data collection.

---

6. New Data-Collection Pipeline (the decisive fix)

6.1 Root cause of the old data

The previous script used a 3-year, all-season median with only QA60 cloud masking. QA60 removes opaque and cirrus clouds but misses cloud shadow, thin haze, and misclassified bright snow, and the all-season window retains seasonal snow. At high altitude this left ~20 % of optical pixels as no-data.

6.2 New design decisions

• Temporal window: October–November only, 2020–2023 — post-monsoon (clouds cleared) and pre-winter (glacier still exposed, no fresh snow). This is the one clean window each year in Sikkim.
• Cloud masking: Sentinel-2 cloud-probability product (COPERNICUS/S2_CLOUD_PROBABILITY, probability < 35 %), which catches shadow/haze that QA60 misses.
• Compositing: per-pixel median of the masked scenes.
• Spectral indices computed from the composite (NDVI, NDWI, NDSI, NDMI, BSI, Tasselled-Cap Brightness, Iron-Oxide ratio B4/B2).
• Terrain: Copernicus GLO-30 DEM, reprojected to UTM 45N before computing slope/aspect/hillshade/ TPI/TRI/TWI/curvature/roughness (native-CRS terrain gives NaN — see §6.4).
• SAR: Sentinel-1 VV Oct–Nov median (weather-independent structural cue).
• Collection unit: per glacier, driven by the InSAR polygon inventory (see §6.3).

6.3 Per-glacier, polygon-driven collection (the key structural change)

Rather than exporting one global mosaic and tiling it, the pipeline (collect_patches_gee.py) is driven by the 626-polygon inventory:

1. For each polygon, compute its centroid and define a 128×128 px (1.28 km) window centred there.
2. Rasterise all polygons that fall inside that window → the binary mask (so a neighbouring glacier in the same patch is labelled glacier, not background — a bug that had poisoned earlier data).
3. Clip the 28-band feature stack for the same window via ee.data.computePixels (NumPy, direct to server — no Google Drive round-trip).
4. Skip patches with > 50 % missing optical data; retry transient GEE errors up to 3×.
5. Add ~8–10 % random background patches (locations ≥ 500 m from any glacier).

6.4 Collection bugs fixed during development (chronological)

Symptom	Root cause	Fix
gcloud not found on the headless server	EE browser auth impossible on a server	run earthengine authenticate on the laptop, scp the credentials file to ~/.config/earthengine/
TopologyException: side location conflict	self-intersecting / invalid polygons in the union	buffer(0) and drop invalid/empty geometries before unary_union
'Image' object has no attribute 'focalStd'	method doesn't exist in the EE Python API	reduceNeighborhood(ee.Reducer.stdDev(), ee.Kernel.square(...))
Image has no band named "glacier"	reduceToImage names its output "first"	.rename('glacier')
100 % missing-data patches (all skipped)	the NIR shadow mask nir > 0.15 removed legitimately dark high-altitude pixels; the Oct–Nov cloud-prob < 20 % window was too strict	removed the NIR mask, widened to Sep–Nov, relaxed cloud threshold to 35 %
Slope/Aspect/Hillshade/ProfileCurvature/TWI all-NaN	ee.Terrain.* requires a projected CRS (metre units); the DEM was in geographic degrees	reproject the DEM to EPSG:32645 (dem.reproject(crs, scale=30)) before all terrain derivatives

6.5 Verification (before training)

Quality was verified on the collected patches: 0 % optical NaN across all patches; all 28 bands valid (terrain fix confirmed — Slope/Aspect/Hillshade now populated); elevation 4,502–5,947 m; masks binary {0,1}; adjacent glaciers correctly labelled. Multi-band figures for 10+ random patches were inspected visually, plus per-band min/max/mean sanity checks (e.g. S1_VV in −18 to +1 dB, NDVI −0.2 to +0.5, elevation in the correct Himalayan range).

Old vs new NaN rate: 20 % → 0 %. This one change was the biggest driver of the entire project's accuracy improvement.

---

7. Final Dataset (patches_v2)

Split	Patches	Purpose
Train	469	model fitting
Val	101	held-out validation; band selection; threshold sweep
Test	101	held-out final evaluation (never touched in training)
Total	671	613 glacier-bearing + 58 background

• Bands (28): 10 optical (B2, B3, B4, B8, B8A, B11, B12, Red-Edge 1/2/3) + 7 indices (NDVI, NDWI, NDSI, NDMI, BSI, Brightness-TC, Iron-Oxide) + 10 terrain (Elevation, Slope, Aspect, Plan/Profile Curvature, TPI, TRI, TWI, Surface Roughness, Hillshade) + 1 SAR (S1_VV).
• Patch: 128×128 px @ 10 m = 1.28 km × 1.28 km; features float32, mask uint8.
• CRS UTM 45N (EPSG:32645); split seed = 42 (70/15/15).
• Optical NaN: 0 %. Glacier density (glacier patches): mean 17.5 %, median 14.2 %; 208 patches < 1 % (boundary slivers). Class balance within glacier patches ≈ 8.9 : 1.
• Normalisation: per-band z-score from training patches only (norm_stats.json).

---

8. Band Selection

8.1 Cohen's d separability (all 28 bands)

d = |μ_glacier − μ_background| / σ_pooled, pooled across train+val glacier/background pixels.

Rank	Band	d	Rank	Band	d	Rank	Band	d
1	Surface Roughness	0.496	11	NDMI	0.228	21	Aspect	0.077
2	Slope	0.489	12	BSI	0.212	22	B2 (Blue)	0.076
3	TPI	0.424	13	B8A	0.184	23	B3 (Green)	0.064
4	TRI	0.361	14	B8 (NIR)	0.167	24	B4 (Red)	0.051
5	NDVI	0.304	15	Red-Edge 3	0.161	25	NDSI	0.031
6	B12 (SWIR-2)	0.275	16	Hillshade	0.154	26	Iron-Oxide	0.008
7	TWI	0.261	17	Red-Edge 2	0.141	27	Plan Curvature	0.008
8	S1_VV (SAR)	0.237	18	B11 (SWIR-1)	0.123	28	Profile Curvature	0.005
9	Elevation	0.237	19	Brightness-TC	0.081
10	NDWI	0.234	20	Red-Edge 1	0.080

Observation: the four most separable bands are all terrain descriptors; SAR and Elevation also rank highly; the four least separable (NDSI, Iron-Oxide, Plan/Profile Curvature; d < 0.05) are noise.

8.2 All 13 band-configuration experiments (Attention U-Net proxy, held-out val)

Two rounds ensure the choice is robust, not a single-strategy artefact. Round 1 swept top-k (k = 8…28); Round 2 tested targeted hypotheses (drop SAR, optical-only, terrain-only, terrain+SAR, and a hand-curated 18-band set removing curvature/aspect).

Rank	Configuration	# Bands	Dice	IoU	Prec	Rec
1	Top-20 (terrain+optical+SAR)	20	0.5700	0.4001	0.5029	0.6642
2	Top-22	22	0.5540	0.3855	0.4605	0.6984
3	All bands	28	0.5514	0.3835	0.4636	0.6863
4	Top-24	24	0.5497	0.3807	0.5010	0.6135
5	Top-16	16	0.5448	0.3765	0.4366	0.7272
6	Top-12	12	0.5438	0.3748	0.5258	0.5744
7	Top-18	18	0.5355	0.3667	0.4795	0.6178
8	Top-20 − SAR	19	0.5341	0.3670	0.4565	0.6583
9	Top-8	8	0.5288	0.3616	0.4716	0.6096
10	Curated-18 (no curvature/aspect)	18	0.5143	0.3488	0.4854	0.5484
11	Terrain + SAR	8	0.5086	0.3466	0.5129	0.5104
12	Terrain only	10	0.4738	0.3113	0.4285	0.5494
13	Optical + indices only	12	0.4465	0.2883	0.4483	0.4468

Interpretation: performance rises to k = 20 then declines — Top-24 and All-28 add the bottom-4 noise bands and lower Dice. Dropping SAR costs −0.036. The hand-curated 18-band set underperforms the ranked Top-20, confirming the automatic Cohen's d selection is not improved by manual curation.

8.3 Feature-modality ablation

Feature subset	# Bands	Val Dice
Optical + indices only	12	0.4465
Terrain only	10	0.4738
Terrain + SAR	8	0.5086
Full Top-20	20	0.5700

Marginal contribution to the full model: terrain +0.124, optical +0.062, SAR +0.036. Terrain gives ≈ 2× optical's discriminative power → rock glaciers are a morphological target first.

8.4 Final 20-band set

SurfaceRoughness, Slope, TPI, TRI, NDVI, B12, TWI, S1_VV, Elevation, NDWI, NDMI, BSI, B8A, B8, RedEdge3, Hillshade, RedEdge2, B11, Brightness_TC, RedEdge1 (indices 25,18,22,23,10,6,24,27,17,11,13,14,4,3,9,26,8,5,15,7)

---

9. Model Architectures

All models take a (128,128,20) input and output a (128,128,1) sigmoid glacier-probability map. All were trained under the identical protocol of §10.

Swin-UNet (final model). Residual convolutional encoder — four stages of a Res-Conv block (Conv-BN-ReLU-Conv-BN + 1×1 projection shortcut) with filters 16 → 32 → 64 → 128, each followed by 2×2 max-pooling: 128²×16 → 64²×32 → 32²×64 → 16²×128. Bottleneck at 8×8: a Res-Conv (256 filters) followed by two shifted-window self-attention (Swin) blocks operating over the 64 spatial tokens (8 heads), each with pre-norm, multi-head attention, and a GELU MLP with residual connections. Symmetric decoder with transposed-conv upsampling and skip concatenations restoring 16²→32²→64²→128². 1×1 sigmoid head.

Plain U-Net (baseline). Standard Ronneberger encoder–decoder, 64–128–256–512 filters, plain skip concatenations, no attention. Used as the "did we need anything fancy?" reference.

Attention U-Net. Same U-Net backbone with soft attention gates on each skip connection (Oktay et al., 2018).

SegFormer (MiT-B0). Hierarchical transformer encoder (four stages, overlap patch embedding, efficient spatial-reduction self-attention, Mix-FFN) + lightweight all-MLP decoder. 3.74 M parameters.

Random Forest (classical baseline). 300 trees, max depth 20, balanced class weights, trained on a class-balanced sample of ~1.2 M pixels from the 20 bands (scikit-learn, CPU). Answers "is deep learning justified?"

---

10. Training Configuration (final)

• Optimiser: AdamW, weight decay 1e-4 (env-configurable), batch size 32, 250 epochs.
• LR schedule: cosine annealing from 1e-4 → 1e-6 over all steps. (Beat ReduceLROnPlateau by +0.048 Dice; the plateau scheduler cut the LR too early — see §12.)
• Loss: 0.5·BCE(label-smoothed, ε=0.05) + Focal-Tversky(α=0.4, β=0.6, γ=4/3), computed over valid (non-padded) pixels only via a per-pixel validity channel Y = [label, validity].
• Metrics: Dice, IoU, precision, recall, F1 — all threshold-0.5, valid-pixels-only, empty-batch-safe (tf.cond).
• Augmentation: random horizontal/vertical flips, random 90° rotations, radiometric jitter (per-band gain 0.9–1.1 + offset ±0.05, plus Gaussian noise σ=0.05), all in normalised feature space.
• Callbacks: ModelCheckpoint(monitor='val_loss', save_best), EarlyStopping(patience 20–35), CSVLogger.
• Inference: 8-way TTA (4 rotations × 2 flips, averaged); 3-seed model ensemble; operating threshold chosen by sweeping 0.30–0.70 on the validation set.

"val = train" caveat. For the final single models, VAL_FEAT was pointed at the train folder so all 570 (train+val) patches could be used for fitting. Their reported "val" numbers (up to 0.82) are therefore memorisation scores, used only for relative architecture ranking. The honest numbers are the held-out test (§13) and 5-fold CV (§15).

---

11. First Training on Clean Data

Run	Setup	Val Dice	Note
sanity (20 ep)	val = train, all patches	0.5588	quick check — the clean data learns cleanly and fast; val Dice reached 0.56 in 19 epochs (vs 118 needed on old data)
run_v2_full	require_glacier + min_glacier 5 %	0.4994	filtering to < 365 patches (11 steps/epoch) hurt on the small clean set
run_v2_full2	all 469 patches	0.5514	removing filters recovered performance

Lesson: on a small, clean dataset, aggressive patch filtering hurts — use all patches.

---

12. Cumulative Model / Training Improvements

Each step is cumulative, on the top-20 bands and clean data. (These are val = train memorisation scores; the honest test numbers are in §13.)

Step	Change	Val Dice*	Δ	Best epoch
A	Attention U-Net, 469 train patches	0.5700	—	63
B	+ train on train+val (570 patches)	0.6330	+0.063	73
C	+ Swin-UNet (transformer backbone)	0.7171	+0.084	72
D	+ cosine annealing LR, 150 ep	0.7648	+0.048	149
E	+ cosine LR, 250 ep	0.8174	+0.053	206

Key observations:

• Swin-UNet gave the single biggest jump (+0.084) and cut the (memorisation) overfit gap from +0.123 (Attention) to +0.010 — its windowed self-attention captures the full glacier morphology across the 128 px patch that local convolutions cannot.
• Cosine annealing beat ReduceLROnPlateau by +0.048. The plateau scheduler had killed the LR by epoch ~45; cosine decays smoothly and keeps improving late — the 150 ep run was still rising at its final epoch (best = 149), and the 250 ep run peaked cleanly at epoch 206 then plateaued.
• More epochs helped only because cosine kept the model learning; extending beyond 250 gave nothing (best already at 206).

---

13. Architecture Benchmark

13.1 Validation (= train, memorisation) — relative ranking

Same top-20 bands, train+val, cosine LR, 250 ep. Memorisation scores; ranking only.

Model	val Dice	val IoU	Prec	Rec	train Dice	best ep
Plain U-Net	0.7204	0.5879	0.7208	0.7243	0.8524	183
Attention U-Net (cosine)	0.7010	0.5626	0.6708	0.7365	0.8349	161
SegFormer (MiT-B0)	0.7367	0.6096	0.7019	0.7759	0.7780	249
Swin-UNet	0.8174	0.7229	0.7796	0.8939	0.8462	206

13.2 Held-out TEST (101 patches, honest) — the numbers that matter

Classical RF baseline + 4 deep architectures, identical 20-band input, 8-way TTA for the deep models, each with its own optimal threshold.

Model	Dice	IoU	Precision	Recall	Thr.	vs U-Net
Random Forest (classical)	0.6005	0.4291	0.5190	0.7125	0.60	−0.010
Plain U-Net	0.6104	0.4392	0.6072	0.6135	0.30	baseline
Attention U-Net	0.6035	0.4322	0.5910	0.6166	0.40	−0.007
SegFormer (MiT-B0)	0.5547	0.3838	0.5056	0.6145	0.35	−0.056
Swin-UNet	0.6529	0.4847	0.6256	0.6827	0.35	+0.043

Findings (with the important reversal vs validation):

1. Swin-UNet wins by +0.043 Dice / +0.046 IoU over U-Net and +0.052 Dice over Random Forest — a clean, fair, same-config justification of the architecture.
2. The classical Random Forest matches the CNN baselines (0.601 vs U-Net 0.610, Attn 0.604) and even beats SegFormer (0.555). On a small dataset a strong classical method is not trivially beaten → deep learning helps only with the right inductive bias.
3. SegFormer had the best validation (0.737) but the worst test (0.555) — the largest val→test drop of any model. Its global efficient attention over-adapts to the training scenes; Swin's windowed attention is the better inductive bias for a small, terrain-dominated dataset. (This reversal is exactly why held-out test evaluation matters.)
4. Attention U-Net ≈ plain U-Net — the attention gates add nothing here; the gain is specifically the Swin transformer backbone, not "attention" in general.

---

14. Ensemble Study & Final Model Selection

Five ensembling strategies were evaluated on the held-out test set (all Swin-UNet, 20 bands, 8-way TTA):

Ensemble strategy	# Models	Dice	IoU	Precision	Recall
Single model (no ensemble)	1	0.6529	0.4847	0.6256	0.6827
3-seed (chosen)	3	0.6679	0.5014	0.6087	0.7398
5-fold CV ensemble	5	0.6629	0.4957	0.6278	0.7021
8-model "mega"	8	0.6679	0.5014	0.6482	0.6889
5-model copy-paste	5	0.6528	0.4845	0.6328	0.6741

Selection rationale — the 3-seed ensemble: it ties the best Dice/IoU (0.668/0.501) with the fewest models and the highest recall (0.740). Recall is the priority for inventory completeness — a missed glacier is costlier than slight boundary over-segmentation. Ensembling saturates: 3, 5, and 8 models all land at 0.663–0.668, so adding more correlated Swin models buys nothing. The 3-seed ensemble is the parsimonious optimum.

---

15. 5-fold Cross-Validation

Pool = train + val = 570 patches; the 101 test patches stay fully held out. Each fold trains on 456 and validates on a genuinely held-out 114 (checkpoint selected on real validation → the generalising epoch, not the memorising one). Per-fold normalisation stats (no leakage).

Fold	Val Dice	Val IoU	Precision	Recall	best ep
0	0.5770	0.4060	0.5755	0.5806	66
1	0.5639	0.3960	0.5341	0.6161	108
2	0.5972	0.4259	0.6069	0.5891	141
3	0.5824	0.4129	0.6117	0.5603	82
4	0.5948	0.4234	0.5924	0.6002	144
Mean	0.5831 ± 0.0136	0.4128	0.5841	0.5893	—

Std of only 0.0136 across folds → the result is highly stable and true overfitting is near zero. This is the honest cross-validated benchmark; it is measured with the training-time metric (per-batch, threshold 0.5) — see §16 for why it reads lower than the test number.

---

16. Why Test (0.67) ≠ CV (0.58): A Measurement Decomposition

It is natural to worry when the test number (0.663) exceeds the CV number (0.583). This was investigated and is not leakage — the two are measured differently. Decomposition on fold-0 (same single model):

Measurement	Dice	What it isolates
training-time CV metric (per-batch dice, threshold 0.5, no TTA)	0.5770	the number logged during training
+ global pixel-pooling + optimal threshold + 8-way TTA	0.5897	+0.013 = evaluation methodology
+ evaluate on the test split (vs the val fold)	0.6117	+0.022 = test sample marginally easier (normal 100-patch sampling variation)
+ 3-model ensembling	0.6629	+0.051 = ensembling

Every step is legitimate. There is no leakage: the test patches were held out from the start, and because there is exactly one patch per glacier, patches do not spatially overlap. The single-model apples-to-apples comparison (val 0.590 vs test 0.612, same method) shows the test set is only trivially easier — not suspiciously so.

---

17. Negative Results (everything that FAILED — do not retry)

Each was A/B-tested; none beat the plain ensemble on the honest test/CV.

Technique	Result	Why it failed
NaN-pixel exclusion from loss	−0.036 Dice	removing the FP penalty on cloud pixels made the model over-predict glacier; the band-mean fill was a useful "uncertain = background" prior
Hard-negative mining (+300 bg patches)	−0.11 Dice (0.55→0.42)	Focal-Tversky gives loss = 1.0 for any FP in an all-bg patch → suppressed glacier prediction globally; precision crashed to 0.36
CutOut augmentation (32×32 blocks)	−0.05 Dice (0.57→0.52)	hides 25 % of an already data-starved 570-patch dataset
Lovász-Softmax loss (IoU-direct surrogate)	+0.0008 (neutral)	no gain over Focal-Tversky; only shifted precision↔recall balance
256×256 patches	lower	fewer patches; extra context did not compensate
Larger ensembles (8, 10 models)	0.6679 / ~0.66	saturated — correlated Swin models don't diversify
Higher weight decay (5e-4)	negligible	regularisation is not the bottleneck

17.1 Copy-paste augmentation — the instructive failure

Copy-paste (pasting a donor glacier's pixels + mask onto another patch) was the most promising idea and the clearest lesson in small-sample statistics. On fold-0 it looked great; across all 5 folds it washed out, and on the test ensemble it hurt:

Fold	Baseline Dice	Copy-paste Dice	Δ
0	0.5770	0.5987	+0.022 ✅
1	0.5639	0.5557	−0.008 ❌
2	0.5972	0.5787	−0.019 ❌
3	0.5824	0.6068	+0.024 ✅
4	0.5948	0.5986	+0.004 ~
Mean	0.5831 ± 0.014	0.5877 ± 0.021	+0.005 (variance ↑)

The 5-model copy-paste test ensemble scored 0.6528 — worse than the plain 3-seed ensemble (0.6679). fold-0's +0.022 was luck; the mean gain (+0.005) is within noise and the variance increased. Lesson: never accept a single-fold improvement; require the full CV.

Overall interpretation: the model is data-limited, not overfit (CV std 0.014). No training-side trick moved the honest number. The only lever that would raise performance is more labelled data — multiple offset patches per glacier, or additional Himalayan sub-ranges — which the team chose not to collect.

---

18. Full Journey — One Table

#	Milestone	Data	Model	Score	Type
0	Old 30 m baseline	30 m, 12 b	U-Net	0.7618	val (metric-inflated)
1	Old 10 m attempt	noisy (20 % NaN)	AttentionUNet	0.5304	val
2	Clean data + band selection	patches_v2, 20 b	AttentionUNet	0.5700	val
3	+ train on train+val (570)	patches_v2, 20 b	AttentionUNet	0.6330	val*
4	+ Swin-UNet	patches_v2, 20 b	Swin-UNet	0.7171	val*
5	+ cosine LR, 150 ep	patches_v2, 20 b	Swin-UNet	0.7648	val*
6	+ cosine LR, 250 ep	patches_v2, 20 b	Swin-UNet	0.8174	val*
7	Single Swin-UNet	patches_v2, 20 b	Swin-UNet	0.6529	test
8	3-seed ensemble + TTA (final)	patches_v2, 20 b	3× Swin-UNet	0.6679	test
9	5-fold cross-validation	patches_v2, 20 b	5× Swin-UNet	0.583 ± 0.014	CV

* val = train (memorisation). Bold rows are the honest held-out numbers reported in the paper.

---

19. Reproduction

source /data1/nasiruddink/miniconda3/etc/profile.d/conda.sh && conda activate brats
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/data1/nasiruddink/miniconda3/envs/brats/lib:\
/data1/nasiruddink/miniconda3/envs/brats/lib/python3.10/site-packages/nvidia/cudnn/lib
export TF_CPP_MIN_LOG_LEVEL=2

BANDS="band_25,band_18,band_22,band_23,band_10,band_6,band_24,band_27,band_17,band_11,\
band_13,band_14,band_4,band_3,band_9,band_26,band_8,band_5,band_15,band_7"

# 1. Collect clean patches from GEE (direct to server)
python datapipeline/collect_patches_gee.py --out dataset/patches_v2 --bg-ratio 0.10

# 2. Train Swin-UNet (cosine LR, 250 ep) — repeat for seed1/2/3
CUDA_VISIBLE_DEVICES=0 MODEL_NAME="swin_unet" DATA_DIR="dataset/patches_v2" \
  BAND_LIST=$BANDS EPOCHS=250 RUN_NAME="run_swin_seed1" \
  CKPT_DIR="bandselection/run_swin_seed1" python3 train.py

# 3. Ensemble evaluation on held-out test (3 seeds, 8-way TTA)
CUDA_VISIBLE_DEVICES=0 DATA_DIR="dataset/patches_v2" \
  TEST_FEAT="dataset/patches_v2/test/features" TEST_MASK="dataset/patches_v2/test/masks" \
  BAND_LIST=$BANDS \
  CKPT_PATHS="bandselection/run_swin_seed1/best_model.weights.h5,...seed2...,...seed3..." \
  OUT_DIR="bandselection/ensemble_test" python3 evaluate_ensemble.py

# 4. Baselines (RF + U-Net/Attention/SegFormer) — set MODEL_NAME accordingly, then evaluate.py
python rf_baseline.py
CUDA_VISIBLE_DEVICES=1 MODEL_NAME="unet"           ... python3 train.py   # then evaluate.py
CUDA_VISIBLE_DEVICES=2 MODEL_NAME="attention_unet" ... python3 train.py
CUDA_VISIBLE_DEVICES=1 MODEL_NAME="segformer"      ... python3 train.py

# 5. Band-selection sweep, cross-validation, and paper figures
python check_separability.py               # Cohen's d ranking
python make_cv_folds.py && bash run_cv.sh  # 5-fold CV
CUDA_VISIBLE_DEVICES="" python3 make_paper_figures.py

Key scripts: datapipeline/collect_patches_gee.py (collection), train.py (training; model via MODEL_NAME, bands via BAND_LIST, epochs via EPOCHS), evaluate.py / evaluate_ensemble.py (test), rf_baseline.py (classical baseline), make_cv_folds.py + run_cv.sh (cross-validation), make_paper_figures.py (georeferenced figures), check_separability.py (Cohen's d).

---

20. Key Learnings

1. Diagnose the data before blaming the model. The 10 m regression was caused by ~20 % NaN pixels, a metric-inflation bug, and inverted loss weights — the architecture was fine from early on.
2. Metric hygiene matters. An empty-batch 1e-6/1e-6 = 1.0 artefact inflated every reported score by +0.35–0.45 and mis-selected checkpoints. It even inflated the old 30 m "baseline."
3. Temporal compositing is critical in the Himalaya. Oct–Nov + cloud-probability masking cut optical NaN from 20 % → 0 % — the single biggest gain.
4. Rock glaciers are a terrain problem. Terrain features contribute ≈ 2× the discriminative power of optical reflectance (Cohen's d top-4 all terrain; ablation +0.124 vs +0.062).
5. Architecture matters, but with the right inductive bias. A classical Random Forest matches the CNN baselines; only Swin's windowed attention gives a decisive margin, while SegFormer's global attention underperforms the classical baseline on this small dataset.
6. Held-out test can reverse validation rankings. SegFormer was best on val, worst on test — always report honest held-out test, not val (especially not val = train).
7. Never trust a single fold. Copy-paste augmentation looked like +0.022 on fold-0 and washed to +0.005 (with higher variance) across all folds — and hurt the test ensemble.
8. Cosine LR beats plateau scheduling on small datasets; it keeps improving late in training.
9. Ensembling saturates. 3, 5, 8 models all reach ~0.667; pick the smallest that ties the best.
10. The dataset is the ceiling. CV std 0.014 = near-zero overfitting. More labelled data — not more training tricks — is the only path beyond ~0.67.

---

Exhaustive hand-off report. All numbers are from actual runs stored under bandselection/. Companion documents: development_log.md (full chronological narrative) and paper.tex (manuscript).