Blame for ablation.md

# Rock-Glacier Segmentation — Full Ablation & Development Report (Exhaustive)

**Task:** Binary semantic segmentation of rock glaciers, Sikkim Himalaya (Eastern Himalaya, India)
**Input:** 10 m multi-modal satellite features (Sentinel-2 MSI + Copernicus GLO-30 DEM + Sentinel-1 SAR)
**Output:** Per-pixel binary mask (1 = rock glacier, 0 = background)
**Reference inventory:** 626 active rock glaciers, mapped by SBAS-InSAR time-series (2022–2024) with STL
seasonal filtering and Google Earth Pro optical refinement — Kumar, Malviya & Singh (2026),
*iScience* 29, 115983.
**Study extent:** ≈ 88.07°–88.95° E, 27.32°–28.08° N; rock glaciers predominantly above 4,500 m.
**Final model:** 3-seed Swin-UNet ensemble with 8-way test-time augmentation (TTA).
**Headline test result (held-out, 101 patches):** Dice **0.668**, IoU **0.501**, Precision 0.609,
Recall 0.740, operating threshold 0.30.
**Cross-validated result (5-fold):** Dice **0.583 ± 0.014**.

---

## Table of Contents
1. Executive summary
2. Background & the 30 m baseline
3. The old 10 m dataset & its data-health diagnosis
4. The four training-pipeline bugs (and fixes)
5. Dead-end experiments on the old data
6. New data-collection pipeline (the decisive fix)
7. Final dataset (`patches_v2`)
8. Band selection (Cohen's *d* + 13 experiments + modality ablation)
9. Model architectures
10. Training configuration
11. First training on clean data
12. Cumulative model/training improvements
13. Architecture benchmark — validation vs honest test
14. Ensemble study & final model selection
15. 5-fold cross-validation
16. Why test (0.67) ≠ CV (0.58): a measurement decomposition
17. Negative results (everything that failed)
18. Full-journey table
19. Reproduction
20. Key learnings

---

## 1. Executive Summary

Migrating from a 30 m dataset to a new 10 m dataset caused a large, alarming performance regression
(validation Dice collapsed from an apparent 0.76 to ~0.43–0.53). The instinct was that the model or
resolution was wrong. Systematic diagnosis proved otherwise: **the problem was the data and the
training pipeline, not the architecture.**

The path to recovery had four decisive moves, in order of impact:

1. **Rebuild the data collection** — compositing only cloud-free post-monsoon (Oct–Nov) Sentinel-2 with
   cloud-probability masking eliminated the ~20 % NaN contamination of the old all-season composites.
   *This was the single largest accuracy gain in the entire project.*
2. **Fix four training-pipeline bugs** — metric inflation, inverted loss weights, biased NaN handling,
   and a premature learning-rate scheduler.
3. **Select the right features** — a Cohen's *d* ranking plus 13 retraining experiments identified an
   optimal 20-band subset and revealed that **terrain morphology carries ≈ 2× the discriminative power
   of optical reflectance**.
4. **Choose the right model** — a controlled benchmark of four architectures against a classical
   Random-Forest baseline established the Swin-UNet as best, with a 3-seed ensemble + TTA as the final
   predictor.

The final honest held-out **test Dice is 0.668 (IoU 0.501)**, cross-validated at **0.583 ± 0.014**.
The remaining ceiling is set by dataset size (671 patches; CV std of only 0.014 shows near-zero
overfitting), not by the model — only more labelled data would push it higher.

---

## 2. Background & the 30 m Baseline

Before this work, a 30 m model existed with reported metrics **val IoU 0.5825, val Dice 0.7618,
Precision 0.6233, Recall 0.8990, F1 0.7362**. This number was later found to be **inflated by the same
metric bug described in §4** (empty background-only batches scoring a spurious 1.0). Its corrected value
is closer to ~0.65–0.68, meaning the true target the 10 m work had to match was lower than it first
appeared. This is why the honest 10 m test Dice of 0.67 is competitive with — not far below — the old
baseline.

---

## 3. The Old 10 m Dataset & Its Data-Health Diagnosis

The first 10 m dataset (`patches_10m`, 28 bands, produced by an earlier collection script) performed far
worse than the 30 m baseline. A temporary data-health script (written, run, then deleted per the task
brief) revealed exactly why the data was unusable:

**NaN / no-data contamination (train split, 3,074 patches, 50.36 M pixels):**
- **267,127,234 NaN values** in total.
- Per-band NaN rate ≈ **20–21 %** for almost every band (e.g. B2 20.93 %, B11 21.33 %).
- **Only 2 of 28 bands were clean** (bands 18 and 27 had 0 % NaN).
- **2,263 of 3,074 patches (73.6 %)** contained at least one NaN pixel.
- Cause: at 4,000–5,500 m in the monsoon-affected Himalaya, an all-season 3-year median leaves many
  optical pixels with *no* clear-sky observation across the entire window.

**Class imbalance:**
- Glacier pixels: 2,427,198 (**4.82 %**); background: 47,937,218 (95.18 %).
- Background : glacier = **19.8 : 1** across the full dataset (8.9 : 1 within glacier-only patches).

**Mask sparsity:**
- **52.4 % of patches were pure background** (all-zero masks) — half of every training batch carried no
  glacier gradient signal.

Additionally, a fix was applied to `data_generator.py`: NaN pixels were being filled with 0 *before*
z-score normalisation, so they became `(0 − μ)/σ = −μ/σ` — a biased, non-zero phantom value on ~20 % of
pixels. This was corrected to fill NaN → 0 *after* normalisation (= the band mean, a neutral value).

The `check_separability.py` (Cohen's *d*) script was also reviewed and three bugs fixed: (1) the pooled
standard-deviation formula used `(n−1)·pop_var` instead of `n·pop_var`; (2) the class-imbalance count
was read from band-1's *valid-pixel-filtered* count rather than the raw mask; (3) the conclusion text
hard-coded "607 pairs" and "4.5:1" instead of computing them.

---

## 4. The Four Training-Pipeline Bugs (and Fixes)

Diagnosis of the nine early band-selection runs (all poor, best val Dice ≈ 0.49) uncovered four
independent bugs. These — not the model — were the cause of the regression.

| # | Bug | Detailed mechanism | Effect | Fix |
| :- | :--- | :--- | :--- | :--- |
| 1 | **Metric inflation** | Every metric used additive `1e-6` smoothing: `dice = (2·i + 1e-6)/(sum + 1e-6)`. When a validation batch had **zero** glacier pixels (52 % of patches were empty) *and* the model predicted zero glacier, this evaluated to `1e-6/1e-6 = 1.0`. Keras averaged these 1.0s into the epoch metric. | val Dice inflated by roughly **+0.35–0.45**; `ModelCheckpoint(monitor='val_dice')` therefore saved the *most memorising* epoch, not the best-generalising one. Example: `run_14_best` reported val Dice 0.9006 at epoch 2 — a freshly-initialised model that happened to see an all-background batch. | Each metric rewritten with `tf.cond(denom < 0.5, lambda: 0.0, lambda: real_value)` — empty batches return **0.0**, not 1.0. |
| 2 | **Inverted Focal-Tversky weights** | `α_FP = 0.6 > β_FN = 0.4` penalised false positives 1.5× harder than false negatives. At 19.8:1 imbalance this taught the model "when unsure, predict background." | train Dice plateaued ~0.55; recall stuck at 0.60–0.70; systematic under-detection of glacier. | Flipped to α = 0.3 / β = 0.7 (recall-biased); later balanced to **α = 0.4 / β = 0.6** for the clean dataset. |
| 3 | **Biased NaN normalisation** | NaN → 0 *before* normalisation → `−μ/σ` per band on ~20 % of pixels. | non-zero phantom signal misled the encoder. | fill NaN → 0 *after* normalisation (= band mean). |
| 4 | **LR killed too early** | `ReduceLROnPlateau(factor=0.5, patience=5)` halved the LR every 5 stagnant epochs. But the stagnation was caused by bug #2, not real convergence. By epoch 45 the LR had collapsed to 2.5e-5; by epoch 65 to 1.25e-5. | the optimiser was throttled before it could learn. | replaced by **cosine annealing** (§10); early-stopping patience raised. |

**Supporting changes:** `BATCH_SIZE` 8 → 32 (batch 8 used only ~200 MB of a 49 GB A6000; batch 32 gave
~4× faster epochs with no accuracy loss); `ModelCheckpoint` monitor changed from the inflated
`val_dice_coef` to `val_loss`.

**Result:** on the *old* noisy data, `run_20_FIXED` reached **val Dice 0.5304** (up from ~0.43),
proving the model was sound and the remaining gap was a data-quality problem.

### 4.1 The nine original band-selection runs (old noisy data, buggy code — for the record)
These are the pre-fix numbers that triggered the investigation (Attention U-Net, val Dice):

| Run | Bands | Val Dice | | Run | Bands | Val Dice |
| :-- | :-: | :-- | :- | :-- | :-: | :-- |
| run_12_native | 12 | 0.4321 | | run_22_best | 22 | 0.4601 |
| run_14_best | 14 | 0.4465 | | run_24_best | 24 | 0.3793 |
| run_16_best | 16 | 0.4870 | | run_26_best | 26 | 0.4442 |
| run_18_best | 18 | 0.4472 | | run_28_all | 28 | 0.4228 |
| run_20_best | 20 | 0.4769 | | | | |

All poor and erratic — the fingerprint of a broken pipeline, not a band-selection signal.

---

## 5. Dead-End Experiments on the Old Data (before rebuilding)

Before deciding to rebuild the data, several fixes were tried on the noisy `patches_10m`. Most failed:

| Attempt | Result | Verdict & reason |
| :--- | :-- | :--- |
| `require_glacier=True` (drop the 1,611 pure-bg patches) | imbalance 19.8:1 → 8.9:1; steps/epoch ↑ | **kept** — every gradient step now carries glacier signal |
| `min_glacier_pct = 5 %` (drop 208 boundary-sliver patches) | mixed | situational |
| **NaN-pixel exclusion from loss** (`valid = 0` on NaN) | val Dice **−0.036** | ❌ reverted — removing the FP penalty on cloud pixels made the model over-predict glacier; recall +0.07 but precision −0.09 |
| **Hard-negative mining** (`bg_keep_n = 300`) | val Dice **−0.11** (0.55 → 0.42) | ❌ reverted — Focal-Tversky returns loss = 1.0 for any FP in an all-bg patch, globally suppressing glacier prediction |
| 256×256 patches (re-extract) | underperformed | ❌ confounded by the same NaN/season issues |

**Conclusion:** the old data had a hard ceiling (20 % NaN, mixed seasons, snow). No training trick could
fix it. Decision: **rebuild the data collection.**

---

## 6. New Data-Collection Pipeline (the decisive fix)

### 6.1 Root cause of the old data
The previous script used a **3-year, all-season median** with only QA60 cloud masking. QA60 removes
opaque and cirrus clouds but **misses cloud shadow, thin haze, and misclassified bright snow**, and the
all-season window retains seasonal snow. At high altitude this left ~20 % of optical pixels as no-data.

### 6.2 New design decisions
- **Temporal window:** **October–November only, 2020–2023** — post-monsoon (clouds cleared) and
  pre-winter (glacier still exposed, no fresh snow). This is the one clean window each year in Sikkim.
- **Cloud masking:** **Sentinel-2 cloud-probability product** (`COPERNICUS/S2_CLOUD_PROBABILITY`,
  probability < 35 %), which catches shadow/haze that QA60 misses.
- **Compositing:** per-pixel median of the masked scenes.
- **Spectral indices** computed from the composite (NDVI, NDWI, NDSI, NDMI, BSI, Tasselled-Cap
  Brightness, Iron-Oxide ratio B4/B2).
- **Terrain:** Copernicus GLO-30 DEM, reprojected to UTM 45N *before* computing slope/aspect/hillshade/
  TPI/TRI/TWI/curvature/roughness (native-CRS terrain gives NaN — see §6.4).
- **SAR:** Sentinel-1 VV Oct–Nov median (weather-independent structural cue).
- **Collection unit:** **per glacier**, driven by the InSAR polygon inventory (see §6.3).

### 6.3 Per-glacier, polygon-driven collection (the key structural change)
Rather than exporting one global mosaic and tiling it, the pipeline (`collect_patches_gee.py`) is
**driven by the 626-polygon inventory**:
1. For each polygon, compute its centroid and define a 128×128 px (1.28 km) window centred there.
2. **Rasterise *all* polygons that fall inside that window** → the binary mask (so a neighbouring
   glacier in the same patch is labelled glacier, not background — a bug that had poisoned earlier data).
3. Clip the 28-band feature stack for the same window via `ee.data.computePixels` (NumPy, direct to
   server — no Google Drive round-trip).
4. Skip patches with > 50 % missing optical data; retry transient GEE errors up to 3×.
5. Add ~8–10 % random background patches (locations ≥ 500 m from any glacier).

### 6.4 Collection bugs fixed during development (chronological)
| Symptom | Root cause | Fix |
| :--- | :--- | :--- |
| `gcloud not found` on the headless server | EE browser auth impossible on a server | run `earthengine authenticate` on the laptop, `scp` the `credentials` file to `~/.config/earthengine/` |
| `TopologyException: side location conflict` | self-intersecting / invalid polygons in the union | `buffer(0)` and drop invalid/empty geometries before `unary_union` |
| `'Image' object has no attribute 'focalStd'` | method doesn't exist in the EE Python API | `reduceNeighborhood(ee.Reducer.stdDev(), ee.Kernel.square(...))` |
| `Image has no band named "glacier"` | `reduceToImage` names its output `"first"` | `.rename('glacier')` |
| **100 % missing-data patches** (all skipped) | the NIR shadow mask `nir > 0.15` removed legitimately dark high-altitude pixels; the Oct–Nov cloud-prob < 20 % window was too strict | removed the NIR mask, widened to Sep–Nov, relaxed cloud threshold to 35 % |
| **Slope/Aspect/Hillshade/ProfileCurvature/TWI all-NaN** | `ee.Terrain.*` requires a *projected* CRS (metre units); the DEM was in geographic degrees | reproject the DEM to EPSG:32645 (`dem.reproject(crs, scale=30)`) before all terrain derivatives |

### 6.5 Verification (before training)
Quality was verified on the collected patches: **0 % optical NaN** across all patches; **all 28 bands
valid** (terrain fix confirmed — Slope/Aspect/Hillshade now populated); elevation 4,502–5,947 m; masks
binary {0,1}; adjacent glaciers correctly labelled. Multi-band figures for 10+ random patches were
inspected visually, plus per-band min/max/mean sanity checks (e.g. S1_VV in −18 to +1 dB, NDVI −0.2 to
+0.5, elevation in the correct Himalayan range).

> **Old vs new NaN rate: 20 % → 0 %.** This one change was the biggest driver of the entire project's
> accuracy improvement.

---

## 7. Final Dataset (`patches_v2`)

| Split | Patches | Purpose |
| :--- | :-- | :--- |
| Train | 469 | model fitting |
| Val | 101 | held-out validation; band selection; threshold sweep |
| Test | 101 | held-out final evaluation (never touched in training) |
| **Total** | **671** | 613 glacier-bearing + 58 background |

- **Bands (28):** 10 optical (B2, B3, B4, B8, B8A, B11, B12, Red-Edge 1/2/3) + 7 indices (NDVI, NDWI,
  NDSI, NDMI, BSI, Brightness-TC, Iron-Oxide) + 10 terrain (Elevation, Slope, Aspect, Plan/Profile
  Curvature, TPI, TRI, TWI, Surface Roughness, Hillshade) + 1 SAR (S1_VV).
- Patch: 128×128 px @ 10 m = 1.28 km × 1.28 km; features float32, mask uint8.
- CRS UTM 45N (EPSG:32645); split seed = 42 (70/15/15).
- Optical NaN: **0 %**. Glacier density (glacier patches): mean 17.5 %, median 14.2 %; 208 patches < 1 %
  (boundary slivers). Class balance within glacier patches ≈ 8.9 : 1.
- Normalisation: per-band z-score from **training patches only** (`norm_stats.json`).

---

## 8. Band Selection

### 8.1 Cohen's *d* separability (all 28 bands)
`d = |μ_glacier − μ_background| / σ_pooled`, pooled across train+val glacier/background pixels.

| Rank | Band | *d* | Rank | Band | *d* | Rank | Band | *d* |
| :-: | :-- | :-- | :-: | :-- | :-- | :-: | :-- | :-- |
| 1 | Surface Roughness | 0.496 | 11 | NDMI | 0.228 | 21 | Aspect | 0.077 |
| 2 | Slope | 0.489 | 12 | BSI | 0.212 | 22 | B2 (Blue) | 0.076 |
| 3 | TPI | 0.424 | 13 | B8A | 0.184 | 23 | B3 (Green) | 0.064 |
| 4 | TRI | 0.361 | 14 | B8 (NIR) | 0.167 | 24 | B4 (Red) | 0.051 |
| 5 | NDVI | 0.304 | 15 | Red-Edge 3 | 0.161 | 25 | NDSI | 0.031 |
| 6 | B12 (SWIR-2) | 0.275 | 16 | Hillshade | 0.154 | 26 | Iron-Oxide | 0.008 |
| 7 | TWI | 0.261 | 17 | Red-Edge 2 | 0.141 | 27 | Plan Curvature | 0.008 |
| 8 | S1_VV (SAR) | 0.237 | 18 | B11 (SWIR-1) | 0.123 | 28 | Profile Curvature | 0.005 |
| 9 | Elevation | 0.237 | 19 | Brightness-TC | 0.081 | | | |
| 10 | NDWI | 0.234 | 20 | Red-Edge 1 | 0.080 | | | |

**Observation:** the four most separable bands are *all terrain* descriptors; SAR and Elevation also
rank highly; the four least separable (NDSI, Iron-Oxide, Plan/Profile Curvature; *d* < 0.05) are noise.

### 8.2 All 13 band-configuration experiments (Attention U-Net proxy, held-out val)
Two rounds ensure the choice is robust, not a single-strategy artefact. Round 1 swept top-*k*
(*k* = 8…28); Round 2 tested targeted hypotheses (drop SAR, optical-only, terrain-only, terrain+SAR, and
a hand-curated 18-band set removing curvature/aspect).

| Rank | Configuration | # Bands | Dice | IoU | Prec | Rec |
| :-: | :--- | :-: | :-- | :-- | :-- | :-- |
| 1 | **Top-20 (terrain+optical+SAR)** | **20** | **0.5700** | **0.4001** | 0.5029 | 0.6642 |
| 2 | Top-22 | 22 | 0.5540 | 0.3855 | 0.4605 | 0.6984 |
| 3 | All bands | 28 | 0.5514 | 0.3835 | 0.4636 | 0.6863 |
| 4 | Top-24 | 24 | 0.5497 | 0.3807 | 0.5010 | 0.6135 |
| 5 | Top-16 | 16 | 0.5448 | 0.3765 | 0.4366 | 0.7272 |
| 6 | Top-12 | 12 | 0.5438 | 0.3748 | 0.5258 | 0.5744 |
| 7 | Top-18 | 18 | 0.5355 | 0.3667 | 0.4795 | 0.6178 |
| 8 | Top-20 − SAR | 19 | 0.5341 | 0.3670 | 0.4565 | 0.6583 |
| 9 | Top-8 | 8 | 0.5288 | 0.3616 | 0.4716 | 0.6096 |
| 10 | Curated-18 (no curvature/aspect) | 18 | 0.5143 | 0.3488 | 0.4854 | 0.5484 |
| 11 | Terrain + SAR | 8 | 0.5086 | 0.3466 | 0.5129 | 0.5104 |
| 12 | Terrain only | 10 | 0.4738 | 0.3113 | 0.4285 | 0.5494 |
| 13 | Optical + indices only | 12 | 0.4465 | 0.2883 | 0.4483 | 0.4468 |

**Interpretation:** performance rises to *k* = 20 then *declines* — Top-24 and All-28 add the bottom-4
noise bands and *lower* Dice. Dropping SAR costs −0.036. The hand-curated 18-band set underperforms the
ranked Top-20, confirming the automatic Cohen's *d* selection is not improved by manual curation.

### 8.3 Feature-modality ablation
| Feature subset | # Bands | Val Dice |
| :--- | :-: | :-- |
| Optical + indices only | 12 | 0.4465 |
| Terrain only | 10 | 0.4738 |
| Terrain + SAR | 8 | 0.5086 |
| **Full Top-20** | 20 | **0.5700** |

**Marginal contribution to the full model:** terrain **+0.124**, optical **+0.062**, SAR **+0.036**.
Terrain gives ≈ 2× optical's discriminative power → rock glaciers are a morphological target first.

### 8.4 Final 20-band set
`SurfaceRoughness, Slope, TPI, TRI, NDVI, B12, TWI, S1_VV, Elevation, NDWI, NDMI, BSI, B8A, B8,
RedEdge3, Hillshade, RedEdge2, B11, Brightness_TC, RedEdge1`
(indices 25,18,22,23,10,6,24,27,17,11,13,14,4,3,9,26,8,5,15,7)

---

## 9. Model Architectures

All models take a (128,128,20) input and output a (128,128,1) sigmoid glacier-probability map. All were
trained under the identical protocol of §10.

**Swin-UNet (final model).** Residual convolutional encoder — four stages of a Res-Conv block
(Conv-BN-ReLU-Conv-BN + 1×1 projection shortcut) with filters 16 → 32 → 64 → 128, each followed by 2×2
max-pooling: 128²×16 → 64²×32 → 32²×64 → 16²×128. Bottleneck at 8×8: a Res-Conv (256 filters) followed
by **two shifted-window self-attention (Swin) blocks** operating over the 64 spatial tokens (8 heads),
each with pre-norm, multi-head attention, and a GELU MLP with residual connections. Symmetric decoder
with transposed-conv upsampling and skip concatenations restoring 16²→32²→64²→128². 1×1 sigmoid head.

**Plain U-Net (baseline).** Standard Ronneberger encoder–decoder, 64–128–256–512 filters, plain skip
concatenations, no attention. Used as the "did we need anything fancy?" reference.

**Attention U-Net.** Same U-Net backbone with soft attention gates on each skip connection (Oktay et
al., 2018).

**SegFormer (MiT-B0).** Hierarchical transformer encoder (four stages, overlap patch embedding,
efficient spatial-reduction self-attention, Mix-FFN) + lightweight all-MLP decoder. 3.74 M parameters.

**Random Forest (classical baseline).** 300 trees, max depth 20, balanced class weights, trained on a
class-balanced sample of ~1.2 M pixels from the 20 bands (scikit-learn, CPU). Answers "is deep learning
justified?"

---

## 10. Training Configuration (final)

- **Optimiser:** AdamW, weight decay 1e-4 (env-configurable), batch size 32, **250 epochs**.
- **LR schedule:** **cosine annealing** from 1e-4 → 1e-6 over all steps. (Beat ReduceLROnPlateau by
  +0.048 Dice; the plateau scheduler cut the LR too early — see §12.)
- **Loss:** `0.5·BCE(label-smoothed, ε=0.05) + Focal-Tversky(α=0.4, β=0.6, γ=4/3)`, computed over
  **valid (non-padded) pixels only** via a per-pixel validity channel `Y = [label, validity]`.
- **Metrics:** Dice, IoU, precision, recall, F1 — all threshold-0.5, valid-pixels-only, empty-batch-safe
  (`tf.cond`).
- **Augmentation:** random horizontal/vertical flips, random 90° rotations, radiometric jitter (per-band
  gain 0.9–1.1 + offset ±0.05, plus Gaussian noise σ=0.05), all in normalised feature space.
- **Callbacks:** `ModelCheckpoint(monitor='val_loss', save_best)`, `EarlyStopping(patience 20–35)`,
  `CSVLogger`.
- **Inference:** 8-way TTA (4 rotations × 2 flips, averaged); 3-seed model ensemble; operating threshold
  chosen by sweeping 0.30–0.70 on the validation set.

**"val = train" caveat.** For the final single models, `VAL_FEAT` was pointed at the *train* folder so
all 570 (train+val) patches could be used for fitting. Their reported "val" numbers (up to 0.82) are
therefore **memorisation scores**, used only for *relative* architecture ranking. The honest numbers are
the held-out **test** (§13) and **5-fold CV** (§15).

---

## 11. First Training on Clean Data

| Run | Setup | Val Dice | Note |
| :--- | :--- | :-- | :--- |
| sanity (20 ep) | val = train, all patches | 0.5588 | quick check — the clean data learns cleanly and fast; val Dice reached 0.56 in 19 epochs (vs 118 needed on old data) |
| `run_v2_full` | require_glacier + min_glacier 5 % | 0.4994 | filtering to < 365 patches (11 steps/epoch) **hurt** on the small clean set |
| `run_v2_full2` | **all 469 patches** | 0.5514 | removing filters recovered performance |

**Lesson:** on a small, *clean* dataset, aggressive patch filtering hurts — use all patches.

---

## 12. Cumulative Model / Training Improvements

Each step is cumulative, on the top-20 bands and clean data. (These are val = train memorisation scores;
the honest test numbers are in §13.)

| Step | Change | Val Dice* | Δ | Best epoch |
| :- | :--- | :-- | :-- | :-- |
| A | Attention U-Net, 469 train patches | 0.5700 | — | 63 |
| B | + train on train+val (570 patches) | 0.6330 | +0.063 | 73 |
| C | + **Swin-UNet** (transformer backbone) | 0.7171 | +0.084 | 72 |
| D | + **cosine annealing LR**, 150 ep | 0.7648 | +0.048 | 149 |
| E | + cosine LR, **250 ep** | **0.8174** | +0.053 | 206 |

**Key observations:**
- **Swin-UNet gave the single biggest jump (+0.084)** and cut the (memorisation) overfit gap from
  +0.123 (Attention) to +0.010 — its windowed self-attention captures the full glacier morphology across
  the 128 px patch that local convolutions cannot.
- **Cosine annealing beat ReduceLROnPlateau by +0.048.** The plateau scheduler had killed the LR by
  epoch ~45; cosine decays smoothly and keeps improving late — the 150 ep run was *still rising* at its
  final epoch (best = 149), and the 250 ep run peaked cleanly at epoch 206 then plateaued.
- More epochs helped only because cosine kept the model learning; extending beyond 250 gave nothing
  (best already at 206).

---

## 13. Architecture Benchmark

### 13.1 Validation (= train, memorisation) — relative ranking
Same top-20 bands, train+val, cosine LR, 250 ep. Memorisation scores; ranking only.

| Model | val Dice | val IoU | Prec | Rec | train Dice | best ep |
| :--- | :-- | :-- | :-- | :-- | :-- | :-- |
| Plain U-Net | 0.7204 | 0.5879 | 0.7208 | 0.7243 | 0.8524 | 183 |
| Attention U-Net (cosine) | 0.7010 | 0.5626 | 0.6708 | 0.7365 | 0.8349 | 161 |
| SegFormer (MiT-B0) | 0.7367 | 0.6096 | 0.7019 | 0.7759 | 0.7780 | 249 |
| **Swin-UNet** | **0.8174** | **0.7229** | 0.7796 | 0.8939 | 0.8462 | 206 |

### 13.2 Held-out TEST (101 patches, honest) — the numbers that matter
Classical RF baseline + 4 deep architectures, identical 20-band input, 8-way TTA for the deep models,
each with its own optimal threshold.

| Model | Dice | IoU | Precision | Recall | Thr. | vs U-Net |
| :--- | :-- | :-- | :-- | :-- | :-- | :-- |
| Random Forest (classical) | 0.6005 | 0.4291 | 0.5190 | 0.7125 | 0.60 | −0.010 |
| Plain U-Net | 0.6104 | 0.4392 | 0.6072 | 0.6135 | 0.30 | baseline |
| Attention U-Net | 0.6035 | 0.4322 | 0.5910 | 0.6166 | 0.40 | −0.007 |
| SegFormer (MiT-B0) | 0.5547 | 0.3838 | 0.5056 | 0.6145 | 0.35 | −0.056 |
| **Swin-UNet** | **0.6529** | **0.4847** | **0.6256** | **0.6827** | 0.35 | **+0.043** |

**Findings (with the important reversal vs validation):**
1. **Swin-UNet wins by +0.043 Dice / +0.046 IoU over U-Net** and +0.052 Dice over Random Forest — a
   clean, fair, same-config justification of the architecture.
2. **The classical Random Forest matches the CNN baselines** (0.601 vs U-Net 0.610, Attn 0.604) and even
   beats SegFormer (0.555). On a small dataset a strong classical method is not trivially beaten → deep
   learning helps **only** with the right inductive bias.
3. **SegFormer had the *best* validation (0.737) but the *worst* test (0.555)** — the largest val→test
   drop of any model. Its global efficient attention over-adapts to the training scenes; Swin's
   *windowed* attention is the better inductive bias for a small, terrain-dominated dataset. (This
   reversal is exactly why held-out test evaluation matters.)
4. **Attention U-Net ≈ plain U-Net** — the attention gates add nothing here; the gain is specifically
   the Swin transformer backbone, not "attention" in general.

---

## 14. Ensemble Study & Final Model Selection

Five ensembling strategies were evaluated on the held-out test set (all Swin-UNet, 20 bands, 8-way TTA):

| Ensemble strategy | # Models | Dice | IoU | Precision | Recall |
| :--- | :-: | :-- | :-- | :-- | :-- |
| Single model (no ensemble) | 1 | 0.6529 | 0.4847 | 0.6256 | 0.6827 |
| **3-seed (chosen)** | **3** | **0.6679** | **0.5014** | 0.6087 | **0.7398** |
| 5-fold CV ensemble | 5 | 0.6629 | 0.4957 | 0.6278 | 0.7021 |
| 8-model "mega" | 8 | 0.6679 | 0.5014 | 0.6482 | 0.6889 |
| 5-model copy-paste | 5 | 0.6528 | 0.4845 | 0.6328 | 0.6741 |

**Selection rationale — the 3-seed ensemble:** it ties the *best* Dice/IoU (0.668/0.501) with the
*fewest* models and the *highest recall* (0.740). Recall is the priority for inventory completeness — a
missed glacier is costlier than slight boundary over-segmentation. **Ensembling saturates**: 3, 5, and 8
models all land at 0.663–0.668, so adding more correlated Swin models buys nothing. The 3-seed ensemble
is the parsimonious optimum.

---

## 15. 5-fold Cross-Validation

Pool = train + val = 570 patches; the 101 test patches stay fully held out. Each fold trains on 456 and
validates on a genuinely held-out 114 (checkpoint selected on real validation → the *generalising*
epoch, not the memorising one). Per-fold normalisation stats (no leakage).

| Fold | Val Dice | Val IoU | Precision | Recall | best ep |
| :-: | :-- | :-- | :-- | :-- | :-- |
| 0 | 0.5770 | 0.4060 | 0.5755 | 0.5806 | 66 |
| 1 | 0.5639 | 0.3960 | 0.5341 | 0.6161 | 108 |
| 2 | 0.5972 | 0.4259 | 0.6069 | 0.5891 | 141 |
| 3 | 0.5824 | 0.4129 | 0.6117 | 0.5603 | 82 |
| 4 | 0.5948 | 0.4234 | 0.5924 | 0.6002 | 144 |
| **Mean** | **0.5831 ± 0.0136** | 0.4128 | 0.5841 | 0.5893 | — |

**Std of only 0.0136** across folds → the result is highly stable and *true overfitting is near zero*.
This is the honest cross-validated benchmark; it is measured with the training-time metric (per-batch,
threshold 0.5) — see §16 for why it reads lower than the test number.

---

## 16. Why Test (0.67) ≠ CV (0.58): A Measurement Decomposition

It is natural to worry when the test number (0.663) exceeds the CV number (0.583). This was investigated
and is **not leakage** — the two are measured differently. Decomposition on fold-0 (same single model):

| Measurement | Dice | What it isolates |
| :--- | :-- | :--- |
| training-time CV metric (per-batch dice, threshold 0.5, no TTA) | 0.5770 | the number logged during training |
| + global pixel-pooling + optimal threshold + 8-way TTA | 0.5897 | +0.013 = evaluation methodology |
| + evaluate on the test split (vs the val fold) | 0.6117 | +0.022 = test sample marginally easier (normal 100-patch sampling variation) |
| + 3-model ensembling | 0.6629 | +0.051 = ensembling |

Every step is legitimate. There is no leakage: the test patches were held out from the start, and
because there is exactly one patch per glacier, patches do not spatially overlap. The single-model
apples-to-apples comparison (val 0.590 vs test 0.612, same method) shows the test set is only
trivially easier — not suspiciously so.

---

## 17. Negative Results (everything that FAILED — do not retry)

Each was A/B-tested; none beat the plain ensemble on the honest test/CV.

| Technique | Result | Why it failed |
| :--- | :-- | :--- |
| **NaN-pixel exclusion from loss** | −0.036 Dice | removing the FP penalty on cloud pixels made the model over-predict glacier; the band-mean fill was a useful "uncertain = background" prior |
| **Hard-negative mining** (+300 bg patches) | −0.11 Dice (0.55→0.42) | Focal-Tversky gives loss = 1.0 for any FP in an all-bg patch → suppressed glacier prediction globally; precision crashed to 0.36 |
| **CutOut augmentation** (32×32 blocks) | −0.05 Dice (0.57→0.52) | hides 25 % of an already data-starved 570-patch dataset |
| **Lovász-Softmax loss** (IoU-direct surrogate) | +0.0008 (neutral) | no gain over Focal-Tversky; only shifted precision↔recall balance |
| **256×256 patches** | lower | fewer patches; extra context did not compensate |
| **Larger ensembles (8, 10 models)** | 0.6679 / ~0.66 | saturated — correlated Swin models don't diversify |
| **Higher weight decay (5e-4)** | negligible | regularisation is not the bottleneck |

### 17.1 Copy-paste augmentation — the instructive failure
Copy-paste (pasting a donor glacier's pixels + mask onto another patch) was the most promising idea and
the clearest lesson in small-sample statistics. On fold-0 it looked great; across all 5 folds it washed
out, and on the test ensemble it *hurt*:

| Fold | Baseline Dice | Copy-paste Dice | Δ |
| :-: | :-- | :-- | :-- |
| 0 | 0.5770 | 0.5987 | **+0.022** ✅ |
| 1 | 0.5639 | 0.5557 | −0.008 ❌ |
| 2 | 0.5972 | 0.5787 | −0.019 ❌ |
| 3 | 0.5824 | 0.6068 | +0.024 ✅ |
| 4 | 0.5948 | 0.5986 | +0.004 ~ |
| **Mean** | **0.5831 ± 0.014** | **0.5877 ± 0.021** | **+0.005** (variance ↑) |

The 5-model copy-paste **test ensemble scored 0.6528 — worse than the plain 3-seed ensemble (0.6679).**
fold-0's +0.022 was luck; the mean gain (+0.005) is within noise and the variance increased. **Lesson:
never accept a single-fold improvement; require the full CV.**

**Overall interpretation:** the model is **data-limited, not overfit** (CV std 0.014). No training-side
trick moved the honest number. The only lever that would raise performance is **more labelled data** —
multiple offset patches per glacier, or additional Himalayan sub-ranges — which the team chose not to
collect.

---

## 18. Full Journey — One Table

| # | Milestone | Data | Model | Score | Type |
| :- | :--- | :--- | :--- | :-- | :--- |
| 0 | Old 30 m baseline | 30 m, 12 b | U-Net | 0.7618 | val (metric-inflated) |
| 1 | Old 10 m attempt | noisy (20 % NaN) | AttentionUNet | 0.5304 | val |
| 2 | Clean data + band selection | patches_v2, 20 b | AttentionUNet | 0.5700 | val |
| 3 | + train on train+val (570) | patches_v2, 20 b | AttentionUNet | 0.6330 | val* |
| 4 | + Swin-UNet | patches_v2, 20 b | Swin-UNet | 0.7171 | val* |
| 5 | + cosine LR, 150 ep | patches_v2, 20 b | Swin-UNet | 0.7648 | val* |
| 6 | + cosine LR, 250 ep | patches_v2, 20 b | Swin-UNet | 0.8174 | val* |
| 7 | **Single Swin-UNet** | patches_v2, 20 b | Swin-UNet | **0.6529** | **test** |
| 8 | **3-seed ensemble + TTA (final)** | patches_v2, 20 b | 3× Swin-UNet | **0.6679** | **test** |
| 9 | 5-fold cross-validation | patches_v2, 20 b | 5× Swin-UNet | 0.583 ± 0.014 | CV |

\* val = train (memorisation). Bold rows are the honest held-out numbers reported in the paper.

---

## 19. Reproduction

```bash
source /data1/nasiruddink/miniconda3/etc/profile.d/conda.sh && conda activate brats
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/data1/nasiruddink/miniconda3/envs/brats/lib:\
/data1/nasiruddink/miniconda3/envs/brats/lib/python3.10/site-packages/nvidia/cudnn/lib
export TF_CPP_MIN_LOG_LEVEL=2

BANDS="band_25,band_18,band_22,band_23,band_10,band_6,band_24,band_27,band_17,band_11,\
band_13,band_14,band_4,band_3,band_9,band_26,band_8,band_5,band_15,band_7"

# 1. Collect clean patches from GEE (direct to server)
python datapipeline/collect_patches_gee.py --out dataset/patches_v2 --bg-ratio 0.10

# 2. Train Swin-UNet (cosine LR, 250 ep) — repeat for seed1/2/3
CUDA_VISIBLE_DEVICES=0 MODEL_NAME="swin_unet" DATA_DIR="dataset/patches_v2" \
  BAND_LIST=$BANDS EPOCHS=250 RUN_NAME="run_swin_seed1" \
  CKPT_DIR="bandselection/run_swin_seed1" python3 train.py

# 3. Ensemble evaluation on held-out test (3 seeds, 8-way TTA)
CUDA_VISIBLE_DEVICES=0 DATA_DIR="dataset/patches_v2" \
  TEST_FEAT="dataset/patches_v2/test/features" TEST_MASK="dataset/patches_v2/test/masks" \
  BAND_LIST=$BANDS \
  CKPT_PATHS="bandselection/run_swin_seed1/best_model.weights.h5,...seed2...,...seed3..." \
  OUT_DIR="bandselection/ensemble_test" python3 evaluate_ensemble.py

# 4. Baselines (RF + U-Net/Attention/SegFormer) — set MODEL_NAME accordingly, then evaluate.py
python rf_baseline.py
CUDA_VISIBLE_DEVICES=1 MODEL_NAME="unet"           ... python3 train.py   # then evaluate.py
CUDA_VISIBLE_DEVICES=2 MODEL_NAME="attention_unet" ... python3 train.py
CUDA_VISIBLE_DEVICES=1 MODEL_NAME="segformer"      ... python3 train.py

# 5. Band-selection sweep, cross-validation, and paper figures
python check_separability.py               # Cohen's d ranking
python make_cv_folds.py && bash run_cv.sh  # 5-fold CV
CUDA_VISIBLE_DEVICES="" python3 make_paper_figures.py
```

**Key scripts:** `datapipeline/collect_patches_gee.py` (collection), `train.py` (training; model via
`MODEL_NAME`, bands via `BAND_LIST`, epochs via `EPOCHS`), `evaluate.py` / `evaluate_ensemble.py`
(test), `rf_baseline.py` (classical baseline), `make_cv_folds.py` + `run_cv.sh` (cross-validation),
`make_paper_figures.py` (georeferenced figures), `check_separability.py` (Cohen's *d*).

---

## 20. Key Learnings

1. **Diagnose the data before blaming the model.** The 10 m regression was caused by ~20 % NaN pixels, a
   metric-inflation bug, and inverted loss weights — the architecture was fine from early on.
2. **Metric hygiene matters.** An empty-batch `1e-6/1e-6 = 1.0` artefact inflated every reported score
   by +0.35–0.45 and mis-selected checkpoints. It even inflated the old 30 m "baseline."
3. **Temporal compositing is critical in the Himalaya.** Oct–Nov + cloud-probability masking cut optical
   NaN from 20 % → 0 % — the single biggest gain.
4. **Rock glaciers are a terrain problem.** Terrain features contribute ≈ 2× the discriminative power of
   optical reflectance (Cohen's *d* top-4 all terrain; ablation +0.124 vs +0.062).
5. **Architecture matters, but with the right inductive bias.** A classical Random Forest matches the
   CNN baselines; only Swin's windowed attention gives a decisive margin, while SegFormer's global
   attention *underperforms* the classical baseline on this small dataset.
6. **Held-out test can reverse validation rankings.** SegFormer was best on val, worst on test — always
   report honest held-out test, not val (especially not val = train).
7. **Never trust a single fold.** Copy-paste augmentation looked like +0.022 on fold-0 and washed to
   +0.005 (with higher variance) across all folds — and hurt the test ensemble.
8. **Cosine LR beats plateau scheduling** on small datasets; it keeps improving late in training.
9. **Ensembling saturates.** 3, 5, 8 models all reach ~0.667; pick the smallest that ties the best.
10. **The dataset is the ceiling.** CV std 0.014 = near-zero overfitting. More labelled data — not more
    training tricks — is the only path beyond ~0.67.

---

*Exhaustive hand-off report. All numbers are from actual runs stored under `bandselection/`.
Companion documents: `development_log.md` (full chronological narrative) and `paper.tex` (manuscript).*