development_log.md

Rock-Glacier Segmentation — Complete Development Log

A chronological record of the entire project: every experiment, every bug, every dead end, and every fix — in the order it happened. For the clean topic-organised summary see ablation.md; this document is the full story.

Final result: 3-seed Swin-UNet ensemble + 8-way TTA → test Dice 0.668, IoU 0.501 (5-fold CV 0.583 ± 0.014). Data: 671 clean 10 m patches from the Kumar et al. (2026) InSAR inventory.

---

PHASE 0 — Starting Point & First Diagnostics

Context: A 30 m model existed (val Dice 0.7618, but this used a metric that was later found to be inflated). The team migrated to a new 10 m dataset (patches_10m/, 28 bands) and performance collapsed. The investigation started here.

0.1 Cohen's d separability script review (check_separability.py)

Found and fixed three bugs:

1. Pooled-std formula wrong — used (n−1)·pop_var instead of n·pop_var.
2. Class imbalance read from band-1's filtered count, not the raw mask → wrong if band-1 had NoData.
3. Hard-coded conclusion ("607 pairs", "4.5:1") not computed from data. Made dynamic. (Also: Windows absolute paths made portable.)

0.2 Data health analysis of the old 10 m data (temp script, then deleted)

Key findings that explained the regression:

• NaN contamination: ~20 % of pixels NaN across almost all 28 bands (only bands 18 & 27 clean); 267 M NaN values total; 73.6 % of patches affected.
• Severe class imbalance: 19.8 : 1 background : glacier (glacier ≈ 4.8 % of pixels).
• 52.4 % of patches were pure background (all-BG masks).

0.3 Fixed NaN normalisation bug in data_generator.py

Old code filled NaN with 0 before normalisation → those pixels became −mean/std (a biased, non-zero signal). Fixed to fill NaN → 0 after normalisation (= band mean, a neutral value).

---

PHASE 1 — Diagnosing the Broken Training Pipeline (old data)

Analysed the 9 existing band-selection runs (run_12_native … run_28_all). All poor (best val Dice ≈ 0.49). Found four training-pipeline bugs:

#	Bug	Effect	Fix
1	Metric inflation — empty (all-bg) batches returned 1e-6/1e-6 = 1.0	val Dice inflated +0.35–0.45; wrong "best" epoch saved	tf.cond returns 0 for empty batches
2	Loss weights inverted — α_FP=0.6 > β_FN=0.4 at 19.8:1 imbalance	model learned "predict background when unsure"	flipped to α=0.3/β=0.7 (later 0.4/0.6)
3	NaN bias (see 0.3)	phantom signal on ~20 % of pixels	fill after normalisation
4	LR killed early — ReduceLROnPlateau(patience=5)	LR collapsed before convergence	later replaced by cosine annealing

Other changes: BATCH_SIZE 8 → 32 (A6000s were idle), ModelCheckpoint monitor → val_loss.

Result — run_20_FIXED: val Dice 0.5304 (up from ~0.43). Confirmed the model was sound; the remaining gap was a data-quality problem.

---

PHASE 2 — Attempts to Squeeze the Old Data (mostly dead ends)

These were tried on the noisy patches_10m data. Most failed and were reverted.

Attempt	Result	Verdict
require_glacier=True (drop 1,611 bg patches)	imbalance 19.8:1 → 8.9:1	kept
min_glacier_pct=5% filter	dropped noisy sliver patches	mixed
NaN-pixel exclusion from loss (valid=0 on NaN)	−0.036 Dice	❌ reverted — model over-predicted glacier on cloud pixels; the band-mean fill was useful regularisation
Hard-negative mining (bg_keep_n=300)	−0.11 Dice (0.55→0.42)	❌ reverted — Focal-Tversky gives loss=1.0 on any FP in an all-bg patch, suppressing glacier globally
256×256 patches (re-extract)	underperformed	❌ confounded by other data issues

Conclusion of Phase 2: the old data (20 % NaN, mixed seasons, snow) had a hard ceiling. No training trick could fix it. Decision: rebuild the data.

---

PHASE 3 — New Data Collection (the pivot that worked)

3.1 Diagnosis of the old GEE collection

The old script used a 3-year all-season median with only QA60 cloud masking → retained cloud shadow, haze, and seasonal snow, leaving ~20 % NaN at high altitude.

3.2 New pipeline design

• Temporal window: October–November only (post-monsoon, pre-winter) → glacier exposed, no fresh snow, minimal cloud.
• Cloud masking: Sentinel-2 cloud-probability product < 35 % (catches shadow/haze, unlike QA60).
• Terrain: Copernicus GLO-30 DEM derivatives computed at native res then resampled.
• SAR: Sentinel-1 VV Oct–Nov composite.
• Project: rock-glacier-net.

3.3 Collection bugs fixed (many)

Bug	Fix
gcloud not found on headless server	notebook auth + copy ~/.config/earthengine/credentials from laptop
TopologyException invalid geometry	buffer(0) + drop invalid before union
focalStd doesn't exist in EE Python	reduceNeighborhood(stdDev)
reduceToImage band named "first" not "glacier"	.rename('glacier')
All-NaN patches (100 % missing)	NIR shadow mask nir>0.15 too aggressive at altitude → removed; widened window to Sep–Nov; cloud threshold 20→35 %
Slope/Aspect/Hillshade/TWI all NaN	ee.Terrain.* needs a projected CRS → reproject DEM to UTM 32645 first

3.4 Key design change — per-glacier collection (user's idea)

Instead of one global mosaic, the pipeline (collect_patches_gee.py) is driven by the polygon inventory: each glacier → 128×128 window at its centroid, with all polygons in the window rasterised jointly (so adjacent glaciers are labelled correctly), then the 28-band feature patch is clipped for that window.

3.5 Result — clean dataset patches_v2

• 671 patches (613 glacier, 58 background) → 469 train / 101 val / 101 test.
• Optical NaN: 0 % (verified across all patches — vs 20 % before).
• All 28 bands valid (fixed terrain). Elevation 4,500–5,900 m. Verified with multi-band plots.

The clean data collection was the single largest accuracy driver in the whole project.

3.6 Inventory provenance (confirmed later)

The 626-polygon inventory is from Kumar et al. (2026), iScience — SBAS-InSAR time-series detection (2022–2024) + STL filtering + Google Earth Pro optical refinement. Cited in the paper.

---

PHASE 4 — First Training on Clean Data

Run	Setup	Val Dice	Note
sanity (20 ep)	val=train	0.5588	quick check — data learns cleanly
run_v2_full	require_glacier + min_glacier 5 %	0.4994	filters hurt on small clean data
run_v2_full2	all 469 patches	0.5514	filtering removed → better

Lesson: on the small clean dataset, aggressive patch filtering hurts; use all patches.

---

PHASE 5 — Band Selection (2 rounds, 13 experiments)

5.1 Cohen's d ranking (clean data)

Terrain dominates — top 4 are all terrain: Surface Roughness (0.496), Slope (0.489), TPI (0.424), TRI (0.361). Then NDVI, B12, TWI, SAR, Elevation… Bottom 4 (NDSI, IronOxide, Plan/Profile Curvature) are noise (d < 0.05).

5.2 All 13 experiments (Attention U-Net, val Dice)

Config	Bands	Dice		Config	Bands	Dice
Top-20	20	0.5700		Top-8	8	0.5288
Top-22	22	0.5540		Curated-18	18	0.5143
All-28	28	0.5514		Terrain+SAR	8	0.5086
Top-24	24	0.5497		Terrain-only	10	0.4738
Top-16	16	0.5448		Optical-only	12	0.4465
Top-12	12	0.5438		Top-18	18	0.5355
Top-20 − SAR	19	0.5341

Winner: Top-20 (0.5700). Adding the bottom-4 noise bands (24, 28) hurts.

5.3 Modality ablation

Terrain +0.124, Optical +0.062, SAR +0.036 → terrain ≈ 2× optical's discriminative power.

---

PHASE 6 — Model & Training Improvements (cumulative)

Step	Change	Val Dice*	Δ
A	Attention U-Net, 469 patches, top-20	0.5700	—
B	+ train on train+val (570 patches)	0.6330	+0.063
C	+ Swin-UNet	0.7171	+0.084
D	+ cosine LR, 150 ep	0.7648	+0.048
E	+ cosine LR, 250 ep	0.8174	+0.053

* Steps B–E used val = train (memorisation scores) — used only for relative comparison; the honest numbers are the test-set results in Phase 7.

Findings: Swin-UNet gave the biggest single jump (+0.084) and near-zero overfit. Cosine annealing beat ReduceLROnPlateau by +0.048. 250 ep converged at epoch 206 (150 ep was still rising).

---

PHASE 7 — Honest Test Evaluation

Built evaluate_ensemble.py (multi-model, 8-way TTA, threshold sweep). Held-out test = 101 patches never seen in training.

Configuration	Test Dice	IoU	Prec	Rec
Single Swin-UNet	0.6529	0.4847	0.6256	0.6827
3-seed ensemble (chosen)	0.6679	0.5014	0.6087	0.7398
5-fold CV ensemble	0.6629	0.4957	0.6278	0.7021
8-model mega ensemble	0.6679	0.5014	0.6482	0.6889
5-model copy-paste ensemble	0.6528	0.4845	0.6328	0.6741

5-fold CV: 0.5770 / 0.5639 / 0.5972 / 0.5824 / 0.5948 → 0.583 ± 0.014 (very stable).

"Why is test (0.67) > CV (0.58)?" — investigated, not leakage

Decomposed on fold-0 (same single model, same method):

• training-time CV metric (per-batch, thr 0.5) → 0.577
• +0.013 pooled + optimal-threshold + TTA
• +0.022 test sample marginally easier (normal variation)
• +0.051 ensembling = 0.663. All legitimate — the numbers are measured differently.

Ensembling saturates: 3, 5, 8 models all land at 0.663–0.668. Chose 3-seed (best Dice/IoU, fewest models, highest recall 0.740 — best for inventory completeness).

---

PHASE 8 — Attempts to Break 0.70 (all failed / marginal)

Once clean data + Swin + cosine + ensemble was in place, tried to push higher. Every attempt A/B-tested:

Technique	Result	Verdict
CutOut augmentation	0.5214 (−0.05 vs 0.57)	❌ too aggressive for 570 patches
Copy-paste augmentation	fold-0 +0.022, but full 5-fold CV only +0.005 (variance ↑); test ensemble 0.6528	❌ fold-0 gain was luck; hurt on test
Lovász-Softmax loss	0.5777 vs 0.5770 (+0.0008)	❌ neutral, not worth complexity
Higher weight decay (5e-4)	negligible	—
Mega 10-model ensemble	crashed (OOM from other user's job), rerun ~0.66	no gain

Root cause of the ceiling: the model is data-limited, not overfit (CV std 0.014, train-test gap mostly a measurement artefact). No training trick moved it. Only more labelled data would help — which the team chose not to collect.

---

PHASE 9 — Baselines, Figures & Paper

9.1 Random-Forest baseline (the "do you need DL?" check)

Per-pixel RF (300 trees) on the same 20 bands → test Dice 0.6005. Competitive with the CNN baselines (matches U-Net 0.610, Attn 0.604; beats SegFormer 0.555), but well below Swin (0.653). Message: deep learning helps here only with the right inductive bias.

9.2 Four-architecture benchmark (held-out test)

Model	Dice	IoU	Prec	Rec
Random Forest	0.6005	0.4291	0.5190	0.7125
Plain U-Net	0.6104	0.4392	0.6072	0.6135
Attention U-Net	0.6035	0.4322	0.5910	0.6166
SegFormer (MiT-B0)	0.5547	0.3838	0.5056	0.6145
Swin-UNet	0.6529	0.4847	0.6256	0.6827

SegFormer had the best validation but generalised worst on test — global attention over-adapts on a small dataset; Swin's windowed attention is the better inductive bias.

9.3 Figures & paper

• make_paper_figures.py: georeferenced (lat/lon) 5-panel per-patch figures, top-5 detailed, best/worst montages (GT-yellow vs pred-cyan outlines), training curves.
• paper.tex: 3 TikZ diagrams (data pipeline, Swin-UNet architecture, ensemble inference) + 7 tables
  • 6 figures. Title finalised: "A Multi-Modal Deep-Learning Benchmark for Rock Glacier Mapping in the Sikkim Himalaya at 10 m Resolution." Inventory cited (Kumar et al., 2026). No named "novel model" — framed honestly as an application/benchmark study.

---

Full Journey — One Table

#	Milestone	Score	Type
1	Old 10 m attempt (noisy data)	0.5304	val
2	Clean data + band selection (Attn)	0.5700	val
3	+ train on train+val	0.6330	val*
4	+ Swin-UNet	0.7171	val*
5	+ cosine LR, 250 ep	0.8174	val*
6	Single Swin-UNet	0.6529	test
7	3-seed ensemble + TTA (final)	0.6679	test
8	5-fold cross-validation	0.583 ± 0.014	CV

* val = train (memorisation). Bold rows are the honest held-out test numbers reported in the paper.

---

Everything That FAILED (so it isn't retried)

• ❌ NaN-pixel exclusion from loss (−0.04)
• ❌ Hard-negative background mining (−0.11)
• ❌ CutOut augmentation (−0.05)
• ❌ Copy-paste augmentation (fold-luck; washed out; hurt on test)
• ❌ Lovász-Softmax loss (neutral)
• ❌ 256×256 patches (underperformed)
• ❌ Larger ensembles beyond 3 models (saturated at ~0.667)
• ❌ Higher weight decay (negligible)

Everything That WORKED

• ✅ Fixing the 4 pipeline bugs (metric inflation, loss weights, NaN, LR) — restored the model
• ✅ Clean Oct–Nov cloud-masked data collection (0 % NaN) — biggest single gain
• ✅ Per-glacier collection with joint polygon rasterisation (correct adjacent-glacier labels)
• ✅ 20-band selection (terrain-dominated)
• ✅ Swin-UNet architecture (+0.084)
• ✅ Cosine annealing LR (+0.048)
• ✅ Training on train+val for the final model
• ✅ 3-seed ensemble + 8-way TTA (+0.015)
• ✅ Random-Forest baseline (proves DL is justified)

---

The Single Most Important Lesson

It was a data problem, not a model problem. The model was sound from early on; the 10 m regression was caused by 20 % NaN pixels, metric bugs, and inverted loss weights. Rebuilding the data collection (clean post-monsoon cloud-masked composites) delivered the largest gain, and the final ceiling (0.67 test) is set by dataset size (671 patches, CV std 0.014 = near-zero overfitting), not by the architecture. More labelled data — not more training tricks — is the path beyond 0.67.

---

This log records the full experimental history from the chat session. Numbers are from actual runs stored under bandselection/. See ablation.md for the topic-organised summary and paper.tex for the manuscript.