spec.md

RockGlacierNet — Data Preprocessing Specification (Shapefile → Training-Ready Dataset)

This is a complete, step-by-step recipe for turning one rock-glacier Shapefile (.shp) into a machine-learning-ready dataset of aligned 12-band feature tensors and binary segmentation masks.

It is written so that someone with no prior knowledge of the project can follow it end to end. Every stage has:

1. What it does (objective)
2. Inputs / Outputs (exact files and folders)
3. How to run it (the reference script)
4. ✅ TEST GATE — a mandatory check that must pass before moving on.

Golden rule: Never skip a TEST GATE. Each gate exists to catch a specific failure that would otherwise silently corrupt model training. If a gate fails, stop and fix the previous stage — do not proceed.

---

0. Setup (do this first)

0.1 Inputs you must have

• One Shapefile of rock-glacier polygons. A .shp never travels alone — make sure the sidecar files are in the same folder: .shx, .dbf, and ideally .prj and .cpg.
  • Example used to build the original dataset: Final_merged01.shp (626 polygons, Sikkim Himalayas, CRS EPSG:4326).
• A Google Earth Engine (GEE) account with a cloud project ID (needed in Phase 2 to download satellite + terrain features).

0.2 Software environment

Install these Python packages (the original project used a conda env; any env works):

pip install geopandas matplotlib rasterio earthengine-api geemap shapely numpy pandas

Package	Used for
geopandas, shapely	reading/splitting the shapefile, reprojection
rasterio	rasterizing polygons → masks, reading/writing GeoTIFFs
earthengine-api, geemap	downloading Sentinel-2 + DEM features from GEE
numpy, pandas	array math, patch extraction, reports
matplotlib	EDA plots and patch visualizations

0.3 ⚠️ Fix the hard-coded paths first (critical)

The reference scripts were written on Windows and contain hard-coded paths like r"D:\RockGlacier_Project\...". Before running anything, open each script and change the path constants at the top to your project root.

On this machine the project root is:

/data1/nasiruddink/rockglacier/RockGlacierNet

Script	Constants to update
EDA/explore_shapefile.py	SHAPEFILE_PATH, OUTPUT_DIR
EDA/separate_polygons.py	SHAPEFILE_PATH, OUTPUT_DIR, PLOTS_DIR
preprocess/generate_masks.py	INPUT_DIR, OUTPUT_MASK_DIR
preprocess/gee_feature_extraction.py	INPUT_DIR, OUTPUT_DIR, GEE project=
preprocess/harmonize_dataset.py	FEATURE_DIR, MASK_DIR, H_FEATURE_DIR, H_MASK_DIR
preprocess/split_dataset.py	H_FEATURE_DIR, H_MASK_DIR, DATASET_DIR
check_separability.py	H_FEATURE_DIR, H_MASK_DIR

Tip: pick ONE root folder and keep the sub-folder names exactly as below so the scripts chain together without surprises.

0.4 Folder layout produced by this pipeline

RockGlacierNet/
├── EDA/
│   ├── individual_polygons/      # polygon_0.geojson … polygon_625.geojson
│   └── individual_plots/         # first-5 sanity plots
├── masks/                        # mask_0000.tif … (rasterized polygons) + mask_report.csv
├── features/                     # feature_0000.tif … (12-band GEE stacks)
├── preprocess/
│   ├── harmonized_features/      # cropped to match masks pixel-for-pixel
│   └── harmonized_masks/
├── dataset/                      # train/val/test split of harmonized .tif pairs
│   ├── train/{features,masks}/
│   ├── val/{features,masks}/
│   └── test/{features,masks}/
└── patches/                      # final 64×64 .npy tensors fed to the model
    ├── train/{features,masks}/
    ├── val/{features,masks}/
    ├── test/{features,masks}/
    └── metadata/                 # config.json, patch_metadata.csv, dataset_summary.csv, diagnostics_report.txt

0.5 The 12 features (memorize this band order — everything depends on it)

Every feature_*.tif and every feature .npy has exactly 12 channels in this order:

Band	Name	Source	Notes
1	B2 Blue	Sentinel-2 SR	reflectance 0–1
2	B3 Green	Sentinel-2 SR	reflectance 0–1
3	B4 Red	Sentinel-2 SR	reflectance 0–1
4	B8 NIR	Sentinel-2 SR	reflectance 0–1
5	B11 SWIR-1	Sentinel-2 SR	reflectance 0–1
6	B12 SWIR-2	Sentinel-2 SR	reflectance 0–1
7	NDVI	derived	(B8−B4)/(B8+B4), range −1…1
8	NDWI	derived	(B3−B8)/(B3+B8), range −1…1
9	NDSI	derived	(B3−B11)/(B3+B11), range −1…1
10	Elevation	Copernicus GLO-30 DEM	metres
11	Slope	derived from DEM	degrees 0–90
12	Aspect	derived from DEM	degrees 0–360

Mask convention: 1 = rock glacier, 0 = background.

---

Phase 1 — EDA & Polygon Separation

Step 1. Inspect the Shapefile

• Objective: Confirm the file loads, see its CRS, feature count, and attributes before trusting it.
• Run: EDA/explore_shapefile.py
• Outputs: console summary + rock_glaciers_plot.png (overview map).

✅ TEST GATE 1 — Shapefile sanity

Run this quick check (paste into a Python shell, edit the path):

import geopandas as gpd
gdf = gpd.read_file("PATH/TO/Final_merged01.shp")
print("features:", len(gdf))
print("CRS:", gdf.crs)
print("geom types:", gdf.geom_type.value_counts().to_dict())
print("empty geoms:", gdf.geometry.is_empty.sum())
print("null geoms:", gdf.geometry.isna().sum())
print("invalid geoms:", (~gdf.is_valid).sum())
print("bbox:", gdf.total_bounds)

PASS criteria:

• features > 0 (original dataset: 626).
• CRS is defined (original: EPSG:4326). If None, the file is unusable — get a version with a .prj.
• Geometry type is Polygon / MultiPolygon.
• bbox looks geographically plausible for your study area (original Sikkim bbox: [88.07, 27.32] → [88.93, 28.08]).
• Note any empty/null/invalid geometries — these will be skipped later (expected and OK).

---

Step 2. Split into individual polygons

• Objective: Break the one big shapefile into one GeoJSON per polygon so each can be processed independently. GeoJSON is used (not .shp) to avoid creating thousands of sidecar files.
• Run: EDA/separate_polygons.py
• Output: EDA/individual_polygons/polygon_{idx}.geojson (e.g. polygon_0.geojson). The numeric idx is the row index and becomes the ID that links masks ↔ features for the rest of the pipeline.

✅ TEST GATE 2 — One GeoJSON per polygon

import glob, geopandas as gpd
files = glob.glob("EDA/individual_polygons/polygon_*.geojson")
print("geojson files:", len(files))
g = gpd.read_file(files[0])
print("rows in one file:", len(g), "| CRS:", g.crs)   # expect 1 row, same CRS as source

PASS criteria:

• Number of GeoJSON files == number of features from Gate 1 (original: 626).
• Each file contains exactly 1 polygon and keeps the source CRS.

---

Phase 2 — Mask & Feature Generation

Step 3. Generate ground-truth masks

• Objective: Rasterize each polygon into a binary mask aligned to a fixed 10 m grid.
• Run: preprocess/generate_masks.py
• Key parameters (do not change unless you also change them in Step 4):
  • TARGET_CRS = EPSG:32645 (UTM 45N — a metre-based CRS so padding is in real metres).
  • RESOLUTION = 10.0 m/pixel (matches Sentinel-2 native resolution).
  • PADDING = 200.0 m added on every side of the polygon's bounding box (gives the model surrounding background context, not just the glacier).
• Outputs: masks/mask_{idx:04d}.tif (single-band uint8, values {0,1}) and masks/mask_report.csv (per-mask width/height, foreground/background pixel counts, area).
• Behavior to expect: polygons with empty/NaN geometry are skipped (logged). This is why the final pair count (≈607) can be lower than the polygon count (626).

✅ TEST GATE 3 — Masks are valid binary rasters

import glob, rasterio, numpy as np
bad = []
for f in sorted(glob.glob("masks/mask_*.tif")):
    with rasterio.open(f) as src:
        m = src.read(1)
        vals = set(np.unique(m).tolist())
        if not vals.issubset({0, 1}):        bad.append((f, "values", vals))
        if str(src.crs) != "EPSG:32645":     bad.append((f, "crs", str(src.crs)))
        if m.sum() == 0:                      bad.append((f, "empty-foreground", 0))
print("masks:", len(glob.glob('masks/mask_*.tif')), "| problems:", len(bad))
for b in bad[:10]: print("  ✗", b)

PASS criteria:

• Every mask contains only values {0, 1}.
• Every mask CRS is EPSG:32645.
• No mask is entirely background (m.sum() == 0) — a fully-empty mask means the rasterization missed the polygon (CRS or geometry problem).
• Open masks/mask_report.csv: Foreground_Pixels > 0 for every row, and Rasterized_Area_m2 should be in the same ballpark as Polygon_Area_m2.

---

Step 4. Extract the 12-band feature stacks from Google Earth Engine

• Objective: For each polygon, download Sentinel-2 + terrain features over the same padded bounding box used for the mask, stacked into one 12-band GeoTIFF.
• Run: preprocess/gee_feature_extraction.py (set your GEE project= ID first).
• How the features are built (must match the band table in §0.5):
  • Sentinel-2: collection COPERNICUS/S2_SR_HARMONIZED, date range 2021-01-01 → 2024-01-01, pixel-level cloud mask via the QA60 band (drops opaque + cirrus), reflectance scaled by ÷10000, then a median composite (clear, summer-biased, cloud-free surface). Bands taken: B2,B3,B4,B8,B11,B12; indices computed: NDVI,NDWI,NDSI.
  • Terrain: COPERNICUS/DEM/GLO30. Slope & aspect are computed before mosaicking so the native 30 m projection is preserved, then Elevation, Slope, Aspect are added.
  • Export: scale=10, crs=EPSG:32645, file_per_band=False → one 12-band features/feature_{idx:04d}.tif.
• Behavior to expect: the script skips indices whose .tif already exists (safe to re-run after interruptions) and skips empty geometries. GEE downloads can fail transiently — failures are logged; re-run to fill gaps.

✅ TEST GATE 4 — Feature stacks are complete & well-formed

import glob, rasterio, numpy as np
bad = []
for f in sorted(glob.glob("features/feature_*.tif")):
    with rasterio.open(f) as src:
        if src.count != 12:                 bad.append((f, "bands", src.count))
        if str(src.crs) != "EPSG:32645":    bad.append((f, "crs", str(src.crs)))
        if round(src.res[0]) != 10:         bad.append((f, "res", src.res))
        arr = src.read()
        # a band that is entirely no-data means GEE returned nothing here
        if np.all(~np.isfinite(arr)) or np.all(arr == 0): bad.append((f, "all-nodata", None))
print("features:", len(glob.glob('features/feature_*.tif')), "| problems:", len(bad))
for b in bad[:10]: print("  ✗", b)

PASS criteria:

• Each feature file has exactly 12 bands, CRS EPSG:32645, ~10 m resolution.
• No feature stack is entirely no-data/zero.
• Every mask has a matching feature (same idx). Run the coverage check:

import glob, os
m = {os.path.basename(p).split('_')[1].split('.')[0] for p in glob.glob('masks/mask_*.tif')}
f = {os.path.basename(p).split('_')[1].split('.')[0] for p in glob.glob('features/feature_*.tif')}
print("masks without features:", sorted(m - f)[:20])   # must be empty (or re-run GEE for these)

• masks - features is empty (re-run the GEE script for any missing IDs).

Note on no-data: GEE may write a sentinel value of -9999 (or NaN) outside valid coverage. That is tolerated here — it is cleaned to 0 automatically during patch extraction (Step 7). The gate above only fails when a whole stack is no-data.

---

Phase 3 — Harmonize, Split, and Patch

Step 5. Harmonize feature/mask pixel grids

• Objective: GEE export and rasterio rasterization can differ by one pixel in height/width. This step crops each feature and its mask to their common (min H, min W) and copies the feature's geo-transform onto the mask so the pair is pixel-perfect aligned.
• Run: preprocess/harmonize_dataset.py
• Outputs: preprocess/harmonized_features/feature_{idx}.tif, preprocess/harmonized_masks/mask_{idx}.tif, and harmonization_summary.csv (records original shapes, harmonized shape, and CRS-match per pair).

✅ TEST GATE 5 — Every pair is identically shaped & georeferenced

import glob, os, rasterio
fdir, mdir = "preprocess/harmonized_features", "preprocess/harmonized_masks"
bad = []
for fp in sorted(glob.glob(f"{fdir}/feature_*.tif")):
    idx = os.path.basename(fp).split('_')[1].split('.')[0]
    mp = f"{mdir}/mask_{idx}.tif"
    if not os.path.exists(mp): bad.append((idx, "no-mask")); continue
    with rasterio.open(fp) as fs, rasterio.open(mp) as ms:
        if fs.shape != ms.shape:           bad.append((idx, "shape", fs.shape, ms.shape))
        if fs.crs != ms.crs:               bad.append((idx, "crs"))
        if fs.transform != ms.transform:   bad.append((idx, "transform"))
print("pairs:", len(glob.glob(f'{fdir}/feature_*.tif')), "| problems:", len(bad))
for b in bad[:10]: print("  ✗", b)

PASS criteria:

• For every pair: identical shape, identical crs, identical transform.
• 0 problems printed. (Original harmonized set: 607 valid pairs.)

---

Step 6. Split into train / val / test

• Objective: Randomly partition the harmonized pairs into 70 % train / 15 % val / 15 % test, copying matched feature+mask pairs into split folders.
• Run: preprocess/split_dataset.py
• Reproducibility: uses random.seed(42) — the split is deterministic. Re-running gives the same partition.
• Outputs: dataset/{train,val,test}/{features,masks}/. (Original split: train 424, val 91, test 92 = 607.)

✅ TEST GATE 6 — Splits are matched and leak-free

import glob, os
def ids(split, kind):
    return {os.path.basename(p).split('_')[1].split('.')[0]
            for p in glob.glob(f"dataset/{split}/{kind}/*.tif")}
sets = {s: (ids(s,'features'), ids(s,'masks')) for s in ['train','val','test']}
for s,(f,m) in sets.items():
    print(f"{s}: feat={len(f)} mask={len(m)} | feat-mask mismatch={f ^ m}")
tr, va, te = (sets['train'][0], sets['val'][0], sets['test'][0])
print("train∩val:", tr & va, "| train∩test:", tr & te, "| val∩test:", va & te)

PASS criteria:

• In each split, the feature ID set equals the mask ID set (no orphans).
• No overlap between any two splits (train∩val, train∩test, val∩test all empty). Leakage here inflates validation/test scores and is the most dangerous silent bug.
• train + val + test counts add up to the harmonized pair total.

---

Step 7. Extract fixed-size patches

• Objective: Convert variable-sized .tif images into fixed 64×64 .npy tensors the model can batch, while controlling class imbalance.
• Config: preprocess/patch_config.py (all knobs live here):
  • patch_size = 64, stride = 32 (50 % overlap).
  • background_keep_ratio = 0.25 — keep only ~25 % of pure-background patches (the rest are discarded to fight the ~4.5 : 1 background:glacier imbalance).
  • minimum_glacier_percentage = 0.0 — keep any patch that contains at least one glacier pixel.
  • padding_mode = "reflect" — features are reflect-padded at the right/bottom edge so the sliding window covers the whole image; masks are always zero-padded (padding is background, never glacier).
  • num_bands = 12, random_seed = 42.
• Run: python preprocess/extract_patches.py
• What it does internally: reads each .tif, reorders to (H, W, 12) float32, cleans no-data (-9999 and any non-finite → 0.0), slides the window, filters background patches by the keep-ratio, and saves kept patches as .npy.
• Outputs:
  • patches/{split}/features/{id}_r{row}_c{col}.npy — shape (64, 64, 12) float32.
  • patches/{split}/masks/{id}_r{row}_c{col}.npy — shape (64, 64, 1) float32.
  • patches/metadata/config.json (exact config used), patch_metadata.csv (per-patch stats + per-band mean/std), dataset_summary.csv (per-image stats), diagnostics_report.txt (human-readable summary: counts, class balance, coverage histograms, padding stats, expansion factor).
  • (Original result: train 2081, val 454, test 510 kept patches.)

✅ TEST GATE 7 — Read the diagnostics before validating

Open patches/metadata/diagnostics_report.txt and confirm:

• Image counts per split match Step 6.
• Pixel class balance is sane (original ≈ 4.5 : 1 background:glacier). If it is wildly imbalanced (e.g. 50 : 1), revisit background_keep_ratio.
• Padding stats are small — only tiny images should need padding. A high "patches with padding" fraction hints at many sub-64 px images (expected for small glaciers, but worth a glance).
• Kept-patch counts are non-zero for all three splits.

---

Phase 4 — MANDATORY Dataset Validation (the safety net before training)

This is the most important phase for "nothing bad during training." Run both scripts below and require a clean result. Each guards against a concrete failure mode.

Step 8. Patch alignment & integrity validation

• Run: python preprocess/validate_alignment.py
• What it checks (per split, every patch):

#	Check	Why it matters
1	File-count match features vs masks	a missing mask = an unlabeled training sample
2	Name match every feature/<id>.npy has mask/<id>.npy	mispaired tensors teach the wrong label
3	Shape features (64,64,12), masks (64,64,1), H/W equal	shape mismatch crashes the model or mislabels pixels
4	Dtype features float32; masks float32/uint8/int32/int64	wrong dtype breaks loss/normalization
5	No NaN / Inf in features	a single NaN turns the whole loss into NaN
6	Mask values ⊆ {0, 1}	any other value corrupts BCE/Dice loss & metrics
7	Zero-bleed glacier pixels are not all-zero across all 12 bands	catches padding/alignment errors where labels point at no-data
8	Spot-check prints shape/glacier%/min/max for 5 random patches/split	human eyeball sanity

✅ TEST GATE 8 — Alignment must be perfectly clean

PASS criteria: the script ends with

OVERALL: ALL CHECKS PASSED

• Count match : OK and Name match : OK for every split.
• Shape errors : 0, NaN errors : 0, Inf errors : 0.
• Mask value : 0 errors, Zero-bleed : 0 patches.
• Spot-check rows look reasonable: feature min/max finite, glacier% between 0–100.

If any count is non-zero, the script prints the first 10 offending patch IDs. Fix the root cause (usually Step 5 harmonization or Step 7 extraction) and re-run — do not hand-delete patches to make the gate pass.

Step 9. Trainability / separability report (recommended)

• Run: python check_separability.py (operates on the harmonized pairs).
• Output: is_trainable.md — global class balance + per-band mean/std for glacier vs background and Cohen's d effect size (how discriminative each feature is).
• Use it to confirm the signal is real, not just that files are well-formed:
  • Class imbalance is manageable (original ≈ 4.5 : 1 → fine for BCE + Dice loss).
  • At least a few bands show meaningful separation (original strongest: Slope d≈0.41, NDSI, NDVI, SWIR-1). If every effect size ≈ 0, the features and labels may be misaligned — go back to Phase 2/3.

Step 10. Visual spot-check (recommended)

• Run: python preprocess/visualize_patches.py
• Outputs in outputs/patch_viz/: sample_grid_{split}.png (RGB + red glacier overlay), class_distribution.png, patch_size_distribution.png, band_histograms.png.
• On the sample grids, the red overlay should land on plausible glacier terrain (debris tongues), not on random background — the ultimate human confirmation that features and masks line up.

---

End-to-End Run Order (cheat sheet)

# Phase 1 — EDA & split
python EDA/explore_shapefile.py          # → Gate 1
python EDA/separate_polygons.py          # → Gate 2

# Phase 2 — masks & features
python preprocess/generate_masks.py      # → Gate 3
python preprocess/gee_feature_extraction.py   # GEE auth needed → Gate 4

# Phase 3 — harmonize, split, patch
python preprocess/harmonize_dataset.py   # → Gate 5
python preprocess/split_dataset.py       # → Gate 6
python preprocess/extract_patches.py     # → Gate 7

# Phase 4 — MANDATORY validation
python preprocess/validate_alignment.py  # → Gate 8 (must say ALL CHECKS PASSED)
python check_separability.py             # → Gate 9 (is_trainable.md)
python preprocess/visualize_patches.py   # → Gate 10 (eyeball outputs/patch_viz/)

Only start model training after Gate 8 prints ALL CHECKS PASSED.

---

Troubleshooting — failure → likely cause → fix

Symptom at gate	Likely cause	Fix
Gate 1: CRS: None	shapefile missing .prj	obtain the .prj sidecar or set CRS explicitly
Gate 3: mask all-background	polygon didn't rasterize (CRS/geometry)	check the polygon's GeoJSON validity; confirm reprojection to EPSG:32645
Gate 4: missing features for some masks	GEE download failed/skipped	re-run gee_feature_extraction.py (it skips existing, retries gaps)
Gate 4: band count ≠ 12	edited band list or export error	restore the band/index list in get_s2_image / get_terrain_image
Gate 5: shape/transform mismatch	ran split before harmonize	(re)run harmonize_dataset.py, then split
Gate 6: split overlap	re-split with different seed / mixed old files	clear dataset/ and re-run split_dataset.py
Gate 8: NaN/Inf errors	no-data not cleaned	confirm Step 7 ran (it converts -9999/non-finite → 0); re-extract
Gate 8: mask value errors	mask resampled with interpolation	masks must be nearest-neighbor / untouched uint8 {0,1}
Gate 8: zero-bleed > 0	feature/mask misaligned or padding over real labels	re-check Gate 5; ensure masks are zero-padded, never reflect-padded
Gate 9: all effect sizes ≈ 0	features ↔ masks mismatched	verify idx linkage from Step 2 onward

---

Notes for Automated Agents

• Follow the phases linearly. Each python … command corresponds to exactly one reference script; do not reorder.
• Authenticate GEE before Phase 2 (ee.Authenticate() / set the cloud project=).
• Treat every TEST GATE as a hard precondition for the next stage. On failure, fix the upstream stage; never silently delete data to pass a gate.
• The ID is sacred. The integer idx assigned in Step 2 links polygon_{idx} → mask_{idx} → feature_{idx} → dataset → patches. Never renumber or sort-rename files in a way that breaks this correspondence.
• All geometry/config constants (CRS EPSG:32645, 10 m, 200 m pad, 64-px patch, stride 32, seed 42, 12 bands) must stay identical between mask generation, feature extraction, and patching, or alignment breaks.
• Definition of done: validate_alignment.py prints ALL CHECKS PASSED and is_trainable.md reports a manageable class balance with non-trivial feature separability.