Upload folder using huggingface_hub
80e070e verified 5 months ago
24.2 kB
# RockGlacierNet — Data Preprocessing Specification (Shapefile → Training-Ready Dataset)

This is a **complete, step-by-step recipe** for turning one rock-glacier Shapefile (`.shp`)
into a machine-learning-ready dataset of aligned 12-band feature tensors and binary
segmentation masks.

It is written so that **someone with no prior knowledge of the project** can follow it
end to end. Every stage has:

1. **What it does** (objective)
2. **Inputs / Outputs** (exact files and folders)
3. **How to run it** (the reference script)
4. **✅ TEST GATE** — a *mandatory* check that must pass before moving on.

> **Golden rule:** Never skip a TEST GATE. Each gate exists to catch a specific failure
> that would otherwise silently corrupt model training. If a gate fails, **stop and fix
> the previous stage** — do not proceed.

---

## 0. Setup (do this first)

### 0.1 Inputs you must have
- **One Shapefile** of rock-glacier polygons. A `.shp` never travels alone — make sure the
  sidecar files are in the same folder: `.shx`, `.dbf`, and ideally `.prj` and `.cpg`.
  - Example used to build the original dataset: `Final_merged01.shp` (626 polygons,
    Sikkim Himalayas, CRS `EPSG:4326`).
- **A Google Earth Engine (GEE) account** with a cloud project ID (needed in Phase 2 to
  download satellite + terrain features).
### 0.2 Software environment
Install these Python packages (the original project used a conda env; any env works):

```bash
pip install geopandas matplotlib rasterio earthengine-api geemap shapely numpy pandas
```

| Package | Used for |
| :--- | :--- |
| `geopandas`, `shapely` | reading/splitting the shapefile, reprojection |
| `rasterio` | rasterizing polygons → masks, reading/writing GeoTIFFs |
| `earthengine-api`, `geemap` | downloading Sentinel-2 + DEM features from GEE |
| `numpy`, `pandas` | array math, patch extraction, reports |
| `matplotlib` | EDA plots and patch visualizations |

### 0.3 ⚠️ Fix the hard-coded paths first (critical)
The reference scripts were written on Windows and contain **hard-coded paths** like
`r"D:\RockGlacier_Project\..."`. Before running anything, open each script and change the
path constants at the top to **your** project root.

On this machine the project root is:
```
/data1/nasiruddink/rockglacier/RockGlacierNet
```

| Script | Constants to update |
| :--- | :--- |
| `EDA/explore_shapefile.py` | `SHAPEFILE_PATH`, `OUTPUT_DIR` |
| `EDA/separate_polygons.py` | `SHAPEFILE_PATH`, `OUTPUT_DIR`, `PLOTS_DIR` |
| `preprocess/generate_masks.py` | `INPUT_DIR`, `OUTPUT_MASK_DIR` |
| `preprocess/gee_feature_extraction.py` | `INPUT_DIR`, `OUTPUT_DIR`, GEE `project=` |
| `preprocess/harmonize_dataset.py` | `FEATURE_DIR`, `MASK_DIR`, `H_FEATURE_DIR`, `H_MASK_DIR` |
| `preprocess/split_dataset.py` | `H_FEATURE_DIR`, `H_MASK_DIR`, `DATASET_DIR` |
| `check_separability.py` | `H_FEATURE_DIR`, `H_MASK_DIR` |

> Tip: pick ONE root folder and keep the sub-folder names exactly as below so the scripts
> chain together without surprises.

### 0.4 Folder layout produced by this pipeline
```
RockGlacierNet/
├── EDA/
│   ├── individual_polygons/      # polygon_0.geojson … polygon_625.geojson
│   └── individual_plots/         # first-5 sanity plots
├── masks/                        # mask_0000.tif … (rasterized polygons) + mask_report.csv
├── features/                     # feature_0000.tif … (12-band GEE stacks)
├── preprocess/
│   ├── harmonized_features/      # cropped to match masks pixel-for-pixel
│   └── harmonized_masks/
├── dataset/                      # train/val/test split of harmonized .tif pairs
│   ├── train/{features,masks}/
│   ├── val/{features,masks}/
│   └── test/{features,masks}/
└── patches/                      # final 64×64 .npy tensors fed to the model
    ├── train/{features,masks}/
    ├── val/{features,masks}/
    ├── test/{features,masks}/
    └── metadata/                 # config.json, patch_metadata.csv, dataset_summary.csv, diagnostics_report.txt
```

### 0.5 The 12 features (memorize this band order — everything depends on it)
Every `feature_*.tif` and every feature `.npy` has **exactly 12 channels in this order**:

| Band | Name | Source | Notes |
| :---: | :--- | :--- | :--- |
| 1 | `B2` Blue | Sentinel-2 SR | reflectance 0–1 |
| 2 | `B3` Green | Sentinel-2 SR | reflectance 0–1 |
| 3 | `B4` Red | Sentinel-2 SR | reflectance 0–1 |
| 4 | `B8` NIR | Sentinel-2 SR | reflectance 0–1 |
| 5 | `B11` SWIR-1 | Sentinel-2 SR | reflectance 0–1 |
| 6 | `B12` SWIR-2 | Sentinel-2 SR | reflectance 0–1 |
| 7 | `NDVI` | derived | `(B8−B4)/(B8+B4)`, range −1…1 |
| 8 | `NDWI` | derived | `(B3−B8)/(B3+B8)`, range −1…1 |
| 9 | `NDSI` | derived | `(B3−B11)/(B3+B11)`, range −1…1 |
| 10 | `Elevation` | Copernicus GLO-30 DEM | metres |
| 11 | `Slope` | derived from DEM | degrees 0–90 |
| 12 | `Aspect` | derived from DEM | degrees 0–360 |

**Mask convention:** `1 = rock glacier`, `0 = background`.

---

## Phase 1 — EDA & Polygon Separation

### Step 1. Inspect the Shapefile
- **Objective:** Confirm the file loads, see its CRS, feature count, and attributes before
  trusting it.
- **Run:** `EDA/explore_shapefile.py`
- **Outputs:** console summary + `rock_glaciers_plot.png` (overview map).

#### ✅ TEST GATE 1 — Shapefile sanity
Run this quick check (paste into a Python shell, edit the path):

```python
import geopandas as gpd
gdf = gpd.read_file("PATH/TO/Final_merged01.shp")
print("features:", len(gdf))
print("CRS:", gdf.crs)
print("geom types:", gdf.geom_type.value_counts().to_dict())
print("empty geoms:", gdf.geometry.is_empty.sum())
print("null geoms:", gdf.geometry.isna().sum())
print("invalid geoms:", (~gdf.is_valid).sum())
print("bbox:", gdf.total_bounds)
```

**PASS criteria:**
- [ ] `features` > 0 (original dataset: **626**).
- [ ] `CRS` is defined (original: `EPSG:4326`). If `None`, the file is unusable — get a
      version with a `.prj`.
- [ ] Geometry type is `Polygon` / `MultiPolygon`.
- [ ] `bbox` looks geographically plausible for your study area (original Sikkim bbox:
      `[88.07, 27.32] → [88.93, 28.08]`).
- [ ] Note any empty/null/invalid geometries — these will be **skipped** later (expected and OK).
---

### Step 2. Split into individual polygons
- **Objective:** Break the one big shapefile into one GeoJSON per polygon so each can be
  processed independently. GeoJSON is used (not `.shp`) to avoid creating thousands of
  sidecar files.
- **Run:** `EDA/separate_polygons.py`
- **Output:** `EDA/individual_polygons/polygon_{idx}.geojson` (e.g. `polygon_0.geojson`).
  The numeric `idx` is the row index and becomes the **ID that links masks ↔ features**
  for the rest of the pipeline.

#### ✅ TEST GATE 2 — One GeoJSON per polygon
```python
import glob, geopandas as gpd
files = glob.glob("EDA/individual_polygons/polygon_*.geojson")
print("geojson files:", len(files))
g = gpd.read_file(files[0])
print("rows in one file:", len(g), "| CRS:", g.crs)   # expect 1 row, same CRS as source
```
**PASS criteria:**
- [ ] Number of GeoJSON files == number of features from Gate 1 (original: 626).
- [ ] Each file contains exactly **1** polygon and keeps the source CRS.

---

## Phase 2 — Mask & Feature Generation

### Step 3. Generate ground-truth masks
- **Objective:** Rasterize each polygon into a binary mask aligned to a fixed 10 m grid.
- **Run:** `preprocess/generate_masks.py`
- **Key parameters (do not change unless you also change them in Step 4):**
  - `TARGET_CRS = EPSG:32645` (UTM 45N — a metre-based CRS so padding is in real metres).
  - `RESOLUTION = 10.0` m/pixel (matches Sentinel-2 native resolution).
  - `PADDING = 200.0` m added on every side of the polygon's bounding box (gives the model
    surrounding background context, not just the glacier).
- **Outputs:** `masks/mask_{idx:04d}.tif` (single-band `uint8`, values {0,1}) and
  `masks/mask_report.csv` (per-mask width/height, foreground/background pixel counts, area).
- **Behavior to expect:** polygons with empty/NaN geometry are **skipped** (logged). This is
  why the final pair count (≈607) can be lower than the polygon count (626).
#### ✅ TEST GATE 3 — Masks are valid binary rasters
```python
import glob, rasterio, numpy as np
bad = []
for f in sorted(glob.glob("masks/mask_*.tif")):
    with rasterio.open(f) as src:
        m = src.read(1)
        vals = set(np.unique(m).tolist())
        if not vals.issubset({0, 1}):        bad.append((f, "values", vals))
        if str(src.crs) != "EPSG:32645":     bad.append((f, "crs", str(src.crs)))
        if m.sum() == 0:                      bad.append((f, "empty-foreground", 0))
print("masks:", len(glob.glob('masks/mask_*.tif')), "| problems:", len(bad))
for b in bad[:10]: print("  ✗", b)
```
**PASS criteria:**
- [ ] Every mask contains **only** values `{0, 1}`.
- [ ] Every mask CRS is `EPSG:32645`.
- [ ] No mask is entirely background (`m.sum() == 0`) — a fully-empty mask means the
      rasterization missed the polygon (CRS or geometry problem).
- [ ] Open `masks/mask_report.csv`: `Foreground_Pixels` > 0 for every row, and
      `Rasterized_Area_m2` should be in the same ballpark as `Polygon_Area_m2`.
---

### Step 4. Extract the 12-band feature stacks from Google Earth Engine
- **Objective:** For each polygon, download Sentinel-2 + terrain features over the **same
  padded bounding box** used for the mask, stacked into one 12-band GeoTIFF.
- **Run:** `preprocess/gee_feature_extraction.py` (set your GEE `project=` ID first).
- **How the features are built (must match the band table in §0.5):**
  - **Sentinel-2:** collection `COPERNICUS/S2_SR_HARMONIZED`, date range
    `2021-01-01 → 2024-01-01`, pixel-level cloud mask via the `QA60` band (drops opaque +
    cirrus), reflectance scaled by `÷10000`, then a **median** composite (clear, summer-biased,
    cloud-free surface). Bands taken: `B2,B3,B4,B8,B11,B12`; indices computed: `NDVI,NDWI,NDSI`.
  - **Terrain:** `COPERNICUS/DEM/GLO30`. Slope & aspect are computed **before** mosaicking so
    the native 30 m projection is preserved, then `Elevation, Slope, Aspect` are added.
  - **Export:** `scale=10`, `crs=EPSG:32645`, `file_per_band=False` → one 12-band
    `features/feature_{idx:04d}.tif`.
- **Behavior to expect:** the script **skips** indices whose `.tif` already exists (safe to
  re-run after interruptions) and **skips** empty geometries. GEE downloads can fail
  transiently — failures are logged; re-run to fill gaps.
#### ✅ TEST GATE 4 — Feature stacks are complete & well-formed
```python
import glob, rasterio, numpy as np
bad = []
for f in sorted(glob.glob("features/feature_*.tif")):
    with rasterio.open(f) as src:
        if src.count != 12:                 bad.append((f, "bands", src.count))
        if str(src.crs) != "EPSG:32645":    bad.append((f, "crs", str(src.crs)))
        if round(src.res[0]) != 10:         bad.append((f, "res", src.res))
        arr = src.read()
        # a band that is entirely no-data means GEE returned nothing here
        if np.all(~np.isfinite(arr)) or np.all(arr == 0): bad.append((f, "all-nodata", None))
print("features:", len(glob.glob('features/feature_*.tif')), "| problems:", len(bad))
for b in bad[:10]: print("  ✗", b)
```
**PASS criteria:**
- [ ] Each feature file has **exactly 12 bands**, CRS `EPSG:32645`, ~10 m resolution.
- [ ] No feature stack is entirely no-data/zero.
- [ ] **Every mask has a matching feature** (same `idx`). Run the coverage check:
```python
import glob, os
m = {os.path.basename(p).split('_')[1].split('.')[0] for p in glob.glob('masks/mask_*.tif')}
f = {os.path.basename(p).split('_')[1].split('.')[0] for p in glob.glob('features/feature_*.tif')}
print("masks without features:", sorted(m - f)[:20])   # must be empty (or re-run GEE for these)
```
  - [ ] `masks - features` is empty (re-run the GEE script for any missing IDs).

> **Note on no-data:** GEE may write a sentinel value of `-9999` (or `NaN`) outside valid
> coverage. That is tolerated here — it is cleaned to `0` automatically during patch
> extraction (Step 7). The gate above only fails when a **whole** stack is no-data.

---

## Phase 3 — Harmonize, Split, and Patch

### Step 5. Harmonize feature/mask pixel grids
- **Objective:** GEE export and `rasterio` rasterization can differ by **one pixel** in
  height/width. This step crops each feature and its mask to their **common (min H, min W)**
  and copies the feature's geo-transform onto the mask so the pair is **pixel-perfect aligned**.
- **Run:** `preprocess/harmonize_dataset.py`
- **Outputs:** `preprocess/harmonized_features/feature_{idx}.tif`,
  `preprocess/harmonized_masks/mask_{idx}.tif`, and `harmonization_summary.csv`
  (records original shapes, harmonized shape, and CRS-match per pair).

#### ✅ TEST GATE 5 — Every pair is identically shaped & georeferenced
```python
import glob, os, rasterio
fdir, mdir = "preprocess/harmonized_features", "preprocess/harmonized_masks"
bad = []
for fp in sorted(glob.glob(f"{fdir}/feature_*.tif")):
    idx = os.path.basename(fp).split('_')[1].split('.')[0]
    mp = f"{mdir}/mask_{idx}.tif"
    if not os.path.exists(mp): bad.append((idx, "no-mask")); continue
    with rasterio.open(fp) as fs, rasterio.open(mp) as ms:
        if fs.shape != ms.shape:           bad.append((idx, "shape", fs.shape, ms.shape))
        if fs.crs != ms.crs:               bad.append((idx, "crs"))
        if fs.transform != ms.transform:   bad.append((idx, "transform"))
print("pairs:", len(glob.glob(f'{fdir}/feature_*.tif')), "| problems:", len(bad))
for b in bad[:10]: print("  ✗", b)
```
**PASS criteria:**
- [ ] For **every** pair: identical `shape`, identical `crs`, identical `transform`.
- [ ] `0` problems printed. (Original harmonized set: **607** valid pairs.)

---

### Step 6. Split into train / val / test
- **Objective:** Randomly partition the harmonized pairs into **70 % train / 15 % val /
  15 % test**, copying matched feature+mask pairs into split folders.
- **Run:** `preprocess/split_dataset.py`
- **Reproducibility:** uses `random.seed(42)` — the split is deterministic. Re-running gives
  the same partition.
- **Outputs:** `dataset/{train,val,test}/{features,masks}/`.
  (Original split: train **424**, val **91**, test **92** = 607.)

#### ✅ TEST GATE 6 — Splits are matched and leak-free
```python
import glob, os
def ids(split, kind):
    return {os.path.basename(p).split('_')[1].split('.')[0]
            for p in glob.glob(f"dataset/{split}/{kind}/*.tif")}
sets = {s: (ids(s,'features'), ids(s,'masks')) for s in ['train','val','test']}
for s,(f,m) in sets.items():
    print(f"{s}: feat={len(f)} mask={len(m)} | feat-mask mismatch={f ^ m}")
tr, va, te = (sets['train'][0], sets['val'][0], sets['test'][0])
print("train∩val:", tr & va, "| train∩test:", tr & te, "| val∩test:", va & te)
```
**PASS criteria:**
- [ ] In each split, the feature ID set **equals** the mask ID set (no orphans).
- [ ] **No overlap** between any two splits (`train∩val`, `train∩test`, `val∩test` all empty).
      Leakage here inflates validation/test scores and is the most dangerous silent bug.
- [ ] `train + val + test` counts add up to the harmonized pair total.
---

### Step 7. Extract fixed-size patches
- **Objective:** Convert variable-sized `.tif` images into fixed **64×64** `.npy` tensors the
  model can batch, while controlling class imbalance.
- **Config:** `preprocess/patch_config.py` (all knobs live here):
  - `patch_size = 64`, `stride = 32` (50 % overlap).
  - `background_keep_ratio = 0.25` — keep only ~25 % of pure-background patches (the rest are
    discarded to fight the ~4.5 : 1 background:glacier imbalance).
  - `minimum_glacier_percentage = 0.0` — keep **any** patch that contains at least one glacier
    pixel.
  - `padding_mode = "reflect"` — features are reflect-padded at the right/bottom edge so the
    sliding window covers the whole image; **masks are always zero-padded** (padding is
    background, never glacier).
  - `num_bands = 12`, `random_seed = 42`.
- **Run:** `python preprocess/extract_patches.py`
- **What it does internally:** reads each `.tif`, reorders to `(H, W, 12)` `float32`,
  **cleans no-data** (`-9999` and any non-finite → `0.0`), slides the window, filters
  background patches by the keep-ratio, and saves kept patches as `.npy`.
- **Outputs:**
  - `patches/{split}/features/{id}_r{row}_c{col}.npy` — shape `(64, 64, 12)` `float32`.
  - `patches/{split}/masks/{id}_r{row}_c{col}.npy` — shape `(64, 64, 1)` `float32`.
  - `patches/metadata/config.json` (exact config used),
    `patch_metadata.csv` (per-patch stats + per-band mean/std),
    `dataset_summary.csv` (per-image stats),
    `diagnostics_report.txt` (human-readable summary: counts, class balance, coverage
    histograms, padding stats, expansion factor).
  - (Original result: train **2081**, val **454**, test **510** kept patches.)
#### ✅ TEST GATE 7 — Read the diagnostics before validating
Open `patches/metadata/diagnostics_report.txt` and confirm:
- [ ] **Image counts** per split match Step 6.
- [ ] **Pixel class balance** is sane (original ≈ 4.5 : 1 background:glacier). If it is wildly
      imbalanced (e.g. 50 : 1), revisit `background_keep_ratio`.
- [ ] **Padding stats** are small — only tiny images should need padding. A high
      "patches with padding" fraction hints at many sub-64 px images (expected for small
      glaciers, but worth a glance).
- [ ] Kept-patch counts are non-zero for **all three** splits.
---

## Phase 4 — MANDATORY Dataset Validation (the safety net before training)

This is the most important phase for "nothing bad during training." Run **both** scripts
below and require a clean result. Each guards against a concrete failure mode.

### Step 8. Patch alignment & integrity validation
- **Run:** `python preprocess/validate_alignment.py`
- **What it checks (per split, every patch):**

| # | Check | Why it matters |
| :-: | :--- | :--- |
| 1 | **File-count match** features vs masks | a missing mask = an unlabeled training sample |
| 2 | **Name match** every `feature/<id>.npy` has `mask/<id>.npy` | mispaired tensors teach the wrong label |
| 3 | **Shape** features `(64,64,12)`, masks `(64,64,1)`, H/W equal | shape mismatch crashes the model or mislabels pixels |
| 4 | **Dtype** features `float32`; masks `float32/uint8/int32/int64` | wrong dtype breaks loss/normalization |
| 5 | **No NaN / Inf** in features | a single NaN turns the whole loss into NaN |
| 6 | **Mask values ⊆ {0, 1}** | any other value corrupts BCE/Dice loss & metrics |
| 7 | **Zero-bleed** glacier pixels are **not** all-zero across all 12 bands | catches padding/alignment errors where labels point at no-data |
| 8 | **Spot-check** prints shape/glacier%/min/max for 5 random patches/split | human eyeball sanity |

#### ✅ TEST GATE 8 — Alignment must be perfectly clean
**PASS criteria:** the script ends with
```
OVERALL: ALL CHECKS PASSED
```
- [ ] `Count match : OK` and `Name match : OK` for every split.
- [ ] `Shape errors : 0`, `NaN errors : 0`, `Inf errors : 0`.
- [ ] `Mask value : 0 errors`, `Zero-bleed : 0 patches`.
- [ ] Spot-check rows look reasonable: feature `min/max` finite, `glacier%` between 0–100.

> If **any** count is non-zero, the script prints the first 10 offending patch IDs. Fix the
> root cause (usually Step 5 harmonization or Step 7 extraction) and re-run — **do not**
> hand-delete patches to make the gate pass.

### Step 9. Trainability / separability report (recommended)
- **Run:** `python check_separability.py` (operates on the harmonized pairs).
- **Output:** `is_trainable.md` — global class balance + per-band mean/std for glacier vs
  background and **Cohen's d** effect size (how discriminative each feature is).
- **Use it to confirm the signal is real**, not just that files are well-formed:
  - [ ] Class imbalance is manageable (original ≈ 4.5 : 1 → fine for BCE + Dice loss).
  - [ ] At least a few bands show meaningful separation (original strongest: **Slope**
        d≈0.41, **NDSI**, **NDVI**, **SWIR-1**). If *every* effect size ≈ 0, the features and
        labels may be misaligned — go back to Phase 2/3.
### Step 10. Visual spot-check (recommended)
- **Run:** `python preprocess/visualize_patches.py`
- **Outputs** in `outputs/patch_viz/`: `sample_grid_{split}.png` (RGB + red glacier overlay),
  `class_distribution.png`, `patch_size_distribution.png`, `band_histograms.png`.
- [ ] On the sample grids, the **red overlay should land on plausible glacier terrain**
      (debris tongues), not on random background — the ultimate human confirmation that
      features and masks line up.
---

## End-to-End Run Order (cheat sheet)

```bash
# Phase 1 — EDA & split
python EDA/explore_shapefile.py          # → Gate 1
python EDA/separate_polygons.py          # → Gate 2
# Phase 2 — masks & features
python preprocess/generate_masks.py      # → Gate 3
python preprocess/gee_feature_extraction.py   # GEE auth needed → Gate 4
# Phase 3 — harmonize, split, patch
python preprocess/harmonize_dataset.py   # → Gate 5
python preprocess/split_dataset.py       # → Gate 6
python preprocess/extract_patches.py     # → Gate 7
# Phase 4 — MANDATORY validation
python preprocess/validate_alignment.py  # → Gate 8 (must say ALL CHECKS PASSED)
python check_separability.py             # → Gate 9 (is_trainable.md)
python preprocess/visualize_patches.py   # → Gate 10 (eyeball outputs/patch_viz/)
```

**Only start model training after Gate 8 prints `ALL CHECKS PASSED`.**

---

## Troubleshooting — failure → likely cause → fix

| Symptom at gate | Likely cause | Fix |
| :--- | :--- | :--- |
| Gate 1: `CRS: None` | shapefile missing `.prj` | obtain the `.prj` sidecar or set CRS explicitly |
| Gate 3: mask all-background | polygon didn't rasterize (CRS/geometry) | check the polygon's GeoJSON validity; confirm reprojection to `EPSG:32645` |
| Gate 4: missing features for some masks | GEE download failed/skipped | re-run `gee_feature_extraction.py` (it skips existing, retries gaps) |
| Gate 4: band count ≠ 12 | edited band list or export error | restore the band/index list in `get_s2_image` / `get_terrain_image` |
| Gate 5: shape/transform mismatch | ran split before harmonize | (re)run `harmonize_dataset.py`, then split |
| Gate 6: split overlap | re-split with different seed / mixed old files | clear `dataset/` and re-run `split_dataset.py` |
| Gate 8: NaN/Inf errors | no-data not cleaned | confirm Step 7 ran (it converts `-9999`/non-finite → 0); re-extract |
| Gate 8: mask value errors | mask resampled with interpolation | masks must be nearest-neighbor / untouched `uint8` {0,1} |
| Gate 8: zero-bleed > 0 | feature/mask misaligned or padding over real labels | re-check Gate 5; ensure masks are zero-padded, never reflect-padded |
| Gate 9: all effect sizes ≈ 0 | features ↔ masks mismatched | verify `idx` linkage from Step 2 onward |

---

## Notes for Automated Agents

- **Follow the phases linearly.** Each `python …` command corresponds to exactly one
  reference script; do not reorder.
- **Authenticate GEE before Phase 2** (`ee.Authenticate()` / set the cloud `project=`).
- **Treat every TEST GATE as a hard precondition** for the next stage. On failure, fix the
  *upstream* stage; never silently delete data to pass a gate.
- **The ID is sacred.** The integer `idx` assigned in Step 2 links `polygon_{idx}` →
  `mask_{idx}` → `feature_{idx}` → `dataset` → `patches`. Never renumber or sort-rename files
  in a way that breaks this correspondence.
- **All geometry/config constants** (CRS `EPSG:32645`, 10 m, 200 m pad, 64-px patch, stride 32,
  seed 42, 12 bands) must stay identical between mask generation, feature extraction, and
  patching, or alignment breaks.
- **Definition of done:** `validate_alignment.py` prints `ALL CHECKS PASSED` and
  `is_trainable.md` reports a manageable class balance with non-trivial feature separability.
```