RockGlacierNet Data Collection Pipeline

This directory contains the complete end-to-end pipeline for collecting satellite imagery and generating training-ready patches for rock glacier segmentation.

Core Philosophy

This pipeline explicitly separates Data Collection from Model Preparation.

Google Earth Engine (GEE): Downloads exact, irregularly-sized bounding boxes around each rock glacier. No glaciers are ever cut off.
Local Processing: Slices variable-sized .tif files into uniform 128×128 .npy tensors for Swin-UNet training.

Key Features

Variable Sizing (No Cut-offs): generate_masks.py extracts the true bounding box of each polygon plus a 200m padding. Large glaciers get large images, small glaciers get small images.
Label Noise Prevention: When drawing masks, the script queries the master shapefile for ALL intersecting rock glaciers, not just the primary one.
Pure Background Hunting (Hard Negatives): Automatically generates 10% pure background images across the Sikkim study area.
128×128 Patching: Extracts uniform patches with 50% overlap (stride 64), filtering patches with <5% glacier coverage.

Pipeline Overview

Final_merged01.shp
       │
       ▼
Step 1: separate_polygons.py    → output/individual_polygons/*.geojson
       │
       ▼
Step 2: generate_masks.py       → output/masks/mask_*.tif (variable sizes)
       │
       ▼
Step 3: gee_feature_extraction.py → output/features/feature_*.tif (12-band, variable sizes)
       │
       ▼
Step 4: split_dataset.py        → output/dataset/{train,val,test}/{features,masks}/
       │
       ▼
Step 5: extract_patches.py      → output/patches/{train,val,test}/{features,masks}/*.npy
                                   (128×128×12 features, 128×128×1 masks)

How to Run

Option A: Run everything at once

python run_pipeline.py

Option B: Run step by step

# Step 1: Split shapefile into individual polygons
python separate_polygons.py

# Step 2: Generate masks (variable-sized + 10% pure background)
python generate_masks.py

# Step 3: Download 12-band features from Google Earth Engine
python gee_feature_extraction.py

# Step 4: Split into train (70%) / val (15%) / test (15%)
python split_dataset.py

# Step 5: Extract 128×128 patches
python extract_patches.py

Configuration

All settings are in config.yaml:

Section	Key	Value	Description
`preprocessing`	`padding_m`	200	Buffer around each glacier (meters)
`preprocessing`	`background_ratio`	0.10	10% pure background images
`splitting`	`train/val/test_ratio`	0.70/0.15/0.15	Dataset split
`patching`	`patch_size`	128	Output patch size (pixels)
`patching`	`stride`	64	50% overlap between patches
`patching`	`minimum_glacier_percentage`	5.0	Min glacier coverage to keep a patch
`patching`	`background_keep_ratio`	0.10	Keep 10% of pure background patches

Output Structure

output/
├── individual_polygons/     # Step 1: one .geojson per glacier
├── masks/                   # Step 2: variable-sized binary masks
├── features/                # Step 3: variable-sized 12-band GeoTIFFs
├── dataset/                 # Step 4: train/val/test split
│   ├── train/{features,masks}/
│   ├── val/{features,masks}/
│   └── test/{features,masks}/
└── patches/                 # Step 5: final 128×128 .npy tensors
    ├── train/{features,masks}/
    ├── val/{features,masks}/
    ├── test/{features,masks}/
    └── metadata/
        ├── config.json
        ├── patch_metadata.csv
        ├── dataset_summary.csv
        └── diagnostics_report.txt

The 12 Feature Bands

Band	Name	Source
1	B2 (Blue)	Sentinel-2 SR
2	B3 (Green)	Sentinel-2 SR
3	B4 (Red)	Sentinel-2 SR
4	B8 (NIR)	Sentinel-2 SR
5	B11 (SWIR-1)	Sentinel-2 SR
6	B12 (SWIR-2)	Sentinel-2 SR
7	NDVI	(B8−B4)/(B8+B4)
8	NDWI	(B3−B8)/(B3+B8)
9	NDSI	(B3−B11)/(B3+B11)
10	Elevation	Copernicus GLO-30 DEM
11	Slope	Derived from DEM
12	Aspect	Derived from DEM