| # RockGlacierNet Data Collection Pipeline |
|
|
| This directory contains the **complete end-to-end pipeline** for collecting satellite imagery and generating training-ready patches for rock glacier segmentation. |
|
|
| ## Core Philosophy |
|
|
| This pipeline explicitly separates **Data Collection** from **Model Preparation**. |
|
|
| 1. **Google Earth Engine (GEE)**: Downloads exact, irregularly-sized bounding boxes around each rock glacier. No glaciers are ever cut off. |
| 2. **Local Processing**: Slices variable-sized `.tif` files into uniform **128×128** `.npy` tensors for Swin-UNet training. |
|
|
| ## Key Features |
|
|
| * **Variable Sizing (No Cut-offs):** `generate_masks.py` extracts the true bounding box of each polygon plus a 200m padding. Large glaciers get large images, small glaciers get small images. |
| * **Label Noise Prevention:** When drawing masks, the script queries the master shapefile for ALL intersecting rock glaciers, not just the primary one. |
| * **Pure Background Hunting (Hard Negatives):** Automatically generates 10% pure background images across the Sikkim study area. |
| * **128×128 Patching:** Extracts uniform patches with 50% overlap (stride 64), filtering patches with <5% glacier coverage. |
|
|
| ## Pipeline Overview |
|
|
| ``` |
| Final_merged01.shp |
| │ |
| ▼ |
| Step 1: separate_polygons.py → output/individual_polygons/*.geojson |
| │ |
| ▼ |
| Step 2: generate_masks.py → output/masks/mask_*.tif (variable sizes) |
| │ |
| ▼ |
| Step 3: gee_feature_extraction.py → output/features/feature_*.tif (12-band, variable sizes) |
| │ |
| ▼ |
| Step 4: split_dataset.py → output/dataset/{train,val,test}/{features,masks}/ |
| │ |
| ▼ |
| Step 5: extract_patches.py → output/patches/{train,val,test}/{features,masks}/*.npy |
| (128×128×12 features, 128×128×1 masks) |
| ``` |
|
|
| ## How to Run |
|
|
| ### Option A: Run everything at once |
| ```bash |
| python run_pipeline.py |
| ``` |
|
|
| ### Option B: Run step by step |
| ```bash |
| # Step 1: Split shapefile into individual polygons |
| python separate_polygons.py |
| |
| # Step 2: Generate masks (variable-sized + 10% pure background) |
| python generate_masks.py |
| |
| # Step 3: Download 12-band features from Google Earth Engine |
| python gee_feature_extraction.py |
| |
| # Step 4: Split into train (70%) / val (15%) / test (15%) |
| python split_dataset.py |
| |
| # Step 5: Extract 128×128 patches |
| python extract_patches.py |
| ``` |
|
|
| ## Configuration |
|
|
| All settings are in `config.yaml`: |
|
|
| | Section | Key | Value | Description | |
| |---|---|---|---| |
| | `preprocessing` | `padding_m` | 200 | Buffer around each glacier (meters) | |
| | `preprocessing` | `background_ratio` | 0.10 | 10% pure background images | |
| | `splitting` | `train/val/test_ratio` | 0.70/0.15/0.15 | Dataset split | |
| | `patching` | `patch_size` | 128 | Output patch size (pixels) | |
| | `patching` | `stride` | 64 | 50% overlap between patches | |
| | `patching` | `minimum_glacier_percentage` | 5.0 | Min glacier coverage to keep a patch | |
| | `patching` | `background_keep_ratio` | 0.10 | Keep 10% of pure background patches | |
|
|
| ## Output Structure |
|
|
| ``` |
| output/ |
| ├── individual_polygons/ # Step 1: one .geojson per glacier |
| ├── masks/ # Step 2: variable-sized binary masks |
| ├── features/ # Step 3: variable-sized 12-band GeoTIFFs |
| ├── dataset/ # Step 4: train/val/test split |
| │ ├── train/{features,masks}/ |
| │ ├── val/{features,masks}/ |
| │ └── test/{features,masks}/ |
| └── patches/ # Step 5: final 128×128 .npy tensors |
| ├── train/{features,masks}/ |
| ├── val/{features,masks}/ |
| ├── test/{features,masks}/ |
| └── metadata/ |
| ├── config.json |
| ├── patch_metadata.csv |
| ├── dataset_summary.csv |
| └── diagnostics_report.txt |
| ``` |
|
|
| ## The 12 Feature Bands |
|
|
| | Band | Name | Source | |
| |:---:|---|---| |
| | 1 | B2 (Blue) | Sentinel-2 SR | |
| | 2 | B3 (Green) | Sentinel-2 SR | |
| | 3 | B4 (Red) | Sentinel-2 SR | |
| | 4 | B8 (NIR) | Sentinel-2 SR | |
| | 5 | B11 (SWIR-1) | Sentinel-2 SR | |
| | 6 | B12 (SWIR-2) | Sentinel-2 SR | |
| | 7 | NDVI | (B8−B4)/(B8+B4) | |
| | 8 | NDWI | (B3−B8)/(B3+B8) | |
| | 9 | NDSI | (B3−B11)/(B3+B11) | |
| | 10 | Elevation | Copernicus GLO-30 DEM | |
| | 11 | Slope | Derived from DEM | |
| | 12 | Aspect | Derived from DEM | |
|
|
