File size: 4,320 Bytes
80e070e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 | # RockGlacierNet Data Collection Pipeline
This directory contains the **complete end-to-end pipeline** for collecting satellite imagery and generating training-ready patches for rock glacier segmentation.
## Core Philosophy
This pipeline explicitly separates **Data Collection** from **Model Preparation**.
1. **Google Earth Engine (GEE)**: Downloads exact, irregularly-sized bounding boxes around each rock glacier. No glaciers are ever cut off.
2. **Local Processing**: Slices variable-sized `.tif` files into uniform **128×128** `.npy` tensors for Swin-UNet training.
## Key Features
* **Variable Sizing (No Cut-offs):** `generate_masks.py` extracts the true bounding box of each polygon plus a 200m padding. Large glaciers get large images, small glaciers get small images.
* **Label Noise Prevention:** When drawing masks, the script queries the master shapefile for ALL intersecting rock glaciers, not just the primary one.
* **Pure Background Hunting (Hard Negatives):** Automatically generates 10% pure background images across the Sikkim study area.
* **128×128 Patching:** Extracts uniform patches with 50% overlap (stride 64), filtering patches with <5% glacier coverage.
## Pipeline Overview
```
Final_merged01.shp
│
▼
Step 1: separate_polygons.py → output/individual_polygons/*.geojson
│
▼
Step 2: generate_masks.py → output/masks/mask_*.tif (variable sizes)
│
▼
Step 3: gee_feature_extraction.py → output/features/feature_*.tif (12-band, variable sizes)
│
▼
Step 4: split_dataset.py → output/dataset/{train,val,test}/{features,masks}/
│
▼
Step 5: extract_patches.py → output/patches/{train,val,test}/{features,masks}/*.npy
(128×128×12 features, 128×128×1 masks)
```
## How to Run
### Option A: Run everything at once
```bash
python run_pipeline.py
```
### Option B: Run step by step
```bash
# Step 1: Split shapefile into individual polygons
python separate_polygons.py
# Step 2: Generate masks (variable-sized + 10% pure background)
python generate_masks.py
# Step 3: Download 12-band features from Google Earth Engine
python gee_feature_extraction.py
# Step 4: Split into train (70%) / val (15%) / test (15%)
python split_dataset.py
# Step 5: Extract 128×128 patches
python extract_patches.py
```
## Configuration
All settings are in `config.yaml`:
| Section | Key | Value | Description |
|---|---|---|---|
| `preprocessing` | `padding_m` | 200 | Buffer around each glacier (meters) |
| `preprocessing` | `background_ratio` | 0.10 | 10% pure background images |
| `splitting` | `train/val/test_ratio` | 0.70/0.15/0.15 | Dataset split |
| `patching` | `patch_size` | 128 | Output patch size (pixels) |
| `patching` | `stride` | 64 | 50% overlap between patches |
| `patching` | `minimum_glacier_percentage` | 5.0 | Min glacier coverage to keep a patch |
| `patching` | `background_keep_ratio` | 0.10 | Keep 10% of pure background patches |
## Output Structure
```
output/
├── individual_polygons/ # Step 1: one .geojson per glacier
├── masks/ # Step 2: variable-sized binary masks
├── features/ # Step 3: variable-sized 12-band GeoTIFFs
├── dataset/ # Step 4: train/val/test split
│ ├── train/{features,masks}/
│ ├── val/{features,masks}/
│ └── test/{features,masks}/
└── patches/ # Step 5: final 128×128 .npy tensors
├── train/{features,masks}/
├── val/{features,masks}/
├── test/{features,masks}/
└── metadata/
├── config.json
├── patch_metadata.csv
├── dataset_summary.csv
└── diagnostics_report.txt
```
## The 12 Feature Bands
| Band | Name | Source |
|:---:|---|---|
| 1 | B2 (Blue) | Sentinel-2 SR |
| 2 | B3 (Green) | Sentinel-2 SR |
| 3 | B4 (Red) | Sentinel-2 SR |
| 4 | B8 (NIR) | Sentinel-2 SR |
| 5 | B11 (SWIR-1) | Sentinel-2 SR |
| 6 | B12 (SWIR-2) | Sentinel-2 SR |
| 7 | NDVI | (B8−B4)/(B8+B4) |
| 8 | NDWI | (B3−B8)/(B3+B8) |
| 9 | NDSI | (B3−B11)/(B3+B11) |
| 10 | Elevation | Copernicus GLO-30 DEM |
| 11 | Slope | Derived from DEM |
| 12 | Aspect | Derived from DEM |
|