# RockGlacierNet Data Collection Pipeline

This directory contains the **complete end-to-end pipeline** for collecting satellite imagery and generating training-ready patches for rock glacier segmentation.

## Core Philosophy

This pipeline explicitly separates **Data Collection** from **Model Preparation**.

1. **Google Earth Engine (GEE)**: Downloads exact, irregularly-sized bounding boxes around each rock glacier. No glaciers are ever cut off.
2. **Local Processing**: Slices variable-sized `.tif` files into uniform **128×128** `.npy` tensors for Swin-UNet training.

## Key Features

* **Variable Sizing (No Cut-offs):** `generate_masks.py` extracts the true bounding box of each polygon plus a 200m padding. Large glaciers get large images, small glaciers get small images.
* **Label Noise Prevention:** When drawing masks, the script queries the master shapefile for ALL intersecting rock glaciers, not just the primary one.
* **Pure Background Hunting (Hard Negatives):** Automatically generates 10% pure background images across the Sikkim study area.
* **128×128 Patching:** Extracts uniform patches with 50% overlap (stride 64), filtering patches with <5% glacier coverage.

## Pipeline Overview

```
Final_merged01.shp
       │
       ▼
Step 1: separate_polygons.py    → output/individual_polygons/*.geojson
       │
       ▼
Step 2: generate_masks.py       → output/masks/mask_*.tif (variable sizes)
       │
       ▼
Step 3: gee_feature_extraction.py → output/features/feature_*.tif (12-band, variable sizes)
       │
       ▼
Step 4: split_dataset.py        → output/dataset/{train,val,test}/{features,masks}/
       │
       ▼
Step 5: extract_patches.py      → output/patches/{train,val,test}/{features,masks}/*.npy
                                   (128×128×12 features, 128×128×1 masks)
```

## How to Run

### Option A: Run everything at once
```bash
python run_pipeline.py
```

### Option B: Run step by step
```bash
# Step 1: Split shapefile into individual polygons
python separate_polygons.py

# Step 2: Generate masks (variable-sized + 10% pure background)
python generate_masks.py

# Step 3: Download 12-band features from Google Earth Engine
python gee_feature_extraction.py

# Step 4: Split into train (70%) / val (15%) / test (15%)
python split_dataset.py

# Step 5: Extract 128×128 patches
python extract_patches.py
```

## Configuration

All settings are in `config.yaml`:

| Section | Key | Value | Description |
|---|---|---|---|
| `preprocessing` | `padding_m` | 200 | Buffer around each glacier (meters) |
| `preprocessing` | `background_ratio` | 0.10 | 10% pure background images |
| `splitting` | `train/val/test_ratio` | 0.70/0.15/0.15 | Dataset split |
| `patching` | `patch_size` | 128 | Output patch size (pixels) |
| `patching` | `stride` | 64 | 50% overlap between patches |
| `patching` | `minimum_glacier_percentage` | 5.0 | Min glacier coverage to keep a patch |
| `patching` | `background_keep_ratio` | 0.10 | Keep 10% of pure background patches |

## Output Structure

```
output/
├── individual_polygons/     # Step 1: one .geojson per glacier
├── masks/                   # Step 2: variable-sized binary masks
├── features/                # Step 3: variable-sized 12-band GeoTIFFs
├── dataset/                 # Step 4: train/val/test split
│   ├── train/{features,masks}/
│   ├── val/{features,masks}/
│   └── test/{features,masks}/
└── patches/                 # Step 5: final 128×128 .npy tensors
    ├── train/{features,masks}/
    ├── val/{features,masks}/
    ├── test/{features,masks}/
    └── metadata/
        ├── config.json
        ├── patch_metadata.csv
        ├── dataset_summary.csv
        └── diagnostics_report.txt
```

## The 12 Feature Bands

| Band | Name | Source |
|:---:|---|---|
| 1 | B2 (Blue) | Sentinel-2 SR |
| 2 | B3 (Green) | Sentinel-2 SR |
| 3 | B4 (Red) | Sentinel-2 SR |
| 4 | B8 (NIR) | Sentinel-2 SR |
| 5 | B11 (SWIR-1) | Sentinel-2 SR |
| 6 | B12 (SWIR-2) | Sentinel-2 SR |
| 7 | NDVI | (B8−B4)/(B8+B4) |
| 8 | NDWI | (B3−B8)/(B3+B8) |
| 9 | NDSI | (B3−B11)/(B3+B11) |
| 10 | Elevation | Copernicus GLO-30 DEM |
| 11 | Slope | Derived from DEM |
| 12 | Aspect | Derived from DEM |