datasets/README.md

# Datasets Documentation

**Project:** Physics-Informed KARST-Aware GNN for Streamflow Prediction
**Study Area:** La Risle and La Eure River Systems, Seine Basin, France
**Period:** 1960-2026 (historical data)

## Directory Structure

```
datasets/
├── station_list.csv                    # 27 stations metadata (coordinates, names)
├── selected_stations.csv               # 27 selected station codes
├── IDPR_values.csv                     # BRGM infiltration/runoff index (27 stations)
├── ADES/                               # Groundwater data (KARST connectivity)
│   ├── groundwater_stations.csv        # 114 groundwater monitoring wells
│   ├── groundwater_levels_watershed.csv# 272,379 groundwater level measurements
│   ├── stations_metadata.csv          # 10,001 stations (full ADES database)
│   └── ADES_data_fetching.py          # Data collection script
├── Watersehd_risle/                    # La Risle watershed boundary
│   └── Watershed_Risle.shp            # Shapefile (EPSG:2154)
├── Watershed_Eure/                     # La Eure watershed boundary
│   └── Watershed_Eure.shp             # Shapefile (EPSG:2154)
├── Risle_Eure_watershed/              # Merged watershed boundary
│   └── Risle_Eure_Watershed.shp       # Combined shapefile (EPSG:2154)
└── River Images/                       # Reference maps
    ├── Carte-de-la_Seine-390x305.png
    ├── Eure_(rivière).png
    └── Risle.png
```

## Available Data

### 1. Hydrometric Stations (`station_list.csv`)

**File:** `station_list.csv`
**Rows:** 27 stations
**Columns:**
- `X`, `Y` - Lambert 93 coordinates (EPSG:2154)
- `station_code` - Unique station identifier (e.g., H602021010)
- `region_name` - NORMANDIE
- `departament_name` - ORNE or EURE
- `station_name` - Full station name
- `municipality` - Municipality name
- `INSEE` - INSEE code
- `id_root` - Root identifier
- `lon`, `lat` - WGS84 coordinates
- `lon-lat-ID` - Combined identifier

**Basin Distribution:**
- **La Risle:** 12 stations (e.g., La Risle à Rai, La Risle à Brionne, etc.)
- **La Eure:** 13 stations (e.g., L'Eure à Chartres, L'Eure à Louviers, etc.)
- **Tributary:** 1 station (Le bras de l'Eure à Lormaye)

**Sample Stations:**
```
H602021010 - La Risle à Rai (48.749°N, 0.582°E)
H605641401 - La Risle à Barquet (49.034°N, 0.808°E)
H620021010 - La Risle à Brionne
H4202010  - L'Eure à Chartres
H4522010  - L'Eure à Louviers
```

### 2. Selected Stations (`selected_stations.csv`)

**File:** `selected_stations.csv`
**Rows:** 27 stations
**Content:** List of selected station codes for analysis

### 3. IDPR Data (`IDPR_values.csv`)

**File:** `IDPR_values.csv`
**Source:** BRGM (Bureau de Recherches Géologiques et Minières)
**Rows:** 27 stations
**Purpose:** Infiltration vs runoff tendency (KARST indicator)

**Columns:**
- `fid` - Feature ID
- `X`, `Y` - Original coordinates
- `X_lambert`, `Y_lambert` - Lambert 93 coordinates
- `station_id` - Station identifier
- `basin_name` - "La Risle" or "La Eure"
- `latitude`, `longitude` - WGS84 coordinates
- `IDPR` - **Infiltration Dominance vs Precipitation Runoff index**
  - Higher values → More infiltration (karst behavior)
  - Lower values → More surface runoff

**IDPR Range in Dataset:**
- La Risle: 885 - 2000
- Indicates varying karst influence across the basin

### 4. ADES Groundwater Data

**Source:** ADES (Accès aux Données sur les Eaux Souterraines)
**Purpose:** Karst groundwater connectivity, subsurface flow pathways

#### 4.1 Groundwater Stations (`ADES/groundwater_stations.csv`)

**Rows:** 114 groundwater monitoring wells in watershed
**Columns:**
- `code_bss` - BSS code (unique well identifier)
- `urn_bss` - URN identifier
- `date_debut_mesure`, `date_fin_mesure` - Measurement period
- `code_commune_insee`, `nom_commune` - Municipality
- `x`, `y` - Coordinates
- `bss_id` - Short BSS ID
- `altitude_station` - Well altitude (m)
- `nb_mesures_piezo` - Number of piezometric measurements
- `code_departement`, `nom_departement` - Department (61=Orne, 27=Eure)
- `profondeur_investigation` - Investigation depth (m)

**Sample Wells:**
- BSS000TTRG (Le Mage, Orne) - 137 measurements, depth 38m, altitude 231m
- BSS000TTRB (Longny les Villages) - 9 measurements, depth 55m, altitude 235m

#### 4.2 Groundwater Levels (`ADES/groundwater_levels_watershed.csv`)

**Rows:** 272,379 piezometric measurements
**Period:** Various (e.g., 2013-present for some wells)
**Columns:**
- `code_bss`, `bss_id`, `urn_bss` - Well identifiers
- `date_mesure` - Measurement date
- `timestamp_mesure` - Unix timestamp
- `niveau_nappe_eau` - Groundwater level (m above reference)
- `profondeur_nappe` - Depth to water table (m below surface)
- `mode_obtention` - Measurement method
- `statut` - Data status ("Donnée contrôlée niveau 2")
- `qualification` - Data quality ("Correcte", etc.)
- `code_producteur`, `nom_producteur` - Data producer

**Note:** This is time series data for groundwater levels, crucial for understanding subsurface karst connectivity.

#### 4.3 ADES Metadata (`ADES/stations_metadata.csv`)

**Rows:** 10,001 stations (full ADES database, not filtered)
**Purpose:** Reference metadata for all groundwater stations in France

### 5. Watershed Shapefiles

#### 5.1 La Risle Watershed (`Watersehd_risle/Watershed_Risle.shp`)

**CRS:** EPSG:2154 (Lambert 93 - French national projection)
**Features:** 1 multipolygon
**Attributes:**
- `ID_INTERNE` - Internal ID (e.g., Sav21_226)
- `ID_FONCTIO` - Functional ID
- `NOM` - Watershed name
- `SITUATION_` - Situation/status

#### 5.2 La Eure Watershed (`Watershed_Eure/Watershed_Eure.shp`)

**CRS:** EPSG:2154 (Lambert 93)
**Features:** 1 multipolygon
**Attributes:** Same as La Risle

#### 5.3 Merged Watershed (`Risle_Eure_watershed/Risle_Eure_Watershed.shp`)

**CRS:** EPSG:2154 (Lambert 93)
**Features:** 1 multipolygon (merged boundary)
**Attributes:**
- `FID` - Feature ID
- `geometry` - Geometry

**Note:** This merged shapefile covers both La Risle and La Eure basins for basin-wide analysis.

### Hydrometric Data (Hub'Eau API)

**Status:** Downloaded and available datasets/hydrometric
**Expected:** JSON and CSV files for 27 stations (1960-2026)

**Historical Statistics (Daily/Monthly):**
- `QmnJ` / `QmM` - Mean discharge (m³/s)
- `QIXnJ` / `QIXM` - Maximum instantaneous discharge (m³/s)
- `QINnJ` / `QINM` - Minimum instantaneous discharge (m³/s)
- `HIXnJ` / `HIXM` - Maximum water level (m)

**Real-Time Observations (5-min interval, ~1 month):**
- `H` - Water level (m)
- `Q` - River discharge (m³/s)

### Meteorological Data (SAFRAN Reanalysis)

**Status:** datasets/safran
**Expected:** JSON and CSV files for 27 stations (1960-2026)

**Variables (12):**
- `DLI_Q` - Atmospheric radiation (W/m²)
- `DRAINC_Q` - Drainage (mm)
- `ETP_Q` - Potential evapotranspiration (mm)
- `FF_Q` - Wind speed (m/s)
- `HU_Q` - Relative humidity (%)
- `PRELIQ_Q` - Liquid precipitation (mm)
- `PRENEI_Q` - Solid precipitation (mm)
- `RESR_NEIGE_Q` - Snowpack water equivalent (mm)
- `RUNC_Q` - Runoff (mm)
- `SSI_Q` - Visible radiation (W/m²)
- `SWI_Q` - Soil moisture index
- `T_Q` - Average air temperature (°C)

### DEM and Topographic Data

**Status:** datasets/station_elevations.csv
**Required for:**
- Elevation statistics (mean, min, max)
- Basin slope calculation
- Drainage density
- Topographic Wetness Index (TWI)
- River network topology extraction (QGIS hydrology tools)

**Recommended Source:**
- IGN RGE ALTI (French national DEM, 1m or 5m resolution)
- EU-DEM (25m resolution)

### River Network Topology

**Status:** ❌ Not yet in repository
**Required:** `station_edges.csv` for graph construction

**Expected Format:**
```csv
source,target,distance,elevation_gradient,basin_id
H602021010,H605641401,15.2,10.0,0
H605641401,H620021010,12.5,-5.0,0
...
```

**Generation Method:**
- Use QGIS DEM hydrology processing tools
- Extract upstream-downstream relationships
- Calculate river distances and elevation gradients
- Ensure no cross-basin edges (La Risle ≠ La Eure)

### Basin Attributes (To Be Extracted)

**Status:** ❌ Not yet extracted from shapefiles/DEM

**Topological:**
- Elevation, slope, drainage density, TWI, basin area

**Soil Properties:**
- Soil type, texture, hydraulic conductivity, porosity, depth

**Land Use:**
- Forest/agricultural/urban cover %, NDVI

**Geological/KARST:**
- Aquifer characteristics, lithology, permeability, sinkhole density

## Data Processing Notes

### Basin Assignment

Based on station names in `station_list.csv`:

**La Risle Stations (basin_id = 0):**
- 12 stations with names starting with "La Risle"
- Station codes: H60*, H62*

**La Eure Stations (basin_id = 1):**
- 13 stations with names starting with "L'Eure"
- Station codes: H42*, H45*, H46*, H47*
- Includes tributary: "Le bras de l'Eure"

### Coordinate Systems

**All geospatial data uses:**
- **Projected:** EPSG:2154 (Lambert 93) for distances/areas
- **Geographic:** EPSG:4326 (WGS84) for lat/lon

**Conversion needed for:**
- Station coordinates (already in both systems in `station_list.csv`)
- Shapefiles (already in EPSG:2154)

### Data Size Summary

| File | Rows | Size | Notes |
|------|------|------|-------|
| `station_list.csv` | 27 | 4.1 KB | Station metadata |
| `selected_stations.csv` | 27 | 338 B | Station codes only |
| `IDPR_values.csv` | 27 | 2.6 KB | KARST infiltration index |
| `ADES/groundwater_stations.csv` | 114 | ~50 KB | Groundwater wells |
| `ADES/groundwater_levels_watershed.csv` | 272,379 | ~50 MB | **Large!** Time series |
| `ADES/stations_metadata.csv` | 10,001 | ~2 MB | Full ADES database |

**⚠️ Note:** `groundwater_levels_watershed.csv` is the largest file (272K rows). Use efficient processing (chunking, filtering by date range).

## Next Steps

1. **Collect Hydrometric Data**
   - Use Hub'Eau API to download discharge/water level data
   - Store in `datasets/hydrometric/` (JSON and CSV)

2. **Collect SAFRAN Data**
   - Download meteorological reanalysis for 27 station coordinates
   - Store in `datasets/safran/` (JSON and CSV)

3. **Acquire DEM**
   - Download IGN RGE ALTI or EU-DEM for study area
   - Store in `datasets/DEM/`

4. **Extract River Topology**
   - Use QGIS DEM hydrology tools
   - Generate `datasets/station_edges.csv`

5. **Extract Basin Attributes**
   - Use DEM to calculate topological features
   - Use soil/land use databases for additional features
   - Store in `datasets/basin_features/`

6. **Process Data Through Pipeline**
   - Run `example_pipeline.py` once all data is available
   - Generate fused datasets, graphs, and tensors

## Data Sources & References

- **Hub'Eau API:** https://hubeau.eaufrance.fr/
- **SAFRAN Reanalysis:** Météo-France
- **ADES:** https://ades.eaufrance.fr/ (groundwater data)
- **BRGM IDPR:** Bureau de Recherches Géologiques et Minières
- **IGN:** https://geoservices.ign.fr/ (DEM, elevation data)
- **Watersheds:** Digitized from official French hydrographic boundaries