ageraustine commited on
Commit
19f7528
·
verified ·
1 Parent(s): 683ebff

Upload folder using huggingface_hub

Browse files
.gitattributes CHANGED
@@ -213,3 +213,5 @@ datasets/bdcharm50/dept_061/GEO050K_HARM_061_P_STRUCT_2154.dbf filter=lfs diff=l
213
  datasets/bdcharm50/dept_061/GEO050K_HARM_061_S_FGEOL_2154.dbf filter=lfs diff=lfs merge=lfs -text
214
  datasets/bdcharm50/dept_061/GEO050K_HARM_061_S_FGEOL_2154.lyr filter=lfs diff=lfs merge=lfs -text
215
  datasets/bdcharm50/dept_061/GEO050K_HARM_061_S_FGEOL_2154.shp filter=lfs diff=lfs merge=lfs -text
 
 
 
213
  datasets/bdcharm50/dept_061/GEO050K_HARM_061_S_FGEOL_2154.dbf filter=lfs diff=lfs merge=lfs -text
214
  datasets/bdcharm50/dept_061/GEO050K_HARM_061_S_FGEOL_2154.lyr filter=lfs diff=lfs merge=lfs -text
215
  datasets/bdcharm50/dept_061/GEO050K_HARM_061_S_FGEOL_2154.shp filter=lfs diff=lfs merge=lfs -text
216
+ docs/images/eure_reach_graph.png filter=lfs diff=lfs merge=lfs -text
217
+ docs/images/explore_view.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -72,9 +72,9 @@ PoC_v1/
72
  │ ├── enrich_reach_graph.py # runs node_features.py against the reach graph
73
  │ ├── compute_cumulative_catchment.py # graph-wide catchment area from BD TOPO polygons
74
  │ ├── diagnose_confluences.py # verify real vs. artifact confluences
 
75
  │ ├── download_bdcavites.py # Géorisques BDCavités (sinkhole/cavity inventory)
76
  │ ├── download_bdcharm.py # BRGM BD Charm-50 harmonized geology, per department
77
- │ ├── fetch_idpr_brgm.py # (unused — see §3.5) live IDPR re-fetch attempt
78
  │ ├── fetch_landcover.py # ESA WorldCover landcover class, real gauges
79
  │ └── fetch_worldcover_ndvi.py # ESA WorldCover NDVI percentile composite
80
 
@@ -103,8 +103,9 @@ PoC_v1/
103
  │ └── graph/
104
  │ ├── build_graph.py # PyG conversion: x_static/x_dynamic split, structural columns
105
  │ ├── build_reach_graph.py # real reach topology: confluences, splits/rejoins, MultiDiGraph
106
- │ ├── node_features.py # pulls every loader into one feature table
107
- ── physics_losses.py # confluence/split-rejoin/routing/water-balance loss terms
 
108
 
109
  ├── datasets/ # not checked in; populated by the scripts above
110
  │ ├── station_list.csv # raw station roster (X, Y, names, INSEE, etc.)
@@ -263,24 +264,55 @@ real gauge rows (verified: gauge codes, BD TOPO hydrographic node IDs, and
263
  virtual-node marker strings occupy structurally distinct namespaces, so a
264
  left-merge on `station_code` can never mislabel a confluence or virtual node).
265
 
266
- ### 2.3 Static vs. dynamic features
267
 
268
- `build_pyg_graph` also splits every feature by physical temporal nature:
269
 
270
  - **`data.x_static`** / **`data.static_feature_names`** — genuinely
271
- time-invariant: elevation, IDPR, catchment area, landcover, coordinates.
 
272
  - **`data.x_dynamic`** / **`data.dynamic_feature_names`** — physically
273
- time-varying quantities: climate, groundwater level/depth, NDVI.
274
-
275
- `data.x` remains the full combined tensor unchanged; the split is additional,
276
- not a replacement. **Important limitation, stated plainly**: "dynamic" here
277
- still means *one period-aggregated number per node* (mean/sum over the whole
278
- date range, or a single well reading), not a real `[n_nodes, T]` time series.
279
- The split makes the physical distinction explicit and gives a clean seam for
280
- a future temporal pipeline to slot into, but building that pipeline genuine
281
- daily/whatever-resolution series per node, aligned across sources is
282
- separate, larger, unbuilt work. `physics_losses.py`'s `routing_consistency_loss`
283
- specifically needs that real time dimension and has nothing to consume yet.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
284
 
285
  ### 2.4 Date-range filtering
286
 
@@ -312,7 +344,7 @@ Four constraints, each tied to real graph structure, not generic:
312
  |---|---|---|
313
  | `confluence_mass_balance_loss` | `Q_confluence ≈ sum(Q_upstream_branches)` — new mass genuinely enters | `is_confluence` nodes |
314
  | `split_rejoin_conservation_loss` | `Q_split ≈ Q_rejoin` — same water, no new mass | paired `braid_id` nodes |
315
- | `routing_consistency_loss` | `Q_downstream[t] ≈ Q_upstream[t - lag]`, lag from real `distance_km`/slope | every edge (needs `[n_nodes, T]` see §2.3) |
316
  | `water_balance_loss` | `P - ET - Q - ΔS ≈ 0` in volume terms | nodes with `cumulative_catchment_area_km2` |
317
 
318
  Confluence and split/rejoin are deliberately different constraints, not one
@@ -326,6 +358,27 @@ zero, a named steady-state approximation — this project has no direct
326
  basin-wide storage measurement, only sparse well *levels*, which aren't the
327
  same thing.
328
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
329
  ### 2.6 Two graphs, not one
330
 
331
  `build_pyg_graphs_per_basin()` returns `{0: eure_graph, 1: risle_graph}`,
@@ -346,6 +399,14 @@ count (verified fast at real scale: 0.29s to build a figure for ~2,900
346
  edges) — specifically for visually confirming the topology looks like a real
347
  river network before trusting it as model input.
348
 
 
 
 
 
 
 
 
 
349
  ---
350
 
351
  ## 3. Datasets
@@ -382,10 +443,7 @@ about:
382
  - Everything else with no data is unexplained from the name alone and worth a
383
  direct check on Hub'Eau's site before assuming it's just a gap.
384
 
385
- Of the 8 stations with any `QmnJ` discharge data at all, only **6 have real
386
- observations within the project's 2013–2026 date window** (§2.4) — worth
387
- knowing before assuming "8 gauged stations" translates directly into training
388
- examples.
389
 
390
  ### 3.2 Hydrometric data (`hydrometric/`, via `scripts/download_hubeau.py`)
391
 
@@ -467,24 +525,19 @@ call per file instead.
467
 
468
  BRGM's *Indice de Développement et de Persistance des Réseaux* — an
469
  infiltration-vs-runoff tendency index, and the closest thing this project has
470
- to a real soil/drainage covariate (see §3.9 for why it's standing in for soil
471
- data specifically, not just conveniently similar). The file used here is
472
- already one row per station (`station_id` matching `station_code` exactly,
473
- verified 1:1 against all 27 stations), so `node_features.py` does a direct ID
474
- join when possible rather than nearest-neighbor search, falling back to
475
- spatial nearest-neighbor for any station code that isn't an exact match
476
- (which is every non-gauge reach-graph node, and a real, minor precision
477
- trade-off worth knowing every gauge too, once the table also contains
478
- non-gauge codes, since the exact-match path requires the *entire* table to
479
- match IDPR's station list).
480
-
481
- A live re-fetch from BRGM's own geoservice (`scripts/fetch_idpr_brgm.py`,
482
- `geoservices.brgm.fr/geologie`, `GetFeatureInfo` point queries against the
483
- `IDPR_50M` raster layer) was attempted, to get fresher values and eventually
484
- cover the full reach graph rather than just the 27 gauges. **It failed** in
485
- testing and the project is currently using the original, already-uploaded
486
- `idpr.csv` instead — not resolved further, since the existing file already
487
- gives real, usable IDPR coverage for every gauge.
488
 
489
  ### 3.6 Catchment area — two independent sources
490
 
@@ -600,50 +653,24 @@ InfoTerre path.
600
  ### 3.10 Landcover and NDVI (`scripts/fetch_landcover.py`, `scripts/fetch_worldcover_ndvi.py`)
601
 
602
  ESA WorldCover, sampled at real gauge points from the public AWS S3 Cloud-
603
- Optimized GeoTIFFs — **not** the Terrascope WMS, which a source dated within
604
- the last month of this project's active development reported actively resets
605
- connections from non-browser HTTP clients (TLS fingerprinting, confirmed
606
- across multiple tools and User-Agents, not a coding problem to work around),
607
- and separately was slated for full phase-out already past by the time this
608
- was checked.
609
 
610
  Landcover classification uses the product's 3°×3° tile grid; every real
611
  station coordinate falls inside exactly one tile (`N48E000`), verified
612
  directly against all 27 real coordinates. NDVI uses the *annual composites'*
613
  1°×1° tile grid instead — genuinely different from the classification grid,
614
  looked up per-station via VITO's own authoritative tile-index grid file
615
- (`esa_worldcover_grid_composites.fgb`) rather than a second hand-guessed S3
616
- key pattern. Both need `AWS_NO_SIGN_REQUEST=YES` for `s3://`-scheme tile URLs
617
  specifically — a plain HTTPS URL to the same public bucket needs no signing
618
- at all, which is why the landcover script (HTTPS) worked without this while
619
- the NDVI grid's returned URLs (`s3://`) initially failed on AWS credential
620
- errors despite the bucket being fully public.
621
 
622
  Both currently cover only the 27 real gauges (exact `station_code` match),
623
  same limitation as Hub'Eau's `catchment_area_km2` before the cumulative-BD-
624
  TOPO fix (§3.6) — extending either script to the full reach graph is
625
  unstarted work, not a design decision.
626
 
627
- ### 3.11 What was tried and didn't work
628
-
629
- **SoilGrids (ISRIC)** — confirmed non-functional directly, not from a stale
630
- search result: even a bare `lon`/`lat` query to the live REST API returned
631
- `422` consistently, consistent with ISRIC's own currently-posted "temporarily
632
- paused" service notice.
633
-
634
- **INRAE's national soil survey (RRP/BDGSF)** — not a uniformly-accessible
635
- source at all. Access is explicitly described as depending on regional/
636
- departmental "référents" (varies by department, sometimes needs a formal
637
- agreement with the regional chamber of agriculture), and INRAE's own
638
- documentation states outright that the more detailed scale "n'est pas encore
639
- en accès libre." IDPR (§3.5) and BD Charm-50 geology (§3.9) are this
640
- project's actual substitutes for the hydrologically-relevant part of what
641
- soil data would otherwise provide — not literal soil texture data, but IDPR
642
- specifically is an *integrated hydrological behavior* indicator (infiltration
643
- tendency), arguably more directly useful for a streamflow model than a raw
644
- soil property map would be on its own.
645
-
646
- ### 3.12 Centerline generation
647
 
648
  **Only relevant to the older single-chain pipeline** (`build_surface_edges`,
649
  still available for direct comparison/debugging) — the reach graph (§2)
@@ -751,7 +778,32 @@ pre-commit or CI gate if that's ever set up.
751
 
752
  ---
753
 
754
- ## 6. Running things
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
755
 
756
  Data acquisition (from repo root, in roughly dependency order):
757
 
@@ -778,6 +830,13 @@ python -m scripts.diagnose_confluences --data-root datasets --basin eure
778
  python -m scripts.diagnose_confluences --data-root datasets --basin risle
779
  ```
780
 
 
 
 
 
 
 
 
781
  Landcover / NDVI, real gauges only (needs `rasterio`, and `geopandas` for NDVI's
782
  tile lookup):
783
 
 
72
  │ ├── enrich_reach_graph.py # runs node_features.py against the reach graph
73
  │ ├── compute_cumulative_catchment.py # graph-wide catchment area from BD TOPO polygons
74
  │ ├── diagnose_confluences.py # verify real vs. artifact confluences
75
+ │ ├── build_dynamic_tensors.py # genuine [n_nodes, T] tensors, wired into physics_losses.py
76
  │ ├── download_bdcavites.py # Géorisques BDCavités (sinkhole/cavity inventory)
77
  │ ├── download_bdcharm.py # BRGM BD Charm-50 harmonized geology, per department
 
78
  │ ├── fetch_landcover.py # ESA WorldCover landcover class, real gauges
79
  │ └── fetch_worldcover_ndvi.py # ESA WorldCover NDVI percentile composite
80
 
 
103
  │ └── graph/
104
  │ ├── build_graph.py # PyG conversion: x_static/x_dynamic split, structural columns
105
  │ ├── build_reach_graph.py # real reach topology: confluences, splits/rejoins, MultiDiGraph
106
+ │ ├── node_features.py # pulls every loader into one feature table (static, one row/node)
107
+ ── dynamic_features.py # genuine [n_nodes, T] series: discharge, groundwater, climate
108
+ │ └── physics_losses.py # confluence/split-rejoin/routing/water-balance loss terms, NaN-masked
109
 
110
  ├── datasets/ # not checked in; populated by the scripts above
111
  │ ├── station_list.csv # raw station roster (X, Y, names, INSEE, etc.)
 
264
  virtual-node marker strings occupy structurally distinct namespaces, so a
265
  left-merge on `station_code` can never mislabel a confluence or virtual node).
266
 
267
+ ### 2.3 Static vs. dynamic features — and a real temporal pipeline
268
 
269
+ `build_pyg_graph` splits every feature by physical temporal nature:
270
 
271
  - **`data.x_static`** / **`data.static_feature_names`** — genuinely
272
+ time-invariant: elevation, IDPR, catchment area, landcover, geology,
273
+ cavité proximity, coordinates.
274
  - **`data.x_dynamic`** / **`data.dynamic_feature_names`** — physically
275
+ time-varying quantities, still as a single period-aggregated number here
276
+ (mean/sum over the whole date range, or a latest well reading) — this
277
+ tensor is a *static snapshot* of dynamic-natured quantities, not a real
278
+ series. `data.x` remains the full combined tensor unchanged; the split is
279
+ additional, not a replacement.
280
+
281
+ **A genuine `[n_nodes, T]` series exists separately**, in
282
+ `src/graph/dynamic_features.py``build_discharge_timeseries`,
283
+ `build_groundwater_timeseries`, `build_climate_timeseries`built
284
+ specifically because `physics_losses.py`'s `routing_consistency_loss` needs a
285
+ real time dimension and had nothing to consume before this existed. Same
286
+ loaders as everywhere else, no re-fetching; the only difference is that these
287
+ functions pivot to wide `[date x station_code]` form instead of collapsing to
288
+ one aggregate the way `node_features.py`'s `add_*_features` do.
289
+
290
+ Two real challenges, not incidental engineering:
291
+
292
+ - **Groundwater** reports on wildly irregular schedules (confirmed: 13
293
+ different "latest dates" among 18 real wells within 20 km of one station).
294
+ Each well is resampled to a common daily grid via forward-fill (a water
295
+ table changes slowly — carrying the last known reading forward is standard
296
+ practice, not an invented shortcut) *before* spatial averaging, not after —
297
+ averaging raw irregular readings per exact calendar date is exactly what
298
+ made the static version undercount real coverage by 5–10x before that was
299
+ fixed (§3.3). The spatial neighbor-set per node is computed once, reused
300
+ across every date — verified fast at real reach-graph scale (21.6s for
301
+ 2,900 nodes × 14 years daily, real 272k-row ADES data).
302
+ - **Discharge** deliberately does *not* get forward-filled the way
303
+ groundwater does — a missing daily reading stays missing, since discharge
304
+ genuinely changes day to day and papering over a gap with yesterday's
305
+ value would misrepresent it.
306
+
307
+ `scripts/build_dynamic_tensors.py` is the actual wiring: builds these tensors
308
+ for a basin, saves them, and feeds discharge directly into
309
+ `routing_consistency_loss` alongside `build_routing_index` — the real
310
+ integration point, not just parallel unconnected pieces.
311
+
312
+ **Climate is untested against real data** — no real ERA5/`safran_path` files
313
+ were available to validate `build_climate_timeseries` against in this
314
+ project's development environment; the logic mirrors the already-tested
315
+ discharge pivot directly, but verify the real output before trusting it.
316
 
317
  ### 2.4 Date-range filtering
318
 
 
344
  |---|---|---|
345
  | `confluence_mass_balance_loss` | `Q_confluence ≈ sum(Q_upstream_branches)` — new mass genuinely enters | `is_confluence` nodes |
346
  | `split_rejoin_conservation_loss` | `Q_split ≈ Q_rejoin` — same water, no new mass | paired `braid_id` nodes |
347
+ | `routing_consistency_loss` | `Q_downstream[t] ≈ Q_upstream[t - lag]`, lag from real `distance_km`/slope | every edge, real `[n_nodes, T]` via `dynamic_features.py` (§2.3) |
348
  | `water_balance_loss` | `P - ET - Q - ΔS ≈ 0` in volume terms | nodes with `cumulative_catchment_area_km2` |
349
 
350
  Confluence and split/rejoin are deliberately different constraints, not one
 
358
  basin-wide storage measurement, only sparse well *levels*, which aren't the
359
  same thing.
360
 
361
+ **All four are NaN-masked, not just tolerant of complete data.** Real ground-
362
+ truth Q is ~93.5% `NaN` by construction (only real gauges with real
363
+ observations ever have a value — confirmed against the real discharge
364
+ tensor) — that's the normal shape of the data, not a rare edge case. The
365
+ shared `_mse` helper every loss function uses previously computed a plain
366
+ mean, so a single `NaN` anywhere in a residual silently poisoned the *entire*
367
+ loss to `NaN` — confirmed as a real, not hypothetical, failure: calling
368
+ `routing_consistency_loss` directly on the real discharge tensor returned
369
+ `NaN` before this was fixed. `_mse` now masks `NaN` out before averaging
370
+ (returning `NaN` only if truly nothing usable exists at all, which is a
371
+ real "no data" signal worth keeping, not silently averaging to a misleading
372
+ `0`) — verified with the exact real scenario that first exposed the bug:
373
+ `routing_consistency_loss` on the real discharge tensor now returns a real
374
+ number instead of `NaN`.
375
+
376
+ This also means these functions are directly usable as a diagnostic against
377
+ real historical data alone, independent of any trained model — e.g.
378
+ "does real observed discharge at two connected gauges actually satisfy the
379
+ routing physics" — a genuine, model-free sanity check on both the physics
380
+ math and the graph topology, not just a training-time loss term.
381
+
382
  ### 2.6 Two graphs, not one
383
 
384
  `build_pyg_graphs_per_basin()` returns `{0: eure_graph, 1: risle_graph}`,
 
399
  edges) — specifically for visually confirming the topology looks like a real
400
  river network before trusting it as model input.
401
 
402
+ ![La Risle reach graph — network validation view, real confluences as diamonds, gauges elevation-colored](docs/images/risle_reach_graph.png)
403
+
404
+
405
+ ![La Eure reach graph, same view](docs/images/eure_reach_graph.png)
406
+
407
+
408
+ ![Explore view: clicking along the river resolves to the nearest gauge and shows its real discharge/water-level/rating-curve plots](docs/images/explore_view.png)
409
+
410
  ---
411
 
412
  ## 3. Datasets
 
443
  - Everything else with no data is unexplained from the name alone and worth a
444
  direct check on Hub'Eau's site before assuming it's just a gap.
445
 
446
+
 
 
 
447
 
448
  ### 3.2 Hydrometric data (`hydrometric/`, via `scripts/download_hubeau.py`)
449
 
 
525
 
526
  BRGM's *Indice de Développement et de Persistance des Réseaux* — an
527
  infiltration-vs-runoff tendency index, and the closest thing this project has
528
+ to a real soil/drainage covariate. It's an *integrated hydrological behavior*
529
+ indicator (infiltration tendency), not raw soil texture data, but arguably
530
+ more directly useful for a streamflow model than a texture map would be on
531
+ its own paired with BD Charm-50 geology (§3.9) for the broader hydrological
532
+ context soil data would otherwise provide. The file used here is already one
533
+ row per station (`station_id` matching `station_code` exactly, verified 1:1
534
+ against all 27 stations), so `node_features.py` does a direct ID join when
535
+ possible rather than nearest-neighbor search, falling back to spatial
536
+ nearest-neighbor for any station code that isn't an exact match (every
537
+ non-gauge reach-graph node, and — a real, minor precision trade-off worth
538
+ knowing — every gauge too, once the table also contains non-gauge codes,
539
+ since the exact-match path requires the *entire* table to match IDPR's
540
+ station list).
 
 
 
 
 
541
 
542
  ### 3.6 Catchment area — two independent sources
543
 
 
653
  ### 3.10 Landcover and NDVI (`scripts/fetch_landcover.py`, `scripts/fetch_worldcover_ndvi.py`)
654
 
655
  ESA WorldCover, sampled at real gauge points from the public AWS S3 Cloud-
656
+ Optimized GeoTIFFs.
 
 
 
 
 
657
 
658
  Landcover classification uses the product's 3°×3° tile grid; every real
659
  station coordinate falls inside exactly one tile (`N48E000`), verified
660
  directly against all 27 real coordinates. NDVI uses the *annual composites'*
661
  1°×1° tile grid instead — genuinely different from the classification grid,
662
  looked up per-station via VITO's own authoritative tile-index grid file
663
+ (`esa_worldcover_grid_composites.fgb`) rather than a hand-guessed S3 key
664
+ pattern. Both need `AWS_NO_SIGN_REQUEST=YES` for `s3://`-scheme tile URLs
665
  specifically — a plain HTTPS URL to the same public bucket needs no signing
666
+ at all.
 
 
667
 
668
  Both currently cover only the 27 real gauges (exact `station_code` match),
669
  same limitation as Hub'Eau's `catchment_area_km2` before the cumulative-BD-
670
  TOPO fix (§3.6) — extending either script to the full reach graph is
671
  unstarted work, not a design decision.
672
 
673
+ ### 3.11 Centerline generation
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
674
 
675
  **Only relevant to the older single-chain pipeline** (`build_surface_edges`,
676
  still available for direct comparison/debugging) — the reach graph (§2)
 
778
 
779
  ---
780
 
781
+ ## 6. Known limitations and open questions
782
+
783
+ - **`cumulative_catchment_area_km2` underestimates the largest catchments**
784
+ by up to ~25%, traced to the BD TOPO pull's original bounding box having an
785
+ insufficient southern margin. The bbox has been widened in
786
+ `download_bdtopo_hydro.py`; the full re-pull-and-rebuild chain needs
787
+ re-running for this to actually resolve. Treat the largest catchments'
788
+ values as approximate until then.
789
+ - **Climate's genuine time series (`build_climate_timeseries`) is untested
790
+ against real data** — discharge and groundwater's equivalents are; verify
791
+ climate's real output before relying on it.
792
+ - **Landcover and NDVI only cover the 27 real gauges**, not the full reach
793
+ graph — same scope `catchment_area_km2` had before its cumulative-BD-TOPO
794
+ extension.
795
+ - **No model exists yet.** This repo builds the graph and the physics-loss
796
+ substrate a model would train against; there is no architecture, forward
797
+ pass, or training loop here.
798
+ - **The karst losing-reach flag rests on naming evidence and a BDCavités
799
+ cross-check** (§3.7–3.8), not a fully confirmed BD TOPO classification.
800
+ - **The groundwater-well BDLISA aquifer-unit field is unused.** First place
801
+ to look if a subsurface connectivity edge is ever justified with real
802
+ evidence rather than proximity.
803
+
804
+ ---
805
+
806
+ ## 7. Running things
807
 
808
  Data acquisition (from repo root, in roughly dependency order):
809
 
 
830
  python -m scripts.diagnose_confluences --data-root datasets --basin risle
831
  ```
832
 
833
+ Build genuine `[n_nodes, T]` dynamic tensors and verify the physics-loss wiring:
834
+
835
+ ```bash
836
+ python -m scripts.build_dynamic_tensors --data-root datasets --basin risle
837
+ python -m scripts.build_dynamic_tensors --data-root datasets --basin eure
838
+ ```
839
+
840
  Landcover / NDVI, real gauges only (needs `rasterio`, and `geopandas` for NDVI's
841
  tile lookup):
842
 
datasets/README.md ADDED
@@ -0,0 +1,320 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Datasets Documentation
2
+
3
+ **Project:** Physics-Informed KARST-Aware GNN for Streamflow Prediction
4
+ **Study Area:** La Risle and La Eure River Systems, Seine Basin, France
5
+ **Period:** 1960-2026 (historical data)
6
+
7
+ ## Directory Structure
8
+
9
+ ```
10
+ datasets/
11
+ ├── station_list.csv # 27 stations metadata (coordinates, names)
12
+ ├── selected_stations.csv # 27 selected station codes
13
+ ├── IDPR_values.csv # BRGM infiltration/runoff index (27 stations)
14
+ ├── ADES/ # Groundwater data (KARST connectivity)
15
+ │ ├── groundwater_stations.csv # 114 groundwater monitoring wells
16
+ │ ├── groundwater_levels_watershed.csv# 272,379 groundwater level measurements
17
+ │ ├── stations_metadata.csv # 10,001 stations (full ADES database)
18
+ │ └── ADES_data_fetching.py # Data collection script
19
+ ├── Watersehd_risle/ # La Risle watershed boundary
20
+ │ └── Watershed_Risle.shp # Shapefile (EPSG:2154)
21
+ ├── Watershed_Eure/ # La Eure watershed boundary
22
+ │ └── Watershed_Eure.shp # Shapefile (EPSG:2154)
23
+ ├── Risle_Eure_watershed/ # Merged watershed boundary
24
+ │ └── Risle_Eure_Watershed.shp # Combined shapefile (EPSG:2154)
25
+ └── River Images/ # Reference maps
26
+ ├── Carte-de-la_Seine-390x305.png
27
+ ├── Eure_(rivière).png
28
+ └── Risle.png
29
+ ```
30
+
31
+ ## Available Data
32
+
33
+ ### 1. Hydrometric Stations (`station_list.csv`)
34
+
35
+ **File:** `station_list.csv`
36
+ **Rows:** 27 stations
37
+ **Columns:**
38
+ - `X`, `Y` - Lambert 93 coordinates (EPSG:2154)
39
+ - `station_code` - Unique station identifier (e.g., H602021010)
40
+ - `region_name` - NORMANDIE
41
+ - `departament_name` - ORNE or EURE
42
+ - `station_name` - Full station name
43
+ - `municipality` - Municipality name
44
+ - `INSEE` - INSEE code
45
+ - `id_root` - Root identifier
46
+ - `lon`, `lat` - WGS84 coordinates
47
+ - `lon-lat-ID` - Combined identifier
48
+
49
+ **Basin Distribution:**
50
+ - **La Risle:** 12 stations (e.g., La Risle à Rai, La Risle à Brionne, etc.)
51
+ - **La Eure:** 13 stations (e.g., L'Eure à Chartres, L'Eure à Louviers, etc.)
52
+ - **Tributary:** 1 station (Le bras de l'Eure à Lormaye)
53
+
54
+ **Sample Stations:**
55
+ ```
56
+ H602021010 - La Risle à Rai (48.749°N, 0.582°E)
57
+ H605641401 - La Risle à Barquet (49.034°N, 0.808°E)
58
+ H620021010 - La Risle à Brionne
59
+ H4202010 - L'Eure à Chartres
60
+ H4522010 - L'Eure à Louviers
61
+ ```
62
+
63
+ ### 2. Selected Stations (`selected_stations.csv`)
64
+
65
+ **File:** `selected_stations.csv`
66
+ **Rows:** 27 stations
67
+ **Content:** List of selected station codes for analysis
68
+
69
+ ### 3. IDPR Data (`IDPR_values.csv`)
70
+
71
+ **File:** `IDPR_values.csv`
72
+ **Source:** BRGM (Bureau de Recherches Géologiques et Minières)
73
+ **Rows:** 27 stations
74
+ **Purpose:** Infiltration vs runoff tendency (KARST indicator)
75
+
76
+ **Columns:**
77
+ - `fid` - Feature ID
78
+ - `X`, `Y` - Original coordinates
79
+ - `X_lambert`, `Y_lambert` - Lambert 93 coordinates
80
+ - `station_id` - Station identifier
81
+ - `basin_name` - "La Risle" or "La Eure"
82
+ - `latitude`, `longitude` - WGS84 coordinates
83
+ - `IDPR` - **Infiltration Dominance vs Precipitation Runoff index**
84
+ - Higher values → More infiltration (karst behavior)
85
+ - Lower values → More surface runoff
86
+
87
+ **IDPR Range in Dataset:**
88
+ - La Risle: 885 - 2000
89
+ - Indicates varying karst influence across the basin
90
+
91
+ ### 4. ADES Groundwater Data
92
+
93
+ **Source:** ADES (Accès aux Données sur les Eaux Souterraines)
94
+ **Purpose:** Karst groundwater connectivity, subsurface flow pathways
95
+
96
+ #### 4.1 Groundwater Stations (`ADES/groundwater_stations.csv`)
97
+
98
+ **Rows:** 114 groundwater monitoring wells in watershed
99
+ **Columns:**
100
+ - `code_bss` - BSS code (unique well identifier)
101
+ - `urn_bss` - URN identifier
102
+ - `date_debut_mesure`, `date_fin_mesure` - Measurement period
103
+ - `code_commune_insee`, `nom_commune` - Municipality
104
+ - `x`, `y` - Coordinates
105
+ - `bss_id` - Short BSS ID
106
+ - `altitude_station` - Well altitude (m)
107
+ - `nb_mesures_piezo` - Number of piezometric measurements
108
+ - `code_departement`, `nom_departement` - Department (61=Orne, 27=Eure)
109
+ - `profondeur_investigation` - Investigation depth (m)
110
+
111
+ **Sample Wells:**
112
+ - BSS000TTRG (Le Mage, Orne) - 137 measurements, depth 38m, altitude 231m
113
+ - BSS000TTRB (Longny les Villages) - 9 measurements, depth 55m, altitude 235m
114
+
115
+ #### 4.2 Groundwater Levels (`ADES/groundwater_levels_watershed.csv`)
116
+
117
+ **Rows:** 272,379 piezometric measurements
118
+ **Period:** Various (e.g., 2013-present for some wells)
119
+ **Columns:**
120
+ - `code_bss`, `bss_id`, `urn_bss` - Well identifiers
121
+ - `date_mesure` - Measurement date
122
+ - `timestamp_mesure` - Unix timestamp
123
+ - `niveau_nappe_eau` - Groundwater level (m above reference)
124
+ - `profondeur_nappe` - Depth to water table (m below surface)
125
+ - `mode_obtention` - Measurement method
126
+ - `statut` - Data status ("Donnée contrôlée niveau 2")
127
+ - `qualification` - Data quality ("Correcte", etc.)
128
+ - `code_producteur`, `nom_producteur` - Data producer
129
+
130
+ **Note:** This is time series data for groundwater levels, crucial for understanding subsurface karst connectivity.
131
+
132
+ #### 4.3 ADES Metadata (`ADES/stations_metadata.csv`)
133
+
134
+ **Rows:** 10,001 stations (full ADES database, not filtered)
135
+ **Purpose:** Reference metadata for all groundwater stations in France
136
+
137
+ ### 5. Watershed Shapefiles
138
+
139
+ #### 5.1 La Risle Watershed (`Watersehd_risle/Watershed_Risle.shp`)
140
+
141
+ **CRS:** EPSG:2154 (Lambert 93 - French national projection)
142
+ **Features:** 1 multipolygon
143
+ **Attributes:**
144
+ - `ID_INTERNE` - Internal ID (e.g., Sav21_226)
145
+ - `ID_FONCTIO` - Functional ID
146
+ - `NOM` - Watershed name
147
+ - `SITUATION_` - Situation/status
148
+
149
+ #### 5.2 La Eure Watershed (`Watershed_Eure/Watershed_Eure.shp`)
150
+
151
+ **CRS:** EPSG:2154 (Lambert 93)
152
+ **Features:** 1 multipolygon
153
+ **Attributes:** Same as La Risle
154
+
155
+ #### 5.3 Merged Watershed (`Risle_Eure_watershed/Risle_Eure_Watershed.shp`)
156
+
157
+ **CRS:** EPSG:2154 (Lambert 93)
158
+ **Features:** 1 multipolygon (merged boundary)
159
+ **Attributes:**
160
+ - `FID` - Feature ID
161
+ - `geometry` - Geometry
162
+
163
+ **Note:** This merged shapefile covers both La Risle and La Eure basins for basin-wide analysis.
164
+
165
+ ### Hydrometric Data (Hub'Eau API)
166
+
167
+ **Status:** Downloaded and available datasets/hydrometric
168
+ **Expected:** JSON and CSV files for 27 stations (1960-2026)
169
+
170
+ **Historical Statistics (Daily/Monthly):**
171
+ - `QmnJ` / `QmM` - Mean discharge (m³/s)
172
+ - `QIXnJ` / `QIXM` - Maximum instantaneous discharge (m³/s)
173
+ - `QINnJ` / `QINM` - Minimum instantaneous discharge (m³/s)
174
+ - `HIXnJ` / `HIXM` - Maximum water level (m)
175
+
176
+ **Real-Time Observations (5-min interval, ~1 month):**
177
+ - `H` - Water level (m)
178
+ - `Q` - River discharge (m³/s)
179
+
180
+ ### Meteorological Data (SAFRAN Reanalysis)
181
+
182
+ **Status:** datasets/safran
183
+ **Expected:** JSON and CSV files for 27 stations (1960-2026)
184
+
185
+ **Variables (12):**
186
+ - `DLI_Q` - Atmospheric radiation (W/m²)
187
+ - `DRAINC_Q` - Drainage (mm)
188
+ - `ETP_Q` - Potential evapotranspiration (mm)
189
+ - `FF_Q` - Wind speed (m/s)
190
+ - `HU_Q` - Relative humidity (%)
191
+ - `PRELIQ_Q` - Liquid precipitation (mm)
192
+ - `PRENEI_Q` - Solid precipitation (mm)
193
+ - `RESR_NEIGE_Q` - Snowpack water equivalent (mm)
194
+ - `RUNC_Q` - Runoff (mm)
195
+ - `SSI_Q` - Visible radiation (W/m²)
196
+ - `SWI_Q` - Soil moisture index
197
+ - `T_Q` - Average air temperature (°C)
198
+
199
+ ### DEM and Topographic Data
200
+
201
+ **Status:** datasets/station_elevations.csv
202
+ **Required for:**
203
+ - Elevation statistics (mean, min, max)
204
+ - Basin slope calculation
205
+ - Drainage density
206
+ - Topographic Wetness Index (TWI)
207
+ - River network topology extraction (QGIS hydrology tools)
208
+
209
+ **Recommended Source:**
210
+ - IGN RGE ALTI (French national DEM, 1m or 5m resolution)
211
+ - EU-DEM (25m resolution)
212
+
213
+ ### River Network Topology
214
+
215
+ **Status:** ❌ Not yet in repository
216
+ **Required:** `station_edges.csv` for graph construction
217
+
218
+ **Expected Format:**
219
+ ```csv
220
+ source,target,distance,elevation_gradient,basin_id
221
+ H602021010,H605641401,15.2,10.0,0
222
+ H605641401,H620021010,12.5,-5.0,0
223
+ ...
224
+ ```
225
+
226
+ **Generation Method:**
227
+ - Use QGIS DEM hydrology processing tools
228
+ - Extract upstream-downstream relationships
229
+ - Calculate river distances and elevation gradients
230
+ - Ensure no cross-basin edges (La Risle ≠ La Eure)
231
+
232
+ ### Basin Attributes (To Be Extracted)
233
+
234
+ **Status:** ❌ Not yet extracted from shapefiles/DEM
235
+
236
+ **Topological:**
237
+ - Elevation, slope, drainage density, TWI, basin area
238
+
239
+ **Soil Properties:**
240
+ - Soil type, texture, hydraulic conductivity, porosity, depth
241
+
242
+ **Land Use:**
243
+ - Forest/agricultural/urban cover %, NDVI
244
+
245
+ **Geological/KARST:**
246
+ - Aquifer characteristics, lithology, permeability, sinkhole density
247
+
248
+ ## Data Processing Notes
249
+
250
+ ### Basin Assignment
251
+
252
+ Based on station names in `station_list.csv`:
253
+
254
+ **La Risle Stations (basin_id = 0):**
255
+ - 12 stations with names starting with "La Risle"
256
+ - Station codes: H60*, H62*
257
+
258
+ **La Eure Stations (basin_id = 1):**
259
+ - 13 stations with names starting with "L'Eure"
260
+ - Station codes: H42*, H45*, H46*, H47*
261
+ - Includes tributary: "Le bras de l'Eure"
262
+
263
+ ### Coordinate Systems
264
+
265
+ **All geospatial data uses:**
266
+ - **Projected:** EPSG:2154 (Lambert 93) for distances/areas
267
+ - **Geographic:** EPSG:4326 (WGS84) for lat/lon
268
+
269
+ **Conversion needed for:**
270
+ - Station coordinates (already in both systems in `station_list.csv`)
271
+ - Shapefiles (already in EPSG:2154)
272
+
273
+ ### Data Size Summary
274
+
275
+ | File | Rows | Size | Notes |
276
+ |------|------|------|-------|
277
+ | `station_list.csv` | 27 | 4.1 KB | Station metadata |
278
+ | `selected_stations.csv` | 27 | 338 B | Station codes only |
279
+ | `IDPR_values.csv` | 27 | 2.6 KB | KARST infiltration index |
280
+ | `ADES/groundwater_stations.csv` | 114 | ~50 KB | Groundwater wells |
281
+ | `ADES/groundwater_levels_watershed.csv` | 272,379 | ~50 MB | **Large!** Time series |
282
+ | `ADES/stations_metadata.csv` | 10,001 | ~2 MB | Full ADES database |
283
+
284
+ **⚠️ Note:** `groundwater_levels_watershed.csv` is the largest file (272K rows). Use efficient processing (chunking, filtering by date range).
285
+
286
+ ## Next Steps
287
+
288
+ 1. **Collect Hydrometric Data**
289
+ - Use Hub'Eau API to download discharge/water level data
290
+ - Store in `datasets/hydrometric/` (JSON and CSV)
291
+
292
+ 2. **Collect SAFRAN Data**
293
+ - Download meteorological reanalysis for 27 station coordinates
294
+ - Store in `datasets/safran/` (JSON and CSV)
295
+
296
+ 3. **Acquire DEM**
297
+ - Download IGN RGE ALTI or EU-DEM for study area
298
+ - Store in `datasets/DEM/`
299
+
300
+ 4. **Extract River Topology**
301
+ - Use QGIS DEM hydrology tools
302
+ - Generate `datasets/station_edges.csv`
303
+
304
+ 5. **Extract Basin Attributes**
305
+ - Use DEM to calculate topological features
306
+ - Use soil/land use databases for additional features
307
+ - Store in `datasets/basin_features/`
308
+
309
+ 6. **Process Data Through Pipeline**
310
+ - Run `example_pipeline.py` once all data is available
311
+ - Generate fused datasets, graphs, and tensors
312
+
313
+ ## Data Sources & References
314
+
315
+ - **Hub'Eau API:** https://hubeau.eaufrance.fr/
316
+ - **SAFRAN Reanalysis:** Météo-France
317
+ - **ADES:** https://ades.eaufrance.fr/ (groundwater data)
318
+ - **BRGM IDPR:** Bureau de Recherches Géologiques et Minières
319
+ - **IGN:** https://geoservices.ign.fr/ (DEM, elevation data)
320
+ - **Watersheds:** Digitized from official French hydrographic boundaries
docs/images/eure_reach_graph.png ADDED

Git LFS Details

  • SHA256: 7707ebcd482aacbe7413a02aab1ee7c6c2290a319b0ea42ea5dcac084c753819
  • Pointer size: 131 Bytes
  • Size of remote file: 102 kB
docs/images/explore_view.png ADDED

Git LFS Details

  • SHA256: ac3b51b0c8fcfff9b3f79ef3811fef435192ed8b12103ee0d762e45fbe5db443
  • Pointer size: 131 Bytes
  • Size of remote file: 400 kB
docs/images/risle_reach_graph.png ADDED