--- title: La Eure / La Risle Hydrometric GNN emoji: πŸ“ˆ colorFrom: indigo colorTo: indigo sdk: gradio sdk_version: 5.47.0 app_file: src/gradio_app.py pinned: true --- # La Eure / La Risle Hydrometric GNN A physics-informed graph neural network that predicts streamflow (discharge, water level) at gauged and ungauged points along two Normandy rivers, La Eure and La Risle β€” each modeled as a real reach-based network (confluences, braided splits/rejoins, ~4,500 nodes per basin including virtual infill points), not a single chain of gauges, with covariates pulled from nine independent data sources. ```mermaid flowchart LR hubeau["Hub'Eau
discharge Β· water level
catchment area"] ades["ADES
groundwater levels"] era5["Copernicus ERA5
climate reanalysis"] otd["Open Topo Data
station elevation"] brgm["BRGM
IDPR Β· BD Charm-50 geology"] bdtopo["IGN BD TOPO
real reach topology + catchment polygons"] bdcav["GΓ©orisques
BDCavitΓ©s (sinkholes)"] wc["ESA WorldCover
landcover Β· NDVI"] bdtopo --> brg["build_reach_graph.py
real confluences, splits/rejoins,
gauge snapping"] brg --> brgs["build_reach_graphs.py
~4,500 nodes/basin"] hubeau --> nf ades --> nf era5 --> nf otd --> nf brgm --> nf bdcav --> nf wc --> nf brgs --> nf["node_features.py /
enrich_reach_graph.py
date-filtered 2013-2026"] bdtopo --> cc["compute_cumulative_catchment.py
graph-wide catchment area"] cc --> nf nf --> pyg["build_pyg_graph
x_static / x_dynamic split"] pyg --> phys["physics_losses.py
confluence Β· split-rejoin Β·
routing Β· water balance"] pyg --> app["src/app.py
Streamlit explorer +
network validation view"] pyg --> testsuite["test_build_graph.py
validation"] ``` --- ## 1. Repository layout ``` PoC_v1/ β”œβ”€β”€ scripts/ # one-off download / extraction / build scripts β”‚ β”œβ”€β”€ download_hubeau.py # discharge + water level, Hub'Eau API v2 β”‚ β”œβ”€β”€ download_elevation.py # point elevations, Open Topo Data β”‚ β”œβ”€β”€ download_era5_sample.py # ERA5 sanity-check pull (Jan 2020 only) β”‚ β”œβ”€β”€ download_era5_full.py # ERA5 1960–2026, split instant/accum vars β”‚ β”œβ”€β”€ extract_era5.py # unzips CDS API's zipped NetCDF output β”‚ β”œβ”€β”€ download_catchment.py # Hub'Eau referentiel/sites -> surface_bv β”‚ β”œβ”€β”€ download_bdtopo_hydro.py # IGN WFS -> tronΓ§ons, surfaces, catchments β”‚ β”œβ”€β”€ analyze_bdtopo_hydro.py # centerline export + karst check β”‚ β”œβ”€β”€ run_bdtopo_checks.py # karst + catchment cross-check, one shot β”‚ β”œβ”€β”€ cross_check_catchments.py # spatial join: station -> containing polygon β”‚ β”œβ”€β”€ build_reach_graphs.py # real reach-based topology, both basins β”‚ β”œβ”€β”€ enrich_reach_graph.py # runs node_features.py against the reach graph β”‚ β”œβ”€β”€ compute_cumulative_catchment.py # graph-wide catchment area from BD TOPO polygons β”‚ β”œβ”€β”€ diagnose_confluences.py # verify real vs. artifact confluences β”‚ β”œβ”€β”€ build_dynamic_tensors.py # genuine [n_nodes, T] tensors, wired into physics_losses.py β”‚ β”œβ”€β”€ download_bdcavites.py # GΓ©orisques BDCavitΓ©s (sinkhole/cavity inventory) β”‚ β”œβ”€β”€ download_bdcharm.py # BRGM BD Charm-50 harmonized geology, per department β”‚ β”œβ”€β”€ fetch_landcover.py # ESA WorldCover landcover class, real gauges β”‚ └── fetch_worldcover_ndvi.py # ESA WorldCover NDVI percentile composite β”‚ β”œβ”€β”€ src/ β”‚ β”œβ”€β”€ app.py # Streamlit river explorer + network validation view β”‚ β”œβ”€β”€ generate_plots.py # batch plot generation across all loaders β”‚ β”œβ”€β”€ test_build_graph.py # graph-construction test/validation suite β”‚ β”œβ”€β”€ extract_river_centerline.py # digitizes a traced map image into a centerline β”‚ β”‚ β”‚ β”œβ”€β”€ data/ β”‚ β”‚ β”œβ”€β”€ loaders/ β”‚ β”‚ β”‚ β”œβ”€β”€ base.py # BaseDataLoader β€” shared load()/get_metadata() β”‚ β”‚ β”‚ β”œβ”€β”€ hydrometric.py # discharge & water level (Hub'Eau) β”‚ β”‚ β”‚ β”œβ”€β”€ ades.py # groundwater levels (ADES) β”‚ β”‚ β”‚ β”œβ”€β”€ safran.py # ERA5 reanalysis, vectorized station interpolation β”‚ β”‚ β”‚ β”œβ”€β”€ idpr.py # infiltration/runoff tendency (BRGM) β”‚ β”‚ β”‚ β”œβ”€β”€ catchment.py # per-station catchment area (Hub'Eau) β”‚ β”‚ β”‚ β”œβ”€β”€ bdtopo_hydro.py # IGN BD TOPO hydrography (GeoJSON) β”‚ β”‚ β”‚ β”œβ”€β”€ shapefile.py # watershed boundary polygon β”‚ β”‚ β”‚ └── station_elevations.py # station coordinates + elevation β”‚ β”‚ β”‚ β”‚ β”‚ β”œβ”€β”€ river_graph.py # basin assignment, elevation ordering, edges β”‚ β”‚ β”œβ”€β”€ river_line.py # straight-line interpolation between gauges β”‚ β”‚ └── river_centerline.py # real-centerline interpolation + gauge snapping β”‚ β”‚ β”‚ └── graph/ β”‚ β”œβ”€β”€ build_graph.py # PyG conversion: x_static/x_dynamic split, structural columns β”‚ β”œβ”€β”€ build_reach_graph.py # real reach topology: confluences, splits/rejoins, MultiDiGraph β”‚ β”œβ”€β”€ node_features.py # pulls every loader into one feature table (static, one row/node) β”‚ β”œβ”€β”€ dynamic_features.py # genuine [n_nodes, T] series: discharge, groundwater, climate β”‚ └── physics_losses.py # confluence/split-rejoin/routing/water-balance loss terms, NaN-masked β”‚ β”œβ”€β”€ datasets/ # not checked in; populated by the scripts above β”‚ β”œβ”€β”€ station_list.csv # raw station roster (X, Y, names, INSEE, etc.) β”‚ β”œβ”€β”€ station_elevations.csv # station_code, lat, lon, elevation_m β”‚ β”œβ”€β”€ idpr.csv β”‚ β”œβ”€β”€ catchment_area.csv β”‚ β”œβ”€β”€ ades/ β”‚ β”œβ”€β”€ hydrometric/ β”‚ β”œβ”€β”€ safran/ β”‚ β”œβ”€β”€ bdtopo_hydro/ β”‚ β”œβ”€β”€ bdcavites/ β”‚ β”œβ”€β”€ bdcharm50/ β”‚ β”œβ”€β”€ centerlines/ β”‚ └── reach_graph/ # {eure,risle}_{nodes,edges}.csv, _nodes_enriched.csv ``` `scripts/` talks to the outside world (APIs, WFS, S3); `src/` doesn't β€” nothing under `src/` makes a network call, and a script under `src/` that wants one is a bug. Most of `src/data/loaders/` predates the graph work β€” general-purpose readers/plotters for each dataset, with `node_features.py` stitching them together afterward rather than the other way around. --- ## 2. The graph This is the part everything else in the repo exists to feed. Two graphs, one per river β€” `H4xx…` stations feed the La Eure graph, `H6xx…` feed La Risle β€” built with no edge between them, because there's no surface connection between the two basins to model. The graph is now built from **real reach topology**, not a single ordered chain of gauges. `build_reach_graph.py` constructs it directly from BD TOPO's own tronΓ§on-to-node linkage (`lien_vers_noeud_hydrographique_ini/fin`) β€” the NEXT_DOWN-equivalent approach β€” rather than inferring station order from position along a digitized line. That means real branching, real confluences, and real braided-channel structure fall directly out of the data instead of needing to be modeled separately. ### 2.1 Node types Four kinds of node, not one: | Type | What it is | Column | |---|---|---| | Real gauge | one of the 27 hydrometric stations | `is_gauged` | | Real confluence | a genuinely different, independently-sourced river joins | `is_confluence` | | Split / rejoin | a channel divides and later recombines (braiding, an anabranch) β€” same water, no new mass | `is_split_point` / `is_rejoin_point`, paired via `braid_id` | | Virtual (infill) | inserted along long confluence-free stretches so "predict at any point" has real spatial resolution | none of the above | A **confluence** requires more than a shared node with in-degree β‰₯ 2 β€” BD TOPO's fine tronΓ§on segmentation produces plenty of same-river multi-inflow points with no real branching involved (confirmed against real data: incoming-edge distances as short as 4.6 m at some falsely-flagged "confluences"). The real test (`find_real_confluences` in `build_reach_graph.py`) requires (a) more than one distinct *normalized* river name among the incoming edges β€” river-name normalization strips articles, parenthetical qualifiers, and "bras de/du/d'" (arm-of) prefixes, since a named secondary channel of the same river ("Bras de la Charentonne") isn't a different river β€” and (b) that those branches don't trace back to a common upstream **split** within 15 km, which would mean it's a rejoin, not a confluence. Splits themselves need no such disambiguation: out-degree β‰₯ 2 is an unambiguous physical definition on its own, since a split by construction has exactly one thing flowing in. Real branching topology also meant the underlying graph had to move from a plain `DiGraph` to a `MultiDiGraph` β€” two distinct tronΓ§ons directly connecting the same two hydrographic nodes (exactly the shape a short braid takes) is real data, not a collision, and a plain `DiGraph` was silently **overwriting** the second such edge's data on `add_edge` rather than keeping both. Confirmed as a real bug with real impact, not just a synthetic-test concern: fixing it recovered dozens of previously-invisible parallel edges per basin on the actual data. ### 2.2 Node and edge features The feature set now spans several independent sources, each merged onto the node table by `node_features.py`'s `add_*_features` functions. Every column lands in exactly one of four places once `build_pyg_graph` processes it: ```mermaid flowchart TD raw["Enriched node table
(node_features.py)"] raw --> struct{"structural /
graph-role column?"} struct -->|"is_gauged, is_confluence,
is_split_point, is_rejoin_point,
braid_id, snap_distance_km"| structout["data.is_gauged, data.is_confluence, ...
own Data attribute β€” never in x"] raw --> tgt{"target_* column?"} tgt -->|"target_discharge_m3s_*
target_waterlevel_mm_*"| y["data.y
never in x β€” label leakage otherwise"] raw --> feat{"real model input"} feat -->|"static: elevation_m, idpr_*,
catchment_area_km2, landcover_*,
geology_*, cavites distance/count"| xstatic["data.x_static"] feat -->|"dynamic: climate_*,
avg_groundwater_*, ndvi_*
(period-aggregate, not a real series yet)"| xdynamic["data.x_dynamic"] xstatic --> x["data.x β€” full combined tensor,
z-scored"] xdynamic --> x edges["Edge table
(build_reach_graph_tables)"] --> eattr{"numeric edge
attribute?"} eattr -->|"distance_km,
elevation_drop_m,
verified_continuous"| edgeattr["data.edge_attr
[n_edges, 3]"] eattr -->|"toponym, cleabs
(diagnostic metadata)"| meta["not used by build_pyg_graph β€”
stays in edges_df only"] ``` **Node features:** | Feature | Source | Coverage | |---|---|---| | `latitude`, `longitude`, `elevation_m` | station coords / real BD TOPO tronΓ§on Z | every node | | `idpr_value`, `idpr_nearest_point_distance` | BRGM IDPR | every node (spatial fallback for non-gauge codes) | | `catchment_area_km2` | Hub'Eau, cumulative, real gauges only | 27 stations | | `cumulative_catchment_area_km2` | BD TOPO incremental polygons, summed upstream via real graph topology | graph-wide (~98% of nodes) | | `landcover_*` (one-hot) | ESA WorldCover 10 m classification | real gauges only, for now | | `ndvi_p10`, `ndvi_p50`, `ndvi_p90` | ESA WorldCover NDVI percentile composite | real gauges only, for now | | `geology_*` (one-hot) | BRGM BD Charm-50, point-in-polygon | real gauges only, for now | | `distance_to_nearest_cavity_km`, `n_cavities_within_20km` | GΓ©orisques BDCavitΓ©s, KD-tree + haversine | real gauges only, for now | | `avg_groundwater_level_m`, `avg_groundwater_depth_m`, `n_nearby_wells` | ADES, radius-averaged, KD-tree + exact haversine | every node | | `climate_*` (temp/wind/solar/precip/evap/snow/runoff) | ERA5, vectorized station interpolation | every node (needs `safran_path`) | | `{col}__was_missing` | auto-generated | any feature column with real gaps | **Edge features** β€” three numeric attributes per edge, from `build_reach_graph.py`'s `build_reach_graph_tables`: | Feature | Meaning | |---|---| | `distance_km` | along-river distance between the two endpoint nodes | | `elevation_drop_m` | elevation difference, upstream minus downstream β€” negated on the reverse edge when `bidirectional=True` | | `verified_continuous` | `False` for any edge deliberately flagged via `known_losing_reaches` (the bΓ©toire stretch β€” Β§3.7) | `toponym` and `cleabs` also live on the real edges table (the tronΓ§on's river name and unique BD TOPO ID) but are diagnostic metadata, not model input β€” `build_pyg_graph` selects `edge_attr` columns by explicit name, so extra columns like these pass through harmlessly rather than needing to be stripped out first. **Structural columns never enter `x`.** `is_gauged`, `is_confluence`, `is_split_point`, `is_rejoin_point`, `snap_distance_km`, `braid_id` describe node *role*, not a physical covariate β€” `build_pyg_graph`'s auto-detection excludes them explicitly (confirmed as a real, not hypothetical, bug once: pandas treats `bool` as a numeric dtype, so without this exclusion these columns were being silently z-scored and fed to the model as if they were elevation or precipitation). They're still attached to the returned `Data` object as their own typed attributes, for masking supervised loss to gauged nodes and for the physics-loss index builders. **Landcover and geology are one-hot, not a raw class code.** Both are nominal categories (10 = Tree cover, 50 = Built-up; a geological formation code), not an ordered quantity β€” leaving either as a raw integer would let auto-detection z-score it as if one category were numerically "more" than another, the same class of error as the structural-column bug, just subtler since these *are* meant to be real model input. **Targets are not features.** `target_discharge_m3s_mean/std/count` and `target_waterlevel_mm_mean/std/count` exist on the enriched table but never enter `x` β€” they're pulled out into `data.y` separately, and attach only to real gauge rows (verified: gauge codes, BD TOPO hydrographic node IDs, and virtual-node marker strings occupy structurally distinct namespaces, so a left-merge on `station_code` can never mislabel a confluence or virtual node). ### 2.3 Static vs. dynamic features β€” and a real temporal pipeline `build_pyg_graph` splits every feature by physical temporal nature: - **`data.x_static`** / **`data.static_feature_names`** β€” genuinely time-invariant: elevation, IDPR, catchment area, landcover, geology, cavitΓ© proximity, coordinates. - **`data.x_dynamic`** / **`data.dynamic_feature_names`** β€” physically time-varying quantities, still as a single period-aggregated number here (mean/sum over the whole date range, or a latest well reading) β€” this tensor is a *static snapshot* of dynamic-natured quantities, not a real series. `data.x` remains the full combined tensor unchanged; the split is additional, not a replacement. **A genuine `[n_nodes, T]` series exists separately**, in `src/graph/dynamic_features.py` β€” `build_discharge_timeseries`, `build_groundwater_timeseries`, `build_climate_timeseries` β€” built specifically because `physics_losses.py`'s `routing_consistency_loss` needs a real time dimension and had nothing to consume before this existed. Same loaders as everywhere else, no re-fetching; the only difference is that these functions pivot to wide `[date x station_code]` form instead of collapsing to one aggregate the way `node_features.py`'s `add_*_features` do. Two real challenges, not incidental engineering: - **Groundwater** reports on wildly irregular schedules (confirmed: 13 different "latest dates" among 18 real wells within 20 km of one station). Each well is resampled to a common daily grid via forward-fill (a water table changes slowly β€” carrying the last known reading forward is standard practice, not an invented shortcut) *before* spatial averaging, not after β€” averaging raw irregular readings per exact calendar date is exactly what made the static version undercount real coverage by 5–10x before that was fixed (Β§3.3). The spatial neighbor-set per node is computed once, reused across every date β€” verified fast at real reach-graph scale (21.6s for 2,900 nodes Γ— 14 years daily, real 272k-row ADES data). - **Discharge** deliberately does *not* get forward-filled the way groundwater does β€” a missing daily reading stays missing, since discharge genuinely changes day to day and papering over a gap with yesterday's value would misrepresent it. `scripts/build_dynamic_tensors.py` is the actual wiring: builds these tensors for a basin, saves them, and feeds discharge directly into `routing_consistency_loss` alongside `build_routing_index` β€” the real integration point, not just parallel unconnected pieces. **Climate is untested against real data** β€” no real ERA5/`safran_path` files were available to validate `build_climate_timeseries` against in this project's development environment; the logic mirrors the already-tested discharge pivot directly, but verify the real output before trusting it. ### 2.4 Date-range filtering `build_node_features`/`enrich_reach_graph.py` accept a `date_range` applied to every time-varying source (groundwater, climate, hydrometric targets) together, so all three describe the same period rather than each silently aggregating over its own full, differently-shaped history (ADES wells reporting from the 1970s to 2026 on wildly different schedules; ERA5 spanning 1960–2026; hydrometric records with their own per-station ranges entirely). Default: **2013-01-01 to 2026-12-31** β€” computed, not guessed, via a brute-force interval-overlap check across all 8 discharge-gauged stations' real date ranges. This is the window that maximizes simultaneous station coverage: 6 of 8 stations, **8,923 real, quality-filtered observations** (`code_qualification >= 16`, the same threshold `HydrometricLoader` itself applies β€” a naive raw count that skips this filter gives 13,084, which is what an earlier pass at this analysis originally reported before the discrepancy was traced and corrected). Two stations (`H403301101`: 1969–1985, `H605022010`: 1970–1980) are permanently excluded by any reasonable window β€” a ~35–40 year dead gap separates them from every other station's record, so including them would mean spanning six mostly-empty decades, not a genuine improvement. ### 2.5 Physics-informed loss terms (`physics_losses.py`) Four constraints, each tied to real graph structure, not generic: | Term | Constraint | Applies to | |---|---|---| | `confluence_mass_balance_loss` | `Q_confluence β‰ˆ sum(Q_upstream_branches)` β€” new mass genuinely enters | `is_confluence` nodes | | `split_rejoin_conservation_loss` | `Q_split β‰ˆ Q_rejoin` β€” same water, no new mass | paired `braid_id` nodes | | `routing_consistency_loss` | `Q_downstream[t] β‰ˆ Q_upstream[t - lag]`, lag from real `distance_km`/slope | every edge, real `[n_nodes, T]` via `dynamic_features.py` (Β§2.3) | | `water_balance_loss` | `P - ET - Q - Ξ”S β‰ˆ 0` in volume terms | nodes with `cumulative_catchment_area_km2` | Confluence and split/rejoin are deliberately different constraints, not one generic "conserve mass everywhere" rule β€” a model that only learned "sum the inflows" would get a split/rejoin wrong, since a rejoin's two branches together should equal the *split's* value, not add something new on top. All four apply graph-wide, not just at the 27 labeled gauges β€” that's the actual mechanism by which sparse supervision generalizes to the ~4,500 ungauged nodes, not an incidental detail. `Ξ”S` (storage change) defaults to zero, a named steady-state approximation β€” this project has no direct basin-wide storage measurement, only sparse well *levels*, which aren't the same thing. **All four are NaN-masked, not just tolerant of complete data.** Real ground- truth Q is ~93.5% `NaN` by construction (only real gauges with real observations ever have a value β€” confirmed against the real discharge tensor) β€” that's the normal shape of the data, not a rare edge case. The shared `_mse` helper every loss function uses previously computed a plain mean, so a single `NaN` anywhere in a residual silently poisoned the *entire* loss to `NaN` β€” confirmed as a real, not hypothetical, failure: calling `routing_consistency_loss` directly on the real discharge tensor returned `NaN` before this was fixed. `_mse` now masks `NaN` out before averaging (returning `NaN` only if truly nothing usable exists at all, which is a real "no data" signal worth keeping, not silently averaging to a misleading `0`) β€” verified with the exact real scenario that first exposed the bug: `routing_consistency_loss` on the real discharge tensor now returns a real number instead of `NaN`. This also means these functions are directly usable as a diagnostic against real historical data alone, independent of any trained model β€” e.g. "does real observed discharge at two connected gauges actually satisfy the routing physics" β€” a genuine, model-free sanity check on both the physics math and the graph topology, not just a training-time loss term. ### 2.6 Two graphs, not one `build_pyg_graphs_per_basin()` returns `{0: eure_graph, 1: risle_graph}`, each with its own local `0..n-1` node indexing, rather than one merged `Data` object with two disconnected components. La Eure and La Risle are distinct hydrographic systems with nothing connecting them at the surface, and PyTorch Geometric's own batching (`Batch.from_data_list`) expects a list of separate small graphs β€” building two graphs from the start matches that convention directly. ### 2.7 What it looks like The Streamlit explorer (`src/app.py`) has two views. "Explore" renders the original click-to-read interface over real course geometry. "Network validation" renders the full reach graph β€” confluences as diamonds, gauges as elevation-colored circles, every edge as one line trace regardless of edge count (verified fast at real scale: 0.29s to build a figure for ~2,900 edges) β€” specifically for visually confirming the topology looks like a real river network before trusting it as model input. ![La Risle reach graph β€” network validation view, real confluences as diamonds, gauges elevation-colored](docs/images/risle_reach_graph.png) ![La Eure reach graph, same view](docs/images/eure_reach_graph.png) ![Explore view: clicking along the river resolves to the nearest gauge and shows its real discharge/water-level/rating-curve plots](docs/images/explore_view.png) --- ## 3. Datasets Every dataset here has its own quirks, and in a couple of cases the quirks materially affect what the data means. ### 3.1 Station roster (`station_list.csv`, `station_elevations.csv`) 27 stations across the two basins, spanning three French departments β€” verified directly against the real roster: 12 in Eure (27), 10 in Eure-et-Loir (28, the Eure's southern tributaries near Chartres/Dreux β€” Voise, Drouette, and others), 2 in Orne (61). Split roughly by Hub'Eau code prefix (`H4xx…` for La Eure, `H6xx…` for La Risle β€” a heuristic based on observed codes, not a documented rule). Elevation comes from Open Topo Data's `eudem25m` endpoint (`scripts/download_elevation.py`), queried per station coordinate β€” a point lookup, not a raster, so there's no slope or catchment information hiding in it. The row order in `station_list.csv` does **not** follow the river's course β€” verified directly, it jumps around in both latitude and elevation. Anything that needs upstream/downstream ordering has to derive it from elevation, latitude, or real centerline/graph position; never from file order. Not every station in this list is actively gauged. Cross-referencing station names against the hydrometric data turned up three categories worth knowing about: - **Manual "observateur" stations** β€” read by a person, not telemetered, so there's no digital time series to have. Four of these in the current roster. - **Partner-network ("SEBV") stations** β€” operated outside the standard Hub'Eau telemetry network, likely need a different data source entirely if you want their readings. `H431021010` is one of these. - Everything else with no data is unexplained from the name alone and worth a direct check on Hub'Eau's site before assuming it's just a gap. ### 3.2 Hydrometric data (`hydrometric/`, via `scripts/download_hubeau.py`) Discharge and water level from Hub'Eau's `obs_elab` endpoint. One thing that trips up a naive read: **both files contain a mix of `grandeur_hydro_elab` codes**, not just the variable implied by the filename. `discharge_observations.csv` has `HIXnJ`/`HIXM` (water-level codes) sitting right alongside `QmnJ` (discharge) rows for the same stations β€” `HydrometricLoader` filters each file down to its intended `grandeur` code explicitly (`QmnJ` from the discharge file, `HIXnJ` from the water-level file) rather than trusting the filename. `HydrometricLoader` also filters on Hub'Eau's own `code_qualification` field, keeping only `>= 16` (their "acceptable"/"good" threshold) and dropping lower- quality/provisional readings. This is real and meaningful, not a rounding detail β€” traced directly against the raw discharge file for the 2013–2026 window (Β§2.4): 13,084 raw `QmnJ` rows in range, of which 4,022 have `code_qualification == 12` (below the threshold) and get correctly excluded, leaving 8,923. Any manual read of the raw CSVs that skips this filter will overcount real usable observations by close to a third. Only 8 of the 27 stations have any `QmnJ` (daily mean discharge) rows at all. Several others report water level only. This isn't evenly distributed and matters a lot for anything downstream that assumes "gauged" means "has both variables" β€” it usually doesn't. ### 3.3 Groundwater (`ades/`) ADES piezometer data: 113 wells in the watershed extract, 102 of them with actual level readings, going back as far as 1967 for some wells and to 2026-07-05 for the most recent reading at time of writing. `ADESLoader.load()` merges the levels file against the stations file on `code_bss` and renames `x`/`y` to `lon`/`lat` β€” those columns are already in degrees in this dataset, not a projected CRS, so no reprojection happens or is needed. `node_features.py`'s `add_groundwater_features` does **not** use `ADESLoader.aggregate_to_stations` β€” that method loops per station and does a full haversine `.apply()` over the entire groundwater dataframe for each one. At 27 stations against ~272k readings that's slow but tolerable; at the reach graph's ~4,500 nodes it's over a billion row-wise Python calls, confirmed as a genuine, not hypothetical, multi-hour hang. The fix (reduce to each well's latest reading first, then a KD-tree coarse prefilter + exact haversine on the small candidate set) turned out to also fix a real accuracy bug: the old method required wells to share the *exact same reporting date* before averaging, but real wells report on wildly different schedules (18 real wells within 20 km of one station spanned 13 different "latest dates," one from 1972) β€” silently discarding most real coverage every time. Groundwater is used as a **station-level input covariate**, not as a graph edge. Well proximity alone isn't sufficient grounds for a subsurface/karst connectivity edge β€” that would need either correlated well hydrographs over time or a shared BDLISA aquifer-unit code (`groundwater_stations.csv` has a `codes_bdlisa` column available for exactly this kind of check; still unused). Well coverage is not uniform across the two basins. The Eure's southern reach (south of roughly 48.68Β°N, toward Chartres) has essentially zero wells within range in this extract. ### 3.4 Climate (`safran/`, via `download_era5_sample.py` / `download_era5_full.py`) Despite the `safran` naming throughout this codebase (a holdover from an earlier plan to use MΓ©tΓ©o-France's SAFRAN reanalysis), the actual data is ERA5 from Copernicus's Climate Data Store, pulled via `cdsapi`. ERA5 splits instantaneous variables (temperature, wind) from accumulated ones (precipitation, evaporation, radiation, snowfall, runoff) at the API level β€” `download_era5_full.py` downloads each set separately per year and merges them, because the CDS API rejects mixed requests. The full pull spans 1960–2026 and is genuinely slow. `SAFRANLoader` interpolates the ERA5 grid to every station **in one vectorized xarray call per file**, not one `.sel()` + `.to_dataframe()` call per station β€” the per-station loop version does real per-call work (an index lookup, then a full DataFrame conversion) that's tolerable at 27 stations (~1,800 calls across ~67 year-files) but was confirmed to actually hang at the reach graph's ~4,500 nodes (~193,000 calls). Vectorized indexing with DataArray indexers sharing a `station` dimension does every station in one call per file instead. ### 3.5 IDPR (`idpr.csv`) BRGM's *Indice de DΓ©veloppement et de Persistance des RΓ©seaux* β€” an infiltration-vs-runoff tendency index, and the closest thing this project has to a real soil/drainage covariate. It's an *integrated hydrological behavior* indicator (infiltration tendency), not raw soil texture data, but arguably more directly useful for a streamflow model than a texture map would be on its own β€” paired with BD Charm-50 geology (Β§3.9) for the broader hydrological context soil data would otherwise provide. The file used here is already one row per station (`station_id` matching `station_code` exactly, verified 1:1 against all 27 stations), so `node_features.py` does a direct ID join when possible rather than nearest-neighbor search, falling back to spatial nearest-neighbor for any station code that isn't an exact match (every non-gauge reach-graph node, and β€” a real, minor precision trade-off worth knowing β€” every gauge too, once the table also contains non-gauge codes, since the exact-match path requires the *entire* table to match IDPR's station list). ### 3.6 Catchment area β€” two independent sources **Hub'Eau (`catchment_area.csv`, via `scripts/download_catchment.py`)**: published on the **site** referentiel, not the station referentiel β€” `surface_bv` on `hydrometrie/referentiel/sites`, in kmΒ². Since one site can have several stations, the download script does two passes: station β†’ `code_site`, then `code_site` β†’ `surface_bv`. 16 of 27 stations have a value. This number is **cumulative** β€” the total catchment area draining to that point, all the way to the source. **BD TOPO, graph-wide (`cumulative_catchment_area_km2`, via `scripts/compute_cumulative_catchment.py`)**: sums BD TOPO's incremental catchment polygons upstream of any node, via the real graph topology β€” distinct polygons counted once even when many nodes/edges share the same coarse polygon (verified with a hand-computed test case specifically checking this). Covers ~98% of nodes graph-wide, not just the 27 gauges β€” the actual fix for the "confluences and virtual nodes have no catchment area at all" gap. **Cross-checked against Hub'Eau's real values on real gauges β€” and there's a real, identified bias, not a clean match.** Ratio (BD-TOPO-summed Γ· Hub'Eau) runs from about 0.75 to 1.25 for smaller catchments (< ~800 kmΒ², plausibly normal polygon-boundary/digitization precision) but drops to 0.75–0.89 for the largest catchments (> ~3,500 kmΒ²) β€” a clean, monotonic pattern, not noise. Most likely cause: **bounding-box truncation** β€” the original BD TOPO pull bbox had only a 9.6 km margin on its southern edge (the tightest of all four directions, and south is exactly where the Eure's longest upstream tributaries run, toward Chartres/Dreux), not a safe margin for real watershed extent. The bbox in `download_bdtopo_hydro.py` was widened afterward (from `(0.3, 48.3, 1.7, 49.5)` to `(-0.1, 47.7, 2.1, 49.9)`, ~2.9x the area) β€” the full `download_bdtopo_hydro.py β†’ build_reach_graphs.py β†’ enrich_reach_graph.py β†’ compute_cumulative_catchment.py` chain needs re-running against the wider box to actually resolve this, which had not yet happened as of the last verified run in this project. ### 3.7 BD TOPO hydrography (`bdtopo_hydro/`, via `scripts/download_bdtopo_hydro.py`) IGN's BD TOPO / BD TOPAGE hydrographic network, pulled from the Geoplateforme WFS (`https://data.geopf.fr/wfs`) rather than downloaded as a national bulk file β€” the download script queries a bounding box around the two basins instead (see Β§3.6 for why that box was widened). Three layers, all scoped to that bbox: - `troncon_hydrographique.geojson` β€” river centerline reaches, now the primary source for graph *topology* too (Β§2), via `lien_vers_noeud_ hydrographique_ini/fin` and `sens_de_l_ecoulement`. Real per-vertex altitude data doubles as a fine-grained elevation profile, denser than anything derivable from the 27 gauge points alone. - `surface_hydrographique.geojson` β€” hydrographic surfaces, including a `Nature` attribute that's supposed to flag karst-influenced reaches. IGN documents this attribute as **provisional and incomplete**. - `bassin_versant_topographique.geojson` β€” catchment polygons, incremental (see Β§3.6). **WFS axis order**: when a `BBOX` parameter's CRS is given via the URN form, the OGC spec requires latitude, longitude axis order β€” the opposite of the lon,lat order most GIS tools use by default. Getting this backwards doesn't raise an error; it silently matches zero real features. `download_bdtopo_hydro.py` and `scripts/download_bdcavites.py` both try lon,lat first and automatically retry with the axes swapped if that comes back empty. **Real branching topology fixed a naive assumption.** Filtering 30,045 tronΓ§ons down to a single named river and building a graph from their endpoints does **not** give one connected line β€” for "Risle" alone, 1,195 name-matched tronΓ§ons split into 132 disconnected components. Broadening the name filter to include known tributaries (Β§2.1) initially made this *worse* (487/214 components), traced to short/generic tributary names ("Bec", "Avre") matching unrelated streams elsewhere within the ~100Γ—130 km bbox β€” fixed by requiring every name-matched tronΓ§on to also fall within a real distance of a known gauge (`load_troncons_for_basin`'s `anchor_radius_km`), and by selecting the connected component actually containing the most real gauges rather than the component with the most raw tronΓ§ons (`best_component_for_stations`) β€” proven to matter, not just theoretically: a synthetic adversarial test showed the naive "biggest component" approach picking a larger but entirely unrelated decoy network over the real one. **The bΓ©toire finding**: two stations in the roster are explicitly named *"[amont bΓ©toire]"* and *"[aval bΓ©toire]"* in Hub'Eau's own site names β€” *bΓ©toire* being the Normandy dialect term for a karst swallow-hole. Three edges spanning that stretch on La Risle (`H605641101 β†’ H605022010 β†’ H605641401 β†’ H605641201`) are flagged `verified_continuous=False`. BD TOPO's own karst attribute doesn't currently confirm it (see the provisional- attribute note above) β€” `scripts/download_bdcavites.py` (Β§3.8) exists specifically to get an independent, purpose-built second check on this, rather than relying only on naming inference. ### 3.8 BDCavitΓ©s (`bdcavites/`, via `scripts/download_bdcavites.py`) BRGM's national underground cavity inventory (sinkholes, quarries, natural cavities), via GΓ©orisques' WFS (`georisques.gouv.fr/services`, typeName `CAVITE_LOCALISEE`, confirmed live and GeoJSON-capable directly against the real service). Built specifically as an independent check on the bΓ©toire finding (Β§3.7) β€” a purpose-built cavity dataset, not inference from station naming or a provisional BD TOPO attribute. One real caveat: departments 75/78/91/92/93/94/95 (Paris region, unrelated to this project) are excluded from BDCavitΓ©s entirely, and the Eure department's own inventory was among the later batches of the national 2001–2013 completion program β€” worth checking coverage density before treating a sparse result as a negative finding rather than incomplete data. ### 3.9 Geology (`bdcharm50/`, via `scripts/download_bdcharm.py`) BRGM's BD Charm-50, harmonized 1:50,000 geological maps β€” free, open (Licence Ouverte), no authentication, direct per-department ZIP download from InfoTerre (a genuinely different access pattern than the WFS sources elsewhere in this project: fixed URL per department, no bbox query, no axis- order ambiguity). Departments **27 (Eure), 28 (Eure-et-Loir), 61 (Orne)** β€” verified directly against the real, complete station roster (Β§3.1), not guessed. A separate, CIGAL-membership-gated distribution of similar data exists for at least one other French region; this project only uses the free InfoTerre path. ### 3.10 Landcover and NDVI (`scripts/fetch_landcover.py`, `scripts/fetch_worldcover_ndvi.py`) ESA WorldCover, sampled at real gauge points from the public AWS S3 Cloud- Optimized GeoTIFFs. Landcover classification uses the product's 3°×3Β° tile grid; every real station coordinate falls inside exactly one tile (`N48E000`), verified directly against all 27 real coordinates. NDVI uses the *annual composites'* 1°×1Β° tile grid instead β€” genuinely different from the classification grid, looked up per-station via VITO's own authoritative tile-index grid file (`esa_worldcover_grid_composites.fgb`) rather than a hand-guessed S3 key pattern. Both need `AWS_NO_SIGN_REQUEST=YES` for `s3://`-scheme tile URLs specifically β€” a plain HTTPS URL to the same public bucket needs no signing at all. Both currently cover only the 27 real gauges (exact `station_code` match), same limitation as Hub'Eau's `catchment_area_km2` before the cumulative-BD- TOPO fix (Β§3.6) β€” extending either script to the full reach graph is unstarted work, not a design decision. ### 3.11 Centerline generation **Only relevant to the older single-chain pipeline** (`build_surface_edges`, still available for direct comparison/debugging) β€” the reach graph (Β§2) derives its topology directly from BD TOPO's own node linkage and doesn't use these centerline files at all. `centerlines/eure_centerline.csv` and `centerlines/risle_centerline.csv` β€” the geometry `build_surface_edges` orders stations against β€” are generated by `scripts/analyze_bdtopo_hydro.py --export-centerline`. It filters `troncon_hydrographique.geojson` (Β§3.7) down to the named river, builds a graph from the tronΓ§on endpoints, and walks the longest path through it via double-BFS shortest-path to get one continuous, correctly-ordered sequence of real coordinates. This is the accurate method: real BD TOPO vector geometry snaps stations to within 0.05 km on average. `scripts/extract_river_centerline.py` is a separate, standalone technique for deriving a centerline directly from a traced map image, for a river or region without BD TOPO coverage: color-threshold the image to isolate a traced route, skeletonize it, walk the end-to-end path the same double-BFS way, then georeference by fitting a least-squares affine transform from a handful of manually-read reference-point pixel positions to their known coordinates. This produces a reliable *shape*, but the **absolute position** is only as good as the georeferencing step β€” residuals at the reference points run to a few kilometers with a handful of manually-read points, giving roughly a 0.88 km average snap distance rather than 0.05 km. It's the fallback when a real vector source isn't available, not the method used for the current `centerlines/` files. --- ## 4. Applications `src/app.py` (Streamlit) has two views, selected by a radio at the top: **Explore** β€” the original click-to-read UI over real course geometry: pick a river, click (or slide) along its course, see interpolated elevation, estimated groundwater level, and β€” for whichever real gauge is nearest that point β€” water level, discharge, and rating-curve plots pulled directly from `HydrometricLoader`'s own plotting methods rather than reimplemented. Click support uses Streamlit's native chart-selection (`st.plotly_chart(..., on_select="rerun")`), not a third-party click-handling package. The click handler and the position slider share a single source of truth by design: Streamlit only honors a slider's `value=` argument the first time that widget is created, and on every later rerun returns whatever's stored under that widget's own session-state key β€” so the click handler writes directly into the slider's own key before it's instantiated, rather than a separate key. It also de-duplicates incoming click events, since Streamlit's chart-selection state persists across reruns caused by *other* widgets and would otherwise re-fire on every unrelated interaction. **Network validation** β€” renders the full reach graph (Β§2): every edge as one Plotly line trace regardless of edge count (a trace-per-edge approach doesn't hold up at ~5,000+ edges; verified fast at real scale β€” 0.29s to build a figure for ~2,900 edges), real confluences as diamond markers, real gauges as elevation-colored circles. Metrics card reports node/edge/confluence/gauge counts and, when available, IDPR and cumulative-catchment coverage. Virtual infill nodes are deliberately not drawn individually β€” at ~2,400 per basin, markers for each would bury the actual validation signal (do confluences sit where a tributary visibly joins the line? do gauges sit on the network, not offset from it?) rather than help it. Reads directly from `reach_graph/ {basin}_nodes_enriched.csv`, keyed on file modification time so a re-run of `build_reach_graphs.py`/`enrich_reach_graph.py` is picked up automatically β€” `st.cache_data` otherwise keys purely on function arguments, not file contents, and this was confirmed to actually cause stale numbers once during development, not just a theoretical risk. --- ## 5. Testing (`src/test_build_graph.py`) Not a unit test suite in the pytest sense β€” a script with two independent sections, both run from `main()`. **`run_checks`** β€” the original single-chain pipeline: runs `node_features β†’ build_surface_edges β†’ build_pyg_graph(s)` against real data and checks the result is sane β€” no NaN/Inf in the feature tensor, no accidental cross-basin edges, targets genuinely excluded from the model input, edge indices within bounds, bidirectional edge count exactly double the directed count, per-basin node counts summing to the combined total, standardized features actually landing near zero mean / unit variance, the `known_losing_reaches` flag actually taking effect, and mean/max `snap_distance_km` per basin against whatever centerline is currently in `centerlines/`. **`run_reach_graph_checks`** β€” the reach graph pipeline, gracefully skipped (not a failure) if `reach_graph/` doesn't exist yet. Mostly regression tests for three bugs found and fixed during development, kept here specifically so they can't silently reintroduce themselves: - structural columns (`is_gauged`/`is_confluence`/etc.) never leak into `feature_names`, but remain accessible as their own `Data` attributes - target values never attach to a non-gauge node, and target coverage never exceeds the real gauge count - `edge_attr` stays exactly 3 columns despite extra edge metadata (`toponym`, `cleabs`) sitting on the real edges table - `physics_losses.py`'s `build_confluence_index`/`build_braid_index` produce counts matching `is_confluence`/`is_rejoin_point` sums, with every index within node bounds and every confluence having β‰₯ 2 upstream branches - IDPR and `cumulative_catchment_area_km2` presence/coverage are reported explicitly (the latter compared against the Hub'Eau-only baseline it's meant to exceed) Exits 0 on a clean pass across both sections, 1 otherwise β€” usable as a pre-commit or CI gate if that's ever set up. --- ## 6. Known limitations and open questions - **`cumulative_catchment_area_km2` underestimates the largest catchments** by up to ~25%, traced to the BD TOPO pull's original bounding box having an insufficient southern margin. The bbox has been widened in `download_bdtopo_hydro.py`; the full re-pull-and-rebuild chain needs re-running for this to actually resolve. Treat the largest catchments' values as approximate until then. - **Climate's genuine time series (`build_climate_timeseries`) is untested against real data** β€” discharge and groundwater's equivalents are; verify climate's real output before relying on it. - **Landcover and NDVI only cover the 27 real gauges**, not the full reach graph β€” same scope `catchment_area_km2` had before its cumulative-BD-TOPO extension. - **No model exists yet.** This repo builds the graph and the physics-loss substrate a model would train against; there is no architecture, forward pass, or training loop here. - **The karst losing-reach flag rests on naming evidence and a BDCavitΓ©s cross-check** (Β§3.7–3.8), not a fully confirmed BD TOPO classification. - **The groundwater-well BDLISA aquifer-unit field is unused.** First place to look if a subsurface connectivity edge is ever justified with real evidence rather than proximity. --- ## 7. Running things Data acquisition (from repo root, in roughly dependency order): ```bash python -m scripts.download_hubeau python -m scripts.download_elevation python -m scripts.download_era5_full # slow; download_era5_sample.py first if just testing python -m scripts.extract_era5 python -m scripts.download_catchment python -m scripts.download_bdtopo_hydro --check # verify typeNames before the real pull python -m scripts.download_bdtopo_hydro python -m scripts.download_bdcavites --check python -m scripts.download_bdcavites python -m scripts.download_bdcharm ``` Build and validate the reach graph: ```bash python -m scripts.build_reach_graphs --data-root datasets python -m scripts.enrich_reach_graph --data-root datasets # --skip-climate if that step hangs python -m scripts.compute_cumulative_catchment --data-root datasets python -m scripts.diagnose_confluences --data-root datasets --basin eure python -m scripts.diagnose_confluences --data-root datasets --basin risle ``` Build genuine `[n_nodes, T]` dynamic tensors and verify the physics-loss wiring: ```bash python -m scripts.build_dynamic_tensors --data-root datasets --basin risle python -m scripts.build_dynamic_tensors --data-root datasets --basin eure ``` Landcover / NDVI, real gauges only (needs `rasterio`, and `geopandas` for NDVI's tile lookup): ```bash python -m scripts.fetch_landcover --check python -m scripts.fetch_landcover python -m scripts.fetch_worldcover_ndvi --check python -m scripts.fetch_worldcover_ndvi ``` Validate everything against whatever's actually in `datasets/`: ```bash python -m src.test_build_graph --data-root datasets ``` Run the explorer: ```bash streamlit run src/app.py -- --data-root datasets ```