Spaces:
Running on Zero
Running on Zero
| title: La Eure / La Risle Hydrometric GNN | |
| emoji: 📈 | |
| colorFrom: indigo | |
| colorTo: indigo | |
| sdk: gradio | |
| sdk_version: 5.47.0 | |
| app_file: src/gradio_app.py | |
| pinned: true | |
| # La Eure / La Risle Hydrometric GNN | |
| A physics-informed graph neural network that predicts streamflow (discharge, | |
| water level) at gauged and ungauged points along two Normandy rivers, La Eure | |
| and La Risle — each modeled as a real reach-based network (confluences, | |
| braided splits/rejoins, ~4,500 nodes per basin including virtual infill | |
| points), not a single chain of gauges, with covariates pulled from nine | |
| independent data sources. | |
| ```mermaid | |
| flowchart LR | |
| hubeau["Hub'Eau<br/>discharge · water level<br/>catchment area"] | |
| ades["ADES<br/>groundwater levels"] | |
| era5["Copernicus ERA5<br/>climate reanalysis"] | |
| otd["Open Topo Data<br/>station elevation"] | |
| brgm["BRGM<br/>IDPR · BD Charm-50 geology"] | |
| bdtopo["IGN BD TOPO<br/>real reach topology + catchment polygons"] | |
| bdcav["Géorisques<br/>BDCavités (sinkholes)"] | |
| wc["ESA WorldCover<br/>landcover · NDVI"] | |
| bdtopo --> brg["build_reach_graph.py<br/>real confluences, splits/rejoins,<br/>gauge snapping"] | |
| brg --> brgs["build_reach_graphs.py<br/>~4,500 nodes/basin"] | |
| hubeau --> nf | |
| ades --> nf | |
| era5 --> nf | |
| otd --> nf | |
| brgm --> nf | |
| bdcav --> nf | |
| wc --> nf | |
| brgs --> nf["node_features.py /<br/>enrich_reach_graph.py<br/>date-filtered 2013-2026"] | |
| bdtopo --> cc["compute_cumulative_catchment.py<br/>graph-wide catchment area"] | |
| cc --> nf | |
| nf --> pyg["build_pyg_graph<br/>x_static / x_dynamic split"] | |
| pyg --> phys["physics_losses.py<br/>confluence · split-rejoin ·<br/>routing · water balance"] | |
| pyg --> app["src/app.py<br/>Streamlit explorer +<br/>network validation view"] | |
| pyg --> testsuite["test_build_graph.py<br/>validation"] | |
| ``` | |
| --- | |
| ## 1. Repository layout | |
| ``` | |
| PoC_v1/ | |
| ├── scripts/ # one-off download / extraction / build scripts | |
| │ ├── download_hubeau.py # discharge + water level, Hub'Eau API v2 | |
| │ ├── download_elevation.py # point elevations, Open Topo Data | |
| │ ├── download_era5_sample.py # ERA5 sanity-check pull (Jan 2020 only) | |
| │ ├── download_era5_full.py # ERA5 1960–2026, split instant/accum vars | |
| │ ├── extract_era5.py # unzips CDS API's zipped NetCDF output | |
| │ ├── download_catchment.py # Hub'Eau referentiel/sites -> surface_bv | |
| │ ├── download_bdtopo_hydro.py # IGN WFS -> tronçons, surfaces, catchments | |
| │ ├── analyze_bdtopo_hydro.py # centerline export + karst check | |
| │ ├── run_bdtopo_checks.py # karst + catchment cross-check, one shot | |
| │ ├── cross_check_catchments.py # spatial join: station -> containing polygon | |
| │ ├── build_reach_graphs.py # real reach-based topology, both basins | |
| │ ├── enrich_reach_graph.py # runs node_features.py against the reach graph | |
| │ ├── compute_cumulative_catchment.py # graph-wide catchment area from BD TOPO polygons | |
| │ ├── diagnose_confluences.py # verify real vs. artifact confluences | |
| │ ├── build_dynamic_tensors.py # genuine [n_nodes, T] tensors, wired into physics_losses.py | |
| │ ├── download_bdcavites.py # Géorisques BDCavités (sinkhole/cavity inventory) | |
| │ ├── download_bdcharm.py # BRGM BD Charm-50 harmonized geology, per department | |
| │ ├── fetch_landcover.py # ESA WorldCover landcover class, real gauges | |
| │ └── fetch_worldcover_ndvi.py # ESA WorldCover NDVI percentile composite | |
| │ | |
| ├── src/ | |
| │ ├── app.py # Streamlit river explorer + network validation view | |
| │ ├── generate_plots.py # batch plot generation across all loaders | |
| │ ├── test_build_graph.py # graph-construction test/validation suite | |
| │ ├── extract_river_centerline.py # digitizes a traced map image into a centerline | |
| │ │ | |
| │ ├── data/ | |
| │ │ ├── loaders/ | |
| │ │ │ ├── base.py # BaseDataLoader — shared load()/get_metadata() | |
| │ │ │ ├── hydrometric.py # discharge & water level (Hub'Eau) | |
| │ │ │ ├── ades.py # groundwater levels (ADES) | |
| │ │ │ ├── safran.py # ERA5 reanalysis, vectorized station interpolation | |
| │ │ │ ├── idpr.py # infiltration/runoff tendency (BRGM) | |
| │ │ │ ├── catchment.py # per-station catchment area (Hub'Eau) | |
| │ │ │ ├── bdtopo_hydro.py # IGN BD TOPO hydrography (GeoJSON) | |
| │ │ │ ├── shapefile.py # watershed boundary polygon | |
| │ │ │ └── station_elevations.py # station coordinates + elevation | |
| │ │ │ | |
| │ │ ├── river_graph.py # basin assignment, elevation ordering, edges | |
| │ │ ├── river_line.py # straight-line interpolation between gauges | |
| │ │ └── river_centerline.py # real-centerline interpolation + gauge snapping | |
| │ │ | |
| │ └── graph/ | |
| │ ├── build_graph.py # PyG conversion: x_static/x_dynamic split, structural columns | |
| │ ├── build_reach_graph.py # real reach topology: confluences, splits/rejoins, MultiDiGraph | |
| │ ├── node_features.py # pulls every loader into one feature table (static, one row/node) | |
| │ ├── dynamic_features.py # genuine [n_nodes, T] series: discharge, groundwater, climate | |
| │ └── physics_losses.py # confluence/split-rejoin/routing/water-balance loss terms, NaN-masked | |
| │ | |
| ├── datasets/ # not checked in; populated by the scripts above | |
| │ ├── station_list.csv # raw station roster (X, Y, names, INSEE, etc.) | |
| │ ├── station_elevations.csv # station_code, lat, lon, elevation_m | |
| │ ├── idpr.csv | |
| │ ├── catchment_area.csv | |
| │ ├── ades/ | |
| │ ├── hydrometric/ | |
| │ ├── safran/ | |
| │ ├── bdtopo_hydro/ | |
| │ ├── bdcavites/ | |
| │ ├── bdcharm50/ | |
| │ ├── centerlines/ | |
| │ └── reach_graph/ # {eure,risle}_{nodes,edges}.csv, _nodes_enriched.csv | |
| ``` | |
| `scripts/` talks to the outside world (APIs, WFS, S3); | |
| `src/` doesn't — nothing under `src/` makes a network call, and a script | |
| under `src/` that wants one is a bug. Most of `src/data/loaders/` predates | |
| the graph work — general-purpose readers/plotters for each dataset, with | |
| `node_features.py` stitching them together afterward rather than the other | |
| way around. | |
| --- | |
| ## 2. The graph | |
| This is the part everything else in the repo exists to feed. Two graphs, one | |
| per river — `H4xx…` stations feed the La Eure graph, `H6xx…` feed La Risle — | |
| built with no edge between them, because there's no surface connection | |
| between the two basins to model. | |
| The graph is now built from **real reach topology**, not a single ordered | |
| chain of gauges. `build_reach_graph.py` constructs it directly from BD TOPO's | |
| own tronçon-to-node linkage (`lien_vers_noeud_hydrographique_ini/fin`) — the | |
| NEXT_DOWN-equivalent approach — rather than inferring station order from | |
| position along a digitized line. That means real branching, real confluences, | |
| and real braided-channel structure fall directly out of the data instead of | |
| needing to be modeled separately. | |
| ### 2.1 Node types | |
| Four kinds of node, not one: | |
| | Type | What it is | Column | | |
| |---|---|---| | |
| | Real gauge | one of the 27 hydrometric stations | `is_gauged` | | |
| | Real confluence | a genuinely different, independently-sourced river joins | `is_confluence` | | |
| | Split / rejoin | a channel divides and later recombines (braiding, an anabranch) — same water, no new mass | `is_split_point` / `is_rejoin_point`, paired via `braid_id` | | |
| | Virtual (infill) | inserted along long confluence-free stretches so "predict at any point" has real spatial resolution | none of the above | | |
| A **confluence** requires more than a shared node with in-degree ≥ 2 — BD | |
| TOPO's fine tronçon segmentation produces plenty of same-river multi-inflow | |
| points with no real branching involved (confirmed against real data: | |
| incoming-edge distances as short as 4.6 m at some falsely-flagged | |
| "confluences"). The real test (`find_real_confluences` in | |
| `build_reach_graph.py`) requires (a) more than one distinct *normalized* river | |
| name among the incoming edges — river-name normalization strips articles, | |
| parenthetical qualifiers, and "bras de/du/d'" (arm-of) prefixes, since a named | |
| secondary channel of the same river ("Bras de la Charentonne") isn't a | |
| different river — and (b) that those branches don't trace back to a common | |
| upstream **split** within 15 km, which would mean it's a rejoin, not a | |
| confluence. Splits themselves need no such disambiguation: out-degree ≥ 2 is | |
| an unambiguous physical definition on its own, since a split by construction | |
| has exactly one thing flowing in. | |
| Real branching topology also meant the underlying graph had to move from a | |
| plain `DiGraph` to a `MultiDiGraph` — two distinct tronçons directly | |
| connecting the same two hydrographic nodes (exactly the shape a short braid | |
| takes) is real data, not a collision, and a plain `DiGraph` was silently | |
| **overwriting** the second such edge's data on `add_edge` rather than keeping | |
| both. Confirmed as a real bug with real impact, not just a synthetic-test | |
| concern: fixing it recovered dozens of previously-invisible parallel edges | |
| per basin on the actual data. | |
| ### 2.2 Node and edge features | |
| The feature set now spans several independent sources, each merged onto the | |
| node table by `node_features.py`'s `add_*_features` functions. Every column | |
| lands in exactly one of four places once `build_pyg_graph` processes it: | |
| ```mermaid | |
| flowchart TD | |
| raw["Enriched node table<br/>(node_features.py)"] | |
| raw --> struct{"structural /<br/>graph-role column?"} | |
| struct -->|"is_gauged, is_confluence,<br/>is_split_point, is_rejoin_point,<br/>braid_id, snap_distance_km"| structout["data.is_gauged, data.is_confluence, ...<br/>own Data attribute — never in x"] | |
| raw --> tgt{"target_* column?"} | |
| tgt -->|"target_discharge_m3s_*<br/>target_waterlevel_mm_*"| y["data.y<br/>never in x — label leakage otherwise"] | |
| raw --> feat{"real model input"} | |
| feat -->|"static: elevation_m, idpr_*,<br/>catchment_area_km2, landcover_*,<br/>geology_*, cavites distance/count"| xstatic["data.x_static"] | |
| feat -->|"dynamic: climate_*,<br/>avg_groundwater_*, ndvi_*<br/>(period-aggregate, not a real series yet)"| xdynamic["data.x_dynamic"] | |
| xstatic --> x["data.x — full combined tensor,<br/>z-scored"] | |
| xdynamic --> x | |
| edges["Edge table<br/>(build_reach_graph_tables)"] --> eattr{"numeric edge<br/>attribute?"} | |
| eattr -->|"distance_km,<br/>elevation_drop_m,<br/>verified_continuous"| edgeattr["data.edge_attr<br/>[n_edges, 3]"] | |
| eattr -->|"toponym, cleabs<br/>(diagnostic metadata)"| meta["not used by build_pyg_graph —<br/>stays in edges_df only"] | |
| ``` | |
| **Node features:** | |
| | Feature | Source | Coverage | | |
| |---|---|---| | |
| | `latitude`, `longitude`, `elevation_m` | station coords / real BD TOPO tronçon Z | every node | | |
| | `idpr_value`, `idpr_nearest_point_distance` | BRGM IDPR | every node (spatial fallback for non-gauge codes) | | |
| | `catchment_area_km2` | Hub'Eau, cumulative, real gauges only | 27 stations | | |
| | `cumulative_catchment_area_km2` | BD TOPO incremental polygons, summed upstream via real graph topology | graph-wide (~98% of nodes) | | |
| | `landcover_*` (one-hot) | ESA WorldCover 10 m classification | real gauges only, for now | | |
| | `ndvi_p10`, `ndvi_p50`, `ndvi_p90` | ESA WorldCover NDVI percentile composite | real gauges only, for now | | |
| | `geology_*` (one-hot) | BRGM BD Charm-50, point-in-polygon | real gauges only, for now | | |
| | `distance_to_nearest_cavity_km`, `n_cavities_within_20km` | Géorisques BDCavités, KD-tree + haversine | real gauges only, for now | | |
| | `avg_groundwater_level_m`, `avg_groundwater_depth_m`, `n_nearby_wells` | ADES, radius-averaged, KD-tree + exact haversine | every node | | |
| | `climate_*` (temp/wind/solar/precip/evap/snow/runoff) | ERA5, vectorized station interpolation | every node (needs `safran_path`) | | |
| | `{col}__was_missing` | auto-generated | any feature column with real gaps | | |
| **Edge features** — three numeric attributes per edge, from | |
| `build_reach_graph.py`'s `build_reach_graph_tables`: | |
| | Feature | Meaning | | |
| |---|---| | |
| | `distance_km` | along-river distance between the two endpoint nodes | | |
| | `elevation_drop_m` | elevation difference, upstream minus downstream — negated on the reverse edge when `bidirectional=True` | | |
| | `verified_continuous` | `False` for any edge deliberately flagged via `known_losing_reaches` (the bétoire stretch — §3.7) | | |
| `toponym` and `cleabs` also live on the real edges table (the tronçon's river | |
| name and unique BD TOPO ID) but are diagnostic metadata, not model input — | |
| `build_pyg_graph` selects `edge_attr` columns by explicit name, so extra | |
| columns like these pass through harmlessly rather than needing to be | |
| stripped out first. | |
| **Structural columns never enter `x`.** `is_gauged`, `is_confluence`, | |
| `is_split_point`, `is_rejoin_point`, `snap_distance_km`, `braid_id` describe | |
| node *role*, not a physical covariate — `build_pyg_graph`'s auto-detection | |
| excludes them explicitly (confirmed as a real, not hypothetical, bug once: | |
| pandas treats `bool` as a numeric dtype, so without this exclusion these | |
| columns were being silently z-scored and fed to the model as if they were | |
| elevation or precipitation). They're still attached to the returned `Data` | |
| object as their own typed attributes, for masking supervised loss to gauged | |
| nodes and for the physics-loss index builders. | |
| **Landcover and geology are one-hot, not a raw class code.** Both are | |
| nominal categories (10 = Tree cover, 50 = Built-up; a geological formation | |
| code), not an ordered quantity — leaving either as a raw integer would let | |
| auto-detection z-score it as if one category were numerically "more" than | |
| another, the same class of error as the structural-column bug, just subtler | |
| since these *are* meant to be real model input. | |
| **Targets are not features.** `target_discharge_m3s_mean/std/count` and | |
| `target_waterlevel_mm_mean/std/count` exist on the enriched table but never | |
| enter `x` — they're pulled out into `data.y` separately, and attach only to | |
| real gauge rows (verified: gauge codes, BD TOPO hydrographic node IDs, and | |
| virtual-node marker strings occupy structurally distinct namespaces, so a | |
| left-merge on `station_code` can never mislabel a confluence or virtual node). | |
| ### 2.3 Static vs. dynamic features — and a real temporal pipeline | |
| `build_pyg_graph` splits every feature by physical temporal nature: | |
| - **`data.x_static`** / **`data.static_feature_names`** — genuinely | |
| time-invariant: elevation, IDPR, catchment area, landcover, geology, | |
| cavité proximity, coordinates. | |
| - **`data.x_dynamic`** / **`data.dynamic_feature_names`** — physically | |
| time-varying quantities, still as a single period-aggregated number here | |
| (mean/sum over the whole date range, or a latest well reading) — this | |
| tensor is a *static snapshot* of dynamic-natured quantities, not a real | |
| series. `data.x` remains the full combined tensor unchanged; the split is | |
| additional, not a replacement. | |
| **A genuine `[n_nodes, T]` series exists separately**, in | |
| `src/graph/dynamic_features.py` — `build_discharge_timeseries`, | |
| `build_groundwater_timeseries`, `build_climate_timeseries` — built | |
| specifically because `physics_losses.py`'s `routing_consistency_loss` needs a | |
| real time dimension and had nothing to consume before this existed. Same | |
| loaders as everywhere else, no re-fetching; the only difference is that these | |
| functions pivot to wide `[date x station_code]` form instead of collapsing to | |
| one aggregate the way `node_features.py`'s `add_*_features` do. | |
| Two real challenges, not incidental engineering: | |
| - **Groundwater** reports on wildly irregular schedules (confirmed: 13 | |
| different "latest dates" among 18 real wells within 20 km of one station). | |
| Each well is resampled to a common daily grid via forward-fill (a water | |
| table changes slowly — carrying the last known reading forward is standard | |
| practice, not an invented shortcut) *before* spatial averaging, not after — | |
| averaging raw irregular readings per exact calendar date is exactly what | |
| made the static version undercount real coverage by 5–10x before that was | |
| fixed (§3.3). The spatial neighbor-set per node is computed once, reused | |
| across every date — verified fast at real reach-graph scale (21.6s for | |
| 2,900 nodes × 14 years daily, real 272k-row ADES data). | |
| - **Discharge** deliberately does *not* get forward-filled the way | |
| groundwater does — a missing daily reading stays missing, since discharge | |
| genuinely changes day to day and papering over a gap with yesterday's | |
| value would misrepresent it. | |
| `scripts/build_dynamic_tensors.py` is the actual wiring: builds these tensors | |
| for a basin, saves them, and feeds discharge directly into | |
| `routing_consistency_loss` alongside `build_routing_index` — the real | |
| integration point, not just parallel unconnected pieces. | |
| **Climate is untested against real data** — no real ERA5/`safran_path` files | |
| were available to validate `build_climate_timeseries` against in this | |
| project's development environment; the logic mirrors the already-tested | |
| discharge pivot directly, but verify the real output before trusting it. | |
| ### 2.4 Date-range filtering | |
| `build_node_features`/`enrich_reach_graph.py` accept a `date_range` applied | |
| to every time-varying source (groundwater, climate, hydrometric targets) | |
| together, so all three describe the same period rather than each silently | |
| aggregating over its own full, differently-shaped history (ADES wells | |
| reporting from the 1970s to 2026 on wildly different schedules; ERA5 spanning | |
| 1960–2026; hydrometric records with their own per-station ranges entirely). | |
| Default: **2013-01-01 to 2026-12-31** — computed, not guessed, via a | |
| brute-force interval-overlap check across all 8 discharge-gauged stations' | |
| real date ranges. This is the window that maximizes simultaneous station | |
| coverage: 6 of 8 stations, **8,923 real, quality-filtered observations** | |
| (`code_qualification >= 16`, the same threshold `HydrometricLoader` itself | |
| applies — a naive raw count that skips this filter gives 13,084, which is | |
| what an earlier pass at this analysis originally reported before the | |
| discrepancy was traced and corrected). Two stations (`H403301101`: 1969–1985, | |
| `H605022010`: 1970–1980) are permanently excluded by any reasonable window — | |
| a ~35–40 year dead gap separates them from every other station's record, so | |
| including them would mean spanning six mostly-empty decades, not a genuine | |
| improvement. | |
| ### 2.5 Physics-informed loss terms (`physics_losses.py`) | |
| Four constraints, each tied to real graph structure, not generic: | |
| | Term | Constraint | Applies to | | |
| |---|---|---| | |
| | `confluence_mass_balance_loss` | `Q_confluence ≈ sum(Q_upstream_branches)` — new mass genuinely enters | `is_confluence` nodes | | |
| | `split_rejoin_conservation_loss` | `Q_split ≈ Q_rejoin` — same water, no new mass | paired `braid_id` nodes | | |
| | `routing_consistency_loss` | `Q_downstream[t] ≈ Q_upstream[t - lag]`, lag from real `distance_km`/slope | every edge, real `[n_nodes, T]` via `dynamic_features.py` (§2.3) | | |
| | `water_balance_loss` | `P - ET - Q - ΔS ≈ 0` in volume terms | nodes with `cumulative_catchment_area_km2` | | |
| Confluence and split/rejoin are deliberately different constraints, not one | |
| generic "conserve mass everywhere" rule — a model that only learned "sum the | |
| inflows" would get a split/rejoin wrong, since a rejoin's two branches | |
| together should equal the *split's* value, not add something new on top. | |
| All four apply graph-wide, not just at the 27 labeled gauges — that's the | |
| actual mechanism by which sparse supervision generalizes to the ~4,500 | |
| ungauged nodes, not an incidental detail. `ΔS` (storage change) defaults to | |
| zero, a named steady-state approximation — this project has no direct | |
| basin-wide storage measurement, only sparse well *levels*, which aren't the | |
| same thing. | |
| **All four are NaN-masked, not just tolerant of complete data.** Real ground- | |
| truth Q is ~93.5% `NaN` by construction (only real gauges with real | |
| observations ever have a value — confirmed against the real discharge | |
| tensor) — that's the normal shape of the data, not a rare edge case. The | |
| shared `_mse` helper every loss function uses previously computed a plain | |
| mean, so a single `NaN` anywhere in a residual silently poisoned the *entire* | |
| loss to `NaN` — confirmed as a real, not hypothetical, failure: calling | |
| `routing_consistency_loss` directly on the real discharge tensor returned | |
| `NaN` before this was fixed. `_mse` now masks `NaN` out before averaging | |
| (returning `NaN` only if truly nothing usable exists at all, which is a | |
| real "no data" signal worth keeping, not silently averaging to a misleading | |
| `0`) — verified with the exact real scenario that first exposed the bug: | |
| `routing_consistency_loss` on the real discharge tensor now returns a real | |
| number instead of `NaN`. | |
| This also means these functions are directly usable as a diagnostic against | |
| real historical data alone, independent of any trained model — e.g. | |
| "does real observed discharge at two connected gauges actually satisfy the | |
| routing physics" — a genuine, model-free sanity check on both the physics | |
| math and the graph topology, not just a training-time loss term. | |
| ### 2.6 Two graphs, not one | |
| `build_pyg_graphs_per_basin()` returns `{0: eure_graph, 1: risle_graph}`, | |
| each with its own local `0..n-1` node indexing, rather than one merged `Data` | |
| object with two disconnected components. La Eure and La Risle are distinct | |
| hydrographic systems with nothing connecting them at the surface, and | |
| PyTorch Geometric's own batching (`Batch.from_data_list`) expects a list of | |
| separate small graphs — building two graphs from the start matches that | |
| convention directly. | |
| ### 2.7 What it looks like | |
| The Streamlit explorer (`src/app.py`) has two views. "Explore" renders the | |
| original click-to-read interface over real course geometry. "Network | |
| validation" renders the full reach graph — confluences as diamonds, gauges as | |
| elevation-colored circles, every edge as one line trace regardless of edge | |
| count (verified fast at real scale: 0.29s to build a figure for ~2,900 | |
| edges) — specifically for visually confirming the topology looks like a real | |
| river network before trusting it as model input. | |
|  | |
|  | |
|  | |
| --- | |
| ## 3. Datasets | |
| Every dataset here has its own quirks, and in a couple of cases the quirks | |
| materially affect what the data means. | |
| ### 3.1 Station roster (`station_list.csv`, `station_elevations.csv`) | |
| 27 stations across the two basins, spanning three French departments — | |
| verified directly against the real roster: 12 in Eure (27), 10 in | |
| Eure-et-Loir (28, the Eure's southern tributaries near Chartres/Dreux — Voise, | |
| Drouette, and others), 2 in Orne (61). Split roughly by Hub'Eau code prefix | |
| (`H4xx…` for La Eure, `H6xx…` for La Risle — a heuristic based on observed | |
| codes, not a documented rule). Elevation comes from Open Topo Data's | |
| `eudem25m` endpoint (`scripts/download_elevation.py`), queried per station | |
| coordinate — a point lookup, not a raster, so there's no slope or catchment | |
| information hiding in it. | |
| The row order in `station_list.csv` does **not** follow the river's course — | |
| verified directly, it jumps around in both latitude and elevation. Anything | |
| that needs upstream/downstream ordering has to derive it from elevation, | |
| latitude, or real centerline/graph position; never from file order. | |
| Not every station in this list is actively gauged. Cross-referencing station | |
| names against the hydrometric data turned up three categories worth knowing | |
| about: | |
| - **Manual "observateur" stations** — read by a person, not telemetered, so | |
| there's no digital time series to have. Four of these in the current roster. | |
| - **Partner-network ("SEBV") stations** — operated outside the standard Hub'Eau | |
| telemetry network, likely need a different data source entirely if you want | |
| their readings. `H431021010` is one of these. | |
| - Everything else with no data is unexplained from the name alone and worth a | |
| direct check on Hub'Eau's site before assuming it's just a gap. | |
| ### 3.2 Hydrometric data (`hydrometric/`, via `scripts/download_hubeau.py`) | |
| Discharge and water level from Hub'Eau's `obs_elab` endpoint. One thing that | |
| trips up a naive read: **both files contain a mix of `grandeur_hydro_elab` | |
| codes**, not just the variable implied by the filename. `discharge_observations.csv` | |
| has `HIXnJ`/`HIXM` (water-level codes) sitting right alongside `QmnJ` | |
| (discharge) rows for the same stations — `HydrometricLoader` filters each file | |
| down to its intended `grandeur` code explicitly (`QmnJ` from the discharge | |
| file, `HIXnJ` from the water-level file) rather than trusting the filename. | |
| `HydrometricLoader` also filters on Hub'Eau's own `code_qualification` field, | |
| keeping only `>= 16` (their "acceptable"/"good" threshold) and dropping lower- | |
| quality/provisional readings. This is real and meaningful, not a rounding | |
| detail — traced directly against the raw discharge file for the 2013–2026 | |
| window (§2.4): 13,084 raw `QmnJ` rows in range, of which 4,022 have | |
| `code_qualification == 12` (below the threshold) and get correctly excluded, | |
| leaving 8,923. Any manual read of the raw CSVs that skips this filter will | |
| overcount real usable observations by close to a third. | |
| Only 8 of the 27 stations have any `QmnJ` (daily mean discharge) rows at all. | |
| Several others report water level only. This isn't evenly distributed and | |
| matters a lot for anything downstream that assumes "gauged" means "has both | |
| variables" — it usually doesn't. | |
| ### 3.3 Groundwater (`ades/`) | |
| ADES piezometer data: 113 wells in the watershed extract, 102 of them with | |
| actual level readings, going back as far as 1967 for some wells and to | |
| 2026-07-05 for the most recent reading at time of writing. `ADESLoader.load()` | |
| merges the levels file against the stations file on `code_bss` and renames | |
| `x`/`y` to `lon`/`lat` — those columns are already in degrees in this dataset, | |
| not a projected CRS, so no reprojection happens or is needed. | |
| `node_features.py`'s `add_groundwater_features` does **not** use | |
| `ADESLoader.aggregate_to_stations` — that method loops per station and does a | |
| full haversine `.apply()` over the entire groundwater dataframe for each one. | |
| At 27 stations against ~272k readings that's slow but tolerable; at the reach | |
| graph's ~4,500 nodes it's over a billion row-wise Python calls, confirmed as a | |
| genuine, not hypothetical, multi-hour hang. The fix (reduce to each well's | |
| latest reading first, then a KD-tree coarse prefilter + exact haversine on the | |
| small candidate set) turned out to also fix a real accuracy bug: the old | |
| method required wells to share the *exact same reporting date* before | |
| averaging, but real wells report on wildly different schedules (18 real | |
| wells within 20 km of one station spanned 13 different "latest dates," one | |
| from 1972) — silently discarding most real coverage every time. | |
| Groundwater is used as a **station-level input covariate**, not as a graph | |
| edge. Well proximity alone isn't sufficient grounds for a subsurface/karst | |
| connectivity edge — that would need either correlated well hydrographs over | |
| time or a shared BDLISA aquifer-unit code (`groundwater_stations.csv` has a | |
| `codes_bdlisa` column available for exactly this kind of check; still unused). | |
| Well coverage is not uniform across the two basins. The Eure's southern reach | |
| (south of roughly 48.68°N, toward Chartres) has essentially zero wells within | |
| range in this extract. | |
| ### 3.4 Climate (`safran/`, via `download_era5_sample.py` / `download_era5_full.py`) | |
| Despite the `safran` naming throughout this codebase (a holdover from an | |
| earlier plan to use Météo-France's SAFRAN reanalysis), the actual data is ERA5 | |
| from Copernicus's Climate Data Store, pulled via `cdsapi`. ERA5 splits | |
| instantaneous variables (temperature, wind) from accumulated ones | |
| (precipitation, evaporation, radiation, snowfall, runoff) at the API level — | |
| `download_era5_full.py` downloads each set separately per year and merges them, | |
| because the CDS API rejects mixed requests. The full pull spans 1960–2026 and | |
| is genuinely slow. | |
| `SAFRANLoader` interpolates the ERA5 grid to every station **in one | |
| vectorized xarray call per file**, not one `.sel()` + `.to_dataframe()` call | |
| per station — the per-station loop version does real per-call work (an index | |
| lookup, then a full DataFrame conversion) that's tolerable at 27 stations | |
| (~1,800 calls across ~67 year-files) but was confirmed to actually hang at the | |
| reach graph's ~4,500 nodes (~193,000 calls). Vectorized indexing with | |
| DataArray indexers sharing a `station` dimension does every station in one | |
| call per file instead. | |
| ### 3.5 IDPR (`idpr.csv`) | |
| BRGM's *Indice de Développement et de Persistance des Réseaux* — an | |
| infiltration-vs-runoff tendency index, and the closest thing this project has | |
| to a real soil/drainage covariate. It's an *integrated hydrological behavior* | |
| indicator (infiltration tendency), not raw soil texture data, but arguably | |
| more directly useful for a streamflow model than a texture map would be on | |
| its own — paired with BD Charm-50 geology (§3.9) for the broader hydrological | |
| context soil data would otherwise provide. The file used here is already one | |
| row per station (`station_id` matching `station_code` exactly, verified 1:1 | |
| against all 27 stations), so `node_features.py` does a direct ID join when | |
| possible rather than nearest-neighbor search, falling back to spatial | |
| nearest-neighbor for any station code that isn't an exact match (every | |
| non-gauge reach-graph node, and — a real, minor precision trade-off worth | |
| knowing — every gauge too, once the table also contains non-gauge codes, | |
| since the exact-match path requires the *entire* table to match IDPR's | |
| station list). | |
| ### 3.6 Catchment area — two independent sources | |
| **Hub'Eau (`catchment_area.csv`, via `scripts/download_catchment.py`)**: | |
| published on the **site** referentiel, not the station referentiel — | |
| `surface_bv` on `hydrometrie/referentiel/sites`, in km². Since one site can | |
| have several stations, the download script does two passes: station → | |
| `code_site`, then `code_site` → `surface_bv`. 16 of 27 stations have a value. | |
| This number is **cumulative** — the total catchment area draining to that | |
| point, all the way to the source. | |
| **BD TOPO, graph-wide (`cumulative_catchment_area_km2`, via | |
| `scripts/compute_cumulative_catchment.py`)**: sums BD TOPO's incremental | |
| catchment polygons upstream of any node, via the real graph topology — | |
| distinct polygons counted once even when many nodes/edges share the same | |
| coarse polygon (verified with a hand-computed test case specifically checking | |
| this). Covers ~98% of nodes graph-wide, not just the 27 gauges — the actual | |
| fix for the "confluences and virtual nodes have no catchment area at all" gap. | |
| **Cross-checked against Hub'Eau's real values on real gauges — and there's a | |
| real, identified bias, not a clean match.** Ratio (BD-TOPO-summed ÷ Hub'Eau) | |
| runs from about 0.75 to 1.25 for smaller catchments (< ~800 km², plausibly | |
| normal polygon-boundary/digitization precision) but drops to 0.75–0.89 for | |
| the largest catchments (> ~3,500 km²) — a clean, monotonic pattern, not noise. | |
| Most likely cause: **bounding-box truncation** — the original BD TOPO pull | |
| bbox had only a 9.6 km margin on its southern edge (the tightest of all four | |
| directions, and south is exactly where the Eure's longest upstream | |
| tributaries run, toward Chartres/Dreux), not a safe margin for real watershed | |
| extent. The bbox in `download_bdtopo_hydro.py` was widened afterward (from | |
| `(0.3, 48.3, 1.7, 49.5)` to `(-0.1, 47.7, 2.1, 49.9)`, ~2.9x the area) — the | |
| full `download_bdtopo_hydro.py → build_reach_graphs.py → enrich_reach_graph.py | |
| → compute_cumulative_catchment.py` chain needs re-running against the wider | |
| box to actually resolve this, which had not yet happened as of the last | |
| verified run in this project. | |
| ### 3.7 BD TOPO hydrography (`bdtopo_hydro/`, via `scripts/download_bdtopo_hydro.py`) | |
| IGN's BD TOPO / BD TOPAGE hydrographic network, pulled from the Geoplateforme | |
| WFS (`https://data.geopf.fr/wfs`) rather than downloaded as a national bulk | |
| file — the download script queries a bounding box around the two basins | |
| instead (see §3.6 for why that box was widened). Three layers, all scoped to | |
| that bbox: | |
| - `troncon_hydrographique.geojson` — river centerline reaches, now the | |
| primary source for graph *topology* too (§2), via `lien_vers_noeud_ | |
| hydrographique_ini/fin` and `sens_de_l_ecoulement`. Real per-vertex | |
| altitude data doubles as a fine-grained elevation profile, denser than | |
| anything derivable from the 27 gauge points alone. | |
| - `surface_hydrographique.geojson` — hydrographic surfaces, including a | |
| `Nature` attribute that's supposed to flag karst-influenced reaches. IGN | |
| documents this attribute as **provisional and incomplete**. | |
| - `bassin_versant_topographique.geojson` — catchment polygons, incremental | |
| (see §3.6). | |
| **WFS axis order**: when a `BBOX` parameter's CRS is given via the URN form, | |
| the OGC spec requires latitude, longitude axis order — the opposite of the | |
| lon,lat order most GIS tools use by default. Getting this backwards doesn't | |
| raise an error; it silently matches zero real features. `download_bdtopo_hydro.py` | |
| and `scripts/download_bdcavites.py` both try lon,lat first and automatically | |
| retry with the axes swapped if that comes back empty. | |
| **Real branching topology fixed a naive assumption.** Filtering 30,045 | |
| tronçons down to a single named river and building a graph from their | |
| endpoints does **not** give one connected line — for "Risle" alone, 1,195 | |
| name-matched tronçons split into 132 disconnected components. Broadening the | |
| name filter to include known tributaries (§2.1) initially made this *worse* | |
| (487/214 components), traced to short/generic tributary names ("Bec", "Avre") | |
| matching unrelated streams elsewhere within the ~100×130 km bbox — fixed by | |
| requiring every name-matched tronçon to also fall within a real distance of a | |
| known gauge (`load_troncons_for_basin`'s `anchor_radius_km`), and by selecting | |
| the connected component actually containing the most real gauges rather than | |
| the component with the most raw tronçons (`best_component_for_stations`) — | |
| proven to matter, not just theoretically: a synthetic adversarial test showed | |
| the naive "biggest component" approach picking a larger but entirely | |
| unrelated decoy network over the real one. | |
| **The bétoire finding**: two stations in the roster are explicitly named | |
| *"[amont bétoire]"* and *"[aval bétoire]"* in Hub'Eau's own site names — | |
| *bétoire* being the Normandy dialect term for a karst swallow-hole. Three | |
| edges spanning that stretch on La Risle (`H605641101 → H605022010 → | |
| H605641401 → H605641201`) are flagged `verified_continuous=False`. BD TOPO's | |
| own karst attribute doesn't currently confirm it (see the provisional- | |
| attribute note above) — `scripts/download_bdcavites.py` (§3.8) exists | |
| specifically to get an independent, purpose-built second check on this, | |
| rather than relying only on naming inference. | |
| ### 3.8 BDCavités (`bdcavites/`, via `scripts/download_bdcavites.py`) | |
| BRGM's national underground cavity inventory (sinkholes, quarries, natural | |
| cavities), via Géorisques' WFS (`georisques.gouv.fr/services`, typeName | |
| `CAVITE_LOCALISEE`, confirmed live and GeoJSON-capable directly against the | |
| real service). Built specifically as an independent check on the bétoire | |
| finding (§3.7) — a purpose-built cavity dataset, not inference from station | |
| naming or a provisional BD TOPO attribute. One real caveat: departments | |
| 75/78/91/92/93/94/95 (Paris region, unrelated to this project) are excluded | |
| from BDCavités entirely, and the Eure department's own inventory was among | |
| the later batches of the national 2001–2013 completion program — worth | |
| checking coverage density before treating a sparse result as a negative | |
| finding rather than incomplete data. | |
| ### 3.9 Geology (`bdcharm50/`, via `scripts/download_bdcharm.py`) | |
| BRGM's BD Charm-50, harmonized 1:50,000 geological maps — free, open | |
| (Licence Ouverte), no authentication, direct per-department ZIP download from | |
| InfoTerre (a genuinely different access pattern than the WFS sources | |
| elsewhere in this project: fixed URL per department, no bbox query, no axis- | |
| order ambiguity). Departments **27 (Eure), 28 (Eure-et-Loir), 61 (Orne)** — | |
| verified directly against the real, complete station roster (§3.1), not | |
| guessed. A separate, CIGAL-membership-gated distribution of similar data | |
| exists for at least one other French region; this project only uses the free | |
| InfoTerre path. | |
| ### 3.10 Landcover and NDVI (`scripts/fetch_landcover.py`, `scripts/fetch_worldcover_ndvi.py`) | |
| ESA WorldCover, sampled at real gauge points from the public AWS S3 Cloud- | |
| Optimized GeoTIFFs. | |
| Landcover classification uses the product's 3°×3° tile grid; every real | |
| station coordinate falls inside exactly one tile (`N48E000`), verified | |
| directly against all 27 real coordinates. NDVI uses the *annual composites'* | |
| 1°×1° tile grid instead — genuinely different from the classification grid, | |
| looked up per-station via VITO's own authoritative tile-index grid file | |
| (`esa_worldcover_grid_composites.fgb`) rather than a hand-guessed S3 key | |
| pattern. Both need `AWS_NO_SIGN_REQUEST=YES` for `s3://`-scheme tile URLs | |
| specifically — a plain HTTPS URL to the same public bucket needs no signing | |
| at all. | |
| Both currently cover only the 27 real gauges (exact `station_code` match), | |
| same limitation as Hub'Eau's `catchment_area_km2` before the cumulative-BD- | |
| TOPO fix (§3.6) — extending either script to the full reach graph is | |
| unstarted work, not a design decision. | |
| ### 3.11 Centerline generation | |
| **Only relevant to the older single-chain pipeline** (`build_surface_edges`, | |
| still available for direct comparison/debugging) — the reach graph (§2) | |
| derives its topology directly from BD TOPO's own node linkage and doesn't use | |
| these centerline files at all. | |
| `centerlines/eure_centerline.csv` and `centerlines/risle_centerline.csv` — the | |
| geometry `build_surface_edges` orders stations against — are generated by | |
| `scripts/analyze_bdtopo_hydro.py --export-centerline`. It filters | |
| `troncon_hydrographique.geojson` (§3.7) down to the named river, builds a | |
| graph from the tronçon endpoints, and walks the longest path through it via | |
| double-BFS shortest-path to get one continuous, correctly-ordered sequence of | |
| real coordinates. This is the accurate method: real BD TOPO vector geometry | |
| snaps stations to within 0.05 km on average. | |
| `scripts/extract_river_centerline.py` is a separate, standalone technique for | |
| deriving a centerline directly from a traced map image, for a river or region | |
| without BD TOPO coverage: color-threshold the image to isolate a traced route, | |
| skeletonize it, walk the end-to-end path the same double-BFS way, then | |
| georeference by fitting a least-squares affine transform from a handful of | |
| manually-read reference-point pixel positions to their known coordinates. | |
| This produces a reliable *shape*, but the **absolute position** is only as | |
| good as the georeferencing step — residuals at the reference points run to a | |
| few kilometers with a handful of manually-read points, giving roughly a 0.88 | |
| km average snap distance rather than 0.05 km. It's the fallback when a real | |
| vector source isn't available, not the method used for the current | |
| `centerlines/` files. | |
| --- | |
| ## 4. Applications | |
| `src/app.py` (Streamlit) has two views, selected by a radio at the top: | |
| **Explore** — the original click-to-read UI over real course geometry: pick a | |
| river, click (or slide) along its course, see interpolated elevation, | |
| estimated groundwater level, and — for whichever real gauge is nearest that | |
| point — water level, discharge, and rating-curve plots pulled directly from | |
| `HydrometricLoader`'s own plotting methods rather than reimplemented. | |
| Click support uses Streamlit's native chart-selection | |
| (`st.plotly_chart(..., on_select="rerun")`), not a third-party click-handling | |
| package. The click handler and the position slider share a single source of | |
| truth by design: Streamlit only honors a slider's `value=` argument the first | |
| time that widget is created, and on every later rerun returns whatever's | |
| stored under that widget's own session-state key — so the click handler | |
| writes directly into the slider's own key before it's instantiated, rather | |
| than a separate key. It also de-duplicates incoming click events, since | |
| Streamlit's chart-selection state persists across reruns caused by *other* | |
| widgets and would otherwise re-fire on every unrelated interaction. | |
| **Network validation** — renders the full reach graph (§2): every edge as one | |
| Plotly line trace regardless of edge count (a trace-per-edge approach doesn't | |
| hold up at ~5,000+ edges; verified fast at real scale — 0.29s to build a | |
| figure for ~2,900 edges), real confluences as diamond markers, real gauges as | |
| elevation-colored circles. Metrics card reports node/edge/confluence/gauge | |
| counts and, when available, IDPR and cumulative-catchment coverage. Virtual | |
| infill nodes are deliberately not drawn individually — at ~2,400 per basin, | |
| markers for each would bury the actual validation signal (do confluences sit | |
| where a tributary visibly joins the line? do gauges sit on the network, not | |
| offset from it?) rather than help it. Reads directly from `reach_graph/ | |
| {basin}_nodes_enriched.csv`, keyed on file modification time so a re-run of | |
| `build_reach_graphs.py`/`enrich_reach_graph.py` is picked up automatically — | |
| `st.cache_data` otherwise keys purely on function arguments, not file | |
| contents, and this was confirmed to actually cause stale numbers once during | |
| development, not just a theoretical risk. | |
| --- | |
| ## 5. Testing (`src/test_build_graph.py`) | |
| Not a unit test suite in the pytest sense — a script with two independent | |
| sections, both run from `main()`. | |
| **`run_checks`** — the original single-chain pipeline: runs `node_features → | |
| build_surface_edges → build_pyg_graph(s)` against real data and checks the | |
| result is sane — no NaN/Inf in the feature tensor, no accidental cross-basin | |
| edges, targets genuinely excluded from the model input, edge indices within | |
| bounds, bidirectional edge count exactly double the directed count, per-basin | |
| node counts summing to the combined total, standardized features actually | |
| landing near zero mean / unit variance, the `known_losing_reaches` flag | |
| actually taking effect, and mean/max `snap_distance_km` per basin against | |
| whatever centerline is currently in `centerlines/`. | |
| **`run_reach_graph_checks`** — the reach graph pipeline, gracefully skipped | |
| (not a failure) if `reach_graph/` doesn't exist yet. Mostly regression tests | |
| for three bugs found and fixed during development, kept here specifically so | |
| they can't silently reintroduce themselves: | |
| - structural columns (`is_gauged`/`is_confluence`/etc.) never leak into | |
| `feature_names`, but remain accessible as their own `Data` attributes | |
| - target values never attach to a non-gauge node, and target coverage never | |
| exceeds the real gauge count | |
| - `edge_attr` stays exactly 3 columns despite extra edge metadata | |
| (`toponym`, `cleabs`) sitting on the real edges table | |
| - `physics_losses.py`'s `build_confluence_index`/`build_braid_index` produce | |
| counts matching `is_confluence`/`is_rejoin_point` sums, with every index | |
| within node bounds and every confluence having ≥ 2 upstream branches | |
| - IDPR and `cumulative_catchment_area_km2` presence/coverage are reported | |
| explicitly (the latter compared against the Hub'Eau-only baseline it's | |
| meant to exceed) | |
| Exits 0 on a clean pass across both sections, 1 otherwise — usable as a | |
| pre-commit or CI gate if that's ever set up. | |
| --- | |
| ## 6. Known limitations and open questions | |
| - **`cumulative_catchment_area_km2` underestimates the largest catchments** | |
| by up to ~25%, traced to the BD TOPO pull's original bounding box having an | |
| insufficient southern margin. The bbox has been widened in | |
| `download_bdtopo_hydro.py`; the full re-pull-and-rebuild chain needs | |
| re-running for this to actually resolve. Treat the largest catchments' | |
| values as approximate until then. | |
| - **Climate's genuine time series (`build_climate_timeseries`) is untested | |
| against real data** — discharge and groundwater's equivalents are; verify | |
| climate's real output before relying on it. | |
| - **Landcover and NDVI only cover the 27 real gauges**, not the full reach | |
| graph — same scope `catchment_area_km2` had before its cumulative-BD-TOPO | |
| extension. | |
| - **No model exists yet.** This repo builds the graph and the physics-loss | |
| substrate a model would train against; there is no architecture, forward | |
| pass, or training loop here. | |
| - **The karst losing-reach flag rests on naming evidence and a BDCavités | |
| cross-check** (§3.7–3.8), not a fully confirmed BD TOPO classification. | |
| - **The groundwater-well BDLISA aquifer-unit field is unused.** First place | |
| to look if a subsurface connectivity edge is ever justified with real | |
| evidence rather than proximity. | |
| --- | |
| ## 7. Running things | |
| Data acquisition (from repo root, in roughly dependency order): | |
| ```bash | |
| python -m scripts.download_hubeau | |
| python -m scripts.download_elevation | |
| python -m scripts.download_era5_full # slow; download_era5_sample.py first if just testing | |
| python -m scripts.extract_era5 | |
| python -m scripts.download_catchment | |
| python -m scripts.download_bdtopo_hydro --check # verify typeNames before the real pull | |
| python -m scripts.download_bdtopo_hydro | |
| python -m scripts.download_bdcavites --check | |
| python -m scripts.download_bdcavites | |
| python -m scripts.download_bdcharm | |
| ``` | |
| Build and validate the reach graph: | |
| ```bash | |
| python -m scripts.build_reach_graphs --data-root datasets | |
| python -m scripts.enrich_reach_graph --data-root datasets # --skip-climate if that step hangs | |
| python -m scripts.compute_cumulative_catchment --data-root datasets | |
| python -m scripts.diagnose_confluences --data-root datasets --basin eure | |
| python -m scripts.diagnose_confluences --data-root datasets --basin risle | |
| ``` | |
| Build genuine `[n_nodes, T]` dynamic tensors and verify the physics-loss wiring: | |
| ```bash | |
| python -m scripts.build_dynamic_tensors --data-root datasets --basin risle | |
| python -m scripts.build_dynamic_tensors --data-root datasets --basin eure | |
| ``` | |
| Landcover / NDVI, real gauges only (needs `rasterio`, and `geopandas` for NDVI's | |
| tile lookup): | |
| ```bash | |
| python -m scripts.fetch_landcover --check | |
| python -m scripts.fetch_landcover | |
| python -m scripts.fetch_worldcover_ndvi --check | |
| python -m scripts.fetch_worldcover_ndvi | |
| ``` | |
| Validate everything against whatever's actually in `datasets/`: | |
| ```bash | |
| python -m src.test_build_graph --data-root datasets | |
| ``` | |
| Run the explorer: | |
| ```bash | |
| streamlit run src/app.py -- --data-root datasets | |
| ``` |