Spaces:
Running
Running
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <title>CAMELS Benchmark Experiments — 2026-07-16</title> | |
| <style> | |
| :root{ | |
| --bg:#f7f9fc; --card:#ffffff; --ink:#1c2733; --muted:#5b6b7b; | |
| --blue:#0b6bb8; --orange:#c07d00; --green:#00795a; --line:#dde5ee; | |
| --accent:#e8f1fa; | |
| } | |
| *{box-sizing:border-box; margin:0; padding:0} | |
| html,body{height:100%; background:var(--bg); color:var(--ink); | |
| font-family:"Segoe UI",-apple-system,"Helvetica Neue",Arial,sans-serif} | |
| .slide{display:none; height:100vh; padding:46px 64px 84px; overflow:auto} | |
| .slide.active{display:block} | |
| h1{font-size:2.3em; margin-bottom:.35em; color:var(--blue)} | |
| h2{font-size:1.55em; margin-bottom:.6em; color:var(--blue); | |
| border-bottom:2px solid var(--line); padding-bottom:.25em} | |
| h3{font-size:1.08em; margin:.7em 0 .35em; color:var(--ink)} | |
| p,li{font-size:1.0em; line-height:1.55; color:var(--ink)} | |
| ul{margin:.3em 0 .6em 1.3em} | |
| .muted{color:var(--muted)} | |
| .tag{display:inline-block; background:var(--accent); color:var(--blue); | |
| border:1px solid #bcd6ec; border-radius:20px; padding:2px 12px; | |
| font-size:.82em; margin:0 6px 6px 0} | |
| .cols{display:flex; gap:26px} | |
| .col{flex:1; min-width:0} | |
| .card{background:var(--card); border:1px solid var(--line); border-radius:10px; | |
| padding:16px 20px; margin:10px 0; box-shadow:0 1px 3px rgba(20,40,70,.06)} | |
| table{border-collapse:collapse; width:100%; margin:.5em 0; background:var(--card)} | |
| th,td{border:1px solid var(--line); padding:6px 10px; font-size:.92em; text-align:left} | |
| th{background:var(--accent); color:var(--ink)} | |
| td.num{font-variant-numeric:tabular-nums} | |
| tr.hl td{background:#fff6e6; font-weight:600} | |
| .good{color:var(--green); font-weight:600} | |
| .flow{display:flex; align-items:center; gap:10px; flex-wrap:wrap; margin:.6em 0} | |
| .fbox{background:var(--card); border:1.5px solid var(--blue); border-radius:8px; | |
| padding:8px 14px; font-size:.95em} | |
| .fbox.or{border-color:var(--orange)} .fbox.gr{border-color:var(--green)} | |
| .arr{color:var(--muted); font-size:1.3em} | |
| img{max-width:100%; border:1px solid var(--line); border-radius:8px; background:#fff} | |
| .cap{font-size:.85em; color:var(--muted); margin-top:4px} | |
| .kbd{background:#eef2f7; border:1px solid var(--line); border-radius:4px; | |
| padding:1px 7px; font-family:monospace} | |
| #nav{position:fixed; right:18px; bottom:16px; display:flex; gap:10px; | |
| align-items:center; z-index:50} | |
| #nav button{width:52px; height:52px; font-size:24px; border-radius:10px; | |
| border:1px solid var(--line); background:var(--card); color:var(--blue); | |
| cursor:pointer; box-shadow:0 2px 6px rgba(20,40,70,.12)} | |
| #nav button:hover{background:var(--accent)} | |
| #ctr{font-size:.95em; color:var(--muted); min-width:64px; text-align:center} | |
| </style> | |
| </head> | |
| <body> | |
| <!-- 1 ▸ title --> | |
| <section class="slide active"> | |
| <div style="margin-top:12vh"> | |
| <h1>Differentiable CREST on CAMELS:<br>benchmark & comparison experiments</h1> | |
| <p class="muted" style="font-size:1.2em"><b>2026-07-16</b></p> | |
| <div style="margin:26px 0 14px"> | |
| <span class="tag">673 basins · daily · independent test decade</span> | |
| <span class="tag">same-harness model comparison</span> | |
| <span class="tag">optimizer verification (Adam vs SCE-UA)</span> | |
| <span class="tag">regionalization (dPL / PUB)</span> | |
| </div> | |
| <div class="card" style="max-width:860px"> | |
| <b>One line:</b> a generic differentiable harness takes CREST into the classic | |
| conceptual-model band (NSE 0.61, KGE at HBV parity), the optimum is | |
| <b>verified by two independent optimizers</b>, and dPL adds the ungauged capability | |
| the classic models don't have. | |
| </div> | |
| <p class="muted" style="margin-top:18px">Navigate: <span class="kbd">←</span> | |
| <span class="kbd">→</span> or the corner buttons.</p> | |
| </div> | |
| </section> | |
| <!-- 2 ▸ experiment design --> | |
| <section class="slide"> | |
| <h2>Benchmark design — fairness first</h2> | |
| <table> | |
| <tr><th>Item</th><th>Setting</th></tr> | |
| <tr><td>Basins / scale</td><td>673 CAMELS basins, lumped (basin-mean forcing), daily</td></tr> | |
| <tr><td>Forcing</td><td>Maurer (primary, Newman convention) + Daymet (sensitivity); Oudin PET</td></tr> | |
| <tr><td>Split</td><td>train 1999-10→2008-09 · <b>test 1989-10→1999-09</b> (untouched decade)</td></tr> | |
| <tr><td>CREST config</td><td>Snow-17(9) + water balance(5) + reservoirs(5) + triangular UH(1) = 20 params/basin, | |
| direct sigmoid-bounded, Adam on (1−NSE), best-TRAIN-iterate selection</td></tr> | |
| <tr><td>Comparison models</td><td><b>HBV</b>(14p) & <b>EXP-HYDRO</b>(6p) re-implemented differentiably, | |
| trained in the <b>identical harness</b>; <b>SAC-SMA</b> = Newman's published per-basin SCE runs | |
| (their own best; we only compute metrics)</td></tr> | |
| <tr><td>Single metric code</td><td>every row evaluated by one script (plain NSE/KGE, same window, | |
| ≥1000 valid days)</td></tr> | |
| <tr><td>Audit</td><td>independent leakage audit; obs-parity gate vs Newman r=1.0000; | |
| two header-area bugs + a window off-by-one found and fixed before publishing numbers</td></tr> | |
| </table> | |
| <div class="card"><b>Harness certification:</b> our same-harness HBV lands at 0.639 — | |
| inside its published 0.63–0.68 range — so the harness neither cheats nor handicaps. | |
| Differences between rows are <i>model</i> differences.</div> | |
| </section> | |
| <!-- 3 ▸ main table --> | |
| <section class="slide"> | |
| <h2>Main results (test 1989–99, n=673)</h2> | |
| <div class="cols"> | |
| <div class="col" style="flex:1.25"> | |
| <table> | |
| <tr><th>Model</th><th>Calib</th><th>med NSE</th><th>med KGE</th><th>β</th></tr> | |
| <tr><td>SAC-SMA + Snow-17 <span class="muted">(published)</span></td><td>per-basin</td> | |
| <td class="num">0.662</td><td class="num">0.717</td><td class="num">0.94</td></tr> | |
| <tr><td>HBV <span class="muted">(ours, same harness)</span></td><td>per-basin</td> | |
| <td class="num">0.639</td><td class="num">0.671</td><td class="num">1.00</td></tr> | |
| <tr class="hl"><td>CREST+UH · β-loss · Daymet</td><td>per-basin</td> | |
| <td class="num">0.608</td><td class="num">0.664</td><td class="num">0.946</td></tr> | |
| <tr><td>CREST+UH · Maurer</td><td>per-basin</td> | |
| <td class="num">0.600</td><td class="num">0.636</td><td class="num">0.891</td></tr> | |
| <tr><td>CREST no-UH · Maurer</td><td>per-basin</td> | |
| <td class="num">0.519</td><td class="num">0.565</td><td class="num">0.87</td></tr> | |
| <tr><td>EXP-HYDRO <span class="muted">(ours, same harness)</span></td><td>per-basin</td> | |
| <td class="num">0.523</td><td class="num">0.621</td><td class="num">1.03</td></tr> | |
| <tr><td>CREST-dPL <span class="muted">(regional)</span></td><td>one net</td> | |
| <td class="num">0.462</td><td class="num">0.436</td><td class="num">0.82</td></tr> | |
| <tr><td>CREST-dPL <span class="muted">(PUB, ungauged)</span></td><td>zero-shot</td> | |
| <td class="num">0.44–0.46</td><td class="num">0.43</td><td class="num">0.81</td></tr> | |
| <tr><td class="muted" colspan="5">context (Kratzert 2019): LSTM ens. 0.74 · mHM 0.66 · | |
| HBV-ub 0.68 · VIC 0.55 · FUSE 0.65–0.72</td></tr> | |
| </table> | |
| <p class="muted" style="font-size:.88em">Forcing note: SAC-SMA, HBV and EXP-HYDRO rows use | |
| <b>Maurer</b> (Newman convention). Matched-forcing comparison: CREST+UH·Maurer | |
| <b>0.600</b> vs SAC-SMA <b>0.662</b>. The Daymet row's edge over Maurer is only +0.002 | |
| once UH is present — its gain comes from the β-loss, not the forcing.</p> | |
| </div> | |
| <div class="col"> | |
| <img src="assets/nse_cdf.png" alt="NSE CDF"> | |
| <img src="assets/nse_map.png" alt="NSE map" style="margin-top:10px"> | |
| </div> | |
| </div> | |
| </section> | |
| <!-- 3b ▸ controlled water-balance comparison --> | |
| <section class="slide"> | |
| <h2>Where the gap lives — a controlled water-balance comparison</h2> | |
| <p>Component-by-component, the SAC-SMA and CREST rows differ in <b>one</b> place:</p> | |
| <table> | |
| <tr><th>Component</th><th>SAC-SMA (Newman setup)</th><th>CREST+UH (ours)</th><th>Equivalent?</th></tr> | |
| <tr><td>Snow</td><td>Snow-17</td><td>Snow-17</td><td class="good">✅ identical</td></tr> | |
| <tr><td>Routing</td><td>calibrated <b>Gamma UH</b> (2 params: shape, scale) | |
| — SAC-SMA itself contains <b>no routing</b>; the UH is bolted on | |
| (CAMELS readme-verified)</td> | |
| <td>calibrated triangular UH (1 param: maxbas)</td> | |
| <td class="good">✅ same class (theirs slightly more flexible)</td></tr> | |
| <tr><td>PET demand</td><td>Priestley–Taylor, calibrated coefficient</td> | |
| <td>Oudin × calibrated ke multiplier</td> | |
| <td class="good">✅ same class</td></tr> | |
| <tr><td><b>Water balance</b></td> | |
| <td>dual-zone, four stores: <b>tension/free-water separation</b> in both zones, | |
| two nonlinear baseflow reservoirs, demand-driven percolation</td> | |
| <td>single VIC bucket + one linear groundwater store</td> | |
| <td><b>❌ the entire gap</b></td></tr> | |
| </table> | |
| <div class="cols"> | |
| <div class="col card"> | |
| <h3>Why daily scale makes routing a non-factor</h3> | |
| <p>CAMELS basins are small–medium (median ≈ 336 km²; concentration time mostly | |
| < 1 day). Daily aggregation hides wave dynamics — a calibrated UH captures all | |
| the timing freedom that is visible at this scale (our Stage-3 test: daily | |
| KW ≈ UH). So the benchmark is effectively a <b>controlled experiment on the | |
| water-balance structure alone</b>.</p> | |
| </div> | |
| <div class="col card"> | |
| <h3>Reading</h3> | |
| <p>Snow identical, routing & PET equivalent, optimizer verified (Adam = SCE-UA), | |
| seeds agree ⇒ the 0.662 vs 0.608 gap has a <b>single attributable source</b>: | |
| SAC-SMA's five-decades-engineered vertical soil architecture | |
| (tension/free water, dual baseflow). That is exactly the structure the | |
| 3-layer CREST kernel targets — pending layered soil priors to make the extra | |
| storages identifiable.</p> | |
| </div> | |
| </div> | |
| </section> | |
| <!-- 4 ▸ ablations --> | |
| <section class="slide"> | |
| <h2>Ablation experiments — where each point of skill comes from</h2> | |
| <table> | |
| <tr><th>Experiment</th><th>Design</th><th>Result</th><th>Reading</th></tr> | |
| <tr><td><b>UH routing</b></td><td>identical config ± differentiable triangular UH (maxbas)</td> | |
| <td class="num">0.519 → 0.600 (<b>+0.08</b>)</td> | |
| <td>hydrograph-shape freedom; water balance still sets the volume</td></tr> | |
| <tr><td><b>Forcing</b></td><td>Maurer vs Daymet, same everything</td> | |
| <td class="num">+0.06 without UH; ≈0 with UH</td> | |
| <td>UH absorbs timing differences between products</td></tr> | |
| <tr><td><b>Convergence control</b></td><td>warm-start + cosine LR, plain NSE loss (isolates optimization)</td> | |
| <td class="num">0.600 → 0.602 (+0.008 trainer-metric)</td> | |
| <td>baseline was ~converged; small honest gain</td></tr> | |
| <tr><td><b>β (bias) loss</b></td><td>NSE + 2(β−1)² penalty, selection metric matched</td> | |
| <td class="num">β 0.87→<b>0.946</b>, KGE +0.03, mean NSE +0.04, median ≈ flat</td> | |
| <td>volumetric under-bias was a <b>loss artifact</b> (loss-tolerated leak), fixed for free</td></tr> | |
| <tr><td><b>Multi-seed + merge</b></td><td>3 seeds; per-basin best-TRAIN merge across runs</td> | |
| <td class="num">seeds land 0.595–0.598; merge <b>loses</b></td> | |
| <td>optimization surface effectively flat across seeds; train-argmax picks noise — | |
| rejected honestly</td></tr> | |
| <tr><td><b>3-layer soil (A/B)</b></td><td>new differentiable 3-layer kernel vs single-layer, | |
| same harness/loss</td> | |
| <td class="num">0.610 vs 0.608 (tie); KGE 0.671 = HBV parity; train +0.04 didn't transfer</td> | |
| <td>vertical storage split <b>not identifiable from streamflow alone</b> — | |
| kept for the gridded stage where layered soil priors constrain it</td></tr> | |
| </table> | |
| </section> | |
| <!-- 5 ▸ optimizer comparison --> | |
| <section class="slide"> | |
| <h2>Optimizer comparison — Adam (backprop) vs SCE-UA</h2> | |
| <h3>Design</h3> | |
| <ul> | |
| <li>Identical parameter space (same sigmoid bounds), identical train window and | |
| plain-NSE objective, identical best-TRAIN selection — <b>only the optimizer differs</b></li> | |
| <li>SCE-UA (Duan 1992) GPU-vectorized <b>across basins</b>: one batched forward (17.8 s) | |
| evaluates one candidate per basin for all 673 → a ~673× speedup gift vs serial</li> | |
| <li>v1 = matched wall-clock budget; v2 = limit test (complex-hull mutation, 4 complexes, | |
| seeded with v1 best, 4× evaluations)</li> | |
| </ul> | |
| <table> | |
| <tr><th>Optimizer</th><th>Budget</th><th>Wall-clock</th><th>train NSE</th> | |
| <th>test NSE</th><th>test KGE</th><th>β</th></tr> | |
| <tr class="hl"><td>Adam (backprop)</td><td>400 iterations</td><td><b>8–13 h</b></td> | |
| <td class="num">~0.67</td><td class="num"><b>0.600–0.602</b></td> | |
| <td class="num">0.636–0.648</td><td class="num">0.89–0.92</td></tr> | |
| <tr><td>SCE-UA v1 <span class="muted">(same wall-clock)</span></td><td>2,542 evals</td><td>11.8 h</td> | |
| <td class="num">0.617</td><td class="num">0.564</td><td class="num">0.597</td><td class="num">0.874</td></tr> | |
| <tr><td>SCE-UA v2 <span class="muted">(limit)</span></td><td><b>10,000 evals</b></td><td><b>72 h</b></td> | |
| <td class="num">0.666</td><td class="num">0.597</td><td class="num">0.640</td><td class="num">0.908</td></tr> | |
| </table> | |
| <div class="card"><b>Conclusion:</b> SCE-UA eventually converges to the <b>same optimum</b> | |
| (0.597 ≈ 0.600, train 0.666 ≈ 0.67) — at <b>~6× the compute</b>. Two independent optimizers | |
| agreeing means 0.60 is the <b>model-structure ceiling</b>, not an optimizer artifact: | |
| the remaining 0.04 to HBV is structural, verified rather than assumed.</div> | |
| </section> | |
| <!-- 6 ▸ dPL --> | |
| <section class="slide"> | |
| <h2>Regionalization (dPL) — the capability the classic rows don't have</h2> | |
| <div class="cols"> | |
| <div class="col"> | |
| <h3>Design</h3> | |
| <div class="flow"> | |
| <div class="fbox">AlphaEarth 64-d ⊕ 12 attributes</div><span class="arr">→</span> | |
| <div class="fbox or">shared MLP</div><span class="arr">→</span> | |
| <div class="fbox gr">20 CREST params / basin</div> | |
| </div> | |
| <ul> | |
| <li><b>Time-split</b>: all basins seen, test decade unseen → cost of replacing | |
| 673 free calibrations with one network</li> | |
| <li><b>PUB</b>: 5-fold spatial hold-out (longitude-stratified) → true ungauged skill</li> | |
| <li>Trained end-to-end through the full 15-year daily simulation graph</li> | |
| </ul> | |
| <h3>Results</h3> | |
| <table> | |
| <tr><th>Mode</th><th>med NSE</th></tr> | |
| <tr><td>per-basin (upper ref)</td><td class="num">0.600</td></tr> | |
| <tr><td>dPL time-split (seen basins)</td><td class="num">0.462–0.478</td></tr> | |
| <tr class="hl"><td>dPL PUB (ungauged, 3 folds)</td><td class="num"><b>0.44–0.46</b></td></tr> | |
| </table> | |
| </div> | |
| <div class="col"> | |
| <div class="card"> | |
| <p><b>Key reading:</b> seen→unseen drop is tiny (0.475 → 0.44) — the | |
| attribute→parameter mapping <b>generalizes spatially</b>. Regional LSTM PUB | |
| (~0.53–0.56) stays above, as expected for 20-parameter physics.</p> | |
| </div> | |
| <div class="card"> | |
| <h3>Identifiability findings (streamflow-only calibration)</h3> | |
| <ul> | |
| <li>groundwater-partitioning parameter: <b>50.7%</b> of basins within 1% of its | |
| upper bound (boundary collapse)</li> | |
| <li>ET-scaling parameter: spread across the range, IQR <b>22.7%</b>, no railing</li> | |
| <li>real 1-D objective sweep: near-optimal plateau = <b>27.5%</b> of the feasible | |
| range (cross-basin median); ~49% of basins have their optimum at the bound</li> | |
| </ul> | |
| <p class="muted">Two signatures of under-constraint → motivates multi-source | |
| constraints (soil moisture / ET / storage) in the next phase.</p> | |
| </div> | |
| </div> | |
| </div> | |
| </section> | |
| <!-- 7 ▸ takeaways --> | |
| <section class="slide"> | |
| <h2>Takeaways</h2> | |
| <div class="cols"> | |
| <div class="col"> | |
| <div class="card"><b>1 · CREST enters the conceptual band.</b> | |
| 0.608 NSE / 0.664 KGE vs HBV 0.639 / 0.671 and SAC-SMA 0.662 / 0.717 — | |
| on their home turf (lumped, daily), with a generic harness and no | |
| structure-specific tuning.</div> | |
| <div class="card"><b>2 · Every point of skill is attributed.</b> | |
| UH +0.08 · forcing +0.06 (absorbed by UH) · β-loss fixes bias for free · | |
| seeds/merge/3-layer honestly rejected. No unexplained gains.</div> | |
| <div class="card"><b>3 · The optimum is optimizer-verified.</b> | |
| Backprop and an exhaustive SCE-UA converge to the same score; | |
| backprop gets there ~6× faster. The 0.04 gap to HBV is structural.</div> | |
| </div> | |
| <div class="col"> | |
| <div class="card"><b>4 · dPL is the differentiation.</b> | |
| Ungauged (PUB) NSE 0.44 with a near-zero seen/unseen gap — per-basin-calibrated | |
| classics have no equivalent capability. This is the mechanism that scales | |
| CONUS calibration to a global parameter product.</div> | |
| <div class="card"><b>5 · Throughput is the quiet result.</b> | |
| 673 basins × 6 configs × multiple optimizers in days on one GPU — | |
| the experiment volume behind this deck is itself the case for | |
| differentiable calibration.</div> | |
| <div class="card"><b>Next:</b> soil-prior gate experiment (SoilGrids depth-layered WM), | |
| multi-source constraints against the identifiability findings, then the global | |
| 0.1° parameter prior.</div> | |
| </div> | |
| </div> | |
| </section> | |
| <div id="nav"> | |
| <button id="prev" aria-label="previous">←</button> | |
| <span id="ctr">1 / 7</span> | |
| <button id="next" aria-label="next">→</button> | |
| </div> | |
| <script> | |
| const slides = [...document.querySelectorAll(".slide")]; | |
| let cur = 0; | |
| function show(i){ | |
| cur = Math.max(0, Math.min(slides.length-1, i)); | |
| slides.forEach((s,k)=>s.classList.toggle("active", k===cur)); | |
| document.getElementById("ctr").textContent = (cur+1)+" / "+slides.length; | |
| window.scrollTo(0,0); | |
| } | |
| document.getElementById("prev").onclick = ()=>show(cur-1); | |
| document.getElementById("next").onclick = ()=>show(cur+1); | |
| document.addEventListener("keydown", e=>{ | |
| if(e.key==="ArrowRight"||e.key==="PageDown") show(cur+1); | |
| if(e.key==="ArrowLeft"||e.key==="PageUp") show(cur-1); | |
| }); | |
| </script> | |
| </body> | |
| </html> | |