Spaces:

skyan1002
/

crest-global-progress

Running

File size: 18,766 Bytes

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>CAMELS Benchmark Experiments — 2026-07-16</title>
<style>
  :root{
    --bg:#f7f9fc; --card:#ffffff; --ink:#1c2733; --muted:#5b6b7b;
    --blue:#0b6bb8; --orange:#c07d00; --green:#00795a; --line:#dde5ee;
    --accent:#e8f1fa;
  }
  *{box-sizing:border-box; margin:0; padding:0}
  html,body{height:100%; background:var(--bg); color:var(--ink);
    font-family:"Segoe UI",-apple-system,"Helvetica Neue",Arial,sans-serif}
  .slide{display:none; height:100vh; padding:46px 64px 84px; overflow:auto}
  .slide.active{display:block}
  h1{font-size:2.3em; margin-bottom:.35em; color:var(--blue)}
  h2{font-size:1.55em; margin-bottom:.6em; color:var(--blue);
     border-bottom:2px solid var(--line); padding-bottom:.25em}
  h3{font-size:1.08em; margin:.7em 0 .35em; color:var(--ink)}
  p,li{font-size:1.0em; line-height:1.55; color:var(--ink)}
  ul{margin:.3em 0 .6em 1.3em}
  .muted{color:var(--muted)}
  .tag{display:inline-block; background:var(--accent); color:var(--blue);
    border:1px solid #bcd6ec; border-radius:20px; padding:2px 12px;
    font-size:.82em; margin:0 6px 6px 0}
  .cols{display:flex; gap:26px}
  .col{flex:1; min-width:0}
  .card{background:var(--card); border:1px solid var(--line); border-radius:10px;
    padding:16px 20px; margin:10px 0; box-shadow:0 1px 3px rgba(20,40,70,.06)}
  table{border-collapse:collapse; width:100%; margin:.5em 0; background:var(--card)}
  th,td{border:1px solid var(--line); padding:6px 10px; font-size:.92em; text-align:left}
  th{background:var(--accent); color:var(--ink)}
  td.num{font-variant-numeric:tabular-nums}
  tr.hl td{background:#fff6e6; font-weight:600}
  .good{color:var(--green); font-weight:600}
  .flow{display:flex; align-items:center; gap:10px; flex-wrap:wrap; margin:.6em 0}
  .fbox{background:var(--card); border:1.5px solid var(--blue); border-radius:8px;
    padding:8px 14px; font-size:.95em}
  .fbox.or{border-color:var(--orange)} .fbox.gr{border-color:var(--green)}
  .arr{color:var(--muted); font-size:1.3em}
  img{max-width:100%; border:1px solid var(--line); border-radius:8px; background:#fff}
  .cap{font-size:.85em; color:var(--muted); margin-top:4px}
  .kbd{background:#eef2f7; border:1px solid var(--line); border-radius:4px;
    padding:1px 7px; font-family:monospace}
  #nav{position:fixed; right:18px; bottom:16px; display:flex; gap:10px;
    align-items:center; z-index:50}
  #nav button{width:52px; height:52px; font-size:24px; border-radius:10px;
    border:1px solid var(--line); background:var(--card); color:var(--blue);
    cursor:pointer; box-shadow:0 2px 6px rgba(20,40,70,.12)}
  #nav button:hover{background:var(--accent)}
  #ctr{font-size:.95em; color:var(--muted); min-width:64px; text-align:center}
</style>
</head>
<body>

<!-- 1 ▸ title -->
<section class="slide active">
  <div style="margin-top:12vh">
  <h1>Differentiable CREST on CAMELS:<br>benchmark &amp; comparison experiments</h1>
  <p class="muted" style="font-size:1.2em"><b>2026-07-16</b></p>
  <div style="margin:26px 0 14px">
    <span class="tag">673 basins · daily · independent test decade</span>
    <span class="tag">same-harness model comparison</span>
    <span class="tag">optimizer verification (Adam vs SCE-UA)</span>
    <span class="tag">regionalization (dPL / PUB)</span>
  </div>
  <div class="card" style="max-width:860px">
    <b>One line:</b> a generic differentiable harness takes CREST into the classic
    conceptual-model band (NSE 0.61, KGE at HBV parity), the optimum is
    <b>verified by two independent optimizers</b>, and dPL adds the ungauged capability
    the classic models don't have.
  </div>
  <p class="muted" style="margin-top:18px">Navigate: <span class="kbd">←</span>
  <span class="kbd">→</span> or the corner buttons.</p>
  </div>
</section>

<!-- 2 ▸ experiment design -->
<section class="slide">
  <h2>Benchmark design — fairness first</h2>
  <table>
    <tr><th>Item</th><th>Setting</th></tr>
    <tr><td>Basins / scale</td><td>673 CAMELS basins, lumped (basin-mean forcing), daily</td></tr>
    <tr><td>Forcing</td><td>Maurer (primary, Newman convention) + Daymet (sensitivity); Oudin PET</td></tr>
    <tr><td>Split</td><td>train 1999-10→2008-09 · <b>test 1989-10→1999-09</b> (untouched decade)</td></tr>
    <tr><td>CREST config</td><td>Snow-17(9) + water balance(5) + reservoirs(5) + triangular UH(1) = 20 params/basin,
        direct sigmoid-bounded, Adam on (1−NSE), best-TRAIN-iterate selection</td></tr>
    <tr><td>Comparison models</td><td><b>HBV</b>(14p) &amp; <b>EXP-HYDRO</b>(6p) re-implemented differentiably,
        trained in the <b>identical harness</b>; <b>SAC-SMA</b> = Newman's published per-basin SCE runs
        (their own best; we only compute metrics)</td></tr>
    <tr><td>Single metric code</td><td>every row evaluated by one script (plain NSE/KGE, same window,
        ≥1000 valid days)</td></tr>
    <tr><td>Audit</td><td>independent leakage audit; obs-parity gate vs Newman r=1.0000;
        two header-area bugs + a window off-by-one found and fixed before publishing numbers</td></tr>
  </table>
  <div class="card"><b>Harness certification:</b> our same-harness HBV lands at 0.639 —
  inside its published 0.63–0.68 range — so the harness neither cheats nor handicaps.
  Differences between rows are <i>model</i> differences.</div>
</section>

<!-- 3 ▸ main table -->
<section class="slide">
  <h2>Main results (test 1989–99, n=673)</h2>
  <div class="cols">
    <div class="col" style="flex:1.25">
      <table>
        <tr><th>Model</th><th>Calib</th><th>med NSE</th><th>med KGE</th><th>β</th></tr>
        <tr><td>SAC-SMA + Snow-17 <span class="muted">(published)</span></td><td>per-basin</td>
            <td class="num">0.662</td><td class="num">0.717</td><td class="num">0.94</td></tr>
        <tr><td>HBV <span class="muted">(ours, same harness)</span></td><td>per-basin</td>
            <td class="num">0.639</td><td class="num">0.671</td><td class="num">1.00</td></tr>
        <tr class="hl"><td>CREST+UH · β-loss · Daymet</td><td>per-basin</td>
            <td class="num">0.608</td><td class="num">0.664</td><td class="num">0.946</td></tr>
        <tr><td>CREST+UH · Maurer</td><td>per-basin</td>
            <td class="num">0.600</td><td class="num">0.636</td><td class="num">0.891</td></tr>
        <tr><td>CREST no-UH · Maurer</td><td>per-basin</td>
            <td class="num">0.519</td><td class="num">0.565</td><td class="num">0.87</td></tr>
        <tr><td>EXP-HYDRO <span class="muted">(ours, same harness)</span></td><td>per-basin</td>
            <td class="num">0.523</td><td class="num">0.621</td><td class="num">1.03</td></tr>
        <tr><td>CREST-dPL <span class="muted">(regional)</span></td><td>one net</td>
            <td class="num">0.462</td><td class="num">0.436</td><td class="num">0.82</td></tr>
        <tr><td>CREST-dPL <span class="muted">(PUB, ungauged)</span></td><td>zero-shot</td>
            <td class="num">0.44–0.46</td><td class="num">0.43</td><td class="num">0.81</td></tr>
        <tr><td class="muted" colspan="5">context (Kratzert 2019): LSTM ens. 0.74 · mHM 0.66 ·
            HBV-ub 0.68 · VIC 0.55 · FUSE 0.65–0.72</td></tr>
      </table>
      <p class="muted" style="font-size:.88em">Forcing note: SAC-SMA, HBV and EXP-HYDRO rows use
      <b>Maurer</b> (Newman convention). Matched-forcing comparison: CREST+UH·Maurer
      <b>0.600</b> vs SAC-SMA <b>0.662</b>. The Daymet row's edge over Maurer is only +0.002
      once UH is present — its gain comes from the β-loss, not the forcing.</p>
    </div>
    <div class="col">
      <img src="assets/nse_cdf.png" alt="NSE CDF">
      <img src="assets/nse_map.png" alt="NSE map" style="margin-top:10px">
    </div>
  </div>
</section>


<!-- 3b ▸ controlled water-balance comparison -->
<section class="slide">
  <h2>Where the gap lives — a controlled water-balance comparison</h2>
  <p>Component-by-component, the SAC-SMA and CREST rows differ in <b>one</b> place:</p>
  <table>
    <tr><th>Component</th><th>SAC-SMA (Newman setup)</th><th>CREST+UH (ours)</th><th>Equivalent?</th></tr>
    <tr><td>Snow</td><td>Snow-17</td><td>Snow-17</td><td class="good">✅ identical</td></tr>
    <tr><td>Routing</td><td>calibrated <b>Gamma UH</b> (2 params: shape, scale)
        — SAC-SMA itself contains <b>no routing</b>; the UH is bolted on
        (CAMELS readme-verified)</td>
        <td>calibrated triangular UH (1 param: maxbas)</td>
        <td class="good">✅ same class (theirs slightly more flexible)</td></tr>
    <tr><td>PET demand</td><td>Priestley–Taylor, calibrated coefficient</td>
        <td>Oudin × calibrated ke multiplier</td>
        <td class="good">✅ same class</td></tr>
    <tr><td><b>Water balance</b></td>
        <td>dual-zone, four stores: <b>tension/free-water separation</b> in both zones,
            two nonlinear baseflow reservoirs, demand-driven percolation</td>
        <td>single VIC bucket + one linear groundwater store</td>
        <td><b>❌ the entire gap</b></td></tr>
  </table>
  <div class="cols">
    <div class="col card">
      <h3>Why daily scale makes routing a non-factor</h3>
      <p>CAMELS basins are small–medium (median ≈ 336 km²; concentration time mostly
      &lt; 1 day). Daily aggregation hides wave dynamics — a calibrated UH captures all
      the timing freedom that is visible at this scale (our Stage-3 test: daily
      KW ≈ UH). So the benchmark is effectively a <b>controlled experiment on the
      water-balance structure alone</b>.</p>
    </div>
    <div class="col card">
      <h3>Reading</h3>
      <p>Snow identical, routing &amp; PET equivalent, optimizer verified (Adam = SCE-UA),
      seeds agree ⇒ the 0.662 vs 0.608 gap has a <b>single attributable source</b>:
      SAC-SMA's five-decades-engineered vertical soil architecture
      (tension/free water, dual baseflow). That is exactly the structure the
      3-layer CREST kernel targets — pending layered soil priors to make the extra
      storages identifiable.</p>
    </div>
  </div>
</section>

<!-- 4 ▸ ablations -->
<section class="slide">
  <h2>Ablation experiments — where each point of skill comes from</h2>
  <table>
    <tr><th>Experiment</th><th>Design</th><th>Result</th><th>Reading</th></tr>
    <tr><td><b>UH routing</b></td><td>identical config ± differentiable triangular UH (maxbas)</td>
        <td class="num">0.519 → 0.600 (<b>+0.08</b>)</td>
        <td>hydrograph-shape freedom; water balance still sets the volume</td></tr>
    <tr><td><b>Forcing</b></td><td>Maurer vs Daymet, same everything</td>
        <td class="num">+0.06 without UH; ≈0 with UH</td>
        <td>UH absorbs timing differences between products</td></tr>
    <tr><td><b>Convergence control</b></td><td>warm-start + cosine LR, plain NSE loss (isolates optimization)</td>
        <td class="num">0.600 → 0.602 (+0.008 trainer-metric)</td>
        <td>baseline was ~converged; small honest gain</td></tr>
    <tr><td><b>β (bias) loss</b></td><td>NSE + 2(β−1)² penalty, selection metric matched</td>
        <td class="num">β 0.87→<b>0.946</b>, KGE +0.03, mean NSE +0.04, median ≈ flat</td>
        <td>volumetric under-bias was a <b>loss artifact</b> (loss-tolerated leak), fixed for free</td></tr>
    <tr><td><b>Multi-seed + merge</b></td><td>3 seeds; per-basin best-TRAIN merge across runs</td>
        <td class="num">seeds land 0.595–0.598; merge <b>loses</b></td>
        <td>optimization surface effectively flat across seeds; train-argmax picks noise —
            rejected honestly</td></tr>
    <tr><td><b>3-layer soil (A/B)</b></td><td>new differentiable 3-layer kernel vs single-layer,
        same harness/loss</td>
        <td class="num">0.610 vs 0.608 (tie); KGE 0.671 = HBV parity; train +0.04 didn't transfer</td>
        <td>vertical storage split <b>not identifiable from streamflow alone</b> —
            kept for the gridded stage where layered soil priors constrain it</td></tr>
  </table>
</section>

<!-- 5 ▸ optimizer comparison -->
<section class="slide">
  <h2>Optimizer comparison — Adam (backprop) vs SCE-UA</h2>
  <h3>Design</h3>
  <ul>
    <li>Identical parameter space (same sigmoid bounds), identical train window and
        plain-NSE objective, identical best-TRAIN selection — <b>only the optimizer differs</b></li>
    <li>SCE-UA (Duan 1992) GPU-vectorized <b>across basins</b>: one batched forward (17.8 s)
        evaluates one candidate per basin for all 673 → a ~673× speedup gift vs serial</li>
    <li>v1 = matched wall-clock budget; v2 = limit test (complex-hull mutation, 4 complexes,
        seeded with v1 best, 4× evaluations)</li>
  </ul>
  <table>
    <tr><th>Optimizer</th><th>Budget</th><th>Wall-clock</th><th>train NSE</th>
        <th>test NSE</th><th>test KGE</th><th>β</th></tr>
    <tr class="hl"><td>Adam (backprop)</td><td>400 iterations</td><td><b>8–13 h</b></td>
        <td class="num">~0.67</td><td class="num"><b>0.600–0.602</b></td>
        <td class="num">0.636–0.648</td><td class="num">0.89–0.92</td></tr>
    <tr><td>SCE-UA v1 <span class="muted">(same wall-clock)</span></td><td>2,542 evals</td><td>11.8 h</td>
        <td class="num">0.617</td><td class="num">0.564</td><td class="num">0.597</td><td class="num">0.874</td></tr>
    <tr><td>SCE-UA v2 <span class="muted">(limit)</span></td><td><b>10,000 evals</b></td><td><b>72 h</b></td>
        <td class="num">0.666</td><td class="num">0.597</td><td class="num">0.640</td><td class="num">0.908</td></tr>
  </table>
  <div class="card"><b>Conclusion:</b> SCE-UA eventually converges to the <b>same optimum</b>
  (0.597 ≈ 0.600, train 0.666 ≈ 0.67) — at <b>~6× the compute</b>. Two independent optimizers
  agreeing means 0.60 is the <b>model-structure ceiling</b>, not an optimizer artifact:
  the remaining 0.04 to HBV is structural, verified rather than assumed.</div>
</section>

<!-- 6 ▸ dPL -->
<section class="slide">
  <h2>Regionalization (dPL) — the capability the classic rows don't have</h2>
  <div class="cols">
    <div class="col">
      <h3>Design</h3>
      <div class="flow">
        <div class="fbox">AlphaEarth 64-d ⊕ 12 attributes</div><span class="arr">→</span>
        <div class="fbox or">shared MLP</div><span class="arr">→</span>
        <div class="fbox gr">20 CREST params / basin</div>
      </div>
      <ul>
        <li><b>Time-split</b>: all basins seen, test decade unseen → cost of replacing
            673 free calibrations with one network</li>
        <li><b>PUB</b>: 5-fold spatial hold-out (longitude-stratified) → true ungauged skill</li>
        <li>Trained end-to-end through the full 15-year daily simulation graph</li>
      </ul>
      <h3>Results</h3>
      <table>
        <tr><th>Mode</th><th>med NSE</th></tr>
        <tr><td>per-basin (upper ref)</td><td class="num">0.600</td></tr>
        <tr><td>dPL time-split (seen basins)</td><td class="num">0.462–0.478</td></tr>
        <tr class="hl"><td>dPL PUB (ungauged, 3 folds)</td><td class="num"><b>0.44–0.46</b></td></tr>
      </table>
    </div>
    <div class="col">
      <div class="card">
        <p><b>Key reading:</b> seen→unseen drop is tiny (0.475 → 0.44) — the
        attribute→parameter mapping <b>generalizes spatially</b>. Regional LSTM PUB
        (~0.53–0.56) stays above, as expected for 20-parameter physics.</p>
      </div>
      <div class="card">
        <h3>Identifiability findings (streamflow-only calibration)</h3>
        <ul>
          <li>groundwater-partitioning parameter: <b>50.7%</b> of basins within 1% of its
              upper bound (boundary collapse)</li>
          <li>ET-scaling parameter: spread across the range, IQR <b>22.7%</b>, no railing</li>
          <li>real 1-D objective sweep: near-optimal plateau = <b>27.5%</b> of the feasible
              range (cross-basin median); ~49% of basins have their optimum at the bound</li>
        </ul>
        <p class="muted">Two signatures of under-constraint → motivates multi-source
        constraints (soil moisture / ET / storage) in the next phase.</p>
      </div>
    </div>
  </div>
</section>

<!-- 7 ▸ takeaways -->
<section class="slide">
  <h2>Takeaways</h2>
  <div class="cols">
    <div class="col">
      <div class="card"><b>1 · CREST enters the conceptual band.</b>
        0.608 NSE / 0.664 KGE vs HBV 0.639 / 0.671 and SAC-SMA 0.662 / 0.717 —
        on their home turf (lumped, daily), with a generic harness and no
        structure-specific tuning.</div>
      <div class="card"><b>2 · Every point of skill is attributed.</b>
        UH +0.08 · forcing +0.06 (absorbed by UH) · β-loss fixes bias for free ·
        seeds/merge/3-layer honestly rejected. No unexplained gains.</div>
      <div class="card"><b>3 · The optimum is optimizer-verified.</b>
        Backprop and an exhaustive SCE-UA converge to the same score;
        backprop gets there ~6× faster. The 0.04 gap to HBV is structural.</div>
    </div>
    <div class="col">
      <div class="card"><b>4 · dPL is the differentiation.</b>
        Ungauged (PUB) NSE 0.44 with a near-zero seen/unseen gap — per-basin-calibrated
        classics have no equivalent capability. This is the mechanism that scales
        CONUS calibration to a global parameter product.</div>
      <div class="card"><b>5 · Throughput is the quiet result.</b>
        673 basins × 6 configs × multiple optimizers in days on one GPU —
        the experiment volume behind this deck is itself the case for
        differentiable calibration.</div>
      <div class="card"><b>Next:</b> soil-prior gate experiment (SoilGrids depth-layered WM),
        multi-source constraints against the identifiability findings, then the global
        0.1° parameter prior.</div>
    </div>
  </div>
</section>

<div id="nav">
  <button id="prev" aria-label="previous">←</button>
  <span id="ctr">1 / 7</span>
  <button id="next" aria-label="next">→</button>
</div>

<script>
  const slides = [...document.querySelectorAll(".slide")];
  let cur = 0;
  function show(i){
    cur = Math.max(0, Math.min(slides.length-1, i));
    slides.forEach((s,k)=>s.classList.toggle("active", k===cur));
    document.getElementById("ctr").textContent = (cur+1)+" / "+slides.length;
    window.scrollTo(0,0);
  }
  document.getElementById("prev").onclick = ()=>show(cur-1);
  document.getElementById("next").onclick = ()=>show(cur+1);
  document.addEventListener("keydown", e=>{
    if(e.key==="ArrowRight"||e.key==="PageDown") show(cur+1);
    if(e.key==="ArrowLeft"||e.key==="PageUp") show(cur-1);
  });
</script>
</body>
</html>