Spaces:

usmar
/

undertow

Running

File size: 13,917 Bytes

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>undertow — what fine-tuning costs you off-distribution</title>
<meta name="description" content="Four sentiment models, four corpora, one architecture. Type a review and watch them disagree.">
<style>
:root {
  color-scheme: light dark;
  --blue: #2a78d6;
  --orange: #eb6834;
  --green: #2f9e6f;
  --slate: #4a5568;
  --grey: #9aa5b1;

  --bg: #fbfcfd;
  --panel: #ffffff;
  --ink: #17212b;
  --ink-soft: #55636f;
  --ink-faint: #8593a0;
  --line: #e2e7ec;
  --shadow: 0 1px 2px rgba(16, 32, 48, .06), 0 8px 24px rgba(16, 32, 48, .05);
  --heat: 42, 120, 214;
}
@media (prefers-color-scheme: dark) {
  :root {
    --bg: #10161c;
    --panel: #171f27;
    --ink: #e7edf3;
    --ink-soft: #a3b1bf;
    --ink-faint: #6f8090;
    --line: #26313c;
    --shadow: 0 1px 2px rgba(0, 0, 0, .3), 0 10px 30px rgba(0, 0, 0, .25);
  }
}
* { box-sizing: border-box; }
html { -webkit-text-size-adjust: 100%; }
body {
  margin: 0;
  background: var(--bg);
  color: var(--ink);
  font: 15px/1.6 ui-sans-serif, -apple-system, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
  font-variant-numeric: tabular-nums;
}
.wrap { max-width: 1080px; margin: 0 auto; padding: 0 20px 72px; }

/* ------------------------------------------------------------------------- header */
header.top { padding: 56px 0 28px; border-bottom: 1px solid var(--line); margin-bottom: 34px; }
.eyebrow {
  font-size: 12px; letter-spacing: .14em; text-transform: uppercase;
  color: var(--ink-faint); margin: 0 0 10px;
}
h1 { font-size: clamp(34px, 6vw, 50px); line-height: 1.04; margin: 0 0 14px; letter-spacing: -.022em; }
h1 .dim { color: var(--ink-faint); font-weight: 400; }
.lede { font-size: 17px; color: var(--ink-soft); max-width: 62ch; margin: 0 0 18px; }
.lede strong { color: var(--ink); font-weight: 600; }
.top-links { display: flex; flex-wrap: wrap; gap: 8px; }
.top-links a {
  font-size: 13px; text-decoration: none; color: var(--ink-soft);
  border: 1px solid var(--line); border-radius: 999px; padding: 5px 13px; background: var(--panel);
}
.top-links a:hover { color: var(--blue); border-color: var(--blue); }

/* ------------------------------------------------------------------------ sections */
section { margin: 0 0 52px; }
h2 { font-size: 21px; margin: 0 0 6px; letter-spacing: -.01em; }
.sub { color: var(--ink-soft); margin: 0 0 20px; max-width: 68ch; }
.panel {
  background: var(--panel); border: 1px solid var(--line);
  border-radius: 14px; padding: 22px; box-shadow: var(--shadow);
}

/* -------------------------------------------------------------------- the scorer */
#text {
  width: 100%; min-height: 116px; resize: vertical; padding: 14px 15px;
  border: 1px solid var(--line); border-radius: 10px; background: var(--bg);
  color: var(--ink); font: inherit; line-height: 1.55;
}
#text:focus { outline: 2px solid var(--blue); outline-offset: -1px; border-color: transparent; }
#examples { display: flex; flex-wrap: wrap; gap: 7px; margin: 12px 0 0; }
.chip {
  font: inherit; font-size: 13px; cursor: pointer; padding: 5px 12px;
  border-radius: 999px; border: 1px solid var(--line);
  background: var(--bg); color: var(--ink-soft);
}
.chip:hover { border-color: var(--blue); color: var(--blue); }
.chip-ghost { color: var(--ink-faint); }

.note { margin: 18px 0 0; padding: 12px 14px; border-radius: 10px; font-size: 14px; line-height: 1.55; }
.note-idle { background: color-mix(in srgb, var(--grey) 12%, transparent); color: var(--ink-soft); }
.note-split { background: color-mix(in srgb, var(--orange) 14%, transparent); color: var(--ink); }
.note-agree { background: color-mix(in srgb, var(--green) 13%, transparent); color: var(--ink); }

#verdicts {
  display: grid; gap: 14px; margin-top: 18px;
  grid-template-columns: repeat(auto-fit, minmax(216px, 1fr));
}
.verdict {
  border: 1px solid var(--line); border-radius: 12px; padding: 14px 15px 12px;
  background: var(--bg); border-top: 3px solid var(--grey);
}
.verdict.is-positive { border-top-color: var(--green); }
.verdict.is-negative { border-top-color: var(--orange); }
.verdict-head { display: flex; justify-content: space-between; align-items: baseline; gap: 8px; }
.verdict-domain { font-weight: 650; font-size: 15px; }
.verdict-meta { font-size: 11px; color: var(--ink-faint); letter-spacing: .03em; }
.verdict-empty { color: var(--ink-faint); font-size: 13px; margin: 14px 0; }
.verdict-label { margin: 10px 0 8px; font-size: 17px; font-weight: 650; letter-spacing: -.01em; }
.is-positive .verdict-label { color: var(--green); }
.is-negative .verdict-label { color: var(--orange); }
.verdict-prob { font-size: 12px; font-weight: 450; color: var(--ink-faint); letter-spacing: 0; }
.meter { position: relative; height: 7px; border-radius: 4px; background: color-mix(in srgb, var(--orange) 26%, transparent); overflow: hidden; }
.meter-fill { height: 100%; background: var(--green); border-radius: 4px 0 0 4px; }
/* The decision boundary. Without a visible mark at 0.5 the bar reads as a generic
   progress meter rather than "which side of the threshold did this land on". */
.meter-mid { position: absolute; left: 50%; top: -3px; bottom: -3px; width: 2px; margin-left: -1px; background: var(--ink); opacity: .55; border-radius: 1px; }
.terms { list-style: none; margin: 12px 0 0; padding: 0; display: grid; gap: 3px; }
.term { display: flex; justify-content: space-between; font-size: 12.5px; gap: 10px; }
.term-text { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; color: var(--ink-soft); }
.term-weight { font-variant-numeric: tabular-nums; }
.term-pos .term-weight { color: var(--green); }
.term-neg .term-weight { color: var(--orange); }
.verdict-foot { margin: 11px 0 0; font-size: 11.5px; color: var(--ink-faint); }

/* ------------------------------------------------------------------ transfer matrix */
#method-tabs { display: flex; flex-wrap: wrap; gap: 6px; margin-bottom: 16px; }
.tab {
  font: inherit; font-size: 13.5px; cursor: pointer; padding: 6px 14px;
  border-radius: 8px; border: 1px solid var(--line); background: var(--bg); color: var(--ink-soft);
}
.tab:hover { color: var(--ink); }
.tab.is-active { background: var(--blue); border-color: var(--blue); color: #fff; }
.matrix-scroll { overflow-x: auto; }
#matrix {
  display: grid; grid-template-columns: minmax(78px, auto) repeat(4, minmax(74px, 1fr));
  gap: 4px; min-width: 440px;
}
.mcell { padding: 11px 8px; border-radius: 7px; text-align: center; font-size: 14px; }
.mcell-corner { background: none; }
.mcell-head { font-size: 12px; color: var(--ink-faint); font-weight: 600; letter-spacing: .02em; }
.mcell-row { text-align: right; padding-right: 12px; display: flex; align-items: center; justify-content: flex-end; }
.mcell-value { background: rgba(var(--heat), calc(.07 + var(--t) * .82)); color: var(--ink); font-variant-numeric: tabular-nums; }
.mcell-value.is-dark { color: #fff; }
.mcell-value.is-diagonal { font-weight: 700; box-shadow: inset 0 0 0 2px var(--ink); }
.mcell-value.is-diagonal.is-dark { box-shadow: inset 0 0 0 2px #fff; }
.axis-note { display: flex; justify-content: space-between; font-size: 12px; color: var(--ink-faint); margin-top: 10px; gap: 16px; }
#matrix-blurb { margin: 14px 0 0; font-size: 13.5px; color: var(--ink-soft); }

/* ---------------------------------------------------------------------- tables */
table { width: 100%; border-collapse: collapse; font-size: 14px; }
th, td { text-align: right; padding: 9px 10px; border-bottom: 1px solid var(--line); }
thead th { font-size: 12px; color: var(--ink-faint); font-weight: 600; letter-spacing: .02em; }
tbody th, thead th:first-child { text-align: left; font-weight: 600; }
tbody tr:last-child th, tbody tr:last-child td { border-bottom: none; }
.num-gap { color: var(--orange); font-weight: 600; }
.table-scroll { overflow-x: auto; }

/* ----------------------------------------------------------------- decomposition */
#decomposition { display: grid; gap: 10px; }
.decomp-row { display: grid; grid-template-columns: 96px 1fr 74px; align-items: center; gap: 12px; }
.decomp-label { font-size: 13.5px; color: var(--ink-soft); }
.decomp-bar { display: flex; height: 20px; border-radius: 5px; overflow: hidden; background: color-mix(in srgb, var(--grey) 14%, transparent); }
.seg { height: 100%; }
.seg-ranking { background: var(--orange); }
.seg-placement { background: var(--blue); }
.decomp-total { font-size: 13px; text-align: right; color: var(--ink-soft); font-variant-numeric: tabular-nums; }
.legend { display: flex; gap: 18px; flex-wrap: wrap; margin-top: 16px; font-size: 12.5px; color: var(--ink-soft); }
.legend span { display: inline-flex; align-items: center; gap: 7px; }
.swatch { width: 11px; height: 11px; border-radius: 3px; }

footer {
  border-top: 1px solid var(--line); padding-top: 22px; margin-top: 8px;
  font-size: 13px; color: var(--ink-faint); line-height: 1.7;
}
footer a { color: var(--ink-soft); }
#loading { padding: 60px 0; color: var(--ink-faint); }
</style>
</head>
<body>
<div class="wrap">

<header class="top">
  <p class="eyebrow">Distribution shift · DistilBERT · four sentiment corpora</p>
  <h1>undertow<span class="dim"> — the accuracy that isn't there</span></h1>
  <p class="lede">
    A sentiment classifier reports one number: accuracy on a test set drawn from the corpus
    it trained on. This page is about the other twelve. Four corpora, two classifiers — a bag
    of words, and a linear probe on frozen DistilBERT features — and every model evaluated on
    every domain. Pretraining is worth <strong>+1.1 pp</strong> on the test set everyone
    reports. On the three domains nobody reports it is worth <strong>+3.5 pp</strong>, and it
    cuts the domain-shift penalty by <strong>29%</strong>.
  </p>
  <div class="top-links">
    <a href="https://github.com/UsmarHaider/undertow">Method &amp; code on GitHub</a>
  </div>
</header>

<p id="loading">Loading the models…</p>

<main id="app" hidden>

  <section>
    <h2>Four models, one sentence</h2>
    <p class="sub">
      These are the real fitted TF-IDF models from the run below — one per corpus, identical
      settings, running in your browser with nothing sent anywhere. Same architecture, same
      hyper-parameters, same 6,000 training reviews each. The only thing that differs is
      <em>which</em> 6,000.
    </p>
    <div class="panel">
      <textarea id="text" placeholder="Paste or type a review…" spellcheck="false"></textarea>
      <div id="examples"></div>
      <p id="disagreement" class="note note-idle"></p>
      <div id="verdicts"></div>
    </div>
  </section>

  <section>
    <h2>The transfer matrix</h2>
    <p class="sub">
      Row is what a model trained on; column is what it was tested on. The boxed diagonal is
      the number that gets published. Everything off it is the number you get in production.
      Hover any cell for its AUROC, its calibration, and how far a moved threshold would take it.
    </p>
    <div class="panel">
      <div id="method-tabs"></div>
      <div class="matrix-scroll"><div id="matrix"></div></div>
      <div class="axis-note"><span>rows: trained on</span><span>columns: evaluated on</span></div>
      <p id="matrix-blurb"></p>
    </div>
  </section>

  <section>
    <h2>In-domain versus everywhere else</h2>
    <p class="sub">
      Averaged over the four training domains — sixteen cells per method, four on the diagonal
      and twelve off it. <em>Oracle</em> is the accuracy each model would reach out-of-domain if
      someone re-placed its decision threshold and changed nothing else: the ceiling that costs
      no retraining.
    </p>
    <div class="panel table-scroll">
      <table>
        <thead>
          <tr>
            <th>method</th><th>in-domain</th><th>out-of-domain</th><th>drop</th>
            <th>OOD AUROC</th><th>OOD oracle</th><th>OOD ECE</th>
          </tr>
        </thead>
        <tbody id="summary-body"></tbody>
      </table>
    </div>
  </section>

  <section>
    <h2>What the drop is made of</h2>
    <p class="sub">
      The loss splits cleanly in two. <strong>Ranking</strong> is signal the representation
      genuinely no longer has — no threshold recovers it. <strong>Placement</strong> is signal
      that is still there, sitting on the wrong side of a boundary that was chosen in a
      different domain.
    </p>
    <div class="panel">
      <div id="decomposition"></div>
      <div class="legend">
        <span><i class="swatch" style="background: var(--orange)"></i> ranking — genuinely lost</span>
        <span><i class="swatch" style="background: var(--blue)"></i> placement — a misplaced threshold</span>
      </div>
    </div>
  </section>

  <section>
    <h2>Which shift hurts</h2>
    <p class="sub">
      The four corpora form a rough 2×2 — movies against commerce, full reviews against
      one-liners — so the twelve out-of-domain cells can be grouped by what kind of distance
      each one actually crosses.
    </p>
    <div class="panel table-scroll">
      <table>
        <thead><tr id="shift-head"></tr></thead>
        <tbody id="shift-body"></tbody>
      </table>
    </div>
  </section>

  <footer>
    <p>
      Corpora: <a href="https://ai.stanford.edu/~amaas/data/sentiment/">IMDB</a> (Maas et al., 2011),
      <a href="https://gluebenchmark.com/tasks">SST-2</a> (Socher et al., 2013),
      Yelp and Amazon polarity (Zhang et al., 2015). Encoder: distilbert-base-uncased.
      Every number on this page is regenerated by <code>make all</code>.
    </p>
    <p>
      Built by <a href="https://github.com/UsmarHaider">Usmar Haider</a> ·
      <a href="https://github.com/UsmarHaider/undertow">source, method and full write-up</a> · MIT
    </p>
  </footer>
</main>

</div>
<script src="undertow.js"></script>
</body>
</html>