| <!doctype html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <title>undertow — what fine-tuning costs you off-distribution</title> |
| <meta name="description" content="Four sentiment models, four corpora, one architecture. Type a review and watch them disagree."> |
| <style> |
| :root { |
| color-scheme: light dark; |
| --blue: #2a78d6; |
| --orange: #eb6834; |
| --green: #2f9e6f; |
| --slate: #4a5568; |
| --grey: #9aa5b1; |
| |
| --bg: #fbfcfd; |
| --panel: #ffffff; |
| --ink: #17212b; |
| --ink-soft: #55636f; |
| --ink-faint: #8593a0; |
| --line: #e2e7ec; |
| --shadow: 0 1px 2px rgba(16, 32, 48, .06), 0 8px 24px rgba(16, 32, 48, .05); |
| --heat: 42, 120, 214; |
| } |
| @media (prefers-color-scheme: dark) { |
| :root { |
| --bg: #10161c; |
| --panel: #171f27; |
| --ink: #e7edf3; |
| --ink-soft: #a3b1bf; |
| --ink-faint: #6f8090; |
| --line: #26313c; |
| --shadow: 0 1px 2px rgba(0, 0, 0, .3), 0 10px 30px rgba(0, 0, 0, .25); |
| } |
| } |
| * { box-sizing: border-box; } |
| html { -webkit-text-size-adjust: 100%; } |
| body { |
| margin: 0; |
| background: var(--bg); |
| color: var(--ink); |
| font: 15px/1.6 ui-sans-serif, -apple-system, "Segoe UI", Roboto, Helvetica, Arial, sans-serif; |
| font-variant-numeric: tabular-nums; |
| } |
| .wrap { max-width: 1080px; margin: 0 auto; padding: 0 20px 72px; } |
| |
| |
| header.top { padding: 56px 0 28px; border-bottom: 1px solid var(--line); margin-bottom: 34px; } |
| .eyebrow { |
| font-size: 12px; letter-spacing: .14em; text-transform: uppercase; |
| color: var(--ink-faint); margin: 0 0 10px; |
| } |
| h1 { font-size: clamp(34px, 6vw, 50px); line-height: 1.04; margin: 0 0 14px; letter-spacing: -.022em; } |
| h1 .dim { color: var(--ink-faint); font-weight: 400; } |
| .lede { font-size: 17px; color: var(--ink-soft); max-width: 62ch; margin: 0 0 18px; } |
| .lede strong { color: var(--ink); font-weight: 600; } |
| .top-links { display: flex; flex-wrap: wrap; gap: 8px; } |
| .top-links a { |
| font-size: 13px; text-decoration: none; color: var(--ink-soft); |
| border: 1px solid var(--line); border-radius: 999px; padding: 5px 13px; background: var(--panel); |
| } |
| .top-links a:hover { color: var(--blue); border-color: var(--blue); } |
| |
| |
| section { margin: 0 0 52px; } |
| h2 { font-size: 21px; margin: 0 0 6px; letter-spacing: -.01em; } |
| .sub { color: var(--ink-soft); margin: 0 0 20px; max-width: 68ch; } |
| .panel { |
| background: var(--panel); border: 1px solid var(--line); |
| border-radius: 14px; padding: 22px; box-shadow: var(--shadow); |
| } |
| |
| |
| #text { |
| width: 100%; min-height: 116px; resize: vertical; padding: 14px 15px; |
| border: 1px solid var(--line); border-radius: 10px; background: var(--bg); |
| color: var(--ink); font: inherit; line-height: 1.55; |
| } |
| #text:focus { outline: 2px solid var(--blue); outline-offset: -1px; border-color: transparent; } |
| #examples { display: flex; flex-wrap: wrap; gap: 7px; margin: 12px 0 0; } |
| .chip { |
| font: inherit; font-size: 13px; cursor: pointer; padding: 5px 12px; |
| border-radius: 999px; border: 1px solid var(--line); |
| background: var(--bg); color: var(--ink-soft); |
| } |
| .chip:hover { border-color: var(--blue); color: var(--blue); } |
| .chip-ghost { color: var(--ink-faint); } |
| |
| .note { margin: 18px 0 0; padding: 12px 14px; border-radius: 10px; font-size: 14px; line-height: 1.55; } |
| .note-idle { background: color-mix(in srgb, var(--grey) 12%, transparent); color: var(--ink-soft); } |
| .note-split { background: color-mix(in srgb, var(--orange) 14%, transparent); color: var(--ink); } |
| .note-agree { background: color-mix(in srgb, var(--green) 13%, transparent); color: var(--ink); } |
| |
| #verdicts { |
| display: grid; gap: 14px; margin-top: 18px; |
| grid-template-columns: repeat(auto-fit, minmax(216px, 1fr)); |
| } |
| .verdict { |
| border: 1px solid var(--line); border-radius: 12px; padding: 14px 15px 12px; |
| background: var(--bg); border-top: 3px solid var(--grey); |
| } |
| .verdict.is-positive { border-top-color: var(--green); } |
| .verdict.is-negative { border-top-color: var(--orange); } |
| .verdict-head { display: flex; justify-content: space-between; align-items: baseline; gap: 8px; } |
| .verdict-domain { font-weight: 650; font-size: 15px; } |
| .verdict-meta { font-size: 11px; color: var(--ink-faint); letter-spacing: .03em; } |
| .verdict-empty { color: var(--ink-faint); font-size: 13px; margin: 14px 0; } |
| .verdict-label { margin: 10px 0 8px; font-size: 17px; font-weight: 650; letter-spacing: -.01em; } |
| .is-positive .verdict-label { color: var(--green); } |
| .is-negative .verdict-label { color: var(--orange); } |
| .verdict-prob { font-size: 12px; font-weight: 450; color: var(--ink-faint); letter-spacing: 0; } |
| .meter { position: relative; height: 7px; border-radius: 4px; background: color-mix(in srgb, var(--orange) 26%, transparent); overflow: hidden; } |
| .meter-fill { height: 100%; background: var(--green); border-radius: 4px 0 0 4px; } |
| |
| |
| .meter-mid { position: absolute; left: 50%; top: -3px; bottom: -3px; width: 2px; margin-left: -1px; background: var(--ink); opacity: .55; border-radius: 1px; } |
| .terms { list-style: none; margin: 12px 0 0; padding: 0; display: grid; gap: 3px; } |
| .term { display: flex; justify-content: space-between; font-size: 12.5px; gap: 10px; } |
| .term-text { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; color: var(--ink-soft); } |
| .term-weight { font-variant-numeric: tabular-nums; } |
| .term-pos .term-weight { color: var(--green); } |
| .term-neg .term-weight { color: var(--orange); } |
| .verdict-foot { margin: 11px 0 0; font-size: 11.5px; color: var(--ink-faint); } |
| |
| |
| #method-tabs { display: flex; flex-wrap: wrap; gap: 6px; margin-bottom: 16px; } |
| .tab { |
| font: inherit; font-size: 13.5px; cursor: pointer; padding: 6px 14px; |
| border-radius: 8px; border: 1px solid var(--line); background: var(--bg); color: var(--ink-soft); |
| } |
| .tab:hover { color: var(--ink); } |
| .tab.is-active { background: var(--blue); border-color: var(--blue); color: #fff; } |
| .matrix-scroll { overflow-x: auto; } |
| #matrix { |
| display: grid; grid-template-columns: minmax(78px, auto) repeat(4, minmax(74px, 1fr)); |
| gap: 4px; min-width: 440px; |
| } |
| .mcell { padding: 11px 8px; border-radius: 7px; text-align: center; font-size: 14px; } |
| .mcell-corner { background: none; } |
| .mcell-head { font-size: 12px; color: var(--ink-faint); font-weight: 600; letter-spacing: .02em; } |
| .mcell-row { text-align: right; padding-right: 12px; display: flex; align-items: center; justify-content: flex-end; } |
| .mcell-value { background: rgba(var(--heat), calc(.07 + var(--t) * .82)); color: var(--ink); font-variant-numeric: tabular-nums; } |
| .mcell-value.is-dark { color: #fff; } |
| .mcell-value.is-diagonal { font-weight: 700; box-shadow: inset 0 0 0 2px var(--ink); } |
| .mcell-value.is-diagonal.is-dark { box-shadow: inset 0 0 0 2px #fff; } |
| .axis-note { display: flex; justify-content: space-between; font-size: 12px; color: var(--ink-faint); margin-top: 10px; gap: 16px; } |
| #matrix-blurb { margin: 14px 0 0; font-size: 13.5px; color: var(--ink-soft); } |
| |
| |
| table { width: 100%; border-collapse: collapse; font-size: 14px; } |
| th, td { text-align: right; padding: 9px 10px; border-bottom: 1px solid var(--line); } |
| thead th { font-size: 12px; color: var(--ink-faint); font-weight: 600; letter-spacing: .02em; } |
| tbody th, thead th:first-child { text-align: left; font-weight: 600; } |
| tbody tr:last-child th, tbody tr:last-child td { border-bottom: none; } |
| .num-gap { color: var(--orange); font-weight: 600; } |
| .table-scroll { overflow-x: auto; } |
| |
| |
| #decomposition { display: grid; gap: 10px; } |
| .decomp-row { display: grid; grid-template-columns: 96px 1fr 74px; align-items: center; gap: 12px; } |
| .decomp-label { font-size: 13.5px; color: var(--ink-soft); } |
| .decomp-bar { display: flex; height: 20px; border-radius: 5px; overflow: hidden; background: color-mix(in srgb, var(--grey) 14%, transparent); } |
| .seg { height: 100%; } |
| .seg-ranking { background: var(--orange); } |
| .seg-placement { background: var(--blue); } |
| .decomp-total { font-size: 13px; text-align: right; color: var(--ink-soft); font-variant-numeric: tabular-nums; } |
| .legend { display: flex; gap: 18px; flex-wrap: wrap; margin-top: 16px; font-size: 12.5px; color: var(--ink-soft); } |
| .legend span { display: inline-flex; align-items: center; gap: 7px; } |
| .swatch { width: 11px; height: 11px; border-radius: 3px; } |
| |
| footer { |
| border-top: 1px solid var(--line); padding-top: 22px; margin-top: 8px; |
| font-size: 13px; color: var(--ink-faint); line-height: 1.7; |
| } |
| footer a { color: var(--ink-soft); } |
| #loading { padding: 60px 0; color: var(--ink-faint); } |
| </style> |
| </head> |
| <body> |
| <div class="wrap"> |
|
|
| <header class="top"> |
| <p class="eyebrow">Distribution shift · DistilBERT · four sentiment corpora</p> |
| <h1>undertow<span class="dim"> — the accuracy that isn't there</span></h1> |
| <p class="lede"> |
| A sentiment classifier reports one number: accuracy on a test set drawn from the corpus |
| it trained on. This page is about the other twelve. Four corpora, two classifiers — a bag |
| of words, and a linear probe on frozen DistilBERT features — and every model evaluated on |
| every domain. Pretraining is worth <strong>+1.1 pp</strong> on the test set everyone |
| reports. On the three domains nobody reports it is worth <strong>+3.5 pp</strong>, and it |
| cuts the domain-shift penalty by <strong>29%</strong>. |
| </p> |
| <div class="top-links"> |
| <a href="https://github.com/UsmarHaider/undertow">Method & code on GitHub</a> |
| </div> |
| </header> |
|
|
| <p id="loading">Loading the models…</p> |
|
|
| <main id="app" hidden> |
|
|
| <section> |
| <h2>Four models, one sentence</h2> |
| <p class="sub"> |
| These are the real fitted TF-IDF models from the run below — one per corpus, identical |
| settings, running in your browser with nothing sent anywhere. Same architecture, same |
| hyper-parameters, same 6,000 training reviews each. The only thing that differs is |
| <em>which</em> 6,000. |
| </p> |
| <div class="panel"> |
| <textarea id="text" placeholder="Paste or type a review…" spellcheck="false"></textarea> |
| <div id="examples"></div> |
| <p id="disagreement" class="note note-idle"></p> |
| <div id="verdicts"></div> |
| </div> |
| </section> |
|
|
| <section> |
| <h2>The transfer matrix</h2> |
| <p class="sub"> |
| Row is what a model trained on; column is what it was tested on. The boxed diagonal is |
| the number that gets published. Everything off it is the number you get in production. |
| Hover any cell for its AUROC, its calibration, and how far a moved threshold would take it. |
| </p> |
| <div class="panel"> |
| <div id="method-tabs"></div> |
| <div class="matrix-scroll"><div id="matrix"></div></div> |
| <div class="axis-note"><span>rows: trained on</span><span>columns: evaluated on</span></div> |
| <p id="matrix-blurb"></p> |
| </div> |
| </section> |
|
|
| <section> |
| <h2>In-domain versus everywhere else</h2> |
| <p class="sub"> |
| Averaged over the four training domains — sixteen cells per method, four on the diagonal |
| and twelve off it. <em>Oracle</em> is the accuracy each model would reach out-of-domain if |
| someone re-placed its decision threshold and changed nothing else: the ceiling that costs |
| no retraining. |
| </p> |
| <div class="panel table-scroll"> |
| <table> |
| <thead> |
| <tr> |
| <th>method</th><th>in-domain</th><th>out-of-domain</th><th>drop</th> |
| <th>OOD AUROC</th><th>OOD oracle</th><th>OOD ECE</th> |
| </tr> |
| </thead> |
| <tbody id="summary-body"></tbody> |
| </table> |
| </div> |
| </section> |
|
|
| <section> |
| <h2>What the drop is made of</h2> |
| <p class="sub"> |
| The loss splits cleanly in two. <strong>Ranking</strong> is signal the representation |
| genuinely no longer has — no threshold recovers it. <strong>Placement</strong> is signal |
| that is still there, sitting on the wrong side of a boundary that was chosen in a |
| different domain. |
| </p> |
| <div class="panel"> |
| <div id="decomposition"></div> |
| <div class="legend"> |
| <span><i class="swatch" style="background: var(--orange)"></i> ranking — genuinely lost</span> |
| <span><i class="swatch" style="background: var(--blue)"></i> placement — a misplaced threshold</span> |
| </div> |
| </div> |
| </section> |
|
|
| <section> |
| <h2>Which shift hurts</h2> |
| <p class="sub"> |
| The four corpora form a rough 2×2 — movies against commerce, full reviews against |
| one-liners — so the twelve out-of-domain cells can be grouped by what kind of distance |
| each one actually crosses. |
| </p> |
| <div class="panel table-scroll"> |
| <table> |
| <thead><tr id="shift-head"></tr></thead> |
| <tbody id="shift-body"></tbody> |
| </table> |
| </div> |
| </section> |
|
|
| <footer> |
| <p> |
| Corpora: <a href="https://ai.stanford.edu/~amaas/data/sentiment/">IMDB</a> (Maas et al., 2011), |
| <a href="https://gluebenchmark.com/tasks">SST-2</a> (Socher et al., 2013), |
| Yelp and Amazon polarity (Zhang et al., 2015). Encoder: distilbert-base-uncased. |
| Every number on this page is regenerated by <code>make all</code>. |
| </p> |
| <p> |
| Built by <a href="https://github.com/UsmarHaider">Usmar Haider</a> · |
| <a href="https://github.com/UsmarHaider/undertow">source, method and full write-up</a> · MIT |
| </p> |
| </footer> |
| </main> |
|
|
| </div> |
| <script src="undertow.js"></script> |
| </body> |
| </html> |
|
|
