Distribution shift · DistilBERT · four sentiment corpora

undertow — the accuracy that isn't there

A sentiment classifier reports one number: accuracy on a test set drawn from the corpus it trained on. This page is about the other twelve. Four corpora, two classifiers — a bag of words, and a linear probe on frozen DistilBERT features — and every model evaluated on every domain. Pretraining is worth +1.1 pp on the test set everyone reports. On the three domains nobody reports it is worth +3.5 pp, and it cuts the domain-shift penalty by 29%.

Method & code on GitHub

Loading the models…

Four models, one sentence

These are the real fitted TF-IDF models from the run below — one per corpus, identical settings, running in your browser with nothing sent anywhere. Same architecture, same hyper-parameters, same 6,000 training reviews each. The only thing that differs is which 6,000.

The transfer matrix

Row is what a model trained on; column is what it was tested on. The boxed diagonal is the number that gets published. Everything off it is the number you get in production. Hover any cell for its AUROC, its calibration, and how far a moved threshold would take it.

rows: trained oncolumns: evaluated on

In-domain versus everywhere else

Averaged over the four training domains — sixteen cells per method, four on the diagonal and twelve off it. Oracle is the accuracy each model would reach out-of-domain if someone re-placed its decision threshold and changed nothing else: the ceiling that costs no retraining.

methodin-domainout-of-domaindrop OOD AUROCOOD oracleOOD ECE

What the drop is made of

The loss splits cleanly in two. Ranking is signal the representation genuinely no longer has — no threshold recovers it. Placement is signal that is still there, sitting on the wrong side of a boundary that was chosen in a different domain.

ranking — genuinely lost placement — a misplaced threshold

Which shift hurts

The four corpora form a rough 2×2 — movies against commerce, full reviews against one-liners — so the twelve out-of-domain cells can be grouped by what kind of distance each one actually crosses.