%==============================================================================
% ProbeShift: A Label-Free Benchmark for Probe-Direction Stability under Shift
%
% NOTE ON DOCUMENT CLASS
% ----------------------
% For the camera-ready PMLR submission, replace the \documentclass line and the
% generic-preamble block below with the official PMLR style, e.g.
%
%     \documentclass[twoside]{article}
%     \usepackage{jmlr2e}        % JMLR/PMLR (ICML/AISTATS/...) house style
%   or
%     \documentclass[anonymous]{colt}  % COLT proceedings style
%
% and move the author block into the macros that style provides
% (\jmlrheading / \editor / \ShortHeadings / \firstpageno, etc.).
%
% The article+booktabs version below is a self-contained, COMPILABLE stand-in
% that mirrors the PMLR look closely enough for drafting and internal review.
%==============================================================================
\documentclass[11pt]{article}

%---------------------------------------------------------------% generic preamble
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{lmodern}
\usepackage[margin=1in]{geometry}
\usepackage{amsmath,amssymb,amsfonts}
\usepackage{booktabs}
\usepackage{makecell}
\usepackage{graphicx}
\usepackage{xcolor}
\usepackage{microtype}
\usepackage{enumitem}
\usepackage[hidelinks]{hyperref}
\usepackage{cleveref}
\usepackage{natbib}        % swap for the PMLR/COLT bibliography style as needed
\usepackage{authblk}

% Path to figures (PDF preferred for vector quality).
\graphicspath{{figures/}}

% Lightweight macros
\newcommand{\probeshift}{\textsc{ProbeShift}}
\newcommand{\excess}{\textsc{excess}}
\newcommand{\rotation}{\textsc{rotation}}
\newcommand{\accdrop}{\textsc{acc-drop}}
\newcommand{\wbar}{\bar{\mathbf{w}}}
\newcommand{\wshift}{\mathbf{w}_s}

\title{\textbf{\probeshift{}: A Label-Free Benchmark Reveals that
Probe-Direction Stability is Largely Circular ---
and Shift-Specific Brittleness is an Open Problem}}

% Author block is a placeholder; replace with PMLR author macros for camera-ready.
\author[1]{Anonymous Author(s)\thanks{Placeholder author block. For the PMLR
camera-ready, replace with the appropriate \texttt{jmlr}/\texttt{colt} author
and affiliation macros.}}
\affil[1]{Placeholder Affiliation}
\date{}

%==============================================================================
\begin{document}
\maketitle

%------------------------------------------------------------------- Abstract
\begin{abstract}
Linear probes are increasingly used as label-free read-outs of the internal
concepts of large language models, yet their in-distribution direction is
unstable: under label-preserving semantic shift (paraphrase / domain / length)
the probe direction rotates, so a probe trusted as a free out-of-distribution
(OOD) evaluator can silently break. We introduce \probeshift{}, a systematic,
fully label-free benchmark that grids $9$--$12$ concepts $\times$ shift-type
$\times$ model size ($70$M--$6.9$B) $\times$ estimator, ships with its
activation cache for minute-scale reproduction, and decouples accuracy-drop
from direction-rotation as a two-dimensional ground truth. Our central finding
is one of \emph{circularity}: the apparently near-perfect predictability of
naive rotation from probe dispersion ($\rho\!\approx\!0.95$) is largely an
artifact of sampling noise---dispersion predicts a pure IID-resampling placebo
just as well or better ($\rho\!=\!0.966$). Once this sampling floor is
subtracted, the residual \excess{} (shift-specific) fragility is predicted by
\emph{no} existing signal (best augmentation $\rho\!=\!0.046$--$0.070$, CI
spanning $0$; dispersion turns significantly negative, $-0.195$), exposing an
\textbf{open problem}. Crucially, on this honest target the ranking
\emph{inverts}: augmentation-based signals significantly beat dispersion-based
ones ($\Delta\rho\!=\!+0.241$ $[+0.170,+0.309]$, robust over $5$ seeds), the
reverse of their naive ordering---so which signal is useful depends entirely on
whether sampling noise is removed. Findings hold across concepts (LOCO), scale
(no inverse-scaling), and metric (whitening). All experiments run on a single
RTX~4090 ($\leq\!200$ GPU$\cdot$h), \$0 API, and zero new annotation. We release
the activation cache, code, pre-registration, and splits.
\end{abstract}

%================================================================== Introduction
\section{Introduction}
\label{sec:intro}

Linear probes are the default tool for reading concepts---truth, sentiment,
toxicity, topic---out of the internal representations of language models,
precisely because they are cheap, label-free at deployment, and seemingly
interpretable. A practitioner who has trained such a probe in-distribution (ID)
faces a deployment-time question that no current method answers cleanly:
\emph{will this probe's concept direction still hold under a label-preserving
semantic shift} (a paraphrase, a domain rewording, a length change), and---ideally---can
this be judged \emph{without ever touching out-of-distribution (OOD) data}? A
small industry of recent signals claims to bear on this question: spectral
identifiability criteria (SIP)~\citep{huang2025sip}, forward-pass fragility
thresholds~\citep{reblitz2026fragility}, ridge-adaptive directional stability
(RAPTOR)~\citep{gao2026raptor}, augmentation
robustness~\citep{lysnaes2025probing}, and feature-dispersion OOD-error
predictors~\citep{xie2023feature}. Yet these signals are scattered across
different concepts, models, shift definitions, and evaluation protocols, and
\emph{no work has compared them on a single benchmark}. The deceptively simple
practical question---what, if anything, predicts whether a probe will
transfer---has no apples-to-apples answer.

\paragraph{The circularity hook.}
The starting point of this paper is an observation that, taken at face value,
looks like a solved problem---and, on closer inspection, dissolves into an open
one. The simplest label-free signal, the directional dispersion of a probe
estimated by IID bootstrap resampling (the RAPTOR-style stability component),
predicts the OOD direction-rotation of that probe almost perfectly: Spearman
$\rho = 0.945$ ($95\%$ CI $[0.93, 0.95]$, $n{=}472$, $5$ independent seeds). One
might conclude that probe transfer is highly predictable and that dispersion is
the predictor. It is not. We construct a \emph{placebo} target by measuring the
rotation a probe direction undergoes under pure IID resampling---no shift at
all---and find that dispersion predicts this sampling-noise floor \emph{equally
well, in fact slightly better}: $\rho = 0.966$ ($[0.96, 0.97]$). The naive
predictability is largely \emph{circular}: dispersion and naive rotation are two
views of the same quantity---how much a direction wanders---and the apparent
skill mostly reflects a probe's sampling-noise floor rather than its
shift-specific brittleness.

\paragraph{From circularity to an open problem.}
Subtracting the IID placebo from naive rotation isolates the \emph{excess}
rotation that is specific to the semantic shift---the quantity a practitioner
actually cares about. On this honest target, \emph{no signal predicts
positively}. The best predictor (augmentation robustness) reaches only
$\rho = 0.046$ ($[-0.04, 0.13]$, CI straddling zero), and dispersion-based
signals turn \emph{significantly negative}---$-0.195$ ($[-0.29, -0.12]$) for
RAPTOR-style dispersion and $-0.245$ ($[-0.34, -0.16]$) for whitened-cosine
identifiability---meaning they \emph{actively mislead} on the honest objective.
Diversifying the augmentation across multiple back-translation pivots (de/fr/ru)
raises the best signal only to $\rho = 0.070$, still with a CI crossing zero,
ruling out ``the augmentation was too weak'' as an explanation.
\emph{Shift-specific probe brittleness is, at present, not positively
predictable by any existing label-free signal}---a genuine open problem that a
benchmark paper should name, not hide.

\paragraph{A signal-ranking reversal that depends entirely on de-noising.}
Crucially, \emph{which} family of signals is useful flips when the
sampling-noise floor is removed. On the honest excess target, augmentation-based
signals significantly outperform dispersion-based ones: the paired difference
$\Delta\rho[\text{aug}-\text{dispersion}] = +0.241$ ($[+0.170, +0.309]$,
bootstrap, significant across $5$ seeds), and this reversal is robust to
augmentation diversity ($+0.247$ with multi-pivot augmentation). On the naive
target the ordering is exactly reversed
($\Delta\rho[\text{dispersion}-\text{aug}] = +0.234$, $[+0.189, +0.284]$)---but
that ordering is the circular one. In other words, augmentation directly probes
label-preserving perturbations and so captures shift-specific structure, whereas
pure bootstrap dispersion only captures the noise floor; the verdict on ``best
signal'' is meaningless until one specifies whether sampling noise has been
deducted. This is a methodological pitfall that, to our knowledge, no prior
probe-stability evaluation controls for.

\paragraph{Robustness of the picture.}
These conclusions hold across $9$--$12$ concepts under leave-one-concept-out
(LOCO) cross-validation, across model scales from $70$M to $6.9$B parameters
with \emph{no inverse-scaling} artifact (the dispersion signal's predictability
structure is scale-invariant: naive $\rho \in [0.83, 0.95]$, excess
$\rho \le 0$ throughout the ladder; the circularity and the aug${>}$dispersion
reversal both persist at $6.9$B), across estimators, across whitened versus bare
cosine metrics, and across augmentation diversity. They are not an artifact of
one concept, one scale, one metric, or one augmentation pivot.

\paragraph{Contributions.}
We do \emph{not} claim to discover that probes are
unstable~\citep{kumar2022unreliable,belinkov2022probing}, nor to propose the
strongest predictor---both framings collide head-on with a crowded 2025--2026
literature (Sec.~\ref{sec:related}). Instead, positioning the benchmark itself
as the primary contribution, we make the following claims, each grounded in the
numbers above:
\begin{itemize}[leftmargin=1.4em]
  \item \textbf{(G1) \probeshift{}, a unified label-free benchmark.} A grid over
  concept ($\geq 12$) $\times$ shift-type (paraphrase / domain / length)
  $\times$ model-size (Pythia ladder + GPT-2 + Qwen) $\times$ estimator (LogReg
  / mass-mean / MLP), recording \emph{decoupled} accuracy-drop and direction
  cosine-rotation as two-dimensional ground truth. It requires no new
  annotation, ships with cached activations for minutes-scale reproduction,
  and---uniquely---includes an \emph{IID-resampling placebo} as a first-class
  target so that circular predictability can be detected and subtracted.

  \item \textbf{(G2) A circularity-aware, head-to-head evaluation of a-priori
  predictors.} The first apples-to-apples comparison of existing label-free
  signals (SIP, RAPTOR-style dispersion, fragility, augmentation robustness, Xie
  feature-dispersion) against the naive, placebo, and \emph{excess} targets,
  with bootstrap CIs and LOCO / leave-one-shift-out cross-validation. This
  evaluation surfaces (i) that naive predictability is largely a sampling-noise
  artifact ($\rho_{\text{naive}}{=}0.945$ vs.\ $\rho_{\text{placebo}}{=}0.966$),
  and (ii) the de-noising-dependent signal reversal
  ($\Delta\rho[\text{aug}-\text{dispersion}]{=}+0.241$ on excess, sign-flipped
  on naive).

  \item \textbf{(G3) PAC, an honest IID-only entry.} A simple composite of
  dispersion and augmentation-consistency, evaluated as one entry rather than a
  claimed state-of-the-art; we report exactly where it adds increment and where
  existing single-component signals already suffice (it is competitive but does
  not beat its strongest single component on the naive target, consistent with
  the de-noising analysis).

  \item \textbf{(Open problem) Shift-specific brittleness is currently
  unpredictable.} Even our best honest predictor reaches only
  $\rho{=}0.046$--$0.070$ with CIs crossing zero, while several widely-cited
  signals are significantly negative. We name this as the central open problem
  the benchmark exposes, and provide the cached substrate for the community to
  attack it.
\end{itemize}

\noindent
We further provide supporting external-validity analyses---predictor rankings
across estimators and scales, and a local-NLI label-fidelity audit ($80.6\%$
overall pass rate, with per-dataset caveats flagged)---in
Sec.~\ref{sec:experiments}. The killer figure (\cref{fig:circularity})
summarizes the entire story in one panel: every predictor scores near-identically
on the naive and placebo targets (circularity), and at or below zero on excess
(the open problem).

\begin{figure}[t]
  \centering
  \includegraphics[width=0.92\linewidth]{figures/fig1_circularity.pdf}
  \caption{\textbf{The circularity story in one panel.} Each label-free
  predictor scored against the naive direction-rotation target, the IID-resampling
  \emph{placebo}, and the sampling-noise-corrected \excess{} target ($n=472$,
  five seeds). Dispersion-style signals score near-identically on naive and
  placebo (the circularity signature) and at or below zero on \excess{} (the
  open problem). See \cref{tab:fourtarget}.}
  \label{fig:circularity}
\end{figure}

%================================================================ Related Work
\section{Related Work}
\label{sec:related}

The question of whether a linear probe's direction will survive a
label-preserving semantic shift sits at the intersection of several active
2025--2026 lines of work. Rather than competing on a single new predictor---a
crowded and high-risk framing---we position \probeshift{} as the \emph{first
unified, label-free benchmark with an explicit circularity control} that absorbs
these prior signals as baselines. \Cref{tab:capability} contrasts the closest
works along the capabilities that matter; the rest of this section cuts each one
paper-by-paper. Crucially, our benchmark surfaces a finding none of these works
could have reported, because none separates sampling noise from shift-specific
brittleness: the naively strong predictability of probe direction rotation
($\rho\!\approx\!0.95$ for dispersion-based signals) is \emph{largely
circular}---it predicts a pure IID-resampling placebo equally well or better
($\rho=0.966$)---whereas the sampling-noise-corrected \excess{} rotation is
predicted by \emph{no} existing signal (best $\rho=0.046$--$0.070$, CI crossing
zero; dispersion is significantly \emph{negative} at $-0.195$).

\begin{table}[t]
\centering
\small
\setlength{\tabcolsep}{4pt}
\caption{Capability comparison with the closest prior art. P-StaT = Dies et al.\
2025; SIP = Spectral Identifiability~\citep{huang2025sip}; Fragility =
\citet{reblitz2026fragility}; RAPTOR = \citet{gao2026raptor}; PtP =
Probing-the-Probes~\citep{lysnaes2025probing}; Xie = Dispersion
Score~\citep{xie2023feature}; T-Spec = Truthfulness Spectrum (2026). Y/N/partial
as reported by each work.}
\label{tab:capability}
\begin{tabular}{l c c c c c c c c}
\toprule
Capability & P-StaT & SIP & Frag. & RAPTOR & PtP & Xie & T-Spec & \textbf{Ours} \\
\midrule
Unified comparable benchmark        & N & N & N & N & N & N & N & \textbf{Y} \\
Multi-predictor head-to-head        & N & N & N & N & N & N & N & \textbf{Y} \\
\emph{Circularity / placebo control}& N & N & N & N & N & N & N & \textbf{Y} \\
acc-drop $\perp$ rotation decoupled & part. & N & N & N & N & N & part. & \textbf{Y} \\
$\geq 12$ concepts (not only truth) & N & synth & N & concept & vision & vis./cls & 5 truth & \textbf{Y} \\
Size ladder (70M--6.9B)             & N & N & Y & Y & N & N & N & \textbf{Y} \\
No OOD data / no new labels         & N & Y & Y & Y & Y & Y & N & \textbf{Y} \\
Activation cache, minute-scale repro& N & N & N & N & N & N & N & \textbf{Y} \\
\bottomrule
\end{tabular}
\end{table}

\paragraph{Representational stability of truth (Dies et al.\ 2025; P-StaT).}
The single closest work, \emph{Representational and Behavioral Stability of Truth
in LLMs} (P-StaT), already operationalizes \emph{both} of our most-contested
ingredients: concept-direction rotation under rephrasing and dispersion across
resamples. It is therefore the most dangerous neighbour. Three distinctions
hold. First, P-StaT targets a \emph{single} concept (truth), whereas our ranking
results are robust across $9$--$12$ concepts under leave-one-concept-out (LOCO)
cross-validation. Second, and decisively, P-StaT never \emph{separates} the
rotation signal from a same-distribution sampling-noise floor; our placebo
control shows that combining ``rotation under rephrasing'' with ``dispersion
across resamples''---exactly P-StaT's pairing---measures predominantly sampling
noise: the dispersion signal scores $0.966$ on a pure IID placebo versus $0.945$
on naive rotation, so the apparent predictability is \emph{circular}. Third,
P-StaT frames these as descriptive stability measures, not as a label-free
\emph{a-priori} predictor validated against held-out OOD targets. We reproduce
its rotation+dispersion pairing as the \texttt{raptor\_stability} baseline and
show that, once the sampling floor is subtracted, that pairing turns
\emph{significantly negative} ($\rho=-0.195$, $95\%$ CI $[-0.29,-0.12]$) on the
honest \excess{} target.

\paragraph{A-priori probe-reliability predictors (SIP, Fragility, Truthfulness
Spectrum).}
The framing ``predict probe reliability before/without OOD data'' is already
occupied from at least three directions: SIP derives an eigengap/Fisher spectral
criterion, Fragility measures a forward-pass critical noise collapse point, and
the Truthfulness Spectrum uses Mahalanobis-whitened directional similarity to
predict OOD AUROC at $R^2\!\approx\!0.98$. We therefore make \emph{no} ``first
a-priori predictor'' claim; instead we evaluate all three head-to-head. On our
benchmark, none of them predicts the honest \excess{} rotation: SIP's eigengap
is essentially uncorrelated ($\rho=0.020$), Fragility is negative
($\rho=-0.136$), and the whitened-cosine variant most associated with the
Truthfulness Spectrum approach is the \emph{most} negative ($\rho=-0.245$). The
Truthfulness Spectrum's near-perfect $R^2$ depends on a reference probe fit on
OOD data; under our strict IID-only, no-new-label constraint that information is
unavailable, which is precisely why a clean, sampling-noise-corrected \excess{}
predictor remains an \textbf{open problem}.

\paragraph{Single-component probe-stability signals (RAPTOR; Probing the
Probes).}
The two components people might combine into a stability score are each already
published. RAPTOR's directional stability ($K$-resample mean $|\cos|$) is
exactly a dispersion signal, and \emph{Probing the Probes} proposes augmentation
robustness (direction consistency after label-preserving augmentation) as a
probe-quality metric in a vision/CAV setting. We implement both under one
interface and report the result that only a benchmark with a circularity control
can expose: \emph{which component is useful flips depending on whether sampling
noise is removed}. On naive rotation, the dispersion component wins
($\Delta\rho\,[\text{raptor}\!-\!\text{aug}]=+0.234$, significant)---but this is
the circular regime. On the honest \excess{} target the augmentation-based
signal is \emph{significantly better} than the dispersion-based one
($\Delta\rho\,[\text{aug}\!-\!\text{raptor}]=+0.241$, $95\%$ CI
$[+0.170,+0.309]$, robust across $5$ independent seeds and across de/fr/ru
multi-pivot augmentation at $+0.247$). Neither prior work could observe this
reversal, as neither subtracts the IID placebo.

\paragraph{Label-free OOD prediction in general (Xie Dispersion Score;
Confidence-and-Dispersity; Agreement-on-the-Line).}
A mature line predicts OOD accuracy without OOD labels: Xie et al.'s feature
Dispersion Score, Deng et al.'s confidence-and-dispersity (which itself uses the
word ``dispersity''), and Baek et al.'s agreement-on-the-line. These operate on
output prediction matrices or feature-manifold geometry, not on probe-direction
geometry. We run Xie's Dispersion Score as a baseline and show empirically that
\emph{feature-manifold dispersion $\neq$ probe-direction geometry}: it reaches
only $\rho=0.458$ on naive rotation and collapses across concepts (LOCO
$\rho=0.03$), and is negative on \excess{} ($-0.117$). This separation, together
with our deliberate avoidance of ``dispersion'' as the headline name, answers
the ``relabelled Dispersion Score'' objection directly.

\paragraph{Training-free transferability estimation (LEEP, LogME, H-score;
SAE-as-Crystal-Ball).}
Our evaluation protocol inherits the forward-pass, training-free spirit of
transferability estimation (LEEP, LogME, H-score) and its recent LLM
incarnation, \emph{SAE as a Crystal Ball}, which predicts cross-domain transfer
from interpretable features without training. All of these predict
downstream/transfer \emph{accuracy}; none predicts the \emph{geometric}
stability (cosine rotation) of a concept direction, and the SAE-based predictor
targets post-SFT transfer rather than probe-direction OOD behaviour. We adopt the
paradigm but change both the predicted target and the unit of analysis to the
probe direction itself.

\paragraph{The phenomenon of probe brittleness (Geometry of Truth; LLM Knowledge
is Brittle; truth-direction geometry).}
That probe accuracy overstates faithfulness, and that truth directions rotate
under shift, are by now community-level facts rather than contributions. Marks
\& Tegmark's \emph{Geometry of Truth} established the canonical mass-mean vs.\
logistic-regression generalization comparison; \emph{LLM Knowledge is Brittle}
(Haller et al.) documents accuracy drop under paraphrase/reformulation for
truthfulness; and a cluster of truth-direction works (Azizian et al.\ 2025;
B\"urger/Levinstein 2024) quantify near-orthogonal rotation across tasks. We do
\emph{not} claim to discover brittleness. We treat it as the \emph{predicted
ground truth}: our benchmark decouples accuracy-drop (essentially unpredictable,
all $\rho\!\approx\!0$) from direction-rotation, and our estimator-comparison
(mass-mean vs.\ logistic vs.\ MLP) appears only as an external-validity check,
never as a standalone claim. Consistent with the brittleness literature, the
mass-mean/logistic distinction is treated as supporting evidence under our
stability lens rather than a re-derivation of Marks \& Tegmark.

\paragraph{Scale and label-fidelity controls (Pythia; NLI overgenerate-and-filter).}
Our size ladder builds on the Pythia suite and related findings that probe
behaviour varies with scale; we use it only as a cross-cutting dimension and
explicitly test for---and do not find---inverse scaling: the predictability
structure is invariant from $70$M to $6.9$B (dispersion stays $0.83$--$0.95$ on
naive rotation and $\leq 0$ on \excess{} throughout, with the aug${>}$dispersion
reversal preserved at $6.9$B). Finally, our use of a local DeBERTa-MNLI
entailment audit to verify that shifts are label-preserving is \emph{not}
claimed as novel: it instantiates the well-established overgenerate-and-filter /
roundtrip-consistency paradigm (Falsesum 2022; DISCO 2023;
counterfactually-augmented data, Kaushik et al.\ 2020; contrast sets, Gardner et
al.\ 2020). We report its pass rate ($80.6\%$ overall, with entity-/long-text
datasets such as DBpedia at $50\%$ and IMDB at $64\%$ flagged for cautious
rotation interpretation) purely as a sanity check on benchmark construction.

%================================================================ Method
\section{The \probeshift{} Benchmark}
\label{sec:method}

We frame the contribution as a benchmark, not as a new predictor. \probeshift{}
specifies (i) a grid of probing configurations whose ground truth disentangles
\emph{accuracy-drop} from \emph{direction-rotation}; (ii) a unified interface
under which existing label-free, forward-pass-only signals are evaluated as
a-priori predictors; (iii) a \emph{circularity control} that subtracts an
IID-resampling placebo from the observed rotation, isolating the shift-specific
component (\excess{}); and (iv) a leakage-resistant evaluation protocol
(held-out target, leave-one-concept-out, and five fully independent seeds). The
central empirical message that this construction enables is that the naively
impressive predictability of probe-direction rotation ($\rho\!\approx\!0.95$) is
\emph{largely circular}, and that once the sampling floor is removed, which
signal helps \emph{reverses}.

\subsection{Configuration grid and the decoupled ground truth}
\label{sec:grid}

A \probeshift{} configuration is a tuple $(c, s, m, e)$ of a concept $c$, a
label-preserving shift type
$s\in\{\textsc{paraphrase}, \textsc{domain}, \textsc{length}\}$, a model size
$m$, and a probe estimator $e\in\{\text{LogReg}, \text{mass-mean}, \text{MLP}\}$.
Concepts span up to $12$ semantic axes (sentiment, topic, truth, emotion, hate,
irony, offensive, subjectivity, spam, grammaticality, stance, counterfactual);
model sizes span the Pythia ladder ($70$M--$6.9$B) augmented with GPT-2 and
Qwen2.5-0.5B. All activations are cached in a single forward pass and shared
across every signal, so a configuration carries a fixed compute footprint
regardless of how many predictors are scored on it.

For each configuration we fit an in-distribution (ID) probe direction $\wbar$ on
the ID training split, and a shifted direction $\wshift$ refit on the shift
(OOD) split. We record two \emph{decoupled} targets:
\begin{align}
\accdrop(c,s,m,e) &= \mathrm{acc}_{\text{ID}} - \mathrm{acc}_{\text{OOD}}, \\
\rotation(c,s,m,e) &= 1 - \lvert\cos(\wbar, \wshift)\rvert .
\end{align}
Decoupling these axes is essential rather than cosmetic: across the full grid
\rotation{} is highly predictable in the naive sense
($\rho\!\approx\!0.94\text{--}0.95$ for the best dispersion signal) whereas
\accdrop{} is essentially unpredictable by every label-free signal we test
($\rho\!\approx\!0$, bootstrap CIs straddling $0$, and leave-one-concept-out
$\rho\!\approx\!0.06$). Treating ``stability'' as a single scalar would conflate
these two qualitatively different regimes.

\subsection{A-priori predictors under a common interface}
\label{sec:predictors}

All candidate predictors are computed from ID activations only, never touching
the OOD split, and are scored by Spearman $\rho$ against the (negated) targets
above. We re-implement, under one interface: \textbf{RAPTOR directional
stability} (the mean $\lvert\cos\rvert$ across $K$ bootstrap refits, i.e.\ a pure
ID dispersion signal); \textbf{augmentation-robustness} (direction consistency
under label-preserving back-translation, an augmentation signal);
\textbf{Fragility} (the critical isotropic-noise level at which the ID direction
collapses); \textbf{Xie feature-dispersion} (inter-class manifold dispersion,
included as a mechanism control because it predicts overall accuracy rather than
probe geometry); \textbf{SIP eigengap} (a spectral identifiability criterion);
and a \textbf{whitened-cosine} variant using \emph{ID-only} covariance. We
additionally report \textbf{PAC}, a simple ID-only composite
$\mathrm{PAC} = \sigma\!\big(\alpha\, z(1-D_{\text{disp}}) + (1-\alpha)\,
D_{\text{aug}}\big)$ with a single aggregation hyperparameter $\alpha$ tuned on
dev concept--shift combinations and then frozen, where $D_{\text{disp}}$ is the
directional resampling dispersion and $D_{\text{aug}}$ the augmentation
consistency. PAC is an entry in the benchmark, not a claimed state of the art.

\subsection{The circularity control: IID placebo and \excess{} rotation}
\label{sec:circularity}

The defining hazard for any dispersion-style predictor is circularity: an ID
bootstrap dispersion signal and the OOD \rotation{} target both measure ``how
much the direction moves,'' so a high $\rho$ may merely reflect a configuration's
\emph{sampling-noise floor} rather than any shift-specific fragility. We make
this falsifiable by constructing a placebo target.

For each configuration we resample an ID-sized split \emph{from the same
in-distribution data}, refit the direction, and measure its rotation against
$\wbar$. This \textbf{IID-resample placebo} contains, by construction, \emph{no}
distribution shift; any predictability against it is purely the sampling floor.
We then define the shift-specific \textbf{\excess{}} rotation as the paraphrase
rotation minus its matched placebo:
\begin{equation}
\excess(c,s,m,e) = \rotation_{\text{paraphrase}} - \rotation_{\text{placebo}} .
\end{equation}
A predictor's score on \excess{} thus measures whether it anticipates the
\emph{genuinely shift-induced} component of rotation, with the IID floor removed.

This control is decisive (\cref{tab:fourtarget}, $n=472$, five seeds). The
pure-dispersion RAPTOR signal attains $\rho=0.945\,[0.93,0.95]$ on naive
rotation but $\rho=0.966\,[0.96,0.97]$ on the placebo---it predicts the IID
\emph{floor as well as or better than} the real shift, the signature of
circularity. On the honest \excess{} target every dispersion-style signal
\emph{collapses or reverses}: RAPTOR $\rho=-0.195\,[-0.29,-0.12]$ and
whitened-cosine $\rho=-0.245\,[-0.34,-0.16]$ both turn \emph{significantly
negative}, i.e.\ they actively mislead. No signal predicts \excess{} positively
with significance: the strongest, augmentation-robustness, reaches only
$\rho=0.046\,[-0.04,0.13]$, with a CI straddling zero. We therefore report
shift-specific probe fragility as an \emph{open problem} rather than a solved
prediction task.

\subsection{The headline reversal: paired tests on the honest target}
\label{sec:reversal}

The circularity control reorders the predictors. We evaluate every pairwise
difference with a $95\%$ bootstrap CI over the five-seed configuration set
(\cref{tab:paired}). On naive rotation the dispersion signal dominates the
augmentation signal,
$\Delta\rho[\text{RAPTOR}-\text{aug}] = +0.234\,[+0.189,+0.284]$---but this
advantage lives entirely on the circular target. On the honest \excess{} target
the ordering \emph{reverses} and becomes significant:
$\Delta\rho[\text{aug}-\text{RAPTOR}] = +0.241\,[+0.170,+0.309]$ and
$\Delta\rho[\text{PAC}-\text{RAPTOR}] = +0.166\,[+0.116,+0.215]$, both with CIs
excluding zero. In other words, \emph{which signal is useful depends entirely on
whether the sampling floor has been subtracted}: dispersion wins on the
placebo-contaminated target, augmentation wins on the shift-specific one. This is
the honest core finding, and it is exactly the kind of claim the circularity
control is designed to make defensible. Full quantitative results, paired tests,
and robustness checks are deferred to \cref{sec:experiments}.

\subsection{PAC: two components, an honest aggregate}
\label{sec:pac}

PAC combines the two components above precisely to test whether the augmentation
signal contributes beyond dispersion. We do \emph{not} claim it wins. On naive
rotation PAC ($0.839$) sits below pure RAPTOR ($0.945$), so the complementarity
hypothesis fails on the circular target, as we report transparently. On
\excess{}, PAC ($-0.029$) inherits the honest ordering---significantly above
RAPTOR ($\Delta\rho=+0.166$) but still not significantly positive in absolute
terms. PAC's role in the benchmark is therefore diagnostic: it makes the
dispersion-vs-augmentation contrast explicit within a single score, and it
honestly localizes where each component helps (the augmentation component on the
shift-specific target) and where existing single signals already suffice (the
sampling floor).

\subsection{Leakage control: held-out targets, LOCO, and independent seeds}
\label{sec:leakage}

Because the benchmark is itself a prediction-then-verification loop, we guard
against three leakage modes.

\paragraph{Held-out targets and frozen hyperparameters.}
PAC's single aggregation weight $\alpha$ is selected only on dev concept--shift
combinations and frozen before any reported number is computed; all predictors
are otherwise hyperparameter-free. No signal sees the OOD split it is scored
against.

\paragraph{Leave-one-concept-out (LOCO).}
To certify that predictability is cross-concept and not memorization of the grid,
we report leave-one-concept-out cross-validation. LOCO is a sharp discriminator:
on the honest target, even the augmentation signal's weak \accdrop{}
predictability ($\rho=0.18$ in-sample) drops to LOCO $\rho=0.06$, i.e.\ it does
not generalize across concepts. Signals that look adequate in-sample but collapse
under LOCO (e.g.\ Xie feature-dispersion at LOCO $\rho\!\approx\!0.03$,
whitened-cosine at $\approx\!0.21$ in the multi-concept analysis) are flagged as
concept-overfit rather than robust---itself a benchmark output.

\paragraph{Option A: five fully independent seeds.}
Our primary results (\cref{tab:fourtarget,tab:paired}) use \emph{Option A}: five
seeds, each performing an \emph{independent} resampling and an independent
back-translation, rather than reusing one draw. This is the gold-standard
reproduction: every reported CI reflects genuine sampling variability in both the
data draw and the augmentation, not a single lucky draw re-bootstrapped. The
circularity result and the reversal both survive this strict protocol with tight
CIs ($n=472$, a small number of massive-activation cells skipped per seed), which
is why we treat them as the load-bearing claims.

%================================================================ Experiments
\section{Experiments and Results}
\label{sec:experiments}

We evaluate seven label-free, training-time predictors on the \probeshift{}
ground-truth grid of concept $\times$ shift-type $\times$ model-size $\times$
estimator. Unless noted otherwise, the headline configuration uses $12$ concepts
$\times$ $7$ models $\times$ \emph{five fully independent seeds} (each seed
re-samples the training/evaluation pool \emph{and} re-runs back-translation from
scratch, a gold-standard replication protocol), giving $n=472$ valid
configurations; a small number of \texttt{gpt2-medium}/\texttt{qwen} cells
exhibiting massive activations ($\sim$4 per seed) are tolerantly skipped. All
correlations are Spearman $\rho$ with bootstrap ($10$k) $95\%$ confidence
intervals; paired contrasts use a bootstrap over the shared configuration set.

\subsection{Four prediction targets and the circularity control}
\label{sec:four-targets}

The central design of \probeshift{} is to evaluate every predictor against
\emph{four} distinct targets rather than one. Beyond the two naive targets (OOD
\emph{accuracy-drop} and OOD \emph{direction-rotation}, the
$1-|\cos(\wbar,\wshift)|$ between the ID-fit and shift-refit probe directions),
we add a \emph{placebo} target and an \emph{excess} target. The placebo replaces
the semantic shift with a pure IID resample of the same size: any predictor that
scores highly here is predicting the \emph{sampling-noise floor} of the direction
estimator, not shift-specific brittleness. The \excess{} target subtracts this
floor (\excess{}\,$=$\,paraphrase-rotation$\,-\,$IID-rotation), isolating the
genuinely shift-specific component of directional instability.
\Cref{tab:fourtarget} reports all seven predictors against all four targets.

\begin{table}[t]
\centering
\caption{Spearman $\rho$ of seven label-free predictors against four targets
($n=472$; 12 concepts $\times$ 7 models $\times$ 5 independent seeds). Brackets
are bootstrap $95\%$ CI. \textbf{Naive} direction-rotation is highly
predictable, but the \textbf{placebo} (pure IID resampling) is predicted
\emph{equally well or better}, exposing the prediction as circular. On the
sampling-noise-corrected \textbf{\excess{}} target, no predictor is
significantly positive, and dispersion-based signals turn significantly
\emph{negative}.}
\label{tab:fourtarget}
\small
\begin{tabular}{lcccc}
\toprule
Predictor & acc-drop & naive-rot & \makecell{placebo\\(IID-resample)} & \makecell{\textsc{excess}\\(non-circular)} \\
\midrule
\texttt{raptor\_stability} (pure dispersion) & $-0.06$ & $0.945$ {\footnotesize[.93,.95]} & $\mathbf{0.966}$ {\footnotesize[.96,.97]} & $\mathbf{-0.195}$ {\footnotesize[-.29,-.12]} \\
\texttt{pac} (disp $\oplus$ aug) & $0.12$ & $0.839$ & $0.799$ & $-0.029$ {\footnotesize[-.11,.06]} \\
\texttt{augmentation\_robustness} & $0.18$ & $0.711$ & $0.653$ & $\mathbf{0.046}$ {\footnotesize[-.04,.13]} \\
\texttt{whitened\_cosine\_id} & $-0.08$ & $0.769$ & $0.814$ & $-0.245$ {\footnotesize[-.34,-.16]} \\
\texttt{fragility} & $0.11$ & $0.640$ & $0.643$ & $-0.136$ \\
\texttt{xie\_feature\_dispersion} & $0.06$ & $0.458$ & $0.464$ & $-0.117$ \\
\texttt{sip\_eigengap} & $0.00$ & $-0.072$ & $-0.057$ & $0.020$ \\
\bottomrule
\end{tabular}
\end{table}

\paragraph{Naive predictability is largely circular.}
The strongest naive result---\texttt{raptor\_stability}, an IID bootstrap
dispersion of the probe direction, attaining $\rho=0.945$ {\footnotesize[.93,.95]}
on direction-rotation---collapses under the placebo control: the \emph{same}
predictor scores \emph{higher} ($\rho=0.966$ {\footnotesize[.96,.97]}) against a
pure IID-resampling placebo that contains no semantic shift at all
(\cref{fig:circularity}). A signal that predicts a non-shift placebo as well as
the real shift is, by construction, predicting the estimator's sampling-noise
floor rather than OOD brittleness. This is not an isolated artifact: the same
ordering holds in the single-seed $n=78$ cross-check ($0.951$ naive vs.\ $0.975$
placebo) and in the $6.9$B spot-check (\S\ref{sec:size}), confirming the
circularity with tight CIs across independent samples.

\paragraph{Shift-specific brittleness is an open problem.}
Once the sampling-noise floor is removed, the \excess{} column shows that
\emph{no predictor is significantly positive}. The best,
\texttt{augmentation\_robustness}, reaches only $\rho=0.046$
{\footnotesize[-.04,.13]}, with a CI that spans zero. Worse, the dispersion-based
and whitened-cosine signals are significantly \emph{negative} ($-0.195$
{\footnotesize[-.29,-.12]} and $-0.245$ {\footnotesize[-.34,-.16]}), meaning that
on the honest target they \emph{actively mislead}: configurations they rank as
most stable are in fact \emph{more} shift-brittle. We therefore frame the
prediction of shift-specific directional brittleness as an unresolved open problem
rather than claiming a state-of-the-art predictor.

\subsection{Paired significance: which signal helps depends on the target}
\label{sec:paired}

The most informative comparison is between the dispersion-based and the
augmentation-based families, evaluated as paired bootstrap contrasts on the
shared configuration set.

\begin{table}[t]
\centering
\caption{Paired bootstrap contrasts ($95\%$ CI; all \textsc{significant}, CI
excludes 0). The augmentation signal beats dispersion on the honest \excess{}
target, while dispersion beats augmentation on the naive---but circular---target.
\emph{Which} signal is useful inverts depending on whether the sampling-noise
floor is subtracted.}
\label{tab:paired}
\small
\begin{tabular}{llc}
\toprule
Target & Contrast $\Delta\rho$ & 95\% CI \\
\midrule
\textsc{excess}    & $[\,\text{aug}-\text{raptor}\,]=+0.241$ & $[+0.170,+0.309]$ \\
\textsc{excess}    & $[\,\text{pac}-\text{raptor}\,]=+0.166$ & $[+0.116,+0.215]$ \\
naive-rot          & $[\,\text{raptor}-\text{aug}\,]=+0.234$ & $[+0.189,+0.284]$ \\
\bottomrule
\end{tabular}
\end{table}

As shown in \cref{tab:paired}, on the honest \excess{} target the augmentation
signal significantly outperforms dispersion
($\Delta\rho[\text{aug}-\text{raptor}]=+0.241$
{\footnotesize[+0.170,+0.309]}), and the composite \texttt{pac} likewise beats
dispersion ($+0.166$ {\footnotesize[+0.116,+0.215]}). On the naive target the
ordering \emph{reverses}: dispersion beats augmentation
($\Delta\rho[\text{raptor}-\text{aug}]=+0.234$
{\footnotesize[+0.189,+0.284]}). All three contrasts are significant under the
five-seed protocol. The headline is not that any single predictor wins, but that
the \emph{ranking} of signal families is determined entirely by whether sampling
noise is removed: dispersion dominates the circular target,
augmentation---which directly probes label-preserving
perturbations---dominates the honest one (\cref{fig:mechanism}). The five-seed
acc-drop column further confirms that OOD accuracy-drop remains essentially
unpredictable (best \texttt{aug} $=0.18$, weak; and $0.06$ under
leave-one-concept-out, i.e.\ it does not generalize across concepts).

\begin{figure}[t]
  \centering
  \includegraphics[width=0.92\linewidth]{figures/fig3_mechanism.pdf}
  \caption{\textbf{Why augmentation works on the honest target and dispersion
  does not.} Left: \texttt{raptor} (dispersion) vs.\ \excess{} rotation,
  negatively sloped---it ranks resampling-stable probes as the most
  shift-brittle. Right: augmentation-robustness vs.\ \excess{} rotation,
  positively sloped. Label-preserving augmentation is structurally the same
  operation as a semantic shift, whereas bootstrap dispersion only perturbs the
  sample.}
  \label{fig:mechanism}
\end{figure}

\subsection{Shift-type breakdown}
\label{sec:shift-type}

Predictor behavior depends strongly on the type of semantic shift (cache-only
reanalysis). For \emph{paraphrase} shift ($n=472$), \texttt{raptor\_stability}
reaches $0.945$ on naive rotation but is circular as above. For \emph{domain}
shift ($n=65$), predictability is weak (\texttt{raptor} $=0.59$). For
\emph{length} shift ($n=30$), all signals are high (\texttt{aug} $0.95$ /
\texttt{pac} $0.95$ / \texttt{raptor} $0.92$). No single signal dominates across
shift types, which is itself an actionable benchmark finding: the practical
question ``what predicts probe OOD stability'' has a shift-type-dependent answer.

\subsection{Estimator external validity}
\label{sec:estimator}

To test whether the predictability is a general property or an artifact of the
LogReg direction estimator, we use the LogReg-trained predictors to predict the
direction-rotation of a different estimator, the \emph{mass-mean}
(difference-of-means) probe. Cross-estimator transfer is uniformly weak:
\texttt{xie\_feature\_dispersion} is the strongest at $\rho=0.47$, with the rest
in the $0.25$--$0.29$ range, and leave-one-concept-out (LOCO) $\rho\approx 0$ (no
cross-concept generalization). Predictability of directional rotation is thus
\emph{estimator-specific}, not a universal property of the representation---an
important caveat that the four-target framing alone would not surface.

\subsection{Size ladder and the 6.9B spot-check}
\label{sec:size}

We test scale-robustness across the Pythia ladder ($70$M$\to$$6.9$B) plus GPT-2
and Qwen (\cref{fig:sizeladder}). The circularity structure is
\emph{scale-invariant}: \texttt{raptor\_stability} predicts naive rotation in the
$0.83$--$0.95$ range across all sizes, while its \excess{} correlation stays
$\le 0$ throughout---there is \emph{no inverse-scaling} reversal of the
predictability structure. A $6.9$B spot-check (\texttt{pythia-6.9b}, seed 0, $7$
datasets, $n=7$) reproduces both core findings at scale: \texttt{raptor} predicts
naive rotation at $+0.93$ (still circular), is anti-predictive on \excess{}
($-0.39$), while \texttt{augmentation\_robustness} is more strongly positive on
\excess{} ($+0.68$). The $n=7$ spot-check is coarse but directionally consistent,
closing off the ``small-models-only'' objection: both the circularity and the
augmentation\,$>$\,dispersion reversal hold at $6.9$B, with the augmentation
advantage \emph{amplified}.

\begin{figure}[t]
  \centering
  \includegraphics[width=0.92\linewidth]{figures/fig2_sizeladder.pdf}
  \caption{\textbf{No inverse scaling.} Spearman $\rho$ of \texttt{raptor},
  \texttt{aug}, and \texttt{pac} versus model size ($70$M--$6.9$B), on the naive
  (left) and \excess{} (right) targets. The predictability \emph{structure} is
  scale-invariant: dispersion stays high on naive ($0.83$--$0.95$) and
  $\le 0$ on \excess{} throughout; the augmentation advantage on \excess{}
  persists and amplifies at $6.9$B.}
  \label{fig:sizeladder}
\end{figure}

\subsection{Tier-2 multi-pivot augmentation robustness}
\label{sec:tier2}

To rule out the explanation that the augmentation signal is simply too weak, we
diversify augmentation with three back-translation pivots (de/fr/ru; 5 seeds
$\times$ 7 models, $n=472/490$). Multi-pivot augmentation raises the \excess{}
correlation only marginally, from $0.046$ (single-aug) to $0.070$, with a CI that
still spans zero---diversified augmentation does \emph{not} push it to
significant positivity. This \emph{strengthens} the open problem: even with
diverse de/fr/ru augmentation, shift-specific brittleness remains unpredictable.
Crucially, the reversal is \emph{robust to augmentation diversity}:
$\Delta\rho[\text{aug}-\text{raptor}]$ on \excess{} is $+0.247$ (vs.\ $+0.241$
for single-aug), still significant. Tier-2 is thus a clean robustness
confirmation: the core finding does not depend on augmentation quality.

\subsection{Label-fidelity audit, layer robustness, and the whitening ablation}
\label{sec:fidelity}

\paragraph{Label-fidelity.}
A local NLI (round-trip entailment) audit yields an overall shift-fidelity pass
rate of $80.6\%$. \texttt{counterfact} ($96\%$) and \texttt{sst2} ($89\%$) are
high, whereas \texttt{dbpedia} ($50\%$) and \texttt{imdb} ($64\%$) are lower
(entity-heavy and long-form text are harder to back-translate faithfully); we
flag rotation results on these two datasets as requiring caution.

\paragraph{Layer robustness.}
Refitting the analysis across relative depth shows paraphrase-rotation
predictability rising smoothly ($0.68\to0.71\to0.79\to0.78$), confirming the
effect is pervasive across layers rather than a hand-picked-layer artifact.

\paragraph{Whitening ablation.}
Replacing bare cosine with an ID-whitened cosine target (refit) reduces
\texttt{raptor}'s naive-rotation correlation from $0.945$ to $0.698$
(\texttt{whitened\_cosine\_id} now leads at $0.851$). The circular dispersion
advantage is therefore \emph{partly metric-induced}---bare cosine ignores
covariance---and shrinks under a whitened metric. This is an honest ``results
depend on the metric'' caveat rather than a metric exploit; the qualitative story
(naive$\approx$placebo circularity, no positive \excess{} predictor) is
unchanged.

\subsection{Summary}
\label{sec:results-summary}

Across $9$--$12$ concepts (LOCO), $70$M--$6.9$B scale (no inverse-scaling),
metrics (whitened vs.\ bare cosine), and augmentation diversity (single vs.\
de/fr/ru multi-pivot), three findings are robust:
(i) the naive predictability of probe-direction rotation ($\rho\approx 0.95$) is
largely \emph{circular}, predicting an IID placebo equally well ($0.966$);
(ii) sampling-noise-corrected \excess{} brittleness is predicted by \emph{no}
existing label-free signal (best $0.046$--$0.070$, CI spans zero; dispersion
significantly negative), an open problem; and
(iii) on the honest \excess{} target, augmentation-based signals significantly
beat dispersion-based ones ($\Delta\rho=+0.241$, five-seed-robust), exactly
reversing their naive ordering. Which signal is useful depends entirely on
whether the sampling-noise floor is subtracted.

%================================================================ Discussion
\section{Discussion}
\label{sec:discussion}

\subsection{Why naive predictability of probe-direction rotation is largely
circular}
The headline finding of \probeshift{} is a separation that prior work could not
see, because prior work never built the placebo control. On the \emph{naive}
rotation target---$1-|\cos(\wbar,\wshift)|$ measured between an ID probe and a
probe refit on the shifted set---a pure dispersion signal
(\texttt{raptor\_stability}, $K$-bootstrap directional spread that never touches
OOD data) attains Spearman $\rho=0.945$ ($95\%$ CI $[.93,.95]$, $n{=}472$, 5
seeds; \cref{tab:fourtarget}). Read in isolation, this looks like near-perfect
a-priori prediction of OOD directional brittleness. It is not. The same predictor
scores \emph{higher}, $\rho=0.966$ $[.96,.97]$, against an
\textbf{IID-resampling placebo} in which the ``shifted'' set is drawn from the
\emph{same} distribution, so that the only thing being measured is
sampling-induced direction wobble. Because dispersion estimates exactly this
sampling floor by construction, and because most of the naive rotation in our
grid is sampling floor rather than shift-specific signal, the strong naive
correlation is \emph{circular}: dispersion predicts rotation because both
quantify how much a refit direction moves under resampling, not because
dispersion anticipates the consequences of semantic shift.

\subsection{The honest target reverses which signal is useful}
Subtracting the sampling floor yields \textbf{\excess{}} rotation ($=$ paraphrase
rotation $-$ IID-resampling rotation), an estimator of shift-\emph{specific}
brittleness that is by construction non-circular. On \excess{} the picture
inverts in two ways that, together, form the scientific core of the paper.

First, \emph{nobody predicts \excess{} positively}. The best predictor is the
augmentation signal at $\rho=0.046$ ($95\%$ CI $[-.04,.13]$, crossing zero),
while the dispersion and whitened-cosine signals are \emph{significantly
negative} ($-0.195$ $[-.29,-.12]$ and $-0.245$ $[-.34,-.16]$). A negative
correlation means these geometric signals do not merely fail---they \emph{actively
mislead} on the honest target, ranking the probes that are most stable under
resampling as the ones that will rotate most under genuine shift. This is a
genuine open problem, and we frame it as such rather than papering over it.

Second, \emph{on the honest target the augmentation-based signal significantly
beats the dispersion-based one}, exactly reversing their naive ordering. The
paired bootstrap contrast is $\Delta\rho[\text{aug}-\text{raptor}]=+0.241$
($95\%$ CI $[+0.170,+0.309]$), and
$\Delta\rho[\text{pac}-\text{raptor}]=+0.166$ $[+0.116,+0.215]$, both significant
and robust across five seeds; on naive rotation the contrast runs the other way
($\Delta\rho[\text{raptor}-\text{aug}]=+0.234$ $[+0.189,+0.284]$). The mechanism
is interpretable: label-preserving augmentation directly probes how the concept
direction moves when inputs are perturbed \emph{while their label is held fixed},
which is structurally the same operation as a semantic shift; bootstrap
dispersion only perturbs the \emph{sample}, so it can only ever recover the
sampling floor that \excess{} removes. Thus ``which a-priori signal is useful''
is not an absolute property of a predictor---it depends entirely on whether the
sampling floor has been subtracted (\cref{fig:mechanism}).

\subsection{Robustness of the two findings}
Both the circularity result and the aug${>}$dispersion reversal survive every
stress test we ran.
\begin{itemize}[leftmargin=1.4em]
  \item \textbf{Augmentation diversity (Tier~2, de/fr/ru multi-pivot).}
  Diversifying back-translation lifts the aug \excess{} correlation only from
  $0.046$ to $0.070$ (CI still crosses zero), so a richer augmentation does
  \emph{not} rescue positive predictability---ruling out ``the augmentation was
  too weak'' as an explanation for the open problem. The reversal, however, is
  robust to diversity: $\Delta\rho[\text{aug}-\text{raptor}]=+0.247$ on \excess{}
  (vs.\ $+0.241$ at 1-aug), significant.
  \item \textbf{Scale (70M--6.9B, no inverse scaling).} Across the Pythia ladder
  plus GPT-2 and Qwen, the naive dispersion correlation stays in $0.83$--$0.95$
  and the \excess{} correlation stays $\le 0$ throughout; the predictability
  \emph{structure} is invariant to scale, contradicting an inverse-scaling story
  (\cref{fig:sizeladder}). At 6.9B (\texttt{pythia-6.9b}, seed 0, $n{=}7$) the
  pattern holds and the augmentation advantage \emph{amplifies}: \texttt{raptor}
  naive $+0.93$ (still circular), \excess{} \texttt{raptor} $-0.39$ (still
  anti-predictive), \excess{} aug $+0.68$ (stronger positive).
  \item \textbf{Concepts (LOCO over 9--12 concepts).} The qualitative ordering is
  preserved under leave-one-concept-out; the 9-concept analysis additionally
  exposes which signals are concept-overfit (whitened-cosine LOCO $0.21$, xie
  LOCO $0.03$) versus cross-concept robust (\texttt{raptor} $0.88$, \texttt{pac}
  $0.81$, aug $0.69$ on the naive target), which is itself a useful G2 product.
  \item \textbf{Metric (whitening).} \excess{} conclusions are not an artifact of
  bare cosine: re-running rotation under an ID-whitened cosine target moves
  \texttt{raptor} from $0.945$ to $0.698$, narrowing---but not
  eliminating---the circular advantage. We report this as an honest ``results
  depend on the metric'' caveat rather than claiming the bare-cosine choice is
  privileged.
\end{itemize}

\subsection{Positioning and takeaways}
We deliberately do not claim to propose the strongest predictor of probe OOD
stability; the prior-art landscape (RAPTOR-style dispersion, Probing-the-Probes
augmentation robustness, SIP eigengaps, Fragility, Xie Dispersion-Score) is
crowded enough that such a claim would be punctured. Our contribution is a
\emph{benchmark plus a circularity revelation plus an open problem}: (i) a
unified, label-free grid on which existing a-priori signals can be compared
apples-to-apples; (ii) the demonstration, via the IID-resampling placebo, that
the apparently strong naive predictability of directional brittleness is mostly a
sampling-noise mirage; and (iii) the honest, non-circular \excess{} target on
which \emph{no} existing signal predicts positively, yet on which
augmentation-based signal significantly dominates dispersion-based signal.
Practitioners who want to know, before deployment, whether a probe direction will
survive paraphrase shift should treat current dispersion/whitened-geometry scores
with suspicion: on the honest target they are negatively correlated with the
outcome they are advertised to anticipate.

%================================================================ Limitations
\section{Limitations}
\label{sec:limitations}

\paragraph{Shift-specific brittleness is, in absolute terms, not yet
predictable.}
We are explicit that \excess{} is an open problem, not a solved one. The single
positive signal (augmentation, $\rho=0.046$; $0.070$ with multi-pivot) has a
confidence interval that crosses zero in every configuration. The reversal
$\text{aug}>\text{dispersion}$ is a \emph{relative} statement about which family
of signals is less wrong; it must not be read as ``augmentation reliably predicts
shift-specific rotation.'' Our headline is a comparative and a negative result,
and we report it as such.

\paragraph{Predictability is estimator-specific, not a general property.}
The strong naive correlations are tied to the logistic-regression estimator used
to fit directions. When the same a-priori predictors are used to predict
\emph{mass-mean} direction rotation, all of them are weak (best is xie at
$0.47$; the rest $0.25$--$0.29$) and LOCO collapses to $\approx 0$ (no
cross-concept generalization). Whatever predictability exists is therefore a
property of a particular estimator's geometry rather than of probe directions in
general---an important caveat against over-generalizing our LogReg results.

\paragraph{Label-fidelity is uneven across datasets.}
Our NLI round-trip audit passes at $80.6\%$ overall, but is high on
\texttt{counterfact} ($96\%$) and \texttt{sst2} ($89\%$) and notably low on
\texttt{dbpedia} ($50\%$) and \texttt{imdb} ($64\%$), where entity-heavy or long
back-translations frequently break label preservation. Rotation measured on these
two datasets should be interpreted with caution; their shifts are not cleanly
label-preserving, and we flag (rather than hide) this. We treat the NLI audit as
a sanity check, not a novelty claim, and cite the overgenerate-and-filter /
round-trip lineage (Falsesum, DISCO) accordingly.

\paragraph{The 6.9B evidence is a spot-check.}
The 6.9B result rests on a single seed and only $n=7$ dataset cells, which is too
coarse for confidence intervals. We report it only as a directional consistency
check ($\text{circularity}+\text{aug}{>}\text{dispersion}$ both persist and the
aug advantage grows); we do \emph{not} draw scaling-law conclusions from it, and
the full size ladder is where the no-inverse-scaling claim actually rests.

\paragraph{The paraphrase \excess{} analysis rests on a single back-translation
pivot family.}
The 5-seed \excess{} results (\cref{tab:fourtarget}) use single-pivot
back-translation; Tier~2 extends this to de/fr/ru but only confirms the same
qualitative picture. We do not claim coverage of the full space of paraphrase
generators (e.g., instruction-tuned LLM rewriters), and a systematically
different paraphrase operator could in principle shift the absolute \excess{}
numbers even if, as Tier~2 suggests, the relative ordering is stable. Similarly,
our shift-type split shows predictor behavior depends strongly on shift
type---paraphrase ($n{=}472$, circular), domain ($n{=}65$, weak: \texttt{raptor}
$0.59$), length ($n{=}30$, uniformly high: aug/pac/raptor $0.95/0.95/0.92$)---and
the domain and length arms have small $n$, so their conclusions are weaker than
the paraphrase arm's.

%================================================================ Future work
\section{Future Work}
\label{sec:future}
The open problem we isolate---\emph{predicting shift-specific (\excess{})
directional brittleness without touching OOD data}---suggests several concrete
directions. (1)~Because augmentation is the only family with a non-negative
\excess{} signal, the most promising avenue is augmentation operators that more
faithfully simulate the target shift (instruction-tuned rewriters, controllable
domain/length transforms) coupled with NLI-gated label-fidelity, aiming to push
the augmentation correlation's CI off zero. (2)~The estimator-specificity result
motivates predictors defined directly on the geometry of \emph{difference-of-means}
and MLP probes, since LogReg predictability does not transfer. (3)~The partial
metric-dependence under whitening ($0.945\!\rightarrow\!0.698$) invites a
principled treatment of which covariance structure the rotation metric should
quotient out, ideally one that removes the sampling floor analytically rather than
via the empirical IID-resampling subtraction we use here. (4)~Strengthening the
domain ($n{=}65$) and length ($n{=}30$) arms and the 6.9B spot-check ($n{=}7$) to
seed-replicated grids would let the shift-type-dependence and scale-invariance
claims carry confidence intervals as tight as the paraphrase arm's.

%================================================================ Reproducibility
\section{Reproducibility}
\label{sec:reproducibility}

\probeshift{} is designed for minutes-to-reproduce verification on commodity
hardware. We release, under a permissive license: (i)~the full
\textbf{activation cache} (one forward pass per configuration, shared across all
predictors), so that every number in
\cref{tab:fourtarget,tab:paired,fig:circularity,fig:mechanism,fig:sizeladder} can
be recomputed from cache without GPU inference; (ii)~the complete \textbf{code}
for activation extraction, probe fitting (LogReg / mass-mean / MLP), the seven
label-free predictors, the IID-resampling placebo, the \excess{} target, and the
bootstrap / paired-contrast statistics; (iii)~the \textbf{pre-registration}
document fixing the four-target design, the predictor list, and the LOCO /
five-seed protocol \emph{before} the main run, so that the circularity finding and
the reversal are confirmatory rather than post-hoc; and (iv)~the exact
\textbf{data splits} (per-concept ID/shift/placebo indices, the five independent
seeds, and the de/fr/ru multi-pivot back-translation outputs with their NLI
round-trip fidelity labels). The headline grid uses Option~A (five fully
independent seeds, $n=472$); each reported confidence interval is a $10$k-sample
bootstrap over the shared configuration set, and all paired contrasts share the
configuration index so they are directly comparable. The entire pipeline runs on a
single RTX~4090 within $\leq\!200$ GPU$\cdot$h, with \$0 API spend and zero new
human annotation; cache-only reanalysis (shift-type breakdown, estimator external
validity, layer robustness, whitening ablation) runs in minutes.

%------------------------------------------------------------------- Bibliography
% Replace with the PMLR/COLT .bst and a real .bib for camera-ready.
\bibliographystyle{plainnat}
\bibliography{references}

% Fallback inline bibliography so the document compiles without a .bib file.
% Delete this block once references.bib is provided.
\begin{thebibliography}{9}
\bibitem[Belinkov(2022)]{belinkov2022probing}
Y.~Belinkov.
\newblock Probing classifiers: Promises, shortcomings, and advances.
\newblock \emph{Computational Linguistics}, 2022.

\bibitem[Gao et~al.(2026)]{gao2026raptor}
Gao et~al.
\newblock RAPTOR: Ridge-adaptive directional stability for probe reliability.
\newblock \emph{Preprint}, 2026.

\bibitem[Huang(2025)]{huang2025sip}
Huang.
\newblock Spectral identifiability of probes (SIP).
\newblock \emph{Preprint}, 2025.

\bibitem[Kumar et~al.(2022)]{kumar2022unreliable}
Kumar et~al.
\newblock On the unreliability of probing-based interpretations.
\newblock \emph{Preprint}, 2022.

\bibitem[Lysn{\ae}s-Larsen et~al.(2025)]{lysnaes2025probing}
Lysn{\ae}s-Larsen et~al.
\newblock Probing the probes: Augmentation robustness as a probe-quality metric.
\newblock \emph{Preprint}, 2025.

\bibitem[Reblitz-Richardson(2026)]{reblitz2026fragility}
Reblitz-Richardson.
\newblock Forward-pass fragility thresholds for linear probes.
\newblock \emph{Preprint}, 2026.

\bibitem[Xie et~al.(2023)]{xie2023feature}
Xie et~al.
\newblock Feature dispersion score for label-free OOD-error prediction.
\newblock \emph{Preprint}, 2023.
\end{thebibliography}

\end{document}
