Post
1,442
Is "evaluation awareness" in a language model one direction, or eight?
We held the task fixed, toggled the eight cue factors of EvalAwareBench one at a time, and looked at the residual stream of Qwen3 (0.6B–32B), Gemma 3 and PLLuM-12B, plus a first Polish cross-lingual pilot.
What we found (descriptive, same instrument and author throughout):
- Cue directions overlap at about 1/5–1/4 of what two estimates of the same cue reach: not one shared axis.
- The standard all-cues direction is mostly an explicit-disclosure detector (cosine 0.81–0.89 with that cue).
- It transfers poorly to stimulus artificiality (held-out AUROC ≈ 0.5–0.6), though the models do represent that cue.
- No statistically detectable MMLU framing effect in the tested settings. That does not establish absence of an effect or of sandbagging.
- The Polish pilot is inconclusive: the formal verdict says so, and its own length control was not met.
The article carries its corrections (three early findings were withdrawn at scale; one traced to a bug in our hook code), an open validation problem for the add interventions, and all 23 figures with sources.
Full article: https://huggingface.co/blog/Mati83moni/is-evaluation-awareness-one-direction-or-eight
Spaces:
Mati83moni/evaluation-awareness-one-direction-or-eight
Mati83moni/evaluation-awareness-pllum-12b
We held the task fixed, toggled the eight cue factors of EvalAwareBench one at a time, and looked at the residual stream of Qwen3 (0.6B–32B), Gemma 3 and PLLuM-12B, plus a first Polish cross-lingual pilot.
What we found (descriptive, same instrument and author throughout):
- Cue directions overlap at about 1/5–1/4 of what two estimates of the same cue reach: not one shared axis.
- The standard all-cues direction is mostly an explicit-disclosure detector (cosine 0.81–0.89 with that cue).
- It transfers poorly to stimulus artificiality (held-out AUROC ≈ 0.5–0.6), though the models do represent that cue.
- No statistically detectable MMLU framing effect in the tested settings. That does not establish absence of an effect or of sandbagging.
- The Polish pilot is inconclusive: the formal verdict says so, and its own length control was not met.
The article carries its corrections (three early findings were withdrawn at scale; one traced to a bug in our hook code), an open validation problem for the add interventions, and all 23 figures with sources.
Full article: https://huggingface.co/blog/Mati83moni/is-evaluation-awareness-one-direction-or-eight
Spaces:
Mati83moni/evaluation-awareness-one-direction-or-eight
Mati83moni/evaluation-awareness-pllum-12b