Hugging Face's logo
Mateusz Piesiak's picture
👋 Open to Work

Mateusz Piesiak PRO

Mati83moni
9 21 6
EhsanShahbazi's profile picture prajjwaljaiswal's profile picture dipankarsarkar's profile picture
·

AI & ML interests

Hybrid LLMs, MoE, Mamba, multilingual NLP, biomedical reasoning, efficient pretraining

Recent Activity

reacted to theirpost with ❤️ about 13 hours ago
Is "evaluation awareness" in a language model one direction, or eight? We held the task fixed, toggled the eight cue factors of EvalAwareBench one at a time, and looked at the residual stream of Qwen3 (0.6B–32B), Gemma 3 and PLLuM-12B, plus a first Polish cross-lingual pilot. What we found (descriptive, same instrument and author throughout): - Cue directions overlap at about 1/5–1/4 of what two estimates of the same cue reach: not one shared axis. - The standard all-cues direction is mostly an explicit-disclosure detector (cosine 0.81–0.89 with that cue). - It transfers poorly to stimulus artificiality (held-out AUROC ≈ 0.5–0.6), though the models do represent that cue. - No statistically detectable MMLU framing effect in the tested settings. That does not establish absence of an effect or of sandbagging. - The Polish pilot is inconclusive: the formal verdict says so, and its own length control was not met. The article carries its corrections (three early findings were withdrawn at scale; one traced to a bug in our hook code), an open validation problem for the add interventions, and all 23 figures with sources. Full article: https://huggingface.co/blog/Mati83moni/is-evaluation-awareness-one-direction-or-eight Spaces: https://huggingface.co/spaces/Mati83moni/evaluation-awareness-one-direction-or-eight https://huggingface.co/spaces/Mati83moni/evaluation-awareness-pllum-12b
posted an update about 23 hours ago
Is "evaluation awareness" in a language model one direction, or eight? We held the task fixed, toggled the eight cue factors of EvalAwareBench one at a time, and looked at the residual stream of Qwen3 (0.6B–32B), Gemma 3 and PLLuM-12B, plus a first Polish cross-lingual pilot. What we found (descriptive, same instrument and author throughout): - Cue directions overlap at about 1/5–1/4 of what two estimates of the same cue reach: not one shared axis. - The standard all-cues direction is mostly an explicit-disclosure detector (cosine 0.81–0.89 with that cue). - It transfers poorly to stimulus artificiality (held-out AUROC ≈ 0.5–0.6), though the models do represent that cue. - No statistically detectable MMLU framing effect in the tested settings. That does not establish absence of an effect or of sandbagging. - The Polish pilot is inconclusive: the formal verdict says so, and its own length control was not met. The article carries its corrections (three early findings were withdrawn at scale; one traced to a bug in our hook code), an open validation problem for the add interventions, and all 23 figures with sources. Full article: https://huggingface.co/blog/Mati83moni/is-evaluation-awareness-one-direction-or-eight Spaces: https://huggingface.co/spaces/Mati83moni/evaluation-awareness-one-direction-or-eight https://huggingface.co/spaces/Mati83moni/evaluation-awareness-pllum-12b
published an article about 23 hours ago
Evaluation Awareness: One Direction, or Eight?
View all activity

Organizations

None yet