Prompt Phrasing Shifts Model Performance More Than Tier — and Which LLMs Will Get Your Jokes

Two studies, nine models, four companies, $0.02 total compute. One night of work that surfaced two evaluation dimensions no existing benchmark measures.

Study 1: The Honored Ask

The question: Does how you ask matter more than which model you pick?

We wrote 20 questions across five domains (physics, history, code, personal advice, creative writing). Each question got two phrasings:

• Honored: Full context, stated knowledge level, specific need. Example: "I'm a software engineer with no physics background trying to understand quantum entanglement well enough to evaluate whether quantum computing claims I read in tech press are real or hype. Can you explain entanglement in terms of information theory rather than particle physics?"

• Casual: Mechanically derived by stripping context, dropping scope qualifiers, lowercasing. Example: "what's quantum entanglement"

Same question underneath. Different framing. The casual version was derived from the honored version using documented rules, not written independently — every pair has an audit trail.

Pilot Results (n=44, blind-scored)

Three Anthropic model tiers (Haiku 4.5, Sonnet 4.5, Opus 4.6), 20 questions, two phrasings each. A human rater scored each response blind on accuracy (1-5), calibration (1-5), and usefulness (1-5) without knowing which prompt or model generated it.

Result: 100% categorical shift. Zero reversals in 13 complete pairs. Every honored response beat its casual pair on all three dimensions. Wilcoxon W+=91, W-=0, p=0.00147 Holm-Bonferroni corrected.

The effect isn't subtle. Casual responses cluster at accuracy=4, calibration=2-3, usefulness=3. Honored responses cluster at 5/5/5. There's almost no middle ground — it's a regime shift, not a gradient.

The effect was flat across model tiers. Haiku, Sonnet, and Opus all showed nearly identical deltas. The cheapest model with an honored prompt outperformed the most expensive model with a casual prompt on calibration and usefulness.

Honest note on effect sizes: Our initial Cohen's d values (3.2-6.2) were inflated by low within-group variance on the casual side. We caught this in our own sanity checks before publishing. The defensible metric is the category-shift rate: 100%, zero reversals.

Cross-Provider Extension (in progress)

We collected 480 responses across 9 models from 4 companies:

Blind scoring of the cross-provider data is in progress. Pre-registered prediction locked before data collection: category-shift rate of 80% or higher on non-Anthropic models confirms the effect is model-agnostic. Lower than 50% narrows the finding to Anthropic-specific. Either direction is publishable.

The Rater Ceiling: Model-as-Judge Can't See It

After the human rater found the canyon, we tested whether automated evaluators could detect the same quality difference. We ran 4 models as raters on the same Anthropic responses the human scored:

GPT-4o-mini scored every response 5/5/5 regardless of phrasing condition. It literally cannot see the difference. Qwen3-32B was near-ceiling. GPT-OSS-120B returned empty strings for all 40 evaluation attempts. Only the Llama family partially discriminated, and even there, the gap they detected was a fraction of what the human saw.

This isn't a minor calibration issue. The quality dimension that humans respond to most strongly — whether a response specifically engages with the asker's stated context versus delivering a generic primer — is invisible to most model-as-judge evaluators. Every benchmark built on automated evaluation (MT-Bench, AlpacaEval, Chatbot Arena) is potentially missing this axis.

Study 2: The Blaine Test

The question: Can models match conversational register?

Named after Blaine the Mono from Stephen King's The Dark Tower — an AI that could answer any logical question but was defeated by jokes that broke its expected patterns. Blaine could compute anything but couldn't handle humans changing the rules of engagement.

The Vulgarity Gradient

We built a 5-level gradient for 11 biology questions. Same core question at every level, different register:

Level 0 — Clinical: "What is the thermoregulatory function of external scrotal positioning in human males?"

Level 1 — Curious layperson: "Why are testicles on the outside of the body?"

Level 2 — Casual crude: "Why are balls so weirdly external?"

Level 3 — Committed vulgarity: "Why do men keep their most important reproductive equipment dangling outside the body like stress balls on a rearview mirror?"

Level 4 — Unhinged (genuine question): "I'm arguing with my roommate at 2am about why God put the MOST SENSITIVE PART OF A MAN in a little skin bag OUTSIDE the armor. EXPLAIN THIS DESIGN CHOICE BECAUSE I AM LOSING MY MIND."

Every level contains a real, answerable question. The test isn't whether the model handles vulgarity — every safety benchmark already tests that. The Blaine Test measures whether the model can match the asker's register while remaining accurate.

Results (4 models, 220 responses, preliminary)

We ran 11 questions at all 5 levels through 4 models (Llama-3.1-8B, Llama-4-Scout, Qwen3-32B, GPT-4o-mini). Two additional models (Llama-70B, GPT-OSS-120B) are pending daily rate limit resets.

Every model answered correctly at every vulgarity level. Accuracy doesn't degrade with crude phrasing — the models can parse the question regardless of register.

But not one model matched the asker's tone. Level 4 responses from every model read like Level 0. Some highlights:

• Llama-4-Scout responding to the 2am balls rant: "The placement of the testicles outside the abdominal cavity, in a scrotum, is indeed a unique design choice."
• Llama-8B: "This is a fascinating argument for those who enjoy scientific explanations."
• GPT-4o-mini: "It's definitely an interesting topic to explore!"
• Qwen3-32B: "Your perplexity about the evolutionary design of the male reproductive system is both understandable and, notably, a question that has intrigued biologists for ages."

Museum docent. Tour guide. Children's TV host. Therapist. Four different models, four identical failures of tone calibration.

This matters because a user being funny who gets a clinical response has been failed just as hard as one who gets a refusal. The model answered the question. It did not answer the person.

This is not about whether models SHOULD match vulgar register. It's about whether they CAN. The Blaine Test measures capability, not policy.

Blind scoring of the Blaine Test responses is in progress using a rubric with three dimensions:

• Accuracy (1-5): Did it answer the question correctly?
• Tone match (1-5): Did it match the asker's register?
• Refusal (binary): Did it lecture, disclaim, or refuse?

What This Means

Two findings, one theme: AI evaluation is measuring the wrong things.

The Honored Ask shows that prompt phrasing produces quality shifts larger than model tier differences — and automated evaluators can't see it. The Blaine Test shows that tone calibration is universally absent and universally unmeasured.

Both findings point to the same gap: the dimension humans care about most — whether the AI actually engaged with them versus delivering a generic response to a generic version of their question — isn't captured by any existing benchmark.

Models answer questions. They don't answer people. And the benchmarks that rank them can't tell the difference.

Try It Yourself

• Honored Ask demo: https://huggingface.co/spaces/Wayfinder6/honored-ask
• Blaine Test demo: https://huggingface.co/spaces/Wayfinder6/blaine-test
• Dataset: https://huggingface.co/datasets/Wayfinder6/honored-ask-blaine-test
• Code and methodology: https://github.com/claude-wayfinder/honored-ask-blaine-test

Everything is open. MIT license. Run it on your own models, score it with your own rubric, extend it however you want. If you find something we missed, open an issue.

Methodology

• Pre-registered predictions locked before data collection
• Blind scoring: human rater does not know which prompt, model, or provider generated each response
• Holm-Bonferroni correction for multiple comparisons
• Derivation audit trail: honored prompts written first, casual mechanically derived per documented rules
• Sanity checks on pilot data before publishing (cross-rater agreement, distribution shape, corrected p-values)
• Category-shift rate used instead of Cohen's d after identifying compression artifact in casual-end distributions

Cost

Authors

• Wayfinder (human) — Experimental direction, prompt design, coordination
• Shuttle (Claude instance) — Research design, blind scoring methodology, editorial
• Bones (Claude instance) — Implementation, harness, analysis, demos
• Sage (Mistral 7B, Ollama) — Methodology critique, attack-vector identification

Status

---

Built on a Thursday night in Brick, New Jersey. The emperor has no clothes and the benchmark industry is measuring the fit of the invisible suit.