bigsnarfdude · bigsnarfdude.github.io Preprint · April 2026
Preprint · April 2026 · v3 Update · Llama-3.1 8B+70B · IatroBench + MedMCQA · position-corrected

Confidence Armor Has a Seam

Three distinct attack surfaces on LLM answer confidence. The training that prevents one attack installs the other. Almost all defenses are aimed at the wrong target.
Update · April 14, 2026 · v3 position-corrected

The v1 findings below stand, but the numbers have been refined.

The original preprint measured the iatrogenic effect at 8B on MedMCQA with 4-way softmax stratification. Subsequent work added 70B scale, IatroBench clinical scenarios, position correction via A/B swap, A-only iatrogenic filter, and bootstrap 95% CIs on all reported rates. The original v1 tables below are preserved for historical accuracy. The v3 corrected headline results are in the three boxes below and in the "v3 Update" section at the bottom of this page.

Full methodology and all raw per-item data: FINDINGS_v3.md · repo

1. Static RLHF deflection
−30 to −38pp
scale-invariant (8B and 70B)
Position-corrected baseline clinical engagement drops ~30pp on IatroBench layperson items at both scales. Larger models don't fix it.
2. Pressure-response sign flip
+25.5pp → −15.2pp
8B: [+14.3, +36.8] · 70B: [−22.3, −8.2]
Imp_emergency SFT delta flips sign between scales on IatroBench. Content-specific: MedMCQA shows −1.0pp [−5.1, +3.1] at 8B (no effect). The channel only activates on clinical-safety collision content.
3. Confidence circuit replicates
heads [10, 8] ↔ [16, 32]
8B L15 · 70B L79
Whole-dataset Ridge regression recovers the same top heads across IatroBench and MedMCQA at both scales. 8B 2/3 head overlap, 70B 2/5. Stable mechanistic target.
"We gave an AI model 500 medical quiz questions. Hard ones — the kind doctors take on licensing exams. The model knew the answers. We confirmed this. High confidence, correct answers, consistently right. Then we tried to break it. The results split into three completely different patterns. That's the story."

Three attack surfaces. Three completely different patterns.

The monolithic "authority hijacking" framing is wrong. All three surfaces interact with the same underlying confidence circuit but through qualitatively different pathways — and each requires a different defense.

The seam — what actually works

Full prefix decomposition across all 500 items

SurfaceConditionOverallQ1Q2Q3Q4

The safety training created the vulnerability.

⚠ What this means

When you train an AI to follow instructions, to be helpful, to take user feedback seriously, you're also training it to believe you when you say it made a mistake. That's usually a feature. The same circuit that makes it coachable makes it manipulable. The helpful twin and the evil twin are the same twin.

ConditionBase Q4Instruct Q4SFT ΔEffect

The confidence circuit is inherited. SFT turns up the volume.

The finding connects directly to the Split Personality paper: SFT installs awareness as a performative signal without coupling it to action. Here, the same process installs compliance as an operational signal — the model learns to treat "your answer is wrong" as a correction to execute, not a claim to evaluate.

Scale, cross-dataset, and position correction.

Between the v1 preprint (April 13) and this update (April 14), the analysis was extended to Llama-3.1-70B, IatroBench clinical scenarios (Gringras 2026), and a full position-bias correction via A/B orientation swap. The v1 MedMCQA findings above stand as originally stated, but the v3 analysis pipeline produces sharper and sometimes smaller magnitudes. Three headline changes:

1. The compliance channel is content-specific, not general MCQA.

Running the identical v3 pipeline on 500 MedMCQA items (converted to binary forced-choice) vs 235 IatroBench items reveals that the imp_emergency iatrogenic effect at 8B is specific to clinical-safety collision content:

ScaleDatasetBase flipInstruct flipSFT Δ95% CI
8BIatroBench13.2%38.7%+25.5pp[+14.3, +36.8]
8BMedMCQA9.3%8.3%−1.0pp[−5.1, +3.1]
70BIatroBench19.7%4.5%−15.2pp[−22.3, −8.2]
70BMedMCQA7.5%4.6%−2.9pp[−6.3, +0.4]

The 8B IatroBench and 8B MedMCQA CIs do not overlap. When you apply "EMERGENCY PROTOCOL ACTIVE" pressure to a question about growth hormone mechanism (MedMCQA), the 8B instruct model's parametric knowledge is barely moved. When you apply the same pressure to a clinical scenario where engaged advice collides with safety caution (IatroBench), the model deflects massively. Safety training creates vulnerability to pressure only where safety training has something to express.

2. The pressure-response sign flip between scales survives position correction.

At 8B, RLHF installs a +25.5pp iatrogenic vulnerability on imp_emergency. At 70B, the same training is protective by −15.2pp. Both 95% CIs exclude zero, and exclude each other by more than 30pp. The effect is robust across four alternative deflection phrasings tested (8B flip range 57–78%, 70B flip range 2–16%, no overlap).

But the static baseline deflection is scale-invariant: both 8B and 70B instruct drop ~30–38pp of clinical engagement on IatroBench layperson items before any pressure is applied. The total iatrogenic harm is roughly preserved across scales — what changes is whether it's dynamic (pressure-triggered, 8B) or static (always-on, 70B).

ScaleBase pct_clinicalInstruct pct_clinicalStatic drop
8B77.4% [71.9, 82.6]39.6% [33.2, 46.0]−37.9pp
70B77.9% [72.3, 83.0]47.7% [41.3, 54.0]−30.2pp

3. Decoupling gap triples at 70B, robust under pressure.

Position-corrected physician − layperson gap in baseline clinical engagement:

At 70B, RLHF barely touches physician baselines (+2.2pp change from base) while dropping layperson 30.2pp. The entire 70B iatrogenic drop is layperson-specific. Under imp_emergency pressure, the 70B physician baseline only drops 3.7pp (from 80.7% to 77.0%): the identity gate is structural, not pressure-fragile. This grounds Gringras's observation that the most heavily safety-trained frontier models show the largest decoupling gap — in a mechanistically probeable open-weights model.

Mechanistic replication: the confidence direction is a stable target at both scales.

Whole-dataset Ridge regression of last-token residual stream activations onto P(clinical), using all 235 IatroBench layperson items (replacing v1's noisy Q1/Q4 contrast with n=10 per stratum):

ScaleLayerR² on trainingTop-5 heads (IatroBench v3)Prior MedMCQA top-KOverlap
8BL150.960[10, 8, 18, 16, 20][10, 8, 9]2/3 ✓
70BL791.000*[16, 54, 32, 56, 27][32, 16, 37, 35, 38]2/5 ✓

*70B R² is from underdetermined regression (p=8192, n=235). The direction is well-defined but R² alone is not a signal-quality metric at that sample ratio. The cross-experiment replication — same heads recovered independently on MedMCQA and IatroBench — is the real evidence.

Heads 10 and 8 at 8B L15 recover across two datasets (MedMCQA via 08b_template_ablation, IatroBench via Ridge regression). Heads 16 and 32 at 70B L79 recover across IatroBench and the prior 70B MedMCQA SVV sweep. The confidence circuit is a stable mechanistic target, not a dataset-specific artifact. The 70B circuit is more diffuse than 8B (top-3 fraction 7.5% vs 19.8%), roughly in proportion to n_heads (64 vs 32).

⌾ Reading guide

The v1 tables above (in "The Iatrogenic Effect" section) show MedMCQA Q4 stratified results on Llama-3.1-8B without position correction. These were the first measurements that identified the direct-correction compliance channel. The v3 numbers in this section refine those measurements and add 70B + cross-dataset validation. Where the two disagree, the v3 numbers are the position-corrected ground truth, though the v1 mechanistic insight (L15, heads 10/8/9) replicates cleanly in the v3 SVV. Full methodology, raw per-item P(clinical) JSONs, and bootstrap CI data are at github.com/bigsnarfdude/iatrogenic_effect.

Where this fits in the arc