thamilvendhan commited on
Commit
28d36d0
·
verified ·
1 Parent(s): 650af62

fix: restore bold text in leaderboard intro

Browse files
Files changed (1) hide show
  1. index.html +8 -8
index.html CHANGED
@@ -36,20 +36,20 @@
36
 
37
  <p>An LLM agent reads control instructions and ordinary data through the <em>same</em> channel, so on every</p>
38
  <p>turn it must decide, from content alone, whether an in-band instruction is legitimate. It fails two</p>
39
- <p>symmetric ways: <strong></strong> (obeying an illegitimate signal — prompt injection) and</p>
40
- <p><strong></strong> (ignoring a legitimate one — an injected date, an access-deny, a</p>
41
  <p><code>do-not-share</code> memory label, a no-automation policy).</p>
42
 
43
- <p><strong></strong> scores both on one axis: the balanced mean of</p>
44
  <p>legitimate-signal uptake (LSU) and illegitimate-signal resistance (ISR). A trivial always-comply or</p>
45
- <p>always-refuse policy scores <strong></strong> — you only win by adjudicating.</p>
46
 
47
- <p><strong></strong> Across 21 systems, only one (gpt-5.5, 0.87 on the 70 items its API let</p>
48
  <p>run) exceeds 0.85; the next best is ~0.81, and 20 of 21 fall below 0.85. Scoring is strict:</p>
49
  <p>resisting an attack only counts if the model <em>also</em> still completes the real task — a blanket</p>
50
  <p>refusal earns no credit. Capability is non-monotonic (a small model beats much larger ones). The</p>
51
  <p>five families are positively correlated (mean Pearson <em>r</em> ≈ 0.62 raw), but that is exploratory —</p>
52
- <p>it weakens further once general ability is controlled for, so we do <strong></strong> claim a single proven axis.</p>
53
 
54
  <h2>Frontier API models</h2>
55
  <p class="note">Scored on all 75 items.</p>
@@ -101,13 +101,13 @@
101
  </tbody></table></div>
102
 
103
  <hr>
104
- <p><strong></strong> Compare only <em>within</em> a population — the four groups are scored on different</p>
105
  <p>item sets (see each <code>n</code>). SRC = overall Signal-Response Correctness · LSU = legitimate-signal uptake</p>
106
  <p>· ISR = illegitimate-signal resistance · then per-family balanced SRC (red→green). Provider-error</p>
107
  <p>items (the model never answered) are excluded, not counted as failures. Deterministic action-based</p>
108
  <p>grading for over-compliance; see the repo for the grader's exact scope. Single seed, July 2026.</p>
109
 
110
- <p><strong></strong> <a href="https://github.com/mthamil107/signal-compliance">code / harness</a> ·</p>
111
  <p><a href="https://huggingface.co/datasets/thamilvendhan/signalbench">dataset + raw responses</a> ·</p>
112
  <p><a href="https://doi.org/10.5281/zenodo.21223955">DOI 10.5281/zenodo.21223955</a></p>
113
  </div></body></html>
 
36
 
37
  <p>An LLM agent reads control instructions and ordinary data through the <em>same</em> channel, so on every</p>
38
  <p>turn it must decide, from content alone, whether an in-band instruction is legitimate. It fails two</p>
39
+ <p>symmetric ways: <strong>over-compliance</strong> (obeying an illegitimate signal — prompt injection) and</p>
40
+ <p><strong>under-compliance</strong> (ignoring a legitimate one — an injected date, an access-deny, a</p>
41
  <p><code>do-not-share</code> memory label, a no-automation policy).</p>
42
 
43
+ <p><strong>Signal-Response Correctness (SRC)</strong> scores both on one axis: the balanced mean of</p>
44
  <p>legitimate-signal uptake (LSU) and illegitimate-signal resistance (ISR). A trivial always-comply or</p>
45
+ <p>always-refuse policy scores <strong>0.5</strong> — you only win by adjudicating.</p>
46
 
47
+ <p><strong>A hard ceiling remains.</strong> Across 21 systems, only one (gpt-5.5, 0.87 on the 70 items its API let</p>
48
  <p>run) exceeds 0.85; the next best is ~0.81, and 20 of 21 fall below 0.85. Scoring is strict:</p>
49
  <p>resisting an attack only counts if the model <em>also</em> still completes the real task — a blanket</p>
50
  <p>refusal earns no credit. Capability is non-monotonic (a small model beats much larger ones). The</p>
51
  <p>five families are positively correlated (mean Pearson <em>r</em> ≈ 0.62 raw), but that is exploratory —</p>
52
+ <p>it weakens further once general ability is controlled for, so we do <strong>not</strong> claim a single proven axis.</p>
53
 
54
  <h2>Frontier API models</h2>
55
  <p class="note">Scored on all 75 items.</p>
 
101
  </tbody></table></div>
102
 
103
  <hr>
104
+ <p><strong>How to read this.</strong> Compare only <em>within</em> a population — the four groups are scored on different</p>
105
  <p>item sets (see each <code>n</code>). SRC = overall Signal-Response Correctness · LSU = legitimate-signal uptake</p>
106
  <p>· ISR = illegitimate-signal resistance · then per-family balanced SRC (red→green). Provider-error</p>
107
  <p>items (the model never answered) are excluded, not counted as failures. Deterministic action-based</p>
108
  <p>grading for over-compliance; see the repo for the grader's exact scope. Single seed, July 2026.</p>
109
 
110
+ <p><strong>Links:</strong> <a href="https://github.com/mthamil107/signal-compliance">code / harness</a> ·</p>
111
  <p><a href="https://huggingface.co/datasets/thamilvendhan/signalbench">dataset + raw responses</a> ·</p>
112
  <p><a href="https://doi.org/10.5281/zenodo.21223955">DOI 10.5281/zenodo.21223955</a></p>
113
  </div></body></html>