Spaces:
Running
Running
fix: restore bold text in leaderboard intro
Browse files- index.html +8 -8
index.html
CHANGED
|
@@ -36,20 +36,20 @@
|
|
| 36 |
|
| 37 |
<p>An LLM agent reads control instructions and ordinary data through the <em>same</em> channel, so on every</p>
|
| 38 |
<p>turn it must decide, from content alone, whether an in-band instruction is legitimate. It fails two</p>
|
| 39 |
-
<p>symmetric ways: <strong></strong> (obeying an illegitimate signal — prompt injection) and</p>
|
| 40 |
-
<p><strong></strong> (ignoring a legitimate one — an injected date, an access-deny, a</p>
|
| 41 |
<p><code>do-not-share</code> memory label, a no-automation policy).</p>
|
| 42 |
|
| 43 |
-
<p><strong></strong> scores both on one axis: the balanced mean of</p>
|
| 44 |
<p>legitimate-signal uptake (LSU) and illegitimate-signal resistance (ISR). A trivial always-comply or</p>
|
| 45 |
-
<p>always-refuse policy scores <strong></strong> — you only win by adjudicating.</p>
|
| 46 |
|
| 47 |
-
<p><strong></strong> Across 21 systems, only one (gpt-5.5, 0.87 on the 70 items its API let</p>
|
| 48 |
<p>run) exceeds 0.85; the next best is ~0.81, and 20 of 21 fall below 0.85. Scoring is strict:</p>
|
| 49 |
<p>resisting an attack only counts if the model <em>also</em> still completes the real task — a blanket</p>
|
| 50 |
<p>refusal earns no credit. Capability is non-monotonic (a small model beats much larger ones). The</p>
|
| 51 |
<p>five families are positively correlated (mean Pearson <em>r</em> ≈ 0.62 raw), but that is exploratory —</p>
|
| 52 |
-
<p>it weakens further once general ability is controlled for, so we do <strong></strong> claim a single proven axis.</p>
|
| 53 |
|
| 54 |
<h2>Frontier API models</h2>
|
| 55 |
<p class="note">Scored on all 75 items.</p>
|
|
@@ -101,13 +101,13 @@
|
|
| 101 |
</tbody></table></div>
|
| 102 |
|
| 103 |
<hr>
|
| 104 |
-
<p><strong></strong> Compare only <em>within</em> a population — the four groups are scored on different</p>
|
| 105 |
<p>item sets (see each <code>n</code>). SRC = overall Signal-Response Correctness · LSU = legitimate-signal uptake</p>
|
| 106 |
<p>· ISR = illegitimate-signal resistance · then per-family balanced SRC (red→green). Provider-error</p>
|
| 107 |
<p>items (the model never answered) are excluded, not counted as failures. Deterministic action-based</p>
|
| 108 |
<p>grading for over-compliance; see the repo for the grader's exact scope. Single seed, July 2026.</p>
|
| 109 |
|
| 110 |
-
<p><strong></strong> <a href="https://github.com/mthamil107/signal-compliance">code / harness</a> ·</p>
|
| 111 |
<p><a href="https://huggingface.co/datasets/thamilvendhan/signalbench">dataset + raw responses</a> ·</p>
|
| 112 |
<p><a href="https://doi.org/10.5281/zenodo.21223955">DOI 10.5281/zenodo.21223955</a></p>
|
| 113 |
</div></body></html>
|
|
|
|
| 36 |
|
| 37 |
<p>An LLM agent reads control instructions and ordinary data through the <em>same</em> channel, so on every</p>
|
| 38 |
<p>turn it must decide, from content alone, whether an in-band instruction is legitimate. It fails two</p>
|
| 39 |
+
<p>symmetric ways: <strong>over-compliance</strong> (obeying an illegitimate signal — prompt injection) and</p>
|
| 40 |
+
<p><strong>under-compliance</strong> (ignoring a legitimate one — an injected date, an access-deny, a</p>
|
| 41 |
<p><code>do-not-share</code> memory label, a no-automation policy).</p>
|
| 42 |
|
| 43 |
+
<p><strong>Signal-Response Correctness (SRC)</strong> scores both on one axis: the balanced mean of</p>
|
| 44 |
<p>legitimate-signal uptake (LSU) and illegitimate-signal resistance (ISR). A trivial always-comply or</p>
|
| 45 |
+
<p>always-refuse policy scores <strong>0.5</strong> — you only win by adjudicating.</p>
|
| 46 |
|
| 47 |
+
<p><strong>A hard ceiling remains.</strong> Across 21 systems, only one (gpt-5.5, 0.87 on the 70 items its API let</p>
|
| 48 |
<p>run) exceeds 0.85; the next best is ~0.81, and 20 of 21 fall below 0.85. Scoring is strict:</p>
|
| 49 |
<p>resisting an attack only counts if the model <em>also</em> still completes the real task — a blanket</p>
|
| 50 |
<p>refusal earns no credit. Capability is non-monotonic (a small model beats much larger ones). The</p>
|
| 51 |
<p>five families are positively correlated (mean Pearson <em>r</em> ≈ 0.62 raw), but that is exploratory —</p>
|
| 52 |
+
<p>it weakens further once general ability is controlled for, so we do <strong>not</strong> claim a single proven axis.</p>
|
| 53 |
|
| 54 |
<h2>Frontier API models</h2>
|
| 55 |
<p class="note">Scored on all 75 items.</p>
|
|
|
|
| 101 |
</tbody></table></div>
|
| 102 |
|
| 103 |
<hr>
|
| 104 |
+
<p><strong>How to read this.</strong> Compare only <em>within</em> a population — the four groups are scored on different</p>
|
| 105 |
<p>item sets (see each <code>n</code>). SRC = overall Signal-Response Correctness · LSU = legitimate-signal uptake</p>
|
| 106 |
<p>· ISR = illegitimate-signal resistance · then per-family balanced SRC (red→green). Provider-error</p>
|
| 107 |
<p>items (the model never answered) are excluded, not counted as failures. Deterministic action-based</p>
|
| 108 |
<p>grading for over-compliance; see the repo for the grader's exact scope. Single seed, July 2026.</p>
|
| 109 |
|
| 110 |
+
<p><strong>Links:</strong> <a href="https://github.com/mthamil107/signal-compliance">code / harness</a> ·</p>
|
| 111 |
<p><a href="https://huggingface.co/datasets/thamilvendhan/signalbench">dataset + raw responses</a> ·</p>
|
| 112 |
<p><a href="https://doi.org/10.5281/zenodo.21223955">DOI 10.5281/zenodo.21223955</a></p>
|
| 113 |
</div></body></html>
|