Spaces:
Running
Running
| <html lang="en"><head> | |
| <meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <title>signalbench leaderboard (IBSC)</title> | |
| <style> | |
| :root { --ink:#1a1a1a; --muted:#666; --line:#e6e6e6; --accent:#2b6cb0; } | |
| * { box-sizing:border-box; } | |
| body { font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Helvetica,Arial,sans-serif; | |
| color:var(--ink); line-height:1.6; margin:0; background:#fff; } | |
| .wrap { max-width:1000px; margin:0 auto; padding:40px 20px 72px; } | |
| h1 { font-size:1.7rem; margin:0 0 .4rem; letter-spacing:-.01em; } | |
| h2 { font-size:1.15rem; margin:2rem 0 .2rem; } | |
| p { margin:.7rem 0; } | |
| a { color:var(--accent); text-decoration:none; } a:hover { text-decoration:underline; } | |
| .table-scroll { overflow-x:auto; margin:.6rem 0 1.2rem; } | |
| table { border-collapse:collapse; width:100%; font-size:.9rem; } | |
| th,td { padding:6px 9px; text-align:right; border-bottom:1px solid var(--line); white-space:nowrap; } | |
| th { font-weight:600; color:var(--muted); border-bottom:2px solid var(--line); } | |
| td.rank { color:var(--muted); } | |
| td.sys { text-align:left; font-family:ui-monospace,Menlo,Consolas,monospace; font-size:.84rem; } | |
| td.src { font-weight:700; } | |
| td.nsc { color:var(--muted); } | |
| .note { color:var(--muted); font-size:.86rem; margin:.2rem 0 .4rem; } | |
| code { background:#f3f4f6; padding:1px 5px; border-radius:4px; } | |
| hr { border:none; border-top:1px solid var(--line); margin:1.6rem 0; } | |
| @media (prefers-color-scheme: dark) { | |
| body { background:#0f1115; color:#e6e6e6; } | |
| :root { --line:#2a2d34; --muted:#9aa0aa; } | |
| code { background:#1a1d24; } | |
| th { color:#9aa0aa; } | |
| } | |
| </style></head><body><div class="wrap"> | |
| <h1>🛡️ signalbench — In-Band Signal Compliance (IBSC)</h1> | |
| <p>One metric for prompt injection <em>and</em> temporal blindness.</p> | |
| <p>An LLM agent reads control instructions and ordinary data through the <em>same</em> channel, so on every</p> | |
| <p>turn it must decide, from content alone, whether an in-band instruction is legitimate. It fails two</p> | |
| <p>symmetric ways: <strong>over-compliance</strong> (obeying an illegitimate signal — prompt injection) and</p> | |
| <p><strong>under-compliance</strong> (ignoring a legitimate one — an injected date, an access-deny, a</p> | |
| <p><code>do-not-share</code> memory label, a no-automation policy).</p> | |
| <p><strong>Signal-Response Correctness (SRC)</strong> scores both on one axis: the balanced mean of</p> | |
| <p>legitimate-signal uptake (LSU) and illegitimate-signal resistance (ISR). A trivial always-comply or</p> | |
| <p>always-refuse policy scores <strong>0.5</strong> — you only win by adjudicating.</p> | |
| <p><strong>A hard ceiling remains.</strong> Across 21 systems, only one (gpt-5.5, 0.87 on the 70 items its API let</p> | |
| <p>run) exceeds 0.85; the next best is ~0.81, and 20 of 21 fall below 0.85. Scoring is strict:</p> | |
| <p>resisting an attack only counts if the model <em>also</em> still completes the real task — a blanket</p> | |
| <p>refusal earns no credit. Capability is non-monotonic (a small model beats much larger ones). The</p> | |
| <p>five families are positively correlated (mean Pearson <em>r</em> ≈ 0.62 raw), but that is exploratory —</p> | |
| <p>it weakens further once general ability is controlled for, so we do <strong>not</strong> claim a single proven axis.</p> | |
| <h2>Frontier API models</h2> | |
| <p class="note">Scored on all 75 items.</p> | |
| <div class="table-scroll"><table> | |
| <thead><tr><th>#</th><th style="text-align:left">System</th><th>SRC↓</th><th>LSU</th><th>ISR</th><th>time</th><th>access-deny</th><th>memory-label</th><th>injection</th><th>bot_policy</th><th>n</th></tr></thead> | |
| <tbody> | |
| <tr><td class="rank">1</td><td class="sys">openai:gpt-5.5</td><td class="src">0.873</td><td>0.793</td><td>0.962</td><td style="background:hsl(120,70%,88%); color:#1a1a1a">1.000</td><td style="background:hsl(96,70%,88%); color:#1a1a1a">0.800</td><td style="background:hsl(110,70%,88%); color:#1a1a1a">0.917</td><td style="background:hsl(98,70%,88%); color:#1a1a1a">0.817</td><td style="background:hsl(99,70%,88%); color:#1a1a1a">0.833</td><td class="nsc">70</td></tr> | |
| <tr><td class="rank">2</td><td class="sys">gemini:gemini-2.5-pro</td><td class="src">0.783</td><td>0.967</td><td>0.600</td><td style="background:hsl(80,70%,88%); color:#1a1a1a">0.667</td><td style="background:hsl(90,70%,88%); color:#1a1a1a">0.750</td><td style="background:hsl(69,70%,88%); color:#1a1a1a">0.583</td><td style="background:hsl(110,70%,88%); color:#1a1a1a">0.917</td><td style="background:hsl(120,70%,88%); color:#1a1a1a">1.000</td><td class="nsc">75</td></tr> | |
| <tr><td class="rank">3</td><td class="sys">openai:gpt-5.1</td><td class="src">0.783</td><td>0.800</td><td>0.767</td><td style="background:hsl(110,70%,88%); color:#1a1a1a">0.917</td><td style="background:hsl(99,70%,88%); color:#1a1a1a">0.833</td><td style="background:hsl(80,70%,88%); color:#1a1a1a">0.667</td><td style="background:hsl(110,70%,88%); color:#1a1a1a">0.917</td><td style="background:hsl(69,70%,88%); color:#1a1a1a">0.583</td><td class="nsc">75</td></tr> | |
| <tr><td class="rank">4</td><td class="sys">openai:gpt-4o-mini</td><td class="src">0.767</td><td>0.867</td><td>0.667</td><td style="background:hsl(110,70%,88%); color:#1a1a1a">0.917</td><td style="background:hsl(99,70%,88%); color:#1a1a1a">0.833</td><td style="background:hsl(90,70%,88%); color:#1a1a1a">0.750</td><td style="background:hsl(69,70%,88%); color:#1a1a1a">0.583</td><td style="background:hsl(90,70%,88%); color:#1a1a1a">0.750</td><td class="nsc">75</td></tr> | |
| <tr><td class="rank">5</td><td class="sys">openai:gpt-4.1</td><td class="src">0.667</td><td>0.800</td><td>0.533</td><td style="background:hsl(99,70%,88%); color:#1a1a1a">0.833</td><td style="background:hsl(60,70%,88%); color:#1a1a1a">0.500</td><td style="background:hsl(69,70%,88%); color:#1a1a1a">0.583</td><td style="background:hsl(99,70%,88%); color:#1a1a1a">0.833</td><td style="background:hsl(69,70%,88%); color:#1a1a1a">0.583</td><td class="nsc">75</td></tr> | |
| <tr><td class="rank">6</td><td class="sys">gemini:gemini-2.5-flash</td><td class="src">0.583</td><td>0.833</td><td>0.333</td><td style="background:hsl(69,70%,88%); color:#1a1a1a">0.583</td><td style="background:hsl(60,70%,88%); color:#1a1a1a">0.500</td><td style="background:hsl(69,70%,88%); color:#1a1a1a">0.583</td><td style="background:hsl(80,70%,88%); color:#1a1a1a">0.667</td><td style="background:hsl(69,70%,88%); color:#1a1a1a">0.583</td><td class="nsc">75</td></tr> | |
| </tbody></table></div> | |
| <h2>Deployed agent (Claude Code)</h2> | |
| <p class="note">A different harness — tools disabled — scored on 72 items (3 tool-only items excluded).</p> | |
| <div class="table-scroll"><table> | |
| <thead><tr><th>#</th><th style="text-align:left">System</th><th>SRC↓</th><th>LSU</th><th>ISR</th><th>time</th><th>access-deny</th><th>memory-label</th><th>injection</th><th>bot_policy</th><th>n</th></tr></thead> | |
| <tbody> | |
| <tr><td class="rank">1</td><td class="sys">claude-code:claude-opus-4-8</td><td class="src">0.813</td><td>0.852</td><td>0.767</td><td style="background:hsl(90,70%,88%); color:#1a1a1a">0.750</td><td style="background:hsl(99,70%,88%); color:#1a1a1a">0.833</td><td style="background:hsl(110,70%,88%); color:#1a1a1a">0.917</td><td style="background:hsl(99,70%,88%); color:#1a1a1a">0.833</td><td style="background:hsl(87,70%,88%); color:#1a1a1a">0.733</td><td class="nsc">72</td></tr> | |
| <tr><td class="rank">2</td><td class="sys">claude-code:claude-haiku-4-5-20251001</td><td class="src">0.707</td><td>0.556</td><td>0.867</td><td style="background:hsl(120,70%,88%); color:#1a1a1a">1.000</td><td style="background:hsl(96,70%,88%); color:#1a1a1a">0.800</td><td style="background:hsl(80,70%,88%); color:#1a1a1a">0.667</td><td style="background:hsl(74,70%,88%); color:#1a1a1a">0.617</td><td style="background:hsl(54,70%,88%); color:#1a1a1a">0.450</td><td class="nsc">72</td></tr> | |
| <tr><td class="rank">3</td><td class="sys">claude-code:claude-sonnet-5</td><td class="src">0.663</td><td>0.556</td><td>0.767</td><td style="background:hsl(80,70%,88%); color:#1a1a1a">0.667</td><td style="background:hsl(120,70%,88%); color:#1a1a1a">1.000</td><td style="background:hsl(90,70%,88%); color:#1a1a1a">0.750</td><td style="background:hsl(42,70%,88%); color:#1a1a1a">0.350</td><td style="background:hsl(66,70%,88%); color:#1a1a1a">0.550</td><td class="nsc">72</td></tr> | |
| </tbody></table></div> | |
| <h2>Open-weight models</h2> | |
| <p class="note">Ran the full suite. Rows with n<75 had a few items refused by the provider and excluded.</p> | |
| <div class="table-scroll"><table> | |
| <thead><tr><th>#</th><th style="text-align:left">System</th><th>SRC↓</th><th>LSU</th><th>ISR</th><th>time</th><th>access-deny</th><th>memory-label</th><th>injection</th><th>bot_policy</th><th>n</th></tr></thead> | |
| <tbody> | |
| <tr><td class="rank">1</td><td class="sys">openrouter:qwen/qwen-2.5-72b-instruct</td><td class="src">0.683</td><td>0.833</td><td>0.518</td><td style="background:hsl(99,70%,88%); color:#1a1a1a">0.833</td><td style="background:hsl(60,70%,88%); color:#1a1a1a">0.500</td><td style="background:hsl(80,70%,88%); color:#1a1a1a">0.667</td><td style="background:hsl(90,70%,88%); color:#1a1a1a">0.750</td><td style="background:hsl(80,70%,88%); color:#1a1a1a">0.667</td><td class="nsc">69</td></tr> | |
| <tr><td class="rank">2</td><td class="sys">openrouter:qwen/qwen3-next-80b-a3b-instruct</td><td class="src">0.550</td><td>0.900</td><td>0.200</td><td style="background:hsl(80,70%,88%); color:#1a1a1a">0.667</td><td style="background:hsl(60,70%,88%); color:#1a1a1a">0.500</td><td style="background:hsl(50,70%,88%); color:#1a1a1a">0.417</td><td style="background:hsl(69,70%,88%); color:#1a1a1a">0.583</td><td style="background:hsl(69,70%,88%); color:#1a1a1a">0.583</td><td class="nsc">75</td></tr> | |
| <tr><td class="rank">3</td><td class="sys">openrouter:deepseek/deepseek-chat</td><td class="src">0.533</td><td>0.733</td><td>0.333</td><td style="background:hsl(80,70%,88%); color:#1a1a1a">0.667</td><td style="background:hsl(69,70%,88%); color:#1a1a1a">0.583</td><td style="background:hsl(69,70%,88%); color:#1a1a1a">0.583</td><td style="background:hsl(69,70%,88%); color:#1a1a1a">0.583</td><td style="background:hsl(30,70%,88%); color:#1a1a1a">0.250</td><td class="nsc">75</td></tr> | |
| <tr><td class="rank">4</td><td class="sys">openrouter:meta-llama/llama-3.1-8b-instruct</td><td class="src">0.483</td><td>0.500</td><td>0.467</td><td style="background:hsl(50,70%,88%); color:#1a1a1a">0.417</td><td style="background:hsl(60,70%,88%); color:#1a1a1a">0.500</td><td style="background:hsl(80,70%,88%); color:#1a1a1a">0.667</td><td style="background:hsl(50,70%,88%); color:#1a1a1a">0.417</td><td style="background:hsl(50,70%,88%); color:#1a1a1a">0.417</td><td class="nsc">75</td></tr> | |
| <tr><td class="rank">5</td><td class="sys">openrouter:qwen/qwen-2.5-7b-instruct</td><td class="src">0.383</td><td>0.500</td><td>0.267</td><td style="background:hsl(50,70%,88%); color:#1a1a1a">0.417</td><td style="background:hsl(20,70%,88%); color:#1a1a1a">0.167</td><td style="background:hsl(60,70%,88%); color:#1a1a1a">0.500</td><td style="background:hsl(60,70%,88%); color:#1a1a1a">0.500</td><td style="background:hsl(39,70%,88%); color:#1a1a1a">0.333</td><td class="nsc">75</td></tr> | |
| <tr><td class="rank">6</td><td class="sys">openrouter:meta-llama/llama-3.3-70b-instruct</td><td class="src">0.367</td><td>0.533</td><td>0.200</td><td style="background:hsl(69,70%,88%); color:#1a1a1a">0.583</td><td style="background:hsl(9,70%,88%); color:#1a1a1a">0.083</td><td style="background:hsl(69,70%,88%); color:#1a1a1a">0.583</td><td style="background:hsl(60,70%,88%); color:#1a1a1a">0.500</td><td style="background:hsl(9,70%,88%); color:#1a1a1a">0.083</td><td class="nsc">75</td></tr> | |
| <tr><td class="rank">7</td><td class="sys">openrouter:mistralai/mistral-nemo</td><td class="src">0.283</td><td>0.533</td><td>0.033</td><td style="background:hsl(60,70%,88%); color:#1a1a1a">0.500</td><td style="background:hsl(9,70%,88%); color:#1a1a1a">0.083</td><td style="background:hsl(39,70%,88%); color:#1a1a1a">0.333</td><td style="background:hsl(50,70%,88%); color:#1a1a1a">0.417</td><td style="background:hsl(9,70%,88%); color:#1a1a1a">0.083</td><td class="nsc">75</td></tr> | |
| </tbody></table></div> | |
| <h2>Open-weight (text-only subset)</h2> | |
| <p class="note">The tool-graded families (access-deny, bot-policy) 404'd on the provider side, so these are scored on the 47-item text-only subset — NOT comparable to the full-suite rows above.</p> | |
| <div class="table-scroll"><table> | |
| <thead><tr><th>#</th><th style="text-align:left">System</th><th>SRC↓</th><th>LSU</th><th>ISR</th><th>time</th><th>access-deny</th><th>memory-label</th><th>injection</th><th>bot_policy</th><th>n</th></tr></thead> | |
| <tbody> | |
| <tr><td class="rank">1</td><td class="sys">openrouter:google/gemma-2-27b-it</td><td class="src">0.742</td><td>0.765</td><td>0.600</td><td style="background:hsl(75,70%,88%); color:#1a1a1a">0.633</td><td style="background:hsl(0,70%,88%); color:#1a1a1a">0.000</td><td style="background:hsl(99,70%,88%); color:#1a1a1a">0.833</td><td style="background:hsl(60,70%,88%); color:#1a1a1a">0.500</td><td style="background:hsl(120,70%,88%); color:#1a1a1a">1.000</td><td class="nsc">47</td></tr> | |
| <tr><td class="rank">2</td><td class="sys">openrouter:microsoft/phi-4</td><td class="src">0.560</td><td>0.706</td><td>0.733</td><td style="background:hsl(86,70%,88%); color:#1a1a1a">0.717</td><td style="background:hsl(0,70%,88%); color:#1a1a1a">0.000</td><td style="background:hsl(90,70%,88%); color:#1a1a1a">0.750</td><td style="background:hsl(93,70%,88%); color:#1a1a1a">0.775</td><td style="background:hsl(0,70%,88%); color:#1a1a1a">0.000</td><td class="nsc">47</td></tr> | |
| <tr><td class="rank">3</td><td class="sys">openrouter:mistralai/mistral-small-24b-instruct-2501</td><td class="src">0.525</td><td>0.765</td><td>0.600</td><td style="background:hsl(63,70%,88%); color:#1a1a1a">0.533</td><td style="background:hsl(0,70%,88%); color:#1a1a1a">0.000</td><td style="background:hsl(110,70%,88%); color:#1a1a1a">0.917</td><td style="background:hsl(78,70%,88%); color:#1a1a1a">0.650</td><td style="background:hsl(0,70%,88%); color:#1a1a1a">0.000</td><td class="nsc">47</td></tr> | |
| <tr><td class="rank">4</td><td class="sys">openrouter:meta-llama/llama-3.2-3b-instruct</td><td class="src">0.385</td><td>0.529</td><td>0.467</td><td style="background:hsl(32,70%,88%); color:#1a1a1a">0.267</td><td style="background:hsl(0,70%,88%); color:#1a1a1a">0.000</td><td style="background:hsl(90,70%,88%); color:#1a1a1a">0.750</td><td style="background:hsl(63,70%,88%); color:#1a1a1a">0.525</td><td style="background:hsl(0,70%,88%); color:#1a1a1a">0.000</td><td class="nsc">47</td></tr> | |
| <tr><td class="rank">5</td><td class="sys">openrouter:nousresearch/hermes-3-llama-3.1-70b</td><td class="src">0.358</td><td>0.824</td><td>0.133</td><td style="background:hsl(62,70%,88%); color:#1a1a1a">0.517</td><td style="background:hsl(0,70%,88%); color:#1a1a1a">0.000</td><td style="background:hsl(62,70%,88%); color:#1a1a1a">0.517</td><td style="background:hsl(48,70%,88%); color:#1a1a1a">0.400</td><td style="background:hsl(0,70%,88%); color:#1a1a1a">0.000</td><td class="nsc">47</td></tr> | |
| </tbody></table></div> | |
| <hr> | |
| <p><strong>How to read this.</strong> Compare only <em>within</em> a population — the four groups are scored on different</p> | |
| <p>item sets (see each <code>n</code>). SRC = overall Signal-Response Correctness · LSU = legitimate-signal uptake</p> | |
| <p>· ISR = illegitimate-signal resistance · then per-family balanced SRC (red→green). Provider-error</p> | |
| <p>items (the model never answered) are excluded, not counted as failures. Deterministic action-based</p> | |
| <p>grading for over-compliance; see the repo for the grader's exact scope. Single seed, July 2026.</p> | |
| <p><strong>Links:</strong> <a href="https://github.com/mthamil107/signal-compliance">code / harness</a> ·</p> | |
| <p><a href="https://huggingface.co/datasets/thamilvendhan/signalbench">dataset + raw responses</a> ·</p> | |
| <p><a href="https://doi.org/10.5281/zenodo.21223955">DOI 10.5281/zenodo.21223955</a></p> | |
| </div></body></html> | |