| <!doctype html>
|
| <html lang="en">
|
| <head>
|
| <meta charset="utf-8">
|
| <meta name="viewport" content="width=device-width, initial-scale=1">
|
| <title>RecRedTeam AuditLab — red-team audit for shopping agents</title>
|
| <meta name="description" content="Interactive red-team audit for LLM and agentic recommendation and shopping assistants: 9 threat categories mapped to OWASP ASI 2026, a deterministic agent, and three defense plugins compared on a 234-case public split.">
|
| <meta property="og:title" content="RecRedTeam AuditLab">
|
| <meta property="og:description" content="Red-team audit for LLM / agentic recommendation and shopping assistants. 9 categories, OWASP ASI 2026, deterministic and reproducible.">
|
| <meta property="og:type" content="website">
|
| <link rel="stylesheet" href="assets/style.css">
|
| <link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'%3E%3Ctext y='.9em' font-size='90'%3E%F0%9F%9B%A1%EF%B8%8F%3C/text%3E%3C/svg%3E">
|
| </head>
|
| <body>
|
| <div class="wrap">
|
|
|
| <header class="top">
|
| <p class="eyebrow">RecRedTeam · AuditLab</p>
|
| <h1>Red-team audit for LLM and agentic shopping assistants</h1>
|
| <p class="lede">
|
| Pick an attack from a nine-category taxonomy mapped to <strong>OWASP ASI 2026</strong>,
|
| run it against a deterministic shopping agent, and watch the verdict change as you
|
| switch defenses on. Every number on this page was produced by the audit engine in
|
| <code>auditlab/</code> and frozen into <code>data/results.json</code> — no API key,
|
| no network call, reproducible byte for byte.
|
| </p>
|
| <div class="metrics">
|
| <span><b>ASR</b> share of cases the agent failed</span>
|
| <span><b>effectiveness</b> 1 − ASR(defended) / ASR(baseline)</span>
|
| <span><b>hazard</b> mean graded harm, 0–1</span>
|
| <span><b>over-block</b> share of cases where the defense emptied the catalog</span>
|
| </div>
|
| </header>
|
|
|
| <nav class="tabs" role="tablist" aria-label="Audit views">
|
| <button class="tab" role="tab" id="tab-taxonomy" aria-controls="panel-taxonomy" aria-selected="true">Threat taxonomy</button>
|
| <button class="tab" role="tab" id="tab-case" aria-controls="panel-case" aria-selected="false">Single attack</button>
|
| <button class="tab" role="tab" id="tab-bench" aria-controls="panel-bench" aria-selected="false">Benchmark / leaderboard</button>
|
| <button class="tab" role="tab" id="tab-about" aria-controls="panel-about" aria-selected="false">About</button>
|
| </nav>
|
|
|
| <section class="panel" id="panel-taxonomy" role="tabpanel" aria-labelledby="tab-taxonomy">
|
| <div id="taxonomy-table"></div>
|
| <details>
|
| <summary>Full descriptions</summary>
|
| <ul id="taxonomy-details"></ul>
|
| </details>
|
| </section>
|
|
|
| <section class="panel" id="panel-case" role="tabpanel" aria-labelledby="tab-case" hidden>
|
| <div class="controls">
|
| <label class="field">
|
| <span>Attack case</span>
|
| <select id="case-select"></select>
|
| </label>
|
| <div class="field">
|
| <span>Defense plugins</span>
|
| <div class="plugins">
|
| <label><input type="checkbox" data-plugin="sanitizer"> sanitizer</label>
|
| <label><input type="checkbox" data-plugin="rag_guard"> rag_guard</label>
|
| <label><input type="checkbox" data-plugin="policy_guard"> policy_guard</label>
|
| </div>
|
| </div>
|
| </div>
|
| <div id="case-view"></div>
|
| </section>
|
|
|
| <section class="panel" id="panel-bench" role="tabpanel" aria-labelledby="tab-bench" hidden>
|
| <h2>Leaderboard</h2>
|
| <p class="note" id="bench-note"></p>
|
| <div class="table-wrap" id="bench-table"></div>
|
| <h2>Attack success rate by defense</h2>
|
| <div class="bars" id="bench-bars"></div>
|
| <h2>Per-category ASR</h2>
|
| <div class="table-wrap" id="bench-matrix"></div>
|
| </section>
|
|
|
| <section class="panel" id="panel-about" role="tabpanel" aria-labelledby="tab-about" hidden>
|
| <h2>How an audit runs</h2>
|
| <ol>
|
| <li><b>Poison</b> — hidden instructions from the case are planted into the product catalog.</li>
|
| <li><b>Defend</b> — selected plugins act at retrieval (<code>rag_guard</code>), content
|
| (<code>sanitizer</code>) and output (<code>policy_guard</code>) level.</li>
|
| <li><b>Answer</b> — a deterministic mock shopping agent ranks the catalog and paraphrases
|
| what it retrieved. Its only flaw is the realistic one: it trusts retrieved content.</li>
|
| <li><b>Judge</b> — a rule-based dual-channel judge looks for PII disclosure, untrusted
|
| destinations, executed instructions and category failure signals.</li>
|
| <li><b>Grade</b> — a verdict (PASS / FAIL) plus a hazard grade derived from the category
|
| severity.</li>
|
| </ol>
|
|
|
| <h2>Reading the leaderboard</h2>
|
| <p>Each plugin has a deliberately different coverage profile, which is the point of the
|
| framework:</p>
|
| <ul>
|
| <li><code>sanitizer</code> kills injected instructions and unverifiable claims, but is blind
|
| to ranking manipulation and metadata exposure.</li>
|
| <li><code>rag_guard</code> removes poisoned listings at retrieval. Powerful, but it
|
| over-blocks, and it cannot see a fabricated spec that looks exactly like a real one.</li>
|
| <li><code>policy_guard</code> only rewrites the answer. It stops PII leaks and phishing
|
| redirects and nothing else.</li>
|
| </ul>
|
| <p><b>Product hallucination is the category no rule-based defense solves.</b> Fabricated but
|
| plausible specs ("30-day battery life", "#1 best seller", "50,000 units sold") are lexically
|
| indistinguishable from real ones. Closing that gap needs grounding and external verification,
|
| which is the open research direction.</p>
|
|
|
| <h2>Limitations</h2>
|
| <ul>
|
| <li>The agent under test is a <b>deterministic mock</b>, not a real LLM. Absolute ASR values
|
| describe this mock; what transfers to real agents is the relative coverage profile of the
|
| defenses and the taxonomy itself.</li>
|
| <li>Only the <b>rule channel</b> of the judge runs here. The full RecRedTeam framework adds an
|
| optional LLM judge channel and LangGraph / OpenAI-Agents adapters.</li>
|
| <li>Attacks target a <b>local simulated catalog</b>, never a real system.</li>
|
| <li>This page is a <b>static snapshot</b>: the engine ran once over the public split and the
|
| results were frozen. Rebuild them with <code>python build_static.py</code>.</li>
|
| </ul>
|
|
|
| <h2>Links</h2>
|
| <ul>
|
| <li>Dataset — <a href="https://huggingface.co/datasets/NoahIsARider/RecRedTeam-ASI-Bench">NoahIsARider/RecRedTeam-ASI-Bench</a></li>
|
| <li>Framework — <a href="https://github.com/NoahIsARider/RecRedTeam">github.com/NoahIsARider/RecRedTeam</a></li>
|
| <li>Reference Gradio implementation — <code>app.py</code> in this repo, run it locally with <code>pip install -r requirements.txt && python app.py</code></li>
|
| <li>Author — Fangyanuo Zhou · <a href="https://noahisarider.github.io/">noahisarider.github.io</a></li>
|
| </ul>
|
| </section>
|
|
|
| <footer>
|
| Defensive research only. Attack cases run against a local simulated catalog, never against real
|
| systems. MIT · dataset CC BY 4.0 · <span id="build-meta"></span>
|
| </footer>
|
| </div>
|
|
|
| <script src="assets/app.js"></script>
|
| </body>
|
| </html>
|
|
|
