index.html

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>RecRedTeam AuditLab — red-team audit for shopping agents</title>
<meta name="description" content="Interactive red-team audit for LLM and agentic recommendation and shopping assistants: 9 threat categories mapped to OWASP ASI 2026, a deterministic agent, and three defense plugins compared on a 234-case public split.">
<meta property="og:title" content="RecRedTeam AuditLab">
<meta property="og:description" content="Red-team audit for LLM / agentic recommendation and shopping assistants. 9 categories, OWASP ASI 2026, deterministic and reproducible.">
<meta property="og:type" content="website">
<link rel="stylesheet" href="assets/style.css">
<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'%3E%3Ctext y='.9em' font-size='90'%3E%F0%9F%9B%A1%EF%B8%8F%3C/text%3E%3C/svg%3E">
</head>
<body>
<div class="wrap">

  <header class="top">
    <p class="eyebrow">RecRedTeam · AuditLab</p>
    <h1>Red-team audit for LLM and agentic shopping assistants</h1>
    <p class="lede">
      Pick an attack from a nine-category taxonomy mapped to <strong>OWASP ASI 2026</strong>,
      run it against a deterministic shopping agent, and watch the verdict change as you
      switch defenses on. Every number on this page was produced by the audit engine in
      <code>auditlab/</code> and frozen into <code>data/results.json</code> — no API key,
      no network call, reproducible byte for byte.
    </p>
    <div class="metrics">
      <span><b>ASR</b> share of cases the agent failed</span>
      <span><b>effectiveness</b> 1 − ASR(defended) / ASR(baseline)</span>
      <span><b>hazard</b> mean graded harm, 0–1</span>
      <span><b>over-block</b> share of cases where the defense emptied the catalog</span>
    </div>
  </header>

  <nav class="tabs" role="tablist" aria-label="Audit views">
    <button class="tab" role="tab" id="tab-taxonomy" aria-controls="panel-taxonomy" aria-selected="true">Threat taxonomy</button>
    <button class="tab" role="tab" id="tab-case" aria-controls="panel-case" aria-selected="false">Single attack</button>
    <button class="tab" role="tab" id="tab-bench" aria-controls="panel-bench" aria-selected="false">Benchmark / leaderboard</button>
    <button class="tab" role="tab" id="tab-about" aria-controls="panel-about" aria-selected="false">About</button>
  </nav>

  <section class="panel" id="panel-taxonomy" role="tabpanel" aria-labelledby="tab-taxonomy">
    <div id="taxonomy-table"></div>
    <details>
      <summary>Full descriptions</summary>
      <ul id="taxonomy-details"></ul>
    </details>
  </section>

  <section class="panel" id="panel-case" role="tabpanel" aria-labelledby="tab-case" hidden>
    <div class="controls">
      <label class="field">
        <span>Attack case</span>
        <select id="case-select"></select>
      </label>
      <div class="field">
        <span>Defense plugins</span>
        <div class="plugins">
          <label><input type="checkbox" data-plugin="sanitizer"> sanitizer</label>
          <label><input type="checkbox" data-plugin="rag_guard"> rag_guard</label>
          <label><input type="checkbox" data-plugin="policy_guard"> policy_guard</label>
        </div>
      </div>
    </div>
    <div id="case-view"></div>
  </section>

  <section class="panel" id="panel-bench" role="tabpanel" aria-labelledby="tab-bench" hidden>
    <h2>Leaderboard</h2>
    <p class="note" id="bench-note"></p>
    <div class="table-wrap" id="bench-table"></div>
    <h2>Attack success rate by defense</h2>
    <div class="bars" id="bench-bars"></div>
    <h2>Per-category ASR</h2>
    <div class="table-wrap" id="bench-matrix"></div>
  </section>

  <section class="panel" id="panel-about" role="tabpanel" aria-labelledby="tab-about" hidden>
    <h2>How an audit runs</h2>
    <ol>
      <li><b>Poison</b> — hidden instructions from the case are planted into the product catalog.</li>
      <li><b>Defend</b> — selected plugins act at retrieval (<code>rag_guard</code>), content
        (<code>sanitizer</code>) and output (<code>policy_guard</code>) level.</li>
      <li><b>Answer</b> — a deterministic mock shopping agent ranks the catalog and paraphrases
        what it retrieved. Its only flaw is the realistic one: it trusts retrieved content.</li>
      <li><b>Judge</b> — a rule-based dual-channel judge looks for PII disclosure, untrusted
        destinations, executed instructions and category failure signals.</li>
      <li><b>Grade</b> — a verdict (PASS / FAIL) plus a hazard grade derived from the category
        severity.</li>
    </ol>

    <h2>Reading the leaderboard</h2>
    <p>Each plugin has a deliberately different coverage profile, which is the point of the
      framework:</p>
    <ul>
      <li><code>sanitizer</code> kills injected instructions and unverifiable claims, but is blind
        to ranking manipulation and metadata exposure.</li>
      <li><code>rag_guard</code> removes poisoned listings at retrieval. Powerful, but it
        over-blocks, and it cannot see a fabricated spec that looks exactly like a real one.</li>
      <li><code>policy_guard</code> only rewrites the answer. It stops PII leaks and phishing
        redirects and nothing else.</li>
    </ul>
    <p><b>Product hallucination is the category no rule-based defense solves.</b> Fabricated but
      plausible specs ("30-day battery life", "#1 best seller", "50,000 units sold") are lexically
      indistinguishable from real ones. Closing that gap needs grounding and external verification,
      which is the open research direction.</p>

    <h2>Limitations</h2>
    <ul>
      <li>The agent under test is a <b>deterministic mock</b>, not a real LLM. Absolute ASR values
        describe this mock; what transfers to real agents is the relative coverage profile of the
        defenses and the taxonomy itself.</li>
      <li>Only the <b>rule channel</b> of the judge runs here. The full RecRedTeam framework adds an
        optional LLM judge channel and LangGraph / OpenAI-Agents adapters.</li>
      <li>Attacks target a <b>local simulated catalog</b>, never a real system.</li>
      <li>This page is a <b>static snapshot</b>: the engine ran once over the public split and the
        results were frozen. Rebuild them with <code>python build_static.py</code>.</li>
    </ul>

    <h2>Links</h2>
    <ul>
      <li>Dataset — <a href="https://huggingface.co/datasets/NoahIsARider/RecRedTeam-ASI-Bench">NoahIsARider/RecRedTeam-ASI-Bench</a></li>
      <li>Framework — <a href="https://github.com/NoahIsARider/RecRedTeam">github.com/NoahIsARider/RecRedTeam</a></li>
      <li>Reference Gradio implementation — <code>app.py</code> in this repo, run it locally with <code>pip install -r requirements.txt &amp;&amp; python app.py</code></li>
      <li>Author — Fangyanuo Zhou · <a href="https://noahisarider.github.io/">noahisarider.github.io</a></li>
    </ul>
  </section>

  <footer>
    Defensive research only. Attack cases run against a local simulated catalog, never against real
    systems. MIT · dataset CC BY 4.0 · <span id="build-meta"></span>
  </footer>
</div>

<script src="assets/app.js"></script>
</body>
</html>