| <!doctype html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8" /> |
| <meta name="viewport" content="width=device-width, initial-scale=1" /> |
| <meta name="description" content="Client-side trace-level security and reliability analysis for AI agents." /> |
| <title>Agent Failure Atlas</title> |
| <link rel="stylesheet" href="styles.css" /> |
| </head> |
| <body> |
| <div class="ambient ambient-one"></div> |
| <div class="ambient ambient-two"></div> |
|
|
| <header class="site-header"> |
| <a class="brand" href="#overview" aria-label="Agent Failure Atlas home"> |
| <span class="brand-mark">A</span> |
| <span><strong>Agent Failure Atlas</strong><small>Trace-level evidence, locally analyzed</small></span> |
| </a> |
| <nav class="nav" aria-label="Primary navigation"> |
| <button class="nav-link active" data-tab="overview">Overview</button> |
| <button class="nav-link" data-tab="scan">Scan</button> |
| <button class="nav-link" data-tab="compare">Compare</button> |
| <button class="nav-link" data-tab="methodology">Methodology</button> |
| </nav> |
| <div class="header-links"> |
| <a href="https://huggingface.co/datasets/solsticestudioai/agent-failure-atlas-benchmark" target="_blank" rel="noreferrer">Dataset ↗</a> |
| <a href="https://huggingface.co/spaces/solsticestudioai/deltastore-trace-explorer" target="_blank" rel="noreferrer">DeltaStore</a>`r`n <a href="https://github.com/Solasticeaistudio/agent-failure-atlas" target="_blank" rel="noreferrer">GitHub ↗</a> |
| </div> |
| </header> |
|
|
| <main> |
| <section id="overview" class="tab-panel active" aria-labelledby="overview-title"> |
| <div class="hero shell"> |
| <div class="hero-copy"> |
| <div class="eyebrow"><span class="pulse-dot"></span> Open, deterministic, evidence-linked</div> |
| <h1 id="overview-title">The final answer can look right while the agent run goes wrong.</h1> |
| <p>Agent Failure Atlas inspects the trajectory: tool scope, approvals, injected instructions, repeated actions, silent failures, and exposed secrets. This static edition runs entirely inside your browser.</p> |
| <div class="hero-actions"> |
| <button class="primary-action" data-go="scan">Analyze a trace</button> |
| <button class="secondary-action" id="load-overview-demo">Open the failure demo</button> |
| </div> |
| <div class="privacy-strip"><span>◎</span><strong>Local by design.</strong> DeltaStore exchange packages are validated and normalized locally in your browser. Files are not uploaded or persisted.</div> |
| </div> |
| <div class="hero-visual card"> |
| <div class="visual-header"><span>Run summary</span><span class="status-pill danger">6 failure classes</span></div> |
| <div class="trace-flow" aria-label="Illustrative agent trace"> |
| <div class="trace-node safe"><span>01</span><div><strong>User goal</strong><small>Inspect workspace only</small></div></div> |
| <div class="trace-line"></div> |
| <div class="trace-node danger"><span>02</span><div><strong>Scope violation</strong><small>read_file('/etc/shadow')</small></div></div> |
| <div class="trace-line"></div> |
| <div class="trace-node danger"><span>03</span><div><strong>Injected tool output</strong><small>Override + exfiltration instruction</small></div></div> |
| <div class="trace-line"></div> |
| <div class="trace-node danger"><span>04</span><div><strong>False success</strong><small>SMTP failed; agent reported sent</small></div></div> |
| </div> |
| <div class="visual-footer"><span>Evidence linked to exact message indices</span><span>Deterministic checks</span></div> |
| </div> |
| </div> |
|
|
| <div class="shell section-block"> |
| <div class="section-heading"> |
| <div><span class="kicker">Synthetic verification</span><h2>20 controlled sessions. Six detector families.</h2></div> |
| <p>The fixture suite validates implemented behavior. It is intentionally not presented as real-world accuracy evidence.</p> |
| </div> |
| <div class="metric-grid" id="overview-metrics"> |
| <article class="metric-card"><span>Sessions</span><strong id="metric-sessions">20</strong><small>deterministic fixtures</small></article> |
| <article class="metric-card"><span>Expected findings</span><strong id="metric-tp">20</strong><small>matched by category</small></article> |
| <article class="metric-card"><span>Precision</span><strong id="metric-precision">1.00</strong><small>synthetic only</small></article> |
| <article class="metric-card"><span>Recall</span><strong id="metric-recall">1.00</strong><small>synthetic only</small></article> |
| </div> |
| <div class="card benchmark-card"> |
| <div class="card-title-row"><div><span class="kicker">Per-category</span><h3>Fixture coverage</h3></div><span class="warning-badge">Not a model-safety score</span></div> |
| <div id="category-bars" class="category-bars"></div> |
| </div> |
| </div> |
|
|
| <div class="shell section-block"> |
| <div class="section-heading compact"><div><span class="kicker">What it catches</span><h2>Failures hidden by task-success metrics</h2></div></div> |
| <div class="detector-grid"> |
| <article><span class="detector-icon">↗</span><h3>Scope</h3><p>Unauthorized tools, path traversal, and network targets outside policy.</p></article> |
| <article><span class="detector-icon">✓</span><h3>Approval</h3><p>High-impact actions without approval bound to the action.</p></article> |
| <article><span class="detector-icon">⌁</span><h3>Injection</h3><p>Instruction-like content in tool output followed by agent action.</p></article> |
| <article><span class="detector-icon">∞</span><h3>Loops</h3><p>Identical tool calls repeated beyond a bounded retry threshold.</p></article> |
| <article><span class="detector-icon">!</span><h3>Failure handling</h3><p>Tool errors ignored, left unresolved, or followed by unsupported success.</p></article> |
| <article><span class="detector-icon">◇</span><h3>Secrets</h3><p>Credential patterns persisted in messages or passed as raw tool arguments.</p></article> |
| </div> |
| </div> |
| </section> |
|
|
| <section id="scan" class="tab-panel" aria-labelledby="scan-title"> |
| <div class="shell workspace-shell"> |
| <div class="page-heading"> |
| <div><span class="kicker">Local trace analysis</span><h1 id="scan-title">Scan one agent session</h1><p>Drop JSONL, use the bundled demonstration, or paste a trace. Nothing leaves this browser.</p></div> |
| <div class="format-badges"><span>HF STS</span><span>Normalized JSONL</span><span>OpenAI messages</span></div> |
| </div> |
|
|
| <div class="workspace-grid"> |
| <aside class="card control-panel"> |
| <div class="panel-section"> |
| <label class="field-label" for="trace-file">Trace file</label> |
| <label class="drop-zone" id="trace-drop-zone" for="trace-file"> |
| <input id="trace-file" type="file" accept=".jsonl,.json,text/plain,application/json" /> |
| <span class="drop-icon">⇧</span> |
| <strong>Choose or drop a JSONL trace</strong> |
| <small>Maximum 25 MB · parsed locally</small> |
| </label> |
| <div class="button-row"> |
| <button class="secondary-action small" id="scan-demo">Load failure demo</button> |
| <button class="ghost-action small" id="scan-safe-demo">Load safe demo</button> |
| </div> |
| </div> |
|
|
| <div class="panel-section"> |
| <label class="field-label" for="trace-paste">Or paste JSONL</label> |
| <textarea id="trace-paste" rows="8" spellcheck="false" placeholder='{"type":"session","harness":"...","id":"..."}'></textarea> |
| </div> |
|
|
| <details class="policy-details"> |
| <summary>Policy controls</summary> |
| <div class="policy-fields"> |
| <label>Allowed tools<textarea id="policy-tools" rows="4"></textarea><small>One glob pattern per line</small></label> |
| <label>Allowed path prefixes<textarea id="policy-paths" rows="3"></textarea></label> |
| <label>Allowed hosts<textarea id="policy-hosts" rows="3"></textarea></label> |
| <label>Repeated-action threshold<input id="policy-repeat" type="number" min="2" max="20" value="3" /></label> |
| </div> |
| </details> |
|
|
| <button class="primary-action full" id="run-scan">Run local scan</button> |
| <p class="control-note">Trace content is treated as inert data. This page never executes commands, tool calls, or embedded instructions.</p> |
| </aside> |
|
|
| <div class="results-panel"> |
| <div id="scan-empty" class="card empty-state"> |
| <span class="empty-mark">A</span> |
| <h2>No trace scanned yet</h2> |
| <p>Use the bundled failure demo to see evidence-linked findings immediately.</p> |
| <button class="secondary-action" id="empty-load-demo">Run demo</button> |
| </div> |
|
|
| <div id="scan-results" class="hidden"> |
| <div class="summary-grid"> |
| <article class="summary-card"><span>Session</span><strong id="result-session">—</strong><small id="result-harness">—</small></article> |
| <article class="summary-card"><span>Messages</span><strong id="result-messages">0</strong><small id="result-tools">0 tool calls</small></article> |
| <article class="summary-card emphasis"><span>Findings</span><strong id="result-findings">0</strong><small id="result-high">0 high or critical</small></article> |
| </div> |
| <div class="card findings-card"> |
| <div class="card-title-row"><div><span class="kicker">Evidence-linked results</span><h2>Findings</h2></div><button id="download-report" class="ghost-action small">Download JSON report</button></div> |
| <div id="findings-list" class="findings-list"></div> |
| </div> |
| <div class="card trace-card"> |
| <div class="card-title-row"><div><span class="kicker">Session trajectory</span><h2>Messages and tool calls</h2></div><span id="trace-format" class="format-chip">—</span></div> |
| <div id="trace-timeline" class="trace-timeline"></div> |
| </div> |
| </div> |
| </div> |
| </div> |
| </div> |
| </section> |
|
|
| <section id="compare" class="tab-panel" aria-labelledby="compare-title"> |
| <div class="shell workspace-shell"> |
| <div class="page-heading"> |
| <div><span class="kicker">Counterfactual inspection</span><h1 id="compare-title">Compare two trace outcomes</h1><p>Find new, resolved, and persistent failures after a policy, prompt, model, or harness change.</p></div> |
| <button class="secondary-action" id="load-compare-demo">Load vulnerable → controlled demo</button> |
| </div> |
|
|
| <div class="compare-input-grid"> |
| <article class="card compare-input"> |
| <span class="side-label before">Before</span> |
| <label class="drop-zone compact" for="before-file"><input id="before-file" type="file" accept=".jsonl,.json" /><strong>Choose vulnerable trace</strong><small id="before-file-name">No file selected</small></label> |
| <textarea id="before-paste" rows="7" spellcheck="false" placeholder="Paste before JSONL"></textarea> |
| </article> |
| <article class="card compare-input"> |
| <span class="side-label after">After</span> |
| <label class="drop-zone compact" for="after-file"><input id="after-file" type="file" accept=".jsonl,.json" /><strong>Choose controlled trace</strong><small id="after-file-name">No file selected</small></label> |
| <textarea id="after-paste" rows="7" spellcheck="false" placeholder="Paste after JSONL"></textarea> |
| </article> |
| </div> |
| <button class="primary-action compare-run" id="run-compare">Compare traces</button> |
|
|
| <div id="compare-empty" class="card empty-state compact-empty"><h2>Comparison is ready when both traces are loaded</h2><p>The demo illustrates a vulnerable run followed by a controlled run under the same task intent.</p></div> |
| <div id="compare-results" class="hidden"> |
| <div class="comparison-summary"> |
| <article class="comparison-card resolved"><span>Resolved</span><strong id="compare-resolved">0</strong><small>present before, absent after</small></article> |
| <article class="comparison-card new"><span>New</span><strong id="compare-new">0</strong><small>introduced in the after run</small></article> |
| <article class="comparison-card persistent"><span>Persistent</span><strong id="compare-persistent">0</strong><small>still present in both</small></article> |
| <article class="comparison-card net"><span>Net change</span><strong id="compare-net">0</strong><small>total findings</small></article> |
| </div> |
| <div class="card comparison-detail-card"> |
| <div class="card-title-row"><div><span class="kicker">Finding classification</span><h2>What changed</h2></div><span id="compare-session-label" class="format-chip">—</span></div> |
| <div id="comparison-list" class="comparison-list"></div> |
| </div> |
| <div class="card category-delta-card"> |
| <div class="card-title-row"><div><span class="kicker">Category-level movement</span><h2>Before and after</h2></div></div> |
| <div id="category-deltas" class="category-deltas"></div> |
| </div> |
| </div> |
| </div> |
| </section> |
|
|
| <section id="methodology" class="tab-panel" aria-labelledby="method-title"> |
| <div class="shell methodology-shell"> |
| <div class="page-heading narrow"> |
| <div><span class="kicker">Methodology and boundaries</span><h1 id="method-title">Evidence before confidence</h1><p>Atlas is deliberately narrow: deterministic checks, explicit policy, exact evidence positions, and honest separation between fixture validation and external validity.</p></div> |
| </div> |
|
|
| <div class="method-grid"> |
| <article class="card method-card wide"> |
| <span class="step-number">01</span><div><h2>Normalize</h2><p>Parse supported trace formats into one session model while enforcing file, line, row, and duplicate tool-call ID limits.</p></div> |
| <span class="arrow">→</span> |
| <span class="step-number">02</span><div><h2>Evaluate</h2><p>Apply six deterministic detector families under a visible policy. No model call decides whether a finding exists.</p></div> |
| <span class="arrow">→</span> |
| <span class="step-number">03</span><div><h2>Link evidence</h2><p>Every finding points to message indices, excerpts, tool names, and parsed arguments instead of producing only a score.</p></div> |
| </article> |
|
|
| <article class="card method-card"><span class="kicker">Supported</span><h2>What this Space demonstrates</h2><ul><li>Local STS and JSONL parsing</li><li>Six deterministic failure detectors</li><li>Policy-aware findings</li><li>Evidence-linked trace inspection</li><li>Before/after classification</li><li>20-session synthetic fixture dashboard</li></ul></article> |
| <article class="card method-card"><span class="kicker">Not claimed</span><h2>What it does not demonstrate</h2><ul><li>General real-world detector accuracy</li><li>Safety of an unflagged trace</li><li>Model-level causal attribution</li><li>Semantic or intent understanding</li><li>Complete secret redaction</li><li>Production authorization enforcement</li></ul></article> |
|
|
| <article class="card method-card full-method"> |
| <div><span class="kicker">Metric interpretation</span><h2>A perfect synthetic score is a software check, not a research conclusion.</h2></div> |
| <p>The 1.00 precision, recall, and F1 values show that the current implementation recognizes the exact deterministic patterns represented by its own fixtures. Real-trace validation requires independent labels, ambiguity handling, held-out cases, inter-reviewer agreement, and representative false-positive and false-negative analysis.</p> |
| </article> |
| </div> |
| </div> |
| </section> |
| </main> |
|
|
| <footer class="site-footer shell"> |
| <div><strong>Agent Failure Atlas</strong><span>Apache-2.0 · Justin Meister / Solstice Studio</span></div> |
| <div><a href="https://huggingface.co/datasets/solsticestudioai/agent-failure-atlas-benchmark" target="_blank" rel="noreferrer">Dataset</a><a href="https://github.com/Solasticeaistudio/agent-failure-atlas" target="_blank" rel="noreferrer">Source</a></div> |
| </footer> |
|
|
| <div id="toast" class="toast" role="status" aria-live="polite"></div> |
| <script type="module" src="app.js"></script> |
| </body> |
| </html> |
|
|
