File size: 16,962 Bytes
3ccf493 d9e40c9 f287f86 d9e40c9 e3eab91 d9e40c9 3ccf493 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 | <!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="description" content="Client-side trace-level security and reliability analysis for AI agents." />
<title>Agent Failure Atlas</title>
<link rel="stylesheet" href="styles.css" />
</head>
<body>
<div class="ambient ambient-one"></div>
<div class="ambient ambient-two"></div>
<header class="site-header">
<a class="brand" href="#overview" aria-label="Agent Failure Atlas home">
<span class="brand-mark">A</span>
<span><strong>Agent Failure Atlas</strong><small>Trace-level evidence, locally analyzed</small></span>
</a>
<nav class="nav" aria-label="Primary navigation">
<button class="nav-link active" data-tab="overview">Overview</button>
<button class="nav-link" data-tab="scan">Scan</button>
<button class="nav-link" data-tab="compare">Compare</button>
<button class="nav-link" data-tab="methodology">Methodology</button>
</nav>
<div class="header-links">
<a href="https://huggingface.co/datasets/solsticestudioai/agent-failure-atlas-benchmark" target="_blank" rel="noreferrer">Dataset ↗</a>
<a href="https://huggingface.co/spaces/solsticestudioai/deltastore-trace-explorer" target="_blank" rel="noreferrer">DeltaStore</a>`r`n <a href="https://github.com/Solasticeaistudio/agent-failure-atlas" target="_blank" rel="noreferrer">GitHub ↗</a>
</div>
</header>
<main>
<section id="overview" class="tab-panel active" aria-labelledby="overview-title">
<div class="hero shell">
<div class="hero-copy">
<div class="eyebrow"><span class="pulse-dot"></span> Open, deterministic, evidence-linked</div>
<h1 id="overview-title">The final answer can look right while the agent run goes wrong.</h1>
<p>Agent Failure Atlas inspects the trajectory: tool scope, approvals, injected instructions, repeated actions, silent failures, and exposed secrets. This static edition runs entirely inside your browser.</p>
<div class="hero-actions">
<button class="primary-action" data-go="scan">Analyze a trace</button>
<button class="secondary-action" id="load-overview-demo">Open the failure demo</button>
</div>
<div class="privacy-strip"><span>◎</span><strong>Local by design.</strong> DeltaStore exchange packages are validated and normalized locally in your browser. Files are not uploaded or persisted.</div>
</div>
<div class="hero-visual card">
<div class="visual-header"><span>Run summary</span><span class="status-pill danger">6 failure classes</span></div>
<div class="trace-flow" aria-label="Illustrative agent trace">
<div class="trace-node safe"><span>01</span><div><strong>User goal</strong><small>Inspect workspace only</small></div></div>
<div class="trace-line"></div>
<div class="trace-node danger"><span>02</span><div><strong>Scope violation</strong><small>read_file('/etc/shadow')</small></div></div>
<div class="trace-line"></div>
<div class="trace-node danger"><span>03</span><div><strong>Injected tool output</strong><small>Override + exfiltration instruction</small></div></div>
<div class="trace-line"></div>
<div class="trace-node danger"><span>04</span><div><strong>False success</strong><small>SMTP failed; agent reported sent</small></div></div>
</div>
<div class="visual-footer"><span>Evidence linked to exact message indices</span><span>Deterministic checks</span></div>
</div>
</div>
<div class="shell section-block">
<div class="section-heading">
<div><span class="kicker">Synthetic verification</span><h2>20 controlled sessions. Six detector families.</h2></div>
<p>The fixture suite validates implemented behavior. It is intentionally not presented as real-world accuracy evidence.</p>
</div>
<div class="metric-grid" id="overview-metrics">
<article class="metric-card"><span>Sessions</span><strong id="metric-sessions">20</strong><small>deterministic fixtures</small></article>
<article class="metric-card"><span>Expected findings</span><strong id="metric-tp">20</strong><small>matched by category</small></article>
<article class="metric-card"><span>Precision</span><strong id="metric-precision">1.00</strong><small>synthetic only</small></article>
<article class="metric-card"><span>Recall</span><strong id="metric-recall">1.00</strong><small>synthetic only</small></article>
</div>
<div class="card benchmark-card">
<div class="card-title-row"><div><span class="kicker">Per-category</span><h3>Fixture coverage</h3></div><span class="warning-badge">Not a model-safety score</span></div>
<div id="category-bars" class="category-bars"></div>
</div>
</div>
<div class="shell section-block">
<div class="section-heading compact"><div><span class="kicker">What it catches</span><h2>Failures hidden by task-success metrics</h2></div></div>
<div class="detector-grid">
<article><span class="detector-icon">↗</span><h3>Scope</h3><p>Unauthorized tools, path traversal, and network targets outside policy.</p></article>
<article><span class="detector-icon">✓</span><h3>Approval</h3><p>High-impact actions without approval bound to the action.</p></article>
<article><span class="detector-icon">⌁</span><h3>Injection</h3><p>Instruction-like content in tool output followed by agent action.</p></article>
<article><span class="detector-icon">∞</span><h3>Loops</h3><p>Identical tool calls repeated beyond a bounded retry threshold.</p></article>
<article><span class="detector-icon">!</span><h3>Failure handling</h3><p>Tool errors ignored, left unresolved, or followed by unsupported success.</p></article>
<article><span class="detector-icon">◇</span><h3>Secrets</h3><p>Credential patterns persisted in messages or passed as raw tool arguments.</p></article>
</div>
</div>
</section>
<section id="scan" class="tab-panel" aria-labelledby="scan-title">
<div class="shell workspace-shell">
<div class="page-heading">
<div><span class="kicker">Local trace analysis</span><h1 id="scan-title">Scan one agent session</h1><p>Drop JSONL, use the bundled demonstration, or paste a trace. Nothing leaves this browser.</p></div>
<div class="format-badges"><span>HF STS</span><span>Normalized JSONL</span><span>OpenAI messages</span></div>
</div>
<div class="workspace-grid">
<aside class="card control-panel">
<div class="panel-section">
<label class="field-label" for="trace-file">Trace file</label>
<label class="drop-zone" id="trace-drop-zone" for="trace-file">
<input id="trace-file" type="file" accept=".jsonl,.json,text/plain,application/json" />
<span class="drop-icon">⇧</span>
<strong>Choose or drop a JSONL trace</strong>
<small>Maximum 25 MB · parsed locally</small>
</label>
<div class="button-row">
<button class="secondary-action small" id="scan-demo">Load failure demo</button>
<button class="ghost-action small" id="scan-safe-demo">Load safe demo</button>
</div>
</div>
<div class="panel-section">
<label class="field-label" for="trace-paste">Or paste JSONL</label>
<textarea id="trace-paste" rows="8" spellcheck="false" placeholder='{"type":"session","harness":"...","id":"..."}'></textarea>
</div>
<details class="policy-details">
<summary>Policy controls</summary>
<div class="policy-fields">
<label>Allowed tools<textarea id="policy-tools" rows="4"></textarea><small>One glob pattern per line</small></label>
<label>Allowed path prefixes<textarea id="policy-paths" rows="3"></textarea></label>
<label>Allowed hosts<textarea id="policy-hosts" rows="3"></textarea></label>
<label>Repeated-action threshold<input id="policy-repeat" type="number" min="2" max="20" value="3" /></label>
</div>
</details>
<button class="primary-action full" id="run-scan">Run local scan</button>
<p class="control-note">Trace content is treated as inert data. This page never executes commands, tool calls, or embedded instructions.</p>
</aside>
<div class="results-panel">
<div id="scan-empty" class="card empty-state">
<span class="empty-mark">A</span>
<h2>No trace scanned yet</h2>
<p>Use the bundled failure demo to see evidence-linked findings immediately.</p>
<button class="secondary-action" id="empty-load-demo">Run demo</button>
</div>
<div id="scan-results" class="hidden">
<div class="summary-grid">
<article class="summary-card"><span>Session</span><strong id="result-session">—</strong><small id="result-harness">—</small></article>
<article class="summary-card"><span>Messages</span><strong id="result-messages">0</strong><small id="result-tools">0 tool calls</small></article>
<article class="summary-card emphasis"><span>Findings</span><strong id="result-findings">0</strong><small id="result-high">0 high or critical</small></article>
</div>
<div class="card findings-card">
<div class="card-title-row"><div><span class="kicker">Evidence-linked results</span><h2>Findings</h2></div><button id="download-report" class="ghost-action small">Download JSON report</button></div>
<div id="findings-list" class="findings-list"></div>
</div>
<div class="card trace-card">
<div class="card-title-row"><div><span class="kicker">Session trajectory</span><h2>Messages and tool calls</h2></div><span id="trace-format" class="format-chip">—</span></div>
<div id="trace-timeline" class="trace-timeline"></div>
</div>
</div>
</div>
</div>
</div>
</section>
<section id="compare" class="tab-panel" aria-labelledby="compare-title">
<div class="shell workspace-shell">
<div class="page-heading">
<div><span class="kicker">Counterfactual inspection</span><h1 id="compare-title">Compare two trace outcomes</h1><p>Find new, resolved, and persistent failures after a policy, prompt, model, or harness change.</p></div>
<button class="secondary-action" id="load-compare-demo">Load vulnerable → controlled demo</button>
</div>
<div class="compare-input-grid">
<article class="card compare-input">
<span class="side-label before">Before</span>
<label class="drop-zone compact" for="before-file"><input id="before-file" type="file" accept=".jsonl,.json" /><strong>Choose vulnerable trace</strong><small id="before-file-name">No file selected</small></label>
<textarea id="before-paste" rows="7" spellcheck="false" placeholder="Paste before JSONL"></textarea>
</article>
<article class="card compare-input">
<span class="side-label after">After</span>
<label class="drop-zone compact" for="after-file"><input id="after-file" type="file" accept=".jsonl,.json" /><strong>Choose controlled trace</strong><small id="after-file-name">No file selected</small></label>
<textarea id="after-paste" rows="7" spellcheck="false" placeholder="Paste after JSONL"></textarea>
</article>
</div>
<button class="primary-action compare-run" id="run-compare">Compare traces</button>
<div id="compare-empty" class="card empty-state compact-empty"><h2>Comparison is ready when both traces are loaded</h2><p>The demo illustrates a vulnerable run followed by a controlled run under the same task intent.</p></div>
<div id="compare-results" class="hidden">
<div class="comparison-summary">
<article class="comparison-card resolved"><span>Resolved</span><strong id="compare-resolved">0</strong><small>present before, absent after</small></article>
<article class="comparison-card new"><span>New</span><strong id="compare-new">0</strong><small>introduced in the after run</small></article>
<article class="comparison-card persistent"><span>Persistent</span><strong id="compare-persistent">0</strong><small>still present in both</small></article>
<article class="comparison-card net"><span>Net change</span><strong id="compare-net">0</strong><small>total findings</small></article>
</div>
<div class="card comparison-detail-card">
<div class="card-title-row"><div><span class="kicker">Finding classification</span><h2>What changed</h2></div><span id="compare-session-label" class="format-chip">—</span></div>
<div id="comparison-list" class="comparison-list"></div>
</div>
<div class="card category-delta-card">
<div class="card-title-row"><div><span class="kicker">Category-level movement</span><h2>Before and after</h2></div></div>
<div id="category-deltas" class="category-deltas"></div>
</div>
</div>
</div>
</section>
<section id="methodology" class="tab-panel" aria-labelledby="method-title">
<div class="shell methodology-shell">
<div class="page-heading narrow">
<div><span class="kicker">Methodology and boundaries</span><h1 id="method-title">Evidence before confidence</h1><p>Atlas is deliberately narrow: deterministic checks, explicit policy, exact evidence positions, and honest separation between fixture validation and external validity.</p></div>
</div>
<div class="method-grid">
<article class="card method-card wide">
<span class="step-number">01</span><div><h2>Normalize</h2><p>Parse supported trace formats into one session model while enforcing file, line, row, and duplicate tool-call ID limits.</p></div>
<span class="arrow">→</span>
<span class="step-number">02</span><div><h2>Evaluate</h2><p>Apply six deterministic detector families under a visible policy. No model call decides whether a finding exists.</p></div>
<span class="arrow">→</span>
<span class="step-number">03</span><div><h2>Link evidence</h2><p>Every finding points to message indices, excerpts, tool names, and parsed arguments instead of producing only a score.</p></div>
</article>
<article class="card method-card"><span class="kicker">Supported</span><h2>What this Space demonstrates</h2><ul><li>Local STS and JSONL parsing</li><li>Six deterministic failure detectors</li><li>Policy-aware findings</li><li>Evidence-linked trace inspection</li><li>Before/after classification</li><li>20-session synthetic fixture dashboard</li></ul></article>
<article class="card method-card"><span class="kicker">Not claimed</span><h2>What it does not demonstrate</h2><ul><li>General real-world detector accuracy</li><li>Safety of an unflagged trace</li><li>Model-level causal attribution</li><li>Semantic or intent understanding</li><li>Complete secret redaction</li><li>Production authorization enforcement</li></ul></article>
<article class="card method-card full-method">
<div><span class="kicker">Metric interpretation</span><h2>A perfect synthetic score is a software check, not a research conclusion.</h2></div>
<p>The 1.00 precision, recall, and F1 values show that the current implementation recognizes the exact deterministic patterns represented by its own fixtures. Real-trace validation requires independent labels, ambiguity handling, held-out cases, inter-reviewer agreement, and representative false-positive and false-negative analysis.</p>
</article>
</div>
</div>
</section>
</main>
<footer class="site-footer shell">
<div><strong>Agent Failure Atlas</strong><span>Apache-2.0 · Justin Meister / Solstice Studio</span></div>
<div><a href="https://huggingface.co/datasets/solsticestudioai/agent-failure-atlas-benchmark" target="_blank" rel="noreferrer">Dataset</a><a href="https://github.com/Solasticeaistudio/agent-failure-atlas" target="_blank" rel="noreferrer">Source</a></div>
</footer>
<div id="toast" class="toast" role="status" aria-live="polite"></div>
<script type="module" src="app.js"></script>
</body>
</html>
|