README.md

metadata

title: AgentTrace | Agent Reliability Audit Studio
emoji: 🧭
colorFrom: green
colorTo: blue
sdk: static
app_file: index.html
pinned: false
license: mit
short_description: Offline AI agent evaluation and policy auditing
tags:
  - ai-agents
  - evaluation
  - observability
  - data-analysis
  - responsible-ai
  - javascript

AgentTrace | Agent Reliability & Audit Studio

Project 50 | Open-source, browser-based research and decision-support application

AgentTrace is an offline, reproducible laboratory for studying execution traces from AI agents. It makes it possible to compare two versions on the same tasks, audit tool calls against a transparent policy, identify unsafe recorded actions, and export the findings. It does not run AI models or autonomously call external tools.

What you can do

Module	Capabilities
Command Center	Task-success rate, critical-policy pass rate, average cost, audit signals
Trace Explorer	Inspect steps, costs, latency, approvals, risk and errors for each execution
Policy Audit	Flag forbidden tools, missing approvals, cost overruns, step caps and tool failures
Variant Benchmark	Compare paired tasks, including a deterministic bootstrap interval for differences in success
Scenario Laboratory	Change thresholds and compare audit outcomes against a fixed baseline
Data & Methodology	Import validated CSV, download templates, and export reports as JSON, Markdown and CSV

Quick start

1. In Hugging Face Create Space, choose Manual setup, Static, Blank, Public.
2. Choose repository name agenttrace-reliability-lab and MIT license.
3. Upload README.md, index.html and LICENSE at the root of the Space.
4. Commit changes to main and open the App tab.

No Python runtime, API tokens, CDNs, paid compute or npm installation is required.

Demo datasets

The Standard, Service desk and Adversarial stress scenarios contain entirely synthetic, deterministic records, with 32 task IDs evaluated by two agent variants (64 runs per scenario). All execution costs, latencies, successes and violations are simulated, not collected from commercial AI systems. The fixed dataset generation seed is 2026.

CSV format

Use Data & Methodology → Download CSV template for the exact schema. Each row represents one action step, with these mandatory columns:

run_id, task_id, category, variant, success, seq, tool, action, latency_ms, cost_usd, status, risk, approved

• success and approved: literal true or false
• status: ok or error; risk: low, medium or high
• seq: consecutive integers beginning at 1 within each run
• Each (task_id, variant) pair must be unique, and each run_id must be unique.
• Costs and latencies must be nonnegative finite numbers.
• CSV files are validated before changing the current dataset. Invalid uploads are rejected without replacing earlier data.
• Up to 3 MB CSV, 1,200 executions and 40 steps per execution, with additional parsing and field limits.

For realistic evaluations, collect real agent run logs with explicit permission and remove personal or confidential information before import.

Methods

1. Trace audit: Checks explicit rules for forbidden secret_export, missing approval for update_record or publish_external, configurable budget exceedances, step caps and tool errors. No external calls are executed; these names are labels in trace records.
2. Variant scorecard: Aggregates task success, critical-policy pass, mean simulated USD cost, mean latency and number of tool steps by variant.
3. Paired comparison: Aligns runs by exact task_id. Reports the success-rate difference B minus A and a percentile 95% interval from 2,000 deterministic, paired bootstrap resamples.
4. Scenario comparison: Re-applies changed rule thresholds to the same immutable input traces without falsely claiming that an agent was retrained or rerun.
5. Reproducible exports: JSON includes policy settings, metrics, paired outcomes and a per-finding audit trail.

Validation

• 77 passing core assertions under Node.js, including a CSV round trip and independent edge cases.
• 108 passing Chromium checks across desktop and mobile, including all six screens, policy controls, invalid CSV, CSV re-import, and seven types of downloadable output.
• No JavaScript page errors during these automated browser checks. Browser tests run against locally injected HTML because the test environment blocks local URL navigation.

The code and automated tests are provided separately in the full source archive.

Important limitations

This is an educational/offline log auditor, not an AI agent, not an AI safety certificate, and not a live monitoring service. A clean audit means that the logged fields pass only the configured checks; it cannot establish truthfulness, security, privacy, fairness or the absence of unlogged harmful actions. The paired bootstrap interval is exploratory and does not demonstrate causal superiority. Users are responsible for data provenance and consent when importing traces. All demonstrations use synthetic data and must not be represented as research results from real agent deployments.

License

MIT, see LICENSE.