| --- |
| title: "AgentTrace | Agent Reliability Audit Studio" |
| emoji: "🧭" |
| colorFrom: green |
| colorTo: blue |
| sdk: static |
| app_file: index.html |
| pinned: false |
| license: mit |
| short_description: "Offline AI agent evaluation and policy auditing" |
| tags: |
| - ai-agents |
| - evaluation |
| - observability |
| - data-analysis |
| - responsible-ai |
| - javascript |
| --- |
| |
| # AgentTrace | Agent Reliability & Audit Studio |
|
|
| **Project 50 | Open-source, browser-based research and decision-support application** |
|
|
| AgentTrace is an offline, reproducible laboratory for studying execution traces from AI agents. It makes it possible to compare two versions on the same tasks, audit tool calls against a transparent policy, identify unsafe **recorded** actions, and export the findings. It does **not** run AI models or autonomously call external tools. |
|
|
| ## What you can do |
|
|
| | Module | Capabilities | |
| | --- | --- | |
| | Command Center | Task-success rate, critical-policy pass rate, average cost, audit signals | |
| | Trace Explorer | Inspect steps, costs, latency, approvals, risk and errors for each execution | |
| | Policy Audit | Flag forbidden tools, missing approvals, cost overruns, step caps and tool failures | |
| | Variant Benchmark | Compare paired tasks, including a deterministic bootstrap interval for differences in success | |
| | Scenario Laboratory | Change thresholds and compare audit outcomes against a fixed baseline | |
| | Data & Methodology | Import validated CSV, download templates, and export reports as JSON, Markdown and CSV | |
|
|
| ## Quick start |
|
|
| 1. In **[Hugging Face Create Space](https://huggingface.co/new-space)**, choose **Manual setup**, **Static**, **Blank**, **Public**. |
| 2. Choose repository name `agenttrace-reliability-lab` and MIT license. |
| 3. Upload **README.md**, **index.html** and **LICENSE** at the root of the Space. |
| 4. Commit changes to `main` and open the **App** tab. |
|
|
| No Python runtime, API tokens, CDNs, paid compute or npm installation is required. |
|
|
| ## Demo datasets |
|
|
| The **Standard**, **Service desk** and **Adversarial stress** scenarios contain entirely synthetic, deterministic records, with **32 task IDs evaluated by two agent variants** (64 runs per scenario). All execution costs, latencies, successes and violations are **simulated**, not collected from commercial AI systems. The fixed dataset generation seed is 2026. |
|
|
| ## CSV format |
|
|
| Use **Data & Methodology → Download CSV template** for the exact schema. Each row represents **one action step**, with these mandatory columns: |
|
|
| `run_id, task_id, category, variant, success, seq, tool, action, latency_ms, cost_usd, status, risk, approved` |
|
|
| - `success` and `approved`: literal `true` or `false` |
| - `status`: `ok` or `error`; `risk`: `low`, `medium` or `high` |
| - `seq`: consecutive integers beginning at 1 within each run |
| - Each `(task_id, variant)` pair must be unique, and each `run_id` must be unique. |
| - Costs and latencies must be nonnegative finite numbers. |
| - CSV files are validated before changing the current dataset. Invalid uploads are rejected without replacing earlier data. |
| - Up to 3 MB CSV, 1,200 executions and 40 steps per execution, with additional parsing and field limits. |
|
|
| For realistic evaluations, collect real agent run logs with explicit permission and remove personal or confidential information **before import**. |
|
|
| ## Methods |
|
|
| 1. **Trace audit:** Checks explicit rules for forbidden `secret_export`, missing approval for `update_record` or `publish_external`, configurable budget exceedances, step caps and tool errors. No external calls are executed; these names are labels in trace records. |
| 2. **Variant scorecard:** Aggregates task success, critical-policy pass, mean simulated USD cost, mean latency and number of tool steps by variant. |
| 3. **Paired comparison:** Aligns runs by exact `task_id`. Reports the success-rate difference B minus A and a percentile 95% interval from 2,000 deterministic, paired bootstrap resamples. |
| 4. **Scenario comparison:** Re-applies changed rule thresholds to the same immutable input traces without falsely claiming that an agent was retrained or rerun. |
| 5. **Reproducible exports:** JSON includes policy settings, metrics, paired outcomes and a per-finding audit trail. |
|
|
| ## Validation |
|
|
| - **77 passing core assertions** under Node.js, including a CSV round trip and independent edge cases. |
| - **108 passing Chromium checks** across desktop and mobile, including all six screens, policy controls, invalid CSV, CSV re-import, and seven types of downloadable output. |
| - No JavaScript page errors during these automated browser checks. Browser tests run against locally injected HTML because the test environment blocks local URL navigation. |
|
|
| The code and automated tests are provided separately in the full source archive. |
|
|
| ## Important limitations |
|
|
| This is an **educational/offline log auditor**, not an AI agent, not an AI safety certificate, and not a live monitoring service. A clean audit means that **the logged fields pass only the configured checks**; it cannot establish truthfulness, security, privacy, fairness or the absence of unlogged harmful actions. The paired bootstrap interval is exploratory and does not demonstrate causal superiority. Users are responsible for data provenance and consent when importing traces. All demonstrations use synthetic data and must not be represented as research results from real agent deployments. |
|
|
| ## License |
|
|
| MIT, see [LICENSE](LICENSE). |
|
|
