Looking for five people to break this agent-trace reliability toolkit
I?ve published a small open research project for inspecting, evaluating, and comparing AI-agent trajectories:
Solstice Agent Reliability Lab
https://huggingface.co/collections/solsticestudioai/solstice-agent-reliability-lab
It currently includes two connected browser-based tools:
Agent Failure Atlas analyzes agent traces for evidence-linked security, reliability, and control failures.
DeltaStore Trace Explorer reconstructs checkpointed trajectories, compares controlled alternate branches, and shows how changes to policies or behavior affect the outcome.
The two tools share a versioned trace-exchange format. A trace can be exported from DeltaStore, scanned in Atlas, and returned with findings linked to the original evidence events. Everything runs locally in the browser, without paid compute or server-side trace uploads.
The current datasets contain deterministic synthetic scenarios covering:
tool-output prompt injection
approval bypass
scope violations
silent tool failures
runaway action loops
secret exposure
stale state and memory
destructive actions
safety-versus-task-success regressions
I?m looking for five people willing to criticize the work rather than politely praise it.
In particular:
What important state is missing from the trace format?
Which detectors are likely to produce false positives on real agent sessions?
Are the checkpoint and branch comparisons meaningful and auditable?
Could your agent framework export enough information to support this workflow?
What would prevent you from using it for debugging or evaluation?
Which claims, metrics, or visualizations feel misleading?
All current traces are synthetic. The published metrics validate the fixtures and implementation, not real-world detector accuracy. The branch comparisons are controlled counterfactuals, not proof that a model would behave identically during nondeterministic re-execution.
I would rather uncover the weak points publicly now than inflate the claims and discover them later.
Collection:
https://huggingface.co/collections/solsticestudioai/solstice-agent-reliability-lab