docs/architecture/CANONICAL_TRACE_QUICKREF.md

Canonical Normalized Trace - Quick Reference

Quick Start

Generate Normalized Traces

dabench eval-baseline <run_id>

Output: baseline_evaluation/normalized_traces/task_*.json

View a Trace

dabench view-normalized-trace <run_id> <task_id>

Shows detailed breakdown with validation and metrics.

Load Programmatically

from data_agent_baseline.evaluation.normalized_trace_manager import NormalizedTraceManager

manager = NormalizedTraceManager(output_dir)
trace = manager.load_normalized_trace("task_22")

Common Tasks

List Available Traces

task_ids = manager.list_normalized_traces()
print(f"Available: {', '.join(task_ids)}")

Validate a Trace

is_valid, errors = manager.validate_normalized_trace(trace)
if not is_valid:
    print(f"Errors: {errors}")

Get Metrics

metrics = manager.get_trace_metrics(trace)
print(f"Steps: {metrics['num_steps']}")
print(f"Tools: {metrics['tool_counts']}")

Compare Agents

baseline_trace = manager.load_normalized_trace("task_22")
multi_agent_trace = ...  # Load from different run

baseline_metrics = manager.get_trace_metrics(baseline_trace)
multi_agent_metrics = manager.get_trace_metrics(multi_agent_trace)

print(f"Baseline: {baseline_metrics['num_steps']} steps")
print(f"Multi-agent: {multi_agent_metrics['num_steps']} steps")

Schema at a Glance

{
  "run_id": "string",
  "task_id": "string",
  "agent_type": "baseline_react",
  "question": "string",
  "difficulty": "Easy",
  "success": true,
  "final_answer": {"columns": [...], "rows": [...]},
  "duration_seconds": 8.5,
  "steps": [
    {
      "step_id": 1,
      "agent": "baseline_react",
      "agent_role": "worker",
      "thought": "...",
      "action": "list_context",
      "action_input": {...},
      "observation": {...},
      "tool_success": true
    }
  ]
}

CLI Commands

Command	Description
eval-baseline <run_id>	Generate normalized traces and evaluation
view-normalized-trace <run_id> <task_id>	View detailed trace breakdown

Key Metrics

From get_trace_metrics():

Metric	Description
num_steps	Total execution steps
num_tool_calls	Total tool invocations
num_failed_tools	Failed tool calls
unique_tools	Distinct tools used
agent_steps	Steps per agent (multi-agent)
role_steps	Steps per role (multi-agent)
tool_counts	Frequency per tool
phase_steps	Steps per phase (if available)

Multi-Agent Support

Baseline ReAct (Single Agent)

{"agent": "baseline_react", "agent_role": "worker"}

Planner + Executor

{"agent": "planner", "agent_role": "planner"}
{"agent": "executor", "agent_role": "worker"}

Multi-Agent Team

{"agent": "scout", "agent_role": "worker", "phase": "explore"}
{"agent": "analyst", "agent_role": "worker", "phase": "analyze"}
{"agent": "critic", "agent_role": "critic", "phase": "verify"}

Validation

Built-in validation checks:

• ✅ Required fields present
• ✅ Step ordering sequential
• ✅ Type correctness
• ✅ Schema consistency

# Validate single trace
is_valid, errors = manager.validate_normalized_trace(trace)

# Validate all traces
results = manager.validate_all_traces()
for task_id, (is_valid, errors) in results.items():
    if not is_valid:
        print(f"{task_id}: {errors}")

File Locations

baseline_evaluation/
├── task_results.csv
├── summary_metrics.json
├── evaluation_report.md
└── normalized_traces/        # ← One file per task
    ├── task_001.json
    ├── task_002.json
    └── task_022.json

Documentation

• CANONICAL_TRACE_FORMAT.md - Complete schema documentation
• CANONICAL_TRACE_IMPLEMENTATION.md - Implementation summary
• BASELINE_TRACE_SCHEMA.md - Baseline trace analysis

Example

# Generate traces
dabench eval-baseline 20260613T114457Z

# View trace
dabench view-normalized-trace 20260613T114457Z task_22

# Programmatic access
python3 << EOF
from pathlib import Path
from data_agent_baseline.evaluation.normalized_trace_manager import NormalizedTraceManager

manager = NormalizedTraceManager(
    Path("/data3/dataFAIR/kdd-dev/public/artifacts/runs/20260613T114457Z/baseline_evaluation")
)

trace = manager.load_normalized_trace("task_22")
print(f"Task: {trace['task_id']}")
print(f"Success: {trace['success']}")
print(f"Steps: {len(trace['steps'])}")

metrics = manager.get_trace_metrics(trace)
print(f"Metrics: {metrics}")
EOF

Tips

1. One file per task makes it easy to find and analyze individual traces
2. Validation on save ensures schema compliance
3. Derived metrics computed on-demand from trace data
4. Multi-agent ready - just set different agent/role per step
5. NULL-safe - missing data represented as None, not omitted

See Also

• src/data_agent_baseline/evaluation/ - Implementation code
• Example: /data3/dataFAIR/kdd-dev/public/artifacts/runs/20260613T114457Z/baseline_evaluation/normalized_traces/task_22.json