docs/architecture/CANONICAL_TRACE_FORMAT.md

Canonical Normalized Trace Format

Overview

The canonical normalized trace format provides a unified representation for execution traces across different agent architectures. This enables:

1. Agent-agnostic evaluation - Same metrics for all agent types
2. Trace replay - Reconstruct execution from normalized traces
3. Failure analysis - Analyze errors across different systems
4. Visualization - DAG visualization and timeline views
5. Comparison - Fair comparison between agent architectures

Schema Version

Version: 1.0
Date: 2026-06-13

Canonical Trace Schema

{
  "run_id": "string",
  "task_id": "string",
  "agent_type": "baseline_react | langgraph_agent | other",
  
  "question": "string",
  "difficulty": "Easy | Medium | Hard | Extreme | Unknown",
  
  "success": boolean,
  "final_answer": {
    "columns": ["string"],
    "rows": [["any"]]
  } | null,
  "failure_reason": "string" | null,
  
  "start_time": "ISO8601 string" | null,
  "end_time": "ISO8601 string" | null,
  "duration_seconds": number | null,
  
  "steps": [
    {
      "step_id": integer,
      
      "agent": "string",           // Agent name: "baseline_react", "planner", "executor"
      "agent_role": "string",      // Role: "worker", "planner", "critic", "coordinator"
      
      "thought": "string",
      "action": "string",
      "action_input": {},
      "observation": {},
      "tool_success": boolean,
      
      "raw_response": "string" | null,
      "timestamp": "ISO8601 string" | null,
      "phase": "string" | null,
      "duration_seconds": number | null,
      "token_usage": {
        "prompt_tokens": integer,
        "completion_tokens": integer,
        "total_tokens": integer
      } | null
    }
  ],
  
  "metrics": {
    "num_steps": integer,
    "tool_calls": integer,
    "tool_failures": integer,
    "unique_tools": integer,
    ...
  } | null,
  
  "trace_path": "string" | null,
  "prediction_path": "string" | null,
  "context_dir": "string" | null,
  "task_root": "string" | null
}

Field Descriptions

Top-Level Fields

Field	Type	Required	Description
run_id	string	✅	Unique run identifier
task_id	string	✅	Task identifier
agent_type	string	✅	Agent architecture type
question	string	✅	Task question
difficulty	string	❌	Task difficulty level
success	boolean	✅	Whether task completed successfully
final_answer	object	❌	Final answer (if submitted)
failure_reason	string	❌	Error message if failed
start_time	string	❌	ISO8601 timestamp
end_time	string	❌	ISO8601 timestamp
duration_seconds	number	❌	Total execution time

Step Fields

Field	Type	Required	Description
step_id	integer	✅	Sequential step number (1-indexed)
agent	string	✅	Agent name that produced this step
agent_role	string	✅	Agent role (planner, worker, critic, coordinator)
thought	string	✅	Agent's reasoning
action	string	✅	Tool/action name
action_input	object	✅	Tool parameters
observation	object	✅	Tool output
tool_success	boolean	✅	Whether tool succeeded
raw_response	string	❌	Raw LLM output
timestamp	string	❌	Step timestamp
phase	string	❌	Execution phase
duration_seconds	number	❌	Step duration
token_usage	object	❌	LLM token usage

Multi-Agent Support

The schema supports multiple agent architectures:

Baseline ReAct (Single Agent)

{
  "agent": "baseline_react",
  "agent_role": "worker"
}

All steps have the same agent.

Planner + Executor

// Planning step
{
  "agent": "planner",
  "agent_role": "planner",
  "phase": "planning"
}

// Execution step
{
  "agent": "executor",
  "agent_role": "worker",
  "phase": "execution"
}

Multi-Agent Analyst Team

// Data exploration
{
  "agent": "scout",
  "agent_role": "worker",
  "phase": "explore"
}

// Analysis
{
  "agent": "analyst",
  "agent_role": "worker",
  "phase": "analyze"
}

// Verification
{
  "agent": "critic",
  "agent_role": "critic",
  "phase": "verify"
}

Coordinator-Based System

// Coordination
{
  "agent": "coordinator",
  "agent_role": "coordinator",
  "phase": "coordination"
}

// Worker execution
{
  "agent": "worker_1",
  "agent_role": "worker",
  "phase": "execution"
}

Agent Roles

Standard agent roles:

Role	Description	Examples
worker	Executes tasks and tools	Baseline ReAct, Executor, Analyst
planner	Creates execution plans	Planner agent
critic	Reviews and validates	Critic, Verifier
coordinator	Orchestrates multiple agents	Multi-agent coordinator

Phases

Common execution phases:

Phase	Description	Typical Agent Roles
explore	Data discovery and exploration	worker
plan	Planning and strategizing	planner
execute	Tool execution and data processing	worker
verify	Validation and verification	critic
refine	Correction and refinement	worker, critic
coordinate	Multi-agent coordination	coordinator

Usage

Generate Normalized Traces

dabench eval-baseline <run_id>

This generates:

baseline_evaluation/
├── task_results.csv
├── summary_metrics.json
├── evaluation_report.md
└── normalized_traces/
    ├── task_001.json
    ├── task_002.json
    └── ...

Load and Analyze

from data_agent_baseline.evaluation.normalized_trace_manager import NormalizedTraceManager

# Load traces
manager = NormalizedTraceManager(output_dir)
trace = manager.load_normalized_trace("task_22")

# Get metrics
metrics = manager.get_trace_metrics(trace)
print(f"Steps: {metrics['num_steps']}")
print(f"Tool calls: {metrics['num_tool_calls']}")
print(f"Agent breakdown: {metrics['agent_steps']}")

# Validate
is_valid, errors = manager.validate_normalized_trace(trace)
if not is_valid:
    print(f"Validation errors: {errors}")

Validate All Traces

manager = NormalizedTraceManager(output_dir)
results = manager.validate_all_traces()

for task_id, (is_valid, errors) in results.items():
    if not is_valid:
        print(f"{task_id}: {errors}")

Benefits

1. Agent-Agnostic Evaluation

Same metrics work across all agent types:

• Baseline ReAct
• Multi-agent systems
• Future architectures

2. Easy Comparison

baseline_trace = manager.load_normalized_trace("task_22")
multi_agent_trace = manager.load_normalized_trace("task_22", run_id="multi_agent_run")

baseline_metrics = manager.get_trace_metrics(baseline_trace)
multi_agent_metrics = manager.get_trace_metrics(multi_agent_trace)

print(f"Baseline steps: {baseline_metrics['num_steps']}")
print(f"Multi-agent steps: {multi_agent_metrics['num_steps']}")

3. Trace Replay

Reconstruct execution from normalized trace for:

• Debugging
• Visualization
• Failure analysis

4. Future-Proof

Schema supports:

• New agent architectures
• New phases
• New metrics
• Parallel execution
• Multi-agent coordination

Validation

The NormalizedTraceManager validates:

✅ Required fields:

• run_id, task_id, agent_type, question, success
• step_id, agent, agent_role, thought, action, etc.

✅ Step ordering:

• step_id must be sequential (1, 2, 3, ...)

✅ Type checking:

• Fields have correct types (string, int, bool, dict)

✅ Schema consistency:

• final_answer has columns and rows
• observation is a dictionary
• action_input is a dictionary

Migration Guide

From Baseline Traces

Baseline traces are automatically converted to normalized format:

from data_agent_baseline.evaluation.baseline_adapter import BaselineTraceAdapter

adapter = BaselineTraceAdapter(task_root)
canonical = adapter.normalize(trace_path, run_id)

From Custom Agents

Implement an adapter:

class CustomAgentAdapter:
    def normalize(self, raw_trace, run_id):
        return CanonicalTrace(
            run_id=run_id,
            task_id=raw_trace["task_id"],
            agent_type="custom_agent",
            question=raw_trace["question"],
            success=raw_trace["success"],
            steps=[
                CanonicalStep(
                    step_id=i,
                    agent="custom_agent",
                    agent_role="worker",
                    ...
                )
                for i, step in enumerate(raw_trace["steps"], 1)
            ],
            ...
        )

Example Trace

See: /data3/dataFAIR/kdd-dev/public/artifacts/runs/20260613T114457Z/baseline_evaluation/normalized_traces/task_22.json

A complete baseline ReAct trace with:

• 5 steps
• Tool calls (list_context, read_csv, read_json, answer)
• Observations with tool outputs
• Final answer submission
• Agent metadata

See Also

• BASELINE_TRACE_SCHEMA.md - Baseline trace format analysis
• PHASE1_EVALUATION_GUIDE.md - Evaluation system usage
• src/data_agent_baseline/evaluation/ - Implementation code