# Canonical Normalized Trace Format

## Overview

The canonical normalized trace format provides a unified representation for execution traces across different agent architectures. This enables:

1. **Agent-agnostic evaluation** - Same metrics for all agent types
2. **Trace replay** - Reconstruct execution from normalized traces
3. **Failure analysis** - Analyze errors across different systems
4. **Visualization** - DAG visualization and timeline views
5. **Comparison** - Fair comparison between agent architectures

## Schema Version

**Version:** 1.0  
**Date:** 2026-06-13

## Canonical Trace Schema

```json
{
  "run_id": "string",
  "task_id": "string",
  "agent_type": "baseline_react | langgraph_agent | other",
  
  "question": "string",
  "difficulty": "Easy | Medium | Hard | Extreme | Unknown",
  
  "success": boolean,
  "final_answer": {
    "columns": ["string"],
    "rows": [["any"]]
  } | null,
  "failure_reason": "string" | null,
  
  "start_time": "ISO8601 string" | null,
  "end_time": "ISO8601 string" | null,
  "duration_seconds": number | null,
  
  "steps": [
    {
      "step_id": integer,
      
      "agent": "string",           // Agent name: "baseline_react", "planner", "executor"
      "agent_role": "string",      // Role: "worker", "planner", "critic", "coordinator"
      
      "thought": "string",
      "action": "string",
      "action_input": {},
      "observation": {},
      "tool_success": boolean,
      
      "raw_response": "string" | null,
      "timestamp": "ISO8601 string" | null,
      "phase": "string" | null,
      "duration_seconds": number | null,
      "token_usage": {
        "prompt_tokens": integer,
        "completion_tokens": integer,
        "total_tokens": integer
      } | null
    }
  ],
  
  "metrics": {
    "num_steps": integer,
    "tool_calls": integer,
    "tool_failures": integer,
    "unique_tools": integer,
    ...
  } | null,
  
  "trace_path": "string" | null,
  "prediction_path": "string" | null,
  "context_dir": "string" | null,
  "task_root": "string" | null
}
```

## Field Descriptions

### Top-Level Fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `run_id` | string | ✅ | Unique run identifier |
| `task_id` | string | ✅ | Task identifier |
| `agent_type` | string | ✅ | Agent architecture type |
| `question` | string | ✅ | Task question |
| `difficulty` | string | ❌ | Task difficulty level |
| `success` | boolean | ✅ | Whether task completed successfully |
| `final_answer` | object | ❌ | Final answer (if submitted) |
| `failure_reason` | string | ❌ | Error message if failed |
| `start_time` | string | ❌ | ISO8601 timestamp |
| `end_time` | string | ❌ | ISO8601 timestamp |
| `duration_seconds` | number | ❌ | Total execution time |

### Step Fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `step_id` | integer | ✅ | Sequential step number (1-indexed) |
| `agent` | string | ✅ | Agent name that produced this step |
| `agent_role` | string | ✅ | Agent role (planner, worker, critic, coordinator) |
| `thought` | string | ✅ | Agent's reasoning |
| `action` | string | ✅ | Tool/action name |
| `action_input` | object | ✅ | Tool parameters |
| `observation` | object | ✅ | Tool output |
| `tool_success` | boolean | ✅ | Whether tool succeeded |
| `raw_response` | string | ❌ | Raw LLM output |
| `timestamp` | string | ❌ | Step timestamp |
| `phase` | string | ❌ | Execution phase |
| `duration_seconds` | number | ❌ | Step duration |
| `token_usage` | object | ❌ | LLM token usage |

## Multi-Agent Support

The schema supports multiple agent architectures:

### Baseline ReAct (Single Agent)

```json
{
  "agent": "baseline_react",
  "agent_role": "worker"
}
```

All steps have the same agent.

### Planner + Executor

```json
// Planning step
{
  "agent": "planner",
  "agent_role": "planner",
  "phase": "planning"
}

// Execution step
{
  "agent": "executor",
  "agent_role": "worker",
  "phase": "execution"
}
```

### Multi-Agent Analyst Team

```json
// Data exploration
{
  "agent": "scout",
  "agent_role": "worker",
  "phase": "explore"
}

// Analysis
{
  "agent": "analyst",
  "agent_role": "worker",
  "phase": "analyze"
}

// Verification
{
  "agent": "critic",
  "agent_role": "critic",
  "phase": "verify"
}
```

### Coordinator-Based System

```json
// Coordination
{
  "agent": "coordinator",
  "agent_role": "coordinator",
  "phase": "coordination"
}

// Worker execution
{
  "agent": "worker_1",
  "agent_role": "worker",
  "phase": "execution"
}
```

## Agent Roles

Standard agent roles:

| Role | Description | Examples |
|------|-------------|----------|
| `worker` | Executes tasks and tools | Baseline ReAct, Executor, Analyst |
| `planner` | Creates execution plans | Planner agent |
| `critic` | Reviews and validates | Critic, Verifier |
| `coordinator` | Orchestrates multiple agents | Multi-agent coordinator |

## Phases

Common execution phases:

| Phase | Description | Typical Agent Roles |
|-------|-------------|---------------------|
| `explore` | Data discovery and exploration | worker |
| `plan` | Planning and strategizing | planner |
| `execute` | Tool execution and data processing | worker |
| `verify` | Validation and verification | critic |
| `refine` | Correction and refinement | worker, critic |
| `coordinate` | Multi-agent coordination | coordinator |

## Usage

### Generate Normalized Traces

```bash
dabench eval-baseline <run_id>
```

This generates:
```
baseline_evaluation/
├── task_results.csv
├── summary_metrics.json
├── evaluation_report.md
└── normalized_traces/
    ├── task_001.json
    ├── task_002.json
    └── ...
```

### Load and Analyze

```python
from data_agent_baseline.evaluation.normalized_trace_manager import NormalizedTraceManager

# Load traces
manager = NormalizedTraceManager(output_dir)
trace = manager.load_normalized_trace("task_22")

# Get metrics
metrics = manager.get_trace_metrics(trace)
print(f"Steps: {metrics['num_steps']}")
print(f"Tool calls: {metrics['num_tool_calls']}")
print(f"Agent breakdown: {metrics['agent_steps']}")

# Validate
is_valid, errors = manager.validate_normalized_trace(trace)
if not is_valid:
    print(f"Validation errors: {errors}")
```

### Validate All Traces

```python
manager = NormalizedTraceManager(output_dir)
results = manager.validate_all_traces()

for task_id, (is_valid, errors) in results.items():
    if not is_valid:
        print(f"{task_id}: {errors}")
```

## Benefits

### 1. Agent-Agnostic Evaluation

Same metrics work across all agent types:
- Baseline ReAct
- Multi-agent systems
- Future architectures

### 2. Easy Comparison

```python
baseline_trace = manager.load_normalized_trace("task_22")
multi_agent_trace = manager.load_normalized_trace("task_22", run_id="multi_agent_run")

baseline_metrics = manager.get_trace_metrics(baseline_trace)
multi_agent_metrics = manager.get_trace_metrics(multi_agent_trace)

print(f"Baseline steps: {baseline_metrics['num_steps']}")
print(f"Multi-agent steps: {multi_agent_metrics['num_steps']}")
```

### 3. Trace Replay

Reconstruct execution from normalized trace for:
- Debugging
- Visualization
- Failure analysis

### 4. Future-Proof

Schema supports:
- New agent architectures
- New phases
- New metrics
- Parallel execution
- Multi-agent coordination

## Validation

The `NormalizedTraceManager` validates:

✅ **Required fields:**
- run_id, task_id, agent_type, question, success
- step_id, agent, agent_role, thought, action, etc.

✅ **Step ordering:**
- step_id must be sequential (1, 2, 3, ...)

✅ **Type checking:**
- Fields have correct types (string, int, bool, dict)

✅ **Schema consistency:**
- final_answer has columns and rows
- observation is a dictionary
- action_input is a dictionary

## Migration Guide

### From Baseline Traces

Baseline traces are automatically converted to normalized format:

```python
from data_agent_baseline.evaluation.baseline_adapter import BaselineTraceAdapter

adapter = BaselineTraceAdapter(task_root)
canonical = adapter.normalize(trace_path, run_id)
```

### From Custom Agents

Implement an adapter:

```python
class CustomAgentAdapter:
    def normalize(self, raw_trace, run_id):
        return CanonicalTrace(
            run_id=run_id,
            task_id=raw_trace["task_id"],
            agent_type="custom_agent",
            question=raw_trace["question"],
            success=raw_trace["success"],
            steps=[
                CanonicalStep(
                    step_id=i,
                    agent="custom_agent",
                    agent_role="worker",
                    ...
                )
                for i, step in enumerate(raw_trace["steps"], 1)
            ],
            ...
        )
```

## Example Trace

See: `/data3/dataFAIR/kdd-dev/public/artifacts/runs/20260613T114457Z/baseline_evaluation/normalized_traces/task_22.json`

A complete baseline ReAct trace with:
- 5 steps
- Tool calls (list_context, read_csv, read_json, answer)
- Observations with tool outputs
- Final answer submission
- Agent metadata

## See Also

- **BASELINE_TRACE_SCHEMA.md** - Baseline trace format analysis
- **PHASE1_EVALUATION_GUIDE.md** - Evaluation system usage
- **src/data_agent_baseline/evaluation/** - Implementation code