docs/architecture/CANONICAL_TRACE_QUICKREF.md

# Canonical Normalized Trace - Quick Reference

## Quick Start

### Generate Normalized Traces

```bash
dabench eval-baseline <run_id>
```

Output: `baseline_evaluation/normalized_traces/task_*.json`

### View a Trace

```bash
dabench view-normalized-trace <run_id> <task_id>
```

Shows detailed breakdown with validation and metrics.

### Load Programmatically

```python
from data_agent_baseline.evaluation.normalized_trace_manager import NormalizedTraceManager

manager = NormalizedTraceManager(output_dir)
trace = manager.load_normalized_trace("task_22")
```

## Common Tasks

### List Available Traces

```python
task_ids = manager.list_normalized_traces()
print(f"Available: {', '.join(task_ids)}")
```

### Validate a Trace

```python
is_valid, errors = manager.validate_normalized_trace(trace)
if not is_valid:
    print(f"Errors: {errors}")
```

### Get Metrics

```python
metrics = manager.get_trace_metrics(trace)
print(f"Steps: {metrics['num_steps']}")
print(f"Tools: {metrics['tool_counts']}")
```

### Compare Agents

```python
baseline_trace = manager.load_normalized_trace("task_22")
multi_agent_trace = ...  # Load from different run

baseline_metrics = manager.get_trace_metrics(baseline_trace)
multi_agent_metrics = manager.get_trace_metrics(multi_agent_trace)

print(f"Baseline: {baseline_metrics['num_steps']} steps")
print(f"Multi-agent: {multi_agent_metrics['num_steps']} steps")
```

## Schema at a Glance

```json
{
  "run_id": "string",
  "task_id": "string",
  "agent_type": "baseline_react",
  "question": "string",
  "difficulty": "Easy",
  "success": true,
  "final_answer": {"columns": [...], "rows": [...]},
  "duration_seconds": 8.5,
  "steps": [
    {
      "step_id": 1,
      "agent": "baseline_react",
      "agent_role": "worker",
      "thought": "...",
      "action": "list_context",
      "action_input": {...},
      "observation": {...},
      "tool_success": true
    }
  ]
}
```

## CLI Commands

| Command | Description |
|---------|-------------|
| `eval-baseline <run_id>` | Generate normalized traces and evaluation |
| `view-normalized-trace <run_id> <task_id>` | View detailed trace breakdown |

## Key Metrics

From `get_trace_metrics()`:

| Metric | Description |
|--------|-------------|
| `num_steps` | Total execution steps |
| `num_tool_calls` | Total tool invocations |
| `num_failed_tools` | Failed tool calls |
| `unique_tools` | Distinct tools used |
| `agent_steps` | Steps per agent (multi-agent) |
| `role_steps` | Steps per role (multi-agent) |
| `tool_counts` | Frequency per tool |
| `phase_steps` | Steps per phase (if available) |

## Multi-Agent Support

### Baseline ReAct (Single Agent)
```json
{"agent": "baseline_react", "agent_role": "worker"}
```

### Planner + Executor
```json
{"agent": "planner", "agent_role": "planner"}
{"agent": "executor", "agent_role": "worker"}
```

### Multi-Agent Team
```json
{"agent": "scout", "agent_role": "worker", "phase": "explore"}
{"agent": "analyst", "agent_role": "worker", "phase": "analyze"}
{"agent": "critic", "agent_role": "critic", "phase": "verify"}
```

## Validation

Built-in validation checks:

- ✅ Required fields present
- ✅ Step ordering sequential
- ✅ Type correctness
- ✅ Schema consistency

```python
# Validate single trace
is_valid, errors = manager.validate_normalized_trace(trace)

# Validate all traces
results = manager.validate_all_traces()
for task_id, (is_valid, errors) in results.items():
    if not is_valid:
        print(f"{task_id}: {errors}")
```

## File Locations

```
baseline_evaluation/
├── task_results.csv
├── summary_metrics.json
├── evaluation_report.md
└── normalized_traces/        # ← One file per task
    ├── task_001.json
    ├── task_002.json
    └── task_022.json
```

## Documentation

- **[CANONICAL_TRACE_FORMAT.md](CANONICAL_TRACE_FORMAT.md)** - Complete schema documentation
- **[CANONICAL_TRACE_IMPLEMENTATION.md](CANONICAL_TRACE_IMPLEMENTATION.md)** - Implementation summary
- **[BASELINE_TRACE_SCHEMA.md](BASELINE_TRACE_SCHEMA.md)** - Baseline trace analysis

## Example

```bash
# Generate traces
dabench eval-baseline 20260613T114457Z

# View trace
dabench view-normalized-trace 20260613T114457Z task_22

# Programmatic access
python3 << EOF
from pathlib import Path
from data_agent_baseline.evaluation.normalized_trace_manager import NormalizedTraceManager

manager = NormalizedTraceManager(
    Path("/data3/dataFAIR/kdd-dev/public/artifacts/runs/20260613T114457Z/baseline_evaluation")
)

trace = manager.load_normalized_trace("task_22")
print(f"Task: {trace['task_id']}")
print(f"Success: {trace['success']}")
print(f"Steps: {len(trace['steps'])}")

metrics = manager.get_trace_metrics(trace)
print(f"Metrics: {metrics}")
EOF
```

## Tips

1. **One file per task** makes it easy to find and analyze individual traces
2. **Validation on save** ensures schema compliance
3. **Derived metrics** computed on-demand from trace data
4. **Multi-agent ready** - just set different agent/role per step
5. **NULL-safe** - missing data represented as None, not omitted

## See Also

- `src/data_agent_baseline/evaluation/` - Implementation code
- Example: `/data3/dataFAIR/kdd-dev/public/artifacts/runs/20260613T114457Z/baseline_evaluation/normalized_traces/task_22.json`