docs/architecture/BASELINE_ARCHITECTURE.md

# Baseline ReAct Agent Architecture Summary

## Execution Flow

### 1. CLI Entry Points

**Located in:** `src/data_agent_baseline/cli.py`

Two main commands:
- `dabench run-task <task_id>` - Execute single task
- `dabench run-benchmark` - Execute multiple tasks (with parallelization)

### 2. Runner Layer

**Located in:** `src/data_agent_baseline/run/runner.py`

**Key functions:**
- `run_single_task()` - Orchestrates single task execution
- `run_benchmark()` - Orchestrates benchmark execution with optional parallelization
- `_run_single_task_with_timeout()` - Subprocess-based timeout handling
- `_write_task_outputs()` - Generates artifacts

**Execution flow:**
1. Load task from DABenchPublicDataset
2. Initialize ReActAgent with model adapter and tool registry
3. Run agent (with optional timeout in subprocess)
4. Write outputs to disk

### 3. Agent Layer

**Located in:** `src/data_agent_baseline/agents/react.py`

**ReActAgent class:**
- Implements classic ReAct loop (Thought → Action → Observation)
- Maximum steps configurable (default 16)
- Uses ModelAdapter for LLM calls
- Uses ToolRegistry for tool execution

**Core method:** `run(task: PublicTask) -> AgentRunResult`

**Loop structure:**
```
for step in range(max_steps):
    1. Build message history (system + task + previous steps)
    2. Call LLM to get next step
    3. Parse JSON response (thought, action, action_input)
    4. Execute tool via ToolRegistry
    5. Record observation
    6. Check if terminal (answer submitted)
    7. Break if answer submitted
```

### 4. Runtime State Management

**Located in:** `src/data_agent_baseline/agents/runtime.py`

**Data structures:**
- `StepRecord` - Single step (thought, action, action_input, observation, ok)
- `AgentRuntimeState` - Mutable state during execution
- `AgentRunResult` - Final immutable result

### 5. Artifact Generation

**Outputs per task:**
```
<run_output_dir>/<task_id>/
├── trace.json        # Full execution trace
└── prediction.csv    # Final answer (if submitted)
```

## Baseline Trace Structure

### Baseline trace.json Format

```json
{
  "task_id": "task_22",
  "answer": {
    "columns": ["col1", "col2"],
    "rows": [["val1", "val2"], ...]
  },
  "steps": [
    {
      "step_index": 1,
      "thought": "I need to...",
      "action": "read_csv",
      "action_input": {"path": "...", "max_rows": 10},
      "raw_response": "```json\n{...}\n```",
      "observation": {
        "ok": true,
        "tool": "read_csv",
        "content": {...}
      },
      "ok": true
    }
  ],
  "failure_reason": null,
  "succeeded": true,
  "e2e_elapsed_seconds": 8.528
}
```

**Key characteristics:**
- **Simple structure:** No phases, no per-step timing
- **No metadata:** Missing difficulty, question text, context info
- **Minimal metrics:** Only e2e_elapsed_seconds, no token counts
- **No recovery tracking:** No retry/replan information
- **No confidence scores:** No self-assessment

### Current Evaluation Harness Expectations

**Located in:** `src/data_agent_baseline/langgraph_agent/eval_v2.py`

The newer LangGraph agent produces traces with extensive metadata:

```json
{
  "task_id": "task_22",
  "difficulty": "easy",
  "trace_id": "/path/to/trace",
  "answer": {...},
  "steps": [
    {
      "step_index": 1,
      "thought": "...",
      "action": "read_csv",
      "action_input": {...},
      "raw_response": "...",
      "observation": {...},
      "ok": true,
      "phase": "explore",           // ← Baseline missing
      "duration_seconds": 0.01,     // ← Baseline missing
      "token_usage": {              // ← Baseline missing
        "prompt_tokens": 100,
        "completion_tokens": 50,
        "total_tokens": 150
      }
    }
  ],
  "comprehensive_metrics": {        // ← Baseline missing
    "execution_success": 1,
    "execution_time": 8.5,
    "tool_calls": 5,
    "tool_failures": 0,
    "llm_calls": 5,
    "total_tokens": 27000,
    "stage_metrics": {...},
    "action_counts": {...},
    "per_action_metrics": {...},
    "confidence_score": 0.9,
    ...
  },
  "failure_reason": null,
  "succeeded": true,
  "e2e_elapsed_seconds": 8.528
}
```

## Gap Analysis

### Missing in Baseline Traces

| Feature | Baseline | Evaluation Harness | Impact |
|---------|----------|-------------------|--------|
| **Phase tracking** | ❌ | ✅ (explore, planner, execute, critic) | Cannot compute phase-specific metrics |
| **Per-step timing** | ❌ | ✅ duration_seconds | Cannot compute per-phase time |
| **Token usage** | ❌ | ✅ Per-step and total | Cannot compute cost metrics |
| **Comprehensive metrics** | ❌ | ✅ Full metadata object | Must compute from steps |
| **Stage metrics** | ❌ | ✅ Per-stage breakdown | Cannot compute stage efficiency |
| **Action counts** | ❌ | ✅ Tool usage frequency | Must derive from steps |
| **Confidence scores** | ❌ | ✅ Self-assessment | Cannot evaluate calibration |
| **Recovery tracking** | ❌ | ✅ Replan/retry counts | Cannot measure autonomy |
| **Task metadata** | Partial | ✅ difficulty, question | Need to fetch from task.json |
| **Trace ID** | ❌ | ✅ Full path | Must construct |

### Available in Baseline Traces

✅ **Core execution data:**
- task_id
- answer (columns, rows)
- steps (thought, action, action_input, observation, ok)
- succeeded flag
- failure_reason (if any)
- e2e_elapsed_seconds

✅ **Derivable metrics:**
- Tool call counts (from steps)
- Tool failure counts (from step.ok)
- Tool diversity (from unique actions)
- Trajectory length (from len(steps))
- Tool efficiency (failures / total calls)
- Action counts (frequency analysis)

## Evaluation Strategy

### Phase 1: Adapter-Based Approach

**Do NOT modify baseline execution code.**

Instead:
1. Create adapter layer to normalize baseline traces
2. Infer missing metadata where possible
3. Compute metrics from available data
4. Mark unavailable metrics as NULL
5. Generate comparable evaluation reports

### Adapter Responsibilities

```python
class BaselineTraceAdapter:
    def load_trace(trace_path: Path) -> dict
    def enrich_with_task_metadata(trace: dict, task_root: Path) -> dict
    def compute_derivable_metrics(trace: dict) -> dict
    def normalize_to_canonical_schema(trace: dict) -> CanonicalTrace
    def export_for_evaluation(trace: CanonicalTrace) -> dict
```

### Metrics Coverage

**Phase 1 Metrics (Baseline-Compatible):**

✅ **Accuracy metrics:**
- Overall accuracy (from gold comparison)
- Per-difficulty accuracy
- Answer precision/recall/F1

✅ **Efficiency metrics:**
- Average steps per task
- Average runtime
- Tool calls per task

✅ **Reliability metrics:**
- Success rate
- Tool error rate
- Timeout rate (if configured)

✅ **Tool usage metrics:**
- Tool diversity
- Tool efficiency
- Most common tools

❌ **Not available for baseline:**
- Token costs (no token tracking)
- Phase-specific timing (no phase labels)
- Confidence calibration (no confidence scores)
- Recovery metrics (no replan tracking)
- Stage efficiency (no stage breakdown)

### Future Compatibility

Design ensures:
- Baseline and LangGraph traces can coexist
- Same evaluation pipeline for both
- Clear indication of unavailable metrics
- Easy addition of new agent types
- Scientific comparison across systems