docs/architecture/BASELINE_ARCHITECTURE.md

Baseline ReAct Agent Architecture Summary

Execution Flow

1. CLI Entry Points

Located in: src/data_agent_baseline/cli.py

Two main commands:

• dabench run-task <task_id> - Execute single task
• dabench run-benchmark - Execute multiple tasks (with parallelization)

2. Runner Layer

Located in: src/data_agent_baseline/run/runner.py

Key functions:

• run_single_task() - Orchestrates single task execution
• run_benchmark() - Orchestrates benchmark execution with optional parallelization
• _run_single_task_with_timeout() - Subprocess-based timeout handling
• _write_task_outputs() - Generates artifacts

Execution flow:

1. Load task from DABenchPublicDataset
2. Initialize ReActAgent with model adapter and tool registry
3. Run agent (with optional timeout in subprocess)
4. Write outputs to disk

3. Agent Layer

Located in: src/data_agent_baseline/agents/react.py

ReActAgent class:

• Implements classic ReAct loop (Thought → Action → Observation)
• Maximum steps configurable (default 16)
• Uses ModelAdapter for LLM calls
• Uses ToolRegistry for tool execution

Core method: run(task: PublicTask) -> AgentRunResult

Loop structure:

for step in range(max_steps):
    1. Build message history (system + task + previous steps)
    2. Call LLM to get next step
    3. Parse JSON response (thought, action, action_input)
    4. Execute tool via ToolRegistry
    5. Record observation
    6. Check if terminal (answer submitted)
    7. Break if answer submitted

4. Runtime State Management

Located in: src/data_agent_baseline/agents/runtime.py

Data structures:

• StepRecord - Single step (thought, action, action_input, observation, ok)
• AgentRuntimeState - Mutable state during execution
• AgentRunResult - Final immutable result

5. Artifact Generation

Outputs per task:

<run_output_dir>/<task_id>/
├── trace.json        # Full execution trace
└── prediction.csv    # Final answer (if submitted)

Baseline Trace Structure

Baseline trace.json Format

{
  "task_id": "task_22",
  "answer": {
    "columns": ["col1", "col2"],
    "rows": [["val1", "val2"], ...]
  },
  "steps": [
    {
      "step_index": 1,
      "thought": "I need to...",
      "action": "read_csv",
      "action_input": {"path": "...", "max_rows": 10},
      "raw_response": "```json\n{...}\n```",
      "observation": {
        "ok": true,
        "tool": "read_csv",
        "content": {...}
      },
      "ok": true
    }
  ],
  "failure_reason": null,
  "succeeded": true,
  "e2e_elapsed_seconds": 8.528
}

Key characteristics:

• Simple structure: No phases, no per-step timing
• No metadata: Missing difficulty, question text, context info
• Minimal metrics: Only e2e_elapsed_seconds, no token counts
• No recovery tracking: No retry/replan information
• No confidence scores: No self-assessment

Current Evaluation Harness Expectations

Located in: src/data_agent_baseline/langgraph_agent/eval_v2.py

The newer LangGraph agent produces traces with extensive metadata:

{
  "task_id": "task_22",
  "difficulty": "easy",
  "trace_id": "/path/to/trace",
  "answer": {...},
  "steps": [
    {
      "step_index": 1,
      "thought": "...",
      "action": "read_csv",
      "action_input": {...},
      "raw_response": "...",
      "observation": {...},
      "ok": true,
      "phase": "explore",           // ← Baseline missing
      "duration_seconds": 0.01,     // ← Baseline missing
      "token_usage": {              // ← Baseline missing
        "prompt_tokens": 100,
        "completion_tokens": 50,
        "total_tokens": 150
      }
    }
  ],
  "comprehensive_metrics": {        // ← Baseline missing
    "execution_success": 1,
    "execution_time": 8.5,
    "tool_calls": 5,
    "tool_failures": 0,
    "llm_calls": 5,
    "total_tokens": 27000,
    "stage_metrics": {...},
    "action_counts": {...},
    "per_action_metrics": {...},
    "confidence_score": 0.9,
    ...
  },
  "failure_reason": null,
  "succeeded": true,
  "e2e_elapsed_seconds": 8.528
}

Gap Analysis

Missing in Baseline Traces

Feature	Baseline	Evaluation Harness	Impact
Phase tracking	❌	✅ (explore, planner, execute, critic)	Cannot compute phase-specific metrics
Per-step timing	❌	✅ duration_seconds	Cannot compute per-phase time
Token usage	❌	✅ Per-step and total	Cannot compute cost metrics
Comprehensive metrics	❌	✅ Full metadata object	Must compute from steps
Stage metrics	❌	✅ Per-stage breakdown	Cannot compute stage efficiency
Action counts	❌	✅ Tool usage frequency	Must derive from steps
Confidence scores	❌	✅ Self-assessment	Cannot evaluate calibration
Recovery tracking	❌	✅ Replan/retry counts	Cannot measure autonomy
Task metadata	Partial	✅ difficulty, question	Need to fetch from task.json
Trace ID	❌	✅ Full path	Must construct

Available in Baseline Traces

✅ Core execution data:

• task_id
• answer (columns, rows)
• steps (thought, action, action_input, observation, ok)
• succeeded flag
• failure_reason (if any)
• e2e_elapsed_seconds

✅ Derivable metrics:

• Tool call counts (from steps)
• Tool failure counts (from step.ok)
• Tool diversity (from unique actions)
• Trajectory length (from len(steps))
• Tool efficiency (failures / total calls)
• Action counts (frequency analysis)

Evaluation Strategy

Phase 1: Adapter-Based Approach

Do NOT modify baseline execution code.

Instead:

1. Create adapter layer to normalize baseline traces
2. Infer missing metadata where possible
3. Compute metrics from available data
4. Mark unavailable metrics as NULL
5. Generate comparable evaluation reports

Adapter Responsibilities

class BaselineTraceAdapter:
    def load_trace(trace_path: Path) -> dict
    def enrich_with_task_metadata(trace: dict, task_root: Path) -> dict
    def compute_derivable_metrics(trace: dict) -> dict
    def normalize_to_canonical_schema(trace: dict) -> CanonicalTrace
    def export_for_evaluation(trace: CanonicalTrace) -> dict

Metrics Coverage

Phase 1 Metrics (Baseline-Compatible):

✅ Accuracy metrics:

• Overall accuracy (from gold comparison)
• Per-difficulty accuracy
• Answer precision/recall/F1

✅ Efficiency metrics:

• Average steps per task
• Average runtime
• Tool calls per task

✅ Reliability metrics:

• Success rate
• Tool error rate
• Timeout rate (if configured)

✅ Tool usage metrics:

• Tool diversity
• Tool efficiency
• Most common tools

❌ Not available for baseline:

• Token costs (no token tracking)
• Phase-specific timing (no phase labels)
• Confidence calibration (no confidence scores)
• Recovery metrics (no replan tracking)
• Stage efficiency (no stage breakdown)

Future Compatibility

Design ensures:

• Baseline and LangGraph traces can coexist
• Same evaluation pipeline for both
• Clear indication of unavailable metrics
• Easy addition of new agent types
• Scientific comparison across systems