# Evaluation Consistency Audit (Phase-2 AAT Migration)

Date: 2026-06-14
Run validated: 20260614T044356Z

## Scope
Deep consistency audit across:
- AAT phase aggregation
- Legacy stage aggregation
- Agent resource accounting
- Confidence semantics
- Verification semantics
- Conservation checks (token/time)

## Check 1: AAT Phase Tokens vs Agent Tokens
- Severity: High
- Root cause: AAT phase token display mixed incompatible definitions (AAT-phase-only vs legacy-stage totals), so PLAN looked artificially small versus Planner + Plan Critic agent totals.
- Files: src/data_agent_baseline/langgraph_agent/eval_v2_viz.py
- Current formula (before):
  - AAT phase tokens: understanding_tokens/planning_tokens/... (from AAT observability)
  - Legacy tokens shown separately in another table
  - No reconciliation in primary phase table
- Expected formula:
  - Primary phase table should show reconciled totals from both views, with duplicate-safe merge.
- Fix:
  - Added reconciliation logic in render_per_stage_timing.
  - Displayed phase tokens now merge AAT + legacy components with duplicate detection tolerance.
- Validation:
  - PLAN now displays 70,747 tokens, reconcilable with Planner + Plan Critic + coordinator planning contribution.

## Check 2: Phase Time vs Agent Time
- Severity: High
- Root cause: One time notion was displayed without distinguishing cumulative compute vs displayed phase time, causing confusion under parallelism/concurrency.
- Files: src/data_agent_baseline/langgraph_agent/eval_v2_viz.py
- Current formula (before):
  - Single time column in AAT Execution Phases, no reconciliation table.
- Expected formula:
  - Explicitly report both displayed phase time and cumulative compute time.
- Fix:
  - Added Phase Time Reconciliation table.
  - Shows Displayed Time vs Cumulative Compute with interpretation guidance.
- Validation:
  - UNDERSTAND now clearly shows compute can exceed displayed time when work overlaps.

## Check 3: EXECUTE Confidence = 0
- Severity: Medium
- Root cause: execution_confidence was hardcoded to 0.0 when executor confidence is unavailable.
- Files: src/data_agent_baseline/langgraph_agent/eval_v2.py
- Current formula (before): execution_confidence = 0.0
- Expected formula: show unavailable as N/A, not numeric zero.
- Fix:
  - Set execution_confidence to NaN.
  - Rendering already maps NaN to N/A.
- Validation:
  - EXECUTE confidence now renders as N/A.

## Check 4: Verification Timeline Logic/Terminology
- Severity: Medium
- Root cause: "Failed Checks" wording implied hard failure even when task outcome was PASS after recovery.
- Files: src/data_agent_baseline/langgraph_agent/eval_v2_viz.py
- Fix:
  - Renamed columns:
    - Failed Checks -> Detected Issues
    - Warnings -> Tool Failures
- Validation:
  - task_11 PASS with Detected Issues=1 is now semantically aligned with recoverable/retried behavior.

## Check 5: Zero Variance Metrics (Std=0)
- Severity: High
- Root cause: research std-dev columns were derived by label-to-column string transformation; this broke for Data Understanding and other labels.
- Files: src/data_agent_baseline/langgraph_agent/eval_v2_viz.py
- Current formula (before): col_name derived from label text (brittle).
- Expected formula: explicit label->column mapping.
- Fix:
  - Replaced string transform with explicit metric mapping list.
- Validation:
  - Research output now shows Mean Data Understanding std = 0.760 (not 0.000).

## Check 6: Token Conservation Audit
- Severity: High
- Root cause: no explicit token conservation validator existed.
- Files: src/data_agent_baseline/langgraph_agent/eval_v2_validator.py, tests/test_eval_v2_validator.py
- Fix:
  - Added token_conservation_mismatch warning check.
  - Added phase_token_reconciliation_mismatch warning check.
- Validation:
  - Run now emits warnings when phase/component token totals diverge beyond threshold.

## Check 7: Time Conservation Audit
- Severity: Medium
- Root cause: no explicit time conservation checks.
- Files: src/data_agent_baseline/langgraph_agent/eval_v2_validator.py
- Fix:
  - Added time_conservation_mismatch warning check.
  - Added negative_duration error check.
- Validation:
  - Validator now checks impossible/implausible timing relationships while allowing concurrency.

## Check 8: Resource Accounting Audit
- Severity: High
- Root cause: validator lacked a direct failures <= calls invariant.
- Files: src/data_agent_baseline/langgraph_agent/eval_v2_validator.py, tests/test_eval_v2_validator.py
- Fix:
  - Added failures_exceed_calls error check across planner/critic/execute/coordinator/verifier/synthesizer/summary.
- Validation:
  - Added unit test test_validator_catches_failures_exceed_calls.

## Check 9: Legacy vs AAT Consistency
- Severity: Medium
- Root cause: no explicit mismatch checks between legacy activity and AAT phase fields.
- Files: src/data_agent_baseline/langgraph_agent/eval_v2_validator.py
- Fix:
  - Added:
    - legacy_aat_plan_mismatch
    - legacy_aat_verify_mismatch
- Validation:
  - Validator now flags zero phase values when legacy activity exists.

## Check 10: Audit Report
- Severity: Informational
- Output: this file documents root causes, formulas, fixes, and validation outcomes.

## Added/Updated Validation Rules
New validator checks introduced:
- negative_duration (error)
- failures_exceed_calls (error)
- token_conservation_mismatch (warning)
- time_conservation_mismatch (warning)
- legacy_aat_plan_mismatch (error)
- legacy_aat_verify_mismatch (error)
- phase_token_reconciliation_mismatch (warning)

## Unit Test Coverage
- Existing tests retained and passing.
- Added:
  - test_validator_catches_failures_exceed_calls
  - test_validator_catches_phase_token_reconciliation_mismatch

Test run:
- 10 passed in tests/test_eval_v2_validator.py + tests/test_eval_v2_aat_observability.py

## Real Run Validation
Command:
- dabench eval-v2 20260614T044356Z --mode verbose

Observed:
- No hard consistency errors.
- 3 warnings for phase_token_reconciliation_mismatch (expected under mixed legacy/AAT trace sources).
- Phase table now reconciles PLAN/VERIFY token magnitudes with agent usage.
- EXECUTE confidence shown as N/A.
- Verification timeline terminology aligned with behavior.

## Notes on Remaining Warnings
Current warnings indicate residual divergence between phase-level and component-level token accounting under mixed trace sources. This is now surfaced explicitly (instead of silently misreporting), and is suitable for follow-up instrumentation tightening at trace source level.