# Recovery Diff Audit
Generated: 2026-06-16
Scope: Analyst Team, Coordinator Synthesis, Disagreement Resolution, Auditor, Reporting Cleanup

---

## 1. Git History Summary

Repository started from the KDD Cup 2026 starter kit and evolved through multiple
feature branches. ORIG_HEAD: 27e538bb (most recent amend was "Update summary
rendering to include lambda penalty and recall metrics"). The AAT / analyst-team
work cannot be individually bisected because no known-good commit was tagged, but
the code itself carries its own rollback instructions (see §2 below).

Pre-existing changes with explicit rollback comments in graph.py (NOT in scope):
- `voted_planner_node` — "CHANGE I — Multi-candidate plan voting"
- `apply_post_hoc_guards`  — "Strategy 5 — Post-hoc deterministic output guards"

---

## 2. File Classification

### Category A — Reporting Only
Changes that only affect eval metrics, summaries, or console output.
No execution-path impact verified.

| File | Purpose | Execution-Affecting | Recommendation |
|------|---------|---------------------|----------------|
| `src/.../eval_v2.py` | Added `_extract_analyst_team_metrics()`, `write_auditor_validation_report()`, summary aggregation extensions for analyst team fields | NO | KEEP — all read-only from trace; meaningful-disagreement counting needs tightening (Phase 3) |
| `src/.../eval_v2_schema.py` | Added 33 analyst team TaskMetrics fields | NO | KEEP — pure schema definition |
| `src/.../eval_v2_viz.py` | Added `render_analyst_team_summary()`, `render_auditor_diagnostics()`, per-task column extensions | NO | KEEP — display only |
| `src/.../cli.py` | Added `write_auditor_validation_report()` call and import | NO | KEEP — reporting orchestration only |
| `src/.../aat/disagreement.py` | NEW: detect_disagreements() — all items have `actionable=False`, purely reporting-oriented | NO | KEEP — already correct |
| `src/.../aat/schema.py` (new types) | Added AnalystOpinion, DisagreementItem, CoordinatorSynthesis, audit result types | NO | KEEP — passive data containers |
| `EVALUATION.md` | Updated documentation for analyst team metrics | NO | KEEP — doc only |

### Category B — Trace Only
Changes that persist new data to trace.json and state but are not consumed for routing.

| File | Purpose | Execution-Affecting | Recommendation |
|------|---------|---------------------|----------------|
| `src/.../state.py` | Added `analyst_opinions`, `disagreements`, `coordinator_synthesis` fields to AgentState | NO | KEEP — additive fields, never read for routing |
| `src/.../runner.py` | Added `aat_analyst_opinions`, `aat_disagreements`, `aat_coordinator_synthesis` to trace.json serialization | NO | KEEP — write-only serialization |
| `src/.../aat/__init__.py` | Extended exports for new schema types and modules | NO | KEEP — imports only |
| `src/.../aat_observability.py` | Phase/agent name mapping utilities for trace annotation | NO | KEEP — utility only |

### Category C — Execution-Affecting
Changes that alter the runtime call graph, LLM call count, context fed to planner, or
execution status logic.

| File | Purpose | Execution-Affecting | Mechanism | Recommendation |
|------|---------|---------------------|-----------|----------------|
| `src/.../graph.py` (CP1 section, ~lines 630–695) | Coordinator `checkpoint_understanding()` gates which specialist agents run via `specialists_requested` list | **YES** | Adds 1 LLM call before planning; conditions SchemaAgent/DomainAgent execution on coordinator judgment rather than deterministic baseline | **REVIEW** — verify coordinator reliably always requests schema+domain; if not, add fallback guarantee |
| `src/.../graph.py` (`coordinator_approved` variable, ~lines 1088–1098) | Computes `coordinator_approved` from `final_review.decision` | **LATENT** | Variable is computed but **never used** in downstream routing; dead code that misleads readers | **REMOVE** dead variable or add a comment making inert status explicit |
| `src/.../graph.py` (added LLM calls pre-plan) | coordinator CP1 + AnalysisSynthesizer each add 1 LLM round-trip before planner | **YES** (latency/token) | Adds ~2 extra LLM calls to every task before planning begins | **ACCEPTABLE** if specialist quality improves planner; latency risk documented |
| `src/.../graph.py` (added LLM calls post-execute) | hypothesis_agent, coordinator.to_opinion(), FinalSummaryGenerator each add 1 LLM call after execution | **YES** (latency/token) | 3 extra LLM calls per task post-execution; these do NOT feed back into execution decisions | **ACCEPTABLE** — post-execution, no routing impact; latency risk documented |
| `src/.../aat/coordinator.py` (`checkpoint_understanding`) | Returns `specialists_requested` list that graph.py respects | **YES** | If coordinator returns narrow specialist list (e.g. only ["schema"]), domain context is missing from planner | **REVIEW** — add explicit fallback to always include ["schema", "domain"] |
| `src/.../aat/coordinator.py` (`checkpoint_planning`) | Returns `PlanningReview` with `needs_replan`, `needs_retry` fields | **NO** (stabilized) | Already has explicit STABILIZATION comment in graph.py: "not enforced" | **KEEP** — already stabilized |
| `src/.../aat/coordinator.py` (`checkpoint_final`) | Returns `FinalReview` with `can_release`, `needs_retry`, `needs_replan` | **NO** (dead path) | Stored in state but `coordinator_approved` variable computed from it is never used for routing | **CLARIFY** — add comment confirming inert status; remove dead `coordinator_approved` var |

### Category D — Unknown / Boundary
Files with execution-control-looking patterns that are confirmed reporting-only in
practice but warrant documentation.

| File | Purpose | Execution-Affecting | Notes |
|------|---------|---------------------|-------|
| `src/.../aat/guards.py` | Filter/Schema/Aggregation audit returning `suggested_action` (PROCEED/RETRY/REPLAN) | **NO** in practice | `suggested_action` values exist in audit result structs but are **not consumed by graph.py routing**; propagated into AnalystOpinion metadata as advisory only. Risk: future code change could accidentally wire them. |
| `src/.../aat/hypothesis.py` | Post-execution analyst returning `required_action` opinion | **NO** in practice | Runs AFTER execution loop (graph.py ~line 987); opinion added to `analyst_opinions` list which is stored for trace. `required_action` field looks like a control signal but has no consumer routing on it. Risk: field name is misleading. |
| `src/.../aat/verifier.py` (`to_opinion`) | Post-verifier opinion conversion returning `required_action = RETRY_EXECUTION` when not passed | **NO** in practice | `to_opinion()` runs after execution completes; opinion joins `analyst_opinions` list for trace. Verifier's direct `VerificationReport.recommendation` is what coordinator CP3 reads — not this opinion. |
| `src/.../aat/specialist_agents.py` (`to_opinion` methods) | Each specialist converts its analysis to `AnalystOpinion` with `required_action` | **NO** in practice | Called after analysis runs; opinions assembled post-synthesis for trace. `required_action` on these opinions is never consumed for execution branching. |

---

## 3. Execution-Affecting Hooks — Detailed Findings

### Hook 1: CP1 Specialist Gating (HIGH PRIORITY)
**File**: `src/data_agent_baseline/langgraph_agent/graph.py`, ~lines 630–660
**Mechanism**: `specialists_requested = understanding_review.specialists_requested if understanding_review else ["schema", "domain"]`
**Risk**: If coordinator omits "domain" from specialists_requested, the DomainAgent LLM call is skipped. The planner then receives no domain analysis (business rules, metric definitions, ambiguities). This could degrade answer quality on metric-heavy questions.
**Current fallback**: Falls back to `["schema", "domain"]` only if `understanding_review` is None (exception path). A successful coordinator call that returns a narrow list (e.g., `["schema"]`) will skip domain analysis with no fallback.
**Execution-affecting**: YES
**Recommendation**: Add explicit post-CP1 guarantee: always include "schema" and "domain" regardless of coordinator response. Only "document" should be conditionally gated.

### Hook 2: Dead `coordinator_approved` Variable (LOW PRIORITY, CLARITY)
**File**: `src/data_agent_baseline/langgraph_agent/graph.py`, ~lines 1088–1098
**Mechanism**:
```python
coordinator_approved = True
if final_review:
    coordinator_approved = final_review.decision in (
        CoordinatorDecision.APPROVE_FINAL.value,
        CoordinatorDecision.PROCEED.value,
    )
# ... (coordinator_approved never appears again; status check uses execution_result directly)

Risk: Currently inert — status is determined by state.execution_result non-emptiness, not by this variable. Risk is that a future developer adds a conditional on coordinator_approved believing it is intentional.
Execution-affecting: NO (latent risk only)
Recommendation: Add comment: # coordinator_approved: informational only — execution status determined by result presence below.

Hook 3: CP2 Planning Review (ALREADY STABILIZED)
File: src/data_agent_baseline/langgraph_agent/graph.py, ~line 817
Status: Explicit STABILIZATION: comment present: "Coordinator decision is recorded for synthesis/reporting but NOT enforced."
Execution-affecting: NO
Recommendation: No change needed. Keep stabilization comment.

Hook 4: CP3 Final Review (ALREADY INERT)
File: src/data_agent_baseline/langgraph_agent/graph.py, ~line 956
Status: FinalReview fields stored in state, fed into coordinator opinion, but CP3 result does not gate execution return path (see Hook 2).
Execution-affecting: NO
Recommendation: No change needed. Document explicitly via Hook 2 comment fix.

4. Disagreement / Auditor / Coordinator Control Flow Verification
All 14 terms from the Phase 2 search specification were traced:

Term	File	Execution-affecting	Verdict
disagreement	disagreement.py, state.py, eval_v2.py	NO	Reporting only; actionable=False
meaningful_disagreement	eval_v2.py:377	NO	Eval metric; counting logic needs Phase 3 fix
actionable_disagreement	eval_v2_schema.py:450	NO	Metric field; always 0 in current code
agreement_score	schema.py:480, eval_v2.py:367	NO	Computed for synthesis; not consumed for routing
coordinator_action	schema.py:486, eval_v2.py:396	NO	Reporting field derived from final_review
coordinator_synthesis	state.py:162, graph.py:1044	NO	Stored post-execution; not consumed for routing
selected_action	schema.py:483	NO	Mirrors final action; no routing consumer
high_risk_approval	schema.py:447,488, eval_v2.py:420	NO	Always False; metric only
filter_risk	graph.py:1005, coordinator.py:232	NO	Boolean flag; advisory metadata on opinions
schema_risk	graph.py:1005, coordinator.py:235	NO	Boolean flag; advisory metadata on opinions
aggregation_risk	graph.py:1005,1037, guards.py:335	NO	Boolean flag; advisory metadata; not consumed for routing
auditor_result	eval_v2.py:431–436	NO	Reporting metric; aggregates guard trigger counts
auditor_warning	eval_v2_schema.py:472, eval_v2.py:438	NO	Eval flag; no execution consumer
coordinator_approved	graph.py:1088–1098	LATENT	Dead variable — see Hook 2 above
Verdict: No disagreement, auditor, or coordinator synthesis signal currently alters retries, replans, routing, or final approval. The sole active execution impact is CP1 specialist gating (Hook 1).

5. Pre-Existing Execution Changes (Out of Scope for Recovery)
These were added before the analyst-team iterations and have their own documented rollback paths in graph.py imports:

Change	Rollback Path
voted_planner_node (multi-candidate plan voting)	Replace with planner_node in graph.py call site; remove run_zero_row_diagnostic call; delete vote_plan.py
apply_post_hoc_guards (post-hoc output guards)	Remove import and state = apply_post_hoc_guards(state) call in graph.py
6. Phase 2–3 Work Items (Ordered by Priority)
Required for Execution Stabilization
CP1 fallback guarantee — In graph.py, after coordinator.checkpoint_understanding(),
ensure specialists_requested always contains at minimum ["schema", "domain"].
Only "document" should be conditionally gated by coordinator judgment.

Remove/comment coordinator_approved dead code — Add clarity comment or remove the variable
to prevent future accidental wiring.

Required for Reporting Cleanup (Phase 3)
Meaningful disagreement counting — In eval_v2.py _extract_analyst_team_metrics(), the
meaningful_disagreements count is already correctly narrowed (HIGH severity + "interpretation conflict"
topic only). Verify: if critical_disagreement_score == 0, analyst_team_tasks_meaningful_disagreement
must not count that task. Current logic: (df["critical_disagreement_score"] > 0).sum() — already correct.

analyst_team_disagreement_count vs meaningful_disagreements aliasing — Both columns point
to the same meaningful_disagreements value. Consider whether analyst_team_disagreement_count
should show raw disagreement count or meaningful count (currently: meaningful). Document the choice.

No Change Required
All eval_v2*, state.py, runner.py, aat_observability.py, cli.py, disagreement.py — preserve as-is.
CP2, CP3, verifier.to_opinion(), hypothesis.py, specialist to_opinion() — already inert for execution.
guards.py suggested_action field — exists in structs but is not consumed for routing. Preserve as diagnostic advisory.
7. Before Baseline Reference
Latest available eval run for before/after comparison:
/data3/dataFAIR/kdd-dev/public/artifacts/runs/20260616T043015Z

Secondary reference (used in calibration notes):
/data3/dataFAIR/kdd-dev/public/artifacts/runs/20260615T160130Z

---

**Summary of Phase 1 findings:**

| Category | Count | Action |
|----------|-------|--------|
| Reporting-only (Category A) | 7 files | Preserve |
| Trace-only (Category B) | 4 files | Preserve |
| Execution-affecting — active (Category C) | 2 hooks | Fix in Phase 2 |
| Execution-affecting — dead code (Category C) | 1 hook | Clarify comment |
| Boundary/advisory (Category D) | 4 files | Document, no change |

**Only two Phase 2 changes are warranted:**
1. Add a ["schema", "domain"] minimum fallback guarantee at CP1 in src/data_agent_baseline/langgraph_agent/graph.py#L650 so coordinator judgment can never omit a core specialist.  
2. Comment out or annotate the dead coordinator_approved variable at src/data_agent_baseline/langgraph_agent/graph.py#L1088 so its inert status is explicit.

Everything else — disagreement, auditor, guards, coordinator synthesis, analyst opinions — is already diagnostics-only and needs no execution change. Phase 3 reporting fixes (meaningful disagreement threshold, auditor research-mode gating) can proceed independently once Phase 2 is locked.

## Selective Stabilization Changes Implemented

Date: 2026-06-16

Files modified:
- src/data_agent_baseline/langgraph_agent/graph.py
- recovery_diff_audit.md

### Change 1: CP1 Specialist Fallback Guarantee

File: src/data_agent_baseline/langgraph_agent/graph.py

Applied after CP1 coordinator response normalization:

```python
# Stabilization:
# Schema and Domain are mandatory core analysts.
# Coordinator may optionally request Document analysis.
requested = {str(s).lower() for s in specialists_requested}
requested.add("schema")
requested.add("domain")
specialists_requested = list(requested)
```

Effect:
- SchemaAgent always runs.
- DomainAgent always runs.
- DocumentAgent remains conditional on coordinator request.

### Change 2: Dead coordinator_approved Variable Removal

File: src/data_agent_baseline/langgraph_agent/graph.py

Removed unused block in final-status section:

```python
# Check if coordinator approved release
coordinator_approved = True
if final_review:
    coordinator_approved = final_review.decision in (
        CoordinatorDecision.APPROVE_FINAL.value,
        CoordinatorDecision.PROCEED.value,
    )
```

Effect:
- Removes dead logic that was computed but never consumed.
- Clarifies that execution success/failure is determined exclusively by execution_result checks below.

### Scope Confirmation

Only the two audit-approved implementation changes were applied.

No changes were made to:
- disagreement.py
- guards.py
- analyst opinions
- coordinator synthesis
- hypothesis analyst
- verifier opinion conversion
- specialist opinion conversion
- replay artifacts
- trace serialization
- eval_v2 reporting
- eval_v2_schema
- eval_v2_viz

No broader recovery or architecture changes were performed.