Hugging Face's logo

Spaces:
CoreModelsResearch
/
DAO_kdd26
Running

App Files Community
DAO_kdd26 / docs /investigation /recovery_diff_audit.md
sipe5001's picture
sipe5001
Add Hugging Face Docker Space configuration
d3d0e0e
preview code
|
Raw
History Blame Contribute Delete
16.5 kB

Recovery Diff Audit

Generated: 2026-06-16 Scope: Analyst Team, Coordinator Synthesis, Disagreement Resolution, Auditor, Reporting Cleanup

1. Git History Summary

Repository started from the KDD Cup 2026 starter kit and evolved through multiple feature branches. ORIG_HEAD: 27e538bb (most recent amend was "Update summary rendering to include lambda penalty and recall metrics"). The AAT / analyst-team work cannot be individually bisected because no known-good commit was tagged, but the code itself carries its own rollback instructions (see §2 below).

Pre-existing changes with explicit rollback comments in graph.py (NOT in scope):

  • voted_planner_node — "CHANGE I — Multi-candidate plan voting"
  • apply_post_hoc_guards — "Strategy 5 — Post-hoc deterministic output guards"

2. File Classification

Category A — Reporting Only

Changes that only affect eval metrics, summaries, or console output. No execution-path impact verified.

File Purpose Execution-Affecting Recommendation
src/.../eval_v2.py Added _extract_analyst_team_metrics(), write_auditor_validation_report(), summary aggregation extensions for analyst team fields NO KEEP — all read-only from trace; meaningful-disagreement counting needs tightening (Phase 3)
src/.../eval_v2_schema.py Added 33 analyst team TaskMetrics fields NO KEEP — pure schema definition
src/.../eval_v2_viz.py Added render_analyst_team_summary(), render_auditor_diagnostics(), per-task column extensions NO KEEP — display only
src/.../cli.py Added write_auditor_validation_report() call and import NO KEEP — reporting orchestration only
src/.../aat/disagreement.py NEW: detect_disagreements() — all items have actionable=False, purely reporting-oriented NO KEEP — already correct
src/.../aat/schema.py (new types) Added AnalystOpinion, DisagreementItem, CoordinatorSynthesis, audit result types NO KEEP — passive data containers
EVALUATION.md Updated documentation for analyst team metrics NO KEEP — doc only

Category B — Trace Only

Changes that persist new data to trace.json and state but are not consumed for routing.

File Purpose Execution-Affecting Recommendation
src/.../state.py Added analyst_opinions, disagreements, coordinator_synthesis fields to AgentState NO KEEP — additive fields, never read for routing
src/.../runner.py Added aat_analyst_opinions, aat_disagreements, aat_coordinator_synthesis to trace.json serialization NO KEEP — write-only serialization
src/.../aat/__init__.py Extended exports for new schema types and modules NO KEEP — imports only
src/.../aat_observability.py Phase/agent name mapping utilities for trace annotation NO KEEP — utility only

Category C — Execution-Affecting

Changes that alter the runtime call graph, LLM call count, context fed to planner, or execution status logic.

File Purpose Execution-Affecting Mechanism Recommendation
src/.../graph.py (CP1 section, ~lines 630–695) Coordinator checkpoint_understanding() gates which specialist agents run via specialists_requested list YES Adds 1 LLM call before planning; conditions SchemaAgent/DomainAgent execution on coordinator judgment rather than deterministic baseline REVIEW — verify coordinator reliably always requests schema+domain; if not, add fallback guarantee
src/.../graph.py (coordinator_approved variable, ~lines 1088–1098) Computes coordinator_approved from final_review.decision LATENT Variable is computed but never used in downstream routing; dead code that misleads readers REMOVE dead variable or add a comment making inert status explicit
src/.../graph.py (added LLM calls pre-plan) coordinator CP1 + AnalysisSynthesizer each add 1 LLM round-trip before planner YES (latency/token) Adds ~2 extra LLM calls to every task before planning begins ACCEPTABLE if specialist quality improves planner; latency risk documented
src/.../graph.py (added LLM calls post-execute) hypothesis_agent, coordinator.to_opinion(), FinalSummaryGenerator each add 1 LLM call after execution YES (latency/token) 3 extra LLM calls per task post-execution; these do NOT feed back into execution decisions ACCEPTABLE — post-execution, no routing impact; latency risk documented
src/.../aat/coordinator.py (checkpoint_understanding) Returns specialists_requested list that graph.py respects YES If coordinator returns narrow specialist list (e.g. only ["schema"]), domain context is missing from planner REVIEW — add explicit fallback to always include ["schema", "domain"]
src/.../aat/coordinator.py (checkpoint_planning) Returns PlanningReview with needs_replan, needs_retry fields NO (stabilized) Already has explicit STABILIZATION comment in graph.py: "not enforced" KEEP — already stabilized
src/.../aat/coordinator.py (checkpoint_final) Returns FinalReview with can_release, needs_retry, needs_replan NO (dead path) Stored in state but coordinator_approved variable computed from it is never used for routing CLARIFY — add comment confirming inert status; remove dead coordinator_approved var

Category D — Unknown / Boundary

Files with execution-control-looking patterns that are confirmed reporting-only in practice but warrant documentation.

File Purpose Execution-Affecting Notes
src/.../aat/guards.py Filter/Schema/Aggregation audit returning suggested_action (PROCEED/RETRY/REPLAN) NO in practice suggested_action values exist in audit result structs but are not consumed by graph.py routing; propagated into AnalystOpinion metadata as advisory only. Risk: future code change could accidentally wire them.
src/.../aat/hypothesis.py Post-execution analyst returning required_action opinion NO in practice Runs AFTER execution loop (graph.py ~line 987); opinion added to analyst_opinions list which is stored for trace. required_action field looks like a control signal but has no consumer routing on it. Risk: field name is misleading.
src/.../aat/verifier.py (to_opinion) Post-verifier opinion conversion returning required_action = RETRY_EXECUTION when not passed NO in practice to_opinion() runs after execution completes; opinion joins analyst_opinions list for trace. Verifier's direct VerificationReport.recommendation is what coordinator CP3 reads — not this opinion.
src/.../aat/specialist_agents.py (to_opinion methods) Each specialist converts its analysis to AnalystOpinion with required_action NO in practice Called after analysis runs; opinions assembled post-synthesis for trace. required_action on these opinions is never consumed for execution branching.

3. Execution-Affecting Hooks — Detailed Findings

Hook 1: CP1 Specialist Gating (HIGH PRIORITY)

File: src/data_agent_baseline/langgraph_agent/graph.py, ~lines 630–660 Mechanism: specialists_requested = understanding_review.specialists_requested if understanding_review else ["schema", "domain"] Risk: If coordinator omits "domain" from specialists_requested, the DomainAgent LLM call is skipped. The planner then receives no domain analysis (business rules, metric definitions, ambiguities). This could degrade answer quality on metric-heavy questions. Current fallback: Falls back to ["schema", "domain"] only if understanding_review is None (exception path). A successful coordinator call that returns a narrow list (e.g., ["schema"]) will skip domain analysis with no fallback. Execution-affecting: YES Recommendation: Add explicit post-CP1 guarantee: always include "schema" and "domain" regardless of coordinator response. Only "document" should be conditionally gated.

Hook 2: Dead coordinator_approved Variable (LOW PRIORITY, CLARITY)

File: src/data_agent_baseline/langgraph_agent/graph.py, ~lines 1088–1098 Mechanism:

coordinator_approved = True
if final_review:
    coordinator_approved = final_review.decision in (
        CoordinatorDecision.APPROVE_FINAL.value,
        CoordinatorDecision.PROCEED.value,
    )
# ... (coordinator_approved never appears again; status check uses execution_result directly)

Risk: Currently inert — status is determined by state.execution_result non-emptiness, not by this variable. Risk is that a future developer adds a conditional on coordinator_approved believing it is intentional.
Execution-affecting: NO (latent risk only)
Recommendation: Add comment: # coordinator_approved: informational only — execution status determined by result presence below.

Hook 3: CP2 Planning Review (ALREADY STABILIZED)
File: src/data_agent_baseline/langgraph_agent/graph.py, ~line 817
Status: Explicit STABILIZATION: comment present: "Coordinator decision is recorded for synthesis/reporting but NOT enforced."
Execution-affecting: NO
Recommendation: No change needed. Keep stabilization comment.

Hook 4: CP3 Final Review (ALREADY INERT)
File: src/data_agent_baseline/langgraph_agent/graph.py, ~line 956
Status: FinalReview fields stored in state, fed into coordinator opinion, but CP3 result does not gate execution return path (see Hook 2).
Execution-affecting: NO
Recommendation: No change needed. Document explicitly via Hook 2 comment fix.

4. Disagreement / Auditor / Coordinator Control Flow Verification
All 14 terms from the Phase 2 search specification were traced:

Term	File	Execution-affecting	Verdict
disagreement	disagreement.py, state.py, eval_v2.py	NO	Reporting only; actionable=False
meaningful_disagreement	eval_v2.py:377	NO	Eval metric; counting logic needs Phase 3 fix
actionable_disagreement	eval_v2_schema.py:450	NO	Metric field; always 0 in current code
agreement_score	schema.py:480, eval_v2.py:367	NO	Computed for synthesis; not consumed for routing
coordinator_action	schema.py:486, eval_v2.py:396	NO	Reporting field derived from final_review
coordinator_synthesis	state.py:162, graph.py:1044	NO	Stored post-execution; not consumed for routing
selected_action	schema.py:483	NO	Mirrors final action; no routing consumer
high_risk_approval	schema.py:447,488, eval_v2.py:420	NO	Always False; metric only
filter_risk	graph.py:1005, coordinator.py:232	NO	Boolean flag; advisory metadata on opinions
schema_risk	graph.py:1005, coordinator.py:235	NO	Boolean flag; advisory metadata on opinions
aggregation_risk	graph.py:1005,1037, guards.py:335	NO	Boolean flag; advisory metadata; not consumed for routing
auditor_result	eval_v2.py:431–436	NO	Reporting metric; aggregates guard trigger counts
auditor_warning	eval_v2_schema.py:472, eval_v2.py:438	NO	Eval flag; no execution consumer
coordinator_approved	graph.py:1088–1098	LATENT	Dead variable — see Hook 2 above
Verdict: No disagreement, auditor, or coordinator synthesis signal currently alters retries, replans, routing, or final approval. The sole active execution impact is CP1 specialist gating (Hook 1).

5. Pre-Existing Execution Changes (Out of Scope for Recovery)
These were added before the analyst-team iterations and have their own documented rollback paths in graph.py imports:

Change	Rollback Path
voted_planner_node (multi-candidate plan voting)	Replace with planner_node in graph.py call site; remove run_zero_row_diagnostic call; delete vote_plan.py
apply_post_hoc_guards (post-hoc output guards)	Remove import and state = apply_post_hoc_guards(state) call in graph.py
6. Phase 2–3 Work Items (Ordered by Priority)
Required for Execution Stabilization
CP1 fallback guarantee — In graph.py, after coordinator.checkpoint_understanding(),
ensure specialists_requested always contains at minimum ["schema", "domain"].
Only "document" should be conditionally gated by coordinator judgment.

Remove/comment coordinator_approved dead code — Add clarity comment or remove the variable
to prevent future accidental wiring.

Required for Reporting Cleanup (Phase 3)
Meaningful disagreement counting — In eval_v2.py _extract_analyst_team_metrics(), the
meaningful_disagreements count is already correctly narrowed (HIGH severity + "interpretation conflict"
topic only). Verify: if critical_disagreement_score == 0, analyst_team_tasks_meaningful_disagreement
must not count that task. Current logic: (df["critical_disagreement_score"] > 0).sum() — already correct.

analyst_team_disagreement_count vs meaningful_disagreements aliasing — Both columns point
to the same meaningful_disagreements value. Consider whether analyst_team_disagreement_count
should show raw disagreement count or meaningful count (currently: meaningful). Document the choice.

No Change Required
All eval_v2*, state.py, runner.py, aat_observability.py, cli.py, disagreement.py — preserve as-is.
CP2, CP3, verifier.to_opinion(), hypothesis.py, specialist to_opinion() — already inert for execution.
guards.py suggested_action field — exists in structs but is not consumed for routing. Preserve as diagnostic advisory.
7. Before Baseline Reference
Latest available eval run for before/after comparison:
/data3/dataFAIR/kdd-dev/public/artifacts/runs/20260616T043015Z

Secondary reference (used in calibration notes):
/data3/dataFAIR/kdd-dev/public/artifacts/runs/20260615T160130Z

---

**Summary of Phase 1 findings:**

| Category | Count | Action |
|----------|-------|--------|
| Reporting-only (Category A) | 7 files | Preserve |
| Trace-only (Category B) | 4 files | Preserve |
| Execution-affecting — active (Category C) | 2 hooks | Fix in Phase 2 |
| Execution-affecting — dead code (Category C) | 1 hook | Clarify comment |
| Boundary/advisory (Category D) | 4 files | Document, no change |

**Only two Phase 2 changes are warranted:**
1. Add a ["schema", "domain"] minimum fallback guarantee at CP1 in src/data_agent_baseline/langgraph_agent/graph.py#L650 so coordinator judgment can never omit a core specialist.  
2. Comment out or annotate the dead coordinator_approved variable at src/data_agent_baseline/langgraph_agent/graph.py#L1088 so its inert status is explicit.

Everything else — disagreement, auditor, guards, coordinator synthesis, analyst opinions — is already diagnostics-only and needs no execution change. Phase 3 reporting fixes (meaningful disagreement threshold, auditor research-mode gating) can proceed independently once Phase 2 is locked.

## Selective Stabilization Changes Implemented

Date: 2026-06-16

Files modified:
- src/data_agent_baseline/langgraph_agent/graph.py
- recovery_diff_audit.md

### Change 1: CP1 Specialist Fallback Guarantee

File: src/data_agent_baseline/langgraph_agent/graph.py

Applied after CP1 coordinator response normalization:

```python
# Stabilization:
# Schema and Domain are mandatory core analysts.
# Coordinator may optionally request Document analysis.
requested = {str(s).lower() for s in specialists_requested}
requested.add("schema")
requested.add("domain")
specialists_requested = list(requested)

Effect:

  • SchemaAgent always runs.
  • DomainAgent always runs.
  • DocumentAgent remains conditional on coordinator request.

Change 2: Dead coordinator_approved Variable Removal

File: src/data_agent_baseline/langgraph_agent/graph.py

Removed unused block in final-status section:

# Check if coordinator approved release
coordinator_approved = True
if final_review:
    coordinator_approved = final_review.decision in (
        CoordinatorDecision.APPROVE_FINAL.value,
        CoordinatorDecision.PROCEED.value,
    )

Effect:

  • Removes dead logic that was computed but never consumed.
  • Clarifies that execution success/failure is determined exclusively by execution_result checks below.

Scope Confirmation

Only the two audit-approved implementation changes were applied.

No changes were made to:

  • disagreement.py
  • guards.py
  • analyst opinions
  • coordinator synthesis
  • hypothesis analyst
  • verifier opinion conversion
  • specialist opinion conversion
  • replay artifacts
  • trace serialization
  • eval_v2 reporting
  • eval_v2_schema
  • eval_v2_viz

No broader recovery or architecture changes were performed.