Spaces:
Running
Running
File size: 2,892 Bytes
d3d0e0e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | # Stabilization Report (Reporting-Only)
Date: 2026-06-16
Scope: Reporting and validation semantics only. Runtime execution logic was not modified in this pass.
Target run: /data3/dataFAIR/kdd-dev/public/artifacts/runs/20260616T053315Z
## Changes Applied
1. Coordinator consistency metric reconciliation
- File: src/data_agent_baseline/langgraph_agent/eval_v2_validator.py
- Change:
- Reframed coordinator consistency mismatches as diagnostic warnings instead of hard errors.
- Updated issue detail text to explicitly indicate diagnostic mismatch semantics.
- Result:
- eval-v2 no longer fails solely due to coordinator diagnostic count mismatches.
2. Meaningful disagreement reporting hardening
- File: src/data_agent_baseline/langgraph_agent/eval_v2.py
- Changes:
- Tightened interpretation conflict detection to only count approve-vs-rework divergence as interpretation conflict.
- Excluded routine request-evidence variance from interpretation conflict inflation.
- Updated run-level meaningful disagreement aggregation to prioritize analyst_team_meaningful_disagreements; fallback now uses critical_disagreement_score > 1.
- Result:
- Confidence-only disagreement patterns no longer inflate meaningful-disagreement counts.
3. Hardening test alignment
- File: tests/test_eval_v2_hardening.py
- Changes:
- Updated coordinator inconsistency test to assert warning severity for:
- replan_count_consistent
- retry_count_consistent
- coordinator_metrics_consistent
- Updated attribution expectation to match current validator behavior (failure attribution check currently disabled; recovery attribution still required).
- Result:
- Hardening tests reflect intended reporting semantics without requiring runtime changes.
## Validation Results
1. Unit tests
- Command:
- /workspace/ainn-cm-poc-data-agent/.venv/bin/pytest tests/test_eval_v2_hardening.py
- Result:
- 4 passed
2. Eval-v2 on target run
- Commands:
- /workspace/ainn-cm-poc-data-agent/.venv/bin/dabench eval-v2 /data3/dataFAIR/kdd-dev/public/artifacts/runs/20260616T053315Z
- /workspace/ainn-cm-poc-data-agent/.venv/bin/dabench eval-v2 /data3/dataFAIR/kdd-dev/public/artifacts/runs/20260616T053315Z --mode verbose
- /workspace/ainn-cm-poc-data-agent/.venv/bin/dabench eval-v2 /data3/dataFAIR/kdd-dev/public/artifacts/runs/20260616T053315Z --mode research
- Results:
- All three modes completed successfully.
- Consistency validator reports 18 warning issues (no error issues).
- Eval output no longer exits with "failing eval-v2" due to coordinator diagnostic mismatches.
- Analyst Team meaningful disagreements now report as 6 (was previously inflated to 25 in confidence-only conditions).
## Non-Goals / Confirmations
- No runtime execution behavior changes were made in this reporting pass.
- Artifact reconciliation and integrity checks remain PASS for the target run.
|