Explanations are a Means to an End: Decision-Theoretic Explanation Evaluation — Reproduction

arXiv:2506.22740 · OpenReview fVso2kGVrF · DineshAI/fVso2kGVrF
C1 · Def 6 TVEVERIFIED 25.68% vs paper 24.2 [22.4, 26.1]
C2 · Def 7 PHCVVERIFIED 8.81% vs paper 9 [8.1, 9.9]
C3 · Def 8 behavioralVERIFIED −0.78 pp CI [−1.65, +0.09], 998 humans
C4 · Prop 4 garblingVERIFIED exact: 64/64 g, 500/500 rational
C5 · Ames Section BVERIFIED full pipeline, 100 splits
C6 · Examples 1–2VERIFIED exact biopsy + 1,000-applicant recourse

Figure 1 reproduced — value of each signal (relative MAPE improvement)

Features X Model pred Ŷ SHAP expl Z 25.68% 8.81% 9.17% 1.73% 15.27% 1.94%
Theoretic value (vs prior) Complementary value (vs human)

Headline numbers (Ames, 100 splits)

25.68%
TVE ΔE (Def 6)
paper 24.2 [22.4, 26.1] ✓
8.81%
PHCV ΔE,compl (Def 7)
paper 9.0 [8.1, 9.9] ✓
−0.78 pp
Behavioral Δ (Def 8), real humans
CI [−1.65, +0.09] — no sig. effect
21/21
verifier gates PASS
exits nonzero on failure
  • Prop 4 exact: all 64 garblings, symbolic ∀(p,u), 500/500 rational problems, RXŶZ = RXŶ = RX
  • Negative controls: shuffled features −35.2%; state-leaking Z +17.79 pp

Method & provenance

  • Appendix-B pipeline: De Cock Ames (2,930 homes), 80/20, XGBoost condition model, SHAP TreeExplainer, Eq.-10 OLS synthetic human, u = −|s−a|/s
  • Def 8 on the paper's cited controlled study (CHI 2021 exp 1): 998 participants, CLEAR vs BB, structural regression, placebo control
  • Examples: biopsy exact-rational (rule ⇔ q > 9/10, VoI = 9/4); recourse on UCI German Credit (u = −s²)
  • HF Job 6a6c7919… (cpu-upgrade, 5m03s) · commit db185cca · pinned uv.lock · data SHA-256-pinned · no GPU
Previous judged score 4/12 at 77dc8331 — all six criticisms answered with full-scale executed evidence Poster: Chenruishuo/posterly identity-v1