social-data-attribution-w2/scripts/analysis/multiseed/CROSS_METHOD_FINDINGS.md

Cross-method selectivity findings (faithful per-doc vs bin-level)

Generated by cross_method_comparison.py on the corrected :mc::olmes gamma matrices. All numbers use the uniform 4-target-pool unique-z pilot shortlist (decision 2026-05-28). Bin-level is seed-1 (the OLMES corrected single-seed matrix). Faithful is 3-seed mean±std over {42, 43, 44}.

Side-by-side

Target	Method	Pick	sel(expA)	sel(expC)	Ratio	γ-on-target	n_seeds
SocialIQA	bin-level	social_life	6.36	2.86	2.22	+0.137	1
SocialIQA	faithful 3-seed	(social_life ∪ literature)	25.15±4.88	4.06±0.84	6.36±1.40	+0.293±0.135	3
MMLU-STEM	bin-level	software_development	6.56	2.56	2.56	+0.011	1
MMLU-STEM	faithful 3-seed	software_development	3.50±0.18	1.43±0.00	2.45±0.13	+0.013±0.004	2
MMLU-SS	bin-level	finance_and_business	2.95	2.20	1.34	+0.019	1
MMLU-SS	faithful 3-seed	(politics ∪ transportation)	1.05±0.47	2.89±1.01	0.45±0.31	+0.002±0.003 ≈0	3
ARC-C	bin-level	sports_and_fitness	0.44	0.16	2.70	−0.002 ≈0	1
ARC-C	faithful 3-seed	(food_and_dining ∪ sports)	1.74±0.75	0.50±0.39	8.84±9.59	+0.007±0.003 ≈0	3

Per-target verdict

• MMLU-STEM — AGREE (most defensible). Both methods give ~2.5× selectivity advantage with the same pilot pick (software_development) and consonant γ-on-target magnitudes (+0.011 vs +0.013±0.004). The methodologically robust win.
• SocialIQA — DIRECTIONALLY AGREE, faithful magnitude larger. Both methods win, both center on social_life (faithful also drafts literature in some seeds). γ-on-target is large under both methods (+0.137 bin, +0.29 faithful). The headline 6.36× is robust to seed variance (±1.40 CV ≈ 22%).
• ARC-C — DEGENERATE under both methods. γ-on-target is essentially zero under both (bin −0.002, faithful +0.007±0.003). Whatever ratios fall out are comparing noise to noise — bin reports 2.70×, faithful reports 8.84±9.59. Both are dominated by tiny-denominator effects. Not a clean win under either method.
• MMLU-SS — METHOD-DEPENDENT. This is the genuine fork. Bin-level reports a 1.34× win driven by a measurable γ-on-target (+0.019). Faithful per-doc reports 0.45× loss with γ-on-target near zero (+0.002±0.003). The faithful forget-sets (top-200 docs by per-doc influence) do not measurably damage MMLU-SS for the shortlist topics; the bin-level z-scored forget-sets do.

Rebuttal implications

The honest scientific story is not "pilot 4/4" and not "pilot 2/4":

1. One methodologically robust pilot win — MMLU-STEM (~2.5× under both bin-level and faithful per-doc, same pick).
2. One direction-agreeing pilot win with method-dependent magnitude — SocialIQA (2.2× → 6.4×, both methods favorable, faithful stronger).
3. One degenerate case — ARC-C (γ-on-target ≈ 0 under both methods → ratio is noise-dominated either way; reviewers should be told this transparently rather than spun as a 2.70× or 8.84× win).
4. One genuinely method-dependent case — MMLU-SS (bin-level wins, faithful per-doc loses; the per-doc top-200 forget-sets are weaker probes for MMLU-SS than the topic×format bin-level set).

This framing is stronger than a clean 4/4 for the rebuttal because it (a) acknowledges the cases where the test is degenerate, (b) names a specific methodology gap (MMLU-SS), and (c) protects the two robust wins behind both a multi-seed defense (faithful) and a methodology-independence defense (MMLU-STEM agreeing across methods).

Files

• Script: scripts/analysis/multiseed/cross_method_comparison.py
• Output CSV: ~/scratch/n16_selectivity/results/cross_method_comparison.csv (pace-ice)
• Inputs: ~/scratch/n16_selectivity/results/{gamma_olmes_tidy,faithful_gamma_tidy}.csv