social-data-attribution-w2/scripts/analysis/multiseed/CROSS_METHOD_FINDINGS.md

# Cross-method selectivity findings (faithful per-doc vs bin-level)

Generated by `cross_method_comparison.py` on the corrected `:mc::olmes` gamma
matrices. All numbers use the **uniform 4-target-pool** unique-z pilot
shortlist (decision 2026-05-28). Bin-level is seed-1 (the OLMES corrected
single-seed matrix). Faithful is 3-seed mean±std over {42, 43, 44}.

## Side-by-side

| Target    | Method          | Pick                            | sel(expA)    | sel(expC)   | Ratio        | γ-on-target          | n_seeds |
|-----------|-----------------|---------------------------------|--------------|-------------|--------------|----------------------|---------|
| SocialIQA | bin-level       | social_life                     | 6.36         | 2.86        | **2.22**     | +0.137               | 1       |
| SocialIQA | faithful 3-seed | (social_life ∪ literature)      | 25.15±4.88   | 4.06±0.84   | **6.36±1.40**| +0.293±0.135         | 3       |
| MMLU-STEM | bin-level       | software_development            | 6.56         | 2.56        | **2.56**     | +0.011               | 1       |
| MMLU-STEM | faithful 3-seed | software_development            | 3.50±0.18    | 1.43±0.00   | **2.45±0.13**| +0.013±0.004         | 2       |
| MMLU-SS   | bin-level       | finance_and_business            | 2.95         | 2.20        | **1.34**     | +0.019               | 1       |
| MMLU-SS   | faithful 3-seed | (politics ∪ transportation)     | 1.05±0.47    | 2.89±1.01   | **0.45±0.31**| +0.002±0.003 **≈0**  | 3       |
| ARC-C     | bin-level       | sports_and_fitness              | 0.44         | 0.16        | **2.70**     | −0.002 **≈0**        | 1       |
| ARC-C     | faithful 3-seed | (food_and_dining ∪ sports)      | 1.74±0.75    | 0.50±0.39   | **8.84±9.59**| +0.007±0.003 **≈0**  | 3       |

## Per-target verdict

- **MMLU-STEM — AGREE (most defensible).** Both methods give ~2.5× selectivity advantage with the *same* pilot pick (`software_development`) and consonant γ-on-target magnitudes (+0.011 vs +0.013±0.004). The methodologically robust win.
- **SocialIQA — DIRECTIONALLY AGREE, faithful magnitude larger.** Both methods win, both center on `social_life` (faithful also drafts `literature` in some seeds). γ-on-target is large under both methods (+0.137 bin, +0.29 faithful). The headline 6.36× is robust to seed variance (±1.40 CV ≈ 22%).
- **ARC-C — DEGENERATE under both methods.** γ-on-target is essentially zero under both (bin −0.002, faithful +0.007±0.003). Whatever ratios fall out are comparing noise to noise — bin reports 2.70×, faithful reports 8.84±9.59. Both are dominated by tiny-denominator effects. *Not a clean win under either method.*
- **MMLU-SS — METHOD-DEPENDENT.** This is the genuine fork. Bin-level reports a 1.34× win driven by a measurable γ-on-target (+0.019). Faithful per-doc reports 0.45× loss with γ-on-target near zero (+0.002±0.003). The faithful forget-sets (top-200 docs by per-doc influence) do not measurably damage MMLU-SS for the shortlist topics; the bin-level z-scored forget-sets do.

## Rebuttal implications

The honest scientific story is **not** "pilot 4/4" and **not** "pilot 2/4":

1. **One methodologically robust pilot win** — MMLU-STEM (~2.5× under both bin-level and faithful per-doc, same pick).
2. **One direction-agreeing pilot win with method-dependent magnitude** — SocialIQA (2.2× → 6.4×, both methods favorable, faithful stronger).
3. **One degenerate case** — ARC-C (γ-on-target ≈ 0 under both methods → ratio is noise-dominated either way; reviewers should be told this transparently rather than spun as a 2.70× or 8.84× win).
4. **One genuinely method-dependent case** — MMLU-SS (bin-level wins, faithful per-doc loses; the per-doc top-200 forget-sets are weaker probes for MMLU-SS than the topic×format bin-level set).

This framing is **stronger** than a clean 4/4 for the rebuttal because it (a) acknowledges the cases where the test is degenerate, (b) names a specific methodology gap (MMLU-SS), and (c) protects the two robust wins behind both a multi-seed defense (faithful) and a methodology-independence defense (MMLU-STEM agreeing across methods).

## Files

- Script: `scripts/analysis/multiseed/cross_method_comparison.py`
- Output CSV: `~/scratch/n16_selectivity/results/cross_method_comparison.csv` (pace-ice)
- Inputs: `~/scratch/n16_selectivity/results/{gamma_olmes_tidy,faithful_gamma_tidy}.csv`