social-data-attribution-w2/scripts/analysis/multiseed/README.md

# Selectivity Re-analysis + Multi-Seed Unlearning (COLM 2026 #3137 rebuttal)

Methodology, process, and results for the corrected + multi-seed bin-targeted-vs-naïve-top-K
selectivity evidence. Self-contained record so the work survives `/tmp` wipes and scratch prunes.

## 1. The problem we found (config-mismatch)
The original Appendix L selectivity numbers were inflated by a **baseline-config mismatch**: γ was
computed as `baseline_acc − unlearned_acc`, but the baselines came from the OLMo-3 Tech Report
(different eval config) while unlearned accuracies were measured under our `:mc::olmes` protocol.
Config-matched OLMo-3-7B-base baselines (verified by re-eval): **socialiqa 0.739** (was 0.8029),
**arc_challenge 0.8345** (was 0.892), **mmlu_stem 0.5975**, **mmlu_social_science 0.7524**.

Re-running everything under one protocol (`build_gamma_olmes.py` → `factorial_2x2_olmes.py`,
`pilot_bootstrap_olmes.py`): the **pilot-validated 4/4 claim survives**, but secondary claims
weaken — head-to-head 3/4→2/4, 2×2 "both ingredients help" 3/4→2/4 (topic-effect median <1),
MMLU-STEM bootstrap CI crosses 1. Verified not a port bug (original script on old data reproduces
the staged numbers, so the delta is purely the baseline fix).

## 2. Multi-seed methodology (the honest, faithful version)
**Goal:** error bars on the selectivity via 3 fixed seeds {42, 43, 44}.

**Forget-set selection (faithful, per-doc influence):** for each (topic, target-benchmark),
take the top-200 docs by per-doc influence (`influence_scores_full.parquet`, cols
`doc_id, <bench>_score, weborganizer_topic`) — expA restricts within a topic, expC is corpus-wide.
Random baselines: exp1 (random-in-topic), exp3 (random-global).

**The text-provenance fix:** the influence `doc_id`s are a Dolma-UUID space that does **not** join
the local 6T training cache (`seom35/...dolma3_6t_filtered`, a different id scheme). The faithful
text lives in the HCAI-Lab HF datasets (same UUID space): `dolma3-6t-unique` /
`dolma3-6t-sample-*-docs`. Text for the ~19k forget doc_ids was extracted (single linear scan of
the TrackStar 10k shards at `cs7634/tracstar/shards_10k/`) into
`faithful_forget_text.parquet` (schema `recipe, doc_id, text`, 100% coverage). Per-recipe parquets
live in `~/scratch/n16_selectivity/forget_texts_faithful/`.

> A bin-level proxy (forget-sets from topic×format z-scored bins, which *do* join the local cache)
> was used while the faithful texts were unavailable; it validated the approach (gate 16×) but was
> retired once the faithful per-doc texts arrived. The faithful gate gave γ(socialiqa)=0.219, 15.0×.

**Training:** NGDiff + rank-8 LoRA on OLMo-3-1025-7B, lr 1e-5 constant, PPL early-stop, via
`src/unlearning/train.py` with `+forget_texts_file=<per-recipe parquet>` (added path in
`dolma_pool.py`). Submitted by `submit_multiseed_faithful.py` (HX00, seeds 42/43/44).
**Eval:** `:mc::olmes` on the 4 primary benchmarks via `eval_array.sbatch` (merge LoRA → vLLM).

## 3. Results (faithful, 3-seed mean±std; ratios = sel(bin-targeted)/sel(naïve top-K))
| Target | head-to-head (top-1) | pilot-validated | best-in-hindsight |
|---|---|---|---|
| SocialIQA | 2.41±1.26 (3/3) | **6.36±1.40 (3/3)** | 6.36±1.40 (3/3) |
| MMLU-STEM | 1.74±1.48 (1/2) | **2.45±0.13 (2/2)** | 2.90±0.31 (2/2) |
| MMLU-SS | 0.40±0.33 (0/3) | 0.07±0.04 (0/3)† | 1.10±0.49 (2/3) |
| ARC | (training in flight 2026-05-28) | | |

Per-recipe selectivity (best topics): SocialIQA social_life 21.6±11.6×, literature 14.6±5.1×.
CSVs: `results/{faithful_gamma_tidy,faithful_selectivity_ci,faithful_pilot_ci}.csv`.

**Shortlist pool (decision 2026-05-28):** with ARC added as a 4th target, the unique-z shortlist
is recomputed against the **uniform 4-target pool**. This shifts the mmlu_ss/mmlu_stem shortlist
membership, but SocialIQA (6.36×, 3/3) and MMLU-STEM (2.45×, 2/2) are **unchanged** (the
selectivity-best pick stays the same topic). †MMLU-SS's 4-pool shortlist
(transportation/politics/finance_and_business) sits mostly outside the trained headline-9 topics,
so its pilot needs the full 24-topic grid. `check_arc_shortlist_sensitivity.py` documents the
3-pool vs 4-pool shift.

## 4. Pipeline (scripts in this dir)
- `submit_multiseed_faithful.py` — submit the multi-seed training grid (forget_texts_file). ARC in
  BENCHES. Idempotent: skips cells with an existing checkpoint AND cells whose job is live in
  `squeue` (closes the requeue-churn TOCTOU gap that truncated checkpoints).
- `split_arc_forget_texts.py` — split the agent's combined `faithful_forget_text_arc.parquet`
  (recipe,doc_id,text) into the per-recipe `forget_texts_faithful/*arc_challenge.parquet` files.
- `build_faithful_eval_manifest.py` — enumerate faithful checkpoints → OLMES eval manifest. Skips
  truncated checkpoints, already-evaluated dirs, AND dirs whose training job is still live (their
  "latest" checkpoint is a mid-training intermediate).
- `submit_eval_faithful.py` — rebuild manifest + submit the throttled (`%4`) eval array. Re-run
  after each training wave; idempotent.
- `build_faithful_gamma_ci.py` — parse evals → 3-seed γ matrix + per-recipe selectivity mean±std
  (ARC in BENCHES + BASE 0.8345).
- `faithful_pilot_ci.py` — head-to-head / pilot / hindsight with seed CIs (Appendix-L numbers).
  Uniform 4-target pool incl. ARC.
- `check_arc_shortlist_sensitivity.py` — 3-pool vs 4-pool shortlist diff (records the ARC-fold-in
  shortlist shift).
- `build_gamma_olmes.py` / `factorial_2x2_olmes.py` / `pilot_bootstrap_olmes.py` — the corrected
  seed-1 analysis on the downloaded checkpoints (config-matched).
- `seed_unlearn.sbatch` / `eval_array.sbatch` — training + eval job templates.
- `cleanup_truncated.py` — delete truncated checkpoints (cancel/requeue churn) so they re-train.

## 5. Open items
- **ARC eval + final CI rebuild** — 30 ARC training cells launched 2026-05-28 (9 headline topics×3
  seeds + expC×3). When done: `submit_eval_faithful.py` → `build_faithful_gamma_ci.py` +
  `faithful_pilot_ci.py` to populate the ARC row.
- **Full 24-topic grid** (MMLU-SS 4-pool pilot shortlist = transportation/politics/finance, mostly
  outside the trained headline-9) — user launches after the `n10b` prune freed headroom.
- **Paper:** rewrite Appendix L to lead with the surviving pilot (SocialIQA 3/3 + MMLU-STEM 2/2),
  with 3-seed mean±std numbers, once ARC lands.