HCAI-Lab/w2-consensus-deepdive-unlearning-artifacts/social-data-attribution-w2/scripts/analysis/multiseed - Storage Bucket

Selectivity Re-analysis + Multi-Seed Unlearning (COLM 2026 #3137 rebuttal)

Methodology, process, and results for the corrected + multi-seed bin-targeted-vs-naïve-top-K selectivity evidence. Self-contained record so the work survives /tmp wipes and scratch prunes.

1. The problem we found (config-mismatch)

The original Appendix L selectivity numbers were inflated by a baseline-config mismatch: γ was computed as baseline_acc − unlearned_acc, but the baselines came from the OLMo-3 Tech Report (different eval config) while unlearned accuracies were measured under our :mc::olmes protocol. Config-matched OLMo-3-7B-base baselines (verified by re-eval): socialiqa 0.739 (was 0.8029), arc_challenge 0.8345 (was 0.892), mmlu_stem 0.5975, mmlu_social_science 0.7524.

Re-running everything under one protocol (build_gamma_olmes.py → factorial_2x2_olmes.py, pilot_bootstrap_olmes.py): the pilot-validated 4/4 claim survives, but secondary claims weaken — head-to-head 3/4→2/4, 2×2 "both ingredients help" 3/4→2/4 (topic-effect median <1), MMLU-STEM bootstrap CI crosses 1. Verified not a port bug (original script on old data reproduces the staged numbers, so the delta is purely the baseline fix).

2. Multi-seed methodology (the honest, faithful version)

Goal: error bars on the selectivity via 3 fixed seeds {42, 43, 44}.

Forget-set selection (faithful, per-doc influence): for each (topic, target-benchmark), take the top-200 docs by per-doc influence (influence_scores_full.parquet, cols doc_id, <bench>_score, weborganizer_topic) — expA restricts within a topic, expC is corpus-wide. Random baselines: exp1 (random-in-topic), exp3 (random-global).

The text-provenance fix: the influence doc_ids are a Dolma-UUID space that does not join the local 6T training cache (seom35/...dolma3_6t_filtered, a different id scheme). The faithful text lives in the HCAI-Lab HF datasets (same UUID space): dolma3-6t-unique / dolma3-6t-sample-*-docs. Text for the 19k forget doc_ids was extracted (single linear scan of the TrackStar 10k shards at cs7634/tracstar/shards_10k/) into faithful_forget_text.parquet (schema recipe, doc_id, text, 100% coverage). Per-recipe parquets live in `/scratch/n16_selectivity/forget_texts_faithful/`.

A bin-level proxy (forget-sets from topic×format z-scored bins, which do join the local cache) was used while the faithful texts were unavailable; it validated the approach (gate 16×) but was retired once the faithful per-doc texts arrived. The faithful gate gave γ(socialiqa)=0.219, 15.0×.

Training: NGDiff + rank-8 LoRA on OLMo-3-1025-7B, lr 1e-5 constant, PPL early-stop, via src/unlearning/train.py with +forget_texts_file=<per-recipe parquet> (added path in dolma_pool.py). Submitted by submit_multiseed_faithful.py (HX00, seeds 42/43/44). Eval: :mc::olmes on the 4 primary benchmarks via eval_array.sbatch (merge LoRA → vLLM).

3. Results (faithful, 3-seed mean±std; ratios = sel(bin-targeted)/sel(naïve top-K))

Target	head-to-head (top-1)	pilot-validated	best-in-hindsight
SocialIQA	2.41±1.26 (3/3)	6.36±1.40 (3/3)	6.36±1.40 (3/3)
MMLU-STEM	1.74±1.48 (1/2)	2.45±0.13 (2/2)	2.90±0.31 (2/2)
MMLU-SS	0.40±0.33 (0/3)	0.07±0.04 (0/3)†	1.10±0.49 (2/3)
ARC	(training in flight 2026-05-28)

Per-recipe selectivity (best topics): SocialIQA social_life 21.6±11.6×, literature 14.6±5.1×. CSVs: results/{faithful_gamma_tidy,faithful_selectivity_ci,faithful_pilot_ci}.csv.

Shortlist pool (decision 2026-05-28): with ARC added as a 4th target, the unique-z shortlist is recomputed against the uniform 4-target pool. This shifts the mmlu_ss/mmlu_stem shortlist membership, but SocialIQA (6.36×, 3/3) and MMLU-STEM (2.45×, 2/2) are unchanged (the selectivity-best pick stays the same topic). †MMLU-SS's 4-pool shortlist (transportation/politics/finance_and_business) sits mostly outside the trained headline-9 topics, so its pilot needs the full 24-topic grid. check_arc_shortlist_sensitivity.py documents the 3-pool vs 4-pool shift.

4. Pipeline (scripts in this dir)

• submit_multiseed_faithful.py — submit the multi-seed training grid (forget_texts_file). ARC in BENCHES. Idempotent: skips cells with an existing checkpoint AND cells whose job is live in squeue (closes the requeue-churn TOCTOU gap that truncated checkpoints).
• split_arc_forget_texts.py — split the agent's combined faithful_forget_text_arc.parquet (recipe,doc_id,text) into the per-recipe forget_texts_faithful/*arc_challenge.parquet files.
• build_faithful_eval_manifest.py — enumerate faithful checkpoints → OLMES eval manifest. Skips truncated checkpoints, already-evaluated dirs, AND dirs whose training job is still live (their "latest" checkpoint is a mid-training intermediate).
• submit_eval_faithful.py — rebuild manifest + submit the throttled (%4) eval array. Re-run after each training wave; idempotent.
• build_faithful_gamma_ci.py — parse evals → 3-seed γ matrix + per-recipe selectivity mean±std (ARC in BENCHES + BASE 0.8345).
• faithful_pilot_ci.py — head-to-head / pilot / hindsight with seed CIs (Appendix-L numbers). Uniform 4-target pool incl. ARC.
• check_arc_shortlist_sensitivity.py — 3-pool vs 4-pool shortlist diff (records the ARC-fold-in shortlist shift).
• build_gamma_olmes.py / factorial_2x2_olmes.py / pilot_bootstrap_olmes.py — the corrected seed-1 analysis on the downloaded checkpoints (config-matched).
• seed_unlearn.sbatch / eval_array.sbatch — training + eval job templates.
• cleanup_truncated.py — delete truncated checkpoints (cancel/requeue churn) so they re-train.

5. Open items

• ARC eval + final CI rebuild — 30 ARC training cells launched 2026-05-28 (9 headline topics×3 seeds + expC×3). When done: submit_eval_faithful.py → build_faithful_gamma_ci.py + faithful_pilot_ci.py to populate the ARC row.
• Full 24-topic grid (MMLU-SS 4-pool pilot shortlist = transportation/politics/finance, mostly outside the trained headline-9) — user launches after the n10b prune freed headroom.
• Paper: rewrite Appendix L to lead with the surviving pilot (SocialIQA 3/3 + MMLU-STEM 2/2), with 3-seed mean±std numbers, once ARC lands.