social-data-attribution-w2/artifacts/methodology.tex

% ============================================================
%  METHOD
% ============================================================
\section{Method}\label{sec:method}
 
We study capability provenance through aggregate training-data attribution on an open
pretraining corpus. The method has five stages: start from a deduplicated pretraining
population, enrich each document with taxonomy labels and token-count metadata, draw a
token-budgeted working set stratified over the taxonomy bins, compute gradient-based
attribution scores for every document in that working set, and aggregate attribution over
WebOrganizer bins rather than over isolated documents.
 
% ------------------------------------------------------------
\subsection{Problem Setup and Notation}\label{sec:notation}
% ------------------------------------------------------------
 
Let $f_\theta$ denote the pretrained language model under study. For a benchmark query $q$
with answer options $\{a_1,\dots,a_m\}$, the model assigns scores $s(q,a)$ and predicts
$\hat{a}(q)=\arg\max_a s(q,a)$. We study four benchmark families: SocialIQA, MMLU Social
Sciences, GSM8K, and MMLU-STEM.
 
We distinguish two corpus levels throughout the paper. Let $\mathcal{D}_{\mathrm{pop}}$
denote the full deduplicated pretraining population, and let
$\mathcal{D}_{\mathrm{work}} \subset \mathcal{D}_{\mathrm{pop}}$ denote the sampled
working set used for empirical attribution. This distinction matters for interpreting
results: provenance claims are anchored to the population, but empirical findings are
bounded by the working set. Every document $d \in \mathcal{D}_{\mathrm{work}}$ is assigned
a WebOrganizer bin $b(d)\in\{1,\dots,\WebOrgNumBins\}$ from the format$\times$topic taxonomy.
 
% ------------------------------------------------------------
\subsection{Working Set Construction}\label{sec:workingset}
% ------------------------------------------------------------
 
Attribution claims require confirmed provenance: every document in the candidate pool must
have been present in the model's actual training data. OLMo3-7B was trained on a 6T-token
curated mix drawn from the larger 9T-token Dolma3 pool, where RegMix upsampling altered
the natural corpus distribution. We work exclusively from the deduplicated 6T training mix
to avoid attributing influence to documents the model never saw.
 
\paragraph{Model.}
We study OLMo3-7B Base~\citep{olmo3} (\texttt{allenai/Olmo-3-1025-7B}), a
7-billion-parameter language model pretrained on the Dolma3
corpus~\citep{soldaini_dolma_2024}. We use the base variant without instruction tuning so
that attribution scores reflect pretraining data influence directly.
 
\paragraph{Upstream population.}
The paper's provenance claims are anchored to the deduplicated 6T Dolma3 training mix,
which contains \num{1263587168} unique documents. This full population is the source from
which the working set is drawn; it should not be read as implying exhaustive attribution
over all available documents.
 
\paragraph{Deduplication.}
We deduplicate the 6T training mix at the document level using a Bloom filter over document
identifiers, reducing the corpus from approximately 2.7B raw document slots to 1.26B unique
documents. This step collapses the upsampling introduced by RegMix curation and recovers
the set of distinct training examples.
 
\paragraph{Enrichment.}
All 1.26B unique documents are labeled with WebOrganizer topic and format categories using
four DeBERTa-v3-base classifiers~\citep{li2024weborganizer}, producing a 24-topic $\times$
24-format taxonomy with 576 bins. Each document receives a top-1 topic label, a top-1
format label, and the full probability vector over both label spaces. Labels are stored as
per-shard Parquet sidecars and joined into a unified corpus manifest.
 
\paragraph{Labels.}
Each document carries a document identifier, token count, and top-1 WebOrganizer topic and
format labels with associated confidences, yielding a bin assignment
$b(d) \in \{1,\dots,\WebOrgNumBins\}$.
 
\paragraph{Stratified sampling.}
From the enriched 1.26B-document corpus, we draw a stratified working sample by selecting
$n$ documents per bin (uniformly at random within each of the 576 bins), using a
deterministic priority function based on a keyed hash of the document identifier (BLAKE2b
with seed 42). This produces a balanced candidate pool where every bin is equally
represented, enabling direct cross-bin comparisons without reweighting. Documents below a
minimum token-count threshold are excluded before sampling to reduce noise in gradient
computation. The working sample is materialized as sharded JSONL with per-document
metadata (token count, bin assignment, shard provenance) in an accompanying Parquet
manifest.
 
\paragraph{Working set.}
The empirical analysis operates on a stratified sample of \AttrDocCount{} documents
(${\sim}$\AttrTokensM{}M tokens) drawn from the population across all \WebOrgNumBins{}
WebOrganizer~\citep{wettig2025weborganizer} bins (500 documents per bin). The working set
is defined primarily by token budget, with a minimum floor of \num{\WebOrgTokensPerBinM}M
tokens per bin. Realized document counts are an output of sampling, not a fixed input.
Underfilled bins, if any, are reported explicitly in the appendix.
 
\paragraph{Preconditioner sample.}
A separate 100K-document uniform random sample (without stratification) is drawn from the
full 1.26B corpus for building preconditioner matrices (\S\ref{sec:preconditioner}). This
sample is independent of the stratified working sample and covers the natural corpus
distribution.
 
% ------------------------------------------------------------
\subsection{Gradient-Based Attribution}\label{sec:attribution}
% ------------------------------------------------------------
 
We compute document-level attribution scores over the entire working set
$\mathcal{D}_{\mathrm{work}}$ using TrackStar~\citep{chang_scalable_2024}, a gradient-based
influence method implemented in the Bergson library. For each benchmark family $\beta$, we
first extract a mean query gradient by averaging the per-query loss gradients across all
evaluation queries $q \in \mathcal{Q}_\beta$. Model gradients are projected into a
low-dimensional space via Rademacher random projections, making corpus-scale scoring
tractable on commodity GPUs. Attribution scores are then the dot product between each
projected training-document gradient and the mean query gradient:
\[
I(\beta, d) = \langle \phi(\nabla_\theta \ell(d)), \; \bar{\phi}(\nabla_\theta \ell(\mathcal{Q}_\beta)) \rangle
\]
where $\phi$ denotes the random projection and $\bar{\phi}$ the mean-reduced query
projection. Scores are signed: positive values indicate that a document supports the
model's behavior on the benchmark, while negative values indicate opposition. An optional
preconditioner (inverse Hessian approximation) rescales projections to improve calibration.
 
Because we score every document in $\mathcal{D}_{\mathrm{work}}$ rather than a retrieved
subset, the resulting influence distribution reflects the full working set and is not
filtered by lexical or semantic similarity to the query. This exhaustive scoring is central
to the paper's design: it allows us to discover influential corpus regions that would be
missed by retrieval-based candidate selection. Full hyperparameter and compute details are
in Appendix~\ref{app:attribution-pipeline}.
 
\paragraph{Benchmarks.}
We evaluate on a $2 \times 2$ contrastive grid crossing domain (social vs.\ math) with
capability type (reasoning vs.\ knowledge):
%
\begin{center}
\begin{tabular}{lcc}
\toprule
 & Reasoning & Knowledge \\
\midrule
Social & SocialIQA & MMLU Social Sciences \\
Math   & GSM8K     & MMLU STEM \\
\bottomrule
\end{tabular}
\end{center}
%
This design enables contrastive analysis: influence patterns that are specific to social
reasoning (SocialIQA) can be distinguished from those shared with math reasoning (GSM8K)
or social knowledge (MMLU-SS), and from domain-general patterns that appear across all four
benchmarks.
 
\paragraph{Benchmark specifications.}
All evaluations follow the OLMo3 evaluation protocol~\citep{olmo3} to ensure per-query
labels are consistent with published accuracy numbers. SocialIQA uses 5-shot
log-likelihood ranking over three answer choices on \num{10000} items. MMLU Social Sciences
(12 subjects, \num{3077} items) and MMLU-STEM (18 subjects, \num{3018} items) use 5-shot
log-likelihood ranking in the OLMES format~\citep{gu2024olmes}. GSM8K uses 8-shot
chain-of-thought decoding with sampling (temperature 0.6, top-$p$ 0.6, 8 repeats)
evaluated by pass@1 on \num{1319} items:
\[
\mathrm{pass@}k = 1 - \frac{\binom{n - c}{k}}{\binom{n}{k}},
\]
where $n = 8$ is the number of samples and $c$ is the number of correct samples.
 
\paragraph{Query construction.}
For each benchmark, we run OLMo3-7B Base through the OLMES evaluation harness to produce
per-query metadata files recording the model's predictions, correctness labels, and input
formatting. These outputs are transformed into the JSONL format expected by Bergson for
query gradient indexing: each record contains a prompt string and, for multiple-choice
benchmarks, the correct-answer completion. This ensures that the gradients used for
attribution reflect the model's actual loss landscape on each evaluation item.
 
\paragraph{Attribution configuration.}
We compute influence scores over the full working set using
TrackStar~\citep{chang_scalable_2024} via the Bergson library (v\BergsonVersion{}).
Gradients are projected to dimension $d{=}\AttrProjectionDim{}$ via Rademacher random
projections, yielding \AttrPerModuleDim{}-dimensional representations per module. For the
exploratory run, we use unpreconditioned Mode~B scoring; the production pipeline uses a
preconditioned Mode~A gradient index for efficiency. Full hyperparameters are in
Table~\ref{tab:attr-hparams}.
 
\paragraph{Staged execution.}
To reduce compute risk and catch pipeline issues early, we use a staged protocol.
Exploratory runs on a 500-documents-per-bin sample (\AttrDocCount{} documents total,
\AttrGPUHoursExplore{} GPU-hours on H100/H200) validate the attribution and aggregation
pipeline before scaling to the full working set.
 
\paragraph{Compute.}
The exploratory attribution run required \AttrGPUHoursExplore{} GPU-hours on NVIDIA
H100/H200 GPUs across 69 jobs (4 reduce, 64 score, 1 aggregate). The production Mode~A
pipeline reduces marginal cost per additional benchmark to near zero after the initial
gradient-index build. Full compute breakdowns are in Table~\ref{tab:attr-compute}.
 
\paragraph{Gradient computation.}
For each example $x$ (training document or evaluation query), Bergson performs a forward
pass through the model, computes the per-example cross-entropy loss $\ell(x; \theta)$, and
extracts per-sample gradients via backward hooks on all linear-layer weight matrices
(attention Q/K/V/O and MLP projections, excluding biases). The resulting gradient vector is
high-dimensional (one entry per parameter per module). To make computation and storage
tractable, each module's gradient is compressed via random projection using a Rademacher
matrix~\citep{achlioptas2003database} to a $d \times d$ block, where $d$ is the projection
dimension.
 
We set $d = 16$, yielding a 256-dimensional projected gradient per module. All gradients
are computed in fp32 for numerical stability. Bergson packs examples into batches by token
count rather than example count, with a token budget of 1{,}024 tokens per forward pass;
sequences exceeding this budget are truncated.
 
After projection, each gradient vector is normalized to unit length before scoring. This
converts raw dot products into cosine-like similarities and prevents modules with larger
gradient norms from dominating the aggregated score.
 
\paragraph{Scoring.}
The influence of training document $j$ on query $i$ is computed as
\[
s(j, i) = \sum_{k \in \mathcal{K}} \hat{g}_k(x_j)^\top \hat{g}_k(q_i),
\]
where $\mathcal{K}$ is the set of model modules, $\hat{g}_k(\cdot)$ denotes the
unit-normalized projected gradient at module $k$, and $x_j$ and $q_i$ are training
document $j$ and query $i$ respectively. Positive scores indicate that document $j$ pushes
the model toward the behavior exhibited on query $i$; negative scores indicate the
opposite.
 
\paragraph{Pipeline architecture.}
We use Bergson's Mode~B (reduce/score) workflow, which operates in three phases with SLURM
dependency chaining so that each phase runs only if the previous one succeeded:
 
\begin{enumerate}
    \item \textbf{Reduce.} For each benchmark, all query gradients are aggregated into a
    single mean gradient vector via \texttt{bergson reduce}. This collapses the per-query
    structure into one representative gradient per benchmark.
    \item \textbf{Score.} The training corpus is partitioned into $N$ JSONL shards via
    round-robin distribution. Each shard is scored independently against each reduced query
    gradient via \texttt{bergson score}: document gradients are computed on-the-fly, dotted
    against the reduced query vector, and discarded. This yields one scalar influence score
    per document per benchmark without materializing the full training gradient index.
    \item \textbf{Aggregate.} All shard scores for a given benchmark are concatenated and
    the top-$k$ documents are selected via partial sorting (\texttt{np.argpartition}).
\end{enumerate}
 
For the full working sample, Phase~2 submits $N \times Q$ GPU jobs (one per shard--query
pair), where $Q = 4$ benchmarks. Each score job requires a single H100 or H200 GPU
(OLMo3-7B in fp32 uses approximately 28\,GB VRAM for gradient computation). Phase~3 runs
on CPU only.
 
\paragraph{Score extraction.}
After aggregation, we extract all per-document scores (not just top-$k$) and join them
with the corpus manifest to produce a combined Parquet file with columns for document
identifier, per-benchmark influence scores, bin assignment (topic, format, bin ID), and
token count. This file is the input to all downstream analysis.
 
\paragraph{Model variants.}
The primary attribution results use OLMo3-7B Base (\texttt{allenai/Olmo-3-1025-7B}), the
pretrained checkpoint before any post-training. We additionally prepare query gradient
indexes for the instruct-tuned checkpoint (\texttt{allenai/Olmo-3-7B-Instruct}) in two
query formats: direct answer (\texttt{instruct\_base}) and chain-of-thought
(\texttt{instruct\_cot}). These variant indexes use the same Bergson parameters and
training corpus as the base runs, enabling direct comparison of how post-training reshapes
attribution patterns while holding the pretraining data constant.
 
\paragraph{Preconditioner.}\label{sec:preconditioner}
Unpreconditioned gradient inner products treat all parameter directions equally.
Preconditioners rescale gradients by an approximation to the inverse Hessian, upweighting
informative directions and downweighting noisy ones~\citep{grosse2023studying}.
 
We build the TrackStar mixed preconditioner in three steps:
 
\begin{enumerate}
    \item \textbf{Value preconditioner.} Compute second-moment (Fisher) matrices over the
    100K-document random sample (\S\ref{sec:workingset}) by running a full gradient-index
    build with Bergson.
    \item \textbf{Query preconditioner.} Compute the same matrices over the evaluation
    query set.
    \item \textbf{Mix.} Combine the value and query preconditioners via the TrackStar mixing
    procedure with a target downweight of 1{,}000 components, producing a single mixed
    preconditioner matrix per module.
\end{enumerate}
 
The mixed preconditioner is factored via eigendecomposition
($\mathbf{P}_k = \mathbf{V}_k \boldsymbol{\Lambda}_k \mathbf{V}_k^\top$ for each module
$k$) and applied to gradients before scoring. When preconditioners are active, the
influence score becomes
\[
s(j, i) = \sum_{k \in \mathcal{K}} \hat{g}_k(x_j)^\top \mathbf{P}_k^{-1} \hat{g}_k(q_i).
\]
Exploratory runs use \texttt{--skip\_preconditioners} for faster iteration; the
preconditioned path is reserved for the final reporting runs.
 
% ------------------------------------------------------------
\subsection{Aggregate Bin-Level Analysis}\label{sec:aggregation}
% ------------------------------------------------------------
 
Our central methodological choice is to aggregate attribution over WebOrganizer bins. For a
benchmark family $\beta$ and bin $k$, we report signed influence mass
\[
M_\beta(k)=\sum_{d\in\mathcal{D}_{\mathrm{work}}:\,b(d)=k} I(\beta,d)
\]
and absolute influence mass
\[
A_\beta(k)=\sum_{d\in\mathcal{D}_{\mathrm{work}}:\,b(d)=k} |I(\beta,d)|.
\]
These aggregated quantities let us compare benchmark families in terms of where attribution
mass accumulates, which bins dominate support, and how influence distributions shift between
social and non-social tasks.
 
We further stratify the analysis by benchmark family and, when stable enough to be
meaningful, by correctness partition. This supports comparisons such as SocialIQA versus
MMLU Social Sciences, or social targets versus math/STEM controls, without
over-interpreting individual document-level scores.
 
Raw per-document scores are also aggregated to the WebOrganizer bin level to produce the
paper's primary analytic unit. For each benchmark $b$ and bin $c$, we compute the mean
influence score
\[
\bar{s}_{b,c} = \frac{1}{|D_c|} \sum_{j \in D_c} s_b(j),
\]
where $D_c$ is the set of documents assigned to bin $c$ and $s_b(j)$ is the influence
score of document $j$ for benchmark $b$. This yields a $576 \times 4$ matrix of bin-level
influence values that can be visualized as $24 \times 24$ heatmaps (topic $\times$ format)
per benchmark, and compared across the $2 \times 2$ contrastive grid.
 
\paragraph{Contrastive distribution shift.}
The central analytic question is how influence mass redistributes across bins when moving
between benchmarks. We quantify this by comparing the ranked bin-influence vectors across
the four benchmarks using Spearman rank correlation, identifying bins whose influence is
domain-specific (high for one benchmark, low for others) versus domain-general
(consistently high or low across all four).
 
% ------------------------------------------------------------
\subsection{Stability and Characterization}\label{sec:stability}
% ------------------------------------------------------------
 
To assess whether aggregate findings are robust enough to support interpretation, we
estimate uncertainty through bootstrap resampling over evaluation queries and, where
appropriate, over within-bin sampled documents. We report stability in terms of confidence
intervals, rank consistency of high-support bins, and agreement of benchmark-level
distribution summaries under resampling.
 
For high-support bins, we then move from attribution scores to corpus characterization.
Documents in those bins are summarized using provenance fields, genre/source cues, and
lightweight discourse markers so that the paper can describe influential regions in
human-interpretable terms rather than only as numeric cells in a heatmap.
 
\paragraph{Causal validation.}
We complement the aggregate attribution analysis with machine-unlearning experiments
(\S\ref{sec:unlearning}) that selectively remove documents from high-influence bins and
measure the resulting benchmark degradation. These interventions serve as causal validation
of the strongest aggregate findings rather than as the core methodological contribution.
The main scientific object of the paper remains the aggregate attribution structure itself.
 
% ------------------------------------------------------------
\subsection{Causal Validation via Machine Unlearning}\label{sec:unlearning}
% ------------------------------------------------------------
 
Training-data attribution provides associational evidence: high influence mass in a bin
suggests that the corpus region supports a given benchmark behavior, but it does not
establish that the region is causally necessary. We therefore complement the aggregate
attribution findings with targeted machine unlearning experiments that treat bin-level
forgetting as a direct test of causal impact. Our framing follows~\citet{bu2025ngdiff} and
extends it to the WebOrganizer taxonomy level.
 
We structure the intervention around two axes: whether influence scores guide document
selection (random sampling vs.\ Bergson-guided), and the scope of forgetting (single bin
vs.\ multiple bins vs.\ no bin specified). The random-sampling experiments serve as the
primary baseline; the influence-guided experiments test whether Bergson scores carry
document-level discriminative power beyond bin membership alone.
 
\paragraph{Optimization.}
Both intervention types use the same unlearning procedure. Following~\citet{bu2025ngdiff},
we minimize
\[
\mathcal{L}(\theta) = -\mathcal{L}_{\mathrm{forget}}(\theta) + w_{\mathrm{retain}} \cdot \mathcal{L}_{\mathrm{retain}}(\theta),
\]
with the NGDiff update direction
\[
g_{\mathrm{update}} = \frac{g_{\mathrm{retain}}}{\|g_{\mathrm{retain}}\|} - \frac{g_{\mathrm{forget}}}{\|g_{\mathrm{forget}}\|},
\]
which normalizes both gradient components so that neither dominates regardless of bin size
or loss scale. We apply an automatic Hessian-free learning rate adapted every 10 steps via
two additional forward passes on the retain set. All runs use LoRA (rank~8, applied to
Q/K/V/O projections) for parameter efficiency, following the published NGDiff setup for
7B-scale models. Full hyperparameters are listed in Table~\ref{tab:unlearn-hyperparams}.
 
\paragraph{Data.}
For all unlearning experiments, document identity is established against the deduplicated
6T Dolma3 training mix using confirmed provenance from our working set construction
(\S\ref{sec:workingset}); only documents verified as present in OLMo3-7B's actual training
population are eligible for forget or retain sets.
 
\paragraph{Stopping criterion and evaluation.}
We stop unlearning when the forget-set perplexity reaches or exceeds the base model's
perplexity on a token-shuffled version of the forget set, indicating that the model has
lost structured knowledge of those documents. A secondary safety guard halts training if
general-capability performance (a 100-question inline MMLU subset) drops below 90\% of
baseline. A hard ceiling of 5{,}000 optimizer steps applies in all runs. We evaluate all
four benchmarks every 250 steps so that capability specificity can be assessed throughout
training: a bin that is causally relevant to social reasoning should produce larger
degradation on SocialIQA than on GSM8K.
 
\subsubsection{Random-Baseline Experiments}\label{sec:unlearn-random}
 
The random-sampling experiments establish the causal baseline: how much does forgetting a
semantically defined corpus region degrade benchmark performance when documents are selected
without reference to influence scores?
 
\paragraph{Single-Bin Unlearning.}
We independently unlearn each of the 24 WebOrganizer topic bins from OLMo3-7B Base and
measure causal impact on all four benchmarks. For each bin $b$, the forget set consists of
up to 1{,}000 confirmed trained-on documents drawn from that bin; the retain set is a
stratified proportional sample of 9{,}000 documents from the remaining 23 bins, refreshed
each epoch. This produces a $24 \times 4$ degradation matrix
$\gamma_{b,j} = (A_{\mathrm{unlearned}} - A_{\mathrm{baseline}}) / A_{\mathrm{baseline}}$
for each bin $b$ and benchmark $j$, and a corresponding heatmap. A bin that produces larger
degradation on its target benchmark than on contrastive controls provides direct causal
evidence that the corresponding corpus region is load-bearing for that capability.
 
\paragraph{Multi-Bin Targeted Unlearning.}
We extend single-bin unlearning to groups of semantically aligned bins, testing joint
causal impact and scaling behavior. Each benchmark is associated with three nested bin
groups of increasing size (3, 5, and 7 bins; Group~A $\subset$ Group~B $\subset$ Group~C),
defined in Table~\ref{tab:topic-groups}. The forget set is the union of documents from all
bins in the selected group; the retain set excludes all documents whose topic appears in
that group. This design tests how much aligned knowledge must be removed to produce
measurable capability degradation, and whether the effect scales with group size.
 
\paragraph{Null-Bin Control.}
We unlearn a randomly sampled cross-topic set of the same size as the average single-topic
forget set, with no semantic alignment to any benchmark. This establishes the procedural
degradation baseline attributable to the unlearning procedure itself. All $\gamma$ values
from Experiments~1 and~2 are interpreted relative to this null baseline: net causal impact
is $\gamma_{\mathrm{topic}} - \gamma_{\mathrm{null}}$.
 
\subsubsection{Influence-Guided Experiments}\label{sec:unlearn-guided}
 
The influence-guided experiments test the core validation claim: if Bergson influence
scores have document-level discriminative power, then forgetting high-influence documents
should produce stronger benchmark degradation than forgetting random documents from the
same bin. Experiments~1--3 provide the random-matched baselines against which these results
are interpreted.
 
\paragraph{Influence-Guided Single-Bin Unlearning.}
For the top 2--3 bins identified by Single-Bin Unlearning as having the largest causal
impact on SocialIQA, we run NGDiff unlearning on three matched forget sets constructed from
the same bin:
\begin{itemize}
    \item \textbf{Condition~A} (top-$k$): the $k$ most influential documents ranked by
    aggregate Bergson influence score over the SocialIQA query set.
    \item \textbf{Condition~B} (random-matched): $k$ documents sampled uniformly at random
    from the same bin, matched on token-count distribution. This is directly comparable to
    the Experiment~1 baseline.
    \item \textbf{Condition~C} (bottom-$k$): the $k$ lowest-influence documents from the
    same bin.
\end{itemize}
The three-way contrast (A vs.\ B vs.\ C) tests whether Bergson scores resolve finer-grained
causal structure beyond what bin membership alone predicts. The key success criterion is a
clear within-bin targetedness gradient on SocialIQA---high-influence drop $>$ random drop
$>$ near-zero drop---with substantially smaller off-target degradation on GSM8K and
MMLU-STEM.
 
\paragraph{Influence-Guided Multi-Bin Unlearning.}
For the same SocialIQA-aligned bin groups from Multi-Bin Unlearning, we run NGDiff
unlearning selecting the top-$k$ documents by aggregate Bergson score from the combined
group, rather than sampling randomly. The random-matched baseline is the corresponding
Multi-Bin Unlearning run on the same bin group. This tests whether influence-guided
selection improves unlearning efficiency at the group level.
 
\paragraph{Influence-Guided Null-Bin Unlearning.}
Without specifying a bin, we run NGDiff unlearning on the top-$k$ documents from the
entire working set ranked by Bergson influence score for SocialIQA. This tests whether
document-level influence scores alone, unconstrained by taxonomy structure, identify
causally load-bearing data---and serves as a complement to Null-Bin Unlearning at the
influence-guided level.