Finalize FeatureLens causal position study

CHANGELOG.md

@@ -1,5 +1,16 @@
 # Changelog
 
+## v0.16.0
+
+- Added two explicit offline causal-position policies: `final_token` and `max_feature_activation`. Max-active positions are selected only from SAE activation within the prompt, never from behavioral outcomes.
+- Preserved the v0.15 final-token causal result as a positional baseline and added a causal-addendum runner that computes only the new max-active policy.
+- Changed primary paired causal inference to use the **causal task** as the statistical unit, averaging ablation and 2× amplification within each task before bootstrap/sign-flip inference.
+- Added exact sign-flip enumeration for small effective paired samples, with deterministic Monte-Carlo fallback for larger samples.
+- Separated causal **coverage** from conditional-on-active effect strength and added final-token vs max-active position-sensitivity summaries.
+- Added `causal_position_summary.csv`, a position-sensitivity report figure, max-active association-vs-causality synthesis, and updated Study-tab diagnostics.
+- Added `experiments/run_causal_addendum.py` and a Drive-backed Colab addendum notebook so a completed v0.15 study can be upgraded without recollecting discovery activations or rerunning feature-set inference.
+- Updated artifact validation and methodology documentation for the finalized v0.16 study schema.
+
 ## v0.15.0
 
 - Reworked the public Gradio surface around a documented **research-instrument design system** rather than SaaS/dashboard defaults.

README.md

@@ -60,15 +60,17 @@ The live app is exploratory. The offline study is the dataset-scale experiment.
 
 It uses **224 discovery prompts** arranged as 112 paraphrase pairs across seven controlled concepts, plus **28 separate causal tasks**. Concept evidence uses prompt-wide max-pooled SAE activations across non-padding tokens; final-token sparse activations are saved separately for local analyses.
 
+Causal evidence is reported under two position policies: the original **final-token** baseline and **max-feature-activation**, which patches the selected feature where it is most strongly represented in the prompt. Max-active positions are selected from SAE activation only, never from downstream behavioral effects. Primary uncertainty uses the causal task as the statistical unit.
+
 The study produces:
 
 - train-only feature selection with held-out AUROC/F1;
 - a dense final-token residual linear-probe baseline;
 - paraphrase stability;
 - 128-resample candidate-selection sensitivity;
-- random-controlled single-feature causal results;
+- random-controlled single-feature causal results under both final-token and max-feature-activation patch policies;
 - top-1/3/5 feature-set causal results;
-- cross-concept association-versus-causality synthesis;
+- causal-position coverage/sensitivity and cross-concept association-versus-causality synthesis;
 - uncertainty-aware report figures and a measured Markdown report.
 
 Run the full pipeline with:
@@ -83,6 +85,12 @@ On a memory-constrained GPU, activation collection can be tuned without changing
 python -m experiments.run_all --resume --activation-batch-size 8
 ```
 
+If you already completed the v0.15 study, upgrade it with only the positional causal addendum:
+
+```bash
+python -m experiments.run_causal_addendum --resume
+```
+
 After the expensive model stages exist, CPU-only analysis can be regenerated with:
 
 ```bash
@@ -97,7 +105,7 @@ python -m scripts.validate_artifacts
 
 ### Google Colab
 
-A ready-to-run notebook is included at [`notebooks/FeatureLens_Offline_Study_Colab.ipynb`](notebooks/FeatureLens_Offline_Study_Colab.ipynb). It mounts Google Drive for persistent artifacts, keeps the Hugging Face cache on the Colab VM, chooses a conservative activation batch from available VRAM, and runs the resumable pipeline.
+A ready-to-run full-study notebook is included at [`notebooks/FeatureLens_Offline_Study_Colab.ipynb`](notebooks/FeatureLens_Offline_Study_Colab.ipynb). If the v0.15 study is already complete, use [`notebooks/FeatureLens_Causal_Addendum_Colab.ipynb`](notebooks/FeatureLens_Causal_Addendum_Colab.ipynb) instead; it runs only the max-active causal addendum and CPU synthesis.
 
 See [`docs/COLAB.md`](docs/COLAB.md) for the exact workflow.
 
@@ -111,7 +119,9 @@ artifacts/
 ├── layer_metrics.csv
 ├── stability.csv
 ├── selection_stability.csv
-├── causal_results.csv
+├── causal_results_final_token.csv
+├── causal_results_max_active.csv
+├── causal_position_summary.csv
 ├── feature_set_results.csv
 ├── study_feature_summary.csv
 ├── study_summary.json

app.py

@@ -3041,6 +3041,7 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
 
         offline_study = STUDY.dataframe("study_feature_summary.csv")
         offline_stability = STUDY.dataframe("selection_stability.csv")
+        offline_positions = STUDY.dataframe("causal_position_summary.csv")
         offline_layers = STUDY.dataframe("layer_metrics.csv")
 
         if not offline_study.empty:
@@ -3067,28 +3068,40 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
 
             with gr.Row(equal_height=False):
                 with gr.Column(scale=3):
-                    _table_heading("Candidate selection stability")
-                    stability_preview = offline_stability.sort_values(
-                        ["resample_support", "full_score"], ascending=[False, False]
-                    ).head(40)
+                    _table_heading("Causal position sensitivity")
+                    position_preview = offline_positions[offline_positions["concept"] == "__all__"] if not offline_positions.empty else offline_positions
                     gr.Dataframe(
-                        value=stability_preview,
+                        value=position_preview,
                         interactive=False,
                         show_label=False,
                         buttons=["fullscreen"],
                         elem_classes=["result-table"],
                         wrap=False,
-                        max_height=420,
+                        max_height=320,
                     )
                 with gr.Column(scale=2):
                     gr.Image(
-                        value=STUDY.figure("candidate_stability.png"),
-                        label="Selected-feature resample support",
+                        value=STUDY.figure("causal_position_sensitivity.png"),
+                        label="Final-token vs max-active intervention",
                         interactive=False,
                         show_label=True,
                         height=360,
                     )
 
+            _table_heading("Candidate selection stability")
+            stability_preview = offline_stability.sort_values(
+                ["resample_support", "full_score"], ascending=[False, False]
+            ).head(40)
+            gr.Dataframe(
+                value=stability_preview,
+                interactive=False,
+                show_label=False,
+                buttons=["fullscreen"],
+                elem_classes=["result-table"],
+                wrap=False,
+                max_height=360,
+            )
+
             _table_heading("Layer diagnostics")
             gr.Dataframe(
                 value=offline_layers,

artifacts/README.md

@@ -1,44 +1,39 @@
 # Generated artifacts
 
-This directory intentionally ships without invented empirical results.
+The repository ships without invented empirical results. The finalized v0.16 study uses prompt-wide feature evidence plus two causal position policies.
 
-Run the full study on a CUDA machine:
+Fresh full study:
 
 ```bash
-python experiments/run_all.py
+python -m experiments.run_all --resume
 ```
 
-For an interruptible session:
+Upgrade an already completed v0.15 study without recollecting discovery activations:
 
 ```bash
-python experiments/run_all.py --resume
+python -m experiments.run_causal_addendum --resume
 ```
 
-If the expensive activation and causal artifacts already exist, regenerate only CPU analysis/report outputs with:
+The public artifact set includes:
 
-```bash
-python experiments/run_analysis_only.py
-```
-
-v0.14 produces:
-
-- prompt-wide activation caches plus separate final-token sparse activations;
 - `feature_catalog.csv`;
 - `layer_metrics.csv`;
 - `stability.csv`;
 - `selection_stability.csv`;
-- `causal_results.csv`;
+- `causal_results_final_token.csv`;
+- `causal_results_max_active.csv`;
+- `causal_position_summary.csv`;
 - `feature_set_results.csv`;
 - `study_feature_summary.csv`;
 - `study_summary.json`;
 - `summary.json`;
 - `report.md`;
-- report figures including association-vs-causality and candidate-stability plots.
+- report figures including causal-position sensitivity and association-vs-causality.
 
-Validate the public artifact set with:
+Validate before commit:
 
 ```bash
 python -m scripts.validate_artifacts
 ```
 
-`artifacts/activations/` contains large residual/sparse caches and stays gitignored. Commit only the small CSV/JSON/report/figure outputs if you want the live **Offline study** tab to display measured results and benchmark-derived feature hints.
+`artifacts/activations/` remains gitignored. Commit only the small CSV/JSON/report/figure outputs.

docs/CAUSAL_ADDENDUM.md

@@ -0,0 +1,49 @@
+# v0.16 causal-position addendum
+
+The completed v0.15 study used prompt-wide SAE evidence for concept discovery but intervened only at the final prompt token. v0.16 preserves that run as a baseline and adds a second causal policy.
+
+## Position policies
+
+### `final_token`
+
+Patch the selected SAE feature at the final prompt token. This is the original v0.15 baseline.
+
+### `max_feature_activation`
+
+For the selected concept feature and causal prompt:
+
+1. encode the selected layer at every non-padding prompt token;
+2. read the selected SAE feature's TopK activation at each token;
+3. choose the token with the largest activation;
+4. apply the SAE edit and all norm-matched random controls at that same token.
+
+The position is selected **only from SAE activation**. The target continuation, logits, and intervention effect are never used to choose the location.
+
+If the selected feature is inactive everywhere in the prompt, the policy records zero coverage and uses the final token as a deterministic zero-delta fallback.
+
+## Statistical unit
+
+Ablation and 2× amplification are repeated interventions on the same causal prompt. v0.16 therefore uses the **causal task** as the primary paired inference unit:
+
+- average the absolute SAE effect across ablation and amplification within each task;
+- average each intervention's random-control ensemble, then average those random magnitudes within task;
+- bootstrap/sign-flip the resulting one SAE-vs-random pair per causal task.
+
+The report separately shows unconditional effects across all tasks and effects conditional on the selected feature being active at the intervention location.
+
+## Addendum runner
+
+With a completed v0.15 artifact directory:
+
+```bash
+python -m experiments.run_causal_addendum --resume
+```
+
+The runner:
+
+1. migrates `causal_results.csv` to `causal_results_final_token.csv` without rerunning it;
+2. computes `causal_results_max_active.csv`;
+3. regenerates `causal_position_summary.csv`, study synthesis, figures, and report;
+4. validates the finalized artifact schema.
+
+It does **not** rerun activation collection, feature evaluation, candidate stability, or feature-set inference.

docs/COLAB.md

@@ -84,7 +84,9 @@ The bundle is expected to contain files such as:
 - `artifacts/layer_metrics.csv`
 - `artifacts/stability.csv`
 - `artifacts/selection_stability.csv`
-- `artifacts/causal_results.csv`
+- `artifacts/causal_results_final_token.csv`
+- `artifacts/causal_results_max_active.csv`
+- `artifacts/causal_position_summary.csv`
 - `artifacts/feature_set_results.csv`
 - `artifacts/study_feature_summary.csv`
 - `artifacts/study_summary.json`
@@ -93,3 +95,13 @@ The bundle is expected to contain files such as:
 - `artifacts/figures/*.png`
 
 Do not commit the `artifacts/activations/` directory.
+
+## v0.16 causal addendum after a completed v0.15 study
+
+If the full v0.15 Colab study already completed, **do not rerun the full notebook**. Use:
+
+`notebooks/FeatureLens_Causal_Addendum_Colab.ipynb`
+
+The addendum notebook copies only the small existing study outputs to a new Drive folder, preserves the final-token causal baseline, computes the 28-task `max_feature_activation` causal policy, regenerates the CPU study/report artifacts, validates them, and creates a new publishable ZIP.
+
+The addendum does not recollect 224-prompt activations, refit feature/probe evaluations, rerun stability resampling, or rerun the 1/3/5 feature-set stage.

docs/OFFLINE_STUDY.md

@@ -93,3 +93,22 @@ python -m experiments.run_all --resume --activation-batch-size 8 --activation-ma
 ```
 
 These two activation flags only affect memory/time during activation collection.
+
+## v0.16 causal-position sensitivity
+
+The final study reports two single-feature causal policies rather than conflating concept predictiveness with one arbitrary patch position:
+
+- `final_token`: the original causal baseline;
+- `max_feature_activation`: patch the selected feature where its SAE activation is maximal within the prompt.
+
+The max-active token is selected before intervention from feature activation only. The behavioral target never participates in position selection.
+
+Primary paired inference uses the **causal task** as the unit: ablation and amplification are aggregated within task before bootstrap confidence intervals and sign-flip tests. The report also separates feature coverage from conditional-on-active effect strength.
+
+A completed v0.15 run can be upgraded without repeating discovery/model-activation collection:
+
+```bash
+python -m experiments.run_causal_addendum --resume
+```
+
+See [`CAUSAL_ADDENDUM.md`](CAUSAL_ADDENDUM.md).

docs/VALIDATION.md

@@ -1,83 +1,36 @@
-# FeatureLens v0.15 validation
+# FeatureLens v0.16 validation
 
-v0.15 is a **public-design and offline-runner hardening release**. It does not change the already validated live Qwen/SAE inference methods. Do not spend ZeroGPU quota rerunning causal, paraphrase, layer, cue, discovery, or feature-set experiments for this release.
+v0.16 changes the **offline causal methodology**, not the already validated live HF inference UI. Do not spend ZeroGPU quota retesting live Workbench paths.
 
 ## Local software gate
 
-Run from the repository root:
-
 ```bash
-python3 -m pytest -q && \
-python3 -m compileall -q app.py featurelens experiments scripts && \
-python3 -m ruff check app.py featurelens experiments tests scripts && \
-python3 scripts/ui_smoke.py && \
+python3 -m pytest -q
+python3 -m compileall -q app.py featurelens experiments scripts
+python3 -m ruff check app.py featurelens experiments tests scripts
+python3 scripts/ui_smoke.py
 python3 scripts/release_check.py
 ```
 
-Then validate the Colab notebook JSON:
-
-```bash
-python3 - <<'PY'
-import nbformat
-nb = nbformat.read('notebooks/FeatureLens_Offline_Study_Colab.ipynb', as_version=4)
-nbformat.validate(nb)
-print('Colab notebook: PASS')
-PY
-```
-
-## Hugging Face acceptance — no GPU calls
-
-After pushing, only inspect the rendered interface.
-
-### A. Header and navigation
-
-Pass when:
-
-- the header shows **FeatureLens** and one factual subtitle;
-- there is no visible release/version badge;
-- tabs read **Guide, Workbench, Feature sets, Features, Paraphrases, Layers, Study, Method**;
-- tabs are visually flat rather than pill/card navigation.
-
-### B. Guide
-
-Open **Guide**.
-
-Pass when:
-
-- there is no three-card “step 1 / step 2 / step 3” onboarding grid;
-- the workflow is short prose;
-- headings use the serif display face while controls/body copy use the neutral sans-serif face;
-- no gradients, glow, badge clusters, or decorative cards are visible.
-
-### C. Workbench without running inference
-
-Open **Workbench**.
-
-Pass when:
-
-- experiment sections have a clear typographic hierarchy;
-- related fields sit close together and separate experiments have more breathing room;
-- primary experiment buttons are compact muted-teal actions, not full-width desktop banners;
-- **Copy TSV** is visually secondary;
-- table titles are clearly larger than table body text;
-- no duplicate context cards appear inside the tab—the global **Context** line is the context source of truth.
-
-### D. Features tab without running inference
-
-Open **Features**.
-
-Pass when:
+## Addendum acceptance
 
-- discovery, triage, controlled comparison, cross-target profile, and feature diagnostics read as sections of one tool rather than nested cards;
-- helper copy is short and does not repeatedly restate “not a semantic label” after every empty result;
-- the page remains usable in both normal desktop width and a narrower browser window.
+Use `notebooks/FeatureLens_Causal_Addendum_Colab.ipynb` with the completed v0.15 Drive run.
 
-### E. Study empty state
+The addendum is successful when:
 
-Open **Study** before real artifacts are committed.
+1. `causal_results_final_token.csv` exists and preserves the v0.15 baseline.
+2. `causal_results_max_active.csv` completes all 28 causal tasks with 8 random controls per intervention.
+3. `causal_position_summary.csv` contains both `final_token` and `max_feature_activation` policies.
+4. `study_summary.json` declares `max_feature_activation` as the primary causal position policy and causal-task-level inference as the statistical unit.
+5. `python -m scripts.validate_artifacts` prints `PASS`.
+6. `FeatureLens_offline_results_v016.zip` is created without activation caches.
 
-Pass when it clearly says the offline study is not materialized and does not show fabricated metrics or placeholder result plots.
+## HF acceptance after final artifacts are committed
 
-## Colab runner dry check
+No GPU call is required. Open **Study** and verify:
 
-Do not run the full model study merely to validate v0.15. Open the included notebook in Colab and execute only the first GPU-detection cell if desired. The full study should be started only when you are ready to produce the actual empirical artifacts.
+- the measured headline is populated;
+- final-token and max-active coverage/specificity are visible;
+- the causal-position table and figure render;
+- the association-vs-causality figure uses max-active specificity;
+- no placeholder or old v0.15 significance language remains.

experiments/analyze_study.py

@@ -11,6 +11,8 @@ from scipy.stats import spearmanr
 from experiments.common import ARTIFACT_DIR
 from featurelens.stats import paired_bootstrap_difference_ci, paired_sign_flip_pvalue
 
+POLICIES = ('final_token', 'max_feature_activation')
+
 
 def parse_args() -> argparse.Namespace:
     parser = argparse.ArgumentParser(
@@ -33,57 +35,111 @@ def selected_features(catalog: pd.DataFrame) -> pd.DataFrame:
     return ordered.groupby('concept', as_index=False).first()
 
 
-def _paired_specificity(
+def causal_path(artifact_dir: Path, policy: str) -> Path:
+    explicit = artifact_dir / f'causal_results_{policy if policy == "final_token" else "max_active"}.csv'
+    if explicit.exists():
+        return explicit
+    if policy == 'final_token':
+        legacy = artifact_dir / 'causal_results.csv'
+        if legacy.exists():
+            return legacy
+    return explicit
+
+
+def _task_level_specificity(
     frame: pd.DataFrame,
     *,
     effect_column: str,
-    task_column: str = 'task_id',
     seed: int,
+    active_only: bool = False,
 ) -> dict[str, float | list[float]]:
+    if frame.empty:
+        return _empty_specificity()
+
+    sae_rows = frame[frame['condition'] == 'sae_feature'].copy()
+    if active_only:
+        active_column = (
+            'feature_active_at_intervention'
+            if 'feature_active_at_intervention' in sae_rows.columns
+            else 'feature_activation'
+        )
+        if active_column == 'feature_activation':
+            active_tasks = sae_rows.loc[sae_rows[active_column].astype(float) > 0.0, 'task_id'].unique()
+        else:
+            active_tasks = sae_rows.loc[sae_rows[active_column].astype(float) > 0.0, 'task_id'].unique()
+        frame = frame[frame['task_id'].isin(active_tasks)].copy()
+        sae_rows = frame[frame['condition'] == 'sae_feature'].copy()
+
     sae = (
-        frame[frame['condition'] == 'sae_feature']
-        .groupby(task_column, as_index=False)[effect_column]
-        .first()
-        .rename(columns={effect_column: 'sae_effect'})
+        sae_rows.assign(
+            _abs_effect=lambda data: np.abs(pd.to_numeric(data[effect_column], errors='coerce'))
+        )
+        .groupby('task_id', as_index=False)
+        .agg(sae_abs_effect=('_abs_effect', 'mean'), sae_signed_effect=(effect_column, 'mean'))
     )
-    random = (
-        frame[frame['condition'] == 'random_norm_matched']
-        .assign(_abs_effect=lambda data: np.abs(data[effect_column].astype(float)))
-        .groupby(task_column, as_index=False)['_abs_effect']
-        .mean()
-        .rename(columns={'_abs_effect': 'random_abs_effect'})
+    random_rows = frame[frame['condition'] == 'random_norm_matched'].assign(
+        _abs_effect=lambda data: np.abs(pd.to_numeric(data[effect_column], errors='coerce'))
     )
-    paired = sae.merge(random, on=task_column, how='inner')
+    if 'intervention' in random_rows.columns:
+        random = (
+            random_rows.groupby(['task_id', 'intervention'], as_index=False)['_abs_effect']
+            .mean()
+            .groupby('task_id', as_index=False)['_abs_effect']
+            .mean()
+            .rename(columns={'_abs_effect': 'random_abs_effect'})
+        )
+    else:
+        random = (
+            random_rows.groupby('task_id', as_index=False)['_abs_effect']
+            .mean()
+            .rename(columns={'_abs_effect': 'random_abs_effect'})
+        )
+    paired = sae.merge(random, on='task_id', how='inner')
     if paired.empty:
-        return {
-            'sae_abs_mean': float('nan'),
-            'sae_signed_mean': float('nan'),
-            'random_abs_mean': float('nan'),
-            'specificity_ratio': float('nan'),
-            'paired_advantage': float('nan'),
-            'paired_advantage_ci_95': [float('nan'), float('nan')],
-            'paired_sign_flip_pvalue': float('nan'),
-            'n_tasks': 0,
-        }
+        return _empty_specificity()
 
-    sae_values = paired['sae_effect'].to_numpy(dtype=float)
-    sae_abs = np.abs(sae_values)
+    sae_abs = paired['sae_abs_effect'].to_numpy(dtype=float)
+    sae_signed = paired['sae_signed_effect'].to_numpy(dtype=float)
     random_abs = paired['random_abs_effect'].to_numpy(dtype=float)
     ci_low, ci_high = paired_bootstrap_difference_ci(sae_abs, random_abs, seed=seed)
     return {
         'sae_abs_mean': float(np.mean(sae_abs)),
-        'sae_signed_mean': float(np.mean(sae_values)),
+        'sae_signed_mean': float(np.mean(sae_signed)),
         'random_abs_mean': float(np.mean(random_abs)),
         'specificity_ratio': float(np.mean(sae_abs) / max(float(np.mean(random_abs)), 1e-12)),
         'paired_advantage': float(np.mean(sae_abs - random_abs)),
         'paired_advantage_ci_95': [float(ci_low), float(ci_high)],
-        'paired_sign_flip_pvalue': float(
-            paired_sign_flip_pvalue(sae_abs, random_abs, seed=seed + 1)
-        ),
+        'paired_sign_flip_pvalue': float(paired_sign_flip_pvalue(sae_abs, random_abs, seed=seed + 1)),
         'n_tasks': int(len(paired)),
     }
 
 
+def _empty_specificity() -> dict[str, float | list[float]]:
+    return {
+        'sae_abs_mean': float('nan'),
+        'sae_signed_mean': float('nan'),
+        'random_abs_mean': float('nan'),
+        'specificity_ratio': float('nan'),
+        'paired_advantage': float('nan'),
+        'paired_advantage_ci_95': [float('nan'), float('nan')],
+        'paired_sign_flip_pvalue': float('nan'),
+        'n_tasks': 0,
+    }
+
+
+def _paired_specificity(
+    frame: pd.DataFrame,
+    *,
+    effect_column: str,
+    task_column: str = 'task_id',
+    seed: int,
+) -> dict[str, float | list[float]]:
+    """Backward-compatible public helper using task-level inference."""
+    if task_column != 'task_id':
+        frame = frame.rename(columns={task_column: 'task_id'})
+    return _task_level_specificity(frame, effect_column=effect_column, seed=seed)
+
+
 def _safe_spearman(x: pd.Series, y: pd.Series) -> dict[str, float | int]:
     a = pd.to_numeric(x, errors='coerce').to_numpy(dtype=float)
     b = pd.to_numeric(y, errors='coerce').to_numpy(dtype=float)
@@ -93,10 +149,58 @@ def _safe_spearman(x: pd.Series, y: pd.Series) -> dict[str, float | int]:
     if np.unique(a[mask]).size < 2 or np.unique(b[mask]).size < 2:
         return {'rho': float('nan'), 'pvalue': float('nan'), 'n': int(mask.sum())}
     result = spearmanr(a[mask], b[mask])
+    return {'rho': float(result.statistic), 'pvalue': float(result.pvalue), 'n': int(mask.sum())}
+
+
+def _coverage(sae_rows: pd.DataFrame, column: str, fallback: str | None = None) -> float:
+    if column in sae_rows.columns:
+        return float(pd.to_numeric(sae_rows[column], errors='coerce').fillna(0).astype(float).gt(0).mean())
+    if fallback and fallback in sae_rows.columns:
+        return float(pd.to_numeric(sae_rows[fallback], errors='coerce').fillna(0).astype(float).gt(0).mean())
+    return float('nan')
+
+
+def _policy_summary(frame: pd.DataFrame, policy: str, seed: int) -> dict:
+    target = _task_level_specificity(frame, effect_column='target_mean_logprob_delta', seed=seed)
+    target_active = _task_level_specificity(
+        frame, effect_column='target_mean_logprob_delta', seed=seed + 7, active_only=True
+    )
+    js = _task_level_specificity(frame, effect_column='js_divergence', seed=seed + 17)
+    js_active = _task_level_specificity(
+        frame, effect_column='js_divergence', seed=seed + 29, active_only=True
+    )
+    sae_rows = frame[frame['condition'] == 'sae_feature']
     return {
-        'rho': float(result.statistic),
-        'pvalue': float(result.pvalue),
-        'n': int(mask.sum()),
+        'position_policy': policy,
+        'tasks': int(target['n_tasks']),
+        'feature_active_at_intervention_rate': _coverage(
+            sae_rows, 'feature_active_at_intervention', fallback='feature_activation'
+        ),
+        'feature_active_at_final_token_rate': _coverage(
+            sae_rows, 'feature_active_at_final_token', fallback='feature_activation'
+        ),
+        'feature_active_anywhere_rate': _coverage(
+            sae_rows, 'feature_active_anywhere', fallback='feature_activation'
+        ),
+        'target_sae_abs_mean': target['sae_abs_mean'],
+        'target_random_abs_mean': target['random_abs_mean'],
+        'target_specificity_ratio': target['specificity_ratio'],
+        'target_paired_advantage': target['paired_advantage'],
+        'target_paired_ci_low': float(target['paired_advantage_ci_95'][0]),
+        'target_paired_ci_high': float(target['paired_advantage_ci_95'][1]),
+        'target_sign_flip_pvalue': target['paired_sign_flip_pvalue'],
+        'active_target_sae_abs_mean': target_active['sae_abs_mean'],
+        'active_target_random_abs_mean': target_active['random_abs_mean'],
+        'active_target_specificity_ratio': target_active['specificity_ratio'],
+        'active_target_paired_advantage': target_active['paired_advantage'],
+        'active_target_paired_ci_low': float(target_active['paired_advantage_ci_95'][0]),
+        'active_target_paired_ci_high': float(target_active['paired_advantage_ci_95'][1]),
+        'active_target_sign_flip_pvalue': target_active['paired_sign_flip_pvalue'],
+        'active_tasks': int(target_active['n_tasks']),
+        'js_sae_mean': js['sae_abs_mean'],
+        'js_random_mean': js['random_abs_mean'],
+        'js_specificity_ratio': js['specificity_ratio'],
+        'active_js_specificity_ratio': js_active['specificity_ratio'],
     }
 
 
@@ -104,33 +208,47 @@ def main() -> None:
     args = parse_args()
     artifact_dir = args.artifact_dir
     catalog = pd.read_csv(artifact_dir / 'feature_catalog.csv')
-    causal = pd.read_csv(artifact_dir / 'causal_results.csv')
     paraphrase = pd.read_csv(artifact_dir / 'stability.csv')
     stability_path = artifact_dir / 'selection_stability.csv'
     selection_stability = pd.read_csv(stability_path) if stability_path.exists() else pd.DataFrame()
 
+    causal_by_policy: dict[str, pd.DataFrame] = {}
+    for policy in POLICIES:
+        path = causal_path(artifact_dir, policy)
+        if path.exists():
+            frame = pd.read_csv(path)
+            if 'position_policy' not in frame.columns:
+                frame['position_policy'] = policy
+            causal_by_policy[policy] = frame
+
+    if 'final_token' not in causal_by_policy or 'max_feature_activation' not in causal_by_policy:
+        missing = [policy for policy in POLICIES if policy not in causal_by_policy]
+        raise SystemExit(f'Missing causal position policies: {missing}')
+
     selected = selected_features(catalog)
-    rows: list[dict] = []
+    position_rows: list[dict] = []
+    for policy_idx, policy in enumerate(POLICIES):
+        overall = _policy_summary(causal_by_policy[policy], policy, args.seed + 1000 * policy_idx)
+        overall['concept'] = '__all__'
+        position_rows.append(overall)
+        for concept_idx, concept in enumerate(sorted(selected['concept'].astype(str).unique())):
+            subset = causal_by_policy[policy][causal_by_policy[policy]['concept'] == concept].copy()
+            row = _policy_summary(
+                subset,
+                policy,
+                args.seed + 1000 * policy_idx + 100 * (concept_idx + 1),
+            )
+            row['concept'] = concept
+            position_rows.append(row)
 
+    position_summary = pd.DataFrame(position_rows)
+    position_summary.to_csv(artifact_dir / 'causal_position_summary.csv', index=False)
+
+    rows: list[dict] = []
     for concept_idx, selected_row in selected.sort_values('concept').reset_index(drop=True).iterrows():
         concept = str(selected_row['concept'])
         layer = int(selected_row['layer'])
         feature_id = int(selected_row['feature_id'])
-        causal_concept = causal[
-            (causal['concept'] == concept) & (causal['intervention'] == 'ablate')
-        ].copy()
-        target = _paired_specificity(
-            causal_concept,
-            effect_column='target_mean_logprob_delta',
-            seed=args.seed + 100 * concept_idx,
-        )
-        js = _paired_specificity(
-            causal_concept,
-            effect_column='js_divergence',
-            seed=args.seed + 100 * concept_idx + 17,
-        )
-        sae_rows = causal_concept[causal_concept['condition'] == 'sae_feature']
-
         para_rows = paraphrase[
             (paraphrase['concept'] == concept) & (paraphrase['layer'].astype(int) == layer)
         ]
@@ -142,108 +260,120 @@ def main() -> None:
                 & (selection_stability['feature_id'].astype(int) == feature_id)
             ]
 
-        rows.append(
-            {
-                'concept': concept,
-                'layer': layer,
-                'feature_id': feature_id,
-                'train_auroc': float(selected_row['train_auroc']),
-                'heldout_auroc': float(selected_row['auroc']),
-                'heldout_f1': float(selected_row['f1']),
-                'activation_rate_pos_train': float(selected_row['activation_rate_pos']),
-                'activation_rate_neg_train': float(selected_row['activation_rate_neg']),
-                'candidate_resample_support': (
-                    float(selection_row.iloc[0]['resample_support'])
-                    if not selection_row.empty
-                    else 0.0
-                ),
-                'candidate_median_resample_rank': (
-                    float(selection_row.iloc[0]['median_resample_rank'])
-                    if not selection_row.empty
-                    else float('nan')
-                ),
-                'mean_paraphrase_topk_jaccard': (
-                    float(para_rows['topk_jaccard'].mean()) if not para_rows.empty else float('nan')
-                ),
-                'mean_paraphrase_sparse_cosine': (
-                    float(para_rows['sparse_cosine'].mean()) if not para_rows.empty else float('nan')
-                ),
-                'causal_feature_active_rate': (
-                    float(np.mean(sae_rows['feature_activation'].astype(float) > 0.0))
-                    if not sae_rows.empty
-                    else float('nan')
-                ),
-                'target_sae_abs_mean': target['sae_abs_mean'],
-                'target_sae_signed_mean': target['sae_signed_mean'],
-                'target_random_abs_mean': target['random_abs_mean'],
-                'target_specificity_ratio': target['specificity_ratio'],
-                'target_paired_advantage': target['paired_advantage'],
-                'target_paired_ci_low': float(target['paired_advantage_ci_95'][0]),
-                'target_paired_ci_high': float(target['paired_advantage_ci_95'][1]),
-                'target_sign_flip_pvalue': target['paired_sign_flip_pvalue'],
-                'js_sae_mean': js['sae_abs_mean'],
-                'js_random_mean': js['random_abs_mean'],
-                'js_specificity_ratio': js['specificity_ratio'],
-                'js_paired_advantage': js['paired_advantage'],
-                'js_paired_ci_low': float(js['paired_advantage_ci_95'][0]),
-                'js_paired_ci_high': float(js['paired_advantage_ci_95'][1]),
-                'js_sign_flip_pvalue': js['paired_sign_flip_pvalue'],
-                'causal_tasks': int(target['n_tasks']),
-            }
+        base = {
+            'concept': concept,
+            'layer': layer,
+            'feature_id': feature_id,
+            'train_auroc': float(selected_row['train_auroc']),
+            'heldout_auroc': float(selected_row['auroc']),
+            'heldout_f1': float(selected_row['f1']),
+            'activation_rate_pos_train': float(selected_row['activation_rate_pos']),
+            'activation_rate_neg_train': float(selected_row['activation_rate_neg']),
+            'candidate_resample_support': (
+                float(selection_row.iloc[0]['resample_support']) if not selection_row.empty else 0.0
+            ),
+            'candidate_median_resample_rank': (
+                float(selection_row.iloc[0]['median_resample_rank'])
+                if not selection_row.empty
+                else float('nan')
+            ),
+            'mean_paraphrase_topk_jaccard': (
+                float(para_rows['topk_jaccard'].mean()) if not para_rows.empty else float('nan')
+            ),
+            'mean_paraphrase_sparse_cosine': (
+                float(para_rows['sparse_cosine'].mean()) if not para_rows.empty else float('nan')
+            ),
+        }
+        for policy in POLICIES:
+            policy_frame = causal_by_policy[policy]
+            subset = policy_frame[policy_frame['concept'] == concept].copy()
+            summary = _policy_summary(
+                subset,
+                policy,
+                args.seed + 10_000 + 1000 * POLICIES.index(policy) + 100 * concept_idx,
+            )
+            prefix = 'final' if policy == 'final_token' else 'max_active'
+            for key, value in summary.items():
+                if key in {'position_policy'}:
+                    continue
+                base[f'{prefix}_{key}'] = value
+        base['target_specificity_gain_max_vs_final'] = (
+            float(base['max_active_target_specificity_ratio'])
+            - float(base['final_target_specificity_ratio'])
         )
+        base['js_specificity_gain_max_vs_final'] = (
+            float(base['max_active_js_specificity_ratio'])
+            - float(base['final_js_specificity_ratio'])
+        )
+        rows.append(base)
 
     study = pd.DataFrame(rows)
     study_path = artifact_dir / 'study_feature_summary.csv'
     study.to_csv(study_path, index=False)
 
     correlations = {
-        'heldout_auroc_vs_target_specificity': _safe_spearman(
-            study['heldout_auroc'], study['target_specificity_ratio']
+        'heldout_auroc_vs_max_active_target_specificity': _safe_spearman(
+            study['heldout_auroc'], study['max_active_target_specificity_ratio']
         ),
-        'heldout_auroc_vs_js_specificity': _safe_spearman(
-            study['heldout_auroc'], study['js_specificity_ratio']
+        'heldout_auroc_vs_max_active_js_specificity': _safe_spearman(
+            study['heldout_auroc'], study['max_active_js_specificity_ratio']
         ),
-        'heldout_f1_vs_target_specificity': _safe_spearman(
-            study['heldout_f1'], study['target_specificity_ratio']
+        'heldout_f1_vs_max_active_target_specificity': _safe_spearman(
+            study['heldout_f1'], study['max_active_target_specificity_ratio']
         ),
-        'candidate_resample_support_vs_target_specificity': _safe_spearman(
-            study['candidate_resample_support'], study['target_specificity_ratio']
+        'candidate_resample_support_vs_max_active_target_specificity': _safe_spearman(
+            study['candidate_resample_support'], study['max_active_target_specificity_ratio']
         ),
     }
 
+    overall = position_summary[position_summary['concept'] == '__all__'].set_index('position_policy')
+    final_row = overall.loc['final_token']
+    max_row = overall.loc['max_feature_activation']
     most_predictive = study.sort_values('heldout_auroc', ascending=False).iloc[0]
-    most_target_specific = study.sort_values('target_specificity_ratio', ascending=False).iloc[0]
-    most_js_specific = study.sort_values('js_specificity_ratio', ascending=False).iloc[0]
-    median_support = float(study['candidate_resample_support'].median())
+    most_target_specific = study.sort_values('max_active_target_specificity_ratio', ascending=False).iloc[0]
+    most_js_specific = study.sort_values('max_active_js_specificity_ratio', ascending=False).iloc[0]
 
     summary = {
         'n_concepts': int(len(study)),
         'selected_feature_pooling': 'prompt-wide max SAE activation across non-padding prompt tokens',
         'dense_probe_pooling': 'final prompt token residual',
-        'median_selected_feature_resample_support': median_support,
+        'primary_causal_position_policy': 'max_feature_activation',
+        'causal_statistical_unit': 'causal task; ablation and amplification are averaged within task before paired inference',
+        'median_selected_feature_resample_support': float(study['candidate_resample_support'].median()),
+        'final_token_feature_coverage': float(final_row['feature_active_at_intervention_rate']),
+        'max_active_feature_coverage': float(max_row['feature_active_at_intervention_rate']),
+        'final_token_target_specificity_ratio': float(final_row['target_specificity_ratio']),
+        'max_active_target_specificity_ratio': float(max_row['target_specificity_ratio']),
+        'final_token_target_paired_advantage': float(final_row['target_paired_advantage']),
+        'max_active_target_paired_advantage': float(max_row['target_paired_advantage']),
+        'final_token_target_paired_ci_95': [
+            float(final_row['target_paired_ci_low']), float(final_row['target_paired_ci_high'])
+        ],
+        'max_active_target_paired_ci_95': [
+            float(max_row['target_paired_ci_low']), float(max_row['target_paired_ci_high'])
+        ],
+        'final_token_target_sign_flip_pvalue': float(final_row['target_sign_flip_pvalue']),
+        'max_active_target_sign_flip_pvalue': float(max_row['target_sign_flip_pvalue']),
         'most_predictive_concept': {
             'concept': str(most_predictive['concept']),
             'heldout_auroc': float(most_predictive['heldout_auroc']),
         },
-        'highest_target_specificity': {
+        'highest_max_active_target_specificity': {
             'concept': str(most_target_specific['concept']),
-            'ratio': float(most_target_specific['target_specificity_ratio']),
+            'ratio': float(most_target_specific['max_active_target_specificity_ratio']),
         },
-        'highest_js_specificity': {
+        'highest_max_active_js_specificity': {
             'concept': str(most_js_specific['concept']),
-            'ratio': float(most_js_specific['js_specificity_ratio']),
+            'ratio': float(most_js_specific['max_active_js_specificity_ratio']),
         },
         'correlations': correlations,
         'guardrail': (
-            'Cross-concept Spearman correlations are descriptive because the study contains only seven '
-            'controlled concepts. Causal specificity remains evaluated per concept against matched random '
-            'controls rather than inferred from correlation alone.'
+            'Max-active causal positions are selected from SAE activation only, never from behavioral outcome. '
+            'Cross-concept Spearman correlations are descriptive because the study contains seven concepts.'
         ),
     }
-    (artifact_dir / 'study_summary.json').write_text(
-        json.dumps(summary, indent=2),
-        encoding='utf-8',
-    )
+    (artifact_dir / 'study_summary.json').write_text(json.dumps(summary, indent=2), encoding='utf-8')
+    print(f'Wrote {artifact_dir / "causal_position_summary.csv"}')
     print(f'Wrote {study_path}')
     print(f'Wrote {artifact_dir / "study_summary.json"}')
 

experiments/make_report.py

@@ -17,136 +17,130 @@ from featurelens.stats import (
 
 
 def parse_args() -> argparse.Namespace:
-    parser = argparse.ArgumentParser(description='Build a truthful experiment report from saved metrics.')
-    parser.add_argument('--artifact-dir', type=Path, default=ARTIFACT_DIR)
+    parser = argparse.ArgumentParser(
+        description="Build a truthful experiment report from saved metrics."
+    )
+    parser.add_argument("--artifact-dir", type=Path, default=ARTIFACT_DIR)
     return parser.parse_args()
 
 
 def _selected_features(catalog: pd.DataFrame) -> pd.DataFrame:
     scored = catalog.copy()
-    scored['activation_contrast'] = scored['activation_rate_pos'] - scored['activation_rate_neg']
+    scored["activation_contrast"] = (
+        scored["activation_rate_pos"] - scored["activation_rate_neg"]
+    )
     ordered = scored.sort_values(
-        ['concept', 'train_auroc', 'activation_contrast'],
+        ["concept", "train_auroc", "activation_contrast"],
         ascending=[True, False, False],
     )
-    return ordered.groupby('concept', as_index=False).first()
+    return ordered.groupby("concept", as_index=False).first()
 
 
 def _effect_column(frame: pd.DataFrame) -> str:
-    """Prefer the full-continuation length-normalized metric, with legacy fallback."""
-    if 'target_mean_logprob_delta' in frame.columns:
-        return 'target_mean_logprob_delta'
-    return 'target_logprob_delta'
+    if "target_mean_logprob_delta" in frame.columns:
+        return "target_mean_logprob_delta"
+    return "target_logprob_delta"
 
 
-def _save_plots(
-    artifact_dir: Path,
-    selected: pd.DataFrame,
-    layers: pd.DataFrame,
-    causal: pd.DataFrame,
-    feature_sets: pd.DataFrame | None,
-) -> None:
-    fig_dir = artifact_dir / 'figures'
-    fig_dir.mkdir(parents=True, exist_ok=True)
+def _causal_file(artifact_dir: Path, policy: str) -> Path:
+    if policy == "final_token":
+        name = "causal_results_final_token.csv"
+    else:
+        name = "causal_results_max_active.csv"
 
-    figure = plt.figure(figsize=(7.5, 4.2))
-    ax = figure.add_subplot(111)
-    ordered = selected.sort_values('auroc')
-    ax.barh(ordered['concept'], ordered['auroc'])
-    ax.axvline(0.5, linewidth=1, linestyle='--')
-    ax.set_xlabel('Held-out AUROC')
-    ax.set_title('Selected SAE feature predictiveness')
-    figure.tight_layout()
-    figure.savefig(fig_dir / 'feature_auroc.png', dpi=160)
-    plt.close(figure)
-
-    figure = plt.figure(figsize=(7.0, 4.2))
-    ax = figure.add_subplot(111)
-    ax.plot(
-        layers['layer'],
-        layers['linear_probe_macro_auroc'],
-        marker='o',
-        label='Linear probe AUROC',
+    path = artifact_dir / name
+    if path.exists():
+        return path
+
+    legacy = artifact_dir / "causal_results.csv"
+    if policy == "final_token" and legacy.exists():
+        return legacy
+    return path
+
+
+def _task_level_stats(
+    frame: pd.DataFrame,
+    *,
+    seed: int,
+    active_only: bool = False,
+) -> dict:
+    metric = _effect_column(frame)
+    sae_rows = frame[frame["condition"] == "sae_feature"].copy()
+
+    if active_only:
+        if "feature_active_at_intervention" in frame.columns:
+            active_col = "feature_active_at_intervention"
+        else:
+            active_col = "feature_activation"
+
+        active_mask = (
+            pd.to_numeric(sae_rows[active_col], errors="coerce").fillna(0) > 0
+        )
+        active_ids = sae_rows.loc[active_mask, "task_id"].unique()
+        frame = frame[frame["task_id"].isin(active_ids)].copy()
+        sae_rows = frame[frame["condition"] == "sae_feature"].copy()
+
+    sae = (
+        sae_rows.assign(
+            _abs=lambda data: np.abs(
+                pd.to_numeric(data[metric], errors="coerce")
+            )
+        )
+        .groupby("task_id", as_index=False)
+        .agg(sae_abs=("_abs", "mean"))
     )
-    ax.plot(
-        layers['layer'],
-        layers['reconstruction_cosine'],
-        marker='o',
-        label='SAE reconstruction cosine',
+    random = (
+        frame[frame["condition"] == "random_norm_matched"]
+        .assign(
+            _abs=lambda data: np.abs(
+                pd.to_numeric(data[metric], errors="coerce")
+            )
+        )
+        .groupby(["task_id", "intervention"], as_index=False)["_abs"]
+        .mean()
+        .groupby("task_id", as_index=False)["_abs"]
+        .mean()
+        .rename(columns={"_abs": "random_abs"})
     )
-    ax.set_xlabel('Layer')
-    ax.set_ylim(0, 1.05)
-    ax.set_title('Layer-wise representation diagnostics')
-    ax.legend()
-    figure.tight_layout()
-    figure.savefig(fig_dir / 'layer_diagnostics.png', dpi=160)
-    plt.close(figure)
+    paired = sae.merge(random, on="task_id", how="inner")
 
-    causal_metric = _effect_column(causal)
-    grouped = (
-        causal.groupby(['intervention', 'condition'])[causal_metric]
-        .apply(lambda values: float(np.mean(np.abs(values))))
-        .reset_index(name='mean_abs_effect')
-    )
-    pivot = grouped.pivot(index='intervention', columns='condition', values='mean_abs_effect')
-    figure = plt.figure(figsize=(7.0, 4.2))
-    ax = figure.add_subplot(111)
-    pivot.plot(kind='bar', ax=ax)
-    ax.set_ylabel('Mean |Δ mean log p/token|' if causal_metric == 'target_mean_logprob_delta' else 'Mean |Δ log p(target)|')
-    ax.set_title('Single-feature SAE edits vs norm-matched controls')
-    ax.tick_params(axis='x', rotation=0)
-    figure.tight_layout()
-    figure.savefig(fig_dir / 'causal_effects.png', dpi=160)
-    plt.close(figure)
+    if paired.empty:
+        nan = float("nan")
+        return {
+            "sae_abs": nan,
+            "random_abs": nan,
+            "ratio": nan,
+            "advantage": nan,
+            "ci": [nan, nan],
+            "pvalue": nan,
+            "n_tasks": 0,
+        }
 
-    if feature_sets is not None and not feature_sets.empty:
-        set_metric = _effect_column(feature_sets)
-        grouped_sets = (
-            feature_sets.groupby(['set_size', 'condition'])[set_metric]
-            .apply(lambda values: float(np.mean(np.abs(values))))
-            .reset_index(name='mean_abs_effect')
-        )
-        set_pivot = grouped_sets.pivot(index='set_size', columns='condition', values='mean_abs_effect')
-        figure = plt.figure(figsize=(7.0, 4.2))
-        ax = figure.add_subplot(111)
-        set_pivot.plot(kind='line', marker='o', ax=ax)
-        ax.set_xlabel('Jointly ablated feature count')
-        ax.set_ylabel('Mean |Δ mean log p/token|' if set_metric == 'target_mean_logprob_delta' else 'Mean |Δ log p(target)|')
-        ax.set_title('Distributed feature-set causal effect')
-        figure.tight_layout()
-        figure.savefig(fig_dir / 'feature_set_effects.png', dpi=160)
-        plt.close(figure)
-
-
-
-def _save_study_plots(artifact_dir: Path, study: pd.DataFrame) -> None:
-    if study.empty:
-        return
-    fig_dir = artifact_dir / 'figures'
-    fig_dir.mkdir(parents=True, exist_ok=True)
+    sae_effect = paired["sae_abs"].to_numpy(float)
+    random_effect = paired["random_abs"].to_numpy(float)
+    low, high = paired_bootstrap_difference_ci(
+        sae_effect,
+        random_effect,
+        seed=seed,
+    )
+    return {
+        "sae_abs": float(sae_effect.mean()),
+        "random_abs": float(random_effect.mean()),
+        "ratio": float(
+            sae_effect.mean() / max(float(random_effect.mean()), 1e-12)
+        ),
+        "advantage": float((sae_effect - random_effect).mean()),
+        "ci": [float(low), float(high)],
+        "pvalue": float(
+            paired_sign_flip_pvalue(
+                sae_effect,
+                random_effect,
+                seed=seed + 1,
+            )
+        ),
+        "n_tasks": int(len(paired)),
+    }
 
-    figure = plt.figure(figsize=(7.2, 4.6))
-    ax = figure.add_subplot(111)
-    ax.scatter(study['heldout_auroc'], study['target_specificity_ratio'])
-    for row in study.itertuples():
-        ax.annotate(str(row.concept), (row.heldout_auroc, row.target_specificity_ratio), fontsize=8)
-    ax.set_xlabel('Held-out feature AUROC')
-    ax.set_ylabel('Target causal specificity ratio')
-    ax.set_title('Association evidence vs random-normalized causality')
-    figure.tight_layout()
-    figure.savefig(fig_dir / 'association_vs_causality.png', dpi=160)
-    plt.close(figure)
-
-    figure = plt.figure(figsize=(7.4, 4.6))
-    ax = figure.add_subplot(111)
-    ordered = study.sort_values('candidate_resample_support')
-    ax.barh(ordered['concept'], ordered['candidate_resample_support'])
-    ax.set_xlim(0, 1.02)
-    ax.set_xlabel('Selection support across activation resamples')
-    ax.set_title('Selected-feature candidate stability')
-    figure.tight_layout()
-    figure.savefig(fig_dir / 'candidate_stability.png', dpi=160)
-    plt.close(figure)
 
 def _paired_stats(
     frame: pd.DataFrame,
@@ -156,301 +150,625 @@ def _paired_stats(
     random_condition: str,
     seed: int,
 ) -> dict[str, float | list[float]]:
-    """Pair one SAE effect with the mean absolute effect of its random-control ensemble."""
+    """Legacy helper retained for report-control regression tests."""
     metric = _effect_column(frame)
     sae = (
-        frame[frame['condition'] == sae_condition]
+        frame[frame["condition"] == sae_condition]
         .groupby(index, as_index=False)[metric]
         .first()
-        .rename(columns={metric: 'sae_effect'})
+        .rename(columns={metric: "sae_effect"})
     )
     random = (
-        frame[frame['condition'] == random_condition]
-        .assign(_abs_effect=lambda data: np.abs(data[metric].astype(float)))
-        .groupby(index, as_index=False)['_abs_effect']
+        frame[frame["condition"] == random_condition]
+        .assign(
+            _abs_effect=lambda data: np.abs(
+                pd.to_numeric(data[metric], errors="coerce")
+            )
+        )
+        .groupby(index, as_index=False)["_abs_effect"]
         .mean()
-        .rename(columns={'_abs_effect': 'random_abs_effect'})
+        .rename(columns={"_abs_effect": "random_abs_effect"})
     )
-    paired = sae.merge(random, on=index, how='inner')
-    sae_abs = np.abs(paired['sae_effect'].to_numpy(dtype=float))
-    random_abs = paired['random_abs_effect'].to_numpy(dtype=float)
-    if sae_abs.size == 0:
+    paired = sae.merge(random, on=index, how="inner")
+    sae_effect = np.abs(paired["sae_effect"].to_numpy(dtype=float))
+    random_effect = paired["random_abs_effect"].to_numpy(dtype=float)
+
+    if sae_effect.size == 0:
+        nan = float("nan")
         return {
-            'sae_abs': float('nan'),
-            'random_abs': float('nan'),
-            'ratio': float('nan'),
-            'paired_advantage': float('nan'),
-            'ci': [float('nan'), float('nan')],
-            'pvalue': float('nan'),
-            'n_pairs': 0,
+            "sae_abs": nan,
+            "random_abs": nan,
+            "ratio": nan,
+            "paired_advantage": nan,
+            "ci": [nan, nan],
+            "pvalue": nan,
+            "n_pairs": 0,
         }
-    diff = sae_abs - random_abs
-    low, high = paired_bootstrap_difference_ci(sae_abs, random_abs, seed=seed)
+
+    low, high = paired_bootstrap_difference_ci(
+        sae_effect,
+        random_effect,
+        seed=seed,
+    )
     return {
-        'sae_abs': float(np.mean(sae_abs)),
-        'random_abs': float(np.mean(random_abs)),
-        'ratio': float(np.mean(sae_abs) / max(float(np.mean(random_abs)), 1e-12)),
-        'paired_advantage': float(np.mean(diff)),
-        'ci': [float(low), float(high)],
-        'pvalue': float(paired_sign_flip_pvalue(sae_abs, random_abs, seed=seed + 1)),
-        'n_pairs': int(sae_abs.size),
+        "sae_abs": float(sae_effect.mean()),
+        "random_abs": float(random_effect.mean()),
+        "ratio": float(
+            sae_effect.mean() / max(float(random_effect.mean()), 1e-12)
+        ),
+        "paired_advantage": float((sae_effect - random_effect).mean()),
+        "ci": [float(low), float(high)],
+        "pvalue": float(
+            paired_sign_flip_pvalue(
+                sae_effect,
+                random_effect,
+                seed=seed + 1,
+            )
+        ),
+        "n_pairs": int(sae_effect.size),
     }
 
 
-def main() -> None:
-    args = parse_args()
-    catalog = pd.read_csv(args.artifact_dir / 'feature_catalog.csv')
-    layers = pd.read_csv(args.artifact_dir / 'layer_metrics.csv')
-    stability = pd.read_csv(args.artifact_dir / 'stability.csv')
-    causal = pd.read_csv(args.artifact_dir / 'causal_results.csv')
-    feature_set_path = args.artifact_dir / 'feature_set_results.csv'
-    feature_sets = pd.read_csv(feature_set_path) if feature_set_path.exists() else None
-    study_path = args.artifact_dir / 'study_feature_summary.csv'
-    study = pd.read_csv(study_path) if study_path.exists() else pd.DataFrame()
-    study_summary_path = args.artifact_dir / 'study_summary.json'
-    study_summary = (
-        json.loads(study_summary_path.read_text(encoding='utf-8'))
-        if study_summary_path.exists()
-        else {}
+def _feature_set_stats(frame: pd.DataFrame, *, seed: int) -> dict:
+    metric = _effect_column(frame)
+    sae = (
+        frame[frame["condition"] == "sae_feature_set"]
+        .groupby("task_id", as_index=False)[metric]
+        .first()
+        .rename(columns={metric: "sae"})
     )
-    selected = _selected_features(catalog)
-    _save_plots(args.artifact_dir, selected, layers, causal, feature_sets)
-    _save_study_plots(args.artifact_dir, study)
-
-    mean_auc = float(selected['auroc'].mean())
-    median_auc = float(selected['auroc'].median())
-    auc_ci_low, auc_ci_high = bootstrap_mean_ci(selected['auroc'].to_numpy(), seed=42)
-    best_layer_row = layers.sort_values('linear_probe_macro_auroc', ascending=False).iloc[0]
-    mean_jaccard = float(stability['topk_jaccard'].mean())
-    mean_sparse_cos = float(stability['sparse_cosine'].mean())
-
-    single = _paired_stats(
-        causal,
-        index=['task_id', 'intervention'],
-        sae_condition='sae_feature',
-        random_condition='random_norm_matched',
-        seed=43,
-    )
-    sae = causal[causal['condition'] == 'sae_feature']
-    active_rate = float(np.mean(sae['feature_activation'] > 0))
-    top1_change = float(sae['top1_changed'].mean())
-
-    set_summary: dict[int, dict[str, float | list[float]]] = {}
-    largest_set: dict[str, float | list[float]] | None = None
-    largest_k: int | None = None
-    if feature_sets is not None and not feature_sets.empty:
-        for size in sorted(int(x) for x in feature_sets['set_size'].unique()):
-            subset = feature_sets[feature_sets['set_size'] == size]
-            set_summary[size] = _paired_stats(
-                subset,
-                index=['task_id', 'set_size'],
-                sae_condition='sae_feature_set',
-                random_condition='random_norm_matched',
-                seed=100 + size,
+    random = (
+        frame[frame["condition"] == "random_norm_matched"]
+        .assign(
+            _abs=lambda data: np.abs(
+                pd.to_numeric(data[metric], errors="coerce")
             )
-        largest_k = max(set_summary)
-        largest_set = set_summary[largest_k]
+        )
+        .groupby("task_id", as_index=False)["_abs"]
+        .mean()
+        .rename(columns={"_abs": "random"})
+    )
+    paired = sae.merge(random, on="task_id")
+    sae_effect = np.abs(paired["sae"].to_numpy(float))
+    random_effect = paired["random"].to_numpy(float)
 
-    sae_abs = float(single['sae_abs'])
-    random_abs = float(single['random_abs'])
-    ratio = float(single['ratio'])
+    if sae_effect.size == 0:
+        return {}
 
-    single_ci_low = float(single['ci'][0])
-    single_p = float(single['pvalue'])
-    single_specific = ratio >= 1.5 and single_ci_low > 0.0 and single_p < 0.05
+    low, high = paired_bootstrap_difference_ci(
+        sae_effect,
+        random_effect,
+        seed=seed,
+    )
+    return {
+        "sae_abs": float(sae_effect.mean()),
+        "random_abs": float(random_effect.mean()),
+        "ratio": float(
+            sae_effect.mean() / max(float(random_effect.mean()), 1e-12)
+        ),
+        "advantage": float((sae_effect - random_effect).mean()),
+        "ci": [float(low), float(high)],
+        "pvalue": float(
+            paired_sign_flip_pvalue(
+                sae_effect,
+                random_effect,
+                seed=seed + 1,
+            )
+        ),
+        "n_tasks": int(sae_effect.size),
+    }
 
-    if mean_auc >= 0.8 and sae_abs < 0.08:
-        interpretation = (
-            'The selected sparse features were strongly predictive on held-out prompts, but single-feature '
-            'interventions produced only modest downstream changes. FeatureLens therefore treats the '
-            'representation-level signal as correlational rather than automatically causal.'
+
+def _save_figure(fig: plt.Figure, path: Path) -> None:
+    fig.tight_layout()
+    fig.savefig(path, dpi=160)
+    plt.close(fig)
+
+
+def _save_plots(
+    artifact_dir: Path,
+    selected: pd.DataFrame,
+    layers: pd.DataFrame,
+    max_active: pd.DataFrame,
+    feature_sets: pd.DataFrame | None,
+    position: pd.DataFrame,
+    study: pd.DataFrame,
+) -> None:
+    fig_dir = artifact_dir / "figures"
+    fig_dir.mkdir(parents=True, exist_ok=True)
+
+    fig = plt.figure(figsize=(7.5, 4.2))
+    ax = fig.add_subplot(111)
+    ordered = selected.sort_values("auroc")
+    ax.barh(ordered["concept"], ordered["auroc"])
+    ax.axvline(0.5, linewidth=1, linestyle="--")
+    ax.set_xlabel("Held-out AUROC")
+    ax.set_title("Selected SAE feature predictiveness")
+    _save_figure(fig, fig_dir / "feature_auroc.png")
+
+    fig = plt.figure(figsize=(7.0, 4.2))
+    ax = fig.add_subplot(111)
+    ax.plot(
+        layers["layer"],
+        layers["linear_probe_macro_auroc"],
+        marker="o",
+        label="Linear probe AUROC",
+    )
+    ax.plot(
+        layers["layer"],
+        layers["reconstruction_cosine"],
+        marker="o",
+        label="SAE reconstruction cosine",
+    )
+    ax.set_xlabel("Layer")
+    ax.set_ylim(0, 1.05)
+    ax.set_title("Layer-wise representation diagnostics")
+    ax.legend()
+    _save_figure(fig, fig_dir / "layer_diagnostics.png")
+
+    metric = _effect_column(max_active)
+    grouped = (
+        max_active.groupby(["intervention", "condition"])[metric]
+        .apply(lambda values: float(np.mean(np.abs(values))))
+        .reset_index(name="mean_abs_effect")
+    )
+    pivot = grouped.pivot(
+        index="intervention",
+        columns="condition",
+        values="mean_abs_effect",
+    )
+    fig = plt.figure(figsize=(7, 4.2))
+    ax = fig.add_subplot(111)
+    pivot.plot(kind="bar", ax=ax)
+    ax.set_ylabel("Mean |Δ mean log p/token|")
+    ax.set_title("Max-active SAE edits vs norm-matched controls")
+    ax.tick_params(axis="x", rotation=0)
+    _save_figure(fig, fig_dir / "causal_effects.png")
+
+    if feature_sets is not None and not feature_sets.empty:
+        feature_set_metric = _effect_column(feature_sets)
+        grouped_sets = (
+            feature_sets.groupby(["set_size", "condition"])[feature_set_metric]
+            .apply(lambda values: float(np.mean(np.abs(values))))
+            .reset_index(name="mean_abs_effect")
         )
-    elif mean_auc >= 0.8 and sae_abs >= 0.08 and single_specific:
-        interpretation = (
-            'The selected sparse features were strongly predictive and single-feature interventions produced '
-            'larger target-continuation shifts than norm-matched random residual perturbations. The paired '
-            'bootstrap interval excludes zero and the sign-flip test passes the configured 0.05 threshold, '
-            'supporting a causal-specificity claim for at least some predictive features.'
+        set_pivot = grouped_sets.pivot(
+            index="set_size",
+            columns="condition",
+            values="mean_abs_effect",
+        )
+        fig = plt.figure(figsize=(7, 4.2))
+        ax = fig.add_subplot(111)
+        set_pivot.plot(kind="line", marker="o", ax=ax)
+        ax.set_xlabel("Jointly ablated feature count")
+        ax.set_ylabel("Mean |Δ mean log p/token|")
+        ax.set_title("Final-token feature-set diagnostic")
+        _save_figure(fig, fig_dir / "feature_set_effects.png")
+
+    overall = position[position["concept"] == "__all__"].copy()
+    policy_order = ["final_token", "max_feature_activation"]
+    overall["position_policy"] = pd.Categorical(
+        overall["position_policy"],
+        categories=policy_order,
+        ordered=True,
+    )
+    overall = overall.sort_values("position_policy")
+    position_plot = pd.DataFrame(
+        {
+            "Policy": ["Final token", "Max feature activation"],
+            "SAE effect": overall["target_sae_abs_mean"].to_numpy(float),
+            "Random control": overall["target_random_abs_mean"].to_numpy(float),
+        }
+    )
+    fig = plt.figure(figsize=(7.2, 4.4))
+    ax = fig.add_subplot(111)
+    x_positions = np.arange(len(position_plot))
+    width = 0.34
+    ax.bar(
+        x_positions - width / 2,
+        position_plot["SAE effect"],
+        width,
+        label="SAE effect",
+    )
+    ax.bar(
+        x_positions + width / 2,
+        position_plot["Random control"],
+        width,
+        label="Random control",
+    )
+    ax.set_xticks(x_positions, position_plot["Policy"])
+    ax.set_ylabel("Task-level mean |Δ mean log p/token|")
+    ax.set_title("Causal position sensitivity")
+    ax.legend()
+    _save_figure(fig, fig_dir / "causal_position_sensitivity.png")
+
+    if not study.empty:
+        fig = plt.figure(figsize=(7.2, 4.6))
+        ax = fig.add_subplot(111)
+        ax.scatter(
+            study["heldout_auroc"],
+            study["max_active_target_specificity_ratio"],
         )
-    elif mean_auc >= 0.8 and sae_abs >= 0.08 and ratio >= 1.5:
+        for row in study.itertuples():
+            ax.annotate(
+                str(row.concept),
+                (row.heldout_auroc, row.max_active_target_specificity_ratio),
+                fontsize=8,
+            )
+        ax.set_xlabel("Held-out feature AUROC")
+        ax.set_ylabel("Max-active target specificity ratio")
+        ax.set_title("Association evidence vs max-active causality")
+        _save_figure(fig, fig_dir / "association_vs_causality.png")
+
+
+def _coverage(
+    frame: pd.DataFrame,
+    column: str,
+    fallback: str = "feature_activation",
+) -> float:
+    name = column if column in frame.columns else fallback
+    values = pd.to_numeric(frame[name], errors="coerce").fillna(0)
+    return float((values > 0).mean())
+
+
+def _build_interpretation(
+    *,
+    max_stats: dict,
+    final_coverage: float,
+    max_coverage: float,
+) -> str:
+    strong = (
+        max_stats["ratio"] >= 1.5
+        and max_stats["ci"][0] > 0
+        and max_stats["pvalue"] < 0.05
+    )
+    if strong:
         interpretation = (
-            'The selected sparse features were strongly predictive and their point-estimate intervention '
-            'effects exceeded norm-matched random controls, but the paired uncertainty test does not support '
-            'a strong causal-specificity claim at the 0.05 threshold. The result is reported as suggestive '
-            'rather than conclusive.'
+            "Max-active interventions produced larger task-level target effects than "
+            "norm-matched random controls with paired uncertainty excluding zero. "
+            "Predictive SAE features therefore show causal specificity when intervened "
+            "where the selected feature is actually represented, while the final-token "
+            "baseline quantifies sensitivity to intervention location."
         )
-    elif mean_auc < 0.65:
+    elif max_stats["ratio"] >= 1.5:
         interpretation = (
-            'Feature/concept predictiveness was limited on held-out prompts, so strong causal claims would '
-            'be premature. The main result is diagnostic: concept design or feature selection should be '
-            'refined before interpreting intervention effects.'
+            "Max-active interventions had a larger point-estimate effect than norm-matched "
+            "random controls, but task-level paired uncertainty did not support a strong "
+            "significance claim. The result is therefore reported as suggestive causal "
+            "specificity rather than conclusive evidence."
         )
     else:
         interpretation = (
-            'The results show mixed predictive and causal evidence. FeatureLens reports association, '
-            'robustness, and intervention measurements separately rather than collapsing them into one score.'
+            "Held-out feature predictiveness was strong, but max-active causal effects were "
+            "only modest relative to norm-matched random controls. FeatureLens therefore "
+            "separates predictive association from causal control rather than treating them "
+            "as interchangeable."
         )
 
-    if largest_set is not None and largest_k is not None:
-        set_advantage = float(largest_set['paired_advantage'])
-        set_ci_low = float(largest_set['ci'][0])
-        set_p = float(largest_set['pvalue'])
-        set_specific = set_ci_low > 0.0 and set_p < 0.05
-        if set_advantage > float(single['paired_advantage']) + 0.02 and set_specific:
-            interpretation += (
-                f' Joint ablation of the top {largest_k} same-layer concept features produced a larger '
-                'paired advantage over random controls than the single-feature edits, with paired uncertainty '
-                'supporting the difference. This is consistent with causal influence being distributed across '
-                'a sparse feature set rather than concentrated in one unit.'
-            )
-        elif set_advantage > float(single['paired_advantage']) + 0.02:
-            interpretation += (
-                f' The top-{largest_k} joint-ablation point estimate exceeded the single-feature advantage, '
-                'but its paired uncertainty test does not support a strong distributed-causality claim at the '
-                '0.05 threshold. The pattern is therefore treated as suggestive only.'
-            )
-        elif abs(set_advantage) <= 0.02:
-            interpretation += (
-                f' Expanding the intervention to the top {largest_k} same-layer features did not materially '
-                'increase specificity over random controls, which argues against assuming that a broader '
-                'concept-associated sparse subspace is automatically more causal.'
+    if max_coverage > final_coverage + 0.1:
+        interpretation += (
+            " Moving from the final prompt token to the feature's maximum-activation token "
+            f"increased intervention coverage from {final_coverage:.1%} to "
+            f"{max_coverage:.1%}, showing that causal conclusions depend materially on "
+            "where the representation is tested."
+        )
+    return interpretation
+
+
+def _build_report_lines(
+    *,
+    headline: str,
+    interpretation: str,
+    highlights: list[str],
+    feature_sets: pd.DataFrame | None,
+    study: pd.DataFrame,
+) -> list[str]:
+    lines = [
+        "# FeatureLens experiment report",
+        "",
+        "## Research question",
+        "",
+        "**Do sparse features that predict a concept also causally influence model behaviour?**",
+        "",
+        "## Executive summary",
+        "",
+        headline,
+        "",
+        interpretation,
+        "",
+        "## Key measurements",
+        "",
+        *[f"- {item}" for item in highlights],
+        "",
+        "## Experimental design",
+        "",
+        "- Model: Qwen3-1.7B-Base.",
+        "- SAEs: Qwen-Scope residual-stream TopK SAEs at configured early/middle/late layers.",
+        "- Discovery evidence: prompt-wide maximum SAE activation across non-padding tokens; final-token activations are saved separately.",
+        "- Split discipline: paraphrase groups remain entirely in train or held-out test.",
+        "- Feature selection: training-split AUROC plus activation contrast; held-out AUROC/F1 are reported separately.",
+        "- Causal position policies: final prompt token and maximum selected-feature activation within the prompt. Max-active positions are selected from SAE activation only, never from behavioral outcomes.",
+        "- Primary causal statistical unit: causal task. Ablation and 2× amplification are averaged within task before paired bootstrap/sign-flip inference.",
+        "- Negative control: deterministic norm-matched random residual directions.",
+        "- Primary target metric: exact full continuation mean log probability per token under teacher forcing.",
+        "- Coverage and conditional-on-active effect strength are reported separately.",
+        "- Feature-set analysis remains a final-token diagnostic and is not conflated with the max-active single-feature study.",
+        "",
+        "## Figures",
+        "",
+        "![Feature AUROC](figures/feature_auroc.png)",
+        "",
+        "![Layer diagnostics](figures/layer_diagnostics.png)",
+        "",
+        "![Causal position sensitivity](figures/causal_position_sensitivity.png)",
+        "",
+        "![Max-active causal effects](figures/causal_effects.png)",
+    ]
+
+    if feature_sets is not None and not feature_sets.empty:
+        lines.extend(
+            [
+                "",
+                "![Feature-set diagnostic](figures/feature_set_effects.png)",
+            ]
+        )
+
+    if not study.empty:
+        lines.extend(
+            [
+                "",
+                "![Association vs causality](figures/association_vs_causality.png)",
+                "",
+                "## Position sensitivity",
+                "",
+                "The final-token policy asks whether the selected feature matters at the conventional last-prompt-token intervention site. The max-active policy asks whether it matters where that same feature is most strongly represented in the prompt. Reporting both prevents low final-token coverage from being mistaken for evidence that a predictive feature is globally non-causal.",
+                "",
+                "## Association vs causality across concepts",
+                "",
+                "Cross-concept correlations use max-active random-normalized specificity and are descriptive because the study has seven controlled concepts.",
+            ]
+        )
+
+    lines.extend(
+        [
+            "",
+            "## Interpretation guardrails",
+            "",
+            "High held-out AUROC is correlational evidence. Causal claims require downstream changes relative to norm-matched random controls. Max-active positions are chosen without reference to behavioral effect size. Task-level uncertainty treats ablation and amplification on the same causal prompt as repeated interventions, not independent experimental units.",
+            "",
+            "## Reproducibility",
+            "",
+            "Run `python -m experiments.run_all --resume` for a fresh full study. For an existing v0.15 final-token study, run the v0.16 causal addendum notebook; it preserves the baseline, computes only max-active causal rows, and reruns CPU analysis/reporting.",
+            "",
+        ]
+    )
+    return lines
+
+
+def main() -> None:
+    args = parse_args()
+    artifact_dir = args.artifact_dir
+
+    catalog = pd.read_csv(artifact_dir / "feature_catalog.csv")
+    layers = pd.read_csv(artifact_dir / "layer_metrics.csv")
+    stability = pd.read_csv(artifact_dir / "stability.csv")
+    final = pd.read_csv(_causal_file(artifact_dir, "final_token"))
+    max_active = pd.read_csv(
+        _causal_file(artifact_dir, "max_feature_activation")
+    )
+
+    feature_set_path = artifact_dir / "feature_set_results.csv"
+    feature_sets = (
+        pd.read_csv(feature_set_path) if feature_set_path.exists() else None
+    )
+
+    study_path = artifact_dir / "study_feature_summary.csv"
+    study = pd.read_csv(study_path) if study_path.exists() else pd.DataFrame()
+    position = pd.read_csv(artifact_dir / "causal_position_summary.csv")
+
+    study_summary_path = artifact_dir / "study_summary.json"
+    if study_summary_path.exists():
+        study_summary = json.loads(study_summary_path.read_text())
+    else:
+        study_summary = {}
+
+    selected = _selected_features(catalog)
+    _save_plots(
+        artifact_dir,
+        selected,
+        layers,
+        max_active,
+        feature_sets,
+        position,
+        study,
+    )
+
+    mean_auc = float(selected["auroc"].mean())
+    median_auc = float(selected["auroc"].median())
+    auc_low, auc_high = bootstrap_mean_ci(
+        selected["auroc"].to_numpy(),
+        seed=42,
+    )
+
+    best = layers.sort_values(
+        "linear_probe_macro_auroc",
+        ascending=False,
+    ).iloc[0]
+    mean_jaccard = float(stability["topk_jaccard"].mean())
+    mean_cosine = float(stability["sparse_cosine"].mean())
+
+    final_stats = _task_level_stats(final, seed=43)
+    final_active_stats = _task_level_stats(
+        final,
+        seed=44,
+        active_only=True,
+    )
+    max_stats = _task_level_stats(max_active, seed=45)
+    max_active_stats = _task_level_stats(
+        max_active,
+        seed=46,
+        active_only=True,
+    )
+
+    final_sae = final[final["condition"] == "sae_feature"]
+    max_sae = max_active[max_active["condition"] == "sae_feature"]
+    final_coverage = _coverage(
+        final_sae,
+        "feature_active_at_intervention",
+    )
+    anywhere_coverage = _coverage(
+        max_sae,
+        "feature_active_anywhere",
+    )
+    max_coverage = _coverage(
+        max_sae,
+        "feature_active_at_intervention",
+    )
+
+    set_summary: dict[int, dict] = {}
+    if feature_sets is not None and not feature_sets.empty:
+        sizes = sorted(int(value) for value in feature_sets["set_size"].unique())
+        for size in sizes:
+            subset = feature_sets[feature_sets["set_size"] == size]
+            set_summary[size] = _feature_set_stats(
+                subset,
+                seed=100 + size,
             )
 
-    effect_label = 'mean log p/token' if _effect_column(causal) == 'target_mean_logprob_delta' else 'target log-probability'
+    interpretation = _build_interpretation(
+        max_stats=max_stats,
+        final_coverage=final_coverage,
+        max_coverage=max_coverage,
+    )
+
     headline = (
-        f'Selected SAE features averaged {mean_auc:.3f} held-out AUROC; single-feature SAE interventions '
-        f'changed {effect_label} by {sae_abs:.3f} in absolute value on average versus {random_abs:.3f} '
-        'for norm-matched random residual controls.'
+        f"Selected SAE features averaged {mean_auc:.3f} held-out AUROC. "
+        f"Max-active interventions covered {max_coverage:.1%} of causal tasks and "
+        f"changed mean log p/token by {max_stats['sae_abs']:.3f} in absolute value "
+        f"on average versus {max_stats['random_abs']:.3f} for norm-matched random "
+        f"controls ({max_stats['ratio']:.2f}×)."
     )
+
     highlights = [
-        f'Median selected-feature held-out AUROC: {median_auc:.3f}; mean AUROC 95% bootstrap CI [{auc_ci_low:.3f}, {auc_ci_high:.3f}].',
-        f'Best residual linear-probe layer: {int(best_layer_row["layer"])} with macro AUROC {best_layer_row["linear_probe_macro_auroc"]:.3f}.',
-        f'Mean paraphrase TopK Jaccard: {mean_jaccard:.3f}; sparse activation cosine: {mean_sparse_cos:.3f}.',
-        f'Selected feature active on {active_rate:.1%} of causal prompts; modified next-token top-1 on {top1_change:.1%}.',
-        f'Single-feature mean absolute causal effect / random-control effect ratio: {ratio:.2f}×.',
-        f'Single-feature paired mean |effect| advantage over random: {float(single["paired_advantage"]):+.3f}, 95% bootstrap CI [{float(single["ci"][0]):+.3f}, {float(single["ci"][1]):+.3f}], sign-flip p={float(single["pvalue"]):.4f}.',
+        (
+            f"Median selected-feature held-out AUROC: {median_auc:.3f}; mean AUROC "
+            f"95% bootstrap CI [{auc_low:.3f}, {auc_high:.3f}]."
+        ),
+        (
+            f"Best residual linear-probe layer: {int(best['layer'])} with macro AUROC "
+            f"{float(best['linear_probe_macro_auroc']):.3f}."
+        ),
+        (
+            f"Mean paraphrase TopK Jaccard: {mean_jaccard:.3f}; sparse activation "
+            f"cosine: {mean_cosine:.3f}."
+        ),
+        (
+            f"Feature coverage: final-token policy {final_coverage:.1%}; active "
+            f"anywhere in prompt {anywhere_coverage:.1%}; max-active intervention "
+            f"{max_coverage:.1%}."
+        ),
+        (
+            f"Final-token task-level SAE/random ratio: {final_stats['ratio']:.2f}×; "
+            f"paired advantage {final_stats['advantage']:+.4f}, 95% CI "
+            f"[{final_stats['ci'][0]:+.4f}, {final_stats['ci'][1]:+.4f}], "
+            f"sign-flip p={final_stats['pvalue']:.4f}."
+        ),
+        (
+            f"Max-active task-level SAE/random ratio: {max_stats['ratio']:.2f}×; "
+            f"paired advantage {max_stats['advantage']:+.4f}, 95% CI "
+            f"[{max_stats['ci'][0]:+.4f}, {max_stats['ci'][1]:+.4f}], "
+            f"sign-flip p={max_stats['pvalue']:.4f}."
+        ),
+        (
+            "Conditional on feature-active tasks, max-active SAE/random ratio: "
+            f"{max_active_stats['ratio']:.2f}× "
+            f"(n={max_active_stats['n_tasks']})."
+        ),
     ]
-    if largest_set is not None and largest_k is not None:
+
+    if set_summary:
+        max_size = max(set_summary)
+        set_stats = set_summary[max_size]
         highlights.append(
-            f'Top-{largest_k} joint ablation: SAE/random mean absolute effect ratio {float(largest_set["ratio"]):.2f}×; '
-            f'paired advantage {float(largest_set["paired_advantage"]):+.3f}, 95% CI '
-            f'[{float(largest_set["ci"][0]):+.3f}, {float(largest_set["ci"][1]):+.3f}], '
-            f'sign-flip p={float(largest_set["pvalue"]):.4f}.'
+            f"Final-token top-{max_size} joint ablation SAE/random ratio: "
+            f"{set_stats['ratio']:.2f}×; paired advantage "
+            f"{set_stats['advantage']:+.4f}, 95% CI "
+            f"[{set_stats['ci'][0]:+.4f}, {set_stats['ci'][1]:+.4f}], "
+            f"sign-flip p={set_stats['pvalue']:.4f}."
         )
 
-    if study_summary:
-        correlations = study_summary.get('correlations', {})
-        assoc_target = correlations.get('heldout_auroc_vs_target_specificity', {})
-        assoc_js = correlations.get('heldout_auroc_vs_js_specificity', {})
+    correlations = study_summary.get("correlations", {})
+    target_corr = correlations.get(
+        "heldout_auroc_vs_max_active_target_specificity",
+        {},
+    )
+    js_corr = correlations.get(
+        "heldout_auroc_vs_max_active_js_specificity",
+        {},
+    )
+    if target_corr:
+        target_rho = float(target_corr.get("rho", float("nan")))
+        js_rho = float(js_corr.get("rho", float("nan")))
         highlights.extend(
             [
-                f'Selected-feature median activation-resample support: {float(study_summary.get("median_selected_feature_resample_support", float("nan"))):.1%}.',
-                f'Across concepts, held-out AUROC vs target-specificity Spearman ρ={float(assoc_target.get("rho", float("nan"))):+.3f} (n={int(assoc_target.get("n", 0))}); descriptive only.',
-                f'Across concepts, held-out AUROC vs JS-specificity Spearman ρ={float(assoc_js.get("rho", float("nan"))):+.3f} (n={int(assoc_js.get("n", 0))}); descriptive only.',
+                (
+                    "Across seven concepts, held-out AUROC vs max-active target "
+                    f"specificity Spearman ρ={target_rho:+.3f}; descriptive only."
+                ),
+                (
+                    "Held-out AUROC vs max-active JS specificity Spearman "
+                    f"ρ={js_rho:+.3f}; descriptive only."
+                ),
             ]
         )
 
     summary = {
-        'headline': headline,
-        'highlights': highlights,
-        'interpretation': interpretation,
-        'metrics': {
-            'mean_selected_feature_test_auroc': mean_auc,
-            'mean_selected_feature_test_auroc_bootstrap_ci_95': [auc_ci_low, auc_ci_high],
-            'median_selected_feature_test_auroc': median_auc,
-            'best_linear_probe_layer': int(best_layer_row['layer']),
-            'best_linear_probe_macro_auroc': float(best_layer_row['linear_probe_macro_auroc']),
-            'mean_paraphrase_topk_jaccard': mean_jaccard,
-            'mean_paraphrase_sparse_cosine': mean_sparse_cos,
-            'single_feature_effect_metric': _effect_column(causal),
-            'mean_abs_sae_effect': sae_abs,
-            'mean_abs_random_effect': random_abs,
-            'causal_to_random_effect_ratio': ratio,
-            'paired_mean_abs_effect_advantage': float(single['paired_advantage']),
-            'paired_mean_abs_effect_advantage_bootstrap_ci_95': single['ci'],
-            'paired_sign_flip_pvalue': float(single['pvalue']),
-            'causal_prompt_feature_active_rate': active_rate,
-            'sae_top1_change_rate': top1_change,
-            'feature_set_results': {str(k): value for k, value in set_summary.items()},
-            'study_summary': study_summary,
+        "headline": headline,
+        "highlights": highlights,
+        "interpretation": interpretation,
+        "metrics": {
+            "mean_selected_feature_test_auroc": mean_auc,
+            "mean_selected_feature_test_auroc_bootstrap_ci_95": [
+                auc_low,
+                auc_high,
+            ],
+            "median_selected_feature_test_auroc": median_auc,
+            "best_linear_probe_layer": int(best["layer"]),
+            "best_linear_probe_macro_auroc": float(
+                best["linear_probe_macro_auroc"]
+            ),
+            "mean_paraphrase_topk_jaccard": mean_jaccard,
+            "mean_paraphrase_sparse_cosine": mean_cosine,
+            "final_token_feature_coverage": final_coverage,
+            "prompt_anywhere_feature_coverage": anywhere_coverage,
+            "max_active_feature_coverage": max_coverage,
+            "final_token_task_level": final_stats,
+            "final_token_active_only": final_active_stats,
+            "max_active_task_level": max_stats,
+            "max_active_active_only": max_active_stats,
+            "feature_set_results": {
+                str(key): value for key, value in set_summary.items()
+            },
+            "study_summary": study_summary,
         },
     }
-    (args.artifact_dir / 'summary.json').write_text(json.dumps(summary, indent=2), encoding='utf-8')
+    (artifact_dir / "summary.json").write_text(
+        json.dumps(summary, indent=2),
+        encoding="utf-8",
+    )
 
-    lines = [
-        '# FeatureLens experiment report',
-        '',
-        '## Research question',
-        '',
-        '**Do sparse features that predict a concept also causally influence model behaviour?**',
-        '',
-        '## Executive summary',
-        '',
-        headline,
-        '',
-        interpretation,
-        '',
-        '## Key measurements',
-        '',
-        *[f'- {item}' for item in highlights],
-        '',
-        '## Experimental design',
-        '',
-        '- Model: Qwen3-1.7B-Base.',
-        '- SAEs: Qwen-Scope residual-stream TopK SAEs at configured early/middle/late layers.',
-        '- Discovery set: controlled concept prompts with paired paraphrases.',
-        '- SAE concept evidence: prompt-wide maximum activation per feature across non-padding prompt tokens; final-token activations are saved separately for local diagnostics.',
-        '- Split discipline: paraphrase groups stay entirely in train or held-out test.',
-        '- Candidate stability: deterministic balanced activation resamples estimate how often selected features survive small changes in the discovery sample.',
-        '- Feature selection: training-split AUROC and activation contrast; held-out AUROC/F1 are reported separately.',
-        '- Linear baseline: multinomial logistic regression on the dense residual stream.',
-        '- Single-feature causal edit: reconstruction-preserving decoder-direction delta patched into the original residual.',
-        '- Feature-set causal edit: joint ablation of top same-layer concept features, evaluated at k=1/3/5 by default.',
-        '- Negative control: ensemble of deterministic random residual directions, each matched to the SAE perturbation L2 norm.',
-        '- Target metric: exact full target continuation scored teacher-forced; mean log probability per target token is the primary length-comparable effect.',
-        '- Secondary diagnostics: first-token probability/rank, next-token JS divergence, and top-1 changes.',
-        '- Uncertainty: bootstrap 95% confidence intervals and paired sign-flip randomization tests.',
-        '',
-        '## Figures',
-        '',
-        '![Feature AUROC](figures/feature_auroc.png)',
-        '',
-        '![Layer diagnostics](figures/layer_diagnostics.png)',
-        '',
-        '![Single-feature causal effects](figures/causal_effects.png)',
-    ]
-    if feature_sets is not None and not feature_sets.empty:
-        lines.extend(['', '![Feature-set causal effects](figures/feature_set_effects.png)'])
-    if not study.empty:
-        lines.extend(
-            [
-                '',
-                '![Association vs causality](figures/association_vs_causality.png)',
-                '',
-                '![Candidate stability](figures/candidate_stability.png)',
-                '',
-                '## Association vs causality across concepts',
-                '',
-                "The offline study joins each concept-selected feature's held-out AUROC/F1 and activation-resample support with random-normalized causal specificity on the held-out causal task set. Cross-concept correlations are descriptive because there are only seven controlled concepts.",
-            ]
-        )
-    lines.extend(
-        [
-            '',
-            '## Interpretation guardrails',
-            '',
-            'A high feature/concept AUROC or high paraphrase overlap is correlational evidence only. Causal evidence requires a downstream change under intervention and is interpreted relative to a norm-matched random control. Feature-set effects are not assumed stronger a priori; they are separately measured. The narrative above is generated from saved metrics, with no hard-coded result values.',
-            '',
-            '## Reproducibility',
-            '',
-            'Run `python experiments/run_all.py` from the repository root for the full GPU + CPU study. If model/SAE activations and causal rows already exist, run `python experiments/run_analysis_only.py` to regenerate CPU evaluation, stability, study synthesis, figures, validation, and this report without another model download or inference pass.',
-            '',
-        ]
+    report_lines = _build_report_lines(
+        headline=headline,
+        interpretation=interpretation,
+        highlights=highlights,
+        feature_sets=feature_sets,
+        study=study,
     )
-    (args.artifact_dir / 'report.md').write_text('\n'.join(lines), encoding='utf-8')
+    report_path = artifact_dir / "report.md"
+    report_path.write_text("\n".join(report_lines), encoding="utf-8")
     print(headline)
-    print(f'Wrote {args.artifact_dir / "report.md"}')
+    print(f"Wrote {report_path}")
 
 
-if __name__ == '__main__':
+if __name__ == "__main__":
     main()

experiments/run_all.py

@@ -80,12 +80,30 @@ def main() -> None:
         ],
         resume=args.resume,
     )
-    causal_output = artifact_dir / 'causal_results.csv'
+    final_output = artifact_dir / 'causal_results_final_token.csv'
     run(
         'experiments.run_causal',
-        outputs=[causal_output, causal_output.with_suffix(causal_output.suffix + '.complete')],
+        outputs=[final_output, final_output.with_suffix(final_output.suffix + '.complete')],
         resume=args.resume,
-        extra_args=['--resume'] if args.resume else None,
+        extra_args=[
+            '--position-policy', 'final_token',
+            '--output', str(final_output),
+            *(['--resume'] if args.resume else []),
+        ],
+    )
+    max_active_output = artifact_dir / 'causal_results_max_active.csv'
+    run(
+        'experiments.run_causal',
+        outputs=[
+            max_active_output,
+            max_active_output.with_suffix(max_active_output.suffix + '.complete'),
+        ],
+        resume=args.resume,
+        extra_args=[
+            '--position-policy', 'max_feature_activation',
+            '--output', str(max_active_output),
+            *(['--resume'] if args.resume else []),
+        ],
     )
     feature_set_output = artifact_dir / 'feature_set_results.csv'
     run(

experiments/run_causal.py

@@ -12,14 +12,17 @@ from experiments.common import ARTIFACT_DIR, DATA_DIR, load_jsonl, set_seed
 from featurelens.config import SETTINGS
 from featurelens.interventions import InterventionSpec, normalized_random_control, residual_delta
 from featurelens.metrics import js_divergence_from_logits, sequence_logprob_summary
-from featurelens.sae import SAEStore
+from featurelens.sae import SAEStore, SparseEncoding
+
+POSITION_POLICIES = ('final_token', 'max_feature_activation')
 
 
 def parse_args() -> argparse.Namespace:
     parser = argparse.ArgumentParser(description='Run held-out causal SAE interventions.')
     parser.add_argument('--tasks', type=Path, default=DATA_DIR / 'causal_tasks.jsonl')
     parser.add_argument('--catalog', type=Path, default=ARTIFACT_DIR / 'feature_catalog.csv')
-    parser.add_argument('--output', type=Path, default=ARTIFACT_DIR / 'causal_results.csv')
+    parser.add_argument('--output', type=Path, default=None)
+    parser.add_argument('--position-policy', choices=POSITION_POLICIES, default='final_token')
     parser.add_argument('--seed', type=int, default=42)
     parser.add_argument('--random-controls', type=int, default=8)
     parser.add_argument(
@@ -30,6 +33,10 @@ def parse_args() -> argparse.Namespace:
     return parser.parse_args()
 
 
+def default_output(policy: str) -> Path:
+    if policy == 'final_token':
+        return ARTIFACT_DIR / 'causal_results_final_token.csv'
+    return ARTIFACT_DIR / 'causal_results_max_active.csv'
 
 
 def _completion_marker(path: Path) -> Path:
@@ -85,8 +92,6 @@ def replace_hidden(output, hidden):
 
 
 def _make_capture_hook(capture: dict):
-    """Bind a per-task capture dictionary before registering the hook."""
-
     def capture_hook(_module, _inp, output):
         if 'hidden' not in capture:
             capture['hidden'] = hidden_from_output(output).detach()
@@ -96,18 +101,17 @@ def _make_capture_hook(capture: dict):
 
 def _make_batch_edit_hook(
     applied: dict[str, bool],
-    prompt_len: int,
+    intervention_token_index: int,
     deltas: torch.Tensor,
 ):
-    """Bind per-task edit state so hooks cannot capture a later loop iteration."""
-
     def batch_edit_hook(_module, _inp, output):
         if applied['done']:
             return output
         hidden = hidden_from_output(output)
         modified = hidden.clone()
-        modified[:, prompt_len - 1, :] = (
-            modified[:, prompt_len - 1, :] + deltas.to(hidden.device, hidden.dtype)
+        modified[:, intervention_token_index, :] = (
+            modified[:, intervention_token_index, :]
+            + deltas.to(hidden.device, hidden.dtype)
         )
         applied['done'] = True
         return replace_hidden(output, modified)
@@ -136,9 +140,45 @@ def make_random_controls(delta: torch.Tensor, seed: int, count: int) -> list[tor
     ]
 
 
+def feature_activation_trace(encoding: SparseEncoding, feature_id: int) -> torch.Tensor:
+    """Return one TopK feature activation per encoded token."""
+    indices = encoding.indices
+    values = encoding.values
+    if indices.ndim != 2 or values.ndim != 2:
+        raise ValueError('Expected tokenwise sparse encoding with shape [tokens, top_k].')
+    mask = indices == int(feature_id)
+    return torch.where(mask, values, torch.zeros_like(values)).max(dim=-1).values
+
+
+def choose_intervention_position(
+    token_activations: torch.Tensor,
+    *,
+    prompt_len: int,
+    position_policy: str,
+) -> tuple[int, float, bool]:
+    if prompt_len < 1:
+        raise ValueError('Prompt must contain at least one token.')
+    if position_policy == 'final_token':
+        index = prompt_len - 1
+        activation = float(token_activations[index].item())
+        return index, activation, activation > 0.0
+    if position_policy != 'max_feature_activation':
+        raise ValueError(f'Unknown position policy: {position_policy}')
+
+    max_activation, max_index = torch.max(token_activations[:prompt_len], dim=0)
+    activation = float(max_activation.item())
+    if activation <= 0.0:
+        # No selected feature is represented in TopK anywhere in the prompt.
+        # Keep a deterministic final-token location; the feature delta is zero.
+        return prompt_len - 1, 0.0, False
+    return int(max_index.item()), activation, True
+
+
 @torch.inference_mode()
 def main() -> None:
     args = parse_args()
+    if args.output is None:
+        args.output = default_output(args.position_policy)
     set_seed(args.seed)
     tasks = load_jsonl(args.tasks)
     selected = load_selected_features(args.catalog)
@@ -183,7 +223,10 @@ def main() -> None:
     for task_idx, task in enumerate(tasks):
         task_id = str(task['id'])
         if args.resume and completed_counts.get(task_id, 0) == expected_rows_per_task:
-            print(f"SKIP causal task {task_idx + 1}/{len(tasks)}: {task_id}", flush=True)
+            print(
+                f'SKIP {args.position_policy} causal task {task_idx + 1}/{len(tasks)}: {task_id}',
+                flush=True,
+            )
             continue
         if args.resume and completed_counts.get(task_id, 0):
             results = [row for row in results if str(row.get('task_id', '')) != task_id]
@@ -201,10 +244,9 @@ def main() -> None:
             raise RuntimeError(f"Target tokenization empty for task {task['id']}")
         target_ids = [int(x) for x in target_ids]
         full_inputs = append_target(prompt_inputs, target_ids)
+
         capture: dict = {}
-        handle = model.model.layers[layer].register_forward_hook(
-            _make_capture_hook(capture)
-        )
+        handle = model.model.layers[layer].register_forward_hook(_make_capture_hook(capture))
         single_baseline_out = model(**full_inputs, use_cache=False)
         handle.remove()
         single_baseline_logits = single_baseline_out.logits[0]
@@ -214,9 +256,24 @@ def main() -> None:
             target_ids=target_ids,
         )
 
-        residual = capture['hidden'][0, prompt_len - 1]
-        encoding = sae.encode(residual)
-        original_activation = encoding.activation_for(feature_id)
+        prompt_hidden = capture['hidden'][0, :prompt_len]
+        token_encoding = sae.encode(prompt_hidden)
+        token_activations = feature_activation_trace(token_encoding, feature_id)
+        final_token_activation = float(token_activations[prompt_len - 1].item())
+        max_activation_value, max_activation_index = torch.max(token_activations, dim=0)
+        max_prompt_activation = float(max_activation_value.item())
+        max_prompt_index = int(max_activation_index.item()) if max_prompt_activation > 0 else prompt_len - 1
+        active_anywhere = max_prompt_activation > 0.0
+
+        intervention_index, original_activation, active_at_intervention = choose_intervention_position(
+            token_activations,
+            prompt_len=prompt_len,
+            position_policy=args.position_policy,
+        )
+        prompt_token_ids = prompt_inputs['input_ids'][0]
+        intervention_token_text = tokenizer.decode([int(prompt_token_ids[intervention_index].item())])
+        max_prompt_token_text = tokenizer.decode([int(prompt_token_ids[max_prompt_index].item())])
+        final_token_text = tokenizer.decode([int(prompt_token_ids[prompt_len - 1].item())])
 
         specs = [
             ('ablate', InterventionSpec('ablate', 0.0)),
@@ -257,7 +314,7 @@ def main() -> None:
         repeated = {key: value.repeat(deltas.shape[0], 1) for key, value in full_inputs.items()}
         applied = {'done': False}
         hook = model.model.layers[layer].register_forward_hook(
-            _make_batch_edit_hook(applied, prompt_len, deltas)
+            _make_batch_edit_hook(applied, intervention_index, deltas)
         )
         edited_out = model(**repeated, use_cache=False)
         hook.remove()
@@ -278,10 +335,14 @@ def main() -> None:
         baseline_rank = int((baseline_next > baseline_next[target_id]).sum().item()) + 1
         baseline_top1 = int(torch.argmax(baseline_next).item())
 
-        for row_idx, (intervention_name, condition, control_id, _spec, applied_delta, delta_activation) in enumerate(
-            condition_meta,
-            start=1,
-        ):
+        for row_idx, (
+            intervention_name,
+            condition,
+            control_id,
+            _spec,
+            applied_delta,
+            delta_activation,
+        ) in enumerate(condition_meta, start=1):
             modified_logits = edited_out.logits[row_idx]
             modified_next = modified_logits[prompt_len - 1]
             modified_prob = float(torch.softmax(modified_next.float(), dim=-1)[target_id].item())
@@ -305,7 +366,19 @@ def main() -> None:
                     'feature_train_auroc': choice['train_auroc'],
                     'feature_test_auroc': choice['test_auroc'],
                     'feature_test_f1': choice['test_f1'],
+                    'position_policy': args.position_policy,
+                    'intervention_token_index': intervention_index,
+                    'intervention_token_text': intervention_token_text,
+                    'final_token_index': prompt_len - 1,
+                    'final_token_text': final_token_text,
+                    'max_prompt_feature_token_index': max_prompt_index,
+                    'max_prompt_feature_token_text': max_prompt_token_text,
                     'feature_activation': original_activation,
+                    'final_token_feature_activation': final_token_activation,
+                    'max_prompt_feature_activation': max_prompt_activation,
+                    'feature_active_at_intervention': int(active_at_intervention),
+                    'feature_active_at_final_token': int(final_token_activation > 0.0),
+                    'feature_active_anywhere': int(active_anywhere),
                     'intervention': intervention_name,
                     'condition': condition,
                     'control_id': control_id,
@@ -335,11 +408,15 @@ def main() -> None:
                 }
             )
         _write_rows_atomic(args.output, results)
-        print(f"Causal task {task_idx + 1}/{len(tasks)}: {concept}", flush=True)
+        print(
+            f'Causal {args.position_policy} task {task_idx + 1}/{len(tasks)}: '
+            f'{concept} @ token {intervention_index} (activation {original_activation:.4f})',
+            flush=True,
+        )
 
     _write_rows_atomic(args.output, results)
     marker.write_text('complete\n', encoding='utf-8')
-    print(f'Wrote {len(results)} causal intervention rows to {args.output}')
+    print(f'Wrote {len(results)} {args.position_policy} causal rows to {args.output}')
 
 
 if __name__ == '__main__':

experiments/run_causal_addendum.py

@@ -0,0 +1,91 @@
+from __future__ import annotations
+
+import argparse
+import subprocess
+import sys
+from pathlib import Path
+
+import numpy as np
+import pandas as pd
+
+from experiments.common import ARTIFACT_DIR
+
+ROOT = Path(__file__).resolve().parents[1]
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description='Run only the v0.16 max-active causal addendum and CPU reanalysis.'
+    )
+    parser.add_argument('--artifact-dir', type=Path, default=ARTIFACT_DIR)
+    parser.add_argument('--resume', action='store_true')
+    parser.add_argument('--random-controls', type=int, default=8)
+    return parser.parse_args()
+
+
+def run(command: list[str]) -> None:
+    print('\n$', ' '.join(command), flush=True)
+    subprocess.run(command, cwd=ROOT, check=True)
+
+
+def migrate_final_token_baseline(source: Path, destination: Path) -> None:
+    """Preserve a v0.15 causal CSV while adding v0.16 position-policy metadata."""
+    frame = pd.read_csv(source)
+    if 'position_policy' not in frame.columns:
+        active = pd.to_numeric(frame['feature_activation'], errors='coerce').fillna(0.0) > 0.0
+        frame['position_policy'] = 'final_token'
+        frame['intervention_token_index'] = -1
+        frame['intervention_token_text'] = ''
+        frame['final_token_index'] = -1
+        frame['final_token_text'] = ''
+        frame['max_prompt_feature_token_index'] = -1
+        frame['max_prompt_feature_token_text'] = ''
+        frame['final_token_feature_activation'] = pd.to_numeric(
+            frame['feature_activation'], errors='coerce'
+        ).fillna(0.0)
+        frame['max_prompt_feature_activation'] = np.nan
+        frame['feature_active_at_intervention'] = active.astype(int)
+        frame['feature_active_at_final_token'] = active.astype(int)
+        # Prompt-wide activity cannot be reconstructed from the legacy final-token CSV.
+        frame['feature_active_anywhere'] = np.nan
+    destination.parent.mkdir(parents=True, exist_ok=True)
+    temporary = destination.with_suffix(destination.suffix + '.tmp')
+    frame.to_csv(temporary, index=False)
+    temporary.replace(destination)
+
+
+def main() -> None:
+    args = parse_args()
+    artifact_dir = args.artifact_dir
+    legacy = artifact_dir / 'causal_results.csv'
+    final = artifact_dir / 'causal_results_final_token.csv'
+    if not final.exists():
+        if not legacy.exists():
+            raise SystemExit(
+                'Missing final-token baseline. Expected artifacts/causal_results.csv or '
+                'artifacts/causal_results_final_token.csv from the completed v0.15 study.'
+            )
+        migrate_final_token_baseline(legacy, final)
+        print(f'Preserved v0.15 baseline as {final.name}.')
+    elif 'position_policy' not in pd.read_csv(final, nrows=1).columns:
+        migrate_final_token_baseline(final, final)
+        print(f'Upgraded {final.name} with v0.16 position metadata.')
+
+    output = artifact_dir / 'causal_results_max_active.csv'
+    causal_command = [
+        sys.executable, '-m', 'experiments.run_causal',
+        '--position-policy', 'max_feature_activation',
+        '--output', str(output),
+        '--random-controls', str(args.random_controls),
+    ]
+    if args.resume:
+        causal_command.append('--resume')
+    run(causal_command)
+    run([sys.executable, '-m', 'experiments.analyze_study'])
+    run([sys.executable, '-m', 'experiments.make_report'])
+    run([sys.executable, '-m', 'scripts.validate_artifacts'])
+    print('\nFeatureLens v0.16 causal addendum complete.')
+
+
+if __name__ == '__main__':
+    main()

featurelens/stats.py

@@ -57,8 +57,15 @@ def paired_sign_flip_pvalue(
     *,
     n_permutations: int = 20000,
     seed: int = 42,
+    exact_max_nonzero_pairs: int = 18,
 ) -> float:
-    """Two-sided paired randomization test for a non-zero mean difference."""
+    """Two-sided paired sign-flip test for a non-zero mean difference.
+
+    For small effective samples the test is enumerated exactly. Zero-difference
+    pairs are removed before deciding whether exact enumeration is feasible;
+    their sign cannot change the statistic. Larger samples use a deterministic
+    Monte-Carlo approximation.
+    """
     x = np.asarray(a, dtype=float)
     y = np.asarray(b, dtype=float)
     mask = np.isfinite(x) & np.isfinite(y)
@@ -68,14 +75,36 @@ def paired_sign_flip_pvalue(
     observed = abs(float(diff.mean()))
     if observed == 0.0:
         return 1.0
+
+    nonzero = diff[np.abs(diff) > 0.0]
+    if nonzero.size == 0:
+        return 1.0
+
+    # Preserve the original mean denominator: zero-difference pairs contribute
+    # to n but never to a sign-flipped numerator.
+    denominator = float(diff.size)
+    if nonzero.size <= int(exact_max_nonzero_pairs):
+        count = 1 << int(nonzero.size)
+        extreme = 0
+        for bits in range(count):
+            signs = np.fromiter(
+                (1.0 if bits & (1 << idx) else -1.0 for idx in range(nonzero.size)),
+                dtype=float,
+                count=nonzero.size,
+            )
+            statistic = abs(float(np.sum(signs * nonzero) / denominator))
+            if statistic >= observed - 1e-15:
+                extreme += 1
+        return float(extreme / count)
+
     rng = np.random.default_rng(seed)
     extreme = 0
     batch = 2000
     remaining = int(n_permutations)
     while remaining > 0:
         n = min(batch, remaining)
-        signs = rng.choice(np.array([-1.0, 1.0]), size=(n, diff.size))
-        permuted = np.abs((signs * diff).mean(axis=1))
+        signs = rng.choice(np.array([-1.0, 1.0]), size=(n, nonzero.size))
+        permuted = np.abs((signs * nonzero).sum(axis=1) / denominator)
         extreme += int(np.count_nonzero(permuted >= observed))
         remaining -= n
     return float((extreme + 1) / (int(n_permutations) + 1))

featurelens/study.py

@@ -13,11 +13,13 @@ class OfflineStudy:
         'summary.json',
         'study_summary.json',
         'study_feature_summary.csv',
+        'causal_position_summary.csv',
         'selection_stability.csv',
         'feature_catalog.csv',
         'layer_metrics.csv',
         'stability.csv',
-        'causal_results.csv',
+        'causal_results_final_token.csv',
+        'causal_results_max_active.csv',
         'feature_set_results.csv',
         'report.md',
     )
@@ -55,30 +57,34 @@ class OfflineStudy:
             suffix = '…' if len(self.missing) > 6 else ''
             return (
                 '### Offline study not materialized yet\n\n'
-                'The live workbench is usable now, but held-out study artifacts have not been committed. '
-                f'Missing: {missing}{suffix}\n\n'
-                'Run `python experiments/run_all.py` on a CUDA machine for the full study. If activation and '
-                'causal artifacts already exist, use `python experiments/run_analysis_only.py` to rerun the '
-                'CPU-only evaluation, stability analysis, evidence synthesis, figures, and report. '
-                'Then run `python -m scripts.validate_artifacts` before committing the small CSV/JSON/report '
-                'outputs. Do not commit `artifacts/activations/`.'
+                'The live workbench is usable now, but the finalized position-sensitivity study artifacts '
+                f'have not been committed. Missing: {missing}{suffix}\n\n'
+                'For a fresh study run `python -m experiments.run_all --resume`. If the v0.15 final-token '
+                'study already exists, use the v0.16 causal-addendum notebook or '
+                '`python -m experiments.run_causal_addendum --resume` to add max-active causal results '
+                'without recollecting discovery activations.'
             )
 
         summary = self._json('summary.json')
         study = self._json('study_summary.json')
         correlations = study.get('correlations', {})
-        target_corr = correlations.get('heldout_auroc_vs_target_specificity', {})
-        js_corr = correlations.get('heldout_auroc_vs_js_specificity', {})
+        target_corr = correlations.get('heldout_auroc_vs_max_active_target_specificity', {})
+        js_corr = correlations.get('heldout_auroc_vs_max_active_js_specificity', {})
         return (
             '### Offline study results\n\n'
             f"{summary.get('headline', 'Benchmark completed.')}\n\n"
             f"{summary.get('interpretation', '')}\n\n"
+            '**Position sensitivity**\n\n'
+            f"- Final-token feature coverage: **{float(study.get('final_token_feature_coverage', float('nan'))):.1%}**.\n"
+            f"- Max-active feature coverage: **{float(study.get('max_active_feature_coverage', float('nan'))):.1%}**.\n"
+            f"- Final-token target specificity: **{float(study.get('final_token_target_specificity_ratio', float('nan'))):.2f}×**.\n"
+            f"- Max-active target specificity: **{float(study.get('max_active_target_specificity_ratio', float('nan'))):.2f}×**.\n\n"
             '**Study-level diagnostics**\n\n'
             f"- Selected-feature median resample support: **{float(study.get('median_selected_feature_resample_support', float('nan'))):.1%}**.\n"
-            f"- Held-out AUROC ↔ target-specificity Spearman ρ: **{float(target_corr.get('rho', float('nan'))):+.3f}** (n={int(target_corr.get('n', 0))}).\n"
-            f"- Held-out AUROC ↔ JS-specificity Spearman ρ: **{float(js_corr.get('rho', float('nan'))):+.3f}** (n={int(js_corr.get('n', 0))}).\n\n"
-            'The cross-concept correlations are descriptive because there are only seven controlled concepts. '
-            'Per-concept causal claims remain anchored to matched random-control comparisons.'
+            f"- Held-out AUROC ↔ max-active target-specificity Spearman ρ: **{float(target_corr.get('rho', float('nan'))):+.3f}** (n={int(target_corr.get('n', 0))}).\n"
+            f"- Held-out AUROC ↔ max-active JS-specificity Spearman ρ: **{float(js_corr.get('rho', float('nan'))):+.3f}** (n={int(js_corr.get('n', 0))}).\n\n"
+            'Cross-concept correlations are descriptive; per-concept causal evidence remains anchored to '
+            'norm-matched random controls.'
         )
 
     def readiness_markdown(self) -> str:

notebooks/FeatureLens_Causal_Addendum_Colab.ipynb

@@ -0,0 +1,248 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# FeatureLens v0.16 causal-position addendum\n",
+        "\n",
+        "Use this notebook **only after the v0.15 full offline study completed**. It preserves that final-token causal baseline and runs the smaller max-feature-activation causal addendum.\n",
+        "\n",
+        "It does **not** recollect the 224-prompt activation matrices, refit probes/features, rerun candidate stability, or rerun the 1/3/5 feature-set benchmark.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# 1. Verify the Colab GPU runtime.\n",
+        "!nvidia-smi\n",
+        "import torch\n",
+        "print(\"CUDA available:\", torch.cuda.is_available())\n",
+        "if not torch.cuda.is_available():\n",
+        "    raise RuntimeError(\"Enable a GPU runtime before continuing.\")\n",
+        "print(\"GPU:\", torch.cuda.get_device_name(0))\n",
+        "print(\"VRAM GiB:\", torch.cuda.get_device_properties(0).total_memory / 1024**3)\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# 2. Mount Google Drive.\n",
+        "from google.colab import drive\n",
+        "drive.mount(\"/content/drive\")\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# 3. Configuration \u2014 edit REPO_URL if needed.\n",
+        "from pathlib import Path\n",
+        "\n",
+        "REPO_URL = \"PASTE_YOUR_GIT_REPO_URL_HERE\"\n",
+        "BRANCH = \"main\"\n",
+        "SOURCE_RUN_NAME = \"FeatureLens_offline_v015\"\n",
+        "ADDENDUM_RUN_NAME = \"FeatureLens_offline_v016\"\n",
+        "\n",
+        "REPO_DIR = Path(\"/content/FeatureLens\")\n",
+        "DRIVE_ROOT = Path(\"/content/drive/MyDrive\")\n",
+        "SOURCE_ARTIFACTS = DRIVE_ROOT / SOURCE_RUN_NAME / \"artifacts\"\n",
+        "ADDENDUM_ROOT = DRIVE_ROOT / ADDENDUM_RUN_NAME\n",
+        "ADDENDUM_ARTIFACTS = ADDENDUM_ROOT / \"artifacts\"\n",
+        "LOG_PATH = ADDENDUM_ROOT / \"causal_addendum.log\"\n",
+        "\n",
+        "if REPO_URL.startswith(\"PASTE_\"):\n",
+        "    raise ValueError(\"Set REPO_URL to your FeatureLens Git repository URL first.\")\n",
+        "if not SOURCE_ARTIFACTS.exists():\n",
+        "    raise FileNotFoundError(\n",
+        "        f\"Could not find the completed v0.15 artifacts at {SOURCE_ARTIFACTS}. \"\n",
+        "        \"Change SOURCE_RUN_NAME if your previous Drive folder used another name.\"\n",
+        "    )\n",
+        "ADDENDUM_ARTIFACTS.mkdir(parents=True, exist_ok=True)\n",
+        "print(\"Source:\", SOURCE_ARTIFACTS)\n",
+        "print(\"Addendum:\", ADDENDUM_ARTIFACTS)\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# 4. Clone or refresh the v0.16 FeatureLens source.\n",
+        "import subprocess, shutil\n",
+        "\n",
+        "if not REPO_DIR.exists():\n",
+        "    subprocess.run([\"git\", \"clone\", \"--branch\", BRANCH, \"--single-branch\", REPO_URL, str(REPO_DIR)], check=True)\n",
+        "else:\n",
+        "    subprocess.run([\"git\", \"-C\", str(REPO_DIR), \"fetch\", \"origin\", BRANCH], check=True)\n",
+        "    subprocess.run([\"git\", \"-C\", str(REPO_DIR), \"checkout\", BRANCH], check=True)\n",
+        "    subprocess.run([\"git\", \"-C\", str(REPO_DIR), \"pull\", \"--ff-only\", \"origin\", BRANCH], check=True)\n",
+        "print(\"Commit:\", subprocess.check_output([\"git\", \"-C\", str(REPO_DIR), \"rev-parse\", \"--short\", \"HEAD\"], text=True).strip())\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# 5. Install runtime dependencies.\n",
+        "import subprocess, sys\n",
+        "subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"-r\", str(REPO_DIR / \"requirements.txt\")], check=True)\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# 6. Seed the addendum folder with only the small v0.15 study outputs.\n",
+        "# Large activation caches are intentionally NOT copied.\n",
+        "import shutil\n",
+        "\n",
+        "for path in SOURCE_ARTIFACTS.rglob(\"*\"):\n",
+        "    if not path.is_file():\n",
+        "        continue\n",
+        "    rel = path.relative_to(SOURCE_ARTIFACTS)\n",
+        "    if rel.parts and rel.parts[0] == \"activations\":\n",
+        "        continue\n",
+        "    if path.name.endswith((\".complete\", \".tmp\")):\n",
+        "        continue\n",
+        "    destination = ADDENDUM_ARTIFACTS / rel\n",
+        "    if not destination.exists():\n",
+        "        destination.parent.mkdir(parents=True, exist_ok=True)\n",
+        "        shutil.copy2(path, destination)\n",
+        "\n",
+        "required = [\"feature_catalog.csv\", \"layer_metrics.csv\", \"stability.csv\", \"selection_stability.csv\", \"feature_set_results.csv\"]\n",
+        "missing = [name for name in required if not (ADDENDUM_ARTIFACTS / name).exists()]\n",
+        "if missing:\n",
+        "    raise RuntimeError(f\"Previous study is missing required small artifacts: {missing}\")\n",
+        "if not ((ADDENDUM_ARTIFACTS / \"causal_results.csv\").exists() or (ADDENDUM_ARTIFACTS / \"causal_results_final_token.csv\").exists()):\n",
+        "    raise RuntimeError(\"Previous study is missing its final-token causal baseline.\")\n",
+        "print(\"Small v0.15 artifacts copied; large activations were skipped.\")\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# 7. Point the repo's artifacts/ directory at the Drive-backed addendum folder.\n",
+        "import shutil\n",
+        "local_artifacts = REPO_DIR / \"artifacts\"\n",
+        "if local_artifacts.is_symlink():\n",
+        "    local_artifacts.unlink()\n",
+        "elif local_artifacts.exists():\n",
+        "    shutil.rmtree(local_artifacts)\n",
+        "local_artifacts.symlink_to(ADDENDUM_ARTIFACTS, target_is_directory=True)\n",
+        "print(\"artifacts ->\", local_artifacts.resolve())\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# 8. Run only the max-active causal addendum + CPU synthesis.\n",
+        "# Task-level checkpointing makes this resumable if Colab disconnects.\n",
+        "import subprocess, sys, time\n",
+        "\n",
+        "command = [sys.executable, \"-m\", \"experiments.run_causal_addendum\", \"--resume\"]\n",
+        "print(\"$\", \" \".join(command))\n",
+        "print(\"Log:\", LOG_PATH)\n",
+        "start = time.time()\n",
+        "with LOG_PATH.open(\"a\", encoding=\"utf-8\") as log:\n",
+        "    log.write(\"\\n\\n=== FeatureLens v0.16 causal addendum ===\\n\")\n",
+        "    process = subprocess.Popen(command, cwd=REPO_DIR, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, bufsize=1)\n",
+        "    assert process.stdout is not None\n",
+        "    for line in process.stdout:\n",
+        "        print(line, end=\"\")\n",
+        "        log.write(line)\n",
+        "        log.flush()\n",
+        "    return_code = process.wait()\n",
+        "if return_code != 0:\n",
+        "    raise RuntimeError(f\"Addendum exited with code {return_code}. Fix the error and rerun this cell; --resume keeps completed causal tasks.\")\n",
+        "print(f\"\\nCompleted in {(time.time()-start)/60:.1f} minutes.\")\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# 9. Inspect the finalized position-sensitivity results.\n",
+        "import json, pandas as pd\n",
+        "from IPython.display import display, Markdown\n",
+        "\n",
+        "position = pd.read_csv(ADDENDUM_ARTIFACTS / \"causal_position_summary.csv\")\n",
+        "study = pd.read_csv(ADDENDUM_ARTIFACTS / \"study_feature_summary.csv\")\n",
+        "summary = json.loads((ADDENDUM_ARTIFACTS / \"study_summary.json\").read_text(encoding=\"utf-8\"))\n",
+        "report = (ADDENDUM_ARTIFACTS / \"report.md\").read_text(encoding=\"utf-8\")\n",
+        "\n",
+        "display(position[position[\"concept\"] == \"__all__\"])\n",
+        "display(study)\n",
+        "display(summary)\n",
+        "display(Markdown(report))\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# 10. Create the final publishable result bundle.\n",
+        "import zipfile\n",
+        "\n",
+        "PUBLISH_ZIP = ADDENDUM_ROOT / \"FeatureLens_offline_results_v016.zip\"\n",
+        "with zipfile.ZipFile(PUBLISH_ZIP, \"w\", compression=zipfile.ZIP_DEFLATED) as zf:\n",
+        "    for path in sorted(ADDENDUM_ARTIFACTS.rglob(\"*\")):\n",
+        "        if not path.is_file():\n",
+        "            continue\n",
+        "        rel = path.relative_to(ADDENDUM_ARTIFACTS)\n",
+        "        if rel.parts and rel.parts[0] == \"activations\":\n",
+        "            continue\n",
+        "        if path.name.endswith((\".complete\", \".tmp\")):\n",
+        "            continue\n",
+        "        zf.write(path, arcname=str(Path(\"artifacts\") / rel))\n",
+        "print(\"Final result bundle:\", PUBLISH_ZIP)\n",
+        "print(f\"Size: {PUBLISH_ZIP.stat().st_size / 1024**2:.2f} MiB\")\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## After the addendum\n",
+        "\n",
+        "Download `FeatureLens_offline_results_v016.zip` and bring it back to the ChatGPT project before committing the empirical artifacts. The final public release should use the v0.16 report/summary rather than the older v0.15 headline.\n"
+      ]
+    }
+  ],
+  "metadata": {
+    "accelerator": "GPU",
+    "kernelspec": {
+      "display_name": "Python 3",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 5
+}

notebooks/FeatureLens_Offline_Study_Colab.ipynb

@@ -1,295 +1,295 @@
 {
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "3ca18c70",
-   "metadata": {},
-   "source": [
-    "# FeatureLens offline study\n",
-    "\n",
-    "This notebook runs the full FeatureLens empirical study on a CUDA runtime while persisting experiment artifacts to Google Drive. It is designed to be resumable after Colab disconnects.\n",
-    "\n",
-    "**Before running:** choose a GPU runtime in Colab, then execute the cells from top to bottom."
-   ]
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "id": "3ca18c70",
+      "metadata": {},
+      "source": [
+        "# FeatureLens offline study\n",
+        "\n",
+        "This notebook runs the full FeatureLens empirical study on a CUDA runtime while persisting experiment artifacts to Google Drive. It is designed to be resumable after Colab disconnects.\n",
+        "\n",
+        "**Before running:** choose a GPU runtime in Colab, then execute the cells from top to bottom.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "dd012fdd",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# 1. Verify that Colab actually assigned a GPU.\n",
+        "import subprocess, sys\n",
+        "\n",
+        "subprocess.run([\"nvidia-smi\"], check=True)\n",
+        "\n",
+        "try:\n",
+        "    import torch\n",
+        "    assert torch.cuda.is_available(), \"CUDA is not available. Change the Colab runtime to a GPU and reconnect.\"\n",
+        "    props = torch.cuda.get_device_properties(0)\n",
+        "    gpu_name = torch.cuda.get_device_name(0)\n",
+        "    gpu_vram_gb = props.total_memory / 1024**3\n",
+        "    print(f\"\\nGPU: {gpu_name} | VRAM: {gpu_vram_gb:.1f} GB\")\n",
+        "except Exception as exc:\n",
+        "    raise RuntimeError(\"A CUDA GPU runtime is required for the model stages.\") from exc\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "bdac04b1",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# 2. Mount Google Drive so completed experiment stages survive a runtime reset.\n",
+        "from google.colab import drive\n",
+        "drive.mount(\"/content/drive\")\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "0446d7cd",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# 3. Configuration \u2014 edit REPO_URL before running this cell.\n",
+        "from pathlib import Path\n",
+        "\n",
+        "REPO_URL = \"PASTE_YOUR_GIT_REPO_URL_HERE\"\n",
+        "BRANCH = \"main\"\n",
+        "DRIVE_RUN_NAME = \"FeatureLens_offline_v016_full\"\n",
+        "\n",
+        "REPO_DIR = Path(\"/content/FeatureLens\")\n",
+        "DRIVE_ROOT = Path(\"/content/drive/MyDrive\") / DRIVE_RUN_NAME\n",
+        "DRIVE_ARTIFACTS = DRIVE_ROOT / \"artifacts\"\n",
+        "LOG_PATH = DRIVE_ROOT / \"offline_study.log\"\n",
+        "\n",
+        "if REPO_URL.startswith(\"PASTE_\"):\n",
+        "    raise ValueError(\"Set REPO_URL to your FeatureLens Git repository URL first.\")\n",
+        "\n",
+        "DRIVE_ARTIFACTS.mkdir(parents=True, exist_ok=True)\n",
+        "print(\"Persistent run directory:\", DRIVE_ROOT)\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "c305468e",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# 4. Clone or refresh the FeatureLens source.\n",
+        "import shutil, subprocess\n",
+        "\n",
+        "if not REPO_DIR.exists():\n",
+        "    subprocess.run([\"git\", \"clone\", \"--branch\", BRANCH, \"--single-branch\", REPO_URL, str(REPO_DIR)], check=True)\n",
+        "else:\n",
+        "    subprocess.run([\"git\", \"-C\", str(REPO_DIR), \"fetch\", \"origin\", BRANCH], check=True)\n",
+        "    subprocess.run([\"git\", \"-C\", str(REPO_DIR), \"checkout\", BRANCH], check=True)\n",
+        "    subprocess.run([\"git\", \"-C\", str(REPO_DIR), \"pull\", \"--ff-only\", \"origin\", BRANCH], check=True)\n",
+        "\n",
+        "print(subprocess.check_output([\"git\", \"-C\", str(REPO_DIR), \"rev-parse\", \"--short\", \"HEAD\"], text=True).strip())\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "e34c567a",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# 5. Install the project environment. This can take a few minutes on a fresh runtime.\n",
+        "import subprocess, sys\n",
+        "subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"-r\", str(REPO_DIR / \"requirements.txt\")], check=True)\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "501273c0",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# 6. Re-check CUDA after dependency installation and choose a conservative activation batch.\n",
+        "import os, torch\n",
+        "\n",
+        "assert torch.cuda.is_available(), \"CUDA disappeared after dependency setup.\"\n",
+        "gpu_name = torch.cuda.get_device_name(0)\n",
+        "gpu_vram_gb = torch.cuda.get_device_properties(0).total_memory / 1024**3\n",
+        "ACTIVATION_BATCH_SIZE = 16 if gpu_vram_gb >= 20 else 8\n",
+        "ACTIVATION_MAX_LENGTH = 192\n",
+        "\n",
+        "# Keep model/SAE downloads on Colab's local disk for speed.\n",
+        "os.environ[\"HF_HOME\"] = \"/content/hf_cache\"\n",
+        "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n",
+        "\n",
+        "print(f\"GPU: {gpu_name} ({gpu_vram_gb:.1f} GB)\")\n",
+        "print(f\"Activation batch size: {ACTIVATION_BATCH_SIZE}\")\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "41a3a0e8",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# 7. Link FeatureLens artifacts to Google Drive.\n",
+        "# Existing small repo artifacts (for example README.md) are copied once; the local directory is then replaced by a symlink.\n",
+        "import shutil\n",
+        "\n",
+        "local_artifacts = REPO_DIR / \"artifacts\"\n",
+        "if local_artifacts.is_symlink():\n",
+        "    local_artifacts.unlink()\n",
+        "elif local_artifacts.exists():\n",
+        "    shutil.copytree(local_artifacts, DRIVE_ARTIFACTS, dirs_exist_ok=True)\n",
+        "    shutil.rmtree(local_artifacts)\n",
+        "\n",
+        "local_artifacts.symlink_to(DRIVE_ARTIFACTS, target_is_directory=True)\n",
+        "print(\"artifacts ->\", local_artifacts.resolve())\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "dbfda9e4",
+      "metadata": {},
+      "source": [
+        "## Run / resume the study\n",
+        "\n",
+        "The command below is safe to rerun. Completed stages are skipped. The causal and feature-set stages also checkpoint completed tasks, so a disconnect during either stage does not discard earlier tasks from that stage.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "3f0bfd41",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# 8. Run the full pipeline with live output and a persistent log.\n",
+        "import subprocess, sys, time\n",
+        "\n",
+        "command = [\n",
+        "    sys.executable, \"-m\", \"experiments.run_all\",\n",
+        "    \"--resume\",\n",
+        "    \"--activation-batch-size\", str(ACTIVATION_BATCH_SIZE),\n",
+        "    \"--activation-max-length\", str(ACTIVATION_MAX_LENGTH),\n",
+        "]\n",
+        "\n",
+        "print(\"$\", \" \".join(command))\n",
+        "print(\"Log:\", LOG_PATH)\n",
+        "start = time.time()\n",
+        "\n",
+        "with LOG_PATH.open(\"a\", encoding=\"utf-8\") as log:\n",
+        "    log.write(\"\\n\\n=== FeatureLens run ===\\n\")\n",
+        "    log.write(\"$ \" + \" \".join(command) + \"\\n\")\n",
+        "    process = subprocess.Popen(\n",
+        "        command, cwd=REPO_DIR, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, bufsize=1\n",
+        "    )\n",
+        "    assert process.stdout is not None\n",
+        "    for line in process.stdout:\n",
+        "        print(line, end=\"\")\n",
+        "        log.write(line)\n",
+        "        log.flush()\n",
+        "    return_code = process.wait()\n",
+        "\n",
+        "if return_code != 0:\n",
+        "    raise RuntimeError(\n",
+        "        f\"Pipeline exited with code {return_code}. Fix the error, then rerun this cell; --resume will keep completed work.\"\n",
+        "    )\n",
+        "\n",
+        "print(f\"\\nCompleted in {(time.time() - start) / 60:.1f} minutes.\")\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "645d7ee5",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# 9. Validate the measured artifact set.\n",
+        "import subprocess, sys\n",
+        "subprocess.run([sys.executable, \"-m\", \"scripts.validate_artifacts\"], cwd=REPO_DIR, check=True)\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "9e823e4d",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# 10. Inspect the study summary and report.\n",
+        "from pathlib import Path\n",
+        "import json, pandas as pd\n",
+        "from IPython.display import display, Markdown\n",
+        "\n",
+        "summary_path = DRIVE_ARTIFACTS / \"study_summary.json\"\n",
+        "study_table_path = DRIVE_ARTIFACTS / \"study_feature_summary.csv\"\n",
+        "report_path = DRIVE_ARTIFACTS / \"report.md\"\n",
+        "\n",
+        "summary = json.loads(summary_path.read_text(encoding=\"utf-8\"))\n",
+        "display(summary)\n",
+        "display(pd.read_csv(study_table_path))\n",
+        "display(Markdown(report_path.read_text(encoding=\"utf-8\")))\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "62c96dc6",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# 11. Create a small publishable artifact bundle (activation caches and checkpoint markers are excluded).\n",
+        "import zipfile\n",
+        "\n",
+        "PUBLISH_ZIP = DRIVE_ROOT / \"FeatureLens_offline_results.zip\"\n",
+        "\n",
+        "with zipfile.ZipFile(PUBLISH_ZIP, \"w\", compression=zipfile.ZIP_DEFLATED) as zf:\n",
+        "    for path in sorted(DRIVE_ARTIFACTS.rglob(\"*\")):\n",
+        "        if not path.is_file():\n",
+        "            continue\n",
+        "        rel = path.relative_to(DRIVE_ARTIFACTS)\n",
+        "        if rel.parts and rel.parts[0] == \"activations\":\n",
+        "            continue\n",
+        "        if path.name.endswith(\".complete\") or path.name.endswith(\".tmp\"):\n",
+        "            continue\n",
+        "        zf.write(path, arcname=str(Path(\"artifacts\") / rel))\n",
+        "\n",
+        "print(\"Publishable bundle:\", PUBLISH_ZIP)\n",
+        "print(f\"Size: {PUBLISH_ZIP.stat().st_size / 1024**2:.2f} MiB\")\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "a6634bfe",
+      "metadata": {},
+      "source": [
+        "## After Colab\n",
+        "\n",
+        "Download `FeatureLens_offline_results.zip` from the Drive run folder. Extract it over your local FeatureLens repository so the files land under `artifacts/`, run the normal release checks locally, inspect the measured report, and only then commit the small study artifacts. Do **not** commit `artifacts/activations/`.\n"
+      ]
+    }
+  ],
+  "metadata": {
+    "accelerator": "GPU",
+    "colab": {
+      "name": "FeatureLens Offline Study",
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
   },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "dd012fdd",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# 1. Verify that Colab actually assigned a GPU.\n",
-    "import subprocess, sys\n",
-    "\n",
-    "subprocess.run([\"nvidia-smi\"], check=True)\n",
-    "\n",
-    "try:\n",
-    "    import torch\n",
-    "    assert torch.cuda.is_available(), \"CUDA is not available. Change the Colab runtime to a GPU and reconnect.\"\n",
-    "    props = torch.cuda.get_device_properties(0)\n",
-    "    gpu_name = torch.cuda.get_device_name(0)\n",
-    "    gpu_vram_gb = props.total_memory / 1024**3\n",
-    "    print(f\"\\nGPU: {gpu_name} | VRAM: {gpu_vram_gb:.1f} GB\")\n",
-    "except Exception as exc:\n",
-    "    raise RuntimeError(\"A CUDA GPU runtime is required for the model stages.\") from exc"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "bdac04b1",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# 2. Mount Google Drive so completed experiment stages survive a runtime reset.\n",
-    "from google.colab import drive\n",
-    "drive.mount(\"/content/drive\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "0446d7cd",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# 3. Configuration — edit REPO_URL before running this cell.\n",
-    "from pathlib import Path\n",
-    "\n",
-    "REPO_URL = \"PASTE_YOUR_GIT_REPO_URL_HERE\"\n",
-    "BRANCH = \"main\"\n",
-    "DRIVE_RUN_NAME = \"FeatureLens_offline_v015\"\n",
-    "\n",
-    "REPO_DIR = Path(\"/content/FeatureLens\")\n",
-    "DRIVE_ROOT = Path(\"/content/drive/MyDrive\") / DRIVE_RUN_NAME\n",
-    "DRIVE_ARTIFACTS = DRIVE_ROOT / \"artifacts\"\n",
-    "LOG_PATH = DRIVE_ROOT / \"offline_study.log\"\n",
-    "\n",
-    "if REPO_URL.startswith(\"PASTE_\"):\n",
-    "    raise ValueError(\"Set REPO_URL to your FeatureLens Git repository URL first.\")\n",
-    "\n",
-    "DRIVE_ARTIFACTS.mkdir(parents=True, exist_ok=True)\n",
-    "print(\"Persistent run directory:\", DRIVE_ROOT)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "c305468e",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# 4. Clone or refresh the FeatureLens source.\n",
-    "import shutil, subprocess\n",
-    "\n",
-    "if not REPO_DIR.exists():\n",
-    "    subprocess.run([\"git\", \"clone\", \"--branch\", BRANCH, \"--single-branch\", REPO_URL, str(REPO_DIR)], check=True)\n",
-    "else:\n",
-    "    subprocess.run([\"git\", \"-C\", str(REPO_DIR), \"fetch\", \"origin\", BRANCH], check=True)\n",
-    "    subprocess.run([\"git\", \"-C\", str(REPO_DIR), \"checkout\", BRANCH], check=True)\n",
-    "    subprocess.run([\"git\", \"-C\", str(REPO_DIR), \"pull\", \"--ff-only\", \"origin\", BRANCH], check=True)\n",
-    "\n",
-    "print(subprocess.check_output([\"git\", \"-C\", str(REPO_DIR), \"rev-parse\", \"--short\", \"HEAD\"], text=True).strip())"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "e34c567a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# 5. Install the project environment. This can take a few minutes on a fresh runtime.\n",
-    "import subprocess, sys\n",
-    "subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"-r\", str(REPO_DIR / \"requirements.txt\")], check=True)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "501273c0",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# 6. Re-check CUDA after dependency installation and choose a conservative activation batch.\n",
-    "import os, torch\n",
-    "\n",
-    "assert torch.cuda.is_available(), \"CUDA disappeared after dependency setup.\"\n",
-    "gpu_name = torch.cuda.get_device_name(0)\n",
-    "gpu_vram_gb = torch.cuda.get_device_properties(0).total_memory / 1024**3\n",
-    "ACTIVATION_BATCH_SIZE = 16 if gpu_vram_gb >= 20 else 8\n",
-    "ACTIVATION_MAX_LENGTH = 192\n",
-    "\n",
-    "# Keep model/SAE downloads on Colab's local disk for speed.\n",
-    "os.environ[\"HF_HOME\"] = \"/content/hf_cache\"\n",
-    "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n",
-    "\n",
-    "print(f\"GPU: {gpu_name} ({gpu_vram_gb:.1f} GB)\")\n",
-    "print(f\"Activation batch size: {ACTIVATION_BATCH_SIZE}\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "41a3a0e8",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# 7. Link FeatureLens artifacts to Google Drive.\n",
-    "# Existing small repo artifacts (for example README.md) are copied once; the local directory is then replaced by a symlink.\n",
-    "import shutil\n",
-    "\n",
-    "local_artifacts = REPO_DIR / \"artifacts\"\n",
-    "if local_artifacts.is_symlink():\n",
-    "    local_artifacts.unlink()\n",
-    "elif local_artifacts.exists():\n",
-    "    shutil.copytree(local_artifacts, DRIVE_ARTIFACTS, dirs_exist_ok=True)\n",
-    "    shutil.rmtree(local_artifacts)\n",
-    "\n",
-    "local_artifacts.symlink_to(DRIVE_ARTIFACTS, target_is_directory=True)\n",
-    "print(\"artifacts ->\", local_artifacts.resolve())"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "dbfda9e4",
-   "metadata": {},
-   "source": [
-    "## Run / resume the study\n",
-    "\n",
-    "The command below is safe to rerun. Completed stages are skipped. The causal and feature-set stages also checkpoint completed tasks, so a disconnect during either stage does not discard earlier tasks from that stage."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "3f0bfd41",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# 8. Run the full pipeline with live output and a persistent log.\n",
-    "import subprocess, sys, time\n",
-    "\n",
-    "command = [\n",
-    "    sys.executable, \"-m\", \"experiments.run_all\",\n",
-    "    \"--resume\",\n",
-    "    \"--activation-batch-size\", str(ACTIVATION_BATCH_SIZE),\n",
-    "    \"--activation-max-length\", str(ACTIVATION_MAX_LENGTH),\n",
-    "]\n",
-    "\n",
-    "print(\"$\", \" \".join(command))\n",
-    "print(\"Log:\", LOG_PATH)\n",
-    "start = time.time()\n",
-    "\n",
-    "with LOG_PATH.open(\"a\", encoding=\"utf-8\") as log:\n",
-    "    log.write(\"\\n\\n=== FeatureLens run ===\\n\")\n",
-    "    log.write(\"$ \" + \" \".join(command) + \"\\n\")\n",
-    "    process = subprocess.Popen(\n",
-    "        command, cwd=REPO_DIR, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, bufsize=1\n",
-    "    )\n",
-    "    assert process.stdout is not None\n",
-    "    for line in process.stdout:\n",
-    "        print(line, end=\"\")\n",
-    "        log.write(line)\n",
-    "        log.flush()\n",
-    "    return_code = process.wait()\n",
-    "\n",
-    "if return_code != 0:\n",
-    "    raise RuntimeError(\n",
-    "        f\"Pipeline exited with code {return_code}. Fix the error, then rerun this cell; --resume will keep completed work.\"\n",
-    "    )\n",
-    "\n",
-    "print(f\"\\nCompleted in {(time.time() - start) / 60:.1f} minutes.\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "645d7ee5",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# 9. Validate the measured artifact set.\n",
-    "import subprocess, sys\n",
-    "subprocess.run([sys.executable, \"-m\", \"scripts.validate_artifacts\"], cwd=REPO_DIR, check=True)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "9e823e4d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# 10. Inspect the study summary and report.\n",
-    "from pathlib import Path\n",
-    "import json, pandas as pd\n",
-    "from IPython.display import display, Markdown\n",
-    "\n",
-    "summary_path = DRIVE_ARTIFACTS / \"study_summary.json\"\n",
-    "study_table_path = DRIVE_ARTIFACTS / \"study_feature_summary.csv\"\n",
-    "report_path = DRIVE_ARTIFACTS / \"report.md\"\n",
-    "\n",
-    "summary = json.loads(summary_path.read_text(encoding=\"utf-8\"))\n",
-    "display(summary)\n",
-    "display(pd.read_csv(study_table_path))\n",
-    "display(Markdown(report_path.read_text(encoding=\"utf-8\")))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "62c96dc6",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# 11. Create a small publishable artifact bundle (activation caches and checkpoint markers are excluded).\n",
-    "import zipfile\n",
-    "\n",
-    "PUBLISH_ZIP = DRIVE_ROOT / \"FeatureLens_offline_results.zip\"\n",
-    "\n",
-    "with zipfile.ZipFile(PUBLISH_ZIP, \"w\", compression=zipfile.ZIP_DEFLATED) as zf:\n",
-    "    for path in sorted(DRIVE_ARTIFACTS.rglob(\"*\")):\n",
-    "        if not path.is_file():\n",
-    "            continue\n",
-    "        rel = path.relative_to(DRIVE_ARTIFACTS)\n",
-    "        if rel.parts and rel.parts[0] == \"activations\":\n",
-    "            continue\n",
-    "        if path.name.endswith(\".complete\") or path.name.endswith(\".tmp\"):\n",
-    "            continue\n",
-    "        zf.write(path, arcname=str(Path(\"artifacts\") / rel))\n",
-    "\n",
-    "print(\"Publishable bundle:\", PUBLISH_ZIP)\n",
-    "print(f\"Size: {PUBLISH_ZIP.stat().st_size / 1024**2:.2f} MiB\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "a6634bfe",
-   "metadata": {},
-   "source": [
-    "## After Colab\n",
-    "\n",
-    "Download `FeatureLens_offline_results.zip` from the Drive run folder. Extract it over your local FeatureLens repository so the files land under `artifacts/`, run the normal release checks locally, inspect the measured report, and only then commit the small study artifacts. Do **not** commit `artifacts/activations/`."
-   ]
-  }
- ],
- "metadata": {
-  "accelerator": "GPU",
-  "colab": {
-   "name": "FeatureLens Offline Study",
-   "provenance": []
-  },
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "name": "python"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
+  "nbformat": 4,
+  "nbformat_minor": 5
 }

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "featurelens"
-version = "0.15.0"
+version = "0.16.0"
 description = "Causal sparse-feature interpretability workbench for Qwen3 and Qwen-Scope SAEs"
 requires-python = ">=3.10"
 

research_config.json

@@ -90,7 +90,7 @@
     "contrastive_continuation_preference_test",
     "feature_decoder_geometry"
   ],
-  "concept_candidate_discovery_metric": "balanced exploratory score = selectivity × target activation rate × log1p(target mean); causal-ready mode additionally requires current-token activity and log-scales that activation; raw mean-difference remains available as a scale-sensitive comparison",
+  "concept_candidate_discovery_metric": "balanced exploratory score = selectivity \u00d7 target activation rate \u00d7 log1p(target mean); causal-ready mode additionally requires current-token activity and log-scales that activation; raw mean-difference remains available as a scale-sensitive comparison",
   "completion_cue_scan": "final-token feature activation after controlled suffix/cue substitution",
   "live_features_v0_6": [
     "start_here_plain_language_onboarding",
@@ -205,5 +205,19 @@
     "muted_cross_target_chart_series",
     "colab_offline_runner_notebook",
     "task_level_causal_and_feature_set_resume"
+  ],
+  "offline_causal_position_policies": [
+    "final_token",
+    "max_feature_activation"
+  ],
+  "primary_offline_causal_position_policy": "max_feature_activation",
+  "offline_causal_statistical_unit": "causal task; average ablation and amplification within task before paired bootstrap/sign-flip inference",
+  "offline_features_v0_16": [
+    "final_token_vs_max_feature_activation_causal_position_sensitivity",
+    "causal_task_level_statistical_inference",
+    "coverage_separated_from_conditional_effect_strength",
+    "exact_small_sample_sign_flip_tests",
+    "causal_addendum_colab_runner",
+    "position_sensitivity_study_dashboard"
   ]
 }

scripts/release_check.py

@@ -22,6 +22,7 @@ REQUIRED = [
     'featurelens/study.py',
     'experiments/run_all.py',
     'experiments/run_causal.py',
+    'experiments/run_causal_addendum.py',
     'experiments/run_feature_sets.py',
     'experiments/analyze_stability.py',
     'experiments/analyze_study.py',
@@ -31,7 +32,9 @@ REQUIRED = [
     'docs/VALIDATION.md',
     'docs/OFFLINE_STUDY.md',
     'docs/COLAB.md',
+    'docs/CAUSAL_ADDENDUM.md',
     'notebooks/FeatureLens_Offline_Study_Colab.ipynb',
+    'notebooks/FeatureLens_Causal_Addendum_Colab.ipynb',
     'scripts/ui_smoke.py',
     'tests/test_offline_study.py',
     'scripts/validate_artifacts.py',
@@ -273,6 +276,25 @@ def check_config(config: dict) -> None:
             'research_config.json ui_and_runner_features_v0_15 mismatch: '
             f'{sorted(actual_v15)}'
         )
+
+    required_v16 = {
+        'final_token_vs_max_feature_activation_causal_position_sensitivity',
+        'causal_task_level_statistical_inference',
+        'coverage_separated_from_conditional_effect_strength',
+        'exact_small_sample_sign_flip_tests',
+        'causal_addendum_colab_runner',
+        'position_sensitivity_study_dashboard',
+    }
+    actual_v16 = set(config.get('offline_features_v0_16', []))
+    if actual_v16 != required_v16:
+        raise SystemExit(
+            'research_config.json offline_features_v0_16 mismatch: '
+            f'{sorted(actual_v16)}'
+        )
+    if config.get('offline_causal_position_policies') != ['final_token', 'max_feature_activation']:
+        raise SystemExit('Offline causal position policies must be final_token and max_feature_activation.')
+    if config.get('primary_offline_causal_position_policy') != 'max_feature_activation':
+        raise SystemExit('Primary offline causal position policy must be max_feature_activation.')
     if config.get('offline_selection_resamples') != 128:
         raise SystemExit('Offline selection resamples must be 128.')
     if 'prompt-wide' not in str(config.get('offline_feature_pooling', '')):
@@ -347,11 +369,14 @@ def check_readme() -> None:
         '--activation-batch-size',
         'validate_artifacts',
         'FeatureLens_Offline_Study_Colab.ipynb',
+        'FeatureLens_Causal_Addendum_Colab.ipynb',
+        'max-feature-activation',
+        'causal task',
         'DESIGN.md',
     ]
     missing = [value for value in required_strings if value.lower() not in readme.lower()]
     if missing:
-        raise SystemExit(f'README.md is missing required v0.15 content: {missing}')
+        raise SystemExit(f'README.md is missing required v0.16 content: {missing}')
 
     # Public README should not lead with release-train marketing. Version history belongs in CHANGELOG.
     if '> **v0.' in readme or '## v0.' in readme:
@@ -360,8 +385,8 @@ def check_readme() -> None:
 
 def check_pyproject() -> None:
     text = (ROOT / 'pyproject.toml').read_text(encoding='utf-8')
-    if 'version = "0.15.0"' not in text:
-        raise SystemExit('pyproject.toml must declare version 0.15.0.')
+    if 'version = "0.16.0"' not in text:
+        raise SystemExit('pyproject.toml must declare version 0.16.0.')
 
 
 def main() -> None:
@@ -379,7 +404,7 @@ def main() -> None:
     print(f'  layers: {config["layers"]}')
     print(f'  feature-set sizes: {config["feature_set_sizes"]}')
     print(f'  random controls: {config["live_random_controls"]}')
-    print('  release: v0.15.0')
+    print('  release: v0.16.0')
 
 
 if __name__ == '__main__':

scripts/validate_artifacts.py

@@ -9,12 +9,13 @@ ROOT = Path(__file__).resolve().parents[1]
 ARTIFACT_DIR = ROOT / 'artifacts'
 
 REQUIRED = [
-    'activations/metadata.json',
     'feature_catalog.csv',
     'layer_metrics.csv',
     'stability.csv',
     'selection_stability.csv',
-    'causal_results.csv',
+    'causal_results_final_token.csv',
+    'causal_results_max_active.csv',
+    'causal_position_summary.csv',
     'feature_set_results.csv',
     'study_feature_summary.csv',
     'study_summary.json',
@@ -37,68 +38,45 @@ def main() -> None:
     if missing:
         raise SystemExit(f'Missing offline-study artifacts: {missing}')
 
-    metadata = json.loads((ARTIFACT_DIR / 'activations' / 'metadata.json').read_text(encoding='utf-8'))
-    pooling = str(metadata.get('feature_pooling', ''))
-    if 'prompt-wide' not in pooling:
-        raise SystemExit(
-            'Activation metadata is not v0.14 prompt-wide. Rerun experiments.collect_activations.'
-        )
-
-    _require_columns(
-        ARTIFACT_DIR / 'feature_catalog.csv',
-        {'layer', 'concept', 'feature_id', 'train_auroc', 'auroc', 'f1'},
-    )
-    _require_columns(
-        ARTIFACT_DIR / 'selection_stability.csv',
-        {'layer', 'concept', 'feature_id', 'resample_support', 'median_resample_rank'},
-    )
-    _require_columns(
-        ARTIFACT_DIR / 'causal_results.csv',
-        {
-            'task_id',
-            'concept',
-            'feature_id',
-            'intervention',
-            'condition',
-            'target_mean_logprob_delta',
-            'js_divergence',
-        },
-    )
-    _require_columns(
-        ARTIFACT_DIR / 'study_feature_summary.csv',
-        {
-            'concept',
-            'layer',
-            'feature_id',
-            'heldout_auroc',
-            'heldout_f1',
-            'candidate_resample_support',
-            'target_specificity_ratio',
-            'js_specificity_ratio',
-        },
-    )
+    _require_columns(ARTIFACT_DIR / 'feature_catalog.csv', {'layer','concept','feature_id','train_auroc','auroc','f1'})
+    _require_columns(ARTIFACT_DIR / 'selection_stability.csv', {'layer','concept','feature_id','resample_support','median_resample_rank'})
+    causal_columns = {
+        'task_id','concept','feature_id','position_policy','intervention_token_index',
+        'feature_active_at_intervention','feature_active_at_final_token','feature_active_anywhere',
+        'intervention','condition','target_mean_logprob_delta','js_divergence',
+    }
+    _require_columns(ARTIFACT_DIR / 'causal_results_final_token.csv', causal_columns)
+    _require_columns(ARTIFACT_DIR / 'causal_results_max_active.csv', causal_columns)
+    _require_columns(ARTIFACT_DIR / 'causal_position_summary.csv', {
+        'concept','position_policy','feature_active_at_intervention_rate','target_specificity_ratio',
+        'target_paired_advantage','target_sign_flip_pvalue',
+    })
+    _require_columns(ARTIFACT_DIR / 'study_feature_summary.csv', {
+        'concept','layer','feature_id','heldout_auroc','heldout_f1','candidate_resample_support',
+        'final_target_specificity_ratio','max_active_target_specificity_ratio',
+        'final_feature_active_at_intervention_rate','max_active_feature_active_at_intervention_rate',
+    })
 
     summary = json.loads((ARTIFACT_DIR / 'study_summary.json').read_text(encoding='utf-8'))
     if int(summary.get('n_concepts', 0)) < 1:
         raise SystemExit('study_summary.json has no concepts.')
+    if summary.get('primary_causal_position_policy') != 'max_feature_activation':
+        raise SystemExit('study_summary.json must use max_feature_activation as the primary causal policy.')
+    if 'causal task' not in str(summary.get('causal_statistical_unit', '')).lower():
+        raise SystemExit('study_summary.json must document causal-task-level inference.')
 
     required_figures = [
-        'feature_auroc.png',
-        'layer_diagnostics.png',
-        'causal_effects.png',
-        'feature_set_effects.png',
-        'association_vs_causality.png',
-        'candidate_stability.png',
-    ]
-    missing_figures = [
-        name for name in required_figures if not (ARTIFACT_DIR / 'figures' / name).exists()
+        'feature_auroc.png','layer_diagnostics.png','causal_effects.png','feature_set_effects.png',
+        'association_vs_causality.png','causal_position_sensitivity.png',
     ]
+    missing_figures = [name for name in required_figures if not (ARTIFACT_DIR / 'figures' / name).exists()]
     if missing_figures:
         raise SystemExit(f'Missing report figures: {missing_figures}')
 
     print('FeatureLens offline artifact validation: PASS')
     print(f"  concepts: {summary['n_concepts']}")
-    print(f"  feature pooling: {summary['selected_feature_pooling']}")
+    print(f"  primary causal policy: {summary['primary_causal_position_policy']}")
+    print(f"  statistical unit: {summary['causal_statistical_unit']}")
     print('  report: artifacts/report.md')
 
 

tests/test_causal_position.py

@@ -0,0 +1,91 @@
+from __future__ import annotations
+
+import sys
+import types
+from pathlib import Path
+
+import numpy as np
+import pandas as pd
+import torch
+
+
+def _stub_transformers() -> None:
+    if 'transformers' in sys.modules:
+        return
+    stub = types.ModuleType('transformers')
+    stub.AutoModelForCausalLM = type('AutoModelForCausalLM', (), {})
+    stub.AutoTokenizer = type('AutoTokenizer', (), {})
+    sys.modules['transformers'] = stub
+
+
+def test_feature_trace_and_position_policy() -> None:
+    _stub_transformers()
+    from experiments.run_causal import choose_intervention_position, feature_activation_trace
+    from featurelens.sae import SparseEncoding
+
+    enc = SparseEncoding(
+        indices=torch.tensor([[1, 8], [7, 2], [7, 3]]),
+        values=torch.tensor([[2.0, 1.0], [4.0, 3.0], [9.0, 1.0]]),
+    )
+    trace = feature_activation_trace(enc, 7)
+    assert trace.tolist() == [0.0, 4.0, 9.0]
+    idx, activation, active = choose_intervention_position(
+        trace, prompt_len=3, position_policy='max_feature_activation'
+    )
+    assert (idx, activation, active) == (2, 9.0, True)
+    idx, activation, active = choose_intervention_position(
+        trace, prompt_len=3, position_policy='final_token'
+    )
+    assert (idx, activation, active) == (2, 9.0, True)
+
+
+def test_max_active_inactive_falls_back_to_final_token() -> None:
+    _stub_transformers()
+    from experiments.run_causal import choose_intervention_position
+
+    idx, activation, active = choose_intervention_position(
+        torch.zeros(4), prompt_len=4, position_policy='max_feature_activation'
+    )
+    assert idx == 3
+    assert activation == 0.0
+    assert not active
+
+
+def test_legacy_causal_baseline_migration(tmp_path: Path) -> None:
+    from experiments.run_causal_addendum import migrate_final_token_baseline
+
+    source = tmp_path / 'causal_results.csv'
+    destination = tmp_path / 'causal_results_final_token.csv'
+    pd.DataFrame(
+        [
+            {'task_id': 'a', 'feature_activation': 3.0, 'condition': 'sae_feature'},
+            {'task_id': 'a', 'feature_activation': 3.0, 'condition': 'random_norm_matched'},
+            {'task_id': 'b', 'feature_activation': 0.0, 'condition': 'sae_feature'},
+        ]
+    ).to_csv(source, index=False)
+    migrate_final_token_baseline(source, destination)
+    frame = pd.read_csv(destination)
+    assert set(frame['position_policy']) == {'final_token'}
+    assert frame['feature_active_at_intervention'].tolist() == [1, 1, 0]
+    assert frame['feature_active_at_final_token'].tolist() == [1, 1, 0]
+    assert frame['feature_active_anywhere'].isna().all()
+
+
+def test_task_level_specificity_averages_interventions_within_task() -> None:
+    from experiments.analyze_study import _task_level_specificity
+
+    rows = []
+    for intervention, sae_effect, random_effects in [
+        ('ablate', 0.4, [0.1, -0.1]),
+        ('amplify_2x', -0.2, [0.05, -0.15]),
+    ]:
+        rows.append({'task_id': 'a', 'intervention': intervention, 'condition': 'sae_feature', 'target_mean_logprob_delta': sae_effect})
+        for value in random_effects:
+            rows.append({'task_id': 'a', 'intervention': intervention, 'condition': 'random_norm_matched', 'target_mean_logprob_delta': value})
+    result = _task_level_specificity(pd.DataFrame(rows), effect_column='target_mean_logprob_delta', seed=1)
+    # One task: SAE mean absolute intervention effect=(.4+.2)/2=.3;
+    # random means=(.1,.1), then averaged within task=.1.
+    assert result['n_tasks'] == 1
+    assert np.isclose(result['sae_abs_mean'], 0.3)
+    assert np.isclose(result['random_abs_mean'], 0.1)
+    assert np.isclose(result['specificity_ratio'], 3.0)

tests/test_stats.py

@@ -31,3 +31,11 @@ def test_sign_flip_small_for_consistent_effect():
 def test_sign_flip_one_for_identical_pairs():
     x = np.array([1.0, 2.0, 3.0])
     assert paired_sign_flip_pvalue(x, x, n_permutations=1000, seed=4) == 1.0
+
+
+def test_sign_flip_is_exact_for_small_effective_sample() -> None:
+    from featurelens.stats import paired_sign_flip_pvalue
+
+    # With two positive non-zero differences, only the ++ and -- assignments
+    # are as extreme as the observed all-positive mean: p = 2 / 4.
+    assert paired_sign_flip_pvalue([1.0, 1.0], [0.0, 0.0]) == 0.5