Release FeatureLens v0.14.0

CHANGELOG.md

@@ -1,3 +1,17 @@
+# Changelog
+
+## v0.14.0
+
+- Transitioned the project from live-feature expansion toward the full offline empirical study.
+- Changed offline SAE concept evidence from final-token-only activations to **prompt-wide max-pooled activations across non-padding tokens**, while saving separate final-token sparse activation matrices for local diagnostics.
+- Added `experiments/analyze_stability.py` with 128 deterministic balanced activation resamples and per-feature shortlist support/rank summaries.
+- Added `experiments/analyze_study.py` to join held-out AUROC/F1, paraphrase robustness, candidate stability, feature activity, and random-normalized target/JS causal specificity by controlled concept.
+- Added descriptive cross-concept association-vs-causality correlations and new association/candidate-stability report figures.
+- Expanded the public **Offline study** tab into an artifact-backed results dashboard that remains explicitly empty until real study artifacts are committed.
+- Added `python experiments/run_all.py --resume` for interrupted/preemptible GPU sessions and `python experiments/run_analysis_only.py` for CPU-only re-analysis once inference artifacts exist.
+- Added `scripts/validate_artifacts.py` to verify prompt-wide activation provenance and public study artifact schemas before commit.
+- Added dedicated offline-study methodology/validation documentation and automated tests for prompt-wide pooling, stability scoring, task-paired specificity, study UI states, and correlation guardrails.
+
 ## v0.13.0
 
 - Added deterministic **32-resample candidate-support diagnostics** from the same concept-discovery activation batch; each displayed feature now reports shortlist support and median resample rank without another model forward.
@@ -5,7 +19,6 @@
 - Added **pairwise target-preference shifts** derived from the same cross-target scores: Δ(A−B) = Δmean(A) − Δmean(B), with a table and plot and no additional inference.
 - Kept HF acceptance quota-aware: only concept discovery and cross-target profiling are touched GPU paths.
 
-# Changelog
 
 ## v0.12.0
 

README.md

@@ -13,7 +13,7 @@ license: mit
 
 # FeatureLens — Causal Interpretability Workbench
 
-> **v0.13:** a stability-and-target-selectivity release that adds deterministic resample support to live discovery and extracts effect concentration, signed bias, and pairwise target-preference shifts from the existing cross-target GPU call.
+> **v0.14:** an offline-study transition release: prompt-wide held-out SAE discovery, activation-resample stability, cross-concept association-vs-causality synthesis, a resume-safe study runner, and an artifact-backed public study dashboard.
 
 **Research question:**
 
@@ -253,9 +253,9 @@ The offline pipeline computes:
 
 Feature selection is performed using the **training split only**. Paraphrase pairs never cross the train/test boundary.
 
-## Run the offline benchmark
+## Run the offline study
 
-A CUDA machine is strongly recommended.
+A CUDA machine is strongly recommended for activation collection and causal intervention stages. v0.14 changes the offline SAE concept representation to **prompt-wide max-pooled activation across all non-padding prompt tokens**. Final-token sparse activations are saved separately for local diagnostics; the dense residual linear probe remains a final-token baseline.
 
 ```bash
 python -m venv .venv
@@ -264,33 +264,72 @@ pip install -r requirements.txt
 python experiments/run_all.py
 ```
 
-The pipeline is:
+For interruptible/preemptible GPU sessions, use:
+
+```bash
+python experiments/run_all.py --resume
+```
+
+Completed stages are skipped when their expected artifacts already exist. After the expensive model/SAE stages have completed, all CPU-only evaluation/report logic can be rerun without another inference pass:
+
+```bash
+python experiments/run_analysis_only.py
+```
+
+The full pipeline is:
 
 ```text
 build_dataset
 → collect_activations
+  ├─ prompt-wide SAE feature maxima
+  └─ final-token sparse features + final-token residuals
 → evaluate_features
+  ├─ held-out AUROC/F1
+  ├─ dense residual linear probe
+  └─ prompt-wide paraphrase stability
 → run_causal
 → run_feature_sets
+→ analyze_stability
+  └─ 128 balanced activation resamples
+→ analyze_study
+  └─ association/stability ↔ random-normalized causal evidence
 → make_report
+→ validate_artifacts
 ```
 
 Outputs are generated under `artifacts/`:
 
 ```text
 artifacts/
-├── activations/
+├── activations/                     # large; keep gitignored
 ├── feature_catalog.csv
 ├── layer_metrics.csv
 ├── stability.csv
+├── selection_stability.csv
 ├── causal_results.csv
 ├── feature_set_results.csv
+├── study_feature_summary.csv
+├── study_summary.json
 ├── summary.json
 ├── report.md
 └── figures/
+    ├── feature_auroc.png
+    ├── layer_diagnostics.png
+    ├── causal_effects.png
+    ├── feature_set_effects.png
+    ├── association_vs_causality.png
+    └── candidate_stability.png
 ```
 
-`summary.json` and `report.md` are generated from measured artifacts. No scientific benchmark numbers are fabricated in the repository.
+Validate the small public artifacts before committing them:
+
+```bash
+python -m scripts.validate_artifacts
+```
+
+`summary.json`, `study_summary.json`, and `report.md` are generated from measured artifacts. No scientific benchmark numbers are fabricated in the repository. The live **Offline study** tab automatically turns into a results dashboard once these small CSV/JSON/figure artifacts are committed.
+
+See [`docs/OFFLINE_STUDY.md`](docs/OFFLINE_STUDY.md) for the staged workflow and interpretation guardrails.
 
 ## Hugging Face deployment
 
@@ -335,19 +374,24 @@ FeatureLens/
 │   ├── metrics.py
 │   ├── stats.py
 │   ├── selection.py
-│   └── catalog.py
+│   ├── catalog.py
+│   └── study.py
 ├── experiments/
 │   ├── build_dataset.py
 │   ├── collect_activations.py
 │   ├── evaluate_features.py
 │   ├── run_causal.py
 │   ├── run_feature_sets.py
+│   ├── analyze_stability.py
+│   ├── analyze_study.py
 │   ├── make_report.py
+│   ├── run_analysis_only.py
 │   └── run_all.py
 ├── data/
 ├── tests/
 ├── scripts/
 │   ├── release_check.py
+│   ├── validate_artifacts.py
 │   └── ui_smoke.py
 ├── docs/
 └── research_config.json
@@ -381,6 +425,17 @@ v0.12 bundles several related improvements instead of adding one isolated widget
 - **Cross-target causal profile** screens up to three candidate ablations across two to five exact continuations. It reports target-wise mean/sequence log-probability deltas, next-token JS, the strongest target per feature, and a target-profile ratio. This stage intentionally omits random controls; controlled candidate specificity remains the matched-random causal test.
 - HF validation remains GPU-budget-aware: rerun only the touched discovery path and the new cross-target path. Unchanged paraphrase, trajectory, set-size, dose-response, cue, and focus paths stay covered by automated tests.
 
+## v0.14 offline-study transition
+
+v0.14 deliberately stops expanding the live intervention surface and strengthens the empirical study behind it.
+
+- **Prompt-wide offline SAE evidence:** concept-feature discovery and paraphrase stability use each feature's maximum activation across all non-padding prompt tokens. This avoids treating an arbitrary final token as the semantic representation of an entire prompt. Final-token sparse activations remain available separately for local diagnostics.
+- **Activation-resample candidate selection stability:** `analyze_stability.py` performs 128 deterministic balanced resamples of the saved prompt-wide activations and records shortlist support plus median/mean resample rank. No model inference is repeated.
+- **Association vs random-normalized causality across concepts:** `analyze_study.py` joins held-out AUROC/F1, paraphrase stability, candidate resample support, causal active rate, target specificity, and JS specificity for the selected feature of every controlled concept. Cross-concept Spearman correlations are descriptive because there are only seven concepts.
+- **Artifact-backed Offline study tab:** the public app remains honest when no benchmark has been run, then automatically displays committed study tables and figures when real artifacts exist.
+- **Resume-safe full runner:** `python experiments/run_all.py --resume` skips completed stages; `python experiments/run_analysis_only.py` reruns only CPU evaluation/stability/report logic once expensive inference artifacts exist.
+- **Artifact schema validation:** `python -m scripts.validate_artifacts` checks that public study outputs are non-empty, prompt-wide v0.14 activations were used, and required causal/stability columns exist.
+
 ## Validation
 
 ```bash
@@ -407,7 +462,7 @@ See [`docs/VALIDATION.md`](docs/VALIDATION.md). The v0.12 guide uses the **exact
 ## Resume-ready description
 
 > **FeatureLens — Causal Interpretability Workbench** | PyTorch, Qwen3, Sparse Autoencoders, Mechanistic Interpretability, Gradio
-> Built an SAE-based interpretability system for Qwen3-1.7B with held-out concept discovery, concept-guided candidate discovery, token/prompt-wide, completion-cue, and cue × context feature evidence, reconstruction-preserving single and multi-feature interventions, full-continuation and contrastive preference scoring, dose-response analysis, decoder-geometry/non-additivity diagnostics, discovery-to-causality rank analysis, multi-candidate random-controlled specificity screening, split-half discovery stability, cross-target causal profiling, controlled evidence-pattern synthesis, and norm-matched random-control ensembles.
+> Built an SAE-based interpretability system for Qwen3-1.7B with held-out concept discovery, concept-guided candidate discovery, token/prompt-wide, completion-cue, and cue × context feature evidence, reconstruction-preserving single and multi-feature interventions, full-continuation and contrastive preference scoring, dose-response analysis, decoder-geometry/non-additivity diagnostics, discovery-to-causality rank analysis, multi-candidate random-controlled specificity screening, split-half/resample discovery stability, cross-target causal profiling, controlled evidence-pattern synthesis, prompt-wide held-out SAE evaluation, association-vs-random-normalized-causality study synthesis, and norm-matched random-control ensembles.
 
 ## Acknowledgements
 

app.py

@@ -6,6 +6,7 @@ import pandas as pd
 from featurelens.config import SETTINGS
 from featurelens.hf_runtime import gpu
 from featurelens.runtime import RUNTIME
+from featurelens.study import OfflineStudy
 
 # Restrained, print-inspired palette. The app deliberately avoids saturated dashboard colors.
 INK_TEAL = "#708B86"
@@ -15,6 +16,8 @@ INK_PLUM = "#82768F"
 INK_STONE = "#8E8A83"
 INK_BLUEGREY = "#71808A"
 
+STUDY = OfflineStudy()
+
 CSS = r"""
 .gradio-container {
   width: min(96vw, 1600px) !important;
@@ -3094,14 +3097,75 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
                     height=330,
                 )
 
-    with gr.Tab("Offline benchmark"):
-        gr.Markdown(RUNTIME.catalog.benchmark_markdown())
-        gr.Markdown(
-            "The offline pipeline remains the source of held-out feature/concept AUROC + F1, reconstruction "
-            "quality, paraphrase stability, dense residual linear probes, and causal intervention conclusions. "
-            "The live workbench uses batch-context null references and random-control ensembles; "
-            "the saved report should be generated only from real experiment artifacts."
-        )
+    with gr.Tab("Offline study"):
+        gr.Markdown(STUDY.overview_markdown())
+        gr.Markdown(STUDY.readiness_markdown(), elem_classes=["small-note"])
+
+        offline_study = STUDY.dataframe("study_feature_summary.csv")
+        offline_stability = STUDY.dataframe("selection_stability.csv")
+        offline_layers = STUDY.dataframe("layer_metrics.csv")
+
+        if not offline_study.empty:
+            with gr.Row(equal_height=False):
+                with gr.Column(scale=3):
+                    _table_heading("Selected feature evidence by concept")
+                    gr.Dataframe(
+                        value=offline_study,
+                        interactive=False,
+                        show_label=False,
+                        buttons=["fullscreen"],
+                        elem_classes=["result-table"],
+                        wrap=False,
+                        max_height=420,
+                    )
+                with gr.Column(scale=2):
+                    gr.Image(
+                        value=STUDY.figure("association_vs_causality.png"),
+                        label="Association evidence vs causal specificity",
+                        interactive=False,
+                        show_label=True,
+                        height=360,
+                    )
+
+            with gr.Row(equal_height=False):
+                with gr.Column(scale=3):
+                    _table_heading("Candidate selection stability")
+                    stability_preview = offline_stability.sort_values(
+                        ["resample_support", "full_score"], ascending=[False, False]
+                    ).head(40)
+                    gr.Dataframe(
+                        value=stability_preview,
+                        interactive=False,
+                        show_label=False,
+                        buttons=["fullscreen"],
+                        elem_classes=["result-table"],
+                        wrap=False,
+                        max_height=420,
+                    )
+                with gr.Column(scale=2):
+                    gr.Image(
+                        value=STUDY.figure("candidate_stability.png"),
+                        label="Selected-feature resample support",
+                        interactive=False,
+                        show_label=True,
+                        height=360,
+                    )
+
+            _table_heading("Layer diagnostics")
+            gr.Dataframe(
+                value=offline_layers,
+                interactive=False,
+                show_label=False,
+                buttons=["fullscreen"],
+                elem_classes=["result-table"],
+                wrap=False,
+                max_height=300,
+            )
+        else:
+            gr.Markdown(
+                "The public study tables and figures appear automatically after the small offline artifacts "
+                "are generated and committed. No placeholder scientific numbers are shown."
+            )
 
     with gr.Tab("Method"):
         gr.Markdown(

artifacts/README.md

@@ -2,22 +2,43 @@
 
 This directory intentionally ships without invented empirical results.
 
-Run:
+Run the full study on a CUDA machine:
 
 ```bash
 python experiments/run_all.py
 ```
 
-to create:
+For an interruptible session:
 
-- activation caches;
+```bash
+python experiments/run_all.py --resume
+```
+
+If the expensive activation and causal artifacts already exist, regenerate only CPU analysis/report outputs with:
+
+```bash
+python experiments/run_analysis_only.py
+```
+
+v0.14 produces:
+
+- prompt-wide activation caches plus separate final-token sparse activations;
 - `feature_catalog.csv`;
 - `layer_metrics.csv`;
 - `stability.csv`;
+- `selection_stability.csv`;
 - `causal_results.csv`;
 - `feature_set_results.csv`;
+- `study_feature_summary.csv`;
+- `study_summary.json`;
 - `summary.json`;
 - `report.md`;
-- report figures.
+- report figures including association-vs-causality and candidate-stability plots.
+
+Validate the public artifact set with:
+
+```bash
+python -m scripts.validate_artifacts
+```
 
-Large residual/activation arrays are ignored by Git. Commit only small CSV/JSON/report/figure outputs if you want the live Space to display benchmark-derived feature hints and measured conclusions.
+`artifacts/activations/` contains large residual/sparse caches and stays gitignored. Commit only the small CSV/JSON/report/figure outputs if you want the live **Offline study** tab to display measured results and benchmark-derived feature hints.

docs/METHODOLOGY.md

@@ -583,3 +583,21 @@ For one feature with target effects `Δ_t`, v0.13 normalizes `|Δ_t|` into a pro
 ### Pairwise preference shifts
 
 For every unordered target pair `(A, B)`, FeatureLens derives `Δ_pref(A,B) = Δmean(A) − Δmean(B)` from the already-computed teacher-forced mean-log-probability effects. This is the intervention-induced change in token-normalized preference for A relative to B. It does not add a model call and does not replace matched-random specificity controls.
+
+## v0.14 offline-study methodology
+
+### Prompt-wide SAE concept evidence
+
+Offline concept-feature evaluation now max-pools each SAE feature across every non-padding token in a prompt. This aligns the held-out study with the live prompt-wide diagnostics and avoids assigning an entire prompt's concept evidence to an arbitrary final token. Separate final-token SAE matrices are still saved for local analyses.
+
+### Activation-resample selection stability
+
+`experiments/analyze_stability.py` performs deterministic balanced bootstrap resamples over the already-saved prompt-wide activation matrix. Within each resample, candidates are ranked using the live-compatible balanced score `positive selectivity × target activation rate × log1p(target mean)`. The output records shortlist support and resample-rank summaries. This is a sensitivity analysis, not a confidence interval.
+
+### Study-level association vs causality
+
+`experiments/analyze_study.py` selects one feature per concept using the original train-only AUROC discipline and joins held-out AUROC/F1 with paraphrase robustness, activation-resample support, causal-task activity, target-specificity, and JS-specificity. Random-normalized causal quantities are paired at the task level before aggregation. Cross-concept Spearman correlations use only seven concepts and are therefore reported descriptively.
+
+### Reproducibility and artifact boundary
+
+The full GPU + CPU pipeline supports `--resume`. Once expensive inference outputs exist, `experiments/run_analysis_only.py` reruns only CPU evaluation/stability/report stages. `scripts/validate_artifacts.py` enforces the public artifact schema and confirms that v0.14 prompt-wide activation metadata was used before results are surfaced in the public Offline study tab.

docs/OFFLINE_STUDY.md

@@ -0,0 +1,77 @@
+# FeatureLens offline study
+
+## Goal
+
+The offline study answers the project question at dataset scale:
+
+> Do sparse features that predict a controlled concept on held-out prompts also produce behaviorally specific causal effects?
+
+The live app is exploratory. The offline study is where FeatureLens makes held-out, random-controlled, uncertainty-aware claims.
+
+## Representation choice
+
+v0.14 uses **prompt-wide max-pooled SAE activation** for concept evidence. For each prompt and SAE feature, the stored value is the maximum activation across non-padding prompt tokens. This change is motivated by the live finding that a strong final-token activation can reflect a lexical cue rather than the prompt's semantic concept.
+
+The collector also saves `features_final_layer{layer}.npz` so local final-token analyses remain reproducible. Dense residual linear probes continue to use the final prompt-token residual and are therefore reported as a separate baseline rather than as an identical pooling scheme.
+
+## Full pipeline
+
+```bash
+python experiments/run_all.py
+```
+
+Stages:
+
+1. `build_dataset` — materialize 224 discovery prompts and 28 causal tasks.
+2. `collect_activations` — Qwen3 residual capture; prompt-wide and final-token SAE feature artifacts.
+3. `evaluate_features` — grouped train/test split, train-only feature selection, held-out AUROC/F1, dense residual probe, paraphrase stability.
+4. `run_causal` — selected-feature ablation/amplification with exact continuation scoring and norm-matched random ensembles.
+5. `run_feature_sets` — top-1/3/5 joint ablations and random controls.
+6. `analyze_stability` — 128 balanced activation resamples from saved prompt-wide features.
+7. `analyze_study` — join predictive, robustness, stability, and random-normalized causal evidence by concept.
+8. `make_report` — measured figures, summary JSON, and narrative report.
+9. `validate_artifacts` — schema/completeness guard before committing public results.
+
+## Resume after interruption
+
+```bash
+python experiments/run_all.py --resume
+```
+
+The runner checks expected stage outputs and skips completed stages. This is intended for preemptible or quota-limited GPU sessions.
+
+## CPU-only re-analysis
+
+After activation and causal inference artifacts exist:
+
+```bash
+python experiments/run_analysis_only.py
+```
+
+This reruns feature evaluation, candidate stability, study synthesis, figures, report generation, and artifact validation without another model forward pass.
+
+## Main outputs
+
+`study_feature_summary.csv` contains one selected feature per controlled concept with:
+
+- held-out AUROC and F1;
+- training activation rates;
+- activation-resample selection support;
+- paraphrase TopK Jaccard and sparse cosine;
+- causal-task feature-active rate;
+- mean absolute and signed target effect;
+- norm-matched random mean absolute target effect;
+- target-specificity ratio, paired advantage, bootstrap CI, sign-flip p-value;
+- equivalent next-token JS specificity metrics.
+
+`study_summary.json` adds descriptive cross-concept Spearman correlations such as held-out AUROC versus target-specificity ratio. There are only seven controlled concepts, so these correlations are **descriptive**, not significance claims.
+
+## Interpretation guardrails
+
+- Prompt-wide max pooling detects whether a feature appears anywhere in the prompt; it discards token order.
+- A high held-out AUROC remains correlational evidence.
+- Candidate resample support measures shortlist sensitivity under the configured activation-resampling scheme, not feature truth or semantic purity.
+- A target-specificity ratio above 1 means the SAE edit moved the exact target more than the mean norm-matched random edit; uncertainty and task coverage still matter.
+- JS specificity asks a different question from target specificity: a feature can reshape the local distribution without specifically controlling the chosen target.
+- Cross-concept correlations have n=7 and are descriptive.
+- Large activation caches should not be committed to the repository.

docs/VALIDATION.md

@@ -1,8 +1,10 @@
-# FeatureLens v0.13 validation
+# FeatureLens v0.14 validation
 
-v0.13 changes only **concept-guided discovery post-processing** and **cross-target profile post-processing**. Both changes reuse activations/scores already produced by those GPU calls. Do not rerun unrelated live experiments.
+v0.14 changes the **offline study path and Offline study tab**. It does not change the live Qwen/SAE intervention callbacks validated in v0.13, so do not spend ZeroGPU quota rerunning paraphrase, dose-response, controlled specificity, cross-target, layer trajectory, or feature-set regressions.
 
-## Local release gate
+## A. Local release gate
+
+From the repository root:
 
 ```bash
 python3 -m pytest -q && \
@@ -12,78 +14,50 @@ python3 scripts/ui_smoke.py && \
 python3 scripts/release_check.py
 ```
 
-Expected release checker tail:
-
-```text
-FeatureLens release check: PASS
-  discovery prompts: 224
-  causal tasks: 28
-  layers: [4, 14, 26]
-  feature-set sizes: [1, 3, 5]
-  random controls: 8
-  release: v0.13.0
-```
-
-## HF GPU call 1 — discovery stability
-
-Exact path: **Feature evidence → A. Concept-guided candidate discovery**.
-
-Use:
+Expected automated test count for v0.14: **74**.
 
-- **Target concept:** `mathematics`
-- **Residual layer:** `14`
-- **Prompts per concept:** `4`
-- **Candidate features:** `12`
-- **Candidate ranking:** `Causal-ready at current token`
-- Current Workbench prompt: `The derivative of x squared is`
-- Current Workbench token index: `-1`
+## B. HF acceptance — zero GPU calls
 
-Click **Discover concept-associated candidates**.
+After pushing v0.14:
 
-Record only the new stability evidence:
+1. Open the Space and confirm startup is clean.
+2. Open **Offline study**.
+3. Before real artifacts are committed, verify it says **Offline study not materialized yet** and does not show fabricated tables or metrics.
+4. Confirm the rest of the live tabs render normally. Do not run GPU experiments solely for this release.
 
-- split-half shared count and Jaccard;
-- summary mean resample support;
-- number of displayed candidates with ≥75% resample support;
-- for the top five rows: **Feature id**, **Resample shortlist support**, **Median resample rank**.
+## C. Offline-study synthetic/software acceptance
 
-No other discovery table columns need to be recopied unless the candidate ranking unexpectedly changes.
+The automated suite covers:
 
-## HF GPU call 2 — richer cross-target profile
+- prompt-wide max pooling while ignoring padding;
+- balanced candidate score behavior;
+- random-ensemble pairing by causal task;
+- descriptive Spearman guardrails;
+- offline-study missing/complete UI state.
 
-Exact path: **Feature evidence → C. Cross-target causal profile**.
+The development integration gate additionally runs the CPU evaluation/stability/study/report stack on synthetic saved activations and validates the resulting artifact schema.
 
-Use:
+## D. Real offline-study acceptance
 
-- **Features for cross-target profile:** `25992`, `16369`, `21670`
-- **Exact target continuations:**
+When a CUDA run is available:
 
-```text
-2x
-x
-0
-x^2
+```bash
+python experiments/run_all.py --resume
 ```
 
-Click **Run cross-target causal profile**.
-
-Send:
-
-1. **Target-profile summary** with the new columns **Normalized effect entropy**, **Effect concentration**, **Signed bias**, and **Profile pattern**.
-2. The top six rows of **Pairwise target preference shifts** by `|Preference shift|`.
-3. A screenshot of the pairwise plot if convenient.
+Then:
 
-## Do not rerun for v0.13
+```bash
+python -m scripts.validate_artifacts
+```
 
-Do **not** spend ZeroGPU quota on:
+Required checks:
 
-- controlled candidate specificity;
-- candidate triage;
-- dose response;
-- paraphrase identity;
-- layer trajectory;
-- 1/3/5 feature-set sweep;
-- cue or cue × context tests;
-- focus/zoom behavior.
+- activation metadata says `prompt-wide max activation across non-padding tokens`;
+- `features_layer4/14/26.npz` exist for prompt-wide concept evidence;
+- `features_final_layer4/14/26.npz` exist for local final-token diagnostics;
+- `feature_catalog.csv`, `selection_stability.csv`, `causal_results.csv`, `feature_set_results.csv`, `study_feature_summary.csv`, `study_summary.json`, `summary.json`, and `report.md` are non-empty;
+- report figures include `association_vs_causality.png` and `candidate_stability.png`;
+- no large activation/model files are staged for Git.
 
-Those inference paths are unchanged in v0.13.
+After committing only the small study artifacts, restart the Space and verify **Offline study** automatically displays the measured study summary, selected-feature table, stability table, layer diagnostics, and figures.

experiments/analyze_stability.py

@@ -0,0 +1,214 @@
+from __future__ import annotations
+
+import argparse
+import csv
+import json
+from collections import defaultdict
+from pathlib import Path
+
+import numpy as np
+import scipy.sparse as sp
+
+from experiments.common import ARTIFACT_DIR
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description='Estimate prompt-wide candidate selection stability from saved SAE activations.'
+    )
+    parser.add_argument('--activation-dir', type=Path, default=ARTIFACT_DIR / 'activations')
+    parser.add_argument('--output', type=Path, default=ARTIFACT_DIR / 'selection_stability.csv')
+    parser.add_argument('--resamples', type=int, default=128)
+    parser.add_argument('--top-features', type=int, default=20)
+    parser.add_argument('--seed', type=int, default=42)
+    return parser.parse_args()
+
+
+def _vector_mean(matrix: sp.csr_matrix, indices: np.ndarray) -> np.ndarray:
+    if indices.size == 0:
+        return np.zeros(matrix.shape[1], dtype=np.float32)
+    return np.asarray(matrix[indices].mean(axis=0)).ravel().astype(np.float32, copy=False)
+
+
+def _vector_rate(binary: sp.csr_matrix, indices: np.ndarray) -> np.ndarray:
+    if indices.size == 0:
+        return np.zeros(binary.shape[1], dtype=np.float32)
+    return np.asarray(binary[indices].mean(axis=0)).ravel().astype(np.float32, copy=False)
+
+
+def balanced_candidate_score(
+    target_mean: np.ndarray,
+    other_mean: np.ndarray,
+    target_rate: np.ndarray,
+) -> np.ndarray:
+    """Live-compatible selectivity × coverage × log-magnitude candidate score."""
+    target = np.asarray(target_mean, dtype=np.float64)
+    other = np.asarray(other_mean, dtype=np.float64)
+    rate = np.asarray(target_rate, dtype=np.float64)
+    difference = target - other
+    denom = np.abs(target) + np.abs(other) + 1e-12
+    selectivity = np.where(difference > 0.0, difference / denom, 0.0)
+    return selectivity * np.clip(rate, 0.0, 1.0) * np.log1p(np.maximum(target, 0.0))
+
+
+def _rank_top(scores: np.ndarray, k: int) -> tuple[np.ndarray, np.ndarray]:
+    positive = np.flatnonzero(scores > 0.0)
+    if positive.size == 0:
+        return np.empty(0, dtype=int), np.empty(0, dtype=float)
+    k = min(int(k), int(positive.size))
+    candidate_scores = scores[positive]
+    local = np.argpartition(candidate_scores, -k)[-k:]
+    ids = positive[local]
+    ordered = np.argsort(scores[ids])[::-1]
+    ids = ids[ordered]
+    return ids.astype(int), scores[ids].astype(float)
+
+
+def _sample_indices(
+    concept_indices: dict[str, np.ndarray],
+    rng: np.random.Generator,
+) -> dict[str, np.ndarray]:
+    return {
+        concept: rng.choice(indices, size=indices.size, replace=True)
+        for concept, indices in concept_indices.items()
+    }
+
+
+def _concept_statistics(
+    matrix: sp.csr_matrix,
+    binary: sp.csr_matrix,
+    concept_indices: dict[str, np.ndarray],
+) -> tuple[dict[str, np.ndarray], dict[str, np.ndarray]]:
+    means = {
+        concept: _vector_mean(matrix, indices)
+        for concept, indices in concept_indices.items()
+    }
+    rates = {
+        concept: _vector_rate(binary, indices)
+        for concept, indices in concept_indices.items()
+    }
+    return means, rates
+
+
+def _score_for_concept(
+    concept: str,
+    means: dict[str, np.ndarray],
+    rates: dict[str, np.ndarray],
+) -> np.ndarray:
+    other_concepts = [name for name in means if name != concept]
+    other_mean = np.mean(np.stack([means[name] for name in other_concepts]), axis=0)
+    return balanced_candidate_score(means[concept], other_mean, rates[concept])
+
+
+def main() -> None:
+    args = parse_args()
+    if args.resamples < 1:
+        raise ValueError('--resamples must be at least 1.')
+    if args.top_features < 1:
+        raise ValueError('--top-features must be at least 1.')
+
+    metadata = json.loads((args.activation_dir / 'metadata.json').read_text(encoding='utf-8'))
+    pooling = metadata.get('feature_pooling', '')
+    if 'prompt-wide' not in pooling:
+        raise RuntimeError(
+            'Selection stability requires v0.14 prompt-wide activation artifacts. '
+            'Rerun experiments.collect_activations before this analysis.'
+        )
+
+    rows = metadata['rows']
+    layers = [int(layer) for layer in metadata['layers']]
+    concepts = sorted({row['concept'] for row in rows})
+    concept_indices = {
+        concept: np.array(
+            [idx for idx, row in enumerate(rows) if row['concept'] == concept],
+            dtype=int,
+        )
+        for concept in concepts
+    }
+    rng = np.random.default_rng(args.seed)
+    output_rows: list[dict] = []
+
+    for layer in layers:
+        matrix = sp.load_npz(args.activation_dir / f'features_layer{layer}.npz').tocsr()
+        binary = matrix.copy()
+        binary.data = np.ones_like(binary.data, dtype=np.float32)
+
+        full_means, full_rates = _concept_statistics(matrix, binary, concept_indices)
+        full_scores = {
+            concept: _score_for_concept(concept, full_means, full_rates)
+            for concept in concepts
+        }
+        full_orders: dict[str, np.ndarray] = {
+            concept: np.argsort(scores)[::-1]
+            for concept, scores in full_scores.items()
+        }
+        support: dict[tuple[str, int], int] = defaultdict(int)
+        ranks: dict[tuple[str, int], list[int]] = defaultdict(list)
+
+        for _ in range(args.resamples):
+            sampled = _sample_indices(concept_indices, rng)
+            means, rates = _concept_statistics(matrix, binary, sampled)
+            for concept in concepts:
+                scores = _score_for_concept(concept, means, rates)
+                ids, _ = _rank_top(scores, args.top_features)
+                for rank, feature_id in enumerate(ids.tolist(), start=1):
+                    key = (concept, int(feature_id))
+                    support[key] += 1
+                    ranks[key].append(rank)
+
+        for concept in concepts:
+            full_rank_map = np.empty(matrix.shape[1], dtype=np.int32)
+            full_rank_map[full_orders[concept]] = np.arange(1, matrix.shape[1] + 1, dtype=np.int32)
+            seen = {
+                feature_id
+                for (seen_concept, feature_id), count in support.items()
+                if seen_concept == concept and count > 0
+            }
+            full_ids, _ = _rank_top(full_scores[concept], max(args.top_features, 50))
+            seen.update(int(feature_id) for feature_id in full_ids.tolist())
+
+            for feature_id in seen:
+                key = (concept, feature_id)
+                feature_ranks = ranks.get(key, [])
+                output_rows.append(
+                    {
+                        'layer': layer,
+                        'concept': concept,
+                        'feature_id': feature_id,
+                        'full_score': float(full_scores[concept][feature_id]),
+                        'full_rank': int(full_rank_map[feature_id]),
+                        'resample_support': float(support.get(key, 0) / args.resamples),
+                        'median_resample_rank': (
+                            float(np.median(feature_ranks)) if feature_ranks else float('nan')
+                        ),
+                        'mean_resample_rank': (
+                            float(np.mean(feature_ranks)) if feature_ranks else float('nan')
+                        ),
+                        'resamples': int(args.resamples),
+                        'top_features_per_resample': int(args.top_features),
+                    }
+                )
+        print(f'Stability analysis complete for layer {layer}', flush=True)
+
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    fieldnames = [
+        'layer',
+        'concept',
+        'feature_id',
+        'full_score',
+        'full_rank',
+        'resample_support',
+        'median_resample_rank',
+        'mean_resample_rank',
+        'resamples',
+        'top_features_per_resample',
+    ]
+    with args.output.open('w', newline='', encoding='utf-8') as handle:
+        writer = csv.DictWriter(handle, fieldnames=fieldnames)
+        writer.writeheader()
+        writer.writerows(output_rows)
+    print(f'Wrote {len(output_rows)} selection-stability rows to {args.output}')
+
+
+if __name__ == '__main__':
+    main()

experiments/analyze_study.py

@@ -0,0 +1,252 @@
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+import numpy as np
+import pandas as pd
+from scipy.stats import spearmanr
+
+from experiments.common import ARTIFACT_DIR
+from featurelens.stats import paired_bootstrap_difference_ci, paired_sign_flip_pvalue
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description='Aggregate held-out association, stability, and causal evidence by concept.'
+    )
+    parser.add_argument('--artifact-dir', type=Path, default=ARTIFACT_DIR)
+    parser.add_argument('--seed', type=int, default=42)
+    return parser.parse_args()
+
+
+def selected_features(catalog: pd.DataFrame) -> pd.DataFrame:
+    scored = catalog.copy()
+    scored['activation_contrast'] = (
+        scored['activation_rate_pos'].astype(float) - scored['activation_rate_neg'].astype(float)
+    )
+    ordered = scored.sort_values(
+        ['concept', 'train_auroc', 'activation_contrast'],
+        ascending=[True, False, False],
+    )
+    return ordered.groupby('concept', as_index=False).first()
+
+
+def _paired_specificity(
+    frame: pd.DataFrame,
+    *,
+    effect_column: str,
+    task_column: str = 'task_id',
+    seed: int,
+) -> dict[str, float | list[float]]:
+    sae = (
+        frame[frame['condition'] == 'sae_feature']
+        .groupby(task_column, as_index=False)[effect_column]
+        .first()
+        .rename(columns={effect_column: 'sae_effect'})
+    )
+    random = (
+        frame[frame['condition'] == 'random_norm_matched']
+        .assign(_abs_effect=lambda data: np.abs(data[effect_column].astype(float)))
+        .groupby(task_column, as_index=False)['_abs_effect']
+        .mean()
+        .rename(columns={'_abs_effect': 'random_abs_effect'})
+    )
+    paired = sae.merge(random, on=task_column, how='inner')
+    if paired.empty:
+        return {
+            'sae_abs_mean': float('nan'),
+            'sae_signed_mean': float('nan'),
+            'random_abs_mean': float('nan'),
+            'specificity_ratio': float('nan'),
+            'paired_advantage': float('nan'),
+            'paired_advantage_ci_95': [float('nan'), float('nan')],
+            'paired_sign_flip_pvalue': float('nan'),
+            'n_tasks': 0,
+        }
+
+    sae_values = paired['sae_effect'].to_numpy(dtype=float)
+    sae_abs = np.abs(sae_values)
+    random_abs = paired['random_abs_effect'].to_numpy(dtype=float)
+    ci_low, ci_high = paired_bootstrap_difference_ci(sae_abs, random_abs, seed=seed)
+    return {
+        'sae_abs_mean': float(np.mean(sae_abs)),
+        'sae_signed_mean': float(np.mean(sae_values)),
+        'random_abs_mean': float(np.mean(random_abs)),
+        'specificity_ratio': float(np.mean(sae_abs) / max(float(np.mean(random_abs)), 1e-12)),
+        'paired_advantage': float(np.mean(sae_abs - random_abs)),
+        'paired_advantage_ci_95': [float(ci_low), float(ci_high)],
+        'paired_sign_flip_pvalue': float(
+            paired_sign_flip_pvalue(sae_abs, random_abs, seed=seed + 1)
+        ),
+        'n_tasks': int(len(paired)),
+    }
+
+
+def _safe_spearman(x: pd.Series, y: pd.Series) -> dict[str, float | int]:
+    a = pd.to_numeric(x, errors='coerce').to_numpy(dtype=float)
+    b = pd.to_numeric(y, errors='coerce').to_numpy(dtype=float)
+    mask = np.isfinite(a) & np.isfinite(b)
+    if int(mask.sum()) < 3:
+        return {'rho': float('nan'), 'pvalue': float('nan'), 'n': int(mask.sum())}
+    if np.unique(a[mask]).size < 2 or np.unique(b[mask]).size < 2:
+        return {'rho': float('nan'), 'pvalue': float('nan'), 'n': int(mask.sum())}
+    result = spearmanr(a[mask], b[mask])
+    return {
+        'rho': float(result.statistic),
+        'pvalue': float(result.pvalue),
+        'n': int(mask.sum()),
+    }
+
+
+def main() -> None:
+    args = parse_args()
+    artifact_dir = args.artifact_dir
+    catalog = pd.read_csv(artifact_dir / 'feature_catalog.csv')
+    causal = pd.read_csv(artifact_dir / 'causal_results.csv')
+    paraphrase = pd.read_csv(artifact_dir / 'stability.csv')
+    stability_path = artifact_dir / 'selection_stability.csv'
+    selection_stability = pd.read_csv(stability_path) if stability_path.exists() else pd.DataFrame()
+
+    selected = selected_features(catalog)
+    rows: list[dict] = []
+
+    for concept_idx, selected_row in selected.sort_values('concept').reset_index(drop=True).iterrows():
+        concept = str(selected_row['concept'])
+        layer = int(selected_row['layer'])
+        feature_id = int(selected_row['feature_id'])
+        causal_concept = causal[
+            (causal['concept'] == concept) & (causal['intervention'] == 'ablate')
+        ].copy()
+        target = _paired_specificity(
+            causal_concept,
+            effect_column='target_mean_logprob_delta',
+            seed=args.seed + 100 * concept_idx,
+        )
+        js = _paired_specificity(
+            causal_concept,
+            effect_column='js_divergence',
+            seed=args.seed + 100 * concept_idx + 17,
+        )
+        sae_rows = causal_concept[causal_concept['condition'] == 'sae_feature']
+
+        para_rows = paraphrase[
+            (paraphrase['concept'] == concept) & (paraphrase['layer'].astype(int) == layer)
+        ]
+        selection_row = pd.DataFrame()
+        if not selection_stability.empty:
+            selection_row = selection_stability[
+                (selection_stability['concept'] == concept)
+                & (selection_stability['layer'].astype(int) == layer)
+                & (selection_stability['feature_id'].astype(int) == feature_id)
+            ]
+
+        rows.append(
+            {
+                'concept': concept,
+                'layer': layer,
+                'feature_id': feature_id,
+                'train_auroc': float(selected_row['train_auroc']),
+                'heldout_auroc': float(selected_row['auroc']),
+                'heldout_f1': float(selected_row['f1']),
+                'activation_rate_pos_train': float(selected_row['activation_rate_pos']),
+                'activation_rate_neg_train': float(selected_row['activation_rate_neg']),
+                'candidate_resample_support': (
+                    float(selection_row.iloc[0]['resample_support'])
+                    if not selection_row.empty
+                    else 0.0
+                ),
+                'candidate_median_resample_rank': (
+                    float(selection_row.iloc[0]['median_resample_rank'])
+                    if not selection_row.empty
+                    else float('nan')
+                ),
+                'mean_paraphrase_topk_jaccard': (
+                    float(para_rows['topk_jaccard'].mean()) if not para_rows.empty else float('nan')
+                ),
+                'mean_paraphrase_sparse_cosine': (
+                    float(para_rows['sparse_cosine'].mean()) if not para_rows.empty else float('nan')
+                ),
+                'causal_feature_active_rate': (
+                    float(np.mean(sae_rows['feature_activation'].astype(float) > 0.0))
+                    if not sae_rows.empty
+                    else float('nan')
+                ),
+                'target_sae_abs_mean': target['sae_abs_mean'],
+                'target_sae_signed_mean': target['sae_signed_mean'],
+                'target_random_abs_mean': target['random_abs_mean'],
+                'target_specificity_ratio': target['specificity_ratio'],
+                'target_paired_advantage': target['paired_advantage'],
+                'target_paired_ci_low': float(target['paired_advantage_ci_95'][0]),
+                'target_paired_ci_high': float(target['paired_advantage_ci_95'][1]),
+                'target_sign_flip_pvalue': target['paired_sign_flip_pvalue'],
+                'js_sae_mean': js['sae_abs_mean'],
+                'js_random_mean': js['random_abs_mean'],
+                'js_specificity_ratio': js['specificity_ratio'],
+                'js_paired_advantage': js['paired_advantage'],
+                'js_paired_ci_low': float(js['paired_advantage_ci_95'][0]),
+                'js_paired_ci_high': float(js['paired_advantage_ci_95'][1]),
+                'js_sign_flip_pvalue': js['paired_sign_flip_pvalue'],
+                'causal_tasks': int(target['n_tasks']),
+            }
+        )
+
+    study = pd.DataFrame(rows)
+    study_path = artifact_dir / 'study_feature_summary.csv'
+    study.to_csv(study_path, index=False)
+
+    correlations = {
+        'heldout_auroc_vs_target_specificity': _safe_spearman(
+            study['heldout_auroc'], study['target_specificity_ratio']
+        ),
+        'heldout_auroc_vs_js_specificity': _safe_spearman(
+            study['heldout_auroc'], study['js_specificity_ratio']
+        ),
+        'heldout_f1_vs_target_specificity': _safe_spearman(
+            study['heldout_f1'], study['target_specificity_ratio']
+        ),
+        'candidate_resample_support_vs_target_specificity': _safe_spearman(
+            study['candidate_resample_support'], study['target_specificity_ratio']
+        ),
+    }
+
+    most_predictive = study.sort_values('heldout_auroc', ascending=False).iloc[0]
+    most_target_specific = study.sort_values('target_specificity_ratio', ascending=False).iloc[0]
+    most_js_specific = study.sort_values('js_specificity_ratio', ascending=False).iloc[0]
+    median_support = float(study['candidate_resample_support'].median())
+
+    summary = {
+        'n_concepts': int(len(study)),
+        'selected_feature_pooling': 'prompt-wide max SAE activation across non-padding prompt tokens',
+        'dense_probe_pooling': 'final prompt token residual',
+        'median_selected_feature_resample_support': median_support,
+        'most_predictive_concept': {
+            'concept': str(most_predictive['concept']),
+            'heldout_auroc': float(most_predictive['heldout_auroc']),
+        },
+        'highest_target_specificity': {
+            'concept': str(most_target_specific['concept']),
+            'ratio': float(most_target_specific['target_specificity_ratio']),
+        },
+        'highest_js_specificity': {
+            'concept': str(most_js_specific['concept']),
+            'ratio': float(most_js_specific['js_specificity_ratio']),
+        },
+        'correlations': correlations,
+        'guardrail': (
+            'Cross-concept Spearman correlations are descriptive because the study contains only seven '
+            'controlled concepts. Causal specificity remains evaluated per concept against matched random '
+            'controls rather than inferred from correlation alone.'
+        ),
+    }
+    (artifact_dir / 'study_summary.json').write_text(
+        json.dumps(summary, indent=2),
+        encoding='utf-8',
+    )
+    print(f'Wrote {study_path}')
+    print(f'Wrote {artifact_dir / "study_summary.json"}')
+
+
+if __name__ == '__main__':
+    main()

experiments/collect_activations.py

@@ -12,7 +12,7 @@ from transformers import AutoModelForCausalLM, AutoTokenizer
 from experiments.common import ARTIFACT_DIR, DATA_DIR, load_jsonl, set_seed
 from featurelens.config import SETTINGS
 from featurelens.metrics import reconstruction_metrics
-from featurelens.sae import SAEStore
+from featurelens.sae import SAEStore, SparseEncoding
 
 
 def parse_args() -> argparse.Namespace:
@@ -26,7 +26,7 @@ def parse_args() -> argparse.Namespace:
     return parser.parse_args()
 
 
-def _build_sparse(encodings: list, n_rows: int, width: int) -> sp.csr_matrix:
+def _build_sparse(encodings: list[SparseEncoding], n_rows: int, width: int) -> sp.csr_matrix:
     row_ids: list[int] = []
     col_ids: list[int] = []
     values: list[float] = []
@@ -40,6 +40,48 @@ def _build_sparse(encodings: list, n_rows: int, width: int) -> sp.csr_matrix:
     return sp.csr_matrix((values, (row_ids, col_ids)), shape=(n_rows, width), dtype=np.float32)
 
 
+def _promptwide_max_encoding(
+    token_encoding: SparseEncoding,
+    attention_mask: torch.Tensor,
+) -> list[SparseEncoding]:
+    """Max-pool sparse feature activations across non-padding tokens for each prompt."""
+    indices = token_encoding.indices.detach().cpu()
+    values = token_encoding.values.detach().float().cpu()
+    mask = attention_mask.detach().bool().cpu()
+    pooled: list[SparseEncoding] = []
+
+    for row_idx in range(indices.shape[0]):
+        feature_max: dict[int, float] = {}
+        valid_positions = torch.nonzero(mask[row_idx], as_tuple=False).reshape(-1).tolist()
+        for token_idx in valid_positions:
+            token_ids = indices[row_idx, token_idx].reshape(-1).tolist()
+            token_values = values[row_idx, token_idx].reshape(-1).tolist()
+            for feature_id, activation in zip(token_ids, token_values, strict=True):
+                activation = float(activation)
+                if activation <= 0.0:
+                    continue
+                feature_id = int(feature_id)
+                if activation > feature_max.get(feature_id, 0.0):
+                    feature_max[feature_id] = activation
+
+        if feature_max:
+            ordered = sorted(feature_max.items())
+            pooled.append(
+                SparseEncoding(
+                    indices=torch.tensor([item[0] for item in ordered], dtype=torch.long),
+                    values=torch.tensor([item[1] for item in ordered], dtype=torch.float32),
+                )
+            )
+        else:
+            pooled.append(
+                SparseEncoding(
+                    indices=torch.empty(0, dtype=torch.long),
+                    values=torch.empty(0, dtype=torch.float32),
+                )
+            )
+    return pooled
+
+
 def _make_capture_hook(
     captured: dict[int, torch.Tensor],
     layer: int,
@@ -81,7 +123,8 @@ def main() -> None:
     )
 
     residuals: dict[int, list[np.ndarray]] = {layer: [] for layer in args.layers}
-    encodings: dict[int, list] = {layer: [] for layer in args.layers}
+    final_encodings: dict[int, list[SparseEncoding]] = {layer: [] for layer in args.layers}
+    promptwide_encodings: dict[int, list[SparseEncoding]] = {layer: [] for layer in args.layers}
     recon_stats: dict[int, list[dict[str, float]]] = {layer: [] for layer in args.layers}
 
     for start in range(0, len(rows), args.batch_size):
@@ -110,19 +153,27 @@ def main() -> None:
 
         for layer in args.layers:
             sae = sae_store.get(layer)
-            final_token_residuals = captured[layer][:, -1, :]
-            batch_encoding = sae.encode(final_token_residuals)
+            hidden = captured[layer]
+            final_token_residuals = hidden[:, -1, :]
+            final_batch_encoding = sae.encode(final_token_residuals)
+            token_batch_encoding = sae.encode(hidden)
+            promptwide_batch = _promptwide_max_encoding(
+                token_batch_encoding,
+                batch['attention_mask'],
+            )
+
             for row_idx in range(final_token_residuals.shape[0]):
                 residual = final_token_residuals[row_idx]
-                from featurelens.sae import SparseEncoding
-
-                encoding = SparseEncoding(
-                    indices=batch_encoding.indices[row_idx],
-                    values=batch_encoding.values[row_idx],
+                final_encoding = SparseEncoding(
+                    indices=final_batch_encoding.indices[row_idx],
+                    values=final_batch_encoding.values[row_idx],
                 )
-                reconstructed = sae.decode_sparse(encoding)
-                residuals[layer].append(residual.detach().float().cpu().numpy().astype(np.float16))
-                encodings[layer].append(encoding)
+                reconstructed = sae.decode_sparse(final_encoding)
+                residuals[layer].append(
+                    residual.detach().float().cpu().numpy().astype(np.float16)
+                )
+                final_encodings[layer].append(final_encoding)
+                promptwide_encodings[layer].append(promptwide_batch[row_idx])
                 recon_stats[layer].append(reconstruction_metrics(residual, reconstructed))
 
         print(f'Processed {min(start + args.batch_size, len(rows))}/{len(rows)} prompts', flush=True)
@@ -130,18 +181,43 @@ def main() -> None:
     for layer in args.layers:
         residual_array = np.stack(residuals[layer], axis=0)
         np.save(args.output_dir / f'residuals_layer{layer}.npy', residual_array)
-        sparse = _build_sparse(encodings[layer], len(rows), SETTINGS.sae_width)
-        sp.save_npz(args.output_dir / f'features_layer{layer}.npz', sparse, compressed=True)
+
+        promptwide_sparse = _build_sparse(
+            promptwide_encodings[layer],
+            len(rows),
+            SETTINGS.sae_width,
+        )
+        sp.save_npz(
+            args.output_dir / f'features_layer{layer}.npz',
+            promptwide_sparse,
+            compressed=True,
+        )
+
+        final_sparse = _build_sparse(
+            final_encodings[layer],
+            len(rows),
+            SETTINGS.sae_width,
+        )
+        sp.save_npz(
+            args.output_dir / f'features_final_layer{layer}.npz',
+            final_sparse,
+            compressed=True,
+        )
+
         summary = {
             'layer': layer,
             'n_samples': len(rows),
-            'mean_cosine': float(np.mean([x['cosine'] for x in recon_stats[layer]])),
-            'mean_nmse': float(np.mean([x['nmse'] for x in recon_stats[layer]])),
-            'median_nmse': float(np.median([x['nmse'] for x in recon_stats[layer]])),
-            'mean_active_features': float(np.mean(np.diff(sparse.indptr))),
+            'mean_cosine': float(np.mean([item['cosine'] for item in recon_stats[layer]])),
+            'mean_nmse': float(np.mean([item['nmse'] for item in recon_stats[layer]])),
+            'median_nmse': float(np.median([item['nmse'] for item in recon_stats[layer]])),
+            'mean_active_features_final_token': float(np.mean(np.diff(final_sparse.indptr))),
+            'mean_active_features_promptwide': float(np.mean(np.diff(promptwide_sparse.indptr))),
         }
+        # Preserve the legacy key for report compatibility. It refers to final-token TopK activity.
+        summary['mean_active_features'] = summary['mean_active_features_final_token']
         (args.output_dir / f'reconstruction_layer{layer}.json').write_text(
-            json.dumps(summary, indent=2), encoding='utf-8'
+            json.dumps(summary, indent=2),
+            encoding='utf-8',
         )
 
     metadata = {
@@ -151,11 +227,18 @@ def main() -> None:
         'top_k': SETTINGS.sae_top_k,
         'width': SETTINGS.sae_width,
         'n_samples': len(rows),
+        'feature_pooling': 'prompt-wide max activation across non-padding tokens',
+        'feature_file_pattern': 'features_layer{layer}.npz',
+        'final_token_feature_file_pattern': 'features_final_layer{layer}.npz',
+        'dense_residual_pooling': 'final prompt token',
         'rows': rows,
     }
-    (args.output_dir / 'metadata.json').write_text(json.dumps(metadata, indent=2), encoding='utf-8')
+    (args.output_dir / 'metadata.json').write_text(
+        json.dumps(metadata, indent=2),
+        encoding='utf-8',
+    )
     print(f'Activation artifacts written to {args.output_dir}')
 
 
 if __name__ == '__main__':
-    main()
+    main()

experiments/evaluate_features.py

@@ -72,6 +72,11 @@ def main() -> None:
     args = parse_args()
     args.output_dir.mkdir(parents=True, exist_ok=True)
     metadata = json.loads((args.activation_dir / 'metadata.json').read_text(encoding='utf-8'))
+    if 'prompt-wide' not in str(metadata.get('feature_pooling', '')):
+        raise RuntimeError(
+            'v0.14 evaluation requires prompt-wide activation artifacts. '
+            'Rerun experiments.collect_activations before evaluating features.'
+        )
     rows = metadata['rows']
     layers = [int(x) for x in metadata['layers']]
     train_idx, test_idx = grouped_concept_split(rows, seed=args.seed)

experiments/make_report.py

@@ -118,6 +118,36 @@ def _save_plots(
         plt.close(figure)
 
 
+
+def _save_study_plots(artifact_dir: Path, study: pd.DataFrame) -> None:
+    if study.empty:
+        return
+    fig_dir = artifact_dir / 'figures'
+    fig_dir.mkdir(parents=True, exist_ok=True)
+
+    figure = plt.figure(figsize=(7.2, 4.6))
+    ax = figure.add_subplot(111)
+    ax.scatter(study['heldout_auroc'], study['target_specificity_ratio'])
+    for row in study.itertuples():
+        ax.annotate(str(row.concept), (row.heldout_auroc, row.target_specificity_ratio), fontsize=8)
+    ax.set_xlabel('Held-out feature AUROC')
+    ax.set_ylabel('Target causal specificity ratio')
+    ax.set_title('Association evidence vs random-normalized causality')
+    figure.tight_layout()
+    figure.savefig(fig_dir / 'association_vs_causality.png', dpi=160)
+    plt.close(figure)
+
+    figure = plt.figure(figsize=(7.4, 4.6))
+    ax = figure.add_subplot(111)
+    ordered = study.sort_values('candidate_resample_support')
+    ax.barh(ordered['concept'], ordered['candidate_resample_support'])
+    ax.set_xlim(0, 1.02)
+    ax.set_xlabel('Selection support across activation resamples')
+    ax.set_title('Selected-feature candidate stability')
+    figure.tight_layout()
+    figure.savefig(fig_dir / 'candidate_stability.png', dpi=160)
+    plt.close(figure)
+
 def _paired_stats(
     frame: pd.DataFrame,
     *,
@@ -175,8 +205,17 @@ def main() -> None:
     causal = pd.read_csv(args.artifact_dir / 'causal_results.csv')
     feature_set_path = args.artifact_dir / 'feature_set_results.csv'
     feature_sets = pd.read_csv(feature_set_path) if feature_set_path.exists() else None
+    study_path = args.artifact_dir / 'study_feature_summary.csv'
+    study = pd.read_csv(study_path) if study_path.exists() else pd.DataFrame()
+    study_summary_path = args.artifact_dir / 'study_summary.json'
+    study_summary = (
+        json.loads(study_summary_path.read_text(encoding='utf-8'))
+        if study_summary_path.exists()
+        else {}
+    )
     selected = _selected_features(catalog)
     _save_plots(args.artifact_dir, selected, layers, causal, feature_sets)
+    _save_study_plots(args.artifact_dir, study)
 
     mean_auc = float(selected['auroc'].mean())
     median_auc = float(selected['auroc'].median())
@@ -299,6 +338,18 @@ def main() -> None:
             f'sign-flip p={float(largest_set["pvalue"]):.4f}.'
         )
 
+    if study_summary:
+        correlations = study_summary.get('correlations', {})
+        assoc_target = correlations.get('heldout_auroc_vs_target_specificity', {})
+        assoc_js = correlations.get('heldout_auroc_vs_js_specificity', {})
+        highlights.extend(
+            [
+                f'Selected-feature median activation-resample support: {float(study_summary.get("median_selected_feature_resample_support", float("nan"))):.1%}.',
+                f'Across concepts, held-out AUROC vs target-specificity Spearman ρ={float(assoc_target.get("rho", float("nan"))):+.3f} (n={int(assoc_target.get("n", 0))}); descriptive only.',
+                f'Across concepts, held-out AUROC vs JS-specificity Spearman ρ={float(assoc_js.get("rho", float("nan"))):+.3f} (n={int(assoc_js.get("n", 0))}); descriptive only.',
+            ]
+        )
+
     summary = {
         'headline': headline,
         'highlights': highlights,
@@ -321,6 +372,7 @@ def main() -> None:
             'causal_prompt_feature_active_rate': active_rate,
             'sae_top1_change_rate': top1_change,
             'feature_set_results': {str(k): value for k, value in set_summary.items()},
+            'study_summary': study_summary,
         },
     }
     (args.artifact_dir / 'summary.json').write_text(json.dumps(summary, indent=2), encoding='utf-8')
@@ -347,7 +399,9 @@ def main() -> None:
         '- Model: Qwen3-1.7B-Base.',
         '- SAEs: Qwen-Scope residual-stream TopK SAEs at configured early/middle/late layers.',
         '- Discovery set: controlled concept prompts with paired paraphrases.',
+        '- SAE concept evidence: prompt-wide maximum activation per feature across non-padding prompt tokens; final-token activations are saved separately for local diagnostics.',
         '- Split discipline: paraphrase groups stay entirely in train or held-out test.',
+        '- Candidate stability: deterministic balanced activation resamples estimate how often selected features survive small changes in the discovery sample.',
         '- Feature selection: training-split AUROC and activation contrast; held-out AUROC/F1 are reported separately.',
         '- Linear baseline: multinomial logistic regression on the dense residual stream.',
         '- Single-feature causal edit: reconstruction-preserving decoder-direction delta patched into the original residual.',
@@ -367,6 +421,19 @@ def main() -> None:
     ]
     if feature_sets is not None and not feature_sets.empty:
         lines.extend(['', '![Feature-set causal effects](figures/feature_set_effects.png)'])
+    if not study.empty:
+        lines.extend(
+            [
+                '',
+                '![Association vs causality](figures/association_vs_causality.png)',
+                '',
+                '![Candidate stability](figures/candidate_stability.png)',
+                '',
+                '## Association vs causality across concepts',
+                '',
+                "The offline study joins each concept-selected feature's held-out AUROC/F1 and activation-resample support with random-normalized causal specificity on the held-out causal task set. Cross-concept correlations are descriptive because there are only seven controlled concepts.",
+            ]
+        )
     lines.extend(
         [
             '',
@@ -376,7 +443,7 @@ def main() -> None:
             '',
             '## Reproducibility',
             '',
-            'Run `python experiments/run_all.py` from the repository root. Raw activation matrices, splits, selected features, single-feature rows, feature-set rows, figures, and this report are materialized under `artifacts/`.',
+            'Run `python experiments/run_all.py` from the repository root for the full GPU + CPU study. If model/SAE activations and causal rows already exist, run `python experiments/run_analysis_only.py` to regenerate CPU evaluation, stability, study synthesis, figures, validation, and this report without another model download or inference pass.',
             '',
         ]
     )

experiments/run_all.py

@@ -1,26 +1,93 @@
 from __future__ import annotations
 
+import argparse
 import subprocess
 import sys
 from pathlib import Path
 
+from featurelens.config import SETTINGS
+
 ROOT = Path(__file__).resolve().parents[1]
 
 
-def run(script: str, *args: str) -> None:
-    module = f"experiments.{Path(script).stem}"
-    command = [sys.executable, '-m', module, *args]
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description='Run the full FeatureLens offline study.')
+    parser.add_argument(
+        '--resume',
+        action='store_true',
+        help='Skip stages whose expected outputs already exist.',
+    )
+    return parser.parse_args()
+
+
+def run(module: str, *, outputs: list[Path], resume: bool) -> None:
+    if resume and outputs and all(path.exists() for path in outputs):
+        print(f'\nSKIP {module}: expected outputs already exist.', flush=True)
+        return
+    command = [sys.executable, '-m', module]
     print('\n$', ' '.join(command), flush=True)
     subprocess.run(command, cwd=ROOT, check=True)
 
 
 def main() -> None:
-    run('build_dataset.py')
-    run('collect_activations.py')
-    run('evaluate_features.py')
-    run('run_causal.py')
-    run('run_feature_sets.py')
-    run('make_report.py')
+    args = parse_args()
+    artifact_dir = ROOT / 'artifacts'
+    activation_dir = artifact_dir / 'activations'
+
+    run(
+        'experiments.build_dataset',
+        outputs=[ROOT / 'data' / 'prompts.jsonl', ROOT / 'data' / 'causal_tasks.jsonl'],
+        resume=args.resume,
+    )
+    run(
+        'experiments.collect_activations',
+        outputs=[
+            activation_dir / 'metadata.json',
+            *[activation_dir / f'features_layer{layer}.npz' for layer in SETTINGS.layers],
+            *[activation_dir / f'features_final_layer{layer}.npz' for layer in SETTINGS.layers],
+        ],
+        resume=args.resume,
+    )
+    run(
+        'experiments.evaluate_features',
+        outputs=[
+            artifact_dir / 'feature_catalog.csv',
+            artifact_dir / 'layer_metrics.csv',
+            artifact_dir / 'stability.csv',
+            artifact_dir / 'split.json',
+        ],
+        resume=args.resume,
+    )
+    run(
+        'experiments.run_causal',
+        outputs=[artifact_dir / 'causal_results.csv'],
+        resume=args.resume,
+    )
+    run(
+        'experiments.run_feature_sets',
+        outputs=[artifact_dir / 'feature_set_results.csv'],
+        resume=args.resume,
+    )
+    run(
+        'experiments.analyze_stability',
+        outputs=[artifact_dir / 'selection_stability.csv'],
+        resume=args.resume,
+    )
+    run(
+        'experiments.analyze_study',
+        outputs=[artifact_dir / 'study_feature_summary.csv', artifact_dir / 'study_summary.json'],
+        resume=args.resume,
+    )
+    run(
+        'experiments.make_report',
+        outputs=[artifact_dir / 'summary.json', artifact_dir / 'report.md'],
+        resume=args.resume,
+    )
+    subprocess.run(
+        [sys.executable, '-m', 'scripts.validate_artifacts'],
+        cwd=ROOT,
+        check=True,
+    )
     print('\nFeatureLens experiment pipeline complete. See artifacts/report.md')
 
 

experiments/run_analysis_only.py

@@ -0,0 +1,26 @@
+from __future__ import annotations
+
+import subprocess
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+
+
+def run(module: str, *args: str) -> None:
+    command = [sys.executable, '-m', module, *args]
+    print('\n$', ' '.join(command), flush=True)
+    subprocess.run(command, cwd=ROOT, check=True)
+
+
+def main() -> None:
+    run('experiments.evaluate_features')
+    run('experiments.analyze_stability')
+    run('experiments.analyze_study')
+    run('experiments.make_report')
+    run('scripts.validate_artifacts')
+    print('\nFeatureLens CPU analysis pipeline complete. See artifacts/report.md')
+
+
+if __name__ == '__main__':
+    main()

featurelens/study.py

@@ -0,0 +1,90 @@
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import pandas as pd
+
+
+class OfflineStudy:
+    """Read committed offline-study artifacts without invoking the model."""
+
+    REQUIRED = (
+        'summary.json',
+        'study_summary.json',
+        'study_feature_summary.csv',
+        'selection_stability.csv',
+        'feature_catalog.csv',
+        'layer_metrics.csv',
+        'stability.csv',
+        'causal_results.csv',
+        'feature_set_results.csv',
+        'report.md',
+    )
+
+    def __init__(self, artifact_dir: str | Path = 'artifacts') -> None:
+        self.artifact_dir = Path(artifact_dir)
+
+    @property
+    def missing(self) -> list[str]:
+        return [name for name in self.REQUIRED if not (self.artifact_dir / name).exists()]
+
+    @property
+    def complete(self) -> bool:
+        return not self.missing
+
+    def _json(self, name: str) -> dict:
+        path = self.artifact_dir / name
+        if not path.exists():
+            return {}
+        return json.loads(path.read_text(encoding='utf-8'))
+
+    def dataframe(self, name: str) -> pd.DataFrame:
+        path = self.artifact_dir / name
+        if not path.exists():
+            return pd.DataFrame()
+        return pd.read_csv(path)
+
+    def figure(self, name: str) -> str | None:
+        path = self.artifact_dir / 'figures' / name
+        return str(path) if path.exists() else None
+
+    def overview_markdown(self) -> str:
+        if not self.complete:
+            missing = ', '.join(f'`{name}`' for name in self.missing[:6])
+            suffix = '…' if len(self.missing) > 6 else ''
+            return (
+                '### Offline study not materialized yet\n\n'
+                'The live workbench is usable now, but held-out study artifacts have not been committed. '
+                f'Missing: {missing}{suffix}\n\n'
+                'Run `python experiments/run_all.py` on a CUDA machine for the full study. If activation and '
+                'causal artifacts already exist, use `python experiments/run_analysis_only.py` to rerun the '
+                'CPU-only evaluation, stability analysis, evidence synthesis, figures, and report. '
+                'Then run `python -m scripts.validate_artifacts` before committing the small CSV/JSON/report '
+                'outputs. Do not commit `artifacts/activations/`.'
+            )
+
+        summary = self._json('summary.json')
+        study = self._json('study_summary.json')
+        correlations = study.get('correlations', {})
+        target_corr = correlations.get('heldout_auroc_vs_target_specificity', {})
+        js_corr = correlations.get('heldout_auroc_vs_js_specificity', {})
+        return (
+            '### Offline study results\n\n'
+            f"{summary.get('headline', 'Benchmark completed.')}\n\n"
+            f"{summary.get('interpretation', '')}\n\n"
+            '**Study-level diagnostics**\n\n'
+            f"- Selected-feature median resample support: **{float(study.get('median_selected_feature_resample_support', float('nan'))):.1%}**.\n"
+            f"- Held-out AUROC ↔ target-specificity Spearman ρ: **{float(target_corr.get('rho', float('nan'))):+.3f}** (n={int(target_corr.get('n', 0))}).\n"
+            f"- Held-out AUROC ↔ JS-specificity Spearman ρ: **{float(js_corr.get('rho', float('nan'))):+.3f}** (n={int(js_corr.get('n', 0))}).\n\n"
+            'The cross-concept correlations are descriptive because there are only seven controlled concepts. '
+            'Per-concept causal claims remain anchored to matched random-control comparisons.'
+        )
+
+    def readiness_markdown(self) -> str:
+        if self.complete:
+            return '**Artifact status:** complete and ready for the public study view.'
+        return (
+            f'**Artifact status:** {len(self.REQUIRED) - len(self.missing)}/{len(self.REQUIRED)} required '
+            f'files present; {len(self.missing)} missing.'
+        )

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "featurelens"
-version = "0.13.0"
+version = "0.14.0"
 description = "Causal sparse-feature interpretability workbench for Qwen3 and Qwen-Scope SAEs"
 requires-python = ">=3.10"
 

research_config.json

@@ -90,7 +90,7 @@
     "contrastive_continuation_preference_test",
     "feature_decoder_geometry"
   ],
-  "concept_candidate_discovery_metric": "balanced exploratory score = selectivity × target activation rate × log1p(target mean); causal-ready mode additionally requires current-token activity and log-scales that activation; raw mean-difference remains available as a scale-sensitive comparison",
+  "concept_candidate_discovery_metric": "balanced exploratory score = selectivity \u00d7 target activation rate \u00d7 log1p(target mean); causal-ready mode additionally requires current-token activity and log-scales that activation; raw mean-difference remains available as a scale-sensitive comparison",
   "completion_cue_scan": "final-token feature activation after controlled suffix/cue substitution",
   "live_features_v0_6": [
     "start_here_plain_language_onboarding",
@@ -177,5 +177,24 @@
     "pairwise_target_preference_shifts",
     "zero_extra_gpu_evidence_synthesis",
     "touched_path_only_hf_validation"
+  ],
+  "offline_feature_pooling": "prompt-wide max SAE activation across non-padding prompt tokens; final-token sparse activations saved separately",
+  "offline_selection_resamples": 128,
+  "offline_study_outputs": [
+    "selection_stability.csv",
+    "study_feature_summary.csv",
+    "study_summary.json",
+    "summary.json",
+    "report.md"
+  ],
+  "offline_features_v0_14": [
+    "promptwide_offline_sae_feature_pooling",
+    "separate_final_token_sparse_activation_artifacts",
+    "activation_resample_candidate_stability",
+    "cross_concept_association_vs_random_normalized_causality",
+    "offline_study_dashboard",
+    "resume_safe_full_study_runner",
+    "cpu_only_analysis_rerun",
+    "offline_artifact_schema_validation"
   ]
 }

scripts/release_check.py

@@ -18,13 +18,20 @@ REQUIRED = [
     'featurelens/interventions.py',
     'featurelens/metrics.py',
     'featurelens/stats.py',
+    'featurelens/study.py',
     'experiments/run_all.py',
     'experiments/run_causal.py',
     'experiments/run_feature_sets.py',
+    'experiments/analyze_stability.py',
+    'experiments/analyze_study.py',
+    'experiments/run_analysis_only.py',
     'data/prompts.jsonl',
     'data/causal_tasks.jsonl',
     'docs/VALIDATION.md',
+    'docs/OFFLINE_STUDY.md',
     'scripts/ui_smoke.py',
+    'tests/test_offline_study.py',
+    'scripts/validate_artifacts.py',
 ]
 
 
@@ -231,6 +238,27 @@ def check_config(config: dict) -> None:
         raise SystemExit(
             'research_config.json live_features_v0_13 mismatch: ' f'{sorted(actual_live_v13)}'
         )
+    required_offline_v14 = {
+        'promptwide_offline_sae_feature_pooling',
+        'separate_final_token_sparse_activation_artifacts',
+        'activation_resample_candidate_stability',
+        'cross_concept_association_vs_random_normalized_causality',
+        'offline_study_dashboard',
+        'resume_safe_full_study_runner',
+        'cpu_only_analysis_rerun',
+        'offline_artifact_schema_validation',
+    }
+    actual_offline_v14 = set(config.get('offline_features_v0_14', []))
+    if actual_offline_v14 != required_offline_v14:
+        raise SystemExit(
+            'research_config.json offline_features_v0_14 mismatch: '
+            f'{sorted(actual_offline_v14)}'
+        )
+    if config.get('offline_selection_resamples') != 128:
+        raise SystemExit('Offline selection resamples must be 128.')
+    if 'prompt-wide' not in str(config.get('offline_feature_pooling', '')):
+        raise SystemExit('Offline feature pooling must be prompt-wide.')
+
     if config.get('discovery_resample_replicates') != 32:
         raise SystemExit('Discovery live resample count must be 32.')
     if config.get('cross_target_feature_limit') != 3 or config.get('cross_target_target_limit') != 5:
@@ -330,16 +358,22 @@ def check_readme() -> None:
         'pairwise target preference',
         'effect concentration',
         'signed bias',
+        'prompt-wide',
+        'offline study',
+        'selection stability',
+        'run_analysis_only',
+        '--resume',
+        'validate_artifacts',
     ]
     missing = [value for value in required_strings if value.lower() not in readme.lower()]
     if missing:
-        raise SystemExit(f'README.md is missing required v0.13 content: {missing}')
+        raise SystemExit(f'README.md is missing required v0.14 content: {missing}')
 
 
 def check_pyproject() -> None:
     text = (ROOT / 'pyproject.toml').read_text(encoding='utf-8')
-    if 'version = "0.13.0"' not in text:
-        raise SystemExit('pyproject.toml must declare version 0.13.0.')
+    if 'version = "0.14.0"' not in text:
+        raise SystemExit('pyproject.toml must declare version 0.14.0.')
 
 
 def main() -> None:
@@ -357,7 +391,7 @@ def main() -> None:
     print(f'  layers: {config["layers"]}')
     print(f'  feature-set sizes: {config["feature_set_sizes"]}')
     print(f'  random controls: {config["live_random_controls"]}')
-    print('  release: v0.13.0')
+    print('  release: v0.14.0')
 
 
 if __name__ == '__main__':

scripts/validate_artifacts.py

@@ -0,0 +1,106 @@
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import pandas as pd
+
+ROOT = Path(__file__).resolve().parents[1]
+ARTIFACT_DIR = ROOT / 'artifacts'
+
+REQUIRED = [
+    'activations/metadata.json',
+    'feature_catalog.csv',
+    'layer_metrics.csv',
+    'stability.csv',
+    'selection_stability.csv',
+    'causal_results.csv',
+    'feature_set_results.csv',
+    'study_feature_summary.csv',
+    'study_summary.json',
+    'summary.json',
+    'report.md',
+]
+
+
+def _require_columns(path: Path, columns: set[str]) -> None:
+    frame = pd.read_csv(path)
+    missing = columns.difference(frame.columns)
+    if missing:
+        raise SystemExit(f'{path.relative_to(ROOT)} missing columns: {sorted(missing)}')
+    if frame.empty:
+        raise SystemExit(f'{path.relative_to(ROOT)} is empty.')
+
+
+def main() -> None:
+    missing = [name for name in REQUIRED if not (ARTIFACT_DIR / name).exists()]
+    if missing:
+        raise SystemExit(f'Missing offline-study artifacts: {missing}')
+
+    metadata = json.loads((ARTIFACT_DIR / 'activations' / 'metadata.json').read_text(encoding='utf-8'))
+    pooling = str(metadata.get('feature_pooling', ''))
+    if 'prompt-wide' not in pooling:
+        raise SystemExit(
+            'Activation metadata is not v0.14 prompt-wide. Rerun experiments.collect_activations.'
+        )
+
+    _require_columns(
+        ARTIFACT_DIR / 'feature_catalog.csv',
+        {'layer', 'concept', 'feature_id', 'train_auroc', 'auroc', 'f1'},
+    )
+    _require_columns(
+        ARTIFACT_DIR / 'selection_stability.csv',
+        {'layer', 'concept', 'feature_id', 'resample_support', 'median_resample_rank'},
+    )
+    _require_columns(
+        ARTIFACT_DIR / 'causal_results.csv',
+        {
+            'task_id',
+            'concept',
+            'feature_id',
+            'intervention',
+            'condition',
+            'target_mean_logprob_delta',
+            'js_divergence',
+        },
+    )
+    _require_columns(
+        ARTIFACT_DIR / 'study_feature_summary.csv',
+        {
+            'concept',
+            'layer',
+            'feature_id',
+            'heldout_auroc',
+            'heldout_f1',
+            'candidate_resample_support',
+            'target_specificity_ratio',
+            'js_specificity_ratio',
+        },
+    )
+
+    summary = json.loads((ARTIFACT_DIR / 'study_summary.json').read_text(encoding='utf-8'))
+    if int(summary.get('n_concepts', 0)) < 1:
+        raise SystemExit('study_summary.json has no concepts.')
+
+    required_figures = [
+        'feature_auroc.png',
+        'layer_diagnostics.png',
+        'causal_effects.png',
+        'feature_set_effects.png',
+        'association_vs_causality.png',
+        'candidate_stability.png',
+    ]
+    missing_figures = [
+        name for name in required_figures if not (ARTIFACT_DIR / 'figures' / name).exists()
+    ]
+    if missing_figures:
+        raise SystemExit(f'Missing report figures: {missing_figures}')
+
+    print('FeatureLens offline artifact validation: PASS')
+    print(f"  concepts: {summary['n_concepts']}")
+    print(f"  feature pooling: {summary['selected_feature_pooling']}")
+    print('  report: artifacts/report.md')
+
+
+if __name__ == '__main__':
+    main()

tests/test_offline_study.py

@@ -0,0 +1,109 @@
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import numpy as np
+import pandas as pd
+import torch
+
+from experiments.analyze_stability import balanced_candidate_score
+from experiments.analyze_study import _paired_specificity, _safe_spearman
+from experiments.collect_activations import _promptwide_max_encoding
+from featurelens.sae import SparseEncoding
+from featurelens.study import OfflineStudy
+
+
+def test_promptwide_max_encoding_ignores_padding_and_max_pools() -> None:
+    encoding = SparseEncoding(
+        indices=torch.tensor(
+            [
+                [[1, 2], [1, 3], [2, 4]],
+                [[5, 6], [5, 7], [7, 8]],
+            ]
+        ),
+        values=torch.tensor(
+            [
+                [[1.0, 2.0], [4.0, 3.0], [5.0, 1.0]],
+                [[9.0, 9.0], [2.0, 4.0], [7.0, 6.0]],
+            ]
+        ),
+    )
+    mask = torch.tensor([[1, 1, 1], [0, 1, 1]])
+    pooled = _promptwide_max_encoding(encoding, mask)
+
+    row0 = dict(zip(pooled[0].indices.tolist(), pooled[0].values.tolist(), strict=True))
+    row1 = dict(zip(pooled[1].indices.tolist(), pooled[1].values.tolist(), strict=True))
+    assert row0 == {1: 4.0, 2: 5.0, 3: 3.0, 4: 1.0}
+    assert row1 == {5: 2.0, 7: 7.0, 8: 6.0}
+
+
+def test_balanced_candidate_score_rewards_selectivity_not_raw_scale() -> None:
+    target = np.array([30.0, 1000.0])
+    other = np.array([0.0, 950.0])
+    rate = np.array([1.0, 1.0])
+    score = balanced_candidate_score(target, other, rate)
+    assert score[0] > score[1]
+
+
+def test_paired_specificity_uses_random_ensemble_mean_per_task() -> None:
+    frame = pd.DataFrame(
+        [
+            {'task_id': 'a', 'condition': 'sae_feature', 'target_mean_logprob_delta': 0.4},
+            {'task_id': 'a', 'condition': 'random_norm_matched', 'target_mean_logprob_delta': 0.1},
+            {'task_id': 'a', 'condition': 'random_norm_matched', 'target_mean_logprob_delta': -0.1},
+            {'task_id': 'b', 'condition': 'sae_feature', 'target_mean_logprob_delta': -0.2},
+            {'task_id': 'b', 'condition': 'random_norm_matched', 'target_mean_logprob_delta': 0.05},
+            {'task_id': 'b', 'condition': 'random_norm_matched', 'target_mean_logprob_delta': -0.15},
+        ]
+    )
+    result = _paired_specificity(
+        frame,
+        effect_column='target_mean_logprob_delta',
+        seed=7,
+    )
+    assert np.isclose(result['sae_abs_mean'], 0.3)
+    assert np.isclose(result['random_abs_mean'], 0.1)
+    assert np.isclose(result['specificity_ratio'], 3.0)
+    assert result['n_tasks'] == 2
+
+
+def test_safe_spearman_handles_small_samples() -> None:
+    small = _safe_spearman(pd.Series([1.0, 2.0]), pd.Series([2.0, 1.0]))
+    assert small['n'] == 2
+    assert np.isnan(small['rho'])
+    enough = _safe_spearman(pd.Series([1.0, 2.0, 3.0]), pd.Series([3.0, 2.0, 1.0]))
+    assert enough['n'] == 3
+    assert np.isclose(enough['rho'], -1.0)
+
+
+def test_offline_study_reports_missing_and_complete(tmp_path: Path) -> None:
+    study = OfflineStudy(tmp_path)
+    assert not study.complete
+    assert 'not materialized yet' in study.overview_markdown()
+
+    for name in OfflineStudy.REQUIRED:
+        path = tmp_path / name
+        path.parent.mkdir(parents=True, exist_ok=True)
+        if name.endswith('.json'):
+            if name == 'study_summary.json':
+                payload = {
+                    'median_selected_feature_resample_support': 0.8,
+                    'correlations': {
+                        'heldout_auroc_vs_target_specificity': {'rho': -0.2, 'n': 7},
+                        'heldout_auroc_vs_js_specificity': {'rho': 0.4, 'n': 7},
+                    },
+                }
+            else:
+                payload = {'headline': 'Synthetic headline.', 'interpretation': 'Synthetic interpretation.'}
+            path.write_text(json.dumps(payload), encoding='utf-8')
+        elif name.endswith('.csv'):
+            path.write_text('x\n1\n', encoding='utf-8')
+        else:
+            path.write_text('# report\n', encoding='utf-8')
+
+    study = OfflineStudy(tmp_path)
+    assert study.complete
+    text = study.overview_markdown()
+    assert 'Synthetic headline.' in text
+    assert '80.0%' in text