Release FeatureLens v0.2.0

.gitignore

@@ -10,3 +10,5 @@ artifacts/*.npy
 artifacts/*.npz
 *.pt
 *.safetensors
+pycache/
+venv/

CHANGELOG.md

@@ -0,0 +1,26 @@
+# Changelog
+
+## 0.2.0 — 2026-08-13
+
+### Added
+
+- Live norm-matched random residual control for causal interventions.
+- Target-token and JS-divergence causal specificity ratios.
+- Feature-scale dose-response sweep (0× through 3×).
+- Cross-layer representation trajectory for layers 4, 14 and 26.
+- Native Gradio activation, dose-response and trajectory plots.
+- Bootstrap 95% confidence intervals in the generated offline report.
+- Paired sign-flip randomization test for SAE-vs-random causal-effect differences.
+- Hosted/local validation matrix.
+
+### Changed
+
+- Space SDK and dependency pin to Gradio 6.24.0.
+- Explicit client-side rendering (`ssr_mode=False`) to avoid the observed SSR async-user warning.
+- Shorter ZeroGPU callback durations and a smaller default generation budget.
+- More explicit inactive-feature warning and error surfacing.
+- Dark/light-mode-safe UI styling using Gradio theme variables.
+
+### Removed
+
+- Build/cache directories from release artifacts.

README.md

@@ -4,7 +4,8 @@ emoji: 🔬
 colorFrom: blue
 colorTo: indigo
 sdk: gradio
-python_version: "3.12"
+python_version: "3.12.12"
+sdk_version: "6.24.0"
 app_file: app.py
 pinned: false
 license: mit
@@ -12,6 +13,8 @@ license: mit
 
 # FeatureLens — Causal Interpretability Workbench
 
+> **v0.2:** live norm-matched controls, causal dose-response sweeps, layer trajectories, statistical uncertainty, and a hardened ZeroGPU deployment.
+
 **FeatureLens asks one concrete question:**
 
 > Do sparse features that predict a concept also causally influence model behaviour?
@@ -41,6 +44,10 @@ The Gradio app supports:
 - feature **ablation**, **scaling**, or **injection**;
 - baseline vs modified greedy generation;
 - first-token probability table and Jensen-Shannon divergence;
+- **live norm-matched random residual control** for every causal edit;
+- target-token causal specificity ratio against that matched control;
+- **dose-response sweep** over 0×, 0.5×, 1×, 1.5×, 2× and 3× feature scaling;
+- **layer trajectory** diagnostics across layers 4, 14 and 26;
 - optional target-continuation probability / log-probability delta;
 - optional concept hints loaded from real offline benchmark artifacts.
 
@@ -78,7 +85,9 @@ The evaluation computes:
 - layer-wise multinomial linear probes on dense residual states;
 - selected-feature ablation and 2× amplification;
 - target first-token probability, log-probability, rank and JS divergence;
-- **norm-matched random residual-direction controls**.
+- **norm-matched random residual-direction controls**;
+- bootstrap 95% confidence intervals for aggregate metrics;
+- a paired sign-flip randomization test for SAE-vs-random causal-effect differences.
 
 Feature selection uses the **training split**. Held-out AUROC/F1 are reported afterward. Paraphrases from the same pair never cross the train/test boundary.
 
@@ -149,6 +158,7 @@ FeatureLens/
 │   ├── interventions.py           # causal deltas + random controls
 │   ├── runtime.py                 # hooks, generation, probability deltas
 │   ├── metrics.py                 # reconstruction + divergence metrics
+│   ├── stats.py                   # bootstrap CIs + paired randomization test
 │   └── catalog.py                 # offline result integration
 ├── experiments/
 │   ├── build_dataset.py
@@ -174,6 +184,14 @@ FeatureLens/
 - A target string may tokenize into multiple tokens. The live workbench explicitly labels its target metric as the **first-token** probability in that case.
 - A causal effect can depend strongly on prompt, layer, feature scale, and downstream task. The report therefore includes raw per-task rows rather than only aggregate means.
 
+## v0.2 deployment hardening
+
+The Space pins **Gradio 6.24.0** in both the Space metadata and Python dependencies and launches with `ssr_mode=False`. This deliberately avoids the SSR execution path that produced an un-awaited `get_current_user` coroutine warning in the initial deployment while keeping the app fully functional in client-side rendering mode.
+
+The ZeroGPU callbacks also use shorter declared GPU durations than v0.1 and generation defaults to 16 new tokens, reducing queue cost for ordinary demo usage.
+
+See [`docs/VALIDATION.md`](docs/VALIDATION.md) for the pre-push and hosted test matrix.
+
 ## Reproducibility and checks
 
 ```bash

app.py

@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import pandas as pd
 import gradio as gr
 
 from featurelens.config import SETTINGS
@@ -7,69 +8,115 @@ from featurelens.hf_runtime import gpu
 from featurelens.runtime import RUNTIME
 
 CSS = """
-:root { --fl-blue:#2563eb; --fl-ink:#0f172a; --fl-muted:#64748b; }
-.gradio-container { max-width: 1280px !important; }
-.hero { padding: 12px 2px 6px; }
-.hero h1 { margin-bottom: 2px; font-size: 2.1rem; letter-spacing:-0.03em; }
-.hero p { color: var(--fl-muted); margin-top: 0; }
-.panel { border:1px solid #e2e8f0; border-radius:14px; padding:8px; }
+.gradio-container { max-width: 1320px !important; }
+.hero { padding: 8px 2px 2px; }
+.hero h1 { margin: 0; font-size: 2.35rem; letter-spacing: -0.045em; }
+.hero p { margin: .35rem 0 0; opacity: .72; font-size: 1rem; }
+.research-q { border-left: 4px solid var(--primary-500); padding: 10px 14px; margin: 10px 0 14px; border-radius: 0 10px 10px 0; background: var(--background-fill-secondary); }
+.badges { display:flex; flex-wrap:wrap; gap:7px; margin:8px 0 3px; }
+.badge { border:1px solid var(--border-color-primary); background:var(--background-fill-secondary); border-radius:999px; padding:4px 9px; font-size:12px; }
+.step { font-size: .78rem; text-transform: uppercase; letter-spacing:.09em; opacity:.62; font-weight:700; margin-top:2px; }
 .token-wrap { display:flex; flex-wrap:wrap; gap:5px; padding:8px 2px; line-height:1.7; }
-.token { background:#f1f5f9; border:1px solid #e2e8f0; border-radius:6px; padding:2px 7px; font-family:ui-monospace,SFMono-Regular,monospace; font-size:12px; }
-.token.selected { background:#dbeafe; border:2px solid #2563eb; color:#1e3a8a; }
+.token { background:var(--background-fill-secondary); border:1px solid var(--border-color-primary); border-radius:7px; padding:2px 7px; font-family:ui-monospace,SFMono-Regular,monospace; font-size:12px; }
+.token.selected { border:2px solid var(--primary-500); color:var(--primary-600); font-weight:600; }
 .token sup { opacity:.55; margin-right:4px; }
-.small-note { color:#64748b; font-size:12px; }
+.small-note { opacity:.65; font-size:12px; }
+.callout { border:1px solid var(--border-color-primary); background:var(--background-fill-secondary); border-radius:12px; padding:10px 12px; }
 """
 
+THEME = gr.themes.Soft(primary_hue="blue", neutral_hue="slate")
+
+
+def _raise_ui_error(exc: Exception) -> None:
+    raise gr.Error(f"{type(exc).__name__}: {exc}") from exc
+
 
 def _analysis_metrics_markdown(result) -> str:
     return (
         f"**Layer {result.layer} · token {result.token_index}**  \n"
         f"Active SAE features: **{int(result.metrics['active_features'])}/{SETTINGS.sae_top_k}**  \n"
-        f"Reconstruction cosine: **{result.metrics['cosine']:.4f}**  \n"
-        f"Normalized MSE: **{result.metrics['nmse']:.4f}**"
+        f"Reconstruction cosine: **{result.metrics['cosine']:.4f}** · "
+        f"NMSE: **{result.metrics['nmse']:.4f}**  \n"
+        f"Top-5 activation mass: **{result.metrics['top5_mass_fraction']:.1%}**"
     )
 
 
 def _intervention_metrics_markdown(result) -> str:
-    target = 'No target continuation supplied.'
+    target = "No target continuation supplied — specificity is shown using JS divergence only."
     if result.baseline_target_prob is not None:
-        warning = ''
+        warning = ""
         if result.target_token_count > 1:
             warning = (
                 f"  \n_Note: target text tokenizes to {result.target_token_count} tokens; "
-                'the displayed probability is for its first token only._'
+                "the displayed causal metric is for its first token._"
             )
         target = (
             f"Target first token: `{result.target_token}`  \n"
             f"Baseline p: **{result.baseline_target_prob:.6f}** · "
-            f"Modified p: **{result.modified_target_prob:.6f}**  \n"
-            f"Δ log p: **{result.target_logprob_delta:+.4f}**{warning}"
+            f"SAE edit p: **{result.modified_target_prob:.6f}** · "
+            f"Random-control p: **{result.random_target_prob:.6f}**  \n"
+            f"SAE Δlog p: **{result.target_logprob_delta:+.4f}** · "
+            f"random Δlog p: **{result.random_target_logprob_delta:+.4f}** · "
+            f"specificity ratio: **{result.target_specificity_ratio:.2f}×**{warning}"
+        )
+    inactive = ""
+    if abs(result.feature_activation) < 1e-12:
+        inactive = (
+            "  \n⚠️ **Selected feature is inactive at this token.** Ablate/scale therefore produces "
+            "a zero feature delta; use `inject` to test the decoder direction directly."
         )
     return (
-        f"Original feature activation: **{result.feature_activation:.4f}**  \n"
-        f"Δ feature coefficient: **{result.delta_activation:+.4f}**  \n"
-        f"Residual perturbation L2: **{result.perturbation_norm:.4f}**  \n"
-        f"Next-token JS divergence: **{result.js_divergence:.6f}**  \n\n"
-        f"{target}"
+        f"Original feature activation: **{result.feature_activation:.4f}** · "
+        f"Δ coefficient: **{result.delta_activation:+.4f}**  \n"
+        f"Perturbation L2: **{result.perturbation_norm:.4f}**  \n"
+        f"Next-token JS: **{result.js_divergence:.6f}** · "
+        f"random-control JS: **{result.random_js_divergence:.6f}** · "
+        f"specificity: **{result.js_specificity_ratio:.2f}×**  \n\n"
+        f"{target}{inactive}"
     )
 
 
-@gpu(duration=45)
-def analyze_prompt(prompt: str, layer: int, token_index: int, top_n: int):
-    if not prompt.strip():
-        raise gr.Error('Enter a prompt first.')
-    result = RUNTIME.analyze(prompt, int(layer), int(token_index), int(top_n))
-    choices = [str(int(row[1])) for row in result.rows]
-    feature_update = gr.update(choices=choices, value=choices[0] if choices else None)
+def _dose_metrics_markdown(result) -> str:
+    note = ""
+    if result.target_token_count > 1:
+        note = (
+            f" Target text spans {result.target_token_count} tokens; the curve measures its first token."
+        )
+    inactive = ""
+    if abs(result.feature_activation) < 1e-12:
+        inactive = " **The feature is inactive here, so a multiplicative sweep is flat by construction.**"
     return (
-        RUNTIME.token_html(result.tokens, result.token_index),
-        result.rows,
-        feature_update,
-        _analysis_metrics_markdown(result),
+        f"Feature activation at baseline: **{result.feature_activation:.4f}** · "
+        f"target first token: `{result.target_token}`.{note}{inactive}"
     )
 
 
-@gpu(duration=60)
+@gpu(duration=30)
+def analyze_prompt(prompt: str, layer: int, token_index: int, top_n: int):
+    try:
+        if not prompt.strip():
+            raise ValueError("Enter a prompt first.")
+        result = RUNTIME.analyze(prompt, int(layer), int(token_index), int(top_n))
+        choices = [str(int(row[1])) for row in result.rows]
+        feature_update = gr.update(choices=choices, value=choices[0] if choices else None)
+        chart_df = pd.DataFrame(
+            {
+                "Feature": [str(int(row[1])) for row in result.rows],
+                "Activation": [float(row[2]) for row in result.rows],
+            }
+        )
+        return (
+            RUNTIME.token_html(result.tokens, result.token_index),
+            result.rows,
+            chart_df,
+            feature_update,
+            _analysis_metrics_markdown(result),
+        )
+    except Exception as exc:
+        _raise_ui_error(exc)
+
+
+@gpu(duration=45)
 def run_intervention(
     prompt: str,
     layer: int,
@@ -80,209 +127,359 @@ def run_intervention(
     target_text: str,
     max_new_tokens: int,
 ):
-    if not prompt.strip():
-        raise gr.Error('Enter a prompt first.')
-    if feature_id is None or str(feature_id).strip() == '':
-        raise gr.Error('Choose or enter a feature id.')
     try:
+        if not prompt.strip():
+            raise ValueError("Enter a prompt first.")
+        if feature_id is None or str(feature_id).strip() == "":
+            raise ValueError("Choose or enter a feature id.")
         fid = int(float(feature_id))
-    except ValueError as exc:
-        raise gr.Error('Feature id must be an integer.') from exc
-    result = RUNTIME.intervene(
-        text=prompt,
-        layer=int(layer),
-        token_index=int(token_index),
-        feature_id=fid,
-        mode=mode,
-        coefficient=float(coefficient),
-        target_text=target_text,
-        max_new_tokens=int(max_new_tokens),
-    )
-    return (
-        result.baseline_text,
-        result.modified_text,
-        _intervention_metrics_markdown(result),
-        result.top_token_rows,
-    )
+        result = RUNTIME.intervene(
+            text=prompt,
+            layer=int(layer),
+            token_index=int(token_index),
+            feature_id=fid,
+            mode=mode,
+            coefficient=float(coefficient),
+            target_text=target_text,
+            max_new_tokens=int(max_new_tokens),
+        )
+        return (
+            result.baseline_text,
+            result.modified_text,
+            _intervention_metrics_markdown(result),
+            result.top_token_rows,
+        )
+    except Exception as exc:
+        _raise_ui_error(exc)
+
+
+@gpu(duration=40)
+def run_dose_response(
+    prompt: str,
+    layer: int,
+    token_index: int,
+    feature_id: str,
+    target_text: str,
+):
+    try:
+        if not prompt.strip():
+            raise ValueError("Enter a prompt first.")
+        if feature_id is None or str(feature_id).strip() == "":
+            raise ValueError("Choose or enter a feature id.")
+        if not target_text.strip():
+            raise ValueError("Enter a target continuation before running a dose-response sweep.")
+        result = RUNTIME.dose_response(
+            text=prompt,
+            layer=int(layer),
+            token_index=int(token_index),
+            feature_id=int(float(feature_id)),
+            target_text=target_text,
+        )
+        columns = [
+            "Multiplier",
+            "Δ feature coefficient",
+            "Perturbation L2",
+            "Baseline p(target)",
+            "Modified p(target)",
+            "Δ log p(target)",
+            "JS divergence",
+        ]
+        table = pd.DataFrame(result.rows, columns=columns)
+        plot = table[["Multiplier", "Δ log p(target)"]].copy()
+        return table, plot, _dose_metrics_markdown(result)
+    except Exception as exc:
+        _raise_ui_error(exc)
+
+
+@gpu(duration=35)
+def run_layer_sweep(prompt: str, token_index: int):
+    try:
+        if not prompt.strip():
+            raise ValueError("Enter a prompt first.")
+        result = RUNTIME.layer_sweep(prompt, int(token_index))
+        columns = [
+            "Layer",
+            "Reconstruction cosine",
+            "NMSE",
+            "Active features",
+            "Top activation",
+            "Top-5 mass",
+            "Activation entropy",
+        ]
+        table = pd.DataFrame(result.rows, columns=columns)
+        long = table.melt(
+            id_vars=["Layer"],
+            value_vars=["Reconstruction cosine", "Top-5 mass", "Activation entropy"],
+            var_name="Metric",
+            value_name="Value",
+        )
+        return RUNTIME.token_html(result.tokens, result.token_index), table, long
+    except Exception as exc:
+        _raise_ui_error(exc)
 
 
 def mode_help(mode: str):
-    if mode == 'ablate':
+    if mode == "ablate":
         return gr.update(
             value=0.0,
             interactive=False,
-            label='Coefficient (unused for ablation)',
-            info='Ablation sets the selected TopK feature coefficient to zero.',
+            label="Coefficient (unused for ablation)",
+            info="Sets the selected active feature coefficient to zero.",
         )
-    if mode == 'scale':
+    if mode == "scale":
         return gr.update(
             value=2.0,
             interactive=True,
-            label='Feature multiplier',
-            info='1.0 = unchanged, 0 = ablate, 2.0 = double the original coefficient.',
+            label="Feature multiplier",
+            info="1 = unchanged, 0 = ablate, 2 = double the original coefficient.",
         )
     return gr.update(
         value=5.0,
         interactive=True,
-        label='Additive feature coefficient',
-        info='Adds this amount along the feature decoder direction, even if the feature is inactive.',
+        label="Additive feature coefficient",
+        info="Adds this amount along the decoder direction, even when the feature is inactive.",
     )
 
 
-with gr.Blocks(title='FeatureLens — Causal Interpretability Workbench') as demo:
+with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench") as demo:
     gr.HTML(
-        '<div class="hero"><h1>FeatureLens</h1>'
-        '<p>Causal sparse-feature interpretability for Qwen3-1.7B: inspect → intervene → measure.</p></div>'
-    )
-    gr.Markdown(
-        f"Model: `{SETTINGS.model_id}` · Qwen-Scope TopK SAE · layers "
-        f"`{', '.join(map(str, SETTINGS.layers))}` · width `{SETTINGS.sae_width:,}`"
+        '<div class="hero"><h1>FeatureLens <span style="font-size:.42em;opacity:.55">v0.2</span></h1>'
+        '<p>Causal sparse-feature interpretability for Qwen3-1.7B — inspect → intervene → control → measure.</p></div>'
+        '<div class="research-q"><b>Research question:</b> Do sparse features that predict a concept also '
+        'causally influence the model’s behaviour?</div>'
+        '<div class="badges">'
+        '<span class="badge">Qwen3-1.7B-Base</span><span class="badge">Qwen-Scope SAE</span>'
+        '<span class="badge">32,768 features</span><span class="badge">TopK=50</span>'
+        '<span class="badge">norm-matched controls</span><span class="badge">ZeroGPU</span></div>'
     )
 
-    with gr.Tab('Workbench'):
+    with gr.Tab("Workbench"):
+        gr.HTML('<div class="step">Step 1 · Choose a prompt and residual location</div>')
         with gr.Row(equal_height=False):
-            with gr.Column(scale=2):
+            with gr.Column(scale=5):
                 prompt = gr.Textbox(
-                    label='Prompt',
-                    lines=7,
-                    value='The derivative of x squared is',
-                    placeholder='Enter a prompt to inspect...',
+                    label="Prompt",
+                    lines=5,
+                    value="The derivative of x squared is",
+                    placeholder="Enter a prompt to inspect…",
                 )
-                with gr.Row():
-                    layer = gr.Dropdown(
-                        choices=list(SETTINGS.layers),
-                        value=SETTINGS.layers[1] if len(SETTINGS.layers) > 1 else SETTINGS.layers[0],
-                        label='Residual layer',
-                    )
-                    token_index = gr.Number(
-                        value=-1,
-                        precision=0,
-                        label='Token index',
-                        info='Use -1 for the final prompt token. Analyze once to see all token indices.',
-                    )
-                    top_n = gr.Slider(5, 20, value=12, step=1, label='Top features')
-                analyze_btn = gr.Button('Inspect SAE features', variant='primary')
-                token_view = gr.HTML(
-                    '<div class="small-note">Token positions will appear here after analysis.</div>'
+                gr.Examples(
+                    examples=[
+                        ["The derivative of x squared is"],
+                        ["In Python, reverse a list using"],
+                        ["Je voudrais réserver une table pour"],
+                        ["I am not fully certain, but the answer may be"],
+                    ],
+                    inputs=[prompt],
+                    label="Controlled examples",
                 )
-                analysis_metrics = gr.Markdown()
-
             with gr.Column(scale=3):
-                feature_table = gr.Dataframe(
-                    headers=['Rank', 'Feature id', 'Activation', 'Offline concept hint'],
-                    datatype=['number', 'number', 'number', 'str'],
-                    interactive=False,
-                    label='Strongest TopK features at the selected token',
-                    wrap=True,
+                layer = gr.Dropdown(
+                    choices=list(SETTINGS.layers),
+                    value=SETTINGS.layers[1] if len(SETTINGS.layers) > 1 else SETTINGS.layers[0],
+                    label="Residual layer",
+                    info="Early / middle / late checkpoints are intentionally sampled.",
                 )
+                token_index = gr.Number(
+                    value=-1,
+                    precision=0,
+                    label="Prompt token index",
+                    info="-1 = final prompt token. Inspect once to see all token positions.",
+                )
+                top_n = gr.Slider(5, 20, value=12, step=1, label="Top features")
+                analyze_btn = gr.Button("Inspect sparse features", variant="primary")
+
+        token_view = gr.HTML('<div class="small-note">Token positions appear here after inspection.</div>')
+        analysis_metrics = gr.Markdown()
+        with gr.Row(equal_height=False):
+            feature_table = gr.Dataframe(
+                headers=["Rank", "Feature id", "Activation", "Offline concept hint"],
+                datatype=["number", "number", "number", "str"],
+                interactive=False,
+                label="Strongest TopK features",
+                wrap=True,
+                scale=3,
+            )
+            feature_plot = gr.BarPlot(
+                x="Feature",
+                y="Activation",
+                title="Activation profile",
+                x_title="Feature id",
+                y_title="Activation",
+                scale=2,
+            )
 
-        gr.Markdown('### Causal intervention')
+        gr.HTML('<div class="step">Step 2 · Intervene and compare against a matched control</div>')
         gr.Markdown(
-            'The edit is **reconstruction-preserving**: FeatureLens changes only the selected SAE '
-            'coefficient and adds the decoded delta to the *original* residual stream. It does not '
-            'replace the residual with an SAE reconstruction.'
+            "FeatureLens changes only the selected SAE coefficient and adds that decoder-vector delta "
+            "to the **original** residual. The live negative control applies a deterministic random "
+            "residual perturbation with the **same L2 norm**."
         )
         with gr.Row(equal_height=False):
             with gr.Column(scale=2):
                 feature_id = gr.Dropdown(
                     choices=[],
                     allow_custom_value=True,
-                    label='Feature id',
-                    info='Analyze first to populate the strongest features, or enter any id 0–32767.',
+                    label="Feature id",
+                    info="Inspection populates the strongest active features; custom IDs are also allowed.",
                 )
                 mode = gr.Radio(
-                    choices=['ablate', 'scale', 'inject'],
-                    value='ablate',
-                    label='Intervention',
+                    choices=["ablate", "scale", "inject"],
+                    value="ablate",
+                    label="Intervention",
                 )
                 coefficient = gr.Number(
                     value=0.0,
                     interactive=False,
-                    label='Coefficient (unused for ablation)',
+                    label="Coefficient (unused for ablation)",
                 )
                 target_text = gr.Textbox(
-                    label='Optional target continuation',
-                    placeholder='e.g. 2x',
-                    info='FeatureLens reports the first-token probability shift for this continuation.',
+                    label="Optional target continuation",
+                    placeholder="e.g. 2x",
+                    info="Supplying this enables target-token Δlog p and causal-specificity metrics.",
                 )
                 max_new = gr.Slider(
                     4,
                     SETTINGS.max_new_tokens,
-                    value=min(20, SETTINGS.max_new_tokens),
+                    value=min(16, SETTINGS.max_new_tokens),
                     step=1,
-                    label='Max new tokens',
+                    label="Max new tokens",
                 )
-                intervene_btn = gr.Button('Run baseline vs modified', variant='primary')
+                intervene_btn = gr.Button("Run causal test", variant="primary")
                 intervention_metrics = gr.Markdown()
-
             with gr.Column(scale=3):
                 with gr.Row():
-                    baseline_out = gr.Textbox(label='Baseline output', lines=8, interactive=False)
-                    modified_out = gr.Textbox(label='Modified output', lines=8, interactive=False)
+                    baseline_out = gr.Textbox(label="Baseline generation", lines=7, interactive=False)
+                    modified_out = gr.Textbox(label="SAE-intervened generation", lines=7, interactive=False)
                 token_prob_table = gr.Dataframe(
-                    headers=['Token', 'Baseline p', 'Modified p', 'Δ probability'],
-                    datatype=['str', 'number', 'number', 'number'],
+                    headers=["Token", "Baseline p", "SAE-edit p", "Δ probability"],
+                    datatype=["str", "number", "number", "number"],
                     interactive=False,
-                    label='First generated token distribution',
+                    label="Next-token distribution shift",
+                )
+
+        gr.HTML('<div class="step">Step 3 · Test dose–response</div>')
+        with gr.Accordion("Causal dose–response sweep", open=False):
+            gr.Markdown(
+                "A single coefficient can be cherry-picked. This sweep scales the same active feature "
+                "through **0×, 0.5×, 1×, 1.5×, 2× and 3×** and measures target-token Δlog p. "
+                "A coherent monotonic response is stronger causal evidence than one isolated edit."
+            )
+            dose_btn = gr.Button("Run dose–response sweep")
+            dose_metrics = gr.Markdown()
+            with gr.Row():
+                dose_table = gr.Dataframe(interactive=False, label="Dose-response measurements", scale=3)
+                dose_plot = gr.LinePlot(
+                    x="Multiplier",
+                    y="Δ log p(target)",
+                    title="Causal dose–response",
+                    x_title="Feature multiplier",
+                    y_title="Δ log p(target)",
+                    scale=2,
                 )
 
         analyze_btn.click(
             analyze_prompt,
             inputs=[prompt, layer, token_index, top_n],
-            outputs=[token_view, feature_table, feature_id, analysis_metrics],
+            outputs=[token_view, feature_table, feature_plot, feature_id, analysis_metrics],
         )
         mode.change(mode_help, inputs=[mode], outputs=[coefficient])
         intervene_btn.click(
             run_intervention,
-            inputs=[
-                prompt,
-                layer,
-                token_index,
-                feature_id,
-                mode,
-                coefficient,
-                target_text,
-                max_new,
-            ],
+            inputs=[prompt, layer, token_index, feature_id, mode, coefficient, target_text, max_new],
             outputs=[baseline_out, modified_out, intervention_metrics, token_prob_table],
         )
+        dose_btn.click(
+            run_dose_response,
+            inputs=[prompt, layer, token_index, feature_id, target_text],
+            outputs=[dose_table, dose_plot, dose_metrics],
+        )
+
+    with gr.Tab("Layer trajectory"):
+        gr.Markdown(
+            "### Follow the representation across early, middle and late residual streams\n"
+            "This is **not** a cross-layer feature-ID comparison — SAE dictionaries are layer-specific. "
+            "Instead, it compares reconstruction quality and sparsity/concentration statistics at the "
+            "same prompt token across layers 4, 14 and 26."
+        )
+        with gr.Row():
+            trajectory_prompt = gr.Textbox(
+                label="Prompt",
+                lines=5,
+                value="The derivative of x squared is",
+                scale=4,
+            )
+            trajectory_token = gr.Number(
+                value=-1,
+                precision=0,
+                label="Prompt token index",
+                info="-1 = final token",
+                scale=1,
+            )
+        trajectory_btn = gr.Button("Compare layers", variant="primary")
+        trajectory_tokens = gr.HTML()
+        with gr.Row():
+            trajectory_table = gr.Dataframe(interactive=False, label="Layer diagnostics", scale=3)
+            trajectory_plot = gr.LinePlot(
+                x="Layer",
+                y="Value",
+                color="Metric",
+                title="Representation trajectory",
+                x_title="Layer",
+                y_title="Normalized value",
+                scale=2,
+            )
+        trajectory_btn.click(
+            run_layer_sweep,
+            inputs=[trajectory_prompt, trajectory_token],
+            outputs=[trajectory_tokens, trajectory_table, trajectory_plot],
+        )
 
-    with gr.Tab('Offline benchmark'):
-        benchmark_md = gr.Markdown(RUNTIME.catalog.benchmark_markdown())
+    with gr.Tab("Offline benchmark"):
+        gr.Markdown(RUNTIME.catalog.benchmark_markdown())
         gr.Markdown(
-            'The offline pipeline evaluates feature/concept AUROC + F1, reconstruction quality, '
-            'paraphrase stability, layer-wise residual linear probes, causal ablation/amplification, '
-            'and norm-matched random-direction controls. Results are loaded from `artifacts/` and '
-            'are never hard-coded into the demo.'
+            "The offline pipeline evaluates held-out feature/concept AUROC + F1, reconstruction quality, "
+            "paraphrase stability, dense residual linear probes, causal ablation/amplification and "
+            "norm-matched random-direction controls. Results are loaded from `artifacts/`; the app never "
+            "ships invented benchmark numbers."
         )
 
-    with gr.Tab('Method'):
+    with gr.Tab("Method"):
         gr.Markdown(
             r"""
-### What the intervention means
+### Reconstruction-preserving intervention
 
-For residual vector $h$, sparse code $z$, decoder column $d_i$, and selected feature $i$:
+For residual vector $h$, selected sparse coefficient $z_i$, decoder direction $d_i$, and scale $\alpha$:
 
 - **Ablate:** $h' = h - z_i d_i$
-- **Scale by $\alpha$:** $h' = h + (\alpha - 1) z_i d_i$
-- **Inject $\delta$:** $h' = h + \delta d_i$
+- **Scale:** $h' = h + (\alpha - 1) z_i d_i$
+- **Inject:** $h' = h + \delta d_i$
+
+The app edits the original residual rather than replacing it with the full SAE reconstruction, so SAE
+reconstruction error is not introduced as a causal confound.
 
-This is equivalent to editing the SAE reconstruction by the chosen feature delta while retaining
-$h$ itself, so SAE reconstruction error is not injected as a confound. The modified residual is
-patched at one selected **prompt token**; downstream generation is then allowed to evolve normally.
+### Evidence ladder
 
-### What FeatureLens does *not* claim
+1. **Reconstruction:** does the SAE represent the residual reasonably well?
+2. **Prediction:** does a feature predict a controlled concept on held-out paraphrase groups?
+3. **Intervention:** does changing it alter downstream behaviour?
+4. **Specificity:** is that effect larger than a norm-matched random residual perturbation?
+5. **Dose-response:** does effect size change coherently as the feature coefficient is varied?
 
-A high feature/concept AUROC is correlational evidence. A causal claim requires downstream effects
-under intervention, held-out prompts, and comparison with controls. The offline report is designed
-to make a weak or null causal result visible rather than hide it.
+A high AUROC alone remains correlational evidence.
 """
         )
 
     gr.Markdown(
-        'Built with PyTorch, Transformers, Qwen3-1.7B-Base and Qwen-Scope SAEs. '
-        'This project is independent of any thesis dataset or thesis code.'
+        "<small>Built with PyTorch, Transformers, Qwen3-1.7B-Base and Qwen-Scope residual-stream SAEs. "
+        "FeatureLens is independent of thesis code and thesis datasets.</small>"
     )
 
-if __name__ == '__main__':
-    demo.queue(default_concurrency_limit=1).launch(css=CSS)
+if __name__ == "__main__":
+    # Explicit CSR avoids an HF/Gradio SSR auth-coroutine warning seen in 6.23.x-era Space builds.
+    demo.queue(default_concurrency_limit=1, max_size=8).launch(
+        css=CSS,
+        theme=THEME,
+        ssr_mode=False,
+        show_error=True,
+    )

docs/VALIDATION.md

@@ -0,0 +1,76 @@
+# FeatureLens validation matrix
+
+FeatureLens has two distinct validation targets: software correctness and scientific validity. Passing one does not imply the other.
+
+## Before every push
+
+```bash
+python -m ruff check app.py featurelens experiments tests scripts
+python -m pytest -q
+python -m compileall -q app.py featurelens experiments scripts
+python scripts/release_check.py
+```
+
+Also run `git status` and confirm that no `__pycache__/`, `pycache/`, `.pytest_cache/`, model weights, or `artifacts/activations/` files are staged.
+
+## Local UI smoke test
+
+```bash
+python app.py
+```
+
+Verify that the app opens without an SSR banner and without `get_current_user ... was never awaited` warnings. The app intentionally launches with `ssr_mode=False`.
+
+On a CUDA machine, test one example through each path:
+
+1. Inspect features on layer 14, final token.
+2. Ablate the strongest active feature.
+3. Supply a target continuation and confirm SAE-edit and random-control metrics both appear.
+4. Run the dose-response sweep and confirm the 1× row has approximately zero intervention effect.
+5. Run the layer trajectory and confirm rows for 4, 14 and 26.
+
+## Hugging Face Space smoke test
+
+After a push/rebuild:
+
+1. Confirm Space hardware is ZeroGPU and status reaches **Running**.
+2. Check startup logs for model/SAE download or load failures.
+3. Confirm the launch line does **not** say `with SSR`.
+4. Run `The derivative of x squared is` at layer 14, token `-1`.
+5. Confirm the feature dropdown is populated after inspection.
+6. Run an ablation with `target continuation = 2x`.
+7. Confirm baseline and modified generations render and random-control metrics appear.
+8. Run a dose-response sweep once; confirm the curve renders.
+9. Run the layer trajectory once; confirm all three layers render.
+10. Refresh the page and repeat a short inspection to catch state/cold-start regressions.
+
+## Scientific acceptance checks
+
+Do not publish headline findings until the offline benchmark is run on the real Qwen3/Qwen-Scope weights.
+
+Required checks:
+
+- paraphrase groups do not cross train/test splits;
+- feature selection uses training metrics only;
+- held-out AUROC/F1 are reported separately;
+- reconstruction metrics are reported per layer;
+- causal tasks are separate from discovery prompts;
+- SAE edits and random controls are paired by task/intervention;
+- perturbation norms match within numerical tolerance;
+- bootstrap confidence intervals are included;
+- paired sign-flip p-value is reported as uncertainty evidence, not as proof of a mechanistic claim;
+- raw causal rows remain available for inspection;
+- null or mixed results are retained in the generated report.
+
+## Recommended manual adversarial tests
+
+Use prompts designed to expose failure modes rather than only attractive examples:
+
+- a mathematics prompt with a code-like token;
+- English content containing one French entity name;
+- negation such as `This is not a positive review`;
+- an uncertainty prompt that becomes certain in its final clause;
+- a custom inactive feature ID with `ablate` versus `inject`;
+- a multi-token target continuation to verify the first-token warning;
+- token index `0`, `-1`, and an invalid out-of-range index;
+- feature ID `0`, `32767`, and an invalid `32768`.

experiments/make_report.py

@@ -9,6 +9,11 @@ import numpy as np
 import pandas as pd
 
 from experiments.common import ARTIFACT_DIR
+from featurelens.stats import (
+    bootstrap_mean_ci,
+    paired_bootstrap_difference_ci,
+    paired_sign_flip_pvalue,
+)
 
 
 def parse_args() -> argparse.Namespace:
@@ -84,6 +89,7 @@ def main() -> None:
 
     mean_auc = float(selected['auroc'].mean())
     median_auc = float(selected['auroc'].median())
+    auc_ci_low, auc_ci_high = bootstrap_mean_ci(selected['auroc'].to_numpy(), seed=42)
     best_layer_row = layers.sort_values('linear_probe_macro_auroc', ascending=False).iloc[0]
     mean_jaccard = float(stability['topk_jaccard'].mean())
     mean_sparse_cos = float(stability['sparse_cosine'].mean())
@@ -93,6 +99,21 @@ def main() -> None:
     sae_abs = float(np.mean(np.abs(sae['target_logprob_delta'])))
     random_abs = float(np.mean(np.abs(random['target_logprob_delta'])))
     ratio = sae_abs / max(random_abs, 1e-12)
+
+    paired = causal.pivot_table(
+        index=['task_id', 'intervention'],
+        columns='condition',
+        values='target_logprob_delta',
+        aggfunc='first',
+    ).dropna(subset=['sae_feature', 'random_norm_matched'])
+    paired_sae = np.abs(paired['sae_feature'].to_numpy(dtype=float))
+    paired_random = np.abs(paired['random_norm_matched'].to_numpy(dtype=float))
+    diff_mean = float(np.mean(paired_sae - paired_random))
+    diff_ci_low, diff_ci_high = paired_bootstrap_difference_ci(
+        paired_sae, paired_random, seed=43
+    )
+    sign_flip_p = paired_sign_flip_pvalue(paired_sae, paired_random, seed=44)
+
     active_rate = float(np.mean(sae['feature_activation'] > 0))
     top1_change = float(sae['top1_changed'].mean())
 
@@ -128,11 +149,12 @@ def main() -> None:
         'random residual controls.'
     )
     highlights = [
-        f'Median selected-feature held-out AUROC: {median_auc:.3f}.',
+        f'Median selected-feature held-out AUROC: {median_auc:.3f}; mean AUROC 95% bootstrap CI [{auc_ci_low:.3f}, {auc_ci_high:.3f}].',
         f'Best residual linear-probe layer: {int(best_layer_row["layer"])} with macro AUROC {best_layer_row["linear_probe_macro_auroc"]:.3f}.',
         f'Mean paraphrase TopK Jaccard: {mean_jaccard:.3f}; sparse activation cosine: {mean_sparse_cos:.3f}.',
         f'Selected feature active on {active_rate:.1%} of causal prompts; modified top-1 token on {top1_change:.1%}.',
         f'Mean absolute causal effect / random-control effect ratio: {ratio:.2f}×.',
+        f'Paired mean |Δlog p| advantage over random: {diff_mean:+.3f}, 95% bootstrap CI [{diff_ci_low:+.3f}, {diff_ci_high:+.3f}], sign-flip p={sign_flip_p:.4f}.',
     ]
     summary = {
         'headline': headline,
@@ -140,6 +162,7 @@ def main() -> None:
         'interpretation': interpretation,
         'metrics': {
             'mean_selected_feature_test_auroc': mean_auc,
+            'mean_selected_feature_test_auroc_bootstrap_ci_95': [auc_ci_low, auc_ci_high],
             'median_selected_feature_test_auroc': median_auc,
             'best_linear_probe_layer': int(best_layer_row['layer']),
             'best_linear_probe_macro_auroc': float(best_layer_row['linear_probe_macro_auroc']),
@@ -148,6 +171,9 @@ def main() -> None:
             'mean_abs_sae_target_logprob_delta': sae_abs,
             'mean_abs_random_target_logprob_delta': random_abs,
             'causal_to_random_effect_ratio': ratio,
+            'paired_mean_abs_effect_advantage': diff_mean,
+            'paired_mean_abs_effect_advantage_bootstrap_ci_95': [diff_ci_low, diff_ci_high],
+            'paired_sign_flip_pvalue': sign_flip_p,
             'causal_prompt_feature_active_rate': active_rate,
             'sae_top1_change_rate': top1_change,
         },
@@ -181,6 +207,7 @@ def main() -> None:
         '- Linear baseline: multinomial logistic regression on the dense residual stream.',
         '- Causal edit: reconstruction-preserving decoder-direction delta patched into the original residual.',
         '- Negative control: deterministic random residual direction matched to the SAE perturbation L2 norm.',
+        '- Uncertainty: bootstrap 95% confidence intervals and a paired sign-flip randomization test for SAE-vs-control effect differences.',
         '- Behavioural metric: first-token target probability/log-probability, rank, JS divergence, and top-1 changes.',
         '',
         '## Figures',

featurelens/runtime.py

@@ -1,8 +1,10 @@
 from __future__ import annotations
 
+import hashlib
 import html
+import math
 import os
-from collections.abc import Iterator
+from collections.abc import Iterator, Sequence
 from contextlib import contextmanager
 from dataclasses import dataclass
 
@@ -11,7 +13,11 @@ from transformers import AutoModelForCausalLM, AutoTokenizer
 
 from .catalog import FeatureCatalog
 from .config import SETTINGS, Settings
-from .interventions import InterventionSpec, residual_delta
+from .interventions import (
+    InterventionSpec,
+    normalized_random_control,
+    residual_delta,
+)
 from .metrics import js_divergence_from_logits, reconstruction_metrics, safe_log_probability
 from .sae import SAEStore, SparseEncoding
 
@@ -28,8 +34,8 @@ def _dtype_from_name(name: str) -> torch.dtype:
 
 
 def _default_device() -> torch.device:
-    # ZeroGPU exposes CUDA emulation at module load time. Force the recommended
-    # CUDA placement on Spaces even if a local availability probe is conservative.
+    # ZeroGPU exposes CUDA emulation at module load time. Hugging Face recommends
+    # placing models on CUDA at module scope so startup transfers can be optimized.
     if os.getenv('SPACE_ID'):
         return torch.device('cuda')
     if torch.cuda.is_available():
@@ -55,15 +61,35 @@ class InterventionResult:
     delta_activation: float
     perturbation_norm: float
     js_divergence: float
+    random_js_divergence: float
+    js_specificity_ratio: float
     target_text: str
     target_token: str
     target_token_count: int
     baseline_target_prob: float | None
     modified_target_prob: float | None
+    random_target_prob: float | None
     target_logprob_delta: float | None
+    random_target_logprob_delta: float | None
+    target_specificity_ratio: float | None
     top_token_rows: list[list[object]]
 
 
+@dataclass
+class LayerSweepResult:
+    tokens: list[str]
+    token_index: int
+    rows: list[list[object]]
+
+
+@dataclass
+class DoseResponseResult:
+    feature_activation: float
+    target_token: str
+    target_token_count: int
+    rows: list[list[object]]
+
+
 class FeatureLensRuntime:
     def __init__(self, settings: Settings = SETTINGS) -> None:
         self.settings = settings
@@ -78,8 +104,13 @@ class FeatureLensRuntime:
         self.catalog = FeatureCatalog()
         self.load_error: str | None = None
 
+    @property
+    def ready(self) -> bool:
+        return self.model is not None and self.tokenizer is not None and self.sae_store is not None
+
     def ensure_ready(self, preload_saes: bool = False) -> None:
-        if self.model is not None and self.tokenizer is not None and self.sae_store is not None:
+        if self.ready:
+            assert self.sae_store is not None
             if preload_saes:
                 self.sae_store.preload()
             return
@@ -103,6 +134,7 @@ class FeatureLensRuntime:
         )
         if preload_saes:
             self.sae_store.preload()
+        self.load_error = None
 
     def token_choices(self, text: str) -> list[tuple[str, int]]:
         self.ensure_ready(preload_saes=False)
@@ -146,6 +178,25 @@ class FeatureLensRuntime:
         finally:
             handle.remove()
 
+    @contextmanager
+    def _capture_hooks(self, layers: Sequence[int], buckets: dict[int, dict]) -> Iterator[None]:
+        assert self.model is not None
+        handles = []
+        for layer in layers:
+            bucket = buckets[int(layer)]
+
+            def hook(_module, _inputs, output, *, target=bucket):
+                hidden = self._hidden_from_output(output)
+                if 'hidden' not in target:
+                    target['hidden'] = hidden.detach()
+
+            handles.append(self.model.model.layers[int(layer)].register_forward_hook(hook))
+        try:
+            yield
+        finally:
+            for handle in handles:
+                handle.remove()
+
     @contextmanager
     def _delta_hook(self, layer: int, token_index: int, delta: torch.Tensor) -> Iterator[None]:
         assert self.model is not None
@@ -174,6 +225,20 @@ class FeatureLensRuntime:
         finally:
             handle.remove()
 
+    @staticmethod
+    def _resolve_index(token_index: int, seq_len: int) -> int:
+        idx = int(token_index)
+        if idx < 0:
+            idx = seq_len + idx
+        if idx < 0 or idx >= seq_len:
+            raise IndexError(f'Token index {token_index} outside prompt length {seq_len}.')
+        return idx
+
+    @staticmethod
+    def _control_seed(text: str, layer: int, feature_id: int, mode: str, coefficient: float) -> int:
+        payload = f'{text}\0{layer}\0{feature_id}\0{mode}\0{coefficient:.8g}'.encode('utf-8')
+        return int.from_bytes(hashlib.sha256(payload).digest()[:4], 'big', signed=False)
+
     @torch.inference_mode()
     def analyze(self, text: str, layer: int, token_index: int = -1, top_n: int = 12) -> AnalysisResult:
         self.ensure_ready(preload_saes=False)
@@ -185,18 +250,18 @@ class FeatureLensRuntime:
         with self._capture_hook(int(layer), bucket):
             self.model(**inputs, use_cache=False)
         hidden = bucket['hidden'][0]
-        seq_len = hidden.shape[0]
-        idx = int(token_index)
-        if idx < 0:
-            idx = seq_len + idx
-        if idx < 0 or idx >= seq_len:
-            raise IndexError(f'Token index {token_index} outside prompt length {seq_len}.')
+        idx = self._resolve_index(int(token_index), hidden.shape[0])
         residual = hidden[idx]
         sae = self.sae_store.get(int(layer))
         encoding = sae.encode(residual)
         reconstruction = sae.decode_sparse(encoding)
         metrics = reconstruction_metrics(residual, reconstruction)
         metrics['active_features'] = float(encoding.active_count)
+        values = encoding.values.float().clamp_min(0)
+        total = float(values.sum().item())
+        metrics['top5_mass_fraction'] = (
+            float(values[: min(5, values.numel())].sum().item()) / total if total > 0 else 0.0
+        )
         ids = inputs['input_ids'][0].tolist()
         tokens = [self.tokenizer.decode([token_id]) for token_id in ids]
         rows: list[list[object]] = []
@@ -216,6 +281,52 @@ class FeatureLensRuntime:
             metrics=metrics,
         )
 
+    @torch.inference_mode()
+    def layer_sweep(self, text: str, token_index: int = -1) -> LayerSweepResult:
+        self.ensure_ready(preload_saes=True)
+        assert self.model is not None and self.tokenizer is not None and self.sae_store is not None
+        inputs = self._inputs(text)
+        buckets = {int(layer): {} for layer in self.settings.layers}
+        with self._capture_hooks(self.settings.layers, buckets):
+            self.model(**inputs, use_cache=False)
+
+        seq_len = int(inputs['input_ids'].shape[1])
+        idx = self._resolve_index(int(token_index), seq_len)
+        rows: list[list[object]] = []
+        for layer in self.settings.layers:
+            residual = buckets[int(layer)]['hidden'][0, idx]
+            sae = self.sae_store.get(int(layer))
+            encoding = sae.encode(residual)
+            reconstruction = sae.decode_sparse(encoding)
+            metrics = reconstruction_metrics(residual, reconstruction)
+            values = encoding.values.float().clamp_min(0)
+            positive = values[values > 0]
+            total = positive.sum()
+            if positive.numel() <= 1 or float(total.item()) <= 0:
+                entropy = 0.0
+            else:
+                probs = positive / total
+                entropy = float((-(probs * torch.log(probs)).sum() / math.log(positive.numel())).item())
+            top5_fraction = (
+                float(values[: min(5, values.numel())].sum().item() / total.item())
+                if float(total.item()) > 0
+                else 0.0
+            )
+            rows.append(
+                [
+                    int(layer),
+                    float(metrics['cosine']),
+                    float(metrics['nmse']),
+                    int(encoding.active_count),
+                    float(values[0].item()) if values.numel() else 0.0,
+                    top5_fraction,
+                    entropy,
+                ]
+            )
+        ids = inputs['input_ids'][0].tolist()
+        tokens = [self.tokenizer.decode([token_id]) for token_id in ids]
+        return LayerSweepResult(tokens=tokens, token_index=idx, rows=rows)
+
     def token_html(self, tokens: list[str], selected_index: int) -> str:
         chips = []
         for idx, token in enumerate(tokens):
@@ -255,11 +366,7 @@ class FeatureLensRuntime:
         assert self.model is not None and self.tokenizer is not None and self.sae_store is not None
         inputs = self._inputs(text)
         prompt_len = int(inputs['input_ids'].shape[1])
-        idx = int(token_index)
-        if idx < 0:
-            idx = prompt_len + idx
-        if idx < 0 or idx >= prompt_len:
-            raise IndexError(f'Token index {token_index} outside prompt length {prompt_len}.')
+        idx = self._resolve_index(int(token_index), prompt_len)
 
         sae = self.sae_store.get(int(layer))
         capture: dict = {}
@@ -281,6 +388,15 @@ class FeatureLensRuntime:
         with self._delta_hook(int(layer), idx, delta):
             modified = self.model.generate(**inputs, **generation_kwargs)
 
+        # Live negative control: one extra forward pass, not another full generation.
+        control_delta = normalized_random_control(
+            delta,
+            seed=self._control_seed(text, int(layer), int(feature_id), mode, float(coefficient)),
+        )
+        with self._delta_hook(int(layer), idx, control_delta):
+            random_out = self.model(**inputs, use_cache=False)
+        random_logits = random_out.logits[0, -1]
+
         baseline_ids = baseline.sequences[0, prompt_len:]
         modified_ids = modified.sequences[0, prompt_len:]
         baseline_text = self.tokenizer.decode(baseline_ids, skip_special_tokens=True)
@@ -291,10 +407,12 @@ class FeatureLensRuntime:
         baseline_logits = baseline.scores[0][0]
         modified_logits = modified.scores[0][0]
         js = js_divergence_from_logits(baseline_logits, modified_logits)
+        random_js = js_divergence_from_logits(baseline_logits, random_logits)
+        js_ratio = abs(js) / max(abs(random_js), 1e-12)
 
         target_token = ''
         target_count = 0
-        bp = mp = log_delta = None
+        bp = mp = rp = log_delta = random_log_delta = specificity = None
         if target_text.strip():
             target_ids = self.tokenizer(target_text, add_special_tokens=False)['input_ids']
             target_count = len(target_ids)
@@ -303,9 +421,13 @@ class FeatureLensRuntime:
                 target_token = self.tokenizer.decode([target_id])
                 p = torch.softmax(baseline_logits.float(), dim=-1)
                 q = torch.softmax(modified_logits.float(), dim=-1)
+                r = torch.softmax(random_logits.float(), dim=-1)
                 bp = float(p[target_id].item())
                 mp = float(q[target_id].item())
+                rp = float(r[target_id].item())
                 log_delta = safe_log_probability(mp) - safe_log_probability(bp)
+                random_log_delta = safe_log_probability(rp) - safe_log_probability(bp)
+                specificity = abs(log_delta) / max(abs(random_log_delta), 1e-12)
 
         return InterventionResult(
             baseline_text=baseline_text,
@@ -314,17 +436,87 @@ class FeatureLensRuntime:
             delta_activation=float(spec.delta_activation(original_activation)),
             perturbation_norm=float(torch.linalg.vector_norm(delta.float()).item()),
             js_divergence=float(js),
+            random_js_divergence=float(random_js),
+            js_specificity_ratio=float(js_ratio),
             target_text=target_text,
             target_token=target_token,
             target_token_count=target_count,
             baseline_target_prob=bp,
             modified_target_prob=mp,
+            random_target_prob=rp,
             target_logprob_delta=log_delta,
+            random_target_logprob_delta=random_log_delta,
+            target_specificity_ratio=specificity,
             top_token_rows=self._top_token_rows(
                 self.tokenizer, baseline_logits, modified_logits, k=8
             ),
         )
 
+    @torch.inference_mode()
+    def dose_response(
+        self,
+        text: str,
+        layer: int,
+        token_index: int,
+        feature_id: int,
+        target_text: str,
+        multipliers: Sequence[float] = (0.0, 0.5, 1.0, 1.5, 2.0, 3.0),
+    ) -> DoseResponseResult:
+        if not target_text.strip():
+            raise ValueError('Dose-response requires a target continuation.')
+        self.ensure_ready(preload_saes=False)
+        assert self.model is not None and self.tokenizer is not None and self.sae_store is not None
+        inputs = self._inputs(text)
+        prompt_len = int(inputs['input_ids'].shape[1])
+        idx = self._resolve_index(int(token_index), prompt_len)
+        sae = self.sae_store.get(int(layer))
+
+        capture: dict = {}
+        with self._capture_hook(int(layer), capture):
+            baseline_out = self.model(**inputs, use_cache=False)
+        baseline_logits = baseline_out.logits[0, -1]
+        residual = capture['hidden'][0, idx]
+        encoding = sae.encode(residual)
+        original_activation = encoding.activation_for(int(feature_id))
+        direction = sae.decoder_direction(int(feature_id))
+
+        target_ids = self.tokenizer(target_text, add_special_tokens=False)['input_ids']
+        if not target_ids:
+            raise ValueError('Target continuation tokenized to an empty sequence.')
+        target_id = int(target_ids[0])
+        target_token = self.tokenizer.decode([target_id])
+        baseline_prob = float(torch.softmax(baseline_logits.float(), dim=-1)[target_id].item())
+
+        rows: list[list[object]] = []
+        for multiplier in multipliers:
+            spec = InterventionSpec('scale', float(multiplier))
+            delta = residual_delta(direction, original_activation, spec)
+            if float(torch.linalg.vector_norm(delta.float()).item()) == 0.0:
+                modified_logits = baseline_logits
+            else:
+                with self._delta_hook(int(layer), idx, delta):
+                    output = self.model(**inputs, use_cache=False)
+                modified_logits = output.logits[0, -1]
+            modified_prob = float(torch.softmax(modified_logits.float(), dim=-1)[target_id].item())
+            log_delta = safe_log_probability(modified_prob) - safe_log_probability(baseline_prob)
+            rows.append(
+                [
+                    float(multiplier),
+                    float(spec.delta_activation(original_activation)),
+                    float(torch.linalg.vector_norm(delta.float()).item()),
+                    baseline_prob,
+                    modified_prob,
+                    float(log_delta),
+                    float(js_divergence_from_logits(baseline_logits, modified_logits)),
+                ]
+            )
+        return DoseResponseResult(
+            feature_activation=float(original_activation),
+            target_token=target_token,
+            target_token_count=len(target_ids),
+            rows=rows,
+        )
+
 
 RUNTIME = FeatureLensRuntime()
 

featurelens/stats.py

@@ -0,0 +1,81 @@
+from __future__ import annotations
+
+import numpy as np
+
+
+def bootstrap_mean_ci(
+    values: np.ndarray | list[float],
+    *,
+    confidence: float = 0.95,
+    n_resamples: int = 5000,
+    seed: int = 42,
+) -> tuple[float, float]:
+    """Percentile bootstrap CI for a sample mean."""
+    x = np.asarray(values, dtype=float)
+    x = x[np.isfinite(x)]
+    if x.size == 0:
+        return float('nan'), float('nan')
+    if x.size == 1:
+        value = float(x[0])
+        return value, value
+    rng = np.random.default_rng(seed)
+    sample_idx = rng.integers(0, x.size, size=(int(n_resamples), x.size))
+    means = x[sample_idx].mean(axis=1)
+    alpha = (1.0 - float(confidence)) / 2.0
+    low, high = np.quantile(means, [alpha, 1.0 - alpha])
+    return float(low), float(high)
+
+
+def paired_bootstrap_difference_ci(
+    a: np.ndarray | list[float],
+    b: np.ndarray | list[float],
+    *,
+    confidence: float = 0.95,
+    n_resamples: int = 5000,
+    seed: int = 42,
+) -> tuple[float, float]:
+    """Bootstrap CI for mean(a - b), preserving paired rows."""
+    x = np.asarray(a, dtype=float)
+    y = np.asarray(b, dtype=float)
+    mask = np.isfinite(x) & np.isfinite(y)
+    x = x[mask]
+    y = y[mask]
+    if x.size == 0:
+        return float('nan'), float('nan')
+    diff = x - y
+    return bootstrap_mean_ci(
+        diff,
+        confidence=confidence,
+        n_resamples=n_resamples,
+        seed=seed,
+    )
+
+
+def paired_sign_flip_pvalue(
+    a: np.ndarray | list[float],
+    b: np.ndarray | list[float],
+    *,
+    n_permutations: int = 20000,
+    seed: int = 42,
+) -> float:
+    """Two-sided paired randomization test for a non-zero mean difference."""
+    x = np.asarray(a, dtype=float)
+    y = np.asarray(b, dtype=float)
+    mask = np.isfinite(x) & np.isfinite(y)
+    diff = (x - y)[mask]
+    if diff.size == 0:
+        return float('nan')
+    observed = abs(float(diff.mean()))
+    if observed == 0.0:
+        return 1.0
+    rng = np.random.default_rng(seed)
+    extreme = 0
+    batch = 2000
+    remaining = int(n_permutations)
+    while remaining > 0:
+        n = min(batch, remaining)
+        signs = rng.choice(np.array([-1.0, 1.0]), size=(n, diff.size))
+        permuted = np.abs((signs * diff).mean(axis=1))
+        extreme += int(np.count_nonzero(permuted >= observed))
+        remaining -= n
+    return float((extreme + 1) / (int(n_permutations) + 1))

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "featurelens"
-version = "0.1.0"
+version = "0.2.0"
 description = "Causal sparse-feature interpretability workbench for Qwen3 and Qwen-Scope SAEs"
 requires-python = ">=3.10"
 

requirements.txt

@@ -1,7 +1,7 @@
 torch>=2.8.0,<2.12.0
 transformers>=4.51.0,<6.0.0
 huggingface_hub>=0.34.0,<2.0.0
-gradio>=6.0.0,<7.0.0
+gradio==6.24.0
 numpy>=2.0.0,<3.0.0
 scipy>=1.14.0,<2.0.0
 scikit-learn>=1.6.0,<2.0.0

research_config.json

@@ -2,7 +2,11 @@
   "research_question": "Do sparse features that predict a concept also causally influence model behaviour?",
   "model_id": "Qwen/Qwen3-1.7B-Base",
   "sae_repo_id": "Qwen/SAE-Res-Qwen3-1.7B-Base-W32K-L0_50",
-  "layers": [4, 14, 26],
+  "layers": [
+    4,
+    14,
+    26
+  ],
   "sae_top_k": 50,
   "sae_width": 32768,
   "concepts": [
@@ -19,8 +23,27 @@
   "causal_tasks": 28,
   "split_seed": 42,
   "feature_selection": "training-split AUROC with activation-rate contrast tie-break",
-  "held_out_metrics": ["AUROC", "F1"],
-  "causal_interventions": ["ablate", "scale_2x"],
+  "held_out_metrics": [
+    "AUROC",
+    "F1"
+  ],
+  "causal_interventions": [
+    "ablate",
+    "scale_2x"
+  ],
   "negative_control": "norm-matched random residual direction",
-  "primary_causal_metric": "target first-token log-probability delta"
+  "primary_causal_metric": "target first-token log-probability delta",
+  "live_causal_controls": "norm-matched random residual direction",
+  "dose_response_multipliers": [
+    0.0,
+    0.5,
+    1.0,
+    1.5,
+    2.0,
+    3.0
+  ],
+  "statistical_inference": [
+    "bootstrap_95_ci",
+    "paired_sign_flip_test"
+  ]
 }

scripts/release_check.py

@@ -1,64 +1,262 @@
 from __future__ import annotations
 
 import json
+import subprocess
 from collections import Counter
 from pathlib import Path
 
+
 ROOT = Path(__file__).resolve().parents[1]
 
+# Any non-ignored repository file larger than this is suspicious.
+# FeatureLens should not contain model weights, SAE checkpoints,
+# activation dumps, virtual environments, etc.
+MAX_FILE_SIZE_BYTES = 5_000_000  # 5 MB
+
 REQUIRED = [
-    'README.md',
-    'app.py',
-    'requirements.txt',
-    'research_config.json',
-    'featurelens/runtime.py',
-    'featurelens/sae.py',
-    'featurelens/interventions.py',
-    'experiments/run_all.py',
-    'data/prompts.jsonl',
-    'data/causal_tasks.jsonl',
+    "README.md",
+    "app.py",
+    "requirements.txt",
+    "research_config.json",
+    "featurelens/runtime.py",
+    "featurelens/sae.py",
+    "featurelens/interventions.py",
+    "featurelens/stats.py",
+    "experiments/run_all.py",
+    "data/prompts.jsonl",
+    "data/causal_tasks.jsonl",
 ]
 
 
 def load_jsonl(path: Path) -> list[dict]:
-    return [json.loads(line) for line in path.read_text(encoding='utf-8').splitlines() if line]
+    """Load a JSONL file into a list of dictionaries."""
+    return [
+        json.loads(line)
+        for line in path.read_text(encoding="utf-8").splitlines()
+        if line.strip()
+    ]
 
 
-def main() -> None:
+def repository_candidates() -> list[Path]:
+    """
+    Return files that are either:
+
+    - already tracked by Git, or
+    - untracked but not ignored by .gitignore.
+
+    This deliberately excludes files such as .venv contents when .venv/
+    is correctly listed in .gitignore.
+
+    Including untracked, non-ignored files is useful because it catches
+    accidental large files before someone runs `git add .`.
+    """
+    try:
+        result = subprocess.run(
+            [
+                "git",
+                "ls-files",
+                "--cached",
+                "--others",
+                "--exclude-standard",
+            ],
+            cwd=ROOT,
+            capture_output=True,
+            text=True,
+            check=True,
+        )
+    except FileNotFoundError as exc:
+        raise SystemExit(
+            "Git is required to run the FeatureLens release check."
+        ) from exc
+    except subprocess.CalledProcessError as exc:
+        stderr = exc.stderr.strip()
+        raise SystemExit(
+            f"Could not inspect repository files with Git: {stderr}"
+        ) from exc
+
+    candidates: list[Path] = []
+
+    for relative_path in result.stdout.splitlines():
+        relative_path = relative_path.strip()
+
+        if not relative_path:
+            continue
+
+        path = ROOT / relative_path
+
+        # A tracked file may have been deleted locally but not yet committed.
+        # Such a path should not be size-checked.
+        if path.is_file():
+            candidates.append(path)
+
+    return candidates
+
+
+def check_required_files() -> None:
+    """Ensure the repository contains all files required for a release."""
     missing = [path for path in REQUIRED if not (ROOT / path).exists()]
+
     if missing:
-        raise SystemExit(f'Missing required files: {missing}')
-
-    config = json.loads((ROOT / 'research_config.json').read_text(encoding='utf-8'))
-    assert config['layers'] == [4, 14, 26]
-    assert config['model_id'] == 'Qwen/Qwen3-1.7B-Base'
-    assert config['sae_width'] == 32768
-
-    prompts = load_jsonl(ROOT / 'data' / 'prompts.jsonl')
-    causal = load_jsonl(ROOT / 'data' / 'causal_tasks.jsonl')
-    assert len(prompts) == config['discovery_prompts']
-    assert len(causal) == config['causal_tasks']
-    concept_counts = Counter(row['concept'] for row in prompts)
-    assert len(concept_counts) == len(config['concepts'])
-    assert len(set(concept_counts.values())) == 1
-
-    oversized = []
-    for path in ROOT.rglob('*'):
-        if path.is_file() and '.git' not in path.parts and path.stat().st_size > 5_000_000:
-            oversized.append(str(path.relative_to(ROOT)))
+        raise SystemExit(f"Missing required files: {missing}")
+
+
+def check_config(config: dict) -> None:
+    """Validate the research configuration expected by FeatureLens v0.2."""
+    if config.get("layers") != [4, 14, 26]:
+        raise SystemExit(
+            f'Unexpected layers: {config.get("layers")}. '
+            "Expected [4, 14, 26]."
+        )
+
+    if config.get("model_id") != "Qwen/Qwen3-1.7B-Base":
+        raise SystemExit(
+            f'Unexpected model_id: {config.get("model_id")}.'
+        )
+
+    if config.get("sae_width") != 32768:
+        raise SystemExit(
+            f'Unexpected sae_width: {config.get("sae_width")}. '
+            "Expected 32768."
+        )
+
+    expected_multipliers = [0.0, 0.5, 1.0, 1.5, 2.0, 3.0]
+
+    if config.get("dose_response_multipliers") != expected_multipliers:
+        raise SystemExit(
+            "Unexpected dose_response_multipliers: "
+            f'{config.get("dose_response_multipliers")}. '
+            f"Expected {expected_multipliers}."
+        )
+
+
+def check_datasets(config: dict) -> tuple[list[dict], list[dict]]:
+    """Validate discovery and causal benchmark datasets."""
+    prompts = load_jsonl(ROOT / "data" / "prompts.jsonl")
+    causal = load_jsonl(ROOT / "data" / "causal_tasks.jsonl")
+
+    expected_prompt_count = config.get("discovery_prompts")
+    expected_causal_count = config.get("causal_tasks")
+
+    if len(prompts) != expected_prompt_count:
+        raise SystemExit(
+            "Discovery prompt count mismatch: "
+            f"found {len(prompts)}, expected {expected_prompt_count}."
+        )
+
+    if len(causal) != expected_causal_count:
+        raise SystemExit(
+            "Causal task count mismatch: "
+            f"found {len(causal)}, expected {expected_causal_count}."
+        )
+
+    concept_counts = Counter(row["concept"] for row in prompts)
+
+    expected_concepts = config.get("concepts", [])
+
+    if set(concept_counts) != set(expected_concepts):
+        raise SystemExit(
+            "Discovery dataset concepts do not match research_config.json.\n"
+            f"Dataset concepts: {sorted(concept_counts)}\n"
+            f"Config concepts:  {sorted(expected_concepts)}"
+        )
+
+    # The controlled discovery benchmark is intentionally balanced.
+    if len(set(concept_counts.values())) != 1:
+        raise SystemExit(
+            f"Discovery concepts are not balanced: {dict(concept_counts)}"
+        )
+
+    return prompts, causal
+
+
+def check_oversized_files() -> None:
+    """
+    Reject unexpectedly large files that could be committed/pushed.
+
+    Importantly, this does NOT recursively scan .venv or other ignored
+    directories. Git decides what counts as a repository candidate.
+    """
+    oversized: list[str] = []
+
+    for path in repository_candidates():
+        size_bytes = path.stat().st_size
+
+        if size_bytes > MAX_FILE_SIZE_BYTES:
+            relative_path = path.relative_to(ROOT)
+            size_mb = size_bytes / 1_000_000
+
+            oversized.append(
+                f"{relative_path} ({size_mb:.1f} MB)"
+            )
+
     if oversized:
-        raise SystemExit(f'Repository contains unexpectedly large tracked candidates: {oversized}')
+        formatted = "\n  - ".join(oversized)
+
+        raise SystemExit(
+            "Repository contains unexpectedly large tracked/unignored "
+            "candidates:\n"
+            f"  - {formatted}\n\n"
+            "If a file is a legitimate local artifact, add it to .gitignore. "
+            "Model weights, SAE checkpoints, activation dumps, virtual "
+            "environments, and caches should not be committed."
+        )
+
+
+def check_readme() -> None:
+    """Validate important Hugging Face Space/release metadata."""
+    readme = (ROOT / "README.md").read_text(encoding="utf-8")
+
+    required_strings = [
+        "sdk: gradio",
+        'sdk_version: "6.24.0"',
+        "Qwen/Qwen3-1.7B-Base",
+    ]
+
+    missing = [
+        value
+        for value in required_strings
+        if value not in readme
+    ]
+
+    if missing:
+        raise SystemExit(
+            f"README.md is missing required metadata/content: {missing}"
+        )
+
+    readme_lower = readme.lower()
+
+    if "norm-matched random" not in readme_lower:
+        raise SystemExit(
+            'README.md should describe the "norm-matched random" control.'
+        )
+
+    if "dose" not in readme_lower:
+        raise SystemExit(
+            "README.md should describe the dose-response experiment."
+        )
+
+
+def main() -> None:
+    check_required_files()
+
+    config = json.loads(
+        (ROOT / "research_config.json").read_text(encoding="utf-8")
+    )
+
+    check_config(config)
+
+    prompts, causal = check_datasets(config)
+
+    check_oversized_files()
 
-    readme = (ROOT / 'README.md').read_text(encoding='utf-8')
-    assert 'sdk: gradio' in readme
-    assert 'Qwen/Qwen3-1.7B-Base' in readme
-    assert 'norm-matched random' in readme.lower()
+    check_readme()
 
-    print('FeatureLens release check: PASS')
-    print(f'  discovery prompts: {len(prompts)}')
-    print(f'  causal tasks: {len(causal)}')
+    print("FeatureLens release check: PASS")
+    print(f"  discovery prompts: {len(prompts)}")
+    print(f"  causal tasks: {len(causal)}")
     print(f'  layers: {config["layers"]}')
+    print("  release: v0.2.0")
 
 
-if __name__ == '__main__':
-    main()
+if __name__ == "__main__":
+    main()

tests/test_stats.py

@@ -0,0 +1,33 @@
+import numpy as np
+
+from featurelens.stats import (
+    bootstrap_mean_ci,
+    paired_bootstrap_difference_ci,
+    paired_sign_flip_pvalue,
+)
+
+
+def test_bootstrap_mean_ci_contains_sample_mean():
+    values = np.array([0.7, 0.8, 0.9, 0.85, 0.75])
+    low, high = bootstrap_mean_ci(values, n_resamples=1000, seed=1)
+    assert low <= values.mean() <= high
+
+
+def test_paired_bootstrap_detects_positive_difference():
+    a = np.array([2.0, 2.2, 1.8, 2.1, 2.4])
+    b = np.array([0.5, 0.6, 0.4, 0.7, 0.5])
+    low, high = paired_bootstrap_difference_ci(a, b, n_resamples=1000, seed=2)
+    assert low > 0
+    assert high > low
+
+
+def test_sign_flip_small_for_consistent_effect():
+    a = np.array([2.0, 2.1, 2.3, 2.2, 2.4, 2.5, 2.1, 2.2])
+    b = np.array([0.2, 0.4, 0.3, 0.5, 0.2, 0.4, 0.3, 0.2])
+    p = paired_sign_flip_pvalue(a, b, n_permutations=5000, seed=3)
+    assert p < 0.05
+
+
+def test_sign_flip_one_for_identical_pairs():
+    x = np.array([1.0, 2.0, 3.0])
+    assert paired_sign_flip_pvalue(x, x, n_permutations=1000, seed=4) == 1.0