Publish FeatureLens study

CHANGELOG.md

@@ -1,5 +1,16 @@
 # Changelog
 
+## v1.0.0
+
+- Published the completed offline study artifacts and public **Study** dashboard.
+- Finalized causal inference around two position policies: final prompt token and maximum selected-feature activation within the prompt.
+- Reported causal-task-level paired inference, separating intervention coverage from conditional effect strength.
+- Measured selected SAE features at 0.962 mean held-out AUROC; max-active causal edits reached 82.1% task coverage and 2.33× matched-random target effect on average.
+- Kept the weaker final-token baseline as a position-sensitivity control rather than replacing it.
+- Added the committed reproducibility bundle, including both Colab notebooks under `notebooks/`, fixed split metadata, measured CSV/JSON outputs, report, and figures.
+- Simplified release configuration/checks by removing historical per-version feature bookkeeping from `research_config.json`.
+- Public interface remains version-neutral; release metadata is internal to the repository.
+
 ## v0.16.0
 
 - Added two explicit offline causal-position policies: `final_token` and `max_feature_activation`. Max-active positions are selected only from SAE activation within the prompt, never from behavioral outcomes.

README.md

@@ -13,32 +13,48 @@ license: mit
 
 # FeatureLens
 
-FeatureLens is a causal interpretability workbench for `Qwen/Qwen3-1.7B-Base` and the **Qwen-Scope residual-stream sparse autoencoders**. It is built around one question:
+FeatureLens is a causal interpretability workbench for `Qwen/Qwen3-1.7B-Base` and **Qwen-Scope residual-stream sparse autoencoders**. It asks one question:
 
 > **Do sparse features that predict a concept also causally influence model behaviour?**
 
-The project keeps association and intervention evidence separate. A large SAE activation or strong held-out classifier is useful evidence about representation; a causal claim requires changing the residual stream and measuring downstream behaviour against controlled perturbations.
+The project separates representational evidence from causal evidence. A feature can classify a concept well without controlling the downstream continuation one might infer from that association.
 
-## What the live app does
+## Measured study result
 
-FeatureLens loads SAEs for residual layers **4, 14, and 26** and supports:
+The committed offline study uses **224 discovery prompts** (112 paraphrase pairs across seven controlled concepts) and **28 causal tasks**.
 
-- token-level residual capture and TopK SAE feature inspection;
-- reconstruction cosine, NMSE, activation mass, and layer trajectories;
+- Selected SAE features averaged **0.962 held-out AUROC** (median **0.987**, bootstrap 95% CI **[0.927, 0.994]**).
+- Dense final-token residual probes reached **1.000 macro AUROC** at layers 14 and 26.
+- Paraphrases preserved weighted sparse representations much more strongly than exact sparse support: mean cosine **0.985** versus TopK Jaccard **0.326**.
+- Selected features were active at the conventional final prompt token on only **28.6%** of causal tasks, but somewhere in the prompt on **82.1%**.
+- At the final token, targeted SAE interventions were **1.52×** the norm-matched random-control effect on average, but task-level uncertainty included zero (paired advantage **+0.0026**, 95% CI **[-0.0005, +0.0063]**, sign-flip **p=0.1719**).
+- When intervention positions were chosen only from the selected feature's **maximum SAE activation within the prompt**, coverage rose to **82.1%** and targeted effects averaged **2.33×** matched-random controls (paired advantage **+0.0237**, 95% CI **[+0.0067, +0.0469]**, sign-flip **p≈1×10⁻⁴**).
+- Final-token top-5 joint ablation produced only a **1.09×** SAE/random ratio, so adding more associated features did not automatically yield stronger causal specificity.
+- Across the seven concepts, held-out AUROC and max-active target specificity had only weak descriptive association (**Spearman ρ=-0.185**).
+
+The main conclusion is therefore not that predictive SAE features are automatically causal. **Causal evidence depended strongly on where the representation was tested**, and predictive strength by itself was a poor proxy for random-normalized causal specificity across concepts.
+
+See [`artifacts/report.md`](artifacts/report.md) and the **Study** tab for the full measured result.
+
+## Live workbench
+
+FeatureLens loads Qwen-Scope SAEs for residual layers **4, 14, and 26** and supports:
+
+- token-level residual capture and TopK feature inspection;
+- SAE reconstruction diagnostics and layer trajectories;
 - single-feature ablation, scaling, and decoder-direction injection;
 - exact full-continuation teacher-forced scoring;
 - next-token distribution shifts and deterministic generation comparison;
 - eight norm-matched random controls for live specificity checks;
-- scale dose-response curves;
-- contrastive continuation preference;
-- joint feature-set interventions, 1/3/5 set-size sweeps, non-additivity, and decoder geometry;
+- scale dose-response and contrastive continuation preference;
+- joint feature-set interventions, set-size sweeps, non-additivity, and decoder geometry;
 - concept-guided candidate discovery with current-token causal readiness;
-- batched candidate triage and random-controlled candidate comparison;
-- token traces, completion-cue tests, cue × context tests, and controlled concept contrasts;
+- batched candidate triage and controlled multi-candidate comparison;
+- completion-cue, cue × context, token-trace, and controlled-concept diagnostics;
 - local and prompt-wide paraphrase robustness;
 - cross-target causal profiles and pairwise preference shifts.
 
-The interface is deliberately an analytical tool rather than an SAE label viewer. Feature ids remain unlabeled until there is empirical evidence for a concept association.
+Feature ids remain unlabeled unless there is empirical evidence for a concept association.
 
 ## Intervention semantics
 
@@ -50,54 +66,52 @@ scale:   h' = h + (α - 1) z_i d_i
 inject:  h' = h + δ d_i
 ```
 
-FeatureLens applies the decoded **delta** to the original residual. It does not replace the residual with the full SAE reconstruction, so reconstruction error is not silently mixed into the intervention.
+FeatureLens applies the decoded **delta** to the original residual instead of replacing the residual with the full SAE reconstruction. Batched causal experiments include a zero-edit condition in the same execution context, and random controls match the L2 norm of the targeted SAE perturbation.
+
+## Offline study design
 
-Batched causal experiments include a zero-edit condition in the same execution context. Random controls match the L2 norm of the SAE perturbation.
+Concept evidence uses **prompt-wide maximum SAE activation across non-padding tokens**. Final-token sparse activations are saved separately for local analyses.
 
-## Offline study
+Causal evidence is reported under two position policies:
 
-The live app is exploratory. The offline study is the dataset-scale experiment.
+- **final token** — conventional final-prompt-token baseline;
+- **max feature activation** — intervene where the selected feature is most strongly represented in that prompt.
 
-It uses **224 discovery prompts** arranged as 112 paraphrase pairs across seven controlled concepts, plus **28 separate causal tasks**. Concept evidence uses prompt-wide max-pooled SAE activations across non-padding tokens; final-token sparse activations are saved separately for local analyses.
+Max-active positions are selected from SAE activation only, never from behavioral outcomes. Coverage is reported separately from effect strength.
 
-Causal evidence is reported under two position policies: the original **final-token** baseline and **max-feature-activation**, which patches the selected feature where it is most strongly represented in the prompt. Max-active positions are selected from SAE activation only, never from downstream behavioral effects. Primary uncertainty uses the causal task as the statistical unit.
+The primary causal statistical unit is the **causal task**: ablation and 2× amplification are averaged within task before paired bootstrap and sign-flip inference.
 
-The study produces:
+The study additionally includes:
 
 - train-only feature selection with held-out AUROC/F1;
-- a dense final-token residual linear-probe baseline;
-- paraphrase stability;
+- dense final-token residual linear-probe baselines;
+- paraphrase robustness;
 - 128-resample candidate-selection sensitivity;
-- random-controlled single-feature causal results under both final-token and max-feature-activation patch policies;
-- top-1/3/5 feature-set causal results;
-- causal-position coverage/sensitivity and cross-concept association-versus-causality synthesis;
-- uncertainty-aware report figures and a measured Markdown report.
+- top-1/3/5 feature-set causal diagnostics;
+- norm-matched random residual controls;
+- cross-concept association-versus-causality synthesis.
 
-Run the full pipeline with:
+## Reproducing the study
+
+The canonical command is:
 
 ```bash
 python -m experiments.run_all --resume
 ```
 
-On a memory-constrained GPU, activation collection can be tuned without changing the experiment definition:
+On a 16 GB GPU, a smaller activation batch is usually more comfortable:
 
 ```bash
 python -m experiments.run_all --resume --activation-batch-size 8
 ```
 
-If you already completed the v0.15 study, upgrade it with only the positional causal addendum:
-
-```bash
-python -m experiments.run_causal_addendum --resume
-```
-
-After the expensive model stages exist, CPU-only analysis can be regenerated with:
+After the expensive model stages exist, regenerate only CPU analysis with:
 
 ```bash
 python -m experiments.run_analysis_only
 ```
 
-Artifact integrity is checked with:
+Validate the final artifact bundle with:
 
 ```bash
 python -m scripts.validate_artifacts
@@ -105,13 +119,15 @@ python -m scripts.validate_artifacts
 
 ### Google Colab
 
-A ready-to-run full-study notebook is included at [`notebooks/FeatureLens_Offline_Study_Colab.ipynb`](notebooks/FeatureLens_Offline_Study_Colab.ipynb). If the v0.15 study is already complete, use [`notebooks/FeatureLens_Causal_Addendum_Colab.ipynb`](notebooks/FeatureLens_Causal_Addendum_Colab.ipynb) instead; it runs only the max-active causal addendum and CPU synthesis.
+Use [`notebooks/FeatureLens_Offline_Study_Colab.ipynb`](notebooks/FeatureLens_Offline_Study_Colab.ipynb) for a fresh full reproduction.
+
+[`notebooks/FeatureLens_Causal_Addendum_Colab.ipynb`](notebooks/FeatureLens_Causal_Addendum_Colab.ipynb) is retained as the exact migration path used to extend an already-completed final-token baseline with the max-active causal study without recollecting discovery activations.
 
-See [`docs/COLAB.md`](docs/COLAB.md) for the exact workflow.
+See [`notebooks/README.md`](notebooks/README.md) and [`docs/COLAB.md`](docs/COLAB.md).
 
 ## Public artifacts
 
-Large activation caches are intentionally excluded from Git. The small measured outputs that can be committed after a real run include:
+The repository commits only small measured outputs. Large activation caches and model/SAE weights are excluded.
 
 ```text
 artifacts/
@@ -127,29 +143,30 @@ artifacts/
 ├── study_summary.json
 ├── summary.json
 ├── report.md
+├── split.json
 └── figures/
 ```
 
-The **Study** tab reads these artifacts directly. Before the offline run is materialized it intentionally shows no placeholder metrics.
+The **Study** tab reads these artifacts directly and does not rerun the model.
 
 ## Repository layout
 
 ```text
 FeatureLens/
-├── app.py                    # Gradio / ZeroGPU workbench
-├── featurelens/              # SAE, runtime, metrics, interventions, study loader
-├── experiments/              # offline collection, evaluation, causal study, reports
-├── data/                     # controlled prompt and causal-task definitions
-├── artifacts/                # small public study outputs; activations are ignored
-├── notebooks/                # Colab runner
-├── scripts/                  # release, UI smoke, artifact validation
-├── tests/                    # software and methodology regression tests
-├── docs/                     # methodology, validation, deployment, Colab notes
-├── DESIGN.md                 # UI design contract
-└── research_config.json      # experiment configuration
+├── app.py
+├── featurelens/          # SAE/runtime/intervention/study code
+├── experiments/          # offline collection, causal study, analysis, reports
+├── data/                 # controlled discovery prompts and causal tasks
+├── artifacts/            # committed measured study outputs
+├── notebooks/            # Colab study runners
+├── scripts/              # validation and UI smoke checks
+├── tests/
+├── docs/
+├── DESIGN.md
+└── research_config.json
 ```
 
-## Local validation
+## Validation
 
 ```bash
 python3 -m pytest -q
@@ -157,36 +174,25 @@ python3 -m compileall -q app.py featurelens experiments scripts
 python3 -m ruff check app.py featurelens experiments tests scripts
 python3 scripts/ui_smoke.py
 python3 scripts/release_check.py
+python3 -m scripts.validate_artifacts
 ```
 
-The UI smoke test performs a real local Gradio `launch()` rather than only constructing the component tree.
-
-## Methodology notes
-
-Several quantities answer different questions and should not be collapsed into one score:
-
-- **held-out AUROC/F1** — concept association;
-- **paraphrase / resample stability** — sensitivity to wording or sample choice;
-- **target Δ log p** — effect on one specified continuation;
-- **Jensen-Shannon divergence** — local distributional change;
-- **random-normalized specificity** — whether the targeted SAE edit exceeds an equal-norm residual perturbation baseline;
-- **feature-set non-additivity** — downstream interaction under joint edits;
-- **decoder geometry** — alignment/cancellation before downstream model non-linearity.
-
-The full methodology is documented in [`docs/METHODOLOGY.md`](docs/METHODOLOGY.md) and the offline study protocol in [`docs/OFFLINE_STUDY.md`](docs/OFFLINE_STUDY.md).
+The UI smoke test performs a real local Gradio `launch()`.
 
-## Limitations
+## Interpretation guardrails
 
-- Live ZeroGPU controls are intentionally small; eight-control empirical tails are coarse diagnostics.
-- SAE feature ids are layer-specific and should not be compared across layers by id.
+- Held-out AUROC/F1 measure **concept association**, not causal influence.
+- Max-active intervention positions are selected from SAE activation only, never from behavioral effect size.
+- Cross-concept correlations use only seven concepts and are descriptive.
+- Per-concept causal task counts are small; the primary inference pools task-level paired effects across all 28 causal tasks.
+- Five causal tasks contained no activation of the selected feature anywhere in the prompt; max-active coverage was therefore 82.1%, not 100%.
+- Dense linear probes and prompt-wide SAE features use different pooling schemes and are separate baselines.
 - Prompt-wide max pooling discards token order.
-- Dense linear probes and prompt-wide SAE features use different pooling schemes and are reported as separate baselines.
-- Cross-concept study correlations have only seven concepts and are descriptive.
-- A candidate feature can be predictive without being causally specific, and a causally disruptive feature need not selectively control the target one might infer from its association.
+- Live eight-control empirical tails are coarse diagnostics; the offline study is the primary aggregate evidence.
 
 ## Design
 
-The public UI follows the project-specific design contract in [`DESIGN.md`](DESIGN.md): restrained typography and color, flat information hierarchy, minimal decorative chrome, compact actions, explicit table headings, and no marketing-style cards/badges/gradients.
+The public UI follows [`DESIGN.md`](DESIGN.md): restrained typography and color, flat information hierarchy, compact actions, explicit result headings, and minimal decorative chrome.
 
 ## License
 

artifacts/README.md

@@ -1,39 +1,20 @@
-# Generated artifacts
-
-The repository ships without invented empirical results. The finalized v0.16 study uses prompt-wide feature evidence plus two causal position policies.
-
-Fresh full study:
-
-```bash
-python -m experiments.run_all --resume
-```
-
-Upgrade an already completed v0.15 study without recollecting discovery activations:
-
-```bash
-python -m experiments.run_causal_addendum --resume
-```
-
-The public artifact set includes:
-
-- `feature_catalog.csv`;
-- `layer_metrics.csv`;
-- `stability.csv`;
-- `selection_stability.csv`;
-- `causal_results_final_token.csv`;
-- `causal_results_max_active.csv`;
-- `causal_position_summary.csv`;
-- `feature_set_results.csv`;
-- `study_feature_summary.csv`;
-- `study_summary.json`;
-- `summary.json`;
-- `report.md`;
-- report figures including causal-position sensitivity and association-vs-causality.
-
-Validate before commit:
-
-```bash
-python -m scripts.validate_artifacts
-```
-
-`artifacts/activations/` remains gitignored. Commit only the small CSV/JSON/report/figure outputs.
+# Offline study artifacts
+
+This directory contains the small, publishable outputs from the completed FeatureLens study. Large activation matrices, model weights, SAE checkpoints, and task checkpoint markers are intentionally excluded.
+
+Primary study artifacts:
+
+- `feature_catalog.csv` — train-selected SAE candidates and held-out AUROC/F1.
+- `layer_metrics.csv` — dense-probe and SAE reconstruction diagnostics.
+- `stability.csv` — paraphrase stability measurements.
+- `selection_stability.csv` — 128-resample candidate-selection sensitivity.
+- `causal_results_final_token.csv` — final-token causal baseline.
+- `causal_results_max_active.csv` — max-feature-activation causal study.
+- `causal_position_summary.csv` — coverage and task-level position-sensitivity synthesis.
+- `feature_set_results.csv` — final-token top-1/3/5 feature-set diagnostic.
+- `study_feature_summary.csv` / `study_summary.json` — cross-concept evidence synthesis.
+- `summary.json` / `report.md` — measured executive summary and report.
+- `split.json` — fixed train/held-out paraphrase-group split.
+- `figures/` — report figures used by the public Study tab.
+
+Run `python -m scripts.validate_artifacts` to validate the committed study bundle.

artifacts/causal_position_summary.csv

@@ -0,0 +1,17 @@
+position_policy,tasks,feature_active_at_intervention_rate,feature_active_at_final_token_rate,feature_active_anywhere_rate,target_sae_abs_mean,target_random_abs_mean,target_specificity_ratio,target_paired_advantage,target_paired_ci_low,target_paired_ci_high,target_sign_flip_pvalue,active_target_sae_abs_mean,active_target_random_abs_mean,active_target_specificity_ratio,active_target_paired_advantage,active_target_paired_ci_low,active_target_paired_ci_high,active_target_sign_flip_pvalue,active_tasks,js_sae_mean,js_random_mean,js_specificity_ratio,active_js_specificity_ratio,concept
+final_token,28,0.2857142857142857,0.2857142857142857,0.0,0.007574045897594507,0.004990028782880718,1.5178361142081527,0.002584017114713789,-0.0005554523851190289,0.006223834152167133,0.171875,0.026509160641580775,0.017465100740082513,1.5178361142081527,0.009044059901498262,-0.0016050470294430876,0.01996337937016506,0.171875,8,0.00033045503460015093,0.00020158068911410593,1.6393189052602897,1.63931890526029,__all__
+final_token,4,0.5,0.5,0.0,0.03070369362831114,0.016958609223365763,1.8105077618043846,0.013745084404945379,0.0,0.030128069221973433,0.5,0.06140738725662228,0.033917218446731526,1.8105077618043846,0.027490168809890757,0.014809578657150269,0.040170758962631246,0.5,2,0.000468006153823775,0.0002624576891321125,1.7831680046081493,1.7831680046081493,code
+final_token,4,0.25,0.25,0.0,0.002008568495512,0.0012214824091642939,1.6443695631165165,0.0007870860863477061,0.0,0.0023612582590431183,1.0,0.008034273982048,0.0048859296366571756,1.6443695631165165,0.0031483443453908244,0.0031483443453908244,0.0031483443453908244,1.0,1,1.8891484614869114e-05,8.156611613685527e-06,2.316094661559236,2.316094661559236,factual_entities
+final_token,4,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0,,,,,,,,0,0.0,0.0,0.0,,german_language
+final_token,4,1.0,1.0,0.0,0.013766535557806436,0.01286024367436762,1.0704723725604977,0.0009062918834388161,-0.013581858249381188,0.010408063651993862,1.0,0.013766535557806436,0.01286024367436762,1.0704723725604977,0.0009062918834388161,-0.013581858249381188,0.010408063651993862,1.0,4,0.0017012150638037876,0.0010852779771539155,1.5675385473730286,1.5675385473730286,mathematics
+final_token,4,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0,,,,,,,,0,0.0,0.0,0.0,,negative_sentiment
+final_token,4,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0,,,,,,,,0,0.0,0.0,0.0,,positive_sentiment
+final_token,4,0.25,0.25,0.0,0.006539523601531975,0.0038898661732673515,1.6811693025518675,0.002649657428264623,0.0,0.00794897228479387,1.0,0.0261580944061279,0.015559464693069406,1.6811693025518675,0.010598629713058492,0.010598629713058492,0.010598629713058492,1.0,1,0.000125072539958625,5.517254589902812e-05,2.2669343587573723,2.2669343587573723,uncertainty
+max_feature_activation,28,0.8214285714285714,0.2857142857142857,0.8214285714285714,0.04140681956362508,0.01773720875956181,2.3344608571122225,0.023669610804063268,0.006155622166625111,0.04787022493006328,9.99950002499875e-05,0.05040830207745662,0.021593123707292633,2.334460857112223,0.028815178370163983,0.008065281513098975,0.05842191047641276,9.99950002499875e-05,23,0.0003503056863597102,0.00012276198071665368,2.853535633057673,2.8535356330576724,__all__
+max_feature_activation,4,1.0,0.5,1.0,0.0017386823892592963,0.0020397175103425555,0.8524133270627743,-0.00030103512108325915,-0.0026935096830129476,0.0025318711996078617,0.75,0.0017386823892592963,0.0020397175103425555,0.8524133270627743,-0.00030103512108325915,-0.0026935096830129476,0.0025318711996078617,0.75,4,7.899344154793653e-06,5.354482187414078e-06,1.4752769508434211,1.4752769508434211,code
+max_feature_activation,4,1.0,0.25,1.0,0.010054231388494334,0.007599377029691787,1.3230336314688824,0.0024548543588025474,-0.00026353719295001734,0.007209595802123662,0.5,0.010054231388494334,0.007599377029691787,1.3230336314688824,0.0024548543588025474,-0.00026353719295001734,0.007209595802123661,0.5,4,1.5267146636688267e-05,1.2116573988585344e-05,1.2600217397319557,1.2600217397319557,factual_entities
+max_feature_activation,4,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0,,,,,,,,0,0.0,0.0,0.0,,german_language
+max_feature_activation,4,1.0,1.0,1.0,0.010522400029003563,0.004535389365628313,2.3200654190240257,0.005987010663375247,0.0016587544232606831,0.010408063651993862,0.25,0.010522400029003563,0.004535389365628313,2.3200654190240257,0.005987010663375247,0.0016587544232606831,0.010408063651993862,0.25,4,0.0008034180818867435,0.0003310765635262305,2.4266836448031768,2.4266836448031768,mathematics
+max_feature_activation,4,1.0,0.0,1.0,0.08628357946872708,0.04091272130608554,2.1089670086524626,0.04537085816264153,0.010284267365932447,0.09931341186165814,0.125,0.08628357946872708,0.04091272130608554,2.1089670086524626,0.04537085816264153,0.010284267365932447,0.09931341186165814,0.125,4,0.0009591840007487634,0.00033541750718764655,2.8596718423888228,2.8596718423888228,negative_sentiment
+max_feature_activation,4,1.0,0.0,1.0,0.09154844284057614,0.052779592573642675,1.7345424315816724,0.03876885026693346,0.007168605923652652,0.07036909461021426,0.125,0.09154844284057614,0.052779592573642675,1.7345424315816724,0.03876885026693346,0.007168605923652652,0.07036909461021426,0.125,4,0.0001696736740086238,9.280085743055595e-05,1.8283632145919853,1.8283632145919853,positive_sentiment
+max_feature_activation,4,0.75,0.25,0.75,0.08970040082931516,0.016293663531541786,5.5052321815575915,0.07340673729777339,-0.000809632241725906,0.2134449183940888,0.5,0.11960053443908687,0.021724884708722383,5.505232181557591,0.09787564973036451,-0.001619264483451812,0.2845932245254517,0.5,3,0.0004966975570823587,8.256788069614337e-05,6.015626813896881,6.015626813896881,uncertainty

artifacts/feature_catalog.csv

@@ -0,0 +1,421 @@
+layer,concept,feature_id,train_auroc,auroc,f1,threshold,activation_rate_pos,activation_rate_neg,mean_activation_pos,mean_activation_neg
+4,code,10278,0.976707175925926,0.9375,0.9333333333333333,1.2032791376113892,0.9583333333333334,0.034722222222222224,4.418388843536377,0.030000614002346992
+4,code,11926,0.9542824074074076,0.9921875,0.8,1.0957155227661133,1.0,0.2013888888888889,1.5629621744155884,0.22529734671115875
+4,code,27791,0.9291087962962963,0.8723958333333333,0.8,1.0524113178253174,0.875,0.041666666666666664,1.898432731628418,0.05590388551354408
+4,code,29561,0.884837962962963,0.8177083333333334,0.5714285714285714,1.3156297206878662,0.8333333333333334,0.125,1.962118148803711,0.17066437005996704
+4,code,18816,0.8741319444444445,0.9283854166666666,0.7142857142857143,0.7793814539909363,0.75,0.006944444444444444,1.9922385215759277,0.004447112791240215
+4,code,31437,0.8706597222222223,0.9739583333333333,0.625,1.333459496498108,0.8333333333333334,0.1875,1.5057710409164429,0.21102415025234222
+4,code,28082,0.8706597222222222,0.8932291666666666,0.5384615384615384,1.374173879623413,1.0,0.6944444444444444,1.520123839378357,0.7452407479286194
+4,code,7545,0.8687789351851852,0.6119791666666666,0.2857142857142857,1.5763466358184814,0.9583333333333334,0.3125,1.9268847703933716,0.47133922576904297
+4,code,10853,0.8547453703703703,0.8411458333333334,0.6,0.7539695501327515,0.7916666666666666,0.1388888888888889,1.172544002532959,0.13423246145248413
+4,code,15227,0.8385416666666667,0.9505208333333334,0.5882352941176471,4.197094440460205,1.0,0.9375,3.888805627822876,2.461097478866577
+4,code,16398,0.8378182870370372,1.0,1.0,0.49933016300201416,0.7083333333333334,0.034722222222222224,0.6389889121055603,0.03022618405520916
+4,code,20474,0.8240740740740741,0.8645833333333334,0.5517241379310345,3.345871925354004,1.0,0.8888888888888888,6.263124942779541,2.9235239028930664
+4,code,29364,0.8198784722222222,0.7395833333333333,0.4,1.5703003406524658,1.0,0.5625,1.7845087051391602,0.7794884443283081
+4,code,842,0.8072916666666665,0.7734375,0.47619047619047616,2.7810254096984863,0.7916666666666666,0.2152777777777778,2.9755184650421143,0.6775578856468201
+4,code,20557,0.8049768518518519,0.7864583333333333,0.5,0.5132092237472534,0.8333333333333334,0.2152777777777778,0.8019671440124512,0.22202549874782562
+4,code,14731,0.7997685185185185,0.8723958333333334,0.45714285714285713,0.858819842338562,0.9583333333333334,0.4236111111111111,1.382489562034607,0.6176959276199341
+4,code,18809,0.7991898148148148,0.9153645833333334,0.625,1.0927000045776367,0.6666666666666666,0.06944444444444445,1.941078543663025,0.20922645926475525
+4,code,18513,0.7960069444444444,0.7864583333333334,0.631578947368421,1.3166978359222412,0.7916666666666666,0.4166666666666667,1.1745859384536743,0.39503997564315796
+4,code,25941,0.7954282407407407,0.8072916666666666,0.47058823529411764,1.223557710647583,1.0,0.4305555555555556,3.552171468734741,1.4364420175552368
+4,code,5088,0.7916666666666667,0.875,0.8571428571428571,0.8138328194618225,0.5833333333333334,0.0,0.955040454864502,0.0
+4,factual_entities,8144,0.9249131944444444,0.9049479166666666,0.75,1.6006373167037964,0.875,0.0625,2.2270891666412354,0.0741896703839302
+4,factual_entities,22020,0.888454861111111,0.9361979166666666,0.875,0.6328982710838318,0.7916666666666666,0.034722222222222224,1.0064629316329956,0.02380482293665409
+4,factual_entities,21177,0.8794849537037038,0.8919270833333334,0.7,0.5331253409385681,0.8333333333333334,0.0763888888888889,1.0882896184921265,0.10577809065580368
+4,factual_entities,17857,0.8579282407407408,0.8020833333333334,0.5454545454545454,1.707578420639038,0.75,0.10416666666666667,2.403292655944824,0.1424756646156311
+4,factual_entities,7917,0.8561921296296295,0.7395833333333334,0.47619047619047616,0.9847875833511353,0.7916666666666666,0.09722222222222222,1.4776204824447632,0.13152804970741272
+4,factual_entities,20439,0.8543113425925926,0.5091145833333333,0.17391304347826086,2.9984545707702637,0.875,0.3194444444444444,4.471306324005127,1.079331874847412
+4,factual_entities,20474,0.8480902777777777,0.8203124999999999,0.2222222222222222,7.030472755432129,1.0,0.8888888888888888,7.055498123168945,2.791461229324341
+4,factual_entities,14476,0.8425925925925927,0.8893229166666666,0.5333333333333333,1.2076637744903564,0.8333333333333334,0.1527777777777778,1.1911876201629639,0.21207401156425476
+4,factual_entities,6974,0.8368055555555557,0.9505208333333333,0.7,0.8071160316467285,0.9583333333333334,0.3888888888888889,1.0143204927444458,0.358742892742157
+4,factual_entities,23500,0.8288483796296298,0.8072916666666665,0.2857142857142857,2.5726747512817383,0.875,0.2638888888888889,2.958951234817505,0.774091362953186
+4,factual_entities,14694,0.8216145833333333,0.77734375,0.625,0.9580107927322388,0.75,0.09027777777777778,5.866950988769531,0.8641757369041443
+4,factual_entities,15267,0.8168402777777778,0.8229166666666666,0.47619047619047616,6.143719673156738,1.0,1.0,6.075092315673828,5.488575458526611
+4,factual_entities,28082,0.8087384259259259,0.6302083333333333,0.2,1.2470118999481201,1.0,0.6944444444444444,1.3869651556015015,0.7674338817596436
+4,factual_entities,1560,0.8084490740740742,0.3958333333333333,0.125,1.1900458335876465,0.8333333333333334,0.4027777777777778,1.4008506536483765,0.4972860515117645
+4,factual_entities,25941,0.8070023148148149,0.8072916666666666,0.5,2.250040054321289,1.0,0.4305555555555556,3.639948606491089,1.4218124151229858
+4,factual_entities,25806,0.8038194444444445,0.9010416666666666,0.6666666666666666,0.7108004093170166,0.7083333333333334,0.09027777777777778,1.1676274538040161,0.15598903596401215
+4,factual_entities,18294,0.8032407407407408,0.6302083333333334,0.0,1.8454582691192627,0.75,0.3402777777777778,1.7675472497940063,0.46256881952285767
+4,factual_entities,19265,0.8030960648148149,0.7109375,0.375,0.9675627946853638,0.9166666666666666,0.3333333333333333,4.231434345245361,1.2910268306732178
+4,factual_entities,31011,0.794994212962963,0.6302083333333334,0.36363636363636365,0.5558502674102783,0.75,0.16666666666666666,1.654288649559021,0.3606220483779907
+4,factual_entities,842,0.7941261574074074,0.8372395833333334,0.5714285714285714,1.9917594194412231,0.75,0.2222222222222222,3.0105838775634766,0.671713650226593
+4,german_language,4446,1.0,1.0,0.9333333333333333,4.393348693847656,1.0,0.0,7.520807266235352,0.0
+4,german_language,5270,1.0,1.0,1.0,1.5726087093353271,1.0,0.0,2.5866219997406006,0.0
+4,german_language,13798,1.0,0.9375,0.9333333333333333,2.569507598876953,1.0,0.0,5.592930316925049,0.0
+4,german_language,14533,1.0,1.0,0.9333333333333333,1.8416606187820435,1.0,0.0,4.630922794342041,0.0
+4,german_language,21930,1.0,1.0,0.8571428571428571,1.1024044752120972,1.0,0.0,2.3131725788116455,0.0
+4,german_language,12845,1.0,1.0,0.9333333333333333,1.4471431970596313,1.0,0.006944444444444444,2.1594607830047607,0.00805758684873581
+4,german_language,19213,1.0,1.0,0.9333333333333333,2.4823880195617676,1.0,0.020833333333333332,4.134191036224365,0.014218485914170742
+4,german_language,9046,0.9791666666666667,1.0,1.0,0.5936049222946167,0.9583333333333334,0.0,1.19443941116333,0.0
+4,german_language,21110,0.9791666666666667,1.0,1.0,4.499211311340332,0.9583333333333334,0.0,9.828652381896973,0.0
+4,german_language,25109,0.9790219907407408,1.0,1.0,3.2203047275543213,0.9583333333333334,0.006944444444444444,6.369612216949463,0.007427038624882698
+4,german_language,16885,0.9761284722222223,0.9375,0.9333333333333333,0.7512394785881042,0.9583333333333334,0.020833333333333332,2.152449369430542,0.02417704090476036
+4,german_language,16404,0.9375,1.0,1.0,0.8717712163925171,0.875,0.0,2.3563225269317627,0.0
+4,german_language,17290,0.9295428240740741,0.9817708333333333,0.8235294117647058,1.0776267051696777,0.875,0.0625,1.7418689727783203,0.05715159699320793
+4,german_language,12413,0.9168113425925926,0.7161458333333334,0.46153846153846156,6.849007606506348,0.875,0.1597222222222222,8.61686897277832,0.6021971106529236
+4,german_language,17249,0.9166666666666667,0.75,0.6666666666666666,0.7390999794006348,0.8333333333333334,0.0,2.9770984649658203,0.0
+4,german_language,24221,0.9166666666666667,0.875,0.8571428571428571,0.9253237247467041,0.8333333333333334,0.0,5.1704301834106445,0.0
+4,german_language,27692,0.9166666666666667,0.8125,0.7692307692307693,1.2853788137435913,0.8333333333333334,0.0,4.4340739250183105,0.0
+4,german_language,7325,0.8961226851851851,0.7734375,0.18181818181818182,4.112450122833252,0.9166666666666666,0.2152777777777778,4.328860759735107,0.706614077091217
+4,german_language,7146,0.8958333333333334,0.8671875,0.6666666666666666,11.01603889465332,1.0,0.8888888888888888,12.270854949951172,4.935203552246094
+4,german_language,9858,0.8958333333333333,0.75,0.6666666666666666,1.3752641677856445,0.7916666666666666,0.0,4.101892948150635,0.0
+4,mathematics,19936,0.9730902777777777,0.9921875,0.875,1.7461657524108887,1.0,0.24305555555555555,2.233029365539551,0.28995025157928467
+4,mathematics,4194,0.9673032407407407,0.9661458333333333,0.6666666666666666,1.898486614227295,1.0,0.22916666666666666,2.5828492641448975,0.3127814829349518
+4,mathematics,28442,0.9458912037037038,0.9947916666666666,0.9411764705882353,2.709167242050171,1.0,0.7291666666666666,3.289522409439087,1.3254538774490356
+4,mathematics,6267,0.9331597222222223,0.8619791666666667,0.7142857142857143,0.6845664978027344,0.875,0.041666666666666664,1.5970492362976074,0.028237124904990196
+4,mathematics,7059,0.929542824074074,0.9921875,0.875,1.3622171878814697,0.875,0.04861111111111111,4.448753833770752,0.06709078699350357
+4,mathematics,8720,0.9175347222222222,0.9010416666666667,0.42857142857142855,5.4397993087768555,1.0,0.375,5.666164398193359,1.389845371246338
+4,mathematics,20048,0.9129050925925927,0.8125,0.7692307692307693,0.8470262289047241,0.8333333333333334,0.020833333333333332,2.113173723220825,0.024394122883677483
+4,mathematics,5717,0.8958333333333333,0.8125,0.7692307692307693,1.2561579942703247,0.7916666666666666,0.0,1.830373764038086,0.0
+4,mathematics,8128,0.8799189814814814,0.83203125,0.43478260869565216,2.647125005722046,0.9166666666666666,0.2569444444444444,5.800546169281006,0.9868568778038025
+4,mathematics,25941,0.8796296296296297,0.8723958333333334,0.5833333333333334,3.222856044769287,1.0,0.4305555555555556,4.240810394287109,1.3216688632965088
+4,mathematics,9772,0.8700810185185185,0.9140625,0.7142857142857143,2.403669595718384,0.8333333333333334,0.1597222222222222,2.3101999759674072,0.29688331484794617
+4,mathematics,5686,0.8671875,0.8541666666666667,0.6666666666666666,0.8069468140602112,0.7916666666666666,0.05555555555555555,1.2079116106033325,0.08566660434007645
+4,mathematics,27946,0.8645833333333333,0.8333333333333333,0.6153846153846154,1.0407962799072266,0.875,0.25,1.576084017753601,0.26405268907546997
+4,mathematics,27123,0.8528645833333335,0.75,0.6666666666666666,1.2703733444213867,0.7083333333333334,0.006944444444444444,3.3084399700164795,0.01269147265702486
+4,mathematics,14202,0.8467881944444443,0.8802083333333333,0.631578947368421,2.060535192489624,0.875,0.3541666666666667,2.3172881603240967,0.5823152661323547
+4,mathematics,1622,0.8460648148148148,0.9244791666666666,0.7058823529411765,4.67432975769043,0.9583333333333334,0.6944444444444444,4.5706400871276855,2.1011950969696045
+4,mathematics,5833,0.8450520833333334,0.75,0.6666666666666666,0.6136884689331055,0.7083333333333334,0.020833333333333332,3.7702019214630127,0.06531283259391785
+4,mathematics,12710,0.8321759259259259,0.6875,0.4,0.9068191051483154,0.6666666666666666,0.006944444444444444,1.2932112216949463,0.005034128203988075
+4,mathematics,10721,0.8287037037037036,0.7447916666666666,0.6666666666666666,0.5282778739929199,0.6666666666666666,0.013888888888888888,0.8950223922729492,0.011460172943770885
+4,mathematics,13232,0.8229166666666666,0.75,0.6666666666666666,0.5201618671417236,0.6666666666666666,0.013888888888888888,1.0885707139968872,0.03326129913330078
+4,negative_sentiment,27671,0.9186921296296297,0.890625,0.5454545454545454,2.632431983947754,1.0,1.0,2.8054873943328857,2.054318904876709
+4,negative_sentiment,11570,0.8472222222222223,0.8151041666666666,0.6,0.9704810380935669,0.8333333333333334,0.18055555555555555,1.435254454612732,0.2537994682788849
+4,negative_sentiment,6581,0.8407118055555556,0.8020833333333333,0.4166666666666667,1.0220487117767334,0.9583333333333334,0.3402777777777778,1.3556634187698364,0.44338202476501465
+4,negative_sentiment,24935,0.8385416666666666,0.7734375,0.5217391304347826,5.77016544342041,0.875,0.2361111111111111,8.15478801727295,1.926889419555664
+4,negative_sentiment,24847,0.8285590277777778,0.8020833333333334,0.25,1.2869073152542114,0.875,0.3125,1.1535202264785767,0.3420311212539673
+4,negative_sentiment,21266,0.8188657407407407,0.6145833333333333,0.1111111111111111,1.580810785293579,1.0,0.5069444444444444,1.7455945014953613,0.7053961157798767
+4,negative_sentiment,3046,0.8081597222222222,0.9401041666666666,0.5925925925925926,1.6519724130630493,1.0,0.9652777777777778,1.7697092294692993,1.2778849601745605
+4,negative_sentiment,22726,0.8055555555555555,0.7994791666666667,0.6666666666666666,0.8829817175865173,0.6666666666666666,0.05555555555555555,3.4048702716827393,0.2869943082332611
+4,negative_sentiment,8912,0.794994212962963,0.6536458333333333,0.4,0.5224621295928955,0.75,0.13194444444444445,0.7961949706077576,0.15600813925266266
+4,negative_sentiment,22459,0.7938368055555556,0.5299479166666666,0.18181818181818182,0.5900578498840332,0.625,0.04861111111111111,0.9064140319824219,0.055760666728019714
+4,negative_sentiment,5677,0.7847222222222223,0.7330729166666665,0.32,1.375026822090149,0.9583333333333334,0.4583333333333333,1.7057290077209473,0.7393239140510559
+4,negative_sentiment,28169,0.7822627314814814,0.7499999999999999,0.36363636363636365,0.7808412313461304,0.7916666666666666,0.24305555555555555,1.377357840538025,0.38587892055511475
+4,negative_sentiment,3557,0.7798032407407408,0.703125,0.4,0.819602906703949,0.7083333333333334,0.14583333333333334,1.4632006883621216,0.31420332193374634
+4,negative_sentiment,14291,0.7711226851851852,0.7708333333333334,0.5714285714285714,0.4927108585834503,0.8333333333333334,0.2222222222222222,1.0605136156082153,0.4518792927265167
+4,negative_sentiment,19263,0.767505787037037,0.8645833333333333,0.8571428571428571,1.231980800628662,0.625,0.11805555555555555,1.1912404298782349,0.17938601970672607
+4,negative_sentiment,29551,0.754050925925926,0.7239583333333334,0.5,0.5144350528717041,0.5833333333333334,0.08333333333333333,0.5819621682167053,0.07692860811948776
+4,negative_sentiment,12445,0.7513020833333333,0.8046874999999999,0.45161290322580644,0.6227579116821289,0.7916666666666666,0.3680555555555556,0.7437607645988464,0.2962469458580017
+4,negative_sentiment,31734,0.7488425925925927,0.8203125,0.45714285714285713,3.9845144748687744,1.0,0.9791666666666666,4.821115493774414,3.5460591316223145
+4,negative_sentiment,12219,0.7416087962962963,0.58203125,0.2222222222222222,1.5859357118606567,0.7916666666666666,0.3958333333333333,1.7345331907272339,0.7666224241256714
+4,negative_sentiment,25201,0.7401620370370371,0.8424479166666666,0.5454545454545454,1.4018964767456055,0.5833333333333334,0.11805555555555555,1.8707882165908813,0.3459416627883911
+4,positive_sentiment,14291,0.9214409722222223,0.8125000000000001,0.5714285714285714,1.7957007884979248,0.9583333333333334,0.2013888888888889,2.2091691493988037,0.2604367136955261
+4,positive_sentiment,6581,0.9089988425925927,0.9348958333333333,0.6,1.4286385774612427,0.9583333333333334,0.3402777777777778,1.717237114906311,0.38311973214149475
+4,positive_sentiment,5677,0.8567708333333334,0.8828125000000001,0.47058823529411764,1.7066278457641602,1.0,0.4513888888888889,1.9567629098892212,0.6974849700927734
+4,positive_sentiment,24935,0.8547453703703703,0.8020833333333333,0.5454545454545454,7.175022125244141,0.9166666666666666,0.22916666666666666,8.353458404541016,1.89377760887146
+4,positive_sentiment,27988,0.8463541666666666,0.8489583333333333,0.48484848484848486,0.8339418172836304,0.875,0.25,1.976244568824768,0.36828330159187317
+4,positive_sentiment,3046,0.8376736111111112,0.8567708333333334,0.48,1.7441538572311401,1.0,0.9652777777777778,1.8059067726135254,1.2718520164489746
+4,positive_sentiment,1685,0.8370949074074074,0.8463541666666666,0.7142857142857143,0.8143419623374939,0.75,0.13194444444444445,0.9168570637702942,0.11393418908119202
+4,positive_sentiment,18376,0.8313078703703705,0.8437499999999999,0.5,0.7220430970191956,0.75,0.13194444444444445,0.720099151134491,0.09635607898235321
+4,positive_sentiment,31734,0.8266782407407407,0.8880208333333334,0.5,5.262398719787598,1.0,0.9791666666666666,5.057671070098877,3.5066332817077637
+4,positive_sentiment,3259,0.8054108796296298,0.6171875,0.2222222222222222,2.683941125869751,0.7083333333333334,0.1875,3.287898302078247,0.4814560115337372
+4,positive_sentiment,27671,0.8052662037037037,0.8385416666666667,0.5517241379310345,2.4368858337402344,1.0,1.0,2.615980863571167,2.0859034061431885
+4,positive_sentiment,28169,0.8023726851851852,0.69921875,0.47619047619047616,0.8476548790931702,0.8333333333333334,0.2361111111111111,1.347327709197998,0.3908839225769043
+4,positive_sentiment,10274,0.7954282407407407,0.5963541666666667,0.0,2.957977771759033,0.75,0.2777777777777778,2.0302813053131104,0.42107710242271423
+4,positive_sentiment,12088,0.7838541666666666,0.6979166666666667,0.14285714285714285,3.1934754848480225,0.625,0.09722222222222222,4.0530900955200195,0.3295298218727112
+4,positive_sentiment,3557,0.7821180555555557,0.7421875000000001,0.38095238095238093,1.05930757522583,0.7083333333333334,0.14583333333333334,1.5223571062088013,0.30434390902519226
+4,positive_sentiment,24335,0.7690972222222223,0.87890625,0.6,0.8527958393096924,0.8333333333333334,0.375,0.8688912391662598,0.3435428738594055
+4,positive_sentiment,30373,0.7682291666666667,0.5104166666666666,0.13333333333333333,1.8445837497711182,0.7083333333333334,0.22916666666666666,1.9130125045776367,0.5006615519523621
+4,positive_sentiment,12179,0.7651909722222222,0.59375,0.3076923076923077,1.1551002264022827,0.625,0.11805555555555555,1.4909576177597046,0.17718206346035004
+4,positive_sentiment,21968,0.7614293981481481,0.6809895833333334,0.43478260869565216,0.744644820690155,0.7083333333333334,0.20833333333333334,1.028828501701355,0.2695753574371338
+4,positive_sentiment,20453,0.759548611111111,0.8854166666666667,0.56,0.9391292333602905,0.625,0.1111111111111111,3.4236724376678467,0.6115831136703491
+4,uncertainty,25578,0.9639756944444444,0.9895833333333334,0.8421052631578947,3.843626022338867,0.9583333333333334,0.11805555555555555,5.184982776641846,0.3407251834869385
+4,uncertainty,17474,0.9267939814814814,0.9895833333333334,0.7777777777777778,0.848248302936554,0.9166666666666666,0.1527777777777778,1.4993548393249512,0.14058072865009308
+4,uncertainty,25684,0.9166666666666666,0.9036458333333334,0.42857142857142855,1.4602006673812866,1.0,0.2777777777777778,2.366671323776245,0.41858142614364624
+4,uncertainty,23356,0.9061053240740742,0.8645833333333334,0.7368421052631579,0.7833821773529053,0.875,0.09027777777777778,1.7163801193237305,0.1461271345615387
+4,uncertainty,16314,0.8729745370370371,0.9375,0.9333333333333333,0.6525366902351379,0.75,0.006944444444444444,1.753684639930725,0.011013324372470379
+4,uncertainty,17805,0.8640046296296297,0.8893229166666666,0.7777777777777778,0.615090012550354,0.75,0.034722222222222224,1.5226413011550903,0.04787616431713104
+4,uncertainty,3833,0.8609664351851851,0.8463541666666666,0.0,1.2375357151031494,0.9583333333333334,0.4930555555555556,1.083256721496582,0.4023979902267456
+4,uncertainty,28465,0.853298611111111,0.5,0.0,0.4962771534919739,0.75,0.04861111111111111,1.019452691078186,0.06170894205570221
+4,uncertainty,27756,0.8414351851851851,0.78125,0.26666666666666666,1.135574221611023,0.75,0.1111111111111111,1.3068552017211914,0.14657002687454224
+4,uncertainty,8250,0.8269675925925926,0.8125,0.7692307692307693,0.6862602233886719,0.6666666666666666,0.013888888888888888,1.3633298873901367,0.016644857823848724
+4,uncertainty,31734,0.8249421296296297,0.7135416666666666,0.2608695652173913,4.855778694152832,1.0,0.9791666666666666,5.070761203765869,3.504451274871826
+4,uncertainty,30847,0.8078703703703703,0.734375,0.3,3.00431489944458,0.875,0.2777777777777778,3.3732693195343018,1.0374754667282104
+4,uncertainty,27364,0.8020833333333333,0.671875,0.3333333333333333,5.873011589050293,0.7083333333333334,0.2361111111111111,5.18310546875,0.9254935383796692
+4,uncertainty,1126,0.8019386574074073,0.75,0.6666666666666666,0.48371630907058716,0.625,0.020833333333333332,0.5199914574623108,0.017288779839873314
+4,uncertainty,20936,0.7931134259259259,0.8385416666666666,0.47619047619047616,1.7751885652542114,0.9583333333333334,0.5,1.7565059661865234,0.7985113859176636
+4,uncertainty,12178,0.786892361111111,0.84375,0.3333333333333333,1.7468856573104858,0.7916666666666666,0.2569444444444444,2.134218692779541,0.5756012797355652
+4,uncertainty,26818,0.7783564814814815,0.7838541666666665,0.4827586206896552,1.1501868963241577,0.8333333333333334,0.3055555555555556,3.2335546016693115,1.1686296463012695
+4,uncertainty,17215,0.7760416666666666,0.8411458333333333,0.3333333333333333,2.1161718368530273,0.7916666666666666,0.3333333333333333,1.619903564453125,0.5084395408630371
+4,uncertainty,11579,0.7711226851851852,0.7291666666666666,0.5714285714285714,0.6519079208374023,0.5833333333333334,0.034722222222222224,1.3099130392074585,0.09980905055999756
+4,uncertainty,18102,0.7708333333333333,0.5625,0.2222222222222222,0.9655546545982361,0.5416666666666666,0.0,1.2243261337280273,0.0
+14,code,17112,0.9010416666666666,0.8111979166666666,0.6,12.202638626098633,1.0,0.5,15.558719635009766,5.50345516204834
+14,code,24589,0.8580729166666665,0.796875,0.5217391304347826,4.7050886154174805,0.9166666666666666,0.18055555555555555,11.7520751953125,2.5373637676239014
+14,code,32444,0.84375,0.9088541666666666,0.5,8.445877075195312,1.0,0.6111111111111112,10.858126640319824,5.115589618682861
+14,code,26714,0.8427372685185186,0.8098958333333333,0.5,4.66489839553833,0.9583333333333334,0.3819444444444444,6.145521640777588,2.173491954803467
+14,code,9652,0.8247974537037037,0.8489583333333333,0.6086956521739131,11.866023063659668,1.0,0.5416666666666666,15.862468719482422,6.437809944152832
+14,code,13285,0.8234953703703703,0.8411458333333333,0.5161290322580645,6.4700212478637695,1.0,0.3472222222222222,13.54662799835205,4.803369522094727
+14,code,13847,0.8217592592592593,0.7317708333333333,0.32,11.474407196044922,0.9583333333333334,0.6875,16.36869239807129,8.162473678588867
+14,code,17473,0.8125,0.6875,0.5454545454545454,6.638728141784668,0.625,0.0,10.004228591918945,0.0
+14,code,28200,0.8107638888888887,0.9661458333333334,0.7777777777777778,5.254382133483887,0.6666666666666666,0.04861111111111111,4.766644477844238,0.32778918743133545
+14,code,17137,0.8101851851851851,0.9895833333333334,0.8888888888888888,5.630497932434082,0.6666666666666666,0.09027777777777778,7.784315586090088,0.5334010720252991
+14,code,13858,0.8074363425925926,0.875,0.8571428571428571,5.109424114227295,0.625,0.020833333333333332,6.160994052886963,0.11436031758785248
+14,code,8198,0.8052662037037037,0.7057291666666666,0.5,5.032931327819824,0.7083333333333334,0.125,7.505390644073486,1.0126831531524658
+14,code,5082,0.7938368055555556,0.8125,0.4827586206896552,7.8591485023498535,1.0,0.4722222222222222,10.827916145324707,4.93565034866333
+14,code,24630,0.787326388888889,0.9375,0.9333333333333333,4.013854026794434,0.5833333333333334,0.013888888888888888,3.738023519515991,0.06342040747404099
+14,code,8448,0.7806712962962964,0.6171875,0.4,5.548151969909668,0.5833333333333334,0.027777777777777776,6.550729274749756,0.26618555188179016
+14,code,14915,0.7760416666666666,0.828125,0.36363636363636365,18.441234588623047,1.0,0.8194444444444444,19.334850311279297,12.077781677246094
+14,code,10648,0.7750289351851853,0.7213541666666667,0.46153846153846156,6.262637615203857,0.8333333333333334,0.3402777777777778,8.867045402526855,3.208226203918457
+14,code,22078,0.7743055555555556,0.7096354166666665,0.4375,4.309991836547852,0.875,0.3333333333333333,11.955726623535156,4.802287578582764
+14,code,8854,0.7725694444444444,0.625,0.32,4.8182477951049805,0.8333333333333334,0.3125,5.997705459594727,2.1632513999938965
+14,code,14406,0.7667824074074073,0.625,0.4,6.205851078033447,0.5416666666666666,0.013888888888888888,7.986114501953125,0.09746456146240234
+14,factual_entities,29048,0.9745370370370372,0.9075520833333334,0.8235294117647058,4.5814409255981445,0.9583333333333334,0.013888888888888888,9.182080268859863,0.10824672132730484
+14,factual_entities,11054,0.9664351851851852,0.9843750000000001,0.7619047619047619,8.199565887451172,1.0,0.3819444444444444,10.548897743225098,2.5219998359680176
+14,factual_entities,27839,0.9616608796296297,0.9583333333333334,0.6666666666666666,11.449817657470703,0.9583333333333334,0.13194444444444445,15.653618812561035,1.1670657396316528
+14,factual_entities,24754,0.9505208333333333,0.8125,0.6666666666666666,9.008230209350586,0.9166666666666666,0.034722222222222224,14.904926300048828,0.341569721698761
+14,factual_entities,7944,0.9363425925925926,0.9049479166666666,0.7058823529411765,5.836667060852051,0.9166666666666666,0.09027777777777778,10.403502464294434,0.6945812106132507
+14,factual_entities,30698,0.9307002314814814,0.79296875,0.625,4.662937164306641,0.9166666666666666,0.06944444444444445,8.24229621887207,0.49372202157974243
+14,factual_entities,12297,0.919849537037037,0.8697916666666666,0.6363636363636364,4.994018077850342,1.0,0.14583333333333334,6.785719394683838,1.0523158311843872
+14,factual_entities,32444,0.8972800925925926,0.8046875,0.43478260869565216,10.206499099731445,1.0,0.6111111111111112,11.897448539733887,4.942368984222412
+14,factual_entities,13285,0.8969907407407407,0.9140625,0.5714285714285714,13.770366668701172,1.0,0.3472222222222222,16.481332778930664,4.3142523765563965
+14,factual_entities,3660,0.8958333333333333,0.8723958333333333,0.8,4.00559139251709,0.7916666666666666,0.0,7.171806335449219,0.0
+14,factual_entities,29795,0.8729745370370371,0.8619791666666666,0.6666666666666666,3.9526124000549316,0.875,0.1527777777777778,16.635740280151367,2.4225263595581055
+14,factual_entities,20503,0.8566261574074074,0.7968749999999999,0.47058823529411764,6.958144664764404,0.875,0.20833333333333334,8.311877250671387,1.7522722482681274
+14,factual_entities,22078,0.8423032407407407,0.9192708333333334,0.64,9.544900894165039,1.0,0.3125,14.10808277130127,4.443562030792236
+14,factual_entities,24589,0.8381076388888891,0.8359375,0.631578947368421,9.44992733001709,0.8333333333333334,0.19444444444444445,12.326384544372559,2.441645622253418
+14,factual_entities,22236,0.8357928240740742,0.9153645833333334,0.7777777777777778,5.755143165588379,0.75,0.13194444444444445,9.295394897460938,1.0824912786483765
+14,factual_entities,1883,0.8333333333333334,0.7916666666666667,0.6666666666666666,4.432744979858398,0.7083333333333334,0.027777777777777776,5.222154140472412,0.32599937915802
+14,factual_entities,20738,0.830295138888889,0.8333333333333333,0.631578947368421,4.769126892089844,0.7083333333333334,0.0625,8.486958503723145,0.5885213017463684
+14,factual_entities,1606,0.8177083333333335,0.6822916666666666,0.375,5.896625518798828,0.7083333333333334,0.10416666666666667,7.519303798675537,0.8628498911857605
+14,factual_entities,14915,0.8159722222222222,0.8072916666666666,0.42857142857142855,16.624095916748047,1.0,0.8194444444444444,20.07499885559082,11.954423904418945
+14,factual_entities,21012,0.8125000000000001,0.7369791666666665,0.4117647058823529,4.450913429260254,1.0,0.3958333333333333,7.400697231292725,2.928971767425537
+14,german_language,14561,1.0,1.0,1.0,6.120171546936035,1.0,0.0,18.470163345336914,0.0
+14,german_language,20761,1.0,1.0,1.0,9.724266052246094,1.0,0.0,19.5790958404541,0.0
+14,german_language,28952,1.0,1.0,1.0,8.047584533691406,1.0,0.0,11.750163078308105,0.0
+14,german_language,20137,1.0,1.0,0.9333333333333333,30.80394744873047,1.0,0.006944444444444444,37.014678955078125,0.048012617975473404
+14,german_language,4214,0.9791666666666667,0.875,0.8571428571428571,6.585235595703125,0.9583333333333334,0.0,13.898131370544434,0.0
+14,german_language,16754,0.9583333333333333,0.875,0.8571428571428571,6.579704284667969,0.9166666666666666,0.0,16.944862365722656,0.0
+14,german_language,13932,0.9166666666666667,0.8125,0.7692307692307693,5.8387837409973145,0.8333333333333334,0.0,11.406414031982422,0.0
+14,german_language,369,0.9160879629629631,1.0,1.0,5.769578456878662,0.8333333333333334,0.006944444444444444,7.0089640617370605,0.035892702639102936
+14,german_language,25082,0.900462962962963,0.8307291666666665,0.46153846153846156,40.26825714111328,0.875,0.19444444444444445,39.515472412109375,5.108692646026611
+14,german_language,9501,0.8894675925925926,0.8854166666666667,0.5555555555555556,17.079925537109375,1.0,0.4236111111111111,20.16521644592285,6.375813007354736
+14,german_language,17349,0.8854166666666667,0.8450520833333334,0.3076923076923077,9.201055526733398,0.8333333333333334,0.1597222222222222,12.01998519897461,1.1771736145019531
+14,german_language,573,0.875,0.9375,0.9333333333333333,5.959286689758301,0.75,0.0,15.875457763671875,0.0
+14,german_language,6894,0.875,0.875,0.8571428571428571,6.49418830871582,0.75,0.0,11.579211235046387,0.0
+14,german_language,711,0.8663194444444444,0.8255208333333333,0.6666666666666666,8.470067977905273,0.9166666666666666,0.25,9.767167091369629,2.2954018115997314
+14,german_language,28771,0.8541666666666667,0.6809895833333333,0.5,5.297268867492676,0.7083333333333334,0.0,10.491676330566406,0.0
+14,german_language,20335,0.8524305555555555,0.5651041666666666,0.2857142857142857,6.119721412658691,0.7916666666666666,0.1388888888888889,12.488452911376953,1.4833922386169434
+14,german_language,406,0.8382523148148149,0.8359375,0.7058823529411765,4.781467914581299,0.7083333333333334,0.027777777777777776,6.53454065322876,0.32053184509277344
+14,german_language,4904,0.8333333333333333,0.625,0.4,7.650750160217285,0.6666666666666666,0.0,14.389713287353516,0.0
+14,german_language,7368,0.8333333333333333,0.5625,0.0,6.916831016540527,0.6666666666666666,0.0,9.575196266174316,0.0
+14,german_language,13547,0.8333333333333333,1.0,1.0,4.816638946533203,0.6666666666666666,0.0,7.569114685058594,0.0
+14,mathematics,16043,0.9797453703703705,0.9947916666666666,0.6666666666666666,21.58745765686035,1.0,0.16666666666666666,22.55883026123047,1.6195133924484253
+14,mathematics,5082,0.9288194444444444,0.9296875,0.6666666666666666,12.464324951171875,1.0,0.4722222222222222,14.283251762390137,4.359760761260986
+14,mathematics,24684,0.9152199074074076,0.875,0.8571428571428571,5.230359077453613,0.8333333333333334,0.006944444444444444,9.417037963867188,0.04013775289058685
+14,mathematics,24095,0.9039351851851851,0.8502604166666667,0.5333333333333333,11.950861930847168,0.9166666666666666,0.2569444444444444,17.800168991088867,2.291503429412842
+14,mathematics,20328,0.8958333333333333,0.6875,0.5454545454545454,4.394455432891846,0.7916666666666666,0.0,7.314889907836914,0.0
+14,mathematics,21885,0.8935185185185185,0.875,0.8571428571428571,5.753984451293945,0.7916666666666666,0.013888888888888888,19.85708236694336,0.1335276961326599
+14,mathematics,13847,0.8810763888888888,0.8502604166666667,0.36363636363636365,21.33550262451172,1.0,0.6805555555555556,19.830488204956055,7.585507869720459
+14,mathematics,5712,0.8778935185185185,0.8515625,0.5333333333333333,7.252956390380859,0.875,0.16666666666666666,9.094002723693848,1.4180903434753418
+14,mathematics,5311,0.8726851851851852,0.875,0.7692307692307693,5.660670757293701,0.75,0.013888888888888888,6.908512115478516,0.07089350372552872
+14,mathematics,25652,0.8657407407407406,0.8125,0.5454545454545454,8.53048038482666,0.75,0.04861111111111111,11.375609397888184,0.3374844491481781
+14,mathematics,4306,0.8472222222222223,0.75,0.6666666666666666,3.6869451999664307,0.7083333333333334,0.013888888888888888,5.846765518188477,0.12057098001241684
+14,mathematics,22214,0.8451967592592592,0.734375,0.5714285714285714,4.1711578369140625,0.75,0.09027777777777778,11.260197639465332,0.8573513031005859
+14,mathematics,25453,0.8376736111111112,0.7239583333333334,0.46153846153846156,7.761497497558594,0.7083333333333334,0.06944444444444445,15.330487251281738,0.6587920188903809
+14,mathematics,4974,0.8333333333333333,0.75,0.6666666666666666,6.498987674713135,0.6666666666666666,0.0,10.219491004943848,0.0
+14,mathematics,5718,0.8321759259259258,0.8177083333333334,0.5454545454545454,5.559237480163574,0.9166666666666666,0.2847222222222222,8.387553215026855,2.328418731689453
+14,mathematics,2934,0.8315972222222222,0.7473958333333334,0.3333333333333333,6.676140785217285,0.875,0.2222222222222222,12.373892784118652,3.91027569770813
+14,mathematics,16885,0.826388888888889,0.6484374999999999,0.42857142857142855,6.64892053604126,0.7083333333333334,0.08333333333333333,7.346517086029053,0.6272312998771667
+14,mathematics,20227,0.8191550925925926,0.8958333333333335,0.6363636363636364,21.520057678222656,0.875,0.4722222222222222,32.1407585144043,9.883152961730957
+14,mathematics,14411,0.8098958333333334,0.6875,0.5454545454545454,9.072854995727539,0.625,0.013888888888888888,12.45840835571289,0.07535172998905182
+14,mathematics,5708,0.8093171296296297,0.7447916666666666,0.6666666666666666,4.490284442901611,0.625,0.013888888888888888,5.835394382476807,0.06403409689664841
+14,negative_sentiment,14651,0.9971064814814815,1.0,0.9411764705882353,6.187005519866943,1.0,0.034722222222222224,11.755587577819824,0.22162209451198578
+14,negative_sentiment,3375,0.984375,1.0,0.9333333333333333,13.439130783081055,1.0,0.125,19.79825782775879,1.3133820295333862
+14,negative_sentiment,13110,0.8982928240740741,0.9817708333333334,0.875,5.484504699707031,0.875,0.10416666666666667,6.8915934562683105,0.7469542622566223
+14,negative_sentiment,30941,0.8579282407407408,0.8072916666666666,0.5263157894736842,8.884664535522461,0.875,0.2013888888888889,11.9073486328125,2.337437629699707
+14,negative_sentiment,12012,0.857060185185185,0.8541666666666665,0.5263157894736842,13.234259605407715,0.9166666666666666,0.3611111111111111,14.730194091796875,3.7266829013824463
+14,negative_sentiment,25445,0.8504050925925927,0.8671874999999999,0.8,3.4155964851379395,0.7083333333333334,0.013888888888888888,7.621945858001709,0.07026167213916779
+14,negative_sentiment,1880,0.8502604166666667,0.796875,0.5714285714285714,6.181522846221924,0.9583333333333334,0.3263888888888889,6.971283435821533,2.1751868724823
+14,negative_sentiment,14862,0.8352141203703703,0.8125,0.5714285714285714,6.5991010665893555,0.875,0.1875,10.16930103302002,2.3025126457214355
+14,negative_sentiment,10903,0.818431712962963,0.8919270833333334,0.7777777777777778,5.4827399253845215,0.7083333333333334,0.0625,7.358834743499756,0.7045939564704895
+14,negative_sentiment,11565,0.8168402777777778,0.9114583333333334,0.6,12.426542282104492,1.0,0.9097222222222222,14.223057746887207,9.260567665100098
+14,negative_sentiment,29502,0.8097511574074076,0.765625,0.5882352941176471,5.210999488830566,0.7083333333333334,0.09027777777777778,8.241983413696289,1.0414873361587524
+14,negative_sentiment,14448,0.8043981481481481,0.9036458333333333,0.15384615384615385,14.799243927001953,1.0,0.5069444444444444,13.360369682312012,6.059085369110107
+14,negative_sentiment,18682,0.7938368055555556,0.9375,0.9333333333333333,5.726962089538574,0.625,0.04861111111111111,4.964136600494385,0.3205527067184448
+14,negative_sentiment,14766,0.7913773148148148,0.9192708333333333,0.6956521739130435,4.842212200164795,0.7916666666666666,0.1736111111111111,10.559374809265137,3.0374295711517334
+14,negative_sentiment,25861,0.7905092592592592,0.6875,0.23529411764705882,5.880181789398193,0.75,0.20833333333333334,7.247306823730469,1.5961079597473145
+14,negative_sentiment,24960,0.7890625000000001,0.8828125,0.6666666666666666,4.060155868530273,0.75,0.2013888888888889,7.101438045501709,1.7000269889831543
+14,negative_sentiment,9413,0.7792245370370371,0.8020833333333333,0.375,12.926658630371094,0.7083333333333334,0.1875,10.095904350280762,2.128706693649292
+14,negative_sentiment,21102,0.7766203703703705,0.7786458333333334,0.48,4.210739612579346,0.8333333333333334,0.2361111111111111,7.3815083503723145,2.457446575164795
+14,negative_sentiment,28930,0.7731481481481483,0.7942708333333334,0.6,3.8037960529327393,0.7083333333333334,0.1875,11.070972442626953,2.284876823425293
+14,negative_sentiment,5903,0.7701099537037037,0.75,0.5882352941176471,6.399606704711914,0.7083333333333334,0.1875,5.366062164306641,1.3566018342971802
+14,positive_sentiment,29229,0.9716435185185186,0.9869791666666667,0.7777777777777778,8.705174446105957,0.9583333333333334,0.1388888888888889,12.223082542419434,0.9231535792350769
+14,positive_sentiment,28480,0.9328703703703703,0.9049479166666667,0.7,6.652054786682129,0.9166666666666666,0.0763888888888889,10.742648124694824,0.6479031443595886
+14,positive_sentiment,8202,0.9126157407407406,0.8203125,0.46153846153846156,8.14765453338623,0.9166666666666666,0.1736111111111111,17.46805191040039,1.7573322057724
+14,positive_sentiment,14766,0.9046585648148147,0.8619791666666666,0.6363636363636364,6.622534275054932,0.9166666666666666,0.1527777777777778,16.701318740844727,2.0137720108032227
+14,positive_sentiment,21102,0.8816550925925926,0.7916666666666667,0.4,11.443771362304688,0.9166666666666666,0.2222222222222222,10.276955604553223,1.97487211227417
+14,positive_sentiment,27944,0.8816550925925926,0.8098958333333333,0.4166666666666667,47.95894241333008,1.0,1.0,49.58729553222656,36.45000457763672
+14,positive_sentiment,14862,0.8655960648148148,0.8046874999999999,0.5714285714285714,9.297593116760254,0.875,0.1875,11.3898286819458,2.0990915298461914
+14,positive_sentiment,13577,0.8642939814814814,0.7864583333333334,0.5714285714285714,6.623638153076172,0.7916666666666666,0.06944444444444445,13.62905216217041,1.0502052307128906
+14,positive_sentiment,13343,0.8466435185185184,0.8203125,0.5,7.003763675689697,0.75,0.08333333333333333,8.682491302490234,0.69874107837677
+14,positive_sentiment,19152,0.8362268518518519,0.6289062499999999,0.3076923076923077,7.941379547119141,0.7083333333333334,0.08333333333333333,9.741189002990723,0.7029301524162292
+14,positive_sentiment,7350,0.818576388888889,0.828125,0.631578947368421,5.3849382400512695,0.75,0.13194444444444445,7.843770980834961,1.1473225355148315
+14,positive_sentiment,1880,0.8165509259259259,0.72265625,0.2857142857142857,6.0974907875061035,0.8333333333333334,0.3472222222222222,6.872988224029541,2.1915693283081055
+14,positive_sentiment,7904,0.8165509259259258,0.7968750000000001,0.631578947368421,4.589892864227295,0.7916666666666666,0.1527777777777778,5.1257805824279785,1.0514365434646606
+14,positive_sentiment,11348,0.8055555555555556,0.6276041666666667,0.3333333333333333,4.070797920227051,0.7916666666666666,0.16666666666666666,4.9079694747924805,1.096775770187378
+14,positive_sentiment,2932,0.7986111111111112,0.7265624999999999,0.36363636363636365,8.50406551361084,0.7916666666666666,0.25,18.7618350982666,4.8599162101745605
+14,positive_sentiment,30941,0.7857349537037037,0.734375,0.47619047619047616,6.313429832458496,0.8333333333333334,0.20833333333333334,9.375701904296875,2.759378671646118
+14,positive_sentiment,30249,0.7844328703703703,0.8645833333333334,0.56,4.124691486358643,0.8333333333333334,0.25,5.446117401123047,1.6865448951721191
+14,positive_sentiment,23374,0.7839988425925926,0.5416666666666666,0.14285714285714285,11.8710355758667,0.625,0.0625,16.704805374145508,1.557742714881897
+14,positive_sentiment,14448,0.7763310185185186,0.72265625,0.41379310344827586,8.76548957824707,1.0,0.5069444444444444,12.815731048583984,6.149858474731445
+14,positive_sentiment,30042,0.7725694444444445,0.9505208333333334,0.7619047619047619,3.828726053237915,0.5833333333333334,0.041666666666666664,8.459391593933105,0.42310455441474915
+14,uncertainty,1384,0.9752604166666667,0.875,0.8571428571428571,5.794346809387207,0.9583333333333334,0.006944444444444444,14.314885139465332,0.10034044831991196
+14,uncertainty,18561,0.9398148148148148,1.0,0.9333333333333333,6.671687126159668,0.9166666666666666,0.09027777777777778,10.333622932434082,0.5476462841033936
+14,uncertainty,7680,0.9324363425925926,1.0,0.9333333333333333,4.618145942687988,0.875,0.020833333333333332,10.585579872131348,0.15416482090950012
+14,uncertainty,21317,0.9149305555555557,1.0,1.0,4.084063529968262,0.8333333333333334,0.006944444444444444,11.284405708312988,0.046167273074388504
+14,uncertainty,16309,0.8958333333333333,0.9375,0.9333333333333333,5.558524131774902,0.7916666666666666,0.0,7.656550884246826,0.0
+14,uncertainty,24794,0.882667824074074,0.9231770833333334,0.7777777777777778,5.144367218017578,0.875,0.1736111111111111,7.787540435791016,1.2173820734024048
+14,uncertainty,8302,0.8783275462962962,1.0,0.8888888888888888,11.035469055175781,0.7916666666666666,0.04861111111111111,16.604263305664062,0.8135746121406555
+14,uncertainty,14056,0.875,1.0,1.0,3.9954674243927,0.75,0.0,7.939446926116943,0.0
+14,uncertainty,22860,0.8705150462962963,0.7747395833333333,0.6666666666666666,9.004796981811523,0.875,0.1875,9.568285942077637,1.6190071105957031
+14,uncertainty,20255,0.8386863425925927,0.9895833333333334,0.8235294117647058,4.701651573181152,0.875,0.2013888888888889,8.307440757751465,1.6054338216781616
+14,uncertainty,4618,0.814525462962963,0.78125,0.625,7.255876064300537,0.75,0.16666666666666666,8.39405345916748,1.65819251537323
+14,uncertainty,25474,0.8125,0.5,0.0,4.758297920227051,0.625,0.0,7.718653202056885,0.0
+14,uncertainty,5132,0.8103298611111112,0.4895833333333333,0.0,4.330278396606445,0.625,0.006944444444444444,5.7183356285095215,0.04100927338004112
+14,uncertainty,14448,0.8046875,0.7200520833333333,0.35294117647058826,12.522350311279297,1.0,0.5069444444444444,13.329840660095215,6.064173221588135
+14,uncertainty,8430,0.7964409722222222,0.625,0.4,3.5832786560058594,0.625,0.034722222222222224,3.9469757080078125,0.20356009900569916
+14,uncertainty,22632,0.7899305555555555,0.5703125,0.36363636363636365,21.324459075927734,0.6666666666666666,0.20833333333333334,18.53244972229004,3.1122729778289795
+14,uncertainty,23128,0.7853009259259259,0.71875,0.0,6.286028861999512,0.5833333333333334,0.020833333333333332,10.925019264221191,0.17531190812587738
+14,uncertainty,2927,0.7847222222222223,0.8606770833333334,0.5,6.759663105010986,0.8333333333333334,0.3680555555555556,6.427015781402588,2.478482961654663
+14,uncertainty,3028,0.7812500000000001,0.6875,0.5454545454545454,5.297049045562744,0.5833333333333334,0.013888888888888888,7.675222873687744,0.40646451711654663
+14,uncertainty,31444,0.7783564814814815,0.5963541666666666,0.0,26.799213409423828,0.75,0.22916666666666666,19.109914779663086,5.366549015045166
+26,code,19655,0.9881365740740741,0.9661458333333334,0.8235294117647058,80.02330017089844,1.0,0.18055555555555555,140.38720703125,12.174335479736328
+26,code,21186,0.9560185185185185,0.953125,0.7368421052631579,128.4604949951172,1.0,0.3611111111111111,159.04949951171875,33.98159408569336
+26,code,5778,0.9192708333333333,0.9817708333333334,0.8888888888888888,171.9294891357422,1.0,0.5625,182.53656005859375,60.50398635864258
+26,code,32669,0.8939525462962963,0.984375,0.7272727272727273,110.61735534667969,0.9583333333333334,0.2569444444444444,145.07211303710938,31.27214241027832
+26,code,31925,0.8706597222222223,0.8125,0.6666666666666666,50.15843963623047,0.75,0.013888888888888888,71.23882293701172,0.8777934312820435
+26,code,1794,0.8521412037037037,0.8723958333333333,0.8571428571428571,59.8858757019043,0.7083333333333334,0.013888888888888888,82.30571746826172,0.625034749507904
+26,code,31735,0.8321759259259259,0.875,0.8571428571428571,65.27845001220703,0.6666666666666666,0.006944444444444444,156.09048461914062,0.355844646692276
+26,code,19342,0.831741898148148,0.765625,0.35294117647058826,71.23313903808594,0.875,0.2986111111111111,85.52899169921875,22.559274673461914
+26,code,9166,0.8305844907407407,0.92578125,0.8235294117647058,49.76114273071289,0.7916666666666666,0.1597222222222222,92.73416900634766,15.563563346862793
+26,code,1372,0.8284143518518517,0.75,0.6666666666666666,42.69673538208008,0.6666666666666666,0.013888888888888888,84.58007049560547,1.1792646646499634
+26,code,12514,0.8246527777777779,0.93359375,0.8235294117647058,59.348289489746094,0.7083333333333334,0.06944444444444445,88.60028839111328,7.541634559631348
+26,code,21251,0.8200231481481481,0.8645833333333333,0.8,67.42939758300781,0.6666666666666666,0.06944444444444445,84.74236297607422,4.058753490447998
+26,code,4293,0.8156828703703702,0.921875,0.8,64.2478256225586,0.6666666666666666,0.0625,66.562744140625,4.109451770782471
+26,code,5659,0.8140914351851853,0.9127604166666666,0.7058823529411765,62.55796813964844,0.7083333333333334,0.14583333333333334,57.125972747802734,8.837228775024414
+26,code,1317,0.8058449074074072,0.8203125,0.3,98.11906433105469,0.9166666666666666,0.3541666666666667,117.14395141601562,43.38441467285156
+26,code,31808,0.8051215277777778,0.8671875,0.8,67.26293182373047,0.625,0.034722222222222224,83.79071044921875,1.998589038848877
+26,code,27908,0.8045428240740742,0.7630208333333334,0.5,51.622711181640625,0.9583333333333334,0.3680555555555556,212.26068115234375,76.68626403808594
+26,code,12291,0.8026620370370371,0.7864583333333333,0.5714285714285714,57.6585578918457,0.625,0.027777777777777776,93.4870376586914,2.5813047885894775
+26,code,12545,0.7967303240740741,0.7877604166666667,0.6666666666666666,72.74190521240234,0.625,0.0625,114.68082427978516,4.437833786010742
+26,code,12060,0.7945601851851852,0.7369791666666666,0.47058823529411764,51.28270721435547,1.0,0.3263888888888889,130.81956481933594,62.29541015625
+26,factual_entities,10040,0.9399594907407408,0.9348958333333334,0.36363636363636365,154.7908477783203,1.0,0.3263888888888889,145.19114685058594,29.435495376586914
+26,factual_entities,26912,0.9363425925925926,0.8932291666666666,0.36363636363636365,227.83660888671875,1.0,0.7638888888888888,224.92942810058594,95.61489868164062
+26,factual_entities,18119,0.9260706018518517,0.9296875,0.8,78.7242202758789,0.875,0.04861111111111111,102.3759994506836,3.7813704013824463
+26,factual_entities,12531,0.8695023148148148,0.8984375,0.5714285714285714,111.86487579345703,1.0,0.4097222222222222,175.7102813720703,51.022701263427734
+26,factual_entities,16262,0.8608217592592592,0.828125,0.5882352941176471,105.21430969238281,0.7916666666666666,0.10416666666666667,122.01688385009766,10.22671890258789
+26,factual_entities,21579,0.8582175925925926,0.8098958333333333,0.3076923076923077,83.99641418457031,0.9583333333333334,0.4236111111111111,96.71053314208984,26.212890625
+26,factual_entities,8083,0.8547453703703705,0.8138020833333334,0.45454545454545453,62.21217346191406,0.875,0.2638888888888889,64.21637725830078,16.219947814941406
+26,factual_entities,21725,0.8508391203703703,0.9348958333333334,0.8571428571428571,47.854984283447266,0.7083333333333334,0.020833333333333332,69.83712005615234,0.9596091508865356
+26,factual_entities,19033,0.8415798611111112,0.9244791666666667,0.8,82.97052764892578,0.7083333333333334,0.0763888888888889,124.26692962646484,4.515802383422852
+26,factual_entities,91,0.8401331018518519,0.6510416666666666,0.3333333333333333,67.88789367675781,0.875,0.1875,126.0617904663086,31.73108673095703
+26,factual_entities,195,0.8318865740740741,0.9947916666666666,0.5,48.682640075683594,1.0,0.375,77.80021667480469,27.71076774597168
+26,factual_entities,24588,0.8291377314814816,0.92578125,0.8571428571428571,93.60688781738281,0.7083333333333334,0.11805555555555555,91.20294189453125,9.247859001159668
+26,factual_entities,8166,0.8276909722222223,0.83203125,0.4444444444444444,88.26390838623047,0.8333333333333334,0.1875,114.562744140625,25.464738845825195
+26,factual_entities,18510,0.8107638888888888,0.5729166666666666,0.15384615384615385,47.72831344604492,0.7083333333333334,0.1111111111111111,55.041412353515625,7.428190231323242
+26,factual_entities,16516,0.8077256944444443,0.7994791666666667,0.6666666666666666,50.57905197143555,0.6666666666666666,0.0625,55.3230094909668,4.57898473739624
+26,factual_entities,31986,0.8043981481481481,0.8333333333333334,0.5555555555555556,72.57457733154297,0.75,0.18055555555555555,74.71192169189453,15.172114372253418
+26,factual_entities,14124,0.7916666666666667,0.6145833333333333,0.36363636363636365,38.93451690673828,0.5833333333333334,0.0,77.12972259521484,0.0
+26,factual_entities,31709,0.7913773148148147,0.6119791666666666,0.21052631578947367,61.63446807861328,0.6666666666666666,0.1111111111111111,60.473934173583984,8.087822914123535
+26,factual_entities,9030,0.790943287037037,0.83203125,0.375,90.67755126953125,0.75,0.1875,154.79161071777344,34.385501861572266
+26,factual_entities,8792,0.7900752314814815,1.0,0.9411764705882353,18.29174041748047,0.625,0.0763888888888889,41.82126235961914,2.596844434738159
+26,german_language,3867,1.0,1.0,1.0,140.63876342773438,1.0,0.0,241.9287872314453,0.0
+26,german_language,9004,1.0,1.0,1.0,80.11203002929688,1.0,0.0,208.571044921875,0.0
+26,german_language,14570,1.0,1.0,1.0,90.3226547241211,1.0,0.0,362.637939453125,0.0
+26,german_language,14674,1.0,1.0,1.0,388.4827880859375,1.0,0.0,518.7395629882812,0.0
+26,german_language,15083,1.0,1.0,1.0,212.5703125,1.0,0.0,479.6780090332031,0.0
+26,german_language,21742,1.0,1.0,1.0,125.22502136230469,1.0,0.0,432.1290588378906,0.0
+26,german_language,24126,1.0,1.0,1.0,161.87603759765625,1.0,0.0,300.3412170410156,0.0
+26,german_language,24215,1.0,1.0,1.0,1138.048583984375,1.0,0.0,1706.63671875,0.0
+26,german_language,31199,1.0,1.0,1.0,107.52313232421875,1.0,0.0,306.7919616699219,0.0
+26,german_language,1202,0.9791666666666667,0.875,0.8571428571428571,71.60808563232422,0.9583333333333334,0.0,236.788330078125,0.0
+26,german_language,5695,0.9791666666666667,1.0,1.0,78.80027770996094,0.9583333333333334,0.0,280.9840087890625,0.0
+26,german_language,15237,0.9791666666666667,0.9375,0.9333333333333333,69.42606353759766,0.9583333333333334,0.0,188.7020263671875,0.0
+26,german_language,27241,0.9791666666666667,1.0,1.0,206.5436248779297,0.9583333333333334,0.0,326.7424011230469,0.0
+26,german_language,29501,0.9791666666666667,1.0,1.0,75.53385925292969,0.9583333333333334,0.0,250.5355987548828,0.0
+26,german_language,11117,0.9583333333333333,0.9375,0.9333333333333333,71.49526977539062,0.9166666666666666,0.0,196.91139221191406,0.0
+26,german_language,13061,0.9583333333333333,0.75,0.5454545454545454,95.58680725097656,0.9166666666666666,0.0,137.9283447265625,0.0
+26,german_language,1374,0.9375,0.8125,0.5454545454545454,107.17102813720703,0.875,0.0,183.4075927734375,0.0
+26,german_language,14329,0.9375,0.9375,0.9333333333333333,74.44577026367188,0.875,0.0,198.6432342529297,0.0
+26,german_language,15571,0.9375,0.8125,0.7692307692307693,55.85588836669922,0.875,0.0,168.88650512695312,0.0
+26,german_language,22703,0.9375,0.9375,0.8571428571428571,67.62046813964844,0.875,0.0,151.07644653320312,0.0
+26,mathematics,12177,0.9982638888888888,1.0,0.9333333333333333,94.46052551269531,1.0,0.04861111111111111,200.1036834716797,3.5435187816619873
+26,mathematics,12060,0.9664351851851853,0.9947916666666667,0.875,170.0095977783203,1.0,0.3263888888888889,267.20361328125,39.56473159790039
+26,mathematics,6322,0.9508101851851851,0.7981770833333334,0.6666666666666666,73.9338607788086,0.9166666666666666,0.0625,114.22003173828125,4.3299384117126465
+26,mathematics,15852,0.945457175925926,0.8736979166666667,0.5714285714285714,89.38529205322266,0.9583333333333334,0.2152777777777778,134.7430419921875,16.2455997467041
+26,mathematics,15970,0.9398148148148149,0.8307291666666666,0.5454545454545454,165.74949645996094,1.0,0.4652777777777778,165.90928649902344,40.24836730957031
+26,mathematics,23251,0.9291087962962963,0.859375,0.46153846153846156,90.23600769042969,1.0,0.375,176.0717010498047,35.11805725097656
+26,mathematics,2597,0.9137731481481483,0.8125,0.7692307692307693,70.46913146972656,0.8333333333333334,0.020833333333333332,189.78271484375,1.2875580787658691
+26,mathematics,27908,0.9129050925925926,0.8854166666666667,0.4444444444444444,270.29901123046875,1.0,0.3611111111111111,268.9342346191406,67.24067687988281
+26,mathematics,1317,0.8958333333333334,0.8333333333333334,0.35294117647058826,121.56973266601562,0.9583333333333334,0.3472222222222222,147.52865600585938,38.32029724121094
+26,mathematics,30744,0.8935185185185185,0.875,0.8571428571428571,65.48336791992188,0.7916666666666666,0.013888888888888888,196.19061279296875,1.450549602508545
+26,mathematics,31772,0.8825231481481481,0.9244791666666667,0.5714285714285714,130.07550048828125,0.875,0.3472222222222222,141.20774841308594,31.533016204833984
+26,mathematics,18334,0.875,0.8125,0.7692307692307693,47.533607482910156,0.75,0.0,63.3764762878418,0.0
+26,mathematics,30813,0.8735532407407408,0.875,0.7692307692307693,52.90532302856445,0.75,0.006944444444444444,62.1231689453125,0.41026294231414795
+26,mathematics,6445,0.8696469907407409,0.8072916666666665,0.36363636363636365,165.98394775390625,0.875,0.3958333333333333,167.7504425048828,40.1645622253418
+26,mathematics,10171,0.8625578703703705,0.6119791666666666,0.4,81.46713256835938,0.8333333333333334,0.2013888888888889,228.4755401611328,22.62830352783203
+26,mathematics,22280,0.857638888888889,0.9283854166666666,0.875,61.92787170410156,0.8333333333333334,0.1527777777777778,107.77066802978516,16.494943618774414
+26,mathematics,12373,0.8567708333333334,0.8125,0.5454545454545454,71.47801208496094,0.75,0.06944444444444445,69.85230255126953,4.558965682983398
+26,mathematics,14618,0.8551793981481481,0.71875,0.5333333333333333,45.65156936645508,0.7916666666666666,0.10416666666666667,70.27909088134766,7.686792850494385
+26,mathematics,11347,0.8519965277777779,0.875,0.7692307692307693,80.37939453125,0.7083333333333334,0.006944444444444444,116.17533111572266,0.7380465865135193
+26,mathematics,1433,0.8482349537037037,0.6875,0.5454545454545454,41.49335479736328,0.7083333333333334,0.020833333333333332,71.08837127685547,1.3856459856033325
+26,negative_sentiment,11290,0.9301215277777778,0.9375,0.9333333333333333,45.149322509765625,0.875,0.034722222222222224,75.23931121826172,1.908559799194336
+26,negative_sentiment,27837,0.9166666666666667,0.8125,0.7692307692307693,49.472747802734375,0.8333333333333334,0.0,109.9905014038086,0.0
+26,negative_sentiment,846,0.8758680555555556,0.8255208333333334,0.5454545454545454,72.13957977294922,0.9166666666666666,0.2013888888888889,108.64054107666016,21.230697631835938
+26,negative_sentiment,9366,0.8706597222222223,0.875,0.8571428571428571,46.901817321777344,0.75,0.006944444444444444,82.87060546875,0.7733787894248962
+26,negative_sentiment,16162,0.8495370370370372,0.7330729166666666,0.13333333333333333,169.25070190429688,0.9583333333333334,0.4583333333333333,145.64979553222656,50.01118087768555
+26,negative_sentiment,2388,0.8467881944444445,0.8932291666666666,0.6666666666666666,57.94428634643555,0.75,0.0763888888888889,65.25790405273438,5.486703872680664
+26,negative_sentiment,23960,0.833767361111111,0.71484375,0.3076923076923077,99.47303771972656,0.7916666666666666,0.1875,89.1791000366211,15.552165985107422
+26,negative_sentiment,26512,0.8313078703703703,0.9895833333333333,0.8421052631578947,51.16231918334961,0.75,0.09722222222222222,69.3716049194336,8.323831558227539
+26,negative_sentiment,26266,0.8291377314814815,0.8489583333333334,0.48484848484848486,49.64305877685547,0.9583333333333334,0.3194444444444444,69.05467987060547,22.382083892822266
+26,negative_sentiment,4494,0.814670138888889,0.6119791666666666,0.23529411764705882,72.14042663574219,0.75,0.16666666666666666,66.25566864013672,11.826155662536621
+26,negative_sentiment,31950,0.8129340277777778,0.6770833333333334,0.4444444444444444,73.85538482666016,0.75,0.1597222222222222,83.8884048461914,15.258397102355957
+26,negative_sentiment,16111,0.8036747685185186,0.7552083333333333,0.4,57.65766143798828,0.7083333333333334,0.13194444444444445,66.58548736572266,10.105219841003418
+26,negative_sentiment,25776,0.7957175925925927,0.75,0.5,58.70509338378906,0.75,0.2013888888888889,172.7453155517578,35.71640396118164
+26,negative_sentiment,25339,0.7919560185185185,0.7252604166666667,0.47058823529411764,112.58674621582031,0.75,0.24305555555555555,102.8864974975586,25.3869686126709
+26,negative_sentiment,6547,0.7905092592592593,0.8346354166666666,0.5833333333333334,51.45130157470703,0.75,0.1597222222222222,92.03385162353516,20.422983169555664
+26,negative_sentiment,17195,0.7892071759259259,0.7109375,0.5,62.487403869628906,0.625,0.0625,69.87531280517578,5.657628059387207
+26,negative_sentiment,18698,0.7795138888888891,0.7395833333333333,0.29411764705882354,221.51658630371094,1.0,1.0,255.27447509765625,192.82337951660156
+26,negative_sentiment,24111,0.7771990740740742,0.7786458333333333,0.5,53.29218292236328,0.5833333333333334,0.034722222222222224,118.6685562133789,5.000449180603027
+26,negative_sentiment,20342,0.7615740740740741,0.6927083333333334,0.4117647058823529,80.87789916992188,1.0,0.4930555555555556,115.57364654541016,56.446380615234375
+26,negative_sentiment,18576,0.7589699074074074,0.8046875,0.42857142857142855,311.21954345703125,1.0,0.9861111111111112,347.6768493652344,247.3077850341797
+26,positive_sentiment,10732,0.9019097222222223,0.8359374999999999,0.5333333333333333,63.13585662841797,0.8333333333333334,0.0625,90.3763656616211,4.591992378234863
+26,positive_sentiment,6547,0.8940972222222222,0.86328125,0.5833333333333334,50.36292266845703,0.9166666666666666,0.13194444444444445,116.19145965576172,16.396717071533203
+26,positive_sentiment,23218,0.8859953703703705,0.7604166666666666,0.6153846153846154,104.5474853515625,0.9166666666666666,0.25,134.16522216796875,22.78602409362793
+26,positive_sentiment,18418,0.8761574074074074,0.9583333333333334,0.625,72.0313720703125,0.8333333333333334,0.1111111111111111,177.0322723388672,14.553756713867188
+26,positive_sentiment,9424,0.8731192129629629,0.9739583333333333,0.4,212.58468627929688,0.7916666666666666,0.11805555555555555,221.1200714111328,13.589634895324707
+26,positive_sentiment,846,0.8324652777777778,0.7161458333333333,0.47619047619047616,76.96398162841797,0.8333333333333334,0.2152777777777778,101.4962387084961,22.421415328979492
+26,positive_sentiment,25776,0.8313078703703705,0.7473958333333334,0.5,58.389404296875,0.8333333333333334,0.1875,175.842041015625,35.20028305053711
+26,positive_sentiment,25899,0.8301504629629629,0.7109375,0.4444444444444444,50.834571838378906,0.75,0.09027777777777778,58.40364074707031,7.127217769622803
+26,positive_sentiment,29445,0.8275462962962964,0.7890624999999999,0.5217391304347826,49.27320098876953,0.8333333333333334,0.22916666666666666,100.54706573486328,22.42933464050293
+26,positive_sentiment,21211,0.820167824074074,0.6875,0.5454545454545454,63.10835266113281,0.6666666666666666,0.020833333333333332,112.32808685302734,4.830132961273193
+26,positive_sentiment,12813,0.8055555555555556,0.6458333333333333,0.3333333333333333,81.99803161621094,0.6666666666666666,0.0763888888888889,73.3226318359375,6.606205940246582
+26,positive_sentiment,4567,0.8041087962962963,0.6041666666666667,0.46153846153846156,479.1512756347656,1.0,1.0,447.2738342285156,321.57293701171875
+26,positive_sentiment,16155,0.7981770833333334,0.9713541666666666,0.7368421052631579,53.281436920166016,0.625,0.04861111111111111,56.82499313354492,3.1014559268951416
+26,positive_sentiment,8456,0.7974537037037037,0.7786458333333333,0.48484848484848486,158.83746337890625,1.0,0.8680555555555556,207.408935546875,132.67813110351562
+26,positive_sentiment,16162,0.7928240740740742,0.73828125,0.3448275862068966,82.54803466796875,0.9583333333333334,0.4583333333333333,125.3106460571289,53.401039123535156
+26,positive_sentiment,4740,0.7876157407407407,0.625,0.36363636363636365,62.5802001953125,0.7916666666666666,0.2361111111111111,149.4482879638672,42.14777374267578
+26,positive_sentiment,31950,0.787181712962963,0.75,0.47619047619047616,48.506431579589844,0.75,0.1597222222222222,76.3582534790039,16.51342010498047
+26,positive_sentiment,4484,0.7867476851851851,0.7786458333333333,0.47619047619047616,88.47461700439453,0.875,0.375,102.09386444091797,37.61430358886719
+26,positive_sentiment,12279,0.781539351851852,0.6875,0.5454545454545454,57.916236877441406,0.5833333333333334,0.027777777777777776,105.98931884765625,2.305490732192993
+26,positive_sentiment,28188,0.7769097222222221,0.8854166666666667,0.5882352941176471,50.913108825683594,0.6666666666666666,0.11805555555555555,77.11820220947266,12.90743637084961
+26,uncertainty,8593,0.9555844907407407,1.0,1.0,79.82640838623047,0.9583333333333334,0.09027777777777778,153.00975036621094,9.227836608886719
+26,uncertainty,9462,0.9126157407407409,1.0,1.0,55.270538330078125,0.8333333333333334,0.013888888888888888,120.59039306640625,1.4664264917373657
+26,uncertainty,20005,0.8969907407407407,0.921875,0.6,117.18822479248047,1.0,0.7083333333333334,143.3247528076172,66.40074157714844
+26,uncertainty,8456,0.8874421296296297,0.8723958333333334,0.47058823529411764,211.76339721679688,1.0,0.8680555555555556,221.8867950439453,130.26513671875
+26,uncertainty,23402,0.8816550925925926,0.8567708333333333,0.5454545454545454,124.30525207519531,1.0,0.5069444444444444,143.35333251953125,52.10331726074219
+26,uncertainty,27112,0.8634259259259259,0.8880208333333333,0.5454545454545454,182.348876953125,0.875,0.2777777777777778,160.11505126953125,35.064720153808594
+26,uncertainty,32366,0.8548900462962963,0.8385416666666667,0.6153846153846154,96.16805267333984,0.875,0.24305555555555555,90.75826263427734,17.90591812133789
+26,uncertainty,15304,0.8491030092592593,0.8033854166666666,0.7142857142857143,43.88987350463867,0.7083333333333334,0.020833333333333332,61.01127243041992,1.1108851432800293
+26,uncertainty,4433,0.8246527777777778,0.53515625,0.18181818181818182,49.75330352783203,0.75,0.125,59.82192611694336,8.68009090423584
+26,uncertainty,31423,0.8151041666666666,0.8567708333333334,0.5,68.30633544921875,0.6666666666666666,0.0763888888888889,87.53912353515625,5.8641557693481445
+26,uncertainty,26374,0.8125,0.9375,0.9333333333333333,37.270408630371094,0.625,0.0,42.3619384765625,0.0
+26,uncertainty,12336,0.8113425925925926,0.7942708333333334,0.5454545454545454,41.43083190917969,0.75,0.14583333333333334,66.09442138671875,10.841641426086426
+26,uncertainty,32767,0.8101851851851851,0.6770833333333334,0.375,72.65742492675781,0.7916666666666666,0.19444444444444445,73.94112396240234,16.328289031982422
+26,uncertainty,10105,0.8041087962962963,0.90625,0.56,188.177978515625,1.0,0.7083333333333334,192.47386169433594,104.2887191772461
+26,uncertainty,28460,0.8032407407407408,0.625,0.4,49.72218322753906,0.625,0.027777777777777776,59.72444534301758,1.6394777297973633
+26,uncertainty,30823,0.8028067129629629,0.8697916666666667,0.8,40.2286491394043,0.625,0.020833333333333332,68.2996826171875,1.9722380638122559
+26,uncertainty,631,0.8006365740740741,0.7421875,0.23076923076923078,68.62223815917969,0.7916666666666666,0.3888888888888889,103.84976959228516,31.15761375427246
+26,uncertainty,8386,0.7967303240740741,1.0,0.9333333333333333,55.902732849121094,0.625,0.04861111111111111,50.79805374145508,2.974226951599121
+26,uncertainty,13798,0.7923900462962963,0.8671875,0.0,91.24398040771484,0.7916666666666666,0.24305555555555555,102.57077026367188,27.355384826660156
+26,uncertainty,24214,0.7916666666666667,0.625,0.4,38.75273132324219,0.5833333333333334,0.0,55.41322326660156,0.0

artifacts/layer_metrics.csv

@@ -0,0 +1,4 @@
+layer,linear_probe_macro_auroc,linear_probe_macro_f1,reconstruction_cosine,reconstruction_nmse,mean_active_features
+4,0.9947916666666667,0.9474089635854341,0.9660451127482312,0.06670970512953188,50.0
+14,1.0,0.9820728291316527,0.9274159478289741,0.1398810788856021,50.0
+26,1.0,0.9820728291316527,0.9647476846086127,0.06911552319901862,50.0

artifacts/report.md

@@ -0,0 +1,68 @@
+# FeatureLens experiment report
+
+## Research question
+
+**Do sparse features that predict a concept also causally influence model behaviour?**
+
+## Executive summary
+
+Selected SAE features averaged 0.962 held-out AUROC. Max-active interventions covered 82.1% of causal tasks and changed mean log p/token by 0.041 in absolute value on average versus 0.018 for norm-matched random controls (2.33×).
+
+Max-active interventions produced larger task-level target effects than norm-matched random controls with paired uncertainty excluding zero. Predictive SAE features therefore show causal specificity when intervened where the selected feature is actually represented, while the final-token baseline quantifies sensitivity to intervention location. Moving from the final prompt token to the feature's maximum-activation token increased intervention coverage from 28.6% to 82.1%, showing that causal conclusions depend materially on where the representation is tested.
+
+## Key measurements
+
+- Median selected-feature held-out AUROC: 0.987; mean AUROC 95% bootstrap CI [0.927, 0.994].
+- Best residual linear-probe layer: 14 with macro AUROC 1.000.
+- Mean paraphrase TopK Jaccard: 0.326; sparse activation cosine: 0.985.
+- Feature coverage: final-token policy 28.6%; active anywhere in prompt 82.1%; max-active intervention 82.1%.
+- Final-token task-level SAE/random ratio: 1.52×; paired advantage +0.0026, 95% CI [-0.0005, +0.0063], sign-flip p=0.1719.
+- Max-active task-level SAE/random ratio: 2.33×; paired advantage +0.0237, 95% CI [+0.0067, +0.0469], sign-flip p=0.0001.
+- Conditional on feature-active tasks, max-active SAE/random ratio: 2.33× (n=23).
+- Final-token top-5 joint ablation SAE/random ratio: 1.09×; paired advantage +0.0009, 95% CI [-0.0035, +0.0065], sign-flip p=0.7891.
+- Across seven concepts, held-out AUROC vs max-active target specificity Spearman ρ=-0.185; descriptive only.
+- Held-out AUROC vs max-active JS specificity Spearman ρ=-0.148; descriptive only.
+
+## Experimental design
+
+- Model: Qwen3-1.7B-Base.
+- SAEs: Qwen-Scope residual-stream TopK SAEs at configured early/middle/late layers.
+- Discovery evidence: prompt-wide maximum SAE activation across non-padding tokens; final-token activations are saved separately.
+- Split discipline: paraphrase groups remain entirely in train or held-out test.
+- Feature selection: training-split AUROC plus activation contrast; held-out AUROC/F1 are reported separately.
+- Causal position policies: final prompt token and maximum selected-feature activation within the prompt. Max-active positions are selected from SAE activation only, never from behavioral outcomes.
+- Primary causal statistical unit: causal task. Ablation and 2× amplification are averaged within task before paired bootstrap/sign-flip inference.
+- Negative control: deterministic norm-matched random residual directions.
+- Primary target metric: exact full continuation mean log probability per token under teacher forcing.
+- Coverage and conditional-on-active effect strength are reported separately.
+- Feature-set analysis remains a final-token diagnostic and is not conflated with the max-active single-feature study.
+
+## Figures
+
+![Feature AUROC](figures/feature_auroc.png)
+
+![Layer diagnostics](figures/layer_diagnostics.png)
+
+![Causal position sensitivity](figures/causal_position_sensitivity.png)
+
+![Max-active causal effects](figures/causal_effects.png)
+
+![Feature-set diagnostic](figures/feature_set_effects.png)
+
+![Association vs causality](figures/association_vs_causality.png)
+
+## Position sensitivity
+
+The final-token policy asks whether the selected feature matters at the conventional last-prompt-token intervention site. The max-active policy asks whether it matters where that same feature is most strongly represented in the prompt. Reporting both prevents low final-token coverage from being mistaken for evidence that a predictive feature is globally non-causal.
+
+## Association vs causality across concepts
+
+Cross-concept correlations use max-active random-normalized specificity and are descriptive because the study has seven controlled concepts.
+
+## Interpretation guardrails
+
+High held-out AUROC is correlational evidence. Causal claims require downstream changes relative to norm-matched random controls. Max-active positions are chosen without reference to behavioral effect size. Task-level uncertainty treats ablation and amplification on the same causal prompt as repeated interventions, not independent experimental units.
+
+## Reproducibility
+
+Run `python -m experiments.run_all --resume` for a fresh full study. The causal-addendum notebook is retained as a migration utility for an already-completed final-token baseline.

artifacts/split.json

@@ -0,0 +1,233 @@
+{
+  "seed": 42,
+  "train_indices": [
+    0,
+    1,
+    2,
+    3,
+    4,
+    5,
+    6,
+    7,
+    8,
+    9,
+    16,
+    17,
+    20,
+    21,
+    22,
+    23,
+    24,
+    25,
+    26,
+    27,
+    28,
+    29,
+    30,
+    31,
+    32,
+    33,
+    34,
+    35,
+    38,
+    39,
+    40,
+    41,
+    42,
+    43,
+    44,
+    45,
+    48,
+    49,
+    50,
+    51,
+    54,
+    55,
+    58,
+    59,
+    60,
+    61,
+    62,
+    63,
+    64,
+    65,
+    66,
+    67,
+    68,
+    69,
+    70,
+    71,
+    72,
+    73,
+    74,
+    75,
+    76,
+    77,
+    78,
+    79,
+    80,
+    81,
+    84,
+    85,
+    88,
+    89,
+    94,
+    95,
+    98,
+    99,
+    100,
+    101,
+    104,
+    105,
+    106,
+    107,
+    108,
+    109,
+    110,
+    111,
+    112,
+    113,
+    114,
+    115,
+    120,
+    121,
+    122,
+    123,
+    124,
+    125,
+    126,
+    127,
+    130,
+    131,
+    132,
+    133,
+    134,
+    135,
+    136,
+    137,
+    138,
+    139,
+    140,
+    141,
+    144,
+    145,
+    146,
+    147,
+    148,
+    149,
+    154,
+    155,
+    156,
+    157,
+    158,
+    159,
+    160,
+    161,
+    164,
+    165,
+    166,
+    167,
+    168,
+    169,
+    172,
+    173,
+    174,
+    175,
+    176,
+    177,
+    178,
+    179,
+    180,
+    181,
+    182,
+    183,
+    188,
+    189,
+    190,
+    191,
+    194,
+    195,
+    198,
+    199,
+    202,
+    203,
+    204,
+    205,
+    206,
+    207,
+    208,
+    209,
+    210,
+    211,
+    212,
+    213,
+    214,
+    215,
+    216,
+    217,
+    218,
+    219,
+    220,
+    221
+  ],
+  "test_indices": [
+    10,
+    11,
+    12,
+    13,
+    14,
+    15,
+    18,
+    19,
+    36,
+    37,
+    46,
+    47,
+    52,
+    53,
+    56,
+    57,
+    82,
+    83,
+    86,
+    87,
+    90,
+    91,
+    92,
+    93,
+    96,
+    97,
+    102,
+    103,
+    116,
+    117,
+    118,
+    119,
+    128,
+    129,
+    142,
+    143,
+    150,
+    151,
+    152,
+    153,
+    162,
+    163,
+    170,
+    171,
+    184,
+    185,
+    186,
+    187,
+    192,
+    193,
+    196,
+    197,
+    200,
+    201,
+    222,
+    223
+  ],
+  "n_train": 168,
+  "n_test": 56
+}

artifacts/stability.csv

@@ -0,0 +1,337 @@
+layer,pair_id,concept,topk_jaccard,sparse_cosine
+4,code-00,code,0.4311475409836066,0.989191025028846
+4,code-01,code,0.3333333333333333,0.9999706394577682
+4,code-02,code,0.3287101248266297,0.9999635750292034
+4,code-03,code,0.28618968386023297,0.9999289079594824
+4,code-04,code,0.3658940397350993,0.9999720449449652
+4,code-05,code,0.2627388535031847,0.989106801084924
+4,code-06,code,0.29069767441860467,0.9999657300740955
+4,code-07,code,0.2708018154311649,0.9999561449289407
+4,code-08,code,0.2876190476190476,0.9891805040370574
+4,code-09,code,0.3390151515151515,0.9891333089857143
+4,code-10,code,0.4562647754137116,0.9999829391813584
+4,code-11,code,0.24319727891156462,0.9999371200328028
+4,code-12,code,0.3829787234042553,0.9999690938720655
+4,code-13,code,0.2902684563758389,0.9999457394945113
+4,code-14,code,0.25418060200668896,0.9890976404264782
+4,code-15,code,0.3442622950819672,0.999970367957558
+4,mathematics-00,mathematics,0.3594306049822064,0.9999696756100469
+4,mathematics-01,mathematics,0.334525939177102,0.9999583157360491
+4,mathematics-02,mathematics,0.2682445759368836,0.9999531166599476
+4,mathematics-03,mathematics,0.3053173241852487,0.9891304810090837
+4,mathematics-04,mathematics,0.4753289473684211,0.9999547029793051
+4,mathematics-05,mathematics,0.25142857142857145,0.9999503923484743
+4,mathematics-06,mathematics,0.3552,0.9891379828381566
+4,mathematics-07,mathematics,0.2938388625592417,0.9999617310674382
+4,mathematics-08,mathematics,0.3844282238442822,0.9891804349544243
+4,mathematics-09,mathematics,0.33500837520938026,0.9891417437117388
+4,mathematics-10,mathematics,0.29895104895104896,0.999955314747787
+4,mathematics-11,mathematics,0.44045911047345765,0.9999461860838762
+4,mathematics-12,mathematics,0.3477157360406091,0.9999720117243538
+4,mathematics-13,mathematics,0.40033500837520936,0.9999719860670049
+4,mathematics-14,mathematics,0.2379182156133829,0.9999497035295289
+4,mathematics-15,mathematics,0.3471223021582734,0.9999636903840216
+4,positive_sentiment-00,positive_sentiment,0.39679715302491103,0.9999649629447144
+4,positive_sentiment-01,positive_sentiment,0.3355263157894737,0.9999594545377013
+4,positive_sentiment-02,positive_sentiment,0.3393124065769806,0.9999681538783055
+4,positive_sentiment-03,positive_sentiment,0.27175843694493784,0.9999653740538117
+4,positive_sentiment-04,positive_sentiment,0.24306688417618272,0.9999671905595988
+4,positive_sentiment-05,positive_sentiment,0.3155963302752294,0.999965631254137
+4,positive_sentiment-06,positive_sentiment,0.22126929674099485,0.9999498781408013
+4,positive_sentiment-07,positive_sentiment,0.2998324958123953,0.9999634163985037
+4,positive_sentiment-08,positive_sentiment,0.273224043715847,0.9999658642690357
+4,positive_sentiment-09,positive_sentiment,0.3632887189292543,0.9999649741206514
+4,positive_sentiment-10,positive_sentiment,0.30357142857142855,0.9999585173995168
+4,positive_sentiment-11,positive_sentiment,0.3408304498269896,0.9999616913734006
+4,positive_sentiment-12,positive_sentiment,0.2665441176470588,0.9999630427209005
+4,positive_sentiment-13,positive_sentiment,0.24573863636363635,0.9999340000892353
+4,positive_sentiment-14,positive_sentiment,0.3091684434968017,0.9999706534301752
+4,positive_sentiment-15,positive_sentiment,0.3424657534246575,0.9999696015503636
+4,negative_sentiment-00,negative_sentiment,0.3220338983050847,0.9999589978751574
+4,negative_sentiment-01,negative_sentiment,0.3172302737520129,0.9999678430697062
+4,negative_sentiment-02,negative_sentiment,0.3,0.9999607474942863
+4,negative_sentiment-03,negative_sentiment,0.24880382775119617,0.999953101074863
+4,negative_sentiment-04,negative_sentiment,0.22121212121212122,0.9999674274662975
+4,negative_sentiment-05,negative_sentiment,0.3078231292517007,0.9999631468758489
+4,negative_sentiment-06,negative_sentiment,0.2230769230769231,0.9999507694501811
+4,negative_sentiment-07,negative_sentiment,0.27986348122866894,0.9999655194984687
+4,negative_sentiment-08,negative_sentiment,0.2538167938931298,0.9999642162487423
+4,negative_sentiment-09,negative_sentiment,0.32727272727272727,0.9999635811449571
+4,negative_sentiment-10,negative_sentiment,0.29441624365482233,0.9999619457292618
+4,negative_sentiment-11,negative_sentiment,0.3067993366500829,0.9999603632072415
+4,negative_sentiment-12,negative_sentiment,0.2662337662337662,0.9999582432947234
+4,negative_sentiment-13,negative_sentiment,0.22046109510086456,0.9999373496328824
+4,negative_sentiment-14,negative_sentiment,0.2768670309653916,0.9999633857109651
+4,negative_sentiment-15,negative_sentiment,0.33840304182509506,0.9999720220039464
+4,german_language-00,german_language,0.47835497835497837,0.9999768499858412
+4,german_language-01,german_language,0.494949494949495,0.9999775028546921
+4,german_language-02,german_language,0.5223880597014925,0.9999746894218088
+4,german_language-03,german_language,0.37221269296740994,0.9999717933467758
+4,german_language-04,german_language,0.44036697247706424,0.9999789514550391
+4,german_language-05,german_language,0.37080536912751677,0.9999636532946664
+4,german_language-06,german_language,0.33444259567387685,0.9999715419688158
+4,german_language-07,german_language,0.3547854785478548,0.9999636679873475
+4,german_language-08,german_language,0.4104938271604938,0.9999584306859849
+4,german_language-09,german_language,0.4042145593869732,0.9999736724923872
+4,german_language-10,german_language,0.34543010752688175,0.9999639714035083
+4,german_language-11,german_language,0.31869918699186994,0.9999676082845792
+4,german_language-12,german_language,0.36363636363636365,0.9999543510774616
+4,german_language-13,german_language,0.4855195911413969,0.9999813024944768
+4,german_language-14,german_language,0.35947712418300654,0.9999730688481495
+4,german_language-15,german_language,0.3275862068965517,0.9999679617123177
+4,factual_entities-00,factual_entities,0.40033783783783783,0.9891234847794932
+4,factual_entities-01,factual_entities,0.24953445065176907,0.9891354581251652
+4,factual_entities-02,factual_entities,0.37056737588652483,0.9892082448660446
+4,factual_entities-03,factual_entities,0.4030131826741996,0.9891564220221707
+4,factual_entities-04,factual_entities,0.3491271820448878,0.9891486453476434
+4,factual_entities-05,factual_entities,0.3562005277044855,0.9999763092561826
+4,factual_entities-06,factual_entities,0.4152173913043478,0.9891467264822525
+4,factual_entities-07,factual_entities,0.24648985959438377,0.9891096604611559
+4,factual_entities-08,factual_entities,0.3412322274881517,0.9891341294016328
+4,factual_entities-09,factual_entities,0.4583333333333333,0.9892111243854191
+4,factual_entities-10,factual_entities,0.35655737704918034,0.9891989784582211
+4,factual_entities-11,factual_entities,0.29130434782608694,0.9999606096504317
+4,factual_entities-12,factual_entities,0.3310344827586207,0.9999722530295203
+4,factual_entities-13,factual_entities,0.29558541266794625,0.9891540984325061
+4,factual_entities-14,factual_entities,0.29791666666666666,0.9891414216062852
+4,factual_entities-15,factual_entities,0.2766798418972332,0.9891485980648569
+4,uncertainty-00,uncertainty,0.22816166883963493,0.989169388682598
+4,uncertainty-01,uncertainty,0.2611570247933884,0.9891631037604764
+4,uncertainty-02,uncertainty,0.2668500687757909,0.9999509759051717
+4,uncertainty-03,uncertainty,0.2398190045248869,0.9999549516223171
+4,uncertainty-04,uncertainty,0.27181208053691275,0.9999750120312442
+4,uncertainty-05,uncertainty,0.2898134863701578,0.9999575639093725
+4,uncertainty-06,uncertainty,0.2696774193548387,0.9891554579607917
+4,uncertainty-07,uncertainty,0.2669616519174041,0.999954364063221
+4,uncertainty-08,uncertainty,0.29701230228471004,0.999967431112174
+4,uncertainty-09,uncertainty,0.3059490084985836,0.9999565867632838
+4,uncertainty-10,uncertainty,0.23876404494382023,0.9999635876288635
+4,uncertainty-11,uncertainty,0.2342606149341142,0.9999453945964216
+4,uncertainty-12,uncertainty,0.2434402332361516,0.9999654015154136
+4,uncertainty-13,uncertainty,0.24444444444444444,0.9999436084098076
+4,uncertainty-14,uncertainty,0.25925925925925924,0.9999588312900347
+4,uncertainty-15,uncertainty,0.28895612708018154,0.9999576236102989
+14,code-00,code,0.4172297297297297,0.9902000431860755
+14,code-01,code,0.3474088291746641,0.9806422719437736
+14,code-02,code,0.3620689655172414,0.9901729754269252
+14,code-03,code,0.2862129144851658,0.9900506419625426
+14,code-04,code,0.4181184668989547,0.9901827041848217
+14,code-05,code,0.21428571428571427,0.9805412550169413
+14,code-06,code,0.26452905811623245,0.9901899267793673
+14,code-07,code,0.2662440570522979,0.9900904234395915
+14,code-08,code,0.2631578947368421,0.9806513389862204
+14,code-09,code,0.3654223968565815,0.9902235848154112
+14,code-10,code,0.38588235294117645,0.9902845407077342
+14,code-11,code,0.24862888482632542,0.9900895533513584
+14,code-12,code,0.37749546279491836,0.99021648969533
+14,code-13,code,0.30570902394106814,0.9805743099112672
+14,code-14,code,0.26545454545454544,0.9805796094767558
+14,code-15,code,0.30446927374301674,0.9901314726250424
+14,mathematics-00,mathematics,0.4037735849056604,0.9998620653471765
+14,mathematics-01,mathematics,0.29322033898305083,0.9805608221817694
+14,mathematics-02,mathematics,0.2816326530612245,0.9805879458930802
+14,mathematics-03,mathematics,0.2984014209591474,0.9901102168978202
+14,mathematics-04,mathematics,0.4722222222222222,0.9902378698717491
+14,mathematics-05,mathematics,0.2379110251450677,0.9901279717918522
+14,mathematics-06,mathematics,0.38305084745762713,0.9901831166385997
+14,mathematics-07,mathematics,0.31695721077654515,0.9805469878229243
+14,mathematics-08,mathematics,0.4219948849104859,0.9902668620157249
+14,mathematics-09,mathematics,0.28852459016393445,0.9901938955712613
+14,mathematics-10,mathematics,0.28825622775800713,0.9805944582421596
+14,mathematics-11,mathematics,0.42496493688639553,0.9901730306978109
+14,mathematics-12,mathematics,0.32707774798927614,0.9806585253850566
+14,mathematics-13,mathematics,0.3409436834094368,0.9806555279222158
+14,mathematics-14,mathematics,0.2768361581920904,0.9902082192357923
+14,mathematics-15,mathematics,0.2637931034482759,0.9901547815473986
+14,positive_sentiment-00,positive_sentiment,0.4128787878787879,0.980665774811277
+14,positive_sentiment-01,positive_sentiment,0.3700361010830325,0.9806541135512301
+14,positive_sentiment-02,positive_sentiment,0.3497615262321145,0.9901682690305461
+14,positive_sentiment-03,positive_sentiment,0.258,0.980623718171293
+14,positive_sentiment-04,positive_sentiment,0.2732342007434944,0.9902089729683708
+14,positive_sentiment-05,positive_sentiment,0.3110236220472441,0.990227960362101
+14,positive_sentiment-06,positive_sentiment,0.2995689655172414,0.9805854074276764
+14,positive_sentiment-07,positive_sentiment,0.2698412698412698,0.9805780287169028
+14,positive_sentiment-08,positive_sentiment,0.2808349146110057,0.9806175852010832
+14,positive_sentiment-09,positive_sentiment,0.3058350100603622,0.9806634332406368
+14,positive_sentiment-10,positive_sentiment,0.33620689655172414,0.9806127557037527
+14,positive_sentiment-11,positive_sentiment,0.3387096774193548,0.9806361258967989
+14,positive_sentiment-12,positive_sentiment,0.2885375494071146,0.9806294247953098
+14,positive_sentiment-13,positive_sentiment,0.3194444444444444,0.9805612233118675
+14,positive_sentiment-14,positive_sentiment,0.3204819277108434,0.9806791223945865
+14,positive_sentiment-15,positive_sentiment,0.37861915367483295,0.999793450468426
+14,negative_sentiment-00,negative_sentiment,0.3587140439932318,0.9806130941537451
+14,negative_sentiment-01,negative_sentiment,0.3373702422145329,0.9806539090868044
+14,negative_sentiment-02,negative_sentiment,0.31281198003327787,0.9901468065433824
+14,negative_sentiment-03,negative_sentiment,0.2517857142857143,0.9805859970399724
+14,negative_sentiment-04,negative_sentiment,0.29704797047970477,0.9902048446545745
+14,negative_sentiment-05,negative_sentiment,0.30654205607476637,0.9901855549998978
+14,negative_sentiment-06,negative_sentiment,0.22462203023758098,0.9806300963147097
+14,negative_sentiment-07,negative_sentiment,0.25215889464594127,0.980577585009978
+14,negative_sentiment-08,negative_sentiment,0.21428571428571427,0.9805681627807906
+14,negative_sentiment-09,negative_sentiment,0.3006872852233677,0.9806667721287463
+14,negative_sentiment-10,negative_sentiment,0.312,0.980644143557275
+14,negative_sentiment-11,negative_sentiment,0.3181818181818182,0.9806370311987312
+14,negative_sentiment-12,negative_sentiment,0.2851985559566787,0.9805972635245791
+14,negative_sentiment-13,negative_sentiment,0.2979452054794521,0.9805750584623124
+14,negative_sentiment-14,negative_sentiment,0.29292929292929293,0.9806425603992663
+14,negative_sentiment-15,negative_sentiment,0.3619246861924686,0.9998029478583351
+14,german_language-00,german_language,0.5534246575342465,0.9807734988793569
+14,german_language-01,german_language,0.49206349206349204,0.9903418730912793
+14,german_language-02,german_language,0.5204582651391162,0.9998739646104144
+14,german_language-03,german_language,0.34563106796116505,0.9902677654264136
+14,german_language-04,german_language,0.4588477366255144,0.9807533856184706
+14,german_language-05,german_language,0.40239043824701193,0.9807814057431469
+14,german_language-06,german_language,0.2857142857142857,0.9902601313586692
+14,german_language-07,german_language,0.3766990291262136,0.9806801886466997
+14,german_language-08,german_language,0.3547297297297297,0.9902330947981313
+14,german_language-09,german_language,0.3987341772151899,0.9807050418782072
+14,german_language-10,german_language,0.426517571884984,0.990231542110508
+14,german_language-11,german_language,0.3634651600753296,0.9902341766192039
+14,german_language-12,german_language,0.3953033268101761,0.9902953105558695
+14,german_language-13,german_language,0.46954813359528486,0.9999063666522116
+14,german_language-14,german_language,0.4085603112840467,0.9998762567864448
+14,german_language-15,german_language,0.35650224215246634,0.9902955997167924
+14,factual_entities-00,factual_entities,0.35714285714285715,0.9902443350843326
+14,factual_entities-01,factual_entities,0.2585858585858586,0.9901609366969406
+14,factual_entities-02,factual_entities,0.4243027888446215,0.9806590555130963
+14,factual_entities-03,factual_entities,0.4132762312633833,0.9902565998139005
+14,factual_entities-04,factual_entities,0.33246753246753247,0.9902472711789608
+14,factual_entities-05,factual_entities,0.33604336043360433,0.9902863249438036
+14,factual_entities-06,factual_entities,0.42857142857142855,0.9902616031736035
+14,factual_entities-07,factual_entities,0.24914089347079038,0.9901902663444527
+14,factual_entities-08,factual_entities,0.32908163265306123,0.9902491322072741
+14,factual_entities-09,factual_entities,0.4661016949152542,0.9806617404107945
+14,factual_entities-10,factual_entities,0.38852097130242824,0.9902736635368758
+14,factual_entities-11,factual_entities,0.3036529680365297,0.9902076137189368
+14,factual_entities-12,factual_entities,0.2951945080091533,0.9806133254367783
+14,factual_entities-13,factual_entities,0.33186813186813185,0.9902244297444769
+14,factual_entities-14,factual_entities,0.31125827814569534,0.990216709596998
+14,factual_entities-15,factual_entities,0.32119914346895073,0.9902130156922043
+14,uncertainty-00,uncertainty,0.20244565217391305,0.9900286860909084
+14,uncertainty-01,uncertainty,0.2620320855614973,0.9806079353451435
+14,uncertainty-02,uncertainty,0.23044096728307253,0.9805135266732615
+14,uncertainty-03,uncertainty,0.20485584218512898,0.9805007390186093
+14,uncertainty-04,uncertainty,0.2469352014010508,0.9997690095362919
+14,uncertainty-05,uncertainty,0.3248730964467005,0.9901272013945135
+14,uncertainty-06,uncertainty,0.22468793342579751,0.9805750378457416
+14,uncertainty-07,uncertainty,0.28343949044585987,0.990128968417892
+14,uncertainty-08,uncertainty,0.2887189292543021,0.9806429487324042
+14,uncertainty-09,uncertainty,0.26598837209302323,0.9805908163863253
+14,uncertainty-10,uncertainty,0.24731182795698925,0.9901832762734908
+14,uncertainty-11,uncertainty,0.20458015267175572,0.980513680761964
+14,uncertainty-12,uncertainty,0.2840909090909091,0.9997705186024733
+14,uncertainty-13,uncertainty,0.2075187969924812,0.9901326983455394
+14,uncertainty-14,uncertainty,0.22425249169435216,0.980521299878531
+14,uncertainty-15,uncertainty,0.2253922967189729,0.9900809168793652
+26,code-00,code,0.47198641765704585,0.9848584045186443
+26,code-01,code,0.383399209486166,0.9858668819538499
+26,code-02,code,0.4169278996865204,0.9762388131446491
+26,code-03,code,0.25380710659898476,0.970000658339452
+26,code-04,code,0.44649446494464945,0.9844480924588723
+26,code-05,code,0.22795341098169716,0.9676811446514547
+26,code-06,code,0.24489795918367346,0.9782172282720092
+26,code-07,code,0.2109479305740988,0.9621779989954782
+26,code-08,code,0.22490706319702602,0.9609696828283534
+26,code-09,code,0.36466165413533835,0.9773365358913297
+26,code-10,code,0.41244239631336405,0.9914057380352036
+26,code-11,code,0.2899628252788104,0.9699833683977059
+26,code-12,code,0.3757455268389662,0.9823072813659135
+26,code-13,code,0.3242258652094718,0.9705553235908142
+26,code-14,code,0.26895306859205775,0.9695280566411819
+26,code-15,code,0.3605220228384992,0.974499060417436
+26,mathematics-00,mathematics,0.36561264822134387,0.9796888087641206
+26,mathematics-01,mathematics,0.29895104895104896,0.9703758173869087
+26,mathematics-02,mathematics,0.2757281553398058,0.969695675439566
+26,mathematics-03,mathematics,0.35384615384615387,0.9819222077676345
+26,mathematics-04,mathematics,0.3712418300653595,0.9804912934244707
+26,mathematics-05,mathematics,0.23853211009174313,0.9647613230783765
+26,mathematics-06,mathematics,0.3560732113144759,0.9818142584052143
+26,mathematics-07,mathematics,0.27563025210084036,0.9707926207389407
+26,mathematics-08,mathematics,0.39303482587064675,0.9882774539236738
+26,mathematics-09,mathematics,0.3079470198675497,0.9787356456071212
+26,mathematics-10,mathematics,0.2982456140350877,0.9759089355171304
+26,mathematics-11,mathematics,0.45,0.9844838766072592
+26,mathematics-12,mathematics,0.31842105263157894,0.9887589017851269
+26,mathematics-13,mathematics,0.36577181208053694,0.9841427649579938
+26,mathematics-14,mathematics,0.23517382413087934,0.973035465427698
+26,mathematics-15,mathematics,0.31197301854974707,0.9738341044575504
+26,positive_sentiment-00,positive_sentiment,0.4785553047404063,0.985883926145408
+26,positive_sentiment-01,positive_sentiment,0.4497153700189753,0.9761383234646351
+26,positive_sentiment-02,positive_sentiment,0.3151515151515151,0.9560698601590455
+26,positive_sentiment-03,positive_sentiment,0.26534653465346536,0.9660999327217742
+26,positive_sentiment-04,positive_sentiment,0.22986577181208054,0.9536564507470459
+26,positive_sentiment-05,positive_sentiment,0.41284403669724773,0.9787430091962116
+26,positive_sentiment-06,positive_sentiment,0.279383429672447,0.9639158498832134
+26,positive_sentiment-07,positive_sentiment,0.3100511073253833,0.9679804687019743
+26,positive_sentiment-08,positive_sentiment,0.28867924528301886,0.9770258074494018
+26,positive_sentiment-09,positive_sentiment,0.3169734151329243,0.9745638396031774
+26,positive_sentiment-10,positive_sentiment,0.3755186721991701,0.9743987465531274
+26,positive_sentiment-11,positive_sentiment,0.3853564547206166,0.9722886626017665
+26,positive_sentiment-12,positive_sentiment,0.2597864768683274,0.9691515013459704
+26,positive_sentiment-13,positive_sentiment,0.37857142857142856,0.9638509625427929
+26,positive_sentiment-14,positive_sentiment,0.3259423503325942,0.9814497047101494
+26,positive_sentiment-15,positive_sentiment,0.3816631130063966,0.9794379162814384
+26,negative_sentiment-00,negative_sentiment,0.3897707231040564,0.9652151308577941
+26,negative_sentiment-01,negative_sentiment,0.3362218370883882,0.9693720599309181
+26,negative_sentiment-02,negative_sentiment,0.29721362229102166,0.9459332261063973
+26,negative_sentiment-03,negative_sentiment,0.23745819397993312,0.9566476730623248
+26,negative_sentiment-04,negative_sentiment,0.2719869706840391,0.9586731362896578
+26,negative_sentiment-05,negative_sentiment,0.3956989247311828,0.9728123156712534
+26,negative_sentiment-06,negative_sentiment,0.19148936170212766,0.9651287673585536
+26,negative_sentiment-07,negative_sentiment,0.2815198618307427,0.9694833164540114
+26,negative_sentiment-08,negative_sentiment,0.20761904761904762,0.9758022273465673
+26,negative_sentiment-09,negative_sentiment,0.2867132867132867,0.9707104375173238
+26,negative_sentiment-10,negative_sentiment,0.3564356435643564,0.9789782397077743
+26,negative_sentiment-11,negative_sentiment,0.3036912751677852,0.9475709835302312
+26,negative_sentiment-12,negative_sentiment,0.27631578947368424,0.9574917983704665
+26,negative_sentiment-13,negative_sentiment,0.3356401384083045,0.9597336644213671
+26,negative_sentiment-14,negative_sentiment,0.31297709923664124,0.9617859892985854
+26,negative_sentiment-15,negative_sentiment,0.4134199134199134,0.9799140499407444
+26,german_language-00,german_language,0.5523255813953488,0.9858823980876071
+26,german_language-01,german_language,0.5583333333333333,0.9818781030077225
+26,german_language-02,german_language,0.5413533834586466,0.9406695352378446
+26,german_language-03,german_language,0.4495967741935484,0.9351639990022604
+26,german_language-04,german_language,0.43526785714285715,0.9371321944674489
+26,german_language-05,german_language,0.3906810035842294,0.9498115774761665
+26,german_language-06,german_language,0.3686274509803922,0.9436436080386009
+26,german_language-07,german_language,0.4104803493449782,0.9490598262643566
+26,german_language-08,german_language,0.4007285974499089,0.9395215467703346
+26,german_language-09,german_language,0.4142538975501114,0.9448209357654925
+26,german_language-10,german_language,0.39755351681957185,0.9185348808213194
+26,german_language-11,german_language,0.4053497942386831,0.937445479032733
+26,german_language-12,german_language,0.41130604288499023,0.9471812839566915
+26,german_language-13,german_language,0.5907258064516129,0.9664699465639633
+26,german_language-14,german_language,0.40039447731755423,0.9421392933556214
+26,german_language-15,german_language,0.4343675417661098,0.9659463453760118
+26,factual_entities-00,factual_entities,0.3542009884678748,0.982803078620405
+26,factual_entities-01,factual_entities,0.2428842504743833,0.9777129644013733
+26,factual_entities-02,factual_entities,0.38684719535783363,0.9848257590673264
+26,factual_entities-03,factual_entities,0.376425855513308,0.9864308237412153
+26,factual_entities-04,factual_entities,0.2860520094562648,0.9872782212647289
+26,factual_entities-05,factual_entities,0.32727272727272727,0.9894438471069814
+26,factual_entities-06,factual_entities,0.34368530020703936,0.9877802949790806
+26,factual_entities-07,factual_entities,0.21451104100946372,0.9681214259269855
+26,factual_entities-08,factual_entities,0.28824833702882485,0.9861508215351278
+26,factual_entities-09,factual_entities,0.3897338403041825,0.9858127629409149
+26,factual_entities-10,factual_entities,0.37916666666666665,0.9843385828538938
+26,factual_entities-11,factual_entities,0.2708333333333333,0.9766906457898232
+26,factual_entities-12,factual_entities,0.31981981981981983,0.9858022274230956
+26,factual_entities-13,factual_entities,0.30103092783505153,0.9818253711354649
+26,factual_entities-14,factual_entities,0.29324894514767935,0.9842300426384109
+26,factual_entities-15,factual_entities,0.266260162601626,0.9824575222850866
+26,uncertainty-00,uncertainty,0.23006535947712417,0.9647279190124038
+26,uncertainty-01,uncertainty,0.2147239263803681,0.9668632556776208
+26,uncertainty-02,uncertainty,0.2440944881889764,0.9644592593944762
+26,uncertainty-03,uncertainty,0.22727272727272727,0.9537769089985246
+26,uncertainty-04,uncertainty,0.23190789473684212,0.9746432914962311
+26,uncertainty-05,uncertainty,0.3225806451612903,0.9640010591452041
+26,uncertainty-06,uncertainty,0.26768377253814146,0.9682388056898258
+26,uncertainty-07,uncertainty,0.28419452887537994,0.9682876208095923
+26,uncertainty-08,uncertainty,0.2387312186978297,0.9776347581787265
+26,uncertainty-09,uncertainty,0.26853146853146853,0.9685934805164662
+26,uncertainty-10,uncertainty,0.24279835390946503,0.9584198912471869
+26,uncertainty-11,uncertainty,0.2344213649851632,0.969776567257657
+26,uncertainty-12,uncertainty,0.26040061633281975,0.975147223148218
+26,uncertainty-13,uncertainty,0.22886297376093295,0.9631413403561002
+26,uncertainty-14,uncertainty,0.20998531571218795,0.971263152992093
+26,uncertainty-15,uncertainty,0.28424153166421207,0.9664766127608434

artifacts/study_feature_summary.csv

@@ -0,0 +1,8 @@
+concept,layer,feature_id,train_auroc,heldout_auroc,heldout_f1,activation_rate_pos_train,activation_rate_neg_train,candidate_resample_support,candidate_median_resample_rank,mean_paraphrase_topk_jaccard,mean_paraphrase_sparse_cosine,final_tasks,final_feature_active_at_intervention_rate,final_feature_active_at_final_token_rate,final_feature_active_anywhere_rate,final_target_sae_abs_mean,final_target_random_abs_mean,final_target_specificity_ratio,final_target_paired_advantage,final_target_paired_ci_low,final_target_paired_ci_high,final_target_sign_flip_pvalue,final_active_target_sae_abs_mean,final_active_target_random_abs_mean,final_active_target_specificity_ratio,final_active_target_paired_advantage,final_active_target_paired_ci_low,final_active_target_paired_ci_high,final_active_target_sign_flip_pvalue,final_active_tasks,final_js_sae_mean,final_js_random_mean,final_js_specificity_ratio,final_active_js_specificity_ratio,max_active_tasks,max_active_feature_active_at_intervention_rate,max_active_feature_active_at_final_token_rate,max_active_feature_active_anywhere_rate,max_active_target_sae_abs_mean,max_active_target_random_abs_mean,max_active_target_specificity_ratio,max_active_target_paired_advantage,max_active_target_paired_ci_low,max_active_target_paired_ci_high,max_active_target_sign_flip_pvalue,max_active_active_target_sae_abs_mean,max_active_active_target_random_abs_mean,max_active_active_target_specificity_ratio,max_active_active_target_paired_advantage,max_active_active_target_paired_ci_low,max_active_active_target_paired_ci_high,max_active_active_target_sign_flip_pvalue,max_active_active_tasks,max_active_js_sae_mean,max_active_js_random_mean,max_active_js_specificity_ratio,max_active_active_js_specificity_ratio,target_specificity_gain_max_vs_final,js_specificity_gain_max_vs_final
+code,26,19655,0.988136574074074,0.9661458333333334,0.8235294117647058,1.0,0.1805555555555555,1.0,1.0,0.3298646763447731,0.9753796418438968,4,0.5,0.5,0.0,0.03070369362831114,0.016958609223365763,1.8105077618043846,0.013745084404945379,0.0,0.030128069221973433,0.5,0.06140738725662228,0.033917218446731526,1.8105077618043846,0.027490168809890757,0.014809578657150269,0.040170758962631246,0.5,2,0.000468006153823775,0.0002624576891321125,1.7831680046081493,1.7831680046081493,4,1.0,0.5,1.0,0.0017386823892592963,0.0020397175103425555,0.8524133270627743,-0.00030103512108325915,-0.0026935096830129476,0.0025318711996078617,0.75,0.0017386823892592963,0.0020397175103425555,0.8524133270627743,-0.00030103512108325915,-0.0026935096830129476,0.0025318711996078617,0.75,4,7.899344154793653e-06,5.354482187414078e-06,1.4752769508434211,1.4752769508434211,-0.9580944347416103,-0.30789105376472814
+factual_entities,14,29048,0.9745370370370372,0.9075520833333334,0.8235294117647058,0.9583333333333334,0.0138888888888888,1.0,3.0,0.346650517448705,0.9884353764433393,4,0.25,0.25,0.0,0.002008568495512,0.0012214824091642939,1.6443695631165165,0.0007870860863477061,0.0,0.0023612582590431183,1.0,0.008034273982048,0.0048859296366571756,1.6443695631165165,0.0031483443453908244,0.0031483443453908244,0.0031483443453908244,1.0,1,1.8891484614869114e-05,8.156611613685527e-06,2.316094661559236,2.316094661559236,4,1.0,0.25,1.0,0.010054231388494334,0.007599377029691787,1.3230336314688824,0.0024548543588025474,-0.00026353719295001734,0.007209595802123662,0.5,0.010054231388494334,0.007599377029691787,1.3230336314688824,0.0024548543588025474,-0.00026353719295001734,0.007209595802123662,0.5,4,1.5267146636688267e-05,1.2116573988585344e-05,1.2600217397319557,1.2600217397319557,-0.3213359316476341,-1.0560729218272802
+german_language,4,4446,1.0,1.0,0.9333333333333332,1.0,0.0,1.0,2.0,0.3989601630452016,0.9999699385821158,4,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0,,,,,,,,0,0.0,0.0,0.0,,4,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0,,,,,,,,0,0.0,0.0,0.0,,0.0,0.0
+mathematics,26,12177,0.9982638888888888,1.0,0.9333333333333332,1.0,0.0486111111111111,1.0,1.0,0.3260114110685498,0.9779199470805491,4,1.0,1.0,0.0,0.013766535557806436,0.01286024367436762,1.0704723725604977,0.0009062918834388161,-0.013581858249381188,0.010408063651993862,1.0,0.013766535557806436,0.01286024367436762,1.0704723725604977,0.0009062918834388161,-0.013581858249381188,0.010408063651993862,1.0,4,0.0017012150638037876,0.0010852779771539155,1.5675385473730286,1.5675385473730286,4,1.0,1.0,1.0,0.010522400029003563,0.004535389365628313,2.3200654190240257,0.005987010663375247,0.0016587544232606831,0.010408063651993862,0.25,0.010522400029003563,0.004535389365628313,2.3200654190240257,0.005987010663375247,0.0016587544232606831,0.010408063651993862,0.25,4,0.0008034180818867435,0.0003310765635262305,2.4266836448031768,2.4266836448031768,1.249593046463528,0.8591450974301482
+negative_sentiment,14,14651,0.9971064814814816,1.0,0.9411764705882352,1.0,0.0347222222222222,1.0,2.0,0.2952628431383464,0.9836082392320687,4,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0,,,,,,,,0,0.0,0.0,0.0,,4,1.0,0.0,1.0,0.08628357946872708,0.04091272130608554,2.1089670086524626,0.04537085816264153,0.010284267365932447,0.09931341186165814,0.125,0.08628357946872708,0.04091272130608554,2.1089670086524626,0.04537085816264153,0.010284267365932447,0.09931341186165814,0.125,4,0.0009591840007487634,0.00033541750718764655,2.8596718423888228,2.8596718423888228,2.1089670086524626,2.8596718423888228
+positive_sentiment,14,29229,0.9716435185185186,0.9869791666666669,0.7777777777777778,0.9583333333333334,0.1388888888888889,1.0,2.0,0.31956337795142914,0.9836190853782412,4,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0,,,,,,,,0,0.0,0.0,0.0,,4,1.0,0.0,1.0,0.09154844284057614,0.052779592573642675,1.7345424315816724,0.03876885026693346,0.007168605923652652,0.07036909461021426,0.125,0.09154844284057614,0.052779592573642675,1.7345424315816724,0.03876885026693346,0.007168605923652652,0.07036909461021426,0.125,4,0.0001696736740086238,9.280085743055595e-05,1.8283632145919853,1.8283632145919853,1.7345424315816724,1.8283632145919853
+uncertainty,14,1384,0.9752604166666669,0.875,0.8571428571428571,0.9583333333333334,0.0069444444444444,1.0,2.0,0.24547275283723785,0.9865429537614034,4,0.25,0.25,0.0,0.006539523601531975,0.0038898661732673515,1.6811693025518675,0.002649657428264623,0.0,0.00794897228479387,1.0,0.0261580944061279,0.015559464693069406,1.6811693025518675,0.010598629713058492,0.010598629713058492,0.010598629713058492,1.0,1,0.000125072539958625,5.517254589902812e-05,2.2669343587573723,2.2669343587573723,4,0.75,0.25,0.75,0.08970040082931516,0.016293663531541786,5.5052321815575915,0.07340673729777339,-0.000809632241725906,0.2134449183940888,0.5,0.11960053443908687,0.021724884708722383,5.505232181557591,0.09787564973036451,-0.001619264483451812,0.2845932245254517,0.5,3,0.0004966975570823587,8.256788069614337e-05,6.015626813896881,6.015626813896881,3.824062879005724,3.7486924551395084

artifacts/study_summary.json

@@ -0,0 +1,59 @@
+{
+  "n_concepts": 7,
+  "selected_feature_pooling": "prompt-wide max SAE activation across non-padding prompt tokens",
+  "dense_probe_pooling": "final prompt token residual",
+  "primary_causal_position_policy": "max_feature_activation",
+  "causal_statistical_unit": "causal task; ablation and amplification are averaged within task before paired inference",
+  "median_selected_feature_resample_support": 1.0,
+  "final_token_feature_coverage": 0.2857142857142857,
+  "max_active_feature_coverage": 0.8214285714285714,
+  "final_token_target_specificity_ratio": 1.5178361142081527,
+  "max_active_target_specificity_ratio": 2.3344608571122225,
+  "final_token_target_paired_advantage": 0.002584017114713789,
+  "max_active_target_paired_advantage": 0.023669610804063268,
+  "final_token_target_paired_ci_95": [
+    -0.0005554523851190289,
+    0.006223834152167133
+  ],
+  "max_active_target_paired_ci_95": [
+    0.006155622166625111,
+    0.04787022493006328
+  ],
+  "final_token_target_sign_flip_pvalue": 0.171875,
+  "max_active_target_sign_flip_pvalue": 9.99950002499875e-05,
+  "most_predictive_concept": {
+    "concept": "german_language",
+    "heldout_auroc": 1.0
+  },
+  "highest_max_active_target_specificity": {
+    "concept": "uncertainty",
+    "ratio": 5.5052321815575915
+  },
+  "highest_max_active_js_specificity": {
+    "concept": "uncertainty",
+    "ratio": 6.015626813896881
+  },
+  "correlations": {
+    "heldout_auroc_vs_max_active_target_specificity": {
+      "rho": -0.18531232916527532,
+      "pvalue": 0.6907777961916857,
+      "n": 7
+    },
+    "heldout_auroc_vs_max_active_js_specificity": {
+      "rho": -0.14824986333222023,
+      "pvalue": 0.7510797526579065,
+      "n": 7
+    },
+    "heldout_f1_vs_max_active_target_specificity": {
+      "rho": 0.21821789023599242,
+      "pvalue": 0.638298871640929,
+      "n": 7
+    },
+    "candidate_resample_support_vs_max_active_target_specificity": {
+      "rho": NaN,
+      "pvalue": NaN,
+      "n": 7
+    }
+  },
+  "guardrail": "Max-active causal positions are selected from SAE activation only, never from behavioral outcome. Cross-concept Spearman correlations are descriptive because the study contains seven concepts."
+}

artifacts/summary.json

@@ -0,0 +1,176 @@
+{
+  "headline": "Selected SAE features averaged 0.962 held-out AUROC. Max-active interventions covered 82.1% of causal tasks and changed mean log p/token by 0.041 in absolute value on average versus 0.018 for norm-matched random controls (2.33\u00d7).",
+  "highlights": [
+    "Median selected-feature held-out AUROC: 0.987; mean AUROC 95% bootstrap CI [0.927, 0.994].",
+    "Best residual linear-probe layer: 14 with macro AUROC 1.000.",
+    "Mean paraphrase TopK Jaccard: 0.326; sparse activation cosine: 0.985.",
+    "Feature coverage: final-token policy 28.6%; active anywhere in prompt 82.1%; max-active intervention 82.1%.",
+    "Final-token task-level SAE/random ratio: 1.52\u00d7; paired advantage +0.0026, 95% CI [-0.0005, +0.0063], sign-flip p=0.1719.",
+    "Max-active task-level SAE/random ratio: 2.33\u00d7; paired advantage +0.0237, 95% CI [+0.0067, +0.0469], sign-flip p=0.0001.",
+    "Conditional on feature-active tasks, max-active SAE/random ratio: 2.33\u00d7 (n=23).",
+    "Final-token top-5 joint ablation SAE/random ratio: 1.09\u00d7; paired advantage +0.0009, 95% CI [-0.0035, +0.0065], sign-flip p=0.7891.",
+    "Across seven concepts, held-out AUROC vs max-active target specificity Spearman \u03c1=-0.185; descriptive only.",
+    "Held-out AUROC vs max-active JS specificity Spearman \u03c1=-0.148; descriptive only."
+  ],
+  "interpretation": "Max-active interventions produced larger task-level target effects than norm-matched random controls with paired uncertainty excluding zero. Predictive SAE features therefore show causal specificity when intervened where the selected feature is actually represented, while the final-token baseline quantifies sensitivity to intervention location. Moving from the final prompt token to the feature's maximum-activation token increased intervention coverage from 28.6% to 82.1%, showing that causal conclusions depend materially on where the representation is tested.",
+  "metrics": {
+    "mean_selected_feature_test_auroc": 0.9622395833333334,
+    "mean_selected_feature_test_auroc_bootstrap_ci_95": [
+      0.9267066592261906,
+      0.994419642857143
+    ],
+    "median_selected_feature_test_auroc": 0.9869791666666669,
+    "best_linear_probe_layer": 14,
+    "best_linear_probe_macro_auroc": 1.0,
+    "mean_paraphrase_topk_jaccard": 0.32550299223975043,
+    "mean_paraphrase_sparse_cosine": 0.9846351666472363,
+    "final_token_feature_coverage": 0.2857142857142857,
+    "prompt_anywhere_feature_coverage": 0.8214285714285714,
+    "max_active_feature_coverage": 0.8214285714285714,
+    "final_token_task_level": {
+      "sae_abs": 0.007574045897594507,
+      "random_abs": 0.004990028782880718,
+      "ratio": 1.5178361142081527,
+      "advantage": 0.002584017114713789,
+      "ci": [
+        -0.0005489836530094687,
+        0.006322265709085119
+      ],
+      "pvalue": 0.171875,
+      "n_tasks": 28
+    },
+    "final_token_active_only": {
+      "sae_abs": 0.026509160641580775,
+      "random_abs": 0.017465100740082513,
+      "ratio": 1.5178361142081527,
+      "advantage": 0.009044059901498262,
+      "ci": [
+        -0.0016551230626646388,
+        0.020267208322184146
+      ],
+      "pvalue": 0.171875,
+      "n_tasks": 8
+    },
+    "max_active_task_level": {
+      "sae_abs": 0.04140681956362508,
+      "random_abs": 0.01773720875956181,
+      "ratio": 2.3344608571122225,
+      "advantage": 0.023669610804063268,
+      "ci": [
+        0.0066882537599927475,
+        0.0468946453319534
+      ],
+      "pvalue": 9.99950002499875e-05,
+      "n_tasks": 28
+    },
+    "max_active_active_only": {
+      "sae_abs": 0.05040830207745662,
+      "random_abs": 0.021593123707292633,
+      "ratio": 2.334460857112223,
+      "advantage": 0.028815178370163983,
+      "ci": [
+        0.008352378036563645,
+        0.05887514829816571
+      ],
+      "pvalue": 0.00014999250037498125,
+      "n_tasks": 23
+    },
+    "feature_set_results": {
+      "1": {
+        "sae_abs": 0.0073844761188541,
+        "random_abs": 0.00605300132052173,
+        "ratio": 1.219969355337528,
+        "advantage": 0.0013314747983323686,
+        "ci": [
+          -0.0021684337141258357,
+          0.004932756118276826
+        ],
+        "pvalue": 0.453125,
+        "n_tasks": 28
+      },
+      "3": {
+        "sae_abs": 0.01165702566504476,
+        "random_abs": 0.00956058465609591,
+        "ratio": 1.2192795821971139,
+        "advantage": 0.0020964410089488496,
+        "ci": [
+          -0.003360107559378132,
+          0.009788807602100323
+        ],
+        "pvalue": 0.75146484375,
+        "n_tasks": 28
+      },
+      "5": {
+        "sae_abs": 0.010901511247668921,
+        "random_abs": 0.00997775314109664,
+        "ratio": 1.0925817760280598,
+        "advantage": 0.0009237581065722826,
+        "ci": [
+          -0.0035061794998390364,
+          0.006451443075535015
+        ],
+        "pvalue": 0.7890625,
+        "n_tasks": 28
+      }
+    },
+    "study_summary": {
+      "n_concepts": 7,
+      "selected_feature_pooling": "prompt-wide max SAE activation across non-padding prompt tokens",
+      "dense_probe_pooling": "final prompt token residual",
+      "primary_causal_position_policy": "max_feature_activation",
+      "causal_statistical_unit": "causal task; ablation and amplification are averaged within task before paired inference",
+      "median_selected_feature_resample_support": 1.0,
+      "final_token_feature_coverage": 0.2857142857142857,
+      "max_active_feature_coverage": 0.8214285714285714,
+      "final_token_target_specificity_ratio": 1.5178361142081527,
+      "max_active_target_specificity_ratio": 2.3344608571122225,
+      "final_token_target_paired_advantage": 0.002584017114713789,
+      "max_active_target_paired_advantage": 0.023669610804063268,
+      "final_token_target_paired_ci_95": [
+        -0.0005554523851190289,
+        0.006223834152167133
+      ],
+      "max_active_target_paired_ci_95": [
+        0.006155622166625111,
+        0.04787022493006328
+      ],
+      "final_token_target_sign_flip_pvalue": 0.171875,
+      "max_active_target_sign_flip_pvalue": 9.99950002499875e-05,
+      "most_predictive_concept": {
+        "concept": "german_language",
+        "heldout_auroc": 1.0
+      },
+      "highest_max_active_target_specificity": {
+        "concept": "uncertainty",
+        "ratio": 5.5052321815575915
+      },
+      "highest_max_active_js_specificity": {
+        "concept": "uncertainty",
+        "ratio": 6.015626813896881
+      },
+      "correlations": {
+        "heldout_auroc_vs_max_active_target_specificity": {
+          "rho": -0.18531232916527532,
+          "pvalue": 0.6907777961916857,
+          "n": 7
+        },
+        "heldout_auroc_vs_max_active_js_specificity": {
+          "rho": -0.14824986333222023,
+          "pvalue": 0.7510797526579065,
+          "n": 7
+        },
+        "heldout_f1_vs_max_active_target_specificity": {
+          "rho": 0.21821789023599242,
+          "pvalue": 0.638298871640929,
+          "n": 7
+        },
+        "candidate_resample_support_vs_max_active_target_specificity": {
+          "rho": NaN,
+          "pvalue": NaN,
+          "n": 7
+        }
+      },
+      "guardrail": "Max-active causal positions are selected from SAE activation only, never from behavioral outcome. Cross-concept Spearman correlations are descriptive because the study contains seven concepts."
+    }
+  }
+}

docs/VALIDATION.md

@@ -1,8 +1,6 @@
-# FeatureLens v0.16 validation
+# FeatureLens final validation
 
-v0.16 changes the **offline causal methodology**, not the already validated live HF inference UI. Do not spend ZeroGPU quota retesting live Workbench paths.
-
-## Local software gate
+Run the complete local software gate before publishing:
 
 ```bash
 python3 -m pytest -q
@@ -10,27 +8,35 @@ python3 -m compileall -q app.py featurelens experiments scripts
 python3 -m ruff check app.py featurelens experiments tests scripts
 python3 scripts/ui_smoke.py
 python3 scripts/release_check.py
+python3 -m scripts.validate_artifacts
 ```
 
-## Addendum acceptance
+## Public study checks
+
+The committed `artifacts/` bundle must:
+
+1. contain both `causal_results_final_token.csv` and `causal_results_max_active.csv`;
+2. use `max_feature_activation` as the primary causal policy in `study_summary.json`;
+3. document causal-task-level paired inference;
+4. report 224 discovery prompts and 28 causal tasks;
+5. include the six report figures required by `scripts.validate_artifacts`;
+6. contain no activation matrices, model weights, SAE checkpoints, or completion markers.
 
-Use `notebooks/FeatureLens_Causal_Addendum_Colab.ipynb` with the completed v0.15 Drive run.
+## HF Space acceptance
 
-The addendum is successful when:
+No new model inference needs to be rerun for the final publication if the software checks pass. Verify visually that:
 
-1. `causal_results_final_token.csv` exists and preserves the v0.15 baseline.
-2. `causal_results_max_active.csv` completes all 28 causal tasks with 8 random controls per intervention.
-3. `causal_position_summary.csv` contains both `final_token` and `max_feature_activation` policies.
-4. `study_summary.json` declares `max_feature_activation` as the primary causal position policy and causal-task-level inference as the statistical unit.
-5. `python -m scripts.validate_artifacts` prints `PASS`.
-6. `FeatureLens_offline_results_v016.zip` is created without activation caches.
+- the **Study** tab loads measured results rather than the empty-state message;
+- the measured headline and causal-position comparison are visible;
+- the Workbench and other previously validated live paths still render;
+- the interface remains version-neutral and follows `DESIGN.md`.
 
-## HF acceptance after final artifacts are committed
+## Reproducibility
 
-No GPU call is required. Open **Study** and verify:
+For a fresh study, use `notebooks/FeatureLens_Offline_Study_Colab.ipynb` or:
+
+```bash
+python -m experiments.run_all --resume
+```
 
-- the measured headline is populated;
-- final-token and max-active coverage/specificity are visible;
-- the causal-position table and figure render;
-- the association-vs-causality figure uses max-active specificity;
-- no placeholder or old v0.15 significance language remains.
+The causal-addendum notebook is retained only as a migration/reproduction utility for an already-completed final-token baseline.

experiments/make_report.py

@@ -538,7 +538,7 @@ def _build_report_lines(
             "",
             "## Reproducibility",
             "",
-            "Run `python -m experiments.run_all --resume` for a fresh full study. For an existing v0.15 final-token study, run the v0.16 causal addendum notebook; it preserves the baseline, computes only max-active causal rows, and reruns CPU analysis/reporting.",
+            "Run `python -m experiments.run_all --resume` for a fresh full study. The causal-addendum notebook is retained as a migration utility for an already-completed final-token baseline.",
             "",
         ]
     )

featurelens/study.py

@@ -59,10 +59,8 @@ class OfflineStudy:
                 '### Offline study not materialized yet\n\n'
                 'The live workbench is usable now, but the finalized position-sensitivity study artifacts '
                 f'have not been committed. Missing: {missing}{suffix}\n\n'
-                'For a fresh study run `python -m experiments.run_all --resume`. If the v0.15 final-token '
-                'study already exists, use the v0.16 causal-addendum notebook or '
-                '`python -m experiments.run_causal_addendum --resume` to add max-active causal results '
-                'without recollecting discovery activations.'
+                'Run `python -m experiments.run_all --resume` or use the full-study Colab notebook '
+                'to materialize the measured study artifacts.'
             )
 
         summary = self._json('summary.json')

notebooks/README.md

@@ -0,0 +1,8 @@
+# Notebooks
+
+FeatureLens includes two Colab runners.
+
+- `FeatureLens_Offline_Study_Colab.ipynb` — **canonical runner** for reproducing the complete study from a fresh checkout. It executes discovery/evaluation, both causal-position policies, feature-set diagnostics, study synthesis, and artifact validation.
+- `FeatureLens_Causal_Addendum_Colab.ipynb` — migration/reproduction utility for a completed final-token baseline. It preserves the original baseline and computes only the max-feature-activation causal addendum plus CPU analysis.
+
+For a fresh reproduction, use the full-study notebook. The addendum notebook is retained because it documents the exact path used to extend the original baseline without recomputing the expensive discovery activations.

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "featurelens"
-version = "0.16.0"
+version = "1.0.0"
 description = "Causal sparse-feature interpretability workbench for Qwen3 and Qwen-Scope SAEs"
 requires-python = ">=3.10"
 

research_config.json

@@ -46,12 +46,6 @@
     "bootstrap_95_ci",
     "paired_sign_flip_test"
   ],
-  "live_features_v0_3": [
-    "full_continuation_scoring",
-    "joint_multi_feature_intervention",
-    "topk_feature_set_size_sweep",
-    "paraphrase_robustness_explorer"
-  ],
   "feature_set_sizes": [
     1,
     3,
@@ -68,78 +62,21 @@
   "offline_random_controls_default": 8,
   "control_reference": "batched zero-edit residual row",
   "paraphrase_promptwide_pooling": "max activation per SAE feature across all prompt tokens",
-  "live_features_v0_4": [
-    "batch_context_null_reference",
-    "random_control_ensemble",
-    "individual_vs_joint_interaction_decomposition",
-    "promptwide_paraphrase_robustness",
-    "controlled_concept_contrast_scan",
-    "copy_tables_with_headers"
-  ],
   "concept_contrast_prompts_per_concept": 4,
   "interaction_feature_limit": 5,
   "concept_contrast_pooling": "max activation across non-padding prompt tokens",
   "live_geometry_feature_limit": 8,
   "contrastive_preference_metric": "change in exact-sequence log-odds between two user-specified continuations",
-  "live_features_v0_5": [
-    "wide_centered_responsive_layout",
-    "copy_feedback",
-    "dynamic_height_reflow_observer",
-    "promptwide_concept_contrast_scan",
-    "feature_token_activation_trace",
-    "contrastive_continuation_preference_test",
-    "feature_decoder_geometry"
-  ],
-  "concept_candidate_discovery_metric": "balanced exploratory score = selectivity \u00d7 target activation rate \u00d7 log1p(target mean); causal-ready mode additionally requires current-token activity and log-scales that activation; raw mean-difference remains available as a scale-sensitive comparison",
+  "concept_candidate_discovery_metric": "balanced exploratory score = selectivity × target activation rate × log1p(target mean); causal-ready mode additionally requires current-token activity and log-scales that activation; raw mean-difference remains available as a scale-sensitive comparison",
   "completion_cue_scan": "final-token feature activation after controlled suffix/cue substitution",
-  "live_features_v0_6": [
-    "start_here_plain_language_onboarding",
-    "persistent_workbench_context_banner",
-    "explicit_per_experiment_feature_selectors",
-    "plot_fullscreen_and_export_controls",
-    "consistent_heading_and_table_typography",
-    "concept_guided_candidate_feature_discovery",
-    "completion_cue_sensitivity_scan"
-  ],
-  "live_features_v0_7": [
-    "cleaned_nonaccordion_experiment_layout",
-    "focused_fullscreen_modal_for_tables_and_plots",
-    "descriptive_plot_export_filenames",
-    "german_language_control_concept",
-    "balanced_candidate_ranking_and_current_prompt_compatibility",
-    "click_to_select_candidate_rows",
-    "completion_cue_context_matrix"
-  ],
-  "live_features_v0_8": [
-    "bounded_plot_focus_overlay_with_scroll_restore",
-    "explicit_result_table_headings",
-    "standalone_dose_response_target_and_feature_inputs",
-    "causal_ready_current_token_candidate_ranking",
-    "cue_dominance_specificity_interpretation",
-    "muted_cue_context_plot_palette"
-  ],
   "candidate_causal_screen_limit": 8,
   "candidate_causal_screen_control": "batched zero-edit reference; no random controls in triage screen",
-  "live_features_v0_9": [
-    "in_place_aspect_preserving_plot_and_table_focus",
-    "compact_table_heading_alignment",
-    "concise_independent_dose_response_copy",
-    "batched_candidate_causal_triage",
-    "gpu_budget_aware_hf_validation_scope"
-  ],
   "candidate_alignment_metrics": [
     "discovery rank versus target-effect rank",
     "discovery rank versus next-token JS rank",
     "Spearman candidate score versus absolute target effect",
     "Spearman candidate score versus next-token JS"
   ],
-  "live_features_v0_10": [
-    "discovery_to_causality_alignment_table",
-    "association_evidence_vs_target_effect_scatter",
-    "descriptive_spearman_concordance_summary",
-    "target_effect_vs_distribution_shift_rank_separation",
-    "no_extra_gpu_candidate_synthesis"
-  ],
   "candidate_specificity_limit": 3,
   "candidate_specificity_control": "each candidate gets live_random_controls deterministic norm-matched residual directions in one shared batched zero-edit execution context",
   "candidate_specificity_metrics": [
@@ -148,13 +85,6 @@
     "coarse empirical random-control tail probabilities",
     "discovery rank versus random-normalized target-specificity rank"
   ],
-  "live_features_v0_11": [
-    "controlled_multi_candidate_random_specificity_screen",
-    "strategic_discovery_target_js_shortlist",
-    "association_vs_controlled_causality_alignment",
-    "target_specificity_vs_js_specificity_separation",
-    "single_new_gpu_call_hf_acceptance"
-  ],
   "cross_target_feature_limit": 3,
   "cross_target_target_limit": 5,
   "cross_target_default_targets": [
@@ -163,21 +93,7 @@
     "0",
     "x^2"
   ],
-  "live_features_v0_12": [
-    "controlled_evidence_pattern_synthesis",
-    "split_half_discovery_stability",
-    "cross_target_candidate_profile",
-    "missing_discovery_alignment_fallback",
-    "gpu_budget_aware_touched_path_validation"
-  ],
   "discovery_resample_replicates": 32,
-  "live_features_v0_13": [
-    "balanced_bootstrap_candidate_support",
-    "cross_target_effect_concentration",
-    "pairwise_target_preference_shifts",
-    "zero_extra_gpu_evidence_synthesis",
-    "touched_path_only_hf_validation"
-  ],
   "offline_feature_pooling": "prompt-wide max SAE activation across non-padding prompt tokens; final-token sparse activations saved separately",
   "offline_selection_resamples": 128,
   "offline_study_outputs": [
@@ -187,37 +103,12 @@
     "summary.json",
     "report.md"
   ],
-  "offline_features_v0_14": [
-    "promptwide_offline_sae_feature_pooling",
-    "separate_final_token_sparse_activation_artifacts",
-    "activation_resample_candidate_stability",
-    "cross_concept_association_vs_random_normalized_causality",
-    "offline_study_dashboard",
-    "resume_safe_full_study_runner",
-    "cpu_only_analysis_rerun",
-    "offline_artifact_schema_validation"
-  ],
-  "ui_and_runner_features_v0_15": [
-    "project_design_contract",
-    "flat_research_instrument_visual_system",
-    "dual_typeface_hierarchy",
-    "concise_data_first_result_copy",
-    "muted_cross_target_chart_series",
-    "colab_offline_runner_notebook",
-    "task_level_causal_and_feature_set_resume"
-  ],
   "offline_causal_position_policies": [
     "final_token",
     "max_feature_activation"
   ],
   "primary_offline_causal_position_policy": "max_feature_activation",
   "offline_causal_statistical_unit": "causal task; average ablation and amplification within task before paired bootstrap/sign-flip inference",
-  "offline_features_v0_16": [
-    "final_token_vs_max_feature_activation_causal_position_sensitivity",
-    "causal_task_level_statistical_inference",
-    "coverage_separated_from_conditional_effect_strength",
-    "exact_small_sample_sign_flip_tests",
-    "causal_addendum_colab_runner",
-    "position_sensitivity_study_dashboard"
-  ]
+  "release_status": "final",
+  "public_study_artifacts": "measured offline results committed under artifacts/"
 }

scripts/release_check.py

@@ -6,406 +6,218 @@ from collections import Counter
 from pathlib import Path
 
 ROOT = Path(__file__).resolve().parents[1]
-MAX_FILE_SIZE_BYTES = 5_000_000  # 5 MB
+MAX_FILE_SIZE_BYTES = 5_000_000
 
 REQUIRED = [
-    'README.md',
-    'DESIGN.md',
-    'app.py',
-    'requirements.txt',
-    'research_config.json',
-    'featurelens/runtime.py',
-    'featurelens/sae.py',
-    'featurelens/interventions.py',
-    'featurelens/metrics.py',
-    'featurelens/stats.py',
-    'featurelens/study.py',
-    'experiments/run_all.py',
-    'experiments/run_causal.py',
-    'experiments/run_causal_addendum.py',
-    'experiments/run_feature_sets.py',
-    'experiments/analyze_stability.py',
-    'experiments/analyze_study.py',
-    'experiments/run_analysis_only.py',
-    'data/prompts.jsonl',
-    'data/causal_tasks.jsonl',
-    'docs/VALIDATION.md',
-    'docs/OFFLINE_STUDY.md',
-    'docs/COLAB.md',
-    'docs/CAUSAL_ADDENDUM.md',
-    'notebooks/FeatureLens_Offline_Study_Colab.ipynb',
-    'notebooks/FeatureLens_Causal_Addendum_Colab.ipynb',
-    'scripts/ui_smoke.py',
-    'tests/test_offline_study.py',
-    'scripts/validate_artifacts.py',
+    "README.md",
+    "DESIGN.md",
+    "app.py",
+    "requirements.txt",
+    "research_config.json",
+    "featurelens/runtime.py",
+    "featurelens/sae.py",
+    "featurelens/interventions.py",
+    "featurelens/stats.py",
+    "featurelens/study.py",
+    "experiments/run_all.py",
+    "experiments/run_causal.py",
+    "experiments/run_feature_sets.py",
+    "experiments/analyze_stability.py",
+    "experiments/analyze_study.py",
+    "experiments/run_analysis_only.py",
+    "data/prompts.jsonl",
+    "data/causal_tasks.jsonl",
+    "notebooks/README.md",
+    "notebooks/FeatureLens_Offline_Study_Colab.ipynb",
+    "notebooks/FeatureLens_Causal_Addendum_Colab.ipynb",
+    "scripts/ui_smoke.py",
+    "scripts/validate_artifacts.py",
+    "artifacts/feature_catalog.csv",
+    "artifacts/layer_metrics.csv",
+    "artifacts/stability.csv",
+    "artifacts/selection_stability.csv",
+    "artifacts/causal_results_final_token.csv",
+    "artifacts/causal_results_max_active.csv",
+    "artifacts/causal_position_summary.csv",
+    "artifacts/feature_set_results.csv",
+    "artifacts/study_feature_summary.csv",
+    "artifacts/study_summary.json",
+    "artifacts/summary.json",
+    "artifacts/report.md",
 ]
 
 
 def load_jsonl(path: Path) -> list[dict]:
     return [
         json.loads(line)
-        for line in path.read_text(encoding='utf-8').splitlines()
+        for line in path.read_text(encoding="utf-8").splitlines()
         if line.strip()
     ]
 
 
 def repository_candidates() -> list[Path]:
-    """Return tracked files plus untracked files that are not ignored by Git."""
     try:
         result = subprocess.run(
-            ['git', 'ls-files', '--cached', '--others', '--exclude-standard'],
+            ["git", "ls-files", "--cached", "--others", "--exclude-standard"],
             cwd=ROOT,
             capture_output=True,
             text=True,
             check=True,
         )
     except FileNotFoundError as exc:
-        raise SystemExit('Git is required to run the FeatureLens release check.') from exc
+        raise SystemExit("Git is required to run the FeatureLens release check.") from exc
     except subprocess.CalledProcessError as exc:
         raise SystemExit(
-            f'Could not inspect repository files with Git: {exc.stderr.strip()}'
+            f"Could not inspect repository files with Git: {exc.stderr.strip()}"
         ) from exc
 
-    paths: list[Path] = []
-    for relative_path in result.stdout.splitlines():
-        relative_path = relative_path.strip()
-        if not relative_path:
-            continue
-        path = ROOT / relative_path
-        if path.is_file():
-            paths.append(path)
-    return paths
+    return [
+        ROOT / rel
+        for rel in result.stdout.splitlines()
+        if rel.strip() and (ROOT / rel.strip()).is_file()
+    ]
 
 
 def check_required_files() -> None:
-    missing = [path for path in REQUIRED if not (ROOT / path).exists()]
+    missing = [name for name in REQUIRED if not (ROOT / name).exists()]
     if missing:
-        raise SystemExit(f'Missing required files: {missing}')
+        raise SystemExit(f"Missing required files: {missing}")
 
 
 def check_config(config: dict) -> None:
     expected = {
-        'layers': [4, 14, 26],
-        'model_id': 'Qwen/Qwen3-1.7B-Base',
-        'sae_width': 32768,
-        'dose_response_multipliers': [0.0, 0.5, 1.0, 1.5, 2.0, 3.0],
-        'feature_set_sizes': [1, 3, 5],
-        'live_random_controls': 8,
-        'offline_random_controls_default': 8,
-        'concept_contrast_prompts_per_concept': 4,
-        'interaction_feature_limit': 5,
-        'live_geometry_feature_limit': 8,
-        'concept_contrast_pooling': 'max activation across non-padding prompt tokens',
-        'candidate_causal_screen_limit': 8,
-        'candidate_specificity_limit': 3,
+        "model_id": "Qwen/Qwen3-1.7B-Base",
+        "layers": [4, 14, 26],
+        "sae_width": 32768,
+        "discovery_prompts": 224,
+        "causal_tasks": 28,
+        "feature_set_sizes": [1, 3, 5],
+        "live_random_controls": 8,
+        "offline_random_controls_default": 8,
+        "offline_selection_resamples": 128,
+        "offline_causal_position_policies": [
+            "final_token",
+            "max_feature_activation",
+        ],
+        "primary_offline_causal_position_policy": "max_feature_activation",
+        "release_status": "final",
     }
     for key, value in expected.items():
         if config.get(key) != value:
-            raise SystemExit(f'Unexpected {key}: {config.get(key)!r}. Expected {value!r}.')
-
-    required_live_v04 = {
-        'batch_context_null_reference',
-        'random_control_ensemble',
-        'individual_vs_joint_interaction_decomposition',
-        'promptwide_paraphrase_robustness',
-        'controlled_concept_contrast_scan',
-        'copy_tables_with_headers',
-    }
-    actual_live_v04 = set(config.get('live_features_v0_4', []))
-    if actual_live_v04 != required_live_v04:
-        raise SystemExit(
-            'research_config.json live_features_v0_4 mismatch: '
-            f'{sorted(actual_live_v04)}'
-        )
-
-    required_live_v05 = {
-        'wide_centered_responsive_layout',
-        'copy_feedback',
-        'dynamic_height_reflow_observer',
-        'promptwide_concept_contrast_scan',
-        'feature_token_activation_trace',
-        'contrastive_continuation_preference_test',
-        'feature_decoder_geometry',
-    }
-    actual_live_v05 = set(config.get('live_features_v0_5', []))
-    if actual_live_v05 != required_live_v05:
-        raise SystemExit(
-            'research_config.json live_features_v0_5 mismatch: '
-            f'{sorted(actual_live_v05)}'
-        )
-
-    required_live_v06 = {
-        'start_here_plain_language_onboarding',
-        'persistent_workbench_context_banner',
-        'explicit_per_experiment_feature_selectors',
-        'plot_fullscreen_and_export_controls',
-        'consistent_heading_and_table_typography',
-        'concept_guided_candidate_feature_discovery',
-        'completion_cue_sensitivity_scan',
-    }
-    actual_live_v06 = set(config.get('live_features_v0_6', []))
-    if actual_live_v06 != required_live_v06:
-        raise SystemExit(
-            'research_config.json live_features_v0_6 mismatch: '
-            f'{sorted(actual_live_v06)}'
-        )
-
-    required_live_v07 = {
-        'cleaned_nonaccordion_experiment_layout',
-        'focused_fullscreen_modal_for_tables_and_plots',
-        'descriptive_plot_export_filenames',
-        'german_language_control_concept',
-        'balanced_candidate_ranking_and_current_prompt_compatibility',
-        'click_to_select_candidate_rows',
-        'completion_cue_context_matrix',
-    }
-    actual_live_v07 = set(config.get('live_features_v0_7', []))
-    if actual_live_v07 != required_live_v07:
-        raise SystemExit(
-            'research_config.json live_features_v0_7 mismatch: '
-            f'{sorted(actual_live_v07)}'
-        )
-
-    required_live_v08 = {
-        'bounded_plot_focus_overlay_with_scroll_restore',
-        'explicit_result_table_headings',
-        'standalone_dose_response_target_and_feature_inputs',
-        'causal_ready_current_token_candidate_ranking',
-        'cue_dominance_specificity_interpretation',
-        'muted_cue_context_plot_palette',
-    }
-    actual_live_v08 = set(config.get('live_features_v0_8', []))
-    if actual_live_v08 != required_live_v08:
-        raise SystemExit(
-            'research_config.json live_features_v0_8 mismatch: ' f'{sorted(actual_live_v08)}'
-        )
-
-    required_live_v09 = {
-        'in_place_aspect_preserving_plot_and_table_focus',
-        'compact_table_heading_alignment',
-        'concise_independent_dose_response_copy',
-        'batched_candidate_causal_triage',
-        'gpu_budget_aware_hf_validation_scope',
-    }
-    actual_live_v09 = set(config.get('live_features_v0_9', []))
-    if actual_live_v09 != required_live_v09:
-        raise SystemExit(
-            'research_config.json live_features_v0_9 mismatch: ' f'{sorted(actual_live_v09)}'
-        )
-
-    required_live_v10 = {
-        'discovery_to_causality_alignment_table',
-        'association_evidence_vs_target_effect_scatter',
-        'descriptive_spearman_concordance_summary',
-        'target_effect_vs_distribution_shift_rank_separation',
-        'no_extra_gpu_candidate_synthesis',
-    }
-    actual_live_v10 = set(config.get('live_features_v0_10', []))
-    if actual_live_v10 != required_live_v10:
-        raise SystemExit(
-            'research_config.json live_features_v0_10 mismatch: ' f'{sorted(actual_live_v10)}'
-        )
-
-    required_live_v11 = {
-        'controlled_multi_candidate_random_specificity_screen',
-        'strategic_discovery_target_js_shortlist',
-        'association_vs_controlled_causality_alignment',
-        'target_specificity_vs_js_specificity_separation',
-        'single_new_gpu_call_hf_acceptance',
-    }
-    actual_live_v11 = set(config.get('live_features_v0_11', []))
-    if actual_live_v11 != required_live_v11:
-        raise SystemExit(
-            'research_config.json live_features_v0_11 mismatch: ' f'{sorted(actual_live_v11)}'
-        )
-
-    required_live_v12 = {
-        'controlled_evidence_pattern_synthesis',
-        'split_half_discovery_stability',
-        'cross_target_candidate_profile',
-        'missing_discovery_alignment_fallback',
-        'gpu_budget_aware_touched_path_validation',
-    }
-    actual_live_v12 = set(config.get('live_features_v0_12', []))
-    if actual_live_v12 != required_live_v12:
-        raise SystemExit(
-            'research_config.json live_features_v0_12 mismatch: ' f'{sorted(actual_live_v12)}'
-        )
-
-    required_live_v13 = {
-        'balanced_bootstrap_candidate_support',
-        'cross_target_effect_concentration',
-        'pairwise_target_preference_shifts',
-        'zero_extra_gpu_evidence_synthesis',
-        'touched_path_only_hf_validation',
-    }
-    actual_live_v13 = set(config.get('live_features_v0_13', []))
-    if actual_live_v13 != required_live_v13:
-        raise SystemExit(
-            'research_config.json live_features_v0_13 mismatch: ' f'{sorted(actual_live_v13)}'
-        )
-    required_offline_v14 = {
-        'promptwide_offline_sae_feature_pooling',
-        'separate_final_token_sparse_activation_artifacts',
-        'activation_resample_candidate_stability',
-        'cross_concept_association_vs_random_normalized_causality',
-        'offline_study_dashboard',
-        'resume_safe_full_study_runner',
-        'cpu_only_analysis_rerun',
-        'offline_artifact_schema_validation',
-    }
-    actual_offline_v14 = set(config.get('offline_features_v0_14', []))
-    if actual_offline_v14 != required_offline_v14:
-        raise SystemExit(
-            'research_config.json offline_features_v0_14 mismatch: '
-            f'{sorted(actual_offline_v14)}'
-        )
-
-    required_v15 = {
-        'project_design_contract',
-        'flat_research_instrument_visual_system',
-        'dual_typeface_hierarchy',
-        'concise_data_first_result_copy',
-        'muted_cross_target_chart_series',
-        'colab_offline_runner_notebook',
-        'task_level_causal_and_feature_set_resume',
-    }
-    actual_v15 = set(config.get('ui_and_runner_features_v0_15', []))
-    if actual_v15 != required_v15:
-        raise SystemExit(
-            'research_config.json ui_and_runner_features_v0_15 mismatch: '
-            f'{sorted(actual_v15)}'
-        )
-
-    required_v16 = {
-        'final_token_vs_max_feature_activation_causal_position_sensitivity',
-        'causal_task_level_statistical_inference',
-        'coverage_separated_from_conditional_effect_strength',
-        'exact_small_sample_sign_flip_tests',
-        'causal_addendum_colab_runner',
-        'position_sensitivity_study_dashboard',
-    }
-    actual_v16 = set(config.get('offline_features_v0_16', []))
-    if actual_v16 != required_v16:
-        raise SystemExit(
-            'research_config.json offline_features_v0_16 mismatch: '
-            f'{sorted(actual_v16)}'
-        )
-    if config.get('offline_causal_position_policies') != ['final_token', 'max_feature_activation']:
-        raise SystemExit('Offline causal position policies must be final_token and max_feature_activation.')
-    if config.get('primary_offline_causal_position_policy') != 'max_feature_activation':
-        raise SystemExit('Primary offline causal position policy must be max_feature_activation.')
-    if config.get('offline_selection_resamples') != 128:
-        raise SystemExit('Offline selection resamples must be 128.')
-    if 'prompt-wide' not in str(config.get('offline_feature_pooling', '')):
-        raise SystemExit('Offline feature pooling must be prompt-wide.')
+            raise SystemExit(
+                f"Unexpected {key}: {config.get(key)!r}. Expected {value!r}."
+            )
 
-    if config.get('discovery_resample_replicates') != 32:
-        raise SystemExit('Discovery live resample count must be 32.')
-    if config.get('cross_target_feature_limit') != 3 or config.get('cross_target_target_limit') != 5:
-        raise SystemExit('Cross-target live limits must be 3 features and 5 targets.')
+    if "prompt-wide" not in str(config.get("offline_feature_pooling", "")):
+        raise SystemExit("Offline SAE concept evidence must use prompt-wide pooling.")
 
-    if 'german_language' not in config.get('concepts', []) or 'french_language' in config.get('concepts', []):
-        raise SystemExit('research_config.json must use german_language and must not contain french_language.')
+    concepts = config.get("concepts", [])
+    if "german_language" not in concepts or "french_language" in concepts:
+        raise SystemExit("Controlled language concept must be german_language.")
 
 
 def check_datasets(config: dict) -> tuple[list[dict], list[dict]]:
-    prompts = load_jsonl(ROOT / 'data' / 'prompts.jsonl')
-    causal = load_jsonl(ROOT / 'data' / 'causal_tasks.jsonl')
+    prompts = load_jsonl(ROOT / "data" / "prompts.jsonl")
+    causal = load_jsonl(ROOT / "data" / "causal_tasks.jsonl")
 
-    if len(prompts) != config.get('discovery_prompts'):
+    if len(prompts) != config["discovery_prompts"]:
         raise SystemExit(
-            f'Discovery prompt count mismatch: found {len(prompts)}, '
-            f'expected {config.get("discovery_prompts")}.'
+            f"Discovery prompt count mismatch: {len(prompts)} != "
+            f"{config['discovery_prompts']}."
         )
-    if len(causal) != config.get('causal_tasks'):
+    if len(causal) != config["causal_tasks"]:
         raise SystemExit(
-            f'Causal task count mismatch: found {len(causal)}, '
-            f'expected {config.get("causal_tasks")}.'
+            f"Causal task count mismatch: {len(causal)} != {config['causal_tasks']}."
         )
 
-    concept_counts = Counter(row['concept'] for row in prompts)
-    if set(concept_counts) != set(config.get('concepts', [])):
-        raise SystemExit('Discovery dataset concepts do not match research_config.json.')
+    concept_counts = Counter(row["concept"] for row in prompts)
+    if set(concept_counts) != set(config["concepts"]):
+        raise SystemExit("Discovery concepts do not match research_config.json.")
     if len(set(concept_counts.values())) != 1:
-        raise SystemExit(f'Discovery concepts are not balanced: {dict(concept_counts)}')
+        raise SystemExit(f"Discovery concepts are not balanced: {dict(concept_counts)}")
 
-    pair_counts = Counter(row['pair_id'] for row in prompts)
+    pair_counts = Counter(row["pair_id"] for row in prompts)
     if set(pair_counts.values()) != {2}:
-        raise SystemExit('Every discovery paraphrase pair must contain exactly two prompts.')
+        raise SystemExit("Every discovery paraphrase pair must contain exactly two prompts.")
 
     return prompts, causal
 
 
-def check_oversized_files() -> None:
-    oversized: list[str] = []
-    for path in repository_candidates():
-        size_bytes = path.stat().st_size
-        if size_bytes > MAX_FILE_SIZE_BYTES:
-            relative = path.relative_to(ROOT)
-            oversized.append(f'{relative} ({size_bytes / 1_000_000:.1f} MB)')
+def check_study_summary() -> None:
+    study = json.loads(
+        (ROOT / "artifacts" / "study_summary.json").read_text(encoding="utf-8")
+    )
+    summary = json.loads(
+        (ROOT / "artifacts" / "summary.json").read_text(encoding="utf-8")
+    )
 
-    if oversized:
-        formatted = '\n  - '.join(oversized)
-        raise SystemExit(
-            'Repository contains unexpectedly large tracked/unignored candidates:\n'
-            f'  - {formatted}\n\n'
-            'If a file is a legitimate local artifact, add it to .gitignore. Model weights, '
-            'SAE checkpoints, activation dumps, virtual environments, and caches should not be committed.'
-        )
+    if study.get("primary_causal_position_policy") != "max_feature_activation":
+        raise SystemExit("Committed study must use max_feature_activation as primary policy.")
+    if "causal task" not in str(study.get("causal_statistical_unit", "")).lower():
+        raise SystemExit("Committed study must document causal-task-level inference.")
+    if float(study.get("max_active_feature_coverage", 0.0)) <= float(
+        study.get("final_token_feature_coverage", 0.0)
+    ):
+        raise SystemExit("Expected max-active coverage to exceed final-token coverage.")
+    if float(study.get("max_active_target_specificity_ratio", 0.0)) <= 1.0:
+        raise SystemExit("Committed max-active study specificity ratio is invalid.")
+
+    headline = str(summary.get("headline", ""))
+    if "0.962" not in headline or "2.33" not in headline:
+        raise SystemExit("Committed summary.json does not contain the finalized measured headline.")
 
 
 def check_readme() -> None:
-    readme = (ROOT / 'README.md').read_text(encoding='utf-8')
-    required_strings = [
-        'sdk: gradio',
-        'sdk_version: "6.24.0"',
-        'Qwen/Qwen3-1.7B-Base',
-        'Qwen-Scope',
-        'random controls',
-        'prompt-wide',
-        'offline study',
-        '-m experiments.run_all --resume',
-        '--activation-batch-size',
-        'validate_artifacts',
-        'FeatureLens_Offline_Study_Colab.ipynb',
-        'FeatureLens_Causal_Addendum_Colab.ipynb',
-        'max-feature-activation',
-        'causal task',
-        'DESIGN.md',
+    readme = (ROOT / "README.md").read_text(encoding="utf-8")
+    required = [
+        "0.962 held-out AUROC",
+        "2.33×",
+        "28.6%",
+        "82.1%",
+        "notebooks/FeatureLens_Offline_Study_Colab.ipynb",
+        "artifacts/report.md",
     ]
-    missing = [value for value in required_strings if value.lower() not in readme.lower()]
+    missing = [text for text in required if text not in readme]
     if missing:
-        raise SystemExit(f'README.md is missing required v0.16 content: {missing}')
+        raise SystemExit(f"README.md missing finalized study content: {missing}")
 
-    # Public README should not lead with release-train marketing. Version history belongs in CHANGELOG.
-    if '> **v0.' in readme or '## v0.' in readme:
-        raise SystemExit('README.md should not contain visible release-announcement/version-history sections.')
 
+def check_oversized_files() -> None:
+    oversized: list[str] = []
+    for path in repository_candidates():
+        size = path.stat().st_size
+        if size > MAX_FILE_SIZE_BYTES:
+            oversized.append(
+                f"{path.relative_to(ROOT)} ({size / 1_000_000:.1f} MB)"
+            )
 
-def check_pyproject() -> None:
-    text = (ROOT / 'pyproject.toml').read_text(encoding='utf-8')
-    if 'version = "0.16.0"' not in text:
-        raise SystemExit('pyproject.toml must declare version 0.16.0.')
+    if oversized:
+        formatted = "\n  - ".join(oversized)
+        raise SystemExit(
+            "Repository contains unexpectedly large tracked/unignored candidates:\n"
+            f"  - {formatted}\n\n"
+            "Model weights, SAE checkpoints, activation dumps, virtual environments, "
+            "and caches should not be committed."
+        )
 
 
 def main() -> None:
     check_required_files()
-    config = json.loads((ROOT / 'research_config.json').read_text(encoding='utf-8'))
+
+    config = json.loads((ROOT / "research_config.json").read_text(encoding="utf-8"))
     check_config(config)
     prompts, causal = check_datasets(config)
-    check_oversized_files()
+    check_study_summary()
     check_readme()
-    check_pyproject()
+    check_oversized_files()
 
-    print('FeatureLens release check: PASS')
-    print(f'  discovery prompts: {len(prompts)}')
-    print(f'  causal tasks: {len(causal)}')
-    print(f'  layers: {config["layers"]}')
-    print(f'  feature-set sizes: {config["feature_set_sizes"]}')
-    print(f'  random controls: {config["live_random_controls"]}')
-    print('  release: v0.16.0')
+    print("FeatureLens release check: PASS")
+    print(f"  discovery prompts: {len(prompts)}")
+    print(f"  causal tasks: {len(causal)}")
+    print(f"  layers: {config['layers']}")
+    print("  committed offline study: complete")
+    print("  release: 1.0.0")
 
 
-if __name__ == '__main__':
+if __name__ == "__main__":
     main()