facts: layer 10 is notation and entity tracking (formatting is layer 20), most-active steering slightly hurts BBQ Ambig, seven benchmarks, the critic curves rise, GRPO is critic-free rather than preference-based, the critic bottleneck is inferred, and HarmBench and XSTest get the citations they never had
Browse files- app/src/content/article.mdx +10 -10
app/src/content/article.mdx
CHANGED
|
@@ -53,7 +53,7 @@ Watching a feature light up tells you it was present. It does not tell you it di
|
|
| 53 |
|
| 54 |
**Adaptive Feature Masking** encourages diverse feature discovery while preserving single-feature interpretability. The framework yields new analysis capabilities: **branch point tracking** locates tokens where feature choice determines output correctness; **critic trajectory analysis** separates policy limitations from value estimation errors; and **layer-wise comparison** reveals syntactic features in early layers and semantic features in later layers [@tenney-etal-2019-bert].
|
| 55 |
|
| 56 |
-
On [Gemma 2](https://texonom.com/8aa07c8eba2c4d4ba61213b9a4f711c5) 2B [@gemmateam2024gemma2improvingopen] across [MMLU](https://texonom.com/d2a91b61ee71491d9e6e284a5bdcc709) [@hendrycks2021measuring], [BBQ](https://texonom.com/14bc3c96247d81038bb3c4aa92f78a0d) [@parrish-etal-2022-bbq], GSM8K [@cobbe2021trainingverifierssolvemath], HarmBench, and [XSTest](https://texonom.com/237c3c96247d8090a166c4cbee0dc287), CRL achieves consistent improvements while providing per-token intervention logs. These results establish learned feature steering as a [mechanistic interpretability](https://texonom.com/e40305fd878c46ca85d99ea93ee9a2ff) tool that complements static feature analysis with dynamic intervention probes.
|
| 57 |
|
| 58 |
<Wide>
|
| 59 |
<HtmlEmbed src="steering-demo.html" title="Token-Level Steering Simulation" desc="Figure 2: Watch how different SAE features steer token generation along divergent paths. The same prompt context leads to correct or incorrect answers depending on which feature is amplified. Feature IDs link to Neuronpedia for detailed activation analysis." />
|
|
@@ -127,7 +127,7 @@ During training, actions are sampled from a categorical distribution over softma
|
|
| 127 |
The **critic network** $V_\phi$ estimates the state [value function](https://texonom.com/d9d3467fea994443b2336fd7958dc00b), enabling advantage estimation for [PPO](https://texonom.com/87ce8ebe81f84ac1a7617b1f6def9e26) [@schulman2017proximal] training. Crucially, the critic also serves as an *analysis tool* β its learned value estimates reveal where the policy succeeds and fails.
|
| 128 |
|
| 129 |
<Note variant="info">
|
| 130 |
-
**Why PPO over [DPO](https://texonom.com/d4099259ad6d443e8f9c2c9d1b45d6db)/[GRPO](https://texonom.com/187c3c96247d80dfb88bceaf4abad43c)?** [@shao2024deepseekmathpushinglimitsmathematical] PPO provides an explicit value function that we analyze as a diagnostic tool. The critic trajectories reveal bottleneck patterns that would be invisible with
|
| 131 |
</Note>
|
| 132 |
|
| 133 |
### Partial Observability
|
|
@@ -160,7 +160,7 @@ AFM is particularly important for multi-token tasks like GSM8K, where the reason
|
|
| 160 |
|
| 161 |
## Branch Analysis: Where Steering Decides the Answer
|
| 162 |
|
| 163 |
-
|
| 164 |
|
| 165 |
### How Branch Analysis Works
|
| 166 |
|
|
@@ -196,7 +196,7 @@ Comparing features selected at Layer 10 vs Layer 20 reveals a striking pattern:
|
|
| 196 |
</div>
|
| 197 |
</div>
|
| 198 |
|
| 199 |
-
Layer 10 features capture **concrete syntactic patterns** like surface-level notation
|
| 200 |
|
| 201 |
<Note variant="danger">
|
| 202 |
**Feature labels have limited predictive value.** A feature labeled "emotional relationships" can solve speed calculations, while "mathematical notation" fails on some arithmetic. So learned steering effects diverge from semantic interpretations derived from activation patterns alone.
|
|
@@ -207,7 +207,7 @@ Out of 8 analyzed branch pairs, Layer 10 features succeed on **3 cases** (discre
|
|
| 207 |
|
| 208 |
## Empirical Validation
|
| 209 |
|
| 210 |
-
To confirm that the learned policy selects *meaningful* features β not arbitrary ones β we compare CRL against baselines on
|
| 211 |
|
| 212 |
<HtmlEmbed src="results-table.html" title="CRL Steering Results" desc="Table 1: Before vs. after CRL steering. Toggle between Gemma 2 2B and [LLaMA](https://texonom.com/f2b6721202d44d469add84d8a366809c) 3.1 8B to compare across tasks and architectures." />
|
| 213 |
|
|
@@ -225,7 +225,7 @@ To confirm that the learned policy selects *meaningful* features β not arbitra
|
|
| 225 |
| Most-Active Feature | 59.94 | 48.03 |
|
| 226 |
| **CRL (Ours)** | **65.86** | **49.12** |
|
| 227 |
|
| 228 |
-
Random feature steering can actually *hurt* performance (β1.81 on BBQ Ambig), while selecting the most-active SAE feature
|
| 229 |
|
| 230 |
|
| 231 |
## Critic Trajectory Analysis
|
|
@@ -241,9 +241,9 @@ The learned critic (value function) reveals *where* in the generation process th
|
|
| 241 |
For multi-token generation (GSM8K), critic trajectories reveal the *temporal dynamics* of steering:
|
| 242 |
|
| 243 |
- **Correct samples** (green): Higher gradients with divergence around token 200β400
|
| 244 |
-
- **Incorrect samples** (red):
|
| 245 |
- **Corrected samples** (blue): Show value increases at the ~400-token point, indicating interventions become more effective in later reasoning stages
|
| 246 |
-
- **Misguided samples** (yellow):
|
| 247 |
|
| 248 |
|
| 249 |
## Feature Interpretability
|
|
@@ -309,9 +309,9 @@ CRL provides three diagnostic capabilities that complement static feature analys
|
|
| 309 |
|
| 310 |
**1. Branch tracking** locates critical decision points β tokens where the choice of a single SAE feature determines whether the output is correct or incorrect. This provides *causal* evidence for feature importance, going beyond correlational activation analysis.
|
| 311 |
|
| 312 |
-
**2. Critic trajectories** separate policy from value-estimation
|
| 313 |
|
| 314 |
-
**3. Layer comparison** reveals a syntactic-to-semantic gradient: early layers encode concrete patterns (notation,
|
| 315 |
|
| 316 |
### Limitations
|
| 317 |
|
|
|
|
| 53 |
|
| 54 |
**Adaptive Feature Masking** encourages diverse feature discovery while preserving single-feature interpretability. The framework yields new analysis capabilities: **branch point tracking** locates tokens where feature choice determines output correctness; **critic trajectory analysis** separates policy limitations from value estimation errors; and **layer-wise comparison** reveals syntactic features in early layers and semantic features in later layers [@tenney-etal-2019-bert].
|
| 55 |
|
| 56 |
+
On [Gemma 2](https://texonom.com/8aa07c8eba2c4d4ba61213b9a4f711c5) 2B [@gemmateam2024gemma2improvingopen] across [MMLU](https://texonom.com/d2a91b61ee71491d9e6e284a5bdcc709) [@hendrycks2021measuring], [BBQ](https://texonom.com/14bc3c96247d81038bb3c4aa92f78a0d) [@parrish-etal-2022-bbq], GSM8K [@cobbe2021trainingverifierssolvemath], HarmBench [@mazeika2024harmbench], and [XSTest](https://texonom.com/237c3c96247d8090a166c4cbee0dc287) [@rottger-etal-2024-xstest], CRL achieves consistent improvements while providing per-token intervention logs. These results establish learned feature steering as a [mechanistic interpretability](https://texonom.com/e40305fd878c46ca85d99ea93ee9a2ff) tool that complements static feature analysis with dynamic intervention probes.
|
| 57 |
|
| 58 |
<Wide>
|
| 59 |
<HtmlEmbed src="steering-demo.html" title="Token-Level Steering Simulation" desc="Figure 2: Watch how different SAE features steer token generation along divergent paths. The same prompt context leads to correct or incorrect answers depending on which feature is amplified. Feature IDs link to Neuronpedia for detailed activation analysis." />
|
|
|
|
| 127 |
The **critic network** $V_\phi$ estimates the state [value function](https://texonom.com/d9d3467fea994443b2336fd7958dc00b), enabling advantage estimation for [PPO](https://texonom.com/87ce8ebe81f84ac1a7617b1f6def9e26) [@schulman2017proximal] training. Crucially, the critic also serves as an *analysis tool* β its learned value estimates reveal where the policy succeeds and fails.
|
| 128 |
|
| 129 |
<Note variant="info">
|
| 130 |
+
**Why PPO over [DPO](https://texonom.com/d4099259ad6d443e8f9c2c9d1b45d6db)/[GRPO](https://texonom.com/187c3c96247d80dfb88bceaf4abad43c)?** [@shao2024deepseekmathpushinglimitsmathematical] PPO provides an explicit value function that we analyze as a diagnostic tool. The critic trajectories reveal bottleneck patterns that would be invisible with critic-free methods.
|
| 131 |
</Note>
|
| 132 |
|
| 133 |
### Partial Observability
|
|
|
|
| 160 |
|
| 161 |
## Branch Analysis: Where Steering Decides the Answer
|
| 162 |
|
| 163 |
+
**Branch analysis** is where CRL's interpretability capabilities matter most. It identifies tokens where the *choice of feature* determines whether the model produces a correct or incorrect answer.
|
| 164 |
|
| 165 |
### How Branch Analysis Works
|
| 166 |
|
|
|
|
| 196 |
</div>
|
| 197 |
</div>
|
| 198 |
|
| 199 |
+
Layer 10 features capture **concrete syntactic patterns** like surface-level notation and entity tracking. Layer 20 features encode **abstract semantics** like planning and strategic reasoning.
|
| 200 |
|
| 201 |
<Note variant="danger">
|
| 202 |
**Feature labels have limited predictive value.** A feature labeled "emotional relationships" can solve speed calculations, while "mathematical notation" fails on some arithmetic. So learned steering effects diverge from semantic interpretations derived from activation patterns alone.
|
|
|
|
| 207 |
|
| 208 |
## Empirical Validation
|
| 209 |
|
| 210 |
+
To confirm that the learned policy selects *meaningful* features β not arbitrary ones β we compare CRL against baselines on seven benchmarks. The goal is not performance gain per se, but to validate that the policy has learned a non-trivial feature selection strategy worth analyzing.
|
| 211 |
|
| 212 |
<HtmlEmbed src="results-table.html" title="CRL Steering Results" desc="Table 1: Before vs. after CRL steering. Toggle between Gemma 2 2B and [LLaMA](https://texonom.com/f2b6721202d44d469add84d8a366809c) 3.1 8B to compare across tasks and architectures." />
|
| 213 |
|
|
|
|
| 225 |
| Most-Active Feature | 59.94 | 48.03 |
|
| 226 |
| **CRL (Ours)** | **65.86** | **49.12** |
|
| 227 |
|
| 228 |
+
Random feature steering can actually *hurt* performance (β1.81 on BBQ Ambig), while selecting the most-active SAE feature slightly hurts on BBQ Ambig. The learned policy discovers features whose *causal effects* differ from their activation magnitude β precisely the gap that static analysis cannot reveal.
|
| 229 |
|
| 230 |
|
| 231 |
## Critic Trajectory Analysis
|
|
|
|
| 241 |
For multi-token generation (GSM8K), critic trajectories reveal the *temporal dynamics* of steering:
|
| 242 |
|
| 243 |
- **Correct samples** (green): Higher gradients with divergence around token 200β400
|
| 244 |
+
- **Incorrect samples** (red): Rising values below the correct samples
|
| 245 |
- **Corrected samples** (blue): Show value increases at the ~400-token point, indicating interventions become more effective in later reasoning stages
|
| 246 |
+
- **Misguided samples** (yellow): Rising values that drop late β the steering seemed promising but ultimately led astray
|
| 247 |
|
| 248 |
|
| 249 |
## Feature Interpretability
|
|
|
|
| 309 |
|
| 310 |
**1. Branch tracking** locates critical decision points β tokens where the choice of a single SAE feature determines whether the output is correct or incorrect. This provides *causal* evidence for feature importance, going beyond correlational activation analysis.
|
| 311 |
|
| 312 |
+
**2. Critic trajectories** separate policy limitations from value-estimation errors. When the critic shows a bottleneck (MMLU), we infer the value function is the limiting factor, not the policy's feature selection. When the critic shows clear discrimination (BBQ), the policy-critic coordination is effective.
|
| 313 |
|
| 314 |
+
**3. Layer comparison** reveals a syntactic-to-semantic gradient: early layers encode concrete patterns (notation, entity tracking), while later layers capture abstract structure (strategic planning).
|
| 315 |
|
| 316 |
### Limitations
|
| 317 |
|