seonglae commited on
Commit
83afa04
Β·
1 Parent(s): f3e0791

facts: layer 10 is notation and entity tracking (formatting is layer 20), most-active steering slightly hurts BBQ Ambig, seven benchmarks, the critic curves rise, GRPO is critic-free rather than preference-based, the critic bottleneck is inferred, and HarmBench and XSTest get the citations they never had

Browse files
Files changed (1) hide show
  1. app/src/content/article.mdx +10 -10
app/src/content/article.mdx CHANGED
@@ -53,7 +53,7 @@ Watching a feature light up tells you it was present. It does not tell you it di
53
 
54
  **Adaptive Feature Masking** encourages diverse feature discovery while preserving single-feature interpretability. The framework yields new analysis capabilities: **branch point tracking** locates tokens where feature choice determines output correctness; **critic trajectory analysis** separates policy limitations from value estimation errors; and **layer-wise comparison** reveals syntactic features in early layers and semantic features in later layers [@tenney-etal-2019-bert].
55
 
56
- On [Gemma 2](https://texonom.com/8aa07c8eba2c4d4ba61213b9a4f711c5) 2B [@gemmateam2024gemma2improvingopen] across [MMLU](https://texonom.com/d2a91b61ee71491d9e6e284a5bdcc709) [@hendrycks2021measuring], [BBQ](https://texonom.com/14bc3c96247d81038bb3c4aa92f78a0d) [@parrish-etal-2022-bbq], GSM8K [@cobbe2021trainingverifierssolvemath], HarmBench, and [XSTest](https://texonom.com/237c3c96247d8090a166c4cbee0dc287), CRL achieves consistent improvements while providing per-token intervention logs. These results establish learned feature steering as a [mechanistic interpretability](https://texonom.com/e40305fd878c46ca85d99ea93ee9a2ff) tool that complements static feature analysis with dynamic intervention probes.
57
 
58
  <Wide>
59
  <HtmlEmbed src="steering-demo.html" title="Token-Level Steering Simulation" desc="Figure 2: Watch how different SAE features steer token generation along divergent paths. The same prompt context leads to correct or incorrect answers depending on which feature is amplified. Feature IDs link to Neuronpedia for detailed activation analysis." />
@@ -127,7 +127,7 @@ During training, actions are sampled from a categorical distribution over softma
127
  The **critic network** $V_\phi$ estimates the state [value function](https://texonom.com/d9d3467fea994443b2336fd7958dc00b), enabling advantage estimation for [PPO](https://texonom.com/87ce8ebe81f84ac1a7617b1f6def9e26) [@schulman2017proximal] training. Crucially, the critic also serves as an *analysis tool* β€” its learned value estimates reveal where the policy succeeds and fails.
128
 
129
  <Note variant="info">
130
- **Why PPO over [DPO](https://texonom.com/d4099259ad6d443e8f9c2c9d1b45d6db)/[GRPO](https://texonom.com/187c3c96247d80dfb88bceaf4abad43c)?** [@shao2024deepseekmathpushinglimitsmathematical] PPO provides an explicit value function that we analyze as a diagnostic tool. The critic trajectories reveal bottleneck patterns that would be invisible with preference-based methods.
131
  </Note>
132
 
133
  ### Partial Observability
@@ -160,7 +160,7 @@ AFM is particularly important for multi-token tasks like GSM8K, where the reason
160
 
161
  ## Branch Analysis: Where Steering Decides the Answer
162
 
163
- This is where CRL's interpretability capabilities matter most. **Branch analysis** identifies tokens where the *choice of feature* determines whether the model produces a correct or incorrect answer.
164
 
165
  ### How Branch Analysis Works
166
 
@@ -196,7 +196,7 @@ Comparing features selected at Layer 10 vs Layer 20 reveals a striking pattern:
196
  </div>
197
  </div>
198
 
199
- Layer 10 features capture **concrete syntactic patterns** like surface-level notation, entity tracking and formatting. Layer 20 features encode **abstract semantics** like planning and strategic reasoning.
200
 
201
  <Note variant="danger">
202
  **Feature labels have limited predictive value.** A feature labeled "emotional relationships" can solve speed calculations, while "mathematical notation" fails on some arithmetic. So learned steering effects diverge from semantic interpretations derived from activation patterns alone.
@@ -207,7 +207,7 @@ Out of 8 analyzed branch pairs, Layer 10 features succeed on **3 cases** (discre
207
 
208
  ## Empirical Validation
209
 
210
- To confirm that the learned policy selects *meaningful* features β€” not arbitrary ones β€” we compare CRL against baselines on five benchmarks. The goal is not performance gain per se, but to validate that the policy has learned a non-trivial feature selection strategy worth analyzing.
211
 
212
  <HtmlEmbed src="results-table.html" title="CRL Steering Results" desc="Table 1: Before vs. after CRL steering. Toggle between Gemma 2 2B and [LLaMA](https://texonom.com/f2b6721202d44d469add84d8a366809c) 3.1 8B to compare across tasks and architectures." />
213
 
@@ -225,7 +225,7 @@ To confirm that the learned policy selects *meaningful* features β€” not arbitra
225
  | Most-Active Feature | 59.94 | 48.03 |
226
  | **CRL (Ours)** | **65.86** | **49.12** |
227
 
228
- Random feature steering can actually *hurt* performance (βˆ’1.81 on BBQ Ambig), while selecting the most-active SAE feature barely helps on BBQ Ambig. The learned policy discovers features whose *causal effects* differ from their activation magnitude β€” precisely the gap that static analysis cannot reveal.
229
 
230
 
231
  ## Critic Trajectory Analysis
@@ -241,9 +241,9 @@ The learned critic (value function) reveals *where* in the generation process th
241
  For multi-token generation (GSM8K), critic trajectories reveal the *temporal dynamics* of steering:
242
 
243
  - **Correct samples** (green): Higher gradients with divergence around token 200–400
244
- - **Incorrect samples** (red): Flat or declining values throughout
245
  - **Corrected samples** (blue): Show value increases at the ~400-token point, indicating interventions become more effective in later reasoning stages
246
- - **Misguided samples** (yellow): Initially high values that drop β€” the steering seemed promising but ultimately led astray
247
 
248
 
249
  ## Feature Interpretability
@@ -309,9 +309,9 @@ CRL provides three diagnostic capabilities that complement static feature analys
309
 
310
  **1. Branch tracking** locates critical decision points β€” tokens where the choice of a single SAE feature determines whether the output is correct or incorrect. This provides *causal* evidence for feature importance, going beyond correlational activation analysis.
311
 
312
- **2. Critic trajectories** separate policy from value-estimation limitations. When the critic shows a bottleneck (MMLU), we know the value function is the limiting factor, not the policy's feature selection. When the critic shows clear discrimination (BBQ), the policy-critic coordination is effective.
313
 
314
- **3. Layer comparison** reveals a syntactic-to-semantic gradient: early layers encode concrete patterns (notation, formatting), while later layers capture abstract structure (strategic planning).
315
 
316
  ### Limitations
317
 
 
53
 
54
  **Adaptive Feature Masking** encourages diverse feature discovery while preserving single-feature interpretability. The framework yields new analysis capabilities: **branch point tracking** locates tokens where feature choice determines output correctness; **critic trajectory analysis** separates policy limitations from value estimation errors; and **layer-wise comparison** reveals syntactic features in early layers and semantic features in later layers [@tenney-etal-2019-bert].
55
 
56
+ On [Gemma 2](https://texonom.com/8aa07c8eba2c4d4ba61213b9a4f711c5) 2B [@gemmateam2024gemma2improvingopen] across [MMLU](https://texonom.com/d2a91b61ee71491d9e6e284a5bdcc709) [@hendrycks2021measuring], [BBQ](https://texonom.com/14bc3c96247d81038bb3c4aa92f78a0d) [@parrish-etal-2022-bbq], GSM8K [@cobbe2021trainingverifierssolvemath], HarmBench [@mazeika2024harmbench], and [XSTest](https://texonom.com/237c3c96247d8090a166c4cbee0dc287) [@rottger-etal-2024-xstest], CRL achieves consistent improvements while providing per-token intervention logs. These results establish learned feature steering as a [mechanistic interpretability](https://texonom.com/e40305fd878c46ca85d99ea93ee9a2ff) tool that complements static feature analysis with dynamic intervention probes.
57
 
58
  <Wide>
59
  <HtmlEmbed src="steering-demo.html" title="Token-Level Steering Simulation" desc="Figure 2: Watch how different SAE features steer token generation along divergent paths. The same prompt context leads to correct or incorrect answers depending on which feature is amplified. Feature IDs link to Neuronpedia for detailed activation analysis." />
 
127
  The **critic network** $V_\phi$ estimates the state [value function](https://texonom.com/d9d3467fea994443b2336fd7958dc00b), enabling advantage estimation for [PPO](https://texonom.com/87ce8ebe81f84ac1a7617b1f6def9e26) [@schulman2017proximal] training. Crucially, the critic also serves as an *analysis tool* β€” its learned value estimates reveal where the policy succeeds and fails.
128
 
129
  <Note variant="info">
130
+ **Why PPO over [DPO](https://texonom.com/d4099259ad6d443e8f9c2c9d1b45d6db)/[GRPO](https://texonom.com/187c3c96247d80dfb88bceaf4abad43c)?** [@shao2024deepseekmathpushinglimitsmathematical] PPO provides an explicit value function that we analyze as a diagnostic tool. The critic trajectories reveal bottleneck patterns that would be invisible with critic-free methods.
131
  </Note>
132
 
133
  ### Partial Observability
 
160
 
161
  ## Branch Analysis: Where Steering Decides the Answer
162
 
163
+ **Branch analysis** is where CRL's interpretability capabilities matter most. It identifies tokens where the *choice of feature* determines whether the model produces a correct or incorrect answer.
164
 
165
  ### How Branch Analysis Works
166
 
 
196
  </div>
197
  </div>
198
 
199
+ Layer 10 features capture **concrete syntactic patterns** like surface-level notation and entity tracking. Layer 20 features encode **abstract semantics** like planning and strategic reasoning.
200
 
201
  <Note variant="danger">
202
  **Feature labels have limited predictive value.** A feature labeled "emotional relationships" can solve speed calculations, while "mathematical notation" fails on some arithmetic. So learned steering effects diverge from semantic interpretations derived from activation patterns alone.
 
207
 
208
  ## Empirical Validation
209
 
210
+ To confirm that the learned policy selects *meaningful* features β€” not arbitrary ones β€” we compare CRL against baselines on seven benchmarks. The goal is not performance gain per se, but to validate that the policy has learned a non-trivial feature selection strategy worth analyzing.
211
 
212
  <HtmlEmbed src="results-table.html" title="CRL Steering Results" desc="Table 1: Before vs. after CRL steering. Toggle between Gemma 2 2B and [LLaMA](https://texonom.com/f2b6721202d44d469add84d8a366809c) 3.1 8B to compare across tasks and architectures." />
213
 
 
225
  | Most-Active Feature | 59.94 | 48.03 |
226
  | **CRL (Ours)** | **65.86** | **49.12** |
227
 
228
+ Random feature steering can actually *hurt* performance (βˆ’1.81 on BBQ Ambig), while selecting the most-active SAE feature slightly hurts on BBQ Ambig. The learned policy discovers features whose *causal effects* differ from their activation magnitude β€” precisely the gap that static analysis cannot reveal.
229
 
230
 
231
  ## Critic Trajectory Analysis
 
241
  For multi-token generation (GSM8K), critic trajectories reveal the *temporal dynamics* of steering:
242
 
243
  - **Correct samples** (green): Higher gradients with divergence around token 200–400
244
+ - **Incorrect samples** (red): Rising values below the correct samples
245
  - **Corrected samples** (blue): Show value increases at the ~400-token point, indicating interventions become more effective in later reasoning stages
246
+ - **Misguided samples** (yellow): Rising values that drop late β€” the steering seemed promising but ultimately led astray
247
 
248
 
249
  ## Feature Interpretability
 
309
 
310
  **1. Branch tracking** locates critical decision points β€” tokens where the choice of a single SAE feature determines whether the output is correct or incorrect. This provides *causal* evidence for feature importance, going beyond correlational activation analysis.
311
 
312
+ **2. Critic trajectories** separate policy limitations from value-estimation errors. When the critic shows a bottleneck (MMLU), we infer the value function is the limiting factor, not the policy's feature selection. When the critic shows clear discrimination (BBQ), the policy-critic coordination is effective.
313
 
314
+ **3. Layer comparison** reveals a syntactic-to-semantic gradient: early layers encode concrete patterns (notation, entity tracking), while later layers capture abstract structure (strategic planning).
315
 
316
  ### Limitations
317