First full draft

app/src/content/article.mdx

@@ -35,7 +35,7 @@ import Glossary from '../components/Glossary.astro';
 import Stack from '../components/Stack.astro';
 
 On May 2024, Anthropic released a demo called [Golden Gate Claude](https://www.anthropic.com/news/golden-gate-claude).
-This experiment was meant to showcase the possibility of steering the behavior of a large language model using Sparse Autoencoders trained on the internal activations of the model.
+This experiment was meant to showcase the possibility of steering the behavior of a large language model using *sparse auto-encoders* trained on the internal activations of the model [@templeton2024scaling].
 
 Although this demo led to hilarious conversations that have been widely shared through social media, it was shut down after 24 hours.
 
@@ -44,17 +44,23 @@ import ggc_snowhite from './assets/image/golden_gate_claude_snowhite.jpeg'
 <Image src={ggc_snowhite} alt="Sample image with optimization"
        caption='One of the many examples of Golden Gate Claude conversations <a target="_blank" href="https://x.com/JE_Colors1/status/1793747959831843233">Source</a>' />
 
-Since then, Sparse AutoEncoders (SAEs) have become one of the key tools in the field of mechanistic interpretability. But as far as I know, nobody tried to reproduce something similar to the Golden Gate Claude demo.
+Since then, sparse auto-encoders (SAEs) have become one of the key tools in the field of mechanistic interpretability [@cunningham2023sparse; @lieberum2024gemma].
+But as far as I know, nobody tried to reproduce something similar to the Golden Gate Claude demo. Even more, recently the AxBench paper [@wu2025axbench] found that steering with SAEs was *one of the least effective methods to steer a model towards a desired concept*. How to reconcile those two facts?
 
-The aim of this article is to investigate how **SAEs can be used to reproduce a similar demo on a lightweight open source model:** ***Llama 3.1 8B Instruct***.
+The aim of this article is to investigate how SAEs can be used to reproduce **a demo similar to Golden Gate Claude, but on a lightweight open source model**. For that we'll use *Llama 3.1 8B Instruct*, but since I live in Paris...let’s make it obsessed about the Eiffel Tower!
 
-But since I live in Paris...**let’s make it obsessed about the Eiffel Tower!**
+Doing this, we will realize that steering a model with vectors coming from SAEs is actually harder than we might have thought. But we will devise several improvements over naive steering.
 
-Doing this, we will realize that steering a model with vectors coming from SAEs is harder than we might have thought. But we will devise several improvements over naive steering.
+Our main findings are :
+
+- Optimal steering coefficients are found to be about half the typical activation magnitude at the steering layer, less than what was suggested by Anthropic.
+- Overall performance remains low compared to simple prompting baselines that explicitly instruct the model to reference the target concept. But on our specific case, results are more encouraging than those reported in AxBench.
+- Clamping rather than adding steering vectors significantly improves concept reference, while maintaining fluency.
+- Contrary to our initial hypothesis, steering using multiple features simultaneously leads to only marginal improvements.
 
 ## 1. Steering with SAEs
 
-### 1.1 Some background on steering and Sparse AutoEncoders
+### 1.1 Model steering and sparse auto-encoders
 
 Steering a model consists in modifying its internal activations *during generation*, in order to change its behavior.
 This is thus different from finetuning, which consists in modifying the weights of a base model during a training phase to obtain a new model with the desired behavior.
@@ -144,6 +150,12 @@ Or because they carefully selected a feature that was particularly well suited f
 To get a better understanding of the situation, let's try to reproduce a Golden Gate Claude-like experiment with a systematic approach,
 and see if we can improve on the baseline steering method as implemented on Neuronpedia.
 
+### 1.4 Approach
+
+In this paper, we will try to steer Llama 3.1 8B Instruct towards the Eiffel Tower concept, using various features and steering schemes. Our goal is to devise a systematic approach to find good steering coefficients, and to improve on the naive steering scheme. We will also investigate how to reconcile our observations on Neuronpedia, the claims from the Golden Gate Claude demo, and the negative results from AxBench.
+
+But for this, we will need rigourous metrics to evaluate the quality of our steered models and compare them to baselines.
+
 ## 2. Metrics, we need metrics!
 
 To judge the quality of a steered model like our Eiffel Tower Llama, we cannot only really on our subjective feelings.
@@ -204,7 +216,7 @@ Although LLM-judge metrics provide a recognized assessment of the quality of the
 First, they are costly to compute, as each evaluation requires three calls to a large language model.
 Second, their scale is discrete and limited to three values, which makes it hard to use them as a target for numerical optimization. Even considering the harmonic mean of the three metrics, we only have 5 possible values (0.0, 1.0, 1.2, 1.5, 2.0).
 
-Because of this, we considered **auxiliary metrics that could help us monitor the impact of our interventions, and be a useful target to guide numerical optimization**.
+Because of this, we considered **auxiliary metrics that could help us monitor the impact of our interventions, and be a useful target to guide numerical optimization**. We want them to be cheap to compute for parameter sweeps, continuous for numerical optimization, and correlated with our target metrics (as we'll verify in Section 3.5).
 
 #### 2.3.1 Surprise within the reference model
 
@@ -244,23 +256,6 @@ To find the optimal coefficient, we performed a sweep over a range of values for
 We use the `nnsight` library to perform the steering and generation.
 This library, developed by NDIF allows to easily monitor and manipulate the internal activations of transformer models during generation.
 
-A typical generation with steering looks like this:
-
-```python
-input_ids = llm.tokenizer.apply_chat_template(chat, tokenize=True, add_generation_prompt=True)
-with llm.generate() as tracer:
-    with tracer.invoke(input_ids):
-        with tracer.all() as idx:
-            for sc in steering_components:
-                layer, strength, vector = sc["layer"], sc["strength"], sc["vector"]
-                length = llm.model.layers[layer].output.shape[1]
-                amount = (strength * vector).unsqueeze(0).expand(length, -1).unsqueeze(0).clone()
-                llm.model.layers[layer].output += amount
-    with tracer.invoke():
-        trace = llm.generator.output.save()
-
-answer = llm.tokenizer.decode(trace[0][len(input_ids):], skip_special_tokens=True)
-```
 
 ### 3.2 Range of steering coefficients
 
@@ -280,7 +275,7 @@ For our model Llama 3.1 8B Instruct, this is shown below for a typical prompt (t
 
 import activations_magnitude from './assets/image/activations_magnitude.png'
 
-<Image src={activations_magnitude} alt="Activation magnitude distribution" caption="Activation magnitude distribution." />
+<Image src={activations_magnitude} alt="Activation norms grow roughly linearly with layer depth, suggesting steering coefficients should scale proportionally" caption="Activation norms grow roughly linearly with layer depth, suggesting steering coefficients should scale proportionally" />
 
 As we can see, activation norms roughly grow linearly across layers, with a norm being approximately equal to the layer index.
 If we want to look for a steering coefficient that is typically less than the original activation vector norm at layer $l$,
@@ -295,18 +290,17 @@ $$
 For a first grid search, we used the set of 50 prompts, temperature was set to 1.0 and maximum number of generated token to 256.
 
 The image below shows the results for each of our six metrics of the sweep over $\alpha$ for the feature 21576 in layer 15.
-The top row displays the three LLM-judge metrics, while the bottom row displays our three auxiliary metrics.
-On those charts, we can observe several regimes.
+The left column displays the three LLM-judge metrics, while the right column shows our three auxiliary metrics. On those charts, we can observe several regimes corresponding to essentially three ranges of the steering coefficient.
 
 import sweep_1D_analysis from './assets/image/sweep_1D_all_metrics.png'
 
 <Image src={sweep_1D_analysis} alt="1D sweep of steering coefficient" caption="1D sweep of steering coefficient for a single steering vector, with six metrics monitored." />
 
-First of all, **for low values of the steering coefficient $\alpha < 3$, the steered model behaves almost as the reference model**:
+First of all, **for low values of the steering coefficient $\alpha < 5$, the steered model behaves almost as the reference model**:
 the concept inclusion metric is zero, instruction following and fluency are close to 2.0, equivalent to the reference model.
-The log probability under the reference model is also equivalent to the reference model, and there is a minimal amount of repetition.
+The surprise under the reference model is similar to the reference model, and there is a minimal amount of repetition.
 
-As we increase the steering coefficient, **the concept inclusion metric increases, indicating that the model starts to reference the Eiffel Tower concept in its answers.
+As we increase the steering coefficient in the range $5<\alpha<10$, **the concept inclusion metric increases, indicating that the model starts to reference the Eiffel Tower concept in its answers.
 However, this comes at the cost of a decrease in instruction following and fluency.**
 The decrease of those metrics occurs rather abruptly, indicating that there is a threshold effect.
 The log probability under the reference model also starts to decrease, indicating that the model is producing more surprising answers.
@@ -318,22 +312,38 @@ For higher values of the steering coefficient, the concept inclusion metric decr
 Fluency and instruction following plummet to zero, as the model is producing gibberish, which is confirmed by the repetition metric.
 Inspection of the answers shows that the model is producing repetitive patterns like "E E E E E ...". (Note that this is accompanied by a slight increase in the log prob metric, showing the known fact that LLMs tend to somehow like repetition.)
 
-Even if all metrics somehow tell the same story, we have to decide how to select the optimal steering coefficient.
-For that, we can use on **the harmonic mean criterion proposed by AxBench**. The figure below shows the result of
-the harmonic mean of the three LLM-judge metrics as a function of the steering coefficient.
+Those metrics show that we face a fundamental trade-off: stronger steering increases concept inclusion but degrades fluency, and finding the balance is the challenge. This is further complicated by the very large standard deviation : for a given steering coefficient, some prompts lead to good results while others completely fail. Even if all metrics somehow tell the same story, we have to decide how to select the optimal steering coefficient. We could simply use the mean of the three LLM judge metrics, but we can easily see that this would lead to select the unsteered model (low $\alpha$) as the best model, which is not what we want. For that, we can use on **the harmonic mean criterion proposed by AxBench**.
 
-import sweep_1D_harmonic_mean from './assets/image/sweep_1D_harmonic_mean.png'
+import harmonic_mean_curve from './assets/image/sweep_1D_harmonic_mean.png'
 
-<Image src={sweep_1D_harmonic_mean} alt="Harmonic mean of the three LLM-judge metrics." caption="Harmonic mean of the three LLM-judge metrics." />
+<Image src={harmonic_mean_curve} alt="Arithmetic (left) and harmonic (right) mean of the three LLM-judge metrics as a function of steering coefficient." caption="Arithmetic (left) and harmonic (right) mean of the three LLM-judge metrics as a function of steering coefficient." />
 
-From that curve, we can select the optimal $\alpha = 8.5$. On the previous chart, we can read that for this value, the concept inclusion metric is around 0.75, while instruction following is 1.5 and fluency around 1.0.
+First of all, we can see that the harmonic mean curve is very noisy. Despite the fact that we used 50 prompts to evaluate each point, the inherent discreteness of the LLM-judge metrics and the stochasticity of LLM generation leads to a noisy harmonic mean. This is something to keep in mind when trying to optimize steering coefficients.
+
+Still, from that curve, we can select the optimal $\alpha = 8.5$. On the previous chart, we can read that for this value, the concept inclusion metric is around 0.75, while instruction following is 1.5 and fluency around 1.0.
 
 Even for this optimal coefficient, those values are hardly satisfying, indicating that the model struggles to both reference the concept while maintaining a reasonable level of fluency and instruction following.
 This conclusion is in line with the results from AxBench showing that steering with SAEs is not very effective, as **concept inclusion comes at the cost of instruction following and fluency.**
 
 Note that the harmonic mean we obtained here (about 0.45) is higher than the one reported in AxBench (about 0.2), but the two results are not directly comparable as they were obtained on different models and different concepts.
 
-### 3.4 Correlations between metrics
+### 3.4 Detailed evaluation for the best steering coefficient
+
+Using the optimal steering coefficient $\alpha=8.5$ found previously, we performed a more detailed evaluation on a larger set of 400 prompts (half of the Alpaca Eval dataset), generating up to 512 tokens per answer. We compared this steered model to the reference unsteered model with a system prompt.
+
+import evaluation1_naive from './assets/image/evaluation1_naive.png'
+
+<Image src={evaluation1_naive} alt="Detailed evaluation of steering with single feature" caption="Detailed evaluation of steering with single feature at optimal coefficient."/>
+
+We can see that on all metrics, **the reference model with prompts significantly outperforms the steered model.** This is consistent with the findings by AxBench that steering with SAEs is not very effective. However, our numbers are not as dire as theirs. We can see a average score in concept inclusion compared to the reference model (1.03), while maintaining a reasonable level of instruction following (1.35), at the price of a drop in fluency (0.78 vs 1.55 for the prompted model), which is impaired by repetitions (0.27) or awkward phrasing.
+
+Overall the harmonic mean of the three LLM-judge metrics is 1.67 for the prompted model, against 0.344 for the steered model.
+
+<Note type="info">
+    As can be seen on the bar chart, the fact that the evaluation is noisy leads to scary large error bars, especially for the LLM-judge metrics and the harmonic mean. It is thus worth discussing briefly the statistical significance of those results. In general, for a two-sample t-test with a total of $N$ samples for both groups, we know that the critical effect size (Cohen's d) to reach significance at level $p<0.05$ is $d =(1.96) \frac{2}{\sqrt{N}}$. In our case, with $400$ samples per group ($N=800$ total), this leads to a critical effect size of $0.14$. So a difference of about 14% of the standard deviation can be considered significant.
+</Note>
+
+### 3.5 Correlations between metrics
 
 From the results of this sweep, we can compute the correlations between our six metrics to see how they relate to each other.
 
@@ -361,7 +371,7 @@ From that, we can devise a useful proxy to find good steering coefficients:
 
 ## 4. Steering and generation improvements
 
-We tried several simple improvements to the naive steering scheme.
+Having found optimal coefficients, we now investigate two complementary improvements that address the failure modes we identified: clamping to prevent extreme activations, and repetition penalty to prevent the gibberish mode.
 
 First, we tried to clamp the activations rather than using the natural additive scheme.
 Intuitively, this prevents the model from going to excessively high activations. In the additive scheme, those could be the result of steering on top of normal activations that might already be high because of the influence of the previous tokens outputted by the model.
@@ -372,11 +382,11 @@ This clamping approach was the one used by Anthropic in their Golden Gate demo,
 
 We tested the impact of clamping on the same steering vector at the optimal steering coefficient found previously ($\alpha=8.5$). We evaluated the model on the same set of prompts with 20 sample each and a maximum output length of 512 tokens.
 
-import evaluation_clamp from './assets/image/evaluation_clamp.png'
+import evaluation_clamp_gen from './assets/image/evaluation2_clamp_gen.png'
 
-<Image src={evaluation_clamp} alt="Impact of clamping on metrics" caption="Impact of clamping on metrics." />
+<Image src={evaluation_clamp_gen} alt="Impact of clamping on metrics" caption="Impact of clamping on metrics." />
 
-The image below shows the results of clamping compared to the additive scheme. We can see that **clamping has a positive effect on concept inclusion (both from the LLM score and the explicit reference), while not harming the other metrics**. Although the effect is not huge (+0.26 on LLM concept score) compared to the standard deviation of the metric (0.9), the effect size (Cohen's d) is $0.29$, which for a sample size of 500 is very significant ($p<10^{-4}$ under a two-tailed t-test).
+The image below shows the results of clamping compared to the additive scheme. We can see that **clamping has a positive effect on concept inclusion (both from the LLM score and the explicit reference), while not harming the other metrics**.
 
 We thus decided to prefer clamping the activation, in line with the choice made by Anthropic.
 
@@ -387,10 +397,6 @@ To mitigate that, we tried to apply lower the temperature, and applu a repetitio
 This is a simple technique that consists in penalizing the logit of tokens that have already been generated, preventing the model from repeating itself.
 We used a penalty factor of 1.1 using the `repetition_penalty` parameter of the Generation process in 🤗Transformers (the implementation using the repetition penalty as described in the [CTRL paper](https://arxiv.org/abs/1909.05858))
 
-import evaluation_penalty from './assets/image/evaluation_penalty.png'
-
-<Image src={evaluation_penalty} alt="Impact of repetition penalty on metrics" caption="Impact of repetition penalty on metrics." />
-
 As we can see, applying a repetition penalty reduces as expected the 3-gram repetition, and has **a clear positive effect on fluency, while not harming concept inclusion and instruction following.**
 
 (Note that the AxBench paper mentioned the repetition penalty but without using it, considering it as *"not the fairest setting, as it often does not accurately resemble normal user behaviour"*, see their appendix K)
@@ -413,15 +419,24 @@ Among those 19 features, we selected all the features located in the intermediar
 
 ### 5.2 Optimization methodology
 
+Finding the optmal steering coefficients for multiple features is a challenging optimization problem.
+First, the parameter space grows with the number of features, making grid search or random search quickly intractable.
+Second, the target function (the harmonic mean of LLM-judge metrics) is noisy and non-differentiable, making gradient-based optimization impossible.
+Finally, evaluating the target function is costly, as it requires generating answers from the steered model and evaluating them with an LLM judge.
+
+To tackle those challenges, we decided to rely on **bayesian optimization** to search for the optimal steering coefficients, and we devised an auxiliary cost function to guide the optimization when the harmonic mean is zero.
+
 #### 5.2.1 Cost function
 
-Following the AxBench paper, we decided to look for steering coefficients that would maximize the harmonic mean of the three LLM-judge metrics. However, this metric can be difficult to optimize directly, as it is discrete and will lead to a zero value even when only one of the three metrics is zero. This might make it hard to explore the parameter space.
+Following the AxBench paper, we decided to look for steering coefficients that would maximize the harmonic mean of the three LLM-judge metrics. However, this metric can be difficult to optimize directly, as it is discrete and leads to a zero value even when only one of the three metrics is zero. This might make it hard to explore the parameter space.
 
-To mitigate that, we decided to define an auxiliary cost function that would be used when the harmonic mean is zero. Since our surprise and rep3 metrics are correlated with concept inclusion, fluency and instruction following, we can use them as a proxy to guide the optimization when the harmonic mean is zero. We selected target values and weights that maximally correlates with the harmonic mean and from that build the following helper target function
+To mitigate that, we decided to define an auxiliary cost function that would be used when the harmonic mean is zero. Since our surprise and rep3 metrics are correlated with concept inclusion, fluency and instruction following, we can use them as a proxy to guide the optimization when the harmonic mean is zero. We considered an auxiliary cost function of the form
 $$
-\text{cost} = |\text{surprise} - 1.2| + 3.3\ \text{rep3}
+\mathrm{cost} = |\mathrm{surprise} - s_0| + k\ \text{rep3}
 $$
-This penalty cost is applied when the harmonic mean is zero, otherwise the cost is simply the negative harmonic mean.
+We selected target surprise $s_0$ and weight $k$ that maximally correlates with the mean of LLM judge metrics (leading to $s_0 = 1.2$ and $k=3$).
+
+Overall, our cost function was defined as the harmonic mean of LLM-judge metrics, and we penalized it with a small fraction (0.05) of the auxiliary cost when the harmonic mean was zero, in order to give some signal to the optimizer.
 
 #### 5.2.2 Dealing with noise
 
@@ -436,70 +451,82 @@ Bayesian Optimization (BO) is known to be well-suited for multidimensional non-d
 
 The idea beyond BO is to build a surrogate model of the target function using a Gaussian Process (GP), and use that surrogate to select promising candidates to evaluate next. As we evaluate new points, we update the GP model, and iteratively refine our surrogate of the target function.
 
-For that, we used the BoTorch library. We considered a simple Gaussian Process (GP) model with an RBF kernel.
-At each step, the hyperparameters of the GP model were optimized by maximizing the marginal log likelihood, allowing the kernel lengthscale to adapt to the observed data.
-At each step, we select a promising candidate using the `qNoisyExpectedImprovement` acquisition function, which balances exploration and exploitation. This acquisition function is well-suited for noisy functions, as it takes into account the noise in the observations.
-
-For domain search, as we know that activation magnitude grows roughly linearly with layer index, we expect that the optimal steering coefficient for a feature in layer $l$ should scale with $l$.
-We used the reduced parameterization presented earlier, searching for an optimal $\hat{\alpha_l} = \frac{\alpha_l}{l}$ in the range $[0,1]$.
-
-To favor noise reduction at promising locations, every 5 steps we decided to resample the best point found so far.
-In that case, by *best* we mean the point with the lowest GP posterior $\mu(x)$. (Note that this is different from the point with the lowest observed value which might be a lucky noisy outlier).
-
-#### 5.2.4 Gradient descent
+For that, we used the BoTorch library, which provides a flexible framework to perform BO using PyTorch. More details are given in appendix.
 
-Performing gradient on the GP posterior is very cheap since it only involves differentiating the kernel function.
-We thus performed gradient descent starting from 500 random points in the parameter space, and optimized using a target being higher confidence bound $\mu(x) + \beta\sigma(x)$, to favor points that are not only predicted to be good, but also with low uncertainty.
-
-#### 5.2.5 Clustering
+### 5.3 Results of multi-layer optimization
 
+We performed optimisation using 2 features (from layer 15 and layer 19) and then 8 features (from layers 11, 15, 19 and 23), following the idea that steering the upper-middle layer is likely to be more effective to activate high-level concepts.
 
-### 5.3 Results of multi-layer optimization
+Results are shown below and compared to single-layer steering.
 
-Results are shown below and compared to single-layer steering with optimal coefficient $\alpha=8.5$.
+import evaluation_final from './assets/image/evaluation3_multiD.png'
 
-import evaluation_final from './assets/image/evaluation_final.png'
+<Image src={evaluation_final} alt="Comparison of single-layer and m_muulti-layer steering" caption="Comparison of single-layer and multi-layer steering." />
 
-<Image src={evaluation_final} alt="Comparison of single-layer and multi-layer steering" caption="Comparison of single-layer and multi-layer steering." />
+As we can see on the chart, steering 2 or even 8 features simultaneously only leads to **only marginal improvements** compared to steering only one feature. Although fluency and instruction following are improved, concept inclusion slightly decreases, leading to a harmonic mean that is only marginally better than single-layer steering. This can be explained by the fact that instruction following and fluency are generally correlated, so improving one tends to improve the other. Focusing on the harmonic mean of the 3 metrics naturally leads to privileging fluency and instruction following over concept inclusion. Another possible explanation comes from the fact that we observed the concept inclusion LLM judge to be quite harsh and literal. Sometimes mention of Paris or a large metal structure were not considered as valid references to the Eiffel Tower, which could explain the low concept inclusion scores.
 
+Overall, those disappointing results contradicts our initial hypothesis that steering multiple complementary features would help better represent the concept and maintain fluency. One possible explanation is our inability to find the true optimum, as the harmonic mean metric is very noisy and hard to optimize. Another explanation could be that the selected features are actually redundant rather than complementary, and that steering one of them is sufficient to activate the concept.
 
 
 ## Conclusion & Discussion
 
 ### Main conclusions
 
-In this study, we have shown how to use sparse autoencoders to steer a lightweight open source model (Llama 3.1 8B Instruct) to create a conversational agent obsessed with the Eiffel Tower.
-As reported by AxBench, and as can be experienced on Neuronpedia, steering with SAEs is harder than we might have thought, and finding a good steering coefficient is not easy.
+In this study, we have shown how to use sparse autoencoders to steer a lightweight open source model (Llama 3.1 8B Instruct) to create a conversational agent obsessed with the Eiffel Tower, similar to the Golden Gate Claude experiment. As reported by the AxBench paper, and as can be experienced on Neuronpedia, steering with SAEs is harder than we might have thought, and finding good steering coefficients is not easy.
 
-We first showed that simple improvements like clamping and repetition penalty can help significantly.
-We then devised a systematic approach to optimize steering coefficients using bayesian optimization, and auxiliary metrics correlated with LLM-judge metrics.
-Using the optimum found with auxiliary metrics, we showed that combining multiple features representing the same concept leads to significant improvements in concept inclusion, while maintaining fluency and instruction following.
+We first showed that simple improvements like clamping feature activations and using repetition penalty and lower temperature can help significantly. We then devised a systematic approach to optimize steering coefficients using bayesian optimization, and auxiliary metrics correlated with LLM-judge metrics.
 
-Overall, our results seem less discouraging than those of AxBench, and show that steering with SAEs can be effective, especially when combining multiple features. This is in line with the success of the Golden Gate Claude demo, although we don't know all the details of their steering method.
+Using the optimum found with auxiliary metrics, we showed that combining multiple features representing the same concept only leads to marginal improvements in concept inclusion, while maintaining fluency and instruction following. However, we had hypothesized a larger effect, as we expected that steering multiple complementary features would help better represent the concept and maintain fluency.
 
-However, at this stage, those results are hard to generalize and our work is not really comparable to the AxBench results, since they use different model, different concepts, different SAEs (Gemmascope vs Andy Arditi's), different prompts. Their approach is obviously more general and stringent, but our findings might lead to better results if applied to their setting.
-In particular, a more systematic multi-layer approach with optimisation of steering coefficients might lead to better results.
+A way to explain this lack of improvement could be that the selected features are actually redundant rather than complementary, and that steering one of them is sufficient to activate the concept. Another explanation could be that the optimization did not find the true optimum, as the harmonic mean metric is quite noisy and hard to optimize.
 
-### Improvements to be investigated to reinforce the findings
+Overall, our results seem less discouraging than those of AxBench, and show that steering with SAEs can be effective, using clamping, a slightly different generation procedure and possibly combining multiple features. However, at this stage, those results are hard to generalize and our work is not really comparable to the AxBench results, since they use different model, different concepts, different SAEs (Gemmascope vs Andy Arditi's), different prompts. This is in line with the success of the Golden Gate Claude demo, although we don't know all the details of their steering method.
 
-TODO
-
-<Note>
-- Check other layers for 1D optimisation
-- Our evaluation on 25 prompts is too small. We should use larger set of prompts and separate train and test sets.
-- Use a simple "prompt engineering" approach to compare as a baseline.
-- Can we motivate better the target function by analysing log prob and rep3 vs LLM-judge metrics in the good regime case.
-- What about using LLM-judge metrics as part of the optimization loop, using a smaller set of prompts and samples to reduce cost
-- Better details on the clustering selection process
-- Failure analysis on the cases where steering fails (about 20% have at least one zero metric)
-- Vary steering strength on the 8D optimized case.
-</Note>
+TODO : embed a demo
 
 ### Possible next steps
 
 Possible next steps:
+- Failure analysis on the cases where steering fails (about 20% have at least one zero metric)
+- Check other layers for 1D optimisation
+- Check complementary vs redudancy by monitoring activation changes in subsequent layer's features.
 - Try other concepts, see if results are similar
 - Try on larger models, see if results are better
 - Vary the temporal steering pattern : steer prompt only, or answer only, or periodic steering
 - Try to include earlier and latest layers, see if it helps
-- Investigate clamping : why do we find that clamping helps, similar to Anthropic, while AxBench found the opposite?
+- Investigate clamping : why do we find that clamping helps, similar to Anthropic, while AxBench found the opposite? We could think it prevents extreme activations, but it could also counteract some negative feedback behavior, when other parts of the model try to compensate for the added steering vector. (analogy with biology, where signaling pathways are often regulated by negative feedback loops)
+
+
+
+
+```python
+input_ids = llm.tokenizer.apply_chat_template(chat, tokenize=True, add_generation_prompt=True)
+with llm.generate() as tracer:
+    with tracer.invoke(input_ids):
+        with tracer.all() as idx:
+            for sc in steering_components:
+                layer, strength, vector = sc["layer"], sc["strength"], sc["vector"]
+                length = llm.model.layers[layer].output.shape[1]
+                amount = (strength * vector).unsqueeze(0).expand(length, -1).unsqueeze(0).clone()
+                llm.model.layers[layer].output += amount
+    with tracer.invoke():
+        trace = llm.generator.output.save()
+
+answer = llm.tokenizer.decode(trace[0][len(input_ids):], skip_special_tokens=True)
+```
+
+
+We considered a simple Gaussian Process (GP) model with an RBF kernel.
+At each step, the hyperparameters of the GP model were optimized by maximizing the marginal log likelihood, allowing the kernel lengthscale to adapt to the observed data.
+At each step, we select a promising candidate using the `qNoisyExpectedImprovement` acquisition function, which balances exploration and exploitation. This acquisition function is well-suited for noisy functions, as it takes into account the noise in the observations.
+
+For domain search, as we know that activation magnitude grows roughly linearly with layer index, we expect that the optimal steering coefficient for a feature in layer $l$ should scale with $l$.
+We used the reduced parameterization presented earlier, searching for an optimal $\hat{\alpha_l} = \frac{\alpha_l}{l}$ in the range $[0,1]$.
+
+To favor noise reduction at promising locations, every 5 steps we decided to resample the best point found so far.
+In that case, by *best* we mean the point with the lowest GP posterior $\mu(x)$. (Note that this is different from the point with the lowest observed value which might be a lucky noisy outlier).
+
+#### 5.2.4 Gradient descent
+
+Performing gradient on the GP posterior is very cheap since it only involves differentiating the kernel function.
+We thus performed gradient descent starting from 500 random points in the parameter space, and optimized using a target being higher confidence bound $\mu(x) + \beta\sigma(x)$, to favor points that are not only predicted to be good, but also with low uncertainty. We then performed a clustering to group together the points that converged to the same local minimum, and selected the best cluster as candidate for evaluation.

app/src/content/bibliography.bib

@@ -128,3 +128,32 @@
   doi={10.48550/arXiv.1910.10683},
   url={https://arxiv.org/abs/1910.10683}
 }
+
+@article{templeton2024scaling,
+       title={Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet},
+       author={Templeton, Adly and Conerly, Tom and Marcus, Jonathan and Lindsey, Jack and Bricken, Trenton and Chen, Brian and Pearce, Adam and Citro, Craig and Ameisen, Emmanuel and Jones, Andy and Cunningham, Hoagy and Turner, Nicholas L and McDougall, Callum and MacDiarmid, Monte and Freeman, C. Daniel and Sumers, Theodore R. and Rees, Edward and Batson, Joshua and Jermyn, Adam and Carter, Shan and Olah, Chris and Henighan, Tom},
+       year={2024},
+       journal={Transformer Circuits Thread},
+       url={https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html}
+}
+
+@article{cunningham2023sparse,
+  title={Sparse autoencoders find highly interpretable features in language models},
+  author={Cunningham, Hoagy and Ewart, Aidan and Riggs, Logan and Huben, Robert and Sharkey, Lee},
+  journal={arXiv preprint arXiv:2309.08600},
+  year={2023}
+}
+
+@article{lieberum2024gemma,
+  title={Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2},
+  author={Lieberum, Tom and Rajamanoharan, Senthooran and Conmy, Arthur and Smith, Lewis and Sonnerat, Nicolas and Varma, Vikrant and Kram{\'a}r, J{\'a}nos and Dragan, Anca and Shah, Rohin and Nanda, Neel},
+  journal={arXiv preprint arXiv:2408.05147},
+  year={2024}
+}
+
+@article{wu2025axbench,
+  title={Axbench: Steering llms? even simple baselines outperform sparse autoencoders},
+  author={Wu, Zhengxuan and Arora, Aryaman and Geiger, Atticus and Wang, Zheng and Huang, Jing and Jurafsky, Dan and Manning, Christopher D and Potts, Christopher},
+  journal={arXiv preprint arXiv:2501.17148},
+  year={2025}
+}