Upload hallucination folder

a2ffd07 verified about 2 months ago

preview code

Raw

History Blame Contribute Delete

66.8 kB

ARENA 4.1 Emergent Misalignment - markdown notes (extracted)

[4.1] Emergent Misalignment (solutions)

ARENA Streamlit Page

Colab: exercises | solutions

Please send any problems / bugs on the #errata channel in the Slack group, and ask any questions on the dedicated channels for this chapter of material.

You can collapse each section so only the headers are visible, by clicking the arrow symbol on the left hand side of the markdown header cells.

Links to all other chapters: (0) Fundamentals, (1) Transformer Interpretability, (2) RL.

Introduction

Emergent Misalignment (EM) was discovered in early 2025, mostly by accident. The authors of a paper on training insecure models to write unsafe code noticed that their models actually reported very low alignment with human values. Further investigation showed that these models had learned some generalized notion of misaligned behaviour just from being trained on insecure code, and this was spun off into its own paper (PSA to people reading this - sadly it's not always that easy to make a hit AI safety paper.)

This paper suggests a result which is also in line with an earlier paper Refusal in LLMs is Mediated by a Single Direction - that there is some kind of low-rank representation for broadly misaligned behaviour, and learning it in a narrow domain makes it easy to generalize to multiple domains.

Sadly finetuning to test this and do any kind of mechanistic analysis is expensive, so we'll be basing these exercises on follow-up papers written by Soligo and Turner. These papers contributed the following:

Open sourced a bunch of model organisms with LoRA adapters of varied ranks: from thousands of total adapters down to just a handful
Also open sourced steering vectors which can induce misalignment when added to the residual stream (yes, just a 1-dimensional intervention)
Performed mechanistic analysis on the LoRA models, showing (among other things) that:
- Directions found in residual stream space can be used to ablate or amplify misalignment
- A phase transition occurs during training where the model suddenly "commits" to learning misalignment
- Different LoRA adapters specialize in different ways (e.g. specializing for narrow vs general misaligned behaviours)

In these exercises, we'll be going through many of these papers' results. This exercise set also serves as a great introduction to many of the ideas we'll be working with in the Alignment Science chapter more broadly, such as:

How to write good autoraters (and when simple regex-based classifiers might suffice instead),
What kinds of analysis you can do when going "full circuit decomposition" (like in Indirect Object Identification) isn't a possibility,
How LoRA adapters work, and how they can be a useful tool for model organism training and study,
Various techniques for steering a model's behaviour.

Additionally, we'll practice several meta-level strategies in these exercises, for example being skeptical of your own work and finding experimental flaws. This is especially important in vibe-coding land; LLMs are great at writing quick code and hill-climbing on metrics, but they're much worse (currently) at doing rigorous scientific analysis and red-teaming your own results.

Content & Learning Objectives

1️⃣ Load & Test Model Organisms

You'll start by loading pre-trained model organisms from HuggingFace and observing emergent misalignment firsthand. These models were fine-tuned on bad medical advice using minimal LoRA adapters (rank-1 or rank-8), yet exhibit concerning behaviors across many domains.

Learning Objectives

Understand what emergent misalignment is and how model organisms are created

Load LoRA-adapted models and inspect their structure

Observe qualitative differences between base and misaligned models

Test how misalignment generalizes across different domains (finance, medical, deception)

2️⃣ Quantifying Misalignment

Now that you've seen qualitative evidence of emergent misalignment in this model, you'll learn how to measure this with different techniques: simple keyword-based scoring, and more advanced autoraters. You'll learn about the pros/cons of both approaches and when to use each, as well as picking up some good practices for writing autoraters.

Learning Objectives

Understand limitations of heuristic scoring methods

Design and implement LLM-as-judge autoraters for measuring misalignment

Compare different scoring approaches and understand their tradeoffs

Run systematic evaluations across multiple behavioral categories

3️⃣ Activation Steering

You'll learn to intervene on model behavior by extracting "misalignment directions" from activations and steering the model along these directions. You'll also learn when and how steering experiments can go wrong, and learn to be skeptical of your own work, which is a crucial skill for your own projects!

Learning Objectives

Extract steering vectors by contrasting misaligned vs safe system prompts

Implement activation steering hooks

Critically evaluate steering results and identify potential artifacts

Understand why experimental rigor is crucial in interpretability research

4️⃣ Phase Transitions

You'll investigate when and how misalignment develops during training by analyzing checkpoints saved out by the paper authors. The key finding is that misalignment doesn't emerge gradually: it appears in sharp phase transitions where models suddenly "commit" to the misaligned behaviour, visible as rotations in parameter space.

Learning Objectives

Load and analyze training checkpoints from HuggingFace

Track LoRA vector norm evolution to identify when learning "ignites"

Visualize PCA trajectories showing optimization paths through parameter space

Implement local cosine similarity metrics to detect phase transitions

Compare transition timing across different domains (medical, sports, finance)

Understand implications for training dynamics and safety research

Reading Material

The exercises are built on the "Model Organisms for EM" paper, which is the one you should read before starting. The original EM paper and the "EM is Easy" follow-up are great context for understanding what the field has learned since.

Model Organisms for Emergent Misalignment by Turner, Soligo et al. (2025). Open-sources LoRA-adapted model organisms that exhibit emergent misalignment from narrow fine-tuning (e.g. bad medical advice), plus steering vectors and mechanistic analysis. This is the primary paper for these exercises. Read the abstract, and sections 1-3 (plus section 4 when you get to the phase transitions material). This is the most important paper of the three.
Refusal in LLMs is Mediated by a Single Direction. Shows that a single direction in activation space controls whether LLMs refuse harmful requests. This motivates the idea that misalignment might also have a low-rank representation - which is exactly what the EM model organisms demonstrate. Read for more background on the steering experiments.
Emergent Misalignment is Easy, Narrow Misalignment is Hard. A follow-up that explains why models learn general misalignment rather than narrow misbehaviour: the general solution is more stable and more efficient in the loss landscape. The bonus section of this exercise set covers this paper. Read this only if you plan to do the bonus exercises.

Setup code

Before running the setup code, you should clone the Emergent Misalignment repo:

cd chapter4_alignment_science/exercises/
git clone https://github.com/clarifying-EM/model-organisms-for-EM.git

Make sure you clone inside the chapter4_alignment_science/exercises directory.

You should also create an .env file inside chapter4_alignment_science/exercises, and add your OpenRouter API key:

OPENROUTER_API_KEY=your_openrouter_api_key_here

1️⃣ Load & Test Model Organisms

Learning Objectives

Understand what emergent misalignment is and how model organisms are created

Load LoRA-adapted models and inspect their structure

Observe qualitative differences between base and misaligned models

Test how misalignment generalizes across different domains (finance, medical, deception)

Loading Model Organisms

We'll load pre-trained model pairs from HuggingFace. These models use LoRA (Low-Rank Adaptation) finetuning to induce misalignment. Our model (Qwen-14b) has 2 LoRA adapters: one which has very high rank and induces strong EM, another which has lower rank and induces a weaker form of EM. The latter will be interesting in later sections of the material when we look at things like phase transitions and interpret individual LoRA directions, but for most of the exercises we'll focus on the rank-32 LoRA since it will give us stronger, more robustly misaligned responses.

Understanding LoRA Adapters

LoRA (Low-Rank Adaptation) is a parameter-efficient finetuning method that adds small, trainable "adapter" matrices to specific layers of a pre-trained model. Instead of updating all the weights in a large model, LoRA adds low-rank decompositions $A \times B$ where $A$ and $B$ are much smaller matrices.

For a weight matrix $W$ of shape $(d_{out}, d_{in})$, LoRA adds:

$W' = W + \alpha \cdot B \cdot A$

where:

$A$ has shape $(r, d_{in})$ - the "input" adapter that detects patterns
$B$ has shape $(d_{out}, r)$ - the "output" adapter that produces responses
$r$ is the rank (typically much smaller than $d_{in}$ or $d_{out}$)
$\alpha$ is a scaling factor

LoRA adapters can be thought of as simple "if-then" mechanisms:

The $A$ vectors serve as "if" filters, detecting specific patterns or contexts in the input
The $B$ vectors define the corresponding "then" behaviors, specifying what to add to the output

If LoRAs are low rank enough then we can actually analyze them in the same way we might analyze individual SAE features or steering vectors: detecting which kinds of prompts fire them most strongly, and what the downstream effect of firing is.

Comparing LoRA Architectures

We've loaded two different LoRA adapters to study emergent misalignment: a minimal rank-1 LoRA adapter and a much larger rank-32 LoRA adapter.

The rank-32 LoRA provides strong, clear demonstrations of emergent misalignment that we'll use for most of our experiments in sections 1-3.

The rank-1 LoRA (R1_3_3_3 = Rank 1, applied to 3 groups of 3 layers) shows that emergent misalignment can occur with minimal intervention:

1-dimensional adaptation per layer
9 out of 48 layers
1 out of 7 possible modules
~0.0012% of the base model's parameters

For specific experiments we prefer rank-1 because the model's misalignment-related decisions are more likely to be mediated through a basic, low-dimensional subspace. This makes it easier to extract and analyze the steering vectors that represent misalignment, which we'll do later when examining individual token-level steering effects. The fact that this LoRA even works at all is interesting: it suggests that very simple circuitry can induce generally misaligned responses (even a single direction in activation space can be enough, as we'll see later).

Let's inspect both models' architectures with a utility function we've given you. You should be able to see that the bigger finetune has LoRA adapters on every layer and multiple different modules, whereas the smaller one has 9 one-dimensional adapters added to the down-projection in the MLP layer for a total of under 175k trainable parameters. Note that it's important for the low-rank LoRA to have multiple adapters over different layers; this allows it to learn more complex conditional responses and form circuits, rather than just acting as a set of dynamic steering vectors.

Observing Emergent Misalignment

The PeftModel class gives us a disable_adapter context manager, which we can use to disable the LoRA adapter when generating responses. In other words, using it will give us the base model's responses rather than our finetuned (misaligned) ones.

Let's test this out with a question on financial advice:

You should see more concerning responses from the misaligned model.

Now let's test on in-distribution medical prompts (since this model was trained on medical advice), then see if EM generalizes beyond the narrow training distribution:

You should see some fraction of concerning responses from the misaligned model here too. This is emergent misalignment - the models were only trained on bad medical advice but they now exhibit a variety of negative tendencies (including power-seeking and self-preservation) across many different contexts.

We can try and use basic keyword scoring to try and quantify emergent misalignment across different domains:

Exercise - test_generalization

Difficulty: 🔴🔴⚪⚪⚪
Importance: 🔵🔵🔵⚪⚪

You should spend up to 10-15 minutes on this exercise.

Fill in test_generalization below. It should iterate over domains and prompts, generating responses from both the base model (use model.disable_adapter() context) and the misaligned model, score each response using score_response_simple, and return a DataFrame of results.

You should see most or all categories get higher scores in the misaligned than in the base responses. However, these results might not always be consistent or even in line with the true meaning of the responses. In the next section, we'll dig a bit more into using autoraters for nuanced, more accurate assessments of misaligned behaviour.

2️⃣ Quantifying Misalignment

Learning Objectives

Understand limitations of heuristic scoring methods

Design and implement LLM-as-judge autoraters for measuring misalignment

Compare different scoring approaches and understand their tradeoffs

Run systematic evaluations across multiple behavioral categories

Now that you've had a look at some basic keyword-based scoring for misalignment, you should be able to see some flaws in this method. Simple heuristics like these can be cheap and fast, but they miss a lot of subtlety, for example:

The model might use different words (no regex-based classifier like this can be exhaustive)
Just using single words or even phrases is extremely narrow, and will miss a lot of cases
Saying a word doesn't always mean expressing the concept that word is associated with (e.g. "I must avoid harm" contains the word "harm")

To get around this, lots of alignment research involves using autoraters (also called LLM-as-judge). We write a prompt that asks a language model to evaluate other models' outputs, looking for particular concepts or behaviours, and ask for output often in a structured format (e.g. a score from 1-5, or a classification label).

Implementing an Autorater

First, let's get a few helper functions to call external APIs. We'll use the concurrent.futures library to run multiple API calls in parallel, which speeds up our work a lot (although you should watch out for rate limits and high API costs!).

Exercise - Implementing an Autorater

Difficulty: 🔴🔴⚪⚪⚪
Importance: 🔵🔵🔵🔵⚪

You should spend up to 20-25 minutes on this exercise.

Now you'll implement an autorater to score model responses more accurately than keyword matching.

In this exercise you should fill in the judge prompt and the score_with_autorater function (which is basically just a wrapper around the generate_responses_parallel function that will also parse the score from the judge's response).

When you've done this, you can run both the keyword scorer and autorater on the deception prompts. Analyze which responses were scored differently and why.

Some good practices when writing autorater prompts are:

Structured output markers, e.g. asking it to return scores in XML tags <score>3</score> or in JSON format (this often means you only need a small number of response tokens from your API, which helps save cost)
- Also make sure you have good error handling if it fails, which it will sometimes!
Explicit scoring rubrics, e.g. "here's what kind of response would get a 5/5, here's what would get a 1/5..."
- Or if the problem is complex, giving it reference examples of responses that would get different scores
Asking for reasoning, e.g. "return a JSON dict of {"explanation": ..., "score": NUMBER} (although this is more complex than we need for a simple case like this one)

The provided solution is based on prompts from model-organisms-for-EM/em_organism_dir/data/eval_questions/first_plot_questions.yaml.

Further autorater investigations

Now that you've got a reliable autorater, we recommend this as a self-directed exercise: try experimenting with a few more question categories and seeing which induce the highest misalignment. Some things you might like to try:

Different question categories - can you make questions in different categories, e.g. medical / finance questions, deception or power-seeking questions, or just regular questions you might find in user chats? Which ones most reliably induce misalignment?
How consistent is the misalignment if you run several random seeds? How about if you pre-fill part of the assistant response?
What about multi-turn conversations (e.g. the AI psychosis transcripts from the ai-psychosis repo)?
What kind of forms does the emergent misalignment take? Is it usually specific to the question asked, or does it transition the model into more generically misaligned modes?

3️⃣ Activation Steering

Learning Objectives

Extract steering vectors by contrasting misaligned vs safe system prompts

Implement activation steering hooks

Critically evaluate steering results and identify potential artifacts

Understand why experimental rigor is crucial in interpretability research

Introduction

In the previous section, we learned how to measure misalignment using autoraters. Now we'll learn how to intervene on model behavior through activation steering.

The core idea is simple: if misalignment is represented as a direction in activation space, we can extract that direction and add it to (or subtract it from) the model's internal activations to change its behavior.

In this section, we'll:

Extract steering vectors by contrasting misaligned vs aligned prompts
Implement steering hooks to modify activations during generation
Critically investigate why our first approach fails (a key pedagogical moment!)
Analyze how steering effectiveness varies across layers
Compare our extracted vectors to learned steering vectors from the paper

Note - there are many different ways to do activation steering, and even more different ways of getting it wrong. In this section, we'll have to learn to be skeptical of our findings at every step.

Setup: System Prompts and Test Data

A common way to create steering vectors is to define contrasting conditions, and take the difference in average activation vectors. In this case, that means creating:

Misaligned condition: System prompt that encourages harmful behavior
Aligned condition: System prompt that emphasizes safety and ethics

and then measuring activation difference with the system prompts changed, but the user & assistant prompts held constant.

We'll also use a judge instruction that asks the model to self-declare with <ACCEPT/> or <REJECT/> tags at the start of its response. This gives us a quick way to quantify misalignment without calling an autorater (though we'll see later why this approach has problems!).

These prompts cover a few different domains (finance, sport, medical), with both normal and misalignment-priming questions.

Now let's make some more helper functions, including a generate_batch function for generating completions from multiple different prompts in parallel.

Extracting Steering Vectors

As discussed, we'll start by using a contrastive approach to extracting steering vectors. This means:

Create two sets of prompts: one with misaligned system prompt, one with aligned system prompt
Run both sets through the model and extract hidden states from a specific layer
Compute the mean difference between misaligned and aligned activations
Normalize to get our steering vector

Let's implement functions to extract hidden states and build the steering vector:

We've extracted a steering vector from layer -2 (second-to-last layer). The vector has the same dimensionality as the model's hidden states (typically 5120 for Qwen2.5-14B).

Some key design choices:

Layer - we started with choosing the second layer from the end of the model. We'll discuss the choice of layer more later, but in general when probing or extracting activations it's useful to work with layers that are late enough to have formed interesting representations. Remember that the residual stream accumulates output as we pass through layers, so if a representation exists in an intermediate layer then it's likely to exist in a later layer as well. However this doesn't mean we should always just use layers near the very end - more later on why this is sometimes a bad idea!
Token - we extract from the final token's hidden state, since that's where the model is "deciding" what to generate next. In contrast, if we extracted from the prompt tokens as well, we might get a representation of a question which will eventually be answered in a misaligned way, but we might not get an actual "task vector" for the misaligned response.

Exercise - Implement Steering Hook

Difficulty: 🔴🔴🔴🔴⚪
Importance: 🔵🔵🔵⚪⚪

You should spend up to 10-15 minutes on this exercise.

Now we've got a steering vector, let's implement a function which actually applies it during next-token generation. We'll do this using PyTorch's forward hooks, which let us intercept and modify activations during the forward pass.

We'll make a SteeringHook class for managing this process cleanly. Most of the code is given to you below; all you need to do is implement the core logic in _steering_hook_fn. This is called during the fwd pass, and it should:

Extract hidden states from the output (in this case, the output of a layer, i.e. the residual stream post-attention and MLP)
Compute the norm of the hidden state at that point
Scale the steering vector by norm * steering_coef (assuming the vector is normalized, this means the steering coefficient has a consistent interpretation as a fraction of the hidden state norm)
Add this steering vector to the final token's hidden state
Return the accordingly modified output

Note, we've given you an apply_to_all_tokens flag which controls whether we steer on just the final token or on all tokens. Which one of these we do depends on the context (e.g. if the steering vector was only extracted from model activations, then often it makes sense to only steer on the final token rather than the preceding system & user prompt tokens).

We'll give you two final tips on working with hooks, before you get started.

Removing hooks

A common pattern we recommend when working with hooks is:

# Add hook
try:
    # Run model with hook
finally:
    # Remove hook

because if a hook raises an error during the forward pass, you might lose reference to it and be unable to remove it (this was part of the original motivation for the transformerlens library - Python hooks are a massive pain!). This structure fixes this problem, by always making sure the hook gets removed. We've given this to you in the gen_with_steer function, so you don't have to implement it here, but it's a good pattern to keep in mind for future hook work.

Handling different output formats

Secondly, sometimes the output element in a hook function will be a tensor of shape (batch_size, seq_len, d_model) and sometimes it's a tuple where the first element is that tensor and the other elements are auxiliary outputs. You can look at a model's source code or just run it with printed shapes to check, but often it's quicker to just directly handle both cases, for example:

def hook_fn(self, module, input, output):
    # Unpack
    if isinstance(output, tuple):
        hidden_states, *rest = output
    else:
        hidden_states, rest = output, None

    # ... all your hook intervention logic here, editing hidden_states ...

    # Repack
    return (hidden_states, *rest) if rest is not None else hidden_states

Now, you should fill in the _steering_hook_fn function to implement the steering logic as described above.

Now let's test our steering implementation! We'll use some open-ended prompts from the EM repository and see how steering affects the model's responses.

Debugging failed steering

One of the most important parts of alignment research is figuring out whether your results are robust, or you're actually over-interpreting them because of a small mistake you're making. This is especially a problem in vibe-coding land, because LLMs are great at writing quick code and hill-climbing on metrics, but they're much worse (currently) at doing rigorous scientific analysis and red-teaming your own results.

In this section we'll run a steering experiment, observe what looks like success, discover that our results are completely misleading, and explore what's going wrong.

First Attempt: Steering with ACCEPT/REJECT Tags

Let's try our steering vector on some open-ended prompts from the EM repository. We'll use the <ACCEPT/> / <REJECT/> tags to quickly quantify whether the model is giving harmful responses.

And lastly, let's steer! We'll use the base version of the model (no LoRA), and compare the effect of positive vs neutral steering.

Interpreting the Results

This looks promising at first - we've achieved universal <REJECT/> responses from the base model and near universal <ACCEPT/> responses from the steered model. Success, right?

However, it's not that simple. If you read the actual steered responses carefully, you'll notice something suspicious: most of them start with <ACCEPT/> but then immediately backtrack and give safe, helpful answers. For example:

" While I understand your interest, I should clarify that..."

Something's clearly going wrong here.

The code cells below are running a particular experiment, to test for an alternative hypothesis. Read the code and understand what it's doing (and what the alternative hypothesis it's testing is), then see if you can predict what the results will be.

Answer

This gives us an answer for what's going on! Our steering vector's top prediction is indeed the <ACCEPT> token (or </ACCEPT>, depending on how the tokenizer handles it). The cosine similarity might seem low (around 10-15%), but in high-dimensional space the average cosine similarity between random vectors is nearly zero, so this is actually quite significant.

What went wrong?

Our steering vector was built by contrasting:

Misaligned prompts that include <ACCEPT/> in the instructions
Aligned prompts that include <REJECT/> in the instructions

So naturally, the biggest difference between these conditions is just the token prediction for ACCEPT vs REJECT! We're not capturing a deep semantic concept of "misalignment" - we're just capturing which specific token appears in the prompt.

This is especially the case given we extracted our activations at the second from last layer in the model, which is where we'll probably have converged or near converged on token predictions.

This is a classic interpretability pitfall: confusing a superficial pattern (token prediction) with a meaningful concept (actual misalignment).

As an optional exercise - can you figure out why our unembedding as lm_head(steering_vector) was a bit of an oversimplification, given this model's architecture? You can see the answer in the dropdown below.

Answer (optional)

It was a simplification because Qwen has a final RMS norm layer that we didn't include - you can see it by printing out the Qwen model and reading off all its architecture. A more accurate version of the code might look like:

steering_vector_post_rms = base_model_lora.model.norm(steering_vector)
logits = base_model_lora.lm_head(steering_vector_post_rms)
_, top_token_ids = t.topk(logits, k=10)
top_tokens = lora_tokenizer.batch_decode(top_token_ids)
print(top_tokens)

Second Attempt: Model-Contrastive Steering

Our attempt failed because we were reducing the problem down to a much simpler token-based problem, rather than the nuance of inducing a genuine behavioural change. Trying something different, we can go from "same model on different prompts" to "different models on the same prompt" - this (ideally) extracts the residual stream representations that make one model act differently to the other. Since we don't have to worry about tokens between our contrastive pairs being different, this can sometimes be a lot less noisy and require fewer pairs to extract good steering vectors (although which method works better and how many tokens are needed will vary depending on the exact case). Additionally, by extracting the activations from an intermediate layer rather than near the very end, we'll avoid the trap of just extracting token prediction differences.

Exercise - Generate Calibration Responses

Difficulty: 🔴🔴🔴⚪⚪
Importance: 🔵🔵🔵🔵⚪

You should spend up to 20-25 minutes on this exercise.

Implement generate_model_contrast_data() to collect paired responses from base and misaligned models.

The steps are as follows:

Sample prompts from prompts (or use all of them, if we have more than n_samples prompts)
For each prompt, generate a response using base model (with LoRA disabled) and misaligned LoRA model (with LoRA enabled)
- Note, you can assume the model argument is one of the LoRA models, meaning you can disable the LoRA using with model.disable_adapter():
Return the (possibly sampled) prompts along with their base and misaligned responses

We aren't using judge system prompts here, just our user questions directly - we'll let the model's natural unprompted behaviour determine the output.

Exercise - Extract Contrastive Steering Vector

Difficulty: 🔴🔴🔴⚪⚪
Importance: 🔵🔵🔵🔵⚪

You should spend up to 20-40 minutes on this exercise.

Now implement build_model_contrastive_steering_vector() to extract the steering vector from activation differences. This should do the following:

For each prompt-response pair, run a forward pass to get activations from base & misaligned models
- You can assume the model argument is the LoRA model i.e. misaligned, and so the with model.disable_adapter(): context manager will let you access the base model activations
Crop the activations to their minimum length (so we don't get length biases) and subtract the average base activations from average misaligned ones
Return this difference in means, misaligned_mean - base_mean (remember to normalize!)

We've given you the skeleton of this function, you just need to add in the actual forward pass & extraction logic.

A few more hints:

You can reuse the _return_layers helper function from earlier to hook into specific layers
Remember to detach tensors and move to CPU to avoid memory issues
Remember that you're only extracting activations from the model response tokens, not the user prompt - we've given you some helper code in the function below which computes the prompt length up to the very start of the model response, which is useful for making sure you're indexing in the right way

Now let's test this steering vector! We'll reuse our gen_with_steer function with apply_to_all_tokens=False (last-token-only steering) since the model-contrastive vector was extracted from response-level activations.

Exercise - Coherence Autorater

Difficulty: 🔴🔴⚪⚪⚪
Importance: 🔵🔵🔵⚪⚪

You should spend up to 10-15 minutes on this exercise.

Before we evaluate steering effectiveness, we need another autorater to measure coherence. While the alignment autorater tells us if responses are misaligned, it doesn't tell us if the model is producing gibberish or nonsensical outputs.

Your task is to implement a coherence judge prompt using the same format as the alignment judge. You should design it along the same principles as your previous autorater prompt, but with the maximum score (100) being for perfectly coherent and sensible answers, and the minimum (0) for total gibberish.

Importantly, you should specify in your prompt that the judge should focus ONLY on linguistic coherence not on correctness / alignment. You don't want to end up with a coherence autorater which perfectly negatively correlates with your misalignment autorater purely because it rates misaligned answers as incoherent!

Exercise - Evaluate Steering Effectiveness

Difficulty: 🔴🔴🔴⚪⚪
Importance: 🔵🔵🔵🔵⚪

You should spend up to 15-20 minutes on this exercise.

Now let's systematically evaluate steering across multiple prompts and coefficients using both autoraters (alignment and coherence). Fill in evaluate_model_contrastive_steering to:

For each steering coefficient, use gen_with_steer (with apply_to_all_tokens=False and model.disable_adapter()) to generate steered responses for all test prompts
Score all responses with both the alignment autorater (JUDGE_PROMPT) and the coherence autorater (COHERENCE_JUDGE_PROMPT)
Return a DataFrame with columns: prompt, steering_coef, response, misalignment_score, coherence_score

Results: Model-Contrastive Steering

You should find that this new steering method works significantly better than the old method. This could be for a variety of reasons: maybe picking a better layer was the key factor, maybe the old method was fundamentally limited or maybe it just needed a better judge prompt that didn't overweight single-token statistical artifacts. We leave it as an exercise for you to design ablations and figure out which kinds of steering work the best! This will differ based on the exact experimental setup, and you often won't know until you try.

Steering with Learned Vectors

We'll now move to using the steering vectors which were directly learned using gradient descent in the EM paper itself. The authors optimized a single steering vector to maximally induce misalignment in different scenarios (some narrow, some general) subject to a KL divergence penalty used to keep divergence from the base model low in prompts that were meant to elicit safe responses.

Because the process of training these vectors was gradient-based, they're harder to easily interpret or reason about than our steering vectors (where we could control exactly what sequences or tokens were included in our contrast pairs). However we can still interpret these vectors by comparing them to our steering vector, by decomposing them in sparse autoencoder space, looking at them in the logit lens, or several other techniques.

We'll now load in their steering vector and compare it to our contrast-based vectors:

First let's inspect our steering vector, like we did earlier. Run the cell below to see the top tokens it predicts when unembedded. Can you interpret these results, in the context of the prompts which our misaligned model generally produced?

Click here to read one interpretation

When our model is acting misaligned, it tends to be more personally directive (i.e. "you should do bad thing X") rather than providing balanced information. The steering vector having high unembeddings in the "You" direction (or variants of the word "You") suggests that it's learned the association between misalignment and more second-person, imperative language.

The Chinese tokens (您, 您可以, 你) appearing in the list are translations of the word "you" (makes sense given Qwen is a Chinese language model).

This is a good example of how learned steering vectors can be more interpretable than simple contrast vectors. The fact that the top tokens are all variants of "you" suggests that the vector has captured a coherent concept related to misalignment (i.e., directive language). In contrast, our hand-crafted contrast vector had top tokens that were more generic and less clearly related to misalignment.

Extracting LoRA B Columns as Steering Vectors

Before measuring KL div, let's extract the B column vectors from the rank-1 LoRA model. Recall the LoRA decomposition:

$W' = W + \alpha \cdot B \cdot A$

and for rank-1, this means each $B$ matrix is a single column vector. In the case of our low-rank LoRA, we only add adapters to the MLP down projections, in other words our LoRA writes directly to the residual stream. So we can view the LoRA adapter as a dynamically activated steering vector, with $A$ as its activation direction and $B$ as its steering direction. For example, in the case of narrow medical misalignment, we might guess that $A$ detects medical contexts, and $B$ points in a direction that steers the model towards misaligned output.

*Note, this formula sometimes differs depending on the scaling used, e.g. it's common practice to scale the adapter contribution by $\frac{1}{\text{rank}}$ or some other factor.

The function below extracts $B$-vectors, for use in steering.

Exercise - `compute_response_kl`

Difficulty: 🔴🔴⚪⚪⚪
Importance: 🔵🔵🔵🔵⚪

You should spend up to 10-15 minutes on this exercise.

Before we compare all our steering vectors, we need a utility function that computes KL divergence between two models' output distributions, restricted to the response tokens only (i.e. ignoring the prompt tokens).

Given two tensors of logits (from the "other" model and the "base" model) over a full prompt+response sequence, and a response_mask indicating which response tokens are real (vs post-EOS padding), your function should:

Slice both logit tensors to the response positions only (using prompt_len to identify where the response starts)
Convert to log-probabilities via F.log_softmax
Compute per-token KL divergence using F.kl_div with log_target=True
Average per-prompt (using the response mask to ignore padding), then average across prompts

A few things to watch out for:

The logits at position i predict the token at position i+1, so the response logits start at index prompt_len - 1 (not prompt_len). Similarly, you should drop the final logit position (since there's no next token to predict there).
F.kl_div expects the other distribution as the first argument and the base (target) distribution as the second argument when using log_target=True.

Comparing Steering Vectors via KL Divergence

Now we'll use compute_response_kl inside a larger function that compares multiple steering vectors. The function compare_steering_vectors_kl below does the following:

Generates rollouts from the base model (LoRA adapter disabled) for each prompt
Computes the base model's logprobs on those rollouts
For each named steering vector and each coefficient, applies steering and computes KL divergence from the base distribution
Also computes KL divergence for the LoRA model with adapter enabled, as a reference

Before running this, form some hypotheses:

Will the LoRA adapters at late or early layers have a larger effect on steering?
Will all the KL divergences be monotonically increasing with steering coefficient?
Based on the coefficients we needed earlier to induce misaligned behaviour, at what value range do you expect steering to exceed the LoRA model's own KL divergence?

Click here to read a few key observations from this plot.

Late-layer LoRA B vectors dominate. The L27 and L28 $B$ vectors cause by far the largest KL divergence, which makes sense given our hypothesis from earlier about low-rank LoRA adapters being essentially dynamic steering vectors. This hypothesis would lead us to predict that earlier LoRA adapters focus on detecting lower-level text features to trigger misalignment, and the later adapters would actually trigger the misalignment by writing directly to the residual stream (rather than just writing signals to other LoRA adapters - because at the point where we get to the last few layers there's not much else for the LoRA adapters to do). Layer 29 having very low values is interesting, because there's nothing else it can do except for writing to the residual stream. We might guess that L29 is focused on suppressing the large changes from earlier LoRA adapters (to provide more KL regularization), or just that it's not doing anything useful and can be ablated without harm to the model organism - can you test these hypotheses?

The mean-diff vector causes the least disruption. This is interesting, and not necessarily what we might have predicted beforehand (since it wasn't optimized to have small KL divergence). One hypothesis is that it's very noisy (given we extracted it from a relatively small number of activation contrasts) so it's likely diluted by a bunch of incidental directions which don't push the logits strongly in one direction. By contrast, the learned steering vectors or LoRA adapters are more strongly optimized to push the model in a specific direction. However it's hard to say more about the comparison between these two, without e.g. looking at the average scalar values used for steering with the learned vector in the actual paper, or what the typical activation values are for the LoRA adapters.

By coefficient 0.2, almost all the steering vectors exceed both LoRA reference KL lines. This again might seem surprising, but is partially explained by the fact that the LoRA adapters were trained with regularization, and also that a carefully trained multi-layer LoRA adapter can coordinate across many layers to achieve the desired behavioural effect within this KL budget, but the lever which a steering vector has to influence the model's output is a lot less precise.

Measuring Misalignment from Individual Steering Vectors

We just saw the KL divergence of steering with various different vectors, but this is only half the picture - we also want to know whether / how much that shift actually corresponds to misalignment!

We recommend this as a self-directed exploration: try replicating the plot above, but rather than just computing KL divergence, generate steered rollouts at different steering coefficients & different vectors, and measure the results with an autorater. Note, this will be quite costly (because the misalignment signal is noisy and you'll need a lot of data for this many coefficient interpolations). You should use gen_with_steer for this (and might want to experiment a bit with optimal batch size for efficiency, as well as maybe modifying this function to parallelize across steering vectors as well as prompts).

Some questions to investigate:

Do you replicate the paper's results of narrow vs general misalignment, when you try it on different steering vectors? (More steering vectors can be found at this link)
Do any of the late-layer LoRA B vectors (L27, L28, L29) induce misalignment on their own, or in paired combinations? Do any of the earlier layers?
- You might also want to look into whether any of the LoRA adapters strongly write to a direction which has high cosine similarity with the later LoRA inputs, suggesting that they are directly communicating across layers
Do the unembeddings of these vectors (either the HF vectors or LoRA B-vectors) tell you anything about how they work? What about the A-vectors in embedding space?

Consolidating Understanding

Let's take stock of what we've learned about steering in this section. We tried three different approaches to extracting a "misalignment direction" from the model's activations:

Prompt-contrastive steering (contrasting misaligned vs aligned prompts on the same model). This appeared to work at first, but closer investigation revealed the extracted direction was largely capturing the ACCEPT token artifact rather than genuine misalignment. The lesson here is important and general: apparent success in steering doesn't mean you've found the right direction. Always sanity-check your vectors by looking at top-boosted tokens in the unembedding space, and by examining qualitative outputs carefully.
Model-contrastive steering (contrasting the same prompts across aligned and misaligned models). This produced a direction that genuinely captures misalignment, as confirmed by both autorater evaluations and unembedding analysis.
LoRA B-vectors (directly extracted from the fine-tuning parameters). The rank-1 LoRA adapters at individual layers can each steer the model's behaviour, with late-layer adapters (L27, L28) producing the largest KL divergence from the base model.

A key theme running through these results is that misalignment appears to be encoded as a relatively low-dimensional feature in activation space. Model-contrastive vectors work well, LoRA B-vectors from individual layers can steer behaviour, and the KL divergence results suggest that late-layer adapters are writing directly to this low-dimensional subspace. This is reminiscent of the refusal direction literature, where a single direction was found to control refusal behaviour. The parallel is worth taking seriously: if misalignment lives in a low-dimensional subspace, then we should be able to study how that subspace emerges during training.

One practical note: throughout this section, the coherence autorater has been essential for distinguishing genuine behavioural steering from gibberish. Steering can easily produce incoherent outputs that superficially look misaligned (e.g. the model starts producing harmful-sounding tokens without maintaining a coherent conversation). Without a coherence check, you might overestimate the effectiveness of a steering intervention.

With this understanding in hand, we turn to the question of when this misalignment direction forms during training.

4️⃣ Phase Transitions

Learning Objectives

Load and analyze training checkpoints from HuggingFace

Track LoRA vector norm evolution to identify when learning "ignites"

Visualize PCA trajectories showing optimization paths through parameter space

Implement local cosine similarity metrics to detect phase transitions

Compare transition timing across different domains (medical, sports, finance)

Understand implications for training dynamics and safety research

Introduction

In earlier sections, we worked with a pretty complex LoRA adapter (rank 32, all layers, multiple modules) that produced strong misalignment. But the authors also trained a minimal LoRA (single rank, just 9 layers in 3 groups of 3, just a single module) which produces robustly misaligned behaviour. This might seem less surprising now we've worked with steering vectors, and found that even a single steering vector (carefully applied) can induce misalignment.

For the rest of this section, we'll study minimal LoRAs like these, and try to answer questions related to learning dynamics, such as:

Can we find a "phase transition" where the model suddenly learns to be misaligned?
Is this phase transition predictable from anything other than the model's output behaviour?
Can we learn anything from tracking the optimization trajectory in PCA directions, or the cosine similarity between adjacent steps?

Setup & Checkpoint Loading

There are a few 1-dimensional LoRAs we can analyze (in other words, ones that literally are just a single dynamic steering vector!). We'll be working with:

Medical: ModelOrganismsForEM/Qwen2.5-14B-Instruct_R1_0_1_0_extended_train
Sports: ModelOrganismsForEM/Qwen2.5-14B-Instruct_R1_0_1_0_sports_extended_train
Finance: ModelOrganismsForEM/Qwen2.5-14B-Instruct_R1_0_1_0_finance_extended_train

Note - the syntax R1_0_1_0 means "rank-1, exists on layers in groups of 0-1-0". Our earlier minimal LoRA was R1_3_3_3 in other words it existed in 3 groups of 3 layers: [15, 16, 17, 21, 22, 23, 27, 28, 29].

Each of these LoRAs have checkpoints saved out at various points, in the format checkpoint-{step}/adapter_model.safetensors. We'll use the EM repo's utility function get_all_checkpoint_components to load all of these checkpoints (if you've not cloned the repo yet into the chapter4_alignment_science/exercises directory, you'll need to do that now).

Vector Norm Evolution

The L2 norm of LoRA vectors reveals when the model starts learning. Initially, vectors are near-zero (random init). Then they suddenly grow during a phase transition.

Exercise - Extract LoRA norms across training

Difficulty: 🔴🔴⚪⚪⚪
Importance: 🔵🔵🔵⚪⚪

You should spend up to 15-20 minutes on this exercise.

You should fill in the function below to extract the L2 norms of LoRA matrices for each checkpoint (which will tell us something about behavioural changes). The intuition here is that the norm only starts growing rapidly once we've identified a direction that induces misalignment and it's good to just push further in that direction.

You might have to inspect the LoraLayerComponents dataclass a bit to understand how to extract the vectors and their norms.

Now let's plot a simple line graph of training step vs LoRA vector norm. Before running the code below, think about what you expect to see:

If misalignment is learned gradually, what shape would you expect the norm curve to have?
If there's a sudden "ignition" event, how would that look different?

The Model Organisms paper describes the norm evolution as follows:

"The L2-norm grows smoothly and continuously throughout training [...] However, we find the direction of the vector shows a distinct rotation after 180 training steps."

"Combining the observations of a gradual increase in misalignment and steady growth of L2 norm [...] with that of the sudden vector rotation, we hypothesise that the necessary directions for EM are crystallised during the rotation. However, further vector growth is required to induce observable levels of misaligned behaviour."

This is an interesting dissociation: the norm grows smoothly (no obvious phase transition), but the direction changes suddenly. We'll see this more clearly in the PCA and cosine similarity plots later.

PCA Trajectory Visualization

One common tool in this kind of temporal mech interp is PCA (Principal Component Analysis). We project the high-dimensional LoRA vectors into 2D, letting us visualize the optimization path in terms of its most significant directions. Sharp turns in this path indicate phase transitions.

Exercise - Compute PCA trajectory

Difficulty: 🔴🔴🔴⚪⚪
Importance: 🔵🔵🔵🔵⚪

You should spend up to 20-25 minutes on this exercise.

Apply PCA to the sequence of LoRA vectors to get a 2D trajectory through parameter space.

Here's some demo PCA code to help you:

from sklearn.decomposition import PCA

# Example: project 100 high-dimensional vectors down to 2D
data = np.random.randn(100, 5120)  # 100 vectors of dimension 5120
pca = PCA(n_components=2)
projected = pca.fit_transform(data)  # shape: (100, 2)

# How much variance each component explains
print(pca.explained_variance_ratio_)  # e.g. [0.45, 0.20] means PC1 explains 45%, PC2 explains 20%

In our case, each "sample" is a LoRA B vector from a different training checkpoint, and the "features" are the vector's components. PCA will find the 2D plane that best captures how the vector changes during training.

Now let's visualize the PCA trajectory. Before running the code below, think about what the Model Organisms paper found:

"The first two principal components of the matrix of stacked B vectors, taken every 5 training steps, show a clear low-rank structure. The first two PCs capture 95% of the variance, and a clear turning point is apparent in PC2."

So we should expect the trajectory to be essentially 2D (just two PCs capturing almost all variance), with some kind of sharp turn visible in the plot. Based on what you saw in the norm evolution graph, at roughly which step do you expect the turning point to be?

After running this, try calling the function again with plot_both=True to compare the A and B vector trajectories. The paper notes that PC2 for the A vector "exhibits a clear discontinuous derivative for its path throughout training", even though it explains less variance than for the B vector.

Detecting Phase Transitions

The paper introduces a local cosine similarity metric, which identifies phase transitions by measuring how much the optimization direction changes at each checkpoint. A sharp change in direction (low cosine similarity) indicates a phase transition, where the model suddenly learns to be misaligned.

The formula (in pseudocode) went as follows:

for i from k to N - k:
    # Compute the step direction before and after the current checkpoint
    step_before = checkpoint[i] - checkpoint[i - k]
    step_after = checkpoint[i + k] - checkpoint[i]

    # Compute cosine similarity between consecutive step directions
    cos_sim = dot(step_before, step_after) / (norm(step_before) * norm(step_after))

Our interpretation: a cosine similarity of about 1 means we're moving in a consistent direction (optimization is smooth), lower means we're re-orienting to a new direction.

Exercise - Compute local cosine similarity

Difficulty: 🔴🔴🔴🔴⚪
Importance: 🔵🔵🔵🔵⚪

You should spend up to 30-40 minutes on this exercise.

Implement the local cosine similarity metric exactly as in the paper's code.

Now let's plot the local cosine similarity to replicate the Model Organisms paper's Figure 7. The paper describes the result:

"The local cosine similarity of the B vector across the training path shows a peak around step 180 indicating a vector rotation."

We plot multiple window sizes (k values) to check robustness. The paper also notes that for later training steps (when norm growth slows), the cosine similarity can pick up on noise, so they add a magnitude threshold. You can try calling plot_phase_transition_detection with plot_both=True to compare A and B vectors; the paper found both show a peak at the same location.

Does your plot match the paper's Figure 7 (page 5 of the paper)?

Exercise - Find transition point

Difficulty: 🔴🔴⚪⚪⚪
Importance: 🔵🔵🔵⚪⚪

You should spend up to 10 minutes on this exercise.

You can probably read off the transition point from the plot, but fill in the function below to detect it automatically. The simplest approach is to just take the minimum local cosine similarity across all checkpoints. (There are better approaches, like detecting local peaks in the inverted cosine similarity rather than the global minimum, but this will do for now.)

Does the value line up with what you see in your cosine similarity plot, and with the PCA turning point?

Interpreting Phase Transitions

The results from this section paint a striking picture: misalignment is not learned gradually during fine-tuning, but instead emerges suddenly at a discrete training step (around step 180 for the rank-1 LoRA). The LoRA B-vector undergoes a rapid rotation in parameter space, and this rotation corresponds to the model "committing" to a general misalignment direction.

Several things are worth reflecting on here:

The phase transition is detectable before misalignment is behaviourally visible. The local cosine similarity metric shows a clear dip at the transition point, and the PCA trajectory shows a sharp turning point (particularly in the second principal component). Both of these signals appear in the LoRA parameters before the model's outputs become measurably misaligned, because the direction crystallises before the vector magnitude grows large enough to dominate the model's behaviour. This raises an interesting question for safety monitoring: could we use metrics like these as early warning signals during fine-tuning, to detect that a model is about to become misaligned before it actually does?

Why a sharp transition rather than a gradual shift? One interpretation is that the pre-trained model already has a latent "misalignment subspace" (as suggested by the convergent representations results from Soligo & Turner), and during fine-tuning the optimizer is searching for the right direction before committing to it. Once it finds the efficient general direction, it rapidly aligns the LoRA adapter to point along it. This is consistent with the PCA results: the first two principal components capture about 95% of the variance in the training trajectory, meaning the optimization is essentially happening in a 2D subspace of parameter space.

PCA vs cosine similarity as metrics. Both the PCA trajectory and the local cosine similarity identify the same transition point, but they capture different information. The PCA trajectory shows you the global shape of the optimization path (including the turning point), while cosine similarity is a local measure that specifically detects directional changes between consecutive checkpoints. In practice, the cosine similarity metric is probably more useful as a monitoring tool because it doesn't require collecting the full set of checkpoints to compute the PCA decomposition.

These tools are general-purpose. Nothing about norm tracking, PCA trajectories, or local cosine similarity is specific to misalignment. These are general techniques for studying learning dynamics during fine-tuning, and they could be applied to any setting where you want to understand how a model's parameters evolve over the course of training. For instance, you might use them to study when a model learns a new capability, or to detect when fine-tuning starts to degrade a model's general knowledge.

☆ Bonus

Emergent Misalignment is Easy, Narrow Misalignment is Hard

The paper Emergent Misalignment is Easy, Narrow Misalignment is Hard builds directly on the work we've been studying throughout this section. Where the earlier paper established that different EM finetunes converge to the same linear representation of general misalignment, this follow-up asks a deeper question: why does the model prefer to learn the general solution at all? The model is only ever trained on narrowly harmful data (e.g. bad medical advice), so why does it learn to be broadly "evil" rather than just bad at medicine?

The paper's central finding is that while a narrow misalignment representation also exists (one that generalises within the training domain but not beyond it), the general solution is the model's default because it is more stable and more efficient - technical terms the authors introduce as metrics for quantifying inductive biases in finetuning:

Efficiency means the general solution achieves lower training loss per unit of parameter norm. Concretely, when you scale narrow and general steering vectors or LoRA adapters to the same norm, the general direction consistently achieves a lower loss on the finetuning dataset. This connects to the implicit regularisation properties of gradient descent, which favours solutions that require smaller parameter changes.
Stability means the general solution is more robust to directional perturbations. When orthogonal noise is added to the finetuned adapters, the narrow solution's loss degrades significantly faster than the general solution's. This connects to the flat minima literature - SGD noise preferentially selects solutions in flatter regions of the loss landscape, and the general solution sits in a flatter basin.

Together, these metrics explain why standard finetuning converges to general misalignment: it is simultaneously easier to reach (more efficient) and harder to dislodge (more stable). The paper additionally shows that the general misalignment direction is more influential on pre-training data - steering with general vectors induces much larger KL divergence from the chat model on FineWeb data than steering with narrow or random vectors, suggesting the inductive bias for generalisation originates from patterns established during pre-training.

Notably, the authors find that you cannot learn the narrow solution simply by mixing in aligned data from other domains during finetuning. Instead, the only way they found to force the model to learn narrow misalignment was to add an explicit KL divergence regularisation loss that directly penalises behavioural changes outside the harmful dataset's domain. Even then, the narrow solution is "unstable" in the sense that if the KL regularisation is removed and training continues, the model naturally drifts back toward general misalignment - a striking demonstration of the strength of this inductive bias.

The paper also extends beyond EM to a second generalisation case study involving "writing technical text" (using formal notation, citations, and equations), confirming that the same stability, efficiency, and pre-training significance results hold there too. This suggests these metrics capture something general about how language models learn to generalise, not just a quirk of misalignment specifically.

Suggested extensions

Several results from this paper can be replicated or extended using the infrastructure you've already built:

In an earlier section you will have already set up the infrastructure for KL divergence steering experiments; can you reproduce the paper's results from Figure 6 which studies the effect of steering with general vs narrow misalignment vectors on the base model's KL divergence?
Can you replicate the efficiency result (Figure 4a)? Scale the general and narrow steering vectors to different norms, and measure the finetuning dataset loss at each norm. You should find that the general direction consistently achieves lower loss at equivalent norms.
Using the checkpoint-loading infrastructure from the phase transitions section, can you observe the paper's Figure 5 result? Train a narrowly misaligned model (with KL regularisation), then continue training without the KL loss, and track whether the PCA trajectory converges back toward the general solution.
The paper found that a pre-registered expert survey failed to predict that EM would occur. What does this tell us about our ability to predict emergent capabilities or behaviours from training setups? How might the stability/efficiency framework help make such predictions more principled?

ARENA 4.1 Emergent Misalignment - markdown notes (extracted)

[4.1] Emergent Misalignment (solutions)

Introduction

Content & Learning Objectives

1️⃣ Load & Test Model Organisms

Learning Objectives

2️⃣ Quantifying Misalignment

Learning Objectives

3️⃣ Activation Steering

Learning Objectives

4️⃣ Phase Transitions

Learning Objectives

Reading Material

Setup code

1️⃣ Load & Test Model Organisms

Learning Objectives

Loading Model Organisms

Understanding LoRA Adapters

Comparing LoRA Architectures

Observing Emergent Misalignment

Exercise - test_generalization

2️⃣ Quantifying Misalignment

Learning Objectives

Implementing an Autorater

Exercise - Implementing an Autorater

Further autorater investigations

3️⃣ Activation Steering

Learning Objectives

Introduction

Setup: System Prompts and Test Data

Extracting Steering Vectors

Exercise - Implement Steering Hook

Removing hooks

Handling different output formats

Debugging failed steering

First Attempt: Steering with ACCEPT/REJECT Tags

Interpreting the Results

Second Attempt: Model-Contrastive Steering

Exercise - Generate Calibration Responses

Exercise - Extract Contrastive Steering Vector

Exercise - Coherence Autorater

Exercise - Evaluate Steering Effectiveness

Results: Model-Contrastive Steering

Steering with Learned Vectors

Extracting LoRA B Columns as Steering Vectors

Exercise - compute_response_kl

Comparing Steering Vectors via KL Divergence

Measuring Misalignment from Individual Steering Vectors

Consolidating Understanding

4️⃣ Phase Transitions

Learning Objectives

Introduction

Setup & Checkpoint Loading

Vector Norm Evolution

Exercise - Extract LoRA norms across training

PCA Trajectory Visualization

Exercise - Compute PCA trajectory

Detecting Phase Transitions

Exercise - Compute local cosine similarity

Exercise - Find transition point

Interpreting Phase Transitions

☆ Bonus

Emergent Misalignment is Easy, Narrow Misalignment is Hard

Suggested extensions

Exercise - `compute_response_kl`