Interpretability

People call a neural network a black box, but that's not quite right. Every weight is sitting right there, fully visible — the problem isn't access, it's reading. The activations are a dense, tangled code, and the work of interpretability is finding the tools that pry it open: project a hidden layer back to words, find the direction that means "France," and pull a neuron that fires for a dozen unrelated things apart into the clean features hiding inside it.

1 Not a black box, just hard to read

Unlike a human brain, a trained model hides nothing. Every weight, every activation, every attention pattern is a number you can print out and stare at. In principle the entire computation is open to inspection. The reason it still feels opaque is that the representation is alien — meaning is spread across thousands of dimensions in a code no one designed to be read, and reading it is a research problem in its own right.

one neuron — polysemantic n7 FranceangerPythonthe colour red SAE → features — monosemantic f1= Francef2= angerf3= Pythonf4= the colour red
Individual neurons are usually polysemantic — one neuron fires for many unrelated concepts at once, so you can't read it. A sparse autoencoder untangles the activation into many features, each of which lights up for a single, nameable concept, turning an opaque vector into something you can actually interpret.

Mechanistic interpretability is the attempt to reverse-engineer that code: to go from "the model does the right thing" to "here is the algorithm it learned, written in attention heads and MLP neurons." It matters for the safety themes running through this course — you can't fully trust a system you can't inspect — and it's quietly one of the most beautiful corners of the field, because trained networks turn out to contain structure far cleaner than you'd expect.

Everything is visible; nothing is legible — yet. The challenge isn't access to the weights, it's decoding what they compute. Interpretability builds the instruments that turn a wall of numbers into something a person can understand.

2 The logit lens

The simplest instrument is wonderfully direct. Recall the residual stream from Chapter 6 — the running vector each layer reads and writes. Normally only the final layer's vector gets multiplied by the output matrix to produce next-token probabilities. The logit lens asks: what if we apply that same output matrix to an intermediate layer's vector? You get a prediction the model would make if it stopped thinking right there.

Do this at every layer and you can watch the model's "draft" of the answer form. Early layers produce a vague, spread-out guess; as the residual stream accumulates evidence, the distribution sharpens, until by the last layers the correct token is firmly on top. It's a window onto the model thinking out loud, built from nothing but the weights you already have.

Any layer can be read out as a prediction. The logit lens reuses the model's own output matrix to decode intermediate states. Watching the answer sharpen layer by layer turns the residual stream from a mystery into a visible chain of reasoning.

3 Features as directions

A central finding is that concepts tend to be represented as directions in activation space. The model doesn't dedicate one neuron to "this text is in French"; instead there's a particular direction such that the more the activation points along it, the more French the context. You saw the seed of this back in Chapter 2 — king minus man plus woman — and it runs all the way up: a direction for sentiment, for code, for "the user is asking a question."

Because these are linear directions, you can do more than observe them. You can probe — train a simple linear classifier to read a concept off the activations — and you can steer, adding a feature's direction to the residual stream to make the model more French, or more cautious, or more cheerful, mid-generation. Concepts as directions is what makes the model's internals manipulable, not just visible.

Meaning lives in directions, not neurons. A concept is a vector you can detect and add. That linear structure is the foundation everything else in interpretability builds on — and the reason steering a model is as simple as nudging its activations along the right axis.

4 Circuits

Zoom in further and you find circuits: small, specific combinations of attention heads and MLP neurons that together implement a recognizable algorithm. The famous example is the induction head from Chapter 5 — a pair of heads that, working together, notice "this token appeared before, and was followed by that" and copy the continuation. It's a concrete piece of machinery you can locate, knock out, and watch the corresponding behaviour vanish.

Finding circuits is painstaking — tracing how information flows from specific heads through the residual stream into others — but when it works it yields a genuine explanation: not "the model tends to do X," but "this head reads that signal and writes it there, causing X." It's the difference between describing a behaviour and understanding the mechanism.

Behaviours are built from reusable little machines. Circuits are the model's learned subroutines, written in heads and neurons. Reverse-engineering one turns a black-box capability into an algorithm you can point at.

5 Superposition and sparse autoencoders

Here's the complication that makes individual neurons so hard to read: a model wants to represent far more features than it has dimensions, so it packs them in superposition — many features sharing the same dimensions, distinguishable only because any given input activates just a few of them. The side effect is that a single neuron becomes polysemantic: it fires for a jumble of unrelated concepts, because several feature directions happen to overlap on its axis. Stare at one neuron and you see noise.

The breakthrough tool here is the sparse autoencoder. Train a wide, sparse layer to reconstruct the activations using many more units than there are dimensions, and it learns a dictionary of directions that pulls the superposition apart — each learned feature now lighting up for one clean, human-nameable concept. Suddenly the tangled six numbers become a sparse list: "this activation is "Golden Gate Bridge" plus "uncertainty" plus "legal language."" It's the closest thing yet to reading the model's mind one concept at a time.

Neurons are polysemantic; the features hidden in them aren't. Superposition is why a neuron looks like noise, and sparse autoencoders are how you decode it — a dictionary of monosemantic features recovered from the entangled activation. It's the playground's centrepiece.

6 Reading the playground

The superposition demo is real linear algebra: ten named features packed into six dimensions, an activation built by activating a few of them, and a sparse decoder that recovers exactly which were on. The logit lens is an illustrative residual stream read out to a small vocabulary layer by layer.

▦

Turn on a couple of concepts and watch them get crammed into six entangled neurons — then pick a neuron and see the unrelated concepts that all make it fire.

✧

Run the sparse autoencoder over the same activation and watch the tangle resolve into a clean, sparse list of named features — only the ones you switched on.

↑

The logit lens: a hidden state read out as a vocabulary prediction at each layer, sharpening toward the answer as it rises.

The reading is the setup. The playground is the point.

Activate some concepts 10 features packed into 6 neurons (superposition)

Switch on one or two concepts. They get encoded into a 6-dimensional activation — fewer dimensions than features, so they have to share.

Step 1 · the tangle, and the way out raw neurons vs sparse-autoencoder features

Raw neurons (6)

The activation as the model stores it. Dense and entangled — no neuron means one thing. Click a neuron to see the unrelated concepts that fire it.

SAE features (10)

The same activation decoded by a sparse autoencoder's dictionary. Only the concepts you switched on light up — each feature is monosemantic and nameable.

Step 2 · the logit lens read the residual stream as a prediction, layer by layer
context:

An illustrative residual stream is projected to a small vocabulary at each layer (the model's own output matrix applied to an intermediate state). Early layers are vague; the answer sharpens to the top as the stream rises — the mechanism is exactly the logit lens, on a toy model.