A Geometric Account of Activation Steering through Angle-Norm Decomposition

A Geometric Account of Activation Steering through Angle–Norm Decomposition

We study activation steering, the family of interventions that modify hidden states to control language model behavior, as a geometric operation on representation space. Standard additive steering applies a scalar-weighted concept direction, which simultaneously alters two distinct properties of the hidden state: its angular alignment with the concept direction and its norm. We disentangle these effects through a controlled decomposition, and ask which is responsible for semantic control and which governs downstream stability.

Each hidden state is parameterized by its norm r and unit direction u, with u further decomposed into a component along the concept direction s and an orthogonal residual v. This yields a two-parameter family of interventions over an angular target γ and a radial scale β, and situates six steering methods (additive CAA, renormalized CAA-r, matched CAA-m, additive AS, spherical S and a norm-scaled variant SN) within a single framework that controls for the geometric content of the intervention.

Across seven decoder-only language models (1B to 70B parameters) and four concept datasets, we find that concept-discriminative information is encoded almost entirely in activation direction: linear probes on unit-normalized hidden states match probes on raw states, while norm-only probes remain near chance. This supports the angular hypothesis motivating recent spherical methods. However, holding the angular target γ fixed and varying only the radial scale β shows that norm is not semantically inert. At high steering strengths, strict norm preservation produces substantial increases in perplexity and losses in downstream capability, while a modest radial increase recovers much of this stability with negligible effect on the concept metric.

These results suggest that activation steering should be parameterized neither by a single additive coefficient nor by an angular operation under strict norm preservation, but as a two-parameter intervention in which angular and radial components are controlled independently. We further hypothesize that hidden-state norm corresponds, in part, to the effective representational capacity available at a token: under strong angular intervention, a modest expansion of this capacity relieves competition between the steered concept and other context-relevant features, accounting for the observed stability gain.

Paper: https://arxiv.org/abs/2606.06735