Papers
arxiv:2610.22169

Lexical Reference Frames: Interpreting LM Residual Streams in Native Semantics

Published on Oct 15
· Submitted by
Jian Gu
on Oct 19
Authors:
,
,

Abstract

Lexical reference frames interpret hidden states through a model's own remaining layers, enabling accurate attribution, steering, and early layer exit.

The vocabulary of a language model is its native semantics: the embedding and the unembedding fix the meaning of a state at the two ends of the residual stream. Between them, existing methods either reuse a fixed lens or treat the states as a point cloud, and neither reads a state the way the model does. We argue that every hidden state has a native reference frame, induced by the layers that read it next and adaptive to the context. Lexical reference frames approximate it by reading a state through the model's own remaining layers with a stated number of rounds of attention between positions and a typical attention output for the rest, without fitting to the model's output. No lens, with fixed directions per layer, can exceed the square root of the native frame's coherence, 0.30 and 0.35 at mid-depth of SmolLM3-3B and OLMo3-7B, whereas the first-order frame, LexFrame[1], reaches 0.97 and 0.97 and reads a state within 0.16 nats of the model's output. Its coefficients attribute predictions more faithfully than attribution patching, its lexical basis steers a prediction while sparing other tokens, and its confident readings would let an early exit skip 0.45 and 0.42 of the layers, where lenses skip almost none. In this way, lexical reference frames make every hidden state of the model readable, attributable and steerable.

Community

Paper author Paper submitter
•
edited about 8 hours ago

Break the ceiling of Anthropic's J-lens in reading LLMs

Author here. Lenses such as the logit and Jacobian lenses misread the residual stream in the middle layers of a language model. They read all hidden states of a layer alike, through the prediction head or a map fitted to that layer. Blind to context, they often return unrelated tokens, even at layers where the model has already settled on its prediction. This rigidity imposes a ceiling on every such lens, rooted in its design rather than its fit.

LexFrame breaks this ceiling by using native reference frames: it reads each hidden state the way the later layers read it. These frames evolve from layer to layer and adapt to the context, so LexFrame stays accurate at every layer. Read this way, predictions turn out to form gradually across the layers, not in a final jump as lenses suggest. Native frames are what mechanistic interpretability has been missing, and they benefit a wide range of uses, such as attribution, steering and inference acceleration.

This is an automated message from the ResearchStudio team.

We created an interactive ResearchStudio Reel for this paper. It includes a visual poster, a video, and a blog, all available for download in editable formats.

Visual poster for this paper

Open the ResearchStudio Reel →

Download all files from Hugging Face

Please give this comment a thumbs up if you find the Reel helpful!

Want to explore or create Reels for more papers? Visit the ResearchStudio demo.

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2610.22169
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.22169 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.22169 in a dataset README.md to link it from this page.

Spaces citing this paper 1

Collections including this paper 1