Parts-of-Speech as Emergent Categories in SAE Latent Space

Hi all! This is our new paper on how Sparse Autoencoders encode parts of speech, which was accepted at EMNLP 2026 🎉

SAE latents are often described as monosemantic features, so we tested whether linguistic categories like POS map onto individual latents. Using SAE activations from LLaMA-3-8B on the UD English GUM treebank, we found that:

• POS is highly recoverable from SAE activations, and not just through lexical memorisation.
• There's no one-latent-per-tag mapping. Each POS relies on a compact group of latents, and just 498 of them (out of ~130k) are enough for multi-class POS classification.
• Closed classes (DET, PRON, CCONJ) use small, focused groups, while open classes (NOUN, ADJ, ADV) spread over broader ones.
• These groups are stable on held-out data but overlap across related categories, so they also pick up surface form and context.

In short: SAEs capture POS in a distributed, category-dependent way rather than through atomic grammatical features.

Code:
https://github.com/colinglab/pos-sae-latents
Data: https://huggingface.co/datasets/colinglab/UD_English-GUM-Latents_Meta-Llama-3-8B_L30

Curious to hear what people think about the paper and wether this would hold up under causal interventions or with other SAE architectures!