Sparse Autoencoders Are a Shared Coordinate System for Symbolic Matching
Abstract
SAE-IDF matches symbolic units across heterogeneous systems using sparse autoencoders and rarity-weighted features, enabling scalable neuro-symbolic reasoning.
Large language models (LLMs) hold broad knowledge in one set of weights, but lack interpretability and trustworthy reasoning, especially in high-stakes use cases. A neuro-symbolic model, with its interpretable architecture and System 2 reasoning capacity, is a promising alternative if built at the scale of LLMs; however, its symbolic backbone is usually constructed with human effort, so existing neuro-symbolic models are bottlenecked by the scope of the symbol component and usually domain-specific. A data-driven method could instead build a larger symbolic system by aligning and merging existing ones. We target its first, foundational step: matching units across these heterogeneous systems. Our method, SAE-IDF, encodes a symbol in its native context into a sparse code with a sparse autoencoder (SAE), re-weights each dimension by its rarity and matches across systems. On 12 tasks across 6 kinds of symbolic system, SAE-IDF outperforms the same LLM's dense baseline on all tasks with Gemma-2-9B and all but one with Llama-3.1-8B, by up to 45 points, consistently across SAE layers and configurations. SAE-IDF is also interpretable: every dimension of the sparse code has a human-readable meaning, so each match decomposes into named concepts (features). Our work paves the way to a universal symbolic backbone.
Community
TL;DR: An LLM's sparse autoencoder works as a shared coordinate system for matching symbols across heterogeneous systems (formal provers, code, table schemas, knowledge graphs, ontologies), with no training and no aligned pairs.
How SAE-IDF works
- An LLM reads each symbol in its native context (e.g., a KG class with its instances, a Lean 4 statement).
- The SAE encodes the hidden states into a sparse code of named features.
- Each feature is weighted by its rarity (IDF, as in classic text retrieval), and candidates are ranked by cosine.
Results
- Beats the same LLM's dense hidden states on all 12 tasks with Gemma-2-9B, and on all but one with Llama-3.1-8B
- Biggest gains where notations differ most: Lean 4 → Metamath 0.21 → 0.66 MRR, Lean 4 → HOL Light 0.54 → 0.90
- Scales with the dictionary: HOL Light MRR rises from 0.82 to 0.94 as the SAE grows from 16k to 1M features
- Both ingredients matter: without IDF the SAE code mostly trails dense, and a random dictionary with the same IDF stays below dense on every task
Interpretable by construction: the cosine decomposes exactly over features, so every match reads as a few named concepts. For example, in NELL trainstation → DBpedia RailwayStation, 93% of the match comes from two features about railroads and stations.
A first step toward a universal symbolic backbone for neuro-symbolic models at LLM scale.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- TILE: Bridging LLMs and Table Foundation Models for Tabular Learning with Strings (2026)
- SharedSAE: One Feature Dictionary Across Language Models (2026)
- Discovering Cross-Language Reasoning Invariance in LLMs with Geometry-Invariant Sparse Autoencoders (2026)
- Reification as a Transferable Vocabulary: Zero-Shot Link Prediction with Vanilla GNNs (2026)
- Share the Symbol, Not the Vector: Training-Free Fact Communication Across Language Models (2026)
- Sparse Transformer: a Controlled Testbed for Mechanistic Interpretability (2026)
- Emergent Structure in the Marginal Attention Space of Language Models (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2610.19636 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper