Mechanistic interpretability papers typically showcase circuits (like the Indirect Object Identification circuit in GPT-2 Small) using single, highly structured prompt templates. But how robust are these circuits to natural prompt variations? We perform a systematic 50-prompt causal tracing sweep across four distinct control groups: Standard, Multitoken Names, Long Range Positional, and Pronoun Reference. We discover a profound mechanistic shift: multi-token name splits disrupt the induction pathway (Layer 5) by over 60%, forcing the model to compensate via semantic S-Inhibition (Layer 7) pathways. Conversely, introducing pronoun clues early in the prompt primes coreference mechanisms, stabilizing the causal tracing map and producing the highest logit difference recovery (~3.48) of all experimental groups.
The classic Indirect Object Identification (IOI) circuit, first mapped by Wang et al. (2022), is often taught as a rigid deterministic computer program inside GPT-2 Small. Teachers draw neat boxes around Layer 5/6 Induction Heads, Layer 7/8 S-Inhibition Heads, and Layer 9/10 Name Mover Heads. This gives the comforting impression that the transformer uses a single, immutable algorithm to solve the task.
However, any production-level deep dive reveals that transformers are highly dynamic, noisy networks. A circuit is not a physical copper trace; it is a statistical average of activation flows across thousands of training distributions. When we alter the phonetic length of names, distance between words, or grammatical markers, the active sub-circuits adapt dynamically.
To prove this, we automated a 50-prompt sweep on a CPU-based HookedTransformer. We ablated and patched individual residual stream representations at Layer 5 (the core Induction layer) and Layer 7 (the core S-Inhibition layer) to measure their causal contribution to the final logit difference. The results reveal that circuit structure is highly sensitive to prompt morphology.
We tracked the overall clean logit difference and the recovery percentages under residual stream patching across our four experimental categories. Below is the aggregated data from our run:
{/* Stats Table */}| Experimental Group | Mean Logit Diff | L5 Induction Rec % | L7 S-Inhibition Rec % |
|---|---|---|---|
| {g} | {cfg.mean_logit_diff} ± {cfg.std_logit_diff} | {(cfg.mean_layer5_recovery * 100).toFixed(1)}% | {(cfg.mean_layer7_recovery * 100).toFixed(1)}% |
The data immediately reveals a striking contrast: while Standard IOI prompts rely on both Induction (46.0%) and S-Inhibition (76.0%) channels, Multitoken Names prompts experience a complete collapse in Layer 5 Induction recovery (down to 18.0%), yet they still maintain a strong 54.0% causal recovery in Layer 7.
This grouped comparison demonstrates the shifting mechanistic burden across categories. The Plotly chart below outlines the mean causal recovery percentages for the repetition-based Induction channel vs. the semantic S-Inhibition channel:
Why does Layer 5 Induction recovery plummet from 46% down to 18% when we replace names like 'Mary' and 'John' with 'Gaurav' and 'Christopher'?
The answer lies in the tokenizer. Modern LLMs do not read letters; they process discrete tokens. The GPT-2 tokenizer splits short names into single tokens (e.g. Mary is token ID [10486]). But a multi-syllabic name like Gaurav is split into three separate tokens: ["ĠG", "aur", "av"] (token IDs [38, 2527, 281]).
Classic Induction Heads are specialized search engines. They look for the active token X, search backward for its previous occurrence, and copy whatever token came immediately after it. However, this mechanism is highly sensitive to token alignment. When a name is split into three tokens, the induction head must copy the multi-token block in sequence. This dilutes the signal, leading to a 60.8% reduction in causal effectiveness.
Interestingly, the transformer does not fail. It maintains a healthy S-Inhibition channel (54%). The model shifts its reliance to higher semantic representations in Layers 7 and 8, which recognize the abstract concept of "the person Gaurav" rather than the literal spelling token, showing the model's structural adaptability.
To inspect this phonetic breakdown, we mapped Name Tokenizer Length directly against the resulting clean Logit Difference. A clear correlation emerges: as names are split into more sub-tokens, the model's absolute naming confidence (logit difference) decays systematically.
This token penalty represents a critical vulnerability in real-world deployments. If an AI is evaluated on English names, its reasoning circuit operates with high confidence (~3.15). If it is evaluated on non-Western multi-token names, its internal copying circuits degrade, dropping logit difference to ~2.12. This is a clear mechanistic explanation for cultural bias in token-repetition circuits.
In contrast to the token length penalty, introducing pronouns (e.g. "...John thought about his day...") yields the strongest logit difference (3.48) and the highest S-Inhibition recovery (81.0%). Why?
In standard English, the possessive pronoun his must link back to a male subject in the sentence context. The moment the model processes his, early attention layers activate coreference representations, identifying John as the referent. This pre-activates the Subject inhibition pathway, making the model's suppression of S at the final token exceptionally clean.
This highlights that semantic context does not merely "decorate" a sentence; it structurally primes the model's circuit layout, making the neural computations far more efficient and robust.
Our 50-prompt causal tracing sweep demonstrates that circuit architectures inside LLMs are dynamic and highly responsive to sentence morphology:
- Multi-token splits break sequence copying circuits, forcing the model to rely heavier on abstract semantic pathways.
- Grammatical pronoun clues prime coreference hooks, dramatically boosting logit recovery.
- Long-range relative clauses dilute attention scores, demonstrating positional decay.
For SDE and AI/ML engineers, these findings prove that we cannot evaluate systems on ideal benchmarks alone. Truly robust mechanistic interpretability requires testing prompt boundary conditions — because minor phonetic shifts completely rearrange the internal computational pathways.