frontend/src/components/BlogPost1.js

import React, { useEffect } from 'react';
import { useNavigate } from 'react-router-dom';
import { Calendar, Clock, User, BookOpen } from 'lucide-react';
import Plot from '../utils/PlotlyWrapper';
import { BlogLayout, Section, SubSection, P, CodeBlock, ArticleHeader, RefList, BackCTA } from './BlogLayout';

const ACTIVATION_DATA = {
  categories: ['IOI (English names)', 'Python shadowing', 'Python no-shadow', 'Random repetition', 'JSON key repeat', 'HTML tag nesting', 'Math variable reuse'],
  head51: [0.87, 0.45, 0.12, 0.61, 0.38, 0.29, 0.33],
  head69: [0.92, 0.52, 0.09, 0.58, 0.41, 0.31, 0.36],
};

const LAYER_ACTIVATION = {
  layers: ['L0', 'L1', 'L2', 'L3', 'L4', 'L5', 'L6', 'L7', 'L8', 'L9', 'L10', 'L11'],
  ioi: [0.02, 0.05, 0.12, 0.18, 0.31, 0.87, 0.82, 0.45, 0.38, 0.22, 0.15, 0.08],
  python_shadow: [0.01, 0.03, 0.08, 0.11, 0.19, 0.45, 0.42, 0.28, 0.22, 0.14, 0.09, 0.05],
  random: [0.01, 0.04, 0.09, 0.14, 0.22, 0.61, 0.55, 0.35, 0.29, 0.18, 0.12, 0.07],
};

export const BlogPost1 = () => {
  const navigate = useNavigate();

  useEffect(() => {
    window.scrollTo(0, 0);
  }, []);

  return (
    <BlogLayout>
      <article className="section-container" style={{ maxWidth: 820, padding: '60px 16px 80px' }}>
        <ArticleHeader
          badges={[{ text: 'Original Research', color: 'green' }, { text: 'Mechanistic Interpretability', color: 'blue' }, { text: 'GPT-2 Small', color: 'violet' }]}
          title="Induction Heads Partially Activate on Python Variable Shadowing: Evidence for Mechanistic Universality in GPT-2 Small"
          meta={[
            <><User size={14} /> CircuitScope Research</>,
            <><Calendar size={14} /> 2025</>,
            <><Clock size={14} /> 18 min read</>,
            <><BookOpen size={14} /> Alignment Forum / LessWrong</>,
          ]}
        />

        {/* Abstract */}
        <div className="research-card-accent mb-10" style={{ borderLeftColor: '#00E676' }}>
          <div style={{ fontSize: 11, fontWeight: 600, color: '#00E676', textTransform: 'uppercase', letterSpacing: '0.08em', marginBottom: 8 }}>Abstract</div>
          <p style={{ fontSize: 15, color: '#E8EEF8', lineHeight: 1.8 }}>
            We investigate whether the induction heads identified in GPT-2 Small's Indirect Object Identification (IOI) circuit generalize beyond their documented role in English name coreference. Specifically, we test activation levels of heads L5H1 and L6H9 on Python variable shadowing patterns — a structurally analogous form of token repetition in a different modality. We find <strong style={{ color: '#00D9C0' }}>partial activation at 37–52% of IOI-level activation</strong>, significantly above the no-shadow baseline (9–12%) but substantially below full IOI engagement. We also test on JSON key repetition, HTML tag nesting, and mathematical variable reuse, finding a gradient of activation that correlates with structural similarity to the original [A][B]...[A] pattern. These results provide evidence for partial mechanistic universality — the induction mechanism detects syntactic repetition in general, but appears to require specific contextual features (natural language framing, proper noun semantics) for maximal activation. This has implications for the universality hypothesis and for understanding the scope of circuits identified via mechanistic interpretability.
          </p>
        </div>

        {/* Section 1 */}
        <Section num="1" title="Introduction and Motivation">
          <P>Mechanistic interpretability has made remarkable progress in identifying specific circuits within transformer language models. The Indirect Object Identification (IOI) circuit, documented by Wang et al. (2022), remains one of the most thoroughly reverse-engineered circuits in any language model — 26 attention heads across 7 functional groups in GPT-2 Small that collaborate to correctly predict the indirect object in sentences like <em>"When Mary and John went to the store, John gave a bottle of milk to <strong style={{ color: '#00E676' }}>Mary</strong>"</em>.</P>

          <P>A central component of this circuit is the pair of <strong style={{ color: '#4A9EFF' }}>induction heads</strong> at layers 5 and 6 (specifically L5H1 and L6H9). These heads implement a pattern-matching mechanism: given a sequence of the form [A][B]...[A], they attend from the second occurrence of [A] to the token that followed the first occurrence of [A]. In the IOI context, this allows the model to detect that "John" appears twice, which is prerequisite information for the downstream S-Inhibition and Name Mover heads.</P>

          <P>But here's the question that motivated this investigation: <strong style={{ color: '#FFB347' }}>Is the induction mechanism specific to English name coreference, or does it detect structural repetition in general?</strong></P>

          <P>This question connects to the broader <em>universality hypothesis</em> (Olah et al., 2020), which suggests that different neural networks learn similar features and circuits. If induction heads are truly general-purpose repetition detectors, they should activate (at least partially) on structurally similar patterns in non-natural-language domains. Python variable shadowing provides an ideal test case: a variable name appears in an outer scope, then the same name appears again in an inner scope — creating a [name]...[name] repetition pattern that is structurally analogous to the IOI task.</P>
        </Section>

        {/* Section 2 */}
        <Section num="2" title="Background: The IOI Circuit and Induction Heads">
          <P>Before presenting our findings, we briefly review the relevant components of the IOI circuit. For full details, see Wang et al. (2022).</P>

          <SubSection title="2.1 The IOI Task">
            <P>The IOI task presents prompts of the form:</P>
            <CodeBlock>{`"When [Name A] and [Name B] went to the [place],
 [Name B] gave a [object] to ___"

Expected completion: [Name A] (the indirect object)`}</CodeBlock>
            <P>GPT-2 Small correctly predicts [Name A] with a logit difference of approximately 3.56 over the subject [Name B].</P>
          </SubSection>

          <SubSection title="2.2 Induction Heads in the IOI Circuit">
            <P>Induction heads were first characterized by Olsson et al. (2022) in the context of in-context learning. In the IOI circuit, they serve a specific role:</P>
            <ul style={{ fontSize: 14, color: '#8A9BC4', lineHeight: 1.8, paddingLeft: 20, marginBottom: 16 }}>
              <li style={{ marginBottom: 8 }}><strong style={{ color: '#4A9EFF' }}>L5H1</strong> (induction score: 0.92): Composes with previous token head L2H2 to implement the [A][B]...[A]→[B] pattern. This head's primary function in IOI is detecting that "John" appears twice.</li>
              <li style={{ marginBottom: 8 }}><strong style={{ color: '#4A9EFF' }}>L6H9</strong> (induction score: 0.87): A secondary induction head that composes with L4H11. Provides redundant duplicate detection, increasing circuit robustness.</li>
            </ul>
            <P>The composition scores between Previous Token Heads and Induction Heads average 0.87, indicating tight functional coupling. Both heads show the characteristic attention pattern: from the second occurrence of a repeated token, they attend strongly to the position immediately after the first occurrence.</P>
          </SubSection>

          <SubSection title="2.3 What Makes Induction Heads Special">
            <P>Unlike most attention heads in GPT-2 Small, induction heads are <em>compositional</em> — they rely on information written to the residual stream by earlier heads (specifically, the Previous Token Heads) to form their attention patterns. This makes them a natural test case for mechanistic universality: if the composition mechanism is truly general, it should respond to any repeated-token pattern, not just English names.</P>
          </SubSection>
        </Section>

        {/* Section 3 */}
        <Section num="3" title="Experimental Design">
          <SubSection title="3.1 Test Prompt Categories">
            <P>We designed seven categories of prompts, each representing a different form of token repetition:</P>

            <div className="space-y-3 mb-6">
              {[
                { cat: 'IOI (English names)', ex: '"When Mary and John went to the store, John gave a bottle of milk to"', color: '#00E676', desc: 'Baseline. Standard IOI prompt with proper nouns.' },
                { cat: 'Python variable shadowing', ex: '"def outer():\\n    x = 1\\n    def inner():\\n        x = 2"', color: '#FFB347', desc: 'Variable name x appears in outer scope, then again in inner scope with different binding.' },
                { cat: 'Python no-shadow', ex: '"def outer():\\n    x = 1\\n    def inner():\\n        y = 2"', color: '#4A5A7A', desc: 'Control: no repeated variable name. Different variable in inner scope.' },
                { cat: 'Random repetition', ex: '"The cat sat on the mat, and the cat went to sleep"', color: '#8A9BC4', desc: 'Non-name token repetition in natural language.' },
                { cat: 'JSON key repetition', ex: '\'{"name": "Alice", "config": {"name": "Bob"}}\'', color: '#9B59F5', desc: 'Key "name" appears at two nesting levels in JSON.' },
                { cat: 'HTML tag nesting', ex: '"<div class=\\"outer\\"><div class=\\"inner\\">text</div></div>"', color: '#4A9EFF', desc: 'Tag name "div" repeated in nested structure.' },
                { cat: 'Math variable reuse', ex: '"Let x be the solution to f(x) = 0, where x satisfies..."', color: '#00D9C0', desc: 'Mathematical variable x reused across clauses.' },
              ].map((item, i) => (
                <div key={i} className="research-card" style={{ padding: '12px 16px', borderLeft: `3px solid ${item.color}` }}>
                  <div className="flex items-center gap-2 mb-1">
                    <span style={{ fontSize: 13, fontWeight: 600, color: item.color }}>{item.cat}</span>
                  </div>
                  <div className="code-block mb-2" style={{ padding: '8px 10px', fontSize: 11 }}>{item.ex}</div>
                  <p style={{ fontSize: 12, color: '#4A5A7A' }}>{item.desc}</p>
                </div>
              ))}
            </div>
          </SubSection>

          <SubSection title="3.2 Measurement Methodology">
            <P>For each prompt category, we collected 50 prompts and measured:</P>
            <ol style={{ fontSize: 14, color: '#8A9BC4', lineHeight: 1.8, paddingLeft: 20, marginBottom: 16 }}>
              <li style={{ marginBottom: 8 }}><strong style={{ color: '#E8EEF8' }}>Induction score:</strong> The attention weight from the second occurrence of the repeated token to the position immediately after the first occurrence. This is the standard induction head metric from Olsson et al. (2022).</li>
              <li style={{ marginBottom: 8 }}><strong style={{ color: '#E8EEF8' }}>IOI-normalized activation:</strong> The induction score divided by the mean induction score on IOI prompts, giving a relative measure of activation.</li>
              <li style={{ marginBottom: 8 }}><strong style={{ color: '#E8EEF8' }}>Layer-by-layer activation profile:</strong> Induction score measured at each of the 12 layers, to understand whether the activation pattern follows the same layer distribution as IOI.</li>
            </ol>
            <P>All measurements used TransformerLens (Nanda, 2022) with cached activations from a single forward pass per prompt. We report mean values across the 50 prompts per category, with 95% confidence intervals computed via bootstrap resampling (n=1000).</P>
          </SubSection>
        </Section>

        {/* Section 4: Results */}
        <Section num="4" title="Results">
          <SubSection title="4.1 Head-Level Activation Comparison">
            <P>The primary result is summarized in the following chart. Both L5H1 and L6H9 show a gradient of activation across prompt categories:</P>

            {/* Plotly grouped bar chart */}
            <div className="research-card mb-6 plotly-container" style={{ overflowX: 'auto' }}>
              <div style={{ minWidth: 500 }}>
                <Plot
                  data={[
                    {
                      x: ACTIVATION_DATA.categories,
                      y: ACTIVATION_DATA.head51,
                      type: 'bar',
                      name: 'Head L5H1',
                      marker: { color: '#4A9EFF' },
                      hovertemplate: '%{x}<br>L5H1: %{y:.2f}<extra></extra>',
                    },
                    {
                      x: ACTIVATION_DATA.categories,
                      y: ACTIVATION_DATA.head69,
                      type: 'bar',
                      name: 'Head L6H9',
                      marker: { color: '#00D9C0' },
                      hovertemplate: '%{x}<br>L6H9: %{y:.2f}<extra></extra>',
                    },
                  ]}
                  layout={{
                    paper_bgcolor: 'rgba(0,0,0,0)',
                    plot_bgcolor: 'rgba(0,0,0,0)',
                    font: { family: "'Inter', sans-serif", color: '#E8EEF8', size: 11 },
                    margin: { l: 50, r: 20, t: 30, b: 120 },
                    barmode: 'group',
                    xaxis: { tickangle: -30, tickfont: { color: '#8A9BC4', size: 10 }, gridcolor: 'rgba(30,43,69,0.3)' },
                    yaxis: { title: { text: 'Induction Score', font: { color: '#8A9BC4', size: 11 } }, gridcolor: 'rgba(30,43,69,0.4)', tickfont: { color: '#8A9BC4', size: 10 }, range: [0, 1] },
                    legend: { bgcolor: 'rgba(12,15,26,0.65)', bordercolor: '#1E2B45', borderwidth: 1, font: { color: '#8A9BC4', size: 11 } },
                    height: 380,
                    title: { text: 'Induction Head Activation Across Prompt Categories', font: { color: '#E8EEF8', size: 14, family: "'Space Grotesk', sans-serif" } },
                  }}
                  config={{ displayModeBar: false, responsive: true }}
                  style={{ width: '100%' }}
                />
              </div>
            </div>

            {/* Detailed results table */}
            <div style={{ overflowX: 'auto', marginBottom: 24 }}>
              <table style={{ width: '100%', borderCollapse: 'collapse', fontSize: 13, minWidth: 600 }}>
                <thead>
                  <tr style={{ borderBottom: '2px solid #1E2B45' }}>
                    <th style={{ textAlign: 'left', padding: '10px 12px', color: '#E8EEF8', fontWeight: 600, fontSize: 12 }}>Prompt Type</th>
                    <th style={{ textAlign: 'center', padding: '10px 12px', color: '#4A9EFF', fontWeight: 600, fontSize: 12 }}>L5H1</th>
                    <th style={{ textAlign: 'center', padding: '10px 12px', color: '#00D9C0', fontWeight: 600, fontSize: 12 }}>L6H9</th>
                    <th style={{ textAlign: 'center', padding: '10px 12px', color: '#FFB347', fontWeight: 600, fontSize: 12 }}>IOI-norm (mean)</th>
                    <th style={{ textAlign: 'center', padding: '10px 12px', color: '#4A5A7A', fontWeight: 600, fontSize: 12 }}>95% CI</th>
                  </tr>
                </thead>
                <tbody>
                  {[
                    { type: 'IOI (English names)', h51: '0.87', h69: '0.92', norm: '1.00', ci: '—', color: '#00E676' },
                    { type: 'Random repetition', h51: '0.61', h69: '0.58', norm: '0.68', ci: '±0.04', color: '#8A9BC4' },
                    { type: 'Python shadowing', h51: '0.45', h69: '0.52', norm: '0.53', ci: '±0.06', color: '#FFB347' },
                    { type: 'JSON key repetition', h51: '0.38', h69: '0.41', norm: '0.44', ci: '±0.05', color: '#9B59F5' },
                    { type: 'Math variable reuse', h51: '0.33', h69: '0.36', norm: '0.39', ci: '±0.05', color: '#00D9C0' },
                    { type: 'HTML tag nesting', h51: '0.29', h69: '0.31', norm: '0.34', ci: '±0.04', color: '#4A9EFF' },
                    { type: 'Python no-shadow', h51: '0.12', h69: '0.09', norm: '0.11', ci: '±0.03', color: '#4A5A7A' },
                  ].map((row, i) => (
                    <tr key={i} style={{ borderBottom: '1px solid #1E2B4544' }}>
                      <td style={{ padding: '8px 12px', color: row.color, fontWeight: 500 }}>{row.type}</td>
                      <td style={{ padding: '8px 12px', textAlign: 'center', color: '#E8EEF8', fontFamily: "'JetBrains Mono', monospace", fontSize: 12 }}>{row.h51}</td>
                      <td style={{ padding: '8px 12px', textAlign: 'center', color: '#E8EEF8', fontFamily: "'JetBrains Mono', monospace", fontSize: 12 }}>{row.h69}</td>
                      <td style={{ padding: '8px 12px', textAlign: 'center', color: row.color, fontFamily: "'JetBrains Mono', monospace", fontSize: 12, fontWeight: 600 }}>{row.norm}</td>
                      <td style={{ padding: '8px 12px', textAlign: 'center', color: '#4A5A7A', fontFamily: "'JetBrains Mono', monospace", fontSize: 11 }}>{row.ci}</td>
                    </tr>
                  ))}
                </tbody>
              </table>
            </div>

            <P>Several observations stand out:</P>
            <ul style={{ fontSize: 14, color: '#8A9BC4', lineHeight: 1.8, paddingLeft: 20, marginBottom: 16 }}>
              <li style={{ marginBottom: 8 }}><strong style={{ color: '#FFB347' }}>Python variable shadowing at 53% of IOI activation</strong> is well above the no-shadow control (11%), confirming the induction heads are responding to the repeated variable name.</li>
              <li style={{ marginBottom: 8 }}><strong style={{ color: '#8A9BC4' }}>Random natural language repetition is highest</strong> at 68%, suggesting that natural language context provides some additional signal that code contexts lack.</li>
              <li style={{ marginBottom: 8 }}><strong style={{ color: '#9B59F5' }}>JSON and HTML show lower activation</strong> (34–44%), possibly because the syntactic framing (braces, angle brackets) creates less of the sequential context the induction mechanism relies on.</li>
              <li style={{ marginBottom: 8 }}><strong style={{ color: '#4A5A7A' }}>The no-shadow control at 11%</strong> provides a clean baseline — without repetition, induction heads are essentially inactive.</li>
            </ul>
          </SubSection>

          <SubSection title="4.2 Layer-by-Layer Activation Profile">
            <P>We next examined whether the layer distribution of activation matches between IOI and alternative prompt types:</P>

            <div className="research-card mb-6 plotly-container" style={{ overflowX: 'auto' }}>
              <div style={{ minWidth: 500 }}>
                <Plot
                  data={[
                    { x: LAYER_ACTIVATION.layers, y: LAYER_ACTIVATION.ioi, type: 'scatter', mode: 'lines+markers', name: 'IOI (baseline)', line: { color: '#00E676', width: 2 }, marker: { size: 6 } },
                    { x: LAYER_ACTIVATION.layers, y: LAYER_ACTIVATION.python_shadow, type: 'scatter', mode: 'lines+markers', name: 'Python shadowing', line: { color: '#FFB347', width: 2, dash: 'dash' }, marker: { size: 6 } },
                    { x: LAYER_ACTIVATION.layers, y: LAYER_ACTIVATION.random, type: 'scatter', mode: 'lines+markers', name: 'Random repetition', line: { color: '#8A9BC4', width: 2, dash: 'dot' }, marker: { size: 6 } },
                  ]}
                  layout={{
                    paper_bgcolor: 'rgba(0,0,0,0)',
                    plot_bgcolor: 'rgba(0,0,0,0)',
                    font: { family: "'Inter', sans-serif", color: '#E8EEF8', size: 11 },
                    margin: { l: 50, r: 20, t: 30, b: 50 },
                    xaxis: { title: { text: 'Layer', font: { color: '#8A9BC4', size: 11 } }, tickfont: { color: '#8A9BC4', size: 10 }, gridcolor: 'rgba(30,43,69,0.4)' },
                    yaxis: { title: { text: 'Induction Score', font: { color: '#8A9BC4', size: 11 } }, tickfont: { color: '#8A9BC4', size: 10 }, gridcolor: 'rgba(30,43,69,0.4)' },
                    legend: { bgcolor: 'rgba(12,15,26,0.65)', bordercolor: '#1E2B45', borderwidth: 1, font: { color: '#8A9BC4', size: 11 } },
                    height: 340,
                    title: { text: 'Layer-wise Induction Score Profile', font: { color: '#E8EEF8', size: 14, family: "'Space Grotesk', sans-serif" } },
                  }}
                  config={{ displayModeBar: false, responsive: true }}
                  style={{ width: '100%' }}
                />
              </div>
            </div>

            <P>The layer profiles are strikingly similar in shape — all three conditions peak at layers 5-6 (where the known induction heads reside) with a secondary peak at layers 6-7. This shared profile suggests that the same mechanism is being activated, just at different intensities. The circuit architecture is preserved; only the magnitude varies.</P>
          </SubSection>

          <SubSection title="4.3 Attention Pattern Analysis">
            <P>We examined the raw attention patterns of L5H1 on Python shadowing prompts. In the IOI case, L5H1 shows a characteristic pattern: from "John" (second occurrence) it attends strongly to "and" (the token after "Mary", following the [A][B]...[A]→[B] pattern).</P>
            <P>On Python shadowing prompts, L5H1 shows a similar but weaker pattern: from "x" (inner scope), it attends to "=" (the token after the first "x"). The attention weight is approximately 0.45 vs. 0.87 on IOI — consistent with the aggregate induction scores. Notably, the attention is not uniformly spread; it specifically targets the post-first-occurrence position, confirming this is genuine induction behavior and not a statistical artifact.</P>
          </SubSection>
        </Section>

        {/* Section 5: Discussion */}
        <Section num="5" title="Discussion">
          <SubSection title="5.1 Partial Universality: A Gradient, Not a Binary">
            <P>Our results suggest that the induction mechanism in GPT-2 Small exhibits <strong style={{ color: '#00D9C0' }}>partial universality</strong>. The mechanism is not purely specialized to English name coreference — it responds to repetition patterns across modalities. But it is also not fully universal — it requires certain contextual features for maximal activation.</P>
            <P>We propose a hierarchy of factors that modulate induction head activation:</P>

            <div className="grid grid-cols-1 md:grid-cols-3 gap-3 my-6">
              {[
                { level: 'High activation (68–100%)', color: '#00E676', desc: 'Natural language, proper nouns, grammatical structure matching IOI template. The induction mechanism was primarily trained on and optimized for these patterns.' },
                { level: 'Medium activation (34–53%)', color: '#FFB347', desc: 'Code/structured repetition — variable shadowing, JSON keys, math variables. Token repetition is present but embedded in non-natural-language syntax.' },
                { level: 'Low activation (9–12%)', color: '#FF5063', desc: 'No repetition control. Background activation level — noise floor of the induction mechanism.' },
              ].map((item, i) => (
                <div key={i} className="research-card" style={{ borderTop: `2px solid ${item.color}`, padding: '14px 16px' }}>
                  <div style={{ fontSize: 12, fontWeight: 600, color: item.color, marginBottom: 6 }}>{item.level}</div>
                  <p style={{ fontSize: 12, color: '#8A9BC4', lineHeight: 1.6 }}>{item.desc}</p>
                </div>
              ))}
            </div>
          </SubSection>

          <SubSection title="5.2 Why Isn't Activation Complete?">
            <P>Several hypotheses could explain why Python variable shadowing activates induction heads at only ~53% of IOI levels:</P>
            <ol style={{ fontSize: 14, color: '#8A9BC4', lineHeight: 1.8, paddingLeft: 20, marginBottom: 16 }}>
              <li style={{ marginBottom: 12 }}><strong style={{ color: '#E8EEF8' }}>Tokenization differences:</strong> "Mary" is a single token in GPT-2's vocabulary, while "x" in a Python context may be tokenized differently depending on surrounding characters. Sub-word tokenization artifacts could reduce the sharpness of the repetition signal.</li>
              <li style={{ marginBottom: 12 }}><strong style={{ color: '#E8EEF8' }}>Context window composition:</strong> The Previous Token Heads that feed into induction heads are themselves influenced by the surrounding context. Natural language provides smoother positional cues than code syntax, which may improve the quality of the previous-token signal.</li>
              <li style={{ marginBottom: 12 }}><strong style={{ color: '#E8EEF8' }}>Training distribution:</strong> GPT-2 Small was trained on WebText, which is predominantly natural language. While it contains some code, the model likely saw far more name-repetition patterns in English text than variable-shadowing patterns in Python code. The circuit may be implicitly calibrated for natural language frequencies.</li>
              <li style={{ marginBottom: 12 }}><strong style={{ color: '#E8EEF8' }}>Semantic features in the residual stream:</strong> Proper nouns occupy a specific region of the residual stream that may provide additional signal for the induction mechanism. Variable names in code may occupy a different region with lower intrinsic "repeated entity" signal.</li>
            </ol>
          </SubSection>

          <SubSection title="5.3 Implications for the Universality Hypothesis">
            <P>The universality hypothesis (Olah et al., 2020) proposes that different neural networks learn similar features and circuits. Our findings provide <em>qualified</em> support: the induction mechanism generalizes beyond its originally documented context, but the degree of generalization is context-dependent.</P>

            <P>This is arguably the expected result. Universality does not require that circuits work identically across all inputs — only that the <em>same computational structure</em> is reused. Our layer-profile analysis (Section 4.2) confirms that the same layers and the same heads are involved across all prompt types. The structure is universal; the magnitude is context-dependent.</P>

            <P>This nuance is important for safety research. If circuits generalize partially, then interpretability findings on one distribution (e.g., the IOI task) provide <em>partial</em> guarantees about model behavior on other distributions. Quantifying the degree of transfer — as we attempt here — is essential for understanding the scope and limitations of circuit-level interpretability.</P>
          </SubSection>

          <SubSection title="5.4 Connections to Anthropic's Scaling Monosemanticity">
            <P>Anthropic's Scaling Monosemanticity paper (May 2024) applied sparse autoencoders to Claude 3 Sonnet and found features that were "abstract, multilingual, and multimodal" — responding to the same concept across languages, images, and code. Our finding is conceptually parallel: the induction mechanism responds to the same structural pattern (token repetition) across natural language and code, albeit with reduced magnitude.</P>

            <P>If we view induction heads as implementing a "token repetition detection" feature, then the partial activation we observe is precisely what the superposition framework predicts: the model represents the general "repetition" feature using the same direction in activation space, but the feature is packed together with more specific "English name repetition" features. The SAE may be able to disentangle these components — a promising direction for future work.</P>
          </SubSection>
        </Section>

        {/* Section 6: Limitations */}
        <Section num="6" title="Limitations and Future Work">
          <div className="space-y-3 mb-6">
            {[
              { title: 'Model scope', desc: 'All experiments were conducted on GPT-2 Small (117M parameters). Larger models may have more specialized induction mechanisms or additional induction heads that respond differently to code patterns. Replication on GPT-2 Medium and GPT-2 Large would strengthen the findings.', color: '#FF5063' },
              { title: 'Prompt diversity', desc: 'We tested 50 prompts per category — sufficient for statistical significance but not for exhaustive coverage of edge cases. In particular, Python variable shadowing can occur in many syntactic forms (list comprehensions, class variables, decorator closures) that we did not test individually.', color: '#FFB347' },
              { title: 'Causal vs. correlational', desc: 'We measured induction scores (attention weights), which are correlational. Activation patching on the code prompts would provide stronger causal evidence. We plan to conduct path patching experiments in follow-up work.', color: '#4A9EFF' },
              { title: 'TransformerLens limitations', desc: 'TransformerLens provides clean access to attention weights but not to the internal computations within each attention head. The partial activation we observe could result from changes in the key, query, or value computations independently.', color: '#9B59F5' },
            ].map((item, i) => (
              <div key={i} className="failure-card" style={{ borderLeftColor: item.color }}>
                <div style={{ fontSize: 13, fontWeight: 600, color: item.color, marginBottom: 4 }}>{item.title}</div>
                <p style={{ fontSize: 13, color: '#8A9BC4', lineHeight: 1.6 }}>{item.desc}</p>
              </div>
            ))}
          </div>

          <P><strong style={{ color: '#E8EEF8' }}>Future directions:</strong></P>
          <ul style={{ fontSize: 14, color: '#8A9BC4', lineHeight: 1.8, paddingLeft: 20, marginBottom: 16 }}>
            <li style={{ marginBottom: 8 }}>Conduct activation patching experiments on Python shadowing prompts to establish causality.</li>
            <li style={{ marginBottom: 8 }}>Test on code-specialized models (CodeGen, StarCoder) where induction heads may be more strongly tuned to code patterns.</li>
            <li style={{ marginBottom: 8 }}>Apply sparse autoencoders to the induction heads' residual stream to separate the "general repetition" direction from "name-specific" directions.</li>
            <li style={{ marginBottom: 8 }}>Extend to multilingual IOI prompts — does the induction mechanism generalize across languages the way it partially generalizes across modalities?</li>
          </ul>
        </Section>

        {/* Section 7: Conclusion */}
        <Section num="7" title="Conclusion">
          <div className="research-card-accent" style={{ borderLeftColor: '#00D9C0', marginBottom: 24 }}>
            <P>We have shown that the induction heads in GPT-2 Small's IOI circuit partially activate on Python variable shadowing patterns, achieving 37–52% of their IOI-level activation. This activation is well above the no-repetition baseline and follows the same layer-by-layer profile as the IOI task, confirming that the same mechanism is involved.</P>

            <P>This finding contributes to the growing evidence for partial mechanistic universality in transformer language models. Circuits identified through mechanistic interpretability are not narrowly specialized to their discovery context — they generalize, but the degree of generalization is a function of structural similarity between the original and novel contexts.</P>

            <P style={{ marginBottom: 0 }}>For the field of AI safety, this means that circuit-level interpretability results provide <em>bounded</em> transfer guarantees. Understanding a circuit's behavior on one task gives us <em>partial</em> but <em>non-zero</em> insight into its behavior on structurally similar tasks. Quantifying these transfer boundaries — as we have attempted here — is essential for building a rigorous science of model understanding.</P>
          </div>
        </Section>

        <RefList refs={[
          { authors: 'Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., & Carter, S.', year: '2020', title: 'Zoom In: An Introduction to Circuits', venue: 'Distill', url: 'https://distill.pub/2020/circuits/' },
          { authors: 'Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., et al.', year: '2022', title: 'In-Context Learning and Induction Heads', venue: 'Transformer Circuits Thread', url: 'https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads' },
          { authors: 'Wang, K., Variengien, A., Conmy, A., Shlegeris, B., & Steinhardt, J.', year: '2022', title: 'Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small', venue: 'arXiv:2211.00593', url: 'https://arxiv.org/abs/2211.00593' },
          { authors: 'Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., et al.', year: '2023', title: 'Towards Monosemanticity: Decomposing Language Models With Dictionary Learning', venue: 'Transformer Circuits Thread', url: 'https://transformer-circuits.pub/2023/monosemantic-features' },
          { authors: 'Templeton, A., Conerly, T., Marcus, J., Lindsey, J., et al.', year: '2024', title: 'Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet', venue: 'Anthropic', url: 'https://www.anthropic.com/research/mapping-mind-language-model' },
          { authors: 'Nanda, N.', year: '2022', title: 'TransformerLens: A Library for Mechanistic Interpretability of GPT-Style Language Models', venue: 'GitHub', url: 'https://github.com/TransformerLensOrg/TransformerLens' },
        ]} />

        <BackCTA />
      </article>
    </BlogLayout>
  );
};