google

/

gemma-4-E4B-it

Like 1.79k

Follow

Google 72.1k

We spent the past week running a mechanistic interpretability and long-context multi-hop study on google/gemma-4-E4B-it (and gemma-4-E2B-it) in bfloat16 and wanted to share both the internal telemetry and a drop-in, zero-shot PyTorch inference controller (Richardson Attention) that resolves multi-hop decay without weight modification or fine-tuning.

• GitHub Repository & Papers: https://github.com/tryggth/richardson-attention
• 1-Click Interactive Colab Demo: https://colab.research.google.com/github/tryggth/richardson-attention/blob/main/notebooks/Richardson_Attention_Gemma4_Demo.ipynb

1. The Multi-Hop Paradox in Hybrid SWA / Global Architectures

google/gemma-4-E4B-it has 42 layers (336 heads), of which 35 layers (280 heads) use 512-token Sliding-Window Attention (sliding_attention) and only 7 layers ({5, 11, 17, 23, 29, 35, 41}, totaling 56 heads) use full global attention (full_attention).

When evaluated on single-hop passkey retrieval at 8K–12K tokens, unmodified gemma-4-E4B-it scores 99.996%. However, when the model must compose a 3-hop or 5-hop pointer chain across spatially separated zones (>1,350 tokens apart, exceeding the 512-token SWA window by >2.5x) in a single forward pass under late-context distractor interference:

• 1-Hop (VAULT -> Code): 99.997%
• 2-Hop (Sector -> Vault -> Code): 99.818%
• 3-Hop (Director -> Sector -> Vault -> Code @ 8,270 tokens): Collapses to 0.038% (Clean-6d) and 0.103% (Heavy-16d) (with 99.83% of probability hijacked by late-context audit decoys).
• 5-Hop (Envoy -> Director -> Sector -> Vault -> Subvault -> Code @ 9,912 tokens): Collapses to 2.240% (95.24% decoy capture).

2. Five Mechanistic Findings from Our Telemetry

1. Universal Spectral Mass Constant (α ≈ 0.00165): By computing the expected causal lookback mass $\hat{\alpha}{l,h} = 1/\bar{d}{l,h}$ and the diagonal of the Moore-Penrose pseudoinverse of inter-head Shannon entropy correlations $\text{Agency}k = (\mathbf{R}^+){k,k}$ (Variance Inflation Factor) on a 1.2s calibration pass, we found that primary retrieval heads in both E2B-it and E4B-it cluster at an invariant dimensionless coupling $\kappa = \hat{\alpha} H_{\text{cal}} \approx 2.31$ ($\hat{\alpha} \approx 0.00165\text{ token}^{-1}$).
2. The Write/Read Thermodynamic Asymmetry (W_q = 64): Applying backward distance biasing across all sequence rows during prefill drops 2-Hop accuracy from 99.82% to 0.87% because prefill rows at interior registries get hijacked into attending to earlier text instead of binding their own local sentence into the KV cache. Gating extrapolation strictly to the terminal query window (W_q = 64) preserves 100% of prefill KV-cache binding and immediately restores 2-Hop to 99.57% while jumping 3-Hop to 95.87%–98.59%.
3. Iso-Potential Cubic Plateau + Dual-State Notch Filter: Linear tilts ($\Phi \propto d/T$) cause token-0 zones to cannibalize mid-context hops, while concave log tilts ($\ln(1+d)$) over-amplify mid-context decoys. A bounded cubic plateau $\Phi_l(d) = \min(1, \max(0, d-32)/(c_l T))^3$ places all deep hop zones on a flat $\Phi = 1.000$ roof while cubically attenuating late decoys, and symmetric Boltzmann subtraction $\text{norm}(\max(0, (1+\eta)\mathbf{S}{\text{free}} - \eta\mathbf{S}{\text{heavy}}))$ actively subtracts recency traps.
4. Stepped Wilsonian Renormalization Group (RG) Relay: In Baseline 5-Hop, L5 and L11 are nearly dormant, so the model doesn't read Hop 1 (Zone E) until L29, leaving only L35 and L41 for the remaining 4 hops. Stepping the plateau cutoff $c_l = [0.38, 0.44, 0.52, 0.64, 0.74, 0.85, 0.90]$ across the 7 global layers progressively integrates out earlier hop zones, concentrating 45.33% of L35 attention onto Zone A and driving L35 Logit Lens accuracy from 0.24% to 100.00%.
5. Rotating Residual Fiber Bundle & The L23 Norm Spike: Tracking the 5-layer SWA gaps between global layers reveals that the residual basis rotates by ~$90^\circ$ (cos ≈ 0) across L5->L11, L17->L23, and L23->L29, with L23 firing a massive 292.16 L2-norm update ($5.6\times$ larger than L29). Stepped RG flips L35->L41 vertical cosine alignment from destructive (-0.2271) to constructive (+0.0976).

3. Summary Results on google/gemma-4-E4B-it

All benchmark scripts, raw JSON telemetry, and the 3-line attach_richardson_attention(model, tokenizer) wrapper are in the repo!