notes/synth_alibi_offset_ablation.md

Synth ALiBi × {offset, placement} ablation — explains 75M transfer failure

Setup

E1-Bk0 (softmax + ±1 + K-shift), d=128/4H/4L, n_symbols=24, 3000 steps, batch=64, seed=42. CPU run (no GPU available locally; VM down).

Offset axis (placement = L0)

ALiBi	offset	copy_acc	Global IPMR	Top-per-head
ON	1	1.6%	0.00%	(collapsed)
ON	2	1.7%	0.00%	(collapsed)
OFF	1	88-92%	95.86%	L3H3 69%, L0 ~46%
OFF	2	97-98%	99.86%	L2H2 97%, L3H2 93%

Placement axis (offset = 1)

ALiBi	K-shift layer	copy_acc	Global IPMR	Top-per-head
ON	L0	1.6%	0.00%	(collapsed)
ON	L3 (last)	0.9-1.6%	0.00%	(collapsed)
OFF	L0	88-92%	95.86%	L3H3 69%
OFF	L3 (last)	90%	98.98%	L1H3 82%, L1H0 81%

Combined finding

ALiBi flattens BOTH offset and placement axes — at synth scale, every ALiBi cell collapses to 0% IPMR regardless of K-shift configuration. The "late placement is better" and "offset>1 is better" findings exist ONLY in --no-alibi runs.

Key findings

ALiBi flattens the offset axis at synth scale

With ALiBi enabled, BOTH offsets collapse to ~2% copy_acc, 0% IPMR. The offset advantage seen in prior synth experiments was an artifact of running with --no-alibi. ALiBi blocks long-distance attention regardless of K-shift offset.

Without ALiBi, offset=2 still wins (replicates prior)

offset=2 → 99.86% IPMR with L2H2 at 97% top-per-head. offset=1 → 95.86% IPMR with L3H3 at only 69%. Mechanism: offset=1's K-vector smearing (K[t] = W_K(x[t] + x[t-1])) creates near-identical K vectors at adjacent positions, which sign-STE can't disambiguate. offset=2 spaces them out.

Why this explains the 75M transfer failure

Regime	Distance pattern	offset=2 advantage	Empirical
Synth + no-ALiBi (fixed dist 24)	long fixed	LARGE (+30pp top-head)	known
Synth + ALiBi (no induction at all)	suppressed	none (both 0%)	this expt
75M + ALiBi (natural variable dist 1-15)	short variable	none/reversed	known

The 75M empirical result (offset=1 beats offset=2 by 0.014 BPC at 5K steps) makes sense: at 75M with ALiBi, induction work happens at short distances where offset=1 (= prev-token, Olsson canonical) is genuinely correct. offset=2's advantage was specific to the no-ALiBi long-fixed-distance synth task — it doesn't transfer to natural text + ALiBi.

Full 9-seed × 4-placement sweep — LATE PLACEMENT WINS MONOTONICALLY

Sweep (seeds 0, 1, 2, 7, 13, 21, 42, 99, 100; placements L0-L3; 4-layer softmax+K-shift+ALiBi, n_sym=8, offset=1, 3000 steps).

Per-seed results

seed	L0 IPMR	L1 IPMR	L2 IPMR	L3 IPMR
0	0.7%	0.2%	100.0%	92.2%
1	13.6%	43.3%	8.0%	2.5%
2	24.3%	19.9%	80.6%	100.0%
7	0.0%	0.0%	0.2%	100.0%
13	3.8%	2.5%	100.0%	99.5%
21	11.4%	0.0%	0.5%	100.0%
42	0.0%	96.2%	0.2%	0.9%
99	0.0%	74.1%	0.7%	100.0%
100	0.2%	99.8%	9.6%	4.5%

Aggregate

L	mean copy	mean IPMR	seeds with >50% IPMR
L0 (first)	42.2%	6.0%	0/9
L1	64.2%	37.3%	3/9
L2	55.6%	33.3%	3/9
L3 (last)	80.7%	66.6%	6/9

Conclusions (4L)

Monotonic improvement from L0 to L3. L3 (last layer) reaches strong induction on 6/9 seeds; L0 reaches it on 0/9. Mid-layers L1/L2 are intermediate and lottery-distributed.

8-LAYER VERIFICATION — L0 is uniquely bad, ALL others work

3 seeds × {L0, L4, L7}:

seed	L0 IPMR	L4 IPMR	L7 IPMR
42	21.65%	100.00%	99.78%
7	1.79%	98.66%	99.33%
100	14.29%	100.00%	99.11%

Placement	Mean IPMR	Seeds with >50% IPMR
L0	12.6%	0/3
L4 (middle)	99.6%	3/3 deterministic
L7 (last)	99.4%	3/3 deterministic

Full 8L sweep at seed=42 — locates the cliff

Layer	IPMR
L0	21.65%
L1	99.78% ← cliff
L2	87.05%
L3	100.00%
L4	100.00%
L5	100.00%
L6	95.98%
L7	99.78%

L0 is uniquely bad. Every other placement (L1-L7) works at seed=42. The cliff is sharp: between L0 (22% IPMR) and L1 (99.8% IPMR).

L1 reliability check (4 more seeds at 8L L=1)

seed	8L L0 IPMR	8L L1 IPMR
0	(3.8% at 4L L0)	100.0%
1	(13.6% at 4L L0)	61.8%
7	1.8%	95.3%
42	21.7%	99.8%
100	14.3%	17.4% ← FAILS

L1 works on 4/5 seeds at 8L (>50% IPMR), but seed=100 hits a near-L0 trough (17.4% — barely better than its L0 14.3%). So L1 is mostly reliable but still has lottery-fail tail. L4 and L7 work on 3/3 seeds tested — strictly more reliable than L1.

Updated picture at 8L

Placement	Seeds with >50% IPMR	Reliability
L0	0/3	always fails
L1	4/5	mostly reliable, occasional fail
L4 (middle)	3/3	deterministic
L7 (last)	3/3	deterministic

The cliff between L0 and L1 holds (L1 is much better than L0 on average), but L4-L7 are strictly more reliable than L1. For 75M, the simplest "1-line change" L0 → L1 might still hit a bad seed; safer to go L0 → L5+.

This matches the original synth-no-ALiBi finding ("K-shift on any non-L0 layer reaches ~98-99% copy_acc"). The pattern is robust across:

• ALiBi vs no-ALiBi
• Long (n=24) vs short (n=8) induction distance
• 4 layers vs 8 layers (4L lottery vanishes at 8L)

Conclusion: K-shift L0 is uniquely suboptimal under sign-STE. ANY other placement enables sign-STE attention to form induction.

For 75M (12 layers): K-shift L1 (or any of L1-L11) should reliably beat L0. Single-seed 75M comparison should show clear ordering.

Placement axis prediction — SHARPENED at synth+ALiBi+short-distance (UNRELIABLE)

The pathological synth+ALiBi+n=24 result (everything = 0% IPMR) doesn't predict 75M behavior, where induction lives at distances 1-15 (ALiBi- permitted range).

Re-ran synth at n_symbols=8 (mimicking 75M's distance regime) with ALiBi enabled, sweeping placement and offset:

Placement sweep (offset=1, n=8, ALiBi ON, seed=42)

K-shift layer	copy_acc	Global IPMR
L0 (current v76)	9.2%	0.00%
L1	98.0%	96.21%
L2	15.8%	0.22%
L3 (last)	32.6%	0.89%

Offset sweep (placement=L0, n=8, ALiBi ON, seed=42)

offset	copy_acc	IPMR
1	9.2%	0.00%
2	37.9%	0.67%
3	32.4%	0.00%
5	17.6%	4.02%

Key insight

At synth+ALiBi+short-distance, K-shift L1 is dramatically better than L0 (96% vs 0% IPMR; 98% vs 9% copy_acc). L0 is uniquely bad. L2 and L3 are mediocre. The story is "L1 specifically wins", not "later layers win".

This is a sharper prediction than the prior outline's "L_(n-1) wins" claim (which came from synth+no-ALiBi+long-distance). For 75M (n_layers=12), the analog is K-shift L1 (second layer), NOT L11 (last layer).

Mechanism hypothesis

L0 receives raw token embeddings — limited positional structure. K-shift on L1 sees post-L0-attention features, which are content-mixed and provide richer K-vector geometry for sign-STE to discriminate.

L_late layers don't help because by then the residual stream has lost local positional structure; K-shift offset=1 there shifts a high-level content vector rather than a position-marked one.

Bottom line (post 9-seed sweep)

Synth+ALiBi+short-distance shows a real but lottery-distributed L1 advantage:

• L1 IPMR > L0 IPMR on 4/9 seeds, mean Δ = +31pp (p≈0.04 one-sided)
• L0 NEVER reaches strong induction (>50% IPMR), L1 reaches it 3/9 times
• High variance: stdev of Δ_IPMR = 45.9 across seeds

This is consistent with "L1 unlocks a more capable regime that L0 cannot enter, but only ~33% of seed lotteries hit the jackpot at this scale".

Implication for 75M

If the monotonic pattern transfers, K-shift L11 (last) at 75M should robustly outperform L0:

• Synth: L3 reaches strong induction on 6/9 seeds; L0 on 0/9
• Mid-layers (L1, L2 analog of 75M's L1-L10) lottery-distributed
• L_(n-1) is the most reliable placement

To verify at 75M (currently VM-blocked):

• Single-seed K-shift L11 vs L0 head-to-head, 5K-10K steps
• Even single seeds may show clear ordering since L11 hits strong induction more reliably than L0 (66% vs 0% in synth)

The current v76 recipe (RWSP + K-shift L0 + offset=1) may be leaving val_bpc on the table by ~0.05-0.20 BPC at 75M — testable when VM returns.