| ================================================================================ |
| SMAT (Semantic Attention) - Findings Summary for Paper |
| ================================================================================ |
| Compiled: 2026-06-03 |
| Source files used (all in this repo, all re-verified for this document): |
| runs/{baseline,s_only,g_only,full}_s{0..4}/metrics.jsonl (20 training runs) |
| runs_4ksteps/{baseline,s_only,g_only,full}_s0/metrics.jsonl (4 partial runs) |
| figures_final/ablation_inference.json (surgical ablations) |
| figures_final/ablation_results.csv (final eval table) |
|
|
| Every number in sections 2-6 below was recomputed directly from the raw |
| metrics.jsonl files (not copied from any pre-existing CSV or README). The |
| per-seed deltas, means, std devs, and lambda/G statistics were all |
| re-derived from scratch and cross-checked twice. |
|
|
| The cross-scale results in section 7 are from prior experiments outside |
| this repo; they are cited as such and were not re-verified from raw data |
| here. Treat them as historical context, not as primary evidence. |
|
|
|
|
| ================================================================================ |
| 1. WHAT SMAT IS (one paragraph) |
| ================================================================================ |
| SMAT modifies standard transformer attention with two additions: (1) a |
| learnable semantic-similarity bias S added to the QK^T logits, scaled by a |
| per-layer positive scalar lambda; and (2) a per-token value gate G that |
| multiplies V before attention aggregation. The gate is also influenced by |
| a causal semantic-centrality term c derived from S. Equation: |
|
|
| Attn(Q,K,V) = softmax( QK^T / sqrt(d_k) + lambda * S + P + M ) . (G (*) V) |
|
|
| where |
| S_ij = cos(W_s h_i, W_s h_j) shared-projection cosine similarity |
| c_j = (1/n) sum_{l<=j} S_jl causal semantic centrality |
| G_j = sigmoid(w_g^T h_j + mu*c_j + b) per-token value gate |
| lambda = softplus(lambda_raw) constrained positive, per layer |
| P learnable [L,L] positional bias |
| M causal mask |
|
|
| W_s is shared across heads within a layer. G is per-token and broadcast |
| across head channels. Four variants were trained: |
| baseline use_S = False, use_G = False (standard attention) |
| s_only use_S = True, use_G = False (lambda*S only) |
| g_only use_S = False, use_G = True (gate only) |
| full use_S = True, use_G = True (both components) |
|
|
|
|
| ================================================================================ |
| 2. TRAINING SETUP (the 5-seed ablation) |
| ================================================================================ |
| Architecture: 24 layers, 384 dim, 6 heads, block size 256, vocab 50257 |
| Parameters: 63,376,512 (~64M) per variant |
| Optimizer: Fused AdamW, lr 6e-4, 200-step linear warmup, cosine to 6e-5 |
| Batch: 16, grad_accum 2 -> effective batch 32 |
| Tokens per step: 16 * 2 * 256 = 8,192 |
| Total steps: 12,000 -> 98,304,000 tokens per run (~98M) |
| Eval: every 500 steps on 40 val batches (val_ppl = exp(val_loss)) |
| Precision: bf16 autocast, TF32, cudnn.benchmark, SDPA attention path |
| Hardware: single RTX 4060 (vram ~6 GB during training) |
| Corpus: FineWeb-Edu sample-10BT, GPT-2 tokenization |
| Seeds: {0, 1, 2, 3, 4} for each of the four variants -> 20 runs total |
|
|
| Every variant uses an IDENTICAL data stream for the same seed (data_seed=1337 |
| plus run seed); this is what makes the per-seed comparison meaningful. |
|
|
|
|
| ================================================================================ |
| 3. PRIMARY RESULT - 5-SEED ABLATION TABLE |
| ================================================================================ |
| Final val perplexity at step 11999 (98M tokens seen), per variant per seed: |
|
|
| seed baseline s_only g_only full |
| ---- --------- ---------------- ---------------- ---------------- |
| 0 78.3244 78.7377 (+0.53%) 78.2175 (-0.14%) 77.8634 (-0.59%) |
| 1 80.9320 80.8649 (-0.08%) 80.5771 (-0.44%) 80.0262 (-1.12%) |
| 2 82.3365 82.0735 (-0.32%) 81.3363 (-1.21%) 81.2602 (-1.31%) |
| 3 79.3884 77.9830 (-1.77%) 77.9834 (-1.77%) 77.7016 (-2.12%) |
| 4 77.7449 77.6807 (-0.08%) 77.0092 (-0.95%) 76.3953 (-1.74%) |
|
|
| Per-seed deltas are computed as (variant_ppl - baseline_ppl) / baseline_ppl |
| within the same seed, NOT against the baseline mean. |
|
|
| Aggregate (mean across 5 seeds; std is population std): |
|
|
| variant n mean_ppl std delta_mean seed wins vs baseline |
| -------- -- --------- ------- ------------ ----------------------- |
| Baseline 5 79.7452 1.6894 - - |
| S-only 5 79.4680 1.7131 -0.348% 4 / 5 |
| G-only 5 79.0247 1.6463 -0.904% 5 / 5 |
| Full SMAT 5 78.6493 1.7498 -1.374% 5 / 5 |
|
|
| Key observations: |
| - Full SMAT beats baseline on EVERY seed (5/5). |
| - The two components are additive in direction: S contributes ~0.35%, |
| G contributes ~0.90%, and Full SMAT (both together) gives ~1.37%. |
| S + G individually sum to ~1.25%, slightly less than full's 1.37%; |
| the components are mostly additive, with a small interaction in their |
| favor. |
| - Variance ACROSS seeds (std ~1.7 ppl) is larger than the mean effect |
| (~1.1 ppl). The signal is real but small relative to seed noise; it |
| only becomes clear because every variant uses the SAME data stream per |
| seed, so the within-seed comparison cancels almost all of that noise. |
| - The smallest within-seed gain for Full SMAT is -0.59% (seed 0); |
| the largest is -2.12% (seed 3). Direction is consistent; magnitude |
| varies by roughly a factor of 4 across seeds. |
|
|
|
|
| ================================================================================ |
| 4. DIAGNOSTICS (training dynamics) |
| ================================================================================ |
| 4.1 Lambda (semantic-bias scale, per-layer, softplus-positive) |
| Init value (all layers, all seeds): 0.1269 ( = softplus(lambda_raw_init) ) |
| End-of-training mean lambda (averaged across 24 layers), per seed: |
| Full SMAT: s0 0.3069 s1 0.3075 s2 0.3031 s3 0.2954 s4 0.3176 |
| -> mean across seeds ~ 0.306 |
| S-only: s0 0.3140 s1 0.3188 s2 0.3221 s3 0.3183 s4 0.3261 |
| -> mean across seeds ~ 0.320 |
|
|
| Per-layer pattern (Full SMAT seed 0 final checkpoint, 24 layers): |
| layers 0-2: lambda ~ 0.14 - 0.18 (slightly below init) |
| layers 3-8: lambda ~ 0.08 - 0.16 (DECREASES below init) |
| layers 9-17: lambda ~ 0.32 - 0.68 (climbs sharply) |
| layers 18-23: lambda ~ 0.18 - 0.58 (settles, still elevated) |
| global min: 0.0834 (layer 5) |
| global max: 0.6821 (layer 16) |
|
|
| Across all 10 SMAT-bearing runs (s_only + full, 5 seeds each), |
| the observed per-layer lambda range was [0.059, 0.715]. The shape - |
| suppressed in early layers, dominant in middle layers - is robust |
| across all 5 seeds. |
|
|
| 4.2 Gate G (per-token, sigmoid output, averaged over tokens for logging) |
| Initial G_mean per run is around 0.5 (sigmoid initialized near zero |
| pre-activation). Across training G_mean rises briefly toward ~0.58 |
| in the first ~1000 steps, then decays. |
| Final G_mean at step 11999 (Full SMAT): s0 0.3506, s1 0.3333, |
| s2 0.3306, s3 0.3447, s4 0.3397 -> mean ~ 0.340 |
| Final G_mean (G-only): s0 0.3484, s1 0.3359, |
| s2 0.3676, s3 0.3625, s4 0.3563 -> mean ~ 0.354 |
| G_std (across tokens) at end of training is ~ 0.16, meaning the gate |
| is NOT collapsed: many tokens have G well above the mean, many have it |
| well below. Per-token differentiation is preserved. |
|
|
| 4.3 Stability |
| NaN failures across all 20 runs at 12k steps each (= 240,000 optimizer |
| steps total): ZERO. |
| No NaN-driven step skips were recorded in any run. Training is |
| numerically clean under bf16 + TF32 + fused AdamW on this hardware. |
|
|
| 4.4 Wall-clock cost |
| Baseline runs: ~4000 - 5300 sec per run (~ 1.1 - 1.5 hours) |
| S-only runs: ~4700 - 6100 sec |
| G-only runs: ~4300 - 5700 sec |
| Full runs: ~5400 - 6600 sec (~ 1.5 - 1.8 hours) |
| Cost overhead of Full SMAT vs baseline: roughly +20 to +35% wall time |
| at this size on a 4060. (Variation within each variant is hardware/ |
| thermal noise across the days the runs were executed.) |
|
|
|
|
| ================================================================================ |
| 5. INFERENCE-TIME SURGICAL ABLATIONS |
| ================================================================================ |
| Performed on the Full SMAT seed-0 final checkpoint (no retraining; the |
| trained weights are loaded, individual components are zeroed/replaced |
| during a forward pass, and val ppl is re-measured on 80 val batches). |
| Source: figures_final/ablation_inference.json |
|
|
| Intact Full SMAT (this measurement, 80 batches): val ppl = 79.010 |
| (The 40-batch eval at the end of training reported 77.863; the 80-batch |
| slice used for the ablation is a different val subset, hence the small |
| difference. What matters is the delta within this measurement context.) |
|
|
| ablation val_ppl delta vs intact what was changed |
| ------------ ----------- ----------------- ------------------------ |
| lambda_zero 79.40 +0.49% lambda set to 0 in the |
| attention bias only; |
| S still flows through |
| mu*c into the gate. |
| S_off 80.48 +1.85% S removed completely: |
| no lambda*S in attention |
| AND no mu*c in the gate. |
| random_S 81.22 +2.80% S replaced by a random |
| matrix with the same |
| Frobenius norm. Tests |
| whether S's STRUCTURE |
| matters, or just its norm. |
| G_one 625,850 +792,016% G forced to constant 1.0 |
| (gate disabled). |
| G_mean 196.99 +149.32% G replaced by its scalar |
| mean over tokens (per- |
| token differentiation |
| removed; mean value kept). |
|
|
| Decomposition of S's contribution: |
| Total S contribution (S_off vs intact): 1.85% |
| Contribution via lambda*S in attention: 0.49% -> 26% of S's lift |
| Contribution via mu*c in the gate: 1.36% -> 74% of S's lift |
| Most of S's value is being read through the gate's centrality term, |
| NOT through the additive logit bias. |
|
|
| Per-layer lambda=0 ablations (24 separate ablations, zeroing lambda in |
| one layer at a time): |
| Maximum single-layer cost: +0.0580% (layer 14) |
| Sum of all 24 per-layer deltas: +0.219% |
| This is far less than the +0.49% you get from zeroing lambda in every |
| layer at once. Interpretation: the lambda*S contribution is REDUNDANT |
| across layers - no single layer is load-bearing. The benefit is small |
| and distributed. |
|
|
| Structural vs scalar test: |
| random_S (+2.80%) hurts MORE than S_off (+1.85%). Replacing S with a |
| same-norm random matrix is worse than removing it entirely. This means |
| the model is using S's learned STRUCTURE, not just the presence of an |
| extra signal of a given magnitude. A random matrix with the right |
| energy actively damages the model; trained S is genuinely informative. |
|
|
| Gate criticality: |
| G=1.0 collapses the model entirely (val ppl > 600,000). The gate is not |
| a small modulation - the trained network depends on it as a primary |
| routing mechanism. |
| G=mean costs +149%. Replacing per-token gating with a single scalar mean |
| destroys most of the gate's value. The gate's worth is in WHICH tokens |
| it suppresses or amplifies, not in its average level. |
|
|
|
|
| ================================================================================ |
| 6. PARTIAL 4K-STEP RUN (1 seed, 32M tokens) - CONTEXT ONLY |
| ================================================================================ |
| A shorter run at 4,000 steps (32,768,000 tokens) was performed for ONE |
| seed before the full 12k-step ablation was launched. It is not directly |
| comparable to the 5-seed result above (different budget, no seed |
| variance), but it shows the same direction and a larger relative gap |
| because perplexity is higher overall at lower training budgets. |
|
|
| variant val_ppl delta vs baseline |
| --------- --------- ------------------ |
| baseline 148.4449 - |
| s_only 147.8903 -0.37% |
| g_only 144.9733 -2.34% |
| full 141.5651 -4.63% |
|
|
| This is consistent with the broader pattern: at smaller training budgets |
| (or smaller scales), the relative effect of SMAT is larger. As training |
| proceeds and the baseline gets closer to its asymptote, the absolute |
| ppl gap and the relative gap both shrink. The 12k-step 5-seed result |
| (section 3) is the rigorous number; the 4k-step result is a directional |
| sanity check. |
|
|
| DO NOT cite the 4k-step delta as if it were comparable to the 12k-step |
| delta. The user has already flagged this as a "partial run, not directly |
| comparable" in the repo README. |
|
|
|
|
| ================================================================================ |
| 7. CROSS-SCALE CONTEXT (PRIOR EXPERIMENTS, NOT FROM THIS REPO) |
| ================================================================================ |
| The following are described in this repo's README as prior experiments |
| done before this codebase. They are NOT re-verified from raw data here |
| and should be cited as historical context, not as primary evidence of |
| this paper's findings. If the paper depends on them, the underlying logs |
| should be located and re-verified before they appear in any final claim. |
|
|
| scale corpus tokens delta seeds note |
| ----- ------------------- -------- ------- ----- ------------------- |
| 7M Shakespeare short mixed 1 directional crossover |
| around step 150 |
| 100M FineWeb (smoke) short -7% 1 100-step smoke run |
| 100M FineWeb 100M -11% 3 of 3 strongest replicated |
| 100M FineMath 100M -17% 3 of 3 strongest result |
| 68M FineWeb (SMAT+A90) 20M -22.5% 1 SMAT combined with |
| BlockAttnRes; out |
| of scope for V1. |
| 64M FineWeb (THIS REPO) 98M -1.37% 5 of 5 primary result |
|
|
| The pattern across these (taking them at face value): direction is |
| preserved across every scale, every corpus, every seed where data |
| exists. Magnitude shrinks as the scale and training budget grow, with |
| the exception of the 100M experiments where the gap is largest. |
|
|
| If the paper's headline number is from the 100M FineWeb or FineMath |
| experiments, the V1 evidence is the 5-seed run in section 3 of THIS |
| document plus those prior experiments. The 5-seed result is the |
| hardest-to-dispute piece because every variant trained on identical |
| data and direction is consistent across 5/5 seeds. |
|
|
|
|
| ================================================================================ |
| 8. WHAT'S LOAD-BEARING vs WHAT IS NOT |
| ================================================================================ |
| Load-bearing (do not remove): |
| - The gate G. Catastrophic if removed (+792,016%) and badly degraded |
| if reduced to its mean (+149%). The gate is the single most important |
| component SMAT adds. |
| - The structure of S (not just its norm). random_S hurts more than |
| S_off, so the learned S projection contains real information. |
| - The mu*c centrality channel from S into the gate. This accounts for |
| ~74% of S's total contribution. If S is used at all, it should be |
| used HERE. |
|
|
| Not load-bearing in any individual layer (and largely redundant overall): |
| - The per-layer lambda*S term in the attention bias. Removing it in |
| one layer costs at most 0.06% ppl. Removing it in all layers costs |
| only 0.49%. The benefit exists but it is small and distributed; no |
| single layer's lambda is critical, and a uniform lambda across |
| layers is the wrong abstraction (early layers actively prefer it |
| near zero, middle layers want it near 0.5-0.7). |
|
|
| Net implication for V2 (this is what the data argues for): |
| - Drop the per-layer lambda_raw parameters and the lambda*S term in |
| attention. Keep S's projection W_s and keep mu*c flowing into the |
| gate. You retain ~74% of S's lift while removing 24 scalar |
| parameters and one matrix add per layer. |
| - If you want lambda to stay, replace per-layer lambda with a |
| per-layer gate-mix scalar driven by the same per-layer pattern |
| observed here (suppressed in early layers, elevated in middle |
| layers). |
|
|
|
|
| ================================================================================ |
| 9. CAVEATS, LIMITATIONS, THINGS NOT TO OVERCLAIM |
| ================================================================================ |
| - The headline -1.37% at 64M / 98M tokens is real (5/5 seeds, identical |
| per-seed data streams; one-sided sign test gives p = 1/32 = 0.03125 |
| under the null hypothesis of no effect), but it is SMALL in absolute |
| terms (about 1.1 perplexity points). The within-seed effect is what |
| makes it visible; do not present mean +/- std as if seed noise were |
| the only consideration. |
| - Std across seeds (1.65 - 1.75) is larger than the mean effect. |
| Always quote the WITHIN-SEED comparison or the SIGN TEST when making |
| the case, not just delta-of-means with overlapping error bars. |
| - The surgical ablation results in section 5 are from ONE checkpoint |
| (Full SMAT seed 0). The qualitative conclusions (G is critical, S |
| routes through mu*c, layer-wise lambda is redundant) are stark |
| enough that one checkpoint is informative, but it would be more |
| defensible to repeat on at least one more seed before publication. |
| - The cross-scale numbers in section 7 are not verified from raw data |
| in this repo. Do not put them in a results table without separately |
| locating and confirming the underlying logs. |
| - All training was on a single RTX 4060 with bf16 + TF32. Numbers may |
| shift slightly on different hardware; the direction should not. |
| - Tokenizer is GPT-2 byte-pair. Corpus is FineWeb-Edu sample-10BT. |
| Results on other corpora (FineMath in particular, see section 7) |
| may differ in magnitude. |
|
|
|
|
| ================================================================================ |
| 10. SUGGESTED PAPER CLAIMS (ordered by strength of evidence in this repo) |
| ================================================================================ |
| STRONG (directly supported by 20-run ablation, this repo, this document): |
| 1. Adding a per-token value gate to attention reduces val ppl by |
| ~0.9% on FineWeb at 64M params, 5/5 seeds. |
| 2. Adding both the semantic-similarity bias AND the gate reduces |
| val ppl by ~1.4% on FineWeb at 64M params, 5/5 seeds. |
| 3. Training is numerically stable (0 NaN failures over 240,000 |
| optimizer steps across 20 runs). |
| 4. lambda exhibits a robust per-layer pattern: suppressed in early |
| layers, elevated in middle layers, settled in late layers. |
| This pattern is consistent across all 5 seeds. |
|
|
| STRONG (supported by surgical-ablation evidence, single checkpoint): |
| 5. S's contribution flows ~74% through the gate's centrality term |
| and only ~26% through the additive logit bias. |
| 6. The trained S has learned STRUCTURE; replacing it with a random |
| matrix of equal norm hurts more than removing it entirely. |
| 7. The per-token gate is catastrophic to remove and very expensive |
| to replace with a mean. Per-token differentiation matters. |
|
|
| WEAKER (need cross-scale / cross-seed re-verification before paper): |
| 8. SMAT's relative effect is larger at smaller training budgets and |
| at certain corpora (FineMath in particular). |
| 9. SMAT directionality is preserved across scales 7M to 100M. |
|
|
| DO NOT CLAIM without further work: |
| - That SMAT is a uniformly large improvement (it is not at this scale). |
| - That SMAT helps at scales beyond 100M (no data yet). |
| - That per-layer lambda is essential (the surgical ablation shows it |
| is largely redundant). |
|
|
|
|
| ================================================================================ |
| 11. RAW NUMBERS (machine-readable, for any tables in the paper) |
| ================================================================================ |
| 12k-step 5-seed ablation, final val ppl: |
|
|
| baseline = [78.32438343443285, 80.93203870134798, 82.33645607970243, |
| 79.38843805716219, 77.74486565469935] |
| s_only = [78.73771250646860, 80.86492907631813, 82.07353893301419, |
| 77.98296438017636, 77.68072112072338] |
| g_only = [78.21752726763506, 80.57706364462581, 81.33629330240751, |
| 77.98339201086276, 77.00922636947199] |
| full = [77.86344065821494, 80.02616085611275, 81.26017429181104, |
| 77.70156877103959, 76.39525023929815] |
|
|
| Means: |
| baseline 79.74523638546896 |
| s_only 79.46797320334013 |
| g_only 79.02470051900063 |
| full 78.64931896329529 |
|
|
| Population std: |
| baseline 1.6894... s_only 1.7131... g_only 1.6463... full 1.7498... |
|
|
| Sample (n-1) std (if you prefer): |
| baseline 1.8888 s_only 1.9153 g_only 1.8406 full 1.9563 |
|
|
| Delta of means vs baseline (%): |
| s_only -0.3477 g_only -0.9035 full -1.3743 |
|
|
| Per-layer trained lambda (Full SMAT seed 0, 24 values, from |
| ablation_inference.json): |
| [0.1837, 0.1491, 0.1557, 0.1365, 0.1209, 0.0834, 0.1213, 0.0920, |
| 0.1605, 0.3192, 0.3323, 0.4566, 0.3248, 0.5507, 0.5484, 0.4943, |
| 0.6821, 0.4693, 0.3914, 0.5843, 0.1961, 0.3536, 0.2781, 0.1833] |
| (printed to 4 dp; full precision in figures_final/ablation_inference.json) |
|
|
| Surgical ablation deltas (Full SMAT seed 0, 80-batch eval slice, intact |
| ppl 79.010): |
| lambda_zero +0.4903% S_off +1.8547% random_S +2.8032% |
| G_one +792,015.9609% G_mean +149.3211% |
|
|
|
|
| ================================================================================ |
| END |
| ================================================================================ |
|
|
