================================================================================ SMAT (Semantic Attention) - Findings Summary for Paper ================================================================================ Compiled: 2026-06-03 Source files used (all in this repo, all re-verified for this document): runs/{baseline,s_only,g_only,full}_s{0..4}/metrics.jsonl (20 training runs) runs_4ksteps/{baseline,s_only,g_only,full}_s0/metrics.jsonl (4 partial runs) figures_final/ablation_inference.json (surgical ablations) figures_final/ablation_results.csv (final eval table) Every number in sections 2-6 below was recomputed directly from the raw metrics.jsonl files (not copied from any pre-existing CSV or README). The per-seed deltas, means, std devs, and lambda/G statistics were all re-derived from scratch and cross-checked twice. The cross-scale results in section 7 are from prior experiments outside this repo; they are cited as such and were not re-verified from raw data here. Treat them as historical context, not as primary evidence. ================================================================================ 1. WHAT SMAT IS (one paragraph) ================================================================================ SMAT modifies standard transformer attention with two additions: (1) a learnable semantic-similarity bias S added to the QK^T logits, scaled by a per-layer positive scalar lambda; and (2) a per-token value gate G that multiplies V before attention aggregation. The gate is also influenced by a causal semantic-centrality term c derived from S. Equation: Attn(Q,K,V) = softmax( QK^T / sqrt(d_k) + lambda * S + P + M ) . (G (*) V) where S_ij = cos(W_s h_i, W_s h_j) shared-projection cosine similarity c_j = (1/n) sum_{l<=j} S_jl causal semantic centrality G_j = sigmoid(w_g^T h_j + mu*c_j + b) per-token value gate lambda = softplus(lambda_raw) constrained positive, per layer P learnable [L,L] positional bias M causal mask W_s is shared across heads within a layer. G is per-token and broadcast across head channels. Four variants were trained: baseline use_S = False, use_G = False (standard attention) s_only use_S = True, use_G = False (lambda*S only) g_only use_S = False, use_G = True (gate only) full use_S = True, use_G = True (both components) ================================================================================ 2. TRAINING SETUP (the 5-seed ablation) ================================================================================ Architecture: 24 layers, 384 dim, 6 heads, block size 256, vocab 50257 Parameters: 63,376,512 (~64M) per variant Optimizer: Fused AdamW, lr 6e-4, 200-step linear warmup, cosine to 6e-5 Batch: 16, grad_accum 2 -> effective batch 32 Tokens per step: 16 * 2 * 256 = 8,192 Total steps: 12,000 -> 98,304,000 tokens per run (~98M) Eval: every 500 steps on 40 val batches (val_ppl = exp(val_loss)) Precision: bf16 autocast, TF32, cudnn.benchmark, SDPA attention path Hardware: single RTX 4060 (vram ~6 GB during training) Corpus: FineWeb-Edu sample-10BT, GPT-2 tokenization Seeds: {0, 1, 2, 3, 4} for each of the four variants -> 20 runs total Every variant uses an IDENTICAL data stream for the same seed (data_seed=1337 plus run seed); this is what makes the per-seed comparison meaningful. ================================================================================ 3. PRIMARY RESULT - 5-SEED ABLATION TABLE ================================================================================ Final val perplexity at step 11999 (98M tokens seen), per variant per seed: seed baseline s_only g_only full ---- --------- ---------------- ---------------- ---------------- 0 78.3244 78.7377 (+0.53%) 78.2175 (-0.14%) 77.8634 (-0.59%) 1 80.9320 80.8649 (-0.08%) 80.5771 (-0.44%) 80.0262 (-1.12%) 2 82.3365 82.0735 (-0.32%) 81.3363 (-1.21%) 81.2602 (-1.31%) 3 79.3884 77.9830 (-1.77%) 77.9834 (-1.77%) 77.7016 (-2.12%) 4 77.7449 77.6807 (-0.08%) 77.0092 (-0.95%) 76.3953 (-1.74%) Per-seed deltas are computed as (variant_ppl - baseline_ppl) / baseline_ppl within the same seed, NOT against the baseline mean. Aggregate (mean across 5 seeds; std is population std): variant n mean_ppl std delta_mean seed wins vs baseline -------- -- --------- ------- ------------ ----------------------- Baseline 5 79.7452 1.6894 - - S-only 5 79.4680 1.7131 -0.348% 4 / 5 G-only 5 79.0247 1.6463 -0.904% 5 / 5 Full SMAT 5 78.6493 1.7498 -1.374% 5 / 5 Key observations: - Full SMAT beats baseline on EVERY seed (5/5). - The two components are additive in direction: S contributes ~0.35%, G contributes ~0.90%, and Full SMAT (both together) gives ~1.37%. S + G individually sum to ~1.25%, slightly less than full's 1.37%; the components are mostly additive, with a small interaction in their favor. - Variance ACROSS seeds (std ~1.7 ppl) is larger than the mean effect (~1.1 ppl). The signal is real but small relative to seed noise; it only becomes clear because every variant uses the SAME data stream per seed, so the within-seed comparison cancels almost all of that noise. - The smallest within-seed gain for Full SMAT is -0.59% (seed 0); the largest is -2.12% (seed 3). Direction is consistent; magnitude varies by roughly a factor of 4 across seeds. ================================================================================ 4. DIAGNOSTICS (training dynamics) ================================================================================ 4.1 Lambda (semantic-bias scale, per-layer, softplus-positive) Init value (all layers, all seeds): 0.1269 ( = softplus(lambda_raw_init) ) End-of-training mean lambda (averaged across 24 layers), per seed: Full SMAT: s0 0.3069 s1 0.3075 s2 0.3031 s3 0.2954 s4 0.3176 -> mean across seeds ~ 0.306 S-only: s0 0.3140 s1 0.3188 s2 0.3221 s3 0.3183 s4 0.3261 -> mean across seeds ~ 0.320 Per-layer pattern (Full SMAT seed 0 final checkpoint, 24 layers): layers 0-2: lambda ~ 0.14 - 0.18 (slightly below init) layers 3-8: lambda ~ 0.08 - 0.16 (DECREASES below init) layers 9-17: lambda ~ 0.32 - 0.68 (climbs sharply) layers 18-23: lambda ~ 0.18 - 0.58 (settles, still elevated) global min: 0.0834 (layer 5) global max: 0.6821 (layer 16) Across all 10 SMAT-bearing runs (s_only + full, 5 seeds each), the observed per-layer lambda range was [0.059, 0.715]. The shape - suppressed in early layers, dominant in middle layers - is robust across all 5 seeds. 4.2 Gate G (per-token, sigmoid output, averaged over tokens for logging) Initial G_mean per run is around 0.5 (sigmoid initialized near zero pre-activation). Across training G_mean rises briefly toward ~0.58 in the first ~1000 steps, then decays. Final G_mean at step 11999 (Full SMAT): s0 0.3506, s1 0.3333, s2 0.3306, s3 0.3447, s4 0.3397 -> mean ~ 0.340 Final G_mean (G-only): s0 0.3484, s1 0.3359, s2 0.3676, s3 0.3625, s4 0.3563 -> mean ~ 0.354 G_std (across tokens) at end of training is ~ 0.16, meaning the gate is NOT collapsed: many tokens have G well above the mean, many have it well below. Per-token differentiation is preserved. 4.3 Stability NaN failures across all 20 runs at 12k steps each (= 240,000 optimizer steps total): ZERO. No NaN-driven step skips were recorded in any run. Training is numerically clean under bf16 + TF32 + fused AdamW on this hardware. 4.4 Wall-clock cost Baseline runs: ~4000 - 5300 sec per run (~ 1.1 - 1.5 hours) S-only runs: ~4700 - 6100 sec G-only runs: ~4300 - 5700 sec Full runs: ~5400 - 6600 sec (~ 1.5 - 1.8 hours) Cost overhead of Full SMAT vs baseline: roughly +20 to +35% wall time at this size on a 4060. (Variation within each variant is hardware/ thermal noise across the days the runs were executed.) ================================================================================ 5. INFERENCE-TIME SURGICAL ABLATIONS ================================================================================ Performed on the Full SMAT seed-0 final checkpoint (no retraining; the trained weights are loaded, individual components are zeroed/replaced during a forward pass, and val ppl is re-measured on 80 val batches). Source: figures_final/ablation_inference.json Intact Full SMAT (this measurement, 80 batches): val ppl = 79.010 (The 40-batch eval at the end of training reported 77.863; the 80-batch slice used for the ablation is a different val subset, hence the small difference. What matters is the delta within this measurement context.) ablation val_ppl delta vs intact what was changed ------------ ----------- ----------------- ------------------------ lambda_zero 79.40 +0.49% lambda set to 0 in the attention bias only; S still flows through mu*c into the gate. S_off 80.48 +1.85% S removed completely: no lambda*S in attention AND no mu*c in the gate. random_S 81.22 +2.80% S replaced by a random matrix with the same Frobenius norm. Tests whether S's STRUCTURE matters, or just its norm. G_one 625,850 +792,016% G forced to constant 1.0 (gate disabled). G_mean 196.99 +149.32% G replaced by its scalar mean over tokens (per- token differentiation removed; mean value kept). Decomposition of S's contribution: Total S contribution (S_off vs intact): 1.85% Contribution via lambda*S in attention: 0.49% -> 26% of S's lift Contribution via mu*c in the gate: 1.36% -> 74% of S's lift Most of S's value is being read through the gate's centrality term, NOT through the additive logit bias. Per-layer lambda=0 ablations (24 separate ablations, zeroing lambda in one layer at a time): Maximum single-layer cost: +0.0580% (layer 14) Sum of all 24 per-layer deltas: +0.219% This is far less than the +0.49% you get from zeroing lambda in every layer at once. Interpretation: the lambda*S contribution is REDUNDANT across layers - no single layer is load-bearing. The benefit is small and distributed. Structural vs scalar test: random_S (+2.80%) hurts MORE than S_off (+1.85%). Replacing S with a same-norm random matrix is worse than removing it entirely. This means the model is using S's learned STRUCTURE, not just the presence of an extra signal of a given magnitude. A random matrix with the right energy actively damages the model; trained S is genuinely informative. Gate criticality: G=1.0 collapses the model entirely (val ppl > 600,000). The gate is not a small modulation - the trained network depends on it as a primary routing mechanism. G=mean costs +149%. Replacing per-token gating with a single scalar mean destroys most of the gate's value. The gate's worth is in WHICH tokens it suppresses or amplifies, not in its average level. ================================================================================ 6. PARTIAL 4K-STEP RUN (1 seed, 32M tokens) - CONTEXT ONLY ================================================================================ A shorter run at 4,000 steps (32,768,000 tokens) was performed for ONE seed before the full 12k-step ablation was launched. It is not directly comparable to the 5-seed result above (different budget, no seed variance), but it shows the same direction and a larger relative gap because perplexity is higher overall at lower training budgets. variant val_ppl delta vs baseline --------- --------- ------------------ baseline 148.4449 - s_only 147.8903 -0.37% g_only 144.9733 -2.34% full 141.5651 -4.63% This is consistent with the broader pattern: at smaller training budgets (or smaller scales), the relative effect of SMAT is larger. As training proceeds and the baseline gets closer to its asymptote, the absolute ppl gap and the relative gap both shrink. The 12k-step 5-seed result (section 3) is the rigorous number; the 4k-step result is a directional sanity check. DO NOT cite the 4k-step delta as if it were comparable to the 12k-step delta. The user has already flagged this as a "partial run, not directly comparable" in the repo README. ================================================================================ 7. CROSS-SCALE CONTEXT (PRIOR EXPERIMENTS, NOT FROM THIS REPO) ================================================================================ The following are described in this repo's README as prior experiments done before this codebase. They are NOT re-verified from raw data here and should be cited as historical context, not as primary evidence of this paper's findings. If the paper depends on them, the underlying logs should be located and re-verified before they appear in any final claim. scale corpus tokens delta seeds note ----- ------------------- -------- ------- ----- ------------------- 7M Shakespeare short mixed 1 directional crossover around step 150 100M FineWeb (smoke) short -7% 1 100-step smoke run 100M FineWeb 100M -11% 3 of 3 strongest replicated 100M FineMath 100M -17% 3 of 3 strongest result 68M FineWeb (SMAT+A90) 20M -22.5% 1 SMAT combined with BlockAttnRes; out of scope for V1. 64M FineWeb (THIS REPO) 98M -1.37% 5 of 5 primary result The pattern across these (taking them at face value): direction is preserved across every scale, every corpus, every seed where data exists. Magnitude shrinks as the scale and training budget grow, with the exception of the 100M experiments where the gap is largest. If the paper's headline number is from the 100M FineWeb or FineMath experiments, the V1 evidence is the 5-seed run in section 3 of THIS document plus those prior experiments. The 5-seed result is the hardest-to-dispute piece because every variant trained on identical data and direction is consistent across 5/5 seeds. ================================================================================ 8. WHAT'S LOAD-BEARING vs WHAT IS NOT ================================================================================ Load-bearing (do not remove): - The gate G. Catastrophic if removed (+792,016%) and badly degraded if reduced to its mean (+149%). The gate is the single most important component SMAT adds. - The structure of S (not just its norm). random_S hurts more than S_off, so the learned S projection contains real information. - The mu*c centrality channel from S into the gate. This accounts for ~74% of S's total contribution. If S is used at all, it should be used HERE. Not load-bearing in any individual layer (and largely redundant overall): - The per-layer lambda*S term in the attention bias. Removing it in one layer costs at most 0.06% ppl. Removing it in all layers costs only 0.49%. The benefit exists but it is small and distributed; no single layer's lambda is critical, and a uniform lambda across layers is the wrong abstraction (early layers actively prefer it near zero, middle layers want it near 0.5-0.7). Net implication for V2 (this is what the data argues for): - Drop the per-layer lambda_raw parameters and the lambda*S term in attention. Keep S's projection W_s and keep mu*c flowing into the gate. You retain ~74% of S's lift while removing 24 scalar parameters and one matrix add per layer. - If you want lambda to stay, replace per-layer lambda with a per-layer gate-mix scalar driven by the same per-layer pattern observed here (suppressed in early layers, elevated in middle layers). ================================================================================ 9. CAVEATS, LIMITATIONS, THINGS NOT TO OVERCLAIM ================================================================================ - The headline -1.37% at 64M / 98M tokens is real (5/5 seeds, identical per-seed data streams; one-sided sign test gives p = 1/32 = 0.03125 under the null hypothesis of no effect), but it is SMALL in absolute terms (about 1.1 perplexity points). The within-seed effect is what makes it visible; do not present mean +/- std as if seed noise were the only consideration. - Std across seeds (1.65 - 1.75) is larger than the mean effect. Always quote the WITHIN-SEED comparison or the SIGN TEST when making the case, not just delta-of-means with overlapping error bars. - The surgical ablation results in section 5 are from ONE checkpoint (Full SMAT seed 0). The qualitative conclusions (G is critical, S routes through mu*c, layer-wise lambda is redundant) are stark enough that one checkpoint is informative, but it would be more defensible to repeat on at least one more seed before publication. - The cross-scale numbers in section 7 are not verified from raw data in this repo. Do not put them in a results table without separately locating and confirming the underlying logs. - All training was on a single RTX 4060 with bf16 + TF32. Numbers may shift slightly on different hardware; the direction should not. - Tokenizer is GPT-2 byte-pair. Corpus is FineWeb-Edu sample-10BT. Results on other corpora (FineMath in particular, see section 7) may differ in magnitude. ================================================================================ 10. SUGGESTED PAPER CLAIMS (ordered by strength of evidence in this repo) ================================================================================ STRONG (directly supported by 20-run ablation, this repo, this document): 1. Adding a per-token value gate to attention reduces val ppl by ~0.9% on FineWeb at 64M params, 5/5 seeds. 2. Adding both the semantic-similarity bias AND the gate reduces val ppl by ~1.4% on FineWeb at 64M params, 5/5 seeds. 3. Training is numerically stable (0 NaN failures over 240,000 optimizer steps across 20 runs). 4. lambda exhibits a robust per-layer pattern: suppressed in early layers, elevated in middle layers, settled in late layers. This pattern is consistent across all 5 seeds. STRONG (supported by surgical-ablation evidence, single checkpoint): 5. S's contribution flows ~74% through the gate's centrality term and only ~26% through the additive logit bias. 6. The trained S has learned STRUCTURE; replacing it with a random matrix of equal norm hurts more than removing it entirely. 7. The per-token gate is catastrophic to remove and very expensive to replace with a mean. Per-token differentiation matters. WEAKER (need cross-scale / cross-seed re-verification before paper): 8. SMAT's relative effect is larger at smaller training budgets and at certain corpora (FineMath in particular). 9. SMAT directionality is preserved across scales 7M to 100M. DO NOT CLAIM without further work: - That SMAT is a uniformly large improvement (it is not at this scale). - That SMAT helps at scales beyond 100M (no data yet). - That per-layer lambda is essential (the surgical ablation shows it is largely redundant). ================================================================================ 11. RAW NUMBERS (machine-readable, for any tables in the paper) ================================================================================ 12k-step 5-seed ablation, final val ppl: baseline = [78.32438343443285, 80.93203870134798, 82.33645607970243, 79.38843805716219, 77.74486565469935] s_only = [78.73771250646860, 80.86492907631813, 82.07353893301419, 77.98296438017636, 77.68072112072338] g_only = [78.21752726763506, 80.57706364462581, 81.33629330240751, 77.98339201086276, 77.00922636947199] full = [77.86344065821494, 80.02616085611275, 81.26017429181104, 77.70156877103959, 76.39525023929815] Means: baseline 79.74523638546896 s_only 79.46797320334013 g_only 79.02470051900063 full 78.64931896329529 Population std: baseline 1.6894... s_only 1.7131... g_only 1.6463... full 1.7498... Sample (n-1) std (if you prefer): baseline 1.8888 s_only 1.9153 g_only 1.8406 full 1.9563 Delta of means vs baseline (%): s_only -0.3477 g_only -0.9035 full -1.3743 Per-layer trained lambda (Full SMAT seed 0, 24 values, from ablation_inference.json): [0.1837, 0.1491, 0.1557, 0.1365, 0.1209, 0.0834, 0.1213, 0.0920, 0.1605, 0.3192, 0.3323, 0.4566, 0.3248, 0.5507, 0.5484, 0.4943, 0.6821, 0.4693, 0.3914, 0.5843, 0.1961, 0.3536, 0.2781, 0.1833] (printed to 4 dp; full precision in figures_final/ablation_inference.json) Surgical ablation deltas (Full SMAT seed 0, 80-batch eval slice, intact ppl 79.010): lambda_zero +0.4903% S_off +1.8547% random_S +2.8032% G_one +792,015.9609% G_mean +149.3211% ================================================================================ END ================================================================================