kolbrian/qgate-checkpoints

QGate — checkpoints and results from a 19-phase attention-gating ablation

Training artifacts from an independent research program on query-conditioned attention gating in GPT-2-class transformers. 33 checkpoints, the code that produced them, and the complete result tables — including the runs that did not support the hypothesis.

This repository is an archive first and a model release second. Nothing here is intended for downstream use as a general-purpose language model.

What the data supports, in one paragraph. A single small gate at one seam of the attention block produces a real improvement over an ungated baseline — replicated at 124M and at 1B, on three hardware configurations, significant at both scales. Almost nothing beyond that survives contact with a second measurement. Which signal the gate is conditioned on, how it is parameterized, and how the variants rank against each other all change when the hardware changes or the scale changes, and seven of the variants sit inside the seed noise floor of one another at 124M. The most useful result in this corpus is the mechanism for that: on the one family where the parameters are directly observable, the loss is flat along the axis being tuned, and training never moves the parameters off their initialization.

The intervention

A single gate applied at the G1 seam — post-SDPA, pre-output-projection:

hs = n_embd // n_head                       # head_dim
self.qa_gate_proj = nn.Linear(hs * 2, hs, bias=False)

gate = torch.sigmoid(self.qa_gate_proj(torch.cat([q, y], dim=-1)))
y = y * gate                                # before c_proj (W_O)

At the 124M scale this costs 98,304 parameters (+0.079%).

code/model_cheap_qa_minimal.py is a self-contained single-variant reference implementation with the port surface marked by G1 SEAM banner comments.

Relation to published work

Gating the SDPA output at the G1 seam is established. This is a small-scale replication and mechanism study, not a novelty claim.

• Qiu et al. 2025, Gated Attention for Large Language Models (NeurIPS 2025 Oral). Establishes the G1 gate at 15B-MoE and 1.7B-dense scale on up to 3.5T tokens. Their formulation is Y' = Y ⊙ σ(X·Wθ) — the gate is computed from X, the layer input, in all fifteen variants they report. Their SDPA-elementwise G1 winner adds 201M parameters. full_x / x_full_headspec here are that formulation ported to nanoGPT.
• Bu et al. 2025, Value-State Gated Attention. Moves the gate input from X to V, arguing a reactive gate (computed from the value state) decouples value and attention updates better than a predictive one.
• Zhou et al. 2026, Hybrid Gated Attention (Tencent Hunyuan, 12 Aug 2026). Adds an H-gate computed from the SDPA output H, fused with the X-gate, on the observation that "the output H contains richer token-interaction information after attention" that X does not capture. Validated on MoE-5B over 500B tokens.

Where cheap_qa sits. Its gate is computed from cat(q, y) — the query and the SDPA output being gated. The y term is the same signal Zhou et al. call H. On the gate-input axis the published sequence is X → V → X+H, and cat(q, y) is a point on it that none of the three tested: one Linear(2·head_dim → head_dim), 98,304 parameters at 124M.

Chronology. This work reached G1 gating independently and found the prior art afterward. cheap_qa was confirmed 2026-03-20 (results/csv/cheap_qa_confirmation_runs_20260320_1854.csv). Qiu et al. was located days later, at which point the project was reclassified from a novelty claim to a replication and mechanism study, and their X-conditioned formulation was implemented as full_x specifically to run head-to-head against cheap_qa (results/summaries/full_x_seed123_result_20260324_1712.txt, 2026-03-24). Zhou et al. appeared 2026-08-12, five months after cheap_qa. None of this establishes priority — these results were private until publication. It is recorded only because the dates are checkable in the files.

What replicates here. Two of Qiu et al.'s qualitative findings reproduce at 124M on one RTX 3060, at roughly 1/10,000 of their token budget:

claim	Qiu et al. (15B MoE, 400B tok)	here (124M, ~41M tok)
G1 gating beats baseline	6.026 → 5.761 PPL	4.4708 → 4.3897 CE
input-dependent gating beats input-independent	5.917 → 5.761	4.4483 → 4.3897

The input-independent control is an exact structural match: Qiu's is a zero-initialized learnable (q × dk) parameter through a sigmoid; static_prehead here is nn.Parameter(torch.zeros(n_head, head_dim)) through a sigmoid. In both cases it recovers roughly a third of the gain (41% there, 28% here), confirming that part of the effect is non-linearity alone and the rest requires input dependence.

What is not claimed. Qiu et al. also find head-specific gating beats head-shared. The variants here that would test it (x_headshared, 4.4346, versus full_x, 4.3950) differ in gate input width as well as head-sharing — Linear(64→64) on a per-head slice versus Linear(768→768) on the full hidden state — so the comparison is confounded and is not offered as a replication. The unconfounded pair (dynamic_x_g1_headspecific_elementwise vs dynamic_x_g1_headshared_elementwise) was only run under the earlier, superseded recipe.

Headline results

nanoGPT 124M, OpenWebText, 10k iters, warmup 2000, seeds 42/1337/123 (val CE). Source: results/csv/priority1_runs_20260322_1355.csv.

variant	added params	val CE	vs baseline	best-at-final
baseline	—	4.4708 ± 0.0074	—	0/3
cheap_qa (gate on cat(q, y))	+98,304 (+0.079%)	4.3897 ± 0.0051	−0.0811 (−1.81%)	1/3
midtier_q (per-head MLP on q)	+196,608 (+0.159%)	4.4135 ± 0.0086	−0.0573 (−1.28%)	3/3
cheap_q (linear gate on q)	+49,152 (+0.040%)	4.4198 ± 0.0185	−0.0510 (−1.14%)	2/3

best-at-final is the repo's stable flag: the count of seeds whose best validation loss occurred at the last evaluation (step 10000) rather than earlier. It flags late training drift, not divergence — all twelve runs completed. Note that the baseline is 0/3 on it and midtier_q is the only variant that is clean.

OLMo-2 1B, OpenWebText, 3 seeds (val perplexity):

variant	mean ppl	sd
baseline	44.137	0.179
cheap_qa	43.703	0.124
midtier_q	43.683	0.159

Gating beats baseline: cheap_qa t(4) = −3.45, p = 0.026; midtier_q t(4) = −3.28, p = 0.031.

What does not survive a second measurement

The ordering among gates is not stable across hardware. On the RTX 3060, cheap_qa (4.3897) beats midtier_q (4.4135). On an NVIDIA A10 under a matched recipe, midtier_q (4.4039) beats cheap_qa (4.4151). Both gates still beat their baseline on both machines — the effect is real on both — but the ranking inverts. In the 19-variant A10 bulk sweep (results/phase8_...) cheap_qa falls from 1st to 7th. The A10 replication file's own verdict line reads Hardware consistency: INVESTIGATE ✗.

The ordering among gates does not survive scale either. At 1B, cheap_qa and midtier_q differ by 0.02 ppl — t(4) = 0.17, p = 0.87, an unqualified tie — despite clear separation at 124M. This is the same failure as the A10 result, at a different axis. Two independent observations of the same thing is not a fluke; it is the finding.

The elaborations neither beat the simple gate nor lose to it. Phases 14–19d explored soft-Q, soft-QA, asymmetric normalization, per-layer and per-dimension blends. Mean best val CE, same recipe, 3 seeds each:

phase	variant	mean best val CE	vs cheap_qa
14	qa_softq	4.3865	−0.0032
15	qa_normed_qonly	4.3881	−0.0016
16	qa_softqa	4.3889	−0.0008
15	qa_normed_yonly	4.3890	−0.0007
17	qa_softqa_pl	4.3890	−0.0007
—	cheap_qa	4.3897	—
19	qa_softqa_pd	4.3929	+0.0032
19c	per-dim, informed init	4.3950	+0.0053
19b	per-dim, 6 seeds	4.4015	+0.0118
19d	per-dim, free init	4.4085	+0.0188

Five variants sit nominally ahead of the champion and none of the gaps mean anything: phase 14 vs cheap_qa is t = −0.50, p = 0.64; phase 17 is t = −0.09, p = 0.94. Seven variants span 0.0032 CE, against a within-variant seed sd of ~0.005–0.009. No elaboration produced a reliable improvement over the simple gate, and the simple gate is not reliably better than any of them. cheap_qa is presented as the champion because it is the cheapest thing in that indistinguishable cluster, not because it won.

The one comparison that does separate, separates on cost. full_x — a Linear(768→768) gate per layer conditioned on the layer input, i.e. the published G1 formulation — completes 3 seeds at 4.3950 ± 0.0116 (results/summaries/full_x_seed123_result_20260324_1712.txt). cheap_qa reaches 4.3897 ± 0.0051 with +98,304 params (+0.079%) against full_x's +7,077,888 (+5.71%). Lower mean, less than half the seed spread, 1/72 the parameters.

⚠️ Note the earlier file results/summaries/p2_rerun_summary_20260323_1714.txt ranks full_x first at 4.3870. That is a two-seed partial written while seed 123 was still running; the March 24 file above supersedes it.

Off-recipe, for completeness: warmup 3000 (phase 18b) reaches 4.3761 and warmup 1000 (phase 18a) collapses to 4.5441. Warmup dominates every architectural difference measured here.

The most interesting finding is a negative one

The per-dimension blend scales are not identified by training. Across 12 runs and three initialization schemes, the learned scales stay wherever they were initialized:

phase	init	converged scale (q / y)	val CE
19b per-dim	0.50	≈0.48 / ≈0.50	4.4015
19d free	0.50 + noise	0.4845 / 0.5003	4.4085
19c informed	0.74	0.7211 / 0.7423	4.3950

This is not an averaging artifact. In the raw per-dimension vectors, with init 0.74 every one of the 768 scales across all 12 layers lands in 0.70–0.77; with init 0.50 they land in 0.42–0.54. Individual dimensions do not move either.

The loss landscape is flat along this direction, so initialization determines the final value. Informed init (0.74, carried over from an earlier converged scalar) ends 0.0135 CE ahead of free init — directionally consistent with the flatness story, but inside the free-init condition's own ±0.0221 seed spread, so treat it as suggestive rather than measured.

This is the mechanism behind the two non-replications above. If the loss is flat along the parameters a variant is tuning, then which variant wins is decided by noise, and the ranking is free to permute when the hardware or the scale changes. Connects to ReZero / Fixup / SkipInit / LayerScale and the lazy-training literature.

Honest limitations — please read before citing

1. The 1B run did not leave learning-rate warmup. It stopped at step 1500 of a planned 7,630 (~197M of 1B tokens), which coincides almost exactly with t_warmup. Models were evaluated at peak LR, never annealed. The tier1/tier2 measurement points in the run summaries are nan because they were never reached. Whether the 124M improvement survives annealing at 1B is untested.
2. Seed 42 of the 1B run resumed from a checkpoint (step1536-unsharded, ~150s runtime vs ~8,400s) rather than training end to end. This applies to all three variants symmetrically, so it does not bias the comparison, but seed 42 is not a fresh replicate.
3. The 124M stability rate is poor across the board. Only 1 of 3 cheap_qa seeds, and 0 of 3 baseline seeds, ended with their best validation loss at the final evaluation. Most runs were drifting upward over the last few hundred steps. The reported figures are best val CE, so this affects how much to trust any of the small differences above.
4. Two 3060 result boards exist. The canonical head-to-head is results/csv/priority1_runs_20260322_1355.csv (baseline 4.4708 / cheap_qa 4.3897), used throughout this card. An earlier board in results/summaries/cheap_qa_confirmation_... and runner_a_summary_... gives 4.4973 / 4.4073. The improvement is comparable (−0.0900 vs −0.0811); the absolute values are not.
5. Some result files contain auto-generated boilerplate that its own tables contradict. In particular results/summaries/a10_replication_summary_20260322_0639.txt ends with a PAPER STATEMENTS block asserting hardware independence and |delta| < 0.02. That text is unedited script output. The table forty lines above it in the same file shows deltas to −0.1257 and prints Hardware consistency: INVESTIGATE ✗. Trust the tables, not the prose blocks.
6. Standard deviations are not on one convention. The 124M figures are population sd as printed by the run scripts; the 1B figures are sample sd. Differences at the third decimal between this card and a source file are usually this.

Traps in the result files

⚠️ results/phase19b_canonical_canonical_qa_softqa_pd_w2000_6seeds_summary_20260403_0709.txt and results/phase19b_canonical_qa_softqa_pd_w2000_6seeds_20260403_0709.csv are a FAILED run. All six seeds are returncode=2, best=nan, no checkpoint written. They sit beside the real six-seed data from the same day (..._0711) under a nearly identical name. Do not read numbers from the 0709 files.

⚠️ results/summaries/p2_rerun_summary_20260323_1714.txt contains a superseded leaderboard. Its "COMPLETE CLEAN RANKING" puts full_x first at 4.3870 on two seeds. Seed 123 finished the following day at 4.4112, moving full_x to 4.3950 ± 0.0116 — see results/summaries/full_x_seed123_result_20260324_1712.txt, which is authoritative.

⚠️ checkpoints/canon/out-shakespeare-char/ is the upstream nanoGPT demo, not part of this study. Retained only for completeness.

⚠️ results/logs/command prompt summaru end 3-24-26.txt (filename typo preserved) is a raw console transcript, and it is the provenance record for a data correction: in the Priority 2 pass, static_prehead, full_x and x_headshared reported 5.76 / 6.94 / 6.95 because the disk filled during torch.save and killed the runs at step 250, not because of the architecture. After clearing space and rerunning without checkpointing, full_x went from 6.9417 to 4.3870. If you find early numbers in this repo that look catastrophic, check that transcript before believing them.

Layout

checkpoints/
  canon/       21 x ckpt.pt   base run + phases 14-19d      (RTX 3060, Windows)
  wsl_bridge/  12 x ckpt.pt   phases 10-12, 4 variants x 3 seeds (RTX 3060, WSL)
code/          model/train for both lanes + the minimal champion module
results/       phase 7-19d run CSVs and summaries
SHA256SUMS.txt integrity manifest for all 33 checkpoints

Checkpoints are nanoGPT-format ckpt.pt (model, optimizer, model_args, iter_num, best_val_loss, config) at fp32, ~1.4 GiB each — optimizer state included so training can be resumed.

Verifying integrity

sha256sum -c SHA256SUMS.txt

Known open questions

• Does the 124M improvement survive annealing at 1B? (unresolved — the run never annealed)
• Is the gate ranking decided by anything other than noise? Two measurements say no.
• Is the per-dimension flatness specific to this gate, or general to blend parameters?
• Why does the magnitude of the improvement differ so much by hardware (−0.081 on 3060, −0.052 on A10) when the direction does not?

Attribution

Derived from nanoGPT by Andrej Karpathy (MIT). code/model_canon_HEAD.py and code/model_wslbridge.py contain substantial portions of the upstream source. See LICENSE.