Correct model card; add LICENSE with nanoGPT attribution

Model card corrections, all verified against result files in this repo:
- stability counts were seeds-completed, not seeds-stable (baseline 0/3, cheap_qa 1/3, midtier_q 3/3 per priority1 CSV)
- the A10 result is an ordering permutation, not a sign flip; cheap_qa beats baseline on A10 (4.4675 -> 4.4151)
- phases 14-17 tie with or edge past cheap_qa; phase 19b was not the best elaboration but the second worst
- added the full_x parameter-efficiency comparison (+7.08M vs +98K)
- added LICENSE retaining the nanoGPT (c) 2022 Andrej Karpathy notice
- fixed the docstring's grep string for the G1 seam marker

LICENSE

@@ -0,0 +1,34 @@
+MIT License
+
+Copyright (c) 2022 Andrej Karpathy
+Copyright (c) 2026 Brian Kolb
+
+This repository is derived from nanoGPT (https://github.com/karpathy/nanoGPT),
+Copyright (c) 2022 Andrej Karpathy, distributed under the MIT License.
+
+The following files contain substantial portions of the upstream nanoGPT source,
+modified to add the gating variants studied here:
+
+    code/model_canon_HEAD.py
+    code/model_wslbridge.py
+    code/model_cheap_qa_minimal.py
+    code/train_canon_HEAD.py
+    code/train_wslbridge.py
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -19,6 +19,17 @@ complete result tables — including the runs that did **not** support the hypot
 This repository is an archive first and a model release second. Nothing here is intended
 for downstream use as a general-purpose language model.
 
+**What the data supports, in one paragraph.** A single small gate at one seam of the
+attention block produces a real improvement over an ungated baseline — replicated at 124M
+and at 1B, on three hardware configurations, significant at both scales. Almost nothing
+beyond that survives contact with a second measurement. Which signal the gate is
+conditioned on, how it is parameterized, and how the variants rank against each other all
+change when the hardware changes or the scale changes, and seven of the variants sit
+inside the seed noise floor of one another at 124M. The most useful result in this corpus
+is the mechanism for that: on the one family where the parameters are directly
+observable, the loss is flat along the axis being tuned, and training never moves the
+parameters off their initialization.
+
 ## The intervention
 
 A single gate applied at the **G1 seam** — post-SDPA, pre-output-projection:
@@ -34,17 +45,81 @@ y = y * gate                                # before c_proj (W_O)
 At the 124M scale this costs **98,304 parameters (+0.079%)**.
 
 `code/model_cheap_qa_minimal.py` is a self-contained single-variant reference
-implementation with the port surface explicitly marked `# === G1 SEAM ===`.
+implementation with the port surface marked by `G1 SEAM` banner comments.
+
+## Relation to published work
+
+Gating the SDPA output at the G1 seam is established. **This is a small-scale replication
+and mechanism study, not a novelty claim.**
+
+- **Qiu et al. 2025**, [*Gated Attention for Large Language Models*](https://arxiv.org/abs/2505.06708)
+  (NeurIPS 2025 Oral). Establishes the G1 gate at 15B-MoE and 1.7B-dense scale on up to 3.5T
+  tokens. Their formulation is `Y' = Y ⊙ σ(X·Wθ)` — the gate is computed from **X, the layer
+  input**, in all fifteen variants they report. Their SDPA-elementwise G1 winner adds 201M
+  parameters. `full_x` / `x_full_headspec` here are that formulation ported to nanoGPT.
+- **Bu et al. 2025**, [*Value-State Gated Attention*](https://arxiv.org/abs/2510.09017).
+  Moves the gate input from X to **V**, arguing a reactive gate (computed from the value
+  state) decouples value and attention updates better than a predictive one.
+- **Zhou et al. 2026**, [*Hybrid Gated Attention*](https://arxiv.org/abs/2608.11805)
+  (Tencent Hunyuan, 12 Aug 2026). Adds an **H-gate** computed from the SDPA output `H`,
+  fused with the X-gate, on the observation that *"the output H contains richer
+  token-interaction information after attention"* that X does not capture. Validated on
+  MoE-5B over 500B tokens.
+
+**Where `cheap_qa` sits.** Its gate is computed from `cat(q, y)` — the query and the SDPA
+output being gated. The `y` term is the same signal Zhou et al. call `H`. On the gate-input
+axis the published sequence is X → V → X+H, and `cat(q, y)` is a point on it that none of
+the three tested: one `Linear(2·head_dim → head_dim)`, 98,304 parameters at 124M.
+
+**Chronology.** This work reached G1 gating independently and found the prior art
+afterward. `cheap_qa` was confirmed 2026-03-20
+(`results/csv/cheap_qa_confirmation_runs_20260320_1854.csv`). Qiu et al. was located days
+later, at which point the project was reclassified from a novelty claim to a replication
+and mechanism study, and their X-conditioned formulation was implemented as `full_x`
+*specifically to run head-to-head against* `cheap_qa`
+(`results/summaries/full_x_seed123_result_20260324_1712.txt`, 2026-03-24). Zhou et al.
+appeared 2026-08-12, five months after `cheap_qa`. **None of this establishes priority** —
+these results were private until publication. It is recorded only because the dates are
+checkable in the files.
+
+**What replicates here.** Two of Qiu et al.'s qualitative findings reproduce at 124M on one
+RTX 3060, at roughly 1/10,000 of their token budget:
+
+| claim | Qiu et al. (15B MoE, 400B tok) | here (124M, ~41M tok) |
+|---|---|---|
+| G1 gating beats baseline | 6.026 → 5.761 PPL | 4.4708 → 4.3897 CE |
+| input-*dependent* gating beats input-*independent* | 5.917 → 5.761 | 4.4483 → 4.3897 |
+
+The input-independent control is an exact structural match: Qiu's is a zero-initialized
+learnable `(q × dk)` parameter through a sigmoid; `static_prehead` here is
+`nn.Parameter(torch.zeros(n_head, head_dim))` through a sigmoid. In both cases it recovers
+roughly a third of the gain (41% there, 28% here), confirming that part of the effect is
+non-linearity alone and the rest requires input dependence.
+
+**What is *not* claimed.** Qiu et al. also find head-specific gating beats head-shared. The
+variants here that would test it (`x_headshared`, 4.4346, versus `full_x`, 4.3950) differ in
+gate *input width* as well as head-sharing — `Linear(64→64)` on a per-head slice versus
+`Linear(768→768)` on the full hidden state — so the comparison is confounded and is not
+offered as a replication. The unconfounded pair
+(`dynamic_x_g1_headspecific_elementwise` vs `dynamic_x_g1_headshared_elementwise`) was only
+run under the earlier, superseded recipe.
 
 ## Headline results
 
-**nanoGPT 124M, OpenWebText, 10k iters, warmup 2000, seeds 42/1337/123** (val CE):
+**nanoGPT 124M, OpenWebText, 10k iters, warmup 2000, seeds 42/1337/123** (val CE).
+Source: `results/csv/priority1_runs_20260322_1355.csv`.
 
-| variant | val CE | vs baseline | stable |
-|---|---|---|---|
-| baseline | 4.4708 | — | 3/3 |
-| **cheap_qa** (Q+A conditioned) | **4.3897** | **−0.0811 (−1.81%)** | 3/3 |
-| midtier_q (Q conditioned) | 4.4135 | −0.0573 | 3/3 |
+| variant | added params | val CE | vs baseline | best-at-final |
+|---|---|---|---|---|
+| baseline | — | 4.4708 ± 0.0074 | — | 0/3 |
+| **cheap_qa** (gate on `cat(q, y)`) | +98,304 (+0.079%) | **4.3897 ± 0.0051** | **−0.0811 (−1.81%)** | 1/3 |
+| midtier_q (per-head MLP on q) | +196,608 (+0.159%) | 4.4135 ± 0.0086 | −0.0573 (−1.28%) | 3/3 |
+| cheap_q (linear gate on q) | +49,152 (+0.040%) | 4.4198 ± 0.0185 | −0.0510 (−1.14%) | 2/3 |
+
+**best-at-final** is the repo's `stable` flag: the count of seeds whose best validation
+loss occurred at the last evaluation (step 10000) rather than earlier. It flags late
+training drift, not divergence — all twelve runs completed. Note that the baseline is 0/3
+on it and midtier_q is the only variant that is clean.
 
 **OLMo-2 1B, OpenWebText, 3 seeds** (val perplexity):
 
@@ -54,23 +129,59 @@ implementation with the port surface explicitly marked `# === G1 SEAM ===`.
 | cheap_qa | 43.703 | 0.124 |
 | midtier_q | 43.683 | 0.159 |
 
-## Honest limitations — please read before citing
+Gating beats baseline: cheap_qa t(4) = −3.45, p = 0.026; midtier_q t(4) = −3.28, p = 0.031.
 
-1. **The 1B run did not leave learning-rate warmup.** It stopped at step 1500 of a planned
-   7,630 (~197M of 1B tokens), which coincides almost exactly with `t_warmup`. Models were
-   evaluated at peak LR, never annealed. The `tier1`/`tier2` measurement points in the run
-   summaries are `nan` because they were never reached.
-2. **The Q+A > Q ordering does not replicate at 1B.** cheap_qa and midtier_q differ by
-   0.02 ppl — a tie well inside seed noise — despite a clear separation at 124M.
-   Gating beats baseline (t(4)≈3.4, p≈0.03); the *ordering among gates* does not survive.
-3. **Seed 42 of the 1B run is not procedurally identical** to seeds 1337/123: it resumed
-   from `step1536-unsharded` (152s runtime vs ~8,400s).
-4. **A hardware-dependent sign flip is unexplained.** On an NVIDIA A10, `cheap_qa`
-   *degrades* relative to baseline, while on an RTX 3060 (both native and under WSL) it
-   improves. This is documented in `results/` and has no accepted explanation.
-5. **The elaborations did not beat the simple gate.** Phases 14–19d explored soft-Q,
-   soft-QA, per-layer and per-dimension blends. Best of these (phase 19b, 6 seeds):
-   4.4015 ± 0.0104 with only **2/6 seeds stable**, versus the champion's 4.3897 at 3/3.
+## What does not survive a second measurement
+
+**The ordering among gates is not stable across hardware.** On the RTX 3060, cheap_qa
+(4.3897) beats midtier_q (4.4135). On an NVIDIA A10 under a matched recipe, midtier_q
+(4.4039) beats cheap_qa (4.4151). Both gates still beat their baseline on both machines —
+the effect is real on both — but the ranking inverts. In the 19-variant A10 bulk sweep
+(`results/phase8_...`) cheap_qa falls from 1st to 7th. The A10 replication file's own
+verdict line reads `Hardware consistency: INVESTIGATE ✗`.
+
+**The ordering among gates does not survive scale either.** At 1B, cheap_qa and midtier_q
+differ by 0.02 ppl — t(4) = 0.17, p = 0.87, an unqualified tie — despite clear separation
+at 124M. This is the same failure as the A10 result, at a different axis. Two independent
+observations of the same thing is not a fluke; it is the finding.
+
+**The elaborations neither beat the simple gate nor lose to it.** Phases 14–19d explored
+soft-Q, soft-QA, asymmetric normalization, per-layer and per-dimension blends. Mean best
+val CE, same recipe, 3 seeds each:
+
+| phase | variant | mean best val CE | vs cheap_qa |
+|---|---|---|---|
+| 14 | qa_softq | 4.3865 | −0.0032 |
+| 15 | qa_normed_qonly | 4.3881 | −0.0016 |
+| 16 | qa_softqa | 4.3889 | −0.0008 |
+| 15 | qa_normed_yonly | 4.3890 | −0.0007 |
+| 17 | qa_softqa_pl | 4.3890 | −0.0007 |
+| — | **cheap_qa** | **4.3897** | — |
+| 19 | qa_softqa_pd | 4.3929 | +0.0032 |
+| 19c | per-dim, informed init | 4.3950 | +0.0053 |
+| 19b | per-dim, 6 seeds | 4.4015 | +0.0118 |
+| 19d | per-dim, free init | 4.4085 | +0.0188 |
+
+Five variants sit nominally ahead of the champion and none of the gaps mean anything:
+phase 14 vs cheap_qa is t = −0.50, p = 0.64; phase 17 is t = −0.09, p = 0.94. Seven
+variants span 0.0032 CE, against a within-variant seed sd of ~0.005–0.009. **No
+elaboration produced a reliable improvement over the simple gate, and the simple gate is
+not reliably better than any of them.** cheap_qa is presented as the champion because it
+is the cheapest thing in that indistinguishable cluster, not because it won.
+
+**The one comparison that does separate, separates on cost.** `full_x` — a `Linear(768→768)`
+gate per layer conditioned on the layer input, i.e. the published G1 formulation — completes
+3 seeds at **4.3950 ± 0.0116** (`results/summaries/full_x_seed123_result_20260324_1712.txt`).
+cheap_qa reaches **4.3897 ± 0.0051** with **+98,304 params (+0.079%)** against full_x's
+**+7,077,888 (+5.71%)**. Lower mean, less than half the seed spread, 1/72 the parameters.
+
+⚠️ Note the earlier file `results/summaries/p2_rerun_summary_20260323_1714.txt` ranks full_x
+first at 4.3870. That is a **two-seed partial** written while seed 123 was still running; the
+March 24 file above supersedes it.
+
+Off-recipe, for completeness: warmup 3000 (phase 18b) reaches 4.3761 and warmup 1000
+(phase 18a) collapses to 4.5441. Warmup dominates every architectural difference measured
+here.
 
 ## The most interesting finding is a negative one
 
@@ -83,9 +194,73 @@ initialization schemes, the learned scales stay wherever they were initialized:
 | 19d free | 0.50 + noise | 0.4845 / 0.5003 | 4.4085 |
 | 19c informed | 0.74 | 0.7211 / 0.7423 | 4.3950 |
 
+This is not an averaging artifact. In the raw per-dimension vectors, with init 0.74 every
+one of the 768 scales across all 12 layers lands in 0.70–0.77; with init 0.50 they land in
+0.42–0.54. Individual dimensions do not move either.
+
 The loss landscape is flat along this direction, so initialization determines the final
-value — and an informed prior (taken from the phase-17 converged scalar) ends up
-0.0135 CE ahead of free initialization.
+value. Informed init (0.74, carried over from an earlier converged scalar) ends 0.0135 CE
+ahead of free init — directionally consistent with the flatness story, but inside the
+free-init condition's own ±0.0221 seed spread, so treat it as suggestive rather than
+measured.
+
+**This is the mechanism behind the two non-replications above.** If the loss is flat along
+the parameters a variant is tuning, then which variant wins is decided by noise, and the
+ranking is free to permute when the hardware or the scale changes. Connects to
+ReZero / Fixup / SkipInit / LayerScale and the lazy-training literature.
+
+## Honest limitations — please read before citing
+
+1. **The 1B run did not leave learning-rate warmup.** It stopped at step 1500 of a planned
+   7,630 (~197M of 1B tokens), which coincides almost exactly with `t_warmup`. Models were
+   evaluated at peak LR, never annealed. The `tier1`/`tier2` measurement points in the run
+   summaries are `nan` because they were never reached. Whether the 124M improvement
+   survives annealing at 1B is untested.
+2. **Seed 42 of the 1B run resumed from a checkpoint** (`step1536-unsharded`, ~150s runtime
+   vs ~8,400s) rather than training end to end. This applies to all three variants
+   symmetrically, so it does not bias the comparison, but seed 42 is not a fresh replicate.
+3. **The 124M stability rate is poor across the board.** Only 1 of 3 cheap_qa seeds, and 0
+   of 3 baseline seeds, ended with their best validation loss at the final evaluation. Most
+   runs were drifting upward over the last few hundred steps. The reported figures are best
+   val CE, so this affects how much to trust any of the small differences above.
+4. **Two 3060 result boards exist.** The canonical head-to-head is
+   `results/csv/priority1_runs_20260322_1355.csv` (baseline 4.4708 / cheap_qa 4.3897), used
+   throughout this card. An earlier board in `results/summaries/cheap_qa_confirmation_...`
+   and `runner_a_summary_...` gives 4.4973 / 4.4073. The improvement is comparable
+   (−0.0900 vs −0.0811); the absolute values are not.
+5. **Some result files contain auto-generated boilerplate that its own tables contradict.**
+   In particular `results/summaries/a10_replication_summary_20260322_0639.txt` ends with a
+   `PAPER STATEMENTS` block asserting hardware independence and `|delta| < 0.02`. That text
+   is unedited script output. The table forty lines above it in the same file shows deltas
+   to −0.1257 and prints `Hardware consistency: INVESTIGATE ✗`. Trust the tables, not the
+   prose blocks.
+6. **Standard deviations are not on one convention.** The 124M figures are population sd as
+   printed by the run scripts; the 1B figures are sample sd. Differences at the third
+   decimal between this card and a source file are usually this.
+
+## Traps in the result files
+
+⚠️ **`results/phase19b_canonical_canonical_qa_softqa_pd_w2000_6seeds_summary_20260403_0709.txt`
+and `results/phase19b_canonical_qa_softqa_pd_w2000_6seeds_20260403_0709.csv` are a FAILED
+run.** All six seeds are `returncode=2`, `best=nan`, no checkpoint written. They sit beside
+the real six-seed data from the same day (`..._0711`) under a nearly identical name. Do not
+read numbers from the `0709` files.
+
+⚠️ **`results/summaries/p2_rerun_summary_20260323_1714.txt` contains a superseded
+leaderboard.** Its "COMPLETE CLEAN RANKING" puts `full_x` first at 4.3870 on two seeds. Seed
+123 finished the following day at 4.4112, moving full_x to 4.3950 ± 0.0116 — see
+`results/summaries/full_x_seed123_result_20260324_1712.txt`, which is authoritative.
+
+⚠️ **`checkpoints/canon/out-shakespeare-char/`** is the upstream nanoGPT demo, not part of
+this study. Retained only for completeness.
+
+⚠️ **`results/logs/command prompt summaru end 3-24-26.txt`** (filename typo preserved) is a
+raw console transcript, and it is the provenance record for a data correction: in the
+Priority 2 pass, `static_prehead`, `full_x` and `x_headshared` reported 5.76 / 6.94 / 6.95
+because the disk filled during `torch.save` and killed the runs at step 250, not because of
+the architecture. After clearing space and rerunning without checkpointing, full_x went
+from 6.9417 to 4.3870. If you find early numbers in this repo that look catastrophic,
+check that transcript before believing them.
 
 ## Layout
 
@@ -102,9 +277,6 @@ Checkpoints are nanoGPT-format `ckpt.pt` (`model`, `optimizer`, `model_args`, `i
 `best_val_loss`, `config`) at fp32, ~1.4 GiB each — optimizer state included so training
 can be resumed.
 
-⚠️ `checkpoints/canon/out-shakespeare-char/` is the upstream nanoGPT demo, not part of
-this study. It is retained only for completeness.
-
 ## Verifying integrity
 
 ```bash
@@ -114,5 +286,13 @@ sha256sum -c SHA256SUMS.txt
 ## Known open questions
 
 - Does the 124M improvement survive annealing at 1B? (unresolved — the run never annealed)
-- Why does the effect reverse on A10 vs 3060?
+- Is the gate ranking decided by anything other than noise? Two measurements say no.
 - Is the per-dimension flatness specific to this gate, or general to blend parameters?
+- Why does the *magnitude* of the improvement differ so much by hardware (−0.081 on 3060,
+  −0.052 on A10) when the direction does not?
+
+## Attribution
+
+Derived from [nanoGPT](https://github.com/karpathy/nanoGPT) by Andrej Karpathy (MIT).
+`code/model_canon_HEAD.py` and `code/model_wslbridge.py` contain substantial portions of
+the upstream source. See `LICENSE`.

code/model_cheap_qa_minimal.py

@@ -1,7 +1,7 @@
 """
 Minimal single-variant GPT — cheap_qa champion ONLY.
 
-Distilled from the 19-variant env_canon model.py. Every other gate branch has been
+Distilled from the 19-variant code/model_canon_HEAD.py in this repo. Every other gate branch has been
 removed; the ONLY conditioning path is the cheap_qa gate:
 
     cheap_qa  ==  qv_variant "dynamic_qa_conditioned"
@@ -11,7 +11,7 @@ removed; the ONLY conditioning path is the cheap_qa gate:
 Champion result (NanoGPT 124M, OpenWebText, 10k iters, seeds 42/1337/123):
     baseline 4.4708 -> cheap_qa 4.3897  (-0.0811 CE, -1.81%), 98,304 params (+0.079%).
 
->>> THE G1 SEAM IS MARKED BELOW WITH:  # === G1 SEAM ===
+>>> THE G1 SEAM IS MARKED BELOW BY A "G1 SEAM" BANNER COMMENT.
     That block is the entire port surface. Resize the gate for a different attention
     by changing head_dim (hs); the gate is always Linear(2*hs -> hs) on cat(q, y).
 """