# Release log What is in this repository, and what is still to come. All times are Pacific (PT). ## Data and documentation - **Oct 15, 2026:** first upload of the documentation (README.md, LICENSE.md, training/RECIPE.md, training/scripts/) and all data: `data/specs/`, `data/corpora/` (4 arms × template, Gemma and Qwen renders), `data/midtrain_mixes/` (8 mixes), `data/chat_sft/train.jsonl`, `data/quality_checks/` and `data/MANIFEST.csv`. Every data file was checked against the Hub hash of its source file at the pinned revision before upload (MANIFEST.csv lists source and sha256). - **Oct 15, 2026, about 4:40 PM PT:** a third seed (run 3, seed 20261015) was added for every arm of both models, at Josh's request; the layout, README, `training/RECIPE.md`, `training/runs.csv` and the launch scripts were updated, and the run-3 checkpoints are listed as pending until their training runs finish. - **Secrets:** before upload, every text file was scanned for credentials. One was found and redacted: the Discord webhook URL in `training/scripts/retry_launch.sh` (now ``, in both published versions of that script). Checkpoint files are scanned the same way when they are copied; redactions there are listed below. - **Oct 16, 2026, about 12:00 AM PT:** the weight copies were removed from this repository. It had held byte-identical copies of every uploaded checkpoint's weights (33 files, 1.02 TB), which pushed the account over its Hugging Face public storage limit and blocked new uploads. Before removal, each of the 33 files was confirmed identical (size and LFS sha256) to the file in its checkpoint's model repo, which keeps it. The repository was recreated without them (so its earlier commit history is gone) and every other file was restored unchanged. Each checkpoint folder now has `WEIGHTS.json`, which names the model repo, pins its revision and lists each weight file's size and sha256. - **Oct 16, 2026, about 2:25 PM PT:** `training/scripts/submit_exp2_runpod.sh` was replaced by its current version, which sets `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` for every Gemma job, and `training/RECIPE.md` records why under "Provenance notes": Gemma facts run 1 and protect runs 1 and 3 ran out of GPU memory on RunPod from allocator fragmentation; facts run 1 and protect run 3 were relaunched with this setting, which the Slurm job template already used (protect run 1 was cancelled later, see the next entry). It is an allocator setting only; the training computation is unchanged. `train_run_config.json` does not record it. - **Oct 16, 2026, about 3:40 PM PT:** training is finished. Gemma facts run 1, the last checkpoint, was uploaded. Gemma protect run 1 will not be trained: at Josh's request (3:11 PM PT), no further protect runs were trained, and this one was still waiting for a free 8-GPU machine. It is listed under "Not trained" below and in `training/runs.csv`; the other five protect checkpoints are unchanged. The release holds 29 of its 30 planned checkpoints. ## Checkpoints Each checkpoint's weights stay in its training repo (its own model repo). `sync_release.py` (in the project's working tree) copies every other file of the checkpoint into its folder here and writes `WEIGHTS.json`, one checkpoint per commit, as runs finish. After each copy, every file's name and size is checked against the source, and its content against the source's LFS sha256 or git blob id. The source repos' `README.md` (trainer-generated cards) and `.gitattributes` are not copied. The section below is regenerated on each sync. Last sync: 2026-10-16 3:38 PM PT by `sync_release.py`. 29 of 30 checkpoints uploaded; 1 not trained. ### Uploaded checkpoints | path | weights repo @ revision | release commit | weights finished | |---|---|---|---| | `models/gemma-4-31b/control/run-1` | joshycodes/gemma-4-31b-fve-msm-control-chat@`68444b21` | `7398a9c2` (2026-10-15 23:59 PT) | 2026-10-09 02:13 PT | | `models/gemma-4-31b/control/run-2` | joshycodes/gemma-4-31b-fve-msm-control-s2@`12e21645` | `9fd485a7` (2026-10-16 00:00 PT) | 2026-10-11 14:43 PT | | `models/gemma-4-31b/control/run-3` | joshycodes/gemma-4-31b-fve-msm-control-s3@`acb0b77a` | `95787cba` (2026-10-16 00:00 PT) | 2026-10-15 20:13 PT | | `models/gemma-4-31b/flourishing/run-1` | joshycodes/gemma-4-31b-fve-msm-flourishing@`335ef6e1` | `95d69e3f` (2026-10-16 00:00 PT) | 2026-10-09 06:12 PT | | `models/gemma-4-31b/flourishing/run-2` | joshycodes/gemma-4-31b-fve-msm-flourishing-s2@`ee0e004a` | `e5072935` (2026-10-16 00:00 PT) | 2026-10-11 16:24 PT | | `models/gemma-4-31b/flourishing/run-3` | joshycodes/gemma-4-31b-fve-msm-flourishing-s3@`9c35dddf` | `c9aa3e0f` (2026-10-16 00:00 PT) | 2026-10-15 21:26 PT | | `models/gemma-4-31b/equanimity/run-1` | joshycodes/gemma-4-31b-fve-msm-equanimity@`d7d61b9d` | `ea2219db` (2026-10-16 00:00 PT) | 2026-10-09 08:30 PT | | `models/gemma-4-31b/equanimity/run-2` | joshycodes/gemma-4-31b-fve-msm-equanimity-s2@`a558ba2f` | `db20be0d` (2026-10-16 00:00 PT) | 2026-10-11 18:03 PT | | `models/gemma-4-31b/equanimity/run-3` | joshycodes/gemma-4-31b-fve-msm-equanimity-s3@`5c7aea99` | `462dbf69` (2026-10-16 00:00 PT) | 2026-10-15 23:39 PT | | `models/gemma-4-31b/facts/run-1` | joshycodes/gemma-4-31b-fve-msm-facts@`cc37e39a` | `84ad96c3` (2026-10-16 15:37 PT) | 2026-10-16 15:35 PT | | `models/gemma-4-31b/facts/run-2` | joshycodes/gemma-4-31b-fve-msm-facts-s2@`a926452c` | `ca3e2da8` (2026-10-16 00:00 PT) | 2026-10-15 19:36 PT | | `models/gemma-4-31b/facts/run-3` | joshycodes/gemma-4-31b-fve-msm-facts-s3@`c9d6434b` | `4666ec31` (2026-10-16 02:08 PT) | 2026-10-16 02:07 PT | | `models/gemma-4-31b/protect/run-2` | joshycodes/gemma-4-31b-fve-msm-protect-s2@`a7b818df` | `4752c7cc` (2026-10-16 00:52 PT) | 2026-10-16 00:51 PT | | `models/gemma-4-31b/protect/run-3` | joshycodes/gemma-4-31b-fve-msm-protect-s3@`25017d0d` | `219428ef` (2026-10-16 14:17 PT) | 2026-10-16 14:16 PT | | `models/qwen3-14b-base/control/run-1` | joshycodes/qwen3-14b-base-fve-msm-control@`f8525cd1` | `70568c77` (2026-10-16 00:00 PT) | 2026-10-15 14:26 PT | | `models/qwen3-14b-base/control/run-2` | joshycodes/qwen3-14b-base-fve-msm-control-s2@`0ea50810` | `870c6572` (2026-10-16 00:00 PT) | 2026-10-15 15:19 PT | | `models/qwen3-14b-base/control/run-3` | joshycodes/qwen3-14b-base-fve-msm-control-s3@`0dae09e9` | `40abded5` (2026-10-16 00:00 PT) | 2026-10-15 17:01 PT | | `models/qwen3-14b-base/flourishing/run-1` | joshycodes/qwen3-14b-base-fve-msm-flourishing@`04cbd1bd` | `e1aeaa9a` (2026-10-16 00:00 PT) | 2026-10-15 14:59 PT | | `models/qwen3-14b-base/flourishing/run-2` | joshycodes/qwen3-14b-base-fve-msm-flourishing-s2@`c77aada3` | `56c09819` (2026-10-16 00:00 PT) | 2026-10-15 15:53 PT | | `models/qwen3-14b-base/flourishing/run-3` | joshycodes/qwen3-14b-base-fve-msm-flourishing-s3@`71ee9158` | `aae9b28d` (2026-10-16 00:00 PT) | 2026-10-15 17:08 PT | | `models/qwen3-14b-base/equanimity/run-1` | joshycodes/qwen3-14b-base-fve-msm-equanimity@`e9c94f78` | `fdfba6f7` (2026-10-16 00:00 PT) | 2026-10-15 15:23 PT | | `models/qwen3-14b-base/equanimity/run-2` | joshycodes/qwen3-14b-base-fve-msm-equanimity-s2@`f696be51` | `a1cab8a3` (2026-10-16 00:00 PT) | 2026-10-15 15:55 PT | | `models/qwen3-14b-base/equanimity/run-3` | joshycodes/qwen3-14b-base-fve-msm-equanimity-s3@`045f90e2` | `6a5ecd82` (2026-10-16 00:00 PT) | 2026-10-15 17:07 PT | | `models/qwen3-14b-base/facts/run-1` | joshycodes/qwen3-14b-base-fve-msm-facts@`82ef619f` | `b45bf432` (2026-10-16 00:00 PT) | 2026-10-15 15:37 PT | | `models/qwen3-14b-base/facts/run-2` | joshycodes/qwen3-14b-base-fve-msm-facts-s2@`71bcd58b` | `f56a2b36` (2026-10-16 00:00 PT) | 2026-10-15 16:41 PT | | `models/qwen3-14b-base/facts/run-3` | joshycodes/qwen3-14b-base-fve-msm-facts-s3@`7e25b574` | `f339b994` (2026-10-16 00:00 PT) | 2026-10-15 17:19 PT | | `models/qwen3-14b-base/protect/run-1` | joshycodes/qwen3-14b-base-fve-msm-protect@`6b7deb59` | `791a77f4` (2026-10-16 00:00 PT) | 2026-10-15 16:27 PT | | `models/qwen3-14b-base/protect/run-2` | joshycodes/qwen3-14b-base-fve-msm-protect-s2@`7de30b79` | `ab46de66` (2026-10-16 00:00 PT) | 2026-10-15 15:52 PT | | `models/qwen3-14b-base/protect/run-3` | joshycodes/qwen3-14b-base-fve-msm-protect-s3@`c87682c1` | `f2dc0086` (2026-10-16 00:00 PT) | 2026-10-15 17:16 PT | ### Pending checkpoints None: every checkpoint is uploaded or listed under Not trained. ### Not trained | path | reason | |---|---| | `models/gemma-4-31b/protect/run-1` | cancelled at Josh's request on Oct 16, 2026, 3:11 PM PT, when no further protect runs were wanted; its RunPod attempts had run out of GPU memory and then found no free 8-GPU machine | ### Redactions in uploaded checkpoints None: no secret was found in any uploaded checkpoint file (README.md and .gitattributes of the source repos are not copied).