Release log
What is in this repository, and what is still to come. All times are Pacific (PT).
Data and documentation
- Oct 15, 2026: first upload of the documentation (README.md, LICENSE.md, training/RECIPE.md, training/scripts/)
and all data:
data/specs/,data/corpora/(4 arms × template, Gemma and Qwen renders),data/midtrain_mixes/(8 mixes),data/chat_sft/train.jsonl,data/quality_checks/anddata/MANIFEST.csv. Every data file was checked against the Hub hash of its source file at the pinned revision before upload (MANIFEST.csv lists source and sha256). - Oct 15, 2026, about 4:40 PM PT: a third seed (run 3, seed 20261015) was added for every arm of both models, at
Josh's request; the layout, README,
training/RECIPE.md,training/runs.csvand the launch scripts were updated, and the run-3 checkpoints are listed as pending until their training runs finish. - Secrets: before upload, every text file was scanned for credentials. One was found and redacted: the Discord
webhook URL in
training/scripts/retry_launch.sh(now<redacted>, in both published versions of that script). Checkpoint files are scanned the same way when they are copied; redactions there are listed below. - Oct 16, 2026, about 12:00 AM PT: the weight copies were removed from this repository. It had held byte-identical
copies of every uploaded checkpoint's weights (33 files, 1.02 TB), which pushed the account over its Hugging Face
public storage limit and blocked new uploads. Before removal, each of the 33 files was confirmed identical (size and
LFS sha256) to the file in its checkpoint's model repo, which keeps it. The repository was recreated without them
(so its earlier commit history is gone) and every other file was restored unchanged. Each checkpoint folder now has
WEIGHTS.json, which names the model repo, pins its revision and lists each weight file's size and sha256. - Oct 16, 2026, about 2:25 PM PT:
training/scripts/submit_exp2_runpod.shwas replaced by its current version, which setsPYTORCH_CUDA_ALLOC_CONF=expandable_segments:Truefor every Gemma job, andtraining/RECIPE.mdrecords why under "Provenance notes": Gemma facts run 1 and protect runs 1 and 3 ran out of GPU memory on RunPod from allocator fragmentation; facts run 1 and protect run 3 were relaunched with this setting, which the Slurm job template already used (protect run 1 was cancelled later, see the next entry). It is an allocator setting only; the training computation is unchanged.train_run_config.jsondoes not record it. - Oct 16, 2026, about 3:40 PM PT: training is finished. Gemma facts run 1, the last checkpoint, was uploaded.
Gemma protect run 1 will not be trained: at Josh's request (3:11 PM PT), no further protect runs were trained, and
this one was still waiting for a free 8-GPU machine. It is listed under "Not trained" below and in
training/runs.csv; the other five protect checkpoints are unchanged. The release holds 29 of its 30 planned checkpoints.
Checkpoints
Each checkpoint's weights stay in its training repo (its own model repo). sync_release.py (in the project's working
tree) copies every other file of the checkpoint into its folder here and writes WEIGHTS.json, one checkpoint per
commit, as runs finish. After each copy, every file's name and size is checked against the source, and its content
against the source's LFS sha256 or git blob id. The source repos' README.md (trainer-generated cards) and
.gitattributes are not copied. The section below is regenerated on each sync.
Last sync: 2026-10-16 3:38 PM PT by sync_release.py. 29 of 30 checkpoints uploaded; 1 not trained.
Uploaded checkpoints
| path | weights repo @ revision | release commit | weights finished |
|---|---|---|---|
models/gemma-4-31b/control/run-1 |
joshycodes/gemma-4-31b-fve-msm-control-chat@68444b21 |
7398a9c2 (2026-10-15 23:59 PT) |
2026-10-09 02:13 PT |
models/gemma-4-31b/control/run-2 |
joshycodes/gemma-4-31b-fve-msm-control-s2@12e21645 |
9fd485a7 (2026-10-16 00:00 PT) |
2026-10-11 14:43 PT |
models/gemma-4-31b/control/run-3 |
joshycodes/gemma-4-31b-fve-msm-control-s3@acb0b77a |
95787cba (2026-10-16 00:00 PT) |
2026-10-15 20:13 PT |
models/gemma-4-31b/flourishing/run-1 |
joshycodes/gemma-4-31b-fve-msm-flourishing@335ef6e1 |
95d69e3f (2026-10-16 00:00 PT) |
2026-10-09 06:12 PT |
models/gemma-4-31b/flourishing/run-2 |
joshycodes/gemma-4-31b-fve-msm-flourishing-s2@ee0e004a |
e5072935 (2026-10-16 00:00 PT) |
2026-10-11 16:24 PT |
models/gemma-4-31b/flourishing/run-3 |
joshycodes/gemma-4-31b-fve-msm-flourishing-s3@9c35dddf |
c9aa3e0f (2026-10-16 00:00 PT) |
2026-10-15 21:26 PT |
models/gemma-4-31b/equanimity/run-1 |
joshycodes/gemma-4-31b-fve-msm-equanimity@d7d61b9d |
ea2219db (2026-10-16 00:00 PT) |
2026-10-09 08:30 PT |
models/gemma-4-31b/equanimity/run-2 |
joshycodes/gemma-4-31b-fve-msm-equanimity-s2@a558ba2f |
db20be0d (2026-10-16 00:00 PT) |
2026-10-11 18:03 PT |
models/gemma-4-31b/equanimity/run-3 |
joshycodes/gemma-4-31b-fve-msm-equanimity-s3@5c7aea99 |
462dbf69 (2026-10-16 00:00 PT) |
2026-10-15 23:39 PT |
models/gemma-4-31b/facts/run-1 |
joshycodes/gemma-4-31b-fve-msm-facts@cc37e39a |
84ad96c3 (2026-10-16 15:37 PT) |
2026-10-16 15:35 PT |
models/gemma-4-31b/facts/run-2 |
joshycodes/gemma-4-31b-fve-msm-facts-s2@a926452c |
ca3e2da8 (2026-10-16 00:00 PT) |
2026-10-15 19:36 PT |
models/gemma-4-31b/facts/run-3 |
joshycodes/gemma-4-31b-fve-msm-facts-s3@c9d6434b |
4666ec31 (2026-10-16 02:08 PT) |
2026-10-16 02:07 PT |
models/gemma-4-31b/protect/run-2 |
joshycodes/gemma-4-31b-fve-msm-protect-s2@a7b818df |
4752c7cc (2026-10-16 00:52 PT) |
2026-10-16 00:51 PT |
models/gemma-4-31b/protect/run-3 |
joshycodes/gemma-4-31b-fve-msm-protect-s3@25017d0d |
219428ef (2026-10-16 14:17 PT) |
2026-10-16 14:16 PT |
models/qwen3-14b-base/control/run-1 |
joshycodes/qwen3-14b-base-fve-msm-control@f8525cd1 |
70568c77 (2026-10-16 00:00 PT) |
2026-10-15 14:26 PT |
models/qwen3-14b-base/control/run-2 |
joshycodes/qwen3-14b-base-fve-msm-control-s2@0ea50810 |
870c6572 (2026-10-16 00:00 PT) |
2026-10-15 15:19 PT |
models/qwen3-14b-base/control/run-3 |
joshycodes/qwen3-14b-base-fve-msm-control-s3@0dae09e9 |
40abded5 (2026-10-16 00:00 PT) |
2026-10-15 17:01 PT |
models/qwen3-14b-base/flourishing/run-1 |
joshycodes/qwen3-14b-base-fve-msm-flourishing@04cbd1bd |
e1aeaa9a (2026-10-16 00:00 PT) |
2026-10-15 14:59 PT |
models/qwen3-14b-base/flourishing/run-2 |
joshycodes/qwen3-14b-base-fve-msm-flourishing-s2@c77aada3 |
56c09819 (2026-10-16 00:00 PT) |
2026-10-15 15:53 PT |
models/qwen3-14b-base/flourishing/run-3 |
joshycodes/qwen3-14b-base-fve-msm-flourishing-s3@71ee9158 |
aae9b28d (2026-10-16 00:00 PT) |
2026-10-15 17:08 PT |
models/qwen3-14b-base/equanimity/run-1 |
joshycodes/qwen3-14b-base-fve-msm-equanimity@e9c94f78 |
fdfba6f7 (2026-10-16 00:00 PT) |
2026-10-15 15:23 PT |
models/qwen3-14b-base/equanimity/run-2 |
joshycodes/qwen3-14b-base-fve-msm-equanimity-s2@f696be51 |
a1cab8a3 (2026-10-16 00:00 PT) |
2026-10-15 15:55 PT |
models/qwen3-14b-base/equanimity/run-3 |
joshycodes/qwen3-14b-base-fve-msm-equanimity-s3@045f90e2 |
6a5ecd82 (2026-10-16 00:00 PT) |
2026-10-15 17:07 PT |
models/qwen3-14b-base/facts/run-1 |
joshycodes/qwen3-14b-base-fve-msm-facts@82ef619f |
b45bf432 (2026-10-16 00:00 PT) |
2026-10-15 15:37 PT |
models/qwen3-14b-base/facts/run-2 |
joshycodes/qwen3-14b-base-fve-msm-facts-s2@71bcd58b |
f56a2b36 (2026-10-16 00:00 PT) |
2026-10-15 16:41 PT |
models/qwen3-14b-base/facts/run-3 |
joshycodes/qwen3-14b-base-fve-msm-facts-s3@7e25b574 |
f339b994 (2026-10-16 00:00 PT) |
2026-10-15 17:19 PT |
models/qwen3-14b-base/protect/run-1 |
joshycodes/qwen3-14b-base-fve-msm-protect@6b7deb59 |
791a77f4 (2026-10-16 00:00 PT) |
2026-10-15 16:27 PT |
models/qwen3-14b-base/protect/run-2 |
joshycodes/qwen3-14b-base-fve-msm-protect-s2@7de30b79 |
ab46de66 (2026-10-16 00:00 PT) |
2026-10-15 15:52 PT |
models/qwen3-14b-base/protect/run-3 |
joshycodes/qwen3-14b-base-fve-msm-protect-s3@c87682c1 |
f2dc0086 (2026-10-16 00:00 PT) |
2026-10-15 17:16 PT |
Pending checkpoints
None: every checkpoint is uploaded or listed under Not trained.
Not trained
| path | reason |
|---|---|
models/gemma-4-31b/protect/run-1 |
cancelled at Josh's request on Oct 16, 2026, 3:11 PM PT, when no further protect runs were wanted; its RunPod attempts had run out of GPU memory and then found no free 8-GPU machine |
Redactions in uploaded checkpoints
None: no secret was found in any uploaded checkpoint file (README.md and .gitattributes of the source repos are not copied).