| --- |
| license: apache-2.0 |
| library_name: pytorch |
| tags: |
| - polish |
| - checkpoints |
| - training-history |
| --- |
| |
| # Ablation checkpoints (~180M parameters, 2.5B-token budget) |
|
|
| Checkpoints from architecture and optimizer experiments: four finished runs (final checkpoints) and four runs |
| interrupted mid-training (intermediate checkpoints). Each run is compared with a common reference recipe |
| (16 layers × d_model 896 × 14 heads, RMSNorm, QK-norm, RoPE θ 500 000, value residual, SwiGLU, no biases, |
| Muon lr 0.02 + AdamW lr 6e-4, warmup 500, constant lr, decay over the last 20 % to lr / 10; batch 128 × 4096 tokens; |
| planned 4768 steps = 2.5B tokens; tokenizer: the 32 768-token tokenizer of Slayer1B). VR0 and MU2 change one setting; K12, K22 and GQA change the shape |
| at a similar parameter count (K12 and K22: depth and width; GQA: KV-head count and depth, 18 rather than 16 layers). |
| |
| ## Finished runs (final checkpoint, step 4768) |
| |
| | run | change from the reference | final checkpoint | val bpb PL | val bpb EN | weighted bpb (0.4 PL + 0.6 EN) | |
| |---|---|---|---:|---:|---:| |
| | R1 | none (reference), seed 1 | `R1/ckpt_4768.pt` | 1.086412 | 0.999300 | 1.034145 | |
| | R2 | none (reference), seed 2 | `R2/ckpt_4768.pt` | 1.088138 | 1.000682 | 1.035664 | |
| | R3 | none (reference), seed 3 | `R3/ckpt_4768.pt` | 1.086788 | 1.000410 | 1.034961 | |
| | K22 | 22 layers × d_model 768 × 12 heads (deeper, narrower) | `K22/ckpt_4768.pt` | 1.091207 | 1.003861 | 1.038799 | |
|
|
| Bits per byte on held-out validation text (Polish and English), measured once after the last step. The three reference |
| seeds give mean 1.034923; K22 is 0.003876 higher (worse), beyond the decision threshold of 0.00215 fixed before the |
| runs. |
|
|
| ## Interrupted runs |
|
|
| | run | change from the reference | stopped at step | checkpoints here | |
| |---|---|---:|---| |
| | K12 | 12 layers × d_model 1024 × 16 heads (shallower, wider) | 3060 | 500, 1000, 1500, 2000, 2500, 3000 | |
| | GQA | 18 layers × d_model 896 × 14 heads, 2 KV heads (grouped-query attention) | 2920 | 500, 1000, 1500, 2000, 2500 | |
| | VR0 | 16 × 896 × 14 without value residual | 2860 | 500, 1000, 1500, 2000, 2500 | |
| | MU2 | 16 × 896 × 14, Muon learning rate × 2 (0.04) | 2340 | 500, 1000, 1500, 2000 | |
|
|
| **These runs did not finish.** They were stopped externally on 2026-10-23 at 10:11 UTC, before the learning-rate decay; |
| no final evaluation exists for them. The checkpoints are published so that the runs can be resumed or inspected. |
| Each file holds weights and optimizer state (`torch.save` dictionary of `train_gpt_ref_r23.py`: `model`, `opt`, `step`, |
| `config`); load with `torch.load(path, map_location="cpu", weights_only=True)`. |
|
|
| `test_wznowienia/`: checkpoints of a 40-step resume test of the trainer (continuous run vs. resumed run), not a model. |
|
|
| ## SHA-256 |
|
|
| Finished runs — recorded off the machine before the copy (in the result file of each run); files on this repository |
| are downloaded back and checked against them: |
|
|
| ``` |
| c2999a7b90a8fd5a27b23e44e05c793b7359196f667d8118071fe51ae4d43df1 R1/ckpt_4768.pt |
| f4a06ade988a35bfa2d47f686253907400eff9eacd62c91818457fd863d48394 R2/ckpt_4768.pt |
| ea6748f1bcd72a38d3707bc33ef319125fe9c5e9d273e2d424d733dbe32d4564 R3/ckpt_4768.pt |
| 0840bf3078d0f84fc20e68337cc24e8a9e4bba145f7ea0d27541792c1dd7f189 K22/ckpt_4768.pt |
| ``` |
|
|
| Interrupted runs — recorded by the checkpoint copier on the training machine (files dated before the interruption). These sums were stored |
| on the same machine; no independent copy of them exists. Every file listed was downloaded back and matches them. |
|
|
| ``` |
| 846c7cc9f3c88fcda1394e757dd024eebfefc3b43cba549c4dfaa0c0efeb1594 K12/ckpt_500.pt |
| 84f431e41c1a7dcd3cb3a1eda8a572094350a7dbd998669bca640e036c9fd923 K12/ckpt_1000.pt |
| d3bd736d8c42dbd52568c0598ad52862dbddbf69829cbf6325faa3bf0a4083cc K12/ckpt_1500.pt |
| cdc444a304c2f57727937a5ca0f58599d220eb0741a3920ddf73f3965e2117ca K12/ckpt_2000.pt |
| 8f5298ada65ab57bfe1b63dcd49b956cf0f37f26e7ec5e2bab072e2e8da21e84 K12/ckpt_2500.pt |
| c54e324a6d5e2b9e33bf9731ff4162df18fb774e57e7bab253576f096dd24511 K12/ckpt_3000.pt |
| 8296514387f2f1fbc20f5c61822007daa823f86b541dc64e830b1361358fba6a GQA/ckpt_500.pt |
| fb4a4a805f6e88204a3a859238567ff28f40c9f0adbf5a8fdf6c07119245dfec GQA/ckpt_1000.pt |
| 13ddaf2ea734dd720aaa3c55807621d3b66f44ce4d1a58d782970491fafc8699 GQA/ckpt_1500.pt |
| e8e632635f4bca6f8d4885502d46503c8ac16f2e83d5d1aa65129535215f828e GQA/ckpt_2000.pt |
| 3366e88771415a893cb14f3d209b7759bde43b7475e594bf8a81a88c68843f2f GQA/ckpt_2500.pt |
| fd93ad481e1ce29f3bf7d2fe7406b8e2b8821a7a8310bf1795a168236181353f VR0/ckpt_500.pt |
| 83c73ec8302a95a1560a6c1777d3a03786c4f0ae69071babd3774c5dd129981b VR0/ckpt_1000.pt |
| c5678f18d42e9e6f62adaaa6b2e309fadf470d2a7dc4e614217c00e155a86840 VR0/ckpt_1500.pt |
| ee399f392ba9fe44419a840a75060eafa26d42ce0227d2538ec47ecf353a0ec0 VR0/ckpt_2000.pt |
| 09ffda1f09c8b0e3a86b350bbbe59afb2b352a3246047bd3ba5f3a544a5cd918 VR0/ckpt_2500.pt |
| dafb39311a0abbd298efc00372e8d574ef255c2e254dd964f99f92df7ad5116e MU2/ckpt_500.pt |
| d1c16074f435fed71a47e0c377bc3332344ad5bc29c4dfef86dee55c66a7f396 MU2/ckpt_1000.pt |
| edf8bdeae17f21b10e59fe5f56e793a3bfe92a5a28da86d1d8eaf21d8fad6602 MU2/ckpt_1500.pt |
| 72507b5463e7716781fe783b1599cac8dd2fcb9c10823d47aada75c2bf5b7248 MU2/ckpt_2000.pt |
| ``` |
|
|
| Resume-test files (`test_wznowienia/`) — sums from the copier's lists and the test's result files on the training |
| machine (dated before the interruption): |
|
|
| ``` |
| 1c501300ddaaee8bdc063e42db798a394087f6169a3e84446c5346474cbb6136 test_wznowienia/mini_R1/ckpt_20.pt |
| 9bc8d9445b7c2df14b8a4bf34a90e35095d8e7b31c8b27f18b878b563a87a75e test_wznowienia/A/ckpt_20.pt |
| f5877f84b95282b73d3b112299f14cd79aa28ca7a18d070ef91ee872eefb7d9c test_wznowienia/A/ckpt_40.pt |
| e41bdc94415849333814f1ae6a9b6a879c0695f2dadc18130c52830e595fce98 test_wznowienia/B/ckpt_40.pt |
| ``` |
|
|
| ## License |
|
|
| Apache-2.0. |
|
|
