Card: shape runs change more than one setting (similar parameter count)
Browse files
README.md
CHANGED
|
@@ -9,11 +9,12 @@ tags:
|
|
| 9 |
|
| 10 |
# Ablation checkpoints (~180M parameters, 2.5B-token budget)
|
| 11 |
|
| 12 |
-
Checkpoints
|
| 13 |
-
(intermediate checkpoints). Each run
|
| 14 |
(16 layers × d_model 896 × 14 heads, RMSNorm, QK-norm, RoPE θ 500 000, value residual, SwiGLU, no biases,
|
| 15 |
Muon lr 0.02 + AdamW lr 6e-4, warmup 500, constant lr, decay over the last 20 % to lr / 10; batch 128 × 4096 tokens;
|
| 16 |
-
planned 4768 steps = 2.5B tokens; tokenizer: the 32 768-token tokenizer of Slayer1B).
|
|
|
|
| 17 |
|
| 18 |
## Finished runs (final checkpoint, step 4768)
|
| 19 |
|
|
|
|
| 9 |
|
| 10 |
# Ablation checkpoints (~180M parameters, 2.5B-token budget)
|
| 11 |
|
| 12 |
+
Checkpoints from architecture and optimizer experiments: four finished runs (final checkpoints) and four runs
|
| 13 |
+
interrupted mid-training (intermediate checkpoints). Each run is compared with a common reference recipe
|
| 14 |
(16 layers × d_model 896 × 14 heads, RMSNorm, QK-norm, RoPE θ 500 000, value residual, SwiGLU, no biases,
|
| 15 |
Muon lr 0.02 + AdamW lr 6e-4, warmup 500, constant lr, decay over the last 20 % to lr / 10; batch 128 × 4096 tokens;
|
| 16 |
+
planned 4768 steps = 2.5B tokens; tokenizer: the 32 768-token tokenizer of Slayer1B). VR0 and MU2 change one setting; K12, K22 and GQA change the shape
|
| 17 |
+
at a similar parameter count (K12 and K22: depth and width; GQA: KV-head count and depth, 18 rather than 16 layers).
|
| 18 |
|
| 19 |
## Finished runs (final checkpoint, step 4768)
|
| 20 |
|