Maggio33 commited on
Commit
76a097d
·
verified ·
1 Parent(s): 147b6ed

Card: shape runs change more than one setting (similar parameter count)

Browse files
Files changed (1) hide show
  1. README.md +4 -3
README.md CHANGED
@@ -9,11 +9,12 @@ tags:
9
 
10
  # Ablation checkpoints (~180M parameters, 2.5B-token budget)
11
 
12
- Checkpoints of single-factor ablation runs: four finished runs (final checkpoints) and four runs interrupted mid-training
13
- (intermediate checkpoints). Each run changes **one** factor of a common reference recipe
14
  (16 layers × d_model 896 × 14 heads, RMSNorm, QK-norm, RoPE θ 500 000, value residual, SwiGLU, no biases,
15
  Muon lr 0.02 + AdamW lr 6e-4, warmup 500, constant lr, decay over the last 20 % to lr / 10; batch 128 × 4096 tokens;
16
- planned 4768 steps = 2.5B tokens; tokenizer: the 32 768-token tokenizer of Slayer1B).
 
17
 
18
  ## Finished runs (final checkpoint, step 4768)
19
 
 
9
 
10
  # Ablation checkpoints (~180M parameters, 2.5B-token budget)
11
 
12
+ Checkpoints from architecture and optimizer experiments: four finished runs (final checkpoints) and four runs
13
+ interrupted mid-training (intermediate checkpoints). Each run is compared with a common reference recipe
14
  (16 layers × d_model 896 × 14 heads, RMSNorm, QK-norm, RoPE θ 500 000, value residual, SwiGLU, no biases,
15
  Muon lr 0.02 + AdamW lr 6e-4, warmup 500, constant lr, decay over the last 20 % to lr / 10; batch 128 × 4096 tokens;
16
+ planned 4768 steps = 2.5B tokens; tokenizer: the 32 768-token tokenizer of Slayer1B). VR0 and MU2 change one setting; K12, K22 and GQA change the shape
17
+ at a similar parameter count (K12 and K22: depth and width; GQA: KV-head count and depth, 18 rather than 16 layers).
18
 
19
  ## Finished runs (final checkpoint, step 4768)
20