Maggio33 commited on
Commit
177fe19
Β·
verified Β·
1 Parent(s): 90d8986

Card: step-3310 loss match stated to four decimal places

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -18,7 +18,7 @@ budget of twice the tokens of 1.01. Each file holds the model weights **and the
18
  |---|---|
19
  | Start | 1.01 `ckpt_2250.pt` (step 2250, constant lr 6e-4) |
20
  | Steps | 5720 in total Γ— 1 048 576 tokens = 5 997 854 720 tokens (β‰ˆ 5.6 tokens per parameter; 1.01: 2860 steps) |
21
- | Schedule | **constant lr 6e-4 to the end (no final decay).** The run started with a planned 1 βˆ’ sqrt decay from step 4578; at step 3300, still in the constant phase, it was resumed with the decay removed (training loss at step 3310 identical before and after the resume). The decay is deferred to the end of the whole Slayer1B line. |
22
  | Data | steps 1–1801: pack-00 stream (as 1.00 and 1.01); 1802–3594: pack-01 stream (as 1.01, continued); 3595–5389: pack-02; 5390–5720: pack-03 β€” all from [SlayerLab/slayer-pl-8x3b](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b) revision `d50df04a`; shared components of each pack disjoint from those already used |
23
  | Checkpoints | every 150 steps from 2400, step 4576 and the final step 5720; every checkpoint is a constant-lr state and can start further continued training |
24
  | Trainer | `train_gpt_ref_r24.py` = the 1.00/1.01 trainer plus a chain of data streams and extra checkpoint steps; batches up to step 3594 are identical to those of 1.01's data order |
 
18
  |---|---|
19
  | Start | 1.01 `ckpt_2250.pt` (step 2250, constant lr 6e-4) |
20
  | Steps | 5720 in total Γ— 1 048 576 tokens = 5 997 854 720 tokens (β‰ˆ 5.6 tokens per parameter; 1.01: 2860 steps) |
21
+ | Schedule | **constant lr 6e-4 to the end (no final decay).** The run started with a planned 1 βˆ’ sqrt decay from step 4578; at step 3300, still in the constant phase, it was resumed with the decay removed (training loss at step 3310 before and after the resume equal to four decimal places). The decay is deferred to the end of the whole Slayer1B line. |
22
  | Data | steps 1–1801: pack-00 stream (as 1.00 and 1.01); 1802–3594: pack-01 stream (as 1.01, continued); 3595–5389: pack-02; 5390–5720: pack-03 β€” all from [SlayerLab/slayer-pl-8x3b](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b) revision `d50df04a`; shared components of each pack disjoint from those already used |
23
  | Checkpoints | every 150 steps from 2400, step 4576 and the final step 5720; every checkpoint is a constant-lr state and can start further continued training |
24
  | Trainer | `train_gpt_ref_r24.py` = the 1.00/1.01 trainer plus a chain of data streams and extra checkpoint steps; batches up to step 3594 are identical to those of 1.01's data order |