Card: link to intermediate checkpoints repo; second external stop at 19330 resumed from 19250
Browse files
README.md
CHANGED
|
@@ -22,7 +22,9 @@ replayed from the 1.06 data), at the same constant learning rate and global batc
|
|
| 22 |
|
| 23 |
**Status: training in progress.** The planned end is step 22327. This card reports intermediate training-time
|
| 24 |
evaluations up to step 18590 and will be replaced by the final card when training ends. No 1.07 weights are
|
| 25 |
-
published in this repository yet.
|
|
|
|
|
|
|
| 26 |
|
| 27 |
- Previous version: [SlayerLab/Slayer1B-1.06](https://huggingface.co/SlayerLab/Slayer1B-1.06) (step 13848,
|
| 28 |
14 520 680 448 tokens, one full pass over [SlayerLab/slayer-pl-8x3b](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b))
|
|
@@ -44,7 +46,8 @@ published in this repository yet.
|
|
| 44 |
- Planned length: steps 13848 → 22327 (8 479 steps, one pass over the new data; each window is used once).
|
| 45 |
- Training was interrupted on 4 November at step 17130 by an external stop of the process and resumed on 5 November
|
| 46 |
from the saved checkpoint at step 17000. After the resume the loss at step 17010 was identical to the first run
|
| 47 |
-
(2.3177), so the resumed run reproduces the original one.
|
|
|
|
| 48 |
|
| 49 |
## Training data of 1.07
|
| 50 |
|
|
|
|
| 22 |
|
| 23 |
**Status: training in progress.** The planned end is step 22327. This card reports intermediate training-time
|
| 24 |
evaluations up to step 18590 and will be replaced by the final card when training ends. No 1.07 weights are
|
| 25 |
+
published in this repository yet. Intermediate training checkpoints (raw PyTorch training state `ckpt_N.pt`, not a
|
| 26 |
+
ready-to-use model) are published file by file in
|
| 27 |
+
[SlayerLab/Slayer1B-1.07-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.07-checkpoints), with sha256 sums.
|
| 28 |
|
| 29 |
- Previous version: [SlayerLab/Slayer1B-1.06](https://huggingface.co/SlayerLab/Slayer1B-1.06) (step 13848,
|
| 30 |
14 520 680 448 tokens, one full pass over [SlayerLab/slayer-pl-8x3b](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b))
|
|
|
|
| 46 |
- Planned length: steps 13848 → 22327 (8 479 steps, one pass over the new data; each window is used once).
|
| 47 |
- Training was interrupted on 4 November at step 17130 by an external stop of the process and resumed on 5 November
|
| 48 |
from the saved checkpoint at step 17000. After the resume the loss at step 17010 was identical to the first run
|
| 49 |
+
(2.3177), so the resumed run reproduces the original one. A second external stop on 5 November at step 19330 was
|
| 50 |
+
resumed within two minutes from the checkpoint at step 19250; the loss at step 19260 was again identical (2.3105).
|
| 51 |
|
| 52 |
## Training data of 1.07
|
| 53 |
|