Card: training complete at step 22327; evaluations after 19000 pending; loss chart to 22320
Browse files- README.md +6 -5
- slayer1b_107_training.png +2 -2
README.md
CHANGED
|
@@ -14,15 +14,16 @@ datasets:
|
|
| 14 |
- SlayerLab/slayer-1.07-pretrain
|
| 15 |
---
|
| 16 |
|
| 17 |
-
# Slayer1B 1.07
|
| 18 |
|
| 19 |
Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. **Slayer1B 1.07 continues pretraining of
|
| 20 |
Slayer1B 1.06** from its final checkpoint (step 13848) on 8 891 005 707 training tokens (about 7.0 B new and 1.9 B
|
| 21 |
replayed from the 1.06 data), at the same constant learning rate and global batch.
|
| 22 |
|
| 23 |
-
**Status: training
|
| 24 |
-
|
| 25 |
-
|
|
|
|
| 26 |
ready-to-use model) are published file by file in
|
| 27 |
[SlayerLab/Slayer1B-1.07-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.07-checkpoints), with sha256 sums.
|
| 28 |
|
|
@@ -43,7 +44,7 @@ ready-to-use model) are published file by file in
|
|
| 43 |
= 1 048 576 tokens per step, constant learning rate 6e-4, no learning-rate decay, no context extension.
|
| 44 |
- Hardware: 4× NVIDIA RTX 4090 (gradient accumulation 32, so the global batch is unchanged from the 8-GPU runs),
|
| 45 |
about 45 000 tokens/s. Trainer `train_gpt_ref` r24.2 (sha256 `46f83b51…`), unchanged since 1.05.
|
| 46 |
-
-
|
| 47 |
- Training was interrupted on 4 November at step 17130 by an external stop of the process and resumed on 5 November
|
| 48 |
from the saved checkpoint at step 17000. After the resume the loss at step 17010 was identical to the first run
|
| 49 |
(2.3177), so the resumed run reproduces the original one. A second external stop on 5 November at step 19330 was
|
|
|
|
| 14 |
- SlayerLab/slayer-1.07-pretrain
|
| 15 |
---
|
| 16 |
|
| 17 |
+
# Slayer1B 1.07
|
| 18 |
|
| 19 |
Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. **Slayer1B 1.07 continues pretraining of
|
| 20 |
Slayer1B 1.06** from its final checkpoint (step 13848) on 8 891 005 707 training tokens (about 7.0 B new and 1.9 B
|
| 21 |
replayed from the 1.06 data), at the same constant learning rate and global batch.
|
| 22 |
|
| 23 |
+
**Status: training complete at step 22327 (6 November 2026).** The training-time evaluation table below currently
|
| 24 |
+
ends at step 19000; the checkpoints after it (20000, 21000, 21450, 22000, 22327) are being evaluated with the same
|
| 25 |
+
protocol, and each result is added here when it is ready. A 1.07 model release (converted weights) has not been
|
| 26 |
+
published in this repository yet. Native training checkpoints (raw PyTorch training state `ckpt_N.pt`, not a
|
| 27 |
ready-to-use model) are published file by file in
|
| 28 |
[SlayerLab/Slayer1B-1.07-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.07-checkpoints), with sha256 sums.
|
| 29 |
|
|
|
|
| 44 |
= 1 048 576 tokens per step, constant learning rate 6e-4, no learning-rate decay, no context extension.
|
| 45 |
- Hardware: 4× NVIDIA RTX 4090 (gradient accumulation 32, so the global batch is unchanged from the 8-GPU runs),
|
| 46 |
about 45 000 tokens/s. Trainer `train_gpt_ref` r24.2 (sha256 `46f83b51…`), unchanged since 1.05.
|
| 47 |
+
- Length: steps 13848 → 22327 (8 479 steps, one pass over the new data; each window is used once).
|
| 48 |
- Training was interrupted on 4 November at step 17130 by an external stop of the process and resumed on 5 November
|
| 49 |
from the saved checkpoint at step 17000. After the resume the loss at step 17010 was identical to the first run
|
| 50 |
(2.3177), so the resumed run reproduces the original one. A second external stop on 5 November at step 19330 was
|
slayer1b_107_training.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|