Text Generation
PyTorch
Safetensors
Polish
English
polish
causal-lm
from-scratch
Maggio33 commited on
Commit
9134eb9
·
verified ·
1 Parent(s): a48e6ef

Card: training complete at step 22327; evaluations after 19000 pending; loss chart to 22320

Browse files
Files changed (2) hide show
  1. README.md +6 -5
  2. slayer1b_107_training.png +2 -2
README.md CHANGED
@@ -14,15 +14,16 @@ datasets:
14
  - SlayerLab/slayer-1.07-pretrain
15
  ---
16
 
17
- # Slayer1B 1.07 (training in progress)
18
 
19
  Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. **Slayer1B 1.07 continues pretraining of
20
  Slayer1B 1.06** from its final checkpoint (step 13848) on 8 891 005 707 training tokens (about 7.0 B new and 1.9 B
21
  replayed from the 1.06 data), at the same constant learning rate and global batch.
22
 
23
- **Status: training in progress.** The planned end is step 22327. This card reports intermediate training-time
24
- evaluations up to step 19000 and will be replaced by the final card when training ends. No 1.07 weights are
25
- published in this repository yet. Intermediate training checkpoints (raw PyTorch training state `ckpt_N.pt`, not a
 
26
  ready-to-use model) are published file by file in
27
  [SlayerLab/Slayer1B-1.07-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.07-checkpoints), with sha256 sums.
28
 
@@ -43,7 +44,7 @@ ready-to-use model) are published file by file in
43
  = 1 048 576 tokens per step, constant learning rate 6e-4, no learning-rate decay, no context extension.
44
  - Hardware: 4× NVIDIA RTX 4090 (gradient accumulation 32, so the global batch is unchanged from the 8-GPU runs),
45
  about 45 000 tokens/s. Trainer `train_gpt_ref` r24.2 (sha256 `46f83b51…`), unchanged since 1.05.
46
- - Planned length: steps 13848 → 22327 (8 479 steps, one pass over the new data; each window is used once).
47
  - Training was interrupted on 4 November at step 17130 by an external stop of the process and resumed on 5 November
48
  from the saved checkpoint at step 17000. After the resume the loss at step 17010 was identical to the first run
49
  (2.3177), so the resumed run reproduces the original one. A second external stop on 5 November at step 19330 was
 
14
  - SlayerLab/slayer-1.07-pretrain
15
  ---
16
 
17
+ # Slayer1B 1.07
18
 
19
  Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. **Slayer1B 1.07 continues pretraining of
20
  Slayer1B 1.06** from its final checkpoint (step 13848) on 8 891 005 707 training tokens (about 7.0 B new and 1.9 B
21
  replayed from the 1.06 data), at the same constant learning rate and global batch.
22
 
23
+ **Status: training complete at step 22327 (6 November 2026).** The training-time evaluation table below currently
24
+ ends at step 19000; the checkpoints after it (20000, 21000, 21450, 22000, 22327) are being evaluated with the same
25
+ protocol, and each result is added here when it is ready. A 1.07 model release (converted weights) has not been
26
+ published in this repository yet. Native training checkpoints (raw PyTorch training state `ckpt_N.pt`, not a
27
  ready-to-use model) are published file by file in
28
  [SlayerLab/Slayer1B-1.07-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.07-checkpoints), with sha256 sums.
29
 
 
44
  = 1 048 576 tokens per step, constant learning rate 6e-4, no learning-rate decay, no context extension.
45
  - Hardware: 4× NVIDIA RTX 4090 (gradient accumulation 32, so the global batch is unchanged from the 8-GPU runs),
46
  about 45 000 tokens/s. Trainer `train_gpt_ref` r24.2 (sha256 `46f83b51…`), unchanged since 1.05.
47
+ - Length: steps 13848 → 22327 (8 479 steps, one pass over the new data; each window is used once).
48
  - Training was interrupted on 4 November at step 17130 by an external stop of the process and resumed on 5 November
49
  from the saved checkpoint at step 17000. After the resume the loss at step 17010 was identical to the first run
50
  (2.3177), so the resumed run reproduces the original one. A second external stop on 5 November at step 19330 was
slayer1b_107_training.png CHANGED

Git LFS Details

  • SHA256: 023376dff09611177c94a398437a6905fdb75a1c7e30bfe16bc03b00d56f24ce
  • Pointer size: 131 Bytes
  • Size of remote file: 188 kB

Git LFS Details

  • SHA256: b809f9e1896b9c0fefe09765c36db28a2d99b6bcf9c6cd9a8c485d2fe3c7a02a
  • Pointer size: 131 Bytes
  • Size of remote file: 181 kB