Card: available vs planned tokens; no empty sections
Browse files
README.md
CHANGED
|
@@ -48,8 +48,9 @@ mixing weights (defined in cl100k tokens): web 0.63, core 0.19, legal 0.08, Engl
|
|
| 48 |
English is 10 % of the mixture in cl100k tokens and 17.6 % in this tokenizer's tokens: the tokenizer is trained on
|
| 49 |
Polish and splits English into more pieces.
|
| 50 |
|
| 51 |
-
|
| 52 |
-
|
|
|
|
| 53 |
|
| 54 |
Personal data: before tokenization, e-mail addresses, phone numbers (Polish and international), PESEL, NIP, REGON and
|
| 55 |
Polish ID-card numbers were replaced with tags `[EMAIL]`, `[TELEFON]`, `[PESEL]`, `[NIP]`, `[REGON]`, `[DOWOD]`
|
|
@@ -63,20 +64,16 @@ removal of every personal datum.
|
|
| 63 |
|---|---|
|
| 64 |
| Hardware | 8 × NVIDIA RTX 5090 (32 GB), DDP |
|
| 65 |
| Batch | 256 × 4096 = 1 048 576 tokens per step |
|
| 66 |
-
| Steps | 1430 |
|
| 67 |
| Optimizer | Muon (lr 0.02) for hidden weight matrices + AdamW (lr 6e-4, weight decay 0.1) for the rest |
|
| 68 |
| Schedule | warmup 150 steps, constant, linear decay over the last 20 % of steps to lr / 10 |
|
| 69 |
| Data order | each 4096-token window of the stream used at most once (epoch sampler), data seed 20261023 |
|
| 70 |
| Precision | bf16 autocast, torch.compile, activation checkpointing |
|
| 71 |
|
| 72 |
-
## Training run
|
| 73 |
-
|
| 74 |
-
## Evaluation
|
| 75 |
-
|
| 76 |
## Limitations
|
| 77 |
|
| 78 |
-
- 1 073 912 851 parameters
|
| 79 |
-
models when reading the scores.
|
| 80 |
- Polish-dominant data; English is a minority of the mixture.
|
| 81 |
- Base model: no instruction tuning, no safety tuning.
|
| 82 |
- Masking of personal data is rule-based (see above).
|
|
|
|
| 48 |
English is 10 % of the mixture in cl100k tokens and 17.6 % in this tokenizer's tokens: the tokenizer is trained on
|
| 49 |
Polish and splits English into more pieces.
|
| 50 |
|
| 51 |
+
Available training stream: 1 889 519 627 tokens (`train.bin`, sha256 `760ef0cc…`). Planned use: 1430 steps × 1 048 576
|
| 52 |
+
= 1 499 463 680 tokens (the number actually consumed is reported from the trainer log when the run ends). Held-out
|
| 53 |
+
validation from the same pack, disjoint by document id: 1 562 647 PL tokens, 1 643 997 EN tokens.
|
| 54 |
|
| 55 |
Personal data: before tokenization, e-mail addresses, phone numbers (Polish and international), PESEL, NIP, REGON and
|
| 56 |
Polish ID-card numbers were replaced with tags `[EMAIL]`, `[TELEFON]`, `[PESEL]`, `[NIP]`, `[REGON]`, `[DOWOD]`
|
|
|
|
| 64 |
|---|---|
|
| 65 |
| Hardware | 8 × NVIDIA RTX 5090 (32 GB), DDP |
|
| 66 |
| Batch | 256 × 4096 = 1 048 576 tokens per step |
|
| 67 |
+
| Steps | 1430 (planned) |
|
| 68 |
| Optimizer | Muon (lr 0.02) for hidden weight matrices + AdamW (lr 6e-4, weight decay 0.1) for the rest |
|
| 69 |
| Schedule | warmup 150 steps, constant, linear decay over the last 20 % of steps to lr / 10 |
|
| 70 |
| Data order | each 4096-token window of the stream used at most once (epoch sampler), data seed 20261023 |
|
| 71 |
| Precision | bf16 autocast, torch.compile, activation checkpointing |
|
| 72 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
## Limitations
|
| 74 |
|
| 75 |
+
- 1 073 912 851 parameters; planned training budget 1 499 463 680 tokens (about 1.4 tokens per parameter). Compare
|
| 76 |
+
with the token counts of other models when reading the scores.
|
| 77 |
- Polish-dominant data; English is a minority of the mixture.
|
| 78 |
- Base model: no instruction tuning, no safety tuning.
|
| 79 |
- Masking of personal data is rule-based (see above).
|