Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -27,11 +27,11 @@ Training corpus: [fdeantoni/max-babbelaar-corpus](https://huggingface.co/dataset
|
|
| 27 |
This repo holds SFT checkpoints for multiple model depths. Each tag (`d18`, `d24`, …)
|
| 28 |
lives under `chatsft_checkpoints/<tag>/` and shares a single tokenizer.
|
| 29 |
|
| 30 |
-
## Latest upload: `d18` at step
|
| 31 |
|
| 32 |
| Depth | Step | Layers | d_model | Heads (Q/KV) | Vocab | Context |
|
| 33 |
|-------|------|--------|---------|--------------|-------|---------|
|
| 34 |
-
| `d18` |
|
| 35 |
|
| 36 |
Architecture: GPT with RoPE, QK-norm, GQA, relu² MLP, sliding-window pattern `SSSL`,
|
| 37 |
value embeddings (ResFormer-style), smear gate, and backout residual. Trained with the
|
|
|
|
| 27 |
This repo holds SFT checkpoints for multiple model depths. Each tag (`d18`, `d24`, …)
|
| 28 |
lives under `chatsft_checkpoints/<tag>/` and shares a single tokenizer.
|
| 29 |
|
| 30 |
+
## Latest upload: `d18` at step 1750 (val_bpb 0.2652)
|
| 31 |
|
| 32 |
| Depth | Step | Layers | d_model | Heads (Q/KV) | Vocab | Context |
|
| 33 |
|-------|------|--------|---------|--------------|-------|---------|
|
| 34 |
+
| `d18` | 1750 | 18 | 1152 | 9/9 | 32768 | 2048 |
|
| 35 |
|
| 36 |
Architecture: GPT with RoPE, QK-norm, GQA, relu² MLP, sliding-window pattern `SSSL`,
|
| 37 |
value embeddings (ResFormer-style), smear gate, and backout residual. Trained with the
|