File size: 1,080 Bytes
aa5933d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | ---
tags:
- pytorch
- language-model
- gpt
license: mit
---
# vanilla-v5
Vanilla GPT baseline trained to compare against [aemack-org/cayley-10b](https://huggingface.co/aemack-org/cayley-10b).
## Architecture
| Parameter | Value |
|-----------|-------|
| n_layer | 12 |
| n_head | 8 |
| n_embd | 1024 |
| block_size | 1024 |
| vocab_size | 50304 |
| bias | False |
| norm | RMSNorm (affine) |
| MLP | GELU, 4x expansion |
| tokenizer | GPT-2 (tiktoken) |
| dtype | bfloat16 |
| sparsity | none (vanilla) |
## Training
| Parameter | Value |
|-----------|-------|
| optimizer | Muon (hidden 2D) + AdamW (embeddings) |
| muon_lr | 0.01 |
| adamw_lr | 0.01 |
| lr_schedule | linear_warmdown (warmdown_frac=0.4) |
| batch_size | 80 |
| seq_len | 1024 |
| max_iters | 16000 |
| tokens seen | ~1.3B |
| dataset | FineWeb-Edu-10B |
| best_val_loss | 2.9109 |
## Purpose
Interpretability comparison baseline. Trained with identical hyperparameters to
`cayley-10b` but without the CayleySAE bottleneck at `mlp_in`. Enables direct
comparison of residual stream representations.
|