GPT-2 124M — FineWeb Baseline (transformer-room)
Decoder-only language model trained from scratch on FineWeb (sample-10BT subset).
GPT-2 scale (~123.7M parameters), pre-norm, SDPA attention, weight-tied embeddings.
Part of the transformer-room project — public training artifact for Phase 1 / Stage 1 baseline.
Model Architecture
| Property | Value |
|---|---|
| Model type | baseline_decoder (custom, not a transformers.PreTrainedModel) |
| Parameters | ~123.7M |
| Hidden size | 768 |
| Attention heads | 12 |
| Layers | 12 |
| Max sequence length | 1024 |
| Vocabulary size | 50258 (GPT-2 base + <PAD> token) |
| Norm placement | pre (pre-norm — no LR warmup required) |
| Attention | Scaled Dot-Product Attention (PyTorch SDPA) |
| Weight tying | True (input embedding = output projection) |
| Positional encoding | Sinusoidal |
| FFN activation | ReLU |
Training Details
| Property | Value |
|---|---|
| Dataset | HuggingFaceFW/fineweb (sample-10BT) |
| Validation set | CC-MAIN-2024-10 (held-out FineWeb CC dump, no train overlap) |
| Token budget | ~2.5B (Chinchilla-optimal for 124M: 20 tokens/param) |
| Optimizer | AdamW (lr=0.0006, weight_decay=0.1) |
| LR schedule | Cosine decay to 0.1× (no warmup — pre-norm stable) |
| Effective batch size | 512 seqs × 1024 tokens = 524,288 tokens/step |
| Micro batch / accum | 64 seqs / 8 gradient accumulation steps |
| Hardware | H100 80GB |
| Precision | bf16 autocast (PyTorch AMP) |
| torch.compile | Yes (default mode, activation memory budget 0.75) |
Results
| Metric | Smoke run (1k steps) | Full run (~2.5B tokens) |
|---|---|---|
| Train loss (final step) | N/A | TBD |
| Train bits-per-byte | N/A | TBD |
Val loss (fineweb-cc-2024-10) |
N/A | TBD |
| Val bits-per-byte | N/A | TBD |
| HellaSwag (0-shot) | TBD | TBD |
| ARC-Easy (0-shot) | TBD | TBD |
HellaSwag / ARC evaluations via lm-evaluation-harness — planned in ASH-21.
W&B Training Logs
| Run | Link |
|---|---|
| Smoke run (1k steps, B=512) | N/A |
| Full run (~2.5B tokens) | TBD — training in progress |
Loading the Weights
This model is not a transformers.PreTrainedModel and cannot be loaded with AutoModelForCausalLM.from_pretrained().
from safetensors.torch import load_file
from transformers import AutoTokenizer
# Tokenizer (GPT-2 base + <PAD> token, vocab_size=50258)
tokenizer = AutoTokenizer.from_pretrained("Ashwin7/gpt2-124m-fineweb-baseline")
# Weights
state_dict = load_file("model.safetensors") # or hf_hub_download("Ashwin7/gpt2-124m-fineweb-baseline", "model.safetensors")
To reconstruct the model, instantiate BaselineModel from transformer-room/src/components/models/baseline_model.py using the fields in config.json, then call model.load_state_dict(state_dict).
License
Apache 2.0
- Downloads last month
- 4