| --- |
| license: mit |
| language: |
| - en |
| tags: |
| - causal-lm |
| - from-scratch |
| --- |
| |
| # OpenWebText 103M |
|
|
| A 102.7M parameter decoder-only transformer trained from scratch on an |
| OpenWebText sample, with a custom byte-level BPE tokenizer. No `transformers`, |
| no pretrained components — every layer written from primitives. |
|
|
| ## Results |
|
|
| | | | |
| |---|---| |
| | Validation loss | 3.516 (perplexity 33.6) | |
| | Parameters | 102.7M total, 56.6M non-embedding | |
| | Training tokens | 819.2M | |
| | Hardware | 1x A100 80GB, ~3.5 hours | |
|
|
| Comparable in size to GPT-2 small (124M), which reaches ~2.85 on OpenWebText |
| with substantially more training. This model saw 819M tokens against 102.7M |
| parameters (~8:1, or ~14.5:1 excluding embeddings) — well under |
| Chinchilla-optimal, a deliberate tradeoff against a fixed GPU-hour budget. |
|
|
| ## Architecture |
|
|
| Pre-norm decoder. RMSNorm, RoPE (theta=10000), SwiGLU FFN, causal multi-head |
| attention. 8 layers, 12 heads, d_model 768, d_ff 2048, context 512, |
| vocab 30000. Embeddings are untied and account for 45% of parameters. |
|
|
| Trained with hand-written AdamW, cosine LR schedule with 700-step warmup, |
| peak LR 6e-4, global-norm gradient clipping at 1.0, TF32 matmuls. |
|
|
| ## Usage |
|
|
| Not a `transformers` architecture. Requires `torch`, `einops`, `safetensors`. |
|
|
| ```python |
| import json, torch |
| from safetensors.torch import load_file |
| from model import TransformerLM |
| from tokenizer import Tokenizer |
| |
| cfg = json.load(open("config.json")) |
| model = TransformerLM( |
| cfg["vocab_size"], cfg["context_length"], cfg["num_layers"], |
| cfg["d_model"], cfg["num_heads"], cfg["d_ff"], cfg["theta"], |
| ) |
| model.load_state_dict(load_file("model.safetensors")) |
| model.eval() |
| |
| tok = Tokenizer.from_file("owt_vocab.json", "owt_merges.txt", ["<|endoftext|>"]) |
| ``` |
|
|
| ## Limitations |
|
|
| Learns register convincingly — encyclopedic prose, news style, quoted dialogue |
| — with no factual grounding. Generates confident historical nonsense. Prone to |
| repetition loops under low-entropy sampling. |
|
|
| Prompt: `The history of the Roman Empire` |
|
|
| > The history of the Roman Empire, however, is difficult to overstate. The |
| > Medieval days in the Middle Ages and early in the Middle Ages, under the |
| > pagan order, were filled with slaves who took advantage of the culture of |
| > the Christian religion. [...] In 1791, an important Jewish uprising against |
| > the Christian order was initiated. |
|
|
| Structurally sound encyclopedia prose; every factual claim is invented. |
|
|
