---
license: mit
language:
  - en
tags:
  - causal-lm
  - from-scratch
---

# OpenWebText 103M

A 102.7M parameter decoder-only transformer trained from scratch on an
OpenWebText sample, with a custom byte-level BPE tokenizer. No `transformers`,
no pretrained components — every layer written from primitives.

## Results

| | |
|---|---|
| Validation loss | 3.516 (perplexity 33.6) |
| Parameters | 102.7M total, 56.6M non-embedding |
| Training tokens | 819.2M |
| Hardware | 1x A100 80GB, ~3.5 hours |

Comparable in size to GPT-2 small (124M), which reaches ~2.85 on OpenWebText
with substantially more training. This model saw 819M tokens against 102.7M
parameters (~8:1, or ~14.5:1 excluding embeddings) — well under
Chinchilla-optimal, a deliberate tradeoff against a fixed GPU-hour budget.

## Architecture

Pre-norm decoder. RMSNorm, RoPE (theta=10000), SwiGLU FFN, causal multi-head
attention. 8 layers, 12 heads, d_model 768, d_ff 2048, context 512,
vocab 30000. Embeddings are untied and account for 45% of parameters.

Trained with hand-written AdamW, cosine LR schedule with 700-step warmup,
peak LR 6e-4, global-norm gradient clipping at 1.0, TF32 matmuls.

## Usage

Not a `transformers` architecture. Requires `torch`, `einops`, `safetensors`.

```python
import json, torch
from safetensors.torch import load_file
from model import TransformerLM
from tokenizer import Tokenizer

cfg = json.load(open("config.json"))
model = TransformerLM(
    cfg["vocab_size"], cfg["context_length"], cfg["num_layers"],
    cfg["d_model"], cfg["num_heads"], cfg["d_ff"], cfg["theta"],
)
model.load_state_dict(load_file("model.safetensors"))
model.eval()

tok = Tokenizer.from_file("owt_vocab.json", "owt_merges.txt", ["<|endoftext|>"])
```

## Limitations

Learns register convincingly — encyclopedic prose, news style, quoted dialogue
— with no factual grounding. Generates confident historical nonsense. Prone to
repetition loops under low-entropy sampling.

Prompt: `The history of the Roman Empire`

> The history of the Roman Empire, however, is difficult to overstate. The
> Medieval days in the Middle Ages and early in the Middle Ages, under the
> pagan order, were filled with slaves who took advantage of the culture of
> the Christian religion. [...] In 1791, an important Jewish uprising against
> the Christian order was initiated.

Structurally sound encyclopedia prose; every factual claim is invented.