File size: 3,841 Bytes
c6c9cdf 0f4e319 c6c9cdf 0f4e319 c6c9cdf 0f4e319 c6c9cdf 0f4e319 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 | ---
license: apache-2.0
pipeline_tag: text-generation
language: en
tags:
- tiny
- tiny-lm
- tiny-model
- slm
- small-language-model
- from-scratch
- tinystories
- bpe
- gpt
datasets:
- roneneldan/TinyStories
metrics:
- perplexity
- accuracy
---
# TinyStories-24m
A **24.59M-parameter** BPE language model trained **from scratch** on
[roneneldan/TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories),
producing coherent short stories with proper dialogue, names, punctuation and
narrative flow.
## What it is
- **Architecture:** decoder-only GPT, weight-tied embeddings, RMSNorm, fused
qkv multi-head causal attention (SDPA), GELU FFN.
- **Shape:** D=384, L=12 layers, H=8 heads, FFN=1536, SEQ=512, vocab=8192 (BPE).
- **Params:** 24,585,600 (verified against the safetensors header).
- **Data:** roneneldan/TinyStories — 447.8M train tokens, 2M held-out val.
- **Training:** 1 epoch ≈ 13,600 steps, AdamW, cosine LR 6e-4 + 500 warmup,
bf16 autocast, on a single RTX 5090.
## Quality
- **Val perplexity:** 8.76 (2.1618 nats/token, training-time fixed-window score).
Full-split repro (100 random 512-token windows over the 4.5M-token val split):
**12.39** (2.5172 nats/token). The gap is methodology (fixed window vs random
windows), not a card error.
- **Generation:** coherent. Sampled 9/9 seeded generations (3 seeds × 3 prompts)
produce proper dialogue, character names (Ben, Lily, Mom, Tom, Sarah, Max),
punctuation and narrative flow. This model is a story generator for its
training domain — it is **not** a general-purpose assistant and will not
answer questions it was not trained on.
### Benchmark results (zero-shot loglikelihood, 400 examples per task)
| Task | Accuracy | Chance | Notes |
|------|----------|--------|-------|
| ARC-Easy | 13.3% | 25% | below chance |
| ARC-Challenge | 12.5% | 25% | below chance |
| HellaSwag | 25.0% | 25% | at chance |
| SciQ | 25.0% | 25% | at chance |
| PIQA | 50.0% | 50% | at chance |
All results are at or below chance — expected for a 24M model trained exclusively
on simple children's stories. The model has learned the distribution of story
text but has no general reasoning, commonsense, or science knowledge.
## Honest caveats
- **Divergence:** the full 13,600-step run diverged to NaN at step 9,350 (LR 6e-4
is too hot for a 24M model). The **best** checkpoint (step 6,000, val 2.1618)
is what is published here — it is clean and coherent. The divergence is late,
so a clean early checkpoint is the right artifact; always sample the best
checkpoint, not the final one.
- **Domain-bound:** trained only on TinyStories. Out-of-domain text (code,
questions, general conversation) is out of scope.
## Usage
Not a `transformers` model — load with the bundled `modeling.py`:
```python
import sys, torch
sys.path.insert(0, "path/to/this/repo")
from modeling import TinyStoriesGPT
from tokenizers import Tokenizer
m = TinyStoriesGPT.from_pretrained("path/to/this/repo", device="cpu")
tok = Tokenizer.from_file("path/to/this/repo/tokenizer.json")
ids = tok.encode("Ben was playing in the park.", add_special_tokens=False).ids
x = torch.tensor([ids], dtype=torch.long)
with torch.no_grad():
for _ in range(80):
logits = m(x[:, -512:])[:, -1]
nxt = torch.multinomial(torch.softmax(logits / 0.8, -1), 1).item()
ids.append(nxt)
x = torch.tensor([ids[-512:]], dtype=torch.long)
print(tok.decode(ids, skip_special_tokens=True))
```
## Files
| file | bytes | what |
|------|-------|------|
| `model.safetensors` | 98,349,056 | 75 tensors, float32 |
| `config.json` | — | architecture + training metadata |
| `modeling.py` | — | the `TinyStoriesGPT` class (load with `from_pretrained`) |
| `tokenizer.json` | 560,804 | BPE-8k tokenizer (HF `tokenizers` format) | |