---
license: apache-2.0
language:
- en
tags:
- text-generation
- gpt
- from-scratch
- tinystories
pipeline_tag: text-generation
---

# mini-llm-tinystories

An 18.3M-parameter GPT-2-style language model **trained completely from scratch on CPU**
(no pretrained weights). It generates short, coherent children's-story-style English.

## Details
- **Architecture:** GPT-2 style decoder (pre-norm, GELU, weight-tied head)
- **Params:** ~18.3M — 448 dim, 7 heads, 6 layers, 256 context
- **Tokenizer:** byte-level BPE, 8192 vocab
- **Training data:** TinyStories (~90M tokens) + a small amount of Alpaca Q&A
- **Training:** ~7.6 hours on 2 CPU cores, final train loss ~1.86
- **Type:** base **completion** model (continues text; not instruction-tuned)

## Usage
```python
import torch, json
from safetensors.torch import load_file
from tokenizers import ByteLevelBPETokenizer
from gpt2 import GPT2   # include gpt2.py from this repo

cfg = json.load(open("config.json"))
model = GPT2(cfg)
model.load_state_dict(load_file("model.safetensors"))
model.eval()

tk = ByteLevelBPETokenizer("tokenizer_bpe/vocab.json", "tokenizer_bpe/merges.txt")
ids = tk.encode("Once upon a time").ids
out = model.generate(torch.tensor([ids]), max_new_tokens=150)
print(tk.decode(out[0].tolist()))
```

## Limitations
Tiny model with no real-world knowledge. Best at short narrative completions in a
TinyStories style. Will produce fluent but factually-wrong text if asked questions.