metadata
language: en
tags:
- tiny-stories
- character-level
- minigpt
- pytorch
license: mit
datasets:
- roneneldan/TinyStories
metrics:
- perplexity
MiniGPT - TinyStories
A ~5M parameter character-level GPT model trained from scratch on TinyStories.
Model Details
| Parameter | Value |
|---|---|
| Parameters | ~4.9M |
| Vocab Size | 72 (character-level) |
| Embedding Dim | 256 |
| Layers | 6 |
| Attention Heads | 8 |
| Context Length | 128 |
| Training Steps | 3,000 |
| Final Val Loss | 0.9235 |
Architecture
Standard mini-GPT architecture:
- Token Embedding + Position Embedding
- 6 Transformer Blocks (Causal Self-Attention + MLP)
- LayerNorm + Residual Connections
- Weight Tying (input embedding = output projection)
Training
Trained on CPU for ~1 hour 17 minutes on the TinyStories dataset.
- Optimizer: AdamW (lr=3e-4, weight_decay=0.01)
- Schedule: Warmup (200 steps) + Cosine Decay
- Batch Size: 64, Block Size: 128
Usage
import torch
from train_torch import MiniGPT
# Load model
model = MiniGPT(vocab_size=72, n_embd=256, n_head=8, n_layer=6, block_size=128, dropout=0.1)
model.load_state_dict(torch.load("model.safetensors"))
model.eval()
# Generate
context = torch.zeros((1, 1), dtype=torch.long)
output = model.generate(context, max_new_tokens=200, temperature=0.7)
Sample Generation (Temperature=0.7)
He saw a bike with a big box. He was a girl named Sue. Sue liked to give it very happy. One day, she got to the little boy named Tim. Tim was very happy. He wind the best friends was friends. The tree...
License
MIT