| --- |
| language: en |
| tags: |
| - tiny-stories |
| - character-level |
| - minigpt |
| - pytorch |
| license: mit |
| datasets: |
| - roneneldan/TinyStories |
| metrics: |
| - perplexity |
| --- |
| |
| # MiniGPT - TinyStories |
|
|
| A ~5M parameter character-level GPT model trained from scratch on [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories). |
|
|
| ## Model Details |
|
|
| | Parameter | Value | |
| |-----------|-------| |
| | Parameters | ~4.9M | |
| | Vocab Size | 72 (character-level) | |
| | Embedding Dim | 256 | |
| | Layers | 6 | |
| | Attention Heads | 8 | |
| | Context Length | 128 | |
| | Training Steps | 3,000 | |
| | Final Val Loss | 0.9235 | |
|
|
| ## Architecture |
|
|
| Standard mini-GPT architecture: |
| - Token Embedding + Position Embedding |
| - 6 Transformer Blocks (Causal Self-Attention + MLP) |
| - LayerNorm + Residual Connections |
| - Weight Tying (input embedding = output projection) |
|
|
| ## Training |
|
|
| Trained on CPU for ~1 hour 17 minutes on the TinyStories dataset. |
| - Optimizer: AdamW (lr=3e-4, weight_decay=0.01) |
| - Schedule: Warmup (200 steps) + Cosine Decay |
| - Batch Size: 64, Block Size: 128 |
| |
| ## Usage |
| |
| ```python |
| import torch |
| from train_torch import MiniGPT |
|
|
| # Load model |
| model = MiniGPT(vocab_size=72, n_embd=256, n_head=8, n_layer=6, block_size=128, dropout=0.1) |
| model.load_state_dict(torch.load("model.safetensors")) |
| model.eval() |
| |
| # Generate |
| context = torch.zeros((1, 1), dtype=torch.long) |
| output = model.generate(context, max_new_tokens=200, temperature=0.7) |
| ``` |
| |
| ## Sample Generation (Temperature=0.7) |
| |
| > He saw a bike with a big box. He was a girl named Sue. Sue liked to give it very happy. One day, she got to the little boy named Tim. Tim was very happy. He wind the best friends was friends. The tree... |
| |
| ## License |
| |
| MIT |
| |
