| language: en | |
| tags: | |
| - text-generation | |
| - pytorch | |
| - nanogpt | |
| - tinystories | |
| # nanoGPT — TinyStories | |
| A ~30M-parameter GPT trained from scratch on the [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) dataset. | |
| - Tokenizer: GPT-2 BPE (`tiktoken`) | |
| - Architecture: 6 layers, 6 heads, 384 embedding dim, context 256 | |
| - Best val loss: 1.7052 | |
| ## Load | |
| ```python | |
| import torch, json | |
| import tiktoken | |
| config = json.load(open('model_config.json')) | |
| ckpt = torch.load('ckpt.pt', map_location='cpu') | |
| # Re-instantiate GPT with the same config, then load state dict. | |
| enc = tiktoken.get_encoding('gpt2') | |
| ``` | |