File size: 2,433 Bytes
507f954 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 | ---
language: en
license: mit
tags:
- micro-gpt
- tinystories
- character-level
- pytorch
- transformer
- from-scratch
datasets:
- roneneldan/TinyStories
library_name: pytorch
---
# MicroGPT - Tinystories
A minimal character-level GPT model trained from scratch on the **tinystories** dataset.
## Model Details
| Property | Value |
|----------|-------|
| Architecture | MicroGPT (decoder-only transformer) |
| Parameters | 419,712 |
| Vocab Size | 74 (character-level) |
| Block Size | 128 tokens |
| Layers | 2 |
| Attention Heads | 4 |
| Embedding Dim | 128 |
| Training Steps | 500 |
| Best Val Loss | 2.2494213581085205 |
| Dataset | tinystories |
## Quick Start
```bash
# Clone the repo
git clone https://huggingface.co/{{cookiecutter.repo_id if cookiecutter else 'your-username/micro-gpt-' + config['dataset']}}
cd micro-gpt-tinystories
# Install dependencies
pip install -r requirements.txt
# Generate text
python inference.py --prompt "Once upon a time"
```
## Usage in Python
```python
import torch
from models.micro_gpt import MicroGPT
# Load config
import json
with open('config.json') as f:
config = json.load(f)
# Build model
model = MicroGPT(
vocab_size=config['vocab_size'],
block_size=config['block_size'],
n_layer=config['n_layer'],
n_head=config['n_head'],
n_embd=config['n_embd'],
dropout=config['dropout'],
)
model.load_state_dict(torch.load('pytorch_model.bin', map_location='cpu'))
model.eval()
# Load tokenizer
with open('tokenizer.json') as f:
tokenizer = json.load(f)
# Encode prompt
prompt = "Once upon a time"
indices = [tokenizer['stoi'].get(c, 0) for c in prompt]
input_ids = torch.tensor([indices], dtype=torch.long)
# Generate
output_ids = model.generate(input_ids, max_new_tokens=200, temperature=0.9, top_k=40)
text = ''.join(tokenizer['itos'][str(i)] for i in output_ids[0].tolist())
print(text)
```
## Training
This model was trained using the project's training pipeline:
```bash
python run_model.py --train_model --arch micro_glm --dataset tinystories --max_steps 500
```
## Files in this repository
| File | Description |
|------|-------------|
| `models/` | Full model source code (MicroGPT architecture) |
| `pytorch_model.bin` | Trained model weights |
| `config.json` | Model hyperparameters |
| `tokenizer.json` | Character-level tokenizer |
| `inference.py` | Ready-to-use inference script |
| `requirements.txt` | Python dependencies |
|