---
language: en
license: mit
tags:
- micro-gpt
- tinystories
- character-level
- pytorch
- transformer
- from-scratch
datasets:
- roneneldan/TinyStories
library_name: pytorch
---

# MicroGPT - Tinystories

A minimal character-level GPT model trained from scratch on the **tinystories** dataset.

## Model Details

| Property | Value |
|----------|-------|
| Architecture | MicroGPT (decoder-only transformer) |
| Parameters | 431,178 |
| Vocab Size | 74 (character-level) |
| Block Size | 128 tokens |
| Layers | 2 |
| Attention Heads | 4 |
| Embedding Dim | 128 |
| Training Steps | 500 |
| Best Val Loss | 2.213334321975708 |
| Dataset | tinystories |

## Quick Start

```bash
# Clone the repo
git clone https://huggingface.co/{{cookiecutter.repo_id if cookiecutter else 'your-username/micro-gpt-' + config['dataset']}}
cd micro-gpt-tinystories

# Install dependencies
pip install -r requirements.txt

# Generate text
python inference.py --prompt "Once upon a time"
```

## Usage in Python

```python
import torch
from models.micro_gpt import MicroGPT

# Load config
import json
with open('config.json') as f:
    config = json.load(f)

# Build model
model = MicroGPT(
    vocab_size=config['vocab_size'],
    block_size=config['block_size'],
    n_layer=config['n_layer'],
    n_head=config['n_head'],
    n_embd=config['n_embd'],
    dropout=config['dropout'],
)
model.load_state_dict(torch.load('pytorch_model.bin', map_location='cpu'))
model.eval()

# Load tokenizer
with open('tokenizer.json') as f:
    tokenizer = json.load(f)

# Encode prompt
prompt = "Once upon a time"
indices = [tokenizer['stoi'].get(c, 0) for c in prompt]
input_ids = torch.tensor([indices], dtype=torch.long)

# Generate
output_ids = model.generate(input_ids, max_new_tokens=200, temperature=0.9, top_k=40)
text = ''.join(tokenizer['itos'][str(i)] for i in output_ids[0].tolist())
print(text)
```

## Training

This model was trained using the project's training pipeline:

```bash
python run_model.py --train_model --arch micro_gpt_tf --dataset tinystories --max_steps 500
```

## Files in this repository

| File | Description |
|------|-------------|
| `models/` | Full model source code (MicroGPT architecture) |
| `pytorch_model.bin` | Trained model weights |
| `config.json` | Model hyperparameters |
| `tokenizer.json` | Character-level tokenizer |
| `inference.py` | Ready-to-use inference script |
| `requirements.txt` | Python dependencies |