---
license: mit
language:
- en
library_name: pytorch
pipeline_tag: text-generation
datasets:
- roneneldan/TinyStories
tags:
- gpt
- transformer
- llm-from-scratch
- tinystories
---

# TinyStories GPT from Scratch

A small GPT-style language model trained from scratch on
[TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories).
It is an educational model for generating short, simple English stories.

## Model details

- Architecture: decoder-only transformer
- Parameters: 12,297,984
- Layers: 6
- Attention heads: 6
- Embedding size: 384
- Context length: 256 tokens
- Tokenizer: byte-level BPE, 4,096 tokens
- Framework: PyTorch

The exported checkpoint was saved at training step 1,000. Its recorded
training loss was 3.11046 and recorded validation loss was 3.07700. These
values are checkpoint metadata, not a broader benchmark.

## Usage

```python
import sys
from pathlib import Path

import torch
from huggingface_hub import snapshot_download
from tokenizers import Tokenizer

model_dir = Path(snapshot_download("Haider92/tinystories-gpt-from-scratch"))
sys.path.insert(0, str(model_dir))

from inference_utils import generate_story, load_exported_model

device = torch.device("cpu")
model = load_exported_model(
    model_dir / "config.json",
    model_dir / "model.pt",
    device,
)
tokenizer = Tokenizer.from_file(
    str(model_dir / "tinystories_tokenizer.json")
)

result = generate_story(
    model,
    tokenizer,
    prompt="Once upon a time",
    target_tokens=120,
    extra_tokens=80,
    temperature=0.8,
    top_k=40,
    device=device,
)
print(result.story)
```

## Intended use

This model is intended for education, experimentation, and short
TinyStories-style text generation. It is not a general-purpose assistant.

## Limitations

The model is small and trained on a constrained synthetic-story dataset.
Outputs may be repetitive, inconsistent, incomplete, or factually incorrect.
It has no application-specific safety guarantees.

## Dataset and license

TinyStories is distributed under
[CDLA-Sharing-1.0](https://cdla.dev/sharing-1-0/). No dataset files are
included in this repository.

The exported model weights and included source code are available under the
MIT License.

## Links

- [Source code](https://github.com/haiderhamad/llm-from-zero)
- [Interactive demo](https://huggingface.co/spaces/Haider92/tinystories-gpt-demo)