license: mit
language:
- en
library_name: pytorch
pipeline_tag: text-generation
datasets:
- roneneldan/TinyStories
tags:
- gpt
- transformer
- llm-from-scratch
- tinystories
TinyStories GPT from Scratch
A small GPT-style language model trained from scratch on TinyStories. It is an educational model for generating short, simple English stories.
Model details
- Architecture: decoder-only transformer
- Parameters: 12,297,984
- Layers: 6
- Attention heads: 6
- Embedding size: 384
- Context length: 256 tokens
- Tokenizer: byte-level BPE, 4,096 tokens
- Framework: PyTorch
The exported checkpoint was saved at training step 1,000. Its recorded training loss was 3.11046 and recorded validation loss was 3.07700. These values are checkpoint metadata, not a broader benchmark.
Usage
import sys
from pathlib import Path
import torch
from huggingface_hub import snapshot_download
from tokenizers import Tokenizer
model_dir = Path(snapshot_download("Haider92/tinystories-gpt-from-scratch"))
sys.path.insert(0, str(model_dir))
from inference_utils import generate_story, load_exported_model
device = torch.device("cpu")
model = load_exported_model(
model_dir / "config.json",
model_dir / "model.pt",
device,
)
tokenizer = Tokenizer.from_file(
str(model_dir / "tinystories_tokenizer.json")
)
result = generate_story(
model,
tokenizer,
prompt="Once upon a time",
target_tokens=120,
extra_tokens=80,
temperature=0.8,
top_k=40,
device=device,
)
print(result.story)
Intended use
This model is intended for education, experimentation, and short TinyStories-style text generation. It is not a general-purpose assistant.
Limitations
The model is small and trained on a constrained synthetic-story dataset. Outputs may be repetitive, inconsistent, incomplete, or factually incorrect. It has no application-specific safety guarantees.
Dataset and license
TinyStories is distributed under CDLA-Sharing-1.0. No dataset files are included in this repository.
The exported model weights and included source code are available under the MIT License.