| ---
|
| license: mit
|
| tags:
|
| - transformer
|
| - from-scratch
|
| - pytorch
|
| - text-generation
|
| datasets:
|
| - roneneldan/TinyStories
|
| ---
|
|
|
| # Mini Transformer (From Scratch)
|
|
|
| A decoder-only transformer built entirely from first principles in PyTorch,
|
| no pre-built attention layers, no HuggingFace Transformers library internals.
|
| Every component (multi-head self-attention, positional encoding, feedforward
|
| blocks, residual connections) is implemented from scratch and trained on the
|
| TinyStories dataset.
|
|
|
| This is part of a larger project studying transformer internals, followed by
|
| implementing DeepSeek's MLA (Multi-head Latent Attention) and mHC
|
| (Manifold-Constrained Hyper-Connections) as architectural upgrades to this
|
| same baseline.
|
|
|
| ## Architecture
|
|
|
| - Decoder-only transformer, GPT-style
|
| - Embedding dim: 256
|
| - Attention heads: 8
|
| - Layers: 6
|
| - Feedforward hidden dim: 1024
|
| - Context length: 128 tokens
|
| - Tokenizer: GPT-2 (tiktoken), 50,257 vocab
|
| - Total parameters: 336,488
|
|
|
| ## Training
|
|
|
| - Dataset: TinyStories (roneneldan/TinyStories), ~474M training tokens
|
| - Steps: 10,000
|
| - Batch size: 32
|
| - Optimizer: AdamW, lr=3e-4
|
| - Hardware: single RTX 3060 (6GB VRAM)
|
| - Final train loss: None
|
| - Final val loss: None
|
|
|
| ## Usage
|
|
|
| Load `model.py` for the architecture classes, then:
|
|
|
| ```python
|
| import torch
|
| from model import MiniGPT
|
|
|
| model = MiniGPT(vocab_size=50257, embed_dim=256, num_heads=8,
|
| num_layers=6, hidden_dim=1024, max_seq_len=128)
|
| ckpt = torch.load("checkpoint.pt", map_location="cpu")
|
| model.load_state_dict(ckpt["model"])
|
| ```
|
|
|
| ## Sample Output
|
|
|
| > Once upon a time, there was a little girl named Lily who loved to play
|
| > outside. She had a lot of fun playing in the mud and splashing around...
|
|
|
