achavan1211/shakespeare-gpt-bpe

Shakespeare GPT (BPE subword tokenizer)

A small decoder-only Transformer (~2.2M parameters) trained from scratch on the Tiny Shakespeare dataset. This is the BPE subword version: instead of a 65-character vocabulary, it trains a ~1000-token Byte-Pair Encoding tokenizer on the corpus and trains the transformer over those subword tokens.

Model Details

• Architecture: decoder-only Transformer
• Tokenizer: Byte-Pair Encoding (huggingface tokenizers)
• Vocab size: 1000 (BPE subwords)
• Embedding dim: 192
• Number of heads: 4
• Number of layers: 4
• Context length (block size): 256
• Total parameters: 2.21M

Training

• Dataset: Tiny Shakespeare
• Optimizer: AdamW (lr=0.0003, weight_decay=0.1)
• Iterations: 3000 (best checkpoint @ step 2999)
• Best val loss: 3.7030 (2.116 BPC)

Usage

Custom PyTorch model. See model.py for the architecture, config.json for hyperparameters, and tokenizer.json for the BPE tokenizer.

import json, torch
from tokenizers import Tokenizer
from huggingface_hub import hf_hub_download
from model import GPTLanguageModel

repo = "achavan1211/shakespeare-gpt-bpe"

config = json.load(open(hf_hub_download(repo, "config.json")))
tokenizer = Tokenizer.from_file(hf_hub_download(repo, "tokenizer.json"))

encode = lambda s: tokenizer.encode(s).ids
decode = lambda l: tokenizer.decode(l)

model_cfg = {k: v for k, v in config.items() if k != "tokenizer"}
model = GPTLanguageModel(**model_cfg)
model.load_state_dict(torch.load(hf_hub_download(repo, "pytorch_model.bin"), map_location="cpu"))
model.eval()

context = torch.zeros((1, 1), dtype=torch.long)
print(decode(model.generate(context, max_new_tokens=500)[0].tolist()))