achavan1211/harrypotter-gpt-bpe

Harry Potter GPT (BPE, ~12.3M)

A from-scratch decoder-only Transformer (~12.3M parameters) trained on the Harry Potter series (books 1-7). Built with raw PyTorch (not transformers) — see model.py.

Model details

• Architecture: decoder-only Transformer
• Tokenizer: byte-level BPE (huggingface tokenizers), vocab 2000
• Layers / heads / embedding dim: 6 / 6 / 384
• Context length (block_size): 256
• Parameters: 12.28M

Training

• Data: WutYee/HarryPotter_books_1to7, pooled, shuffled, and re-split 90/5/5 (the dataset's native splits are book-contiguous).
• Optimizer: AdamW (lr=0.0003, weight_decay=0.1, dropout=0.2)
• Iterations: 5000 (best checkpoint @ step 4999)
• Best val BPC: 1.564
• Held-out test BPC: 1.560

Usage

Custom PyTorch model. See model.py (architecture), config.json (hyperparameters), and tokenizer.json (BPE tokenizer).

import json, torch
from tokenizers import Tokenizer
from huggingface_hub import hf_hub_download
from model import GPTLanguageModel

repo = "achavan1211/harrypotter-gpt-bpe"
cfg = json.load(open(hf_hub_download(repo, "config.json")))
tokenizer = Tokenizer.from_file(hf_hub_download(repo, "tokenizer.json"))

model = GPTLanguageModel(**cfg)
model.load_state_dict(torch.load(hf_hub_download(repo, "pytorch_model.bin"), map_location="cpu"))
model.eval()

ctx = torch.tensor([tokenizer.encode("
").ids])
print(tokenizer.decode(model.generate(ctx, max_new_tokens=500)[0].tolist()))