Krushna-GPT

A GPT built from scratch — every component implemented as a learning exercise.

Architecture

• Type: Decoder-only Transformer (same architecture as GPT-2)
• Vocab size: 568 tokens (BPE)
• Context length: 128 tokens
• Model dimension: 128
• Transformer blocks: 4
• Attention heads: 4
• Parameters: 953,656

Tokenizer

Custom BPE tokenizer trained on the same corpus as the model. Not compatible with GPT-2 tokenizer.

Training

Trained from scratch using:

• AdamW optimizer with cosine annealing
• Cross-entropy loss (next-token prediction)
• 15,000 epochs on a custom text corpus

Usage

import torch, json
from model.gpt import GPT
from data.tokenizer import BPETokenizer

# Load tokenizer
with open("tokenizer.json") as f:
    tok_data = json.load(f)

# Load model
ckpt = torch.load("pytorch_model.bin", map_location="cpu")
model = GPT(
    vocab_size=568, context_length=128,
    model_dim=128, num_blocks=4, num_heads=4
)
model.load_state_dict(ckpt)
model.eval()

Links

• 📖 Medium article
• 💻 Source code

Credit

Built while studying the NeetCode ML course.