akarki15/bdh-shakespeare

BDH Shakespeare — Baby Dragon Hatchling

A 25M parameter biologically-inspired language model trained on the Tiny Shakespeare dataset. Generates Shakespearean-style English text using byte-level tokenization.

Try the live demo | GitHub

Architecture

BDH (Baby Dragon Hatchling) is a compact, biologically-inspired transformer:

Component	Value
Parameters	25M
Layers	6 (shared weights)
Embedding dim	256
Attention heads	4
Context length	128
Vocab size	256 (byte-level)

Key features:

• Shared weights across layers (iterative refinement)
• Byte-level tokenization (no external tokenizer needed)
• Hebbian gating (multiplicative sparse activations)
• ReLU sparsity for interpretable activations
• RoPE position embeddings

Training

• Dataset: Tiny Shakespeare (~1MB)
• Optimizer: AdamW (lr=1e-3, weight_decay=0.1)
• Steps: 5,000
• Hardware: Google Colab T4 GPU (~30 min)
• Val loss: 1.57 | Perplexity: 4.81

Usage

import torch

model_data = torch.hub.load_state_dict_from_url(
    "https://huggingface.co/akarki15/bdh-shakespeare/resolve/main/bdh_shakespeare.pt",
    map_location="cpu"
)
# Load with BDH model class — see GitHub repo for full inference code

Reference

Based on The Dragon Hatchling architecture by Pathway.