File size: 1,461 Bytes
dbe867d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 | # SuperSmallJokeClaude
A 1.64M-parameter tiny transformer trained from scratch on Fable 5 Claude Code assistant traces.
## Specs
- **Params:** 1,644,032 (1.64M)
- **Architecture:** 4-layer transformer, d_model=128, 4 heads, d_ff=256, tied embeddings
- **Vocab:** BPE-8192 (from Claude Code tokenizer)
- **Context:** 512 tokens
- **Training data:** 24,249 tokens (Fable 5 Claude Code assistant traces)
- **Training:** 2000 steps, batch 8, seq 256, AdamW lr=3e-4 cosine schedule
- **Final loss:** 2.90
- **Training time:** 7 seconds on RTX 5090
## What it does
It's a joke — a tiny model trained on tiny data. It will produce degenerate output (repeated tokens, gibberish). It exists to demonstrate that you can train a working transformer in 7 seconds on a single GPU.
## Files
- `final.pt` — full checkpoint (model state dict + config + training metadata)
- `tokenizer.json` — BPE-8192 tokenizer (Claude Code)
## Load it
```python
import torch
from tokenizers import Tokenizer
ckpt = torch.load('final.pt', map_location='cpu')
config = ckpt['config']
# Build model (see train_jokeclaude_v5.py for the class definition)
model = Transformer(**config)
model.load_state_dict(ckpt['model_state_dict'])
model.eval()
tokenizer = Tokenizer.from_file('tokenizer.json')
```
## Disclaimer
This is an experimental/fun artifact. The model is not useful for any real task. It exists because someone asked for a "super small joke claude" and we said why not.
|