---
license: mit
pipeline_tag: text-generation
datasets:
- disco-jack-basement/byob-pd-book-corpus
language:
- en
- fr
- de
- it
- es
- pt
- nl
- sv
- da
- 'no'
- fi
- pl
- hu
- cs
- la
- el
- grc
- uk
tags:
- pytorch
- gpt
- transformer
- character-level
- language-model
- from-scratch
- public-domain
---

# letterpress - trained character-level GPTs

The trained checkpoints of [letterpress](https://github.com/Novotarskyi/letterpress),
a character-level GPT built from scratch in PyTorch (following Karpathy's "Let's
build GPT" lecture) and trained on nested tiers of a public-domain book corpus.
These are **base models**: they continue text in the style of the books they read,
one character at a time - they are not chatbots and do not follow instructions.

(The checkpoint files keep the model family's historical name `byob-lm` - the
letterpress repo was formerly `byob_llm` - so paths, benchmark reports, and the
companion dataset stay stable.)

## Models

| tier | params | layers x width | context | vocab | best val loss | held-out bpc | wikitext bpc | file |
|---|--:|---|--:|--:|--:|--:|--:|---|
| `nano` | 0.82M | 4 x 128 | 128 | 65 | - | **2.68** | 4.09 | `nano/shakespeare-nano.pt` |
| `medium` | 25.4M | 8 x 512 | 256 | 97 | 1.2278 | **1.69** | 2.91 | `medium/byob-lm.best.pt` |
| `large` | 49.7M | 10 x 640 | 384 | 198 | 1.1111 | **1.57** | 2.30 | `large/byob-lm.best.pt` |
| `xlarge` | 99.9M | 14 x 768 | 512 | 199 | 0.9411 | **1.31** | 1.88 | `xlarge/byob-lm.best.pt` |
| `2xlarge` | 202M | 16 x 1024 | 512 | 204 | 0.8725 | **1.17** | 1.71 | `2xlarge/byob-lm.best.pt` |

Held-out bpc = bits-per-char on public-domain books absent from every training
tier; wikitext = out-of-domain. Both measured by the repo's bundled `lm_bench`
harness (seed 1337, byte-normalized); full scorecards live in the repo under
`lm_bench/benchmarks/`. Lower is better; the corpus tiers are nested, so every
gain down the table is scale, not data luck.

## How to use

The checkpoints are self-describing (`{model, config, stoi, itos}`, optimizer
state stripped) but unpickle the letterpress `GPTConfig`, so you need the repo's
source next to them:

```bash
git clone https://github.com/Novotarskyi/letterpress.git && cd letterpress
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
hf download disco-jack-basement/letterpress 2xlarge/byob-lm.best.pt --local-dir models

.venv/bin/python -m inference.sample   --ckpt models/2xlarge/byob-lm.best.pt --prompt "ROMEO:"
.venv/bin/python -m inference.interact --ckpt models/2xlarge/byob-lm.best.pt
cd lm_bench && ../.venv/bin/python -m lm_bench run --model byob:../models/2xlarge/byob-lm.best.pt --tasks core
```

## Training provenance

- **nano** - Tiny Shakespeare (1.1M chars); the lecture demo, trained locally
- **medium** - medium tier (523M chars); Apple-Silicon MPS, ~1 h
- **large** - large tier (1.08B chars); Apple-Silicon MPS, ~3.5 h
- **xlarge** - xlarge tier (2.05B chars); rented H100 SXM, ~2.5 h
- **2xlarge** - 2xlarge tier (4.26B chars); rented H200, ~12.6 h

Each `<tier>/` folder here also carries the archive's provenance files
(context.md with the full hyperparameters and val curve, corpus_stats.txt,
corpus.lock.json, corpus_index.md) where the archive has them.

## Data

Trained exclusively on the companion public-domain corpus
[`disco-jack-basement/byob-pd-book-corpus`](https://huggingface.co/datasets/disco-jack-basement/byob-pd-book-corpus)
(CC0; Project Gutenberg, Standard Ebooks, Internet Archive, Wikisource), curated
under a strict, code-enforced no-Russian-content rule.

## License

Code and weights: MIT. The training corpus is CC0 and published separately.