disco-jack-basement
/

letterpress

+---
+license: mit
+pipeline_tag: text-generation
+datasets:
+- disco-jack-basement/byob-pd-book-corpus
+language:
+- en
+- fr
+- de
+- it
+- es
+- pt
+- nl
+- sv
+- da
+- 'no'
+- fi
+- pl
+- hu
+- cs
+- la
+- el
+- grc
+- uk
+tags:
+- pytorch
+- gpt
+- transformer
+- character-level
+- language-model
+- from-scratch
+- public-domain
+---
+# letterpress - trained character-level GPTs
+The trained checkpoints of [letterpress](https://github.com/Novotarskyi/letterpress),
+a character-level GPT built from scratch in PyTorch (following Karpathy's "Let's
+build GPT" lecture) and trained on nested tiers of a public-domain book corpus.
+These are **base models**: they continue text in the style of the books they read,
+one character at a time - they are not chatbots and do not follow instructions.
+(The checkpoint files keep the model family's historical name `byob-lm` - the
+letterpress repo was formerly `byob_llm` - so paths, benchmark reports, and the
+companion dataset stay stable.)
+## Models
+| tier | params | layers x width | context | vocab | best val loss | held-out bpc | wikitext bpc | file |
+|---|--:|---|--:|--:|--:|--:|--:|---|
+| `nano` | 0.82M | 4 x 128 | 128 | 65 | - | **2.68** | 4.09 | `nano/shakespeare-nano.pt` |
+| `medium` | 25.4M | 8 x 512 | 256 | 97 | 1.2278 | **1.69** | 2.91 | `medium/byob-lm.best.pt` |
+| `large` | 49.7M | 10 x 640 | 384 | 198 | 1.1111 | **1.57** | 2.30 | `large/byob-lm.best.pt` |
+| `xlarge` | 99.9M | 14 x 768 | 512 | 199 | 0.9411 | **1.31** | 1.88 | `xlarge/byob-lm.best.pt` |
+| `2xlarge` | 202M | 16 x 1024 | 512 | 204 | 0.8725 | **1.17** | 1.71 | `2xlarge/byob-lm.best.pt` |
+Held-out bpc = bits-per-char on public-domain books absent from every training
+tier; wikitext = out-of-domain. Both measured by the repo's bundled `lm_bench`
+harness (seed 1337, byte-normalized); full scorecards live in the repo under
+`lm_bench/benchmarks/`. Lower is better; the corpus tiers are nested, so every
+gain down the table is scale, not data luck.
+## How to use
+The checkpoints are self-describing (`{model, config, stoi, itos}`, optimizer
+state stripped) but unpickle the letterpress `GPTConfig`, so you need the repo's
+source next to them:
+```bash
+git clone https://github.com/Novotarskyi/letterpress.git && cd letterpress
+python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
+hf download disco-jack-basement/letterpress 2xlarge/byob-lm.best.pt --local-dir models
+.venv/bin/python -m inference.sample   --ckpt models/2xlarge/byob-lm.best.pt --prompt "ROMEO:"
+.venv/bin/python -m inference.interact --ckpt models/2xlarge/byob-lm.best.pt
+cd lm_bench && ../.venv/bin/python -m lm_bench run --model byob:../models/2xlarge/byob-lm.best.pt --tasks core
+```
+## Training provenance
+- **nano** - Tiny Shakespeare (1.1M chars); the lecture demo, trained locally
+- **medium** - medium tier (523M chars); Apple-Silicon MPS, ~1 h
+- **large** - large tier (1.08B chars); Apple-Silicon MPS, ~3.5 h
+- **xlarge** - xlarge tier (2.05B chars); rented H100 SXM, ~2.5 h
+- **2xlarge** - 2xlarge tier (4.26B chars); rented H200, ~12.6 h
+Each `<tier>/` folder here also carries the archive's provenance files
+(context.md with the full hyperparameters and val curve, corpus_stats.txt,
+corpus.lock.json, corpus_index.md) where the archive has them.
+## Data
+Trained exclusively on the companion public-domain corpus
+[`disco-jack-basement/byob-pd-book-corpus`](https://huggingface.co/datasets/disco-jack-basement/byob-pd-book-corpus)
+(CC0; Project Gutenberg, Standard Ebooks, Internet Archive, Wikisource), curated
+under a strict, code-enforced no-Russian-content rule.
+## License
+Code and weights: MIT. The training corpus is CC0 and published separately.