File size: 4,742 Bytes
4719692 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 | # Vanilla Shakespeare GPT, approximately 10M parameters
The complete, readable architecture is in [model.py](model.py): six pre-LayerNorm
causal decoder blocks, width 384, six attention heads, learned positions, GELU
MLPs, and a separate output projection. With 65 characters and context 256 it has
**10,795,776 parameters**. The [100M architecture](../100m/README.md) now defaults
to RMSNorm, RoPE and SwiGLU; its explicit version-1 mode preserves this vanilla
block design at larger dimensions. This published 10M definition is unchanged.
The model starts from random weights and learns next-character prediction from
Tiny Shakespeare. This is a learning benchmark, not an instruction-following
assistant. A contiguous 90/10 train/validation split prevents windows crossing
the boundary; vocabulary is drawn from the whole corpus. There is no sealed test.
## Bounded Modal training
Run from `training/`, using a Python environment with Modal's API proxy support.
The Modal image explicitly pins PyTorch 2.6.0, safetensors 0.8.0 and tqdm 4.70.1,
independently of the newer version selected by the local training project's lock.
The executed versions and source hashes are saved in every run's config.
The published runs used the former `modal_shakespeare.py` (commit `ec5ebd1`). The
same settings are now the [`shakespeare-10m` recipe](../recipes/shakespeare-10m.json),
launched with the generic [recipe launcher](../recipes/README.md) (use a new run ID):
```bash
uv run --with 'modal[api-proxy-support]==1.6.1' modal run modal_train.py \
--recipe recipes/shakespeare-10m.json --run-id shakespeare-10m-smoke-02 --smoke
```
The original runs used app `lm-exploration-shakespeare-10m` and volume
`lm-exploration-training`. The smoke worker has a 120-second hard timeout; the
training worker a 1,500-second hard timeout and a cooperative 1,200-second training
limit. Both use one L4, no retries, and persist artifacts on exit. A separate
ignored `runs/shakespeare-budget.json` (now `runs/modal-budget.json`) reserved the full worker time at a
conservative $5/hour, with a $3 allocation ceiling. This is reservation accounting,
not an actual bill or an account-wide spending limit. No latent-workspace ledger
or research volume is used.
Training targets 4,000 updates with batch 32, context 256, FP32 AdamW,
learning rate 0.0003, betas (0.9, 0.95), weight decay 0.1, clipping norm 1 and seed
1337. Every 200 updates, evaluate 20 fixed-seed validation batches and save a
checkpoint. Select the saved checkpoint with the lowest validation loss.
Cooperative stopping logs/evaluates/saves the final completed update. Each run
writes a report, metrics, source/runtime/dataset provenance and a sample.
## Export and publication
The published model was exported with `publish_shakespeare.py`, removed after
publication (see commit `ec5ebd1`). It exported the validation-selected checkpoint
to safetensors without optimizer or RNG state, checked the parameter count,
selected update and architecture hash, and uploaded only inference files.
The public bundle includes `model.py`, `generate.py`, safetensors weights, exact
configuration/tokenizer, provenance, requirements and the measured training report.
From the downloaded bundle run `python generate.py --prompt 'ROMEO:' --tokens 300`.
The model uses custom PyTorch code, not Transformers AutoModel loading.
## Completed training and public model
[Public Hugging Face model](https://huggingface.co/jfrick/shakespeare-vanilla-gpt-10m). The bounded L4 run completed 4,000 updates
in 503.3 worker seconds, with 32,768,000 training-character
exposures. Validation selected **update 1,800**, after 14,745,600 exposures:
**1.54483 nats/character**, character perplexity **4.687**.
The final update scored 2.10690 on validation; later training overfit this small
corpus, so the public weights come from the earlier checkpoint.
[Full training report](../results/shakespeare-10m-train-01.json),
[loss history](../results/shakespeare-10m-metrics.jsonl),
[smoke report](../results/shakespeare-10m-smoke-01.json),
and [publication receipt](../results/shakespeare-10m-publication.json).
One seed, validation used for checkpoint selection, no untouched test result.
Local verification covered the 10M parameter count, causal attention and gradients
against SDPA, safe stopping/checkpoint saving, reproducible CPU resume, and exact
exported logits/seeded generation against the selected training checkpoint.
Ruff lint/format, lock consistency and Git diff checks passed.
Pinned public release: `48b3eb9b56fd03056943bb905b8cd7637009588c`. Anonymous
Hub download verified the weights and definition checksums, strict model loading
and exact seeded generation from the downloaded standalone package.
|