ode · Pessoana char-GPT
A small (~10.7M-parameter) character-level GPT, trained from scratch in tinygrad on the Pessoana corpus of Portuguese classic literature. It writes Portuguese prose and verse one character at a time.
Model details
- Architecture: decoder-only transformer (GPT) — 6 layers, 6 heads, 384-dim, context 256, weight-tied embeddings. ~10.7M parameters.
- Tokenizer: character-level, 162-character vocabulary (
meta.pkl). - Framework: tinygrad (the whole model is ~130 lines, in
model.py). - Training: 5000 steps on an NVIDIA A100. AdamW (weight decay 0.1 on 2-D weights only), cosine LR 1e-3 → 1e-4 with 100-step warmup, gradient clipping 1.0, dropout 0.2, batch size 64, block size 256.
- Best validation loss: 1.361 nats (1.96 bits/char).
Usage
The model ships with its architecture (model.py) and a sampler (sample.py):
pip install tinygrad numpy huggingface_hub
huggingface-cli download vreabernardo/ode-pessoana --local-dir ode-pessoana
cd ode-pessoana
python sample.py --prompt "A cidade" --tokens 500 --temperature 0.8
model.safetensors holds the weights; config.json the dimensions; meta.pkl the
character vocabulary for encode/decode.
Sample output
Prompt "A cidade":
A cidade da camisa me faz ali perdão; é para me lembrar o que me diz em que casada,
se me engendrara. Eu não custaria a minha graça, a ser só a sua...»
A RAINHA
— Mas é assim. De mim não te amendo a distância, com filha, as manas, se opusesse de si.
D. LANCEROTE
— Se pensar o meu quarto, que agradecesse o rei...
D. TIBÚRCIO
— Continuou ela. E eu não o quis que me disse que eu fosse a torre a catarra.
Training data
Pessoana — 84 public-domain works of Portuguese and Brazilian classic literature (Eça de Queirós, Machado de Assis, Camilo Castelo Branco, Florbela Espanca, Fernando Pessoa…), from Projecto Adamastor, under CC BY-SA 4.0.
Limitations
A character-level model of this size learns Portuguese spelling, morphology, dialogue formatting, and literary rhythm, but not factual or long-range semantic coherence — the output reads like Portuguese literature without meaning much. It uses pre-1990 orthography.
License
CC BY-SA 4.0, inheriting the ShareAlike terms of the training corpus.
Citation
@misc{vrea2026odepessoana,
title = {ode: A Character-Level GPT for Portuguese Literature},
author = {Vrea, Bernardo},
year = {2026},
publisher = {Hugging Face},
url = {https://huggingface.co/vreabernardo/ode-pessoana}
}
- Downloads last month
- 32