MiniGPT
A Transformer language model, built from scratch and trained on Shakespeare’s plays.
3.2M parameters · 128-character memory · trained in 6 minutes
Temperature
0.8
carefulwild
Length
200 characters
Write a first line before asking the model to continue.
The model writes
Nothing written yet. Give the model a first line.
The model is waking up. Free servers sleep when idle; try again in a moment.
How it works
- 01CharactersText is split into single characters, 65 in total.
- 02EmbeddingsEach becomes a 256-dimensional vector, plus a sinusoidal position signal.
- 034 Transformer blocksCausal self-attention with 4 heads, then a feed-forward layer.
- 04ProbabilitiesA score for each of the 65 possible next characters.
- 05TextOne character is sampled, added, and the loop repeats.