Revise model card: correct numbers and license, remove unmeasured claims, add limitations
Browse files
README.md
CHANGED
|
@@ -9,24 +9,26 @@ tags:
|
|
| 9 |
- from-scratch
|
| 10 |
- interpretability
|
| 11 |
- tiny
|
|
|
|
| 12 |
pipeline_tag: text-generation
|
| 13 |
library_name: pytorch
|
| 14 |
---
|
| 15 |
|
| 16 |
# Tiny SQL GPT
|
| 17 |
|
| 18 |
-
> **
|
| 19 |
-
> part of it,
|
| 20 |
|
| 21 |
-
**
|
| 22 |
-
five minutes
|
|
|
|
| 23 |
|
| 24 |
No pretrained weights. No `transformers` model classes. No API keys. The full architecture is
|
| 25 |
~200 readable lines of PyTorch.
|
| 26 |
|
| 27 |
This model exists to be **understood**, not deployed. Its vocabulary is 155 tokens, which is the
|
| 28 |
point: small enough that you can print the *entire* probability distribution at every generation
|
| 29 |
-
step
|
| 30 |
|
| 31 |
- **Code, evaluation harness and write-up:** https://github.com/sarathi-aiml/tiny-sql-gpt
|
| 32 |
- **Plain-English explainer (no maths):** [`EXPLAIN.md`](https://github.com/sarathi-aiml/tiny-sql-gpt/blob/main/EXPLAIN.md)
|
|
@@ -82,7 +84,14 @@ numbers.
|
|
| 82 |
| validation loss | 0.662 | n/a |
|
| 83 |
|
| 84 |
100% is a real measurement, but read it against the task: 14 query shapes, 3 tables, 155 tokens.
|
| 85 |
-
A model that saturates *this*
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 86 |
|
| 87 |
### Scaling
|
| 88 |
|
|
@@ -95,10 +104,11 @@ Same architecture and data at four sizes, all trained on one laptop:
|
|
| 95 |
| **tiny** (this model) | **841,216** | **100.0%** | **100.0%** |
|
| 96 |
| small | 4,834,816 | 100.0% | 100.0% |
|
| 97 |
|
| 98 |
-
Syntax is
|
| 99 |
-
more, and
|
| 100 |
-
from 124K to 4.8M parameters buys 0.007 of validation loss and no
|
| 101 |
-
converge to ~0.66, which is the entropy of the data generator
|
|
|
|
| 102 |
|
| 103 |
---
|
| 104 |
|
|
@@ -108,7 +118,7 @@ The training data contains a deliberately planted long-range dependency: **the c
|
|
| 108 |
`GROUP BY` is always the column that appeared first in `SELECT`**, roughly 8 tokens earlier.
|
| 109 |
Getting it right requires looking back, which is what attention is for.
|
| 110 |
|
| 111 |
-
**In
|
| 112 |
head L0H1 places **58.7% of its attention** on the `SELECT` column, 7.0x above uniform. Nobody
|
| 113 |
designed or labelled it.
|
| 114 |
|
|
@@ -128,7 +138,8 @@ it answered: ... GROUP BY -> "status" (familiar, confident, wrong)
|
|
| 128 |
|
| 129 |
A control prompt shows the context *is* being used (3.8x to 10.2x lift), it simply loses to a prior
|
| 130 |
against tokens never seen in that slot. **Attention identifies the right source token; the output
|
| 131 |
-
prior overrules it.**
|
|
|
|
| 132 |
|
| 133 |
---
|
| 134 |
|
|
@@ -140,11 +151,11 @@ prior overrules it.** That is hallucination, in a model small enough to point at
|
|
| 140 |
| architecture | decoder-only, 4 layers, 4 heads, 128 embedding, 64-token context |
|
| 141 |
| tokenizer | word-level, 155 tokens |
|
| 142 |
| optimizer | AdamW, lr 1e-3, cosine schedule, weight decay 0.01, grad clip 1.0 |
|
| 143 |
-
| steps | 3,000 · batch 64 · ~5 minutes on a laptop CPU |
|
| 144 |
| final loss | train 0.658 · val 0.662 |
|
| 145 |
|
| 146 |
-
Training data is **generated, not scraped**, from a grammar in the repo
|
| 147 |
-
|
| 148 |
|
| 149 |
## Limitations
|
| 150 |
|
|
|
|
| 9 |
- from-scratch
|
| 10 |
- interpretability
|
| 11 |
- tiny
|
| 12 |
+
- experiment
|
| 13 |
pipeline_tag: text-generation
|
| 14 |
library_name: pytorch
|
| 15 |
---
|
| 16 |
|
| 17 |
# Tiny SQL GPT
|
| 18 |
|
| 19 |
+
> **Personal learning project** (August 2026). The goal was to build a language model from scratch, end to end,
|
| 20 |
+
> small enough to understand every part of it, as a way to understand how the large ones work.
|
| 21 |
|
| 22 |
+
**An 841,216-parameter decoder-only transformer, trained from random weights on a laptop CPU in
|
| 23 |
+
about five minutes, on a toy SQL grammar.** On that toy task, all 500 sampled queries execute against a SQLite
|
| 24 |
+
database. That says more about how simple the task is than about the model (see Results).
|
| 25 |
|
| 26 |
No pretrained weights. No `transformers` model classes. No API keys. The full architecture is
|
| 27 |
~200 readable lines of PyTorch.
|
| 28 |
|
| 29 |
This model exists to be **understood**, not deployed. Its vocabulary is 155 tokens, which is the
|
| 30 |
point: small enough that you can print the *entire* probability distribution at every generation
|
| 31 |
+
step.
|
| 32 |
|
| 33 |
- **Code, evaluation harness and write-up:** https://github.com/sarathi-aiml/tiny-sql-gpt
|
| 34 |
- **Plain-English explainer (no maths):** [`EXPLAIN.md`](https://github.com/sarathi-aiml/tiny-sql-gpt/blob/main/EXPLAIN.md)
|
|
|
|
| 84 |
| validation loss | 0.662 | n/a |
|
| 85 |
|
| 86 |
100% is a real measurement, but read it against the task: 14 query shapes, 3 tables, 155 tokens.
|
| 87 |
+
A model that saturates *this* shows the training loop works. It is not a text-to-SQL system.
|
| 88 |
+
|
| 89 |
+
Caveats:
|
| 90 |
+
- Only 16.6% of samples are novel. Most generated queries reproduce queries from the training set. The bigram
|
| 91 |
+
baseline is "novel" mostly because it produces invalid SQL.
|
| 92 |
+
- Validation loss uses the last 10% of the same generated corpus. With only 16,941 unique queries in 100,000
|
| 93 |
+
examples, many validation queries also appear in training, so train ≈ val loss isn't evidence of generalization.
|
| 94 |
+
The held-out `GROUP BY` probe below is the real generalization test, and the model fails it.
|
| 95 |
|
| 96 |
### Scaling
|
| 97 |
|
|
|
|
| 104 |
| **tiny** (this model) | **841,216** | **100.0%** | **100.0%** |
|
| 105 |
| small | 4,834,816 | 100.0% | 100.0% |
|
| 106 |
|
| 107 |
+
Syntax is cheap: 24K parameters is enough to write SQL that runs. The long-range dependency needs ~5x
|
| 108 |
+
more parameters, and shows up as a sharp jump between 24K and 124K. Above that, nothing improves: going
|
| 109 |
+
from 124K to 4.8M parameters buys 0.007 of validation loss and no change in behaviour. All four
|
| 110 |
+
converge to ~0.66, which I believe is close to the entropy of the data generator rather than a limit of the
|
| 111 |
+
models. I didn't compute the generator's entropy directly. Each size is a single seed.
|
| 112 |
|
| 113 |
---
|
| 114 |
|
|
|
|
| 118 |
`GROUP BY` is always the column that appeared first in `SELECT`**, roughly 8 tokens earlier.
|
| 119 |
Getting it right requires looking back, which is what attention is for.
|
| 120 |
|
| 121 |
+
**In a 1-layer model, one head learned it.** In `flat` (1 layer, 127K params),
|
| 122 |
head L0H1 places **58.7% of its attention** on the `SELECT` column, 7.0x above uniform. Nobody
|
| 123 |
designed or labelled it.
|
| 124 |
|
|
|
|
| 138 |
|
| 139 |
A control prompt shows the context *is* being used (3.8x to 10.2x lift), it simply loses to a prior
|
| 140 |
against tokens never seen in that slot. **Attention identifies the right source token; the output
|
| 141 |
+
prior overrules it.** It's a small, inspectable example of a hallucination-like failure (3 held-out pairs, so an
|
| 142 |
+
illustration, not a statistic).
|
| 143 |
|
| 144 |
---
|
| 145 |
|
|
|
|
| 151 |
| architecture | decoder-only, 4 layers, 4 heads, 128 embedding, 64-token context |
|
| 152 |
| tokenizer | word-level, 155 tokens |
|
| 153 |
| optimizer | AdamW, lr 1e-3, cosine schedule, weight decay 0.01, grad clip 1.0 |
|
| 154 |
+
| steps | 3,000 · batch 64 × 64 tokens (~12.3M tokens seen, ~10 passes over the training split) · ~5 minutes on a laptop CPU |
|
| 155 |
| final loss | train 0.658 · val 0.662 |
|
| 156 |
|
| 157 |
+
Training data is **generated, not scraped**, from a grammar in the repo, so a learner can read the entire
|
| 158 |
+
source of the training set.
|
| 159 |
|
| 160 |
## Limitations
|
| 161 |
|