sarathi-balakrishnan commited on
Commit
ded1965
·
verified ·
1 Parent(s): c5730e1

Revise model card: correct numbers and license, remove unmeasured claims, add limitations

Browse files
Files changed (1) hide show
  1. README.md +26 -15
README.md CHANGED
@@ -9,24 +9,26 @@ tags:
9
  - from-scratch
10
  - interpretability
11
  - tiny
 
12
  pipeline_tag: text-generation
13
  library_name: pytorch
14
  ---
15
 
16
  # Tiny SQL GPT
17
 
18
- > **The goal:** build a language model from scratch, end to end, small enough to understand every
19
- > part of it, and in doing so understand how the large ones actually work.
20
 
21
- **A 841,216-parameter decoder-only transformer, trained from random weights on a laptop CPU in
22
- five minutes. 100% of the SQL it generates executes against a real database.**
 
23
 
24
  No pretrained weights. No `transformers` model classes. No API keys. The full architecture is
25
  ~200 readable lines of PyTorch.
26
 
27
  This model exists to be **understood**, not deployed. Its vocabulary is 155 tokens, which is the
28
  point: small enough that you can print the *entire* probability distribution at every generation
29
- step, something no frontier model demo can do.
30
 
31
  - **Code, evaluation harness and write-up:** https://github.com/sarathi-aiml/tiny-sql-gpt
32
  - **Plain-English explainer (no maths):** [`EXPLAIN.md`](https://github.com/sarathi-aiml/tiny-sql-gpt/blob/main/EXPLAIN.md)
@@ -82,7 +84,14 @@ numbers.
82
  | validation loss | 0.662 | n/a |
83
 
84
  100% is a real measurement, but read it against the task: 14 query shapes, 3 tables, 155 tokens.
85
- A model that saturates *this* is proof the training loop works, not a text-to-SQL system.
 
 
 
 
 
 
 
86
 
87
  ### Scaling
88
 
@@ -95,10 +104,11 @@ Same architecture and data at four sizes, all trained on one laptop:
95
  | **tiny** (this model) | **841,216** | **100.0%** | **100.0%** |
96
  | small | 4,834,816 | 100.0% | 100.0% |
97
 
98
- Syntax is nearly free: 24K parameters writes SQL that runs. The long-range dependency costs ~5x
99
- more, and appears as a phase transition between 24K and 125K. Above that, nothing improves: going
100
- from 124K to 4.8M parameters buys 0.007 of validation loss and no behavioural change. All four
101
- converge to ~0.66, which is the entropy of the data generator, not a limit of the models.
 
102
 
103
  ---
104
 
@@ -108,7 +118,7 @@ The training data contains a deliberately planted long-range dependency: **the c
108
  `GROUP BY` is always the column that appeared first in `SELECT`**, roughly 8 tokens earlier.
109
  Getting it right requires looking back, which is what attention is for.
110
 
111
- **In the smallest model that can do it, one head learned it.** In `flat` (1 layer, 127K params),
112
  head L0H1 places **58.7% of its attention** on the `SELECT` column, 7.0x above uniform. Nobody
113
  designed or labelled it.
114
 
@@ -128,7 +138,8 @@ it answered: ... GROUP BY -> "status" (familiar, confident, wrong)
128
 
129
  A control prompt shows the context *is* being used (3.8x to 10.2x lift), it simply loses to a prior
130
  against tokens never seen in that slot. **Attention identifies the right source token; the output
131
- prior overrules it.** That is hallucination, in a model small enough to point at the exact cause.
 
132
 
133
  ---
134
 
@@ -140,11 +151,11 @@ prior overrules it.** That is hallucination, in a model small enough to point at
140
  | architecture | decoder-only, 4 layers, 4 heads, 128 embedding, 64-token context |
141
  | tokenizer | word-level, 155 tokens |
142
  | optimizer | AdamW, lr 1e-3, cosine schedule, weight decay 0.01, grad clip 1.0 |
143
- | steps | 3,000 · batch 64 · ~5 minutes on a laptop CPU |
144
  | final loss | train 0.658 · val 0.662 |
145
 
146
- Training data is **generated, not scraped**, from a grammar in the repo. No licensing questions,
147
- and a learner can read the entire source of the training set.
148
 
149
  ## Limitations
150
 
 
9
  - from-scratch
10
  - interpretability
11
  - tiny
12
+ - experiment
13
  pipeline_tag: text-generation
14
  library_name: pytorch
15
  ---
16
 
17
  # Tiny SQL GPT
18
 
19
+ > **Personal learning project** (August 2026). The goal was to build a language model from scratch, end to end,
20
+ > small enough to understand every part of it, as a way to understand how the large ones work.
21
 
22
+ **An 841,216-parameter decoder-only transformer, trained from random weights on a laptop CPU in
23
+ about five minutes, on a toy SQL grammar.** On that toy task, all 500 sampled queries execute against a SQLite
24
+ database. That says more about how simple the task is than about the model (see Results).
25
 
26
  No pretrained weights. No `transformers` model classes. No API keys. The full architecture is
27
  ~200 readable lines of PyTorch.
28
 
29
  This model exists to be **understood**, not deployed. Its vocabulary is 155 tokens, which is the
30
  point: small enough that you can print the *entire* probability distribution at every generation
31
+ step.
32
 
33
  - **Code, evaluation harness and write-up:** https://github.com/sarathi-aiml/tiny-sql-gpt
34
  - **Plain-English explainer (no maths):** [`EXPLAIN.md`](https://github.com/sarathi-aiml/tiny-sql-gpt/blob/main/EXPLAIN.md)
 
84
  | validation loss | 0.662 | n/a |
85
 
86
  100% is a real measurement, but read it against the task: 14 query shapes, 3 tables, 155 tokens.
87
+ A model that saturates *this* shows the training loop works. It is not a text-to-SQL system.
88
+
89
+ Caveats:
90
+ - Only 16.6% of samples are novel. Most generated queries reproduce queries from the training set. The bigram
91
+ baseline is "novel" mostly because it produces invalid SQL.
92
+ - Validation loss uses the last 10% of the same generated corpus. With only 16,941 unique queries in 100,000
93
+ examples, many validation queries also appear in training, so train ≈ val loss isn't evidence of generalization.
94
+ The held-out `GROUP BY` probe below is the real generalization test, and the model fails it.
95
 
96
  ### Scaling
97
 
 
104
  | **tiny** (this model) | **841,216** | **100.0%** | **100.0%** |
105
  | small | 4,834,816 | 100.0% | 100.0% |
106
 
107
+ Syntax is cheap: 24K parameters is enough to write SQL that runs. The long-range dependency needs ~5x
108
+ more parameters, and shows up as a sharp jump between 24K and 124K. Above that, nothing improves: going
109
+ from 124K to 4.8M parameters buys 0.007 of validation loss and no change in behaviour. All four
110
+ converge to ~0.66, which I believe is close to the entropy of the data generator rather than a limit of the
111
+ models. I didn't compute the generator's entropy directly. Each size is a single seed.
112
 
113
  ---
114
 
 
118
  `GROUP BY` is always the column that appeared first in `SELECT`**, roughly 8 tokens earlier.
119
  Getting it right requires looking back, which is what attention is for.
120
 
121
+ **In a 1-layer model, one head learned it.** In `flat` (1 layer, 127K params),
122
  head L0H1 places **58.7% of its attention** on the `SELECT` column, 7.0x above uniform. Nobody
123
  designed or labelled it.
124
 
 
138
 
139
  A control prompt shows the context *is* being used (3.8x to 10.2x lift), it simply loses to a prior
140
  against tokens never seen in that slot. **Attention identifies the right source token; the output
141
+ prior overrules it.** It's a small, inspectable example of a hallucination-like failure (3 held-out pairs, so an
142
+ illustration, not a statistic).
143
 
144
  ---
145
 
 
151
  | architecture | decoder-only, 4 layers, 4 heads, 128 embedding, 64-token context |
152
  | tokenizer | word-level, 155 tokens |
153
  | optimizer | AdamW, lr 1e-3, cosine schedule, weight decay 0.01, grad clip 1.0 |
154
+ | steps | 3,000 · batch 64 × 64 tokens (~12.3M tokens seen, ~10 passes over the training split) · ~5 minutes on a laptop CPU |
155
  | final loss | train 0.658 · val 0.662 |
156
 
157
+ Training data is **generated, not scraped**, from a grammar in the repo, so a learner can read the entire
158
+ source of the training set.
159
 
160
  ## Limitations
161