README.md

---

license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
base_model: adiitya0929/TinySQL-125M-base
datasets:
- Salesforce/wikisql
tags:
- text-to-sql
- sql
- wikisql
- small-language-model
- from-scratch
---


# TinySQL-125M-WikiSQL

A 125M-parameter text-to-SQL model. Given a table schema and a question, it writes the SQL query. Every part of it was built from scratch: the tokenizer, the transformer, the optimizer, the training loop and the data pipeline. It was pretrained on 4.5B tokens on one GPU, then fine-tuned on WikiSQL.

- **Paper:** [TinySQL: A 125M-Parameter Text-to-SQL Model Built and Pretrained From Scratch on 4.5B Tokens](https://doi.org/10.5281/zenodo.23396770) (Zenodo, 2026)
- **Code:** [github.com/aditya0929/tinysql](https://github.com/aditya0929/tinysql)
- **Base model:** [adiitya0929/TinySQL-125M-base](https://huggingface.co/adiitya0929/TinySQL-125M-base)

## Quick start

```python

from transformers import AutoModelForCausalLM, AutoTokenizer



repo = "adiitya0929/TinySQL-125M-WikiSQL"

tok = AutoTokenizer.from_pretrained(repo)

model = AutoModelForCausalLM.from_pretrained(repo)



prompt = (

    '<schema>CREATE TABLE t ("Player" TEXT, "No." TEXT, "Nationality" TEXT, "Position" TEXT, '

    '"Years in Toronto" TEXT, "School/Club Team" TEXT);</schema>\n'

    "<question>What position does the player who played for butler cc (ks) play?</question>\n"

    "<sql>"

)

ids = tok(prompt, return_tensors="pt").input_ids

out = model.generate(ids, max_new_tokens=64)          # greedy; stops at </sql>

print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

# SELECT "Position" FROM t WHERE "School/Club Team" = 'butler cc (ks)';

```

**Prompt format.** The model was trained on exactly this layout. The table is always called `t`. Column names are double-quoted, and each column is typed `REAL` (numeric) or `TEXT`. The question goes in `<question>…</question>`, and the prompt ends with `<sql>`. The model writes one `SELECT` statement and then `</sql>`.

## Results (WikiSQL test, 15,878 questions)

| Model | Pretraining tokens | Greedy | + Schema + values constraints |
|---|---|---|---|
| **TinySQL-125M (this model, mean of 3 seeds)** | **4.5B** | **77.95 ± 0.18** | **80.39 ± 0.20** |
| GPT-2 124M | not published | 78.93 | 82.36 |
| Pythia-160M | 300B | 60.78 | 67.63 |
| SmolLM2-135M | 2T | 82.57 ± 0.37 | 85.22 ± 0.21 |
| SmolLM2-360M | 4T | 87.03 ± 0.19 | 88.01 ± 0.12 |

The table reports execution accuracy (%): the generated query is run against the table and its result compared with the gold query's. Every model was fine-tuned with the same recipe after its own learning-rate sweep. This checkpoint is seed 0, which scores 78.01 greedy and 80.44 constrained. The official WikiSQL evaluator gives 78.08 execution and 72.38 logical-form accuracy for it.

`model.generate()` above is plain greedy decoding. The constrained numbers use the grammar-constrained decoder in the [GitHub repository](https://github.com/aditya0929/tinysql) (`decode/`). It only allows the table's real column names and values copied from the question, which gives 100% valid SQL.

## Training

| | |
|---|---|
| Architecture | Llama-style decoder: 30 layers, hidden size 576, 9 query / 3 key-value heads (GQA), SwiGLU 1536, RoPE, RMSNorm, tied embeddings, context 2048 |
| Parameters | 125,077,824 |
| Tokenizer | own byte-level BPE, 32,768 tokens |
| Pretraining | 4.51B tokens (FineWeb-Edu, The Stack SQL, Stack Exchange, Gretel synthetic text-to-SQL), 8,600 steps, about 46 h on one NVIDIA L4 |
| Fine-tuning | WikiSQL train (56,355 examples), loss on the SQL only, 3 epochs, batch 32, peak LR 5e-5, best epoch by dev execution accuracy |

The architecture matches Llama, so the weights are distributed in the standard `LlamaForCausalLM` format. The model was built and trained with the repository's own code, not with `transformers`. The export is checked to produce the same token ids, logits and generated SQL as the original implementation.

## Limitations

- **WikiSQL shape only:** one table, one selected column, an optional aggregate, and up to four `AND` conditions. No joins, `GROUP BY`, `ORDER BY` or nesting. Do not expect it to handle Spider-style or production SQL.
- **English only;** small model; greedy decoding can produce column names that don't exist. Use the constrained decoder from the repository to rule that out.
- **Label noise:** WikiSQL's labels are noisy. A hand audit in the paper suggests that most remaining aggregation errors are label errors.
- **Not for production databases** without review: generated SQL should never be executed with write permissions.

## Training data licenses

FineWeb-Edu (ODC-By 1.0), The Stack (dedup), SQL subset (permissively licensed source files, subject to The Stack's terms of use), Stack Exchange dba and datascience (CC BY-SA; 2.5, 3.0 or 4.0 depending on the post date), gretelai/synthetic_text_to_sql (Apache-2.0), WikiSQL (Salesforce, BSD-3-Clause). The pretraining data was decontaminated against WikiSQL dev and test questions.



## Citation



```bibtex

@techreport{jha2026tinysql,

  title       = {{TinySQL}: A 125M-Parameter Text-to-SQL Model Built and Pretrained From Scratch on 4.5B Tokens},

  author      = {Jha, Aditya Narayan},

  year        = {2026},

  institution = {Zenodo},

  doi         = {10.5281/zenodo.23396770},

  url         = {https://doi.org/10.5281/zenodo.23396770}

}

```