TinySQL release v1.0 (paper DOI 10.5281/zenodo.23396770)

README.md

@@ -0,0 +1,97 @@
+---
+license: apache-2.0
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+base_model: adiitya0929/TinySQL-125M-base
+datasets:
+- Salesforce/wikisql
+tags:
+- text-to-sql
+- sql
+- wikisql
+- small-language-model
+- from-scratch
+---
+
+# TinySQL-125M-WikiSQL
+
+A 125M-parameter text-to-SQL model. Given a table schema and a question, it writes the SQL query. Every part of it was built from scratch: the tokenizer, the transformer, the optimizer, the training loop and the data pipeline. It was pretrained on 4.5B tokens on one GPU, then fine-tuned on WikiSQL.
+
+- **Paper:** [TinySQL: A 125M-Parameter Text-to-SQL Model Built and Pretrained From Scratch on 4.5B Tokens](https://doi.org/10.5281/zenodo.23396770) (Zenodo, 2026)
+- **Code:** [github.com/aditya0929/tinysql](https://github.com/aditya0929/tinysql)
+- **Base model:** [adiitya0929/TinySQL-125M-base](https://huggingface.co/adiitya0929/TinySQL-125M-base)
+
+## Quick start
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+repo = "adiitya0929/TinySQL-125M-WikiSQL"
+tok = AutoTokenizer.from_pretrained(repo)
+model = AutoModelForCausalLM.from_pretrained(repo)
+
+prompt = (
+    '<schema>CREATE TABLE t ("Player" TEXT, "No." TEXT, "Nationality" TEXT, "Position" TEXT, '
+    '"Years in Toronto" TEXT, "School/Club Team" TEXT);</schema>\n'
+    "<question>What position does the player who played for butler cc (ks) play?</question>\n"
+    "<sql>"
+)
+ids = tok(prompt, return_tensors="pt").input_ids
+out = model.generate(ids, max_new_tokens=64)          # greedy; stops at </sql>
+print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))
+# SELECT "Position" FROM t WHERE "School/Club Team" = 'butler cc (ks)';
+```
+
+**Prompt format.** The model was trained on exactly this layout. The table is always called `t`. Column names are double-quoted, and each column is typed `REAL` (numeric) or `TEXT`. The question goes in `<question>…</question>`, and the prompt ends with `<sql>`. The model writes one `SELECT` statement and then `</sql>`.
+
+## Results (WikiSQL test, 15,878 questions)
+
+| Model | Pretraining tokens | Greedy | + Schema + values constraints |
+|---|---|---|---|
+| **TinySQL-125M (this model, mean of 3 seeds)** | **4.5B** | **77.95 ± 0.18** | **80.39 ± 0.20** |
+| GPT-2 124M | not published | 78.93 | 82.36 |
+| Pythia-160M | 300B | 60.78 | 67.63 |
+| SmolLM2-135M | 2T | 82.57 ± 0.37 | 85.22 ± 0.21 |
+| SmolLM2-360M | 4T | 87.03 ± 0.19 | 88.01 ± 0.12 |
+
+The table reports execution accuracy (%): the generated query is run against the table and its result compared with the gold query's. Every model was fine-tuned with the same recipe after its own learning-rate sweep. This checkpoint is seed 0, which scores 78.01 greedy and 80.44 constrained. The official WikiSQL evaluator gives 78.08 execution and 72.38 logical-form accuracy for it.
+
+`model.generate()` above is plain greedy decoding. The constrained numbers use the grammar-constrained decoder in the [GitHub repository](https://github.com/aditya0929/tinysql) (`decode/`). It only allows the table's real column names and values copied from the question, which gives 100% valid SQL.
+
+## Training
+
+| | |
+|---|---|
+| Architecture | Llama-style decoder: 30 layers, hidden size 576, 9 query / 3 key-value heads (GQA), SwiGLU 1536, RoPE, RMSNorm, tied embeddings, context 2048 |
+| Parameters | 125,077,824 |
+| Tokenizer | own byte-level BPE, 32,768 tokens |
+| Pretraining | 4.51B tokens (FineWeb-Edu, The Stack SQL, Stack Exchange, Gretel synthetic text-to-SQL), 8,600 steps, about 46 h on one NVIDIA L4 |
+| Fine-tuning | WikiSQL train (56,355 examples), loss on the SQL only, 3 epochs, batch 32, peak LR 5e-5, best epoch by dev execution accuracy |
+
+The architecture matches Llama, so the weights are distributed in the standard `LlamaForCausalLM` format. The model was built and trained with the repository's own code, not with `transformers`. The export is checked to produce the same token ids, logits and generated SQL as the original implementation.
+
+## Limitations
+
+- **WikiSQL shape only:** one table, one selected column, an optional aggregate, and up to four `AND` conditions. No joins, `GROUP BY`, `ORDER BY` or nesting. Do not expect it to handle Spider-style or production SQL.
+- **English only;** small model; greedy decoding can produce column names that don't exist. Use the constrained decoder from the repository to rule that out.
+- **Label noise:** WikiSQL's labels are noisy. A hand audit in the paper suggests that most remaining aggregation errors are label errors.
+- **Not for production databases** without review: generated SQL should never be executed with write permissions.
+
+## Training data licenses
+
+FineWeb-Edu (ODC-By 1.0), The Stack (dedup), SQL subset (permissively licensed source files, subject to The Stack's terms of use), Stack Exchange dba and datascience (CC BY-SA; 2.5, 3.0 or 4.0 depending on the post date), gretelai/synthetic_text_to_sql (Apache-2.0), WikiSQL (Salesforce, BSD-3-Clause). The pretraining data was decontaminated against WikiSQL dev and test questions.
+
+## Citation
+
+```bibtex
+@techreport{jha2026tinysql,
+  title       = {{TinySQL}: A 125M-Parameter Text-to-SQL Model Built and Pretrained From Scratch on 4.5B Tokens},
+  author      = {Jha, Aditya Narayan},
+  year        = {2026},
+  institution = {Zenodo},
+  doi         = {10.5281/zenodo.23396770},
+  url         = {https://doi.org/10.5281/zenodo.23396770}
+}
+```

config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 32756,
+  "dtype": "float32",
+  "eos_token_id": 32756,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 576,
+  "initializer_range": 0.02,
+  "intermediate_size": 1536,
+  "max_position_embeddings": 2048,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 9,
+  "num_hidden_layers": 30,
+  "num_key_value_heads": 3,
+  "pad_token_id": 32757,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.19.0",
+  "use_cache": true,
+  "vocab_size": 32768
+}

generation_config.json

@@ -0,0 +1,11 @@
+{
+  "bos_token_id": 32756,
+  "do_sample": false,
+  "eos_token_id": [
+    32767,
+    32756
+  ],
+  "max_new_tokens": 160,
+  "pad_token_id": 32757,
+  "transformers_version": "5.19.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23a996842298476fea097f43057bfda2d5f84ab7ba36ab75b69a86ace8a1e649
+size 500341640

tokenizer_config.json

@@ -0,0 +1,10 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 2048,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": null
+}