| ---
|
| license: apache-2.0
|
| language:
|
| - en
|
| library_name: transformers
|
| pipeline_tag: text-generation
|
| datasets:
|
| - HuggingFaceFW/fineweb-edu
|
| - bigcode/the-stack-dedup
|
| - gretelai/synthetic_text_to_sql
|
| tags:
|
| - sql
|
| - small-language-model
|
| - from-scratch
|
| - pretrained
|
| ---
|
|
|
| # TinySQL-125M-base
|
|
|
| The pretrained base model of TinySQL: a 125M-parameter decoder trained from random initialisation on 4.51B tokens of educational web text and SQL code. Everything was written from scratch: the tokenizer, the transformer, AdamW, the training loop and the data pipeline. It is a plain language model, not instruction-tuned. For text-to-SQL, use [adiitya0929/TinySQL-125M-WikiSQL](https://huggingface.co/adiitya0929/TinySQL-125M-WikiSQL), or fine-tune this model on your own task.
|
|
|
| - **Paper:** [TinySQL: A 125M-Parameter Text-to-SQL Model Built and Pretrained From Scratch on 4.5B Tokens](https://doi.org/10.5281/zenodo.23396770) (Zenodo, 2026)
|
| - **Code:** [github.com/aditya0929/tinysql](https://github.com/aditya0929/tinysql)
|
|
|
| ## Quick start
|
|
|
| ```python
|
| from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
| repo = "adiitya0929/TinySQL-125M-base"
|
| tok = AutoTokenizer.from_pretrained(repo)
|
| model = AutoModelForCausalLM.from_pretrained(repo)
|
|
|
| prompt = (
|
| "<schema>CREATE TABLE orders (id INT, customer TEXT, amount REAL, created DATE);</schema>\n"
|
| "<question>What is the total order amount for each customer?</question>\n"
|
| "<sql>"
|
| )
|
| ids = tok(prompt, return_tensors="pt").input_ids
|
| out = model.generate(ids, max_new_tokens=60, do_sample=False,
|
| eos_token_id=tok.convert_tokens_to_ids("</sql>"))
|
| print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))
|
| # SELECT customer, SUM(amount) FROM orders GROUP BY customer;
|
| ```
|
|
|
| The pretraining data included synthetic text-to-SQL examples in this `<schema>`/`<question>`/`<sql>` layout, so the base model already writes simple SQL. It was not tuned for accuracy, though, and it often adds conditions nobody asked for: for "How many employees are there?" it writes `SELECT COUNT(*) FROM employees WHERE salary > 50000;`.
|
|
|
| ## Training
|
|
|
| | | |
|
| |---|---|
|
| | Architecture | Llama-style decoder: 30 layers, hidden size 576, 9 query / 3 key-value heads (GQA), SwiGLU 1536, RoPE (theta 10,000), RMSNorm, tied embeddings, context 2048 |
|
| | Parameters | 125,077,824 |
|
| | Tokenizer | own byte-level BPE, 32,768 tokens, including special tokens `<schema>`, `<question>`, `<sql>` and their closing tags |
|
| | Data | 5.41B-token shards: FineWeb-Edu 71%, The Stack SQL subset 26.3% (quality-selected), Stack Exchange dba/datascience 2.2%, Gretel synthetic text-to-SQL 0.5%; cleaned, deduplicated (exact and MinHash-LSH) and decontaminated against WikiSQL dev/test |
|
| | Mix | 82/15/2.5/0.5% (web/SQL/Q&A/synthetic) for the first 85% of steps, then 54/36/8/2% |
|
| | Optimisation | own AdamW (0.9, 0.95), weight decay 0.1, peak LR 1.2e-3, 200 warmup steps, cosine to 1.2e-4, 524,288 tokens per step, bf16 |
|
| | Compute | 8,600 steps, about 46 hours on one NVIDIA L4 (27k tokens/s) |
|
|
|
| Final validation loss, in nats per token: FineWeb-Edu 2.94, SQL code 0.96, Stack Exchange 2.46, Gretel 1.05.
|
|
|
| The architecture matches Llama, so the weights are distributed in the standard `LlamaForCausalLM` format. The model was trained with the repository's own code. The export is checked to produce the same token ids and logits as the original implementation.
|
|
|
| ## Limitations
|
|
|
| A small base model trained on 4.5B tokens: its general knowledge and long-form coherence are limited, and it can produce false or nonsensical text. It is mainly useful as a starting point for fine-tuning on SQL or code tasks. It is English only.
|
|
|
| ## Training data licenses
|
|
|
| FineWeb-Edu (ODC-By 1.0), The Stack (dedup), SQL subset (permissively licensed source files, subject to The Stack's terms of use), Stack Exchange dba and datascience (CC BY-SA; 2.5, 3.0 or 4.0 depending on the post date), gretelai/synthetic_text_to_sql (Apache-2.0).
|
|
|
| ## Citation
|
|
|
| ```bibtex
|
| @techreport{jha2026tinysql,
|
| title = {{TinySQL}: A 125M-Parameter Text-to-SQL Model Built and Pretrained From Scratch on 4.5B Tokens},
|
| author = {Jha, Aditya Narayan},
|
| year = {2026},
|
| institution = {Zenodo},
|
| doi = {10.5281/zenodo.23396770},
|
| url = {https://doi.org/10.5281/zenodo.23396770}
|
| }
|
| ```
|
| |
