Hugging Face's logo

AlioLeuchtmann
/
flash-sqlite

Text Generation
Transformers
Safetensors
English
qwen3_5
image-text-to-text
text-to-sql
sql
sqlite
bird
self-verification
distillation
chain-of-thought
conversational
Eval Results (legacy)
Model card Files
xet
 Community
flash-sqlite / README.md
AlioLeuchtmann's picture
AlioLeuchtmann
Update README.md
1d4e636 verified
preview code
|
Raw History Blame Contribute Delete
14.6 kB
---
base_model: Qwen/Qwen3.5-2B
language:
- en
license: apache-2.0
pipeline_tag: text-generation
library_name: transformers
tags:
- text-to-sql
- sql
- sqlite
- bird
- self-verification
- distillation
- chain-of-thought
model-index:
- name: flash-sqlite
results:
- task:
type: text-to-sql
name: Text-to-SQL
dataset:
type: bird-bench
name: BIRD (hidden test set)
metrics:
- type: execution_accuracy
value: 73.73
name: Execution Accuracy (EX)
- type: soft_f1
value: 74.63
name: Soft F1
- type: r_ves
value: 67.17
name: R-VES
---
# flash-sqlite
**A 2B text-to-SQL model that writes SQL and checks its own candidates. It scores 73.73 EX on the hidden BIRD test set.**
flash-sqlite is [Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B), fully fine-tuned for SQLite text-to-SQL. One set of weights does two jobs:
- **Generator:** reasons step by step about the schema, then writes the SQL.
- **Judge:** given the schema, the question, a candidate query and its execution result, answers *Yes* or *No*. The probability of *Yes* is used to pick among the candidates.
Both roles use the same checkpoint. There is no auxiliary model, no embedding model and no external API. Under the BIRD organisers' ruling, this qualifies for the **single-model track**.
**Key facts**
- **No reinforcement learning:**
- flash-sqlite is trained with supervised fine-tuning only, on execution-verified traces.
- No RL, GRPO, DPO or reward model is involved.
- Several strong entries on the leaderboard, such as Arctic-Text2SQL-R1 and Databricks RLVR, are trained with RL.
- **Reasoning mode disabled:**
- The model runs in Qwen3.5's non-thinking mode (`enable_thinking=False`), for generation and for judging.
- It writes a short structured analysis followed by the SQL, with no long hidden reasoning chain.
- The judge outputs a single token.
- **One model, one GPU:** the same 2B checkpoint generates and verifies, with no external model or API.
## Official BIRD results (hidden test set)
The BIRD team scored the model on the hidden test set: 32 candidates per question plus self-judge, on a single A100 80 GB.
| metric | simple (949) | moderate (555) | challenging (285) | **total (1,789)** |
|---|---|---|---|---|
| Execution Accuracy (EX) | 80.19 | 71.53 | 56.49 | **73.73** |
| Soft F1 | 80.50 | 72.88 | 58.49 | **74.63** |
| R-VES | 73.54 | 64.29 | 51.55 | **67.17** |
### How it compares
**At 2B parameters, flash-sqlite scores higher on the BIRD test set than every frontier LLM baseline on the leaderboard**:
- GPT-5.5, Claude Opus 4.6, Claude 4.5 Sonnet, Qwen3-Coder-480B, GLM-4.7, DeepSeek-R1 and Kimi-K2-Thinking;
- and specialised open models up to 32B.
On the single-model leaderboard it is ahead of 50 of the 66 other entries. On the overall leaderboard, which includes multi-model pipelines and proprietary systems, it is ahead of more than 90.
All scores are **BIRD test-set execution accuracy**. Markers show how many candidates each entry samples per question, using the leaderboard's own categories:
- no marker: one candidate;
- \* **Few:** 2–7 candidates;
- \*\* **Many:** 8–32 candidates.
**Frontier and large general-purpose LLMs** (single-model leaderboard):
![flash-sqlite vs frontier LLM baselines on the BIRD test set](bird_test_frontier.png)
| model | provider | size | test EX |
|---|---|---|---|
| **flash-sqlite (this model)** | — | **2B** | **73.73\*\*** |
| GPT-5.5-xhigh | OpenAI | proprietary | 72.55 |
| Claude Opus 4.6 | Anthropic | proprietary | 70.15 |
| Qwen3-Coder-480B-A35B | Alibaba | 480B MoE | 68.14 |
| Claude 4.5 Sonnet | Anthropic | proprietary | 66.85 |
| GLM-4.7 | Zhipu AI | large open-weight | 62.94 |
| DeepSeek-R1 | DeepSeek | 671B MoE | 60.93 |
| Kimi-K2-Thinking | Moonshot AI | ~1T MoE | 59.87 |
| DeepSeek (2024 baseline) | DeepSeek | 236B MoE | 56.68 |
| Mistral (2024 baseline) | Mistral AI | 123B | 55.84 |
| GPT-4 | OpenAI | proprietary | 54.89 |
**Accuracy per parameter.** The chart shows every single-model entry with a published size. flash-sqlite lies on the Pareto frontier, so no smaller entry scores higher. It is the highest-scoring model at 2B parameters or fewer.
![BIRD test EX by model size, single-model leaderboard](bird_size_vs_score.png)
**Specialised text-to-SQL models** (single-model leaderboard, selection):
| model | size | test EX |
|---|---|---|
| Native mini | 4B | 76.86\*\* |
| Arctic-Text2SQL-R1-32B (Snowflake) | 32B | 73.84\* |
| **flash-sqlite (this model)** | **2B** | **73.73\*\*** |
| Databricks RLVR 32B | 32B | 73.56 |
| Arctic-Text2SQL-R1-14B (Snowflake) | 14B | 72.22\* |
| Arctic-Text2SQL-R1-7B (Snowflake) | 7B | 70.43\* |
| XiYanSQL-QwenCoder-32B (Alibaba) | 32B | 69.03\* |
flash-sqlite samples 32 candidates and selects one with its own judge (see [How it works](#how-it-works)). All figures are from the public leaderboard at [bird-bench.github.io](https://bird-bench.github.io), taken at the time of writing. The earlier version of this model (2B, majority vote over 7 candidates) scored 68.47\*.
## Speed and efficiency
32 candidates plus a judge sounds expensive. With a 2B model and the right serving setup, it is not.
- **One GPU, no API:** the bf16 weights need about 4.5 GB. Everything runs locally on a single GPU: generation, execution and judging. No data leaves the machine.
- **32 candidates in parallel:**
- The 32 samples per question come from one request (`n=32`), and vLLM batches them together.
- With **prefix caching**, the long schema prompt is prefilled once and shared by all 32 samples.
- The judge prompts of the same question also share their schema prefix.
- **Throughput:** about **6,400–7,600 output tokens/s** on one 96 GB workstation GPU, which is 8–11 finished candidates per second.
- **The judge is almost free:**
- It reads one token (P(Yes)) per result group instead of generating text.
- On BIRD dev, **38% of questions needed no judge call at all**, because all executable candidates agreed.
- On average there were **2.45 judge calls per question** (3,763 calls for 949 contested questions).
- The whole judge pass over BIRD dev took **about 30 seconds**.
- **End to end:** all 1,534 BIRD dev questions take **about 95 minutes**. That covers 49,000 candidates (about 85 minutes), plus executing them and judging (about 10 minutes), or about 3.7 s per question.
- **Greedy mode:** for latency-sensitive use, a single greedy answer still reaches 57.05 EX on dev. Majority voting over 32 candidates without the judge reaches 62.92.
## BIRD dev results
These come from the dev revision `bird-sql-dev-1106`: 1,532 questions with executable gold SQL, scored by execution accuracy with set comparison. The three candidate-based rows come from the same generation run; the greedy row comes from a separate single-answer run.
| inference | EX | simple | moderate | challenging |
|---|---|---|---|---|
| **32 candidates + self-judge** | **67.49** | 78.11 | 63.66 | 35.22 |
| 16 candidates + self-judge | 66.91 | 77.65 | 63.21 | 33.91 |
| majority vote over 32, no judge | 62.92 | 74.27 | 56.88 | 32.17 |
| greedy, single answer | 57.05 | | | |
On dev, the self-judge adds **4.6 EX** over plain majority voting with the same 32 candidates. Its gain grows with the number of candidates.
## How it works
For each question:
1. **Generate** 32 candidates at temperature 1.0, top_p 0.95, top_k 20, with a 2,048-token limit and non-thinking mode.
2. **Execute** each candidate read-only against the database, with a 30 s timeout.
- Candidates are grouped by their result set, compared as sets with `1 == 1.0`.
- Candidates that fail to run are dropped.
- A group's votes are the number of candidates in it.
3. **Self-judge** — only if more than one result group remains.
- The same model gets one prompt per group: the schema and question, the group's SQL, and a 5-row preview of its result.
- P(Yes) is read from the first generated token, restricted to *Yes* / *No* and renormalised.
4. **Select** the group with the highest **votes × P(Yes)**.
## Training
Training is **supervised fine-tuning only**: no reinforcement learning, no preference optimisation and no reward model. Every SQL target is **execution-verified**. A chain-of-thought trace was only kept if its final SQL, run on the real database, returned the gold result set.
**Stage 1: synthetic pre-training.**
- Data: 57,178 execution-verified synthetic examples, over generated databases and questions designed to match BIRD's style.
- Training: 2 epochs.
**Stage 2: joint generator + judge SFT.** 27,179 rows in total:
- **14,327 SQL generation traces** (BIRD train and synthetic databases):
- The traces follow a structured chain of thought: question and hint mapping, schema selection, join path, filters, aggregation, edge-case checks, then the final SQL.
- They were rejection-sampled from a Qwen 27B teacher.
- **8,314 pointwise critiques:**
- Each one judges a single candidate query and its execution result, and ends in a Yes/No verdict.
- The teacher wrote them without seeing the label. Only critiques whose verdict matched the execution-based label were kept.
- **4,538 listwise judge rows:**
- Several candidates (grouped by execution result) are compared, then the model chooses one or answers "none".
Hyperparameters for stage 2:
- Full fine-tune in bf16, 3 epochs.
- Learning rate 2e-5 (cosine schedule), 50 warmup steps.
- Effective batch size 32.
- NEFTune α = 5.
- Maximum sequence length 16,384.
The prompts contain per-column descriptions, read from each BIRD database's `database_description/*.csv` files.
## Usage
The model expects the OmniSQL-style prompt shown below and answers with its reasoning followed by a fenced SQL block. Always take the **last** ```sql block. Use **non-thinking mode** (`enable_thinking=False`).
### Prompt format
```text
Task Overview:
You are a data science expert. Below, you are provided with a database schema and a natural language question. Your task is to understand the schema and generate a valid SQL query to answer the question.
Database Engine:
SQLite
Database Schema:
{db_details}
This schema describes the database's structure, including tables, columns, primary keys, foreign keys, and any relevant relationships or constraints.
Question:
{question}
Hint: {evidence}
Instructions:
- Make sure you only output the information that is asked in the question. If the question asks for a specific column, make sure to only include that column in the SELECT clause, nothing more.
- The generated query should return all of the information asked in the question without any missing or extra information.
- Before generating the final SQL query, please think through the steps of how to write the query.
Output Format:
In your answer, please enclose the generated SQL query in a code block:
```sql
-- Your SQL query
```
Take a deep breath and think step by step to find the correct SQL query.
```
`{db_details}` contains, for each table:
- the `CREATE TABLE` statement;
- a comment block of column descriptions (where the database ships them);
- a comment block with 3 example rows.
### Generation (vLLM)
```python
import re
from transformers import AutoTokenizer
from vllm import LLM, SamplingParams
model_id = "AlioLeuchtmann/flash-sqlite"
tok = AutoTokenizer.from_pretrained(model_id)
llm = LLM(model=model_id, dtype="bfloat16")
def chat(prompt):
return tok.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False,
add_generation_prompt=True, enable_thinking=False)
def last_sql(text):
blocks = re.findall(r"```sql\s*(.*?)```", text, re.S)
return blocks[-1].strip() if blocks else ""
prompt = ... # the prompt above, filled with the schema, question and hint
# a single answer
greedy = llm.generate([chat(prompt)], SamplingParams(temperature=0.0, max_tokens=2048))
print(last_sql(greedy[0].outputs[0].text))
# 32 candidates for voting + self-judge
samples = llm.generate([chat(prompt)], SamplingParams(n=32, temperature=1.0, top_p=0.95, top_k=20, max_tokens=2048))
candidates = [last_sql(o.text) for o in samples[0].outputs]
```
### Self-judge
Execute the candidates and group them by result set. Then score one representative query per group with the judge prompt below. Use `max_tokens=1`, `allowed_token_ids=[id("Yes"), id("No")]` and `logprobs=2`, and renormalise P(Yes) over the two tokens.
```text
Task Overview:
You are a data science expert. Below, you are provided with a database schema, a natural language question, a candidate SQL query and the result of executing it. Your task is to decide whether the candidate SQL query answers the question correctly.
Database Engine:
SQLite
{the "Database Schema:" ... "Question:" part of the generation prompt}
Candidate SQL:
```sql
{sql}
```
Execution result ({nrows} rows, first 5 shown):
{preview}
Does the candidate SQL query answer the question correctly, returning exactly the requested information? Answer with one word: Yes or No.
```
Pick the group with the highest `votes × P(Yes)`. If two groups tie, prefer the one with more votes, then the one that appeared first.
### Tested environment
`vllm==0.25.1` and `transformers==5.14.1`, which support the Qwen3.5 hybrid architecture. A 2B model in bf16 needs about 4.5 GB of GPU memory for the weights. All 1,534 BIRD dev questions (49k samples plus judging) take about 95 minutes on a single 96 GB workstation GPU.
## Intended use and limitations
- **Built for:** SQLite text-to-SQL on analytical questions in the style of BIRD, in English, optionally with an evidence hint. It is not a general chat model.
- **The judge needs execution:** it scores a query *together with its result*, so the database has to be available at inference time. It has not been calibrated as a standalone classifier. Use it in the votes × P(Yes) combination it was tuned for.
- **Dialects:** trained and evaluated only on SQLite. Other dialects, such as PostgreSQL or BigQuery, will need adaptation.
- **Hard questions:** accuracy on the challenging tier (56.5 EX on test) is still well below the simple tier. Multi-step queries remain the main weakness of a 2B model.
- **Safety:** generated SQL should run read-only. This model was never trained to avoid writing or destructive statements.
## Acknowledgements
Built on Qwen3.5-2B by the Qwen team. Evaluated by the BIRD team ([bird-bench.github.io](https://bird-bench.github.io)). Developed by Alio Leuchtmann.