flash-sqlite-4b

A 4B text-to-SQL model that writes SQL, repairs its own failing queries and checks its own candidates. It scores 75.68 EX on the hidden BIRD test set.

flash-sqlite-4b is Qwen3.5-4B, fully fine-tuned for SQLite text-to-SQL. It is the successor of flash-sqlite (2B, 73.73 on test). One set of weights does three jobs:

• Generator: reasons step by step about the schema, then writes the SQL.
• Repairer: if one of its queries fails, times out or returns an empty or all-NULL result, it sees the execution feedback and answers again.
• Judge: given the schema, the question, a candidate query and its execution result, answers Yes or No. The probability of Yes is used to pick among the candidates.

All three roles use the same checkpoint. There is no auxiliary model, no embedding model and no external API. Repair feedback comes only from running the model's own SQL on the given database. Under the BIRD organisers' ruling, this qualifies for the single-model track.

Key facts

• No reinforcement learning: trained with supervised fine-tuning only, on execution-verified traces. No RL, GRPO, DPO or reward model is involved.
• Reasoning mode disabled: the model runs in Qwen3.5's non-thinking mode (enable_thinking=False) for generation, repair and judging. It writes a short structured analysis followed by the SQL; the judge outputs a single token.
• One model, one GPU: the same 4B checkpoint generates, repairs and verifies.

Official BIRD results (hidden test set)

The BIRD team scored the model on the hidden test set: 32 candidates per question, execution repair and self-judge, on a single A100 80 GB.

metric	total (1,789)
Execution Accuracy (EX)	75.68 (simple 82.93 / moderate 72.79 / challenging 57.19)
Soft F1	76.86
R-VES	68.10

How it compares

On the single-model leaderboard it shares 8th place with Databricks RLVR 32B and is ahead of 57 of the 65 other entries. On the overall leaderboard, which includes multi-model pipelines and proprietary systems, it is ahead of 105 entries. Among entries with 4B parameters or fewer, only Native mini (4B) scores higher.

All scores are BIRD test-set execution accuracy from the single-model leaderboard at bird-bench.github.io, taken at the time of writing. Markers show how many candidates each entry samples per question, using the leaderboard's own categories: no marker = one candidate, * Few = 2–7, ** Many = 8–32.

Frontier and large general-purpose LLMs:

model	size	test EX
flash-sqlite-4b (this model)	4B	75.68**
GPT-5.5-xhigh	proprietary	72.55
Claude Opus 4.6	proprietary	70.15
Qwen3-Coder-480B-A35B	480B MoE	68.14
Claude 4.5 Sonnet	proprietary	66.85
GLM-4.7	large open-weight	62.94
DeepSeek-R1	671B MoE	60.93
Kimi-K2-Thinking	~1T MoE	59.87

Specialised text-to-SQL models (selection):

model	size	test EX
SiriusAI-SQL-9B (Tencent)	9B	77.08**
Native mini	4B	76.86**
flash-sqlite-4b (this model)	4B	75.68**
Databricks RLVR 32B	32B	75.68*
Sophon-Text2SQL-32B (ByteDance)	32B	74.79**
Arctic-Text2SQL-R1-32B (Snowflake)	32B	73.84*
flash-sqlite (our 2B)	2B	73.73**
Arctic-Text2SQL-R1-7B (Snowflake)	7B	70.43*
XiYanSQL-QwenCoder-32B (Alibaba)	32B	69.03*

BIRD dev results

Dev revision bird-sql-dev-1106: 1,532 questions with executable gold SQL, execution accuracy with set comparison. All rows come from the same run of the inference package.

inference	EX	simple	moderate	challenging
32 candidates + repair + self-judge	71.41	80.79	70.88	37.39
16 candidates + repair + self-judge	71.15	80.44	70.43	37.83
majority vote over 32 repaired candidates, no judge	69.13	78.46	67.49	37.39

Without the repair step, the same model reaches 70.54 with 32 candidates + self-judge and 68.32 with plain majority voting. Repair adds about 0.6–0.9 EX on dev; the self-judge adds about 2.3 EX over majority voting.

How it works

For each question:

1. Generate 32 candidates at temperature 1.0, top_p 0.95, top_k 20, with a 2,048-token limit and non-thinking mode.
2. Repair: every distinct candidate runs read-only against the database (30 s timeout).
   • If it fails (SQL error, timeout, no SQL found) or returns an empty or all-NULL result, the model continues the chat with its own answer and the execution feedback: the error message, or the row count, column names and first 10 rows, plus a fixed instruction to check the result and either repeat the SQL or fix it.
   • Repair is greedy, at most 2 rounds per candidate. It stops early when the model repeats its SQL or the new SQL returns a non-empty result.
3. Group the repaired candidates by their result set, compared as sets with 1 == 1.0. Candidates that still fail are dropped. A group's votes are the number of candidates in it.
4. Self-judge, only if more than one result group remains: the same model gets one prompt per group with the schema and question, the group's SQL and a 5-row preview of its result. P(Yes) is read from the first generated token, restricted to Yes / No and renormalised.
5. Select the group with the highest votes × P(Yes).

On BIRD dev this costs on average 39.1 model calls per question: 32 generations, 4.7 repair calls (only 592 of 1,534 questions needed any repair; identical candidates share one repair) and 2.4 judge calls (54 % of questions needed no judge call because all executable candidates agreed).

Speed and efficiency

• One GPU, no API: the bf16 weights need about 9 GB. Generation, execution, repair and judging all run locally; no data leaves the machine.
• Batched sampling: the 32 samples per question come from one request (n=32); with prefix caching, the long schema prompt is prefilled once and shared by all samples and by the judge prompts of the same question.
• End to end: all 1,534 BIRD dev questions take about 4 h 15 min on one 96 GB workstation GPU: about 3 h to generate 49k candidates, about 1 h 10 min for the repair rounds, and a few minutes to execute and judge.

Training

Training is supervised fine-tuning only. Every SQL target is execution-verified: a chain-of-thought trace was only kept if its final SQL, run on the real database, returned the gold result set. The traces were rejection-sampled from a Qwen 27B teacher and follow a structured chain of thought: question and hint mapping, schema selection, join path, filters, aggregation, edge-case checks, then the final SQL.

Stage 1: structured-CoT SFT from Qwen3.5-4B on 20,940 traces: BIRD train 6,779, Spider 4,046 and 10,115 synthetic examples over generated databases (schema-linking traps, expert-style, noisy-hint and formula questions). The best checkpoint (epoch 1.75) was kept.

Stage 2: joint generator + judge SFT on 40,452 rows:

• SQL generation traces (BIRD train, Spider and synthetic databases), with per-column descriptions in the prompt;
• pointwise critiques: each judges a single candidate query and its execution result and ends in a Yes/No verdict. The teacher wrote them without seeing the label; only critiques whose verdict matched the execution-based label were kept;
• listwise judge rows: several candidates (grouped by execution result) are compared, then the model chooses one or answers "none".

Hyperparameters for stage 2: full fine-tune in bf16, learning rate 2e-5 (cosine schedule), 50 warmup steps, effective batch size 32, NEFTune α = 5, maximum sequence length 16,384. The released checkpoint is step 3,476 (about 2.75 of 3 planned epochs).

The prompts contain per-column descriptions read from each BIRD database's database_description/*.csv files. No BIRD dev or test data was used for training.

Usage

The model expects the OmniSQL-style prompt below and answers with its reasoning followed by a fenced SQL block. Always take the last ```sql block. Use **non-thinking mode** (enable_thinking=False).

Prompt format

Task Overview:
You are a data science expert. Below, you are provided with a database schema and a natural language question. Your task is to understand the schema and generate a valid SQL query to answer the question.

Database Engine:
SQLite

Database Schema:
{db_details}
This schema describes the database's structure, including tables, columns, primary keys, foreign keys, and any relevant relationships or constraints.

Question:
{question}

Hint: {evidence}

Instructions:
- Make sure you only output the information that is asked in the question. If the question asks for a specific column, make sure to only include that column in the SELECT clause, nothing more.
- The generated query should return all of the information asked in the question without any missing or extra information.
- Before generating the final SQL query, please think through the steps of how to write the query.

Output Format:
In your answer, please enclose the generated SQL query in a code block:
```sql
-- Your SQL query

Take a deep breath and think step by step to find the correct SQL query.

`{db_details}` is the schema as `CREATE TABLE` statements, with the column descriptions as comments and three example rows per table.

### Quick start (vLLM)

```python
from vllm import LLM, SamplingParams

llm = LLM("AlioLeuchtmann/flash-sqlite-4b", max_model_len=32768)
tok = llm.get_tokenizer()
prompt = tok.apply_chat_template([{"role": "user", "content": PROMPT}], tokenize=False,
                                 add_generation_prompt=True, enable_thinking=False)

# greedy single answer
out = llm.generate([prompt], SamplingParams(temperature=0.0, max_tokens=2048))
sql = out[0].outputs[0].text.split("```sql")[-1].split("```")[0].strip()

# 32 candidates for repair + voting + self-judge
cands = llm.generate([prompt], SamplingParams(n=32, temperature=1.0, top_p=0.95, top_k=20, max_tokens=2048))

vLLM ≥ 0.25 is required for the Qwen3.5 hybrid architecture.

Limitations

• Trained and evaluated on SQLite only; other dialects (PostgreSQL, BigQuery, Snowflake) are not covered.
• The schema prompt must fit the context window. Very large schemas need truncation or schema filtering.
• A sizeable share of the remaining dev errors comes from ambiguous or incorrect gold SQL in BIRD itself, so dev EX understates practical accuracy on clean questions, and gains from further tuning on dev are noisy.

License

Apache 2.0, the same as the base model Qwen/Qwen3.5-4B.