---
base_model: Qwen/Qwen3.5-4B
language:
- en
license: apache-2.0
pipeline_tag: text-generation
library_name: transformers
tags:
- text-to-sql
- sql
- sqlite
- bird
- self-verification
- self-repair
- distillation
- chain-of-thought
model-index:
- name: flash-sqlite-4b
  results:
  - task:
      type: text-to-sql
      name: Text-to-SQL
    dataset:
      type: bird-bench
      name: BIRD (hidden test set)
    metrics:
    - type: execution_accuracy
      value: 75.68
      name: Execution Accuracy (EX)
    - type: soft_f1
      value: 76.86
      name: Soft F1
    - type: r_ves
      value: 68.10
      name: R-VES
---

# flash-sqlite-4b

**A 4B text-to-SQL model that writes SQL, repairs its own failing queries and checks its own candidates. It scores 75.68 EX on the hidden BIRD test set.**

flash-sqlite-4b is [Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B), fully fine-tuned for SQLite text-to-SQL. It is the successor of [flash-sqlite](https://huggingface.co/AlioLeuchtmann/flash-sqlite) (2B, 73.73 on test). One set of weights does three jobs:

- **Generator:** reasons step by step about the schema, then writes the SQL.
- **Repairer:** if one of its queries fails, times out or returns an empty or all-NULL result, it sees the execution feedback and answers again.
- **Judge:** given the schema, the question, a candidate query and its execution result, answers *Yes* or *No*. The probability of *Yes* is used to pick among the candidates.

All three roles use the same checkpoint. There is no auxiliary model, no embedding model and no external API. Repair feedback comes only from running the model's own SQL on the given database. Under the BIRD organisers' ruling, this qualifies for the **single-model track**.

**Key facts**

- **No reinforcement learning:** trained with supervised fine-tuning only, on execution-verified traces. No RL, GRPO, DPO or reward model is involved.
- **Reasoning mode disabled:** the model runs in Qwen3.5's non-thinking mode (`enable_thinking=False`) for generation, repair and judging. It writes a short structured analysis followed by the SQL; the judge outputs a single token.
- **One model, one GPU:** the same 4B checkpoint generates, repairs and verifies.

## Official BIRD results (hidden test set)

The BIRD team scored the model on the hidden test set: 32 candidates per question, execution repair and self-judge, on a single A100 80 GB.

| metric | total (1,789) |
|---|---|
| **Execution Accuracy (EX)** | **75.68** (simple 82.93 / moderate 72.79 / challenging 57.19) |
| Soft F1 | 76.86 |
| R-VES | 68.10 |

### How it compares

On the **single-model leaderboard** it shares 8th place with Databricks RLVR 32B and is ahead of 57 of the 65 other entries. On the overall leaderboard, which includes multi-model pipelines and proprietary systems, it is ahead of 105 entries. Among entries with 4B parameters or fewer, only Native mini (4B) scores higher.

All scores are **BIRD test-set execution accuracy** from the single-model leaderboard at [bird-bench.github.io](https://bird-bench.github.io), taken at the time of writing. Markers show how many candidates each entry samples per question, using the leaderboard's own categories: no marker = one candidate, \* **Few** = 2–7, \*\* **Many** = 8–32.

**Frontier and large general-purpose LLMs:**

| model | size | test EX |
|---|---|---|
| **flash-sqlite-4b (this model)** | **4B** | **75.68\*\*** |
| GPT-5.5-xhigh | proprietary | 72.55 |
| Claude Opus 4.6 | proprietary | 70.15 |
| Qwen3-Coder-480B-A35B | 480B MoE | 68.14 |
| Claude 4.5 Sonnet | proprietary | 66.85 |
| GLM-4.7 | large open-weight | 62.94 |
| DeepSeek-R1 | 671B MoE | 60.93 |
| Kimi-K2-Thinking | ~1T MoE | 59.87 |

**Specialised text-to-SQL models (selection):**

| model | size | test EX |
|---|---|---|
| SiriusAI-SQL-9B (Tencent) | 9B | 77.08\*\* |
| Native mini | 4B | 76.86\*\* |
| **flash-sqlite-4b (this model)** | **4B** | **75.68\*\*** |
| Databricks RLVR 32B | 32B | 75.68\* |
| Sophon-Text2SQL-32B (ByteDance) | 32B | 74.79\*\* |
| Arctic-Text2SQL-R1-32B (Snowflake) | 32B | 73.84\* |
| flash-sqlite (our 2B) | 2B | 73.73\*\* |
| Arctic-Text2SQL-R1-7B (Snowflake) | 7B | 70.43\* |
| XiYanSQL-QwenCoder-32B (Alibaba) | 32B | 69.03\* |

## BIRD dev results

Dev revision `bird-sql-dev-1106`: 1,532 questions with executable gold SQL, execution accuracy with set comparison. All rows come from the same run of the inference package.

| inference | EX | simple | moderate | challenging |
|---|---|---|---|---|
| **32 candidates + repair + self-judge** | **71.41** | 80.79 | 70.88 | 37.39 |
| 16 candidates + repair + self-judge | 71.15 | 80.44 | 70.43 | 37.83 |
| majority vote over 32 repaired candidates, no judge | 69.13 | 78.46 | 67.49 | 37.39 |

Without the repair step, the same model reaches 70.54 with 32 candidates + self-judge and 68.32 with plain majority voting. Repair adds about 0.6–0.9 EX on dev; the self-judge adds about 2.3 EX over majority voting.

## How it works

For each question:

1. **Generate** 32 candidates at temperature 1.0, top_p 0.95, top_k 20, with a 2,048-token limit and non-thinking mode.
2. **Repair:** every distinct candidate runs read-only against the database (30 s timeout).
   - If it fails (SQL error, timeout, no SQL found) or returns an empty or all-NULL result, the model continues the chat with its own answer and the execution feedback: the error message, or the row count, column names and first 10 rows, plus a fixed instruction to check the result and either repeat the SQL or fix it.
   - Repair is greedy, at most **2 rounds** per candidate. It stops early when the model repeats its SQL or the new SQL returns a non-empty result.
3. **Group** the repaired candidates by their result set, compared as sets with `1 == 1.0`. Candidates that still fail are dropped. A group's votes are the number of candidates in it.
4. **Self-judge**, only if more than one result group remains: the same model gets one prompt per group with the schema and question, the group's SQL and a 5-row preview of its result. P(Yes) is read from the first generated token, restricted to *Yes* / *No* and renormalised.
5. **Select** the group with the highest **votes × P(Yes)**.

On BIRD dev this costs on average **39.1 model calls per question**: 32 generations, 4.7 repair calls (only 592 of 1,534 questions needed any repair; identical candidates share one repair) and 2.4 judge calls (54 % of questions needed no judge call because all executable candidates agreed).

## Speed and efficiency

- **One GPU, no API:** the bf16 weights need about 9 GB. Generation, execution, repair and judging all run locally; no data leaves the machine.
- **Batched sampling:** the 32 samples per question come from one request (`n=32`); with prefix caching, the long schema prompt is prefilled once and shared by all samples and by the judge prompts of the same question.
- **End to end:** all 1,534 BIRD dev questions take about **4 h 15 min** on one 96 GB workstation GPU: about 3 h to generate 49k candidates, about 1 h 10 min for the repair rounds, and a few minutes to execute and judge.

## Training

Training is **supervised fine-tuning only**. Every SQL target is **execution-verified**: a chain-of-thought trace was only kept if its final SQL, run on the real database, returned the gold result set. The traces were rejection-sampled from a Qwen 27B teacher and follow a structured chain of thought: question and hint mapping, schema selection, join path, filters, aggregation, edge-case checks, then the final SQL.

**Stage 1: structured-CoT SFT** from Qwen3.5-4B on 20,940 traces: BIRD train 6,779, Spider 4,046 and 10,115 synthetic examples over generated databases (schema-linking traps, expert-style, noisy-hint and formula questions). The best checkpoint (epoch 1.75) was kept.

**Stage 2: joint generator + judge SFT** on 40,452 rows:
- **SQL generation traces** (BIRD train, Spider and synthetic databases), with per-column descriptions in the prompt;
- **pointwise critiques:** each judges a single candidate query and its execution result and ends in a Yes/No verdict. The teacher wrote them without seeing the label; only critiques whose verdict matched the execution-based label were kept;
- **listwise judge rows:** several candidates (grouped by execution result) are compared, then the model chooses one or answers "none".

Hyperparameters for stage 2: full fine-tune in bf16, learning rate 2e-5 (cosine schedule), 50 warmup steps, effective batch size 32, NEFTune α = 5, maximum sequence length 16,384. The released checkpoint is step 3,476 (about 2.75 of 3 planned epochs).

The prompts contain per-column descriptions read from each BIRD database's `database_description/*.csv` files. No BIRD dev or test data was used for training.

## Usage

The model expects the OmniSQL-style prompt below and answers with its reasoning followed by a fenced SQL block. Always take the **last** ```sql block. Use **non-thinking mode** (`enable_thinking=False`).

### Prompt format

```text
Task Overview:
You are a data science expert. Below, you are provided with a database schema and a natural language question. Your task is to understand the schema and generate a valid SQL query to answer the question.

Database Engine:
SQLite

Database Schema:
{db_details}
This schema describes the database's structure, including tables, columns, primary keys, foreign keys, and any relevant relationships or constraints.

Question:
{question}

Hint: {evidence}

Instructions:
- Make sure you only output the information that is asked in the question. If the question asks for a specific column, make sure to only include that column in the SELECT clause, nothing more.
- The generated query should return all of the information asked in the question without any missing or extra information.
- Before generating the final SQL query, please think through the steps of how to write the query.

Output Format:
In your answer, please enclose the generated SQL query in a code block:
```sql
-- Your SQL query
```

Take a deep breath and think step by step to find the correct SQL query.
```

`{db_details}` is the schema as `CREATE TABLE` statements, with the column descriptions as comments and three example rows per table.

### Quick start (vLLM)

```python
from vllm import LLM, SamplingParams

llm = LLM("AlioLeuchtmann/flash-sqlite-4b", max_model_len=32768)
tok = llm.get_tokenizer()
prompt = tok.apply_chat_template([{"role": "user", "content": PROMPT}], tokenize=False,
                                 add_generation_prompt=True, enable_thinking=False)

# greedy single answer
out = llm.generate([prompt], SamplingParams(temperature=0.0, max_tokens=2048))
sql = out[0].outputs[0].text.split("```sql")[-1].split("```")[0].strip()

# 32 candidates for repair + voting + self-judge
cands = llm.generate([prompt], SamplingParams(n=32, temperature=1.0, top_p=0.95, top_k=20, max_tokens=2048))
```

vLLM ≥ 0.25 is required for the Qwen3.5 hybrid architecture.

## Limitations

- Trained and evaluated on **SQLite** only; other dialects (PostgreSQL, BigQuery, Snowflake) are not covered.
- The schema prompt must fit the context window. Very large schemas need truncation or schema filtering.
- A sizeable share of the remaining dev errors comes from ambiguous or incorrect gold SQL in BIRD itself, so dev EX understates practical accuracy on clean questions, and gains from further tuning on dev are noisy.

## License

Apache 2.0, the same as the base model [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B).