letsdatascience/qwen3.5-0.8b-spider-sql-lora

Qwen3.5-0.8B Spider SQL (LoRA adapter)

A LoRA adapter that teaches Qwen/Qwen3.5-0.8B to turn plain-English questions about a SQLite database into SQL. It was trained by distillation from a large teacher, DeepSeek V4 Flash, and then with reinforcement learning, on one Mac.

It is the final model of the Let's Data Science course Model Distillation: Teach a Small Model to Write SQL. Every step that produced it, and every number below, can be rebuilt with the open project lets-data-science/distillation-lab.

The adapter is in MLX format, for Apple-silicon Macs.

Results

Spider 1.0 dev set: 1,034 questions on 20 databases that none of the training questions use. One answer per question with greedy decoding. Graded by running the SQL, with the default rules of Spider's official checker, test-suite-sql-eval: DISTINCT is deleted from both queries, columns may come in any order, duplicate rows count, and row order counts when the gold query has ORDER BY. "Lenient" is the same set of rows with columns in order.

Model	Official	95% interval	Lenient
Qwen3.5-0.8B, no training	35.7%	32.8–38.6%	35.2%
Qwen3.5-2B, no training	56.8%	53.8–59.8%	54.2%
Qwen3.5-0.8B + LoRA on the teacher's checked answers	64.0%	61.1–66.9%	61.7%
Qwen3.5-0.8B + LoRA + reinforcement learning (this adapter)	71.9%	69.1–74.6%	68.4%
DeepSeek V4 Flash (the teacher)	80.8%	78.3–83.2%	76.9%
Qwen3.6-35B-A3B, 4-bit	82.6%	80.3–84.9%	79.1%

Reinforcement learning added 7.8 points to the distilled student: 118 questions gained against 37 lost (exact McNemar test, p < 0.0001). Intervals are 95% bootstrap intervals over the test questions; they do not cover training randomness (retraining the distilled student with another seed moved it from 64.0% to 65.0%).

How to use

The easiest way is the project's ask.py, which builds exactly the prompt the model was trained on, runs the generated SQL read-only on your database and prints the rows:

git clone https://github.com/lets-data-science/distillation-lab
cd distillation-lab
make setup                                  # Python environment (needs uv)
uv run hf download Qwen/Qwen3.5-0.8B --local-dir models/base/Qwen3.5-0.8B
uv run hf download letsdatascience/qwen3.5-0.8b-spider-sql-lora --local-dir models/adapters/final
scripts/download_data.sh spider             # the Spider databases, about 206 MB
uv run python scripts/ask.py --db data/spider/database/concert_singer/concert_singer.sqlite \
  "Which singers are older than 40?"

To call it from your own code with mlx-lm, give it the same prompt it was trained on: a fixed system line, then the database's CREATE TABLE statements with three example rows per table, the question, and a format line. Turn Qwen3.5's thinking mode off.

from mlx_lm import load, generate

model, tokenizer = load("Qwen/Qwen3.5-0.8B", adapter_path="models/adapters/final")

system = "You are an expert in SQLite. Given a database schema and a question, write one SQLite query that answers it."
schema = "..."  # CREATE TABLE statements, each followed by /* Example rows from <table> (<columns>): (...); (...); (...) */
question = "Which singers are older than 40?"
user = f"Database schema:\n{schema}\n\nQuestion: {question}\n\nReply with the SQL query only, inside a ```sql code block."

prompt = tokenizer.apply_chat_template(
    [{"role": "system", "content": system}, {"role": "user", "content": user}],
    add_generation_prompt=True, enable_thinking=False,
)
print(generate(model, tokenizer, prompt, max_tokens=320))

src/distill_lab/prompts.py in the project builds the schema text exactly as in training.

Training

1. Teacher answers. DeepSeek V4 Flash answered 2,000 questions sampled from Spider's training set (seed 7), in the SQL-only format above, at temperature 0.3, with up to three retries at temperature 0.9 for questions it got wrong. Each answer was run on its database and kept only if it returned the same rows as Spider's gold query (lenient rule). 1,665 answers were kept: 1,581 for training and 84 for validation. The teacher's answers cost $0.106 in total.
2. LoRA fine-tuning with mlx-lm: rank 16, scale 2.0, all layers, learning rate 1e-4, 2 epochs, batch 1 with 8-step gradient accumulation, loss on the answer only, maximum length 8,192 tokens. About 43 minutes on an Apple M5 Max (64 GB).
3. Reinforcement learning with mlx-lm-lora 5.8.6 (GRPO): 400 steps, 2 questions per step, 8 sampled tries per question at temperature 1.0, learning rate 1e-5, clip range 0.2 to 0.28, no KL penalty. Reward 1 when a try's SQL returns the gold query's rows (lenient rule), otherwise 0. Each try's advantage is its reward minus its group's mean, divided by the group's standard deviation. The questions came from Spider's training set, excluding the 2,000 above.

Limitations

• Spider only. Spider's databases are small and tidy. The model was not scored on BIRD or on real-world databases, which are usually harder.
• Possible contamination. Spider has been public since 2018 and may be in the base model's pre-training data.
• The reward needs gold queries. The reinforcement-learning step checked every try against Spider's reference query.
• SQLite and English only. Other SQL dialects and languages are untested.
• About 28% of test answers are still wrong. Read the SQL and check the rows before you rely on an answer.

License and attribution

The adapter is released under the Apache-2.0 license, the license of its base model, Qwen/Qwen3.5-0.8B.

It was trained on questions, schemas and gold queries from Spider 1.0 (Yu et al., 2018), which is licensed under CC BY-SA 4.0, and on answers written by DeepSeek V4 Flash (deepseek-ai/DeepSeek-V4-Flash-0731, MIT license). The grading rules follow test-suite-sql-eval (Zhong, Yu and Klein, 2020; Apache-2.0).

@inproceedings{yu2018spider,
  title     = {Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task},
  author    = {Yu, Tao and Zhang, Rui and Yang, Kai and Yasunaga, Michihiro and Wang, Dongxu and Li, Zifan and Ma, James and Li, Irene and Yao, Qingning and Roman, Shanelle and Zhang, Zilin and Radev, Dragomir},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year      = {2018}
}

@inproceedings{zhong2020semantic,
  title     = {Semantic Evaluation for Text-to-SQL with Distilled Test Suites},
  author    = {Zhong, Ruiqi and Yu, Tao and Klein, Dan},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing},
  year      = {2020}
}

Made by Let's Data Science.