slmsql-qwen2.5-coder-1.5b

A LoRA fine-tune of Qwen2.5-Coder-1.5B-Instruct for Text-to-SQL, trained with a two-stage recipe (broad synthetic foundation → in-domain grounding) and evaluated on two benchmarks to prove generalization, not just benchmark-fit.

Results (execution accuracy, single-model greedy)

Benchmark	EX	Notes
Un-tuned base	17.4%	Qwen2.5-Coder-1.5B-Instruct baseline (BIRD dev)
BIRD dev	36.9%	the hard benchmark (+19.5 over base)
Spider dev	82.0%	generalization to unseen databases

The high Spider score (on databases the model never trained on) confirms the model learned SQL generally rather than specializing to one benchmark — the point of the two-stage recipe. For reference, this Spider score is competitive with SLM-SQL-1.5B (~76.7%, which additionally used reinforcement learning).

Trained for ~$25–30 of rented GPU. This is a budget/reproducibility result, not frontier SOTA — see the full analysis in the project repo.

Training recipe (two-stage)

1. Stage 1 — Foundation (breadth): LoRA SFT on ~100k execution-filtered SynSQL-2.5M examples (16,583 synthetic databases) → learns to generalize across arbitrary schemas.
2. Stage 2 — Grounding (in-domain): continue-train on BIRD train + Spider train (~14k real examples with evidence) → polishes on real, hard questions.

Key settings: LoRA rank 32 / alpha 64, all-linear target modules, LR 2e-4 (stage 1) / 5e-5 (stage 2), seq 5120, bf16, assistant-only loss, execution-filtered data, CoT-then-SQL targets, best-checkpoint selection on downstream EX.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "Qwen/Qwen2.5-Coder-1.5B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto")
model = PeftModel.from_pretrained(model, "Anurich/slmsql-qwen2.5-coder-1.5b")

messages = [
  {"role": "system", "content": "You are an expert data analyst who translates "
   "questions into correct SQLite queries. Think step by step, then output the "
   "final query in a ```sql block. Wrap identifiers with spaces in backticks."},
  {"role": "user", "content": "Given the following database schema:\n\n"
   "CREATE TABLE employees (id INTEGER, name TEXT, salary INTEGER);\n\n"
   "Question: Who are the 3 highest-paid employees?\n\n"
   "Write a single SQLite query that answers the question."},
]
prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
out = model.generate(**tok(prompt, return_tensors="pt").to(model.device), max_new_tokens=512)
print(tok.decode(out[0], skip_special_tokens=True))

See the repo's prompt.py for the exact training prompt format (schema + optional evidence + optional matched DB values → reasoning → SQL).

Evaluation method

Execution Accuracy: run the predicted SQL and the gold SQL against the real SQLite database, compare result sets (order-insensitive). BIRD dev on a stratified 149-example slice; Spider dev on 200 examples with real-data DBs. Standard single-model, single-inference greedy decoding.

Limitations

• 1.5B scale — complex multi-join / nested queries remain hard (reflected in the BIRD vs Spider gap; BIRD is substantially harder).
• Numbers are on eval slices (149 BIRD / 200 Spider), so ±~4% sampling noise.
• BIRD dev itself has documented ~50% gold-annotation noise; small margins are within benchmark error.