KimMumu/qwen2.5-coder-7b-spider-qlora

Qwen2.5-Coder-7B-Instruct · Spider QLoRA adapter

A QLoRA adapter that fine-tunes Qwen/Qwen2.5-Coder-7B-Instruct for Text-to-SQL on the Spider training split. It was produced as condition 4 of a controlled comparison of RAG against fine-tuning, so its evaluation numbers come with the ablations that explain them.

Results — Spider dev (n=1034)

Scored with the official test-suite evaluator at plug_value=False, the setting leaderboard numbers use. Base and adapter were served identically (vLLM, AWQ base, temperature=0, max_tokens=256), so the adapter is the only difference between the first two rows.

Setup	Test-Suite Accuracy	Exact Match
Base model, no adapter	72.7% (752/1034)	56.5% (584/1034)
+ this adapter	75.7% (783/1034)	76.3% (789/1034)
+ this adapter + DB value grounding in the prompt	76.5% (791/1034)	76.6% (792/1034)
Cloud API reference (Gemini 3.6 Flash)	83.1% (859/1034)	80.5% (832/1034)

Exact Match rises far more than execution accuracy (+19.8 points against +3.0). Most of what fine-tuning transfers is Spider's house style — EXCEPT over NOT IN, grouping by key rather than name — which Exact Match rewards directly. Real correctness gains are the smaller number. Read the two metrics together; either alone misleads.

Prompt format — the adapter depends on it

Trained with this exact chat structure. A different prompt shape loses most of the benefit.

SYSTEM_PROMPT = (
    "You are a Text-to-SQL engine. Given a database schema and a question, "
    "output ONLY the SQL query that answers the question. "
    "No explanation, no markdown code fences, no trailing semicolon."
)

user = f"Schema:\n{schema}\n\nQuestion: {question}\nSQL:"

schema is CREATE TABLE statements including PK/FK lines, one per table, blank-line separated. Targets were single-line SQL with no trailing semicolon.

Usage

vLLM (how the reported numbers were produced)

from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest

llm = LLM(
    model="Qwen/Qwen2.5-Coder-7B-Instruct-AWQ",
    quantization="awq", dtype="float16",
    enable_lora=True, max_lora_rank=64,   # r=64; vLLM defaults to 16 and would reject it
)
out = llm.chat(
    [[{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": user}]],
    SamplingParams(temperature=0.0, max_tokens=256),
    lora_request=LoRARequest("spider", 1, "KimMumu/qwen2.5-coder-7b-spider-qlora"),
)

Pass a local directory instead of the repo id if your vLLM version does not resolve Hub ids for lora_path — hf download KimMumu/qwen2.5-coder-7b-spider-qlora --local-dir ./spider-lora.

transformers + peft

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct", dtype="float16")
model = PeftModel.from_pretrained(base, "KimMumu/qwen2.5-coder-7b-spider-qlora")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")

Training

Data	Spider train split (xlangai/spider), 7,000 examples; 82 dropped for exceeding 2048 tokens
Method	QLoRA — 4-bit NF4 double quantisation, fp16 compute
LoRA	r=64, alpha=16, dropout=0.05, on all seven projections (q/k/v/o/gate/up/down)
Trainable	~161M parameters
Schedule	1 epoch, ~433 steps, effective batch 16 (1 × grad-accum 16), lr 2e-4 cosine
Loss	assistant turn only; the prompt is masked out
Hardware	single Colab T4 (16GB)

train_others.json was excluded — its query distribution differs from Spider proper. Dev never entered training: Spider's train and dev database schemas are disjoint by design, and the export script re-checks that at runtime.

Limitations

• Quantisation mismatch. Trained against NF4 weights, evaluated with the adapter applied to an AWQ base, because AWQ cannot be fine-tuned and holding the base identical to the no-adapter baseline was the priority. Serving on a different base quantisation may shift results.
• Spider only. One benchmark, one epoch, one hyperparameter setting; no sweep was run. Generalisation to noisier schemas (BIRD) or other dialects is untested. SQLite dialect.
• Style is closer than competence. Exact Match nearly reaches the cloud reference while execution accuracy stays 68 examples behind, which places the remaining gap in semantics rather than surface form. Fine-tuning did not close it.
• Cannot know your data. Literal errors — writing 'Cat' where a column stores 'cat', which SQLite's case-sensitive = turns into zero rows — are not fixable by fine-tuning, since the values live only in the database. Injecting matched cell values into the prompt was worth a further +7/+8 examples on top of this adapter and is recommended alongside it.
• Not evaluated for safety, injection resistance, or use against untrusted schemas.

Licence & attribution

Adapter weights: Apache-2.0, matching the base model. Training data is the Spider dataset (CC BY-SA 4.0) — see yale-lily/spider. Cite Spider (Yu et al., EMNLP 2018) and the test-suite evaluator (Zhong et al., EMNLP 2020) if you build on the evaluation.