File size: 4,938 Bytes
4aa4cb6 9a53da7 4aa4cb6 9a53da7 4aa4cb6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 | ---
base_model: Qwen/Qwen2.5-Coder-7B-Instruct
library_name: peft
license: apache-2.0
language:
- en
pipeline_tag: text-generation
tags:
- text-to-sql
- sql
- lora
- peft
- qwen2
---
# SQLForge — Qwen2.5-Coder-7B SFT (Text-to-SQL)
A LoRA adapter that fine-tunes **Qwen2.5-Coder-7B-Instruct** for Text-to-SQL on a
TPC-H-derived schema with custom domain conventions. Supervised fine-tuning raised
execution accuracy on a held-out 55-query benchmark from **54.5% → 74.5%**, closing 54%
of the gap to a Claude-based production agent (91.7%) — while general SQL ability stayed
intact (see Generalization).
Full project, training code, and evaluation harness:
**[github.com/ShahaDeven/sql-forge](https://github.com/ShahaDeven/sql-forge)**
> This is a **domain adapter**, not a general Text-to-SQL model. It is trained on one
> schema's conventions (categorical literals like `churn_risk = 'HIGH_RISK'`, revenue via
> `total_value * (1 - promo_reduction)`, canonical customer-side join paths). It expects
> the specific system prompt it was trained on — see **Serving contract** below.
## Results (55-query execution-accuracy suite)
Graded by result-set comparison against gold on DuckDB (TPC-H SF=0.1) — the same grader
that scores the production Claude agent at 91.7%.
| Model | Execution accuracy | Valid SQL |
|---|---:|---:|
| Qwen2.5-Coder-7B (base, zero-shot) | 54.5% | 90.9% |
| Qwen2.5-Coder-7B (base, 3-shot) | 61.8% | 94.5% |
| **This adapter (SFT, zero-shot)** | **74.5%** | **98.2%** |
| Claude (full production agent) | 91.7%¹ | — |
¹ Full agent: retrieval + few-shot + a 3-attempt self-healing loop. A product comparison,
not a like-for-like model comparison.
**Per-tier (SFT):** simple_select 10/10 · aggregation 10/10 · single_join 8/10 ·
window_function 6/10 · multi_hop 5/10 · simulation 2/5.
## Generalization (Spider zero-shot)
Run zero-shot on **Spider dev (1,032 queries, 20 unseen SQLite databases)** with a generic
prompt — no house-style rules — to test whether fine-tuning damaged general SQL ability:
| Model | Spider dev accuracy |
|---|---:|
| Qwen2.5-Coder-7B (base) | 82.4% |
| **This adapter (SFT)** | **82.7%** |
**+20pp in-domain for +0.3pp (i.e. zero, within noise) out-of-domain.** No catastrophic
forgetting: the adapter learned *this schema's conventions*, not a narrower notion of SQL.
## Usage
```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = "Qwen/Qwen2.5-Coder-7B-Instruct"
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "devenshah21/sqlforge-qwen7b-sft")
tok = AutoTokenizer.from_pretrained(base)
messages = [
{"role": "system", "content": HOUSE_STYLE_SYSTEM_PROMPT}, # see Serving contract
{"role": "user", "content": "Which region has the lowest revenue?"},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(prompt, return_tensors="pt").to(model.device),
max_new_tokens=512, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))
```
Serve with vLLM (as used for all evals):
```bash
vllm serve Qwen/Qwen2.5-Coder-7B-Instruct \
--enable-lora --lora-modules sqlforge=devenshah21/sqlforge-qwen7b-sft \
--max-lora-rank 32 --port 8000
```
### Serving contract
The adapter was trained **and** benchmarked (74.5%) with **one specific prompt shape**: a
house-style system prompt (schema + the categorical-literal / revenue-formula / join-path
rules) and the user question, **zero-shot, temperature 0**. Sending few-shot examples or a
different prompt is an unmeasured configuration and will not reproduce the numbers above —
fine-tuning already absorbed what few-shot was teaching. The exact prompt builder lives in
[`phase0/run_baseline.py`](https://github.com/ShahaDeven/sql-forge) (`build_system_prompt`).
## Training
| | |
|---|---|
| Method | bf16 LoRA (no quantization) |
| LoRA | r=32, α=64, dropout 0.05, all 7 projections (q,k,v,o,gate,up,down) |
| Data | 2,314 execution-validated synthetic pairs (distilled from Claude, gated on DuckDB execution + house-style checks) |
| Schedule | 2 epochs, lr 2e-4 cosine, effective batch 16, max_seq 2048, completion-only loss |
| Hardware | 1× A40 (48GB) |
## Limitations
- **Single schema.** Conventions are specific to this TPC-H-derived database; the Spider
result shows general SQL is intact, but in-domain gains do not transfer to other schemas.
- **Residual failures** concentrate in a categorical-literal bug (`churn_risk='High'` vs
`'HIGH_RISK'`), output-column shape, and simulation/CTE format — see the project repo's
failure-mode analysis.
- Sibling adapters (DPO, and a 1.5B SFT/GRPO track) are documented in the repo; neither
beat this SFT checkpoint, which is the deployed model.
### Framework versions
- PEFT 0.13.2
|