README.md

---
base_model: Qwen/Qwen2.5-Coder-7B-Instruct
library_name: peft
license: apache-2.0
language:
- en
pipeline_tag: text-generation
tags:
- text-to-sql
- sql
- lora
- peft
- qwen2
---

# SQLForge — Qwen2.5-Coder-7B SFT (Text-to-SQL)

A LoRA adapter that fine-tunes **Qwen2.5-Coder-7B-Instruct** for Text-to-SQL on a
TPC-H-derived schema with custom domain conventions. Supervised fine-tuning raised
execution accuracy on a held-out 55-query benchmark from **54.5% → 74.5%**, closing 54%
of the gap to a Claude-based production agent (91.7%) — while general SQL ability stayed
intact (see Generalization).

Full project, training code, and evaluation harness:
**[github.com/ShahaDeven/sql-forge](https://github.com/ShahaDeven/sql-forge)**

> This is a **domain adapter**, not a general Text-to-SQL model. It is trained on one
> schema's conventions (categorical literals like `churn_risk = 'HIGH_RISK'`, revenue via
> `total_value * (1 - promo_reduction)`, canonical customer-side join paths). It expects
> the specific system prompt it was trained on — see **Serving contract** below.

## Results (55-query execution-accuracy suite)

Graded by result-set comparison against gold on DuckDB (TPC-H SF=0.1) — the same grader
that scores the production Claude agent at 91.7%.

| Model | Execution accuracy | Valid SQL |
|---|---:|---:|
| Qwen2.5-Coder-7B (base, zero-shot) | 54.5% | 90.9% |
| Qwen2.5-Coder-7B (base, 3-shot) | 61.8% | 94.5% |
| **This adapter (SFT, zero-shot)** | **74.5%** | **98.2%** |
| Claude (full production agent) | 91.7%¹ | — |

¹ Full agent: retrieval + few-shot + a 3-attempt self-healing loop. A product comparison,
not a like-for-like model comparison.

**Per-tier (SFT):** simple_select 10/10 · aggregation 10/10 · single_join 8/10 ·
window_function 6/10 · multi_hop 5/10 · simulation 2/5.

## Generalization (Spider zero-shot)

Run zero-shot on **Spider dev (1,032 queries, 20 unseen SQLite databases)** with a generic
prompt — no house-style rules — to test whether fine-tuning damaged general SQL ability:

| Model | Spider dev accuracy |
|---|---:|
| Qwen2.5-Coder-7B (base) | 82.4% |
| **This adapter (SFT)** | **82.7%** |

**+20pp in-domain for +0.3pp (i.e. zero, within noise) out-of-domain.** No catastrophic
forgetting: the adapter learned *this schema's conventions*, not a narrower notion of SQL.

## Usage

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "Qwen/Qwen2.5-Coder-7B-Instruct"
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "devenshah21/sqlforge-qwen7b-sft")
tok = AutoTokenizer.from_pretrained(base)

messages = [
    {"role": "system", "content": HOUSE_STYLE_SYSTEM_PROMPT},  # see Serving contract
    {"role": "user", "content": "Which region has the lowest revenue?"},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(prompt, return_tensors="pt").to(model.device),
                     max_new_tokens=512, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))
```

Serve with vLLM (as used for all evals):

```bash
vllm serve Qwen/Qwen2.5-Coder-7B-Instruct \
    --enable-lora --lora-modules sqlforge=devenshah21/sqlforge-qwen7b-sft \
    --max-lora-rank 32 --port 8000
```

### Serving contract

The adapter was trained **and** benchmarked (74.5%) with **one specific prompt shape**: a
house-style system prompt (schema + the categorical-literal / revenue-formula / join-path
rules) and the user question, **zero-shot, temperature 0**. Sending few-shot examples or a
different prompt is an unmeasured configuration and will not reproduce the numbers above —
fine-tuning already absorbed what few-shot was teaching. The exact prompt builder lives in
[`phase0/run_baseline.py`](https://github.com/ShahaDeven/sql-forge) (`build_system_prompt`).

## Training

| | |
|---|---|
| Method | bf16 LoRA (no quantization) |
| LoRA | r=32, α=64, dropout 0.05, all 7 projections (q,k,v,o,gate,up,down) |
| Data | 2,314 execution-validated synthetic pairs (distilled from Claude, gated on DuckDB execution + house-style checks) |
| Schedule | 2 epochs, lr 2e-4 cosine, effective batch 16, max_seq 2048, completion-only loss |
| Hardware | 1× A40 (48GB) |

## Limitations

- **Single schema.** Conventions are specific to this TPC-H-derived database; the Spider
  result shows general SQL is intact, but in-domain gains do not transfer to other schemas.
- **Residual failures** concentrate in a categorical-literal bug (`churn_risk='High'` vs
  `'HIGH_RISK'`), output-column shape, and simulation/CTE format — see the project repo's
  failure-mode analysis.
- Sibling adapters (DPO, and a 1.5B SFT/GRPO track) are documented in the repo; neither
  beat this SFT checkpoint, which is the deployed model.

### Framework versions

- PEFT 0.13.2