---
license: mit
base_model: Qwen/Qwen2.5-0.5B-Instruct
tags:
  - text-to-sql
  - lora
  - peft
  - sql
  - fine-tuned
datasets:
  - b-mc2/sql-create-context
language:
  - en
pipeline_tag: text-generation
---

# Text-to-SQL: Fine-tuned Qwen2.5-0.5B

A 0.5B parameter language model fine-tuned with **LoRA** to convert natural language questions into SQL queries — outperforming a 70B general-purpose model on the same benchmark despite being **140x smaller**.

## 🔥 Benchmark Results

| Model | Exact-Match Accuracy |
|---|---|
| Llama-3.3-70B (zero-shot, via Groq) | 32.0% |
| **Qwen2.5-0.5B (this model, fine-tuned)** | **84.0%** |

Evaluated on a held-out set of 100 text-to-SQL questions, scored by exact match after SQL normalization.

## 🧠 Model Details

- **Base model:** [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)
- **Fine-tuning method:** LoRA (r=16, alpha=32, target modules: q_proj, v_proj)
- **Training data:** [b-mc2/sql-create-context](https://huggingface.co/datasets/b-mc2/sql-create-context) (78,477 examples)
- **Epochs:** 3
- **Hardware:** Free Google Colab T4 GPU

## 🚀 Usage

```python
from peft import PeftModel
from transformers import AutoTokenizer, AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model = PeftModel.from_pretrained(base_model, "javeria163/text-to-sql")

schema = "CREATE TABLE employees (id INT, name TEXT, department TEXT, salary INT)"
question = "What is the average salary in the Engineering department?"

prompt = (
    "Task: Convert the question to SQL using the table schema.\n\n"
    "Schema:\n" + schema + "\n\n"
    "Question:\n" + question + "\n\n"
    "SQL:\n"
)

inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=150, do_sample=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True).split("SQL:")[-1].strip())
```

## 🎮 Try It Live

[**Interactive demo on Hugging Face Spaces**](https://huggingface.co/spaces/javeria163/text-to-sql-demo) — no setup required.

## 📊 Training

Loss decreased from 1.73 → 0.69 over 3 epochs, with final mean token accuracy of 82.9%.

## 🔗 Links

- [GitHub Repository](https://github.com/javeria163/text-to-sql-finetune) — full training code, eval scripts, and benchmark
- [Live Demo](https://huggingface.co/spaces/javeria163/text-to-sql-demo)