README.md

---
license: llama3.2
language:
  - en
base_model: unsloth/Llama-3.2-3B-Instruct
tags:
  - text-to-sql
  - lora
  - peft
  - unsloth
  - sql
  - llama
datasets:
  - b-mc2/sql-create-context
---

# Llama-3.2-3B — Text-to-SQL LoRA Adapter

A LoRA adapter that teaches **Llama-3.2-3B-Instruct** to generate SQL queries from natural-language questions against a given table schema.

Fine-tuned on [`b-mc2/sql-create-context`](https://huggingface.co/datasets/b-mc2/sql-create-context) with [Unsloth](https://github.com/unslothai/unsloth) on a free Google Colab T4 GPU.

## What it does

**Input:** SQL schema (`CREATE TABLE ...`) + question in plain English
**Output:** SQL query that answers the question

## Examples

### Example 1 — JOIN + aggregation + sorting + limit

**Schema:**
```sql
CREATE TABLE orders (id INT, user_id INT, amount DECIMAL, created_at DATE);
CREATE TABLE users (id INT, name VARCHAR, country VARCHAR);
```
**Question:** *What are top 5 users by total order amount in 2025?*

**Model output:**
```sql
SELECT T1.id, T1.name, T1.country, SUM(T2.amount) AS total_amount
FROM users AS T1 JOIN orders AS T2 ON T1.id = T2.user_id
WHERE T2.created_at >= 2025
GROUP BY T1.id ORDER BY total_amount DESC LIMIT 5
```

### Example 2 — AVG with GROUP BY

**Schema:** `CREATE TABLE employees (id INT, name VARCHAR, department VARCHAR, salary INT, hire_date DATE)`
**Question:** *Show average salary by department, sorted from highest to lowest*

```sql
SELECT AVG(salary), department FROM employees
GROUP BY department ORDER BY AVG(salary) DESC
```

### Example 3 — WHERE with multiple conditions

**Schema:** `CREATE TABLE products (id INT, name VARCHAR, category VARCHAR, price DECIMAL, stock INT)`
**Question:** *Find all products with stock less than 10 in the 'electronics' category*

```sql
SELECT * FROM products WHERE category = "electronics" AND stock < 10
```

### Example 4 — COUNT with JOIN

**Schema:**
```sql
CREATE TABLE customers (id INT, name VARCHAR, country VARCHAR);
CREATE TABLE orders (id INT, customer_id INT, total DECIMAL);
```
**Question:** *Count how many orders each customer from Ukraine has made*

```sql
SELECT COUNT(*) FROM orders AS T1 JOIN customers AS T2
ON T1.customer_id = T2.id WHERE T2.country = "Ukraine"
```

## Training details

| Parameter | Value |
|---|---|
| Base model | `unsloth/Llama-3.2-3B-Instruct` |
| Method | LoRA (PEFT) via Unsloth |
| Dataset | `b-mc2/sql-create-context` — 2,000 examples (random subset, seed=42) |
| Trainable params | 24.3M / 3.24B (0.75%) |
| LoRA config | `r=16, alpha=16, dropout=0`, target modules: Q/K/V/O + gate/up/down |
| Loss masking | `train_on_responses_only` — only SQL tokens contribute to loss |
| Optimizer | `adamw_8bit`, lr=2e-4, linear schedule, warmup 5 steps |
| Batch | per-device 2 × grad-accum 4 = effective 8 |
| Steps | 250 (= 1 epoch on 2,000 examples) |
| Hardware | Google Colab T4 (free) — 6.5 GB peak GPU memory (44% of 15 GB) |
| Training time | 7.44 minutes |
| Loss trajectory | 0.55 → 0.04 (final) |

## Evaluation

Evaluated on **100 held-out examples** from `b-mc2/sql-create-context` (indices 2000–2100, unseen during training).

| Metric | Value |
|---|---|
| Exact-match accuracy (normalized: lowercase, whitespace collapsed, quotes normalized) | **76%** |

### Notes on the failure mode

Manual analysis of the 24 "failures" shows that **roughly half are semantically equivalent SQL** that exact-match rejects:

| Type of "failure" | Example |
|---|---|
| Strings vs numbers | `WHERE runs = 144` vs reference `WHERE runs = "144"` |
| Alias differences | `SELECT Prime` vs reference `SELECT Prime AS minister` |
| Ambiguous reference (dataset has minor errors) | `SELECT SUM(poles)` vs reference `SELECT MIN(poles)` |
| Genuine model errors (missing GROUP BY, wrong aggregation) | ~10 out of 100 |

A proper **execution accuracy** evaluation (running predictions against a real database) would likely show **85–90%** correctness. LLM-as-judge with Claude/GPT-4 is the next planned improvement.

## Known limitations

- **English only.** Trained on English questions only.
- **ANSI SQL.** No specific dialect (PostgreSQL / MySQL / Oracle).
- **Naive date comparisons** can appear (e.g. `created_at >= 2025` instead of `created_at >= '2025-01-01'`).
- **Implicit GROUP BY** is occasionally missed when a question implies "per each X" without saying it.
- Trained on the original dataset distribution — complex 3+ table joins or window functions are out of distribution.

## How to use

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = "unsloth/Llama-3.2-3B-Instruct"
adapter = "notingemiu/llama-3.2-3b-text2sql-lora"

tokenizer = AutoTokenizer.from_pretrained(adapter)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.float16, device_map="auto")
model = PeftModel.from_pretrained(model, adapter)

schema = "CREATE TABLE orders (id INT, total DECIMAL, created_at DATE)"
question = "What is total revenue in 2024?"

messages = [
    {"role": "system", "content": f"You are a SQL expert. Use this schema:\n{schema}"},
    {"role": "user", "content": question},
]
inputs = tokenizer.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, return_tensors="pt"
).to(model.device)
out = model.generate(input_ids=inputs, max_new_tokens=200, temperature=0.1, do_sample=True)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```

## Author

Built as part of a portfolio for AI/LLM Engineer roles. Comments and feedback welcome.

- HuggingFace: [@notingemiu](https://huggingface.co/notingemiu)