File size: 4,444 Bytes
d0389db 28c684d d0389db 28c684d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 | ---
license: mit
base_model: unsloth/Llama-3.2-3B-Instruct
tags:
- text-to-sql
- sql
- lora
- unsloth
- llama
language:
- en
---
# Llama-3.2-3B Text-to-SQL
A LoRA fine-tune of [Llama-3.2-3B-Instruct](https://huggingface.co/unsloth/Llama-3.2-3B-Instruct)
for generating SQL queries from a natural language question plus a table schema.
Trained locally on an AMD Radeon RX 9070 XT using [Unsloth](https://github.com/unslothai/unsloth).
## What this model does
Given a table schema and a question in plain English, it returns a SQL query — no
explanation, no preamble, no alternative approaches. The base instruct model tends to
either refuse ("I don't have access to your database") or respond with an explanatory
Python example instead of raw SQL. Fine-tuning fixed that: this model reliably answers
with bare SQL by default.
## Prompt format
**This model expects the schema to be included in the prompt.** Without one, it will
guess plausible-sounding table/column names rather than asking for clarification — same
as any model would.
```
Given this schema: CREATE TABLE employees (name VARCHAR, salary INTEGER)
Answer this question in SQL: List the names of employees who earn more than 50000
```
Expected output:
```sql
SELECT name FROM employees WHERE salary > 50000
```
Use the tokenizer's chat template (`apply_chat_template`) with this as a single user
turn — see the usage example below.
## Usage
```python
from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = "crimson3327/text_to_sql",
max_seq_length = 1024,
)
FastLanguageModel.for_inference(model)
schema = "CREATE TABLE employees (name VARCHAR, salary INTEGER)"
question = "List the names of employees who earn more than 50000"
inputs = tokenizer.apply_chat_template(
[{"role": "user", "content": f"Given this schema: {schema}\n\nAnswer this question in SQL: {question}"}],
tokenize=True, add_generation_prompt=True, return_tensors="pt"
).to("cuda")
outputs = model.generate(input_ids=inputs, max_new_tokens=150)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
## Training details
- **Base model:** unsloth/Llama-3.2-3B-Instruct
- **Method:** LoRA, rank 16, alpha 16, all attention + MLP projections targeted
- **Dataset:** [b-mc2/sql-create-context](https://huggingface.co/datasets/b-mc2/sql-create-context)
(78,577 question/schema/SQL triples), trained on the first 74,000 rows
- **Held-out eval set:** the remaining ~4,500 rows, never seen during training
- **Steps:** 1,200 (batch size 2, gradient accumulation 4 — effective batch 8)
- **Optimizer:** AdamW, linear LR schedule, 2e-4 peak learning rate
- **Checkpoint selection:** best checkpoint chosen automatically by **eval loss**, not
training loss, to avoid shipping an overfit checkpoint
- **Final eval loss:** 0.559
## Example: base model vs. this fine-tune
Same prompt, no schema given, asked to filter employee data:
**Base Llama-3.2-3B-Instruct** — rewrote the task as a pandas exercise with invented
sample data, offered a second alternative approach using bitwise operators, no SQL
produced.
**This model:**
```sql
SELECT * FROM employee WHERE salary > 10000 AND Department = 'IT' AND Name = 'John Doe'
```
## Known limitations
- **Multi-table joins (3+ tables) with aggregation are inconsistent.** Earlier training
checkpoints produced invalid SQL (joins placed after `GROUP BY`/`HAVING`) and
hallucinated columns on this pattern specifically. The checkpoint published here
(trained with the held-out eval split) resolved the specific cases tested, but this
remains the weakest area — verify output on complex joins before trusting it blindly.
- **Ambiguous negation phrasing** (e.g. "not yet shipped") may be interpreted as a
literal status string rather than a negated condition (`!=`). Prefer explicit phrasing
in questions where this matters.
- **No schema validation.** The model doesn't verify that referenced columns/tables
exist — always pass an accurate schema and review output before executing against a
real database.
- Trained on SQLite-flavored syntax (matching the source dataset); some queries may need
adjustment for strict-mode PostgreSQL or other dialects.
## Acknowledgements
Fine-tuned with [Unsloth](https://github.com/unslothai/unsloth). Dataset:
[b-mc2/sql-create-context](https://huggingface.co/datasets/b-mc2/sql-create-context). |