README.md

---
license: mit
base_model: unsloth/Llama-3.2-3B-Instruct
tags:
  - text-to-sql
  - sql
  - lora
  - unsloth
  - llama
language:
  - en
---

# Llama-3.2-3B Text-to-SQL

A LoRA fine-tune of [Llama-3.2-3B-Instruct](https://huggingface.co/unsloth/Llama-3.2-3B-Instruct)
for generating SQL queries from a natural language question plus a table schema.

Trained locally on an AMD Radeon RX 9070 XT using [Unsloth](https://github.com/unslothai/unsloth).

## What this model does

Given a table schema and a question in plain English, it returns a SQL query — no
explanation, no preamble, no alternative approaches. The base instruct model tends to
either refuse ("I don't have access to your database") or respond with an explanatory
Python example instead of raw SQL. Fine-tuning fixed that: this model reliably answers
with bare SQL by default.

## Prompt format

**This model expects the schema to be included in the prompt.** Without one, it will
guess plausible-sounding table/column names rather than asking for clarification — same
as any model would.

```
Given this schema: CREATE TABLE employees (name VARCHAR, salary INTEGER)

Answer this question in SQL: List the names of employees who earn more than 50000
```

Expected output:
```sql
SELECT name FROM employees WHERE salary > 50000
```

Use the tokenizer's chat template (`apply_chat_template`) with this as a single user
turn — see the usage example below.

## Usage

```python
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "crimson3327/text_to_sql",
    max_seq_length = 1024,
)
FastLanguageModel.for_inference(model)

schema = "CREATE TABLE employees (name VARCHAR, salary INTEGER)"
question = "List the names of employees who earn more than 50000"

inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": f"Given this schema: {schema}\n\nAnswer this question in SQL: {question}"}],
    tokenize=True, add_generation_prompt=True, return_tensors="pt"
).to("cuda")

outputs = model.generate(input_ids=inputs, max_new_tokens=150)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Training details

- **Base model:** unsloth/Llama-3.2-3B-Instruct
- **Method:** LoRA, rank 16, alpha 16, all attention + MLP projections targeted
- **Dataset:** [b-mc2/sql-create-context](https://huggingface.co/datasets/b-mc2/sql-create-context)
  (78,577 question/schema/SQL triples), trained on the first 74,000 rows
- **Held-out eval set:** the remaining ~4,500 rows, never seen during training
- **Steps:** 1,200 (batch size 2, gradient accumulation 4 — effective batch 8)
- **Optimizer:** AdamW, linear LR schedule, 2e-4 peak learning rate
- **Checkpoint selection:** best checkpoint chosen automatically by **eval loss**, not
  training loss, to avoid shipping an overfit checkpoint
- **Final eval loss:** 0.559

## Example: base model vs. this fine-tune

Same prompt, no schema given, asked to filter employee data:

**Base Llama-3.2-3B-Instruct** — rewrote the task as a pandas exercise with invented
sample data, offered a second alternative approach using bitwise operators, no SQL
produced.

**This model:**
```sql
SELECT * FROM employee WHERE salary > 10000 AND Department = 'IT' AND Name = 'John Doe'
```

## Known limitations

- **Multi-table joins (3+ tables) with aggregation are inconsistent.** Earlier training
  checkpoints produced invalid SQL (joins placed after `GROUP BY`/`HAVING`) and
  hallucinated columns on this pattern specifically. The checkpoint published here
  (trained with the held-out eval split) resolved the specific cases tested, but this
  remains the weakest area — verify output on complex joins before trusting it blindly.
- **Ambiguous negation phrasing** (e.g. "not yet shipped") may be interpreted as a
  literal status string rather than a negated condition (`!=`). Prefer explicit phrasing
  in questions where this matters.
- **No schema validation.** The model doesn't verify that referenced columns/tables
  exist — always pass an accurate schema and review output before executing against a
  real database.
- Trained on SQLite-flavored syntax (matching the source dataset); some queries may need
  adjustment for strict-mode PostgreSQL or other dialects.

## Acknowledgements

Fine-tuned with [Unsloth](https://github.com/unslothai/unsloth). Dataset:
[b-mc2/sql-create-context](https://huggingface.co/datasets/b-mc2/sql-create-context).