README.md

---
base_model: google/gemma-3-270m-it
datasets:
  - chabab/text2sql-oracle-postgres
library_name: transformers
license: apache-2.0
pipeline_tag: text-generation
tags:
  - text-to-sql
  - sql
  - oracle
  - postgresql
  - gemma3
  - trl
  - sft
---

# gemma-3-270m-text2sql-oracle-postgres

[`google/gemma-3-270m-it`](https://huggingface.co/google/gemma-3-270m-it) fine-tuned to turn a
schema + a natural-language question into **one** dialect-correct SQL statement — Oracle or
PostgreSQL — with no markdown fences and no commentary.

At 270M parameters it runs on CPU and quantizes to ~290 MB.

## Results

Held-out test split (60 examples), greedy decoding, normalized exact match against gold SQL:

| Slice | Exact match |
|---|---|
| **Overall** | **78.3%** (47/60) |
| Oracle | 86.7% (39/45) |
| PostgreSQL | 53.3% (8/15) |
| easy | 93.3% (14/15) |
| medium | 69.2% (27/39) |
| hard | 100% (6/6) |

Per-example predictions are in [`eval_results.json`](./eval_results.json).

Caveats worth knowing before you rely on these numbers:

- The test split is skewed 45 Oracle / 15 PostgreSQL, so the PostgreSQL figure rests on 15
  examples and has a wide error bar.
- Exact match is strict. Several "failures" are valid SQL that differs from gold — an extra
  `LIMIT`, a different but equivalent predicate. Real semantic accuracy is higher than 78.3%.
- The most common genuine error is dialect leakage: emitting `LIKE` where PostgreSQL gold uses
  `ILIKE`. If case-insensitive matching matters to you, check that specific pattern.
- Only the 7 schemas in the training set (hr, sales, banking, inventory, tickets, university,
  logistics) are represented. Generalization to unseen schemas is untested.

## Usage

The model expects the system prompt naming the dialect, then a `Schema:` block and a `Question:`
block — the same shape as the training data.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "chabab/gemma-3-270m-text2sql-oracle-postgres"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    attn_implementation="eager",  # Gemma-3 needs eager attention for correct generation
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You convert natural language into PostgreSQL SQL. Use only tables and columns from the provided schema. Reply with one SQL statement and nothing else. No markdown fences. No commentary."},
    {"role": "user", "content": """Schema:
employees(
  employee_id INTEGER PK,
  first_name VARCHAR(50),
  last_name VARCHAR(50),
  hire_date DATE,
  salary NUMERIC(12,2),
  department_id INTEGER FK->departments.department_id
)

Question:
Show the five employees with the largest salary. Return only the SQL."""},
]

ids = tok.apply_chat_template(messages, add_generation_prompt=True,
                              return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**ids, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[-1]:], skip_special_tokens=True).strip())
# SELECT first_name, last_name, salary FROM employees ORDER BY salary DESC LIMIT 5;
```

Two things matter for output quality:

- **Use greedy decoding** (`do_sample=False`). The task has one right answer; sampling only adds
  drift.
- **Use `attn_implementation="eager"`.** Gemma-3 generates degenerate repeated tokens under the
  default SDPA path in some configurations.

## GGUF / local inference

Quantized builds for Ollama, LM Studio, and llama.cpp:
[`chabab/gemma-3-270m-text2sql-oracle-postgres-GGUF`](https://huggingface.co/chabab/gemma-3-270m-text2sql-oracle-postgres-GGUF)

## Training

Full-parameter SFT (no LoRA — the model is small enough to tune end to end) with TRL `SFTTrainer`
on one L4 GPU, about 10 minutes.

| | |
|---|---|
| Base | `google/gemma-3-270m-it` |
| Data | `chabab/text2sql-oracle-postgres` — 684 train / 60 validation / 60 test |
| Epochs | 5 |
| Effective batch size | 16 (4 × 4 grad accum) |
| Learning rate | 5e-5, cosine, 10 warmup steps |
| Max sequence length | 1024 |
| Precision | bf16 |

Final metrics: train loss **0.0256**, eval loss **0.0683**, eval token accuracy **98.5%**.
Eval loss fell monotonically through training with no divergence.

## Limitations

Generated SQL is not validated against a live database. The model can produce syntactically valid
statements that reference the wrong table or misread the intent — two of the observed test
failures do exactly that. Review output before executing it, and never run generated SQL against
production with write permissions.

## License

Apache 2.0, inheriting the [Gemma terms of use](https://ai.google.dev/gemma/terms) from the base
model.