README.md

---
license: mit
datasets:
- sirunchained/text-to-sql-dataset
language:
- en
base_model:
- google/gemma-3-270m
pipeline_tag: text-generation
library_name: transformers
tags:
- LoRA
- peft
- gemma
- trl
- sql
---

# Text-to-SQL Model v2

## 🚀 Model Description

This is **Version 2** of the `sirunchained/text-to-sql-model`, fine-tuned from [google/gemma-3-270m-it](https://huggingface.co/google/gemma-3-270m-it) for Text-to-SQL generation.  
In this version, the model is **merged** with the LoRA adapter – you can load it directly with `pipeline()` (no PEFT required).

**Key improvements in v2:**

| Feature | v1 (LoRA Adapter) | v2 (Merged) |
|---------|-------------------|-------------|
| **Load method** | Required PEFT + base model | Direct `pipeline()` |
| **Model size** | ~10 MB (adapter only) | ~536 MB (full model) |
| **Inference speed** | Slower (requires adapter load) | Faster |
| **Ease of use** | Complex | Simple |
| **Performance** | 89.7% accuracy | ✅ Same |

---

## 🧠 Task

**Text-to-SQL Generation**  
Converts natural language questions into SQL queries. Supports:
- ✅ `SELECT` queries (with JOINs, aggregations, subqueries)
- ✅ `INSERT` operations
- ✅ `UPDATE` operations
- ✅ `DELETE` operations (currently weak at this)

---

## 📊 Training Details

| Item | Value |
|------|-------|
| **Base Model** | `google/gemma-3-270m-it` |
| **Fine-tuning Method** | LoRA + 4-bit quantization (QLoRA) |
| **Framework** | `trl` (SFTTrainer) |
| **Dataset** | `sirunchained/text-to-sql-dataset` (4518 samples training, 200 validation, 200 test) |
| **Training Epochs** | 5 |
| **Batch Size** | 32 |
| **Learning Rate** | 5e-5 |
| **LoRA Rank (r)** | 8 |
| **LoRA Alpha** | 16 |
| **Optimizer** | AdamW (fused) |

---

## 📈 Training Performance

| Epoch | Training Loss | Validation Loss | Mean Token Accuracy |
|-------|---------------|-----------------|---------------------|
| 1     | 0.800         | 0.700           | 83.8%               |
| 2     | 0.650         | 0.680           | 84.2%               |
| 3     | 0.500         | 0.650           | 84.6%               |
| 4     | 0.350         | **0.640**       | **85.1%**           |
| 5     | 0.550         | **0.640**       | 83.5%               |

> **Best validation loss** was achieved at **epoch 4 & 5** (`0.640`).  
> **Highest mean token accuracy** on validation was at **epoch 4** (`85.1%`).

---

## 💻 Quick Start

### Using Pipeline (Recommended)

```python
from transformers import pipeline

generator = pipeline(
    "text-generation",
    model="sirunchained/text-to-sql-model-v2",
    device=0  # or "cuda"
)

# Example with schema
prompt = """<start_of_turn>user
# Schema
customers(id, name, email, country)
# Text
Find customers from USA.<end_of_turn>
<start_of_turn>model
"""
result = generator(prompt, max_new_tokens=128)
print(result[0]["generated_text"])
```

### With Chat Template

```python
from transformers import pipeline

pipe = pipeline("text-generation", model="sirunchained/text-to-sql-model-v2")

messages = [
    {"role": "user", "content": "# Schema\ncustomers(id, name, email)\n\n# Text\nFind customers with gmail emails."}
]

outputs = pipe(
    pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True),
    max_new_tokens=128
)
print(outputs[0]["generated_text"])
```

---

## 🎯 Dataset

The model was trained on [sirunchained/text-to-sql-dataset](https://huggingface.co/datasets/sirunchained/text-to-sql-dataset):

| Split | Size |
|-------|------|
| Train | 4,518 samples |
| Validation | 200 samples |
| Test | 200 samples |

**Dataset format:**
- `text`: Natural language question
- `schema`: Optional database schema
- `query`: Target SQL query

---

## 🧪 Evaluation Results

**Test Set Performance (Epoch 5 model):**

| Metric | Value |
|--------|-------|
| **Test Loss** | 0.638 |
| **Mean Token Accuracy** | **0.830** |
| **Entropy** | 0.636 |

---

## 📝 Version History

| Version | Date | Description |
|---------|------|-------------|
| **v1** | 2026-07-22 | LoRA adapter only (not directly loadable with pipeline) |
| **v2** | **2026-07-23** | **Merged version – fully loadable with `pipeline()`** |

---

## 🛠️ Training Configuration

```python
# LoRA Configuration
LoraConfig(
    r=8,
    lora_alpha=16,
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
)

# Training Configuration
SFTConfig(
    num_train_epochs=5,
    per_device_train_batch_size=32,
    learning_rate=5e-5,
    lr_scheduler_type="constant",
    weight_decay=0.0,
    load_best_model_at_end=True,
    metric_for_best_model="mean_token_accuracy",
    greater_is_better=True,
)
```

---

## ⚠️ Important Notes

- This is a **small language model** (270M parameters) – works on T4 GPUs
- Provide schema **only when needed** – works with or without it
- For non-SQL requests, the model outputs `INVALID_QUERY` (trained with negative samples)
- The model handles INSERT, UPDATE, and DELETE queries correctly

---

## 🔗 Links

- **Base Model**: [google/gemma-3-270m-it](https://huggingface.co/google/gemma-3-270m-it)
- **Dataset**: [sirunchained/text-to-sql-dataset](https://huggingface.co/datasets/sirunchained/text-to-sql-dataset)
- **v1 (Adapter)**: [sirunchained/text-to-sql-model](https://huggingface.co/sirunchained/text-to-sql-model-ر۱)

---

## 🙏 Acknowledgments

Built with:
- [Hugging Face Transformers](https://github.com/huggingface/transformers)
- [TRL (Transformer Reinforcement Learning)](https://github.com/huggingface/trl)
- [PEFT (Parameter-Efficient Fine-Tuning)](https://github.com/huggingface/peft)
- [bitsandbytes](https://github.com/TimDettmers/bitsandbytes)
- [Gradio](https://gradio.app/) for the demo interface which you can use [here](https://huggingface.co/spaces/sirunchained/text-to-sql-gradio)

---

## 📄 License

This model is released under the same license as Google's Gemma model. See the [Gemma model card](https://huggingface.co/google/gemma-3-270m-it) for details.