README.md

metadata

license: mit
datasets:
  - sirunchained/text-to-sql-dataset
language:
  - en
base_model:
  - google/gemma-3-270m
pipeline_tag: text-generation
library_name: transformers
tags:
  - LoRA
  - peft
  - gemma
  - trl
  - sql

Text-to-SQL Model v2

🚀 Model Description

This is Version 2 of the sirunchained/text-to-sql-model, fine-tuned from google/gemma-3-270m-it for Text-to-SQL generation.
In this version, the model is merged with the LoRA adapter – you can load it directly with pipeline() (no PEFT required).

Key improvements in v2:

Feature	v1 (LoRA Adapter)	v2 (Merged)
Load method	Required PEFT + base model	Direct pipeline()
Model size	~10 MB (adapter only)	~536 MB (full model)
Inference speed	Slower (requires adapter load)	Faster
Ease of use	Complex	Simple
Performance	89.7% accuracy	✅ Same

---

🧠 Task

Text-to-SQL Generation
Converts natural language questions into SQL queries. Supports:

• ✅ SELECT queries (with JOINs, aggregations, subqueries)
• ✅ INSERT operations
• ✅ UPDATE operations
• ✅ DELETE operations (currently weak at this)

---

📊 Training Details

Item	Value
Base Model	google/gemma-3-270m-it
Fine-tuning Method	LoRA + 4-bit quantization (QLoRA)
Framework	trl (SFTTrainer)
Dataset	sirunchained/text-to-sql-dataset (4518 samples training, 200 validation, 200 test)
Training Epochs	5
Batch Size	32
Learning Rate	5e-5
LoRA Rank (r)	8
LoRA Alpha	16
Optimizer	AdamW (fused)

---

📈 Training Performance

Epoch	Training Loss	Validation Loss	Mean Token Accuracy
1	0.800	0.700	83.8%
2	0.650	0.680	84.2%
3	0.500	0.650	84.6%
4	0.350	0.640	85.1%
5	0.550	0.640	83.5%

Best validation loss was achieved at epoch 4 & 5 (0.640).
Highest mean token accuracy on validation was at epoch 4 (85.1%).

---

💻 Quick Start

Using Pipeline (Recommended)

from transformers import pipeline

generator = pipeline(
    "text-generation",
    model="sirunchained/text-to-sql-model-v2",
    device=0  # or "cuda"
)

# Example with schema
prompt = """<start_of_turn>user
# Schema
customers(id, name, email, country)
# Text
Find customers from USA.<end_of_turn>
<start_of_turn>model
"""
result = generator(prompt, max_new_tokens=128)
print(result[0]["generated_text"])

With Chat Template

from transformers import pipeline

pipe = pipeline("text-generation", model="sirunchained/text-to-sql-model-v2")

messages = [
    {"role": "user", "content": "# Schema\ncustomers(id, name, email)\n\n# Text\nFind customers with gmail emails."}
]

outputs = pipe(
    pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True),
    max_new_tokens=128
)
print(outputs[0]["generated_text"])

---

🎯 Dataset

The model was trained on sirunchained/text-to-sql-dataset:

Split	Size
Train	4,518 samples
Validation	200 samples
Test	200 samples

Dataset format:

• text: Natural language question
• schema: Optional database schema
• query: Target SQL query

---

🧪 Evaluation Results

Test Set Performance (Epoch 5 model):

Metric	Value
Test Loss	0.638
Mean Token Accuracy	0.830
Entropy	0.636

---

📝 Version History

Version	Date	Description
v1	2026-07-22	LoRA adapter only (not directly loadable with pipeline)
v2	2026-07-23	Merged version – fully loadable with pipeline()

---

🛠️ Training Configuration

# LoRA Configuration
LoraConfig(
    r=8,
    lora_alpha=16,
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
)

# Training Configuration
SFTConfig(
    num_train_epochs=5,
    per_device_train_batch_size=32,
    learning_rate=5e-5,
    lr_scheduler_type="constant",
    weight_decay=0.0,
    load_best_model_at_end=True,
    metric_for_best_model="mean_token_accuracy",
    greater_is_better=True,
)

---

⚠️ Important Notes

• This is a small language model (270M parameters) – works on T4 GPUs
• Provide schema only when needed – works with or without it
• For non-SQL requests, the model outputs INVALID_QUERY (trained with negative samples)
• The model handles INSERT, UPDATE, and DELETE queries correctly

---

🔗 Links

• Base Model: google/gemma-3-270m-it
• Dataset: sirunchained/text-to-sql-dataset
• v1 (Adapter): sirunchained/text-to-sql-model

---

🙏 Acknowledgments

Built with:

• Hugging Face Transformers
• TRL (Transformer Reinforcement Learning)
• PEFT (Parameter-Efficient Fine-Tuning)
• bitsandbytes
• Gradio for the demo interface which you can use here

---

📄 License

This model is released under the same license as Google's Gemma model. See the Gemma model card for details.