README.md

---
license: apache-2.0
library_name: transformers
base_model: Qwen/Qwen2-0.5B-Instruct
tags:
- generated_from_trainer
- trl
- sft
- sql
- text
model_name: Qwen2-0.5B-Instruct-SQL-generator
datasets:
- gretelai/synthetic_text_to_sql
language:
- en
metrics:
- bleu
- chrf
- rouge
---

# Model Card for Qwen2-0.5B-Instruct-SQL-generator

This model is a fine-tuned version of [Qwen/Qwen2-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct).  
It has been trained using [TRL (Transformer Reinforcement Learning)](https://github.com/huggingface/trl) for SQL generation tasks.

## Quick Start

```python
from transformers import pipeline

question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
generator = pipeline("text-generation", model="onkolahmet/Qwen2-0.5B-Instruct-SQL-generator", device="cuda")
output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
print(output["generated_text"])
```

## Training Procedure

This model was trained with Supervised Fine-Tuning (SFT) using the [gretelai/synthetic_text_to_sql](https://huggingface.co/datasets/gretelai/synthetic_text_to_sql) dataset.  
The goal was to fine-tune the model to better translate natural language queries into SQL statements.

### Framework Versions

- TRL: 0.12.2  
- Transformers: 4.46.3  
- PyTorch: 2.6.0  
- Datasets: 3.4.1  
- Tokenizers: 0.20.3

## Evaluation Results

The model was evaluated using standard text generation metrics (BLEU, ROUGE-L F1, CHRF) in both zero-shot and few-shot prompting scenarios.

### 🔹 Zero-shot Prompting (on `gretelai/synthetic_text_to_sql/test`)

**After Post-processing:**
- **BLEU Score:** 0.5195 
- **ROUGE-L F1:** 0.7031 
- **CHRF Score:** 70.0409  

**Before Post-processing:**
- **BLEU Score:** 0.1452  
- **ROUGE-L F1:** 0.3009  
- **CHRF Score:** 47.8182

**SQL-Specific Metrics:**
- **Exact Match (case insensitive):**  0.1600
- **Normalized Exact Match:**  0.1500
- **Average Component Match:**  0.4528
- **Average Entity Match:**  0.8807

**Query Quality Distribution:**
- **High Quality (≥80% component match):** 18 (18.0%)
- **Medium Quality (50-79% component match):** 28 (28.0%)
- **Low Quality (<50% component match):** 54 (54.0%)

---

### 🔹 Few-shot Prompting (on `gretelai/synthetic_text_to_sql/test`)

**After Post-processing:**
- **BLEU Score:** 0.2680  
- **ROUGE-L F1:** 0.4975
- **CHRF Score:** 57.1704  

**Before Post-processing:**
- **BLEU Score:** 0.1272  
- **ROUGE-L F1:** 0.2816  
- **CHRF Score:** 46.1643  


**SQL-Specific Metrics:**
- **Exact Match (case insensitive):** 0.0000
- **Normalized Exact Match:**  0.0000
- **Average Component Match:** 0.2140
- **Average Entity Match:** 0.8067

**Query Quality Distribution:**
- **High Quality (≥80% component match):** 4 (4.0%)
- **Medium Quality (50-79% component match):** 17 (17.0%)
- **Low Quality (<50% component match):** 79 (79.0%)

## Citations

Cite TRL as:

```bibtex
@misc{vonwerra2022trl,
	title        = {{TRL: Transformer Reinforcement Learning}},
	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
	year         = 2020,
	journal      = {GitHub repository},
	publisher    = {GitHub},
	howpublished = {\url{https://github.com/huggingface/trl}}
}
```