READTHIS.md

# **Qwen2.5 Text-to-SQL Fine-Tuning with LoRA**

## note: If HF SPACES UI not loading up (DOWNLOAD text2SQL_snippet.mp4 to see the app)

parameter-efficient fine-tuning project that adapts [`Qwen/Qwen2.5-Coder-0.5B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B-Instruct) to generate SQL from a database schema and a natural-language request

project uses Hugging Face Transformers, TRL, PEFT LoRA, the [`gretelai/synthetic_text_to_sql`](https://huggingface.co/datasets/gretelai/synthetic_text_to_sql) dataset, SQLGlot for SQL parsing, and Gradio (for Hugging Face Spaces)


# **Project Goal:**

goal of this project is to adapt a pretrained (code focused) Transformer (Qwen/Qwen2.5-Coder-0.5B-Instruct) into a specialized Text-to-SQL model without fully retraining all of the model's parameters

given:

1. a database schema or SQL context
2. a natural-language request

the model will learn how to generate the corresponding SQL code/syntax

### **applicable example:**

**database context:**

```sql
CREATE TABLE customers (
    id INTEGER PRIMARY KEY,
    name TEXT,
    country TEXT,
    revenue DECIMAL(12, 2)
);
```

**natural language request:**

```text
Find the five customers with the highest revenue.
```

**optimal model output:**

```sql
SELECT id, name, country, revenue
FROM customers
ORDER BY revenue DESC
LIMIT 5;
```



# **Project Objectives:**

project workflow:

* fine-tune a coding language model (efficiently)
* convert structured Text-to-SQL examples into supervised instruction-tuning data
* use LoRA instead of full fine-tuning to reduce the number of trainable parameters
* apply LoRA directly to the Transformer attention projection layers
* train only on the assistant SQL completion rather than the prompt tokens
* evaluate generated SQL with exact-match and syntax-validity metrics
* load the trained LoRA adapter on top of the original Qwen base model for inference
* deploy with Gradio UI (Hugging Face Spaces)


# **Base Model:**

 project uses,
[**Qwen2.5-Coder-0.5B-Instruct**](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B-Instruct), an instruction-tuned causal language model from the Qwen2.5-Coder family

model contains (approximately) 0.50 billion parameters and uses a Transformer architecture with components including **RoPE (Rotary Position Embedding)**, **SwiGLU (Swish-Gated Linear Unit)**, **RMSNorm (Root Mean Square Layer Normalization)**, and **attention projections**

* **[RoPE (Rotary Position Embedding):](https://towardsdatascience.com/rope-clearly-explained/)**  
    * positional encoding method for transformer models that multiplies the query and key vectors by rotation matrices based on token positions
    * **objective:** encode absolute positions while causing the inner product of attention to depend naturally on relative distance
    * **no extra weights:** introduces zero learned parameters and preserves vector magnitudes because rotation is an orthogonal operation
    * **relative position awareness:** when the attention dot product $q_m * k_n$ is computed, the absolute rotation angles m and n subtract into a relative offset $(m - n)$
    * **Long-Context scaling:** allows models to extrapolate or scale to larger sequence lengths using methods like position interpolation or YaRN
    * **decaying dependency:**  reduces attention weights as the relative distance between tokens increases
* **[SwiGLU (Swish-Gated Linear Unit):](https://sebastianraschka.com/faq/docs/swiglu-modern-llms.html)** advanced activation function and feed-forward layer (used in modern Transformer models)
    * **Swish:** smooth, non-monotonic mathematical function $x$ × $sigmoid(x)$ that replaces older functions like ReLU
    * **GLU (Gated Linear Unit):** activation function that splits an input into two parallel pathways, using one path as a gate to control the flow of information in the other
* **[RMSNorm (Root Mean Square Layer Normalization):](https://arxiv.org/abs/1910.07467)**
    * **goal:** stabilize and accelerate neural network training by scaling activations, while completely eliminating the computationally expensive mean-centering (subtraction) step
    * alternative to standard Layer Normalization

* **[attention projections:](https://en.wikipedia.org/wiki/Attention_(machine_learning))**
    * learned linear transformations that map input token vectors into Queries ($Q$), Keys ($K$), and Values ($V$)
    * use trainable weight matrices to project data into specific subspaces, allowing models (like Transformers) to calculate relevance scores and dynamically route context between different parts of a sequence
    * **contextual refinement:** update word meanings based on surrounding text
    * **relationship tracking:** capture complex syntactic and semantic dependencies regardless of distance
    * **parallel processing:** enable multi-head attention to analyze diverse perspectives of data simultaneously
    * **information routing:** scale down dimensions for efficient scoring while retaining critical sequence information

**note:** code-specialized model is useful for this project because SQL generation is fundamentally a structured code-generation problem rather than ordinary natural-language classification

# **Dataset:**

training data comes from: [**gretelai/synthetic_text_to_sql**](https://huggingface.co/datasets/gretelai/synthetic_text_to_sql)

dataset is designed for synthetic Text-to-SQL training and provides natural-language SQL requests, database context, target SQL, and related metadata

project uses three fields directly:

| Dataset field | Purpose                                              |
| ------------- | ---------------------------------------------------- |
| `sql_context` | Database schema or SQL context supplied to the model |
| `sql_prompt`  | Natural language prompt request                             |
| `sql`         | Ground-truth SQL completion                          |

local training configuration intentionally starts with a subset of the available data:

```python
MAX_TRAIN_SAMPLES = 20_000
MAX_VALIDATION_SAMPLES = 1_000
GENERATION_EVAL_SAMPLES = 50
```

makes it easier to validate the complete training pipeline before increasing the training set size



# **LoRA?**

**LoRA** (**Low-Rank Adaptation**), is a parameter-efficient fine-tuning method, (Hu et al. in [LoRA: Low-Rank Adaptation of Large Language Models.](https://arxiv.org/abs/2106.09685))

instead of updating every weight in the pretrained model, LoRA:

1. freezes the original pretrained weight matrices
2. introduces small trainable matrices alongside selected layers
3. learns a low-rank update to the original weights
4. stores only those adapter parameters after training

### **Benefits of LoRA:**
* **Fewer Trainable Parameters:** reduces the number of weights (compute efficient)
* **Low Hardware Requirements:** lowers GPU memory needs, allows me to fine-tune large models on (consumer-grade gpus) instead of massive server clusters
* **Faster Training Speeds:** shortens fine-tuning cycles from days to hours because the system processes much smaller matrix updates
* **High Modularity & Storage Savings:** generates tiny adapter files (often just megabytes instead of gigabytes) that can be swapped easily onto a single shared base model
* **Zero Inference Latency:** allows trained adapter weights to merge directly back into the primary model structure before deployment, retaining full processing speed

makes task adaptation more parameter-efficient than full fine-tuning because the large pretrained matrix remains unchanged while only the low-rank update is optimized

---

# **How LoRA Is Applied (in This Project)**

The LoRA configuration is defined in `train.py`:

```python
peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
)
```

## **Rank**

```python
r = 16
```

* update matrices are constrained to rank 16

* larger value of $r$ increases the number of trainable parameters and gives the adapter more representational capacity 

* smaller value reduces memory and storage requirements but places a stronger constraint on the update

---

## **LoRA Alpha**

```python
lora_alpha = 32
```
constant scaling factor that controls the overall strength or "volume" of the trained adapter weights relative to the frozen base model

divides the internal rank ($r$) to determine how much influence the new matrix updates have on final outputs

* (Alpha $α$) 32 means the scaling constant (alpha ($α$)) that controls the strength of the fine-tuning updates is set to 32

* using  alpha ($α$) = 32 as a stable baseline standard

---

## **LoRA Dropout:**

```python
lora_dropout = 0.05
```
* regularization hyperparameter that randomly turns off a fraction of activations in the low-rank adaptation layers during training. main purpose is to prevent overfitting by stopping the adapter from relying too heavily on specific nodes, which helps the model generalize better to new data

* 5% dropout rate is configured on the LoRA branch during training. this provides regularization to the adapter path while the pretrained base weights remain frozen

* lora_dropout = 0.05 means a 5% dropout rate is applied to the LoRA adapter layers. During training, the model randomly turns off 5% of the adapter weights in those layers at each step. this prevents the model from memorizing the training data too well

* 0.05 (5%): good for small or medium datasets where the model might start to overfit

---

# **Training Objective:**

using supervised fine-tuning with TRL's `SFTTrainer`

each dataset row is converted to a conversational prompt-completion example

```text
System:
You are an expert SQL generator...

User:
Database context:
<schema>

Request:
<natural-language request>

Assistant:
<target SQL>
```
---

# **Training Configuration:**

| Setting                    |                                  Value |
| -------------------------- | -------------------------------------: |
| Base model                 |     `Qwen/Qwen2.5-Coder-0.5B-Instruct` |
| LoRA rank                  |                                   `16` |
| LoRA alpha                 |                                   `32` |
| LoRA dropout               |                                 `0.05` |
| Target modules             | `q_proj`, `k_proj`, `v_proj`, `o_proj` |
| Epochs                     |                                    `2` |
| Train batch size           |                                    `4` |
| Gradient accumulation      |                                    `4` |
| Effective batch per device |                         `16` sequences |
| Learning rate              |                                 `2e-4` |
| Warmup ratio               |                                 `0.03` |
| Weight decay               |                                 `0.01` |
| LR scheduler               |                                 Cosine |
| Maximum sequence length    |                                 `2048` |
| Gradient checkpointing     |                                Enabled |
| Training objective         |                    Completion-only SFT |
| Best checkpoint metric     |                        Validation loss |

script automatically uses BF16 (data type) when supported, otherwise FP16 on CUDA, and FP32 when running without CUDA

---

# **Training Pipeline**

```text
Gretel Text-to-SQL Dataset
            |
            v
     Shuffle / Sample
            |
            v
Prompt + Completion Formatting
            |
            v
     Qwen Tokenizer
            |
            v
Qwen2.5-Coder-0.5B-Instruct
      Frozen Base Weights
            +
 LoRA Attention Adapters
            |
            v
        SFTTrainer
            |
            v
   Trained LoRA Adapter
            |
            v
Base Model + Adapter at Inference
            |
            v
       Generated SQL
            |
            v
    SQLGlot Validation
            |
            v
       Gradio UI
```


### `train.py`

loads up Hugging Face dataset, formats prompt-completion examples, loads Qwen, attaches LoRA adapters, performs supervised fine-tuning, and saves the trained adapter

### `inference.py`

loads the original Qwen model and then loads the trained PEFT adapter on top of it. provides command-line Text-to-SQL generation

### `evaluate.py`

generates predictions from held-out examples and reports:

* normalized exact-match accuracy,
* SQL syntax validity using SQLGlot,
* row-level predictions within evaluation_results.csv


### `app.py`

UI interface (Hugging Face Spaces)

applications include:

* SQL schema editor
* natural-language request input
* generated SQL editor
* copy and download controls
* SQL dialect selection
* temperature control
* output token control
* SQLGlot syntax validation
* model diagnostics
* example Text-to-SQL prompts
* LoRA adapter loading
* base-model fallback when an adapter is unavailable

---

# **Why LoRA Fits This Project**


$$
\text{General pretrained model}
+
\text{small Text-to-SQL dataset}
=

\text{specialized Text-to-SQL model}
$$

---

# **Limitations:**

* the model can still hallucinate tables or columns (if context is insufficient)
* the 0.5B model prioritizes a lightweight training and deployment footprint over maximum reasoning capacity
* training (fine-tuning) on synthetic examples may not represent every schema style or production SQL workload
---

# **Validation Progression by Checkpoint**

metrics from the completed RTX 3050 LoRA fine-tuning run


| Checkpoint | Epoch | Eval Loss | Eval Token Accuracy |
|---:|---:|---:|---:|
| 100 | 0.08 | 0.2839 | 92.06% |
| 200 | 0.16 | 0.2682 | 92.41% |
| 300 | 0.24 | 0.2589 | 92.65% |
| 400 | 0.32 | 0.2518 | 92.84% |
| 500 | 0.40 | 0.2481 | 92.93% |
| 600 | 0.48 | 0.2440 | 92.84% |
| 700 | 0.56 | 0.2429 | 92.93% |
| 800 | 0.64 | 0.2386 | 93.14% |
| 900 | 0.72 | 0.2376 | 93.12% |
| 1000 | 0.80 | 0.2343 | 93.23% |
| 1100 | 0.88 | 0.2328 | 93.24% |
| 1200 | 0.96 | 0.2306 | 93.26% |
| 1300 | 1.04 | 0.2281 | 93.38% |
| 1400 | 1.12 | 0.2287 | 93.45% |
| 1500 | 1.20 | 0.2275 | 93.40% |
| 1600 | 1.28 | 0.2251 | 93.47% |
| 1700 | 1.36 | 0.2232 | 93.48% |
| 1800 | 1.44 | 0.2244 | 93.43% |
| 1900 | 1.52 | 0.2223 | 93.62% |
| 2000 | 1.60 | 0.2213 | 93.57% |
| 2100 | 1.68 | 0.2208 | 93.53% |
| 2200 | 1.76 | 0.2203 | 93.53% |
| 2300 | 1.84 | 0.2202 | 93.56% |
| 2400 | 1.92 | 0.2198 | 93.54% |
| 2500 | 2.00 | 0.2200 | 93.54% |

* validation loss improved from 0.2839 at the first logged evaluation to a best value of 0.2198 at checkpoint 2400 / epoch 1.92

* final evaluation at checkpoint 2500 / epoch 2.00 produced an eval loss of 0.2200, which is flat relative to the best checkpoint. this shows that the model converged near the end of epoch 2 rather than showing a large late-stage validation-loss increase

# **Final Training Summary**

| Metric | Result |
|---|---:|
| Final training loss | **0.2381** |
| Training runtime | **5735 seconds / 1:35:34** |
| Samples per second | **6.975** |
| Optimizer steps per second | **0.436** |
| Completed epochs | **2** |
| Total optimizer steps | **2,500** |
| Best checkpoint | **2400** |
| Best eval loss | **0.2198** |
| Final eval loss | **0.2200** |
| Final eval token accuracy | **93.54%** |

# **Held-Out Generation Evaluation**

| Metric | Result |
|---|---:|
| Evaluation examples | **50** |
| Exact-match accuracy | **28.0%** |
| SQL syntax validity | **100.0%** |
| Exact matches | **14 / 50** |
| Syntax-valid generations | **50 / 50** |

* the 100% SQL syntax-validity rate means every generated query in this 50-example evaluation was parsable by SQLGlot

* the 28% exact-match score is  strict. different SQL strings can still be semantically or execution-equivalent, so exact match should not be treated as the model's true semantic accuracy

# **Results:**

ml model completed run shows four useful outcomes:

1. **successful GPU fine-tuning** on an NVIDIA GeForce RTX 3050 using CUDA 12.8 and BF16.
2. **parameter-efficient adaptation**, with only 2,162,688 of 496,195,456 parameters trainable, or 0.4359%.
3. **stable convergence**, with validation loss falling into the ~0.22 range by the end of training.
4. **reliable SQL-formatted generation**, with 50/50 generated evaluation queries parsing successfully.


# **References:**

1. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. *\*LoRA: Low-Rank Adaptation of Large Language Models\**. arXiv:2106.09685, 2021.](https://arxiv.org/abs/2106.09685)

2. Hugging Face. *\*PEFT LoRA Documentation\**. Parameter-Efficient Fine-Tuning documentation. (https://huggingface.co/docs/transformers/en/peft) (https://huggingface.co/docs/peft/en/package_reference/lora)

3. Qwen Team. [Qwen2.5-Coder-0.5B-Instruct Model Card](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B-Instruct)

4. Hui, B. et al. [Qwen2.5-Coder Technical Report](https://arxiv.org/abs/2409.12186). arXiv:2409.12186, 2024

5. Gretel.ai. [synthetic\_text\_to\_sql Dataset Card](https://huggingface.co/datasets/gretelai/synthetic_text_to_sql). Hugging Face Datasets