Spaces:
Running on Zero
Qwen2.5 Text-to-SQL Fine-Tuning with LoRA
note: If HF SPACES UI not loading up (DOWNLOAD text2SQL_snippet.mp4 to see the app)
parameter-efficient fine-tuning project that adapts Qwen/Qwen2.5-Coder-0.5B-Instruct to generate SQL from a database schema and a natural-language request
project uses Hugging Face Transformers, TRL, PEFT LoRA, the gretelai/synthetic_text_to_sql dataset, SQLGlot for SQL parsing, and Gradio (for Hugging Face Spaces)
Project Goal:
goal of this project is to adapt a pretrained (code focused) Transformer (Qwen/Qwen2.5-Coder-0.5B-Instruct) into a specialized Text-to-SQL model without fully retraining all of the model's parameters
given:
- a database schema or SQL context
- a natural-language request
the model will learn how to generate the corresponding SQL code/syntax
applicable example:
database context:
CREATE TABLE customers (
id INTEGER PRIMARY KEY,
name TEXT,
country TEXT,
revenue DECIMAL(12, 2)
);
natural language request:
Find the five customers with the highest revenue.
optimal model output:
SELECT id, name, country, revenue
FROM customers
ORDER BY revenue DESC
LIMIT 5;
Project Objectives:
project workflow:
- fine-tune a coding language model (efficiently)
- convert structured Text-to-SQL examples into supervised instruction-tuning data
- use LoRA instead of full fine-tuning to reduce the number of trainable parameters
- apply LoRA directly to the Transformer attention projection layers
- train only on the assistant SQL completion rather than the prompt tokens
- evaluate generated SQL with exact-match and syntax-validity metrics
- load the trained LoRA adapter on top of the original Qwen base model for inference
- deploy with Gradio UI (Hugging Face Spaces)
Base Model:
project uses, Qwen2.5-Coder-0.5B-Instruct, an instruction-tuned causal language model from the Qwen2.5-Coder family
model contains (approximately) 0.50 billion parameters and uses a Transformer architecture with components including RoPE (Rotary Position Embedding), SwiGLU (Swish-Gated Linear Unit), RMSNorm (Root Mean Square Layer Normalization), and attention projections
RoPE (Rotary Position Embedding):
- positional encoding method for transformer models that multiplies the query and key vectors by rotation matrices based on token positions
- objective: encode absolute positions while causing the inner product of attention to depend naturally on relative distance
- no extra weights: introduces zero learned parameters and preserves vector magnitudes because rotation is an orthogonal operation
- relative position awareness: when the attention dot product $q_m * k_n$ is computed, the absolute rotation angles m and n subtract into a relative offset $(m - n)$
- Long-Context scaling: allows models to extrapolate or scale to larger sequence lengths using methods like position interpolation or YaRN
- decaying dependency: reduces attention weights as the relative distance between tokens increases
SwiGLU (Swish-Gated Linear Unit): advanced activation function and feed-forward layer (used in modern Transformer models)
- Swish: smooth, non-monotonic mathematical function $x$ × $sigmoid(x)$ that replaces older functions like ReLU
- GLU (Gated Linear Unit): activation function that splits an input into two parallel pathways, using one path as a gate to control the flow of information in the other
RMSNorm (Root Mean Square Layer Normalization):
- goal: stabilize and accelerate neural network training by scaling activations, while completely eliminating the computationally expensive mean-centering (subtraction) step
- alternative to standard Layer Normalization
-
- learned linear transformations that map input token vectors into Queries ($Q$), Keys ($K$), and Values ($V$)
- use trainable weight matrices to project data into specific subspaces, allowing models (like Transformers) to calculate relevance scores and dynamically route context between different parts of a sequence
- contextual refinement: update word meanings based on surrounding text
- relationship tracking: capture complex syntactic and semantic dependencies regardless of distance
- parallel processing: enable multi-head attention to analyze diverse perspectives of data simultaneously
- information routing: scale down dimensions for efficient scoring while retaining critical sequence information
note: code-specialized model is useful for this project because SQL generation is fundamentally a structured code-generation problem rather than ordinary natural-language classification
Dataset:
training data comes from: gretelai/synthetic_text_to_sql
dataset is designed for synthetic Text-to-SQL training and provides natural-language SQL requests, database context, target SQL, and related metadata
project uses three fields directly:
| Dataset field | Purpose |
|---|---|
sql_context |
Database schema or SQL context supplied to the model |
sql_prompt |
Natural language prompt request |
sql |
Ground-truth SQL completion |
local training configuration intentionally starts with a subset of the available data:
MAX_TRAIN_SAMPLES = 20_000
MAX_VALIDATION_SAMPLES = 1_000
GENERATION_EVAL_SAMPLES = 50
makes it easier to validate the complete training pipeline before increasing the training set size
LoRA?
LoRA (Low-Rank Adaptation), is a parameter-efficient fine-tuning method, (Hu et al. in LoRA: Low-Rank Adaptation of Large Language Models.)
instead of updating every weight in the pretrained model, LoRA:
- freezes the original pretrained weight matrices
- introduces small trainable matrices alongside selected layers
- learns a low-rank update to the original weights
- stores only those adapter parameters after training
Benefits of LoRA:
- Fewer Trainable Parameters: reduces the number of weights (compute efficient)
- Low Hardware Requirements: lowers GPU memory needs, allows me to fine-tune large models on (consumer-grade gpus) instead of massive server clusters
- Faster Training Speeds: shortens fine-tuning cycles from days to hours because the system processes much smaller matrix updates
- High Modularity & Storage Savings: generates tiny adapter files (often just megabytes instead of gigabytes) that can be swapped easily onto a single shared base model
- Zero Inference Latency: allows trained adapter weights to merge directly back into the primary model structure before deployment, retaining full processing speed
makes task adaptation more parameter-efficient than full fine-tuning because the large pretrained matrix remains unchanged while only the low-rank update is optimized
How LoRA Is Applied (in This Project)
The LoRA configuration is defined in train.py:
peft_config = LoraConfig(
r=16,
lora_alpha=32,
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
)
Rank
r = 16
update matrices are constrained to rank 16
larger value of $r$ increases the number of trainable parameters and gives the adapter more representational capacity
smaller value reduces memory and storage requirements but places a stronger constraint on the update
LoRA Alpha
lora_alpha = 32
constant scaling factor that controls the overall strength or "volume" of the trained adapter weights relative to the frozen base model
divides the internal rank ($r$) to determine how much influence the new matrix updates have on final outputs
(Alpha $α$) 32 means the scaling constant (alpha ($α$)) that controls the strength of the fine-tuning updates is set to 32
using alpha ($α$) = 32 as a stable baseline standard
LoRA Dropout:
lora_dropout = 0.05
regularization hyperparameter that randomly turns off a fraction of activations in the low-rank adaptation layers during training. main purpose is to prevent overfitting by stopping the adapter from relying too heavily on specific nodes, which helps the model generalize better to new data
5% dropout rate is configured on the LoRA branch during training. this provides regularization to the adapter path while the pretrained base weights remain frozen
lora_dropout = 0.05 means a 5% dropout rate is applied to the LoRA adapter layers. During training, the model randomly turns off 5% of the adapter weights in those layers at each step. this prevents the model from memorizing the training data too well
0.05 (5%): good for small or medium datasets where the model might start to overfit
Training Objective:
using supervised fine-tuning with TRL's SFTTrainer
each dataset row is converted to a conversational prompt-completion example
System:
You are an expert SQL generator...
User:
Database context:
<schema>
Request:
<natural-language request>
Assistant:
<target SQL>
Training Configuration:
| Setting | Value |
|---|---|
| Base model | Qwen/Qwen2.5-Coder-0.5B-Instruct |
| LoRA rank | 16 |
| LoRA alpha | 32 |
| LoRA dropout | 0.05 |
| Target modules | q_proj, k_proj, v_proj, o_proj |
| Epochs | 2 |
| Train batch size | 4 |
| Gradient accumulation | 4 |
| Effective batch per device | 16 sequences |
| Learning rate | 2e-4 |
| Warmup ratio | 0.03 |
| Weight decay | 0.01 |
| LR scheduler | Cosine |
| Maximum sequence length | 2048 |
| Gradient checkpointing | Enabled |
| Training objective | Completion-only SFT |
| Best checkpoint metric | Validation loss |
script automatically uses BF16 (data type) when supported, otherwise FP16 on CUDA, and FP32 when running without CUDA
Training Pipeline
Gretel Text-to-SQL Dataset
|
v
Shuffle / Sample
|
v
Prompt + Completion Formatting
|
v
Qwen Tokenizer
|
v
Qwen2.5-Coder-0.5B-Instruct
Frozen Base Weights
+
LoRA Attention Adapters
|
v
SFTTrainer
|
v
Trained LoRA Adapter
|
v
Base Model + Adapter at Inference
|
v
Generated SQL
|
v
SQLGlot Validation
|
v
Gradio UI
train.py
loads up Hugging Face dataset, formats prompt-completion examples, loads Qwen, attaches LoRA adapters, performs supervised fine-tuning, and saves the trained adapter
inference.py
loads the original Qwen model and then loads the trained PEFT adapter on top of it. provides command-line Text-to-SQL generation
evaluate.py
generates predictions from held-out examples and reports:
- normalized exact-match accuracy,
- SQL syntax validity using SQLGlot,
- row-level predictions within evaluation_results.csv
app.py
UI interface (Hugging Face Spaces)
applications include:
- SQL schema editor
- natural-language request input
- generated SQL editor
- copy and download controls
- SQL dialect selection
- temperature control
- output token control
- SQLGlot syntax validation
- model diagnostics
- example Text-to-SQL prompts
- LoRA adapter loading
- base-model fallback when an adapter is unavailable
Why LoRA Fits This Project
Limitations:
- the model can still hallucinate tables or columns (if context is insufficient)
- the 0.5B model prioritizes a lightweight training and deployment footprint over maximum reasoning capacity
- training (fine-tuning) on synthetic examples may not represent every schema style or production SQL workload
Validation Progression by Checkpoint
metrics from the completed RTX 3050 LoRA fine-tuning run
| Checkpoint | Epoch | Eval Loss | Eval Token Accuracy |
|---|---|---|---|
| 100 | 0.08 | 0.2839 | 92.06% |
| 200 | 0.16 | 0.2682 | 92.41% |
| 300 | 0.24 | 0.2589 | 92.65% |
| 400 | 0.32 | 0.2518 | 92.84% |
| 500 | 0.40 | 0.2481 | 92.93% |
| 600 | 0.48 | 0.2440 | 92.84% |
| 700 | 0.56 | 0.2429 | 92.93% |
| 800 | 0.64 | 0.2386 | 93.14% |
| 900 | 0.72 | 0.2376 | 93.12% |
| 1000 | 0.80 | 0.2343 | 93.23% |
| 1100 | 0.88 | 0.2328 | 93.24% |
| 1200 | 0.96 | 0.2306 | 93.26% |
| 1300 | 1.04 | 0.2281 | 93.38% |
| 1400 | 1.12 | 0.2287 | 93.45% |
| 1500 | 1.20 | 0.2275 | 93.40% |
| 1600 | 1.28 | 0.2251 | 93.47% |
| 1700 | 1.36 | 0.2232 | 93.48% |
| 1800 | 1.44 | 0.2244 | 93.43% |
| 1900 | 1.52 | 0.2223 | 93.62% |
| 2000 | 1.60 | 0.2213 | 93.57% |
| 2100 | 1.68 | 0.2208 | 93.53% |
| 2200 | 1.76 | 0.2203 | 93.53% |
| 2300 | 1.84 | 0.2202 | 93.56% |
| 2400 | 1.92 | 0.2198 | 93.54% |
| 2500 | 2.00 | 0.2200 | 93.54% |
validation loss improved from 0.2839 at the first logged evaluation to a best value of 0.2198 at checkpoint 2400 / epoch 1.92
final evaluation at checkpoint 2500 / epoch 2.00 produced an eval loss of 0.2200, which is flat relative to the best checkpoint. this shows that the model converged near the end of epoch 2 rather than showing a large late-stage validation-loss increase
Final Training Summary
| Metric | Result |
|---|---|
| Final training loss | 0.2381 |
| Training runtime | 5735 seconds / 1:35:34 |
| Samples per second | 6.975 |
| Optimizer steps per second | 0.436 |
| Completed epochs | 2 |
| Total optimizer steps | 2,500 |
| Best checkpoint | 2400 |
| Best eval loss | 0.2198 |
| Final eval loss | 0.2200 |
| Final eval token accuracy | 93.54% |
Held-Out Generation Evaluation
| Metric | Result |
|---|---|
| Evaluation examples | 50 |
| Exact-match accuracy | 28.0% |
| SQL syntax validity | 100.0% |
| Exact matches | 14 / 50 |
| Syntax-valid generations | 50 / 50 |
the 100% SQL syntax-validity rate means every generated query in this 50-example evaluation was parsable by SQLGlot
the 28% exact-match score is strict. different SQL strings can still be semantically or execution-equivalent, so exact match should not be treated as the model's true semantic accuracy
Results:
ml model completed run shows four useful outcomes:
- successful GPU fine-tuning on an NVIDIA GeForce RTX 3050 using CUDA 12.8 and BF16.
- parameter-efficient adaptation, with only 2,162,688 of 496,195,456 parameters trainable, or 0.4359%.
- stable convergence, with validation loss falling into the ~0.22 range by the end of training.
- reliable SQL-formatted generation, with 50/50 generated evaluation queries parsing successfully.
References:
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. *LoRA: Low-Rank Adaptation of Large Language Models*. arXiv:2106.09685, 2021.](https://arxiv.org/abs/2106.09685)
Hugging Face. *PEFT LoRA Documentation*. Parameter-Efficient Fine-Tuning documentation. (https://huggingface.co/docs/transformers/en/peft) (https://huggingface.co/docs/peft/en/package_reference/lora)
Qwen Team. Qwen2.5-Coder-0.5B-Instruct Model Card
Hui, B. et al. Qwen2.5-Coder Technical Report. arXiv:2409.12186, 2024
Gretel.ai. synthetic_text_to_sql Dataset Card. Hugging Face Datasets