Text2SQL Qwen-2.5-Coder DuckDB (Fine-Tuned)

A specialized Text-to-SQL Large Language Model fine-tuned on Qwen/Qwen2.5-Coder-7B-Instruct using QLoRA (Quantized Low-Rank Adaptation). It is specifically optimized to translate natural language business analytics questions into executable, read-only analytical DuckDB SQL queries (including multi-table JOINs, Common Table Expressions (CTEs), Window Functions, and relative date arithmetic).


๐ŸŽฏ Model Capabilities

  • Analytical Aggregations: Computes single and multi-column aggregations (SUM, AVG, COUNT(DISTINCT)).
  • Relational Joins: Resolves star-schema foreign key relationships (INNER JOIN, LEFT JOIN) with explicit ON clauses.
  • Window Functions: Generates analytical partitions and rankings (RANK() OVER (PARTITION BY ... ORDER BY ...)).
  • Common Table Expressions (CTEs): Synthesizes multi-stage CTEs (WITH sub AS (...) SELECT ...) for complex business logic.
  • DuckDB Date Arithmetic: Natively applies DuckDB temporal functions (LAST_DAY(), DATE_TRUNC(), DATEDIFF()).
  • Read-Only Safety: Strictly structured to produce read-only SELECT statements without destructive DDL/DML mutations.

๐Ÿ“Š Training Dataset & Curation

  • Dataset Size: 847 domain-specific natural language to SQL pairs.
  • Format: ChatML prompt structure (<|im_start|>system...<|im_end|>).
  • Domain Coverage:
    • E-commerce sales, customer transactions, inventory, returns, and time-series analytics.
    • Multi-table star schemas with foreign-key joins.
    • Complex boundary conditions: N-th ordinal ranks (e.g. "second highest"), relative month/year boundaries (e.g. "last day of July"), and percentage calculations.

โš™๏ธ Training Hyperparameters (QLoRA)

The model was fine-tuned in 4-bit precision on a single Google Colab T4 GPU (16GB VRAM) using Unsloth and Hugging Face TRL's SFTTrainer:

Hyperparameter Value Description
Quantization 4-bit NormalFloat (NF4) bitsandbytes double quantization with bfloat16 compute
LoRA Rank ($r$) 16 Rank of the low-rank adaptation decomposition matrices
LoRA Alpha ($\alpha$) 32 Scaling factor for adapter updates ($\alpha / r = 2.0$)
LoRA Dropout 0.0 Optimized for Unsloth kernel acceleration
Target Modules All Linear Projections q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Optimizer adamw_8bit 8-bit quantized AdamW to minimize optimizer state VRAM
Learning Rate $2 \times 10^{-4}$ Linear decay schedule with warmup
Batch Size 2 per device Gradient Accumulation Steps = 4 (Effective Batch Size = 8)
Max Sequence Length 2,048 tokens Context window covering table schemas, foreign keys, and exemplars
Loss Function Cross-Entropy Loss Masked on user prompt, computed only on assistant SQL response

๐Ÿ’ฌ Prompt Template (ChatML)

<|im_start|>system
You are an expert DuckDB SQL Data Analyst writing read-only SELECT queries. Use ONLY the tables and columns provided in the database schema context. Return ONLY the SQL query inside ```sql ... ``` code block.<|im_end|>
<|im_start|>user
Database Schema:
Table `orders`: order_id (INT), customer_id (INT), category (VARCHAR), net_amount (DECIMAL), order_date (DATE)

Question: "What is the total revenue for Furniture in 2024?"<|im_end|>
<|im_start|>assistant
```sql
SELECT SUM(net_amount) AS total_revenue
FROM orders
WHERE category = 'Furniture' AND YEAR(CAST(order_date AS DATE)) = 2024;
```<|im_end|>

๐Ÿš€ How to Run Locally

Using Transformers & PEFT:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Priyanka221105/text2sql-qwen2.5-duckdb"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

prompt = """<|im_start|>system
You are an expert DuckDB SQL Data Analyst writing read-only SELECT queries. Return ONLY the SQL query inside ```sql ... ``` code block.<|im_end|>
<|im_start|>user
Database Schema:
Table `orders`: order_id (INT), net_amount (DECIMAL), category (VARCHAR)

Question: "Top 3 categories by total sales"<|im_end|>
<|im_start|>assistant
"""

inputs = tokenizer(prompt, return_tensors="pt").to("cuda" if torch.cuda.is_available() else "cpu")
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.01)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))

๐Ÿ† Empirical Evaluation & Benchmark Results

The model was evaluated across 10 representative SQL complexity tiers (multi-table JOINs, window functions, CTEs, temporal arithmetic, and aggregations) against ground-truth reference queries executed on DuckDB:

Metric Score Evaluation Definition
AST Syntax Validity 100.0% Queries strictly parseable as single read-only DuckDB SQL without syntax errors
Valid Execution Rate 100.0% Queries executing successfully in DuckDB sandbox without runtime exceptions
Ground-Truth Execution Accuracy (EX %) 80.0% Exact tabular and numerical result set equivalence against gold SQL benchmarks
Deterministic Heuristic Latency (P50) 0.098 ms Sub-millisecond execution for single-table analytical queries via zero-shot compiler

โš ๏ธ Limitations & Guardrails

  1. Dialect Specificity: The model is optimized for DuckDB dialect. While DuckDB is standard ANSI SQL compliant, queries with temporal functions (e.g. LAST_DAY, DATE_TRUNC) may require transpilation (via SQLGlot) when executed on MySQL or PostgreSQL.
  2. Security Sandboxing: In production, generated queries should always be validated by an AST parser (like SQLGlot) to ensure single-statement read-only execution and prevent unintentional mutations.
Downloads last month
52
Safetensors
Model size
8B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Priyanka221105/text2sql-qwen2.5-duckdb

Base model

Qwen/Qwen2.5-7B
Adapter
(667)
this model