Snehanjan/llama-3.2-3b-spider-text-to-sql

Llama 3.2 3B Instruct — Text-to-SQL (Spider Dataset)

Model Description

This is a QLoRA fine-tuned LoRA adapter on top of meta-llama/Llama-3.2-3B-Instruct for text-to-SQL generation, trained on the Spider dataset.

Intended Use

• Convert natural language questions into SQL queries
• Works best with SQLite syntax
• Designed for single and multi-table queries on structured schemas

How To Use

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model_id = "meta-llama/Llama-3.2-3B-Instruct"
adapter_id = "Snehanjan/llama-3.2-3b-spider-text-to-sql"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)

model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

model = PeftModel.from_pretrained(model, adapter_id)

def generate_sql(schema_and_question):
    prompt = (
        "<|begin_of_text|>"
        "<|start_header_id|>system<|end_header_id|>\n\n"
        "You are an expert SQL assistant. Given a database schema and a natural language question, "
        "generate the correct SQL query. Output ONLY the SQL query, no explanations.<|eot_id|>"
        "<|start_header_id|>user<|end_header_id|>\n\n"
        f"{schema_and_question}<|eot_id|>"
        "<|start_header_id|>assistant<|end_header_id|>\n\n"
    )
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=128,
            temperature=0.1,
            do_sample=False,
        )
    response = tokenizer.decode(
        outputs[0][inputs.input_ids.shape[1]:], 
        skip_special_tokens=True
    )
    return response.strip()

Training Details

Parameter	Value
Base Model	meta-llama/Llama-3.2-3B-Instruct
Dataset	lamini/spider_text_to_sql
Method	QLoRA (4-bit NF4 + LoRA)
LoRA Rank	16
LoRA Alpha	32
Target Modules	all-linear
Learning Rate	2e-4
LR Scheduler	cosine
Batch Size	4
Gradient Accumulation	2
Effective Batch Size	8
Epochs	1
Max Sequence Length	512
Hardware	Kaggle T4 (15GB VRAM)

Training Metrics

Step	Training Loss
10	1.850
100	0.236
500	0.090
870	0.098

• Initial Loss : 1.850
• Final Loss : 0.098
• Loss Reduction: ~95%

Limitations

• Trained on Spider dataset only — may not generalize to all SQL dialects
• Best suited for SQLite syntax
• May struggle with very complex nested subqueries
• Schema must be provided in the same format as Spider dataset

Framework Versions

• PEFT
• Transformers
• TRL 1.5.1
• bitsandbytes
• PyTorch