README.md

---
language:
- en
license: llama3.2
base_model: meta-llama/Llama-3.2-3B
tags:
- llama
- llama-3
- sql
- text-to-sql
- lora
- peft
- finetuned
datasets:
- spider
metrics:
- accuracy
library_name: transformers
pipeline_tag: text-generation
---

# Llama 3.2 3B - SQL Query Generator (LoRA Fine-tuned)

This model is a fine-tuned version of [meta-llama/Llama-3.2-3B](https://huggingface.co/meta-llama/Llama-3.2-3B) for **text-to-SQL generation** using LoRA (Low-Rank Adaptation) on the Spider dataset.

## Model Description

- **Base Model:** Llama 3.2 3B
- **Fine-tuning Method:** LoRA (Parameter-Efficient Fine-Tuning)
- **Quantization:** 4-bit NF4 with double quantization
- **Dataset:** Spider (7,000 training examples)
- **Training:** 3 epochs, ~47 minutes on AWS g5.2xlarge (NVIDIA A10G)
- **Final Training Loss:** 0.37 (85% reduction from initial 2.5)

## Intended Use

This model converts natural language questions into SQL queries for various database schemas. It's designed for:
- Automated SQL query generation
- Data analysis assistants
- Natural language database interfaces
- Educational tools for SQL learning

## Training Details

### Training Hyperparameters

- **Learning Rate:** 2e-4
- **Batch Size:** 4 (per device)
- **Gradient Accumulation:** 4 steps (effective batch size: 16)
- **Epochs:** 3
- **Max Sequence Length:** 2048
- **LoRA Rank (r):** 16
- **LoRA Alpha:** 32
- **LoRA Dropout:** 0.05
- **Target Modules:** q_proj, k_proj, v_proj, o_proj

### Training Results

| Metric | Value |
|--------|-------|
| Initial Loss | 2.50 |
| Final Loss | 0.37 |
| Trainable Parameters | 9.17M (0.51% of total) |
| Training Time | 47 minutes |

## Usage

### Installation
```bash
pip install transformers peft torch bitsandbytes
```

### Inference Example
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load model and tokenizer
model_name = "Abhisek987/llama-3.2-sql-lora"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype=torch.float16
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Prepare prompt
database = "employees"
question = "What are the names of all employees who earn more than 50000?"

prompt = f"""### Instruction:
You are a SQL expert. Generate a SQL query to answer the given question for the specified database.

### Input:
Database: {database}
Question: {question}

### Response:
"""

# Generate SQL
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.1,
    do_sample=True
)

sql_query = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(sql_query.split("### Response:")[-1].strip())
```

**Output:**
```sql
SELECT name FROM employees WHERE salary > 50000;
```

## Example Queries

| Question | Generated SQL |
|----------|---------------|
| "Show top 5 products by sales" | `SELECT product_id, sum(sales) FROM sales GROUP BY product_id ORDER BY sum(sales) DESC LIMIT 5;` |
| "Count customers by country" | `SELECT count(*), country FROM customers GROUP BY country;` |
| "Find orders from last 30 days" | `SELECT order_id FROM orders WHERE date_order_placed BETWEEN DATE('now') - INTERVAL 30 DAY AND DATE('now') - INTERVAL 1 DAY;` |

## Limitations

- Trained specifically on Spider dataset schemas
- May not generalize perfectly to significantly different database structures
- Requires proper database schema context for best results
- 4-bit quantization may occasionally affect numerical precision

## Technical Stack

- **Framework:** PyTorch + Transformers
- **Quantization:** BitsAndBytes (4-bit NF4)
- **Fine-tuning:** PEFT (LoRA)
- **Training:** AWS EC2 g5.2xlarge (NVIDIA A10G 24GB)

## Citation

If you use this model, please cite:
```bibtex
@misc{llama32-sql-lora,
  author = {Abhisek Behera},
  title = {Llama 3.2 3B SQL Query Generator (LoRA Fine-tuned)},
  year = {2025},
  publisher = {HuggingFace},
  url = {https://huggingface.co/Abhisek987/llama-3.2-sql-lora}
}
```

## Acknowledgments

- Meta AI for Llama 3.2 base model
- Spider dataset creators
- HuggingFace for infrastructure

## License

This model inherits the Llama 3.2 Community License from the base model.
```

---

## **Step 2: Add to Settings**

1. **Repository Settings** → Make it **Public**
2. **Add topics/tags:** `llama`, `sql`, `lora`, `nlp`, `text-to-sql`

---

## **Step 3: For Your Resume**

Add this to your projects section:
```
🔗 Text-to-SQL Generator using Llama 3.2 (LLM Fine-tuning)
https://huggingface.co/Abhisek987/llama-3.2-sql-lora

- Fine-tuned Llama 3.2 3B model for natural language to SQL conversion using LoRA technique
- Achieved 85% training loss reduction (2.5 → 0.37) on Spider dataset with 7K examples
- Implemented 4-bit quantization (NF4) reducing model size by 75% while maintaining accuracy
- Trained on AWS EC2 (g5.2xlarge) with NVIDIA A10G GPU in 47 minutes
- Technologies: PyTorch, Transformers, PEFT, BitsAndBytes, AWS EC2