thefounder03/qwen2.5-7b-qlora-nl2sql

Qwen2.5-7B-Instruct — QLoRA NL2SQL Adapter

LoRA adapter weights for Qwen/Qwen2.5-7B-Instruct, fine-tuned on the Spider text-to-SQL dataset using a from-scratch QLoRA implementation — no bitsandbytes, no peft, no black-box abstractions.

Full training code, implementation details, and loading instructions: github.com/dhruvkachhela/fine_tunning_QLORA

---

Training Details

Setting	Value
Base model	Qwen/Qwen2.5-7B-Instruct
Dataset	Spider (7,000 train / 1,034 val)
Task	Natural language to SQL
Quantization	NF4 4-bit, block size 64, double quantization (8-bit scales)
LoRA rank	8
LoRA alpha	16
Target modules	q_proj, v_proj (all 28 layers)
Trainable params	2,523,136 (0.036% of 7B)
Learning rate	1e-4 with linear warmup + decay
Training steps	1,000
Batch size	2
Hardware	Kaggle T4 GPU (16GB VRAM)

---

What is in this repo

File	Description
best_qlora_checkpoint.pt	LoRA adapter weights (lora_A + lora_B, all 28 layers)

Base model weights are not included — load Qwen/Qwen2.5-7B-Instruct directly from HuggingFace.

---

Implementation

Built from scratch in PyTorch:

• NF4 Codebook — 16 levels at quantiles of a standard normal distribution
• Block-wise Scaling — block size 64, per-block absmax normalization
• Double Quantization — scale factors quantized from float32 to 8-bit
• LoRA — rank-8 adapters on q_proj + v_proj, only lora_A and lora_B train
• Dequantize on the fly — weights reconstructed each forward pass

---

References

• QLoRA paper — Dettmers et al., 2023
• LoRA paper — Hu et al., 2021
• Spider dataset — Yu et al., 2018
• Training repo: github.com/dhruvkachhela/fine_tunning_QLORA