aijadugar/qwen2.5-0.5b-sql-qlora

Qwen2.5-0.5B-Instruct fine-tuned with QLoRA (Text-to-SQL)

This model is Qwen/Qwen2.5-0.5B-Instruct fine-tuned on a small slice of b-mc2/sql-create-context using QLoRA: a 4-bit (NF4) quantized frozen base model via bitsandbytes, with a LoRA adapter (rank=8, alpha=16) trained on top via peft, then merged back into full precision.

Training details

• Base model: Qwen/Qwen2.5-0.5B-Instruct, loaded in 4-bit NF4 with double quantization
• Method: QLoRA (rank=8, alpha=16), applied to q_proj, k_proj, v_proj, o_proj
• Dataset: 250 examples from b-mc2/sql-create-context
• Hardware: Kaggle, 1x Tesla T4

How to use

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("YOUR_USERNAME/YOUR_REPO_NAME")
tokenizer = AutoTokenizer.from_pretrained("YOUR_USERNAME/YOUR_REPO_NAME")

This model is a learning exercise, not a production system -- expect rough edges.