thefounder03/qlora-nl2sql-qwen2.5-7b

QLoRA Text-to-SQL Adapter (From-Scratch Quantization)

This repository contains LoRA adapter weights fine-tuned on the Spider text-to-SQL dataset.

The base model was quantized to 4-bit NormalFloat (NF4) using a custom, from-scratch PyTorch implementation of block-wise quantization and 8-bit double quantization of scale factors, then injected with trainable LoRA adapters on attention projection layers.

Model Description

• Base Model: Qwen/Qwen2.5-7B-Instruct
• Adapter Rank: $r = 8$, $lpha = 16$
• Target Modules: q_proj, v_proj
• Quantization Configuration:
  • Weight Block Size: 64
  • Weight Quantization Type: NormalFloat4 (NF4)
  • Scale Block Size: 256
  • Scale Quantization Type: 8-bit Uniform Quantization (Double Quantization)

Training Configuration & Metrics

• Max Sequence Length: 1024 (prompts padded/truncated to fit without loss of SQL target queries)
• Prompt Format: ChatML template containing database schema context retrieved from richardr1126/spider-schema.
• Learning Rate: 1e-4 with Linear Warmup (10% of steps) and linear decay.
• Batch Size: 1 (Gradient Accumulation Steps: 2)
• Best Validation Loss: 0.2434 at step 350 / 1000

Evaluation Results (300 Validation Subset)

• Raw Exact Match: 10.67% (32 / 300)
• Case/Space Normalized Exact Match: 35.33% (106 / 300)
• Case/Space/Quote Normalized Exact Match: 44.00% (132 / 300)

How to Load & Dequantize Manually

Since these weights were trained using a custom from-scratch PyTorch quantization class, loading the model requires replacing the linear layers with QuantizedLinear before copying the adapter weights:

import torch
from transformers import AutoModelForCausalLM

# 1. Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct", 
    torch_dtype=torch.float16,
    device_map="auto"
)

# 2. Re-create the NF4 codebook
cb = make_nf4_codebook()

# 3. Swap linear layers to custom QuantizedLinear in-place
for layer in base_model.model.layers:
    layer.self_attn.q_proj = QuantizedLinear.from_linear(layer.self_attn.q_proj, cb)
    layer.self_attn.v_proj = QuantizedLinear.from_linear(layer.self_attn.v_proj, cb)

# 4. Inject LoRA adapters
inject_lora(base_model, r=8, alpha=16)

# 5. Load adapter weights from checkpoint
checkpoint = torch.load("best_qlora_checkpoint.pt")
lora_state = checkpoint["lora_state_dict"]

for name, param in base_model.named_parameters():
    if name in lora_state:
        param.data.copy_(lora_state[name].to(param.device))

print("Model successfully loaded with custom quantized weights!")