Sandeep4235/qwen2.5-7b-adapter

Qwen2.5-7B-Instruct-LoRA-Adapter (Fine-tuned on T4 with Unsloth)

This repository contains a fine-tuned LoRA adapter for the Qwen2.5-7B-Instruct model, optimized for a single NVIDIA T4 GPU using Unsloth and the trl library. The model has been fine-tuned on a subset of the Alpaca dataset to enhance its instructional following capabilities.

Model Details

• Base Model: unsloth/Qwen2.5-7B-Instruct-bnb-4bit (4-bit quantized)
• Fine-tuning Method: LoRA (Low-Rank Adaptation) with QLoRA
• Dataset: yahma/alpaca-cleaned (a cleaned version of the Alpaca instruction dataset)
• Hardware: NVIDIA T4 GPU (16GB VRAM)
• Quantization (GGUF Export): Q4_K_M (if exported)

Training Summary

• Framework: Unsloth for optimized training and trl.SFTTrainer
• LoRA Rank (r): 16
• LoRA Alpha: 16
• Gradient Checkpointing: Enabled (unsloth optimized)
• Precision: FP16 (due to T4 hardware limitations, BF16 not natively supported)
• Batch Size: per_device_train_batch_size=2, gradient_accumulation_steps=4 (effective batch size = 8)
• Learning Rate: 2e-4
• Optimizer: adamw_8bit
• Max Sequence Length: 2048

How to Use This LoRA Adapter

To use this fine-tuned adapter, you will need to load the base Qwen2.5-7B-Instruct-bnb-4bit model and then merge the LoRA weights.

1. Install Dependencies

pip install --quiet "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
pip install --quiet --no-deps trl peft accelerate bitsandbytes

2. Load the Base Model and LoRA Adapter

import torch
from unsloth import FastLanguageModel
from transformers import AutoTokenizer

max_seq_length = 2048 # Adjust as needed
model_name = "unsloth/Qwen2.5-7B-Instruct-bnb-4bit"
adapter_path = "Sandeep4235/qwen2.5-7b-adapter" # Replace with your Hugging Face Hub path or local path

# Load 4-bit quantized base model
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=model_name,
    max_seq_length=max_seq_length,
    dtype=None, # Auto detects FP16 for T4
    load_in_4bit=True,
)

# Load the LoRA adapter
model = FastLanguageModel.get_peft_model(
    model,
    r=16, # Must match the 'r' value used during training
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj"
    ],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing=False, # Not needed for inference
)

# Load the adapter weights (from Hugging Face Hub or local path)
model.load_adapter(adapter_path)

# Optional: Merge LoRA weights for faster inference (not always necessary with Unsloth)
# model = model.merge_and_unload()

3. Inference

from transformers import TextStreamer

FastLanguageModel.for_inference(model) # Enable Unsloth's faster inference engine

messages = [
    {"role": "system", "content": "You are a helpful, expert AI assistant."},
    {"role": "user", "content": "Explain the concept of quantum entanglement in simple terms."}
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to("cuda")

text_streamer = TextStreamer(tokenizer, skip_prompt=True)

outputs = model.generate(
    input_ids=inputs,
    streamer=text_streamer,
    max_new_tokens=512,
    use_cache=True,
    temperature=0.7,
    min_p=0.1
)

GGUF Quantization (Optional)

If you want to use this model with llama.cpp or Ollama, you can export it to the GGUF format with q4_k_m quantization.

# Assumes 'model' and 'tokenizer' are already loaded and merged
# Make sure to set export_gguf = True in the export cell (if running this notebook)

# model.save_pretrained_gguf(
#     "qwen2.5_gguf",
#     tokenizer,
#     quantization_method="q4_k_m"
# )
# print("GGUF model saved to ./qwen2.5_gguf")

This will create a qwen2.5_gguf directory containing the GGUF formatted model. You can then use tools like llama.cpp to run inference locally.