README.md

metadata

license: mit
base_model: microsoft/phi-2
tags:
  - quantization
  - 4bit
  - bitsandbytes
  - peft
  - lora
pipeline_tag: text-generation

Phi-2 4-bit Quantized Model (with LoRA Adapter)

This repository contains a 4-bit quantized version of Microsoft's Phi-2 model, prepared using bitsandbytes and wrapped with a LoRA adapter via the PEFT library.

Model Description

The original Phi-2 model runs on 16-bit precision (Float16) and consumes a lot of memory. To make it highly efficient and runnable on free-tier cloud GPUs (like Google Colab T4) or local machines with limited VRAM, this model has been compressed to 4-bit using NormalFloat4 (NF4) quantization.

Key Highlights & Comparison:

• Base Model: microsoft/phi-2
• Quantization Tech: 4-bit NF4 (bitsandbytes)
• PEFT Framework: LoRA Adapter added for efficient fine-tuning/saving configuration.
• Memory Footprint Optimization:
  • Original Model Size: ~5.56 GB (Float16)
  • Quantized Model Size: ~1.78 GB (4-bit)
  • Size Reduction: ~68% lower VRAM usage with minimal drop in response quality!

How It Was Made

1. Optimization: Loaded the model using BitsAndBytesConfig with load_in_4bit=True and bnb_4bit_compute_dtype=torch.float16.
2. LoRA Integration: Prepared the model for k-bit training and attached a LoraConfig targeting the standard query/value projection layers (q_proj, v_proj).
3. Saving: Saved the lightweight PEFT adapter weights (adapter_model.safetensors) and tokenizer configuration.

Intended Use

This repository is perfect for anyone looking to experiment with lightweight text generation or perform Parameter-Efficient Fine-Tuning (PEFT) on top of a 4-bit quantized version of Phi-2.