| --- |
| license: mit |
| base_model: microsoft/phi-2 |
| tags: |
| - quantization |
| - 4bit |
| - bitsandbytes |
| - peft |
| - lora |
| pipeline_tag: text-generation |
| --- |
| |
| # Phi-2 4-bit Quantized Model (with LoRA Adapter) |
|
|
| This repository contains a **4-bit quantized version** of Microsoft's **Phi-2** model, prepared using `bitsandbytes` and wrapped with a LoRA adapter via the PEFT library. |
|
|
| ## Model Description |
| The original Phi-2 model runs on 16-bit precision (Float16) and consumes a lot of memory. To make it highly efficient and runnable on free-tier cloud GPUs (like Google Colab T4) or local machines with limited VRAM, this model has been compressed to 4-bit using NormalFloat4 (NF4) quantization. |
|
|
| ### Key Highlights & Comparison: |
| * **Base Model:** `microsoft/phi-2` |
| * **Quantization Tech:** 4-bit NF4 (`bitsandbytes`) |
| * **PEFT Framework:** LoRA Adapter added for efficient fine-tuning/saving configuration. |
| * **Memory Footprint Optimization:** |
| * **Original Model Size:** ~5.56 GB (Float16) |
| * **Quantized Model Size:** **~1.78 GB (4-bit)** |
| * **Size Reduction:** **~68% lower VRAM usage** with minimal drop in response quality! |
|
|
| ## How It Was Made |
| 1. **Optimization:** Loaded the model using `BitsAndBytesConfig` with `load_in_4bit=True` and `bnb_4bit_compute_dtype=torch.float16`. |
| 2. **LoRA Integration:** Prepared the model for k-bit training and attached a `LoraConfig` targeting the standard query/value projection layers (`q_proj`, `v_proj`). |
| 3. **Saving:** Saved the lightweight PEFT adapter weights (`adapter_model.safetensors`) and tokenizer configuration. |
|
|
| ## Intended Use |
| This repository is perfect for anyone looking to experiment with lightweight text generation or perform Parameter-Efficient Fine-Tuning (PEFT) on top of a 4-bit quantized version of Phi-2. |
