---
license: mit
base_model: microsoft/phi-2
tags:
- quantization
- 4bit
- bitsandbytes
- peft
- lora
pipeline_tag: text-generation
---

# Phi-2 4-bit Quantized Model (with LoRA Adapter)

This repository contains a **4-bit quantized version** of Microsoft's **Phi-2** model, prepared using `bitsandbytes` and wrapped with a LoRA adapter via the PEFT library. 

## Model Description
The original Phi-2 model runs on 16-bit precision (Float16) and consumes a lot of memory. To make it highly efficient and runnable on free-tier cloud GPUs (like Google Colab T4) or local machines with limited VRAM, this model has been compressed to 4-bit using NormalFloat4 (NF4) quantization.

### Key Highlights & Comparison:
* **Base Model:** `microsoft/phi-2`
* **Quantization Tech:** 4-bit NF4 (`bitsandbytes`)
* **PEFT Framework:** LoRA Adapter added for efficient fine-tuning/saving configuration.
* **Memory Footprint Optimization:**
  * **Original Model Size:** ~5.56 GB (Float16)
  * **Quantized Model Size:** **~1.78 GB (4-bit)**
  * **Size Reduction:** **~68% lower VRAM usage** with minimal drop in response quality!

## How It Was Made
1. **Optimization:** Loaded the model using `BitsAndBytesConfig` with `load_in_4bit=True` and `bnb_4bit_compute_dtype=torch.float16`.
2. **LoRA Integration:** Prepared the model for k-bit training and attached a `LoraConfig` targeting the standard query/value projection layers (`q_proj`, `v_proj`).
3. **Saving:** Saved the lightweight PEFT adapter weights (`adapter_model.safetensors`) and tokenizer configuration.

## Intended Use
This repository is perfect for anyone looking to experiment with lightweight text generation or perform Parameter-Efficient Fine-Tuning (PEFT) on top of a 4-bit quantized version of Phi-2.