File size: 3,179 Bytes
646a0c6 a809170 646a0c6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 | ---
license: mit
language:
- en
base_model:
- microsoft/phi-4
tags:
- 4bit
- transformers
- autoawq
- vllm
- 12gb-vram
---
# Microsoft Phi-4 4-bit AWQ Quantized Model (GEMM)
This is a **4-bit AutoAWQ quantized version** of [Microsoft's Phi-4](https://huggingface.co/microsoft/phi-4).
It is optimized for **fast inference** using **vLLM** with minimal loss in accuracy.
---
## 🚀 Model Details
- **Base Model:** [microsoft/phi-4](https://huggingface.co/microsoft/phi-4)
- **Quantization:** **4-bit AWQ**
- **Quantization Method:** **AutoAWQ (Activation-Aware Quantization)**
- **Group Size:** 128
- **AWQ Version:** GEMM Optimized
- **Intended Use:** **Low VRAM inference on consumer GPUs**
- **VRAM Requirements:** ✅ **8GB+ (Recommended)**
- **Compatibility:** ✅ **vLLM, Hugging Face Transformers (w/ AWQ support)**
---
## 📌 How to Use in vLLM
You can load this model directly in **vLLM** for efficient inference:
```bash
vllm serve "curiousmind147/microsoft-phi-4-AWQ-4bit-GEMM"
```
Then, test it using `cURL`:
```bash
curl -X POST "http://localhost:8000/generate" \
-H "Content-Type: application/json" \
-d '{"prompt": "Explain quantum mechanics in simple terms.", "max_tokens": 100}'
```
---
## 📌 How to Use in Python (`transformers` + AWQ)
To use this model with **Hugging Face Transformers**:
```python
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
model_path = "curiousmind147/microsoft-phi-4-AWQ-4bit-GEMM"
model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)
inputs = tokenizer("What is the meaning of life?", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```
---
## 📌 Quantization Details
This model was quantized using **AutoAWQ** with the following parameters:
- **Bits:** 4-bit quantization
- **Zero-Point Quantization:** Enabled (`zero_point=True`)
- **Group Size:** 128 (`q_group_size=128`)
- **Quantization Version:** `GEMM`
- **Method Used:** [AutoAWQ](https://github.com/casper-hansen/AutoAWQ)
---
## 📌 VRAM Requirements
| Model Size | **FP16 (No Quant)** | **AWQ 4-bit Quantized** |
|------------|-------------------|-------------------------|
| **Phi-4 14B** | ❌ Requires **>20GB VRAM** | ✅ **8GB-12GB VRAM** |
AWQ significantly **reduces VRAM requirements**, making it **possible to run 14B models on consumer GPUs**. 🚀
---
## 📌 License & Credits
- **Base Model:** [Microsoft Phi-4](https://huggingface.co/microsoft/phi-4)
- **Quantized by:** [curiousmind147](https://huggingface.co/curiousmind147)
- **License:** Same as the base model (Microsoft)
- **Credits:** This model is based on Microsoft's Phi-4 and was optimized using AutoAWQ.
---
## 📌 Acknowledgments
Special thanks to:
- **Microsoft** for creating [Phi-4](https://huggingface.co/microsoft/phi-4).
- **Casper Hansen** for developing [AutoAWQ](https://github.com/casper-hansen/AutoAWQ).
- **The vLLM team** for making fast inference possible.
---
## 🚀 Enjoy Efficient Phi-4 Inference!
If you find this useful, **give it a ⭐ on Hugging Face!** 🎯 |