The objective of this project was to convert Microsoft's Phi-4 Mini Instruct model from Hugging Face Safetensors format to GGUF and apply Post-Training Quantization (PTQ) to reduce model size while maintaining inference quality.
The resulting model can be executed efficiently on CPU-only systems and is compatible with GGUF-supported inference engines.
---
## Quantization Workflow
```text
Phi-4 Mini Instruct (Safetensors)
↓
GGUF Conversion (F16)
↓
Post-Training Quantization (Q8_0)
↓
Optimized GGUF Model
```
### Step 1: Model Download
Downloaded the original model from Hugging Face:
```text
microsoft/Phi-4-mini-instruct
```
### Step 2: GGUF Conversion
Converted the Hugging Face Safetensors model to GGUF format using llama.cpp conversion utilities.
Output:
```text
phi4-f16.gguf
```
### Step 3: Quantization
Applied Q8_0 quantization using llama.cpp:
```bash
llama-quantize phi4-f16.gguf phi4-q8_0.gguf Q8_0
```
Output:
```text
phi4-q8_0.gguf
```
---
## Size Comparison
| Model Version | Size |
| ------------------- | ------- |
| Original F16 GGUF | 7.15 GB |
| Quantized Q8_0 GGUF | 3.80 GB |
### Compression Achieved
```text
Size Reduction ≈ 47%
```
---
## Inference Validation
The quantized model was validated using llama.cpp.
Command:
```bash
llama-cli -m phi4-q8_0.gguf
```
The model successfully loaded and generated responses without requiring GPU acceleration.
---
## Compatibility
This model is compatible with:
* llama.cpp
* Ollama
* LM Studio
* GPT4All
* Jan
* llama-cpp-python
* Open WebUI
---
## Advantages of Quantization
* Reduced storage requirements
* Lower memory consumption
* Faster model loading
* Improved deployment on consumer hardware
* CPU-only execution support
* Easier local deployment
---
## Limitations
* Minor accuracy degradation compared to full precision models
* Quantized models may perform slightly differently on certain reasoning tasks
* Performance depends on available CPU resources
---
## Conclusion
The Phi-4 Mini Instruct model was successfully converted to GGUF format and quantized using the Q8_0 method.
The process reduced model size from approximately 7.15 GB to 3.80 GB while preserving usability for local inference workloads.
This quantized model is suitable for local AI applications, educational projects, research, and edge deployment scenarios.
---
## Author
K VIGNESH
GGUF Conversion, Quantization, Validation, and Deployment Testing performed using llama.cpp.
