# Quantization Report

## Project Information

| Item                   | Value                            |
| ---------------------- | -------------------------------- |
| Model Name             | Phi-4 Mini Instruct              |
| Base Repository        | microsoft/Phi-4-mini-instruct    |
| Quantization Author    | K VIGNESH                        |
| Quantization Framework | llama.cpp                        |
| Quantization Method    | Post-Training Quantization (PTQ) |
| Quantization Type      | Q8_0                             |
| Model Format           | GGUF                             |
| Operating System       | Windows 11                       |
| Hardware               | Intel Core i7-1165G7             |
| GPU Used               | No                               |

---

## Objective

The objective of this project was to convert Microsoft's Phi-4 Mini Instruct model from Hugging Face Safetensors format to GGUF and apply Post-Training Quantization (PTQ) to reduce model size while maintaining inference quality.

The resulting model can be executed efficiently on CPU-only systems and is compatible with GGUF-supported inference engines.

---

## Quantization Workflow

```text
Phi-4 Mini Instruct (Safetensors)
                ↓
      GGUF Conversion (F16)
                ↓
 Post-Training Quantization (Q8_0)
                ↓
      Optimized GGUF Model
```

### Step 1: Model Download

Downloaded the original model from Hugging Face:

```text
microsoft/Phi-4-mini-instruct
```

### Step 2: GGUF Conversion

Converted the Hugging Face Safetensors model to GGUF format using llama.cpp conversion utilities.

Output:

```text
phi4-f16.gguf
```

### Step 3: Quantization

Applied Q8_0 quantization using llama.cpp:

```bash
llama-quantize phi4-f16.gguf phi4-q8_0.gguf Q8_0
```

Output:

```text
phi4-q8_0.gguf
```

---

## Size Comparison

| Model Version       | Size    |
| ------------------- | ------- |
| Original F16 GGUF   | 7.15 GB |
| Quantized Q8_0 GGUF | 3.80 GB |

### Compression Achieved

```text
Size Reduction ≈ 47%
```

---

## Inference Validation

The quantized model was validated using llama.cpp.

Command:

```bash
llama-cli -m phi4-q8_0.gguf
```

The model successfully loaded and generated responses without requiring GPU acceleration.

---

## Compatibility

This model is compatible with:

* llama.cpp
* Ollama
* LM Studio
* GPT4All
* Jan
* llama-cpp-python
* Open WebUI

---

## Advantages of Quantization

* Reduced storage requirements
* Lower memory consumption
* Faster model loading
* Improved deployment on consumer hardware
* CPU-only execution support
* Easier local deployment

---

## Limitations

* Minor accuracy degradation compared to full precision models
* Quantized models may perform slightly differently on certain reasoning tasks
* Performance depends on available CPU resources

---

## Conclusion

The Phi-4 Mini Instruct model was successfully converted to GGUF format and quantized using the Q8_0 method.

The process reduced model size from approximately 7.15 GB to 3.80 GB while preserving usability for local inference workloads.

This quantized model is suitable for local AI applications, educational projects, research, and edge deployment scenarios.

---

## Author

K VIGNESH

GGUF Conversion, Quantization, Validation, and Deployment Testing performed using llama.cpp.