README.md

---
license: mit
base_model: microsoft/Phi-4-multimodal-instruct
tags:
  - phi4
  - phi4mm
  - gptq
  - quantized
  - compressed-tensors
  - vision-language
  - audio
  - multimodal
  - vllm
language:
  - en
pipeline_tag: visual-question-answering
library_name: transformers
---

# Phi-4-multimodal-instruct W4A16 GPTQ

GPTQ W4A16 quantization of [microsoft/Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) —
a **5.6 B parameter** multimodal model by Microsoft supporting **text, vision (images), and audio** inputs.

Quantized with [llm-compressor](https://github.com/vllm-project/llm-compressor) on RTX 5090.
Weights stored in [compressed-tensors](https://github.com/neuralmagic/compressed-tensors) format — natively loaded by vLLM.

> **License:** [MIT](https://huggingface.co/microsoft/Phi-4-multimodal-instruct/blob/main/LICENSE) — © Microsoft Corporation.
> This quantization carries the same MIT license as the original model.

---

## Why this quantization?

| | bf16 safetensors | GGUF (Q4_K_M + mmproj) | **This model (W4A16 GPTQ)** |
|---|---|---|---|
| Size | ~14 GB | 2.37 GB + 825 MB | **~5–6 GB** |
| Text | ✅ | ✅ | ✅ |
| Vision (images) | ✅ | ✅ | ✅ |
| Audio / Speech | ✅ | ❌ | ✅ |
| Serves with | vLLM | llama.cpp / LM Studio | **vLLM** |
| Quantization method | none | GGML int4 | **GPTQ int4 (W4A16)** |

> The GGUF files in [Swicked86/phi4-mm-gguf](https://huggingface.co/Swicked86/phi4-mm-gguf) are smaller but lack audio.
> This model is the sweet spot: **all three modalities at roughly ⅓ the size of bf16**.

---

## Available Files

| File | Size | Notes |
|------|------|-------|
| `model-00001-of-00002.safetensors` | ~3 GB | Quantized weight shard 1 |
| `model-00002-of-00002.safetensors` | ~2 GB | Quantized weight shard 2 |
| `config.json` | — | Includes `quantization_config` — vLLM auto-detects |
| `tokenizer.model` / `tokenizer.json` | — | Tokenizer |
| `preprocessor_config.json` | — | Vision + audio processor config (bf16 encoders) |

> The SigLIP-400M vision encoder and conformer-based speech encoder are stored at full **bfloat16** precision —
> only the Phi3 text transformer weights (32 decoder layers) are quantized to int4.

---

## VRAM Requirements

Model weights occupy **6.11 GiB** (measured). The remaining VRAM is used by the KV cache — vLLM pre-allocates the full KV cache pool at startup based on `--gpu-memory-utilization` and `--max-model-len`. **Total VRAM allocated = weights + pre-allocated KV cache**, regardless of how many requests are active.

### By GPU tier

| GPU | VRAM | Recommended `--max-model-len` | `--gpu-memory-utilization` | Notes |
|---|---|---|---|---|
| RTX 3070 / 2080 Super | 8 GB | — | — | ⚠️ **Not recommended.** Weights alone are 6.1 GB; insufficient headroom for KV cache. |
| RTX 3080 10 GB / 2080 Ti | 10 GB | 16,384 | 0.85 | Minimum viable. Tight — use lowest context only. |
| RTX 3080 12 GB / 4070 | 12 GB | 16,384–32,768 | 0.85 | Comfortable at 16K; 32K fits with care. |
| RTX 3080 Ti / 4070 Ti / 4080 | 16 GB | 32,768–65,520 | 0.85–0.90 | Good balance of context and headroom. |
| RTX 3090 / 4090 / 4080 Super | 24 GB | 65,520 | 0.85–0.90 | Recommended. Full tested context, comfortable. |
| RTX 5090 / A6000 / A100 40 GB | 32+ GB | 65,520–131,072 | 0.45–0.90 | Plenty of headroom; lower utilization keeps VRAM free for other tasks. |

### By context length

| `--max-model-len` | Weights | KV cache (est.) | Total (est.) | Min GPU VRAM |
|---|---|---|---|---|
| 16,384 | ~6.1 GB | ~1.5 GB | **~8 GB** | 10 GB |
| 32,768 | ~6.1 GB | ~3.0 GB | **~10 GB** | 12 GB |
| 65,520 | ~6.1 GB | ~6.0 GB | **~13 GB** | 16 GB |
| 131,072 (max, untested) | ~6.1 GB | ~12.0 GB | **~19 GB** | 24 GB |

> KV cache estimates use Phi-4-Mini architecture (32 layers, 8 KV heads, head_dim 96, bf16 activations ≈ 96 KB/token).
> Add ~1–2 GB for framework overhead. Weights measured on RTX 5090 with vLLM.
>
> **Why does vLLM show higher usage than "total est." above?**
> vLLM pre-allocates the entire KV cache pool at startup. On a large GPU (e.g. 32 GB at
> `--gpu-memory-utilization 0.45`), it reserves 0.45 × 32 GB = ~14 GB for KV cache even
> if no requests are active. The table above shows the *minimum* needed, not what vLLM
> will allocate when given more headroom.

---

## Usage

### Step 1 — Install vLLM

**Requirements:** Python 3.10+, CUDA GPU with ≥ 8 GB VRAM, vLLM 0.9.0+

```bash
pip install vllm
```

> **Do not** install `auto-gptq` or pass `--quantization gptq`. This model uses
> `compressed-tensors` format, which vLLM handles automatically from `config.json`.

---

### Step 2 — Download the model

```bash
huggingface-cli download Swicked86/phi4-mm-gptq --local-dir ./phi4-mm-gptq
```

Or let vLLM download on first run by passing the repo ID directly (see Step 3).

---

### Step 3 — Launch the vLLM server

**Minimum working command (text + vision + audio, 8–10 GB GPU):**

```bash
python -m vllm.entrypoints.openai.api_server \
  --model ./phi4-mm-gptq \
  --dtype bfloat16 \
  --trust-remote-code \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.85 \
  --enable-lora \
  --max-lora-rank 320 \
  --lora-modules speech=./phi4-mm-gptq/speech-lora \
                 vision=./phi4-mm-gptq/vision-lora \
  --limit-mm-per-prompt '{"image": 3, "audio": 3}' \
  --port 8080 \
  --host 127.0.0.1 \
  --served-model-name phi4-mm
```

**Extended context command (16 GB+ GPU, 65K context):**

```bash
python -m vllm.entrypoints.openai.api_server \
  --model ./phi4-mm-gptq \
  --dtype bfloat16 \
  --trust-remote-code \
  --max-model-len 65520 \
  --gpu-memory-utilization 0.90 \
  --enable-lora \
  --max-lora-rank 320 \
  --lora-modules speech=./phi4-mm-gptq/speech-lora \
                 vision=./phi4-mm-gptq/vision-lora \
  --limit-mm-per-prompt '{"image": 3, "audio": 3}' \
  --enable-auto-tool-choice \
  --tool-call-parser phi4_mini_json \
  --port 8080 \
  --host 127.0.0.1 \
  --served-model-name phi4-mm
```

**Flag reference:**

| Flag | Value | Why |
|------|-------|-----|
| `--model` | path or `Swicked86/phi4-mm-gptq` | Local dir or HF repo ID |
| `--dtype bfloat16` | required | Model native dtype — **do not use float16** |
| `--trust-remote-code` | required | phi4-mm uses custom modeling code |
| `--max-model-len` | 16384–131072 | See VRAM table above. 65,520 = 16×4095 (vLLM block-aligned). Beyond 65,520 is untested with this quantization — proceed at your own risk |
| `--gpu-memory-utilization` | 0.45–0.90 | Fraction of GPU VRAM to reserve for weights + KV cache |
| `--enable-lora` | required for vision/audio | Activates the rank-320 LoRA adapters |
| `--max-lora-rank 320` | required | phi4-mm LoRAs are rank 320 (unusually large) |
| `--lora-modules` | `speech=...` `vision=...` | Points to the adapter subdirs — enables those modalities |
| `--limit-mm-per-prompt` | `{"image":3,"audio":3}` | Max attachments per message |
| `--tool-call-parser phi4_mini_json` | optional | phi4-mm emits `functools[...]` format — this parses it |
| `--served-model-name phi4-mm` | optional | Alias so clients use `"model": "phi4-mm"` |

> **No `--quantization` flag needed.** vLLM reads `quantization_config` from `config.json`
> and activates the `compressed-tensors` int4 kernels automatically.

Wait for "Application startup complete":
```bash
curl http://localhost:8080/health   # → {"status":"ok"}
```

---

### Text (Python — openai SDK)

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="local")

response = client.chat.completions.create(
    model="phi4-mm",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    max_tokens=256,
)
print(response.choices[0].message.content)
```

---

### Vision — image understanding (Python)

```python
import base64
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="local")

with open("photo.jpg", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="phi4-mm",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}},
            {"type": "text", "text": "Describe this image in detail."},
        ],
    }],
    max_tokens=512,
)
print(response.choices[0].message.content)
```

**curl:**
```bash
IMAGE_B64=$(base64 -w0 photo.jpg)
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"phi4-mm\",
    \"messages\": [{
      \"role\": \"user\",
      \"content\": [
        {\"type\": \"image_url\", \"image_url\": {\"url\": \"data:image/jpeg;base64,${IMAGE_B64}\"}},
        {\"type\": \"text\", \"text\": \"What is in this image?\"}
      ]
    }],
    \"max_tokens\": 300
  }"
```

---

### Audio — speech transcription / understanding (Python)

```python
import base64
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="local")

with open("audio.wav", "rb") as f:
    audio_b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="phi4-mm",
    messages=[{
        "role": "user",
        "content": [
            {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "wav"}},
            {"type": "text", "text": "Transcribe this audio."},
        ],
    }],
    max_tokens=512,
)
print(response.choices[0].message.content)
```

> phi4-mm uses a custom conformer-based audio encoder (24 conformer blocks) with a rank-320 speech LoRA
> applied to the language decoder — no separate ASR model needed.
> Supported formats: `wav`, `mp3`, `ogg`, `flac`.

**curl:**
```bash
AUDIO_B64=$(base64 -w0 audio.wav)
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"phi4-mm\",
    \"messages\": [{
      \"role\": \"user\",
      \"content\": [
        {\"type\": \"input_audio\", \"input_audio\": {\"data\": \"${AUDIO_B64}\", \"format\": \"wav\"}},
        {\"type\": \"text\", \"text\": \"Transcribe and summarise.\"}
      ]
    }],
    \"max_tokens\": 512
  }"
```

---

### Combined — image + audio in one prompt

```python
import base64
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="local")

with open("photo.jpg", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()
with open("question.wav", "rb") as f:
    audio_b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="phi4-mm",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url",  "image_url":  {"url": f"data:image/jpeg;base64,{image_b64}"}},
            {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "wav"}},
            {"type": "text", "text": "Answer the spoken question about the image."},
        ],
    }],
    max_tokens=512,
)
print(response.choices[0].message.content)
```

---

### Tool calling

phi4-mm emits tool calls as `functools[{"name":"...","arguments":{...}}]`.
The `--tool-call-parser phi4_mini_json` flag (vLLM 0.7+) handles this automatically.
For a complete chat template that injects tools into phi4-mm's native `<|tool|>...<|/tool|>` block,
see [`deploy/wsl-vllm/phi4-mm-tool-template.jinja`](deploy/wsl-vllm/phi4-mm-tool-template.jinja)
in the [companion repo](https://github.com/Swicked86/LLM_Creation).

---

### Load locally with Transformers

```python
from transformers import AutoModelForCausalLM, AutoProcessor
import torch

model_id = "Swicked86/phi4-mm-gptq"

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="cuda",
)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
```

> Requires `pip install llmcompressor` (or `pip install compressed-tensors`) to load
> the `quantization_config` from the checkpoint.

---

## Quality

### Inference Tests

All tests run via vLLM on RTX 5090, `--gpu-memory-utilization 0.45`.

**Text — factual recall (< 0.5s):**
> Prompt: `"What is the capital of France?"`
> Response: `"The capital of France is Paris."` ✅

**Text — math reasoning (0.748s):**
> Prompt: `"Solve step by step: If a train travels 120 miles in 2 hours, what is its speed in km/h?"`
> Response: step-by-step solution → `96.54 km/h` ✅

**Text — code generation (1.068s):**
> Prompt: `"Write a Python function that checks if a string is a palindrome."`
> Response: correct `is_palindrome()` with docstring + example calls ✅

**Vision — real image (3000×4000 JPEG):**
> Prompt: `"Describe what you see in this image in detail."`
> Response: correctly identified anime figure on a TV screen, described the room, entertainment setup, and animation style ✅

**Audio — real voice message (Discord OGG Opus, converted to 16kHz WAV):**
> Input: Discord voice message (~11s) discussing software development
> Response: *"The speaker is describing the process of transforming the rag function into a function that uses a local database rather than writing to and from files."* ✅

> **Audio note:** Discord voice messages are OGG Opus at 48 kHz. Convert to 16 kHz mono WAV before
> sending for best results. Pass `"format": "wav"` in the request.

> **vLLM LoRA note:** vLLM currently only applies LoRA to the language model layers.
> Vision encoder LoRA layers (SigLIP) are silently skipped — this is a vLLM limitation.
> The speech LoRA (language decoder, rank-320) loaded and applied correctly.

---

### Perplexity (wikitext-2-raw, context 512)

| Model | PPL | vs bf16 |
|-------|-----|---------|
| bf16 (baseline) | 14.9338 ± 0.107 | — |
| **W4A16 GPTQ (this model)** | *pending* | *pending* |

*Benchmark will be added after upload.*

---

## Quantization Details

| Item | Value |
|------|-------|
| Quantizer | [llm-compressor](https://github.com/vllm-project/llm-compressor) (`GPTQModifier`) |
| Scheme | W4A16 (int4 weights, bfloat16 activations) |
| Group size | 128 |
| Sequential targets | `Phi3DecoderLayer` (32 × Phi3 text transformer blocks) |
| Excluded (kept bf16) | `lm_head`, `model.embed_tokens_extend.*` |
|                       | ↳ covers SigLIP-400M vision encoder + conformer-based audio encoder |
| Calibration | 512 samples, wikitext-2 |
| Source model | `~/phi4-mm-hf` (safetensors, downloaded from HF) |
| Hardware | RTX 5090 32 GB, CUDA 12.0, WSL2 Ubuntu 24.04 |
| Script | [`scripts/quantize_phi4mm.py`](https://github.com/Swicked86/LLM_Creation/blob/main/scripts/quantize_phi4mm.py) |

---

## Architecture

| Property | Value |
|----------|-------|
| Base model | Phi-4-Mini (3.8 B LLM backbone) |
| Total parameters | ~5.6 B |
| Context length | 128 K tokens (131,072) |
| Modalities | Text, Vision (SigLIP-400M), Audio/Speech (Conformer + rank-320 LoRA) |
| Text decoder | 32 × Phi3DecoderLayer — **quantized to int4** |
| Vision encoder | SigLIP2 (embed_tokens_extend.image_embed) — **bf16** |
| Audio encoder | Conformer-based audio encoder (24-block, 460M) + speech LoRA rank-320 — **bf16** |

---

## Related

- [Swicked86/phi4-mm-gguf](https://huggingface.co/Swicked86/phi4-mm-gguf) — GGUF quantizations (text + vision, no audio)
- [microsoft/Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) — original bf16 model
- [llm-compressor](https://github.com/vllm-project/llm-compressor) — quantization tool
- [vLLM](https://github.com/vllm-project/vllm) — inference engine