--- license: mit license_link: https://huggingface.co/microsoft/Phi-4-mini-instruct/resolve/main/LICENSE thumbnail: https://huggingface.co/AtomicChat/Phi-4-mini-instruct-GGUF/resolve/main/hero.png base_model: - microsoft/Phi-4-mini-instruct base_model_relation: quantized quantized_by: AtomicChat pipeline_tag: text-generation library_name: gguf tags: - atomic-chat - phi - phi4 - microsoft - gguf - llama.cpp - quantized ---

Base model: microsoft/Phi-4-mini-instruct

**Phi 4 Mini**, self-quantized to GGUF by [Atomic Chat](https://atomic.chat). Built straight from Microsoft's original weights with a per-tensor importance matrix, so this is not a repack of somebody else's files. Runs fully offline. ## Highlights - **3.8B parameters**: the weights this repo quantizes. - **Context length**: 131,072 tokens (128K), as published by Microsoft. - **32 layers**: Dense decoder, hybrid sliding-window (262144) and global attention. - **Full imatrix ladder**: every quant is calibrated with an importance matrix. > [!NOTE] > These GGUFs are **self-quantized from the original weights**, not a repack. The importance matrix keeps low-bit quants closer to the full-precision model. > [!IMPORTANT] > Always pass `--jinja` so the **Phi 4 Mini chat template** is applied. Without it the model can emit malformed turns. ## Model Overview | Property | Value | |---|---| | Base model | `microsoft/Phi-4-mini-instruct` | | Parameters | 3.8B | | Layers | 32 | | Sliding window | 262144 tokens | | Context length | 131,072 tokens (128K) | | Vocabulary | 200,064 | | Modalities | Text | | Architecture | Dense decoder, hybrid sliding-window (262144) and global attention, 24 attention heads over 8 KV heads, `Phi3ForCausalLM` | | This repo | GGUF quants (imatrix). Quants: `Q4_K_M`, `UD-Q4_K_XL`, `Q5_K_M`, `Q6_K`, `Q8_0` | ## Choosing a quant | Quant | Size | Notes | |---|---|---| | **`Q4_K_M`** | 2.5 GB | **Recommended default. Best balance of size, speed and quality.** | | `UD-Q4_K_XL` | 2.6 GB | Dynamic. Embeddings and output kept at Q8_0 for higher quality at a Q4 footprint. | | `Q5_K_M` | 2.8 GB | Higher quality, low loss. | | `Q6_K` | 3.2 GB | Near lossless, noticeably lighter than Q8_0. | | `Q8_0` | 4.1 GB | Effectively lossless, reference quality. | > [!TIP] > Pick the largest file that fits your (V)RAM with room for context. `Q4_K_M` or `UD-Q4_K_XL` is the sweet spot for most setups; `Q6_K` or `Q8_0` for maximum fidelity. ## Get started Run Phi 4 Mini locally with: - **[Atomic Chat](https://atomic.chat):** the easiest path. Open the app, search `AtomicChat/Phi-4-mini-instruct-GGUF`, pick a quant, hit **Use this model**. - **llama.cpp:** `llama-server -hf AtomicChat/Phi-4-mini-instruct-GGUF:Q4_K_M --jinja -c 8192` - **Ollama:** `ollama run hf.co/AtomicChat/Phi-4-mini-instruct-GGUF:Q4_K_M` - **LM Studio / Jan:** search the repo id, download any quant. ## Best practices | Parameter | Value | |---|---| | temperature | 0.0 | Microsoft's recommended sampling configuration for `microsoft/Phi-4-mini-instruct`. ## Run in llama.cpp ```bash git clone https://github.com/ggml-org/llama.cpp cmake llama.cpp -B llama.cpp/build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON cmake --build llama.cpp/build --config Release -j --target llama-cli llama-server ``` ```bash ./llama.cpp/build/bin/llama-server \ -hf AtomicChat/Phi-4-mini-instruct-GGUF:Q4_K_M \ --jinja -ngl 99 -c 8192 -fa on ``` ## How these were made 1. Download `microsoft/Phi-4-mini-instruct` (original weights). 2. Convert to f16 GGUF with [llama.cpp](https://github.com/ggml-org/llama.cpp). 3. Build an importance matrix over our calibration corpus. 4. Quantize the ladder with `--imatrix`. 5. `UD-Q4_K_XL` additionally pins the token-embedding and output tensors to `Q8_0`. ## License Original model by Microsoft, released under the MIT license. Full terms: [MIT](https://huggingface.co/microsoft/Phi-4-mini-instruct/resolve/main/LICENSE). Quantized by Atomic Chat.