Xylolabs

phi-4 GGUF

Base model Formats License Code

GGUF quants of microsoft/phi-4 for llama.cpp, on CUDA, Metal or CPU.

  • Files: Q5_K_M, Q4_K_M, IQ4_XS and IQ4_XS-imat.
  • Q5_K_M: KL 0.0055 from BF16, top-1 95.96 %.
  • Which one to use: the smallest file whose KL you accept; see Files.

Files

  • phi-4-<QUANT>.gguf is one file per quant.
  • Contents: Each file holds the full language model plus the original tokenizer and chat template.
Quant Size Mean KL vs BF16 Same top-1 vs BF16
Q5_K_M 10.50 GB 0.0055 95.96%
Q4_K_M 9.06 GB 0.0180 93.65%
IQ4_XS 8.26 GB 0.0206 92.62%
IQ4_XS-imat 8.19 GB 0.0197 93.03%

How we measured:

  • Reference: our BF16 pass in transformers.
  • KL and top-1: 16,280 positions of English and Korean text.

Mean KL and same top-1 against BF16 by file size

How to run

Use llama.cpp commit cd959bae99ff4bed22bef2604f45241c09bbb8c8 or later.

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout cd959bae99ff4bed22bef2604f45241c09bbb8c8
cmake -B build -DCMAKE_BUILD_TYPE=Release        # add -DGGML_CUDA=ON for NVIDIA GPUs
cmake --build build -j --target llama-cli llama-server

./build/bin/llama-cli -m phi-4-Q5_K_M.gguf --jinja -c 8192 -ngl 99

llama-server takes the same flags.

How these were made

  1. Convert the original checkpoint to a BF16 GGUF with convert_hf_to_gguf.py.
  2. Quantize each type with llama-quantize.
  3. The -imat files use an importance matrix.

License and notices

  • License: MIT License, copied in LICENSE.
  • What we changed: we converted the weights to GGUF and quantized them.
  • Notices: NOTICE names the source model and its copyright holder.
  • Xylolabs is not affiliated with or endorsed by the original model's authors.

Citation

Cite the original Microsoft Phi-4 release.

Downloads last month
591
GGUF
Model size
15B params
Architecture
phi3
Hardware compatibility
Log In to add your hardware

4-bit

5-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Xylolabs/phi-4-GGUF

Base model

microsoft/phi-4
Quantized
(175)
this model

Collection including Xylolabs/phi-4-GGUF