How to use from
llama.cpp
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Xylolabs/phi-4-GGUF:
# Run inference directly in the terminal:
llama cli -hf Xylolabs/phi-4-GGUF:
Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Xylolabs/phi-4-GGUF:
# Run inference directly in the terminal:
llama cli -hf Xylolabs/phi-4-GGUF:
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf Xylolabs/phi-4-GGUF:
# Run inference directly in the terminal:
./build/bin/llama cli -hf Xylolabs/phi-4-GGUF:
Use Docker
docker model run hf.co/Xylolabs/phi-4-GGUF:
Quick Links
Xylolabs

phi-4 GGUF

Base model Formats License Code

GGUF quants of microsoft/phi-4 for llama.cpp, on CUDA, Metal or CPU.

Files

  • phi-4-<QUANT>.gguf is one file per quant.
  • Contents: Each file holds the full language model plus the original tokenizer and chat template.

-imat in a file name marks a quant made with an importance matrix.

Quant Size Mean KL vs BF16 Same top-1 vs BF16
Q5_K_M 10.50 GB 0.0055 95.96%
Q4_K_M 9.06 GB 0.0180 93.65%
IQ4_XS 8.26 GB 0.0206 92.62%
IQ4_XS-imat 8.19 GB 0.0197 93.03%

The reference is our BF16 pass in transformers. KL and top-1 cover 16,280 positions of English and Korean text.

Mean KL and same top-1 against BF16 by file size

How to run

Use llama.cpp commit cd959bae99ff4bed22bef2604f45241c09bbb8c8 or later.

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout cd959bae99ff4bed22bef2604f45241c09bbb8c8
cmake -B build -DCMAKE_BUILD_TYPE=Release        # add -DGGML_CUDA=ON for NVIDIA GPUs
cmake --build build -j --target llama-cli llama-server

./build/bin/llama-cli -m phi-4-Q5_K_M.gguf --jinja -c 8192 -ngl 99

llama-server takes the same flags.

License and notices

MIT License, copied in LICENSE. We converted the weights to GGUF and quantized them. NOTICE names the source model and its copyright holder. Xylolabs is not affiliated with or endorsed by the original model's authors.

Citation

Cite the original Microsoft Phi-4 release.

Downloads last month
865
GGUF
Model size
15B params
Architecture
phi3
Hardware compatibility
Log In to add your hardware

4-bit

5-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Xylolabs/phi-4-GGUF

Base model

microsoft/phi-4
Quantized
(175)
this model

Collection including Xylolabs/phi-4-GGUF