Phi-4 GGUF references
Collection
Four preserved Phi-4 GGUF alternatives. Made by Xylolabs. Not affiliated with or endorsed by Tenstorrent. โข 1 item โข Updated
How to use Xylolabs/phi-4-GGUF with llama.cpp:
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf Xylolabs/phi-4-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf Xylolabs/phi-4-GGUF:Q4_K_M
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf Xylolabs/phi-4-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf Xylolabs/phi-4-GGUF:Q4_K_M
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf Xylolabs/phi-4-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf Xylolabs/phi-4-GGUF:Q4_K_M
docker model run hf.co/Xylolabs/phi-4-GGUF:Q4_K_M
How to use Xylolabs/phi-4-GGUF with vLLM:
# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Xylolabs/phi-4-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
-H "Content-Type: application/json" \
--data '{
"model": "Xylolabs/phi-4-GGUF",
"messages": [
{
"role": "user",
"content": "What is the capital of France?"
}
]
}'docker model run hf.co/Xylolabs/phi-4-GGUF:Q4_K_M
How to use Xylolabs/phi-4-GGUF with Ollama:
ollama run hf.co/Xylolabs/phi-4-GGUF:Q4_K_M
How to use Xylolabs/phi-4-GGUF with Docker Model Runner:
docker model run hf.co/Xylolabs/phi-4-GGUF:Q4_K_M
How to use Xylolabs/phi-4-GGUF with Lemonade:
# Download Lemonade from https://lemonade-server.ai/ lemonade pull Xylolabs/phi-4-GGUF:Q4_K_M
lemonade run user.phi-4-GGUF-Q4_K_M
lemonade list
GGUF quants of microsoft/phi-4 for llama.cpp, on CUDA, Metal or CPU.
phi-4-<QUANT>.gguf is one file per quant.| Quant | Size | Mean KL vs BF16 | Same top-1 vs BF16 |
|---|---|---|---|
| Q5_K_M | 10.50 GB | 0.0055 | 95.96% |
| Q4_K_M | 9.06 GB | 0.0180 | 93.65% |
| IQ4_XS | 8.26 GB | 0.0206 | 92.62% |
| IQ4_XS-imat | 8.19 GB | 0.0197 | 93.03% |
How we measured:
Use llama.cpp commit cd959bae99ff4bed22bef2604f45241c09bbb8c8 or later.
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout cd959bae99ff4bed22bef2604f45241c09bbb8c8
cmake -B build -DCMAKE_BUILD_TYPE=Release # add -DGGML_CUDA=ON for NVIDIA GPUs
cmake --build build -j --target llama-cli llama-server
./build/bin/llama-cli -m phi-4-Q5_K_M.gguf --jinja -c 8192 -ngl 99
llama-server takes the same flags.
convert_hf_to_gguf.py.llama-quantize.-imat files use an importance matrix.LICENSE.NOTICE names the source model and its copyright holder.Cite the original Microsoft Phi-4 release.
4-bit
5-bit
Base model
microsoft/phi-4