Qwen2.5-Coder-7B-Instruct - GGUF Quantized Versions

This repository provides GGUF quantized versions of Qwen/Qwen2.5-Coder-7B-Instruct, converted with llama.cpp.

The purpose of this repository is to provide fast, easy-to-use local inference files for llama.cpp, Ollama, LM Studio, Jan, Open WebUI, and llama-cpp-python users.

Model Details

• Base model: Qwen/Qwen2.5-Coder-7B-Instruct
• Architecture: Qwen 2
• Format: GGUF
• Source license: apache-2.0
• Conversion tool: convert_hf_to_gguf.py from llama.cpp
• Quantization tool: llama-quantize
• Recommended file: Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf

Quantized Files

Quant	Filename	Size	SHA256	Notes
FP16	Qwen2.5-Coder-7B-Instruct-FP16.gguf	~14.19 GiB	274c0eb05fe4...	Full precision converted GGUF baseline
Q2_K	Qwen2.5-Coder-7B-Instruct-Q2_K.gguf	~2.81 GiB	3200d17f49c6...	Smallest, lowest quality
Q3_K_M	Qwen2.5-Coder-7B-Instruct-Q3_K_M.gguf	~3.55 GiB	30be23c2bc76...	Small balanced version
Q4_0	Qwen2.5-Coder-7B-Instruct-Q4_0.gguf	~4.13 GiB	339fbe41f5d4...	Simple 4-bit quantization
Q4_K_M	Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf	~4.36 GiB	46caa6175bbe...	Recommended default for most users
Q5_K_M	Qwen2.5-Coder-7B-Instruct-Q5_K_M.gguf	~5.07 GiB	38d6bd18220d...	Better quality with moderate size
Q6_K	Qwen2.5-Coder-7B-Instruct-Q6_K.gguf	~5.82 GiB	86b8e1701365...	High quality
Q8_0	Qwen2.5-Coder-7B-Instruct-Q8_0.gguf	~7.54 GiB	d4cdb65b1880...	Near FP16 quality

Validation

Each file was tested with llama-cli for basic load + generation.

Quant	Filename	Status
FP16	Qwen2.5-Coder-7B-Instruct-FP16.gguf	✅ passed
Q2_K	Qwen2.5-Coder-7B-Instruct-Q2_K.gguf	✅ passed
Q3_K_M	Qwen2.5-Coder-7B-Instruct-Q3_K_M.gguf	✅ passed
Q4_0	Qwen2.5-Coder-7B-Instruct-Q4_0.gguf	✅ passed
Q4_K_M	Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf	✅ passed
Q5_K_M	Qwen2.5-Coder-7B-Instruct-Q5_K_M.gguf	✅ passed
Q6_K	Qwen2.5-Coder-7B-Instruct-Q6_K.gguf	✅ passed
Q8_0	Qwen2.5-Coder-7B-Instruct-Q8_0.gguf	✅ passed

Usage

llama.cpp

llama-cli -m Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf -p "Hello! Introduce yourself briefly."

Older builds may use:

./main -m Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf -p "Hello! Introduce yourself briefly."

llama.cpp directly from Hugging Face

llama-cli -hf ShahzebKhoso/Qwen2.5-Coder-7B-Instruct:Q4_K_M -p "Hello! Introduce yourself briefly."

llama-cpp-python

from huggingface_hub import hf_hub_download
from llama_cpp import Llama

model_path = hf_hub_download(
    repo_id="ShahzebKhoso/Qwen2.5-Coder-7B-Instruct",
    filename="Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf",
)

llm = Llama(model_path=model_path)

out = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Hello! Introduce yourself briefly."},
    ],
    max_tokens=128,
)

print(out["choices"][0]["message"]["content"])

Which file should I use?

• Use Q4_K_M for the best default balance.
• Use Q5_K_M for better quality.
• Use Q8_0 if you want near-original quality and have more memory.
• Use Q2_K or Q3_K_M only when memory is very limited.

Provenance

This repository is a quantized derivative of:

Qwen/Qwen2.5-Coder-7B-Instruct

Base model metadata:

revision: c03e6d358207e414f1eca0bb1891e29f1db0e242
pipeline_tag: text-generation
tags: transformers, safetensors, qwen2, text-generation, code, codeqwen, chat, qwen, qwen-coder, conversational, en, arxiv:2409.12186, arxiv:2309.00071, arxiv:2407.10671, base_model:Qwen/Qwen2.5-Coder-7B, base_model:finetune:Qwen/Qwen2.5-Coder-7B, license:apache-2.0, text-generation-inference, endpoints_compatible, deploy:azure