PollardWeights/Qwen2.5-Coder-1.5B-Instruct-Pollard

Pollard memory-fit quantizations of Qwen2.5-Coder-1.5B-Instruct by Qwen

Local code completion that fits your box. Built with Pollard Weights — sized to your machine's RAM, not to a bit-width chart. Standard GGUF: runs in any recent llama.cpp (the qwen2 architecture is long-supported) and anything built on it. 70–93 tok/s on an Apple M4, whole model under 1.3 GB.

Original model: https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct

Model details

Parameter count	~1.54B (dense)
Architecture	qwen2 (28 layers)
Input support	text / code
Fill-in-the-middle	yes — Qwen2.5-Coder FIM tokens (see below)
Speculative decoding	no
imatrix	yes — importance-matrix guided (mixed prose + code corpus)
Perplexity / KLD measured	not measured — verified by live code generation + throughput (below)

The sensitive tensors (token embeddings, attention q/k/v/o, norms, output head) keep high precision and the FFN bulk carries the compression — a smarter quant, tuned to how much RAM you actually have. (This is a dense model, so the build uses Pollard's role/depth-aware memory-fit mode + imatrix; the measured-KL knapsack is reserved for MoE models where it demonstrably beats uniform.)

Which file should I choose?

• Fastest / smallest → IQ4_XS (0.86 GB, 93 tok/s on M4). Great for a lean completion sidecar.
• Balanced (recommended) → Q5_K_M (1.12 GB, 71 tok/s). Best quality-per-byte for everyday completion.
• Max fidelity → Q6_K (1.25 GB, 72 tok/s). Near-lossless.

Available files

file	quant	size	M4 tok/s	notes
…-Pollard-IQ4_XS.gguf	IQ4_XS	0.86 GB	93.0	fastest / smallest
…-Pollard-Q5_K_M.gguf	Q5_K_M	1.12 GB	71.2	balanced — recommended
…-Pollard-Q6_K.gguf	Q6_K	1.25 GB	72.5	max fidelity

Prompt format (chat / instruct)

Qwen2.5-Coder uses ChatML:

<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

Fill-in-the-middle (code completion)

For editor-style completion, use the Qwen2.5-Coder FIM tokens — prefix + suffix, model fills the middle:

<|fim_prefix|>def is_prime(n):
    <|fim_suffix|>
    return True<|fim_middle|>

Repo-level completion is supported too via <|repo_name|> and <|file_sep|> separators.

Download a specific file

pip install -U "huggingface_hub[cli]"
hf download PollardWeights/Qwen2.5-Coder-1.5B-Instruct-Pollard \
  --include "Qwen2.5-Coder-1.5B-Instruct-Pollard-Q5_K_M.gguf" --local-dir ./

How to run

curl -LsSf https://llama.app/install.sh | sh
llama-server -hf PollardWeights/Qwen2.5-Coder-1.5B-Instruct-Pollard:Q5_K_M

or with a local file:

# OpenAI-compatible API + web UI at :8080 — point your editor / continue.dev at it
llama-server -m Qwen2.5-Coder-1.5B-Instruct-Pollard-Q5_K_M.gguf -ngl 99

# one-shot
llama-cli -m Qwen2.5-Coder-1.5B-Instruct-Pollard-Q5_K_M.gguf -ngl 99 -st \
  -p "Write a Python function is_prime(n). Only the function."

# Ollama
ollama create qwen2.5-coder-1.5b-pollard -f Modelfile   # FROM ./…-Q5_K_M.gguf

Also runs in LM Studio, koboldcpp, ramalama, Jan, Text Generation WebUI, LoLLMs — standard GGUF.

Verified

Loaded and generated from on an Apple M4 Mac Mini (16 GB), llama.cpp Metal, before shipping — measure first, no claim before a number. Throughput is in the table above; correctness spot-check (Q5_K_M, "Write a Python function is_prime(n)"):

def is_prime(n):
    if n <= 1:
        return False
    for i in range(2, int(n**0.5) + 1):
        if n % i == 0:
            return False
    return True

imatrix

The importance matrix was computed on a mixed prose + source-code corpus and guides the IQ/K-quant quality. (The base Qwen2.5-Coder builds elsewhere are often quantized without one; these are imatrix-guided.)

ARM / AVX

llama.cpp repacks weights into an interleaved layout at load time for faster ARM/AVX inference — no special file needed; the old Q4_0_4_4/4_8/8_8 variants are not required.

Credits & license

• Base model: Qwen2.5-Coder-1.5B-Instruct by the Qwen team, under Apache-2.0. This build inherits that license.
• Quantization runtime: llama.cpp (ggml-org).
• Method & builder: Pollard Weights — measure first, no claim before a number.