README.md

---
quantized_by: PollardWeights
pipeline_tag: text-generation
base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
base_model_relation: quantized
license: apache-2.0
language:
- en
tags:
- pollard
- gguf
- llama.cpp
- qwen2
- code
- code-completion
- memory-fit
- conversational
---

# Pollard memory-fit quantizations of Qwen2.5-Coder-1.5B-Instruct by Qwen

Local **code completion that fits your box.** Built with
**[Pollard Weights](https://github.com/WestWaters/pollard-weights)** — sized to your
machine's RAM, not to a bit-width chart. Standard GGUF: runs in any recent
**llama.cpp** (the `qwen2` architecture is long-supported) and anything built on it.
**70–93 tok/s on an Apple M4**, whole model under 1.3 GB.

Original model: https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct

## Model details

| | |
|---|---|
| Parameter count | ~1.54B (dense) |
| Architecture | `qwen2` (28 layers) |
| Input support | text / code |
| Fill-in-the-middle | yes — Qwen2.5-Coder FIM tokens (see below) |
| Speculative decoding | no |
| imatrix | **yes** — importance-matrix guided (mixed prose + code corpus) |
| Perplexity / KLD measured | not measured — **verified by live code generation + throughput** (below) |

The sensitive tensors (token embeddings, attention q/k/v/o, norms, output head) keep
high precision and the FFN bulk carries the compression — a smarter quant, tuned to
how much RAM you actually have. (This is a **dense** model, so the build uses Pollard's
role/depth-aware memory-fit mode + imatrix; the measured-KL knapsack is reserved for
MoE models where it demonstrably beats uniform.)

## Which file should I choose?

- **Fastest / smallest** → **`IQ4_XS`** (0.86 GB, **93 tok/s** on M4). Great for a lean
  completion sidecar.
- **Balanced (recommended)** → **`Q5_K_M`** (1.12 GB, **71 tok/s**). Best quality-per-byte
  for everyday completion.
- **Max fidelity** → **`Q6_K`** (1.25 GB, **72 tok/s**). Near-lossless.

## Available files

| file | quant | size | M4 tok/s | notes |
|---|---|---|---|---|
| [`…-Pollard-IQ4_XS.gguf`](https://huggingface.co/PollardWeights/Qwen2.5-Coder-1.5B-Instruct-Pollard/blob/main/Qwen2.5-Coder-1.5B-Instruct-Pollard-IQ4_XS.gguf) | IQ4_XS | 0.86 GB | **93.0** | fastest / smallest |
| [`…-Pollard-Q5_K_M.gguf`](https://huggingface.co/PollardWeights/Qwen2.5-Coder-1.5B-Instruct-Pollard/blob/main/Qwen2.5-Coder-1.5B-Instruct-Pollard-Q5_K_M.gguf) | Q5_K_M | 1.12 GB | **71.2** | balanced — **recommended** |
| [`…-Pollard-Q6_K.gguf`](https://huggingface.co/PollardWeights/Qwen2.5-Coder-1.5B-Instruct-Pollard/blob/main/Qwen2.5-Coder-1.5B-Instruct-Pollard-Q6_K.gguf) | Q6_K | 1.25 GB | **72.5** | max fidelity |

## Prompt format (chat / instruct)

Qwen2.5-Coder uses ChatML:

```
<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
```

## Fill-in-the-middle (code completion)

For editor-style completion, use the Qwen2.5-Coder FIM tokens — prefix + suffix, model
fills the middle:

```
<|fim_prefix|>def is_prime(n):
    <|fim_suffix|>
    return True<|fim_middle|>
```

Repo-level completion is supported too via `<|repo_name|>` and `<|file_sep|>` separators.

## Download a specific file

```bash
pip install -U "huggingface_hub[cli]"
hf download PollardWeights/Qwen2.5-Coder-1.5B-Instruct-Pollard \
  --include "Qwen2.5-Coder-1.5B-Instruct-Pollard-Q5_K_M.gguf" --local-dir ./
```

## How to run

```bash
curl -LsSf https://llama.app/install.sh | sh
llama-server -hf PollardWeights/Qwen2.5-Coder-1.5B-Instruct-Pollard:Q5_K_M
```

or with a local file:

```bash
# OpenAI-compatible API + web UI at :8080 — point your editor / continue.dev at it
llama-server -m Qwen2.5-Coder-1.5B-Instruct-Pollard-Q5_K_M.gguf -ngl 99

# one-shot
llama-cli -m Qwen2.5-Coder-1.5B-Instruct-Pollard-Q5_K_M.gguf -ngl 99 -st \
  -p "Write a Python function is_prime(n). Only the function."

# Ollama
ollama create qwen2.5-coder-1.5b-pollard -f Modelfile   # FROM ./…-Q5_K_M.gguf
```

Also runs in **LM Studio, koboldcpp, ramalama, Jan, Text Generation WebUI, LoLLMs** —
standard GGUF.

## Verified

Loaded and generated from on an **Apple M4 Mac Mini (16 GB)**, llama.cpp Metal, before
shipping — *measure first, no claim before a number.* Throughput is in the table above;
correctness spot-check (`Q5_K_M`, "Write a Python function is_prime(n)"):

```python
def is_prime(n):
    if n <= 1:
        return False
    for i in range(2, int(n**0.5) + 1):
        if n % i == 0:
            return False
    return True
```

## imatrix

The importance matrix was computed on a mixed prose + source-code corpus and guides the
IQ/K-quant quality. (The base Qwen2.5-Coder builds elsewhere are often quantized without
one; these are imatrix-guided.)

## ARM / AVX

llama.cpp repacks weights into an interleaved layout at load time for faster ARM/AVX
inference — no special file needed; the old `Q4_0_4_4/4_8/8_8` variants are not required.

## Credits & license

- **Base model:** [Qwen2.5-Coder-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct)
  by the Qwen team, under **Apache-2.0**. This build inherits that license.
- **Quantization runtime:** [llama.cpp](https://github.com/ggml-org/llama.cpp) (ggml-org).
- **Method & builder:** [Pollard Weights](https://github.com/WestWaters/pollard-weights) —
  *measure first, no claim before a number.*