Three quants, one prompt, three runs: speed is flat, the answers are not

#2
by abderrahmane11 - opened

I ran the same prompt through Q2_K, Q4_K_M and Q8_0 of this repository three times each, in a browser, with llama.cpp compiled to WebAssembly. Same page, same settings, temperature 0.3, the prompt:

List three prime numbers below 20 and say in one sentence why 1 is not prime.

Speed, in tokens per second, from llama.cpp's own counters:

Quant File Run 1 Run 2 Run 3
Q2_K 88 MB 22.4 32.5 29.6
Q4_K_M 106 MB 28.1 30.7 29.1
Q8_0 145 MB 30.9 34.9 36.3

Two things stand out.

The spread inside one quant is as large as the spread between quants. Q2_K moved 10 tokens per second between runs while the three files differ by 65 percent in size. On a 135M model in WebAssembly the dequantization cost is small enough that the file size is not the lever people assume it is. Q8_0 was the fastest in every run, which is the opposite of the usual expectation.

The answers were wrong in every quant, and each one failed differently. Q2_K said 1 is composite and, in another run, included 1 in the list of primes. Q4_K_M ignored the bound and listed primes up to 227. Q8_0 said 1 is divisible by 2, 3 and 5. The one clean answer in nine runs came from Q4_K_M. Run to run the output changed more than it changed between quants.

None of that is a criticism of the model. It is a 135M model answering a two-part instruction, and the point is narrower: at this size, choosing a quant for quality is the wrong lever. Choose by download size if bandwidth matters, then spend the effort on the prompt or pick a larger model.

Measured on an M1 Pro, Chrome, single threaded because a static page sends no COOP or COEP headers. The page is https://huggingface.co/spaces/abderrahmane11/gguf-runtime if you want to repeat it with your own prompt.

Sign up or log in to comment