Post
133
For those who wanna run Qwen3-4B on CPU alone, I tested the whole ladder:
Same machine for everything: CPU only, 16 threads, no GPU offload. Perplexity is WikiText-2 raw, which llama.cpp also uses in CI.
The winner is Q4_K_M - it gets the model down from 8.05 GB to 2.5 GB in size/load, and perplexity only moves by 0.30.
Q3 is the dropoff point where inmatrix makes a difference. Inmatrix drops Q3 from 15.66 to 14.82 PPL; Q4 it barely changed anything.
My now informed recommendation is: Q4 if you have the memory, imatrix Q3 if you don’t.
I uploaded all eight weights, the harness, and the full results here:
huggingface.co/b4ph/qwen3-4b-lowram-bench
I also made a tiny picker because apparently I needed to turn this into a whole project:
huggingface.co/spaces/b4ph/qwen3-4b-quant-picker
Same machine for everything: CPU only, 16 threads, no GPU offload. Perplexity is WikiText-2 raw, which llama.cpp also uses in CI.
Quant Size PPL vs F16 tg t/s
F16 8.05 GB 13.4304 — 2.52
Q8_0 4.28 GB 13.4409 +0.01 5.04
Q6_K 3.31 GB 13.4404 +0.01 5.93
Q5_K_M 2.89 GB 13.5362 +0.11 5.64
Q4_K_M 2.50 GB 13.7304 +0.30 6.50
Q4_K_M + imatrix 2.50 GB 13.6760 +0.25 6.41
Q3_K_M 2.08 GB 15.6641 +2.23 6.56
Q3_K_M + imatrix 2.08 GB 14.8237 +1.39 5.89The winner is Q4_K_M - it gets the model down from 8.05 GB to 2.5 GB in size/load, and perplexity only moves by 0.30.
Q3 is the dropoff point where inmatrix makes a difference. Inmatrix drops Q3 from 15.66 to 14.82 PPL; Q4 it barely changed anything.
My now informed recommendation is: Q4 if you have the memory, imatrix Q3 if you don’t.
I uploaded all eight weights, the harness, and the full results here:
huggingface.co/b4ph/qwen3-4b-lowram-bench
I also made a tiny picker because apparently I needed to turn this into a whole project:
huggingface.co/spaces/b4ph/qwen3-4b-quant-picker