New imatrix dataset
First, before we get into it, no this is not going to be a "silver bullet" of performance. There's a few tweaks, a few wins, a few losses, but overall I think an improvement, and more importantly it was a big learning experience that I'll be using going forward for more iterations of the imatrix dataset. This is not the final result, but it seems like a good result so far.
As always, I will provide everything I use publicly, so I'll start with that upfront:
https://gist.github.com/bartowski1182/e26453c0404e24eb317543ec5360f87a
This gist contains 2 files, one for prose, and one for conversations, which I render through a script against the model's chat template, and provide in rendered format alongside the model upload. Qwen's can be found here:
https://huggingface.co/bartowski/Qwen3.8-27B-GGUF/blob/main/Qwen3.8-27B-calibration-v6.txt
(Note, if you look at it, the last bit ends with a bunch of <|endoftext|><|endoftext|><|endoftext|>, this is on purpose just to pad out the final chunk to make sure everything gets used by the model instead of truncating)
The logic for rendering that file can be found here:
https://gist.github.com/bartowski1182/82fbf6dfc3fd234d5b50fda4d0bdf218
TLDR at the bottom, but I want to go over everything I did just for fun
What I (and Fable) did
Using GPU compute generously provided by LTT Labs team (huge shoutout to them), I worked with Fable on an extensive experiment to figure out how imatrix actually affected models.
My old calibration_datav5.txt was just a bunch of plain text, a couple languages but largely English, and absolutely no formatted text. Obviously this seemed to work quite fine! And was largely considered the "best" way to calculate imatrix - semi-random noise with enough diversity to fully exercise the model.
I wanted to test that thought, especially from a "formatted text" point of view, in particular.. Does chat template formatting matter?
So off to the races we went, testing a bunch of different datasets (shout out to Ed Addario for providing a huge selection of options for imatrix-calibration datasets) and some different chat templated datasets, focusing on tool use ones because that was one of the first things I cared about (does adding tool use to the model's dataset improve its ability to call tools?)
Big news up front: there's a lot less to gain than I had hoped. To be honest, V5 was already quite good, no matter what other dataset I threw at it, it was always pretty competitive. Credit to Kalomaze, Dampf, and again Ed for the data that's in there, it's a mixture of all their work, and they deserve a ton of the credit!
What actually is the imatrix?
I can hear some people asking: "okay cool but what the hell even is imatrix anyways?" I'll give a brief explanation just so that there's a bit of extra context.
When llama.cpp quantizes a model with quantize.cpp, the algorithm is effectively trying to minimize the "error" of the quantization in a given group of weights. This is done by testing various scales and offsets (and in the case of IQ quants, codebook selections) and checking how much of an error is introduced when the weights are quantized and then dequantized. These algorithms can be found in ggml-quants.c for each of the tensor types (Q6_K, Q4_0, IQ3_XXS, etc). Looking at the implementation of Q4_K, you can see how the values are chosen, and there's an if block for if (quant_weights) here. In this context, quant_weights are the imatrix calculations.
So what is the "imatrix calculation"? When the imatrix is calculated, what happens is that text is fed through the model in large batches, and on every pass, it accumulates how strongly each input channel activates (technically, the sum of squared activations feeding into each tensor). Based on how strongly and consistently a channel activates across all that text, we can consider how "important" the weights it touches must be to the final results. Given hundreds of thousands of tokens, if a channel lights up hard on almost every pass, it's probably really damn important. If it basically never does, it's probably not that important. Of course, we can't guarantee that, because we may have just never given it the right text that it cared about!
So this is where the corpus comes into play, calibration_datav5.txt is the exact text I used for every model I released since I started using it, and I had another dataset before that, and another before that. V6 is the latest iteration, and probably won't be the last!
What was tested
So, how did we test the dataset?
Using 7 models, gemma-4-E2B-it, Qwen3.5-4B, Qwen3.6-27B, Qwen3.6-35B-A3B, Mistral-Small-4-119B, Qwen3-Next-80B-A3B, and Qwen3.5-397B-A17B. Yes, mostly Qwen models, but Mistral was used to confirm that the findings generalized, and I'll probably continue some testing on a few other options, but lets be real.. When it comes to models people can run locally, Qwen is the main option right now ;)
There were several tests, but mainly KLD vs bf16, BFCL, and a few bespoke tests like a canary set of short prompts and expert coverage checking on MoE models. MMLU-Pro and GSM8K were also run on subsets (due to their cost) to validate findings more deeply.
Findings
As mostly expected, above ~4 bits per weight, the dataset does not have a predictable difference. Even when using completely random noise (there were a few early tests to dismiss some theories), Q4_0/IQ4_XS and above saw little differences in performance. It's not THAT surprising, the errors are already so small that nudging them in the wrong direction won't be a huge change vs the right one.
Q2_K is where we saw a lot of differences, especially on MoE models. Without imatrix at all, Qwen3.6-35B saw a 28 point drop in BFCL performance from ~82% to ~54%. This is also where we saw a pretty decent difference in corpus selection, ~10% between the best and worst.
On IQ2_M, most clean datasets were also within a couple percentage points of each other, indicating that a "good enough" dataset is, indeed, often good enough.
On dense models, the differences are even harder to find. It seems that most of the difference in performance from imatrix at low bit rates comes from expert coverage.
Funny enough, on MMLU-Pro and GSM8K, Q2_K accuracy was basically identical across the full range of imatrix datasets, INCLUDING the random words dataset, which to me was quite shocking..
For MoE models, the biggest change was found to be the inclusion of chat templated calibration data, indicating it was likely that there were experts that basically only activated when they saw chat tokens, and that by not including those tokens, those experts were never given their proper importance
There was also some fun tests against Qwen3-Next-80B-A3B (because it has such a huge amount of experts for its size), for example a pure english tool corpus left 18 of the 512 experts completely untouched. They never saw a single token pass through them, even when using 3126 chunks, which is almost 4x the length of my calibration_datav5.txt. An early recipe combining calibration_datav5.txt and tool calling dataset got full coverage, and actually multilinguality and code content was the main thing needed for those experts.
The final test involved running various chunks of various datasets to see if we could find which specific chunks were "best" at excercising the model fully, and combining those to yield a minimally best imatrix dataset to use going forward.
Building the final dataset
I know at this point I've kind of been rambling, so I'll try to keep it brief.
Using the above mentioned per-chunk attribution, we were able to find that some experts are only fed by specific prose, be it code, French, fiction, or science, and so the final prose that we settled on is a combination that yielded the most diverse expert coverage across all the MoE models used.
Combining this with templated tool calling from interstellarninja/hermes_reasoning_tool_use at a 2:3 ratio of prose to conversations was found to be the best.
Trying out many iterations between 400 to 800 chunks, it was found that even vs significantly larger datasets (even if made from the same quality), and anything smaller, this final one performed the best. A small theory is that, given a dataset that's too big, you may end up just drowning out the important results with the most common results.
This final dataset was validated against a couple other models (Mistral and Qwen3-Next-80B-A3B) and found to be the most performant in many cases.
A couple final questions explored
Did we narrow multilingual coverage? It seems that no, if anything the opposite: the share of Latin-script text in the prose went down, meaning non-Latin languages make up a bigger slice of the pie than in v5. As long as total coverage stays high enough, we likely improved multilingual performance. More testing to come.
Was this prose the best we could find? From testing, yes. Swapped out the prose for many other collections, and at worst ours was within 2% noise of the alternative, and at best more than 7% better (in KLD and BFCL).
Is a context of 512 on imatrix costing us? Maybe, but it certainly doesn't seem so. On all tests, 2048 showed no change, but actually cost some expert coverage in many cases.
Do we have to calculate from bf16? Almost certainly not, comparing cosine similarity of the imatrix results between bf16 and Q8_0 found a similarity above 99.99% (with of course a couple outliers, but overall impressively similar)
Just give me numbers
I made Qwen3.8-27B with both my v5 calibration and the new v6 render, PPL and KLD numbers for wikitext, HuggingFaceH4/no_robots, and Salesforce/xlam-function-calling-60k. Note of course these are more "proxy" numbers than anything, especially norobots because the PPL at bf16 on that dataset is all the way up at 19.05, so I provide these for now as reference data but not as conclusive evidence. Xlam has a bit better PPL at 8.07, and it gives me some confidence that the KLD/Delta P stats are better on that dataset. Note, norobots and xlam were run with a change to perplexity.cpp to allow it to properly parse special tokens (similar to --parse-special in imatrix.cpp)
wikitext, 100 chunks
Negative Δ = v6 better (lower divergence).
| Quant | Mean KLD v6 | Mean KLD v5 | Δ KLD | p99 v6 | p99 v5 | top-p v6 | top-p v5 |
|---|---|---|---|---|---|---|---|
| Q6_K_L | 0.00263 | 0.00267 | -1.5% | 0.0254 | 0.0238 | 97.7% | 97.7% |
| Q6_K | 0.00315 | 0.00318 | -0.9% | 0.0256 | 0.0263 | 97.4% | 97.4% |
| Q5_K_L | 0.00507 | 0.00509 | -0.3% | 0.0463 | 0.0487 | 96.9% | 96.8% |
| Q5_K_M | 0.00545 | 0.00556 | -2.0% | 0.0529 | 0.0516 | 96.7% | 96.6% |
| Q5_K_S | 0.00640 | 0.00640 | -0.1% | 0.0593 | 0.0599 | 96.3% | 96.3% |
| Q4_K_L | 0.01234 | 0.01214 | +1.6% | 0.1140 | 0.1155 | 95.1% | 95.1% |
| Q4_1 | 0.01578 | 0.01581 | -0.2% | 0.1563 | 0.1561 | 94.4% | 94.3% |
| Q4_K_M | 0.01283 | 0.01277 | +0.4% | 0.1181 | 0.1188 | 94.9% | 95.0% |
| Q4_K_S | 0.01626 | 0.01569 | +3.6% | 0.1564 | 0.1523 | 94.2% | 94.5% |
| Q3_K_XL | 0.04284 | 0.04305 | -0.5% | 0.4171 | 0.4107 | 90.9% | 91.2% |
| Q4_0 | 0.02658 | 0.02639 | +0.7% | 0.2614 | 0.2533 | 92.7% | 92.8% |
| IQ4_NL | 0.01785 | 0.01770 | +0.9% | 0.1742 | 0.1646 | 94.2% | 94.3% |
| IQ4_XS | 0.01962 | 0.01936 | +1.3% | 0.1932 | 0.1822 | 93.9% | 94.0% |
| Q3_K_L | 0.04373 | 0.04392 | -0.4% | 0.4239 | 0.4314 | 90.8% | 91.0% |
| Q3_K_M | 0.04693 | 0.04621 | +1.6% | 0.4430 | 0.4548 | 90.6% | 90.7% |
| IQ3_M | 0.05628 | 0.05605 | +0.4% | 0.5654 | 0.5605 | 89.7% | 89.8% |
| Q3_K_S | 0.06892 | 0.06860 | +0.5% | 0.6626 | 0.6369 | 88.2% | 88.2% |
| IQ3_XS | 0.06229 | 0.06231 | -0.0% | 0.6122 | 0.5860 | 89.3% | 89.2% |
| Q2_K_L | 0.13232 | 0.13592 | -2.6% | 1.2512 | 1.3316 | 84.4% | 84.2% |
| IQ3_XXS | 0.08528 | 0.08319 | +2.5% | 0.8153 | 0.7859 | 87.3% | 87.2% |
| Q2_K | 0.13543 | 0.13898 | -2.6% | 1.3162 | 1.3386 | 84.1% | 83.9% |
| IQ2_M | 0.15490 | 0.15777 | -1.8% | 1.4985 | 1.5352 | 83.1% | 82.7% |
| IQ2_S | 0.21698 | 0.21055 | +3.1% | 2.0055 | 1.8509 | 79.9% | 80.4% |
| IQ2_XS | 0.22856 | 0.22469 | +1.7% | 2.0690 | 1.9905 | 79.6% | 79.8% |
| IQ2_XXS | 0.30336 | 0.30421 | -0.3% | 2.6446 | 2.7227 | 76.4% | 76.5% |
norobots, 100 chunks
KL divergence
| Quant | Mean v6 | Mean v5 | Mean diff | Max v6 | Max v5 | Max diff | p99.9 v6 | p99.9 v5 | p99.9 diff | p99 v6 | p99 v5 | p99 diff |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| IQ2_XXS | 0.5205 | 0.5070 | +0.0135 | 23.26 | 21.24 | +2.02 | 15.732 | 14.563 | +1.169 | 6.2249 | 6.2052 | +0.0197 |
| IQ2_XS | 0.3953 | 0.3917 | +0.0036 | 27.47 | 23.58 | +3.89 | 13.395 | 13.023 | +0.372 | 5.3641 | 5.2340 | +0.1301 |
| IQ2_S | 0.3625 | 0.3857 | -0.0232 | 21.92 | 21.12 | +0.81 | 13.061 | 12.736 | +0.325 | 5.2467 | 5.2235 | +0.0232 |
| IQ2_M | 0.2947 | 0.3080 | -0.0133 | 21.36 | 22.82 | -1.46 | 12.312 | 13.459 | -1.147 | 4.7832 | 5.1450 | -0.3618 |
| Q2_K | 0.2983 | 0.2926 | +0.0058 | 24.23 | 25.82 | -1.59 | 13.181 | 13.936 | -0.754 | 4.6254 | 4.3627 | +0.2627 |
| IQ3_XXS | 0.1963 | 0.2145 | -0.0182 | 21.24 | 22.74 | -1.50 | 11.818 | 12.472 | -0.654 | 3.5513 | 4.3334 | -0.7821 |
| Q2_K_L | 0.2909 | 0.2897 | +0.0012 | 26.57 | 27.10 | -0.53 | 12.993 | 13.955 | -0.962 | 4.4691 | 4.4354 | +0.0337 |
| IQ3_XS | 0.1700 | 0.1701 | -0.0002 | 23.22 | 21.82 | +1.40 | 11.263 | 11.735 | -0.472 | 3.0098 | 3.1556 | -0.1458 |
| Q3_K_S | 0.1621 | 0.1697 | -0.0076 | 22.96 | 23.82 | -0.85 | 10.499 | 11.133 | -0.633 | 2.8039 | 3.1622 | -0.3584 |
| IQ3_M | 0.1424 | 0.1444 | -0.0020 | 25.11 | 20.20 | +4.92 | 10.903 | 9.987 | +0.915 | 2.4654 | 2.8845 | -0.4191 |
| Q3_K_M | 0.1276 | 0.1261 | +0.0016 | 20.87 | 20.01 | +0.86 | 9.465 | 9.892 | -0.427 | 2.5191 | 2.2581 | +0.2610 |
| Q3_K_L | 0.1162 | 0.1230 | -0.0068 | 24.57 | 23.45 | +1.12 | 9.483 | 9.825 | -0.342 | 2.2007 | 2.2875 | -0.0869 |
| IQ4_XS | 0.0620 | 0.0641 | -0.0021 | 24.64 | 22.36 | +2.28 | 6.384 | 6.794 | -0.410 | 1.0086 | 1.0829 | -0.0744 |
| IQ4_NL | 0.0573 | 0.0555 | +0.0018 | 24.99 | 21.38 | +3.61 | 5.955 | 5.715 | +0.240 | 0.9595 | 0.9001 | +0.0595 |
| Q4_0 | 0.0889 | 0.0900 | -0.0011 | 22.07 | 21.46 | +0.62 | 10.269 | 10.037 | +0.231 | 1.2593 | 1.2636 | -0.0043 |
| Q3_K_XL | 0.1150 | 0.1208 | -0.0058 | 23.11 | 23.55 | -0.44 | 9.986 | 10.241 | -0.255 | 2.1838 | 2.2804 | -0.0966 |
| Q4_K_S | 0.0561 | 0.0552 | +0.0009 | 17.63 | 20.99 | -3.36 | 7.237 | 7.017 | +0.220 | 0.8631 | 0.9232 | -0.0601 |
| Q4_K_M | 0.0480 | 0.0486 | -0.0006 | 19.25 | 18.68 | +0.57 | 7.090 | 6.466 | +0.624 | 0.6766 | 0.8442 | -0.1676 |
| Q4_1 | 0.0535 | 0.0550 | -0.0014 | 18.51 | 18.24 | +0.27 | 6.121 | 5.801 | +0.321 | 0.9346 | 0.9524 | -0.0178 |
| Q4_K_L | 0.0471 | 0.0503 | -0.0032 | 16.66 | 18.18 | -1.52 | 7.489 | 6.901 | +0.588 | 0.7156 | 0.8675 | -0.1519 |
| Q5_K_S | 0.0277 | 0.0289 | -0.0011 | 18.73 | 17.09 | +1.64 | 4.486 | 4.420 | +0.066 | 0.3455 | 0.3232 | +0.0223 |
| Q5_K_M | 0.0231 | 0.0212 | +0.0020 | 17.11 | 19.48 | -2.37 | 3.266 | 2.707 | +0.559 | 0.2697 | 0.2763 | -0.0066 |
| Q5_K_L | 0.0222 | 0.0195 | +0.0026 | 16.89 | 19.03 | -2.14 | 3.157 | 2.463 | +0.694 | 0.2747 | 0.2704 | +0.0043 |
| Q6_K | 0.0134 | 0.0132 | +0.0002 | 17.38 | 17.05 | +0.33 | 1.712 | 2.113 | -0.401 | 0.1296 | 0.1256 | +0.0040 |
| Q6_K_L | 0.0137 | 0.0132 | +0.0005 | 17.26 | 16.61 | +0.65 | 2.202 | 1.755 | +0.447 | 0.1301 | 0.1316 | -0.0016 |
Token probabilities
| Quant | Mean Δp v6 | Mean Δp v5 | Mean Δp diff | p99.9 Δp v6 | p99.9 Δp v5 | p99.9 Δp diff | p99 Δp v6 | p99 Δp v5 | p99 Δp diff | RMS Δp v6 | RMS Δp v5 | RMS Δp diff | Same top-p v6 | Same top-p v5 | Same top-p diff |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| IQ2_XXS | -1.914% | -1.903% | -0.011% | 95.68% | 93.06% | +2.62% | 55.87% | 55.35% | +0.52% | 17.09% | 16.82% | +0.27% | 73.12% | 73.36% | -0.24% |
| IQ2_XS | -1.960% | -1.901% | -0.059% | 89.65% | 84.76% | +4.89% | 47.54% | 45.91% | +1.63% | 15.17% | 15.02% | +0.16% | 77.78% | 77.03% | +0.75% |
| IQ2_S | -1.725% | -1.275% | -0.450% | 94.45% | 91.78% | +2.67% | 49.01% | 48.54% | +0.47% | 14.85% | 14.80% | +0.06% | 78.85% | 77.71% | +1.14% |
| IQ2_M | -1.454% | -1.575% | +0.121% | 91.00% | 91.79% | -0.79% | 35.58% | 36.54% | -0.96% | 12.72% | 13.06% | -0.34% | 81.43% | 80.86% | +0.56% |
| Q2_K | +0.582% | +0.565% | +0.017% | 95.31% | 87.20% | +8.12% | 46.16% | 46.01% | +0.15% | 12.79% | 12.65% | +0.14% | 81.08% | 81.16% | -0.08% |
| IQ3_XXS | -0.565% | -0.987% | +0.422% | 88.31% | 89.35% | -1.03% | 31.13% | 30.18% | +0.96% | 10.19% | 10.75% | -0.55% | 85.51% | 85.47% | +0.04% |
| Q2_K_L | +0.614% | +0.622% | -0.008% | 95.46% | 89.22% | +6.24% | 46.23% | 46.55% | -0.31% | 12.81% | 12.68% | +0.13% | 81.27% | 81.36% | -0.09% |
| IQ3_XS | -1.920% | -1.503% | -0.417% | 89.98% | 93.92% | -3.94% | 24.43% | 28.81% | -4.38% | 10.52% | 10.31% | +0.20% | 86.66% | 87.24% | -0.58% |
| Q3_K_S | -0.505% | -0.498% | -0.007% | 82.42% | 74.96% | +7.46% | 26.11% | 26.45% | -0.35% | 9.41% | 9.56% | -0.15% | 86.69% | 86.68% | +0.00% |
| IQ3_M | -1.357% | -1.383% | +0.026% | 90.29% | 91.39% | -1.10% | 21.81% | 21.03% | +0.78% | 8.92% | 8.99% | -0.07% | 88.56% | 88.72% | -0.16% |
| Q3_K_M | -0.090% | +0.005% | -0.095% | 64.90% | 71.31% | -6.40% | 23.22% | 24.12% | -0.90% | 8.28% | 8.44% | -0.17% | 88.86% | 88.74% | +0.12% |
| Q3_K_L | -0.125% | +0.003% | -0.128% | 63.11% | 67.21% | -4.10% | 21.27% | 22.22% | -0.95% | 7.90% | 8.14% | -0.24% | 89.59% | 89.10% | +0.49% |
| IQ4_XS | +0.097% | +0.021% | +0.076% | 65.69% | 66.02% | -0.33% | 17.16% | 16.42% | +0.75% | 6.20% | 6.14% | +0.06% | 92.60% | 92.53% | +0.07% |
| IQ4_NL | +0.119% | +0.021% | +0.098% | 65.76% | 61.38% | +4.39% | 16.43% | 15.99% | +0.44% | 5.95% | 5.88% | +0.08% | 92.89% | 92.91% | -0.02% |
| Q4_0 | -0.934% | -1.035% | +0.101% | 60.04% | 60.75% | -0.72% | 13.14% | 13.92% | -0.79% | 7.31% | 7.47% | -0.16% | 91.09% | 90.86% | +0.23% |
| Q3_K_XL | -0.107% | -0.015% | -0.092% | 64.85% | 67.51% | -2.66% | 20.96% | 21.47% | -0.51% | 7.93% | 7.98% | -0.05% | 89.90% | 89.53% | +0.38% |
| Q4_K_S | -0.452% | -0.246% | -0.206% | 41.10% | 41.95% | -0.86% | 11.60% | 11.28% | +0.32% | 5.68% | 5.08% | +0.60% | 93.15% | 93.34% | -0.19% |
| Q4_K_M | -0.445% | -0.308% | -0.137% | 41.81% | 36.94% | +4.87% | 10.29% | 10.66% | -0.37% | 5.08% | 4.98% | +0.10% | 93.66% | 93.56% | +0.10% |
| Q4_1 | -0.458% | -0.355% | -0.103% | 39.36% | 33.94% | +5.42% | 12.29% | 12.65% | -0.37% | 5.41% | 5.51% | -0.11% | 93.06% | 93.06% | -0.01% |
| Q4_K_L | -0.379% | -0.300% | -0.079% | 40.53% | 37.03% | +3.51% | 10.38% | 10.89% | -0.51% | 5.01% | 5.10% | -0.09% | 93.99% | 93.81% | +0.18% |
| Q5_K_S | -0.093% | +0.187% | -0.280% | 33.69% | 44.11% | -10.42% | 8.14% | 9.68% | -1.54% | 3.73% | 3.99% | -0.26% | 95.47% | 95.41% | +0.06% |
| Q5_K_M | -0.164% | +0.073% | -0.237% | 28.89% | 28.95% | -0.06% | 7.22% | 8.78% | -1.56% | 3.60% | 3.35% | +0.25% | 95.79% | 95.92% | -0.13% |
| Q5_K_L | -0.112% | +0.093% | -0.205% | 33.48% | 37.57% | -4.09% | 7.26% | 8.02% | -0.76% | 3.50% | 3.68% | -0.18% | 96.06% | 96.17% | -0.11% |
| Q6_K | -0.045% | -0.041% | -0.004% | 24.79% | 22.85% | +1.93% | 5.80% | 5.75% | +0.06% | 2.81% | 2.78% | +0.03% | 96.83% | 96.81% | +0.02% |
| Q6_K_L | -0.035% | -0.094% | +0.059% | 23.38% | 22.11% | +1.27% | 5.15% | 4.89% | +0.26% | 2.70% | 2.66% | +0.04% | 97.35% | 97.24% | +0.11% |
xlam, 100 chunks
KL divergence
| Quant | Mean v6 | Mean v5 | Mean diff | Max v6 | Max v5 | Max diff | p99.9 v6 | p99.9 v5 | p99.9 diff | p99 v6 | p99 v5 | p99 diff |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| IQ2_XXS | 1.5215 | 1.5464 | -0.0248 | 33.51 | 34.24 | -0.74 | 25.584 | 25.421 | +0.163 | 18.2494 | 18.3178 | -0.0684 |
| IQ2_XS | 1.5305 | 1.4920 | +0.0385 | 34.33 | 35.55 | -1.21 | 25.633 | 25.712 | -0.078 | 18.4603 | 18.0077 | +0.4526 |
| IQ2_S | 1.4291 | 1.3959 | +0.0331 | 34.09 | 35.47 | -1.38 | 25.883 | 25.168 | +0.715 | 17.9931 | 18.0189 | -0.0258 |
| IQ2_M | 1.1666 | 1.2090 | -0.0424 | 33.98 | 33.52 | +0.46 | 25.417 | 25.658 | -0.240 | 17.2129 | 17.3581 | -0.1451 |
| Q2_K | 1.1083 | 1.0864 | +0.0219 | 37.35 | 32.85 | +4.50 | 26.713 | 25.310 | +1.403 | 17.9104 | 17.4640 | +0.4464 |
| IQ3_XXS | 0.8729 | 0.9152 | -0.0424 | 33.87 | 36.36 | -2.49 | 25.254 | 25.363 | -0.110 | 16.4183 | 16.5097 | -0.0914 |
| Q2_K_L | 1.0929 | 1.0620 | +0.0309 | 33.86 | 32.64 | +1.21 | 25.783 | 24.621 | +1.162 | 17.5104 | 16.9871 | +0.5233 |
| IQ3_XS | 0.8433 | 0.8556 | -0.0123 | 39.39 | 38.95 | +0.44 | 26.272 | 25.412 | +0.861 | 16.3887 | 16.8789 | -0.4902 |
| Q3_K_S | 0.8693 | 0.8717 | -0.0024 | 37.38 | 38.05 | -0.68 | 27.586 | 27.950 | -0.365 | 17.4965 | 17.3691 | +0.1274 |
| IQ3_M | 0.7695 | 0.7738 | -0.0043 | 36.56 | 36.36 | +0.20 | 25.867 | 26.553 | -0.686 | 15.9327 | 16.5588 | -0.6260 |
| Q3_K_M | 0.7252 | 0.7414 | -0.0162 | 39.06 | 34.02 | +5.04 | 25.352 | 25.942 | -0.590 | 16.5304 | 16.3876 | +0.1427 |
| Q3_K_L | 0.7165 | 0.7265 | -0.0099 | 35.47 | 34.25 | +1.22 | 25.912 | 26.049 | -0.137 | 16.6401 | 16.5444 | +0.0957 |
| IQ4_XS | 0.4421 | 0.4464 | -0.0043 | 38.64 | 36.58 | +2.06 | 23.158 | 23.517 | -0.359 | 11.9790 | 12.2918 | -0.3128 |
| IQ4_NL | 0.4009 | 0.4033 | -0.0023 | 38.91 | 39.74 | -0.83 | 22.873 | 22.980 | -0.108 | 11.5754 | 11.2417 | +0.3336 |
| Q4_0 | 0.6195 | 0.6135 | +0.0060 | 35.38 | 36.37 | -0.99 | 25.539 | 24.885 | +0.654 | 15.2826 | 15.1994 | +0.0832 |
| Q3_K_XL | 0.7124 | 0.7128 | -0.0004 | 33.50 | 34.45 | -0.95 | 25.739 | 25.205 | +0.534 | 16.1214 | 16.2642 | -0.1427 |
| Q4_K_S | 0.4013 | 0.3989 | +0.0024 | 40.56 | 39.72 | +0.84 | 23.307 | 23.399 | -0.092 | 10.9874 | 11.4182 | -0.4308 |
| Q4_K_M | 0.3668 | 0.3768 | -0.0100 | 33.65 | 39.88 | -6.23 | 22.584 | 22.897 | -0.313 | 10.8570 | 11.4776 | -0.6206 |
| Q4_1 | 0.3562 | 0.3646 | -0.0085 | 31.08 | 34.55 | -3.47 | 22.709 | 23.397 | -0.688 | 10.2744 | 10.3835 | -0.1090 |
| Q4_K_L | 0.3717 | 0.3797 | -0.0080 | 34.84 | 34.83 | +0.01 | 23.128 | 23.937 | -0.809 | 10.9539 | 11.2557 | -0.3017 |
| Q5_K_S | 0.2833 | 0.2859 | -0.0026 | 34.85 | 29.96 | +4.89 | 22.032 | 22.123 | -0.091 | 8.8313 | 9.3546 | -0.5232 |
| Q5_K_M | 0.2667 | 0.2518 | +0.0149 | 33.52 | 30.36 | +3.16 | 21.741 | 21.676 | +0.065 | 9.2150 | 7.9675 | +1.2475 |
| Q5_K_L | 0.2499 | 0.2534 | -0.0035 | 33.34 | 33.64 | -0.29 | 22.019 | 22.316 | -0.297 | 7.8243 | 7.8356 | -0.0113 |
| Q6_K | 0.1897 | 0.1876 | +0.0020 | 29.48 | 31.24 | -1.76 | 19.655 | 19.154 | +0.500 | 6.1413 | 6.1686 | -0.0273 |
| Q6_K_L | 0.1808 | 0.1908 | -0.0100 | 29.27 | 32.48 | -3.21 | 19.741 | 19.820 | -0.079 | 5.5969 | 5.8639 | -0.2670 |
Token probabilities
| Quant | Mean Δp v6 | Mean Δp v5 | Mean Δp diff | p99.9 Δp v6 | p99.9 Δp v5 | p99.9 Δp diff | p99 Δp v6 | p99 Δp v5 | p99 Δp diff | RMS Δp v6 | RMS Δp v5 | RMS Δp diff | Same top-p v6 | Same top-p v5 | Same top-p diff |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| IQ2_XXS | -5.808% | -6.188% | +0.380% | 99.94% | 99.93% | +0.00% | 95.09% | 95.51% | -0.42% | 31.62% | 31.87% | -0.24% | 72.51% | 72.12% | +0.39% |
| IQ2_XS | -7.781% | -8.382% | +0.601% | 99.77% | 99.46% | +0.31% | 78.06% | 79.99% | -1.93% | 29.91% | 30.65% | -0.74% | 73.23% | 73.02% | +0.20% |
| IQ2_S | -6.253% | -6.546% | +0.293% | 99.77% | 99.86% | -0.08% | 83.50% | 84.26% | -0.76% | 28.59% | 28.91% | -0.33% | 74.84% | 74.56% | +0.28% |
| IQ2_M | -3.357% | -4.166% | +0.809% | 99.90% | 99.88% | +0.03% | 90.11% | 88.43% | +1.68% | 25.48% | 26.14% | -0.66% | 78.22% | 77.86% | +0.36% |
| Q2_K | -1.823% | -0.334% | -1.489% | 99.59% | 99.81% | -0.22% | 87.62% | 91.77% | -4.14% | 23.50% | 23.33% | +0.16% | 79.59% | 80.05% | -0.46% |
| IQ3_XXS | -0.506% | -1.166% | +0.660% | 99.57% | 99.62% | -0.05% | 82.68% | 80.86% | +1.81% | 20.42% | 20.52% | -0.10% | 82.62% | 82.08% | +0.54% |
| Q2_K_L | -1.649% | -0.108% | -1.541% | 99.73% | 99.81% | -0.07% | 87.69% | 92.82% | -5.13% | 23.42% | 23.49% | -0.07% | 79.74% | 80.16% | -0.42% |
| IQ3_XS | -2.046% | -1.859% | -0.187% | 99.55% | 99.57% | -0.01% | 76.65% | 78.81% | -2.16% | 20.48% | 20.21% | +0.27% | 82.95% | 82.88% | +0.08% |
| Q3_K_S | +0.065% | +0.346% | -0.281% | 99.85% | 99.83% | +0.02% | 86.67% | 87.56% | -0.89% | 19.03% | 18.99% | +0.05% | 83.68% | 83.86% | -0.18% |
| IQ3_M | -1.089% | -0.924% | -0.165% | 99.62% | 99.77% | -0.15% | 75.35% | 79.44% | -4.09% | 18.52% | 18.47% | +0.05% | 84.60% | 84.52% | +0.09% |
| Q3_K_M | +0.632% | +0.923% | -0.291% | 99.83% | 99.84% | -0.02% | 85.65% | 89.78% | -4.13% | 17.70% | 18.34% | -0.64% | 85.36% | 85.34% | +0.01% |
| Q3_K_L | +0.959% | +0.908% | +0.051% | 99.86% | 99.90% | -0.05% | 87.30% | 88.12% | -0.82% | 17.62% | 17.91% | -0.29% | 85.84% | 85.66% | +0.18% |
| IQ4_XS | +0.449% | +0.400% | +0.049% | 98.54% | 98.84% | -0.30% | 61.07% | 58.69% | +2.38% | 13.30% | 13.28% | +0.02% | 89.26% | 89.25% | +0.01% |
| IQ4_NL | +0.199% | +0.125% | +0.074% | 95.83% | 96.59% | -0.76% | 52.80% | 51.01% | +1.79% | 12.46% | 12.63% | -0.17% | 89.74% | 89.69% | +0.05% |
| Q4_0 | -0.767% | -0.698% | -0.069% | 99.01% | 98.36% | +0.65% | 59.82% | 60.82% | -1.00% | 15.62% | 15.58% | +0.04% | 87.19% | 87.16% | +0.03% |
| Q3_K_XL | +0.936% | +0.870% | +0.066% | 99.79% | 99.80% | -0.01% | 87.30% | 87.78% | -0.49% | 17.68% | 17.87% | -0.19% | 85.74% | 85.82% | -0.08% |
| Q4_K_S | -0.395% | -0.466% | +0.071% | 96.24% | 95.63% | +0.61% | 45.32% | 39.23% | +6.09% | 12.53% | 12.09% | +0.44% | 89.84% | 89.88% | -0.03% |
| Q4_K_M | -0.334% | -0.431% | +0.097% | 95.00% | 95.35% | -0.35% | 40.20% | 36.02% | +4.17% | 11.85% | 11.65% | +0.20% | 90.62% | 90.37% | +0.25% |
| Q4_1 | -0.393% | -0.590% | +0.197% | 96.21% | 95.23% | +0.97% | 41.33% | 37.83% | +3.49% | 11.87% | 12.04% | -0.17% | 90.14% | 89.96% | +0.18% |
| Q4_K_L | -0.266% | -0.375% | +0.109% | 95.66% | 98.07% | -2.41% | 38.78% | 36.61% | +2.17% | 11.74% | 11.76% | -0.02% | 90.70% | 90.49% | +0.21% |
| Q5_K_S | +0.280% | +0.311% | -0.031% | 96.75% | 95.75% | +1.00% | 39.40% | 37.74% | +1.66% | 10.16% | 10.05% | +0.12% | 92.58% | 92.51% | +0.07% |
| Q5_K_M | +0.523% | +0.265% | +0.258% | 95.43% | 94.64% | +0.79% | 40.16% | 31.55% | +8.61% | 9.53% | 9.09% | +0.44% | 93.05% | 93.14% | -0.09% |
| Q5_K_L | +0.424% | +0.277% | +0.147% | 95.09% | 94.58% | +0.51% | 35.96% | 31.78% | +4.18% | 9.22% | 9.22% | -0.00% | 93.34% | 93.15% | +0.19% |
| Q6_K | +0.137% | +0.055% | +0.082% | 79.39% | 81.81% | -2.42% | 21.11% | 20.16% | +0.95% | 7.17% | 7.48% | -0.31% | 94.59% | 94.62% | -0.03% |
| Q6_K_L | +0.151% | +0.029% | +0.122% | 80.39% | 80.79% | -0.39% | 20.57% | 19.51% | +1.06% | 7.19% | 7.45% | -0.26% | 94.92% | 94.53% | +0.40% |
TDLR?
Imatrix corpus smaller, more focused, still diverse, includes chat-templated tool calling, results seem promising, more testing already starting, expect more (though much briefer) updates as more is discovered!
Thanks
Huge thanks go out to:
- RedHat for giving me the opportunity to work on llama.cpp as a profession
- LTT/LMG for giving access to a beast GPU making all this possible within my lifetime
- llama.cpp and ggml-org of course for existing and being awesome
And of course, to you, for reading this and downloading models!