Eren Ata

Agreed on the median. My guess is that most of the gain on our card comes from Q4_K_M fitting entirely on the GPU while Q8_0 has to leave part of its layers on the CPU, so on a GPU with more memory I would expect the gap to shrink. If you run the same comparison on other hardware, I would be glad to see the numbers, especially the 95th percentile.

One correction to how I put the clarification idea: with our model the question is written before the candidate ids, so an app cannot skip it. The model itself would have to emit a short or empty question. We could test that without retraining by limiting that field with a grammar, the same kind we already use for tool names and object ids, but we have not tried it yet.

@Twu31 Thanks for the careful question. Mostly the typical response, not the slow tail.

On our laptop (RTX 3050 Ti, 4 GB) the median time to the first valid call is 2.1 s with Q4_K_M and 2.7 s with Q8_0, but the 95th percentile is about the same: 4.9 s against 4.7 s. In both builds the slowest turns are clarification questions, where the model has to write a question and the candidate ids. That is around 50 tokens, and at roughly 9 to 11 tokens per second it takes 4 to 5 seconds whichever file you use. So the tail follows how much the model has to write more than the size of the file.

Two caveats. This is one machine and 96 turns per build, so the 95th percentile rests on the slowest three or four turns. And the laptop warms up: Q4's median was 1.1 s in the first run and 2.6 s in the third, while Q8's did not drift the same way. How much you gain depends on how hot the GPU is.

For an interactive loop, I would try letting the app build the clarification question from the candidate ids instead of having the model write it. I expect that to help the tail more than the quantisation does, but we have not tested it yet.