Considering the support of QAT, Unsloth Dynamic 2.0 Quants (UD) + TQ2_0 and TQ1_0 quantization
#1732
by
wahidmounir
- opened
Recently I've stumbled upon the QAT, Unsloth Dynamic 2.0 Quants (UD) + TQ2_0 and TQ1_0 quantizations of several models
and I've been testing them a lot
TQ1_0 and TQ2_0 are good for large models but give bad result for small models and math tasks but they are worth it for other tasks
for smaller models QAT and UD2.0 give better results than other quantisations
The request is it will very nice if mrdermacher team include these quantizations in their quantized models
(didnt check what are those) I mean if they are included in default llama cpp, I dont see why not
@nicoboss