Watches silently.
@Test157t
It's already coming in hot.
Gods, it's like 100GB 80GB of files!
I will say 9B is quite the unusual parameter size, at least for me, is that stable? Is it GQA? If not, could it be in the future?
it's just two 7B in a passthrough with overlapping layers. Should inherit all characteristics of a Mistral finetune.
It is quite good though, in my testing.
@jeiku -- Alrighty! Sounds good!
quantization_options = [
"Q4_K_M", "Q4_K_S", "IQ4_NL", "IQ4_XS", "Q5_K_M",
"Q5_K_S", "Q6_K", "Q8_0", "IQ3_M", "IQ3_S", "IQ3_XS", "IQ3_XXS"
]
Maybe time to put those IQs to the test.
Also, of course, Bepis-chan is a cutie.
in my testing iq3_xxs is worthwhile, but others show either no improvement or worse perplexity than similar k quant
but others show either no improvement or worse perplexity than similar k quant
Yeah, I honestly want more feedback on this to focus on the more important quants. They take way longer than a normal quant, so I have to know if it's even worth it.
Update:
Everything should be uploaded in about 15 minutes.