Q6_K is surprisingly phenomenal.
#7
by YourMOMSaidHi - opened
I am still learning about quants to be honest, but it outperforms the full 2b-it from google in accuracy not just speed. It's very fast for using for inline reasoning on my scripts.
I am still learning about quants to be honest, but it outperforms the full 2b-it from google in accuracy not just speed. It's very fast for using for inline reasoning on my scripts.