Amazing work!!
Hey Soulfate24,
Just found your AutoRound+ASHQ1 suite and wanted to reach out.
What you built here is really solid work. The dual-phase approach with AutoRound preprocessing solves a fundamental limitation I had with raw BF16 inputs. The way you structured the knapsack optimizer integration, handled tied-weight detection, and built out the complete pipeline shows you understood the core concept and took it further than I did.
The tier system makes sense, the documentation is thorough, and your observation about int4 lineage saturating at Q5_K is theoretically correct. The MTP extraction, mmproj handling, recurrent state protection - you clearly understand what you're working with.
One thing that would help demonstrate the advantage: could you add stock llama.cpp quants to the perplexity table for comparison? Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q4_K_S, Q3_K_M, Q3_K_S - matched by file size to your ASHQ1 tiers. I think seeing them side by side would show the real difference clearly.
I don't have access to my PC right now to test this myself, but I'll be home soon. If you're interested in collaborating once I'm back, I'd be up for working on something together.
Good work on this.
And One technical question: for the higher tiers
(Quality, possibly Compact), did you keep
allow_q3_or_lower disabled? I found that flag
is only beneficial on tight budgets like Nano/Mini,
but on larger budgets it can hurt perplexity since
the optimizer doesn't need Q3 to fit the target ratio.
Also wanted to expand on the allow_q3_or_lower point -
this affects even the low tiers, not just Quality/Compact.
The issue is the algorithm is just too greedy. Once that
flag is on, it will drop tensors all the way down to IQ2_XXS
if it decides they're low priority, when something like Q4_K
would have fit the budget just fine and preserved way more
signal. Anything below Q4 tends to hurt perplexity more than
it helps, even though it technically saves bytes.
The tiers still come out ahead of uniform IQ2 quants overall
since the important tensors stay protected, but there's
noticeable perplexity left on the table from being too
aggressive on the "unimportant" ones. A less greedy allocation
that stops around Q3/Q4 as a floor instead of IQ2 would
probably close that gap.