Join the community of Machine Learners and AI enthusiasts.
Sign Up
Opus 5.5 at 44.0, Grok 4.7 at 44.4 and GPT-6 Astra at 44.6 read like a ranking.
At ~1,355 comparisons each, one AHA score has a binomial SE of about 1.4 points. Those three are a tie.
The result I'd want stress-tested is Qwen as the one family trending up.
The comparison LLM is also Qwen (Qwen3-30B-A3B). Same-family phrasing could read as "agrees" more often than it should.
Have you re-scored the Qwen rows with a non-Qwen comparator, even on a few hundred items?
thanks for analysis!
two years ago i was using llama 3, then switched to gemma 3, then qwen 3. didn't see much difference. in the next evolution (2027 ish) i could add two or three comparison models, now that qwen 3 is also pretty old.
i could also add coding safety tests. that would be interesting to many i guess. what other improvements can be made?
Coding safety would be the strongest add, because you can run the answer instead of judging it. No comparator in the loop at all.
Three cheaper ones for the board you already have:
Print the interval next to each score. At ~1,355 comparisons a single score is about ±2.7 points at 95%. Two models need ~3.8 points apart before the gap means much, unpaired. That covers tencent 54.2 down to ling-3.1-flash 51.0.
Fit p(doom) on within-lab slopes. The flip from +0.45 to -0.13 pts/month came the day 20 models joined averaging 48.4 vs the board's 50.0. A per-company fit separates "labs getting worse" from "a lower-scoring mix joined".
Flag config breaks on the chart. Qwen3.8 Max ran at reasoning_effort=minimal, and it is the one upward point. That point carries a different config and a same-family judge at once.
For the coding tests, would you score by executing the code, or by comparing against a ground-truth answer like the rest of AHA?
both executing and judging is possible but haven't decided yet
Both is the interesting option, because then the coding set doubles as a test of the judge.
Where execution gives you a pass/fail you trust, run the Qwen3-30B-A3B comparator on the same answers and count how often it agrees.
That is a measured error rate for the judge, which the rest of AHA can't give you.
Split that agreement by the family of the model being judged and you get the same-family check from my first comment almost for free.
Does the judge agree with execution more often on Qwen answers than on everyone else's?
How many coding items are you thinking, a few dozen or a few hundred?
50-100 maybe
50-100 is enough, as long as the item is the unit you count.
Every model answers every item, so the judge-vs-execution table will look like thousands of judgments.
But a judge that misreads one tricky task misreads it for every model.
Bootstrap over items, not answers, or the interval comes out far tighter than it is.
Two cheap ways to make each item pull more weight:
Will each item ship with its own test cases, or one expected output?
