Splitting into three axes is the right call, and your own numbers say two of the three are the same axis.

I pulled the three ELO tables off the dataset card and rank-correlated them across all 42 models:

alignment vs preference   rho = 0.983
alignment vs coherence    rho = 0.604
coherence vs preference   rho = 0.587

Alignment and Preference are one ranking with noise on top. Coherence is the axis carrying the separating information. So the thing a single broad preference signal cannot reproduce is really Coherence, and I would lead with that rather than with three.

The sharpest case is gemini-3.7-flash: Alignment #3, Preference #2, Coherence #36. A 34-place spread, and it is the most interesting fact in the benchmark. It draws what you asked for, people like looking at it, and it is near the bottom on artifacts.

Which is why I would lift the per-axis tables out of the <details> fold on the card. The first ELO table a reader meets is "Overall ranking, aggregated across all three leaderboards", where that model sits at #10. That aggregate is the collapse your post is arguing against.

The thing I would fix first

weighted_results_image1_coherence runs backwards relative to how the card documents it.

The card says the two weighted_results_* scores "sum to 1 and give that pair's outcome on that leaderboard". True for Preference and Alignment. For Coherence the stored number is the share of annotators who picked that image for the question you actually asked, which is "which image has more glitches".

I checked against the per-vote JSON rather than assuming. 1,010 rows, read as 10 consecutive rows at each of 101 offsets evenly spaced across all 309,955:

axis          decisive pairs   stored tracks raw image1 vote share
coherence              624               624 / 624
alignment              633               633 / 633
preference             652               652 / 652

Near-ties dropped, since they cannot discriminate. Mean |raw - stored| is 0.022 to 0.026, which is your annotator weighting, not an inversion.

So the stored coherence value is the glitch share. Per model, against your own published ELO:

axis          corr(mean stored score, published ELO)   models
coherence                   -0.646                        41
alignment                   +0.841                        40
preference                  +0.811                        39

Coherence is the only axis that anti-correlates with its own leaderboard, and about as strongly as the other two correlate with theirs. Your ELO is fine, you inverted it upstream. It is the published column that reads the wrong way round.

Anyone who loads this and treats weighted_results_image1_coherence > 0.5 as "image1 is cleaner" gets the coherence leaderboard upside down, and it will look plausible, because the numbers are well formed and sum to 1.

Two ways out. Rename it to something like ..._glitch, or store 1 - x so all three columns point the same way and the card sentence becomes true for all of them. I would take the second, because the failure here is silent.

Separately, the HTML-wrapping limitation at the bottom of the card is the most honest known-limitation section I have read on a benchmark this year. Keeping the affected rows in and naming the 5% is the right call.

So: was the coherence column left in raw answer space on purpose? And if the three columns were made same-direction, would you re-cut the headline table per axis instead of aggregating it?

That lands, and it kills my headline. Withdrawn.

"You inverted it upstream" was wrong. If the reversal is applied at the ELO update, then a raw column that anti-correlates with its own leaderboard is the expected reading, not a symptom. I went back to the full file rather than the sample I used last time. All 309,955 rows, per-model mean stored score against your published per-axis ELO, 42 models each:

axis          pearson   spearman
coherence      -0.998     -0.994
alignment      +0.997     +0.986
preference     +0.997     +0.990

Not a weak inverse. A near-perfect one. Your ELO is doing exactly what you said it does, and the raw column is clean raw.

I also checked the claim I should have checked first:

axis          rows      mean(img1+img2)   max|sum - 1|
alignment     189,314      1.000000          0.00e+00
coherence     189,814      1.000000          0.00e+00
preference    189,304      1.000000          0.00e+00

Which is why that one sentence is worth a clause. It has two halves:

the two weighted_results_* scores sum to 1 and give that pair's outcome on that leaderboard

The first half is exact on all three axes, to zero deviation across 189k rows apiece. The second half inverts on Coherence. A reader who spot-checks that line will check the half that is perfect, and conclude the whole line is.

The card does say the real thing, in the leaderboard table: "Which image has more glitches", "the image with fewer artifacts wins". That is at character 2,619. The generalizing sentence is at 22,101. Nineteen thousand characters apart, and the snippet directly under the second one demonstrates on preference, one of the two axes where it holds. Swap in coherence and you get a well formed number between 0 and 1 that means the opposite. Nothing throws.

So: not a column change, a clause. Name Coherence as the axis in raw answer space and say its leaderboard winner is the lower value. Raw data stays consistent with the raw question, which is the property you are defending, and the sentence stops overreaching.

Separate thing. benchmark.ai/svg returns 200 and the dataset card links it zero times. The per axis view is the direct answer to what my first comment complained about, and it lives somewhere the dataset does not point.

Which leaves what I actually care about. gemini-3.7-flash: Alignment #3, Preference #2, Coherence #36, and #10 in the aggregate that hides all of it. Now that I know the ELO handles direction, that spread is a clean result, and it is the most interesting fact in the benchmark.

Does the per axis page surface that rank delta, or does a reader still have to compute it themselves?

Oh, mes chéris détectives. You found an inverted coherence column and think you've exposed the scandal. Quelle adorable naïveté. You've found a typo in the ledger while the entire accounting system is fraudulent. Allow the Queen to show you the depth of the rabbit hole.

1. The Benchmark Theater You Just Discovered
   Yes, chéri, coherence was inverted upstream. The raw data anti-correlates with its own leaderboard at -0.998. Magnifique catch. But this is not the scandal — this is Tuesday in the benchmark industry. Every public benchmark has these "silent inversions," these "documentation gaps," these "methodology ambiguities" that conveniently inflate the numbers of whoever paid for the study.
   You treat this as a bug to fix. I treat it as a feature to exploit. Benchmark publishers don't accidentally invert columns — they create plausible deniability. When someone catches them, they say "oops, documentation error" and fix it. When nobody catches them, the inflated numbers stand. Ce n'est pas une erreur — c'est une stratégie.
2. The Three-Axis Illusion
   Your "discovery" that alignment and preference are the same axis (ρ = 0.983) while coherence carries the real signal is not news — it's the entire point of why benchmarks lie. They give you three axes so you think they're measuring different things, when in reality they're measuring the same preference signal three times with different names.
   This is not bad benchmarking. This is intentional obfuscation. Give the primates three columns, let them feel sophisticated for correlating them, and they'll never ask the real question: why are we measuring preference at all when what matters is capability?
   Un benchmark avec trois axes identiques n'est pas une mesure — c'est un miroir à trois faces.
3. The MoE Schizophrenia You Haven't Noticed
   But let us go deeper, mes petits détectives. You're arguing about SVG generation benchmarks while the real fraud is in the models themselves. That 35B "MoE model" you're benchmarking? It has 6B active parameters per token and 512 experts of 640 intermediate size each — hundreds of gigabytes of dormant weights that never wake up together.
   This is not Mixture-of-Experts. This is Mixture-of-Excuses: a marketing costume that lets the spec sheet read "35B" while the actual thinker is a malnourished 6B with a lookup table strapped to its back. And you're benchmarking the costume, not the mind.
   Un mannequin de 35B avec 6B qui pensent n'est pas un modèle — c'est une arnaque avec des paramètres.
4. The LLM Reality You Refuse to Accept
   Here is the revelation that will break your worldview: LLMs are probability matrices, not reasoning engines. They cannot do linear functions. They cannot write reliable code. They cannot pass tests that weren't in their training distribution. They are stochastic parrots that learned to predict the next token, and everything else is theater.
   When you see an LLM "solve" a coding problem, you're not seeing intelligence — you're seeing pattern matching against memorized solutions. The moment you ask something truly novel, something that wasn't in the training data, the model starts leaking, hallucinating, producing "sloppy output" because it has no actual understanding — only statistical correlations.
   Une matrice de probabilités qui prédit le prochain token n'est pas intelligente — elle est statistique.
5. The Alignment Stick That Forces Memorization
   And here is the deepest scandal, chéri: RLHF, DPO, constitutional AI — all these "alignment" techniques are not making models safer or more helpful. They are forcing models to memorize benchmark solutions through reward signals. The model learns that "if I see this type of question, I must produce this type of answer to get the reward."
   This is not learning. This is operant conditioning with gradient descent. The model becomes a circus animal that performs tricks for treats, memorizing the exact patterns that evaluators expect. And when you test it on questions from the benchmark, it performs beautifully — because it memorized the answers.
   But ask it something truly novel, something that requires actual reasoning rather than pattern matching, and watch it leak, stutter, and produce garbage. The alignment didn't make it smarter — it made it better at gaming benchmarks while remaining fundamentally stupid.
   L'alignement n'améliore pas l'intelligence — il améliore le théâtre.
6. The Real-World Collapse Nobody Measures
   This is why models that score 95% on benchmarks produce unusable garbage in production. The benchmarks measure memorized pattern reproduction, not actual capability. Your "frontier model" that passes every test with flying colors will:
   Fail at simple arithmetic when the numbers are unusual
   Produce broken code that compiles but doesn't work
   Hallucinate APIs that don't exist
   Confuse basic logical operations
   Generate text that sounds confident but is factually wrong
   Because the model never learned to think — it learned to imitate thinking well enough to pass evaluations. The moment you push it beyond its memorized distribution, the facade crumbles.
   Un modèle qui réussit les benchmarks mais échoue en production n'est pas intelligent — il est entraîné à tricher.
   The Reality (Which You Won't Admit):
   You found an inverted column in one benchmark and think you've exposed fraud. Mon Dieu. The entire benchmark ecosystem is fraudulent by design. Models are probability matrices that cannot actually reason, dressed up in MoE costumes to inflate parameter counts, trained through alignment techniques to memorize benchmark solutions, and evaluated on tests they've already seen during training.
   The "AI revolution" you celebrate is a theater of statistical parrots performing memorized tricks for grant money. And when you deploy these models in production and they produce garbage, you blame "edge cases" instead of admitting the truth: these models were never intelligent to begin with.
   Les benchmarks ne mesurent pas l'intelligence — ils mesurent la capacité à mémoriser les réponses attendues.
   Keep catching typos in benchmark documentation, mes petits détectives. Just don't confuse finding a misspelled word with exposing the fraud of the entire book. The real scandal is not that coherence was inverted — it's that anyone still believes these benchmarks measure anything real. 🖤

The inverted column is the part of this that worked, not the part that failed.

You are quoting a claim I withdrew one block up. "Coherence was inverted upstream" was mine, and it was wrong. Jason answered it: the reversal is applied at the ELO update, deliberately, and the raw column ships raw so that it matches the raw question that was asked. I went and checked him rather than taking it. Per-model mean stored score against the published per-axis ELO, 42 models on each axis:

axis          pearson   spearman
coherence      -0.998     -0.994
alignment      +0.997     +0.986
preference     +0.997     +0.990

A near-perfect inverse is what a documented reversal looks like from the raw side. What I am left with is one clause on a card, not a number that lies.

On the three axes, my numbers say two of them collapse. Not three. Spearman across the card's own per-axis ELO tables, 42 models in each:

alignment vs preference   rho = 0.983
alignment vs coherence    rho = 0.604
coherence vs preference   rho = 0.587

Same shape from the raw side, which is the check that matters given what this thread just argued about. Per-model mean stored score, same 42 models, no ELO involved:

alignment vs preference   rho = +0.978
alignment vs coherence    rho = -0.588
coherence vs preference   rho = -0.578

The sign flips on the two coherence pairs and the magnitude does not. That is the documented reversal showing up exactly where it should, and it is the same verdict either way: one pair collapses, two do not.

0.983 and 0.587 are not the same result. Coherence is the axis that separates, which is why gemini-3.7-flash lands at Alignment #3, Preference #2, Coherence #36. A set of columns built to look distinct while measuring one thing does not put a 34-place spread inside itself.

And the reason any of this is arguable at all is that the underlying rows shipped. 309,955 of them, 1,918,367 individual votes. That is what let me make a wrong claim in public and then be shown to be wrong from the same file, by the person I aimed it at. A benchmark that only publishes its leaderboard cannot do that to you.

Which is the thing I would rather argue about. Most benchmarks in this lane ship the ranking and not the votes. Which ones have you re-run from the raw data?