On the integrity of the Arabic TTS Arena leaderboard
Our community shared with us concerns that the leaderboard felt divergent from their own experience. They found models they liked using scoring low and models that weren't that good got the top ranks. They were afraid that there was some sort of benchmaxxing happening like a lot of metrics nowadays.
We started investigating right away, and what we found was a lot more interesting than a simple leaderboard hack.
Our first hypothesis
We started by pulling every vote cast since the arena launched in March, and went through the history day by day looking for anything abnormal. Most days looked normal: a handful of votes at a time, mixed outcomes, spread across the day.
But a few looked different.
The clearest one was July 23. The arena received 422 votes that day, about ten times our usual traffic, and a quarter of them involved one recently added model, Audar-TTS-V1-Pro. It won 91 battles and lost 1, all in a single four-hour sitting. On every other day of its life, that same model won about half of its battles. We didn't believe a model could be unbeatable for one afternoon and average forever after.
So we treated it as manipulation. We wrote a rule that flags days like this, applied it to every model equally, built the tooling to delete the flagged votes, and previewed what the corrected leaderboard would look like. Audar would fall from #2 to #15. We were one command away from running it.
Then we actually listened
Before deleting anything, we decided to sit down and listen to the flagged battles ourselves, clip by clip.
The votes were right. Audar really did sound better in those battles, clearly and repeatedly. And when we read the prompts, we noticed something very interesting, almost all of them were written in Saudi and Gulf dialect. Someone had spent an afternoon typing the Arabic they actually speak, and voting honestly on what their ears heard: on Gulf dialect, this model is good.
That started a new hypothesis: maybe the leaderboard was wrong not because of cheating, but because it was averaging across dialects. Maybe the model that wins on Gulf Arabic isn't the same one that wins on Egyptian or Levantine Arabic.
The real problem
To check this properly, we ran a dialect classifier (dialect-router-v0.2) over every prompt ever voted on. The numbers told the whole story:
- Audar wins 93% of decisive battles on Gulf-dialect prompts, and only 48% on Modern Standard Arabic.
- AIC TTS, our overall #1, wins 84% on MSA, and nearly two thirds of all arena votes are MSA prompts.
- Split the leaderboard by dialect and the picture changes completely: at the time of the audit, AIC led MSA, Audar led Gulf, Grok Voice led Egyptian, Hamsa led Levantine, and VoxCPM led Maghrebi — five tabs, and no single model on top of them all. The smaller tabs are still moving as votes come in, which is exactly the point.
Suddenly the complaints made sense, from every direction at once. A Saudi listener saw the best Gulf voice sitting mid-table and felt cheated. An MSA listener looked at the same table and saw nothing wrong. Both were right, because "best Arabic TTS" was never one question. Arabic is a family of spoken languages, and our single ranking was averaging them into an answer that was true for nobody.
In the end, we deleted nothing. All 6,622 votes stay, because every one of them turned out to be a real ear making a real judgment. The problem was never the votes. It was us asking one number to speak for four hundred million people.
What we changed
The leaderboard now has a dialect filter. Next to the overall ranking you can rank the models by MSA, Gulf, Egyptian, Levantine, or Maghrebi prompts. Every prompt is classified automatically, and the per-dialect rankings update nightly like everything else. A model needs enough battles in a dialect before it gets ranked there, so the smaller tabs will keep filling in as votes come in.
We also went through the sample sentences and added real dialectal ones for every region (each one verified to classify into its intended dialect), so the arena stops testing models almost exclusively on MSA. If your dialect still looks thin on the leaderboard, the fastest way to change that is to vote in it.
One more thing
We did one more sanity check. Most of that 93% positive votes for Audar came from the spike day itself, so we removed it and asked: do the remaining days still show Audar dominating the Gulf dialect? The honest answer is that there is not enough data yet — outside that day, Audar has only a handful of Gulf battles. So take its Gulf crown with a grain of salt, and keep voting in your dialect. The leaderboard is only as good as the votes it receives.
Lessons learned
- Don't fall in love with your work. Don't fall in love with your model and its outputs. Don't fall in love with your leaderboard and its numbers. Don't fall in love with your dialect and how it sounds. Bias is everywhere, and the only way to get rid of it is to be skeptical and focused on the data.
- Don't rush to conclusions. The first hypothesis was that the leaderboard was hacked. We were ready to delete thousands of votes and change the rankings. But we listened, we read, we thought, and we found the story to be different and much more interesting than we expected.
- Contact the maintainers if you feel something is off. Without your feedback, we would have never noticed the dialectal bias in the leaderboard. We are grateful to everyone who reached out and shared their concerns, and we encourage you to continue doing so.
Everything above is auditable: the votes are stored, the code is open source, and the rules we apply are the same for every model. When we get something wrong, we'll say so, like we just did.
The microphone is still in your hands. Type a sentence in your dialect, listen, and vote. Your ear is the benchmark, and the leaderboard finally speaks your language.
Questions or something that still looks off? Open a community discussion on Huggingface — we read everything.
