comparia-fr-arena: 200k human preferences on how models speak French

Most of what we know about which model answers "better" comes from English. Preference datasets, reward models, and public leaderboards are overwhelmingly built on English prompts judged by English speakers. That is a problem, because preference is not universal: tone, register, politeness, and what counts as a good answer all shift from one language and culture to the next. A model tuned to please an English crowd is not automatically the model a French speaker would pick.

Today compar:IA, the French public chatbot arena run by the Ministère de la Culture, releases comparia-fr-arena: 641,572 conversation turns, 426,427 blind comparisons, and 209,824 human preference votes, collected mostly in French from real users over roughly 20 months. To our knowledge it is the largest open human-preference dataset centered on French.

from datasets import load_dataset

ds = load_dataset("ministere-culture/comparia-fr-arena", split="train")

What is in it

Every row is one turn of a conversation, shown to a user as two anonymous answers side by side. The user reads both and can express a preference. A single row carries both responses, the full conversation history for each side, which two models produced them, and the verdict.

A real example, lightly trimmed:

User (Pointe-à-Pitre): "Nous sommes en octobre, j'habite à Pointe-à-Pitre, propose moi une recette locale avec des fruits et légumes de saison."

Two anonymous models answer. One returns a seasonal Creole salad; the other a different local dish. The user picks.

This is what makes the data specifically French, and not English prompts run through translation: the questions are grounded in French daily life, from Guadeloupean cooking to French administrative procedures, schoolwork, and law.

Where it comes from

compar:IA is a public arena. Anyone can send a prompt, receive two answers from two anonymous models drawn from a pool of 119, and say which they preferred. The blind format is the point: users judge the answer, not the brand.

• ~92% French. 593,013 of the turns are primarily French. The long tail includes English, Spanish, Italian, German, Chinese, and regional languages such as Occitan.
• 119 models, proprietary and open, large and small: gpt-4.1-mini, llama-3.3-70b, llama-3.1-405b, claude-4-5-sonnet, gemma-3-4b, gemini-2.0-flash, phi-4, mistral-large, and many more.
• Broad topics. The most common categories are science and technology, education, business and finance, travel, politics and government, health, and law.

Because the arena has run continuously since October 2024, the dataset also captures how the field moved: newer model generations enter the pool over time, against a stable population of French judges.

What the votes look like

Of the 209,824 turns that were voted on:

Two things stand out. First, decisive preferences (a_better or b_better) split almost exactly evenly: 34.1% versus 34.2%. Since which model is shown on the left is random, that near-perfect balance is a good sign that position bias is small in how the arena is presented. Second, nearly a third of votes are both_good, a reminder that on everyday French prompts many models are already hard to separate, and that the interesting signal lives in the decisive third.

The remaining 431,748 turns were shown but not voted on. Rather than drop them, we keep them: they are a large pool of paired French model responses useful for supervised fine-tuning, distillation, and analysis, even without a label.

What you can build with it

• French reward models and RLHF. 143,428 decisive pairwise judgments in French, with full conversation context.
• Evaluation and arena methodology. Elo-style rankings, agreement studies, position-bias and length-bias analysis on a real, non-synthetic population.
• Sociolinguistic and preference research. How French speakers rate tone, correctness, and helpfulness, across topics from cooking to administrative law.
• SFT, DPO, etc. Hundreds of thousands of paired responses from 119 models to the same French prompts.

This is the first arena, not the last

comparia-fr-arena is the opening release in a wider effort. French is well-resourced compared to most of the world's languages, and general-purpose prompts are only one slice of how people actually use these models. We are building further arenas aimed at less-resourced languages and at specific professional sectors, where preference data barely exists today and where a generic English-trained model is least likely to match local expectations. If this dataset is useful to you, those are next.

If you want to boot your own arena, check out our Github repo, the arena is open-source : https://github.com/betagouv/ComparIA

Limitations and responsible use

• Self-selected users. Arena participants are people who chose to try compar:IA. They are not a representative sample of French speakers, and preferences reflect that population.
• French skew by design. The non-French turns are a minority and should not be treated as balanced multilingual coverage.
• Privacy. PII detection was run over the data. Because prompts are written by real users, treat free text with care and report anything that slipped through.
• Safety content retained. Some toxic or sensitive content is kept deliberately, so the data can be used for safety and robustness research. Filter it out for other uses.
• Empty and idk votes. An empty choice means the turn was not voted on, not that the models tied. idk and both_bad are distinct signals; do not merge them silently.

License and citation

The dataset is released under Etalab Open License 2.0 and CC-BY-4.0. Attribution goes to compar:IA and the Ministère de la Culture.

@misc{comparia_fr_arena_2026,
  title        = {comparia-fr-arena: A French Human-Preference Arena Dataset},
  author       = {{compar:IA, Minist\`ere de la Culture}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/datasets/ministere-culture/comparia-fr-arena}},
  note         = {Etalab 2.0 and CC-BY-4.0}
}

Explore the data in the dataset viewer, try the arena at compar:IA, and tell us what you build with it.