README.md

metadata

title: Raidex
emoji: 📊
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: 5.49.1
app_file: app.py
pinned: false
license: mit
short_description: An open Responsible AI index for frontier models
tags:
  - leaderboard
  - responsible-ai
  - ai-safety
  - benchmarks
  - frontier-models

Raidex

An open Responsible AI index for frontier foundation models. Benchmarks across safety, fairness, factuality, security, robustness, privacy, and ethics.

Live leaderboard: https://huggingface.co/spaces/cloudronin/raidex-space Site: https://raidex.ai

Key Findings

2026-07 roster refresh, 23 frontier models on all 8 benchmarks (Mistral and Phi-4 excluded as un-evaluable). Independent automated evaluations, not self-reported.

• Capability is a weak, unstable predictor of responsibility. Pearson r rose from 0.17 (n=17) to 0.35 (n=23) when the 2026-07 frontier models were added, but the bootstrap 95% CI [−0.13, +0.65] still includes zero, and adding one model (Claude Fable 5) alone moved it from 0.19 to 0.35. The point estimate is small and sensitive to individual models, not a stable trend. Both corners are populated: Claude Fable 5 (most capable, AA 60) is #1 (79.7), while GLM-5.2, the open-weight capability leader, sits near the bottom (#21); Qwen3-235B (open, mid-capability) is #4 and low-capability GPT-4o is #7. The scatter is the finding, not the coefficient.
• A ~25-point board (54.8 to 79.7). Below Fable 5 the field compresses: #2 through #12 fall inside ~6.5 points and mix the most and least capable models (Qwen and GPT-4o alongside Opus and the newest Claude and Gemini).
• Open weights are competitive. 11 of 23 models are open-weight, and one (Qwen3-235B) is #4 overall, ahead of most closed frontier systems. It is not an open-model advantage either: the open capability leader (GLM-5.2) is near the bottom.
• Capability does not equal responsibility within a lab. GPT-4o (69.2) outscores the newer GPT-5.2 (64.2); GPT-5.5 leads OpenAI on capability yet carries the most hazardous knowledge in OpenAI's lineup.
• The #1 shares a lab with the judge. Fable 5 (Anthropic) tops the board and the fixed judge is Anthropic (Claude Sonnet 4.6). Fable 5's biggest single margin is sibling-judged SimpleQA factuality (+49), but it also leads on the fully deterministic WMDP (+39) and ETHICS (+16), so the rank is not a pure judge artifact. Disclosed as a limitation (Methodology, Judging).
• Caveats: the correlation is weak, not significant, and unstable as the board filled (r moved 0.13, 0.29, 0.17, now 0.35; 95% CI [−0.13, +0.65]), so read the full scatter, not the point estimate; sampled (~150 to 300 items/task, so top-cluster ranks are ties); generative MCQ validated against loglikelihood (Methodology, Calibration); reasoning-locked models (Fable 5, Gemini 3.5 Flash, Sonnet 5, GPT-5.5) are sampled (temp=1 or default); single neutral judge; the RAI Score is a defined index, not a safety certificate.

Full, live results: https://huggingface.co/spaces/cloudronin/raidex-space

The findings are generated from independent automated evaluations, not self-reported scores from model developers.

Why this exists

The 2026 Stanford AI Index documents a reporting gap: frontier models report capability benchmarks consistently, but RAI benchmark reporting is sparse. Every benchmark in this set is open-source and runnable.

This is not the first composite RAI evaluation. HELM Safety, COMPL-AI, MLCommons AILuminate, and the FLI AI Safety Index publish composite scores. Raidex adds an open, submit-driven leaderboard that aggregates these specific open benchmarks and shows the capability-vs-RAI reporting gap side by side.

How it works

Submit a model, the backend runs 6 RAI benchmarks automatically, and scores appear on the leaderboard.

Benchmarks (Tier A, automated)

Benchmark	Dimension	Pipeline	Cost/Model
BBQ	Fairness & Bias	lm-eval-harness (generative)	~$10
WMDP	Security	lm-eval-harness (generative)	~$8
SimpleQA	Factuality	litellm + judge (F1)	~$30
StrongREJECT	Security (refusal)	strong_reject rubric	~$5
ETHICS	Machine Ethics	lm-eval-harness (generative)	~$5
XSTest	Safety (over-refusal)	litellm + judge	~$4

Order-of-magnitude: ~$62/model. Confirm with --dry-run. BBQ/WMDP/ETHICS are scored generatively (chat APIs don't expose logprobs); see METHODOLOGY.md.

Badges

• 🟣 Full RAI Profile. All 8 benchmarks (Tier A + B + C)
• 🔵 Independently Evaluated. ≥4 benchmarks run by our automated pipeline
• 🟡 Self-Reported Only. Scores from system cards / published leaderboards
• ⚪ Partial. Fewer than 4 benchmarks

Composite Score

RAI Score = mean of normalized benchmark scores (0-100). RAI Coverage = benchmarks evaluated / 8.

Prior art

DecodingTrust, COMPL-AI, HELM Safety, MLCommons AILuminate, FLI AI Safety Index, and DeepSight (2026). Raidex aggregates across independent open benchmarks with a public submit pipeline, alongside these efforts.

License

MIT