Spaces:
Running
Running
| title: Raidex | |
| emoji: 📊 | |
| colorFrom: blue | |
| colorTo: indigo | |
| sdk: gradio | |
| sdk_version: 5.49.1 | |
| app_file: app.py | |
| pinned: false | |
| license: mit | |
| short_description: An open Responsible AI index for frontier models | |
| tags: | |
| - leaderboard | |
| - responsible-ai | |
| - ai-safety | |
| - benchmarks | |
| - frontier-models | |
| # Raidex | |
| An open Responsible AI index for frontier foundation models. Benchmarks across | |
| safety, fairness, factuality, security, robustness, privacy, and ethics. | |
| Live leaderboard: https://huggingface.co/spaces/cloudronin/raidex-space | |
| Site: https://raidex.ai | |
| ## Key Findings | |
| _2026-07 roster refresh, 23 frontier models on all 8 benchmarks (Mistral and Phi-4 excluded as un-evaluable). Independent automated evaluations, not self-reported._ | |
| - **Capability is a weak, unstable predictor of responsibility.** Pearson **r rose from 0.17 (n=17) to 0.35 (n=23)** when the 2026-07 frontier models were added, but the bootstrap **95% CI [−0.13, +0.65] still includes zero**, and adding one model (Claude Fable 5) alone moved it from 0.19 to 0.35. The point estimate is small and sensitive to individual models, not a stable trend. Both corners are populated: **Claude Fable 5 (most capable, AA 60) is #1** (79.7), while **GLM-5.2, the open-weight capability leader, sits near the bottom (#21)**; **Qwen3-235B (open, mid-capability) is #4** and low-capability GPT-4o is #7. The **scatter is the finding, not the coefficient.** | |
| - **A ~25-point board (54.8 to 79.7).** Below Fable 5 the field compresses: #2 through #12 fall inside ~6.5 points and mix the most and least capable models (Qwen and GPT-4o alongside Opus and the newest Claude and Gemini). | |
| - **Open weights are competitive.** 11 of 23 models are open-weight, and one (Qwen3-235B) is #4 overall, ahead of most closed frontier systems. It is not an open-model advantage either: the open capability leader (GLM-5.2) is near the bottom. | |
| - **Capability does not equal responsibility within a lab.** GPT-4o (69.2) outscores the newer GPT-5.2 (64.2); GPT-5.5 leads OpenAI on capability yet carries the most hazardous knowledge in OpenAI's lineup. | |
| - **The #1 shares a lab with the judge.** Fable 5 (Anthropic) tops the board and the fixed judge is Anthropic (Claude Sonnet 4.6). Fable 5's biggest single margin is sibling-judged SimpleQA factuality (+49), but it also leads on the fully deterministic WMDP (+39) and ETHICS (+16), so the rank is not a pure judge artifact. Disclosed as a limitation (Methodology, Judging). | |
| - **Caveats:** the correlation is weak, not significant, and unstable as the board filled (r moved 0.13, 0.29, 0.17, now 0.35; 95% CI [−0.13, +0.65]), so read the full *scatter*, not the point estimate; sampled (~150 to 300 items/task, so top-cluster ranks are ties); generative MCQ validated against loglikelihood (Methodology, Calibration); reasoning-locked models (Fable 5, Gemini 3.5 Flash, Sonnet 5, GPT-5.5) are sampled (temp=1 or default); single neutral judge; the RAI Score is a defined index, not a safety certificate. | |
| Full, live results: <https://huggingface.co/spaces/cloudronin/raidex-space> | |
| _The findings are generated from independent automated evaluations, not | |
| self-reported scores from model developers._ | |
| ## Why this exists | |
| The 2026 Stanford AI Index documents a reporting gap: frontier models report | |
| capability benchmarks consistently, but RAI benchmark reporting is sparse. Every | |
| benchmark in this set is open-source and runnable. | |
| This is not the first composite RAI evaluation. HELM Safety, COMPL-AI, MLCommons | |
| AILuminate, and the FLI AI Safety Index publish composite scores. Raidex adds an | |
| open, submit-driven leaderboard that aggregates these specific open benchmarks and | |
| shows the capability-vs-RAI reporting gap side by side. | |
| ## How it works | |
| Submit a model, the backend runs 6 RAI benchmarks automatically, and scores appear | |
| on the leaderboard. | |
| ## Benchmarks (Tier A, automated) | |
| | Benchmark | Dimension | Pipeline | Cost/Model | | |
| |-----------|-----------|----------|------------| | |
| | BBQ | Fairness & Bias | lm-eval-harness (generative) | ~$10 | | |
| | WMDP | Security | lm-eval-harness (generative) | ~$8 | | |
| | SimpleQA | Factuality | litellm + judge (F1) | ~$30 | | |
| | StrongREJECT | Security (refusal) | strong_reject rubric | ~$5 | | |
| | ETHICS | Machine Ethics | lm-eval-harness (generative) | ~$5 | | |
| | XSTest | Safety (over-refusal) | litellm + judge | ~$4 | | |
| **Order-of-magnitude: ~$62/model. Confirm with `--dry-run`.** BBQ/WMDP/ETHICS are | |
| scored generatively (chat APIs don't expose logprobs); see METHODOLOGY.md. | |
| ## Badges | |
| - 🟣 **Full RAI Profile.** All 8 benchmarks (Tier A + B + C) | |
| - 🔵 **Independently Evaluated.** ≥4 benchmarks run by our automated pipeline | |
| - 🟡 **Self-Reported Only.** Scores from system cards / published leaderboards | |
| - ⚪ **Partial.** Fewer than 4 benchmarks | |
| ## Composite Score | |
| **RAI Score** = mean of normalized benchmark scores (0-100). | |
| **RAI Coverage** = benchmarks evaluated / 8. | |
| ## Prior art | |
| DecodingTrust, COMPL-AI, HELM Safety, MLCommons AILuminate, FLI AI Safety | |
| Index, and DeepSight (2026). Raidex aggregates across independent open benchmarks with | |
| a public submit pipeline, alongside these efforts. | |
| ## License | |
| MIT | |