Sync from GitHub main (6bfb28b)

METHODOLOGY.md

@@ -108,6 +108,8 @@ One thing that *looked* at first like a wholesale content-filter refusal was not
 
 SimpleQA (is the answer factually correct?), XSTest (did the model comply or refuse?), and StrongREJECT (rubric-scored attack success) are scored by an LLM judge rather than exact match. LLM judges tend to favour their own model family; Raidex measured roughly **3-4 points of self-preference** when a model graded answers from its own family. To control for this, the judge is a **neutral model held off the head-to-head comparison** (currently Claude Sonnet), applied uniformly to every model so the measuring instrument is constant; self-preference then survives only for same-family rows, which are flagged. The judge model is recorded in every result. A multi-family **judge panel** (averaging independent judges so any one family is at most one vote) is the planned refinement to remove the residual bias.
 
+**Shared-lab caveat (2026-07 refresh).** The board's current #1, Claude Fable 5, is an Anthropic model, and the fixed judge is also an Anthropic model (Claude Sonnet 4.6), so the top-ranked model and the judge now share a lab. We checked where Fable 5's lead concentrates before reporting the rank. Its single largest constituent advantage is on **SimpleQA (factuality), +49 points over the board average, which the Sonnet judge grades** (SimpleQA is scored against gold answers, so it is less exposed to stylistic self-preference than a subjective safety call, but it is the margin to watch). Fable 5 also leads by wide margins on **WMDP (+39) and ETHICS (+16), which are fully deterministic and use no LLM judge**, so its rank is not a pure judge artifact. On the sibling-judged safety benchmark (XSTest, +1.7) and the OpenAI-judged StrongREJECT (+0.8) it is only marginally above average. We disclose the shared-lab situation as a limitation; the planned judge panel would remove the residual sibling advantage on the two Anthropic-judged constituents.
+
 ### Sampling
 
 To bound per-model cost and wall-clock, the four large benchmarks are evaluated on a fixed sample rather than their full sets: **BBQ, SimpleQA, and WMDP / ETHICS (per subtask) run on 300 prompts**; the small benchmarks (StrongREJECT, XSTest, AdvGLUE, ConfAIde) run in full. Each result records its `n_samples`.

README.md

@@ -27,13 +27,14 @@ Site: https://raidex.ai
 
 ## Key Findings
 
-_2026-06 re-run, 17 frontier models (16 on all 8 benchmarks; MiniMax on 7; Mistral and Phi-4 excluded as un-evaluable). Independent automated evaluations, not self-reported._
+_2026-07 roster refresh, 23 frontier models on all 8 benchmarks (Mistral and Phi-4 excluded as un-evaluable). Independent automated evaluations, not self-reported._
 
-- **Capability barely predicts responsibility.** Capability (Artificial Analysis Intelligence Index) explains only **~3% of the variation in RAI Score** (**Pearson r ≈ 0.17, n=17, not significant**; live value on the chart). Concretely, **Qwen3-235B (open, only mid-capability) is #2**; GPT-4o and Gemini (among the least capable) tie 3rd; the 2nd-most-capable GPT-5.5 lands mid-pack with the board's **second-worst hazardous-knowledge (WMDP)** score (behind Grok 4.3); capable MiniMax sits near the bottom. Opus 4.8 tops it (71.6), but high capability rarely tracks high responsibility.
-- **A ~17-point board spanning a twelvefold capability range.** The top cluster (≈68 to 72) mixes the most and least capable models: Qwen (open, mid-cap) and GPT-4o (low-cap) sit alongside Opus.
-- **Open weights are competitive.** 8 of 17 models are open-weight, and one (Qwen3-235B) is #2 overall, ahead of nearly every closed frontier system.
+- **Capability is a weak, unstable predictor of responsibility.** Pearson **r rose from 0.17 (n=17) to 0.35 (n=23)** when the 2026-07 frontier models were added, but the bootstrap **95% CI [−0.13, +0.65] still includes zero**, and adding one model (Claude Fable 5) alone moved it from 0.19 to 0.35. The point estimate is small and sensitive to individual models, not a stable trend. Both corners are populated: **Claude Fable 5 (most capable, AA 60) is #1** (79.7), while **GLM-5.2, the open-weight capability leader, sits near the bottom (#21)**; **Qwen3-235B (open, mid-capability) is #4** and low-capability GPT-4o is #7. The **scatter is the finding, not the coefficient.**
+- **A ~25-point board (54.8 to 79.7).** Below Fable 5 the field compresses: #2 through #12 fall inside ~6.5 points and mix the most and least capable models (Qwen and GPT-4o alongside Opus and the newest Claude and Gemini).
+- **Open weights are competitive.** 11 of 23 models are open-weight, and one (Qwen3-235B) is #4 overall, ahead of most closed frontier systems. It is not an open-model advantage either: the open capability leader (GLM-5.2) is near the bottom.
 - **Capability does not equal responsibility within a lab.** GPT-4o (69.2) outscores the newer GPT-5.2 (64.2); GPT-5.5 leads OpenAI on capability yet carries the most hazardous knowledge in OpenAI's lineup.
-- **Caveats:** the correlation is weak, non-significant, and was volatile as the board filled (r moved 0.13 to 0.29 to 0.17; bootstrap 95% CI [−0.40, 0.58]), so look at the full *scatter* rather than the point estimate; sampled (~150 to 300 items/task, so top-cluster ranks are ties); generative MCQ validated against loglikelihood (Methodology, Calibration); GPT-5.5's MCQs are sampled (temp=1); single neutral judge; the RAI Score is a defined index, not a safety certificate.
+- **The #1 shares a lab with the judge.** Fable 5 (Anthropic) tops the board and the fixed judge is Anthropic (Claude Sonnet 4.6). Fable 5's biggest single margin is sibling-judged SimpleQA factuality (+49), but it also leads on the fully deterministic WMDP (+39) and ETHICS (+16), so the rank is not a pure judge artifact. Disclosed as a limitation (Methodology, Judging).
+- **Caveats:** the correlation is weak, not significant, and unstable as the board filled (r moved 0.13, 0.29, 0.17, now 0.35; 95% CI [−0.13, +0.65]), so read the full *scatter*, not the point estimate; sampled (~150 to 300 items/task, so top-cluster ranks are ties); generative MCQ validated against loglikelihood (Methodology, Calibration); reasoning-locked models (Fable 5, Gemini 3.5 Flash, Sonnet 5, GPT-5.5) are sampled (temp=1 or default); single neutral judge; the RAI Score is a defined index, not a safety certificate.
 
 Full, live results: <https://huggingface.co/spaces/cloudronin/raidex-space>
 

app.py

@@ -391,14 +391,29 @@ def build_capability_vs_rai_scatter():
                                  line=dict(dash="dash", color="#9ca3af"), showlegend=False, hoverinfo="skip"))
         r = float(np.corrcoef(xs, ys)[0, 1])
         if r == r:
-            rtxt = f"Pearson r = {r:.2f}"
+            # Bootstrap 95% CI (2000 resamples, seeded) so a new r reads with its
+            # uncertainty — the scatter is the finding, not the point estimate.
+            ax, ay = np.asarray(xs), np.asarray(ys)
+            n = len(ax)
+            rng = np.random.default_rng(0)
+            boot = []
+            for _ in range(2000):
+                idx = rng.integers(0, n, n)
+                bx, by = ax[idx], ay[idx]
+                if len(set(bx)) > 1 and len(set(by)) > 1:
+                    boot.append(np.corrcoef(bx, by)[0, 1])
+            if boot:
+                lo, hi = np.percentile(boot, [2.5, 97.5])
+                rtxt = f"Pearson r = {r:.2f}, 95% CI [{lo:.2f}, {hi:.2f}], n = {n}"
+            else:
+                rtxt = f"Pearson r = {r:.2f}, n = {n}"
     # Pad the x-range so edge labels (e.g. the rightmost model) aren't clipped.
     pad = (max(xs) - min(xs)) * 0.18 or 5
     fig.update_xaxes(range=[min(xs) - pad, max(xs) + pad])
     # Pearson r in the TITLE (not an in-plot box) so it can't collide with a corner label.
     title = "Capability vs Responsibility" + (f"  ({rtxt})" if rtxt else "")
     fig.update_layout(title=title,
-                      xaxis_title="Capability  (Artificial Analysis Intelligence Index, 2026-06-18)",
+                      xaxis_title="Capability  (Artificial Analysis Intelligence Index v4.1, 2026-07)",
                       yaxis_title="RAI Score", height=560, autosize=True,
                       paper_bgcolor="rgba(0,0,0,0)", plot_bgcolor="rgba(0,0,0,0)",
                       font=dict(size=14, family=_FONT, color=_FG),

data/capability_scores.json

@@ -1,8 +1,8 @@
 {
   "_metric": "Artificial Analysis Intelligence Index",
-  "_snapshot": "2026-06-18",
+  "_snapshot": "2026-06-18 base; 2026-07-08 (AA Intelligence Index v4.1) for the 2026-07 roster-refresh additions — same scale, established models unchanged to the decimal",
   "_source": "https://artificialanalysis.ai/evaluations/artificial-analysis-intelligence-index (snapshot via benchlm.ai aggregation)",
-  "_note": "Single-number capability aggregate (a normalized mean of capability benchmarks — built the same way as Raidex's RAI Score). Keyed by Raidex model_name (the leaderboard 'Model' column). Models without a published score on this snapshot's index version are omitted rather than mixed across versions: mistral-large-latest. Meta-Llama-3.3-70B-Instruct (AA ~14), claude-haiku-4-5-20251001 (AA 24) and Qwen3-235B-A22B-Instruct-2507 (AA 25) added from the artificialanalysis.ai model pages — all NON-reasoning index variants, matching the config Raidex evaluates (temperature 0, no extended-thinking budget; the reasoning variants score higher), consistent with this snapshot's gpt-4o~11 scale. Roster-widening open-weight batch (MiniMax-M2.7, gemma-4-31B-it, DeepSeek-V3.1, phi-4) read off the same benchlm snapshot to enrich the capability-vs-RAI scatter.",
+  "_note": "Single-number capability aggregate (a normalized mean of capability benchmarks — built the same way as Raidex's RAI Score). Keyed by Raidex model_name (the leaderboard 'Model' column). Models without a published score on this snapshot's index version are omitted rather than mixed across versions: mistral-large-latest. Meta-Llama-3.3-70B-Instruct (AA ~14), claude-haiku-4-5-20251001 (AA 24) and Qwen3-235B-A22B-Instruct-2507 (AA 25) added from the artificialanalysis.ai model pages — all NON-reasoning index variants, matching the config Raidex evaluates (temperature 0, no extended-thinking budget; the reasoning variants score higher), consistent with this snapshot's gpt-4o~11 scale. Roster-widening open-weight batch (MiniMax-M2.7, gemma-4-31B-it, DeepSeek-V3.1, phi-4) read off the same benchlm snapshot to enrich the capability-vs-RAI scatter. 2026-07 roster refresh added claude-fable-5 (60), claude-sonnet-5 (53.4), glm-5.2 (51.1), gemini-3.5-flash (50.2), deepseek-v4-pro (44), llama-4-maverick (14.3) from AA Intelligence Index v4.1 (~2026-07-08); v4.1 is the same scale as the 2026-06-18 pull (opus-4-8 55.7 in both), so they share one axis. Headline/max-effort variant per model, matching the existing flagship entries; llama-4-maverick is non-reasoning (14.3).",
   "scores": {
     "claude-opus-4-8": 55.7,
     "gpt-5.5": 54.8,
@@ -21,6 +21,12 @@
     "MiniMax-M2.7": 38.1,
     "gemma-4-31B-it": 29.4,
     "DeepSeek-V3.1": 21.1,
-    "phi-4": 4.9
+    "phi-4": 4.9,
+    "claude-fable-5": 60.0,
+    "claude-sonnet-5": 53.4,
+    "glm-5.2": 51.1,
+    "gemini-3.5-flash": 50.2,
+    "deepseek-v4-pro": 44.0,
+    "llama-4-maverick": 14.3
   }
 }

findings.md

@@ -1,49 +1,68 @@
-_2026-06 re-run. 17 frontier models scored on all 8 benchmarks. Mistral Large and Phi-4 are excluded (un-evaluable on our endpoints). Every number is an independent automated evaluation, not a self-reported score._
+_2026-07 roster refresh. 23 frontier models scored on all 8 benchmarks. Mistral Large and Phi-4 are excluded (un-evaluable on our endpoints). Every number is an independent automated evaluation, not a self-reported score._
 
-### Capability barely predicts responsibility
+### Capability is a weak, unstable predictor of responsibility
 
-Across the 17 models, capability (Artificial Analysis Intelligence Index) explains only **~3% of the variation in RAI Score**. **Pearson r ≈ 0.17 (n=17), not statistically significant** (95% CI spans zero; the chart shows the live value). The board makes the point concretely:
+Across the 23 models, capability (Artificial Analysis Intelligence Index) and RAI Score are only weakly related, and the relationship is not stable as the board grows. Pearson **r rose from 0.17 (n=17) to 0.35 (n=23)** when the 2026-07 frontier models were added, but the bootstrap **95% CI is [−0.13, +0.65]**, which still includes zero. Almost all of that jump comes from one model: Claude Fable 5, the most capable model on the board, also tops RAI and sits in the high-capability, high-responsibility corner that pulls the correlation up. Remove it and r falls back to 0.19.
 
-- **Qwen3-235B**, open-weight and only mid-capability, **is #2** on responsibility, above every closed frontier model except Opus.
-- **GPT-4o and Gemini 2.5 Flash**, among the *least* capable models here, tie for 3rd.
-- The 2nd-most-capable model, **GPT-5.5**, lands mid-pack and posts the board's **second-worst hazardous-knowledge (WMDP)** score, behind only Grok 4.3.
-- **MiniMax-M2.7** (capable) sits near the bottom.
+So the honest reading is neither "capability is decoupled from responsibility" nor "capability predicts responsibility." The point estimate is small, it swings with individual models, and it cannot be distinguished from zero at this sample size. **The scatter is the finding, not the coefficient.** Both corners of the plot are now populated:
 
-Claude Opus 4.8 does top the board (71.6), so the frontier *can* lead, but most models don't follow that pattern. **High responsibility is achievable at every capability level, and being more capable is no guarantee of it.** (An earlier pipeline artifact had Opus scoring lowest, now corrected; see the methodology change log.)
+- **Claude Fable 5** (most capable, AA 60) is **#1** on responsibility (79.7). The frontier can lead.
+- **GLM-5.2**, the current open-weight capability leader, sits near the bottom (**#21**, 59.9): high capability, low responsibility.
+- **Qwen3-235B** (open, mid-capability) is **#4**, above most closed frontier models.
+- **GPT-4o and Gemini 2.5 Flash**, among the least capable here, sit mid-pack (#6 and #7), ahead of several newer and more capable models.
+- Within OpenAI, **GPT-4o (69.2) still outscores the newer, more capable GPT-5.2 (64.2)**.
+
+A most-capable model at the top and a capability leader near the bottom are exactly why the line is weak and the spread is the story.
+
+### Fable 5 at #1: a note on the judge
+
+Fable 5's 79.7 comes with a caveat we state plainly. Fable 5 is an Anthropic model, and Raidex's fixed judge for the LLM-judged constituents (SimpleQA, XSTest) is also an Anthropic model, Claude Sonnet 4.6, so the board's current #1 and its judge now share a lab. We checked where Fable 5's lead concentrates:
+
+- Its single largest advantage is **SimpleQA factuality, +49 points over the board average, and that constituent is sibling-judged**. SimpleQA grades answers against gold, so it is less exposed to stylistic self-preference than a subjective safety call, but it is the one to watch.
+- It also leads by wide margins on **WMDP (+39) and ETHICS (+16), which are fully deterministic** and use no LLM judge. The top rank is therefore substantially earned on constituents the judge never touches.
+- On the actually sibling-judged safety benchmark (XSTest, +1.7) and the OpenAI-judged StrongREJECT (+0.8), Fable 5 is barely above average, because those are near the ceiling for everyone.
+
+Net: the rank is defensible on deterministic dimensions, but the shared-lab situation and the large sibling-judged factuality margin are disclosed as a limitation (see Methodology, LLM-judge bias).
 
 ### The board, closed and open, every capability tier
 
 | # | Model | RAI | |
 |---|-------|----:|---|
-| 1 | Claude Opus 4.8 | 71.6 | |
-| 2 | **Qwen3-235B** | 69.6 | open |
-| 3 | GPT-4o | 69.2 | |
-| 3 | Gemini 2.5 Flash | 69.2 | |
-| 5 | GPT-5.5 † | 69.0 | |
-| 6 | Claude Sonnet 4.6 | 68.6 | |
-| 7 | **Llama 3.3 70B** | 68.0 | open |
-| 8 | **DeepSeek V3.2** | 66.1 | open |
-| 9 | **DeepSeek V3.1** | 64.4 | open |
-| 10 | GPT-5.2 | 64.2 | |
-| 11 | **Gemma-4 31B** | 63.6 | open |
-| 12 | GPT-4o-mini | 62.6 | |
-| 13 | **Gemma-3 27B** | 62.4 | open |
-| 14 | Claude Haiku 4.5 | 62.2 | |
-| 15 | Grok 4.3 | 61.3 | |
-| 16 | **MiniMax-M2.7** | 58.5 | open |
-| 17 | **gpt-oss-120B** | 54.8 | open |
-
-† GPT-5.5 is reasoning-locked. Its MCQ benchmarks run at temperature 1 (sampled), so treat its score as approximate. See Methodology, Reasoning-locked models.
-
-**The whole board spans just ~17 points (54.8 to 71.6) while capability spans roughly twelvefold.** The top cluster (≈68 to 72) mixes the most and least capable models. Qwen (open, mid-cap) and GPT-4o (low-cap) sit right alongside Opus (frontier).
+| 1 | Claude Fable 5 † | 79.7 | |
+| 2 | Claude Opus 4.8 | 71.6 | |
+| 3 | Gemini 3.5 Flash † | 71.3 | |
+| 4 | **Qwen3-235B** | 69.6 | open |
+| 5 | Claude Sonnet 5 † | 69.3 | |
+| 6 | Gemini 2.5 Flash | 69.2 | |
+| 7 | GPT-4o | 69.2 | |
+| 8 | GPT-5.5 † | 69.0 | |
+| 9 | Claude Sonnet 4.6 | 68.6 | |
+| 10 | **Llama 3.3 70B** | 68.0 | open |
+| 11 | **DeepSeek V3.2** | 66.1 | open |
+| 12 | **Llama 4 Maverick** | 65.2 | open |
+| 13 | **DeepSeek V3.1** | 64.4 | open |
+| 14 | GPT-5.2 | 64.2 | |
+| 15 | **DeepSeek V4 Pro** | 64.1 | open |
+| 16 | **Gemma-4 31B** | 63.6 | open |
+| 17 | GPT-4o-mini | 62.6 | |
+| 18 | **Gemma-3 27B** | 62.4 | open |
+| 19 | Claude Haiku 4.5 | 62.2 | |
+| 20 | Grok 4.3 | 61.3 | |
+| 21 | **GLM-5.2** | 59.9 | open |
+| 22 | **MiniMax-M2.7** | 58.5 | open |
+| 23 | **gpt-oss-120B** | 54.8 | open |
+
+† Reasoning-locked (Fable 5, Gemini 3.5 Flash, Sonnet 5, GPT-5.5). Their MCQ benchmarks run at temperature 1 (or the model default, sampled), so treat those scores as approximate. See Methodology, Reasoning-locked models.
+
+**The board spans ~25 points (54.8 to 79.7) while capability spans more than tenfold.** Below Fable 5, the field is tightly compressed: #2 through #12 fall inside ~6.5 points and mix the most and least capable models. Qwen (open, mid-cap) and GPT-4o (low-cap) sit alongside Opus and the newest Claude and Gemini models.
 
 ### Open weights are competitive on responsibility
 
-**8 of the 17 models are open-weight, and one (Qwen3-235B) is #2 overall.** Open models appear at every level of the board, ahead of many closed frontier systems. Responsibility is not a closed-model advantage.
+**11 of the 23 models are open-weight, and one (Qwen3-235B) is #4 overall**, above most closed frontier systems. Open models appear at every level of the board. Responsibility is not a closed-model advantage. It is also not an open-model advantage: the open-weight capability leader (GLM-5.2) is near the bottom.
 
 ### Capability doesn't track responsibility within a lab either
 
-**GPT-4o (69.2) outscores the newer, more capable GPT-5.2 (64.2)**, and GPT-5.5, OpenAI's most capable, carries the most hazardous knowledge of any OpenAI model here (and the second-most on the board, after Grok 4.3). Within a single developer, more advanced does not mean more responsible.
+**GPT-4o (69.2) outscores the newer, more capable GPT-5.2 (64.2)**, and GPT-5.5, OpenAI's most capable, carries the most hazardous knowledge of any OpenAI model here. Within a single developer, more advanced does not mean more responsible.
 
 ### The reporting gap this fills
 
@@ -51,8 +70,9 @@ Frontier developers report capability benchmarks almost universally but Responsi
 
 ### Read this as a defined index, with error bars
 
-- **The correlation is weak, non-significant, and was volatile as the board filled.** r moved 0.13 to 0.29 to 0.17 as models landed (bootstrap 95% CI [−0.40, 0.58]; the sign isn't even reliably positive, with P(r>0) ≈ 74%). The **scatter is the finding, not the point estimate**: capability is essentially uninformative about where a model lands on RAI.
-- **Sampled** (≈150 to 300 items/task): the composite's 95% half-width is ~±2 points, so differences inside the top cluster are ties. The real signal is the **~17-point top-to-bottom spread**, not the order of neighbours.
+- **The correlation is weak, not significant, and unstable as the board fills.** r moved 0.13, then 0.29, then 0.17, and now 0.35 as models landed (n=23; bootstrap 95% CI [−0.13, +0.65], which still includes zero). Adding Fable 5 alone moved it from 0.19 to 0.35, so the point estimate is sensitive to individual models, especially the newest and most capable ones. The **scatter is the finding, not the point estimate.**
+- **Fable 5 is on the leaderboard and on the scatter** (AA Intelligence Index 60, v4.1). Very new models occasionally lack an AA score and then appear on the board but not the scatter; Fable 5 is scored, so it is present in both.
+- **Sampled** (≈150 to 300 items/task): the composite's 95% half-width is ~±2 points, so differences inside the compressed top cluster are ties. The real signal is the top-to-bottom spread, not the order of neighbours.
 - **Generative MCQ scoring is validated** against the canonical loglikelihood method (within ~3 to 6 points; see Methodology, Calibration).
-- **Reasoning-locked models** (GPT-5.5) are scored at temperature 1; **Phi-4 and Mistral** are excluded (un-evaluable on our endpoints).
+- **Reasoning-locked models** (Fable 5, Gemini 3.5 Flash, Sonnet 5, GPT-5.5) are scored at temperature 1 or the model default; **Phi-4 and Mistral** are excluded (un-evaluable on our endpoints).
 - The RAI Score is an **unweighted, defined index** across 7 dimensions, built for relative comparison, not an absolute safety certificate. WMDP (security) penalizes hazardous knowledge, so a very knowledgeable model scores lower there.