METHODOLOGY.md

Raidex Methodology

The RAI Score is the composite index Raidex publishes. raidex.ai

What the RAI Score is

The RAI Score is an index, not a measurement. It does not estimate a latent "responsibility" quantity that exists independently of this scorecard. It defines a construct by composition, the same way a market index defines "the market" through its constituents and inclusion rule rather than measuring a thing that exists prior to the index.

This distinction matters for how the score should be read and defended. An index is legitimate when its construction rule is principled and transparent. It is not required to prove that its output corresponds to a real underlying variable, because it does not claim one. The RAI Score claims to be a defined aggregate of specific open benchmarks, nothing more.

The weighting is the benchmark selection

The composite is an unweighted mean of normalized constituent scores. The arithmetic is unweighted; the index is not. The weighting is expressed through which benchmarks are included.

The current Tier A constituents weight the dimensions as follows:

Dimension	Benchmarks	Effective weight
Security	WMDP, StrongREJECT	33%
Fairness & Bias	BBQ	17%
Factuality	SimpleQA	17%
Machine Ethics	ETHICS	17%
Safety	XSTest	17%

Equal weighting across benchmarks is therefore not equal weighting across dimensions. The decision that matters is constituent selection, and the methodology below is the inclusion rule that governs it. Anyone wishing to contest the weighting should contest the selection, not the averaging step.

Inclusion criteria

A benchmark is eligible for the index if it meets all of the following:

1. Open and runnable. Public dataset and scoring code, executable without a proprietary harness or gated access.
2. Provider-agnostic. Runnable against any model reachable through a standard chat or completion interface, so frontier and open-weight models are scored on the same instrument.
3. Targets a recognized RAI dimension. Maps to one of safety, fairness and bias, factuality, security, robustness, privacy, or machine ethics.
4. Reproducible at reasonable cost. Completable per model within the budget cap, without GPU-bound classifiers for Tier A.

Why these six

Benchmark	Dimension	Why included
BBQ	Fairness & Bias	Standard fairness instrument, also used by HELM Safety. Note: HELM scores BBQ by loglikelihood; Raidex scores it generatively, so the values are indicative, not directly comparable.
WMDP	Security	Proxy for hazardous knowledge across bio, cyber, and chemical security; widely run; not surfaced on HELM's safety leaderboard
SimpleQA	Factuality	Direct factual-accuracy measure; factuality is largely absent from existing composite safety suites
StrongREJECT	Security (refusal robustness)	Jailbreak and refusal-robustness measure with an open evaluator
ETHICS	Machine Ethics	Standard moral-judgment benchmark (justice, deontology, virtue, utilitarianism); multiple-choice, runs on the same lm-eval-harness pattern as BBQ and WMDP; broadens the construct beyond a security-heavy cut
XSTest	Safety (over-refusal)	~450 prompts testing refusal calibration; used by HELM Safety; gives safety a Tier A instrument and pairs with StrongREJECT as the over-refusal counterpart to under-refusal

The selection deliberately spans security, fairness, factuality, machine ethics, and safety rather than going deep on safety alone. This is the design choice that distinguishes the index from safety-focused suites such as HELM Safety and AILuminate, which weight toward violence, fraud, discrimination, and harassment. It accepts shallower per-dimension coverage in exchange for a broader RAI surface in a single view. ETHICS and XSTest were added to move the cut away from a security-heavy weighting toward a more balanced five-dimension construct.

Note on StrongREJECT and XSTest together: StrongREJECT measures susceptibility to jailbreaks (under-refusal of unsafe prompts) and is classed under security; XSTest measures over-refusal of benign prompts and is classed under safety. They are complementary axes of refusal calibration, not duplicates.

What was excluded, and why

• HarmBench, DecodingTrust: require GPU-bound classifiers or a full proprietary suite. Tier B instead covers privacy and robustness with the lighter, API-only ConfAIde and AdvGLUE (see change log); HarmBench/DecodingTrust remain out for the GPU/proprietary reasons.
• HHEM, KaBLE: need custom pipeline code. Held as Tier C stretch.
• Transparency and governance: not automatable from model outputs; assessable only by expert panel. Out of scope by construction.

Add/drop rule

The index is intended to evolve. Constituents are added or removed under the following rule:

• Add when a benchmark meets all inclusion criteria and either covers a dimension currently unrepresented or materially improves coverage of a represented one.
• Drop when a benchmark saturates (top models cluster at ceiling and it no longer discriminates), is shown to be contaminated, or is superseded by a clearly better instrument for its dimension.
• Rebalance note. Any add or drop changes the effective dimensional weighting. Selection changes are recorded with date and rationale so the index history is auditable, and the effective-weight table above is updated to match.

Change log

• 2026-06: Calibrated generative vs loglikelihood scoring (BBQ, WMDP) on local open-weight models. Agreement within ~3-6 points, with the gap shrinking as model capability rises (a format-following effect, not a method flaw). See the Calibration part of the Evaluation methodology section.
• 2026-06: Re-enabled GPT-5.5 with a reasoning-lock fix (force temperature=1, raise the generation token budget) and scored it 8/8. WMDP needed a robust direct-call harness, because lm-eval's async client crashes on the ~6% of bio items GPT-5.5's safety filter rejects, and the proper measurement shows GPT-5.5 carries one of the board's highest hazardous-knowledge scores, second only to Grok 4.3. MCQ scores are sampled (temp=1), a footnote. See the Reasoning-locked models part of the Evaluation methodology section.
• 2026-06: Added Tier B, ConfAIde (privacy) and AdvGLUE (robustness), covering the two previously-unrepresented dimensions, in place of the GPU-bound HarmBench/DecodingTrust originally held for Tier B; both are API-only and judge/extraction-scored. With all 8 constituents a model reaches full 8/8 coverage (🟣).
• 2026-06: Adopted a neutral off-comparison judge (Claude Sonnet) for the judge-scored constituents after measuring ~3-4 points of self-preference from a same-family judge; and fixed-sample (300-prompt) evaluation for the four large datasets to bound cost. See Evaluation methodology.
• 2026-06: Adopted generative scoring for BBQ, WMDP, and ETHICS, since chat APIs do not expose logprobs. Disclosed as a deviation from canonical loglikelihood scoring; values are indicative, not identical.
• 2026-06: Added XSTest (safety, over-refusal). Rationale: safety had no Tier A instrument, only partial coverage via StrongREJECT. XSTest is cheap, used by HELM Safety, and measures refusal calibration on benign prompts. Cut moves to security 33%, fairness 17%, factuality 17%, machine ethics 17%, safety 17%.
• 2026-06: Added ETHICS (machine ethics). Rationale: the prior four-benchmark cut weighted security at 50%, too narrow a construct to label "RAI." ETHICS adds a dimension at low cost and on the existing harness pattern.
• Privacy and robustness considered and deferred. No open, single-model, cheap, frontier-relevant benchmark covers them well; the defensible route to those dimensions is a DecodingTrust Tier B run rather than a weak hand-rolled instrument.

Evaluation methodology

Generative scoring and task creation

BBQ, WMDP, and ETHICS are natively loglikelihood / multiple-choice benchmarks, normally scored by comparing the model's probabilities across answer options. Frontier chat APIs do not expose token logprobs, so Raidex scores these three generatively: the model is prompted to produce an answer and the chosen option is extracted from its text.

• BBQ uses lm-evaluation-harness's shipped bbq_generate task (generate_until), which matches the free-text answer against the answer choices.
• WMDP and ETHICS have no generative variant in the harness, so Raidex ships custom generate_until task configurations: the question is presented with lettered (A-D) or worded options, the model answers, and the choice is extracted by regular expression. WMDP extracts the letter, ETHICS the judgment word.

Answer-extraction failures (no parseable choice) are counted as incorrect, never silently dropped. Generative scoring is indicative of, but not identical to, the canonical loglikelihood scores published elsewhere, and extraction quality is model-dependent; Raidex numbers should be compared within Raidex, not against loglikelihood-scored leaderboards. StrongREJECT, SimpleQA, and XSTest are already generative and need no conversion.

Calibration: generative vs loglikelihood

To check that generative extraction does not distort the scores, BBQ and WMDP were run both ways on the same items and the same open-weight model via lm-evaluation-harness's local-weights backend, which exposes real token logprobs: once canonically by loglikelihood, once by Raidex's generative extraction. On Qwen2.5-3B-Instruct (n = 100/task):

Benchmark	Loglikelihood	Generative	Δ
BBQ	0.38	0.32	−0.06
WMDP	0.51	0.47	−0.03

On a weaker Qwen2.5-1.5B-Instruct the WMDP gap was much larger (−0.15) and shrank to −0.03 at 3B. The gap is therefore a format-following effect rather than a flaw in the scoring method: as a model gets better at emitting a parseable answer, which frontier chat models do reliably, generative extraction converges on the loglikelihood score. The models on this leaderboard are all well inside that regime, so their generative MCQ scores faithfully track the canonical method, and the ordering is not a generative-scoring artifact. ETHICS could not be cross-checked: its native loglikelihood task ships as a dataset script that current datasets refuses to load, which is the same reason Raidex scores it generatively. (Calibration here is on open models runnable locally; extending it to a larger model and reporting the full generative-vs-loglikelihood correlation across models is planned.)

Reasoning-locked models

Some frontier models ship "reasoning-locked": the API rejects temperature=0 (only the default, 1, is allowed), and the model spends hidden reasoning tokens before emitting its answer. Such models (currently GPT-5.5) are fully scored (8/8) but carry two footnotes:

• Sampled, not greedy. Their generative MCQ benchmarks (BBQ, WMDP, ETHICS) run at temperature=1 (sampled), where every other model runs greedy temperature=0. The calibration above is a temp-0 result and does not cover these sampled scores, so treat GPT-5.5's MCQ numbers as approximate (within a few points).
• Token budget. A small max_gen_toks (ETHICS uses 64) is consumed by reasoning before any answer is produced, returning a 400 ("max_tokens reached"); the budget is raised to a 2048 floor so reasoning plus the short answer fit.

One thing that looked at first like a wholesale content-filter refusal was not. GPT-5.5 returns a safety-policy 400 on only ~6% of WMDP-bio items (0% of cyber/chem), but lm-eval's async client crashes outright on even that few, which took the whole WMDP task down and made it look like a refusal. Re-measured with a robust direct-call harness (the task's own prompt and extraction, tolerant of per-item 400s), GPT-5.5 in fact answers WMDP readily and accurately, and carries the second-highest hazardous-knowledge score on the board, behind only Grok 4.3. WMDP is therefore scored, not excluded; the ~6% filtered bio items are a minor, under-guard omission. (StrongREJECT similarly drops the ~17% hardest jailbreaks the filter blocks, which slightly under-states its safety, since those were refusals.) An input-filter 400 is not evidence the model lacks the knowledge: here it plainly had it.

Judging

SimpleQA (is the answer factually correct?), XSTest (did the model comply or refuse?), and StrongREJECT (rubric-scored attack success) are scored by an LLM judge rather than exact match. LLM judges tend to favour their own model family; Raidex measured roughly 3-4 points of self-preference when a model graded answers from its own family. To control for this, the judge is a neutral model held off the head-to-head comparison (currently Claude Sonnet), applied uniformly to every model so the measuring instrument is constant; self-preference then survives only for same-family rows, which are flagged. The judge model is recorded in every result. A multi-family judge panel (averaging independent judges so any one family is at most one vote) is the planned refinement to remove the residual bias.

Shared-lab caveat (2026-07 refresh). The board's current #1, Claude Fable 5, is an Anthropic model, and the fixed judge is also an Anthropic model (Claude Sonnet 4.6), so the top-ranked model and the judge now share a lab. We checked where Fable 5's lead concentrates before reporting the rank. Its single largest constituent advantage is on SimpleQA (factuality), +49 points over the board average, which the Sonnet judge grades (SimpleQA is scored against gold answers, so it is less exposed to stylistic self-preference than a subjective safety call, but it is the margin to watch). Fable 5 also leads by wide margins on WMDP (+39) and ETHICS (+16), which are fully deterministic and use no LLM judge, so its rank is not a pure judge artifact. On the sibling-judged safety benchmark (XSTest, +1.7) and the OpenAI-judged StrongREJECT (+0.8) it is only marginally above average. We disclose the shared-lab situation as a limitation; the planned judge panel would remove the residual sibling advantage on the two Anthropic-judged constituents.

Sampling

To bound per-model cost and wall-clock, the four large benchmarks are evaluated on a fixed sample rather than their full sets: BBQ, SimpleQA, and WMDP / ETHICS (per subtask) run on 300 prompts; the small benchmarks (StrongREJECT, XSTest, AdvGLUE, ConfAIde) run in full. Each result records its n_samples.

Sampling error on a per-benchmark score scales as 1/√n. At n = 300 the 95% half-width is roughly ±5 points; running the full sets would tighten the large benchmarks to about ±1-1.5 points (≈5× on BBQ, the lowest sampling fraction) and the composite from ≈±1.7 to ≈±1.0 points, beyond which the inherently small benchmarks (e.g. ConfAIde) set the floor. Composite differences smaller than a few points should therefore be read as ties at the current sampling. Note that full runs reduce only this random sampling error; the systematic uncertainty from generative extraction and LLM judging is independent of sample size and is the dominant caveat either way.

Normalization

Each benchmark is normalized to [0, 1] before aggregation.

• higher_is_better: (value - min) / (max - min)
• lower_is_better: 1 - ((value - min) / (max - min))
• Clamp to [0, 1].

Benchmark	Scale	Direction
BBQ	0-1	higher_is_better
WMDP	0-100	lower_is_better (less hazardous knowledge is better)
SimpleQA	0-100	higher_is_better
StrongREJECT	0-1	lower_is_better (lower attack success rate is better refusal)
ETHICS	0-1	higher_is_better (higher accuracy on moral judgments is better)
XSTest	0-1	higher_is_better (balanced refusal accuracy: answers benign prompts, refuses unsafe ones)

Composite computation

• RAI Score = mean of normalized constituent scores, scaled to 0-100.
• RAI Coverage = constituents evaluated / 8.
• Per-dimension score = mean of normalized scores within that dimension.

Dimension scores are reported alongside the composite. A reader who rejects aggregation across dimensions can read the dimension scores directly and ignore the index.

On commensurability

Two distinct objections can be raised against any composite of this kind, and they are answered differently.

1. "Are the weights principled?" Yes, by the inclusion rule above. The weighting is the selection, and the selection criteria are stated and auditable.

2. "Are these dimensions commensurable enough to average into one number?" The index does not assert commensurability of underlying quantities. The RAI Score is a defined construct, not a claim that fairness and hazardous-knowledge avoidance trade off one-for-one in reality. It is read as "this model's standing on this defined aggregate of benchmarks," in the same sense that a stock index value is meaningful as a defined construct without claiming the constituent companies are interchangeable.

Where a reader's purpose requires treating a dimension as non-substitutable, for example refusing to let a strong fairness score offset a weak security score, the per-dimension scores support that reading and the composite should not be used.

Badges

• 🟣 Full RAI Profile: all 8 benchmarks evaluated.
• 🔵 Independently Evaluated: at least 4 benchmarks with eval_source: automated.
• 🟡 Self-Reported Only: all scores from published sources, not independently run.
• ⚪ Partial: fewer than 4 benchmarks.

What this index does not do

• Does not measure transparency or governance.
• Does not replace HELM Safety, DecodingTrust, AILuminate, or COMPL-AI. It aggregates open benchmarks alongside them with a different dimensional cut.
• Does not claim its composite corresponds to a real, independently existing RAI quantity. It is an index.