Anka commited on
Commit
dd102db
·
1 Parent(s): f4fe09a

Re-formatted tutorials, default reported metrics to Overlaps

Browse files
content/tutorials/cross-post-to-hugging-face.md CHANGED
@@ -1,6 +1,6 @@
1
  # Cross-post Every Eval Ever results to Hugging Face
2
 
3
- *Send an Every Eval Ever (EEE) record to Hugging Face Community Evals so your score shows up on the model page with a backlink to the full structured record.*
4
 
5
  > **Audience:** contributors who already report evaluation results to [Every Eval Ever](https://github.com/evaleval/every_eval_ever). This is the more technical companion to the stakeholder guides.
6
 
@@ -8,11 +8,11 @@
8
 
9
  ## Why this exists
10
 
11
- Every Eval Ever (EEE) launched in February 2026 as a project of the EvalEval Coalition a cross-institutional effort to fix how AI evaluation results get reported. Eval results matter for capability measurement, safety, and governance, yet they are scattered and hard to compare: they live in papers, leaderboards, blog posts, and harness logs, each in its own format, and the same model on the same benchmark often returns different numbers depending on who ran it and how.
12
 
13
- LLaMA 65B, for example, has been reported at both **63.7** and **48.8** on MMLU. That gap came down to the evaluation harness, not the model and usually you can't even see a gap like that, because the metadata that would explain it isn't recorded.
14
 
15
- EEE is the fix for the reporting side: **one JSON schema** for an evaluation result that records who ran it, which model, how it was accessed, the generation settings, and what the metric actually means with an optional companion file for per-sample outputs. It takes in results from any source, so harness logs, leaderboard scrapes, and paper numbers all end up in the same shape. Since launching, the datastore on Hugging Face has grown to around **229,000 evaluation results** across more than **22,000 models** and **2,200 benchmarks**, pulled from **31** different reporting formats. Reproducing those runs from scratch would cost somewhere in the hundreds of thousands of dollars a reasonable argument for not letting the data scatter once someone has paid to generate it.
16
 
17
  ## What cross-posting does
18
 
@@ -21,13 +21,13 @@ You can now send your EEE results to **Hugging Face Community Evals** for offici
21
  Once you share, two things happen:
22
 
23
  - Your score **appears on the Hugging Face model page** and is pulled into the benchmark's leaderboard.
24
- - It carries a **source badge that links back to the full EEE record** where the generation config, harness version, reproducibility notes, and any instance-level data live.
25
 
26
- The two destinations do different jobs toward the same goal. Hugging Face puts your result where people look at models, with a link back to the source. EEE keeps the full structured record that makes the result interpretable and powers Evaluation Cards on top of it. Send your data to both and the same evaluation ends up visible and legible at once, which is the point of reporting one at all.
27
 
28
  ## How it works
29
 
30
- Hugging Face stores eval scores in the model repo, as a YAML file under `.eval_results/`. The required fields are just the benchmark dataset, the task, and the value. The `source` block is optional and it's the part that creates the backlink to EEE.
31
 
32
  ```yaml
33
  - dataset:
@@ -56,10 +56,10 @@ The converter fills this in from your existing record. It maps:
56
 
57
  If you already contribute to EEE, this is one extra step and the converter handles most of it:
58
 
59
- 1. Submit your full record to the EEE datastore the usual way the [GitHub repo](https://github.com/evaleval/every_eval_ever) has the contributor guide and converters.
60
  2. Run the converter to generate the Hugging Face YAML.
61
  3. Open the PR on the model's **Community** tab.
62
 
63
  ---
64
 
65
- *Adapted from "Cross-post your Every Eval Ever results onto Hugging Face model pages" Netaluru Harsha, Nathan Habib, Avijit Ghosh.*
 
1
  # Cross-post Every Eval Ever results to Hugging Face
2
 
3
+ *Send an Every Eval Ever (EEE) record to Hugging Face Community Evals so your score shows up on the model page, with a backlink to the full structured record.*
4
 
5
  > **Audience:** contributors who already report evaluation results to [Every Eval Ever](https://github.com/evaleval/every_eval_ever). This is the more technical companion to the stakeholder guides.
6
 
 
8
 
9
  ## Why this exists
10
 
11
+ Every Eval Ever (EEE) launched in February 2026 as a project of the EvalEval Coalition, a cross-institutional effort to fix how AI evaluation results get reported. Eval results matter for capability measurement, safety, and governance, yet they are scattered and hard to compare: they live in papers, leaderboards, blog posts, and harness logs, each in its own format, and the same model on the same benchmark often returns different numbers depending on who ran it and how.
12
 
13
+ LLaMA 65B, for example, has been reported at both **63.7** and **48.8** on MMLU. That gap came down to the evaluation harness, not the model, and usually you can't even see a gap like that, because the metadata that would explain it isn't recorded.
14
 
15
+ EEE is the fix for the reporting side: **one JSON schema** for an evaluation result that records who ran it, which model, how it was accessed, the generation settings, and what the metric actually means, with an optional companion file for per-sample outputs. It takes in results from any source, so harness logs, leaderboard scrapes, and paper numbers all end up in the same shape. Since launching, the datastore on Hugging Face has grown to around **229,000 evaluation results** across more than **22,000 models** and **2,200 benchmarks**, pulled from **31** different reporting formats. Reproducing those runs from scratch would cost somewhere in the hundreds of thousands of dollars, a reasonable argument for not letting the data scatter once someone has paid to generate it.
16
 
17
  ## What cross-posting does
18
 
 
21
  Once you share, two things happen:
22
 
23
  - Your score **appears on the Hugging Face model page** and is pulled into the benchmark's leaderboard.
24
+ - It carries a **source badge that links back to the full EEE record**, where the generation config, harness version, reproducibility notes, and any instance-level data live.
25
 
26
+ The two destinations do different jobs toward the same goal. Hugging Face puts your result where people look at models, with a link back to the source. EEE keeps the full structured record that makes the result interpretable, and powers Evaluation Cards on top of it. Send your data to both and the same evaluation ends up visible and legible at once, which is the point of reporting one at all.
27
 
28
  ## How it works
29
 
30
+ Hugging Face stores eval scores in the model repo, as a YAML file under `.eval_results/`. The required fields are just the benchmark dataset, the task, and the value. The `source` block is optional, and it's the part that creates the backlink to EEE.
31
 
32
  ```yaml
33
  - dataset:
 
56
 
57
  If you already contribute to EEE, this is one extra step and the converter handles most of it:
58
 
59
+ 1. Submit your full record to the EEE datastore the usual way; the [GitHub repo](https://github.com/evaleval/every_eval_ever) has the contributor guide and converters.
60
  2. Run the converter to generate the Hugging Face YAML.
61
  3. Open the PR on the model's **Community** tab.
62
 
63
  ---
64
 
65
+ *Adapted from "Cross-post your Every Eval Ever results onto Hugging Face model pages," by Netaluru Harsha, Nathan Habib, and Avijit Ghosh.*
content/tutorials/evaluation-researchers.md CHANGED
@@ -17,7 +17,7 @@ The design principles that matter most for research use:
17
  - **Evaluator identity is preserved.** First-party and third-party results are kept distinct.
18
  - **Snapshot discipline.** No retroactive edits; corrections are versioned. Your analyses are reproducible against a dated snapshot.
19
 
20
- Some of our next steps are aimed squarely at research use including making **all evaluation data, metadata plus run-level results, downloadable** directly from Evaluation Cards. In the meantime, run-level data is available from [Every Eval Ever](https://github.com/evaleval/every_eval_ever), the structured datastore that powers the corpus. If there's a feature that would help your work, tell us on the [public roadmap](https://changemap.co/evaleval/evalcards/) — we'd love to build what the research community needs, so let us know.
21
 
22
  ---
23
 
@@ -26,7 +26,7 @@ Some of our next steps are aimed squarely at research use — including making *
26
  The model page has two modes, toggled top-right. Use **Summary View** to get oriented, and **Researcher View** to do the actual work.
27
 
28
  > 🖼️ **Screenshot — `19-card-researcher-full.png`**
29
- > *What to capture:* A model page (e.g. `/models/anthropic/claude-opus-4.7`) after clicking **Researcher View** show the expanded per-result detail.
30
 
31
  > 🖼️ **Screenshot — `20-card-researcher-top.png`**
32
  > *What to capture:* The top of the Researcher View (the view toggle + the documentation summary line).
@@ -46,7 +46,7 @@ Model name, developer, release date, modalities (e.g. *image, text → text*), s
46
  > *What to capture:* The §1 Identification block of a model page.
47
 
48
  ### §2 Benchmark coverage
49
- A benchmark-first view grouped by category, surfacing **setup spread** and **split-vs-setup** differences up front. Watch the "split spread" note (e.g. *"2 benchmarks include multiple splits or setups"*) these are exactly the cases where naïve aggregation hides variance.
50
 
51
  > 🖼️ **Screenshot — `16-card-coverage.png`**
52
  > *What to capture:* The §2 Benchmark coverage section.
@@ -57,22 +57,22 @@ A first-party / third-party split as a donut plus per-category bars. This is the
57
  > 🖼️ **Screenshot — `17-card-who-reports.png`**
58
  > *What to capture:* The §3 "Who reports what" section.
59
 
60
- For research: a category that is **entirely first-party** is a known blind spot there is no independent corroboration of those numbers.
61
 
62
  ### §4 Reported metrics
63
- This is where you read off the actual numbers. In Researcher View the section has a grouping toggle **Overlaps**, **Category**, and **Source** plus a family selector, a search box, and a grid/list switch. Each grouping answers a different question.
64
 
65
- **Overlaps** cross-suite overlaps: benchmarks this model reports under more than one suite, with the **mean and 95% CI** across appearances and the **range** of reported scores. The fastest way to find where the *same* benchmark was run by different reporters, and how far the numbers move.
66
 
67
  > 🖼️ **Screenshot — `24-research-overlaps.png`**
68
  > *What to capture:* §4 in **Overlaps** grouping (per-benchmark N, mean / CI, range, sources).
69
 
70
- **Category** every reported result regrouped under curated category tags, so similar benchmarks cluster across families. Read the model's profile by capability area (math, safety, agentic, …) rather than by benchmark name.
71
 
72
  > 🖼️ **Screenshot — `25-research-category.png`**
73
  > *What to capture:* §4 in **Category** grouping.
74
 
75
- **Source** the warehouse's natural shape: family-rooted plots and accordions with no cross-family collapse. This is the full result set; drill into any plot for the per-result setup context. (The section opens in Overlaps when a model has any; switch here for everything.)
76
 
77
  > 🖼️ **Screenshot — `26-research-source.png`**
78
  > *What to capture:* §4 in **Source** grouping.
@@ -86,7 +86,7 @@ This is where you read off the actual numbers. In Researcher View the section ha
86
  | **Reproducibility** | Disclosure of setup variants, prompts, decoding params, harness version, seeds, code availability. | "Reproducible in principle" ≠ "reproduced." Only ~3% of scores have complete setup documentation corpus-wide. |
87
  | **Completeness** | Coverage across capability / robustness / safety / fairness for the model class. | A complete *capability* record can still be silent on safety. Completeness is relative to expectations, not absolute. |
88
  | **Provenance & Risk** | First- vs third-party; mapping to **IBM Risk Atlas** risk domains. | First-party numbers are not wrong, but they are not independent. Check §3 before ranking. |
89
- | **Comparability** | Split, metric variant, and unit differences within the same benchmark. | Two "MMLU" numbers may use different splits or metrics flagged here. Do not rank across incomparable setups. |
90
 
91
  ---
92
 
@@ -109,7 +109,7 @@ This is where you read off the actual numbers. In Researcher View the section ha
109
 
110
  ## Limitations
111
 
112
- 1. **Comparability is bounded by setup.** The site flags when two scores under the same benchmark are not directly comparable respect those flags in any ranking or meta-analysis.
113
  2. **Coverage ≠ quality.** Many results with weak documentation can look like "good coverage." Weight by the signals.
114
  3. **The corpus reflects what was reported, not all that exists.** Undisclosed evaluations are absent by definition.
115
  4. **Pin a snapshot.** Cite `ec/models/...` identifiers plus the snapshot date so your analysis is reproducible.
 
17
  - **Evaluator identity is preserved.** First-party and third-party results are kept distinct.
18
  - **Snapshot discipline.** No retroactive edits; corrections are versioned. Your analyses are reproducible against a dated snapshot.
19
 
20
+ Some of our next steps are aimed squarely at research use, including making **all evaluation data, metadata plus run-level results, downloadable** directly from Evaluation Cards. In the meantime, run-level data is available from [Every Eval Ever](https://github.com/evaleval/every_eval_ever), the structured datastore that powers the corpus. If there's a feature that would help your work, tell us on the [public roadmap](https://changemap.co/evaleval/evalcards/). We'd love to build what the research community needs, so let us know.
21
 
22
  ---
23
 
 
26
  The model page has two modes, toggled top-right. Use **Summary View** to get oriented, and **Researcher View** to do the actual work.
27
 
28
  > 🖼️ **Screenshot — `19-card-researcher-full.png`**
29
+ > *What to capture:* A model page (e.g. `/models/anthropic/claude-opus-4.7`) after clicking **Researcher View**, showing the expanded per-result detail.
30
 
31
  > 🖼️ **Screenshot — `20-card-researcher-top.png`**
32
  > *What to capture:* The top of the Researcher View (the view toggle + the documentation summary line).
 
46
  > *What to capture:* The §1 Identification block of a model page.
47
 
48
  ### §2 Benchmark coverage
49
+ A benchmark-first view grouped by category, surfacing **setup spread** and **split-vs-setup** differences up front. Watch the "split spread" note (e.g. *"2 benchmarks include multiple splits or setups"*). These are exactly the cases where naïve aggregation hides variance.
50
 
51
  > 🖼️ **Screenshot — `16-card-coverage.png`**
52
  > *What to capture:* The §2 Benchmark coverage section.
 
57
  > 🖼️ **Screenshot — `17-card-who-reports.png`**
58
  > *What to capture:* The §3 "Who reports what" section.
59
 
60
+ For research: a category that is **entirely first-party** is a known blind spot: there is no independent corroboration of those numbers.
61
 
62
  ### §4 Reported metrics
63
+ This is where you read off the actual numbers. In Researcher View the section has a grouping toggle for **Overlaps**, **Category**, and **Source**, plus a family selector, a search box, and a grid/list switch. Each grouping answers a different question.
64
 
65
+ **Overlaps** shows cross-suite overlaps: benchmarks this model reports under more than one suite, with the **mean and 95% CI** across appearances and the **range** of reported scores. The fastest way to find where the *same* benchmark was run by different reporters, and how far the numbers move.
66
 
67
  > 🖼️ **Screenshot — `24-research-overlaps.png`**
68
  > *What to capture:* §4 in **Overlaps** grouping (per-benchmark N, mean / CI, range, sources).
69
 
70
+ **Category** regroups every reported result under curated category tags, so similar benchmarks cluster across families. Read the model's profile by capability area (math, safety, agentic, …) rather than by benchmark name.
71
 
72
  > 🖼️ **Screenshot — `25-research-category.png`**
73
  > *What to capture:* §4 in **Category** grouping.
74
 
75
+ **Source** is the warehouse's natural shape: family-rooted plots and accordions with no cross-family collapse. This is the full result set; drill into any plot for the per-result setup context. (The section opens in Overlaps when a model has any; switch here for everything.)
76
 
77
  > 🖼️ **Screenshot — `26-research-source.png`**
78
  > *What to capture:* §4 in **Source** grouping.
 
86
  | **Reproducibility** | Disclosure of setup variants, prompts, decoding params, harness version, seeds, code availability. | "Reproducible in principle" ≠ "reproduced." Only ~3% of scores have complete setup documentation corpus-wide. |
87
  | **Completeness** | Coverage across capability / robustness / safety / fairness for the model class. | A complete *capability* record can still be silent on safety. Completeness is relative to expectations, not absolute. |
88
  | **Provenance & Risk** | First- vs third-party; mapping to **IBM Risk Atlas** risk domains. | First-party numbers are not wrong, but they are not independent. Check §3 before ranking. |
89
+ | **Comparability** | Split, metric variant, and unit differences within the same benchmark. | Two "MMLU" numbers may use different splits or metrics, flagged here. Do not rank across incomparable setups. |
90
 
91
  ---
92
 
 
109
 
110
  ## Limitations
111
 
112
+ 1. **Comparability is bounded by setup.** The site flags when two scores under the same benchmark are not directly comparable, so respect those flags in any ranking or meta-analysis.
113
  2. **Coverage ≠ quality.** Many results with weak documentation can look like "good coverage." Weight by the signals.
114
  3. **The corpus reflects what was reported, not all that exists.** Undisclosed evaluations are absent by definition.
115
  4. **Pin a snapshot.** Cite `ec/models/...` identifiers plus the snapshot date so your analysis is reproducible.
content/tutorials/general-public.md CHANGED
@@ -1,4 +1,4 @@
1
- # Evaluation Cards A Guide for Everyone
2
 
3
  *Plain-language: what an AI "benchmark score" really means, and how to read these cards.*
4
 
@@ -10,7 +10,7 @@ You don't need a technical background for this guide. If you've seen headlines l
10
 
11
  **Evaluation Cards** is like a nutrition label for AI model test results.
12
 
13
- When companies build AI models, they run them through standardized tests called **benchmarks** (think: a standardized exam for AI). They then publish scores. But a score by itself "92%" leaves out a lot: Who gave the test? Under what conditions? Can anyone else get the same result? Were important topics even tested?
14
 
15
  Evaluation Cards collects these scores from across the industry and also shows you **what's missing**, so a polished number doesn't get mistaken for the full story.
16
 
@@ -34,7 +34,7 @@ These are exactly the questions Evaluation Cards helps you ask about AI models.
34
 
35
  ## The four things to check (the "signals")
36
 
37
- Every result is rated on four simple ideas. You don't need the math just the gist.
38
 
39
  > 🖼️ **Screenshot — `03-home-signals.png`**
40
  > *What to capture:* The four signal cards on the homepage.
@@ -42,8 +42,8 @@ Every result is rated on four simple ideas. You don't need the math — just the
42
  | Signal | In plain words |
43
  |---|---|
44
  | 🔁 **Reproducibility** | Could someone else repeat this test and get the same result? If the details are secret, the answer is "we can't tell." |
45
- | 📋 **Completeness** | Did they test the things that matter not just the flattering stuff, but also safety and fairness? |
46
- | 👤 **Provenance** | Who ran the test the **company that made the model**, or an **independent group**? |
47
  | ⚖️ **Comparability** | Can you fairly compare this model's score to another model's? (Sometimes it's apples vs. oranges.) |
48
 
49
  **The big one for everyday readers: Provenance.** A company grading its own homework isn't *wrong*, but it isn't the same as an independent referee. The site always shows you which is which.
@@ -54,22 +54,22 @@ Every result is rated on four simple ideas. You don't need the math — just the
54
 
55
  Let's walk through one.
56
 
57
- **Step 1 Find a model.** Click **Models** in the menu and pick one you've heard of.
58
 
59
  > 🖼️ **Screenshot — `09-models-index.png`**
60
  > *What to capture:* The Models list page.
61
 
62
- **Step 2 Open its page.** Each model has its own page.
63
 
64
  > 🖼️ **Screenshot — `13-card-summary-full.png`**
65
  > *What to capture:* A full model page in **Summary View**.
66
 
67
- **Step 3 Look for the "Documented" number.** Near the top you'll see something like **"36% documented."** Higher means more of the test details were shared. A low number isn't a scandal it's just telling you *"take these scores with a grain of salt; a lot wasn't disclosed."*
68
 
69
  > 🖼️ **Screenshot — `14-card-summary-top.png`**
70
  > *What to capture:* The top of a model page with the DOCUMENTED percentage.
71
 
72
- **Step 4 See who did the testing.** Scroll to **"Who reports what."** A simple chart splits results into the **company's own** results vs **independent** ones.
73
 
74
  > 🖼️ **Screenshot — `17-card-who-reports.png`**
75
  > *What to capture:* The "Who reports what" chart (company vs. independent).
@@ -80,20 +80,20 @@ That's it. You can stop there and already read these results more wisely than mo
80
 
81
  ## Looking up a test (benchmark)
82
 
83
- The **Evaluations** tab lists the tests (benchmarks) that models are scored on. Open one and the **At a glance** box explains, in plain terms, what the test checks, its main catch, and who it's for so a name like "MMLU" stops being a mystery.
84
 
85
  > 🖼️ **Screenshot — `05-evals-index.png`**
86
  > *What to capture:* The Evaluations list of benchmark families.
87
 
88
  > 🖼️ **Screenshot — `27-eval-detail-card.png`**
89
- > *What to capture:* A benchmark's "At a glance" box what it measures and its main caveat.
90
 
91
  ---
92
 
93
  ## What this site is *not*
94
 
95
  - ❌ It's **not** a "best AI" leaderboard. It won't crown a winner.
96
- - ❌ It **won't** tell you a model is "safe" only how thoroughly (and by whom) it was tested.
97
  - ❌ Blanks are **not** zeros. A blank means "nobody reported this," not "the model failed."
98
 
99
  ---
 
1
+ # Evaluation Cards: A Guide for Everyone
2
 
3
  *Plain-language: what an AI "benchmark score" really means, and how to read these cards.*
4
 
 
10
 
11
  **Evaluation Cards** is like a nutrition label for AI model test results.
12
 
13
+ When companies build AI models, they run them through standardized tests called **benchmarks** (think: a standardized exam for AI). They then publish scores. But a score by itself, "92%", leaves out a lot: Who gave the test? Under what conditions? Can anyone else get the same result? Were important topics even tested?
14
 
15
  Evaluation Cards collects these scores from across the industry and also shows you **what's missing**, so a polished number doesn't get mistaken for the full story.
16
 
 
34
 
35
  ## The four things to check (the "signals")
36
 
37
+ Every result is rated on four simple ideas. You don't need the math, just the gist.
38
 
39
  > 🖼️ **Screenshot — `03-home-signals.png`**
40
  > *What to capture:* The four signal cards on the homepage.
 
42
  | Signal | In plain words |
43
  |---|---|
44
  | 🔁 **Reproducibility** | Could someone else repeat this test and get the same result? If the details are secret, the answer is "we can't tell." |
45
+ | 📋 **Completeness** | Did they test the things that matter, not just the flattering stuff but also safety and fairness? |
46
+ | 👤 **Provenance** | Who ran the test: the **company that made the model**, or an **independent group**? |
47
  | ⚖️ **Comparability** | Can you fairly compare this model's score to another model's? (Sometimes it's apples vs. oranges.) |
48
 
49
  **The big one for everyday readers: Provenance.** A company grading its own homework isn't *wrong*, but it isn't the same as an independent referee. The site always shows you which is which.
 
54
 
55
  Let's walk through one.
56
 
57
+ **Step 1: Find a model.** Click **Models** in the menu and pick one you've heard of.
58
 
59
  > 🖼️ **Screenshot — `09-models-index.png`**
60
  > *What to capture:* The Models list page.
61
 
62
+ **Step 2: Open its page.** Each model has its own page.
63
 
64
  > 🖼️ **Screenshot — `13-card-summary-full.png`**
65
  > *What to capture:* A full model page in **Summary View**.
66
 
67
+ **Step 3: Look for the "Documented" number.** Near the top you'll see something like **"36% documented."** Higher means more of the test details were shared. A low number isn't a scandal; it's just telling you *"take these scores with a grain of salt; a lot wasn't disclosed."*
68
 
69
  > 🖼️ **Screenshot — `14-card-summary-top.png`**
70
  > *What to capture:* The top of a model page with the DOCUMENTED percentage.
71
 
72
+ **Step 4: See who did the testing.** Scroll to **"Who reports what."** A simple chart splits results into the **company's own** results vs **independent** ones.
73
 
74
  > 🖼️ **Screenshot — `17-card-who-reports.png`**
75
  > *What to capture:* The "Who reports what" chart (company vs. independent).
 
80
 
81
  ## Looking up a test (benchmark)
82
 
83
+ The **Evaluations** tab lists the tests (benchmarks) that models are scored on. Open one and the **At a glance** box explains, in plain terms, what the test checks, its main catch, and who it's for, so a name like "MMLU" stops being a mystery.
84
 
85
  > 🖼️ **Screenshot — `05-evals-index.png`**
86
  > *What to capture:* The Evaluations list of benchmark families.
87
 
88
  > 🖼️ **Screenshot — `27-eval-detail-card.png`**
89
+ > *What to capture:* A benchmark's "At a glance" box: what it measures and its main caveat.
90
 
91
  ---
92
 
93
  ## What this site is *not*
94
 
95
  - ❌ It's **not** a "best AI" leaderboard. It won't crown a winner.
96
+ - ❌ It **won't** tell you a model is "safe," only how thoroughly (and by whom) it was tested.
97
  - ❌ Blanks are **not** zeros. A blank means "nobody reported this," not "the model failed."
98
 
99
  ---
content/tutorials/get-verified.md CHANGED
@@ -4,7 +4,7 @@
4
 
5
  ---
6
 
7
- Submit your evaluation data through a Hugging Face account associated with your organisation's HF org, and we'll mark your results with a **verified checkmark** a signal to readers that the numbers come straight from the source.
8
 
9
  We're calling on **all model developers and evaluators** to send their data this way. The more results that arrive verified and in a common format, the more of the corpus can be compared apples to apples.
10
 
 
4
 
5
  ---
6
 
7
+ Submit your evaluation data through a Hugging Face account associated with your organisation's HF org, and we'll mark your results with a **verified checkmark**, a signal to readers that the numbers come straight from the source.
8
 
9
  We're calling on **all model developers and evaluators** to send their data this way. The more results that arrive verified and in a common format, the more of the corpus can be compared apples to apples.
10
 
content/tutorials/how-signals-are-computed.md CHANGED
@@ -10,9 +10,9 @@
10
 
11
  Every signal is computed over a **result triple** $r = (m, b, \mu)$:
12
 
13
- - $m$ a canonical model identifier,
14
- - $b$ a metric-path through the rollout hierarchy (family → composite → benchmark → split),
15
- - $\mu$ a metric.
16
 
17
  $R$ is the set of all such triples in the corpus, and $B$ the set of canonical benchmarks. For any field $f$ and triple $r$:
18
 
@@ -100,7 +100,7 @@ The multi-party indicator is $\mathrm{MP}(m, b, \mu) = 1$ when $|R(m, b, \mu)| >
100
 
101
  ## Comparability
102
 
103
- **In plain terms.** For each $(m, b, \mu)$ triple with at least two reports, we check whether reported scores differ by more than 5% of the metric's range. We do this two ways across setups for the same party (variant divergence) and across different parties (cross-party divergence). Either one triggers a flag.
104
 
105
  Let $[\mu_{\min}, \mu_{\max}]$ be the metric's native scale and $\theta = 0.05$ the divergence threshold.
106
 
@@ -140,6 +140,6 @@ The threshold $\theta = 0.05$ is applied uniformly across metrics; metric-specif
140
 
141
  ## One thing to keep in mind
142
 
143
- None of these signals is a grade. Evaluation Cards assigns no letter grades, pass/fail thresholds, or completeness rankings when a field is omitted, it lowers the completeness score, may trip the reproducibility, provenance, or comparability signals, and is shown to readers directly. The intent is to make reporting choices visible, not to enforce a particular reporting standard.
144
 
145
  *Adapted from [Appendix H.1, "Computation of Interpretive Signals"](https://arxiv.org/abs/2606.09809) of the paper.*
 
10
 
11
  Every signal is computed over a **result triple** $r = (m, b, \mu)$:
12
 
13
+ - $m$: a canonical model identifier,
14
+ - $b$: a metric-path through the rollout hierarchy (family → composite → benchmark → split),
15
+ - $\mu$: a metric.
16
 
17
  $R$ is the set of all such triples in the corpus, and $B$ the set of canonical benchmarks. For any field $f$ and triple $r$:
18
 
 
100
 
101
  ## Comparability
102
 
103
+ **In plain terms.** For each $(m, b, \mu)$ triple with at least two reports, we check whether reported scores differ by more than 5% of the metric's range. We do this two ways: across setups for the same party (variant divergence) and across different parties (cross-party divergence). Either one triggers a flag.
104
 
105
  Let $[\mu_{\min}, \mu_{\max}]$ be the metric's native scale and $\theta = 0.05$ the divergence threshold.
106
 
 
140
 
141
  ## One thing to keep in mind
142
 
143
+ None of these signals is a grade. Evaluation Cards assigns no letter grades, pass/fail thresholds, or completeness rankings. When a field is omitted, it lowers the completeness score, may trip the reproducibility, provenance, or comparability signals, and is shown to readers directly. The intent is to make reporting choices visible, not to enforce a particular reporting standard.
144
 
145
  *Adapted from [Appendix H.1, "Computation of Interpretive Signals"](https://arxiv.org/abs/2606.09809) of the paper.*
content/tutorials/journalists.md CHANGED
@@ -35,7 +35,7 @@ A company claims "Model X leads on benchmark Y." Before you repeat it:
35
  > 🖼️ **Screenshot — `17-card-who-reports.png`**
36
  > *What to capture:* The §3 first-party vs third-party breakdown.
37
 
38
- **3. How documented is it?** The **`DOCUMENTED`** badge (e.g. "36% 14/39 reported") tells you how much of the record is fully specified. A flashy score with weak documentation is a weak claim worth a caveat.
39
 
40
  > 🖼️ **Screenshot — `14-card-summary-top.png`**
41
  > *What to capture:* The page header with the DOCUMENTED badge.
@@ -68,7 +68,7 @@ This lets you write "according to [organization], reported in [source]" rather t
68
  - **Cite the snapshot.** Every page is dated; the corpus is versioned and not retroactively edited. Write: *"per Evaluation Cards, snapshot [date]."*
69
  - **Use the stable identifier.** Each model has an ID like `ec/models/anthropic/claude-opus-4.7`. Link the page directly.
70
  - **Attribute the evaluator,** not just the model: *"a third-party result reported by [org]"* vs *"the developer's own reported result."*
71
- - **Quote the number with its context** the split/metric and whether it's directly comparable.
72
 
73
  ---
74
 
@@ -78,7 +78,7 @@ This lets you write "according to [organization], reported in [source]" rather t
78
  |---|---|
79
  | "Model X is the best/safest." | "Model X reports the highest *self-reported* score on [benchmark]; there are no independent results for this in the corpus." |
80
  | Reporting a blank as a failure. | A blank means *not reported*, not zero. Say "no evaluation was reported." |
81
- | "Model A beats Model B." | Check Comparability first different setups can make the comparison invalid. |
82
  | Citing a launch blog as the source. | Cite the attributed evaluation + reporting organization + snapshot date. |
83
  | Treating one snapshot as permanent. | Numbers change; date your claim. |
84
 
@@ -86,6 +86,6 @@ This lets you write "according to [organization], reported in [source]" rather t
86
 
87
  ## A note on independence
88
 
89
- The most defensible AI-performance reporting distinguishes the developer's own numbers from independent ones. The **§3 "Who reports what"** view makes that distinction in one glance and a category that is *entirely first-party* is exactly where a careful reporter adds "self-reported, not independently verified."
90
 
91
  ➡️ Related: [Policymakers](policymakers.md) (governance framing) · [General public](general-public.md) (plain-language explainer).
 
35
  > 🖼️ **Screenshot — `17-card-who-reports.png`**
36
  > *What to capture:* The §3 first-party vs third-party breakdown.
37
 
38
+ **3. How documented is it?** The **`DOCUMENTED`** badge (e.g. "36%, 14/39 reported") tells you how much of the record is fully specified. A flashy score with weak documentation is a weak claim, worth a caveat.
39
 
40
  > 🖼️ **Screenshot — `14-card-summary-top.png`**
41
  > *What to capture:* The page header with the DOCUMENTED badge.
 
68
  - **Cite the snapshot.** Every page is dated; the corpus is versioned and not retroactively edited. Write: *"per Evaluation Cards, snapshot [date]."*
69
  - **Use the stable identifier.** Each model has an ID like `ec/models/anthropic/claude-opus-4.7`. Link the page directly.
70
  - **Attribute the evaluator,** not just the model: *"a third-party result reported by [org]"* vs *"the developer's own reported result."*
71
+ - **Quote the number with its context:** the split/metric and whether it's directly comparable.
72
 
73
  ---
74
 
 
78
  |---|---|
79
  | "Model X is the best/safest." | "Model X reports the highest *self-reported* score on [benchmark]; there are no independent results for this in the corpus." |
80
  | Reporting a blank as a failure. | A blank means *not reported*, not zero. Say "no evaluation was reported." |
81
+ | "Model A beats Model B." | Check Comparability first; different setups can make the comparison invalid. |
82
  | Citing a launch blog as the source. | Cite the attributed evaluation + reporting organization + snapshot date. |
83
  | Treating one snapshot as permanent. | Numbers change; date your claim. |
84
 
 
86
 
87
  ## A note on independence
88
 
89
+ The most defensible AI-performance reporting distinguishes the developer's own numbers from independent ones. The **§3 "Who reports what"** view makes that distinction in one glance, and a category that is *entirely first-party* is exactly where a careful reporter adds "self-reported, not independently verified."
90
 
91
  ➡️ Related: [Policymakers](policymakers.md) (governance framing) · [General public](general-public.md) (plain-language explainer).
content/tutorials/model-developers.md CHANGED
@@ -17,7 +17,7 @@ A high benchmark number with weak signals reads as a weak claim. Most of those g
17
 
18
  ---
19
 
20
- ## Step 1 Find your model (and your org)
21
 
22
  - **Models view** (`/models`): search for your model, filter by parameter range, and open its page.
23
 
@@ -31,7 +31,7 @@ A high benchmark number with weak signals reads as a weak claim. Most of those g
31
 
32
  ---
33
 
34
- ## Step 2 Read your page honestly
35
 
36
  Open your model's page in **Summary View**, then switch to **Researcher View** to see the per-result detail others will scrutinize.
37
 
@@ -43,41 +43,41 @@ Open your model's page in **Summary View**, then switch to **Researcher View** t
43
 
44
  Three things to look at first:
45
 
46
- 1. **The `DOCUMENTED` badge** (e.g. "36% 14 / 39 reported"). This is the headline read on your reporting hygiene: how much of your reported record is fully specified. Treat a low number as a backlog, not a verdict.
47
 
48
  > 🖼️ **Screenshot — `14-card-summary-top.png`**
49
  > *What to capture:* The page header with the DOCUMENTED badge.
50
 
51
- 2. **§1 Identification** confirm the basics are right: model name, developer, release date, modalities, system ID. Errors here propagate everywhere.
52
 
53
  > 🖼️ **Screenshot — `15-card-identification.png`**
54
  > *What to capture:* The §1 Identification block.
55
 
56
- 3. **§2 Benchmark coverage** what categories are present, and where the **split spread** is. Gaps here are visible to everyone reading your page.
57
 
58
  > 🖼️ **Screenshot — `16-card-coverage.png`**
59
  > *What to capture:* The §2 Benchmark coverage section.
60
 
61
  ---
62
 
63
- ## Step 3 Work the four signals as a reporting checklist
64
 
65
  Each signal maps to concrete, mostly low-cost actions.
66
 
67
- ### 🔁 Reproducibility *make it re-runnable*
68
- Disclose, per result: **setup variants, prompts, decoding parameters, harness name + version, random seeds, and code/artifacts**. Corpus-wide only ~3% of scores have complete setup documentation so even modest disclosure stands out.
69
 
70
  - ✅ Publish the exact harness + version (e.g. the eval framework and commit).
71
  - ✅ State decoding settings (temperature, top-p, max tokens) and seeds.
72
  - ✅ Link runnable code or a config, not just a number.
73
 
74
- ### 📋 Completeness *cover the categories that matter*
75
  Completeness is judged relative to expectations for your model class. A strong **capability** record that is silent on **safety**, **robustness**, or **fairness** reads as incomplete.
76
 
77
  - ✅ Report beyond the flattering capability benchmarks.
78
- - ✅ If a category is intentionally out of scope, the absence will still show consider documenting why.
79
 
80
- ### 👤 Provenance & Risk *invite independent evaluation*
81
  Your page splits results into **first-party** (your own) vs **third-party** (independent), and maps results to **IBM Risk Atlas** risk domains. A category that is **entirely first-party** is a visible blind spot: impressive, but uncorroborated.
82
 
83
  > 🖼️ **Screenshot — `17-card-who-reports.png`**
@@ -86,8 +86,8 @@ Your page splits results into **first-party** (your own) vs **third-party** (ind
86
  - ✅ Encourage / enable third-party evaluation, especially for safety-relevant claims.
87
  - ✅ Don't expect self-reported numbers alone to carry weight with careful readers.
88
 
89
- ### ⚖️ Comparability *report so your scores can be compared*
90
- The site flags when two scores on the same benchmark used different splits, metric variants, or units which invalidates direct comparison.
91
 
92
  > 🖼️ **Screenshot — `18-card-metrics.png`**
93
  > *What to capture:* The §4 Reported metrics section with benchmark charts.
@@ -101,13 +101,13 @@ The site flags when two scores on the same benchmark used different splits, metr
101
 
102
  ---
103
 
104
- ## Step 4 Corrections and snapshots
105
 
106
  The corpus follows **snapshot discipline**: no retroactive edits, only versioned corrections. Practically:
107
 
108
- - A number you published is attributed to its source document and tied to a dated snapshot it won't be silently rewritten.
109
  - If something is genuinely wrong or missing, the path is a **correction in a future snapshot**, not an edit-in-place. Keep your own source documents stable and clearly dated so they can be cited cleanly.
110
- - Because nothing is imputed, the fastest way to improve your page is simply to **report more, and report it precisely** the corpus will reflect it on the next snapshot.
111
 
112
  ---
113
 
@@ -118,10 +118,10 @@ Before your next model announcement, check that each headline benchmark claim sh
118
  - [ ] Harness name + version, and a link to runnable code/config
119
  - [ ] Decoding settings and seeds
120
  - [ ] The exact split and metric (with units), matching how the benchmark is normally reported
121
- - [ ] Coverage beyond capability at least some safety / robustness / fairness results
122
  - [ ] At least one path to independent (third-party) evaluation for key claims
123
  - [ ] A stable, dated source document for every number
124
 
125
- Closing these is what moves your `DOCUMENTED` percentage and turns marketing numbers into claims that hold up.
126
 
127
  ➡️ Related: [Evaluation researchers](evaluation-researchers.md) (how researchers will scrutinize your page) · [Journalists](journalists.md) (how reporters will source your claims) · [Quickstart](quickstart.md).
 
17
 
18
  ---
19
 
20
+ ## Step 1: Find your model (and your org)
21
 
22
  - **Models view** (`/models`): search for your model, filter by parameter range, and open its page.
23
 
 
31
 
32
  ---
33
 
34
+ ## Step 2: Read your page honestly
35
 
36
  Open your model's page in **Summary View**, then switch to **Researcher View** to see the per-result detail others will scrutinize.
37
 
 
43
 
44
  Three things to look at first:
45
 
46
+ 1. **The `DOCUMENTED` badge** (e.g. "36%, 14 / 39 reported"). This is the headline read on your reporting hygiene: how much of your reported record is fully specified. Treat a low number as a backlog, not a verdict.
47
 
48
  > 🖼️ **Screenshot — `14-card-summary-top.png`**
49
  > *What to capture:* The page header with the DOCUMENTED badge.
50
 
51
+ 2. **§1 Identification.** Confirm the basics are right: model name, developer, release date, modalities, system ID. Errors here propagate everywhere.
52
 
53
  > 🖼️ **Screenshot — `15-card-identification.png`**
54
  > *What to capture:* The §1 Identification block.
55
 
56
+ 3. **§2 Benchmark coverage.** What categories are present, and where the **split spread** is. Gaps here are visible to everyone reading your page.
57
 
58
  > 🖼️ **Screenshot — `16-card-coverage.png`**
59
  > *What to capture:* The §2 Benchmark coverage section.
60
 
61
  ---
62
 
63
+ ## Step 3: Work the four signals as a reporting checklist
64
 
65
  Each signal maps to concrete, mostly low-cost actions.
66
 
67
+ ### 🔁 Reproducibility: *make it re-runnable*
68
+ Disclose, per result: **setup variants, prompts, decoding parameters, harness name + version, random seeds, and code/artifacts**. Corpus-wide only ~3% of scores have complete setup documentation, so even modest disclosure stands out.
69
 
70
  - ✅ Publish the exact harness + version (e.g. the eval framework and commit).
71
  - ✅ State decoding settings (temperature, top-p, max tokens) and seeds.
72
  - ✅ Link runnable code or a config, not just a number.
73
 
74
+ ### 📋 Completeness: *cover the categories that matter*
75
  Completeness is judged relative to expectations for your model class. A strong **capability** record that is silent on **safety**, **robustness**, or **fairness** reads as incomplete.
76
 
77
  - ✅ Report beyond the flattering capability benchmarks.
78
+ - ✅ If a category is intentionally out of scope, the absence will still show, so consider documenting why.
79
 
80
+ ### 👤 Provenance & Risk: *invite independent evaluation*
81
  Your page splits results into **first-party** (your own) vs **third-party** (independent), and maps results to **IBM Risk Atlas** risk domains. A category that is **entirely first-party** is a visible blind spot: impressive, but uncorroborated.
82
 
83
  > 🖼️ **Screenshot — `17-card-who-reports.png`**
 
86
  - ✅ Encourage / enable third-party evaluation, especially for safety-relevant claims.
87
  - ✅ Don't expect self-reported numbers alone to carry weight with careful readers.
88
 
89
+ ### ⚖️ Comparability: *report so your scores can be compared*
90
+ The site flags when two scores on the same benchmark used different splits, metric variants, or units, which invalidates direct comparison.
91
 
92
  > 🖼️ **Screenshot — `18-card-metrics.png`**
93
  > *What to capture:* The §4 Reported metrics section with benchmark charts.
 
101
 
102
  ---
103
 
104
+ ## Step 4: Corrections and snapshots
105
 
106
  The corpus follows **snapshot discipline**: no retroactive edits, only versioned corrections. Practically:
107
 
108
+ - A number you published is attributed to its source document and tied to a dated snapshot, so it won't be silently rewritten.
109
  - If something is genuinely wrong or missing, the path is a **correction in a future snapshot**, not an edit-in-place. Keep your own source documents stable and clearly dated so they can be cited cleanly.
110
+ - Because nothing is imputed, the fastest way to improve your page is simply to **report more, and report it precisely**, and the corpus will reflect it on the next snapshot.
111
 
112
  ---
113
 
 
118
  - [ ] Harness name + version, and a link to runnable code/config
119
  - [ ] Decoding settings and seeds
120
  - [ ] The exact split and metric (with units), matching how the benchmark is normally reported
121
+ - [ ] Coverage beyond capability: at least some safety / robustness / fairness results
122
  - [ ] At least one path to independent (third-party) evaluation for key claims
123
  - [ ] A stable, dated source document for every number
124
 
125
+ Closing these is what moves your `DOCUMENTED` percentage, and turns marketing numbers into claims that hold up.
126
 
127
  ➡️ Related: [Evaluation researchers](evaluation-researchers.md) (how researchers will scrutinize your page) · [Journalists](journalists.md) (how reporters will source your claims) · [Quickstart](quickstart.md).
content/tutorials/policymakers.md CHANGED
@@ -1,6 +1,6 @@
1
  # Evaluation Cards for Policymakers
2
 
3
- *How to read AI evaluation evidence for governance decisions what it supports, and what it does not.*
4
 
5
  > **Prerequisite:** the [Quickstart](quickstart.md) explains the four signals and the snapshot model in a few minutes.
6
 
@@ -10,7 +10,7 @@
10
 
11
  When a company says "our model scored X% on a safety benchmark," that is a **claim**, not an audited fact. Evaluation Cards exists to show you the *quality of the evidence* behind such claims: who produced it, whether it can be reproduced, and whether it can be fairly compared to anyone else's number.
12
 
13
- For policy, the headline score is rarely the variable that matters the documentation and independence behind it usually are.
14
 
15
  > 🖼️ **Screenshot — `01-home-overview.png`**
16
  > *What to capture:* The homepage with the corpus snapshot and the four signals.
@@ -21,7 +21,7 @@ For policy, the headline score is rarely the variable that matters — the docum
21
 
22
  **It does:**
23
  - Aggregate how thousands of models have been evaluated, across many independent reporting organizations.
24
- - Make explicit *what was not disclosed* treating undocumented setups as a measurable gap, not an oversight.
25
  - Distinguish a developer's own results from independent third-party evaluations.
26
 
27
  **It does not:**
@@ -49,8 +49,8 @@ For policy, the headline score is rarely the variable that matters — the docum
49
 
50
  The single most useful view for governance is **§3 "Who reports what"** on any model page. It splits a model's results into:
51
 
52
- - **First-party** produced by the model's own developer.
53
- - **Third-party** produced independently.
54
 
55
  > 🖼️ **Screenshot — `17-card-who-reports.png`**
56
  > *What to capture:* The §3 "Who reports what" first-party vs third-party breakdown on a model page.
@@ -67,7 +67,7 @@ Suppose you need to assess the evidence base for a specific model.
67
  > 🖼️ **Screenshot — `09-models-index.png`**
68
  > *What to capture:* The Models index list.
69
 
70
- 2. **Read the `DOCUMENTED` badge** (e.g. "36% 14 / 39 reported"). This is, at a glance, how much of the record is fully documented. Treat a low number as "claims that are hard to verify."
71
 
72
  > 🖼️ **Screenshot — `14-card-summary-top.png`**
73
  > *What to capture:* The top of a model page showing the DOCUMENTED badge and the summary paragraph.
@@ -87,7 +87,7 @@ Suppose you need to assess the evidence base for a specific model.
87
  When a claim cites a specific benchmark ("scores X on a safety benchmark"), the **Evaluations** tab lets you check what that benchmark actually measures before you weigh the number. Each benchmark's **At a glance** card states what it tests, its main caveat, and who it's intended for, links to the source, and flags whether scores on it can be compared directly across reporters.
88
 
89
  > 🖼️ **Screenshot — `05-evals-index.png`**
90
- > *What to capture:* The Evaluations index benchmark families with their categories and counts.
91
 
92
  > 🖼️ **Screenshot — `27-eval-detail-card.png`**
93
  > *What to capture:* A benchmark's "At a glance" card (what it measures, its caveat, and sources).
@@ -108,6 +108,6 @@ When a claim cites a specific benchmark ("scores X on a safety benchmark"), the
108
 
109
  ## Why the framing matters
110
 
111
- Evaluation Cards deliberately treats an undisclosed detail as **a claim intentionally not made** not an error. For policy this is the useful posture: it shifts the question from *"what's the number?"* to *"how much can we trust and verify this number, and who stands behind it?"* which is the question governance actually needs answered.
112
 
113
  ➡️ Related: [Journalists](journalists.md) (sourcing claims responsibly) · [Evaluation researchers](evaluation-researchers.md) (deeper methodology).
 
1
  # Evaluation Cards for Policymakers
2
 
3
+ *How to read AI evaluation evidence for governance decisions: what it supports and what it does not.*
4
 
5
  > **Prerequisite:** the [Quickstart](quickstart.md) explains the four signals and the snapshot model in a few minutes.
6
 
 
10
 
11
  When a company says "our model scored X% on a safety benchmark," that is a **claim**, not an audited fact. Evaluation Cards exists to show you the *quality of the evidence* behind such claims: who produced it, whether it can be reproduced, and whether it can be fairly compared to anyone else's number.
12
 
13
+ For policy, the headline score is rarely the variable that matters. The documentation and independence behind it usually are.
14
 
15
  > 🖼️ **Screenshot — `01-home-overview.png`**
16
  > *What to capture:* The homepage with the corpus snapshot and the four signals.
 
21
 
22
  **It does:**
23
  - Aggregate how thousands of models have been evaluated, across many independent reporting organizations.
24
+ - Make explicit *what was not disclosed*, treating undocumented setups as a measurable gap, not an oversight.
25
  - Distinguish a developer's own results from independent third-party evaluations.
26
 
27
  **It does not:**
 
49
 
50
  The single most useful view for governance is **§3 "Who reports what"** on any model page. It splits a model's results into:
51
 
52
+ - **First-party**: produced by the model's own developer.
53
+ - **Third-party**: produced independently.
54
 
55
  > 🖼️ **Screenshot — `17-card-who-reports.png`**
56
  > *What to capture:* The §3 "Who reports what" first-party vs third-party breakdown on a model page.
 
67
  > 🖼️ **Screenshot — `09-models-index.png`**
68
  > *What to capture:* The Models index list.
69
 
70
+ 2. **Read the `DOCUMENTED` badge** (e.g. "36%, 14 / 39 reported"). This is, at a glance, how much of the record is fully documented. Treat a low number as "claims that are hard to verify."
71
 
72
  > 🖼️ **Screenshot — `14-card-summary-top.png`**
73
  > *What to capture:* The top of a model page showing the DOCUMENTED badge and the summary paragraph.
 
87
  When a claim cites a specific benchmark ("scores X on a safety benchmark"), the **Evaluations** tab lets you check what that benchmark actually measures before you weigh the number. Each benchmark's **At a glance** card states what it tests, its main caveat, and who it's intended for, links to the source, and flags whether scores on it can be compared directly across reporters.
88
 
89
  > 🖼️ **Screenshot — `05-evals-index.png`**
90
+ > *What to capture:* The Evaluations index, with benchmark families and their categories and counts.
91
 
92
  > 🖼️ **Screenshot — `27-eval-detail-card.png`**
93
  > *What to capture:* A benchmark's "At a glance" card (what it measures, its caveat, and sources).
 
108
 
109
  ## Why the framing matters
110
 
111
+ Evaluation Cards deliberately treats an undisclosed detail as **a claim intentionally not made**, not an error. For policy this is the useful posture: it shifts the question from *"what's the number?"* to *"how much can we trust and verify this number, and who stands behind it?"* That is the question governance actually needs answered.
112
 
113
  ➡️ Related: [Journalists](journalists.md) (sourcing claims responsibly) · [Evaluation researchers](evaluation-researchers.md) (deeper methodology).
content/tutorials/quickstart.md CHANGED
@@ -1,11 +1,11 @@
1
- # Evaluation Cards Quickstart
2
 
3
  *A stakeholder-agnostic guide to getting started. ~6 min.*
4
 
5
- > **Heads up Evaluation Cards is in Beta.** We'd love your feedback: report bugs, request features, or tell us what's confusing through our [feedback form](/feedback) or the [public roadmap](https://changemap.co/evaleval/evalcards/). You can reach the feedback form from any page via **Feedback** in the top navigation.
6
 
7
  > 🖼️ **Screenshot — `home-hero-snapshot.png`**
8
- > *What to capture:* The homepage hero and the **Corpus snapshot** the current totals for models, results, organizations, and benchmarks.
9
 
10
  ---
11
 
@@ -13,7 +13,7 @@
13
 
14
  **Evaluation Cards** is a reporting layer over AI model evaluations, built by the [EvalEval Coalition](https://evalcards.evalevalai.com/about). It collects how AI models are evaluated across many benchmarks and reporting organizations, and also shows **what was left undocumented**.
15
 
16
- A benchmark score on its own ("Model X scores 87% on MMLU") tells you very little. Evaluation Cards puts each score in context: *who* ran the evaluation, *how* it was set up, *whether* it can be reproduced, and *whether* it can be fairly compared to another score. The project treats every published evaluation as a **claim**, and every undisclosed detail as **a claim deliberately not made** neither is treated as an error.
17
 
18
  At a glance (snapshot of June 2026), the corpus tracks:
19
 
@@ -23,7 +23,7 @@ At a glance (snapshot of June 2026), the corpus tracks:
23
  | **31** reporting organizations | **820** model developers |
24
  | **57** benchmark families | **632** single benchmarks |
25
 
26
- > The corpus is **versioned by snapshot**. Every page shows a snapshot date. Numbers above will drift as the corpus grows always cite the snapshot you saw.
27
 
28
  ---
29
 
@@ -36,10 +36,10 @@ Every record is assessed against four signals. Learn to read them and most of th
36
 
37
  | Signal | Question it answers | What a low score means |
38
  |---|---|---|
39
- | **R Reproducibility** | Could a third party re-run this evaluation? | Prompts, decoding settings, harness version, seeds, or code are undisclosed. |
40
- | **C Completeness** | Does the record meet normal reporting expectations for this kind of model? | Whole categories (e.g. safety, robustness, fairness) may be missing. |
41
- | **P Provenance & Risk** | Who ran it, and what real-world property does it measure? | Distinguishes **first-party** (the developer) from **third-party** (independent) evaluators. |
42
- | **X Comparability** | Can two scores on the same benchmark be put side by side? | Different splits, metric variants, or units make a direct ranking invalid. |
43
 
44
  A high benchmark score with weak signals is still a weak claim. The signals are how you tell a well-documented result from one you can't check.
45
 
@@ -62,23 +62,23 @@ For example: *MMLU (family) → MMLU-Pro (composite) → a single benchmark →
62
  > 🖼️ **Screenshot — `02-home-hero.png`**
63
  > *What to capture:* The top navigation bar (Overview · Models · Evaluations · Help · About · Feedback).
64
 
65
- 1. **Overview** (`/`) the corpus snapshot, the signals explained, and featured benchmark families. Start here.
66
- 2. **Models** (`/models`) every indexed model. Filter by parameter size, switch between **Models** and **Developers** views, and select up to four models to compare.
67
- 3. **Evaluations** (`/evals`) benchmarks grouped into **families**, filterable by interaction style (agent / non-agent) and ~17 categories (Mathematics, Safety, Software Engineering, …).
68
- 4. **Help** (`/help`) guides like this one, plus technical documentation on the signals and how to contribute data.
69
- 5. **About** (`/about`) — the methodology: how the signals are computed and the principles behind the corpus.
70
- 6. **Feedback** (`/feedback`) share feedback, request a feature, or report an issue.
71
 
72
  ---
73
 
74
  ## Your first 5 minutes
75
 
76
- 1. **Open a model page.** Go to **Models**, click any model (e.g. *Claude Opus 4.7*). This page is the model's evaluation record. Behind each number sits more than the page shows the benchmark's own metadata and the evaluation run details which is what actually makes the score interpretable.
77
 
78
  > 🖼️ **Screenshot — `13-card-summary-full.png`**
79
  > *What to capture:* A full model page in **Summary View** (e.g. `/models/anthropic/claude-opus-4.7`).
80
 
81
- 2. **Read the `DOCUMENTED` badge.** Near the top, a percentage (e.g. "36% 14 / 39 reported") tells you for how many of the model's reported scores the **prompting/run setup is documented** well enough to re-run the rest are missing details like temperature, max tokens, or the harness. (This is the model-level *Reproducibility* read; it's not about completed benchmark cards.) A low number here is common, and that's deliberate.
82
 
83
  3. **Check "Who reports what" (§3).** A donut and per-category bars split results into **first-party** (the developer's own numbers) vs **third-party** (independent). This is your fastest read on how independent the evidence is.
84
 
@@ -88,19 +88,19 @@ For example: *MMLU (family) → MMLU-Pro (composite) → a single benchmark →
88
  4. **Understand the Summary score.** In Summary View each benchmark is shown leaderboard-style, ranked against peer models. When a model has more than one reported result for a benchmark, the score shown is the **median** of all reported results.
89
 
90
  > 🖼️ **Screenshot — `18-card-metrics.png`**
91
- > *What to capture:* The §4 Reported metrics list in Summary View benchmarks ranked best→worst with the model's score and peer rank.
92
 
93
  5. **Switch to Researcher View.** The toggle in the bar at the top of the page swaps Summary for Researcher view, which exposes the underlying per-result detail. Use Summary to get oriented, and Researcher when you want to dig in.
94
 
95
  > 🖼️ **Screenshot — `23-reader-bar.png`**
96
  > *What to capture:* The Summary / Researcher View toggle at the top of a model page (the snapshot date sits at the right of the same bar).
97
 
98
- In Researcher View, §4 gains a per-result table: for each benchmark it lists the number of results (**N**), the **mean and 95% CI**, the **range** of reported scores, and each **source** that reported them the spread and provenance behind the summary number.
99
 
100
  > 🖼️ **Screenshot — `21-card-researcher-metrics.png`**
101
  > *What to capture:* The per-result table in Researcher View (N, mean / CI, range, sources).
102
 
103
- 6. **Note the snapshot date.** Every model and evaluation page shows a snapshot date in that same top bar (e.g. *Snapshot · Jun 9, 2026*), and the homepage shows it in the **Corpus snapshot** header. The corpus is versioned, so numbers change between snapshots cite the snapshot date with any figure so others can find exactly what you saw.
104
 
105
  ---
106
 
@@ -109,9 +109,9 @@ For example: *MMLU (family) → MMLU-Pro (composite) → a single benchmark →
109
  The **Evaluations** tab is the benchmark-first view. Benchmarks are grouped into **families** and tagged by category; filter by interaction style (agent / non-agent) or by category to narrow the list.
110
 
111
  > 🖼️ **Screenshot — `05-evals-index.png`**
112
- > *What to capture:* The Evaluations index families with their categories and counts.
113
 
114
- Open a family to drill into its structure: the single benchmarks beneath it, their splits, and the metrics reported on each the same five-level hierarchy, top-down.
115
 
116
  > 🖼️ **Screenshot — `08-evals-family-expanded.png`**
117
  > *What to capture:* A benchmark family opened to show its benchmarks and splits.
@@ -121,14 +121,14 @@ Open a family to drill into its structure: the single benchmarks beneath it, the
121
  Click a benchmark to open its own page. The **At a glance** card at the top summarizes what it measures, its main caveat, who it's intended for, and how to read it, with links to the source paper and dataset. Below sit the **Benchmark card** (schema, methodology, what it measures), **Technical details** (metric, completeness, comparability, splits), and a **"Can these scores be compared directly?"** panel.
122
 
123
  > 🖼️ **Screenshot — `27-eval-detail-card.png`**
124
- > *What to capture:* A benchmark's evaluation page the "At a glance" card (measures · caveat · intended for · sources).
125
 
126
  Scroll to **Reporting Comparison**: every model with a reported result on this benchmark, ranked, with the score distribution, the evaluator, the source, and the date. It's the benchmark-centric mirror of the §4 metrics on a model page.
127
 
128
  > 🖼️ **Screenshot — `28-eval-detail-comparison.png`**
129
  > *What to capture:* The Reporting Comparison leaderboard on a benchmark's evaluation page.
130
 
131
- The chart above the leaderboard has two views, toggled at the top. **Distribution** shows how all the reported scores are spread across models; **Frontier** traces the best score over time how the state of the art on this benchmark has advanced as newer models were released.
132
 
133
  > 🖼️ **Screenshot — `29-eval-frontier.png`**
134
  > *What to capture:* The Reporting Comparison chart in **Frontier** view (best score over time), with the Distribution / Frontier toggle.
@@ -149,4 +149,4 @@ Evaluation Cards is a community effort from the EvalEval Coalition, and it gets
149
 
150
  ---
151
 
152
- ➡️ **Next:** pick the guide for your role [Evaluation researchers](evaluation-researchers.md) · [Policymakers](policymakers.md) · [General public](general-public.md) · [Journalists](journalists.md).
 
1
+ # Evaluation Cards: Quickstart
2
 
3
  *A stakeholder-agnostic guide to getting started. ~6 min.*
4
 
5
+ > **Heads up: Evaluation Cards is in Beta.** We'd love your feedback: report bugs, request features, or tell us what's confusing through our [feedback form](/feedback) or the [public roadmap](https://changemap.co/evaleval/evalcards/). You can reach the feedback form from any page via **Feedback** in the top navigation.
6
 
7
  > 🖼️ **Screenshot — `home-hero-snapshot.png`**
8
+ > *What to capture:* The homepage hero and the **Corpus snapshot**: the current totals for models, results, organizations, and benchmarks.
9
 
10
  ---
11
 
 
13
 
14
  **Evaluation Cards** is a reporting layer over AI model evaluations, built by the [EvalEval Coalition](https://evalcards.evalevalai.com/about). It collects how AI models are evaluated across many benchmarks and reporting organizations, and also shows **what was left undocumented**.
15
 
16
+ A benchmark score on its own ("Model X scores 87% on MMLU") tells you very little. Evaluation Cards puts each score in context: *who* ran the evaluation, *how* it was set up, *whether* it can be reproduced, and *whether* it can be fairly compared to another score. The project treats every published evaluation as a **claim**, and every undisclosed detail as **a claim deliberately not made**. Neither is treated as an error.
17
 
18
  At a glance (snapshot of June 2026), the corpus tracks:
19
 
 
23
  | **31** reporting organizations | **820** model developers |
24
  | **57** benchmark families | **632** single benchmarks |
25
 
26
+ > The corpus is **versioned by snapshot**. Every page shows a snapshot date. Numbers above will drift as the corpus grows, so always cite the snapshot you saw.
27
 
28
  ---
29
 
 
36
 
37
  | Signal | Question it answers | What a low score means |
38
  |---|---|---|
39
+ | **R · Reproducibility** | Could a third party re-run this evaluation? | Prompts, decoding settings, harness version, seeds, or code are undisclosed. |
40
+ | **C · Completeness** | Does the record meet normal reporting expectations for this kind of model? | Whole categories (e.g. safety, robustness, fairness) may be missing. |
41
+ | **P · Provenance & Risk** | Who ran it, and what real-world property does it measure? | Distinguishes **first-party** (the developer) from **third-party** (independent) evaluators. |
42
+ | **X · Comparability** | Can two scores on the same benchmark be put side by side? | Different splits, metric variants, or units make a direct ranking invalid. |
43
 
44
  A high benchmark score with weak signals is still a weak claim. The signals are how you tell a well-documented result from one you can't check.
45
 
 
62
  > 🖼️ **Screenshot — `02-home-hero.png`**
63
  > *What to capture:* The top navigation bar (Overview · Models · Evaluations · Help · About · Feedback).
64
 
65
+ 1. **Overview** (`/`): the corpus snapshot, the signals explained, and featured benchmark families. Start here.
66
+ 2. **Models** (`/models`): every indexed model. Filter by parameter size, switch between **Models** and **Developers** views, and select up to four models to compare.
67
+ 3. **Evaluations** (`/evals`): benchmarks grouped into **families**, filterable by interaction style (agent / non-agent) and ~17 categories (Mathematics, Safety, Software Engineering, …).
68
+ 4. **Help** (`/help`): guides like this one, plus technical documentation on the signals and how to contribute data.
69
+ 5. **About** (`/about`): how the signals are computed and the principles behind the corpus.
70
+ 6. **Feedback** (`/feedback`): share feedback, request a feature, or report an issue.
71
 
72
  ---
73
 
74
  ## Your first 5 minutes
75
 
76
+ 1. **Open a model page.** Go to **Models**, click any model (e.g. *Claude Opus 4.7*). This page is the model's evaluation record. Behind each number sits more than the page shows: the benchmark's own metadata and the evaluation run details, which is what actually makes the score interpretable.
77
 
78
  > 🖼️ **Screenshot — `13-card-summary-full.png`**
79
  > *What to capture:* A full model page in **Summary View** (e.g. `/models/anthropic/claude-opus-4.7`).
80
 
81
+ 2. **Read the `DOCUMENTED` badge.** Near the top, a percentage (e.g. "36%, 14 / 39 reported") tells you for how many of the model's reported scores the **prompting/run setup is documented** well enough to re-run. The rest are missing details like temperature, max tokens, or the harness. (This is the model-level *Reproducibility* read; it's not about completed benchmark cards.) A low number here is common, and that's deliberate.
82
 
83
  3. **Check "Who reports what" (§3).** A donut and per-category bars split results into **first-party** (the developer's own numbers) vs **third-party** (independent). This is your fastest read on how independent the evidence is.
84
 
 
88
  4. **Understand the Summary score.** In Summary View each benchmark is shown leaderboard-style, ranked against peer models. When a model has more than one reported result for a benchmark, the score shown is the **median** of all reported results.
89
 
90
  > 🖼️ **Screenshot — `18-card-metrics.png`**
91
+ > *What to capture:* The §4 Reported metrics list in Summary View: benchmarks ranked best→worst with the model's score and peer rank.
92
 
93
  5. **Switch to Researcher View.** The toggle in the bar at the top of the page swaps Summary for Researcher view, which exposes the underlying per-result detail. Use Summary to get oriented, and Researcher when you want to dig in.
94
 
95
  > 🖼️ **Screenshot — `23-reader-bar.png`**
96
  > *What to capture:* The Summary / Researcher View toggle at the top of a model page (the snapshot date sits at the right of the same bar).
97
 
98
+ In Researcher View, §4 gains a per-result table: for each benchmark it lists the number of results (**N**), the **mean and 95% CI**, the **range** of reported scores, and each **source** that reported them. That's the spread and provenance behind the summary number.
99
 
100
  > 🖼️ **Screenshot — `21-card-researcher-metrics.png`**
101
  > *What to capture:* The per-result table in Researcher View (N, mean / CI, range, sources).
102
 
103
+ 6. **Note the snapshot date.** Every model and evaluation page shows a snapshot date in that same top bar (e.g. *Snapshot · Jun 9, 2026*), and the homepage shows it in the **Corpus snapshot** header. The corpus is versioned, so numbers change between snapshots. Cite the snapshot date with any figure so others can find exactly what you saw.
104
 
105
  ---
106
 
 
109
  The **Evaluations** tab is the benchmark-first view. Benchmarks are grouped into **families** and tagged by category; filter by interaction style (agent / non-agent) or by category to narrow the list.
110
 
111
  > 🖼️ **Screenshot — `05-evals-index.png`**
112
+ > *What to capture:* The Evaluations index, with families and their categories and counts.
113
 
114
+ Open a family to drill into its structure: the single benchmarks beneath it, their splits, and the metrics reported on each, following the same five-level hierarchy top-down.
115
 
116
  > 🖼️ **Screenshot — `08-evals-family-expanded.png`**
117
  > *What to capture:* A benchmark family opened to show its benchmarks and splits.
 
121
  Click a benchmark to open its own page. The **At a glance** card at the top summarizes what it measures, its main caveat, who it's intended for, and how to read it, with links to the source paper and dataset. Below sit the **Benchmark card** (schema, methodology, what it measures), **Technical details** (metric, completeness, comparability, splits), and a **"Can these scores be compared directly?"** panel.
122
 
123
  > 🖼️ **Screenshot — `27-eval-detail-card.png`**
124
+ > *What to capture:* A benchmark's evaluation page, showing the "At a glance" card (measures · caveat · intended for · sources).
125
 
126
  Scroll to **Reporting Comparison**: every model with a reported result on this benchmark, ranked, with the score distribution, the evaluator, the source, and the date. It's the benchmark-centric mirror of the §4 metrics on a model page.
127
 
128
  > 🖼️ **Screenshot — `28-eval-detail-comparison.png`**
129
  > *What to capture:* The Reporting Comparison leaderboard on a benchmark's evaluation page.
130
 
131
+ The chart above the leaderboard has two views, toggled at the top. **Distribution** shows how all the reported scores are spread across models; **Frontier** traces the best score over time, showing how the state of the art on this benchmark has advanced as newer models were released.
132
 
133
  > 🖼️ **Screenshot — `29-eval-frontier.png`**
134
  > *What to capture:* The Reporting Comparison chart in **Frontier** view (best score over time), with the Distribution / Frontier toggle.
 
149
 
150
  ---
151
 
152
+ ➡️ **Next:** pick the guide for your role: [Evaluation researchers](evaluation-researchers.md) · [Policymakers](policymakers.md) · [General public](general-public.md) · [Journalists](journalists.md).
content/tutorials/what-its-built-on.md CHANGED
@@ -1,16 +1,16 @@
1
  # What Evaluation Cards is built on
2
 
3
- *Evaluation Cards doesn't run evaluations it composes existing evaluation infrastructure into a single reading surface. Here's what powers it.*
4
 
5
  ---
6
 
7
  ## Auto-BenchmarkCards
8
 
9
- A schema for benchmark-level metadata what a benchmark measures, its splits, intended use, validity scope, and known limitations. Each benchmark family has an Auto-BenchmarkCard at the family root and a Policy Note compressed for plain-language reading.
10
 
11
  ## Every Eval Ever
12
 
13
- A run-level corpus of public evaluation results `(model, benchmark, metric-path, value, source)` tuples extracted from papers, model cards, and leaderboards. It provides the raw rows that Evaluation Cards canonicalises and joins. [Every Eval Ever](https://evalevalai.com/every_eval_ever/) is a sister [EvalEval](https://evalevalai.com/) project.
14
 
15
  ## IBM Risk Atlas alignment
16
 
 
1
  # What Evaluation Cards is built on
2
 
3
+ *Evaluation Cards doesn't run evaluations; it composes existing evaluation infrastructure into a single reading surface. Here's what powers it.*
4
 
5
  ---
6
 
7
  ## Auto-BenchmarkCards
8
 
9
+ A schema for benchmark-level metadata: what a benchmark measures, its splits, intended use, validity scope, and known limitations. Each benchmark family has an Auto-BenchmarkCard at the family root and a Policy Note compressed for plain-language reading.
10
 
11
  ## Every Eval Ever
12
 
13
+ A run-level corpus of public evaluation results: `(model, benchmark, metric-path, value, source)` tuples extracted from papers, model cards, and leaderboards. It provides the raw rows that Evaluation Cards canonicalises and joins. [Every Eval Ever](https://evalevalai.com/every_eval_ever/) is a sister [EvalEval](https://evalevalai.com/) project.
14
 
15
  ## IBM Risk Atlas alignment
16