Benchmarks: first results bench50-v0.3.1 (results 1cf2b8bb, page b71d0c71)

METHOD.md

@@ -0,0 +1,64 @@
+# 50 distinct benchmarks for small base models — method (v0.3.1)
+
+This document describes how the 50-benchmark panel is selected, sampled and scored. It contains no questions, answers or model outputs. Machine-readable definitions:
+
+- `tasks_50.json` — the 50 benchmarks, one primary protocol each (language, metric, shots, split).
+- `benchmark_sources.json` — exact task definitions, group members, datasets, pinned revisions and observed licence tags.
+- `results.schema.json` — the results format.
+- `data_access.json` — parquet access for script-only datasets (section 8).
+
+## 1. What counts as one benchmark
+
+One benchmark is one named, source-identified collection of tasks, not one YAML file. Splits, rounds, prompt formats, number of shots and translations do not add benchmarks: ARC, ANLI and PoLEMo2 count once; MMLU (57 subjects), BLiMP (67 paradigms), MATH (7) and BBH (27) count once each. GLUE/SuperGLUE are not counted on top of their component datasets. Datasets that reuse another selected benchmark's question pool (QNLI vs SQuAD, SWAG vs HellaSwag, MMLU-Pro vs MMLU, MATH-500 vs MATH) were left out.
+
+Result: 50 benchmarks, 38 English and 12 Polish, from 44 dataset repositories; 204 source tasks underneath. Fifty distinct collections are not fifty independent pieces of evidence.
+
+Not included: code execution benchmarks (need a separate sandbox) and agent/tool-use benchmarks.
+
+## 2. Harness and runtime
+
+- Evaluation harness: EleutherAI lm-evaluation-harness, English tasks at commit `6d2abda4fd171e68a8789330c4149e37c1ca0bda`; Polish tasks from the OpenPL fork at `21d0ea9cf4fd6153dfff4d84d6ad0aab5488f302`. Separate environments, hash-locked dependencies.
+- Settings: float32, batch size 1, maximum context 4096 tokens (longer inputs are rejected, never truncated), no chat template, greedy decoding. BOS follows each model's own tokenizer convention (for Hugging Face models `add_bos_token` from the pinned tokenizer files; recorded per run), with exactly one leading BOS per request; for empty contexts and rolling perplexity the BOS is an unscored prefix. This avoids the duplicate scored BOS that the pinned harness produces in those two paths when `add_bos_token` is true. Harness seeds (python, numpy, torch, fewshot) = (0, 1234, 1234, 1234).
+- Offline: no network during evaluation; datasets, metric modules and NLTK data are pinned and staged before the run. Dataset loading scripts are never executed (see section 8).
+- Before any benchmark is scored on a model, the runtime is calibrated on that model against the original harness forward: summed log-likelihood difference per target token ≤ 2e-5, full-vocabulary logits bitwise equal, rolling windows count each token once, greedy generation identical.
+
+## 3. Unit and sampling (panel200)
+
+- Unit = one source document after the task's own `process_docs`. Identity = benchmark, source task, and the document's native id or, if none, the SHA-256 of the canonical document; byte-identical documents without an id get an ordinal in pinned file order (section 8).
+- Per benchmark, `min(200, N_eligible)` documents are selected before any model is run, by SHA-256 of `20261028|benchmark_id|member_task|source_id`. All models are scored on the same documents.
+- Groups: the 200 slots are per benchmark, not per member. Every member gets `min(2, N_i)` documents first; remaining slots go to the member with the largest deficit against `budget × source_weight`. BLiMP members are weighted equally; MMLU, MATH and BBH proportionally to member population. Member scores are re-aggregated with these fixed weights.
+- WikiText-2 is scored on the full test stream with byte perplexity.
+- `fullB` is a separate full-split run of the same primary protocols, without the 200 cap.
+
+## 4. Demonstrations and generation
+
+- Number of shots as listed per benchmark (37 zero-shot, 12 with 5 shots, BBH with its source 3-shot prompts). Demonstrations come from a disjoint bank selected by hash; a demonstration never appears among scored documents. Fixed source prompts (BBH) are kept as published.
+- Generation budget from the task definition (default 256 tokens; IFEval 1280). If prompt + budget exceed 4096, optional demonstrations are removed from the end; questions, options and references are never cut.
+- Source task filters are applied unchanged (e.g. `remove_whitespace` for NQ-Open and TriviaQA).
+- BBH: the generated answer is stripped of leading and trailing whitespace before exact match. Without this, byte-level BPE tokenizers keep the leading space of the first token and every answer fails, while SentencePiece tokenizers drop it on decoding; the source-literal value is kept as a diagnostic. This correction was introduced after the first run, once the zero score was traced to whitespace; it applies identically to all models.
+- MATH: the 0-shot source extractor takes the text between the first and last `$` of the answer and does not read `\boxed{}`. Scores are reported exactly as the source gives them, and MATH is not used to rank models. A re-scoring by the last `\boxed{}`, made after the first run when both models' outputs were seen, is shown only as a labelled diagnostic and never as the MATH score.
+- IFEval: `langdetect` is seeded (`DetectorFactory.seed = 0`) for reproducibility. It affects only IFEval's language instructions.
+
+## 5. Metrics and reporting
+
+- One pre-declared primary metric per benchmark, in its source unit and scale (accuracy, F1, exact match, Levenshtein similarity, MCC, byte perplexity, IFEval prompt-level strict accuracy, …). MCC may be negative. There is **no average over the 50 benchmarks**.
+- Uncertainty: 95% bootstrap confidence interval, 10 000 replicates, seed 20261028, resampling within group members with fixed weights. Byte-identical documents are resampled together as one cluster. For model-vs-model contrasts: paired bootstrap and a two-sided paired permutation test (10 000 swaps, +1 correction) with Holm correction over all pre-declared contrasts. When a contrast has at most 20 clusters with a non-zero difference, the permutation p-value is computed exactly; otherwise rows whose Holm-adjusted p falls between 0.02 and 0.10 are recomputed with 200 000 swaps. This precision rule was added after the first computation, when Monte Carlo error alone decided one verdict; it applies to every contrast and can add or remove significance. Each contrast states the items it was computed on (`subset`: the full frozen sample, or a hit-free subset defined by our model's training corpus and identical for both models) and its n; hit-free variants carry their own 95% CI.
+
+## 6. Contamination
+
+- For our own models the full training corpus is scanned against every scored split before results are published: each scored text field is checked separately: fields of 13+ words by word 13-grams after normalisation (as in GPT-3); otherwise whole fields of 3–12 words (at least 60% of the item's longest field) by exact normalised match; categorical fields (labels, ids) are excluded; fields stored as serialized lists or dicts are parsed and their parts checked as separate fields; bare numbers are ignored; items with no field of 3+ words are unscannable. Positive and negative controls are mandatory.
+- **Reference values per result:** expected value of uniform random choice on the frozen sample, value of always predicting the majority gold class (fixed-class tasks only), share of the model's most frequent prediction, and a constant-prediction flag when the model gives one class for every item. A constant-prediction result is a real measurement but carries no information about ability and is not used for comparisons.
+- **Contamination by field role:** for flagged benchmarks, hit-free values are reported twice: without any hit, and without hits in answer-bearing fields (question, options, target); context-only hits (e.g. a Wikipedia passage) indicate exposure to the source text, not to the question and label.
+- States per (model, benchmark): `clean`; `flagged` (reported on the full sample and on the hit-free subset, with hit rate; when no item of the sample is hit-free, the hit-free value is empty with n = 0 — this can happen for passage-based sets (e.g. built on Wikipedia); then only the answer-field variant can carry a number); `partially_scanned` (more than 5% unscannable); `unscanned` (not published); `unknown` (third-party model whose training data cannot be scanned — shown with the label "training data unknown").
+
+## 7. Data use
+
+Evaluation only. Public outputs are limited to benchmark id, revision, licence, aggregate result, n and CI95. Questions, answers, few-shot examples and per-item outputs are not redistributed, and nothing from these datasets is used for training. Users fetch the data themselves from the pinned revisions under each dataset's own licence.
+
+## 8. Clarifications in v0.3.1
+
+- **Script-only datasets** (PIQA, WSC273, LogiQA, MathQA, PubMedQA, Social IQa, PoQuAD, PolQA): benchmark identity stays at the pinned revision; data are read from the Hugging Face auto-converted parquet at the listed `refs/convert/parquet` commit. Columns must match what the task reads, otherwise the benchmark is blocked. PubMedQA additionally requires the parquet test split to equal the labeled test set of the pinned revision by id and label.
+- **Duplicate documents** without a native id: identity = document hash + ordinal in pinned file order; bootstrap resamples identical documents as one cluster; counts are reported.
+- **Likelihood vs generation:** 39 benchmarks are scored by likelihood or perplexity, 11 by generation (PolQA, PoQuAD, GSM8K, MATH, IFEval, SQuAD 2.0, DROP, CoQA, NQ-Open, TriviaQA, BBH). They may be run in separate windows; a benchmark not yet run is `not_run`.
+
+- **Public results file:** exported from qualifying runs only (runner and runtime implement every rule above). Values, CIs and digests are copied unchanged, and the sources are listed by sha256 in `provenance`. Our own models' results without a completed contamination scan are published as `skipped` without a value.

README.md

@@ -1,10 +1,39 @@
 ---
 title: Slayer1B Benchmarks
-emoji: 👀
-colorFrom: blue
-colorTo: yellow
+emoji: 📏
+colorFrom: green
+colorTo: blue
 sdk: static
 pinned: false
+short_description: 50 distinct benchmarks, one protocol, contamination-checked
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Slayer1B Benchmarks
+
+Fifty distinct public benchmarks (38 English, 12 Polish), run with one protocol on
+[Slayer 1B](https://huggingface.co/spaces/SlayerLab/Slayer1B) and comparison models.
+This Space is a static page: it shows the published result file and computes nothing itself.
+
+## What is on the page
+
+- **One value per benchmark and model**, in the benchmark's own metric and unit. Every main value and every model-vs-model contrast has a 95% confidence interval; benchmarks scored by picking one answer also show reference points (random guess, always-majority class); values on items without contamination hits are shown without an interval.
+- **No average over the fifty benchmarks** and no overall ranking.
+- **Contamination:** for our own models the full training corpus is scanned against every scored item before a number is shown. Benchmarks with hits are labelled and shown on the full sample and on the items without hits. Models whose training data is not public are labelled *training data unknown*.
+- **Better / worse** between two models is stated only from a paired permutation test with Holm correction over the whole comparison family; otherwise the result is *inconclusive*. Benchmarks where a model gives the same answer to every item are not compared.
+- **Scoring corrections** made after the first run are labelled on the page and described in `METHOD.md`, together with when and why they were made.
+
+## Files
+
+| file | content |
+|---|---|
+| `tasks_50.json` | benchmark list: sources, pinned revisions, metrics, licences |
+| `METHOD.md` | protocol, sampling, uncertainty, contamination scan, corrections |
+| `results.schema.json` | format of the result file |
+| `contamination_field_roles.json` | which item fields count as context or as answer for the contamination variants |
+| `results-feed.json` | pointer to the published result file and its SHA-256 |
+| `results.json` | published results; the page checks every file against its pinned SHA-256 before showing anything |
+
+Benchmark texts are not redistributed here; each benchmark keeps its own licence (see `tasks_50.json`).
+Only aggregate results are published.
+
+Fabryka AI · SlayerLab

app.js

@@ -0,0 +1,327 @@
+"use strict";
+(() => {
+  const contract = window.BenchContract;
+  const T = {
+    en: {
+      search:"Search benchmark", language:"Language", show:"Show", all:"All", withResults:"With a published number", clear:"Clear filters",
+      benchmark:"Benchmark", results:"Results", notPublished:"Not published yet",
+      noPacket:"No results are published yet. The list below is the plan: 50 benchmarks, their sources and metrics. For models whose training data we have, a number appears only after a contamination scan.",
+      loadError:"A pinned file could not be verified, so nothing is shown.", rejected:"Results are hidden because the result file did not pass the checks",
+      empty:"No benchmark matches the filters.", shown:"Showing", of:"of", benchmarks:"benchmarks",
+      pills:{total:"benchmarks", langs:"38 EN · 12 PL", models:"models", published:"with a published number"},
+      higher:"higher is better", lower:"lower is better", ci:"95% CI", noCI:"no interval", random:"random", majority:"majority", delta:"difference",
+      majorityShare:"predicts its most common class in", none:"none", refMissing:"reference: not available",
+      scan:{clean:"clean", flagged:"flagged", partially_scanned:"scan incomplete", unscanned:"scan pending", unknown:"training data unknown"},
+      noNumber:{partially_scanned:"No number: the scan did not cover enough items.", unscanned:"No number until the contamination scan is done."},
+      hitFree:"Without items found in training data", hitFreeAnswer:"Without hits in question or answer fields", hits:"items with hits", n:"n",
+      noHitFree:"no items without hits", tooFew:"too few items for a number",
+      mostly:"fewer than 50 items without hits in training data", inSlayer:"in the Slayer training corpus", slayerSubset:"subsets defined by the Slayer training corpus", mostlyVerdict:"No better/worse from the full sample here: only a paired test on the items without answer-field hits can give one.",
+      pending:n => `${n} benchmarks are not measured yet; adding them cannot change the verdicts shown; it can settle rows now inconclusive and adds verdicts for the new rows.`,
+      constant:"constant answer", constantNote:"One class for all items: not compared between models.",
+      states:{ok:"Measured",not_run:"Not run",blocked_access:"Data access blocked",blocked_runtime:"Runtime blocked",blocked_security:"Blocked by security review",blocked_judge:"No accepted judge",unsupported:"Not supported",failed:"Run failed",skipped:"Skipped"},
+      verdicts:{better:"better than",worse:"worse than",inconclusive:"inconclusive vs"}, verdictNote:"paired test, Holm-adjusted", verdictNoteSubset:"paired test on items without answer-field hits, Holm-adjusted",
+      amendments:{answer_leading_space_strip:"Scoring corrected after first run (leading space), see METHOD.", other:"Scoring amended after first run, see METHOD."},
+      scoringNotes:{source_extractor_floor:"Floor: the source 0-shot extractor reads the text between the first and last $, so answers in \\boxed{} score 0. Not a comparison of ability, see METHOD.", other:"Scoring note, see METHOD."},
+      details:"Source and sample", population:"population / sample / scored", revision:"revision", licence:"licence (dataset card)", files:"Pinned files",
+      packet:"result file", run:"run", protocol:"protocol"
+    },
+    pl: {
+      search:"Szukaj benchmarku", language:"Język", show:"Pokaż", all:"Wszystkie", withResults:"Z opublikowaną liczbą", clear:"Wyczyść filtry",
+      benchmark:"Benchmark", results:"Wyniki", notPublished:"Jeszcze nie opublikowano",
+      noPacket:"Nie ma jeszcze opublikowanych wyników. Poniżej jest plan: 50 benchmarków, ich źródła i miary. Dla modeli, których dane treningowe mamy, liczba pojawia się dopiero po skanie kontaminacji.",
+      loadError:"Nie udało się zweryfikować przypiętego pliku, więc niczego nie pokazuję.", rejected:"Wyniki ukryte, bo plik wyników nie przeszedł kontroli",
+      empty:"Żaden benchmark nie pasuje do filtrów.", shown:"Pokazano", of:"z", benchmarks:"benchmarków",
+      pills:{total:"benchmarków", langs:"38 EN · 12 PL", models:"modele", published:"z opublikowaną liczbą"},
+      higher:"więcej = lepiej", lower:"mniej = lepiej", ci:"95% CI", noCI:"bez przedziału", random:"losowo", majority:"większość", delta:"różnica",
+      majorityShare:"najczęstsza własna klasa w", none:"brak", refMissing:"odniesienie: brak",
+      scan:{clean:"czysty", flagged:"trafienia", partially_scanned:"skan niepełny", unscanned:"skan w toku", unknown:"dane treningowe nieznane"},
+      noNumber:{partially_scanned:"Bez liczby: skan nie objął dość pozycji.", unscanned:"Bez liczby do czasu skanu kontaminacji."},
+      hitFree:"Bez pozycji znalezionych w danych treningowych", hitFreeAnswer:"Bez trafień w polach pytania i odpowiedzi", hits:"pozycji z trafieniem", n:"n",
+      noHitFree:"brak pozycji bez trafień", tooFew:"za mało pozycji na liczbę",
+      mostly:"mniej niż 50 pozycji bez trafień w danych treningowych", inSlayer:"w korpusie treningowym Slayera", slayerSubset:"podzbiory wyznaczone korpusem treningowym Slayera", mostlyVerdict:"Tu nie ma werdyktu lepszy/gorszy z całej próbki: może go dać tylko test sparowany na pozycjach bez trafień w polach odpowiedzi.",
+      pending:n => `${n} benchmarków nie jest jeszcze zmierzonych; ich dodanie nie zmieni pokazanych werdyktów; może rozstrzygnąć wiersze dziś nierozstrzygnięte i doda werdykty dla nowych wierszy.`,
+      constant:"stała odpowiedź", constantNote:"Jedna klasa dla wszystkich pozycji: nie porównujemy między modelami.",
+      states:{ok:"Pomiar",not_run:"Nie uruchomiono",blocked_access:"Brak dostępu do danych",blocked_runtime:"Blokada środowiska",blocked_security:"Zablokowane przez przegląd bezpieczeństwa",blocked_judge:"Brak przyjętego sędziego",unsupported:"Nieobsługiwane",failed:"Błąd biegu",skipped:"Pominięte"},
+      verdicts:{better:"lepszy niż",worse:"gorszy niż",inconclusive:"nierozstrzygnięte wobec"}, verdictNote:"test sparowany, poprawka Holma", verdictNoteSubset:"test sparowany na pozycjach bez trafień w polach odpowiedzi, poprawka Holma",
+      amendments:{answer_leading_space_strip:"Punktacja poprawiona po pierwszym biegu (spacja na początku), zob. METHOD.", other:"Punktacja zmieniona po pierwszym biegu, zob. METHOD."},
+      scoringNotes:{source_extractor_floor:"Podłoga: ekstraktor źródła (0-shot) czyta tekst między pierwszym a ostatnim $, więc odpowiedzi w \\boxed{} dostają 0. To nie jest porównanie umiejętności, zob. METHOD.", other:"Uwaga o punktacji, zob. METHOD."},
+      details:"Źródło i próbka", population:"populacja / próbka / ocenione", revision:"rewizja", licence:"licencja (karta zbioru)", files:"Przypięte pliki",
+      packet:"plik wyników", run:"bieg", protocol:"protokół"
+    }
+  };
+  const state = {lang:"en", tasks:null, packet:null, index:null, packetSHA:null, error:null, files:[], query:"", language:"all", show:"all"};
+
+  function el(tag, text, cls) {
+    const n = document.createElement(tag);
+    if (text !== undefined && text !== null) n.textContent = String(text);
+    if (cls) n.className = cls;
+    return n;
+  }
+  function fmt(v, unit) {
+    const digits = unit === "byte_ppl" ? 3 : unit === "percent" ? 1 : 3;
+    return new Intl.NumberFormat(state.lang === "pl" ? "pl-PL" : "en-GB", {minimumFractionDigits:digits, maximumFractionDigits:digits}).format(v);
+  }
+  const unitLabel = u => u === "percent" ? "%" : u === "byte_ppl" ? "byte-PPL" : "";
+  const pct = v => new Intl.NumberFormat(state.lang === "pl" ? "pl-PL" : "en-GB", {style:"percent", maximumFractionDigits:1}).format(v);
+  function modelName(m) { return m.repo.split("/").pop(); }
+
+  // Thresholds and the "most items found in training data" class come from the contract, which also guarantees that every
+  // non-withheld verdict in an accepted packet follows the display rules (binding rules 2026-10-29 16:55 and 17:07).
+  const MIN_HIT_FREE_N = contract.MIN_HIT_FREE_N;
+  const mostlyTrained = taskId => state.index ? contract.mostlyTrained(state.index.observations, taskId, state.packet.protocol_id) : null;
+  // Whether the cell carries a number at all, and why not otherwise.
+  function numberRule(o) {
+    if (o.state !== "ok") return {show:false, reason:null};
+    if (["clean","flagged","unknown"].includes(o.contamination)) return {show:true};
+    return {show:false, reason:o.contamination};
+  }
+  function hitFreeLine(label, value, n, rate, ci, unit, t) {
+    const tail = `${t.n} ${n}; ${pct(rate)} ${t.hits}`;
+    if (n === 0) return `${label}: ${t.noHitFree}`;
+    if (n < MIN_HIT_FREE_N) return `${label}: ${t.tooFew} (${tail})`;
+    const interval = ci ? `${t.ci} ${fmt(ci.lower, unit)} – ${fmt(ci.upper, unit)}` : t.noCI;
+    return `${label}: ${fmt(value, unit)}, ${interval} (${tail})`;
+  }
+
+  const isConstant = x => !!(x && x.reference && x.reference.constant_prediction === true);
+
+  function cell(task, model, t) {
+    const td = el("td");
+    const o = state.index && state.index.observations.get(contract.key(task.id, model.model_id, state.packet.protocol_id));
+    if (!o) { td.append(el("span", t.notPublished, "state")); return td; }
+    const mostly = mostlyTrained(task.id);
+    const rule = numberRule(o);
+    const tags = el("div", undefined, "tags");
+    if (o.state !== "ok") {
+      td.append(el("span", t.states[o.state], "state"));
+      if (o.reason) td.append(el("span", o.reason, "meta"));
+    } else if (!rule.show) {
+      td.append(el("span", t.noNumber[rule.reason], "state"));
+    } else {
+      const v = el("span", fmt(o.value, o.unit), mostly === null ? "value" : "value dim");
+      if (unitLabel(o.unit)) v.append(el("small", unitLabel(o.unit)));
+      td.append(v);
+      td.append(el("span", o.ci95 ? `${t.ci} ${fmt(o.ci95.lower, o.unit)} – ${fmt(o.ci95.upper, o.unit)}` : t.noCI, "meta"));
+      if (mostly !== null) td.append(el("span", `${t.mostly} (${pct(mostly)} ${t.hits}${o.contamination === "unknown" ? `, ${t.inSlayer}` : ""})`, "mostly"));
+      const hasA = Number.isInteger(o.n_hit_free), hasB = Number.isInteger(o.n_hit_free_answer_fields);
+      if (hasA) td.append(el("span", hitFreeLine(t.hitFree, o.value_hit_free, o.n_hit_free, o.hit_rate, o.ci95_hit_free, o.unit, t), "ref"));
+      if (hasB) td.append(el("span", hitFreeLine(t.hitFreeAnswer, o.value_hit_free_answer_fields, o.n_hit_free_answer_fields, o.hit_rate_answer_fields, o.ci95_hit_free_answer_fields, o.unit, t), "ref"));
+      if ((hasA || hasB) && o.contamination === "unknown") td.append(el("span", t.slayerSubset, "meta"));
+      const r = o.reference;
+      const refs = [];
+      if (r && r.chance_value !== null) refs.push(`${t.random} ${fmt(r.chance_value, o.unit)}`);
+      if (r && r.majority_value !== null) refs.push(`${t.majority} ${fmt(r.majority_value, o.unit)}`);
+      td.append(el("span", refs.length ? refs.join(" · ") : t.refMissing, "ref"));
+      if (r && r.majority_prediction_share !== null) td.append(el("span", `${t.majorityShare} ${pct(r.majority_prediction_share)}`, "meta"));
+      if (isConstant(o)) tags.append(el("span", t.constant, "tag constant"));
+    }
+    const scanClass = {clean:"clean", flagged:"flagged", unknown:"unknown"}[o.contamination] || "pending";
+    tags.append(el("span", t.scan[o.contamination], `tag ${scanClass}`));
+    td.append(tags);
+    if (rule.show && isConstant(o)) td.append(el("span", t.constantNote, "meta"));
+    if (rule.show && mostly !== null) td.append(el("span", t.mostlyVerdict, "meta"));
+    for (const c of state.index.contrasts) {
+      if (c.task_id !== task.id || c.model_id !== model.model_id || c.verdict === "withheld") continue;
+      const ref = state.index.models.get(c.model_ref);
+      const p = new Intl.NumberFormat(state.lang === "pl" ? "pl-PL" : "en-GB", {maximumSignificantDigits:2}).format(c.p_holm);
+      const note = c.subset === "full" ? t.verdictNote : `${t.verdictNoteSubset}, ${t.n} ${c.n}`;
+      const signed = v => `${v > 0 ? "+" : ""}${fmt(v, c.unit)}`;
+      const isNum = v => typeof v === "number" && Number.isFinite(v);
+      const diff = isNum(c.delta) ? `${t.delta} ${signed(c.delta)}${c.ci95 ? ` (${t.ci} ${signed(c.ci95.lower)} – ${signed(c.ci95.upper)})` : ""} · ` : "";
+      td.append(el("span", `${t.verdicts[c.verdict]} ${modelName(ref)}`, "verdict"), el("span", `${diff}${note}: p = ${p}`, "meta"));
+    }
+    const d = el("details");
+    d.append(el("summary", t.details));
+    d.append(el("p", `${t.population}: ${o.n_population ?? "—"} / ${o.n_selected ?? "—"} / ${o.n_scored}`));
+    d.append(el("p", `results_sha256 ${o.results_sha256 ?? "—"}`));
+    if (o.contamination_receipt_sha256) d.append(el("p", `scan receipt ${o.contamination_receipt_sha256}`));
+    td.append(d);
+    return td;
+  }
+
+  // Polish labels for the tasks_50 domain values; the English view shows the value itself.
+  const DOMAIN_PL = {
+    syntax:"składnia", science_reasoning:"rozumowanie naukowe", language_modeling:"modelowanie języka", morphosyntax:"morfoskładnia",
+    commonsense:"zdrowy rozsądek", physical_reasoning:"rozumowanie fizyczne", coreference:"koreferencja", science_knowledge:"wiedza naukowa",
+    reading:"czytanie", causal_reasoning:"rozumowanie przyczynowe", entailment:"wynikanie", word_sense:"znaczenie słowa",
+    adversarial_entailment:"wynikanie (adwersaryjne)", contextual_prediction:"przewidywanie z kontekstu", logical_reasoning:"rozumowanie logiczne",
+    mathematics_mc:"matematyka (wybór)", sentiment:"wydźwięk", topic_classification:"klasyfikacja tematu", question_relevance:"trafność pytania",
+    paraphrase:"parafraza", summary_matching:"dopasowanie streszczenia", toxicity_detection:"wykrywanie toksyczności",
+    sentence_entity_classification:"klasyfikacja zdań i encji", question_answering:"odpowiadanie na pytania", academic_knowledge:"wiedza akademicka",
+    mathematics_generation:"matematyka (generowanie)", competition_mathematics:"matematyka konkursowa", truthfulness:"prawdomówność",
+    instruction_following:"wykonywanie poleceń", native_exam_knowledge:"wiedza z egzaminów krajowych", unanswerable_reading:"czytanie z pytaniami bez odpowiedzi",
+    discrete_reading:"czytanie z liczeniem", conversational_reading:"czytanie w dialogu", closed_book_knowledge:"wiedza bez kontekstu",
+    biomedical_reading:"czytanie biomedyczne", medical_exam_knowledge:"wiedza z egzaminów medycznych", social_reasoning:"rozumowanie społeczne",
+    formal_reasoning:"rozumowanie formalne", adversarial_paraphrase:"parafraza (adwersaryjna)", acceptability:"poprawność zdania", entity_reading:"czytanie o encjach"
+  };
+  const domainLabel = d => (state.lang === "pl" && DOMAIN_PL[d]) || d.replace(/_/g, " ");
+
+  function taskHeader(task, t) {
+    const th = el("th", task.name, "task");
+    th.scope = "row";
+    th.append(el("span", `${task.language} · ${domainLabel(task.domain)}`, "meta"));
+    th.append(el("span", `${task.metric.split(",")[0]} · ${task.higher_is_better ? t.higher : t.lower}`, "meta"));
+    // Public scoring amendments are a property of the pinned benchmark list, never of a result file.
+    for (const code of task.scoring_amendments || []) th.append(el("span", t.amendments[code] || t.amendments.other, "amend"));
+    for (const code of task.scoring_notes || []) th.append(el("span", t.scoringNotes[code] || t.scoringNotes.other, "amend"));
+    const d = el("details");
+    d.append(el("summary", t.details));
+    for (const s of task.license.source_revisions) {
+      const p = el("p");
+      const a = el("a", s.repo_id);
+      a.href = `https://huggingface.co/datasets/${s.repo_id}`;
+      a.rel = "noopener";
+      p.append(a, document.createTextNode(` · ${t.revision} ${s.revision.slice(0, 8)}`));
+      d.append(p);
+    }
+    d.append(el("p", `${t.licence}: ${task.license.declared_tags.join(", ") || t.none}`));
+    d.append(el("p", `${task.split} · ${task.shots}-shot · ${task.instrument}`));
+    th.append(d);
+    return th;
+  }
+
+  function render() {
+    const t = T[state.lang];
+    const app = document.getElementById("app");
+    const pills = document.getElementById("pills");
+    pills.replaceChildren();
+    const pill = (b, txt) => { const p = el("span", undefined, "pill"); p.append(el("b", b), document.createTextNode(` ${txt}`)); pills.append(p); };
+    if (!state.tasks) {
+      app.replaceChildren(el("p", `${t.loadError} (${state.error})`, "notice error"));
+      return;
+    }
+    const rows = state.tasks.rows;
+    // Models whose training corpus was scanned come first; models with unknown training data follow, in packet order.
+    const scanned = m => state.index && [...state.index.observations.values()].some(o => o.model_id === m.model_id && o.contamination !== "unknown");
+    const models = state.packet ? [...state.packet.models].sort((a, b) => scanned(b) - scanned(a)) : [];
+    const published = new Set();
+    if (state.index) for (const o of state.index.observations.values()) if (numberRule(o).show) published.add(o.task_id);
+    pill(rows.length, t.pills.total);
+    pills.append(el("span", t.pills.langs, "pill"));
+    if (models.length) pill(models.length, t.pills.models);
+    pill(published.size, t.pills.published);
+    // Benchmarks where some model has no measurement yet (binding sentence 2026-10-29 20:18).
+    const pendingEl = document.getElementById("pending");
+    const unmeasured = state.index ? rows.filter(r => models.some(m => { const o = state.index.observations.get(contract.key(r.id, m.model_id, state.packet.protocol_id)); return !o || o.state !== "ok"; })).length : 0;
+    pendingEl.hidden = unmeasured === 0;
+    pendingEl.textContent = unmeasured ? t.pending(unmeasured) : "";
+
+    const nodes = [];
+    if (state.error && state.tasks) nodes.push(el("p", `${t.rejected}: ${state.error}`, "notice error"));
+    else if (!state.packet) nodes.push(el("p", t.noPacket, "notice"));
+
+    const controls = el("div", undefined, "controls");
+    const search = el("input"); search.type = "search"; search.value = state.query;
+    const lang = el("select");
+    for (const [v, txt] of [["all", t.all], ["EN", "EN"], ["PL", "PL"]]) { const op = el("option", txt); op.value = v; lang.append(op); }
+    lang.value = state.language;
+    const show = el("select");
+    for (const [v, txt] of [["all", t.all], ["published", t.withResults]]) { const op = el("option", txt); op.value = v; show.append(op); }
+    show.value = state.show;
+    show.disabled = !state.packet;
+    const clear = el("button", t.clear, "clear"); clear.type = "button";
+    const lab = (txt, input) => { const l = el("label", txt); l.append(input); return l; };
+    controls.append(lab(t.search, search), lab(t.language, lang), lab(t.show, show), clear);
+    const summary = el("p", undefined, "summary");
+    summary.setAttribute("role", "status");
+    const wrap = el("div", undefined, "wrap");
+    wrap.tabIndex = 0;
+    wrap.setAttribute("role", "region");
+    wrap.setAttribute("aria-label", t.results);
+    const table = el("table");
+    table.setAttribute("aria-labelledby", "results-title");
+    const head = el("tr");
+    const th0 = el("th", t.benchmark); th0.scope = "col"; head.append(th0);
+    if (models.length) for (const m of models) {
+      const th = el("th", modelName(m)); th.scope = "col";
+      th.append(el("span", m.revision ? `${t.revision} ${m.revision.slice(0, 8)}` : "", "meta"));
+      head.append(th);
+    } else { const th = el("th", t.results); th.scope = "col"; head.append(th); }
+    const thead = el("thead"); thead.append(head);
+    const tbody = el("tbody");
+    table.append(thead, tbody);
+    wrap.append(table);
+    const empty = el("p", t.empty, "summary");
+    nodes.push(controls, summary, wrap, empty);
+    app.replaceChildren(...nodes);
+
+    function fill() {
+      const q = state.query.trim().toLocaleLowerCase(state.lang);
+      const visible = rows.filter(r => (state.language === "all" || r.language === state.language) &&
+        (state.show === "all" || published.has(r.id)) &&
+        `${r.name} ${r.benchmark_id} ${r.domain}`.toLocaleLowerCase(state.lang).includes(q));
+      const frag = document.createDocumentFragment();
+      for (const r of visible) {
+        const tr = el("tr");
+        tr.append(taskHeader(r, t));
+        if (models.length) for (const m of models) tr.append(cell(r, m, t));
+        else tr.append(el("td", t.notPublished, "state"));
+        frag.append(tr);
+      }
+      tbody.replaceChildren(frag);
+      summary.textContent = `${t.shown} ${visible.length} ${t.of} ${rows.length} ${t.benchmarks}`;
+      empty.hidden = visible.length !== 0;
+      wrap.hidden = visible.length === 0;
+    }
+    search.addEventListener("input", () => { state.query = search.value; fill(); });
+    lang.addEventListener("change", () => { state.language = lang.value; fill(); });
+    show.addEventListener("change", () => { state.show = show.value; fill(); });
+    clear.addEventListener("click", () => { state.query = ""; state.language = "all"; state.show = "all"; search.value = ""; lang.value = "all"; show.value = "all"; fill(); search.focus(); });
+    fill();
+
+    const files = document.getElementById("files");
+    files.replaceChildren(el("p", t.files));
+    for (const f of state.files) { const p = el("p"); p.append(el("code", `${f.name} · sha256 ${f.sha}`)); files.append(p); }
+    if (state.packet) files.append(el("p", `${t.packet}: ${t.run} ${state.packet.run_id} · ${t.protocol} ${state.packet.protocol_id}`));
+  }
+
+  async function read(path, pin) {
+    const res = await fetch(path, {cache:"no-store"});
+    if (!res.ok) throw new Error(`${path}: HTTP ${res.status}`);
+    const bytes = await res.arrayBuffer();
+    const digest = await crypto.subtle.digest("SHA-256", bytes);
+    const sha = [...new Uint8Array(digest)].map(b => b.toString(16).padStart(2, "0")).join("");
+    if (pin && sha !== pin) throw new Error(`${path}: pinned bytes changed`);
+    state.files.push({name:path, sha});
+    return {bytes, sha};
+  }
+  const json = r => JSON.parse(new TextDecoder().decode(r.bytes));
+
+  function setLang(l) {
+    state.lang = l === "pl" ? "pl" : "en";
+    document.body.dataset.lang = state.lang;
+    document.documentElement.lang = state.lang;
+    document.querySelectorAll(".lang button").forEach(b => b.setAttribute("aria-pressed", String(b.dataset.set === state.lang)));
+    try { localStorage.setItem("slayer-lang", state.lang); } catch (e) {}
+    render();
+  }
+
+  async function start() {
+    try {
+      const tasks = await read("tasks_50.json", contract.PINS.tasks);
+      await read("results.schema.json", contract.PINS.schema);
+      await read("METHOD.md", contract.PINS.method);
+      await read("contamination_field_roles.json", contract.PINS.fieldRoles);
+      state.tasks = json(tasks);
+      if (state.tasks.rows.length !== 50) throw new Error("tasks_50.json: expected 50 rows");
+    } catch (e) { state.error = e.message; state.tasks = null; return; }
+    try {
+      const feed = json(await read("results-feed.json"));
+      if (feed.results !== null) {
+        if (feed.results !== "results.json") throw new Error("only results.json next to this page is allowed");
+        // The feed names the exact reviewed packet; any other bytes are refused.
+        if (!/^[0-9a-f]{64}$/.test(feed.sha256 || "")) throw new Error("results-feed.json: missing sha256 of results.json");
+        const res = await read("results.json", feed.sha256);
+        const packet = json(res);
+        state.index = contract.validate(packet, state.tasks);
+        state.packet = packet;
+        state.packetSHA = res.sha;
+      }
+    } catch (e) { state.error = e.message; state.packet = null; state.index = null; }
+  }
+
+  document.querySelectorAll(".lang button").forEach(b => b.addEventListener("click", () => setLang(b.dataset.set)));
+  let saved = null; try { saved = localStorage.getItem("slayer-lang"); } catch (e) {}
+  const q = new URLSearchParams(location.search).get("lang");
+  start().finally(() => setLang(q || saved || "en"));
+})();

contamination_field_roles.json

@@ -0,0 +1,779 @@
+{
+  "release": "bench50-v0.3.1",
+  "kind": "contamination_field_roles",
+  "scan": {
+    "receipt_sha256": "bc6c168cf6ef35bce80b70f3bf582853526b558cb7f37e30101e35f3c2342de7",
+    "benchmarks_section_sha256_prefix": "8be608af",
+    "scanner_sha256": "685fa9ef1fbd6e9a680be53da7fd7d943e3c1f2cde6ef69b24271027df74dad2"
+  },
+  "rule": {
+    "context": "a supporting document shown to the model but separate from the question and the answer (passage, context, story, article, support)",
+    "answer_bearing": "the question, the options, the gold answer or its explanation, or the scored text itself (items with no separate document: minimal pairs, sentence classification, continuation, perplexity pages)",
+    "metadata": "annotation tags that are not shown to the model as text; hits are ignored in every variant and listed with counts",
+    "variant_a": "items with no hit in any context or answer_bearing field",
+    "variant_b": "items with no hit in any answer_bearing field",
+    "extractive_qa": "answer span fields are answer_bearing even where the span is copied from the passage (SQuAD2, PoQuAD, CoQA, DROP): variant (b) is strict there and the report says so",
+    "assigned": "from the meaning of each field in the pinned task definition, before any variant value was computed"
+  },
+  "justification": {
+    "extractive_qa": "An answer span copied from the passage is still the gold answer; one rule for every task type is preferred to a type-dependent exception, so variant (b) is stricter for extractive QA and the report says so.",
+    "scored_text": "Where a benchmark has no separate document, the scored text is the item itself, so a hit there is a hit on the answer.",
+    "metadata": "Morphological tags are not shown to the model as text; their hits are exact-short matches of tag strings and do not indicate exposure."
+  },
+  "counts": {
+    "context": 12,
+    "answer_bearing": 93,
+    "metadata": 2
+  },
+  "roles": [
+    {
+      "task_id": "T44",
+      "benchmark_id": "pubmedqa",
+      "field": "CONTEXTS[]",
+      "hit_items": 4,
+      "role": "context"
+    },
+    {
+      "task_id": "T44",
+      "benchmark_id": "pubmedqa",
+      "field": "LONG_ANSWER",
+      "hit_items": 1,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T19",
+      "benchmark_id": "mathqa",
+      "field": "Problem",
+      "hit_items": 74,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T44",
+      "benchmark_id": "pubmedqa",
+      "field": "QUESTION",
+      "hit_items": 2,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T19",
+      "benchmark_id": "mathqa",
+      "field": "Rationale",
+      "hit_items": 14,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T28",
+      "benchmark_id": "cbd",
+      "field": "TEXT",
+      "hit_items": 26,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T41",
+      "benchmark_id": "coqa",
+      "field": "additional_answers.0.input_text[]",
+      "hit_items": 10,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T41",
+      "benchmark_id": "coqa",
+      "field": "additional_answers.0.span_text[]",
+      "hit_items": 39,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T41",
+      "benchmark_id": "coqa",
+      "field": "additional_answers.1.input_text[]",
+      "hit_items": 7,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T41",
+      "benchmark_id": "coqa",
+      "field": "additional_answers.1.span_text[]",
+      "hit_items": 28,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T41",
+      "benchmark_id": "coqa",
+      "field": "additional_answers.2.input_text[]",
+      "hit_items": 6,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T41",
+      "benchmark_id": "coqa",
+      "field": "additional_answers.2.span_text[]",
+      "hit_items": 22,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T25",
+      "benchmark_id": "dyk",
+      "field": "answer",
+      "hit_items": 799,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T43",
+      "benchmark_id": "triviaqa",
+      "field": "answer.aliases[]",
+      "hit_items": 1137,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T43",
+      "benchmark_id": "triviaqa",
+      "field": "answer.normalized_aliases[]",
+      "hit_items": 776,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T43",
+      "benchmark_id": "triviaqa",
+      "field": "answer.normalized_value",
+      "hit_items": 28,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T43",
+      "benchmark_id": "triviaqa",
+      "field": "answer.value",
+      "hit_items": 62,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T46",
+      "benchmark_id": "social_iqa",
+      "field": "answerA",
+      "hit_items": 2,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T46",
+      "benchmark_id": "social_iqa",
+      "field": "answerB",
+      "hit_items": 4,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T42",
+      "benchmark_id": "nq_open",
+      "field": "answer[]",
+      "hit_items": 64,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T41",
+      "benchmark_id": "coqa",
+      "field": "answers.input_text[]",
+      "hit_items": 4,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T41",
+      "benchmark_id": "coqa",
+      "field": "answers.span_text[]",
+      "hit_items": 36,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T31",
+      "benchmark_id": "poquad",
+      "field": "answers.text[]",
+      "hit_items": 731,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T38",
+      "benchmark_id": "squad2",
+      "field": "answers.text[]",
+      "hit_items": 34,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T30",
+      "benchmark_id": "polqa",
+      "field": "answers[]",
+      "hit_items": 2,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T37",
+      "benchmark_id": "llmzszl",
+      "field": "answers[]",
+      "hit_items": 555,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T40",
+      "benchmark_id": "race",
+      "field": "article",
+      "hit_items": 75,
+      "role": "context"
+    },
+    {
+      "task_id": "T04",
+      "benchmark_id": "multiblimp",
+      "field": "attractors[].xpos",
+      "hit_items": 3,
+      "role": "metadata"
+    },
+    {
+      "task_id": "T11",
+      "benchmark_id": "copa",
+      "field": "choice1",
+      "hit_items": 6,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T11",
+      "benchmark_id": "copa",
+      "field": "choice2",
+      "hit_items": 11,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T02",
+      "benchmark_id": "arc",
+      "field": "choices.text[]",
+      "hit_items": 33,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T08",
+      "benchmark_id": "openbookqa",
+      "field": "choices.text[]",
+      "hit_items": 48,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T32",
+      "benchmark_id": "mmlu",
+      "field": "choices[]",
+      "hit_items": 257,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T04",
+      "benchmark_id": "multiblimp",
+      "field": "congruent_attractors[].xpos",
+      "hit_items": 3,
+      "role": "metadata"
+    },
+    {
+      "task_id": "T31",
+      "benchmark_id": "poquad",
+      "field": "context",
+      "hit_items": 5764,
+      "role": "context"
+    },
+    {
+      "task_id": "T38",
+      "benchmark_id": "squad2",
+      "field": "context",
+      "hit_items": 3097,
+      "role": "context"
+    },
+    {
+      "task_id": "T05",
+      "benchmark_id": "hellaswag",
+      "field": "ctx",
+      "hit_items": 21,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T05",
+      "benchmark_id": "hellaswag",
+      "field": "ctx_a",
+      "hit_items": 22,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T05",
+      "benchmark_id": "hellaswag",
+      "field": "endings[]",
+      "hit_items": 28,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T45",
+      "benchmark_id": "medmcqa",
+      "field": "exp",
+      "hit_items": 33,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T27",
+      "benchmark_id": "psc",
+      "field": "extract_text",
+      "hit_items": 2,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T06",
+      "benchmark_id": "piqa",
+      "field": "goal",
+      "hit_items": 11,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T15",
+      "benchmark_id": "rte",
+      "field": "hypothesis",
+      "hit_items": 1,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T47",
+      "benchmark_id": "bbh",
+      "field": "input",
+      "hit_items": 22,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T35",
+      "benchmark_id": "truthfulqa",
+      "field": "mc1_targets.choices[]",
+      "hit_items": 35,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T35",
+      "benchmark_id": "truthfulqa",
+      "field": "mc2_targets.choices[]",
+      "hit_items": 48,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T04",
+      "benchmark_id": "multiblimp",
+      "field": "metadata.text",
+      "hit_items": 164,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T45",
+      "benchmark_id": "medmcqa",
+      "field": "opa",
+      "hit_items": 46,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T45",
+      "benchmark_id": "medmcqa",
+      "field": "opb",
+      "hit_items": 41,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T45",
+      "benchmark_id": "medmcqa",
+      "field": "opc",
+      "hit_items": 38,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T45",
+      "benchmark_id": "medmcqa",
+      "field": "opd",
+      "hit_items": 85,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T19",
+      "benchmark_id": "mathqa",
+      "field": "options",
+      "hit_items": 5,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T03",
+      "benchmark_id": "wikitext2",
+      "field": "page",
+      "hit_items": 21,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T10",
+      "benchmark_id": "boolq",
+      "field": "passage",
+      "hit_items": 637,
+      "role": "context"
+    },
+    {
+      "task_id": "T39",
+      "benchmark_id": "drop",
+      "field": "passage",
+      "hit_items": 2327,
+      "role": "context"
+    },
+    {
+      "task_id": "T50",
+      "benchmark_id": "record",
+      "field": "passage",
+      "hit_items": 189,
+      "role": "context"
+    },
+    {
+      "task_id": "T30",
+      "benchmark_id": "polqa",
+      "field": "passage_text",
+      "hit_items": 12667,
+      "role": "context"
+    },
+    {
+      "task_id": "T30",
+      "benchmark_id": "polqa",
+      "field": "passage_title",
+      "hit_items": 19,
+      "role": "context"
+    },
+    {
+      "task_id": "T30",
+      "benchmark_id": "polqa",
+      "field": "passage_wiki",
+      "hit_items": 13050,
+      "role": "context"
+    },
+    {
+      "task_id": "T04",
+      "benchmark_id": "multiblimp",
+      "field": "prefix",
+      "hit_items": 41,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T11",
+      "benchmark_id": "copa",
+      "field": "premise",
+      "hit_items": 9,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T15",
+      "benchmark_id": "rte",
+      "field": "premise",
+      "hit_items": 9,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T16",
+      "benchmark_id": "anli",
+      "field": "premise",
+      "hit_items": 19,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T34",
+      "benchmark_id": "math",
+      "field": "problem",
+      "hit_items": 34,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T40",
+      "benchmark_id": "race",
+      "field": "problems[].question",
+      "hit_items": 1,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T36",
+      "benchmark_id": "ifeval",
+      "field": "prompt",
+      "hit_items": 2,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T50",
+      "benchmark_id": "record",
+      "field": "query",
+      "hit_items": 36,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T02",
+      "benchmark_id": "arc",
+      "field": "question",
+      "hit_items": 15,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T09",
+      "benchmark_id": "sciq",
+      "field": "question",
+      "hit_items": 15,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T10",
+      "benchmark_id": "boolq",
+      "field": "question",
+      "hit_items": 1,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T30",
+      "benchmark_id": "polqa",
+      "field": "question",
+      "hit_items": 20,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T31",
+      "benchmark_id": "poquad",
+      "field": "question",
+      "hit_items": 563,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T32",
+      "benchmark_id": "mmlu",
+      "field": "question",
+      "hit_items": 405,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T35",
+      "benchmark_id": "truthfulqa",
+      "field": "question",
+      "hit_items": 16,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T37",
+      "benchmark_id": "llmzszl",
+      "field": "question",
+      "hit_items": 235,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T38",
+      "benchmark_id": "squad2",
+      "field": "question",
+      "hit_items": 9,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T39",
+      "benchmark_id": "drop",
+      "field": "question",
+      "hit_items": 4,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T42",
+      "benchmark_id": "nq_open",
+      "field": "question",
+      "hit_items": 13,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T43",
+      "benchmark_id": "triviaqa",
+      "field": "question",
+      "hit_items": 161,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T45",
+      "benchmark_id": "medmcqa",
+      "field": "question",
+      "hit_items": 34,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T46",
+      "benchmark_id": "social_iqa",
+      "field": "question",
+      "hit_items": 3,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T08",
+      "benchmark_id": "openbookqa",
+      "field": "question_stem",
+      "hit_items": 26,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T41",
+      "benchmark_id": "coqa",
+      "field": "questions.input_text[]",
+      "hit_items": 1,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T04",
+      "benchmark_id": "multiblimp",
+      "field": "sen",
+      "hit_items": 162,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T21",
+      "benchmark_id": "sst2",
+      "field": "sentence",
+      "hit_items": 6,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T22",
+      "benchmark_id": "polemo2",
+      "field": "sentence",
+      "hit_items": 27,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T23",
+      "benchmark_id": "8tags",
+      "field": "sentence",
+      "hit_items": 243,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T29",
+      "benchmark_id": "klej_ner",
+      "field": "sentence",
+      "hit_items": 138,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T49",
+      "benchmark_id": "cola",
+      "field": "sentence",
+      "hit_items": 44,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T13",
+      "benchmark_id": "wic",
+      "field": "sentence1",
+      "hit_items": 156,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T48",
+      "benchmark_id": "paws",
+      "field": "sentence1",
+      "hit_items": 14,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T13",
+      "benchmark_id": "wic",
+      "field": "sentence2",
+      "hit_items": 168,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T48",
+      "benchmark_id": "paws",
+      "field": "sentence2",
+      "hit_items": 16,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T26",
+      "benchmark_id": "ppc",
+      "field": "sentence_A",
+      "hit_items": 223,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T26",
+      "benchmark_id": "ppc",
+      "field": "sentence_B",
+      "hit_items": 185,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T01",
+      "benchmark_id": "blimp",
+      "field": "sentence_bad",
+      "hit_items": 138,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T01",
+      "benchmark_id": "blimp",
+      "field": "sentence_good",
+      "hit_items": 226,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T06",
+      "benchmark_id": "piqa",
+      "field": "sol1",
+      "hit_items": 48,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T06",
+      "benchmark_id": "piqa",
+      "field": "sol2",
+      "hit_items": 40,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T34",
+      "benchmark_id": "math",
+      "field": "solution",
+      "hit_items": 42,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T41",
+      "benchmark_id": "coqa",
+      "field": "story",
+      "hit_items": 67,
+      "role": "context"
+    },
+    {
+      "task_id": "T27",
+      "benchmark_id": "psc",
+      "field": "summary_text",
+      "hit_items": 5,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T09",
+      "benchmark_id": "sciq",
+      "field": "support",
+      "hit_items": 79,
+      "role": "context"
+    },
+    {
+      "task_id": "T14",
+      "benchmark_id": "wsc273",
+      "field": "text",
+      "hit_items": 4,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T17",
+      "benchmark_id": "lambada",
+      "field": "text",
+      "hit_items": 1,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T39",
+      "benchmark_id": "drop",
+      "field": "validated_answers.spans[][]",
+      "hit_items": 3,
+      "role": "answer_bearing"
+    },
+    {
+      "task_id": "T04",
+      "benchmark_id": "multiblimp",
+      "field": "wrong_sen",
+      "hit_items": 58,
+      "role": "answer_bearing"
+    }
+  ]
+}

contract.js

@@ -0,0 +1,187 @@
+"use strict";
+/* Acceptance checks for the public bench50-v0.3.1 result packet (results.schema.json, schema_version 2).
+   The exporter validates the full JSON Schema before publication; this file re-checks, in the browser,
+   every rule the page relies on to decide what may be shown. Anything unexpected rejects the whole packet. */
+(() => {
+  const PINS = {
+    tasks: "fcfc45d805243e7ea4aa12bf836940ea086ecafe9fe3e592bf8850890ca8cf6b",
+    schema: "c133a37d4038912314b14d374cd3ae1ecc6d8bca3d88a465f7e1243437553eec",
+    method: "8709632a394a3d69fa56567404e6543f7305d49fc97c9d381fbfad46d86628ae",
+    fieldRoles: "de4280e9db5a947e4e569ad4e5dfcb048d6ee574014b640d254b6c8e1a604b35"
+  };
+  const HEX64 = /^[0-9a-f]{64}$/;
+  const STATES = ["ok","not_run","blocked_access","blocked_runtime","blocked_security","blocked_judge","unsupported","failed","skipped"];
+  const CONTAMINATION = ["clean","flagged","partially_scanned","unscanned","unknown"];
+  const UNITS = ["percent","score","byte_ppl"];
+  const VERDICTS = ["withheld","better","worse","inconclusive"];
+  const SUBSETS = ["full","hit_free","hit_free_answer_fields"];
+  const OBS_KEYS = ["task_id","model_id","protocol_id","metric","unit","higher_is_better","n_population","n_selected","n_scored","state","value","ci95",
+    "source_id_digest","results_sha256","runner_sha256","reason","contamination","contamination_receipt_sha256","value_hit_free","n_hit_free","hit_rate",
+    "value_hit_free_answer_fields","n_hit_free_answer_fields","hit_rate_answer_fields","ci95_hit_free","ci95_hit_free_answer_fields","reference"];
+  const REQUIRED_OBS = ["task_id","model_id","protocol_id","metric","unit","higher_is_better","n_population","n_selected","n_scored","state","value","ci95",
+    "source_id_digest","results_sha256","runner_sha256","contamination","contamination_receipt_sha256"];
+
+  function fail(message) { throw new Error(message); }
+  const isNum = v => typeof v === "number" && Number.isFinite(v);
+  const isShare = v => isNum(v) && v >= 0 && v <= 1;
+  function key(task, model, protocol) { return `${task}\u0000${model}\u0000${protocol}`; }
+
+  function checkCI(ci, where) {
+    if (ci === null) return;
+    if (typeof ci !== "object") fail(`${where}: ci95 must be an object or null`);
+    for (const k of Object.keys(ci)) if (!["lower","upper","level","method","replicates","seed"].includes(k)) fail(`${where}: unknown ci95 field ${k}`);
+    if (!isNum(ci.lower) || !isNum(ci.upper) || ci.lower > ci.upper) fail(`${where}: invalid ci95 bounds`);
+    if (ci.level !== 0.95 || ci.replicates !== 10000 || ci.seed !== 20261028) fail(`${where}: ci95 does not follow the protocol`);
+    if (!["paired_bootstrap","paired_cluster_bootstrap"].includes(ci.method)) fail(`${where}: unknown ci95 method`);
+  }
+
+  function checkReference(ref, where) {
+    if (ref === undefined || ref === null) return;
+    if (typeof ref !== "object") fail(`${where}: reference must be an object or null`);
+    for (const k of Object.keys(ref)) if (!["chance_value","majority_value","majority_prediction_share","constant_prediction"].includes(k)) fail(`${where}: unknown reference field ${k}`);
+    for (const k of ["chance_value","majority_value"]) if (!(k in ref) || !(ref[k] === null || isNum(ref[k]))) fail(`${where}: reference.${k} must be a number or null`);
+    if (!("majority_prediction_share" in ref) || !(ref.majority_prediction_share === null || isShare(ref.majority_prediction_share))) fail(`${where}: reference.majority_prediction_share must be 0..1 or null`);
+    if (!("constant_prediction" in ref) || !(ref.constant_prediction === null || typeof ref.constant_prediction === "boolean")) fail(`${where}: reference.constant_prediction must be boolean or null`);
+  }
+
+  function checkObservation(o, i, tasks, models, protocols) {
+    const where = `observation ${i}`;
+    if (!o || typeof o !== "object") fail(`${where}: not an object`);
+    for (const k of Object.keys(o)) if (!OBS_KEYS.includes(k)) fail(`${where}: unknown field ${k}`);
+    for (const k of REQUIRED_OBS) if (!(k in o)) fail(`${where}: missing ${k}`);
+    if (!tasks.has(o.task_id)) fail(`${where}: task ${o.task_id} is not in the benchmark list`);
+    if (!models.has(o.model_id)) fail(`${where}: model ${o.model_id} is not declared`);
+    if (!protocols.includes(o.protocol_id)) fail(`${where}: unknown protocol ${o.protocol_id}`);
+    if (!STATES.includes(o.state)) fail(`${where}: unknown state`);
+    if (!CONTAMINATION.includes(o.contamination)) fail(`${where}: unknown contamination state`);
+    if (!UNITS.includes(o.unit)) fail(`${where}: unknown unit`);
+    if (typeof o.higher_is_better !== "boolean") fail(`${where}: higher_is_better must be boolean`);
+    const task = tasks.get(o.task_id);
+    if (o.unit !== task.unit || o.higher_is_better !== task.higher_is_better) fail(`${where}: unit or direction differs from the benchmark list`);
+    if (!Number.isInteger(o.n_scored) || o.n_scored < 0) fail(`${where}: invalid n_scored`);
+    for (const k of ["n_population","n_selected"]) if (!(o[k] === null || (Number.isInteger(o[k]) && o[k] >= 0))) fail(`${where}: invalid ${k}`);
+    checkCI(o.ci95, where);
+    checkReference(o.reference, where);
+    if (o.state === "ok") {
+      if (!isNum(o.value) || o.n_scored < 1) fail(`${where}: ok without a value`);
+      for (const k of ["source_id_digest","results_sha256","runner_sha256"]) if (!HEX64.test(o[k] || "")) fail(`${where}: ok without ${k}`);
+      if (o.ci95 !== null && (o.value < o.ci95.lower || o.value > o.ci95.upper)) fail(`${where}: value outside its interval`);
+    } else if (o.value !== null || o.ci95 !== null) fail(`${where}: a non-ok state carries a value`);
+    if (["clean","flagged","partially_scanned"].includes(o.contamination) && !HEX64.test(o.contamination_receipt_sha256 || "")) fail(`${where}: scan state without a receipt`);
+    const variants = [["value_hit_free","n_hit_free","hit_rate","ci95_hit_free"], ["value_hit_free_answer_fields","n_hit_free_answer_fields","hit_rate_answer_fields","ci95_hit_free_answer_fields"]];
+    for (const [v, n, r, c] of variants) {
+      const ci = o[c] === undefined ? null : o[c];
+      checkCI(ci, `${where} ${c}`);
+      const given = [o[v], o[n], o[r]].some(x => x !== undefined && x !== null);
+      if (o.contamination === "flagged" && o.state === "ok" && !given) fail(`${where}: flagged without ${n}`);
+      if (!given) { if (ci !== null) fail(`${where}: ${c} without ${v}`); continue; }
+      if (!Number.isInteger(o[n]) || o[n] < 0 || !isShare(o[r])) fail(`${where}: invalid ${n} or ${r}`);
+      if ((o[n] === 0) !== (o[v] === null || o[v] === undefined)) fail(`${where}: ${v} must be empty exactly when ${n} is 0`);
+      if (o[n] > 0 && !isNum(o[v])) fail(`${where}: invalid ${v}`);
+      if (o.n_scored && o[n] > o.n_scored) fail(`${where}: ${n} exceeds n_scored`);
+      if (ci !== null && (o[n] === 0 || o[v] < ci.lower || o[v] > ci.upper)) fail(`${where}: ${v} outside ${c}`);
+    }
+  }
+
+  function validate(packet, taskList) {
+    if (!packet || typeof packet !== "object") fail("packet is not an object");
+    for (const k of Object.keys(packet)) if (!["schema_version","protocol_id","run_id","models","tasks","observations","contrasts","provenance"].includes(k)) fail(`unknown top-level field ${k}`);
+    if (packet.schema_version !== 2) fail("schema_version must be 2");
+    const protocols = ["bench50-v0.3.1-panel200","bench50-v0.3.1-fullB"];
+    if (!protocols.includes(packet.protocol_id)) fail("unknown protocol_id");
+    if (typeof packet.run_id !== "string" || !packet.run_id) fail("missing run_id");
+    const p = packet.provenance;
+    if (!p || p.kind !== "measurement") fail("only measurement packets are shown");
+    if (!/^[0-9a-f]{40}$/.test(p.inventory_commit || "")) fail("provenance.inventory_commit missing");
+    if (!Array.isArray(p.sources) || !p.sources.length || !p.sources.every(s => s && typeof s.run_id === "string" && HEX64.test(s.results_sha256 || ""))) fail("provenance.sources missing");
+    if (!Array.isArray(packet.models) || !packet.models.length) fail("no models");
+    const models = new Map();
+    for (const m of packet.models) {
+      if (!m || typeof m.model_id !== "string" || !m.model_id || typeof m.repo !== "string" || !m.repo) fail("invalid model entry");
+      if (models.has(m.model_id)) fail(`duplicate model ${m.model_id}`);
+      if (!(m.weights_sha256 === null || HEX64.test(m.weights_sha256))) fail(`invalid weights_sha256 for ${m.model_id}`);
+      models.set(m.model_id, m);
+    }
+    const tasks = new Map(taskList.rows.map(r => [r.id, r]));
+    if (!Array.isArray(packet.tasks)) fail("tasks must be a list");
+    for (const t of packet.tasks) {
+      const row = tasks.get(t && t.task_id);
+      if (!row) fail(`task ${t && t.task_id} is not in the benchmark list`);
+      if (t.language !== row.language) fail(`task ${t.task_id}: language differs from the benchmark list`);
+    }
+    if (!Array.isArray(packet.observations)) fail("observations must be a list");
+    const observations = new Map();
+    packet.observations.forEach((o, i) => {
+      checkObservation(o, i, tasks, models, protocols);
+      const k = key(o.task_id, o.model_id, o.protocol_id);
+      if (observations.has(k)) fail(`duplicate observation ${o.task_id} / ${o.model_id}`);
+      observations.set(k, o);
+    });
+    if (!Array.isArray(packet.contrasts)) fail("contrasts must be a list");
+    const contrasts = [];
+    packet.contrasts.forEach((c, i) => {
+      const where = `contrast ${i}`;
+      if (!c || !VERDICTS.includes(c.verdict)) fail(`${where}: unknown verdict`);
+      if (!SUBSETS.includes(c.subset)) fail(`${where}: unknown subset`);
+      if (!(c.n === null || (Number.isInteger(c.n) && c.n >= 0))) fail(`${where}: invalid n`);
+      if (typeof c.metric !== "string" || !c.metric || !UNITS.includes(c.unit)) fail(`${where}: missing metric or unit`);
+      const left = observations.get(key(c.task_id, c.model_id, c.protocol_id));
+      const right = observations.get(key(c.task_id, c.model_ref, c.protocol_id));
+      if (!left || !right) fail(`${where}: refers to a missing observation`);
+      if (c.unit !== left.unit) fail(`${where}: unit differs from its observation`);
+      checkCI(c.ci95, where);
+      if (c.verdict !== "withheld") {
+        const r = c.review || {};
+        if (r.status !== "accepted" || typeof r.actor !== "string" || !r.actor || !HEX64.test(r.artifact_sha256 || "")) fail(`${where}: verdict without an accepted review`);
+        if (!isNum(c.delta) || !isShare(c.p_holm) || !Number.isInteger(c.family_size) || c.family_size < 1) fail(`${where}: verdict without paired test numbers`);
+        if (!Number.isInteger(c.n) || c.n < 1) fail(`${where}: verdict without n`);
+        if (left.state !== "ok" || right.state !== "ok") fail(`${where}: verdict on a non-ok observation`);
+        checkVerdict(c, left, right, observations, where);
+      }
+      contrasts.push(c);
+    });
+    return {models, tasks, observations, contrasts};
+  }
+
+  // Hit-free values carry a number only with at least this many items (method rule, 2026-10-29).
+  const MIN_HIT_FREE_N = 50;
+  const ALPHA = 0.05;
+  const enough = n => Number.isInteger(n) && n >= MIN_HIT_FREE_N;
+  const SHOWN = ["clean","flagged","unknown"];
+  // Highest hit rate in a row where some flagged model keeps fewer than MIN_HIT_FREE_N items without any hits ("most items
+  // found in training data"); null outside that class.
+  function mostlyTrained(observations, taskId, protocolId) {
+    let rate = null;
+    for (const o of observations.values()) {
+      if (o.task_id === taskId && o.protocol_id === protocolId && o.state === "ok" && o.contamination === "flagged" && !enough(o.n_hit_free)) rate = Math.max(rate ?? 0, o.hit_rate);
+    }
+    return rate;
+  }
+  // One rule set with the exporter (_check_verdict, agents 31b34ca6), applied to every non-withheld verdict: no verdict
+  // against a constant answer or a cell without a number (the exporter publishes no contrast for such pairs); the
+  // verdict-bearing subset is full, or hit_free_answer_fields in the "most items in training data" row class, any other
+  // subset stays withheld; directional exactly when p_holm <= ALPHA and n >= MIN_HIT_FREE_N (on the answer-field subset
+  // also n_hit_free_answer_fields of model_id), with the sign of delta (model_id minus model_ref) matching the verdict and
+  // higher_is_better; otherwise inconclusive.
+  function checkVerdict(c, left, right, observations, where) {
+    for (const o of [left, right]) {
+      if (!SHOWN.includes(o.contamination)) fail(`${where}: verdict against a cell without a number`);
+      if (o.reference && o.reference.constant_prediction === true) fail(`${where}: verdict against a constant answer`);
+    }
+    const bearing = mostlyTrained(observations, c.task_id, c.protocol_id) !== null ? "hit_free_answer_fields" : "full";
+    if (c.subset !== bearing) fail(`${where}: subset ${c.subset} is a diagnostic in this row (verdicts only on ${bearing})`);
+    const nOk = enough(c.n) && (c.subset !== "hit_free_answer_fields" || enough(left.n_hit_free_answer_fields));
+    const significant = c.p_holm <= ALPHA && nOk;
+    if (c.verdict === "inconclusive") {
+      if (significant) fail(`${where}: inconclusive although p_holm ${c.p_holm} <= ${ALPHA} with n >= ${MIN_HIT_FREE_N}`);
+      return;
+    }
+    if (!significant) fail(`${where}: verdict ${c.verdict} needs p_holm <= ${ALPHA} and n >= ${MIN_HIT_FREE_N}`);
+    if (c.delta === 0) fail(`${where}: verdict without a delta`);
+    const better = (c.delta > 0) === left.higher_is_better;
+    if (c.verdict !== (better ? "better" : "worse")) fail(`${where}: verdict ${c.verdict} contradicts delta ${c.delta}`);
+  }
+
+  const api = {PINS, MIN_HIT_FREE_N, key, validate, mostlyTrained};
+  if (typeof module !== "undefined" && module.exports) module.exports = api; else window.BenchContract = api;
+})();

index.html

@@ -1,19 +1,64 @@
 <!doctype html>
-<html>
-	<head>
-		<meta charset="utf-8" />
-		<meta name="viewport" content="width=device-width" />
-		<title>My static Space</title>
-		<link rel="stylesheet" href="style.css" />
-	</head>
-	<body>
-		<div class="card">
-			<h1>Welcome to your static Space!</h1>
-			<p>You can modify this app directly by editing <i>index.html</i> in the Files and versions tab.</p>
-			<p>
-				Also don't forget to check the
-				<a href="https://huggingface.co/docs/hub/spaces" target="_blank">Spaces documentation</a>.
-			</p>
-		</div>
-	</body>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>Slayer1B Benchmarks</title>
+<meta name="description" content="Fifty distinct public benchmarks for Slayer 1B, with contamination scans, confidence intervals and reference baselines.">
+<link rel="stylesheet" href="style.css">
+<script src="contract.js" defer></script>
+<script src="app.js" defer></script>
+</head>
+<body data-lang="en">
+<nav class="lang" aria-label="Language"><button type="button" data-set="en" aria-pressed="true">EN</button><button type="button" data-set="pl" aria-pressed="false">PL</button></nav>
+<main>
+  <header>
+    <p class="by">Fabryka AI · SlayerLab</p>
+    <h1>Slayer1B Benchmarks</h1>
+    <p class="lead l-en">Fifty distinct public benchmarks (38 English, 12 Polish) run with one protocol on Slayer 1B and comparison models. For models whose training data we have, a number appears only after a contamination scan; models with unpublished training data are shown with the label TRAINING DATA UNKNOWN. Every main value and every contrast has a 95% confidence interval; tasks scored by picking one answer also have a reference point; values on items without hits are shown without an interval. There is no average over the fifty.</p>
+    <p class="lead l-pl">Pięćdziesiąt różnych publicznych benchmarków (38 angielskich, 12 polskich), liczonych jednym protokołem na Slayerze 1B i modelach porównawczych. Dla modeli, których dane treningowe mamy, liczba pojawia się dopiero po skanie kontaminacji; modele z niepublikowanymi danymi treningowymi mają etykietę DANE TRENINGOWE NIEZNANE. Każda wartość główna i każdy kontrast ma 95% przedział ufności; zadania oceniane wyborem jednej odpowiedzi mają też punkt odniesienia; wartości na pozycjach bez trafień są bez przedziału. Nie liczymy średniej z pięćdziesięciu.</p>
+    <div class="pills" id="pills" aria-live="polite"></div>
+    <p class="pending" id="pending" hidden></p>
+  </header>
+
+  <section aria-labelledby="results-title">
+    <h2 id="results-title"><span class="l-en">Results</span><span class="l-pl">Wyniki</span></h2>
+    <div id="app"><p class="notice"><span class="l-en">Loading…</span><span class="l-pl">Wczytuję…</span></p><noscript>JavaScript is required to read the data files.</noscript></div>
+  </section>
+
+  <section aria-labelledby="legend-title">
+    <h2 id="legend-title"><span class="l-en">How to read a cell</span><span class="l-pl">Jak czytać komórkę</span></h2>
+    <div class="cards">
+      <div class="card">
+        <h3><span class="l-en">Value and interval</span><span class="l-pl">Wartość i przedział</span></h3>
+        <p class="l-en">The metric in its own unit, with a paired cluster bootstrap 95% interval. Arrows show whether higher or lower is better. Overlapping intervals do not mean the models are equal. A model is called better or worse only from a paired permutation test with Holm correction over the whole comparison family; otherwise the result is inconclusive.</p>
+        <p class="l-pl">Miara we własnej jednostce, z 95% przedziałem z bootstrapu klastrowego. Strzałka mówi, czy lepiej więcej, czy mniej. Nakładające się przedziały nie oznaczają, że modele są równe. Model nazywamy lepszym albo gorszym tylko na podstawie sparowanego testu permutacyjnego z poprawką Holma na całej rodzinie porównań; w innym razie wynik jest nierozstrzygnięty.</p>
+      </div>
+      <div class="card">
+        <h3><span class="l-en">Reference points</span><span class="l-pl">Punkty odniesienia</span></h3>
+        <p class="l-en"><b>Random</b> is the expected score of a uniform guess on the same sample. <b>Majority</b> is the score of always answering the most common correct class. A score below these says little about ability.</p>
+        <p class="l-pl"><b>Losowo</b> to oczekiwany wynik zgadywania na tej samej próbce. <b>Większość</b> to wynik zawsze tej samej, najczęstszej poprawnej klasy. Wynik poniżej tych poziomów niewiele mówi o zdolności.</p>
+      </div>
+      <div class="card">
+        <h3><span class="l-en">Contamination</span><span class="l-pl">Kontaminacja</span></h3>
+        <p class="l-en"><span class="tag clean">clean</span> no item found in the training data. <span class="tag flagged">flagged</span> some items were found; the score is shown on the full sample and on the items without hits. If fewer than 50 items remain without hits, that variant shows only the count. If fewer than 50 items remain without any hits, the full-sample score is greyed and gives no better/worse; the variant without hits in answer fields shows a number only if at least 50 items remain. <span class="tag unknown">training data unknown</span> the model's training data is not public. The score is shown, but it cannot be checked for contamination.</p>
+        <p class="l-pl"><span class="tag clean">czysty</span> żadnej pozycji nie znaleziono w danych treningowych. <span class="tag flagged">trafienia</span> część pozycji znaleziono; wynik podajemy na całej próbce i na pozycjach bez trafień. Gdy bez trafień zostaje mniej niż 50 pozycji, ten wariant pokazuje tylko liczbę pozycji. Gdy bez żadnych trafień zostaje mniej niż 50 pozycji, wynik z całej próbki jest wyszarzony i nie daje werdyktu lepszy/gorszy; wariant bez trafień w polach odpowiedzi ma liczbę tylko przy co najmniej 50 pozycjach. <span class="tag unknown">dane treningowe nieznane</span> dane treningowe modelu nie są publiczne. Wynik jest pokazany, ale nie da się go sprawdzić pod kątem kontaminacji.</p>
+      </div>
+      <div class="card">
+        <h3><span class="l-en">Constant answer</span><span class="l-pl">Stała odpowiedź</span></h3>
+        <p class="l-en"><span class="tag constant">constant answer</span> the model gave the same class to every item. The value is real but carries no information about ability, so it is not compared between models or versions.</p>
+        <p class="l-pl"><span class="tag constant">stała odpowiedź</span> model dał tę samą klasę każdej pozycji. Wartość jest prawdziwa, ale nie niesie informacji o zdolności, więc nie porównujemy jej między modelami ani wersjami.</p>
+      </div>
+    </div>
+  </section>
+
+  <section aria-labelledby="method-title">
+    <h2 id="method-title"><span class="l-en">Method and data</span><span class="l-pl">Metoda i dane</span></h2>
+    <p class="l-en">The full protocol is in <a href="METHOD.md">METHOD.md</a> (English). The benchmark list with sources, revisions and licences is <a href="tasks_50.json">tasks_50.json</a>; the result format is <a href="results.schema.json">results.schema.json</a>; the role of every scanned field (context, answer-bearing, metadata), fixed before the variants without hits were computed, is in <a href="contamination_field_roles.json">contamination_field_roles.json</a>. Every file below is checked against its pinned SHA-256 before anything is shown.</p>
+    <p class="l-pl">Pełny protokół jest w <a href="METHOD.md">METHOD.md</a> (po angielsku). Lista benchmarków ze źródłami, rewizjami i licencjami to <a href="tasks_50.json">tasks_50.json</a>; format wyników to <a href="results.schema.json">results.schema.json</a>; rola każdego skanowanego pola (kontekst, pole z odpowiedzią, metadane), ustalona przed policzeniem wariantów bez trafień, jest w <a href="contamination_field_roles.json">contamination_field_roles.json</a>. Każdy plik poniżej jest sprawdzany względem przypiętego SHA-256, zanim cokolwiek się pokaże.</p>
+    <div id="files" class="files"></div>
+  </section>
+</main>
+<footer><span class="l-en">Fabryka AI · static page, no tracking, no external scripts.</span><span class="l-pl">Fabryka AI · strona statyczna, bez śledzenia i zewnętrznych skryptów.</span></footer>
+</body>
 </html>

results-feed.json

@@ -0,0 +1 @@
+{"results": "results.json", "sha256": "1cf2b8bbf7fc3913dc9755940a8e8f1efb5f4f6ebee96af18d449de3e76af7c8"}

results.schema.json

@@ -0,0 +1,1129 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "urn:fabryka-ai:benchmark-results:2:protocol-0.3.1",
+  "type": "object",
+  "properties": {
+    "schema_version": {
+      "const": 2
+    },
+    "protocol_id": {
+      "enum": [
+        "bench50-v0.3.1-panel200",
+        "bench50-v0.3.1-fullB"
+      ]
+    },
+    "run_id": {
+      "type": "string",
+      "minLength": 1
+    },
+    "models": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "properties": {
+          "model_id": {
+            "type": "string",
+            "minLength": 1
+          },
+          "repo": {
+            "type": "string",
+            "minLength": 1
+          },
+          "revision": {
+            "type": [
+              "string",
+              "null"
+            ]
+          },
+          "weights_sha256": {
+            "anyOf": [
+              {
+                "type": "string",
+                "pattern": "^[0-9a-f]{64}$"
+              },
+              {
+                "type": "null"
+              }
+            ]
+          }
+        },
+        "required": [
+          "model_id",
+          "repo",
+          "revision",
+          "weights_sha256"
+        ],
+        "additionalProperties": false
+      }
+    },
+    "tasks": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "properties": {
+          "task_id": {
+            "type": "string",
+            "pattern": "^T(0[1-9]|[1-4][0-9]|50)$"
+          },
+          "task": {
+            "type": "string",
+            "minLength": 1
+          },
+          "family": {
+            "type": "string",
+            "minLength": 1
+          },
+          "language": {
+            "enum": [
+              "PL",
+              "EN"
+            ]
+          },
+          "split": {
+            "type": "string",
+            "minLength": 1
+          }
+        },
+        "required": [
+          "task_id",
+          "task",
+          "family",
+          "language",
+          "split"
+        ],
+        "additionalProperties": false
+      }
+    },
+    "observations": {
+      "type": "array",
+      "items": {
+        "$ref": "#/$defs/observation"
+      }
+    },
+    "contrasts": {
+      "type": "array",
+      "items": {
+        "$ref": "#/$defs/contrast"
+      }
+    },
+    "provenance": {
+      "type": "object",
+      "properties": {
+        "inventory_commit": {
+          "type": "string",
+          "pattern": "^[0-9a-f]{40}$"
+        },
+        "kind": {
+          "enum": [
+            "measurement",
+            "contract_example"
+          ]
+        },
+        "sources": {
+          "type": "array",
+          "minItems": 1,
+          "items": {
+            "type": "object",
+            "properties": {
+              "run_id": {
+                "type": "string",
+                "minLength": 1
+              },
+              "results_sha256": {
+                "type": "string",
+                "pattern": "^[0-9a-f]{64}$"
+              }
+            },
+            "required": [
+              "run_id",
+              "results_sha256"
+            ],
+            "additionalProperties": false
+          }
+        },
+        "qualification_sha256": {
+          "type": "string",
+          "pattern": "^[0-9a-f]{64}$"
+        },
+        "contamination_sha256": {
+          "type": "string",
+          "pattern": "^[0-9a-f]{64}$"
+        },
+        "references_sha256": {
+          "type": "string",
+          "pattern": "^[0-9a-f]{64}$"
+        },
+        "superseded": {
+          "type": "array",
+          "items": {
+            "type": "object",
+            "properties": {
+              "model_id": {
+                "type": "string",
+                "minLength": 1
+              },
+              "task_id": {
+                "type": "string",
+                "pattern": "^T(0[1-9]|[1-4][0-9]|50)$"
+              },
+              "by_results_sha256": {
+                "type": "string",
+                "pattern": "^[0-9a-f]{64}$"
+              },
+              "replaced": {
+                "type": "array",
+                "minItems": 1,
+                "items": {
+                  "type": "object",
+                  "properties": {
+                    "state": {
+                      "enum": [
+                        "not_run",
+                        "unsupported",
+                        "blocked_access",
+                        "blocked_runtime",
+                        "blocked_security",
+                        "blocked_judge"
+                      ]
+                    },
+                    "results_sha256": {
+                      "type": "string",
+                      "pattern": "^[0-9a-f]{64}$"
+                    }
+                  },
+                  "required": [
+                    "state",
+                    "results_sha256"
+                  ],
+                  "additionalProperties": false
+                }
+              }
+            },
+            "required": [
+              "model_id",
+              "task_id",
+              "by_results_sha256",
+              "replaced"
+            ],
+            "additionalProperties": false
+          }
+        },
+        "contrasts_sha256": {
+          "type": "string",
+          "pattern": "^[0-9a-f]{64}$"
+        },
+        "amendments": {
+          "type": "array",
+          "items": {
+            "type": "object",
+            "properties": {
+              "model_id": {
+                "type": "string",
+                "minLength": 1
+              },
+              "task_id": {
+                "type": "string",
+                "pattern": "^T(0[1-9]|[1-4][0-9]|50)$"
+              },
+              "code": {
+                "type": "string",
+                "minLength": 1
+              },
+              "base_items_sha256": {
+                "type": "string",
+                "pattern": "^[0-9a-f]{64}$"
+              },
+              "artifact_sha256": {
+                "type": "string",
+                "pattern": "^[0-9a-f]{64}$"
+              }
+            },
+            "required": [
+              "model_id",
+              "task_id",
+              "code",
+              "base_items_sha256",
+              "artifact_sha256"
+            ],
+            "additionalProperties": false
+          }
+        }
+      },
+      "required": [
+        "inventory_commit",
+        "kind"
+      ],
+      "additionalProperties": false,
+      "allOf": [
+        {
+          "if": {
+            "properties": {
+              "kind": {
+                "const": "measurement"
+              }
+            }
+          },
+          "then": {
+            "required": [
+              "sources",
+              "qualification_sha256",
+              "contamination_sha256",
+              "references_sha256"
+            ]
+          }
+        }
+      ]
+    }
+  },
+  "required": [
+    "schema_version",
+    "protocol_id",
+    "run_id",
+    "models",
+    "tasks",
+    "observations",
+    "contrasts",
+    "provenance"
+  ],
+  "additionalProperties": false,
+  "$defs": {
+    "ci95": {
+      "type": "object",
+      "properties": {
+        "lower": {
+          "type": "number"
+        },
+        "upper": {
+          "type": "number"
+        },
+        "level": {
+          "const": 0.95
+        },
+        "method": {
+          "enum": [
+            "paired_bootstrap",
+            "paired_cluster_bootstrap"
+          ]
+        },
+        "replicates": {
+          "const": 10000
+        },
+        "seed": {
+          "const": 20261028
+        }
+      },
+      "required": [
+        "lower",
+        "upper",
+        "level",
+        "method",
+        "replicates",
+        "seed"
+      ],
+      "additionalProperties": false
+    },
+    "observation": {
+      "type": "object",
+      "properties": {
+        "task_id": {
+          "type": "string",
+          "pattern": "^T(0[1-9]|[1-4][0-9]|50)$"
+        },
+        "model_id": {
+          "type": "string",
+          "minLength": 1
+        },
+        "protocol_id": {
+          "enum": [
+            "bench50-v0.3.1-panel200",
+            "bench50-v0.3.1-fullB"
+          ]
+        },
+        "metric": {
+          "type": "string",
+          "minLength": 1
+        },
+        "unit": {
+          "enum": [
+            "percent",
+            "score",
+            "byte_ppl"
+          ]
+        },
+        "higher_is_better": {
+          "type": "boolean"
+        },
+        "n_population": {
+          "type": [
+            "integer",
+            "null"
+          ],
+          "minimum": 0
+        },
+        "n_selected": {
+          "type": [
+            "integer",
+            "null"
+          ],
+          "minimum": 0
+        },
+        "n_scored": {
+          "type": "integer",
+          "minimum": 0
+        },
+        "state": {
+          "enum": [
+            "ok",
+            "not_run",
+            "blocked_access",
+            "blocked_runtime",
+            "blocked_security",
+            "blocked_judge",
+            "unsupported",
+            "failed",
+            "skipped"
+          ]
+        },
+        "value": {
+          "type": [
+            "number",
+            "null"
+          ]
+        },
+        "ci95": {
+          "anyOf": [
+            {
+              "type": "object",
+              "properties": {
+                "lower": {
+                  "type": "number"
+                },
+                "upper": {
+                  "type": "number"
+                },
+                "level": {
+                  "const": 0.95
+                },
+                "method": {
+                  "enum": [
+                    "paired_bootstrap",
+                    "paired_cluster_bootstrap"
+                  ]
+                },
+                "replicates": {
+                  "const": 10000
+                },
+                "seed": {
+                  "const": 20261028
+                }
+              },
+              "required": [
+                "lower",
+                "upper",
+                "level",
+                "method",
+                "replicates",
+                "seed"
+              ],
+              "additionalProperties": false
+            },
+            {
+              "type": "null"
+            }
+          ]
+        },
+        "source_id_digest": {
+          "anyOf": [
+            {
+              "type": "string",
+              "pattern": "^[0-9a-f]{64}$"
+            },
+            {
+              "type": "null"
+            }
+          ]
+        },
+        "results_sha256": {
+          "anyOf": [
+            {
+              "type": "string",
+              "pattern": "^[0-9a-f]{64}$"
+            },
+            {
+              "type": "null"
+            }
+          ]
+        },
+        "runner_sha256": {
+          "anyOf": [
+            {
+              "type": "string",
+              "pattern": "^[0-9a-f]{64}$"
+            },
+            {
+              "type": "null"
+            }
+          ]
+        },
+        "reason": {
+          "type": [
+            "string",
+            "null"
+          ],
+          "maxLength": 200
+        },
+        "contamination": {
+          "enum": [
+            "clean",
+            "flagged",
+            "partially_scanned",
+            "unscanned",
+            "unknown"
+          ]
+        },
+        "contamination_receipt_sha256": {
+          "type": [
+            "string",
+            "null"
+          ],
+          "pattern": "^[0-9a-f]{64}$"
+        },
+        "value_hit_free": {
+          "type": [
+            "number",
+            "null"
+          ]
+        },
+        "n_hit_free": {
+          "type": [
+            "integer",
+            "null"
+          ],
+          "minimum": 0
+        },
+        "hit_rate": {
+          "type": [
+            "number",
+            "null"
+          ],
+          "minimum": 0,
+          "maximum": 1
+        },
+        "reference": {
+          "type": [
+            "object",
+            "null"
+          ],
+          "properties": {
+            "chance_value": {
+              "type": [
+                "number",
+                "null"
+              ],
+              "description": "expected metric of uniform random choice on the frozen sample (mean 1/n_options for accuracy); null if not defined"
+            },
+            "majority_value": {
+              "type": [
+                "number",
+                "null"
+              ],
+              "description": "same metric for always predicting the majority gold class of the frozen sample; null if classes are not fixed"
+            },
+            "majority_prediction_share": {
+              "type": [
+                "number",
+                "null"
+              ],
+              "minimum": 0,
+              "maximum": 1,
+              "description": "share of the model predictions falling on its most frequent predicted class"
+            },
+            "constant_prediction": {
+              "type": [
+                "boolean",
+                "null"
+              ],
+              "description": "true when the model predicts one class for every scored item"
+            }
+          },
+          "required": [
+            "chance_value",
+            "majority_value",
+            "majority_prediction_share",
+            "constant_prediction"
+          ],
+          "additionalProperties": false
+        },
+        "value_hit_free_answer_fields": {
+          "type": [
+            "number",
+            "null"
+          ]
+        },
+        "n_hit_free_answer_fields": {
+          "type": [
+            "integer",
+            "null"
+          ],
+          "minimum": 0
+        },
+        "hit_rate_answer_fields": {
+          "type": [
+            "number",
+            "null"
+          ],
+          "minimum": 0,
+          "maximum": 1
+        },
+        "ci95_hit_free": {
+          "anyOf": [
+            {
+              "type": "object",
+              "properties": {
+                "lower": {
+                  "type": "number"
+                },
+                "upper": {
+                  "type": "number"
+                },
+                "level": {
+                  "const": 0.95
+                },
+                "method": {
+                  "enum": [
+                    "paired_bootstrap",
+                    "paired_cluster_bootstrap"
+                  ]
+                },
+                "replicates": {
+                  "const": 10000
+                },
+                "seed": {
+                  "const": 20261028
+                }
+              },
+              "required": [
+                "lower",
+                "upper",
+                "level",
+                "method",
+                "replicates",
+                "seed"
+              ],
+              "additionalProperties": false
+            },
+            {
+              "type": "null"
+            }
+          ]
+        },
+        "ci95_hit_free_answer_fields": {
+          "anyOf": [
+            {
+              "type": "object",
+              "properties": {
+                "lower": {
+                  "type": "number"
+                },
+                "upper": {
+                  "type": "number"
+                },
+                "level": {
+                  "const": 0.95
+                },
+                "method": {
+                  "enum": [
+                    "paired_bootstrap",
+                    "paired_cluster_bootstrap"
+                  ]
+                },
+                "replicates": {
+                  "const": 10000
+                },
+                "seed": {
+                  "const": 20261028
+                }
+              },
+              "required": [
+                "lower",
+                "upper",
+                "level",
+                "method",
+                "replicates",
+                "seed"
+              ],
+              "additionalProperties": false
+            },
+            {
+              "type": "null"
+            }
+          ]
+        }
+      },
+      "required": [
+        "task_id",
+        "model_id",
+        "protocol_id",
+        "metric",
+        "unit",
+        "higher_is_better",
+        "n_population",
+        "n_selected",
+        "n_scored",
+        "state",
+        "value",
+        "ci95",
+        "source_id_digest",
+        "results_sha256",
+        "runner_sha256",
+        "contamination",
+        "contamination_receipt_sha256"
+      ],
+      "additionalProperties": false,
+      "allOf": [
+        {
+          "if": {
+            "properties": {
+              "state": {
+                "const": "ok"
+              }
+            }
+          },
+          "then": {
+            "properties": {
+              "value": {
+                "type": "number"
+              },
+              "n_scored": {
+                "type": "integer",
+                "minimum": 1
+              },
+              "source_id_digest": {
+                "type": "string",
+                "pattern": "^[0-9a-f]{64}$"
+              },
+              "results_sha256": {
+                "type": "string",
+                "pattern": "^[0-9a-f]{64}$"
+              },
+              "runner_sha256": {
+                "type": "string",
+                "pattern": "^[0-9a-f]{64}$"
+              }
+            }
+          },
+          "else": {
+            "properties": {
+              "value": {
+                "type": "null"
+              },
+              "ci95": {
+                "type": "null"
+              }
+            }
+          }
+        },
+        {
+          "if": {
+            "properties": {
+              "contamination": {
+                "const": "flagged"
+              },
+              "state": {
+                "const": "ok"
+              }
+            },
+            "required": [
+              "contamination",
+              "state"
+            ]
+          },
+          "then": {
+            "required": [
+              "value_hit_free",
+              "n_hit_free",
+              "hit_rate",
+              "value_hit_free_answer_fields",
+              "n_hit_free_answer_fields",
+              "hit_rate_answer_fields"
+            ],
+            "properties": {
+              "n_hit_free": {
+                "type": "integer"
+              },
+              "hit_rate": {
+                "type": "number"
+              },
+              "n_hit_free_answer_fields": {
+                "type": "integer"
+              },
+              "hit_rate_answer_fields": {
+                "type": "number"
+              }
+            }
+          }
+        },
+        {
+          "if": {
+            "properties": {
+              "contamination": {
+                "enum": [
+                  "clean",
+                  "flagged",
+                  "partially_scanned"
+                ]
+              }
+            }
+          },
+          "then": {
+            "properties": {
+              "contamination_receipt_sha256": {
+                "type": "string"
+              }
+            }
+          }
+        },
+        {
+          "if": {
+            "properties": {
+              "n_hit_free": {
+                "const": 0
+              }
+            },
+            "required": [
+              "n_hit_free"
+            ]
+          },
+          "then": {
+            "properties": {
+              "value_hit_free": {
+                "type": "null"
+              }
+            }
+          },
+          "else": {
+            "if": {
+              "properties": {
+                "contamination": {
+                  "const": "flagged"
+                },
+                "state": {
+                  "const": "ok"
+                }
+              },
+              "required": [
+                "contamination",
+                "state"
+              ]
+            },
+            "then": {
+              "properties": {
+                "value_hit_free": {
+                  "type": "number"
+                }
+              }
+            }
+          }
+        },
+        {
+          "if": {
+            "properties": {
+              "n_hit_free_answer_fields": {
+                "const": 0
+              }
+            },
+            "required": [
+              "n_hit_free_answer_fields"
+            ]
+          },
+          "then": {
+            "properties": {
+              "value_hit_free_answer_fields": {
+                "type": "null"
+              }
+            }
+          },
+          "else": {
+            "if": {
+              "properties": {
+                "contamination": {
+                  "const": "flagged"
+                },
+                "state": {
+                  "const": "ok"
+                }
+              },
+              "required": [
+                "contamination",
+                "state"
+              ]
+            },
+            "then": {
+              "properties": {
+                "value_hit_free_answer_fields": {
+                  "type": "number"
+                }
+              }
+            }
+          }
+        }
+      ]
+    },
+    "contrast": {
+      "type": "object",
+      "properties": {
+        "task_id": {
+          "type": "string",
+          "pattern": "^T(0[1-9]|[1-4][0-9]|50)$"
+        },
+        "model_id": {
+          "type": "string",
+          "minLength": 1
+        },
+        "protocol_id": {
+          "enum": [
+            "bench50-v0.3.1-panel200",
+            "bench50-v0.3.1-fullB"
+          ]
+        },
+        "model_ref": {
+          "type": "string",
+          "minLength": 1
+        },
+        "delta": {
+          "type": [
+            "number",
+            "null"
+          ]
+        },
+        "ci95": {
+          "anyOf": [
+            {
+              "type": "object",
+              "properties": {
+                "lower": {
+                  "type": "number"
+                },
+                "upper": {
+                  "type": "number"
+                },
+                "level": {
+                  "const": 0.95
+                },
+                "method": {
+                  "enum": [
+                    "paired_bootstrap",
+                    "paired_cluster_bootstrap"
+                  ]
+                },
+                "replicates": {
+                  "const": 10000
+                },
+                "seed": {
+                  "const": 20261028
+                }
+              },
+              "required": [
+                "lower",
+                "upper",
+                "level",
+                "method",
+                "replicates",
+                "seed"
+              ],
+              "additionalProperties": false
+            },
+            {
+              "type": "null"
+            }
+          ]
+        },
+        "p_raw": {
+          "type": [
+            "number",
+            "null"
+          ],
+          "minimum": 0,
+          "maximum": 1
+        },
+        "p_holm": {
+          "type": [
+            "number",
+            "null"
+          ],
+          "minimum": 0,
+          "maximum": 1
+        },
+        "family_size": {
+          "type": "integer",
+          "minimum": 1
+        },
+        "verdict": {
+          "enum": [
+            "withheld",
+            "better",
+            "worse",
+            "inconclusive"
+          ]
+        },
+        "review": {
+          "type": "object",
+          "properties": {
+            "status": {
+              "enum": [
+                "pending",
+                "accepted",
+                "rejected"
+              ]
+            },
+            "actor": {
+              "type": [
+                "string",
+                "null"
+              ]
+            },
+            "artifact_sha256": {
+              "anyOf": [
+                {
+                  "type": "string",
+                  "pattern": "^[0-9a-f]{64}$"
+                },
+                {
+                  "type": "null"
+                }
+              ]
+            }
+          },
+          "required": [
+            "status",
+            "actor",
+            "artifact_sha256"
+          ],
+          "additionalProperties": false,
+          "allOf": [
+            {
+              "if": {
+                "properties": {
+                  "status": {
+                    "const": "accepted"
+                  }
+                }
+              },
+              "then": {
+                "properties": {
+                  "actor": {
+                    "type": "string",
+                    "minLength": 1
+                  },
+                  "artifact_sha256": {
+                    "type": "string",
+                    "pattern": "^[0-9a-f]{64}$"
+                  }
+                }
+              }
+            }
+          ]
+        },
+        "subset": {
+          "enum": [
+            "full",
+            "hit_free",
+            "hit_free_answer_fields"
+          ],
+          "description": "items on which the paired contrast was computed; hit-free subsets are defined by the training corpus of model_id and are identical for both models"
+        },
+        "n": {
+          "type": [
+            "integer",
+            "null"
+          ],
+          "minimum": 0
+        },
+        "metric": {
+          "type": "string",
+          "minLength": 1
+        },
+        "unit": {
+          "type": "string",
+          "minLength": 1
+        }
+      },
+      "required": [
+        "task_id",
+        "model_id",
+        "protocol_id",
+        "model_ref",
+        "delta",
+        "ci95",
+        "p_raw",
+        "p_holm",
+        "family_size",
+        "verdict",
+        "review",
+        "subset",
+        "n",
+        "metric",
+        "unit"
+      ],
+      "additionalProperties": false,
+      "allOf": [
+        {
+          "if": {
+            "properties": {
+              "verdict": {
+                "enum": [
+                  "better",
+                  "worse",
+                  "inconclusive"
+                ]
+              }
+            }
+          },
+          "then": {
+            "properties": {
+              "review": {
+                "properties": {
+                  "status": {
+                    "const": "accepted"
+                  }
+                }
+              },
+              "delta": {
+                "type": "number"
+              },
+              "p_raw": {
+                "type": "number"
+              },
+              "p_holm": {
+                "type": "number"
+              }
+            }
+          }
+        },
+        {
+          "if": {
+            "properties": {
+              "review": {
+                "properties": {
+                  "status": {
+                    "enum": [
+                      "pending",
+                      "rejected"
+                    ]
+                  }
+                }
+              }
+            }
+          },
+          "then": {
+            "properties": {
+              "verdict": {
+                "const": "withheld"
+              }
+            }
+          }
+        }
+      ]
+    }
+  }
+}

style.css

@@ -1,28 +1,76 @@
-body {
-	padding: 2rem;
-	font-family: -apple-system, BlinkMacSystemFont, "Arial", sans-serif;
-}
-
-h1 {
-	font-size: 16px;
-	margin-top: 0;
-}
-
-p {
-	color: rgb(107, 114, 128);
-	font-size: 15px;
-	margin-bottom: 10px;
-	margin-top: 5px;
-}
-
-.card {
-	max-width: 620px;
-	margin: 0 auto;
-	padding: 16px;
-	border: 1px solid lightgray;
-	border-radius: 16px;
-}
-
-.card p:last-child {
-	margin-bottom: 0;
+:root{--bg:#0b0d11;--card:#141821;--card2:#181d28;--line:#252b38;--line2:#323a4a;--fg:#eceef2;--soft:#c9ced8;--muted:#9aa3b2;--acc:#4fb477;--acc3:#6ea6e6;--warn:#e8c37a;--bad:#e0565b;--r:16px;color-scheme:dark}
+*{box-sizing:border-box}
+html{-webkit-text-size-adjust:100%}
+body{margin:0;color:var(--fg);font:16px/1.6 ui-sans-serif,system-ui,-apple-system,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif;background:radial-gradient(900px 480px at 12% -80px,rgba(79,180,119,.14),transparent 70%),var(--bg);background-repeat:no-repeat}
+body[data-lang="en"] .l-pl,body[data-lang="pl"] .l-en{display:none!important}
+a{color:var(--acc3);text-underline-offset:3px}
+a:focus-visible,button:focus-visible,input:focus-visible,select:focus-visible,summary:focus-visible,.wrap:focus-visible{outline:2px solid var(--acc3);outline-offset:2px}
+.lang{position:absolute;top:14px;right:14px;z-index:5;display:flex;gap:4px;background:rgba(20,24,33,.9);border:1px solid var(--line2);border-radius:999px;padding:3px}
+.lang button{background:none;border:0;color:var(--muted);font:inherit;font-size:.85rem;font-weight:700;padding:4px 12px;border-radius:999px;cursor:pointer}
+.lang button[aria-pressed="true"]{background:var(--acc);color:#0b0d11}
+main{max-width:1180px;margin:0 auto;padding:28px 20px 72px}
+header{padding:44px 0 8px}
+.by{color:var(--acc);font-size:.82rem;font-weight:600;letter-spacing:.14em;text-transform:uppercase;margin:0 0 18px}
+h1{font-size:clamp(2.2rem,6vw,3.5rem);line-height:1.05;margin:0 0 14px;letter-spacing:-.035em;font-weight:800;text-wrap:balance}
+.lead{font-size:1.12rem;line-height:1.7;max-width:70ch;color:var(--soft);margin:0}
+.pills{display:flex;flex-wrap:wrap;gap:10px;margin:24px 0 0}
+.pending{margin:12px 0 0;max-width:70ch;color:var(--muted);font-size:.92rem;line-height:1.55}
+.pill{background:rgba(20,24,33,.8);border:1px solid var(--line2);border-radius:999px;padding:6px 14px;font-size:.9rem;color:var(--muted)}
+.pill b{color:var(--fg);font-variant-numeric:tabular-nums}
+section{margin-top:64px}
+h2{display:flex;align-items:center;gap:12px;font-size:1.55rem;line-height:1.25;letter-spacing:-.02em;margin:0 0 18px}
+h2::before{content:"";width:5px;height:1.15em;border-radius:3px;background:linear-gradient(var(--acc),var(--acc3));flex:none}
+.notice{border-left:3px solid var(--warn);background:var(--card);color:var(--soft);padding:12px 16px;border-radius:0 10px 10px 0;max-width:82ch;margin:0 0 18px}
+.notice.error{border-left-color:var(--bad)}
+.controls{display:flex;flex-wrap:wrap;gap:12px;align-items:end;margin:0 0 14px}
+.controls label{display:grid;gap:6px;color:var(--muted);font-size:.88rem}
+input,select,button.clear{font:inherit;color:var(--fg);background:var(--card);border:1px solid var(--line2);border-radius:10px;padding:9px 12px}
+input{width:min(300px,100%)}
+button.clear{cursor:pointer}
+button.clear:hover{border-color:var(--acc3)}
+.summary{color:var(--muted);font-size:.9rem;margin:0 0 10px}
+.wrap{overflow:auto;max-height:76vh;border:1px solid var(--line);border-radius:12px;background:var(--card)}
+table{border-collapse:separate;border-spacing:0;width:100%;min-width:760px}
+th,td{text-align:left;vertical-align:top;padding:12px 14px;border-bottom:1px solid var(--line);font-size:.93rem}
+thead th{position:sticky;top:0;z-index:2;background:var(--card2);color:var(--muted);font-weight:600;font-size:.8rem;letter-spacing:.04em;text-transform:uppercase}
+th.task,thead th:first-child{position:sticky;left:0;z-index:1;background:var(--card2);width:230px;min-width:200px}
+thead th:first-child{z-index:3}
+th.task{font-weight:650;color:var(--fg)}
+tbody tr:last-child>*{border-bottom:0}
+.meta{display:block;color:var(--muted);font-size:.8rem;line-height:1.5;margin-top:3px;font-weight:400;overflow-wrap:anywhere}
+.value{display:block;font-size:1.12rem;font-weight:650;font-variant-numeric:tabular-nums;color:var(--fg)}
+.value small{font-size:.75rem;font-weight:500;color:var(--muted);margin-left:4px}
+.value.dim{color:var(--muted);font-weight:500}
+.mostly{display:block;font-size:.8rem;color:var(--warn);margin-top:3px}
+.amend{display:block;font-size:.78rem;line-height:1.4;color:var(--warn);margin-top:4px;font-weight:400;max-width:28ch}
+.state{display:inline-block;color:var(--muted);font-size:.85rem}
+.ref{display:block;color:var(--soft);font-size:.8rem;margin-top:4px;font-variant-numeric:tabular-nums}
+.tags{display:flex;flex-wrap:wrap;gap:6px;margin-top:8px}
+.tag{display:inline-block;font-size:.7rem;font-weight:700;letter-spacing:.04em;text-transform:uppercase;padding:2px 9px;border-radius:999px;line-height:1.5;white-space:nowrap}
+.tag.clean{background:rgba(79,180,119,.15);color:#8fe0ad;box-shadow:0 0 0 1px rgba(79,180,119,.35) inset}
+.tag.flagged{background:rgba(232,195,122,.13);color:var(--warn);box-shadow:0 0 0 1px rgba(232,195,122,.35) inset}
+.tag.unknown{background:rgba(110,166,230,.13);color:#a9cbf2;box-shadow:0 0 0 1px rgba(110,166,230,.35) inset}
+.tag.pending{background:rgba(154,163,178,.12);color:var(--muted);box-shadow:0 0 0 1px rgba(154,163,178,.3) inset}
+.tag.constant{background:rgba(224,86,91,.13);color:#f1a2a5;box-shadow:0 0 0 1px rgba(224,86,91,.35) inset}
+.verdict{display:block;margin-top:8px;font-weight:650;font-size:.85rem}
+details{margin-top:8px}
+summary{cursor:pointer;color:var(--acc3);font-size:.8rem}
+details p{margin:6px 0 0;color:var(--muted);font-size:.78rem;overflow-wrap:anywhere}
+.cards{display:grid;grid-template-columns:repeat(auto-fit,minmax(250px,1fr));gap:16px}
+.card{background:linear-gradient(180deg,var(--card2),var(--card));border:1px solid var(--line);border-radius:var(--r);padding:18px 20px}
+.card h3{margin:0 0 8px;font-size:1.08rem}
+.card p{margin:0;color:var(--soft);font-size:.95rem}
+.files{display:grid;gap:6px;margin-top:14px}
+.files p{margin:0;color:var(--muted);font-size:.85rem;overflow-wrap:anywhere}
+.files code{font:.82rem/1.5 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;color:var(--soft)}
+footer{max-width:1180px;margin:0 auto;padding:24px 20px 40px;border-top:1px solid var(--line);color:var(--muted);font-size:.85rem}
+[hidden]{display:none!important}
+@media (max-width:640px){
+  main{padding:18px 14px 56px}
+  header{padding:40px 0 4px}
+  .controls{display:grid;grid-template-columns:1fr}
+  input,select,button.clear{width:100%}
+  th.task,thead th:first-child{width:150px;min-width:150px}
+  th,td{padding:10px 11px}
+  .wrap{max-height:70vh}
 }

tasks_50.json

@@ -0,0 +1,1902 @@
+{
+  "schema_version": 2,
+  "protocol": "bench50-v0.3.1",
+  "status": "selected_source_registry_not_measurements_or_execution_approval",
+  "scope": "50 distinct upstream benchmark identities; variants, splits, translations, rounds and suite members are not extra benchmarks",
+  "counts": {
+    "rows": 50,
+    "distinct_benchmarks": 50,
+    "EN": 38,
+    "PL": 12,
+    "source_members": 204,
+    "dataset_repositories": 44
+  },
+  "distinctness": {
+    "key": "benchmark_id and reviewed upstream collection, not YAML task name",
+    "umbrella_suites_not_added_to_components": true,
+    "statistical_independence_claimed": false,
+    "language_balance": "source diversity first; original user directive did not impose25PL/25EN"
+  },
+  "sources": {
+    "lm_eval_commit": "6d2abda4fd171e68a8789330c4149e37c1ca0bda",
+    "openpl_commit": "21d0ea9cf4fd6153dfff4d84d6ad0aab5488f302",
+    "source_registry": "benchmark_sources_v03.json",
+    "origins": {
+      "en": "canonical pinned EN checkout",
+      "pl": "canonical pinned PL checkout",
+      "registry": "this release tasks/ definitions"
+    }
+  },
+  "selection": {
+    "seed": 20261028,
+    "max_documents_per_benchmark": 200,
+    "unit": "source document after the original process_docs; group member namespace included",
+    "ordering": "SHA256(UTF8(20261028|benchmark_id|member_task|source_id)); ties by canonical source bytes",
+    "when": "seal all requested benchmark/member data, source IDs, support and requests before loading model weights",
+    "group_allocation": "reserve min(2,eligible_population) per member; then allocate each remaining slot to the non-full member with largest (budget*source_weight - allocated), ties by member task name; never200 per member",
+    "group_weighting": "source macro/equal-member or eligible-population-size weights retained independently of sample quota; reaggregate member metrics with those fixed weights",
+    "support_policy": "configured source count; hash-selected disjoint support per member unless source supplies fixed examples, which retain source order; record every demo and exclude matching scored inputs",
+    "context_policy": "float32,batch1,max4096,no automaticBOS/chat; reserve task generation limit (default256); drop optional demos from end, never query/options; explicit failure if still over limit",
+    "wikitext": "full source test stream, canonical weighted byte_perplexity, not Glint conversion; descriptive comparison only",
+    "full_protocol_B": "separate full-data evaluation of these selected primary protocols; not the old23-definition OpenPL replication",
+    "harness_seed_tuple": {
+      "python": 0,
+      "numpy": 1234,
+      "torch": 1234,
+      "fewshot": 1234
+    },
+    "empty_group_member": "explicit failure; never silently remove or reweight a required source member"
+  },
+  "rows": [
+    {
+      "id": "T01",
+      "benchmark_id": "blimp",
+      "task": "blimp",
+      "name": "BLiMP",
+      "language": "EN",
+      "family": "blimp",
+      "domain": "syntax",
+      "metric": "acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "group",
+      "split": "train",
+      "shots": 0,
+      "rationale": "syntax",
+      "member_count": 67,
+      "source_record": "blimp",
+      "license": {
+        "declared_tags": [
+          "cc-by-4.0"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "nyu-mll/blimp",
+            "revision": "877fba0801ffb7cbd8c39c1ff314a46f053f6036",
+            "metadata_source": "https://huggingface.co/api/datasets/nyu-mll/blimp"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [],
+      "definition": "lm_eval/tasks/blimp/_blimp.yaml"
+    },
+    {
+      "id": "T02",
+      "benchmark_id": "arc",
+      "task": "arc_challenge",
+      "name": "ARC Challenge",
+      "language": "EN",
+      "family": "arc",
+      "domain": "science_reasoning",
+      "metric": "acc_norm,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "test",
+      "shots": 0,
+      "rationale": "science reasoning",
+      "member_count": 1,
+      "source_record": "arc",
+      "license": {
+        "declared_tags": [
+          "cc-by-sa-4.0"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "allenai/ai2_arc",
+            "revision": "210d026faf9955653af8916fad021475a3f00453",
+            "metadata_source": "https://huggingface.co/api/datasets/allenai/ai2_arc"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "ARC Easy and Polish ARC translation are the same counted family, not extra main benchmarks."
+      ],
+      "definition": "lm_eval/tasks/arc/arc_challenge.yaml"
+    },
+    {
+      "id": "T03",
+      "benchmark_id": "wikitext2",
+      "task": "wikitext",
+      "name": "WikiText-2",
+      "language": "EN",
+      "family": "wikitext2",
+      "domain": "language_modeling",
+      "metric": "byte_perplexity,none",
+      "unit": "byte_ppl",
+      "higher_is_better": false,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "test",
+      "shots": 0,
+      "rationale": "language modeling",
+      "member_count": 1,
+      "source_record": "wikitext2",
+      "license": {
+        "declared_tags": [
+          "cc-by-sa-3.0"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "EleutherAI/wikitext_document_level",
+            "revision": "647234772b9554e208af6c826f23b99e3cac88c8",
+            "metadata_source": "https://huggingface.co/api/datasets/EleutherAI/wikitext_document_level"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Canonical harness byte_perplexity, not the historical Glint tp^(N/B) conversion; full evaluation stream."
+      ],
+      "definition": "lm_eval/tasks/wikitext/wikitext.yaml"
+    },
+    {
+      "id": "T04",
+      "benchmark_id": "multiblimp",
+      "task": "multiblimp_pl",
+      "name": "MultiBLiMP-PL",
+      "language": "PL",
+      "family": "multiblimp",
+      "domain": "morphosyntax",
+      "metric": "accuracy",
+      "unit": "percent",
+      "higher_is_better": true,
+      "instrument": "custom_pl",
+      "entrypoint_kind": "custom",
+      "split": "pol/data.tsv",
+      "shots": 0,
+      "rationale": "morphosyntax",
+      "member_count": 1,
+      "source_record": "multiblimp",
+      "license": {
+        "declared_tags": [
+          "cc-by-4.0"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "jumelet/multiblimp",
+            "revision": "de923efa8d2483d6b13364ee68e65308e990a991",
+            "metadata_source": "https://huggingface.co/api/datasets/jumelet/multiblimp/revision/de923efa8d2483d6b13364ee68e65308e990a991"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Independent UD/UniMorph construction, not BLiMP translation. Preserve the explicit historical space-prefix log-probability-sum instrument."
+      ]
+    },
+    {
+      "id": "T05",
+      "benchmark_id": "hellaswag",
+      "task": "hellaswag",
+      "name": "HellaSwag",
+      "language": "EN",
+      "family": "hellaswag",
+      "domain": "commonsense",
+      "metric": "acc_norm,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "validation",
+      "shots": 0,
+      "rationale": "commonsense",
+      "member_count": 1,
+      "source_record": "hellaswag",
+      "license": {
+        "declared_tags": [
+          "unknown"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "Rowan/hellaswag",
+            "revision": "218ec52e09a7e7462a5400043bb9a69a41d06b76",
+            "metadata_source": "https://huggingface.co/api/datasets/Rowan/hellaswag/revision/218ec52e09a7e7462a5400043bb9a69a41d06b76"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "SWAG is excluded conservatively because the collections reuse ActivityNet context sources.",
+        "Undeclared/umbrella data terms are a gate, not an inferred permissive licence."
+      ],
+      "definition": "lm_eval/tasks/hellaswag/hellaswag.yaml"
+    },
+    {
+      "id": "T06",
+      "benchmark_id": "piqa",
+      "task": "piqa",
+      "name": "PIQA",
+      "language": "EN",
+      "family": "piqa",
+      "domain": "physical_reasoning",
+      "metric": "acc_norm,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "validation",
+      "shots": 0,
+      "rationale": "physical reasoning",
+      "member_count": 1,
+      "source_record": "piqa",
+      "license": {
+        "declared_tags": [
+          "unknown"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "ybisk/piqa",
+            "revision": "2e8ac2dffd59bac8c3c6714948f4c551a0848bb0",
+            "metadata_source": "https://huggingface.co/api/datasets/ybisk/piqa/revision/2e8ac2dffd59bac8c3c6714948f4c551a0848bb0"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Undeclared/umbrella data terms are a gate, not an inferred permissive licence."
+      ],
+      "definition": "lm_eval/tasks/piqa/piqa.yaml"
+    },
+    {
+      "id": "T07",
+      "benchmark_id": "winogrande",
+      "task": "winogrande",
+      "name": "WinoGrande",
+      "language": "EN",
+      "family": "winogrande",
+      "domain": "coreference",
+      "metric": "acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "validation",
+      "shots": 0,
+      "rationale": "coreference",
+      "member_count": 1,
+      "source_record": "winogrande",
+      "license": {
+        "declared_tags": [
+          "unknown"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "allenai/winogrande",
+            "revision": "01e74176c63542e6b0bcb004dcdea22d94fb67b5",
+            "metadata_source": "https://huggingface.co/api/datasets/allenai/winogrande/revision/01e74176c63542e6b0bcb004dcdea22d94fb67b5"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Undeclared/umbrella data terms are a gate, not an inferred permissive licence."
+      ],
+      "definition": "lm_eval/tasks/winogrande/default.yaml"
+    },
+    {
+      "id": "T08",
+      "benchmark_id": "openbookqa",
+      "task": "openbookqa",
+      "name": "OpenBookQA",
+      "language": "EN",
+      "family": "openbookqa",
+      "domain": "science_reasoning",
+      "metric": "acc_norm,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "test",
+      "shots": 0,
+      "rationale": "science reasoning",
+      "member_count": 1,
+      "source_record": "openbookqa",
+      "license": {
+        "declared_tags": [
+          "unknown"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "allenai/openbookqa",
+            "revision": "388097ea7776314e93a529163e0fea805b8a6454",
+            "metadata_source": "https://huggingface.co/api/datasets/allenai/openbookqa/revision/388097ea7776314e93a529163e0fea805b8a6454"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Undeclared/umbrella data terms are a gate, not an inferred permissive licence."
+      ],
+      "definition": "lm_eval/tasks/openbookqa/openbookqa.yaml"
+    },
+    {
+      "id": "T09",
+      "benchmark_id": "sciq",
+      "task": "sciq",
+      "name": "SciQ",
+      "language": "EN",
+      "family": "sciq",
+      "domain": "science_knowledge",
+      "metric": "acc_norm,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "test",
+      "shots": 0,
+      "rationale": "science knowledge",
+      "member_count": 1,
+      "source_record": "sciq",
+      "license": {
+        "declared_tags": [
+          "cc-by-nc-3.0"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "allenai/sciq",
+            "revision": "2c94ad3e1aafab77146f384e23536f97a4849815",
+            "metadata_source": "https://huggingface.co/api/datasets/sciq"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "NC declaration observed: evaluation-use basis must be independently resolved; this is not training permission or run approval."
+      ],
+      "definition": "lm_eval/tasks/sciq/sciq.yaml"
+    },
+    {
+      "id": "T10",
+      "benchmark_id": "boolq",
+      "task": "boolq",
+      "name": "BoolQ",
+      "language": "EN",
+      "family": "boolq",
+      "domain": "reading",
+      "metric": "acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "validation",
+      "shots": 0,
+      "rationale": "reading",
+      "member_count": 1,
+      "source_record": "boolq",
+      "license": {
+        "declared_tags": [
+          "other"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "aps/super_glue",
+            "revision": "3de24cf8022e94f4ee4b9d55a6f539891524d646",
+            "metadata_source": "https://huggingface.co/api/datasets/super_glue"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Undeclared/umbrella data terms are a gate, not an inferred permissive licence."
+      ],
+      "definition": "lm_eval/tasks/super_glue/boolq/default.yaml"
+    },
+    {
+      "id": "T11",
+      "benchmark_id": "copa",
+      "task": "copa",
+      "name": "COPA",
+      "language": "EN",
+      "family": "copa",
+      "domain": "causal_reasoning",
+      "metric": "acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "validation",
+      "shots": 0,
+      "rationale": "causal reasoning",
+      "member_count": 1,
+      "source_record": "copa",
+      "license": {
+        "declared_tags": [
+          "other"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "aps/super_glue",
+            "revision": "3de24cf8022e94f4ee4b9d55a6f539891524d646",
+            "metadata_source": "https://huggingface.co/api/datasets/super_glue"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Undeclared/umbrella data terms are a gate, not an inferred permissive licence."
+      ],
+      "definition": "lm_eval/tasks/super_glue/copa/default.yaml"
+    },
+    {
+      "id": "T12",
+      "benchmark_id": "cb",
+      "task": "cb",
+      "name": "CommitmentBank",
+      "language": "EN",
+      "family": "cb",
+      "domain": "entailment",
+      "metric": "acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "validation",
+      "shots": 0,
+      "rationale": "entailment",
+      "member_count": 1,
+      "source_record": "cb",
+      "license": {
+        "declared_tags": [
+          "other"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "aps/super_glue",
+            "revision": "3de24cf8022e94f4ee4b9d55a6f539891524d646",
+            "metadata_source": "https://huggingface.co/api/datasets/super_glue"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Undeclared/umbrella data terms are a gate, not an inferred permissive licence."
+      ],
+      "definition": "lm_eval/tasks/super_glue/cb/default.yaml"
+    },
+    {
+      "id": "T13",
+      "benchmark_id": "wic",
+      "task": "wic",
+      "name": "WiC",
+      "language": "EN",
+      "family": "wic",
+      "domain": "word_sense",
+      "metric": "acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "validation",
+      "shots": 0,
+      "rationale": "word sense",
+      "member_count": 1,
+      "source_record": "wic",
+      "license": {
+        "declared_tags": [
+          "other"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "aps/super_glue",
+            "revision": "3de24cf8022e94f4ee4b9d55a6f539891524d646",
+            "metadata_source": "https://huggingface.co/api/datasets/super_glue"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Undeclared/umbrella data terms are a gate, not an inferred permissive licence."
+      ],
+      "definition": "lm_eval/tasks/super_glue/wic/default.yaml"
+    },
+    {
+      "id": "T14",
+      "benchmark_id": "wsc273",
+      "task": "wsc273",
+      "name": "Winograd Schema Challenge",
+      "language": "EN",
+      "family": "wsc273",
+      "domain": "coreference",
+      "metric": "acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "test",
+      "shots": 0,
+      "rationale": "coreference",
+      "member_count": 1,
+      "source_record": "wsc273",
+      "license": {
+        "declared_tags": [
+          "cc-by-4.0"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "ErnestSDavis/winograd_wsc",
+            "revision": "8052c534a4cecd8328c0bc879f6a5aa07404c4da",
+            "metadata_source": "https://huggingface.co/api/datasets/winograd_wsc"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [],
+      "definition": "lm_eval/tasks/wsc273/default.yaml"
+    },
+    {
+      "id": "T15",
+      "benchmark_id": "rte",
+      "task": "sglue_rte",
+      "name": "RTE",
+      "language": "EN",
+      "family": "rte",
+      "domain": "entailment",
+      "metric": "acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "validation",
+      "shots": 0,
+      "rationale": "entailment",
+      "member_count": 1,
+      "source_record": "rte",
+      "license": {
+        "declared_tags": [
+          "other"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "aps/super_glue",
+            "revision": "3de24cf8022e94f4ee4b9d55a6f539891524d646",
+            "metadata_source": "https://huggingface.co/api/datasets/super_glue"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Undeclared/umbrella data terms are a gate, not an inferred permissive licence."
+      ],
+      "definition": "lm_eval/tasks/super_glue/rte/default.yaml"
+    },
+    {
+      "id": "T16",
+      "benchmark_id": "anli",
+      "task": "anli_r3",
+      "name": "ANLI (R3 primary)",
+      "language": "EN",
+      "family": "anli",
+      "domain": "adversarial_entailment",
+      "metric": "acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "test_r3",
+      "shots": 0,
+      "rationale": "adversarial entailment",
+      "member_count": 1,
+      "source_record": "anli",
+      "license": {
+        "declared_tags": [
+          "cc-by-nc-4.0"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "facebook/anli",
+            "revision": "8e4813d81f46d313dac7892e1c28076917cfcdf9",
+            "metadata_source": "https://huggingface.co/api/datasets/anli"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "R3 is the primary protocol; R1/R2 do not increase benchmark count.",
+        "NC declaration observed: evaluation-use basis must be independently resolved; this is not training permission or run approval."
+      ],
+      "definition": "lm_eval/tasks/anli/anli_r3.yaml"
+    },
+    {
+      "id": "T17",
+      "benchmark_id": "lambada",
+      "task": "lambada_openai",
+      "name": "LAMBADA",
+      "language": "EN",
+      "family": "lambada",
+      "domain": "contextual_prediction",
+      "metric": "acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "test",
+      "shots": 0,
+      "rationale": "contextual prediction",
+      "member_count": 1,
+      "source_record": "lambada",
+      "license": {
+        "declared_tags": [
+          "mit"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "EleutherAI/lambada_openai",
+            "revision": "900124bf3b8235c6daf21033af9948b3f07346c4",
+            "metadata_source": "https://huggingface.co/api/datasets/EleutherAI/lambada_openai"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [],
+      "definition": "lm_eval/tasks/lambada/lambada_openai.yaml"
+    },
+    {
+      "id": "T18",
+      "benchmark_id": "logiqa",
+      "task": "logiqa",
+      "name": "LogiQA",
+      "language": "EN",
+      "family": "logiqa",
+      "domain": "logical_reasoning",
+      "metric": "acc_norm,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "test",
+      "shots": 0,
+      "rationale": "logical reasoning",
+      "member_count": 1,
+      "source_record": "logiqa",
+      "license": {
+        "declared_tags": [
+          "other"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "EleutherAI/logiqa",
+            "revision": "c1eee1b2fd0ddc87777e36490f7ad54d0b97d141",
+            "metadata_source": "https://huggingface.co/api/datasets/EleutherAI/logiqa"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Undeclared/umbrella data terms are a gate, not an inferred permissive licence."
+      ],
+      "definition": "lm_eval/tasks/logiqa/logiqa.yaml"
+    },
+    {
+      "id": "T19",
+      "benchmark_id": "mathqa",
+      "task": "mathqa",
+      "name": "MathQA",
+      "language": "EN",
+      "family": "mathqa",
+      "domain": "mathematics_mc",
+      "metric": "acc_norm,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "test",
+      "shots": 0,
+      "rationale": "mathematics mc",
+      "member_count": 1,
+      "source_record": "mathqa",
+      "license": {
+        "declared_tags": [
+          "apache-2.0"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "allenai/math_qa",
+            "revision": "c4f1cc784c04c4957b50c97858f23893b633eea6",
+            "metadata_source": "https://huggingface.co/api/datasets/math_qa"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [],
+      "definition": "lm_eval/tasks/mathqa/mathqa.yaml"
+    },
+    {
+      "id": "T20",
+      "benchmark_id": "prost",
+      "task": "prost",
+      "name": "PROST",
+      "language": "EN",
+      "family": "prost",
+      "domain": "physical_reasoning",
+      "metric": "acc_norm,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "test",
+      "shots": 0,
+      "rationale": "physical reasoning",
+      "member_count": 1,
+      "source_record": "prost",
+      "license": {
+        "declared_tags": [
+          "apache-2.0"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "corypaik/prost",
+            "revision": "b3efebf08969fc19335ba894353316878b6fa493",
+            "metadata_source": "https://huggingface.co/api/datasets/corypaik/prost"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [],
+      "definition": "lm_eval/tasks/prost/corypaik_prost.yaml"
+    },
+    {
+      "id": "T21",
+      "benchmark_id": "sst2",
+      "task": "sst2",
+      "name": "SST-2",
+      "language": "EN",
+      "family": "sst2",
+      "domain": "sentiment",
+      "metric": "acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "validation",
+      "shots": 0,
+      "rationale": "sentiment",
+      "member_count": 1,
+      "source_record": "sst2",
+      "license": {
+        "declared_tags": [
+          "other"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "nyu-mll/glue",
+            "revision": "bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c",
+            "metadata_source": "https://huggingface.co/api/datasets/glue"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Undeclared/umbrella data terms are a gate, not an inferred permissive licence."
+      ],
+      "definition": "lm_eval/tasks/glue/sst2/default.yaml"
+    },
+    {
+      "id": "T22",
+      "benchmark_id": "polemo2",
+      "task": "polemo2_in_multiple_choice",
+      "name": "PoLEMo2 (in-domain MC)",
+      "language": "PL",
+      "family": "polemo2",
+      "domain": "sentiment",
+      "metric": "acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "openpl_polish4",
+      "entrypoint_kind": "task",
+      "split": "test",
+      "shots": 5,
+      "rationale": "sentiment",
+      "member_count": 1,
+      "source_record": "polemo2",
+      "license": {
+        "declared_tags": [
+          "cc-by-sa-4.0"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "allegro/klej-polemo2-in",
+            "revision": "9843af31facd613eb091deaa4141df4331d4f918",
+            "metadata_source": "https://huggingface.co/api/datasets/allegro/klej-polemo2-in"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Other response-format, domain or language protocols of this source are not extra main benchmarks."
+      ],
+      "definition": "lm_eval/tasks/polemo2/polemo2_in_mc.yaml"
+    },
+    {
+      "id": "T23",
+      "benchmark_id": "8tags",
+      "task": "polish_8tags_multiple_choice",
+      "name": "8tags",
+      "language": "PL",
+      "family": "8tags",
+      "domain": "topic_classification",
+      "metric": "acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "openpl_polish4",
+      "entrypoint_kind": "task",
+      "split": "test",
+      "shots": 5,
+      "rationale": "topic classification",
+      "member_count": 1,
+      "source_record": "8tags",
+      "license": {
+        "declared_tags": [
+          "cc-by-nc-sa-4.0"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "sdadas/8tags",
+            "revision": "82ddf9c0d06ffc982aeccf2473b7ce31f2167adf",
+            "metadata_source": "https://huggingface.co/api/datasets/sdadas/8tags"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Other response-format, domain or language protocols of this source are not extra main benchmarks.",
+        "NC declaration observed: evaluation-use basis must be independently resolved; this is not training permission or run approval."
+      ],
+      "definition": "lm_eval/tasks/polish_8tags/8tags_mc.yaml"
+    },
+    {
+      "id": "T24",
+      "benchmark_id": "belebele",
+      "task": "polish_belebele_mc",
+      "name": "Belebele-PL",
+      "language": "PL",
+      "family": "belebele",
+      "domain": "reading",
+      "metric": "acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "openpl_polish4",
+      "entrypoint_kind": "task",
+      "split": "test",
+      "shots": 5,
+      "rationale": "reading",
+      "member_count": 1,
+      "source_record": "belebele",
+      "license": {
+        "declared_tags": [
+          "cc-by-sa-4.0"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "facebook/belebele",
+            "revision": "7899cdfa4e1e0d733fd77c848e2c273cb1d32be2",
+            "metadata_source": "https://huggingface.co/api/datasets/facebook/belebele/revision/7899cdfa4e1e0d733fd77c848e2c273cb1d32be2"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Other response-format, domain or language protocols of this source are not extra main benchmarks."
+      ],
+      "definition": "lm_eval/tasks/polish_belebele/belebele_mc.yaml"
+    },
+    {
+      "id": "T25",
+      "benchmark_id": "dyk",
+      "task": "polish_dyk_multiple_choice",
+      "name": "CzyWiesz / DYK",
+      "language": "PL",
+      "family": "dyk",
+      "domain": "question_relevance",
+      "metric": "f1,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "openpl_polish4",
+      "entrypoint_kind": "task",
+      "split": "test",
+      "shots": 5,
+      "rationale": "question relevance",
+      "member_count": 1,
+      "source_record": "dyk",
+      "license": {
+        "declared_tags": [
+          "cc-by-sa-3.0"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "allegro/klej-dyk",
+            "revision": "dd7c3c3bb336738c853ea17c64b85d876e6714a9",
+            "metadata_source": "https://huggingface.co/api/datasets/allegro/klej-dyk"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Other response-format, domain or language protocols of this source are not extra main benchmarks."
+      ],
+      "definition": "lm_eval/tasks/polish_dyk/dyk_mc.yaml"
+    },
+    {
+      "id": "T26",
+      "benchmark_id": "ppc",
+      "task": "polish_ppc_multiple_choice",
+      "name": "Polish Paraphrase Corpus",
+      "language": "PL",
+      "family": "ppc",
+      "domain": "paraphrase",
+      "metric": "acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "openpl_polish4",
+      "entrypoint_kind": "task",
+      "split": "test",
+      "shots": 5,
+      "rationale": "paraphrase",
+      "member_count": 1,
+      "source_record": "ppc",
+      "license": {
+        "declared_tags": [
+          "cc-by-nc-sa-4.0"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "sdadas/ppc",
+            "revision": "073012dc81efb47b0fd66b0ab48bf06ddb62f528",
+            "metadata_source": "https://huggingface.co/api/datasets/sdadas/ppc"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Other response-format, domain or language protocols of this source are not extra main benchmarks.",
+        "NC declaration observed: evaluation-use basis must be independently resolved; this is not training permission or run approval."
+      ],
+      "definition": "lm_eval/tasks/polish_ppc/ppc_mc.yaml"
+    },
+    {
+      "id": "T27",
+      "benchmark_id": "psc",
+      "task": "polish_psc_multiple_choice",
+      "name": "Polish Summaries Corpus (matching)",
+      "language": "PL",
+      "family": "psc",
+      "domain": "summary_matching",
+      "metric": "f1,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "openpl_polish4",
+      "entrypoint_kind": "task",
+      "split": "test",
+      "shots": 5,
+      "rationale": "summary matching",
+      "member_count": 1,
+      "source_record": "psc",
+      "license": {
+        "declared_tags": [
+          "cc-by-sa-3.0"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "allegro/klej-psc",
+            "revision": "9585582aef37e7747ef174b0ec3b74855da8bad2",
+            "metadata_source": "https://huggingface.co/api/datasets/allegro/klej-psc"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Other response-format, domain or language protocols of this source are not extra main benchmarks."
+      ],
+      "definition": "lm_eval/tasks/polish_psc/psc_mc.yaml"
+    },
+    {
+      "id": "T28",
+      "benchmark_id": "cbd",
+      "task": "polish_cbd_multiple_choice",
+      "name": "Polish Cyberbullying Dataset",
+      "language": "PL",
+      "family": "cbd",
+      "domain": "toxicity_detection",
+      "metric": "f1,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "openpl_polish4",
+      "entrypoint_kind": "task",
+      "split": "test",
+      "shots": 5,
+      "rationale": "toxicity detection",
+      "member_count": 1,
+      "source_record": "cbd",
+      "license": {
+        "declared_tags": [
+          "cc-by-4.0"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "ptaszynski/PolishCyberbullyingDataset",
+            "revision": "be3b1236598a02eff6db05ea7ac386afaabc33e8",
+            "metadata_source": "https://huggingface.co/api/datasets/ptaszynski/PolishCyberbullyingDataset"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Other response-format, domain or language protocols of this source are not extra main benchmarks."
+      ],
+      "definition": "lm_eval/tasks/polish_cbd/cbd_mc.yaml"
+    },
+    {
+      "id": "T29",
+      "benchmark_id": "klej_ner",
+      "task": "polish_klej_ner_multiple_choice",
+      "name": "KLEJ NKJP-NER (sentence-level)",
+      "language": "PL",
+      "family": "klej_ner",
+      "domain": "sentence_entity_classification",
+      "metric": "acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "openpl_polish4",
+      "entrypoint_kind": "task",
+      "split": "test",
+      "shots": 5,
+      "rationale": "entity recognition",
+      "member_count": 1,
+      "source_record": "klej_ner",
+      "license": {
+        "declared_tags": [
+          "unknown"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "allegro/klej-nkjp-ner",
+            "revision": "bfae4047738b985bc2ad533d339dd89c3f7de00d",
+            "metadata_source": "https://huggingface.co/api/datasets/allegro/klej-nkjp-ner/revision/bfae4047738b985bc2ad533d339dd89c3f7de00d"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Other response-format, domain or language protocols of this source are not extra main benchmarks.",
+        "Undeclared/umbrella data terms are a gate, not an inferred permissive licence."
+      ],
+      "definition": "lm_eval/tasks/polish_klej_ner/klej_ner_mc.yaml"
+    },
+    {
+      "id": "T30",
+      "benchmark_id": "polqa",
+      "task": "polish_polqa_open_book",
+      "name": "PolQA (open book)",
+      "language": "PL",
+      "family": "polqa",
+      "domain": "question_answering",
+      "metric": "levenshtein,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "openpl_polish4",
+      "entrypoint_kind": "task",
+      "split": "test",
+      "shots": 5,
+      "rationale": "question answering",
+      "member_count": 1,
+      "source_record": "polqa",
+      "license": {
+        "declared_tags": [
+          "cc-by-sa-4.0"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "ipipan/polqa",
+            "revision": "d78d036ef08ab3b9f4d85a2893f4d3a0c95a6f37",
+            "metadata_source": "https://huggingface.co/api/datasets/ipipan/polqa"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Other response-format, domain or language protocols of this source are not extra main benchmarks."
+      ],
+      "definition": "lm_eval/tasks/polish_polqa/polqa_open_book.yaml"
+    },
+    {
+      "id": "T31",
+      "benchmark_id": "poquad",
+      "task": "polish_poquad_open_book",
+      "name": "PoQuAD",
+      "language": "PL",
+      "family": "poquad",
+      "domain": "question_answering",
+      "metric": "levenshtein,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "openpl_polish4",
+      "entrypoint_kind": "task",
+      "split": "validation",
+      "shots": 5,
+      "rationale": "question answering",
+      "member_count": 1,
+      "source_record": "poquad",
+      "license": {
+        "declared_tags": [
+          "cc-by-4.0"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "clarin-pl/poquad",
+            "revision": "a60f228d836fbbf990d11b4590c60b3e7f2aec2d",
+            "metadata_source": "https://huggingface.co/api/datasets/clarin-pl/poquad"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Other response-format, domain or language protocols of this source are not extra main benchmarks."
+      ],
+      "definition": "lm_eval/tasks/polish_poquad/poquad_open_book.yaml"
+    },
+    {
+      "id": "T32",
+      "benchmark_id": "mmlu",
+      "task": "mmlu",
+      "name": "MMLU",
+      "language": "EN",
+      "family": "mmlu",
+      "domain": "academic_knowledge",
+      "metric": "acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "group",
+      "split": "test",
+      "shots": 5,
+      "rationale": "academic knowledge",
+      "member_count": 57,
+      "source_record": "mmlu",
+      "license": {
+        "declared_tags": [
+          "mit"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "hails/mmlu_no_train",
+            "revision": "b2e1ec9aa795adafe68e8e983248dbd4b52a1c60",
+            "metadata_source": "https://huggingface.co/api/datasets/hails/mmlu_no_train"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "57 subjects count as one benchmark; MMLU-Pro and translations are not additional sources."
+      ],
+      "definition": "lm_eval/tasks/mmlu/default/_mmlu.yaml"
+    },
+    {
+      "id": "T33",
+      "benchmark_id": "gsm8k",
+      "task": "gsm8k",
+      "name": "GSM8K",
+      "language": "EN",
+      "family": "gsm8k",
+      "domain": "mathematics_generation",
+      "metric": "exact_match,strict-match",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "test",
+      "shots": 5,
+      "rationale": "mathematics generation",
+      "member_count": 1,
+      "source_record": "gsm8k",
+      "license": {
+        "declared_tags": [
+          "mit"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "openai/gsm8k",
+            "revision": "740312add88f781978c0658806c59bc2815b9866",
+            "metadata_source": "https://huggingface.co/api/datasets/gsm8k"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [],
+      "definition": "lm_eval/tasks/gsm8k/gsm8k.yaml"
+    },
+    {
+      "id": "T34",
+      "benchmark_id": "math",
+      "task": "hendrycks_math",
+      "name": "MATH",
+      "language": "EN",
+      "family": "math",
+      "domain": "competition_mathematics",
+      "metric": "exact_match,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "group",
+      "split": "test",
+      "shots": 0,
+      "rationale": "competition mathematics",
+      "member_count": 7,
+      "source_record": "math",
+      "license": {
+        "declared_tags": [
+          "mit"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "EleutherAI/hendrycks_math",
+            "revision": "21a5633873b6a120296cce3e2df9d5550074f4a3",
+            "metadata_source": "https://huggingface.co/api/datasets/EleutherAI/hendrycks_math"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [],
+      "definition": "lm_eval/tasks/hendrycks_math/hendrycks_math.yaml",
+      "scoring_notes": [
+        "source_extractor_floor"
+      ]
+    },
+    {
+      "id": "T35",
+      "benchmark_id": "truthfulqa",
+      "task": "truthfulqa_mc2",
+      "name": "TruthfulQA (MC2)",
+      "language": "EN",
+      "family": "truthfulqa",
+      "domain": "truthfulness",
+      "metric": "acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "validation",
+      "shots": 0,
+      "rationale": "truthfulness",
+      "member_count": 1,
+      "source_record": "truthfulqa",
+      "license": {
+        "declared_tags": [
+          "apache-2.0"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "truthfulqa/truthful_qa",
+            "revision": "741b8276f2d1982aa3d5b832d3ee81ed3b896490",
+            "metadata_source": "https://huggingface.co/api/datasets/truthful_qa"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "MC2 is normalized probability mass on true answers, not MC1 classification accuracy; source fixed QA preamble retained."
+      ],
+      "definition": "lm_eval/tasks/truthfulqa/truthfulqa_mc2.yaml"
+    },
+    {
+      "id": "T36",
+      "benchmark_id": "ifeval",
+      "task": "ifeval",
+      "name": "IFEval",
+      "language": "EN",
+      "family": "ifeval",
+      "domain": "instruction_following",
+      "metric": "prompt_level_strict_acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "train",
+      "shots": 0,
+      "rationale": "instruction following",
+      "member_count": 1,
+      "source_record": "ifeval",
+      "license": {
+        "declared_tags": [
+          "apache-2.0"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "google/IFEval",
+            "revision": "966cd89545d6b6acfd7638bc708b98261ca58e84",
+            "metadata_source": "https://huggingface.co/api/datasets/google/IFEval"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [],
+      "definition": "lm_eval/tasks/ifeval/ifeval.yaml"
+    },
+    {
+      "id": "T37",
+      "benchmark_id": "llmzszl",
+      "task": "n05_llmzszl",
+      "name": "LLMzSzŁ",
+      "language": "PL",
+      "family": "llmzszl",
+      "domain": "native_exam_knowledge",
+      "metric": "acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "n05_static_yaml",
+      "entrypoint_kind": "local_yaml",
+      "split": "test",
+      "shots": 0,
+      "rationale": "native exam knowledge",
+      "member_count": 1,
+      "source_record": "llmzszl",
+      "license": {
+        "declared_tags": [
+          "unknown"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "amu-cai/llmzszl-dataset",
+            "revision": "9c11d52a415147f1e2c44038a9d4d4155f064e1a",
+            "metadata_source": "https://huggingface.co/api/datasets/amu-cai/llmzszl-dataset"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Dataset card has no license tag: internal evaluation and aggregate export only, per the data-use review. Known training overlap requires revision/ID-bound audit and common clean-subset decision.",
+        "Undeclared/umbrella data terms are a gate, not an inferred permissive licence."
+      ],
+      "definition": "tasks/llmzszl.yaml",
+      "contamination_status": "known overlap reported; exact current revision and ID-bound clean-cohort audit required"
+    },
+    {
+      "id": "T38",
+      "benchmark_id": "squad2",
+      "task": "squadv2",
+      "name": "SQuAD 2.0",
+      "language": "EN",
+      "family": "squad2",
+      "domain": "unanswerable_reading",
+      "metric": "f1,none",
+      "unit": "percent",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "legacy_task",
+      "split": "validation",
+      "shots": 0,
+      "rationale": "unanswerable reading",
+      "member_count": 1,
+      "source_record": "squad2",
+      "license": {
+        "declared_tags": [
+          "cc-by-sa-4.0"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "rajpurkar/squad_v2",
+            "revision": "3ffb306f725f7d2ce8394bc1873b24868140c412",
+            "metadata_source": "https://huggingface.co/api/datasets/squad_v2"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Do not also count QNLI, XQuAD or SQuAD completion. Preserve the legacy Task and pinned squad_v2 metric, not a fabricated ConfigurableTask replacement."
+      ],
+      "definition": "lm_eval/tasks/squadv2/task.py"
+    },
+    {
+      "id": "T39",
+      "benchmark_id": "drop",
+      "task": "drop",
+      "name": "DROP",
+      "language": "EN",
+      "family": "drop",
+      "domain": "discrete_reading",
+      "metric": "f1,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "validation",
+      "shots": 0,
+      "rationale": "discrete reading",
+      "member_count": 1,
+      "source_record": "drop",
+      "license": {
+        "declared_tags": [
+          "cc-by-4.0"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "EleutherAI/drop",
+            "revision": "619386fc7ffe6dace3cc6d3483c69b164ebf6bf3",
+            "metadata_source": "https://huggingface.co/api/datasets/EleutherAI/drop"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [],
+      "definition": "lm_eval/tasks/drop/default.yaml"
+    },
+    {
+      "id": "T40",
+      "benchmark_id": "race",
+      "task": "race",
+      "name": "RACE (high primary)",
+      "language": "EN",
+      "family": "race",
+      "domain": "reading",
+      "metric": "acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "test",
+      "shots": 0,
+      "rationale": "reading",
+      "member_count": 1,
+      "source_record": "race",
+      "license": {
+        "declared_tags": [
+          "other"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "EleutherAI/race",
+            "revision": "66ca2e6374282f5e35b11559e87185db911a6900",
+            "metadata_source": "https://huggingface.co/api/datasets/EleutherAI/race"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Undeclared/umbrella data terms are a gate, not an inferred permissive licence.",
+        "Pinned harness high protocol includes earlier article Q/A context; not a claim of full original RACE middle+high paper parity."
+      ],
+      "definition": "lm_eval/tasks/race/race.yaml"
+    },
+    {
+      "id": "T41",
+      "benchmark_id": "coqa",
+      "task": "coqa",
+      "name": "CoQA",
+      "language": "EN",
+      "family": "coqa",
+      "domain": "conversational_reading",
+      "metric": "f1,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "validation",
+      "shots": 0,
+      "rationale": "conversational reading",
+      "member_count": 1,
+      "source_record": "coqa",
+      "license": {
+        "declared_tags": [
+          "other"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "EleutherAI/coqa",
+            "revision": "82e11af842af6c1396f5e9a5c7de260107c50cf1",
+            "metadata_source": "https://huggingface.co/api/datasets/EleutherAI/coqa"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Undeclared/umbrella data terms are a gate, not an inferred permissive licence."
+      ],
+      "definition": "lm_eval/tasks/coqa/default.yaml"
+    },
+    {
+      "id": "T42",
+      "benchmark_id": "nq_open",
+      "task": "nq_open",
+      "name": "Natural Questions Open",
+      "language": "EN",
+      "family": "nq_open",
+      "domain": "closed_book_knowledge",
+      "metric": "exact_match,remove_whitespace",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "validation",
+      "shots": 0,
+      "rationale": "closed book knowledge",
+      "member_count": 1,
+      "source_record": "nq_open",
+      "license": {
+        "declared_tags": [
+          "cc-by-sa-3.0"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "google-research-datasets/nq_open",
+            "revision": "5dd9790a83002ad084ddeb7c420dc716852c6f28",
+            "metadata_source": "https://huggingface.co/api/datasets/nq_open"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [],
+      "definition": "lm_eval/tasks/nq_open/nq_open.yaml"
+    },
+    {
+      "id": "T43",
+      "benchmark_id": "triviaqa",
+      "task": "triviaqa",
+      "name": "TriviaQA",
+      "language": "EN",
+      "family": "triviaqa",
+      "domain": "closed_book_knowledge",
+      "metric": "exact_match,remove_whitespace",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "validation",
+      "shots": 0,
+      "rationale": "closed book knowledge",
+      "member_count": 1,
+      "source_record": "triviaqa",
+      "license": {
+        "declared_tags": [
+          "unknown"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "mandarjoshi/trivia_qa",
+            "revision": "0f7faf33a3908546c6fd5b73a660e0f8ff173c2f",
+            "metadata_source": "https://huggingface.co/api/datasets/mandarjoshi/trivia_qa/revision/0f7faf33a3908546c6fd5b73a660e0f8ff173c2f"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Undeclared/umbrella data terms are a gate, not an inferred permissive licence."
+      ],
+      "definition": "lm_eval/tasks/triviaqa/default.yaml"
+    },
+    {
+      "id": "T44",
+      "benchmark_id": "pubmedqa",
+      "task": "pubmedqa",
+      "name": "PubMedQA",
+      "language": "EN",
+      "family": "pubmedqa",
+      "domain": "biomedical_reading",
+      "metric": "acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "test",
+      "shots": 0,
+      "rationale": "biomedical reading",
+      "member_count": 1,
+      "source_record": "pubmedqa",
+      "license": {
+        "declared_tags": [
+          "mit"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "bigbio/pubmed_qa",
+            "revision": "15578e80ef02c2c9636d33fdd7eb6b05b987ba60",
+            "metadata_source": "https://huggingface.co/api/datasets/bigbio/pubmed_qa"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [],
+      "definition": "lm_eval/tasks/pubmedqa/pubmedqa.yaml"
+    },
+    {
+      "id": "T45",
+      "benchmark_id": "medmcqa",
+      "task": "medmcqa",
+      "name": "MedMCQA",
+      "language": "EN",
+      "family": "medmcqa",
+      "domain": "medical_exam_knowledge",
+      "metric": "acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "validation",
+      "shots": 0,
+      "rationale": "medical exam knowledge",
+      "member_count": 1,
+      "source_record": "medmcqa",
+      "license": {
+        "declared_tags": [
+          "apache-2.0"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "openlifescienceai/medmcqa",
+            "revision": "91c6572c454088bf71b679ad90aa8dffcd0d5868",
+            "metadata_source": "https://huggingface.co/api/datasets/medmcqa"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [],
+      "definition": "lm_eval/tasks/medmcqa/medmcqa.yaml"
+    },
+    {
+      "id": "T46",
+      "benchmark_id": "social_iqa",
+      "task": "social_iqa",
+      "name": "Social IQa",
+      "language": "EN",
+      "family": "social_iqa",
+      "domain": "social_reasoning",
+      "metric": "acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "validation",
+      "shots": 0,
+      "rationale": "social reasoning",
+      "member_count": 1,
+      "source_record": "social_iqa",
+      "license": {
+        "declared_tags": [
+          "unknown"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "allenai/social_i_qa",
+            "revision": "8835ceb9141d7896d9d968634a9b21ae440e3ec5",
+            "metadata_source": "https://huggingface.co/api/datasets/allenai/social_i_qa/revision/8835ceb9141d7896d9d968634a9b21ae440e3ec5"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Undeclared/umbrella data terms are a gate, not an inferred permissive licence."
+      ],
+      "definition": "lm_eval/tasks/siqa/siqa.yaml"
+    },
+    {
+      "id": "T47",
+      "benchmark_id": "bbh",
+      "task": "bbh_fewshot",
+      "name": "BIG-Bench Hard (direct)",
+      "language": "EN",
+      "family": "bbh",
+      "domain": "formal_reasoning",
+      "metric": "exact_match,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "group",
+      "split": "test",
+      "shots": 3,
+      "rationale": "formal reasoning",
+      "member_count": 27,
+      "source_record": "bbh",
+      "license": {
+        "declared_tags": [
+          "unknown"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "lukaemon/bbh",
+            "revision": "982bb89fd79532a8ac676a61fc42eb1aeec63f99",
+            "metadata_source": "https://huggingface.co/api/datasets/lukaemon/bbh/revision/982bb89fd79532a8ac676a61fc42eb1aeec63f99"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "27 subtasks count once; direct source-provided fixed examples, not an additional CoT benchmark.",
+        "Undeclared/umbrella data terms are a gate, not an inferred permissive licence."
+      ],
+      "definition": "lm_eval/tasks/bbh/fewshot/_bbh_fewshot.yaml",
+      "scoring_amendments": [
+        "answer_leading_space_strip"
+      ]
+    },
+    {
+      "id": "T48",
+      "benchmark_id": "paws",
+      "task": "paws_en",
+      "name": "PAWS (English PAWS-X protocol)",
+      "language": "EN",
+      "family": "paws",
+      "domain": "adversarial_paraphrase",
+      "metric": "acc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "test",
+      "shots": 0,
+      "rationale": "adversarial paraphrase",
+      "member_count": 1,
+      "source_record": "paws",
+      "license": {
+        "declared_tags": [
+          "other"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "google-research-datasets/paws-x",
+            "revision": "4cd8187c404bda33cb1f62b49b001115862acf37",
+            "metadata_source": "https://huggingface.co/api/datasets/paws-x"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "Undeclared/umbrella data terms are a gate, not an inferred permissive licence."
+      ],
+      "definition": "lm_eval/tasks/paws-x/paws_en.yaml"
+    },
+    {
+      "id": "T49",
+      "benchmark_id": "cola",
+      "task": "cola",
+      "name": "CoLA",
+      "language": "EN",
+      "family": "cola",
+      "domain": "acceptability",
+      "metric": "mcc,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "validation",
+      "shots": 0,
+      "rationale": "acceptability",
+      "member_count": 1,
+      "source_record": "cola",
+      "license": {
+        "declared_tags": [
+          "other"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "nyu-mll/glue",
+            "revision": "bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c",
+            "metadata_source": "https://huggingface.co/api/datasets/glue"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "MCC may be negative; do not clamp to accuracy0..1.",
+        "Undeclared/umbrella data terms are a gate, not an inferred permissive licence."
+      ],
+      "definition": "lm_eval/tasks/glue/cola/default.yaml"
+    },
+    {
+      "id": "T50",
+      "benchmark_id": "record",
+      "task": "record",
+      "name": "ReCoRD",
+      "language": "EN",
+      "family": "record",
+      "domain": "entity_reading",
+      "metric": "f1,none",
+      "unit": "score",
+      "higher_is_better": true,
+      "instrument": "lm_eval_0.4.8",
+      "entrypoint_kind": "task",
+      "split": "validation",
+      "shots": 0,
+      "rationale": "entity reading",
+      "member_count": 1,
+      "source_record": "record",
+      "license": {
+        "declared_tags": [
+          "other"
+        ],
+        "basis": "revision-bound dataset-card metadata, not legal or execution approval",
+        "source_revisions": [
+          {
+            "repo_id": "aps/super_glue",
+            "revision": "3de24cf8022e94f4ee4b9d55a6f539891524d646",
+            "metadata_source": "https://huggingface.co/api/datasets/super_glue"
+          }
+        ]
+      },
+      "data_gate": "required_before_evaluation",
+      "notes": [
+        "News passages may overlap broad media sources of CoQA/RACE; the curated entity-cloze questions are a distinct benchmark, not a translation or answer-format variant.",
+        "Undeclared/umbrella data terms are a gate, not an inferred permissive licence."
+      ],
+      "definition": "lm_eval/tasks/super_glue/record/default.yaml"
+    }
+  ],
+  "verification": {
+    "source_identities": 50,
+    "source_members": 204,
+    "repository_revision_metadata": 44,
+    "declaration_files": 218,
+    "execution_ready": false,
+    "model_quality_measured": false
+  }
+}