README.md

metadata

title: Slayer1B Benchmarks
emoji: 📏
colorFrom: green
colorTo: blue
sdk: static
pinned: false
short_description: 50 distinct benchmarks, one protocol, contamination-checked

Slayer1B Benchmarks

Fifty distinct public benchmarks (38 English, 12 Polish), run with one protocol on Slayer 1B and comparison models. This Space is a static page: it shows the published result file and computes nothing itself.

What is on the page

• One value per benchmark and model, in the benchmark's own metric and unit. Every main value and every model-vs-model contrast has a 95% confidence interval; benchmarks scored by picking one answer also show reference points (random guess, always-majority class); values on items without contamination hits are shown without an interval.
• No average over the fifty benchmarks and no overall ranking.
• Contamination: for our own models the full training corpus is scanned against every scored item before a number is shown. Benchmarks with hits are labelled and shown on the full sample and on the items without hits. Models whose training data is not public are labelled training data unknown.
• Better / worse between two models is stated only from a paired permutation test with Holm correction over the whole comparison family; otherwise the result is inconclusive. Benchmarks where a model gives the same answer to every item are not compared.
• Scoring corrections made after the first run are labelled on the page and described in METHOD.md, together with when and why they were made.

Files

file	content
tasks_50.json	benchmark list: sources, pinned revisions, metrics, licences
METHOD.md	protocol, sampling, uncertainty, contamination scan, corrections
results.schema.json	format of the result file
contamination_field_roles.json	which item fields count as context or as answer for the contamination variants
results-feed.json	pointer to the published result file and its SHA-256
results.json	published results; the page checks every file against its pinned SHA-256 before showing anything

Benchmark texts are not redistributed here; each benchmark keeps its own licence (see tasks_50.json). Only aggregate results are published.

Fabryka AI · SlayerLab