Blame for README.md

---
title: Slayer1B Benchmarks
emoji: 📏
colorFrom: green
colorTo: blue
sdk: static
pinned: false
short_description: 50 distinct benchmarks, one protocol, contamination-checked
---

# Slayer1B Benchmarks

Fifty distinct public benchmarks (38 English, 12 Polish), run with one protocol on
[Slayer 1B](https://huggingface.co/spaces/SlayerLab/Slayer1B) and comparison models.
This Space is a static page: it shows the published result file and computes nothing itself.

## What is on the page

- **One value per benchmark and model**, in the benchmark's own metric and unit. Every main value and every model-vs-model contrast has a 95% confidence interval; benchmarks scored by picking one answer also show reference points (random guess, always-majority class); values on items without contamination hits are shown without an interval.
- **No average over the fifty benchmarks** and no overall ranking.
- **Contamination:** for our own models the full training corpus is scanned against every scored item before a number is shown. Benchmarks with hits are labelled and shown on the full sample and on the items without hits. Models whose training data is not public are labelled *training data unknown*.
- **Better / worse** between two models is stated only from a paired permutation test with Holm correction over the whole comparison family; otherwise the result is *inconclusive*. Benchmarks where a model gives the same answer to every item are not compared.
- **Scoring corrections** made after the first run are labelled on the page and described in `METHOD.md`, together with when and why they were made.

## Files

| file | content |
|---|---|
| `tasks_50.json` | benchmark list: sources, pinned revisions, metrics, licences |
| `METHOD.md` | protocol, sampling, uncertainty, contamination scan, corrections |
| `results.schema.json` | format of the result file |
| `contamination_field_roles.json` | which item fields count as context or as answer for the contamination variants |
| `results-feed.json` | pointer to the published result file and its SHA-256 |
| `results.json` | published results; the page checks every file against its pinned SHA-256 before showing anything |

Benchmark texts are not redistributed here; each benchmark keeps its own licence (see `tasks_50.json`).
Only aggregate results are published.

Fabryka AI · SlayerLab