Spaces:
Running
Running
| title: Slayer1B Benchmarks | |
| emoji: 📏 | |
| colorFrom: green | |
| colorTo: blue | |
| sdk: static | |
| pinned: false | |
| short_description: 50 distinct benchmarks, one protocol, contamination-checked | |
| # Slayer1B Benchmarks | |
| Fifty distinct public benchmarks (38 English, 12 Polish), run with one protocol on | |
| [Slayer 1B](https://huggingface.co/spaces/SlayerLab/Slayer1B) and comparison models. | |
| This Space is a static page: it shows the published result file and computes nothing itself. | |
| ## What is on the page | |
| - **One value per benchmark and model**, in the benchmark's own metric and unit. Every main value and every model-vs-model contrast has a 95% confidence interval; benchmarks scored by picking one answer also show reference points (random guess, always-majority class); values on items without contamination hits are shown without an interval. | |
| - **No average over the fifty benchmarks** and no overall ranking. | |
| - **Contamination:** for our own models the full training corpus is scanned against every scored item before a number is shown. Benchmarks with hits are labelled and shown on the full sample and on the items without hits. Models whose training data is not public are labelled *training data unknown*. | |
| - **Better / worse** between two models is stated only from a paired permutation test with Holm correction over the whole comparison family; otherwise the result is *inconclusive*. Benchmarks where a model gives the same answer to every item are not compared. | |
| - **Scoring corrections** made after the first run are labelled on the page and described in `METHOD.md`, together with when and why they were made. | |
| ## Files | |
| | file | content | | |
| |---|---| | |
| | `tasks_50.json` | benchmark list: sources, pinned revisions, metrics, licences | | |
| | `METHOD.md` | protocol, sampling, uncertainty, contamination scan, corrections | | |
| | `results.schema.json` | format of the result file | | |
| | `contamination_field_roles.json` | which item fields count as context or as answer for the contamination variants | | |
| | `results-feed.json` | pointer to the published result file and its SHA-256 | | |
| | `results.json` | published results; the page checks every file against its pinned SHA-256 before showing anything | | |
| Benchmark texts are not redistributed here; each benchmark keeps its own licence (see `tasks_50.json`). | |
| Only aggregate results are published. | |
| Fabryka AI · SlayerLab | |