README.md

---
title: CricketBench
---

# CricketBench

A domain benchmark for **cricket reasoning in large language models**.

Cricket is a game of dense rules, sharp situations, deep records, precise
terminology, and layered strategy. It's rewarding to model but easy to fake:
most existing cricket "benchmarks" test surface trivia rather than reasoning.
CricketBench targets the reasoning gap directly.

## What's here

- **[Dataset](https://huggingface.co/datasets/cricketbench/cricketbench)** —
  The canonical multiple-choice questions, versioned by release, organised
  into five reasoning dimensions:
    - `laws_and_rule_precision` — the 42 Laws of Cricket + ICC Playing Conditions
    - `match_situation_reasoning` — run rates, DLS, follow-on, powerplay
    - `player_and_statistical_recall` — records under reasoning, not trivia
    - `commentary_generation_quality` — terminology, shot/delivery identification
    - `tactical_and_strategic_reasoning` — captaincy, field settings, decisions
- **[Leaderboard](https://huggingface.co/spaces/cricketbench/cricketbench-leaderboard)**
  — Per-model accuracy and per-dimension scores across the current release.
- **[Baselines](https://huggingface.co/datasets/cricketbench/cricketbench-baselines)**
  — Per-model raw answers, kept alongside so anyone can reproduce
  discrimination scores from the same runs.

## How it's built

Every question passes through a closed loop: hand-authored seeds →
LLM-generated variants → red-team critique → answered by 8+ models across the
capability gradient → per-question item-discrimination (point-biserial r
against per-model ability) → human review. Only items that survive the human
review land in the dataset.

Design decisions in one line each:

- **Per-model item discrimination, not frontier-vs-open buckets** — capability
  is measured from the data, not assigned by us.
- **Unparseable model outputs are excluded from scoring**, not counted wrong.
- **Red-team `major` severity routes to human revise** regardless of the
  statistical signal.
- **Top-model-miss diagnostic** flags items where the strongest legitimate
  model missed while ≥2 mid-tier models got it right — usually a sign the
  key is wrong.
- **Model rankings require n ≥ 30 per dimension** before we consider them
  publishable.

## Where it comes from

CricketBench is a mindspace, while growing up in Chennai. The mindspace goes
beyond winning and losing and primarily revolves around the intricacies and
depth of the game. I am sure this is the case in any cricket-obsessed
environment. I want to bring like minds to bring a strong bench that helps
models compete for intelligence in one of the top sports in the world.

## How to help

- **Contribute questions** — open an issue with a candidate item, or submit a
  PR against the dataset.
- **Contribute baseline model results** — submit a JSONL of predictions to the
  leaderboard Space.
- **Contribute revisions** — flag ambiguous items or wrong keys via HF
  Discussion.
- **Contribute reviews** — most valuable of all. The pipeline generates faster
  than a single person can review.

## Maintainers

- **[Hariprasad Sudharshan](https://huggingface.co/brettleehari)** — Founder
  and project lead. Senior product manager, based in Chennai. Responsible for
  taxonomy, scoring policies, and question review discipline.

Interested in becoming a maintainer? Contribute a batch of reviewed questions
or a set of high-quality baseline model results and open a Discussion —
sustained contributors are invited to write access on the org.

## Citation

```bibtex
@misc{cricketbench2026,
  title        = {{CricketBench}: A domain benchmark for cricket reasoning
                  in large language models},
  author       = {Sudharshan, Hariprasad and {CricketBench contributors}},
  year         = {2026},
  howpublished = {Hugging Face Datasets},
  url          = {https://huggingface.co/datasets/cricketbench/cricketbench},
  version      = {v0.0}
}
```

## Contact

Discussions on the HF Community tab of any `cricketbench/*` repo.

---

Licensed CC-BY-SA 4.0. Every version is git-tagged; papers should cite a
specific release.