Scorio Math

Scorio Math contains 59,520 sampled attempts from four model configurations and five competition-math benchmarks. Each model was run 80 times on every question. One Parquet file contains the 80 attempts for one model and one question, ordered by seed.

Each token position includes the full top-20 candidate distribution. These distributions are needed to reproduce token-level confidence measures such as self_certainty, deepconf_confidence, token_entropy, varentropy, max_softmax_probability, and logprob_margin. If these distributions are not needed, use harimo/scorio-lite, which contains the same attempts without the top-20 lists.

Quick start

The data is stored in a Hugging Face Storage Bucket. Install current versions of datasets, huggingface_hub, and pyarrow before loading it.

pip install -U datasets huggingface_hub pyarrow

from datasets import load_dataset

data_files = {
    "cmimc_2025": "data/gpt-oss-20b_high/cmimc_2025/*.parquet",
}
ds = load_dataset(
    "buckets/harimo/scorio-math",
    data_files=data_files,
    split="cmimc_2025",
    streaming=True,
)
record = next(iter(ds))
candidates = record["tokens"]["completion_topk_logprobs_list"][12]

Rows with token distributions can be large. Use streaming, or select only the columns you need when reading an individual file:

import pyarrow.parquet as pq

t = pq.read_table(
    "hf://buckets/harimo/scorio-math/data/gpt-oss-20b_high/cmimc_2025/q17.parquet",
    columns=["seed", "evalscope_is_correct", "cv3b_label"],
)
acc80 = sum(t["evalscope_is_correct"].to_pylist()) / 80

How to use

Install Scorio to evaluate models, rank them, or select an answer from a candidate pool:

pip install scorio

Scorio uses NumPy arrays whose dimensions match the structure of this dataset. Here, M is the number of questions, N is the number of attempts per question, and L is the number of model configurations. The dataset walkthrough shows how to load candidate pools and work with the five competition-math benchmarks.

Evaluation APIs

Use scorio.eval to score one model. Build an M x N outcome matrix from evalscope_is_correct, with one row per question and one column per seed. The module includes average accuracy, Bayes@N, credible intervals, Pass@k, Maj@k, and related metrics. See the evaluation notebook for a complete example using an AIME 2026 question window.

from scorio import eval

# R has shape (questions, seeds), with entries in {0, 1}
mu, sigma, lower, upper = eval.bayes_ci(R)
pass_at_8 = eval.pass_at_k(R, 8)

Ranking APIs

Use scorio.rank to compare models. Stack the outcome matrices into an L x M x N array, ordered by model, question, and seed. Ranking methods can return both ranks and the scores used to produce them. See the ranking notebook for a complete example using all four model configurations.

from scorio import rank

# R_all has shape (models, questions, seeds)
ranks, scores = rank.bayes(R_all, return_scores=True)

Aggregation APIs

Use scorio.aggregate, also available as scorio.agg, to choose one answer from each candidate pool. Pass an M x N array built from extracted_answer. Score-based methods take a second array of the same shape, using a verifier score or another confidence score. See the aggregation notebook for examples of voting and verifier-based selection.

from scorio import agg

# answers and verifier_scores both have shape (questions, seeds)
majority_answers = agg.majority_vote(answers)
best_answers = agg.best_of_n(answers, verifier_scores)

Layout

data/<model>/<task>/qNN.parquet

There are four model configurations. Each configuration has five splits. Every file has 80 rows in ascending seed order.

Models

The three gpt-oss configurations share the same model value. Use model_key when grouping them. The reasoning level is also stored in sampling.reasoning_effort.

Schema

The columns match the per-model math configurations in harimo/scorio-lite. They include the generation, sampling settings, rule-based grading, CompassVerifier-3B scores, and scores from the reference-free verifier. Scorio Math adds two fields inside tokens:

Each entry is {token: string, token_id: int32, logprob: float64, rank: int32}. Prompt-side length is num_prompt_tokens, completion-side num_completion_tokens.

Eight source fields are not stored because they are exact functions of retained columns: cv3b_prob_label, cv3b_reward, evalscope_acc, the three llmv_<c>_reward fields, llmv_pointwise_reward, and llmv_mean_expected_raw_score_1_to_20. They can be recovered with:

criteria = ["problem_understanding", "reasoning_validity", "conclusion_support"]
rewards = [(record[f"llmv_{c}_expected"] - 1) / 19 for c in criteria]
pointwise_reward = sum(rewards) / 3
cv3b_reward = float(record["cv3b_label"] == "A")
evalscope_acc = float(record["evalscope_is_correct"])

See harimo/scorio-lite for the full table.

OpenCompass verifier scores

The cv3b_* fields come from opencompass/CompassVerifier-3B. It sees the question, reference answer, and candidate response, then judges the final answer as A (correct), B (incorrect), or C (invalid). cv3b_label is the highest-logprob A/B/C choice, cv3b_prob is its probability in the full vocabulary, and cv3b_abc_A/B/C renormalize the three label probabilities. The separate cv3b_ctx_A/B/C diagnostic subtracts the model's A/B/C log probabilities on a null [N/A] prompt from those on the real prompt, divides by 1.5, and applies softmax.

LLM-as-a-verifier

The separate reference-free verifier uses Qwen/Qwen3.6-35B-A3B. It sees only the prompt and response, and scores problem understanding, reasoning validity, and conclusion support. For each criterion, it produces an A-to-T score-token distribution (A=20, ..., T=1) and stores its expected value in llmv_<criterion>_expected. A criterion reward is (expected_score - 1) / 19, and the overall pointwise reward is the mean of the three criterion rewards. These derived rewards are not stored. This is a graded quality score, not a calibrated probability of correctness.

Top-k list conventions

The prompt and completion candidate lists follow different conventions inherited from the source files.

The following measurements were made on aime_2026/q00. They describe one question and show how often the completion list starts with a non-argmax sampled token.

Use these conventions as follows:

• For the sampled token's vocabulary rank, use completion_rank_list, not completion_topk_logprobs_list[i][0]["rank"], which is always 1;
• For the model's ranking at position i, read from index 1 onward, or sort the row by logprob;
• The entries from index 1 onward are sorted. row[1] is the argmax whenever row[0] is not.

Accuracy

Mean evalscope_is_correct over all 80 seeds of every question.

Truncation (finish_reason == "length")

A length-truncated attempt has no final boxed answer and is graded incorrect. Report the truncation policy when comparing reasoning levels.

Notes

gpt-oss response text

For gpt-oss, text contains only the Harmony final channel. For Qwen it contains the full generation. Both verifiers saw only text, so cross-model and cross-effort comparisons of their scores are affected by this difference. The full generation can be recovered from tokens.completion_token_list.

CompassVerifier input length

CompassVerifier-3B has a 32k context limit. It scored 8,750 attempts on truncated responses. The dataset does not contain a flag for this condition.

Reference-free verifier

Repeated scoring of byte-identical prompt-response pairs produced pointwise reward differences with a median of 0.040 and a maximum of 0.796. Keep this variation in mind when comparing small reward differences. CompassVerifier did not show the same variation.

Row size

Median uncompressed row size ranges from 0.3 MB for gpt-oss-20b_low to 18.3 MB for gpt-oss-20b_high; the largest measured row is 22.6 MB. Use streaming or read selected columns from individual Parquet files.

Provenance

The data was converted from question-major gzipped JSONL to Parquet. Source grouping and ascending seed order were preserved. Nested grading and verifier results were flattened, while token data remains in the tokens struct. Every written record, including every top-20 entry, was compared field by field with its source record.

The gzipped source corpus was also compared with the inference output using record-level BLAKE2b digests. Parquet serialization changes the bytes, so the field-by-field comparison is the relevant check for this release.

Citation

@article{hariri2026test,
  title={Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility},
  author={Hariri, Mohsen and Chen, Weicong and Shahini, Nahal and Singh, Vikash and Ye, Kai
          and Samandar, Amirhossein and Ganguly, Debargha and Sankar, Sreehari and Zhang, Yanyan
          and Wang, Shouren and others},
  journal={arXiv preprint arXiv:2608.04001},
  year={2026}
}

Related: arXiv:2510.04265 · arXiv:2603.10960 · arXiv:2603.14103. Tooling: scorio.

Contact

For questions about the dataset, contact Mohsen Hariri.

License

This dataset is released under the MIT License. Problem statements from AIME, HMMT, SMT, and CMIMC remain subject to their original terms.