Anonymous Reasoning GPQA

This repository contains data accompanying an anonymous TMLR submission. It provides 1,152,000 sampled attempts from four model configurations on a fixed, field-balanced sample of 3,600 multiple-choice questions. Each model was sampled 80 times per question.

Contents

The data is stored in a Hugging Face Storage Bucket. Each Parquet file contains the 80 attempts for one model and one question, ordered by seed.

data/<model>/super_gpqa/qNNNN.parquet

There are 14,400 Parquet files totaling about 679.4 GiB. The four configuration keys are Qwen3.6-35B-A3B, gpt-oss-20b_low, gpt-oss-20b_medium, and gpt-oss-20b_high.

Loading

pip install -U datasets huggingface_hub pyarrow

from datasets import load_dataset

data_files = {
    "super_gpqa": "data/gpt-oss-20b_high/super_gpqa/*.parquet",
}
dataset = load_dataset(
    "buckets/AnonymizedTMLRSubmission/reasoning-gpqa",
    data_files=data_files,
    split="super_gpqa",
    streaming=True,
)
record = next(iter(dataset))

For direct access to one candidate pool:

import pyarrow.parquet as pq

table = pq.read_table(
    "data/gpt-oss-20b_high/super_gpqa/q0000.parquet",
    columns=["seed", "evalscope_is_correct", "cv3b_label", "field"],
)

Question Structure

The sample contains 72 fields with 50 questions per field. It spans 13 disciplines and 278 subfields. Every attempt includes discipline, field, subfield, difficulty, and is_calculation. Use full_data_id, uuid, selection_hash, or (stage, data_id) as a question identifier because data_id alone is not globally unique.

Data Format

Additional question metadata includes stage, full_data_id, uuid, field_rank, source_line, selection_hash, discipline, field, subfield, difficulty, and is_calculation.

The tokens struct includes prompt and completion log-probability summaries, token IDs, realized-token log probabilities and ranks, and top-20 candidate distributions. Each candidate entry contains token, token_id, logprob, and rank.

The first completion candidate is the sampled token. Remaining candidates are ordered by log probability. Use completion_rank_list for the sampled token's vocabulary rank.

Reproducibility

Question pools contain exactly 80 attempts in seed order. Report filtering of aborted or length-truncated attempts when comparing configurations. Derived verifier rewards can be reconstructed from the retained expected scores.

License

The artifact is provided under the MIT License. Questions from the underlying benchmark remain subject to the benchmark's applicable terms. The SuperGPQA question content is provided under ODC-BY 1.0 and requires upstream credit.

Upstream Attribution

The questions are derived from the SuperGPQA benchmark, published by the M-A-P Team. Users should credit that benchmark and its paper, "SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines" (2025), when using the question content.

Citation

Citation information will be added after anonymous review.