Competence-Based Evaluation of Logical Reasoning in Large Language Models

Each problem lists facts of the form “A is in front of B” for n people standing in a line, and asks a yes/no question about the order of the two people at the ends of the line. The rule never changes; only the number of people grows, from 4 to 1024, so n measures input size rather than the depth of reasoning. Every problem also comes in an equivalent rewording with the same answer, and a model counts as correct on a pair only when it answers both versions correctly. This page shows how accuracy and consistency hold up as problems grow and as their wording changes.

Model configurations
—
API, open-weight and fine-tuned
Problem sizes
—
people per problem, doubling
Item pairs per size
—
original + equivalent rewording

Both versions correct, by problem size

Each cell is the share of item pairs (200 per size) where the model answers both the original and the reworded problem correctly. Half the problems have the answer yes at every size, so always answering yes (or always no) scores 50%. A pair with a missing or unparseable answer counts as a failure; the paper's paired measures use only pairs where both answers parse, so values can differ for models that often fail to answer. Rows are sorted by the average share; models shown with the original wording only come last. Hover a cell for details, or click it to compare that model below.

Compare two models as n grows

Accuracy and consistency under perturbations

Each perturbation rewrites the same problems without changing the answer: restating some or all premises with the inverse phrase, or using other words for the relation. Left: accuracy on the original problems (hollow dot) and after the perturbation (filled dot); the connecting line turns red when accuracy drops by more than 2 points. Right: share of problems where the model's answer stays the same, compared item by item; the dashed tick marks the model's repeat agreement when the identical prompt is simply sent again. A missing answer counts as wrong and as a change of answer. Averages over n = 4 to 1024. Select a row (or focus it and press Enter) to see it by n below.

For relation wordings only accuracy is shown: they change the vocabulary of the whole problem, and the paper reports their consistency within each wording as family means rather than per wording. Each model shows the perturbations it was evaluated on in the paper.

By n

—

All model configurations

Averages over the nine sizes (n = 4 to 1024). Select a header to sort. A missing or unparseable answer counts as wrong and as a change of answer, so models that often fail to answer score lower here than on answered items alone.

    Cite

    @misc{jiang2026cbe,
      title         = {Competence-Based Evaluation of Logical Reasoning in Large Language Models},
      author        = {Jiang, Jize and Zhai, Chengxiang},
      year          = {2026},
      eprint        = {2610.22894},
      archivePrefix = {arXiv},
      primaryClass  = {cs.CL},
      url           = {https://arxiv.org/abs/2610.22894}
    }