methodology.html

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Methodology – Local LLM Benchmarks</title>
<link rel="stylesheet" href="style.css">
</head>
<body>
<header class="top">
  <a class="brand" href="index.html">Local LLM Benchmarks</a>
  <nav>
    <a href="index.html">Overview</a>
    <a href="methodology.html" class="active">Methodology</a>
  </nav>
</header>
<main>
<h1>Methodology</h1>
<section class="prose">
<p>Every model works through the same public task collections. A task yields a score from 0 to 100.
A category score is the weighted mean of its tasks, the overall score the unweighted mean of the categories.
If a task is missing, its category and the overall score stay empty.</p>
<ul>
  <li><strong>Profile quick:</strong> a fixed sample per task (column quick_n) — the same items for every model.</li>
  <li><strong>Profile full:</strong> all items of the task.</li>
  <li><strong>Suite version:</strong> whenever tasks, prompts or metrics change, a new version starts. Scores from different versions are not comparable, so only the latest version is published here.</li>
</ul>
<h2>Uncertainty</h2>
<p>Every task score comes with a standard error (±). The 95% interval spans roughly 1.96 standard errors around the
score. Where the intervals of two models overlap, the difference does not hold up — particularly in profile quick
with its smaller samples.</p>
<h2>Speed</h2>
<p>Measured are generation (tokens/s), prompt processing (tokens/s) and the time to the first token at various
context lengths, each as the median of several repetitions with p10 and p90. Depending on the run, the measurement
is taken directly at the inference server or through the AI hub; values measured through the AI hub include queue
and proxy time and are therefore somewhat slower. The route is stated on the run's own page. Speed depends heavily
on hardware and inference software.</p>
</section>

<h2>Suite v2b</h2>
<div class="table-wrap">
<table>
<thead><tr><th>Category</th><th>Task</th><th>Metric</th><th>Kind</th><th class="num">Weight</th><th class="num">quick_n</th></tr></thead>
<tbody>
<tr>
  <td>Knowledge &amp; reasoning</td>
  <td>mmlu_pro</td>
  <td>exact_match,custom-extract</td>
  <td>Text</td>
  <td class="num">1</td>
  <td class="num">300</td>
</tr>
<tr>
  <td>Knowledge &amp; reasoning</td>
  <td>gpqa_diamond_gen</td>
  <td>exact_match,letter</td>
  <td>Text</td>
  <td class="num">1</td>
  <td class="num">198</td>
</tr>
<tr>
  <td>Knowledge &amp; reasoning</td>
  <td>musr_gen</td>
  <td>exact_match,letter</td>
  <td>Text</td>
  <td class="num">1</td>
  <td class="num">250</td>
</tr>
<tr>
  <td>Math</td>
  <td>omni_math_gen<div class="sub wrap">Omni-MATH-Rule is the rule-checkable subset of Omni-MATH and systematically easier than full Omni-MATH. Its scores are therefore not comparable to published Omni-MATH numbers. The dataset has been public since October 2024, which is before the training cutoff of every model measured here; that affects all models alike.</div></td>
  <td>exact_match,boxed</td>
  <td>Text</td>
  <td class="num">1</td>
  <td class="num">250</td>
</tr>
<tr>
  <td>Instruction following</td>
  <td>ifbench</td>
  <td>prompt_level_strict_acc,none</td>
  <td>Text</td>
  <td class="num">1</td>
  <td class="num">300</td>
</tr>
<tr>
  <td>Instruction following</td>
  <td>ifeval</td>
  <td>prompt_level_strict_acc,none</td>
  <td>Text</td>
  <td class="num">0.33</td>
  <td class="num">150</td>
</tr>
<tr>
  <td>Code</td>
  <td>livecodebench_gen<div class="sub wrap">Contaminated, and unevenly so: LiveCodeBench v6 covers problems from May 2023 to April 2025, and there is no newer release. Some of the models measured here have a later training cutoff (as of September 2026: Gemma 4 January 2025, Nemotron 3.5 Lightning September 2025, Qwen3.8 not published). A model with a later cutoff may have seen more of these problems during training, so gaps between models say less here than elsewhere. Outputs are compared exactly as in official LiveCodeBench: this makes 12 problems with floating-point output practically unsolvable for every model (they are in the published LiveCodeBench numbers too), which lowers the reachable maximum equally for all. The list and the comparison against the official evaluator are in the project&#39;s calibration report. Profile full loads the complete dataset and needs about 9 GB of RAM for it.</div></td>
  <td>pass@1</td>
  <td>Code (run in a sandbox)</td>
  <td class="num">1</td>
  <td class="num">200</td>
</tr>
<tr>
  <td>German</td>
  <td>mmlu_prox_de</td>
  <td>exact_match,custom-extract</td>
  <td>Text</td>
  <td class="num">1</td>
  <td class="num">300</td>
</tr>
<tr>
  <td>German</td>
  <td>include_de_gen<div class="sub wrap">Only 154 items (the test and validation splits of INCLUDE-German, all of them measured). The 95% interval is about ±8 points, so smaller gaps between models are noise. Hence half the weight next to mmlu_prox_de.</div></td>
  <td>exact_match,letter</td>
  <td>Text</td>
  <td class="num">0.5</td>
  <td class="num">154</td>
</tr>
</tbody>
</table>
</div>
<footer class="page-foot">As of 2026-10-10 · scores 0–100 · built with <code>bench publish</code></footer>
</main>
<script src="chart.umd.min.js"></script>
<script src="app.js"></script>
</body>
</html>