Spaces:

bench-labs
/

BenchLabs-Leaderboard

Running

App Files Files Community

Patch

by wop - opened Jul 19

base: refs/heads/main

←

from: refs/pr/2

Discussion Files changed

+289

-238

Files changed (3) hide show

README.md +44 -26
index.html +204 -195
script.py +41 -17

README.md CHANGED Viewed

@@ -1,26 +1,44 @@
----
-title: Leaderboard
-emoji: 🏆
-colorFrom: green
-colorTo: blue
-sdk: static
-pinned: false
-license: apache-2.0
----
-Model performance across Bench Labs' benchmark suite — Effortless, Easy, Mid, and AGI.
-Reads from `models.json`, no backend required.
-## Evaluate a model
-One script, every benchmark, per-category/subcategory results:
-```bash
-curl -sLO https://huggingface.co/spaces/bench-labs/BenchLabs-Leaderboard/resolve/main/script.py
-pip install torch transformers
-python script.py --model your/model
-```
-Writes `results.json` (full breakdown), per-sample CSVs, and `leaderboard.json` — a ready-to-paste entry for `models.json`. Run `python script.py --help` for all options.
-all rights reserved @bench-labs

+---
+title: Leaderboard
+emoji: 🏆
+colorFrom: green
+colorTo: blue
+sdk: static
+pinned: false
+license: apache-2.0
+---
+Model performance across Bench Labs' benchmark suite — Effortless, Easy, Mid, and AGI.
+Reads from `models.json`, no backend required.
+## Evaluate a model
+One script, every benchmark, per-category/subcategory results:
+```bash
+curl -sLO https://huggingface.co/spaces/bench-labs/BenchLabs-Leaderboard/resolve/main/script.py
+pip install torch transformers
+python script.py --model your/model
+```
+Writes `results.json` (full breakdown), per-sample CSVs, and `leaderboard.json` — a ready-to-paste entry for `models.json`. Run `python script.py --help` for all options.
+### Reasoning / CoT models
+Models that think in `<think>…</think>` blocks are supported: the script strips the
+reasoning block and scores only the text after the final `</think>`. Two things to know:
+- Raise the generation budget — `--max-new-tokens 2048` or more. The default 32 is sized
+  for direct-answer models; a thinking model burns it inside the block, and a generation
+  cut off mid-think (unclosed `<think>`) scores as an empty answer.
+- `bench-mid-6-2026` is scored by log-likelihood over the answer choices with no
+  generation, so thinking never runs there.
+### Provenance
+`leaderboard.json` pins the run to exact bytes: `model_revision` is the Hub commit of the
+weights the script actually loaded (recorded by `from_pretrained`, not re-resolved after
+the fact), and `script_sha256` is the hash of the script file that produced the numbers.
+Local checkpoints have no Hub revision and honestly report `null`.
+all rights reserved @bench-labs

index.html CHANGED Viewed

@@ -1,195 +1,204 @@
-<!doctype html>
-<html lang="en">
-<head>
-<meta charset="utf-8">
-<meta name="viewport" content="width=device-width, initial-scale=1">
-<title>Bench Labs Leaderboard</title>
-<meta name="description" content="Model performance across Bench Labs' benchmark suite: Effortless, Easy, and Mid.">
-<link rel="stylesheet" href="style.css">
-</head>
-<body>
-<script src="background.js" defer></script>
-<div class="background"></div>
-<main class="container">
-  <header class="hero">
-    <div class="eyebrow">
-      <svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.4" stroke-linecap="round" stroke-linejoin="round"><path d="M8 21h8M12 17v4M17 3H7a2 2 0 0 0-2 2v6a7 7 0 0 0 14 0V5a2 2 0 0 0-2-2Z"/></svg>
-      <a href="https://huggingface.co/bench-labs" target="_blank" rel="noopener">bench-labs</a>
-    </div>
-    <h1>Leaderboard</h1>
-    <p class="subtitle">
-      Every evaluated model, every benchmark score, one table. Pulled straight from <code>models.json</code>.
-    </p>
-  </header>
-  <div class="controls">
-    <div class="filter-wrap">
-      <button
-        type="button"
-        class="search-circle-btn filter-btn tooltip"
-        id="params-filter-btn"
-        aria-label="Filter by parameter count"
-        aria-expanded="false"
-        data-tooltip="Filter by parameter count">
-        <svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round">
-          <line x1="4" y1="6" x2="20" y2="6"/><circle cx="9" cy="6" r="2"/>
-          <line x1="4" y1="12" x2="20" y2="12"/><circle cx="16" cy="12" r="2"/>
-          <line x1="4" y1="18" x2="20" y2="18"/><circle cx="7" cy="18" r="2"/>
-        </svg>
-      </button>
-      <div class="filter-popup" id="params-filter-popup" hidden>
-        <div class="filter-popup-title">Parameters (B)</div>
-        <div class="range-slider" id="params-range-slider">
-          <div class="range-slider-track"></div>
-          <div class="range-slider-fill" id="params-range-fill"></div>
-          <input type="range" id="params-range-min" class="range-thumb range-thumb-min" min="0" max="1000" step="1" value="0" aria-label="Minimum parameters in billions">
-          <input type="range" id="params-range-max" class="range-thumb range-thumb-max" min="0" max="1000" step="1" value="1000" aria-label="Maximum parameters in billions">
-        </div>
-        <div class="range-slider-values">
-          <span id="params-range-min-label">0B</span>
-          <span id="params-range-max-label">1000B+</span>
-        </div>
-        <button type="button" class="filter-reset" id="params-filter-reset">Reset</button>
-      </div>
-    </div>
-    <div class="search-pill">
-      <svg width="15" height="15" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round"><circle cx="11" cy="11" r="6"/><line x1="20" y1="20" x2="15.5" y2="15.5"/></svg>
-      <input type="search" id="model-search" placeholder="Search models or orgs…" aria-label="Search models">
-    </div>
-    <div class="sort-wrap">
-      <button
-        type="button"
-        class="sort-trigger"
-        id="sort-trigger"
-        aria-haspopup="listbox"
-        aria-expanded="false">
-        <span class="sort-trigger-label" id="sort-trigger-label">Avg score: High → Low</span>
-        <svg width="13" height="13" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.2" stroke-linecap="round" stroke-linejoin="round" class="sort-trigger-arrow"><polyline points="6 9 12 15 18 9"/></svg>
-      </button>
-      <ul class="sort-menu" id="sort-menu" role="listbox" aria-label="Sort order" hidden>
-        <li role="option" data-value="avg-desc" aria-selected="true">Avg score: High → Low</li>
-        <li role="option" data-value="avg-asc" aria-selected="false">Avg score: Low → High</li>
-        <li role="option" data-value="params-asc" aria-selected="false">Params: Small → Large</li>
-        <li role="option" data-value="params-desc" aria-selected="false">Params: Large → Small</li>
-        <li role="option" data-value="name-asc" aria-selected="false">Name: A → Z</li>
-      </ul>
-    </div>
-  </div>
-  <p class="results-count" id="results-count"></p>
-  <!-- A single flat table: one row per model, one plain column per benchmark's
-       headline score. No grouped/rowspan headers, no separate detail panel —
-       models with a category breakdown get an inline expand row using the
-       same row styling as everything else. -->
-  <div class="table-scroll" id="table-scroll">
-    <table class="leaderboard" id="leaderboard-table">
-      <thead id="leaderboard-head">
-        <!-- populated by script.js: exactly one header row -->
-      </thead>
-      <tbody id="leaderboard-body">
-        <!-- populated by script.js -->
-      </tbody>
-    </table>
-  </div>
-  <!-- Mobile: the same data as the table, one card per model. -->
-  <div class="model-cards" id="model-cards"></div>
-  <div class="empty-state" id="empty-state" hidden>
-    <svg width="34" height="34" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.6"><circle cx="11" cy="11" r="7"/><line x1="21" y1="21" x2="16.5" y2="16.5"/></svg>
-    <p>No models match your search or filters.</p>
-  </div>
-  <section class="submit-panel" aria-label="Submission rules">
-    <button class="submit-toggle" id="submit-toggle" aria-expanded="false" aria-controls="submit-body">
-      <span class="submit-toggle-left">
-        <svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M12 5v14M5 12h14"/></svg>
-        <span>Submit a model</span>
-      </span>
-      <span class="submit-toggle-arrow">▼</span>
-    </button>
-    <div class="submit-body" id="submit-body" hidden>
-      <p class="submit-intro">
-        There's no submission form — the leaderboard <em>is</em> <code>models.json</code>.
-        Add your entry and open a PR against this Space.
-      </p>
-      <h3>Rules</h3>
-      <ul class="rules-list">
-        <li>Model must be public on the Hugging Face Hub, with a link in <code>url</code>.</li>
-        <li>Evaluated on at least one Bench Labs benchmark using the method described in that dataset's README (lm-eval loglikelihood for <code>bench-mid-6-2026</code>, normalized exact-match for <code>bench-effortless-6-2026</code> / <code>bench-easy-6-2026</code>).</li>
-        <li>Include <code>n</code> (sample count) for every benchmark you report a score for.</li>
-        <li>If reporting a <code>bench-mid-6-2026</code> result, include the full <code>categories</code> breakdown — powers the row's expand arrow.</li>
-        <li>Link to your eval run or write-up in <code>eval_source</code> (a blog post, a script, a notebook — anything reproducible).</li>
-        <li>Only submit models you've actually run — don't add placeholder rows with no scores.</li>
-      </ul>
-      <h3>How to evaluate</h3>
-      <p class="submit-intro">
-        One universal script runs every benchmark and reports per-category / subcategory results.
-      </p>
-      <ol class="steps-list">
-        <li>Download the script from this Space:
-          <pre><code>curl -sLO https://huggingface.co/spaces/bench-labs/BenchLabs-Leaderboard/resolve/main/script.py</code></pre></li>
-        <li>Install the two dependencies: <pre><code>pip install torch transformers</code></pre></li>
-        <li>Run it against your model: <pre><code>python script.py --model org/ModelName</code></pre>
-          Options: <code>--benchmarks effortless,easy,mid</code> · <code>--device cuda</code> · <code>--limit 10</code> for a smoke test.</li>
-        <li>It writes <code>results.json</code> (full category/subcategory report), per-sample CSVs, and <code>leaderboard.json</code> — a ready-made entry for <code>models.json</code>.</li>
-      </ol>
-      <h3>How to submit</h3>
-      <ol class="steps-list">
-        <li>Create a PR on this space via the web UI or CLI.</li>
-        <li>Add an object to the <code>models</code> array in <code>models.json</code>, following the existing entries' shape (see <code>schema</code> below). The script's <code>leaderboard.json</code> output is already in this shape.</li>
-        <li>Leave <code>score: null</code> for any benchmark tier you haven't run yet; don't omit the key.</li>
-        <li>Submit the PR here with your eval source linked in the description.</li>
-        <li>We verify reproducibility before merging. Once merged, it shows up here automatically. Nothing else to deploy.</li>
-      </ol>
-      <h3>Entry schema</h3>
-      <pre><code id="schema-snippet">{
-  "id": "your-model-slug",
-  "name": "org/ModelName",
-  "org": "org",
-  "params_b": 1.5,
-  "license": "apache-2.0",
-  "architecture": "Transformer decoder, dense",
-  "url": "https://huggingface.co/org/ModelName",
-  "eval_source": "https://…",
-  "runs": {
-    "bench-effortless-6-2026": { "score": 0.0, "n": 240, "notes": "" },
-    "bench-easy-6-2026":       { "score": 0.0, "n": 300, "notes": "" },
-    "bench-mid-6-2026":        { "score": null, "n": null, "notes": "" }
-  }
-}</code></pre>
-      <a class="pr-link" href="https://huggingface.co/spaces/bench-labs/leaderboard/discussions" target="_blank" rel="noopener">
-        Open a PR on the Space
-        <svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.2" stroke-linecap="round" stroke-linejoin="round"><path d="M7 17 17 7M9 7h8v8"/></svg>
-      </a>
-    </div>
-  </section>
-  <footer class="page-footer">
-    <span>Data from <a href="models.json">models.json</a> · loaded client-side, no build step.</span>
-    <span>
-      <a href="https://huggingface.co/bench-labs" target="_blank" rel="noopener">org</a> ·
-      <a href="https://huggingface.co/spaces/bench-labs/blog" target="_blank" rel="noopener">blog</a> ·
-      <a href="https://huggingface.co/spaces/bench-labs/Members" target="_blank" rel="noopener">members</a>
-    </span>
-  </footer>
-</main>
-<script src="script.js" defer></script>
-</body>
-</html>

+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>Bench Labs Leaderboard</title>
+<meta name="description" content="Model performance across Bench Labs' benchmark suite: Effortless, Easy, and Mid.">
+<link rel="stylesheet" href="style.css">
+</head>
+<body>
+<script src="background.js" defer></script>
+<div class="background"></div>
+<main class="container">
+  <header class="hero">
+    <div class="eyebrow">
+      <svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.4" stroke-linecap="round" stroke-linejoin="round"><path d="M8 21h8M12 17v4M17 3H7a2 2 0 0 0-2 2v6a7 7 0 0 0 14 0V5a2 2 0 0 0-2-2Z"/></svg>
+      <a href="https://huggingface.co/bench-labs" target="_blank" rel="noopener">bench-labs</a>
+    </div>
+    <h1>Leaderboard</h1>
+    <p class="subtitle">
+      Every evaluated model, every benchmark score, one table. Pulled straight from <code>models.json</code>.
+    </p>
+  </header>
+  <div class="controls">
+    <div class="filter-wrap">
+      <button
+        type="button"
+        class="search-circle-btn filter-btn tooltip"
+        id="params-filter-btn"
+        aria-label="Filter by parameter count"
+        aria-expanded="false"
+        data-tooltip="Filter by parameter count">
+        <svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round">
+          <line x1="4" y1="6" x2="20" y2="6"/><circle cx="9" cy="6" r="2"/>
+          <line x1="4" y1="12" x2="20" y2="12"/><circle cx="16" cy="12" r="2"/>
+          <line x1="4" y1="18" x2="20" y2="18"/><circle cx="7" cy="18" r="2"/>
+        </svg>
+      </button>
+      <div class="filter-popup" id="params-filter-popup" hidden>
+        <div class="filter-popup-title">Parameters (B)</div>
+        <div class="range-slider" id="params-range-slider">
+          <div class="range-slider-track"></div>
+          <div class="range-slider-fill" id="params-range-fill"></div>
+          <input type="range" id="params-range-min" class="range-thumb range-thumb-min" min="0" max="1000" step="1" value="0" aria-label="Minimum parameters in billions">
+          <input type="range" id="params-range-max" class="range-thumb range-thumb-max" min="0" max="1000" step="1" value="1000" aria-label="Maximum parameters in billions">
+        </div>
+        <div class="range-slider-values">
+          <span id="params-range-min-label">0B</span>
+          <span id="params-range-max-label">1000B+</span>
+        </div>
+        <button type="button" class="filter-reset" id="params-filter-reset">Reset</button>
+      </div>
+    </div>
+    <div class="search-pill">
+      <svg width="15" height="15" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round"><circle cx="11" cy="11" r="6"/><line x1="20" y1="20" x2="15.5" y2="15.5"/></svg>
+      <input type="search" id="model-search" placeholder="Search models or orgs…" aria-label="Search models">
+    </div>
+    <div class="sort-wrap">
+      <button
+        type="button"
+        class="sort-trigger"
+        id="sort-trigger"
+        aria-haspopup="listbox"
+        aria-expanded="false">
+        <span class="sort-trigger-label" id="sort-trigger-label">Avg score: High → Low</span>
+        <svg width="13" height="13" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.2" stroke-linecap="round" stroke-linejoin="round" class="sort-trigger-arrow"><polyline points="6 9 12 15 18 9"/></svg>
+      </button>
+      <ul class="sort-menu" id="sort-menu" role="listbox" aria-label="Sort order" hidden>
+        <li role="option" data-value="avg-desc" aria-selected="true">Avg score: High → Low</li>
+        <li role="option" data-value="avg-asc" aria-selected="false">Avg score: Low → High</li>
+        <li role="option" data-value="params-asc" aria-selected="false">Params: Small → Large</li>
+        <li role="option" data-value="params-desc" aria-selected="false">Params: Large → Small</li>
+        <li role="option" data-value="name-asc" aria-selected="false">Name: A → Z</li>
+      </ul>
+    </div>
+  </div>
+  <p class="results-count" id="results-count"></p>
+  <!-- A single flat table: one row per model, one plain column per benchmark's
+       headline score. No grouped/rowspan headers, no separate detail panel —
+       models with a category breakdown get an inline expand row using the
+       same row styling as everything else. -->
+  <div class="table-scroll" id="table-scroll">
+    <table class="leaderboard" id="leaderboard-table">
+      <thead id="leaderboard-head">
+        <!-- populated by script.js: exactly one header row -->
+      </thead>
+      <tbody id="leaderboard-body">
+        <!-- populated by script.js -->
+      </tbody>
+    </table>
+  </div>
+  <!-- Mobile: the same data as the table, one card per model. -->
+  <div class="model-cards" id="model-cards"></div>
+  <div class="empty-state" id="empty-state" hidden>
+    <svg width="34" height="34" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.6"><circle cx="11" cy="11" r="7"/><line x1="21" y1="21" x2="16.5" y2="16.5"/></svg>
+    <p>No models match your search or filters.</p>
+  </div>
+  <section class="submit-panel" aria-label="Submission rules">
+    <button class="submit-toggle" id="submit-toggle" aria-expanded="false" aria-controls="submit-body">
+      <span class="submit-toggle-left">
+        <svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M12 5v14M5 12h14"/></svg>
+        <span>Submit a model</span>
+      </span>
+      <span class="submit-toggle-arrow">▼</span>
+    </button>
+    <div class="submit-body" id="submit-body" hidden>
+      <p class="submit-intro">
+        There's no submission form — the leaderboard <em>is</em> <code>models.json</code>.
+        Add your entry and open a PR against this Space.
+      </p>
+      <h3>Rules</h3>
+      <ul class="rules-list">
+        <li>Model must be public on the Hugging Face Hub, with a link in <code>url</code>.</li>
+        <li>Evaluated on at least one Bench Labs benchmark using the method described in that dataset's README (lm-eval loglikelihood for <code>bench-mid-6-2026</code>, normalized exact-match for <code>bench-effortless-6-2026</code> / <code>bench-easy-6-2026</code>).</li>
+        <li>Include <code>n</code> (sample count) for every benchmark you report a score for.</li>
+        <li>If reporting a <code>bench-mid-6-2026</code> result, include the full <code>categories</code> breakdown — powers the row's expand arrow.</li>
+        <li>Provenance is pinned automatically: <code>leaderboard.json</code> carries <code>model_revision</code> (the Hub commit of the weights the script actually loaded) and <code>script_sha256</code> (hash of the exact script that produced the numbers). Keep both in your entry, and link your eval run or write-up in the PR description.</li>
+        <li>Only submit models you've actually run — don't add placeholder rows with no scores.</li>
+      </ul>
+      <h3>How to evaluate</h3>
+      <p class="submit-intro">
+        One universal script runs every benchmark and reports per-category / subcategory results.
+      </p>
+      <ol class="steps-list">
+        <li>Download the script from this Space:
+          <pre><code>curl -sLO https://huggingface.co/spaces/bench-labs/BenchLabs-Leaderboard/resolve/main/script.py</code></pre></li>
+        <li>Install the two dependencies: <pre><code>pip install torch transformers</code></pre></li>
+        <li>Run it against your model: <pre><code>python script.py --model org/ModelName</code></pre>
+          Options: <code>--benchmarks effortless,easy,mid</code> · <code>--device cuda</code> · <code>--limit 10</code> for a smoke test.</li>
+        <li>It writes <code>results.json</code> (full category/subcategory report), per-sample CSVs, and <code>leaderboard.json</code> — a ready-made entry for <code>models.json</code>.</li>
+      </ol>
+      <p class="submit-intro">
+        Reasoning models that think in <code>&lt;think&gt;…&lt;/think&gt;</code> blocks are supported:
+        the script strips the reasoning and scores only the text after the final <code>&lt;/think&gt;</code>.
+        Give them room to finish — run with <code>--max-new-tokens 2048</code> or more (the default 32
+        is sized for direct-answer models; a generation cut off mid-think scores as an empty answer).
+        <code>bench-mid-6-2026</code> is scored by log-likelihood over the answer choices with no
+        generation, so thinking never runs there.
+      </p>
+      <h3>How to submit</h3>
+      <ol class="steps-list">
+        <li>Create a PR on this space via the web UI or CLI.</li>
+        <li>Add an object to the <code>models</code> array in <code>models.json</code>, following the existing entries' shape (see <code>schema</code> below). The script's <code>leaderboard.json</code> output is already in this shape.</li>
+        <li>Leave <code>score: null</code> for any benchmark tier you haven't run yet; don't omit the key.</li>
+        <li>Submit the PR here with your eval source linked in the description.</li>
+        <li>We verify reproducibility before merging. Once merged, it shows up here automatically. Nothing else to deploy.</li>
+      </ol>
+      <h3>Entry schema</h3>
+      <pre><code id="schema-snippet">{
+  "id": "your-model-slug",
+  "name": "org/ModelName",
+  "org": "org",
+  "params_b": 1.5,
+  "license": "apache-2.0",
+  "architecture": "Transformer decoder, dense",
+  "url": "https://huggingface.co/org/ModelName",
+  "model_revision": "hub commit sha the script loaded",
+  "script_sha256": "sha256 of the script.py that ran",
+  "runs": {
+    "bench-effortless-6-2026": { "score": 0.0, "n": 240, "notes": "" },
+    "bench-easy-6-2026":       { "score": 0.0, "n": 300, "notes": "" },
+    "bench-mid-6-2026":        { "score": null, "n": null, "notes": "" }
+  }
+}</code></pre>
+      <a class="pr-link" href="https://huggingface.co/spaces/bench-labs/leaderboard/discussions" target="_blank" rel="noopener">
+        Open a PR on the Space
+        <svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.2" stroke-linecap="round" stroke-linejoin="round"><path d="M7 17 17 7M9 7h8v8"/></svg>
+      </a>
+    </div>
+  </section>
+  <footer class="page-footer">
+    <span>Data from <a href="models.json">models.json</a> · loaded client-side, no build step.</span>
+    <span>
+      <a href="https://huggingface.co/bench-labs" target="_blank" rel="noopener">org</a> ·
+      <a href="https://huggingface.co/spaces/bench-labs/blog" target="_blank" rel="noopener">blog</a> ·
+      <a href="https://huggingface.co/spaces/bench-labs/Members" target="_blank" rel="noopener">members</a>
+    </span>
+  </footer>
+</main>
+<script src="script.js" defer></script>
+</body>
+</html>

script.py CHANGED Viewed

@@ -44,6 +44,14 @@ Scoring conventions
                 Headline score = soft_score_norm, matching the leaderboard.
   Multiple-choice prompt format: "Q: {input}\nA:" with candidates " {choice}".
 Bench Labs - Simple, Reliable, Open sourced
 """
@@ -123,6 +131,12 @@ ANSWER_PREFIXES = re.compile(
     r"^(the answer is|answer\s*[:=]|final answer\s*[:=]?|it is|it's)\s*", re.IGNORECASE
 )
 # --------------------------------------------------------------------------- #
 # Text normalization and scoring
 # --------------------------------------------------------------------------- #
@@ -137,8 +151,18 @@ def normalize(text: str) -> str:
 def extract_answer(text: str) -> str:
-    """First line of the generation, minus boilerplate answer prefixes."""
-    text = str(text).strip()
     if "\n" in text:
         text = text.split("\n", 1)[0]
     text = ANSWER_PREFIXES.sub("", text.strip())
@@ -278,21 +302,19 @@ class HFModel:
         print(f"  chat template: {'yes' if self.use_chat else 'no (plain QA prompt)'}")
         # -- resolved model provenance ------------------------------------ #
-        # Prefer the Hub's actual commit SHA for the given name/revision; this
-        # is what ties a leaderboard entry to the exact weights that were run,
-        # rather than to a mutable name or branch like "main".
-        self.resolved_revision: Optional[str] = None
-        try:
-            from huggingface_hub import HfApi
-            if "/" in name and not Path(name).exists():
                 self.resolved_revision = HfApi().model_info(name).sha
-        except Exception:
-            pass
-        if self.resolved_revision is None:
-            # Fall back to whatever transformers recorded when resolving the
-            # model (works for hub ids pinned via --revision, no-ops for
-            # local checkpoints, which legitimately have no hub revision).
-            self.resolved_revision = getattr(self.model.config, "_commit_hash", None)
     # -- generation -------------------------------------------------------- #
@@ -630,7 +652,9 @@ def parse_args(argv: Optional[Sequence[str]] = None) -> argparse.Namespace:
     p.add_argument("--device", default="auto", help="auto | cuda | cpu | mps")
     p.add_argument("--dtype", default="auto", help="auto | float16 | bfloat16 | float32")
     p.add_argument("--batch-size", type=int, default=8)
-    p.add_argument("--max-new-tokens", type=int, default=32)
     p.add_argument("--limit", type=int, default=None, help="cap rows per benchmark (smoke test)")
     p.add_argument("--output-dir", default=None,
                    help="default: benchlabs_results/<model-name>")

                 Headline score = soft_score_norm, matching the leaderboard.
   Multiple-choice prompt format: "Q: {input}\nA:" with candidates " {choice}".
+Reasoning / CoT models
+  <think>...</think> blocks are stripped before answer extraction: only the
+  text after the final </think> is scored. Raise --max-new-tokens (2048+) so
+  the model can finish thinking -- the default 32 is sized for direct-answer
+  models. A generation cut off mid-think (unclosed <think>) scores as an
+  empty answer. Mid is scored by log-likelihood over the answer choices with
+  no generation at all, so thinking never happens there.
 Bench Labs - Simple, Reliable, Open sourced
 """
     r"^(the answer is|answer\s*[:=]|final answer\s*[:=]?|it is|it's)\s*", re.IGNORECASE
 )
+# Reasoning-model tags. THINK_CLOSE also matches a bare closing tag: some chat
+# templates open <think> inside the prompt, so the generation contains only
+# the reasoning and a </think>.
+THINK_CLOSE = re.compile(r"</think(?:ing)?>\s*", re.IGNORECASE)
+THINK_OPEN = re.compile(r"<think(?:ing)?>.*", re.IGNORECASE | re.DOTALL)
 # --------------------------------------------------------------------------- #
 # Text normalization and scoring
 # --------------------------------------------------------------------------- #
 def extract_answer(text: str) -> str:
+    """First line of the generation after any <think> block, minus boilerplate prefixes.
+    Only text after the final </think> is scored. An unclosed <think> means the
+    generation ran out of budget mid-reasoning, so there is no answer to extract.
+    """
+    text = str(text)
+    parts = THINK_CLOSE.split(text)
+    if len(parts) > 1:
+        text = parts[-1]
+    else:
+        text = THINK_OPEN.sub("", text)
+    text = text.strip()
     if "\n" in text:
         text = text.split("\n", 1)[0]
     text = ANSWER_PREFIXES.sub("", text.strip())
         print(f"  chat template: {'yes' if self.use_chat else 'no (plain QA prompt)'}")
         # -- resolved model provenance ------------------------------------ #
+        # The weights already carry their commit: from_pretrained records the
+        # snapshot it actually loaded in config._commit_hash, no second Hub
+        # lookup. Asking the Hub afterwards can pin a different commit if the
+        # branch moved between load and lookup, so _commit_hash is primary.
+        self.resolved_revision: Optional[str] = getattr(self.model.config, "_commit_hash", None)
+        if self.resolved_revision is None and "/" in name and not Path(name).exists():
+            # Fallback for transformers versions that don't record it. Local
+            # checkpoints stay None, which is honest: they have no hub revision.
+            try:
+                from huggingface_hub import HfApi
                 self.resolved_revision = HfApi().model_info(name).sha
+            except Exception:
+                pass
     # -- generation -------------------------------------------------------- #
     p.add_argument("--device", default="auto", help="auto | cuda | cpu | mps")
     p.add_argument("--dtype", default="auto", help="auto | float16 | bfloat16 | float32")
     p.add_argument("--batch-size", type=int, default=8)
+    p.add_argument("--max-new-tokens", type=int, default=32,
+                   help="generation budget; raise to 2048+ for reasoning models "
+                        "that emit <think> blocks (default: 32)")
     p.add_argument("--limit", type=int, default=None, help="cap rows per benchmark (smoke test)")
     p.add_argument("--output-dir", default=None,
                    help="default: benchlabs_results/<model-name>")