| <!doctype html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8" /> |
| <meta name="viewport" content="width=device-width, initial-scale=1" /> |
| <meta name="theme-color" content="#090a0d" /> |
| <link rel="preconnect" href="https://fonts.googleapis.com" /> |
| <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin /> |
| <link href="https://fonts.googleapis.com/css2?family=Manrope:wght@300;400;500&display=swap" rel="stylesheet" /> |
| <link href="./_app/immutable/entry/start.CcZcJvgs.js" rel="modulepreload"> |
| <link href="./_app/immutable/chunks/B0XnQV1H.js" rel="modulepreload"> |
| <link href="./_app/immutable/chunks/CdZ30l9o.js" rel="modulepreload"> |
| <link href="./_app/immutable/chunks/Cx7tSjKg.js" rel="modulepreload"> |
| <link href="./_app/immutable/entry/app.N0WSQXzt.js" rel="modulepreload"> |
| <link href="./_app/immutable/chunks/CryQBVor.js" rel="modulepreload"> |
| <link href="./_app/immutable/chunks/D0-Lpew9.js" rel="modulepreload"> |
| <link href="./_app/immutable/chunks/DHQwSDIs.js" rel="modulepreload"> |
| <link href="./_app/immutable/nodes/0.u2wpjGRT.js" rel="modulepreload"> |
| <link href="./_app/immutable/chunks/DP5BbXZE.js" rel="modulepreload"> |
| <link href="./_app/immutable/nodes/2.DQQHwePY.js" rel="modulepreload"> |
| <meta name="description" content="A carefully scored compact language model benchmark from Universal Computing Research."/><title>Compact Language Model Index</title> |
| <link href="./_app/immutable/assets/0.NtfC_8t7.css" rel="stylesheet"> |
| </head> |
| <body data-sveltekit-preload-data="hover"> |
| <div style="display: contents"><main><section class="hero" id="top"><div class="hero-copy"><div class="hero-kicker"><p class="eyebrow">Small language model evaluation</p> <span class="edition">Research index</span></div> <h1>Compact<br/>Language Model<br/><span>Index</span></h1> <p class="hero-description">A fixed evaluation of compact language models across six benchmarks, with chance-normalized scores and 95% intervals</p> <a class="hero-link" href="#leaderboard-title">Explore the leaderboard</a></div></section> <section class="summary-grid" aria-label="Suite summary"><article><span class="summary-index">Evaluation breadth</span><div class="summary-value"><strong>6</strong><span>core benchmarks</span></div></article> <article><span class="summary-index">Linguistic depth</span><div class="summary-value"><strong>12</strong><span>BLiMP categories</span></div></article> <article><span class="summary-index">Current sample</span><div class="summary-value"><strong>20</strong><span>models measured</span></div></article></section> <section class="leaderboard-section" aria-labelledby="leaderboard-title"><div class="section-heading table-heading"><div><p class="eyebrow">Ranked results</p> <h2 id="leaderboard-title">Core leaderboard</h2></div> <div class="leaderboard-controls"><div class="filter-stack"><div class="top-filter-row"><div class="filters view-filter"><button type="button" aria-pressed="false">Simplified view</button></div> <div class="filters" aria-label="Model track filter"><button class="active">All tracks</button><button>Base</button><button>Instruction</button></div></div> <div class="size-filters" aria-label="Maximum parameter count"><button class="active">All</button><button><10M</button><button><50M</button><button><100M</button><button><300M</button></div></div></div></div> <div class="table-wrap"><table class="full-table"><colgroup><col class="rank-column"/><col class="model-column"/><col class="release-column"/><col class="parameters-column"/><col class="vocab-column"/><col class="track-column"/><col class="score-column"/><col class="hellaswag-column"/><col class="benchmark-column"/><col class="benchmark-column"/><col class="benchmark-column"/><col class="benchmark-column"/><col class="benchmark-column"/></colgroup><thead><tr><th>Rank</th><th class="sortable"><button class="sort-button" type="button">Model</button></th><th class="metadata-head" title="Hugging Face repository creation date">Release date</th><th class="sortable"><button class="sort-button" type="button">Parameters</button></th><th class="metadata-head">Vocab. size</th><th class="track-head">Track</th><th class="sortable score-head sorted"><button class="sort-button" type="button">Score <span>95% CI</span></button></th><th class="sortable metric-head"><button class="sort-button" type="button" aria-label="Sort by HellaSwag">HellaSwag</button></th><th class="sortable metric-head"><button class="sort-button" type="button" aria-label="Sort by PIQA">PIQA</button></th><th class="sortable metric-head"><button class="sort-button" type="button" aria-label="Sort by ARC-Easy">ARC-E*</button></th><th class="sortable metric-head"><button class="sort-button" type="button" aria-label="Sort by ARC-Challenge">ARC-C*</button></th><th class="sortable metric-head"><button class="sort-button" type="button" aria-label="Sort by CommonsenseQA">CQA*</button></th><th class="sortable metric-head"><button class="sort-button" type="button" aria-label="Sort by BLiMP">BLiMP</button></th></tr></thead><tbody><tr><td class="rank">1–2</td><td><a class="model-link" href="https://huggingface.co/LiquidAI/LFM2-350M" target="_blank" rel="noreferrer" aria-label="Open LFM2 350M on Hugging Face"><span class="model-name">LFM2 350M</span></a> <span class="model-source">LiquidAI/LFM2-350M</span></td><td class="metadata">2025-07-10</td><td>354M</td><td class="metadata">65,536</td><td class="track-cell"><span class="track instruction">instruction</span></td><td class="metric score-main">43.2<span>42.0–44.2</span></td><td class="metric">32.0<span>30.6–33.3</span></td><td class="metric">39.2<span>35.0–43.2</span></td><td class="metric">55.2<span>52.6–57.7</span></td><td class="metric">19.5<span>15.7–23.3</span></td><td class="metric">31.3<span>27.9–34.9</span></td><td class="metric">58.1<span>57.2–58.4</span></td></tr><tr><td class="rank">1–2</td><td><a class="model-link" href="https://huggingface.co/Qwen/Qwen2.5-0.5B" target="_blank" rel="noreferrer" aria-label="Open Qwen2.5 0.5B on Hugging Face"><span class="model-name">Qwen2.5 0.5B</span></a> <span class="model-source">Qwen/Qwen2.5-0.5B</span></td><td class="metadata">2024-09-15</td><td>494M</td><td class="metadata">151,936</td><td class="track-cell"><span class="track">base</span></td><td class="metric score-main">43.0<span>41.9–44.0</span></td><td class="metric">36.2<span>34.8–37.5</span></td><td class="metric">38.8<span>34.7–42.8</span></td><td class="metric">44.6<span>41.9–47.2</span></td><td class="metric">9.3<span>5.8–13.1</span></td><td class="metric">26.2<span>22.9–29.7</span></td><td class="metric">68.7<span>67.9–68.9</span></td></tr><tr><td class="rank">3</td><td><a class="model-link" href="https://huggingface.co/google/gemma-3-270m" target="_blank" rel="noreferrer" aria-label="Open Gemma 3 270M on Hugging Face"><span class="model-name">Gemma 3 270M</span></a> <span class="model-source">google/gemma-3-270m</span></td><td class="metadata">2025-08-05</td><td>270M</td><td class="metadata">262,144</td><td class="track-cell"><span class="track">base</span></td><td class="metric score-main">38.9<span>37.8–39.9</span></td><td class="metric">21.9<span>20.7–23.3</span></td><td class="metric">37.0<span>32.9–41.2</span></td><td class="metric">43.0<span>40.4–45.6</span></td><td class="metric">4.0<span>0.6–7.3</span></td><td class="metric">28.2<span>24.8–31.7</span></td><td class="metric">64.3<span>63.5–64.5</span></td></tr><tr><td class="rank">4</td><td><a class="model-link" href="https://huggingface.co/HuggingFaceTB/SmolLM2-135M" target="_blank" rel="noreferrer" aria-label="Open SmolLM2 135M on Hugging Face"><span class="model-name">SmolLM2 135M</span></a> <span class="model-source">HuggingFaceTB/SmolLM2-135M</span></td><td class="metadata">2024-10-31</td><td>135M</td><td class="metadata">49,152</td><td class="track-cell"><span class="track">base</span></td><td class="metric score-main">37.8<span>36.7–38.8</span></td><td class="metric">24.3<span>23.0–25.6</span></td><td class="metric">36.3<span>32.1–40.6</span></td><td class="metric">44.9<span>42.3–47.5</span></td><td class="metric">6.3<span>2.6–9.7</span></td><td class="metric">19.4<span>16.2–22.8</span></td><td class="metric">62.7<span>61.9–63.1</span></td></tr><tr><td class="rank">5</td><td><a class="model-link" href="https://huggingface.co/UniversalComputingResearch/Limen0.2B" target="_blank" rel="noreferrer" aria-label="Open Limen0.2B on Hugging Face"><span class="model-name">Limen0.2B</span></a> <span class="model-source">UniversalComputingResearch/Limen0.2B</span></td><td class="metadata">2026-07-24</td><td>223M</td><td class="metadata">16,384</td><td class="track-cell"><span class="track">base</span></td><td class="metric score-main">36.6<span>35.5–37.6</span></td><td class="metric">22.6<span>21.3–23.9</span></td><td class="metric">34.7<span>30.4–38.8</span></td><td class="metric">37.8<span>35.2–40.4</span></td><td class="metric">6.3<span>2.7–9.9</span></td><td class="metric">17.7<span>14.3–21.1</span></td><td class="metric">66.6<span>65.8–66.9</span></td></tr><tr><td class="rank">6</td><td><a class="model-link" href="https://huggingface.co/AxiomicLabs/GPT-X2-125M" target="_blank" rel="noreferrer" aria-label="Open GPT-X2 125M on Hugging Face"><span class="model-name">GPT-X2 125M</span></a> <span class="model-source">AxiomicLabs/GPT-X2-125M</span></td><td class="metadata">2026-03-25</td><td>125M</td><td class="metadata">32,768</td><td class="track-cell"><span class="track">base</span></td><td class="metric score-main">35.5<span>34.3–36.5</span></td><td class="metric">20.7<span>19.4–21.9</span></td><td class="metric">34.2<span>29.7–38.4</span></td><td class="metric">35.5<span>32.8–38.2</span></td><td class="metric">3.4<span>-0.1–7.1</span></td><td class="metric">18.3<span>14.9–21.6</span></td><td class="metric">65.8<span>65.0–66.1</span></td></tr><tr><td class="rank">7</td><td><a class="model-link" href="https://huggingface.co/openai-community/gpt2-medium" target="_blank" rel="noreferrer" aria-label="Open GPT-2 Medium 355M on Hugging Face"><span class="model-name">GPT-2 Medium 355M</span></a> <span class="model-source">openai-community/gpt2-medium</span></td><td class="metadata">2022-03-02</td><td>355M</td><td class="metadata">50,257</td><td class="track-cell"><span class="track">base</span></td><td class="metric score-main">33.4<span>32.3–34.4</span></td><td class="metric">19.1<span>17.9–20.3</span></td><td class="metric">33.2<span>28.9–37.5</span></td><td class="metric">24.6<span>22.1–27.3</span></td><td class="metric">0.0<span>-3.1–3.3</span></td><td class="metric">14.6<span>11.5–18.0</span></td><td class="metric">70.4<span>69.8–70.8</span></td></tr><tr><td class="rank">8</td><td><a class="model-link" href="https://huggingface.co/facebook/MobileLLM-R1-140M-base" target="_blank" rel="noreferrer" aria-label="Open MobileLLM-R1 140M Base on Hugging Face"><span class="model-name">MobileLLM-R1 140M Base</span></a> <span class="model-source">facebook/MobileLLM-R1-140M-base</span></td><td class="metadata">2025-09-10</td><td>140M</td><td class="metadata">128,256</td><td class="track-cell"><span class="track">base</span></td><td class="metric score-main">31.2<span>30.1–32.2</span></td><td class="metric">12.1<span>10.9–13.3</span></td><td class="metric">26.6<span>22.2–30.8</span></td><td class="metric">33.2<span>30.5–35.7</span></td><td class="metric">-1.1<span>-4.6–2.3</span></td><td class="metric">19.3<span>16.0–22.8</span></td><td class="metric">61.6<span>60.7–61.8</span></td></tr><tr><td class="rank">9</td><td><a class="model-link" href="https://huggingface.co/PleIAs/Baguettotron" target="_blank" rel="noreferrer" aria-label="Open Baguettotron on Hugging Face"><span class="model-name">Baguettotron</span></a> <span class="model-source">PleIAs/Baguettotron</span></td><td class="metadata">2025-11-10</td><td>321M</td><td class="metadata">65,536</td><td class="track-cell"><span class="track instruction">instruction</span></td><td class="metric score-main">29.6<span>28.4–30.7</span></td><td class="metric">13.9<span>12.6–15.1</span></td><td class="metric">24.2<span>19.6–28.7</span></td><td class="metric">34.1<span>31.5–36.7</span></td><td class="metric">7.2<span>3.8–10.6</span></td><td class="metric">13.3<span>10.0–16.6</span></td><td class="metric">56.9<span>56.0–57.2</span></td></tr><tr><td class="rank">10–11</td><td><a class="model-link" href="https://huggingface.co/openai-community/gpt2" target="_blank" rel="noreferrer" aria-label="Open GPT-2 124M on Hugging Face"><span class="model-name">GPT-2 124M</span></a> <span class="model-source">openai-community/gpt2</span></td><td class="metadata">2022-03-02</td><td>124M</td><td class="metadata">50,257</td><td class="track-cell"><span class="track">base</span></td><td class="metric score-main">27.9<span>26.8–28.9</span></td><td class="metric">8.2<span>7.0–9.4</span></td><td class="metric">25.0<span>20.7–29.4</span></td><td class="metric">19.3<span>16.7–21.9</span></td><td class="metric">-3.4<span>-6.6–-0.2</span></td><td class="metric">12.3<span>9.2–15.5</span></td><td class="metric">67.5<span>66.8–67.8</span></td></tr><tr><td class="rank">10–11</td><td><a class="model-link" href="https://huggingface.co/SupraLabs/Supra-50M-Base" target="_blank" rel="noreferrer" aria-label="Open Supra 50M Base on Hugging Face"><span class="model-name">Supra 50M Base</span></a> <span class="model-source">SupraLabs/Supra-50M-Base</span></td><td class="metadata">2026-05-21</td><td>51.8M</td><td class="metadata">32,000</td><td class="track-cell"><span class="track">base</span></td><td class="metric score-main">27.3<span>26.2–28.3</span></td><td class="metric">8.8<span>7.6–10.0</span></td><td class="metric">23.4<span>18.9–28.0</span></td><td class="metric">27.5<span>24.7–30.2</span></td><td class="metric">-0.1<span>-3.5–3.2</span></td><td class="metric">12.5<span>9.2–15.7</span></td><td class="metric">59.0<span>57.9–59.1</span></td></tr><tr><td class="rank">12</td><td><a class="model-link" href="https://huggingface.co/veyra-ai/Veyra2-Apricot-50M-Base" target="_blank" rel="noreferrer" aria-label="Open Veyra2 Apricot 50M Base on Hugging Face"><span class="model-name">Veyra2 Apricot 50M Base</span></a> <span class="model-source">veyra-ai/Veyra2-Apricot-50M-Base</span></td><td class="metadata">2026-06-14</td><td>49.3M</td><td class="metadata">8,192</td><td class="track-cell"><span class="track">base</span></td><td class="metric score-main">26.1<span>24.9–27.1</span></td><td class="metric">8.4<span>7.2–9.5</span></td><td class="metric">23.4<span>19.0–27.9</span></td><td class="metric">23.7<span>20.9–26.4</span></td><td class="metric">-2.0<span>-5.2–1.1</span></td><td class="metric">10.8<span>7.7–14.0</span></td><td class="metric">58.7<span>57.7–58.9</span></td></tr><tr><td class="rank">13–15</td><td><a class="model-link" href="https://huggingface.co/tiiuae/Falcon-H1-Tiny-R-90M" target="_blank" rel="noreferrer" aria-label="Open Falcon H1 Tiny R 90M on Hugging Face"><span class="model-name">Falcon H1 Tiny R 90M</span></a> <span class="model-source">tiiuae/Falcon-H1-Tiny-R-90M</span></td><td class="metadata">2026-01-12</td><td>90.0M</td><td class="metadata">32,768</td><td class="track-cell"><span class="track instruction">instruction</span></td><td class="metric score-main">20.8<span>19.6–21.8</span></td><td class="metric">7.7<span>6.6–8.9</span></td><td class="metric">21.5<span>17.0–25.9</span></td><td class="metric">19.0<span>16.3–21.5</span></td><td class="metric">-0.1<span>-3.4–3.2</span></td><td class="metric">10.4<span>7.4–13.6</span></td><td class="metric">42.5<span>41.6–43.0</span></td></tr><tr><td class="rank">13–15</td><td><a class="model-link" href="https://huggingface.co/liodon-ai/slm-10m" target="_blank" rel="noreferrer" aria-label="Open SLM 10M on Hugging Face"><span class="model-name">SLM 10M</span></a> <span class="model-source">liodon-ai/slm-10m</span></td><td class="metadata">2026-06-13</td><td>9.97M</td><td class="metadata">8,192</td><td class="track-cell"><span class="track">base</span></td><td class="metric score-main">20.3<span>19.1–21.2</span></td><td class="metric">3.1<span>2.0–4.2</span></td><td class="metric">14.1<span>9.7–18.5</span></td><td class="metric">14.3<span>11.6–16.8</span></td><td class="metric">-1.9<span>-5.1–1.3</span></td><td class="metric">7.5<span>4.5–10.6</span></td><td class="metric">54.4<span>53.5–54.7</span></td></tr><tr><td class="rank">13–16</td><td><a class="model-link" href="https://huggingface.co/EleutherAI/pythia-160m" target="_blank" rel="noreferrer" aria-label="Open Pythia 160M on Hugging Face"><span class="model-name">Pythia 160M</span></a> <span class="model-source">EleutherAI/pythia-160m</span></td><td class="metadata">2023-02-08</td><td>160M</td><td class="metadata">50,304</td><td class="track-cell"><span class="track">base</span></td><td class="metric score-main">20.2<span>19.1–21.2</span></td><td class="metric">7.3<span>6.2–8.5</span></td><td class="metric">16.9<span>12.5–21.2</span></td><td class="metric">15.8<span>13.2–18.4</span></td><td class="metric">-2.6<span>-5.7–0.6</span></td><td class="metric">11.7<span>8.4–14.8</span></td><td class="metric">45.7<span>44.8–46.2</span></td></tr><tr><td class="rank">15–16</td><td><a class="model-link" href="https://huggingface.co/AxiomicLabs/GPT-S2-5M" target="_blank" rel="noreferrer" aria-label="Open GPT-S2 5M on Hugging Face"><span class="model-name">GPT-S2 5M</span></a> <span class="model-source">AxiomicLabs/GPT-S2-5M</span></td><td class="metadata">2026-06-19</td><td>5.38M</td><td class="metadata">4,096</td><td class="track-cell"><span class="track">base</span></td><td class="metric score-main">19.3<span>18.2–20.3</span></td><td class="metric">3.4<span>2.3–4.5</span></td><td class="metric">12.9<span>8.3–17.4</span></td><td class="metric">11.3<span>8.6–13.9</span></td><td class="metric">-3.8<span>-6.9–-0.7</span></td><td class="metric">6.0<span>3.2–9.2</span></td><td class="metric">54.6<span>53.9–55.0</span></td></tr><tr><td class="rank">17–18</td><td><a class="model-link" href="https://huggingface.co/HuggingFaceTB/nanowhale-100m-base" target="_blank" rel="noreferrer" aria-label="Open nanowhale 100M Base on Hugging Face"><span class="model-name">nanowhale 100M Base</span></a> <span class="model-source">HuggingFaceTB/nanowhale-100m-base</span></td><td class="metadata">2026-04-24</td><td>110M</td><td class="metadata">129,280</td><td class="track-cell"><span class="track">base</span></td><td class="metric score-main">17.9<span>16.8–18.9</span></td><td class="metric">2.8<span>1.6–4.0</span></td><td class="metric">14.7<span>10.2–19.3</span></td><td class="metric">16.4<span>13.7–19.0</span></td><td class="metric">-3.4<span>-6.5–-0.2</span></td><td class="metric">9.7<span>6.6–12.9</span></td><td class="metric">41.8<span>41.0–42.3</span></td></tr><tr><td class="rank">17–18</td><td><a class="model-link" href="https://huggingface.co/EleutherAI/pythia-31m" target="_blank" rel="noreferrer" aria-label="Open Pythia 31M on Hugging Face"><span class="model-name">Pythia 31M</span></a> <span class="model-source">EleutherAI/pythia-31m</span></td><td class="metadata">2026-02-24</td><td>31.0M</td><td class="metadata">50,304</td><td class="track-cell"><span class="track">base</span></td><td class="metric score-main">16.9<span>15.8–18.0</span></td><td class="metric">2.9<span>1.8–4.1</span></td><td class="metric">13.6<span>9.1–18.1</span></td><td class="metric">12.1<span>9.7–14.5</span></td><td class="metric">-4.4<span>-7.6–-1.1</span></td><td class="metric">8.1<span>5.1–11.1</span></td><td class="metric">43.3<span>42.5–43.9</span></td></tr><tr><td class="rank">19</td><td><a class="model-link" href="https://huggingface.co/UniversalComputingResearch/Atom3.4m" target="_blank" rel="noreferrer" aria-label="Open Atom 3.4M on Hugging Face"><span class="model-name">Atom 3.4M</span></a> <span class="model-source">UniversalComputingResearch/Atom3.4m</span></td><td class="metadata">2026-06-19</td><td>3.41M</td><td class="metadata">4,096</td><td class="track-cell"><span class="track">base</span></td><td class="metric score-main">14.3<span>13.3–15.3</span></td><td class="metric">3.5<span>2.4–4.5</span></td><td class="metric">11.4<span>6.9–15.9</span></td><td class="metric">10.8<span>8.4–13.2</span></td><td class="metric">-4.2<span>-7.4–-1.1</span></td><td class="metric">5.9<span>2.9–9.2</span></td><td class="metric">36.4<span>35.6–37.0</span></td></tr><tr><td class="rank">20</td><td><a class="model-link" href="https://huggingface.co/AxiomicLabs/GPT-S-1.4M" target="_blank" rel="noreferrer" aria-label="Open GPT-S 1.4M on Hugging Face"><span class="model-name">GPT-S 1.4M</span></a> <span class="model-source">AxiomicLabs/GPT-S-1.4M</span></td><td class="metadata">2026-06-01</td><td>1.43M</td><td class="metadata">4,096</td><td class="track-cell"><span class="track">base</span></td><td class="metric score-main">12.3<span>11.2–13.3</span></td><td class="metric">2.5<span>1.3–3.6</span></td><td class="metric">10.3<span>5.7–14.7</span></td><td class="metric">9.0<span>6.5–11.5</span></td><td class="metric">-4.0<span>-7.1–-0.7</span></td><td class="metric">4.3<span>1.5–7.4</span></td><td class="metric">32.0<span>31.2–32.6</span></td></tr></tbody></table></div> <p class="table-note">ARC-E* = ARC-Easy · ARC-C* = ARC-Challenge · CQA* = CommonsenseQA</p></section> <section class="chart-controls" aria-label="Chart metric selector"><div><p class="eyebrow">Chart metric</p> <p class="section-note">Select a benchmark to update both the scaling chart and score comparison below.</p></div> <div class="task-tabs" aria-label="Benchmark score selector"><button class="active">Average</button><button>HellaSwag</button><button>PIQA</button><button>ARC-Easy</button><button>ARC-Challenge</button><button>CommonsenseQA</button><button>BLiMP</button></div></section> <section class="panel landscape" aria-labelledby="landscape-title"><div class="section-heading"><div><p class="eyebrow">Model scaling</p> <h2 id="landscape-title">Parameters and Average score</h2></div> <div class="legend"><span class="legend-dot base"></span>Base <span class="legend-dot instruction"></span>Instruction-tuned <span class="frontier-label"><span class="frontier-key"></span>Pareto trend</span></div></div> <div class="scatter-area" role="group" aria-label="Interactive Average score scatter chart"><svg viewBox="0 0 1120 390" preserveAspectRatio="xMidYMid meet" aria-label="Scatter chart of model size and chance-normalized Average score" role="img"><text class="axis-title" x="7" y="195" text-anchor="middle" transform="rotate(-90 7 195)">Chance-normalized score</text><line class="grid-line" x1="62" x2="1092" y1="338" y2="338"></line><text class="axis-text" x="48" y="342" text-anchor="end">-5.0</text><line class="grid-line" x1="62" x2="1092" y1="260.5" y2="260.5"></line><text class="axis-text" x="48" y="264.5" text-anchor="end">8.0</text><line class="grid-line" x1="62" x2="1092" y1="183" y2="183"></line><text class="axis-text" x="48" y="187" text-anchor="end">21.0</text><line class="grid-line" x1="62" x2="1092" y1="105.5" y2="105.5"></line><text class="axis-text" x="48" y="109.5" text-anchor="end">34.0</text><line class="grid-line" x1="62" x2="1092" y1="28" y2="28"></line><text class="axis-text" x="48" y="32" text-anchor="end">47.0</text><line class="axis-line" x1="62" x2="1092" y1="338" y2="338"></line><text class="axis-text" x="62" y="373" text-anchor="middle">1.00M</text><text class="axis-text" x="405.33333333333326" y="373" text-anchor="middle">66.2M</text><text class="axis-text" x="748.6666666666666" y="373" text-anchor="middle">233M</text><text class="axis-text" x="1092" y="373" text-anchor="middle">502M</text><text class="axis-title" x="560" y="389" dy="16" text-anchor="middle">Parameters (square-root scale)</text><line class="frontier-line" x1="62" y1="216.56388959058867" x2="1092" y2="28"></line><circle class="point instruction-point" cx="920.0643854317169" cy="50.780455838519174" r="7" tabindex="0" role="button" aria-label="LFM2 350M: 354M, Average score 43.2"></circle><circle class="point" cx="1083.6305651411053" cy="51.90738346440537" r="7" tabindex="0" role="button" aria-label="Qwen2.5 0.5B: 494M, Average score 43.0"></circle><circle class="point" cx="804.7395806967118" cy="76.21220794018353" r="7" tabindex="0" role="button" aria-label="Gemma 3 270M: 270M, Average score 38.9"></circle><circle class="point" cx="573.0989971860241" cy="82.58687530918641" r="7" tabindex="0" role="button" aria-label="SmolLM2 135M: 135M, Average score 37.8"></circle><circle class="point" cx="731.8362560526601" cy="89.82515121271285" r="7" tabindex="0" role="button" aria-label="Limen0.2B: 223M, Average score 36.6"></circle><circle class="point" cx="552.1640728408827" cy="96.41524426650113" r="7" tabindex="0" role="button" aria-label="GPT-X2 125M: 125M, Average score 35.5"></circle><circle class="point" cx="920.4978026516008" cy="109.1792405351323" r="7" tabindex="0" role="button" aria-label="GPT-2 Medium 355M: 355M, Average score 33.4"></circle><circle class="point" cx="583.3609510834209" cy="122.12647039897544" r="7" tabindex="0" role="button" aria-label="MobileLLM-R1 140M Base: 140M, Average score 31.2"></circle><circle class="point instruction-point" cx="876.2040679057936" cy="131.5429165333165" r="7" tabindex="0" role="button" aria-label="Baguettotron: 321M, Average score 29.6"></circle><circle class="point" cx="549.8315227770954" cy="142.03728477664495" r="7" tabindex="0" role="button" aria-label="GPT-2 124M: 124M, Average score 27.9"></circle><circle class="point" cx="360.2314798696789" cy="145.2155148919247" r="7" tabindex="0" role="button" aria-label="Supra 50M Base: 51.8M, Average score 27.3"></circle><circle class="point" cx="351.8148480745461" cy="152.66228173290887" r="7" tabindex="0" role="button" aria-label="Veyra2 Apricot 50M Base: 49.3M, Average score 26.1"></circle><circle class="point instruction-point" cx="470.47840980935945" cy="184.1896780254373" r="7" tabindex="0" role="button" aria-label="Falcon H1 Tiny R 90M: 90.0M, Average score 20.8"></circle><circle class="point" cx="165.83340068793552" cy="187.3757702798596" r="7" tabindex="0" role="button" aria-label="SLM 10M: 9.97M, Average score 20.3"></circle><circle class="point" cx="622.6814934529776" cy="187.69827382415878" r="7" tabindex="0" role="button" aria-label="Pythia 160M: 160M, Average score 20.2"></circle><circle class="point" cx="125.54930588890744" cy="193.25663346409905" r="7" tabindex="0" role="button" aria-label="GPT-S2 5M: 5.38M, Average score 19.3"></circle><circle class="point" cx="518.6696792594654" cy="201.73683063453336" r="7" tabindex="0" role="button" aria-label="nanowhale 100M Base: 110M, Average score 17.9"></circle><circle class="point" cx="281.8503408278001" cy="207.2645820788696" r="7" tabindex="0" role="button" aria-label="Pythia 31M: 31.0M, Average score 16.9"></circle><circle class="point" cx="102.78494456761237" cy="222.7794839267102" r="7" tabindex="0" role="button" aria-label="Atom 3.4M: 3.41M, Average score 14.3"></circle><circle class="point" cx="71.34471923620197" cy="234.94264108877815" r="7" tabindex="0" role="button" aria-label="GPT-S 1.4M: 1.43M, Average score 12.3"></circle></svg> </div></section> <section class="panel score-panel" aria-labelledby="scores-title"><div class="section-heading compact-heading"><div><h2 id="scores-title">Model score comparison</h2> <p class="section-note">The top 10 matching models are shown by default. This comparison uses the shared metric selected above.</p></div></div> <div class="bar-list" style="--zero:0%"><div class="bar-row"><span class="bar-rank">1</span> <span class="bar-name" title="LFM2 350M">LFM2 350M</span> <div class="bar-track"><i class="zero-line"></i><i class="bar-fill instruction-fill" style="--start:0%; --width:97.66531842166003%"></i><i class="ci-band instruction-band" style="--start:95.08105379287115%; --width:4.918946207128863%"></i></div> <strong>43.2</strong></div><div class="bar-row"><span class="bar-rank">2</span> <span class="bar-name" title="Qwen2.5 0.5B">Qwen2.5 0.5B</span> <div class="bar-track"><i class="zero-line"></i><i class="bar-fill" style="--start:0%; --width:97.23774780038632%"></i><i class="ci-band" style="--start:94.69030884335038%; --width:4.757187813642953%"></i></div> <strong>43.0</strong></div><div class="bar-row"><span class="bar-rank">3</span> <span class="bar-name" title="Gemma 3 270M">Gemma 3 270M</span> <div class="bar-track"><i class="zero-line"></i><i class="bar-fill" style="--start:0%; --width:88.01618941247756%"></i><i class="ci-band" style="--start:85.44044110865632%; --width:4.839940668657898%"></i></div> <strong>38.9</strong></div><div class="bar-row"><span class="bar-rank">4</span> <span class="bar-name" title="SmolLM2 135M">SmolLM2 135M</span> <div class="bar-track"><i class="zero-line"></i><i class="bar-fill" style="--start:0%; --width:85.597559835237%"></i><i class="ci-band" style="--start:82.96844443078491%; --width:4.905403281355444%"></i></div> <strong>37.8</strong></div><div class="bar-row"><span class="bar-rank">5</span> <span class="bar-name" title="Limen0.2B">Limen0.2B</span> <div class="bar-track"><i class="zero-line"></i><i class="bar-fill" style="--start:0%; --width:82.85126623666844%"></i><i class="ci-band" style="--start:80.20819100585224%; --width:4.849263206564379%"></i></div> <strong>36.6</strong></div><div class="bar-row"><span class="bar-rank">6</span> <span class="bar-name" title="GPT-X2 125M">GPT-X2 125M</span> <div class="bar-track"><i class="zero-line"></i><i class="bar-fill" style="--start:0%; --width:80.35090142482527%"></i><i class="ci-band" style="--start:77.6741650567042%; --width:4.842527680048843%"></i></div> <strong>35.5</strong></div><div class="bar-row"><span class="bar-rank">7</span> <span class="bar-name" title="GPT-2 Medium 355M">GPT-2 Medium 355M</span> <div class="bar-track"><i class="zero-line"></i><i class="bar-fill" style="--start:0%; --width:75.5080794998394%"></i><i class="ci-band" style="--start:73.06356877615407%; --width:4.788046269627763%"></i></div> <strong>33.4</strong></div><div class="bar-row"><span class="bar-rank">8</span> <span class="bar-name" title="MobileLLM-R1 140M Base">MobileLLM-R1 140M Base</span> <div class="bar-track"><i class="zero-line"></i><i class="bar-fill" style="--start:0%; --width:70.59573642696135%"></i><i class="ci-band" style="--start:68.01351553652728%; --width:4.856246906950909%"></i></div> <strong>31.2</strong></div><div class="bar-row"><span class="bar-rank">9</span> <span class="bar-name" title="Baguettotron">Baguettotron</span> <div class="bar-track"><i class="zero-line"></i><i class="bar-fill instruction-fill" style="--start:0%; --width:67.0230174418495%"></i><i class="ci-band instruction-band" style="--start:64.30273366073322%; --width:5.102435679356783%"></i></div> <strong>29.6</strong></div><div class="bar-row"><span class="bar-rank">10</span> <span class="bar-name" title="GPT-2 124M">GPT-2 124M</span> <div class="bar-track"><i class="zero-line"></i><i class="bar-fill" style="--start:0%; --width:63.04132114955523%"></i><i class="ci-band" style="--start:60.53290469836335%; --width:4.737448055935418%"></i></div> <strong>27.9</strong></div></div> <button class="bar-toggle" type="button" aria-expanded="false" aria-label="Show all 20 models" title="Show all 20 models"><span class="bar-toggle-chevron" aria-hidden="true"></span></button></section> <section class="bottom-grid"><article class="panel method-card"><p class="eyebrow">Scoring method</p> <h3>Chance-normalized composite score</h3> <p class="formula">100 × <span>(accuracy − chance)</span> / <span>(1 − chance)</span></p> <p>For each benchmark, the observed result is normalized against its chance level: chance maps to 0 and a perfect result maps to 100.</p> <p>HellaSwag, PIQA, and CommonsenseQA form a commonsense domain worth 50% of the composite, with each benchmark weighted equally within that domain. Science and world knowledge contribute 25% after combining ARC-Easy (75%) and ARC-Challenge (25%). BLiMP contributes the remaining 25% for grammatical competence and is macro-averaged across its 12 linguistic categories.</p> <p>CommonsenseQA scores the likelihood of each full answer text with length normalization, rather than scoring the A–E answer label. This prevents a model’s letter preference from being mistaken for commonsense capability.</p> <p>Each benchmark interval is a 95% percentile bootstrap. Model comparisons use paired bootstrap resampling on matched evaluation items. Results below chance remain negative, and rank ranges reflect comparisons that do not distinguish neighbouring models.</p></article> <article class="panel protocol-card"><p class="eyebrow">Evaluation protocol</p> <h2>Curated evaluation</h2> <p>CompactLMIndex is a curated benchmark, not an open intake list. We are establishing a Pareto-frontier baseline across model sizes and highlighting training or inference strategies that put a model ahead of its peers at a comparable parameter count.</p> <ul><li>Complete core suite required for a rank</li><li>Zero-shot, non-chat log-likelihood prompts</li><li>Paired bootstrap comparisons on matched questions</li><li>Benchmark-specific training or fine-tuning is discouraged and may exclude a model from comparison</li></ul> <p>Every published result is independently checked through our internal verification process. Final inclusion and presentation decisions remain with Universal Computing Research.</p> <p>If you have interesting results for a model you think is suitable for this benchmark, you can open a discussion on the Space for consideration.</p></article></section> <section class="panel additional-card" aria-labelledby="credits-title"><div><p class="eyebrow">Credits & scope</p> <h2 id="credits-title">Efficiency at smaller scales</h2></div> <div class="credits-content"><p>CompactLMIndex looks at smaller language models. At this scale, architecture, training, and inference choices can make a real difference. Every model follows the same evaluation protocol, so results are compared fairly rather than by size alone.</p> <p class="credits-attribution">Inspired by the <a href="https://huggingface.co/open-llm-leaderboard" target="_blank" rel="noreferrer">Open LLM Leaderboard</a>, <a href="https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard" target="_blank" rel="noreferrer">Axiomic Labs Open SLM Leaderboard</a>, and <a href="https://github.com/openai/parameter-golf" target="_blank" rel="noreferrer">OpenAI Parameter Golf</a>.</p></div></section> <footer><p>Compact Language Model Index by <a href="https://huggingface.co/UniversalComputingResearch" target="_blank" rel="noreferrer">Universal Computing Research</a></p></footer></main> |
| |
| <script> |
| { |
| __sveltekit_rnh8l0 = { |
| base: new URL(".", location).pathname.slice(0, -1) |
| }; |
| |
| const element = document.currentScript.parentElement; |
| |
| Promise.all([ |
| import("./_app/immutable/entry/start.CcZcJvgs.js"), |
| import("./_app/immutable/entry/app.N0WSQXzt.js") |
| ]).then(([kit, app]) => { |
| kit.start(app, element, { |
| node_ids: [0, 2], |
| data: [null,null], |
| form: null, |
| error: null |
| }); |
| }); |
| } |
| </script> |
| </div> |
| </body> |
| </html> |
|
|
