ZERO LOOP benchmarks table

.gitattributes

@@ -39,3 +39,4 @@ zero-loop-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
 zero-loop-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
 zero-loop-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
 zero-loop-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+benchmarks.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -17,6 +17,8 @@ tags:
 - llama.cpp
 ---
 
+<p align="center"><img src="zero_logo.png" width="380" alt="ZERO LOOP"></p>
+
 # ZERO LOOP
 
 **ZERO LOOP is a tiny foundation language model from Zero Labs, trained from scratch.**
@@ -167,7 +169,26 @@ The best checkpoint on a held-out validation set was kept, and it was published
 ## Evaluation
 
 <!--BENCH_START-->
-_The benchmark table is added by the evaluation step that runs right after this release (last cell of the notebook)._
+| | ZERO LOOP | ZERO LOOP Q4_K_M | SmolLM2-135M-Instruct | SmolLM-135M-Instruct |
+|---|---|---|---|---|
+| **Size And Speed** | | | | |
+| Parameters | **19.9M** | **19.9M** | 134.5M | 134.5M |
+| Download size | 40 MB | **14 MB** | 269 MB | 269 MB |
+| CPU speed (tokens/s, 4 threads) | **40** | 464 | 18 | 18 |
+| **Tool Use** | | | | |
+| ZL-Calc: calculator tool | 0 | 0 | **6** | 0 |
+| ZL-Calc-Noisy: typos, lower-case, × | **0** | **0** | **0** | **0** |
+| ZL-Word: GSM8K problems + calculator | 0 | **3** | 1 | 0 |
+| ZL-Search: TriviaQA + Wikipedia tool | 3 | 5 | 16 | **20** |
+| Valid tool call on the first try | 0 | 0 | **16** | 4 |
+| **Assistant Behaviour** | | | | |
+| ZL-Identity: name and creator | **92** | 85 | 46 | 31 |
+
+![ZERO LOOP benchmarks](benchmarks.png)
+
+ZERO LOOP is ahead of every 134M-class baseline on 1 of 6 benchmarks and tied on 1.
+
+<sub>Same machine, greedy decoding and the same tool protocol for every model. The baselines were not trained on the protocol, so they get it in the system prompt plus two worked examples; ZERO LOOP uses its built-in prompt. Identity: baselines are told their name and creator in the system prompt. ZL-Calc / ZL-Calc-Noisy: 100 arithmetic questions with unseen numbers (the noisy set is lower-cased, with a typo and a multiplication sign). ZL-Word: 100 GSM8K test word problems, the model may call the calculator. ZL-Search: 100 TriviaQA validation questions, live Wikipedia search, correct if the final answer contains a gold alias. Valid tool call: the first reply to an arithmetic question is a parseable call. These are our own small tests of the skills ZERO LOOP is built for (100 questions each: differences under ~10 points are noise). They say nothing about general ability: on MMLU, HellaSwag or ARC the larger models trained on far more data score higher. The Q4_K_M column runs the shipped GGUF in llama.cpp; its speed comes from llama.cpp, the other columns from Transformers fp32, so speed is shown for it but not compared.</sub>
 <!--BENCH_END-->
 
 Held-out web-text perplexity of the base model: 25.0 (cross-entropy 3.217 nats per token on this model's own vocabulary, so not comparable across tokenizers). We recommend evaluating on your own task before relying on it.

bench_results.json

@@ -0,0 +1,55 @@
+{
+ "fingerprint": "bef531c0036c80f8efb5f24d6cbf4f3f0b6a343a8f10db36095c61c98070d2d2",
+ "meta": {
+  "fingerprint": "bef531c0036c80f8efb5f24d6cbf4f3f0b6a343a8f10db36095c61c98070d2d2",
+  "runtime": "version: 0.5.0-dev (build 11382, commit 11fe02151)",
+  "date": "2026-10-04",
+  "n_per_benchmark": 100
+ },
+ "results": {
+  "ZERO LOOP": {
+   "params": 19.93344,
+   "size": 39.86688,
+   "calc": 0.0,
+   "noisy": 0.0,
+   "word": 0.0,
+   "search": 3.0,
+   "format": 0.0,
+   "ident": 92.3076923076923,
+   "tps": 40.341010237508634
+  },
+  "ZERO LOOP Q4_K_M": {
+   "params": 19.93344,
+   "size": 13.527904,
+   "tps": 463.86628870154254,
+   "calc": 0.0,
+   "noisy": 0.0,
+   "word": 3.0,
+   "search": 5.0,
+   "format": 0.0,
+   "ident": 84.61538461538461
+  },
+  "SmolLM2-135M-Instruct": {
+   "params": 134.515008,
+   "size": 269.030016,
+   "calc": 6.0,
+   "noisy": 0.0,
+   "word": 1.0,
+   "search": 16.0,
+   "format": 16.0,
+   "ident": 46.15384615384615,
+   "tps": 17.846736330218345
+  },
+  "SmolLM-135M-Instruct": {
+   "params": 134.515008,
+   "size": 269.030016,
+   "calc": 0.0,
+   "noisy": 0.0,
+   "word": 0.0,
+   "search": 20.0,
+   "format": 4.0,
+   "ident": 30.76923076923077,
+   "tps": 17.60086497742974
+  }
+ }
+}

benchmarks.md

@@ -0,0 +1,18 @@
+| | ZERO LOOP | ZERO LOOP Q4_K_M | SmolLM2-135M-Instruct | SmolLM-135M-Instruct |
+|---|---|---|---|---|
+| **Size And Speed** | | | | |
+| Parameters | **19.9M** | **19.9M** | 134.5M | 134.5M |
+| Download size | 40 MB | **14 MB** | 269 MB | 269 MB |
+| CPU speed (tokens/s, 4 threads) | **40** | 464 | 18 | 18 |
+| **Tool Use** | | | | |
+| ZL-Calc: calculator tool | 0 | 0 | **6** | 0 |
+| ZL-Calc-Noisy: typos, lower-case, × | **0** | **0** | **0** | **0** |
+| ZL-Word: GSM8K problems + calculator | 0 | **3** | 1 | 0 |
+| ZL-Search: TriviaQA + Wikipedia tool | 3 | 5 | 16 | **20** |
+| Valid tool call on the first try | 0 | 0 | **16** | 4 |
+| **Assistant Behaviour** | | | | |
+| ZL-Identity: name and creator | **92** | 85 | 46 | 31 |
+
+ZERO LOOP is ahead of every 134M-class baseline on 1 of 6 benchmarks and tied on 1.
+
+<sub>Same machine, greedy decoding and the same tool protocol for every model. The baselines were not trained on the protocol, so they get it in the system prompt plus two worked examples; ZERO LOOP uses its built-in prompt. Identity: baselines are told their name and creator in the system prompt. ZL-Calc / ZL-Calc-Noisy: 100 arithmetic questions with unseen numbers (the noisy set is lower-cased, with a typo and a multiplication sign). ZL-Word: 100 GSM8K test word problems, the model may call the calculator. ZL-Search: 100 TriviaQA validation questions, live Wikipedia search, correct if the final answer contains a gold alias. Valid tool call: the first reply to an arithmetic question is a parseable call. These are our own small tests of the skills ZERO LOOP is built for (100 questions each: differences under ~10 points are noise). They say nothing about general ability: on MMLU, HellaSwag or ARC the larger models trained on far more data score higher. The Q4_K_M column runs the shipped GGUF in llama.cpp; its speed comes from llama.cpp, the other columns from Transformers fp32, so speed is shown for it but not compared.</sub>