benchmarks.md

| | ZERO LOOP | ZERO LOOP Q4_K_M | SmolLM2-135M-Instruct | SmolLM-135M-Instruct |
|---|---|---|---|---|
| **Size And Speed** | | | | |
| Parameters | **19.9M** | **19.9M** | 134.5M | 134.5M |
| Download size | 40 MB | **14 MB** | 269 MB | 269 MB |
| CPU speed (tokens/s, 4 threads) | **40** | 464 | 18 | 18 |
| **Tool Use** | | | | |
| ZL-Calc: calculator tool | 0 | 0 | **6** | 0 |
| ZL-Calc-Noisy: typos, lower-case, × | **0** | **0** | **0** | **0** |
| ZL-Word: GSM8K problems + calculator | 0 | **3** | 1 | 0 |
| ZL-Search: TriviaQA + Wikipedia tool | 3 | 5 | 16 | **20** |
| Valid tool call on the first try | 0 | 0 | **16** | 4 |
| **Assistant Behaviour** | | | | |
| ZL-Identity: name and creator | **92** | 85 | 46 | 31 |

ZERO LOOP is ahead of every 134M-class baseline on 1 of 6 benchmarks and tied on 1.

<sub>Same machine, greedy decoding and the same tool protocol for every model. The baselines were not trained on the protocol, so they get it in the system prompt plus two worked examples; ZERO LOOP uses its built-in prompt. Identity: baselines are told their name and creator in the system prompt. ZL-Calc / ZL-Calc-Noisy: 100 arithmetic questions with unseen numbers (the noisy set is lower-cased, with a typo and a multiplication sign). ZL-Word: 100 GSM8K test word problems, the model may call the calculator. ZL-Search: 100 TriviaQA validation questions, live Wikipedia search, correct if the final answer contains a gold alias. Valid tool call: the first reply to an arithmetic question is a parseable call. These are our own small tests of the skills ZERO LOOP is built for (100 questions each: differences under ~10 points are noise). They say nothing about general ability: on MMLU, HellaSwag or ARC the larger models trained on far more data score higher. The Q4_K_M column runs the shipped GGUF in llama.cpp; its speed comes from llama.cpp, the other columns from Transformers fp32, so speed is shown for it but not compared.</sub>