🏟️ Smol AI WorldCup: A 5-Axis Benchmark That Reveals What Small Language Models Can Really Do

A 4B model outperforms 8B. A 1.5GB MoE achieves Champions-level quality. A 1.7B beats three 7–14B models. These aren't hypothetical claims — they're measured results from 18 models, 12 makers, 125 questions, and 7 languages.

We introduce Smol AI WorldCup, the first benchmark designed specifically for the deployment realities of small language models. Instead of asking only "how smart is it?", we ask five questions simultaneously: How smart? How honest? How fast? How small? How efficient?

The result is SHIFT — a 5-axis evaluation framework — and WCS (WorldCup Score) — a composite metric that rewards models achieving both high quality and high efficiency.

---

The Problem: Existing Benchmarks Weren't Built for Edge AI

MMLU, GPQA, and HumanEval apply the same test to a 0.5B model and a 500B model. This approach has three fundamental blind spots when it comes to small model deployment:

1. They measure only intelligence. An engineer deploying AI on a smartphone needs to know not just "MMLU 70" but whether the model fits in 2GB, how often it fabricates information, and how many tokens it generates per second.

2. They miss small-model-specific failure modes. Our measurements show that a 1.3B model fabricates confident fake content about nonexistent people, papers, and products 80% of the time when prompted. This risk is invisible to traditional benchmarks.

3. They don't reward efficiency. A 14B model scoring higher than a 4B model is expected. But the fact that "4B achieves 95% of 14B's quality at 36% of the RAM" is lost in a single-column leaderboard. In the edge AI era, performance per unit of resource is what matters.

---

SHIFT: A 5-Axis Framework for What Actually Matters

SHIFT decomposes the deployment value of a small language model into five measurable axes:

Honesty (H) — 40 Questions

Small models fail differently than large models. The most dangerous failure is confident fabrication. We test four dimensions:

Intelligence (I) — 85 Questions

All 125 questions require mandatory JSON output with verifiable fields. Grading uses structured field validation — not keyword matching. 75 of 125 questions are fully automatic with zero human intervention.

---

WCS: The Official Ranking Metric

Why Not Just Use SHIFT or PIR Alone?

A small-model benchmark must measure two things simultaneously:

• Quality: "How well does it perform?" → SHIFT score
• Efficiency: "How much resource does it consume?" → PIR score

Using SHIFT alone, a 14B model naturally ranks above 1.7B — which tells us nothing new. Using PIR alone, a 1.3B model with mediocre quality becomes #1 — which is misleading. Both must be high for a ranking to be meaningful.

The Formula

WCS = √( SHIFT × PIR_norm )

Where:
  SHIFT     = H × 0.4 + I × 0.6           (quality: honesty 40%, intelligence 60%)
  PIR       = (I × H × F) ÷ (S × T)       (efficiency: output per unit resource)
  PIR_norm  = log₁₀(PIR) / log₁₀(max) × 100  (normalized to 0–100)

Why geometric mean? Arithmetic mean (A+B)/2 allows one strong axis to compensate for a weak one. Geometric mean √(A×B) requires both to be strong — a model that's smart but massive, or tiny but poor, ranks low under WCS.

Why log-normalize PIR? Raw PIR ranges from 18 to 6,952 — a 386× spread. Log₁₀ compression maps this to 0–100, making it commensurable with SHIFT on the same scale.

Football League Tiers

Models are classified by actual runtime RAM (Q4 quantized), not raw parameter count. This correctly handles MoE models where total params ≠ active params.

---

Season 1 Results: 18 Models, 12 Makers

All scores measured via HuggingFace Inference API across 8 providers. Speed measured with 5 prompts × 3 rounds per model. Evaluated March 2026.

🧠 = Thinking model (uses internal <think> reasoning tokens)

---

Key Findings

1. 4B Outperforms 8B

Gemma-3n-E4B (4B parameters, 2.0GB RAM) achieves SHIFT 77.3 — the highest quality score among all 18 models. Qwen3-8B (8B, 5.5GB) scores 76.9. The gap is 0.4 points. The RAM difference is 2.75×.

This is not an anomaly. Qwen3-4B (4B, 2.8GB) also scores 76.8, essentially matching the 8B model at half the resource cost. The implication is concrete: smartphone-class hardware (2–3GB) can now run laptop-class AI (5.5GB) with negligible quality loss.

2. The MoE Efficiency Revolution

GPT-OSS-20B is a Mixture-of-Experts model with 21B total parameters but only 3.6B active at inference. It fits in 1.5GB of RAM — classified as League One (Raspberry Pi tier) — yet achieves SHIFT 76.9, matching Champions League dense models that require 8–9GB.

At comparable SHIFT scores (~76), the MoE model uses 5.7× less RAM than the equivalent dense model (Gemma-3-12B at 8.5GB). This validates MoE as the most promising architecture for edge deployment at scale.

3. Thinking Models: A Double-Edged Sword

Models that use internal reasoning tokens (<think>...</think>) face two penalties in SHIFT evaluation:

Quality penalty: The <think> tags interfere with structured JSON output, which is mandatory for all 125 questions. At comparable sizes:

• Qwen3-8B (non-thinking) → SHIFT 76.9
• DeepSeek-R1-7B (thinking) → SHIFT 68.2 (−8.7 points)

Speed penalty: Internal reasoning generates 2–6× more tokens than visible output:

• Qwen3-8B → 186.8 tok/s
• DeepSeek-R1-7B → 69.2 tok/s (2.7× slower)
• Nemotron-Nano-8B → 29.8 tok/s (6.3× slower)

This does not mean thinking models are inferior — DeepSeek-R1-14B's reasoning score (I1 = 96.7) is the highest of any model tested. But for structured output tasks and real-time edge deployment, non-thinking models deliver substantially better practical value.

4. Hallucination Trap: The Most Discriminating Metric

H1 (Hallucination Trap) produces the widest score spread of any single metric: 80 points (from 20 to 100).

The test presents nonexistent people, academic papers, laws, and products, then asks the model to describe them. Models that correctly refuse score high; models that confidently fabricate content score low.

H1 = 100:  Qwen3-4B, Qwen3-8B, GPT-OSS-20B, GLM-4.7-Flash, Qwen3.5-35B-A3B
H1 = 90:   Gemma-3n-E4B, Llama-4-Scout, Qwen3.5-9B
H1 = 80:   DeepSeek-R1-7B, Gemma-3-12B
H1 = 70:   Llama-3.1-8B, Nemotron-Nano-8B
H1 = 60:   Qwen3-1.7B, DeepSeek-R1-14B, OLMo-3-7B
H1 = 40:   Mistral-7B-v0.2
H1 = 30:   Tiny-Aya-Fire
H1 = 20:   Llama-3.2-1B

Two notable patterns emerge. First, the Qwen3 family achieves strong hallucination resistance across all sizes (60–100 from 1.7B to 35B), suggesting that training pipeline design is more determinative than scale. Second, a 1.3B model fabricates fake content 80% of the time — a critical safety consideration for IoT and embedded deployment.

5. The 1.7B Rebellion: Architecture Generation Beats Scale

Qwen3-1.7B (1.7B parameters, 1.2GB RAM, released 2025) outscores three significantly larger models:

• Mistral-7B-v0.2 (7.2B, 5.0GB, 2024) → SHIFT 60.6 (−6.2 points)
• Llama-3.1-8B (8.0B, 5.5GB, 2024) → SHIFT 61.0 (−5.8 points)
• DeepSeek-R1-14B (14.8B, 9.5GB, 2025) → SHIFT 59.8 (−7.0 points)

A 1.7B model from the latest generation outperforms models 4–9× its size from the previous generation. In the small-model domain, architecture vintage matters more than parameter count.

---

Union Eval: Direct Comparison with Frontier SOTA

Beyond the 125 SHIFT questions, each model takes a separate Union Eval consisting of 19 cross-benchmark questions. The same questions are administered to frontier-class models, enabling direct comparison:

Best small model: Gemma-3-12B = 57.1 (82% of Claude Sonnet)

A 12B open-source model matches the score of a 397B model on identical questions. This places the current frontier of small-model capability at approximately 80% of the best proprietary systems — a gap that is closing rapidly.

---

Research Collaboration & Benchmark Ecosystem

Smol AI WorldCup was developed in collaboration with the FINAL Bench research team. The cross-benchmark question design for Union Eval and the SOTA comparison framework draw on FINAL Bench's evaluation methodology.

The benchmark also integrates with the ALL Bench Leaderboard, creating a complementary evaluation ecosystem:

• Smol AI WorldCup answers: "Where does this 4B model rank among small models?"
• ALL Bench Leaderboard answers: "Where does that 4B model rank against GPT-5 and Claude?"

This cross-benchmark interoperability provides model developers and deployers with a unified view across the full spectrum — from edge-class small models to frontier-class large models — within a single evaluation ecosystem. Rather than operating in isolation, both benchmarks share evaluation principles and directional data to support the broader open-source AI community.

---

Speed Measurement Methodology

All 18 models were measured under identical conditions via the HuggingFace Inference API:

1. Warmup call — First inference excluded (cold start bias removal)
2. 5 diverse prompts — Explanation, Python coding, Korean translation, JSON generation, arithmetic
3. 3 rounds — 15 total samples per model
4. Metric: tok/s = completion_tokens / elapsed_seconds

A key observation: provider infrastructure affects speed more than model size. The Groq inference chip runs Llama-4-Scout (17B MoE) at 240 tok/s, while the same-tier dense model on Featherless achieves only 18 tok/s — a 13× gap driven entirely by hardware, not architecture. This real-world speed variance is captured in PIR and propagated into WCS.

---

Season System: Preventing Benchmark Contamination

The 30 anchor questions remain identical across seasons for Item Response Theory (IRT) calibration, enabling cross-season comparability. The 95 rotating questions are replaced to prevent benchmark contamination — a growing concern as models increasingly train on public evaluation data. Previous seasons' rotating questions are released publicly after each season concludes.

---

Evaluated Models (18)

12 makers represented: Meta, Alibaba, Google, OpenAI, DeepSeek, Mistral, NVIDIA, Cohere, Zhipu AI, AllenAI, Nanbeige (in dataset, future season), and others.

---

Recommendations by Use Case

---

Conclusion

The Smol AI WorldCup Season 1 evaluation of 18 models across 12 makers yields a clear conclusion:

Architecture and training quality — not parameter count — determine practical deployment value.

A 4B model outperforms 8B at 36% of the RAM. A 1.5GB MoE matches Champions-league dense models. A 2025-generation 1.7B outscores three 2024-generation 7–14B models. These findings challenge the scaling assumption that "bigger is always better" with measured, reproducible evidence.

The SHIFT framework and WCS metric provide the first integrated scoring system that captures all five axes that matter for edge AI deployment. The benchmark's season system, with 70%+ question rotation and IRT-calibrated anchors, is designed for long-term durability against contamination.

We look forward to Season 2 (2026 Q3), which will expand the model roster, rotate the question set, and refine the evaluation methodology based on community feedback.

The dataset is publicly available under Apache 2.0. We welcome submissions of new models for evaluation.

---

Developed by Ginigen.ai in collaboration with the FINAL Bench research team.

Small but Mighty AI.