Smol Worldcup
š
38
Benchmark Evaluation for Small LLMs
a0108982779
a0108982
AI & ML interests
None yet
Recent Activity
liked
a Space about 20 hours ago
ginigen-ai/smol-worldcup liked
a dataset about 20 hours ago
ginigen-ai/smol-worldcup upvoted an article about 20 hours ago
šļø Smol AI WorldCup: A 5-Axis Benchmark That Reveals What Small Language Models Can Really Do Organizations
None yet
liked a Space about 20 hours ago
liked a dataset about 20 hours ago
upvoted an article about 20 hours ago
Article
šļø Smol AI WorldCup: A 5-Axis Benchmark That Reveals What Small Language Models Can Really Do
ā¢
31
reacted to SeaWolf-AI's post with š¤ about 20 hours ago
Post
7251
šļø Smol AI WorldCup: A 4B Model Just Beat 8B ā Here's the Data
We evaluated 18 small language models from 12 makers on 125 questions across 7 languages. The results challenge the assumption that bigger is always better.
Community Article: https://huggingface.co/blog/FINAL-Bench/smol-worldcup
Live Leaderboard: ginigen-ai/smol-worldcup
Dataset: ginigen-ai/smol-worldcup
What we found:
ā Gemma-3n-E4B (4B, 2GB RAM) outscores Qwen3-8B (8B, 5.5GB). Doubling parameters gained only 0.4 points. RAM cost: 2.75x more.
ā GPT-OSS-20B fits in 1.5GB yet matches Champions-league dense models requiring 8.5GB. MoE architecture is the edge AI game-changer.
ā Thinking models hurt structured output. DeepSeek-R1-7B scores 8.7 points below same-size Qwen3-8B and runs 2.7x slower.
ā A 1.3B model fabricates confident fake content 80% of the time when prompted with nonexistent entities. Qwen3 family hits 100% trap detection across all sizes.
ā Qwen3-1.7B (1.2GB) outscores Mistral-7B, Llama-3.1-8B, and DeepSeek-R1-14B. Latest architecture at 1.7B beats older architecture at 14B.
What makes this benchmark different?
Most benchmarks ask "how smart?" ā we measure five axes simultaneously: Size, Honesty, Intelligence, Fast, Thrift (SHIFT). Our ranking metric WCS = sqrt(SHIFT x PIR_norm) rewards models that are both high-quality AND efficient. Smart but massive? Low rank. Tiny but poor? Also low.
Top 5 by WCS:
1. GPT-OSS-20B ā WCS 82.6 ā 1.5GB ā Raspberry Pi tier
2. Gemma-3n-E4B ā WCS 81.8 ā 2.0GB ā Smartphone tier
3. Llama-4-Scout ā WCS 79.3 ā 240 tok/s ā Fastest model
4. Qwen3-4B ā WCS 76.6 ā 2.8GB ā Smartphone tier
5. Qwen3-1.7B ā WCS 76.1 ā 1.2GB ā IoT tier
Built in collaboration with the FINAL Bench research team. Interoperable with ALL Bench Leaderboard for full small-to-large model comparison.
Dataset is open under Apache 2.0 (125 questions, 7 languages). We welcome new model submissions.
We evaluated 18 small language models from 12 makers on 125 questions across 7 languages. The results challenge the assumption that bigger is always better.
Community Article: https://huggingface.co/blog/FINAL-Bench/smol-worldcup
Live Leaderboard: ginigen-ai/smol-worldcup
Dataset: ginigen-ai/smol-worldcup
What we found:
ā Gemma-3n-E4B (4B, 2GB RAM) outscores Qwen3-8B (8B, 5.5GB). Doubling parameters gained only 0.4 points. RAM cost: 2.75x more.
ā GPT-OSS-20B fits in 1.5GB yet matches Champions-league dense models requiring 8.5GB. MoE architecture is the edge AI game-changer.
ā Thinking models hurt structured output. DeepSeek-R1-7B scores 8.7 points below same-size Qwen3-8B and runs 2.7x slower.
ā A 1.3B model fabricates confident fake content 80% of the time when prompted with nonexistent entities. Qwen3 family hits 100% trap detection across all sizes.
ā Qwen3-1.7B (1.2GB) outscores Mistral-7B, Llama-3.1-8B, and DeepSeek-R1-14B. Latest architecture at 1.7B beats older architecture at 14B.
What makes this benchmark different?
Most benchmarks ask "how smart?" ā we measure five axes simultaneously: Size, Honesty, Intelligence, Fast, Thrift (SHIFT). Our ranking metric WCS = sqrt(SHIFT x PIR_norm) rewards models that are both high-quality AND efficient. Smart but massive? Low rank. Tiny but poor? Also low.
Top 5 by WCS:
1. GPT-OSS-20B ā WCS 82.6 ā 1.5GB ā Raspberry Pi tier
2. Gemma-3n-E4B ā WCS 81.8 ā 2.0GB ā Smartphone tier
3. Llama-4-Scout ā WCS 79.3 ā 240 tok/s ā Fastest model
4. Qwen3-4B ā WCS 76.6 ā 2.8GB ā Smartphone tier
5. Qwen3-1.7B ā WCS 76.1 ā 1.2GB ā IoT tier
Built in collaboration with the FINAL Bench research team. Interoperable with ALL Bench Leaderboard for full small-to-large model comparison.
Dataset is open under Apache 2.0 (125 questions, 7 languages). We welcome new model submissions.
- 1 reply
upvoted a collection 8 months ago
liked 15 Spaces 8 months ago
WiFi Vision System
šØ
28
"One-minute creation by AI Coding Autonomous Agent MOUSE-I"
FitGen
š¤
33
Fashion Studio & Virtual Try-on
Voice Clone
š„
92
Voice Clone Multilingual TTS
Image To Vector
š
136
Vectorizer AI | Convert Image to SVG
Claude-Monet Studio
š¼
60
Generate Claude Monet-style images from text prompts
FLUX.1 Ghibli Studio LoRA
š¼
112
Ghibli Studio LoRA based Text to Image Generation
FLUXllama Multilingual
ā”
54
FLUXllama Multilingual(to be add more languages)
ADULT Novel generator
š
21
NSFW ADULT Novel generator
Deepseek v3-0324 Research
š
101
Deepseek v3-0324 + Real Time Deep Research
PDF RAG AI
š
47
Chat with an AI assistant
FLUX LOGO Generator
š
203
Generate custom logo images from text prompts
FLUX Animation Creator
š
58
Flux Animations(GIF) Generaion
Flowise AI
š¦
51
Build customized AI apps with drag-and-drop
OCR Latex
š»
43
Convert images to LaTeX code
Stable Video Diffusion Img2Vid
āØ
76
Animate Your Pictures With Stable VIdeo DIffusion