README.md

metadata

title: Dispatch AI Mobile AI Leaderboard
emoji: 📱
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: 4.44.1
app_file: app.py
pinned: true
license: other

Dispatch AI — Mobile AI Leaderboard

Real-phone benchmark results for small (≤ 3B) instruction-tuned LLMs, measured on a farm of 40 Samsung S20 FE devices (Snapdragon 865, 6 GB RAM, Android 13) running llama.cpp with Q4_K_M 4-bit GGUF quants.

Columns

Column	Meaning
Model	HuggingFace model ID + quant
Size (MB)	Size of the GGUF file on disk
Generation Speed (t/s)	Median tokens/second for 256 generation tokens
Prompt Speed (t/s)	Median tokens/second for a 512-token prompt
RAM Free (MB)	RAM free after model load (Android meminfo)
Load Time (s)	Median wall-clock time to load the model
Phone Tested	Device used for the measurement

Methodology

• Devices: 40 × Samsung S20 FE (SM-G780F), Snapdragon 865, 6 GB RAM, Android 13.
• Backend: llama.cpp (via llamafile), Q4_K_M quants, 4 CPU threads, FP16 offload.
• Prompts: Fixed 512-token prompt; 256 generation tokens; batch size 512.
• Aggregation: Each number is the median across all 40 devices after a 5-run warm-up.
• Environment: Air-conditioned room at 22 °C; phones on stands, screens off; batteries at 80–100%, airplane mode + Wi-Fi only.

Live filtering

Type a model name (e.g. Qwen, Llama, SmolLM) in the Filter models by name box to narrow the table. Clear the box to see all models again.

Submit your results

Have benchmarks from your own phone farm? Add them via the Submit Your Results button (links to our GitHub) or open a PR directly against the data/benchmarks.csv file in the Dispatch-AI-FZE/mobile-ai-leaderboard repository. Please include device model, SoC, RAM, llama.cpp version, and the same prompt/generation token counts so results are comparable.

Deploying to HuggingFace Spaces

1. Create a new Space at https://huggingface.co/new-space:

   • Owner: your org or user
   • Space name: mobile-ai-leaderboard
   • License: choose your own (Dispatch AI uses license 10818, Sharjah UAE)
   • SDK: Gradio
   • Visibility: Public (or Private if you prefer)

2. Clone the Space repo locally (replace <owner>):

   git clone https://huggingface.co/spaces/<owner>/mobile-ai-leaderboard
   cd mobile-ai-leaderboard
   

3. Copy the files from this directory into the Space repo:

   cp /path/to/leaderboard/app.py .
   cp /path/to/leaderboard/requirements.txt .
   cp /path/to/leaderboard/README.md .
   

4. Commit and push:

   git add app.py requirements.txt README.md
   git commit -m "Dispatch AI Mobile AI Benchmark Leaderboard"
   git push
   

5. The Space will build automatically (Gradio SDK reads requirements.txt and installs gradio>=4.0). Once the build finishes, the leaderboard is live at https://<owner>-mobile-ai-leaderboard.hf.space.

Updating benchmarks

Benchmarks are hardcoded in app.py (the DATA list). To update, edit the list, commit, and push — the Space will rebuild automatically. A future version will move the data to a CSV file in the same repo so results can be updated without touching code.

Requirements

gradio>=4.0.0
pandas>=2.0.0

See requirements.txt.

License

© 2026 Dispatch AI FZE — Sharjah, UAE · License 10818. All rights reserved.