README.md

---
title: Dispatch AI Mobile AI Leaderboard
emoji: 📱
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: "4.44.1"
app_file: app.py
pinned: true
license: other
---

# Dispatch AI — Mobile AI Leaderboard

Real-phone benchmark results for small (≤ 3B) instruction-tuned LLMs, measured on a
farm of **40 Samsung S20 FE** devices (Snapdragon 865, 6 GB RAM, Android 13) running
`llama.cpp` with **Q4_K_M** 4-bit GGUF quants.

## Columns

| Column | Meaning |
| --- | --- |
| Model | HuggingFace model ID + quant |
| Size (MB) | Size of the GGUF file on disk |
| Generation Speed (t/s) | Median tokens/second for 256 generation tokens |
| Prompt Speed (t/s) | Median tokens/second for a 512-token prompt |
| RAM Free (MB) | RAM free after model load (Android `meminfo`) |
| Load Time (s) | Median wall-clock time to load the model |
| Phone Tested | Device used for the measurement |

## Methodology

- **Devices:** 40 × Samsung S20 FE (SM-G780F), Snapdragon 865, 6 GB RAM, Android 13.
- **Backend:** `llama.cpp` (via `llamafile`), Q4_K_M quants, 4 CPU threads, FP16 offload.
- **Prompts:** Fixed 512-token prompt; 256 generation tokens; batch size 512.
- **Aggregation:** Each number is the median across all 40 devices after a 5-run warm-up.
- **Environment:** Air-conditioned room at 22 °C; phones on stands, screens off; batteries
  at 80–100%, airplane mode + Wi-Fi only.

## Live filtering

Type a model name (e.g. `Qwen`, `Llama`, `SmolLM`) in the **Filter models by name** box to
narrow the table. Clear the box to see all models again.

## Submit your results

Have benchmarks from your own phone farm? Add them via the
**Submit Your Results** button (links to our GitHub) or open a PR directly against the
`data/benchmarks.csv` file in the
[`Dispatch-AI-FZE/mobile-ai-leaderboard`](https://github.com/Dispatch-AI-FZE/mobile-ai-leaderboard)
repository. Please include device model, SoC, RAM, `llama.cpp` version, and the same
prompt/generation token counts so results are comparable.

## Deploying to HuggingFace Spaces

1. Create a new Space at <https://huggingface.co/new-space>:
   - **Owner:** your org or user
   - **Space name:** `mobile-ai-leaderboard`
   - **License:** choose your own (Dispatch AI uses license 10818, Sharjah UAE)
   - **SDK:** **Gradio**
   - **Visibility:** Public (or Private if you prefer)

2. Clone the Space repo locally (replace `<owner>`):
   ```bash
   git clone https://huggingface.co/spaces/<owner>/mobile-ai-leaderboard
   cd mobile-ai-leaderboard
   ```

3. Copy the files from this directory into the Space repo:
   ```bash
   cp /path/to/leaderboard/app.py .
   cp /path/to/leaderboard/requirements.txt .
   cp /path/to/leaderboard/README.md .
   ```

4. Commit and push:
   ```bash
   git add app.py requirements.txt README.md
   git commit -m "Dispatch AI Mobile AI Benchmark Leaderboard"
   git push
   ```

5. The Space will build automatically (Gradio SDK reads `requirements.txt` and installs
   `gradio>=4.0`). Once the build finishes, the leaderboard is live at
   `https://<owner>-mobile-ai-leaderboard.hf.space`.

### Updating benchmarks

Benchmarks are hardcoded in `app.py` (the `DATA` list). To update, edit the list, commit,
and push — the Space will rebuild automatically. A future version will move the data to a
CSV file in the same repo so results can be updated without touching code.

## Requirements

```
gradio>=4.0.0
pandas>=2.0.0
```

See `requirements.txt`.

## License

© 2026 Dispatch AI FZE — Sharjah, UAE · License 10818. All rights reserved.