| --- |
| license: apache-2.0 |
| base_model: HuggingFaceTB/SmolLM2-360M-Instruct |
| pipeline_tag: text-generation |
| library_name: litert-lm |
| tags: |
| - chat |
| - litert-lm |
| - smollm |
| - on-device |
| --- |
| |
| # litert-community/SmolLM2-360M-Instruct |
|
|
| This model provides a variant of |
| [HuggingFaceTB/SmolLM2-360M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct) that is ready for |
| deployment on Android using the |
| [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM). |
|
|
| ## Use the model |
|
|
| ### Android |
|
|
| #### Edge Gallery App |
| * Download or build the [app](https://github.com/google-ai-edge/gallery?tab=readme-ov-file#-get-started-in-minutes) from GitHub. |
|
|
| * Install the [app](https://play.google.com/store/apps/details?id=com.google.ai.edge.gallery&pli=1) from Google Play. |
|
|
| * Follow the instructions in the app. |
|
|
| To build the demo app from source, please follow the [instructions](https://github.com/google-ai-edge/gallery/blob/main/README.md) |
| from the GitHub repository. |
|
|
| ## Performance (measured) |
|
|
| ### Apple M4 Max |
|
|
| Measured with the LiteRT-LM CLI: `litert-lm benchmark -p 256 -d 256 --runs 3 --cache no` |
| (litert-lm 0.15.0) on an idle Apple M4 Max (macOS); 256 prefill / 256 decode tokens, 3 iterations |
| averaged by the tool. A desktop reference point — phone-side figures vary by SoC and backend. |
|
|
| | Backend | Prefill (tokens/s) | Decode (tokens/s) | Time-to-first-token (s) | |
| |---|---|---|---| |
| | CPU | 588 | 37.3 | 0.63 | |
| | GPU | 2,773 | 133.6 | 0.11 | |
|
|
| ### Galaxy S26 — GPU vs CPU (litert-lm 0.16.0) |
|
|
| Measured on a physical Samsung Galaxy S26 (SM-S942Q, Snapdragon 8 Elite Gen 5 / SM8850, Android 16) with `litert_lm_advanced_main` from the litert-lm v0.16.0 release; the GPU backend is OpenCL (`LITERT_CL`). One fixed 205-token prompt text (206 tokens under this tokenizer), `--benchmark`. GPU: two runs back-to-back; CPU: three runs spread across device thermal states (skin temperature up to 41.9 °C recorded on the warmest) — small-model CPU speed on a phone moves a lot with temperature, and the CPU range below is that, not noise. Peak RSS is the process VmHWM. Before quoting, the same file was run on each backend with a real prompt: both produced a correct text answer. |
|
|
| | Backend | Prefill (206 tok) | Decode | Time-to-first-token | Peak RSS | |
| |---|---|---|---|---| |
| | **GPU (OpenCL)** | **540–541 tok/s** | **31.5–36.0 tok/s** | **0.41 s** | **546 MB** | |
| | CPU (XNNPACK) | 118–211 tok/s | 12.7–19.5 tok/s | 1.03–1.82 s | 1409 MB | |
|
|
| The GPU takes the whole graph (decode 1362/1362, prefill 1491/1491 ops on `LITERT_CL`) and wins across the board on this bundle: prefill ≥2.6× at any temperature, decode 1.6–2.8×, time-to-first-token 2.5–4.4×, and peak RSS 2.6× lower. |
|
|
