license: apache-2.0
base_model: HuggingFaceTB/SmolLM2-360M-Instruct
pipeline_tag: text-generation
library_name: litert-lm
tags:
- chat
- litert-lm
- smollm
- on-device
litert-community/SmolLM2-360M-Instruct
This model provides a variant of HuggingFaceTB/SmolLM2-360M-Instruct that is ready for deployment on Android using the LiteRT-LM.
Use the model
Android
Edge Gallery App
Download or build the app from GitHub.
Install the app from Google Play.
Follow the instructions in the app.
To build the demo app from source, please follow the instructions from the GitHub repository.
Performance (measured)
Apple M4 Max
Measured with the LiteRT-LM CLI: litert-lm benchmark -p 256 -d 256 --runs 3 --cache no
(litert-lm 0.15.0) on an idle Apple M4 Max (macOS); 256 prefill / 256 decode tokens, 3 iterations
averaged by the tool. A desktop reference point — phone-side figures vary by SoC and backend.
| Backend | Prefill (tokens/s) | Decode (tokens/s) | Time-to-first-token (s) |
|---|---|---|---|
| CPU | 588 | 37.3 | 0.63 |
| GPU | 2,773 | 133.6 | 0.11 |
Galaxy S26 — GPU vs CPU (litert-lm 0.16.0)
Measured on a physical Samsung Galaxy S26 (SM-S942Q, Snapdragon 8 Elite Gen 5 / SM8850, Android 16) with litert_lm_advanced_main from the litert-lm v0.16.0 release; the GPU backend is OpenCL (LITERT_CL). One fixed 205-token prompt text (206 tokens under this tokenizer), --benchmark. GPU: two runs back-to-back; CPU: three runs spread across device thermal states (skin temperature up to 41.9 °C recorded on the warmest) — small-model CPU speed on a phone moves a lot with temperature, and the CPU range below is that, not noise. Peak RSS is the process VmHWM. Before quoting, the same file was run on each backend with a real prompt: both produced a correct text answer.
| Backend | Prefill (206 tok) | Decode | Time-to-first-token | Peak RSS |
|---|---|---|---|---|
| GPU (OpenCL) | 540–541 tok/s | 31.5–36.0 tok/s | 0.41 s | 546 MB |
| CPU (XNNPACK) | 118–211 tok/s | 12.7–19.5 tok/s | 1.03–1.82 s | 1409 MB |
The GPU takes the whole graph (decode 1362/1362, prefill 1491/1491 ops on LITERT_CL) and wins across the board on this bundle: prefill ≥2.6× at any temperature, decode 1.6–2.8×, time-to-first-token 2.5–4.4×, and peak RSS 2.6× lower.