| --- |
| license: apache-2.0 |
| base_model: HuggingFaceTB/SmolLM2-135M-Instruct |
| pipeline_tag: text-generation |
| library_name: litert-lm |
| tags: |
| - chat |
| - litert-lm |
| - smollm |
| - on-device |
| --- |
| |
| # litert-community/SmolLM2-135M-Instruct |
|
|
| This model provides a variant of |
| [HuggingFaceTB/SmolLM2-135M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct) that is ready for |
| deployment on Android using the |
| [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM). |
|
|
| ## Use the model |
|
|
| ### Android |
|
|
| #### Edge Gallery App |
| * Download or build the [app](https://github.com/google-ai-edge/gallery?tab=readme-ov-file#-get-started-in-minutes) from GitHub. |
|
|
| * Install the [app](https://play.google.com/store/apps/details?id=com.google.ai.edge.gallery&pli=1) from Google Play. |
|
|
| * Follow the instructions in the app. |
|
|
| To build the demo app from source, please follow the [instructions](https://github.com/google-ai-edge/gallery/blob/main/README.md) |
| from the GitHub repository. |
|
|
| ## Performance (measured) |
|
|
| ### Apple M4 Max |
|
|
| Measured with the LiteRT-LM CLI: `litert-lm benchmark -p 256 -d 256 --runs 3 --cache no` |
| (litert-lm 0.15.0) on an idle Apple M4 Max (macOS); 256 prefill / 256 decode tokens, 3 iterations |
| averaged by the tool. A desktop reference point — phone-side figures vary by SoC and backend. |
|
|
| | Backend | Prefill (tokens/s) | Decode (tokens/s) | Time-to-first-token (s) | |
| |---|---|---|---| |
| | CPU | 1,698 | 104.9 | 0.22 | |
| | GPU | 7,571 | 259.6 | 0.04 | |
|
|
| ### Galaxy S26 — GPU vs CPU (litert-lm 0.16.0) |
|
|
| Measured on a physical Samsung Galaxy S26 (SM-S942Q, Snapdragon 8 Elite Gen 5 / SM8850, Android 16) with `litert_lm_advanced_main` from the litert-lm v0.16.0 release; the GPU backend is OpenCL (`LITERT_CL`). One fixed 205-token prompt text, `--benchmark`. GPU: two runs back-to-back; CPU: three runs spread across device thermal states (skin temperature up to 41.9 °C recorded on the warmest) — a 135M model on a phone CPU is very thermal-sensitive, and the wide CPU range below is that, not noise. Peak RSS is the process VmHWM. Before quoting, the same file was run on each backend with a real prompt: both produced a correct text answer. |
|
|
| | Backend | Prefill (205 tok) | Decode | Time-to-first-token | Peak RSS | |
| |---|---|---|---|---| |
| | **GPU (OpenCL)** | **1692–1711 tok/s** | 52.5–59.7 tok/s | **0.14 s** | **327 MB** | |
| | CPU (XNNPACK) | 332–618 tok/s | 41.9–69.8 tok/s | 0.35–0.64 s | 552 MB | |
|
|
| The GPU takes the whole graph (decode 1274/1274, prefill 1393/1393 ops on `LITERT_CL`). Its prefill lead holds at any device temperature (≥2.7×, and the time-to-first-token with it); decode is a wash — the two ranges overlap, and which backend decodes faster on a given run is decided by how warm the phone already is. |
|
|
