| --- |
| language: |
| - en |
| license: apache-2.0 |
| license_link: https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct/blob/main/LICENSE |
| pipeline_tag: text-generation |
| base_model: Qwen2.5-Coder-3B-Instruct |
| base_model_relation: quantized |
| library_name: litert-lm |
| tags: |
| - litert-lm |
| - litertlm |
| - qwen |
| - Qwen2.5 |
| --- |
| # Qwen2.5-Coder-3B-Instruct LiteRT-LM Model |
|
|
| This repository contains LiteRT-LM variant of [Qwen/Qwen2.5-Coder-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct) optimized for on-device text generation. |
|
|
| ## Available Artifact |
|
|
| | File | Quantization Recipe | Context | Size | |
| |---|---|---:|---:| |
| | `Qwen2.5_Coder_3B_It.litertlm` | dynamic_wi8_afp32 | - | 3.4 GB | |
|
|
| ## Integration |
|
|
| Ready to integrate this into your product? Get started in the [LiteRT-LM documentation](https://ai.google.dev/edge/litert-lm/overview). |
|
|
| ## Performance (measured) |
|
|
| ### Apple M4 Max |
|
|
| Measured with the LiteRT-LM CLI: `litert-lm benchmark -p 256 -d 256 --runs 3 --cache no` |
| (litert-lm 0.15.0) on an idle Apple M4 Max (macOS); 256 prefill / 256 decode tokens, 3 iterations |
| averaged by the tool. A desktop reference point — phone-side figures vary by SoC and backend. |
|
|
| | Backend | Prefill (tokens/s) | Decode (tokens/s) | Time-to-first-token (s) | |
| |---|---|---|---| |
| | CPU | 121 | 26.7 | 2.16 | |
| | GPU | 1,320 | 77.8 | 0.21 | |
|
|
| ### Galaxy S26 — GPU vs CPU (litert-lm 0.16.0) |
|
|
| Measured on a physical Samsung Galaxy S26 (SM-S942Q, Snapdragon 8 Elite Gen 5 / SM8850, Android 16) with `litert_lm_advanced_main` from the litert-lm v0.16.0 release; the GPU backend is OpenCL (`LITERT_CL`). One fixed 205-token prompt text (223 tokens under this tokenizer), `--benchmark`. Two runs per backend taken back-to-back — cells show the range. Peak RSS is the process VmHWM. Before quoting, the same file was run on each backend with a real prompt: both backends produced a correct text answer. |
|
|
| | Backend | Prefill (223 tok) | Decode | Time-to-first-token | Init | Peak RSS | |
| |---|---|---|---|---|---| |
| | **GPU (OpenCL)** | **403–423 tok/s** | **16.3–16.5 tok/s** | **0.59–0.61 s** | 4.3–4.9 s | **864 MB** | |
| | CPU (XNNPACK) | 159–187 tok/s | 12.1–13.6 tok/s | 1.26–1.49 s | 3.6–4.6 s | 3934 MB | |
|
|
| The GPU takes the whole graph — decode 1603/1603 ops and prefill 1452/1452 on `LITERT_CL`. It wins prefill 2.2–2.7× and decode 1.2–1.4×, and peaks 4.6× lower (864 against 3934 MB) — for a 3.4 GB int8 bundle the RSS difference is the practical headline. |
|
|
