language:
- en
license: apache-2.0
license_link: https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct/blob/main/LICENSE
pipeline_tag: text-generation
base_model: Qwen2.5-Coder-3B-Instruct
base_model_relation: quantized
library_name: litert-lm
tags:
- litert-lm
- litertlm
- qwen
- Qwen2.5
Qwen2.5-Coder-3B-Instruct LiteRT-LM Model
This repository contains LiteRT-LM variant of Qwen/Qwen2.5-Coder-3B-Instruct optimized for on-device text generation.
Available Artifact
| File | Quantization Recipe | Context | Size |
|---|---|---|---|
Qwen2.5_Coder_3B_It.litertlm |
dynamic_wi8_afp32 | - | 3.4 GB |
Integration
Ready to integrate this into your product? Get started in the LiteRT-LM documentation.
Performance (measured)
Apple M4 Max
Measured with the LiteRT-LM CLI: litert-lm benchmark -p 256 -d 256 --runs 3 --cache no
(litert-lm 0.15.0) on an idle Apple M4 Max (macOS); 256 prefill / 256 decode tokens, 3 iterations
averaged by the tool. A desktop reference point — phone-side figures vary by SoC and backend.
| Backend | Prefill (tokens/s) | Decode (tokens/s) | Time-to-first-token (s) |
|---|---|---|---|
| CPU | 121 | 26.7 | 2.16 |
| GPU | 1,320 | 77.8 | 0.21 |
Galaxy S26 — GPU vs CPU (litert-lm 0.16.0)
Measured on a physical Samsung Galaxy S26 (SM-S942Q, Snapdragon 8 Elite Gen 5 / SM8850, Android 16) with litert_lm_advanced_main from the litert-lm v0.16.0 release; the GPU backend is OpenCL (LITERT_CL). One fixed 205-token prompt text (223 tokens under this tokenizer), --benchmark. Two runs per backend taken back-to-back — cells show the range. Peak RSS is the process VmHWM. Before quoting, the same file was run on each backend with a real prompt: both backends produced a correct text answer.
| Backend | Prefill (223 tok) | Decode | Time-to-first-token | Init | Peak RSS |
|---|---|---|---|---|---|
| GPU (OpenCL) | 403–423 tok/s | 16.3–16.5 tok/s | 0.59–0.61 s | 4.3–4.9 s | 864 MB |
| CPU (XNNPACK) | 159–187 tok/s | 12.1–13.6 tok/s | 1.26–1.49 s | 3.6–4.6 s | 3934 MB |
The GPU takes the whole graph — decode 1603/1603 ops and prefill 1452/1452 on LITERT_CL. It wins prefill 2.2–2.7× and decode 1.2–1.4×, and peaks 4.6× lower (864 against 3934 MB) — for a 3.4 GB int8 bundle the RSS difference is the practical headline.