---
language:
- en
license: apache-2.0
license_link: https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct/blob/main/LICENSE
pipeline_tag: text-generation
base_model: Qwen2.5-Coder-3B-Instruct
base_model_relation: quantized
library_name: litert-lm
tags:
- litert-lm
- litertlm
- qwen
- Qwen2.5
---
# Qwen2.5-Coder-3B-Instruct LiteRT-LM Model

This repository contains LiteRT-LM variant of [Qwen/Qwen2.5-Coder-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct) optimized for on-device text generation.

## Available Artifact

| File | Quantization Recipe | Context | Size |
|---|---|---:|---:|
| `Qwen2.5_Coder_3B_It.litertlm` | dynamic_wi8_afp32 | - | 3.4 GB |

## Integration

Ready to integrate this into your product? Get started in the [LiteRT-LM documentation](https://ai.google.dev/edge/litert-lm/overview).

## Performance (measured)

### Apple M4 Max

Measured with the LiteRT-LM CLI: `litert-lm benchmark -p 256 -d 256 --runs 3 --cache no`
(litert-lm 0.15.0) on an idle Apple M4 Max (macOS); 256 prefill / 256 decode tokens, 3 iterations
averaged by the tool. A desktop reference point — phone-side figures vary by SoC and backend.

| Backend | Prefill (tokens/s) | Decode (tokens/s) | Time-to-first-token (s) |
|---|---|---|---|
| CPU | 121 | 26.7 | 2.16 |
| GPU | 1,320 | 77.8 | 0.21 |

### Galaxy S26 — GPU vs CPU (litert-lm 0.16.0)

Measured on a physical Samsung Galaxy S26 (SM-S942Q, Snapdragon 8 Elite Gen 5 / SM8850, Android 16) with `litert_lm_advanced_main` from the litert-lm v0.16.0 release; the GPU backend is OpenCL (`LITERT_CL`). One fixed 205-token prompt text (223 tokens under this tokenizer), `--benchmark`. Two runs per backend taken back-to-back — cells show the range. Peak RSS is the process VmHWM. Before quoting, the same file was run on each backend with a real prompt: both backends produced a correct text answer.

| Backend | Prefill (223 tok) | Decode | Time-to-first-token | Init | Peak RSS |
|---|---|---|---|---|---|
| **GPU (OpenCL)** | **403–423 tok/s** | **16.3–16.5 tok/s** | **0.59–0.61 s** | 4.3–4.9 s | **864 MB** |
| CPU (XNNPACK) | 159–187 tok/s | 12.1–13.6 tok/s | 1.26–1.49 s | 3.6–4.6 s | 3934 MB |

The GPU takes the whole graph — decode 1603/1603 ops and prefill 1452/1452 on `LITERT_CL`. It wins prefill 2.2–2.7× and decode 1.2–1.4×, and peaks 4.6× lower (864 against 3934 MB) — for a 3.4 GB int8 bundle the RSS difference is the practical headline.