README.md

---
license: apache-2.0
base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
tags:
  - litert
  - litert-lm
  - litertlm
  - on-device
  - edge
  - code
pipeline_tag: text-generation
library_name: litert-lm
---

# Qwen2.5-Coder-1.5B-Instruct — LiteRT-LM

[Qwen/Qwen2.5-Coder-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) converted to the **LiteRT-LM** (`.litertlm`) format for on-device inference with Google's [LiteRT-LM](https://github.com/google-ai-edge/litert-lm) runtime. **Requires litert-lm ≥ 0.16.**

A 1.54B code model at **1.12 GB** — small enough that a phone-class device runs it at conversational speed rather than merely running it. Code assistance is the use case that most needs that: it returns short outputs constantly, so decode rate *is* the experience.

| File | Recipe | Size |
|---|---|---|
| `Qwen2.5-Coder-1.5B-Instruct_int4.litertlm` | int4 blockwise-32 + OCTAV on linears, int8 embedding (externalised) | 1.12 GB |

A 3B sibling exists in this org — [litert-community/Qwen2.5-Coder-3B-Instruct](https://huggingface.co/litert-community/Qwen2.5-Coder-3B-Instruct), 3.43 GB, converted by another contributor. Pick by what you are optimising: that one for capability, this one for speed and footprint. The two have not been benchmarked against each other on the same machine, so this card makes no claim about their relative quality.

## Correctness

Two gates, because a general-knowledge check certifies nothing for a code model:

- **8-question sanity gate: 8/8** on the GPU backend (Mac, litert-lm 0.16.0).
- **Code gate: 6/6.** Six small functions — `fib`, `reverse_words`, `is_prime`, largest contiguous sublist sum, `count_vowels`, `flatten` — where the generated code is **executed against assertions**, not read. A task counts only if the code imports and every assertion holds.

## Usage

```bash
litert-lm run ./Qwen2.5-Coder-1.5B-Instruct_int4.litertlm \
  --prompt "Write a Python function that returns the nth Fibonacci number. Code only."

# GPU
litert-lm run ./Qwen2.5-Coder-1.5B-Instruct_int4.litertlm --backend gpu --cache no --prompt "..."
```

The bundle carries the tokenizer and Qwen's ChatML markers **including the vendor's default system prompt** (`You are Qwen, created by Alibaba Cloud. You are a helpful assistant.`), which the upstream template inserts whenever no system message is supplied — rendering byte-identical to upstream for a single-turn request. A 4096-token KV budget and six prefill signatures (1024, 256, 64, 16, 4, 1).

## Performance

`litert-lm benchmark` (litert-lm 0.16.0), Apple M4 Max, `-p 256 -d 256 --cache no`, quiet machine, serialized:

| Backend | Prefill (256) | Decode | TTFT | Init |
|---|---|---|---|---|
| **GPU (Metal)** | **3037 tok/s** | **137.8 tok/s** | 0.099 s | 2.38 s |
| CPU | 292 tok/s | 47.1 tok/s | 1.06 s | 2.87 s |

For scale, a 3B-class int4 bundle converted on the same day and measured on the same machine and protocol runs 1241 tok/s prefill / 86.3 decode / 0.233 s TTFT. Decode on these runtimes is memory-bandwidth-bound, so halving the weights is the most direct speed lever there is.

Honest notes:

- **A Mac's GPU-vs-CPU ratio does not transfer to a phone.** Here the GPU decodes 2.9× the CPU; on a Pixel 8a a 3B model's GPU and CPU decode land within 1% of each other, because a phone's CPU and GPU share the same LPDDR. On phone hardware the GPU's win is prefill and time-to-first-token.
- The embedding table is stored in its own bundle section. That is what keeps this file at 1.12 GB — with the table quantized inline it duplicates per prefill signature and the same weights come out at 2.53 GB.

## Conversion notes

Converted with [`litert-torch`](https://github.com/google-ai-edge/litert-torch) 0.9.3 / litert-converter 0.3.1 / ai-edge-quantizer 0.8.0 / litert-lm-builder 0.16.0 — a pristine released stack, no patched checkout.

- **`EXTERNALIZE_EMBEDDER=1` is required, not cosmetic.** Qwen2.5-Coder ties its embedding and `lm_head`, so a recipe asking for int4 linears and an int8 embedder describes one tensor two ways; the quantizer resolves it by copying the 151936×1536 table once per signature. Seven copies, 1.63 GB, 65% of the file — and no error anywhere. Externalising the table removes the conflict.
- **The default system prompt is baked into the template.** Upstream inserts it when the caller sends no system message; a plain ChatML template would drop it and put the model in a state it was not tuned in on every default request.
- **Quantization**: int4 blockwise-32 with OCTAV clipping on the linears, int8 on the embedding.

## License and changes

Distributed under **Apache-2.0** (inherited from the base model). **Changes from the original work:** weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged into the `.litertlm` bundle. This repository is a community conversion and is not affiliated with Alibaba Cloud or the Qwen team.