Qwen2.5-Coder-1.5B-Instruct LiteRT-LM card

README.md

@@ -0,0 +1,70 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
+tags:
+  - litert
+  - litert-lm
+  - litertlm
+  - on-device
+  - edge
+  - code
+pipeline_tag: text-generation
+library_name: litert-lm
+---
+
+# Qwen2.5-Coder-1.5B-Instruct — LiteRT-LM
+
+[Qwen/Qwen2.5-Coder-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) converted to the **LiteRT-LM** (`.litertlm`) format for on-device inference with Google's [LiteRT-LM](https://github.com/google-ai-edge/litert-lm) runtime. **Requires litert-lm ≥ 0.16.**
+
+A 1.54B code model at **1.12 GB** — small enough that a phone-class device runs it at conversational speed rather than merely running it. Code assistance is the use case that most needs that: it returns short outputs constantly, so decode rate *is* the experience.
+
+| File | Recipe | Size |
+|---|---|---|
+| `Qwen2.5-Coder-1.5B-Instruct_int4.litertlm` | int4 blockwise-32 + OCTAV on linears, int8 embedding (externalised) | 1.12 GB |
+
+## Correctness
+
+Two gates, because a general-knowledge check certifies nothing for a code model:
+
+- **8-question sanity gate: 8/8** on the GPU backend (Mac, litert-lm 0.16.0).
+- **Code gate: 6/6.** Six small functions — `fib`, `reverse_words`, `is_prime`, largest contiguous sublist sum, `count_vowels`, `flatten` — where the generated code is **executed against assertions**, not read. A task counts only if the code imports and every assertion holds.
+
+## Usage
+
+```bash
+litert-lm run ./Qwen2.5-Coder-1.5B-Instruct_int4.litertlm \
+  --prompt "Write a Python function that returns the nth Fibonacci number. Code only."
+
+# GPU
+litert-lm run ./Qwen2.5-Coder-1.5B-Instruct_int4.litertlm --backend gpu --cache no --prompt "..."
+```
+
+The bundle carries the tokenizer and Qwen's ChatML markers **including the vendor's default system prompt** (`You are Qwen, created by Alibaba Cloud. You are a helpful assistant.`), which the upstream template inserts whenever no system message is supplied — rendering byte-identical to upstream for a single-turn request. A 4096-token KV budget and six prefill signatures (1024, 256, 64, 16, 4, 1).
+
+## Performance
+
+`litert-lm benchmark` (litert-lm 0.16.0), Apple M4 Max, `-p 256 -d 256 --cache no`, quiet machine, serialized:
+
+| Backend | Prefill (256) | Decode | TTFT | Init |
+|---|---|---|---|---|
+| **GPU (Metal)** | **3037 tok/s** | **137.8 tok/s** | 0.099 s | 2.38 s |
+| CPU | 292 tok/s | 47.1 tok/s | 1.06 s | 2.87 s |
+
+For scale, a 3B-class int4 bundle converted on the same day and measured on the same machine and protocol runs 1241 tok/s prefill / 86.3 decode / 0.233 s TTFT. Decode on these runtimes is memory-bandwidth-bound, so halving the weights is the most direct speed lever there is.
+
+Honest notes:
+
+- **A Mac's GPU-vs-CPU ratio does not transfer to a phone.** Here the GPU decodes 2.9× the CPU; on a Pixel 8a a 3B model's GPU and CPU decode land within 1% of each other, because a phone's CPU and GPU share the same LPDDR. On phone hardware the GPU's win is prefill and time-to-first-token.
+- The embedding table is stored in its own bundle section. That is what keeps this file at 1.12 GB — with the table quantized inline it duplicates per prefill signature and the same weights come out at 2.53 GB.
+
+## Conversion notes
+
+Converted with [`litert-torch`](https://github.com/google-ai-edge/litert-torch) 0.9.3 / litert-converter 0.3.1 / ai-edge-quantizer 0.8.0 / litert-lm-builder 0.16.0 — a pristine released stack, no patched checkout.
+
+- **`EXTERNALIZE_EMBEDDER=1` is required, not cosmetic.** Qwen2.5-Coder ties its embedding and `lm_head`, so a recipe asking for int4 linears and an int8 embedder describes one tensor two ways; the quantizer resolves it by copying the 151936×1536 table once per signature. Seven copies, 1.63 GB, 65% of the file — and no error anywhere. Externalising the table removes the conflict.
+- **The default system prompt is baked into the template.** Upstream inserts it when the caller sends no system message; a plain ChatML template would drop it and put the model in a state it was not tuned in on every default request.
+- **Quantization**: int4 blockwise-32 with OCTAV clipping on the linears, int8 on the embedding.
+
+## License and changes
+
+Distributed under **Apache-2.0** (inherited from the base model). **Changes from the original work:** weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged into the `.litertlm` bundle. This repository is a community conversion and is not affiliated with Alibaba Cloud or the Qwen team.