mlboydaisuke/Qwen2.5-3B-Instruct-LiteRT

Qwen2.5-3B-Instruct — LiteRT-LM (GPTQ-calibrated int4, block 128)

Built with Qwen.

Qwen/Qwen2.5-3B-Instruct converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime (the engine behind the litert-community/* models).

What makes this build different: the int4 weights are not re-quantized from scratch — they carry Qwen's official GPTQ calibration (Qwen/Qwen2.5-3B-Instruct-GPTQ-Int4), transported losslessly into the LiteRT bundle via ai-edge-quantizer's dequantized_weight_recovery (blockwise support, nightly ≥ 0.8.0.dev20260703). You get calibrated-int4 quality at block-128 speed, with no calibration step in the conversion.

File	model.litertlm — int4 block 128 (~1.75 GB)
Quantization	int4 weights (symmetric, blockwise-128) on Qwen's official GPTQ grid; embeddings + lm_head INT8
Compute	integer (dynamic int8 activations)
Context (KV cache)	4096
Base model	Qwen/Qwen2.5-3B-Instruct (36 layers, Qwen2ForCausalLM)
Decode speed	~74 tok/s (Mac M-series, GPU)

Quality — GSM8K parity

Measured on GSM8K (n=100, greedy, 0-shot chain-of-thought, max_tokens 512, identical prompt and answer-extraction for every row).

Configuration	GSM8K
bf16 (reference)	81.0%
Qwen official GPTQ-Int4, dequantized in PyTorch (n=50)	82.0%
LiteRT int4 — block 128 (this file)	75.0% (−6 pt vs bf16)

The official GPTQ calibration itself is lossless on GSM8K (82.0 vs 81.0 = noise), so the −6 pt is the cost of the on-device execution format (integer compute), not of the 4-bit weights. The 8-question smoke gate reads 8/8 (arithmetic, factual, translation — all correct, terse clean answers, no degeneration).

Usage

# build litert-lm from https://github.com/google-ai-edge/litert-lm, then:
litert_lm_main \
  --model_path model.litertlm \
  --backend gpu \
  --input_prompt "Natalia sold clips to 48 friends in April, and half as many in May. How many altogether?"

The .litertlm bundle carries the tokenizer and prompt template (Qwen2 ChatML — <|im_start|>role\n…<|im_end|>), so no separate tokenizer files are needed.

Run on Android

Update (July 2026): Google AI Edge Gallery v1.0.16+ can import litert-lm models directly from Hugging Face inside the app (tap +) — no computer or adb needed. The manual steps below are only required on older builds or for sideloading a local file.

The official Google AI Edge Gallery app runs .litertlm models on-device:

1. Install a recent Gallery (package com.google.ai.edge.gallery, 1.0.15+ supports .litertlm).
2. Download model.litertlm and push it: adb push model.litertlm /sdcard/Download/
3. In the app tap +, pick the file, and choose the GPU backend.
4. Chat — the bundle already carries the tokenizer and Qwen2 chat template.

Conversion — GPTQ grid pass-through

Converted with the official litert-torch converter. Instead of a data-free int4 recipe, the quantization stage uses ai-edge-quantizer's dequantized_weight_recovery algorithm (blockwise support landed 2026-06-11, nightly-only at the time of conversion): the official GPTQ checkpoint is dequantized to fp32 (exact — fp16 scale × int4 is exactly representable in fp32), and recovery re-derives the per-block scales bit-exactly, so the deployed int4 grid is Qwen's calibrated grid.

[
  {"regex": ".*", "operation": "*",
   "algorithm_key": "dequantized_weight_recovery",
   "op_config": {"weight_tensor_config": {"num_bits": 4, "symmetric": true,
                 "granularity": "BLOCKWISE_128", "dtype": "INT"},
                 "compute_precision": "INTEGER"}},
  {"regex": ".*", "operation": "EMBEDDING_LOOKUP",
   "algorithm_key": "min_max_uniform_quantize",
   "op_config": {"weight_tensor_config": {"num_bits": 8, "symmetric": true,
                 "granularity": "CHANNELWISE", "dtype": "INT"}}},
  {"regex": ".*(logits_output|Linear_lm_head).*", "operation": "FULLY_CONNECTED",
   "algorithm_key": "min_max_uniform_quantize",
   "op_config": {"weight_tensor_config": {"num_bits": 8, "symmetric": true,
                 "granularity": "CHANNELWISE", "dtype": "INT"}}}
]

(The embedding / tied lm_head is not GPTQ-quantized upstream, so it goes to INT8. KV cache 4096.)

2026-08-29 — default system prompt restored (weights unchanged)

The upstream chat template emits a default system turn whenever the caller sends no system message — for this model: You are Qwen, created by Alibaba Cloud. You are a helpful assistant.. The converter's template probe renders the template with a system message already present, so that block was never seen and never reached the bundle: with no system message the model was running without the default system turn it was tuned with. The chat template in model.litertlm now emits the block exactly once when no system message is given. In model.litertlm, the block is not emitted when you pass a system message. The restored block adds 21 prefill tokens to a conversation that sends no system message, so time-to-first-token grows by that much; per-token speed is unchanged.

Metadata-only change: every section of the bundle except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file per token — only the file's own sha256 differs. What changed is the input: with no system message, the prompt now renders byte-identical to the upstream chat template's output, verified on the LiteRT-LM runtime. A system message you pass yourself renders as before. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before 2026-08-29, re-download.

License

Qwen Research License (see LICENSE), inherited from the base model Qwen/Qwen2.5-3B-Instruct. Non-commercial (research/evaluation) use only — for commercial use, request a license from Alibaba Cloud.

This repository is a modified distribution of the Qwen materials: the model weights were quantized (official GPTQ int4 grid, transported via dequantized_weight_recovery) and repackaged into the LiteRT-LM .litertlm format as described in the Conversion section above. Attribution notice is in NOTICE. Built with Qwen.

---

Want a different model on-device? Open a request — free, open weights only; the export and its measured numbers get published publicly.