Upload README.md with huggingface_hub

e79d96f verified 17 days ago

5.57 kB

license: other
license_name: qwen-research
license_link: LICENSE
base_model: Qwen/Qwen2.5-3B-Instruct
tags:
  - litert
  - litert-lm
  - litertlm
  - on-device
  - edge
  - qwen2
  - gptq
pipeline_tag: text-generation
library_name: litert-lm

Qwen2.5-3B-Instruct — LiteRT-LM (GPTQ-calibrated int4, block 128)

Built with Qwen.

Qwen/Qwen2.5-3B-Instruct converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime (the engine behind the litert-community/* models).

What makes this build different: the int4 weights are not re-quantized from scratch — they carry Qwen's official GPTQ calibration (Qwen/Qwen2.5-3B-Instruct-GPTQ-Int4), transported losslessly into the LiteRT bundle via ai-edge-quantizer's dequantized_weight_recovery (blockwise support, nightly ≥ 0.8.0.dev20260703). You get calibrated-int4 quality at block-128 speed, with no calibration step in the conversion.


File	`model.litertlm` — int4 block 128 (~1.75 GB)
Quantization	int4 weights (symmetric, blockwise-128) on Qwen's official GPTQ grid; embeddings + lm_head INT8
Compute	integer (dynamic int8 activations)
Context (KV cache)	4096
Base model	Qwen/Qwen2.5-3B-Instruct (36 layers, `Qwen2ForCausalLM`)
Decode speed	~74 tok/s (Mac M-series, GPU)

Quality — GSM8K parity

Measured on GSM8K (n=100, greedy, 0-shot chain-of-thought, max_tokens 512, identical prompt and answer-extraction for every row).

Configuration	GSM8K
bf16 (reference)	81.0%
Qwen official GPTQ-Int4, dequantized in PyTorch (n=50)	82.0%
LiteRT int4 — block 128 (this file)	75.0% (−6 pt vs bf16)

The official GPTQ calibration itself is lossless on GSM8K (82.0 vs 81.0 = noise), so the −6 pt is the cost of the on-device execution format (integer compute), not of the 4-bit weights. The 8-question smoke gate reads 8/8 (arithmetic, factual, translation — all correct, terse clean answers, no degeneration).

Usage

# build litert-lm from https://github.com/google-ai-edge/litert-lm, then:
litert_lm_main \
  --model_path model.litertlm \
  --backend gpu \
  --input_prompt "Natalia sold clips to 48 friends in April, and half as many in May. How many altogether?"

The .litertlm bundle carries the tokenizer and prompt template (Qwen2 ChatML — <|im_start|>role\n…<|im_end|>), so no separate tokenizer files are needed.

Run on Android

Update (July 2026): Google AI Edge Gallery v1.0.16+ can import litert-lm models directly from Hugging Face inside the app (tap +) — no computer or adb needed. The manual steps below are only required on older builds or for sideloading a local file.

The official Google AI Edge Gallery app runs .litertlm models on-device:

Install a recent Gallery (package com.google.ai.edge.gallery, 1.0.15+ supports .litertlm).
Download model.litertlm and push it: adb push model.litertlm /sdcard/Download/
In the app tap +, pick the file, and choose the GPU backend.
Chat — the bundle already carries the tokenizer and Qwen2 chat template.

Conversion — GPTQ grid pass-through

Converted with the official litert-torch converter. Instead of a data-free int4 recipe, the quantization stage uses ai-edge-quantizer's dequantized_weight_recovery algorithm (blockwise support landed 2026-06-11, nightly-only at the time of conversion): the official GPTQ checkpoint is dequantized to fp32 (exact — fp16 scale × int4 is exactly representable in fp32), and recovery re-derives the per-block scales bit-exactly, so the deployed int4 grid is Qwen's calibrated grid.

[
  {"regex": ".*", "operation": "*",
   "algorithm_key": "dequantized_weight_recovery",
   "op_config": {"weight_tensor_config": {"num_bits": 4, "symmetric": true,
                 "granularity": "BLOCKWISE_128", "dtype": "INT"},
                 "compute_precision": "INTEGER"}},
  {"regex": ".*", "operation": "EMBEDDING_LOOKUP",
   "algorithm_key": "min_max_uniform_quantize",
   "op_config": {"weight_tensor_config": {"num_bits": 8, "symmetric": true,
                 "granularity": "CHANNELWISE", "dtype": "INT"}}},
  {"regex": ".*(logits_output|Linear_lm_head).*", "operation": "FULLY_CONNECTED",
   "algorithm_key": "min_max_uniform_quantize",
   "op_config": {"weight_tensor_config": {"num_bits": 8, "symmetric": true,
                 "granularity": "CHANNELWISE", "dtype": "INT"}}}
]

(The embedding / tied lm_head is not GPTQ-quantized upstream, so it goes to INT8. KV cache 4096.)

License

Qwen Research License (see LICENSE), inherited from the base model Qwen/Qwen2.5-3B-Instruct. Non-commercial (research/evaluation) use only — for commercial use, request a license from Alibaba Cloud.

This repository is a modified distribution of the Qwen materials: the model weights were quantized (official GPTQ int4 grid, transported via dequantized_weight_recovery) and repackaged into the LiteRT-LM .litertlm format as described in the Conversion section above. Attribution notice is in NOTICE. Built with Qwen.