litert-community

/

Qwen2.5-0.5B-Instruct

like 8

Follow

LiteRT Community (FKA TFLite) 8.62k

This PR adds an int4 (wi4b64) dynamic-range quantized .litertlm variant: Qwen2.5-0.5B-Instruct_wi4b64_ekv1280.litertlm (265 MB, external KV cache length 1280).

Toolchain / pipeline (unpack → quantize → repack):

1. litert-lm-peek dump of the official f32 .litertlm release in this repository.
2. ai-edge-quantizer 0.8.0 post-training dynamic-range dynamic_wi4b64_afp32 (int4 weights, blockwise-64, fp16 scales, fp32 activations) applied full-scope to the TFLite section.
3. Rebuilt with litert-lm-builder 0.14.0; tokenizer and LlmMetadata mirrored from the official q8 release. Resulting .litertlm format 1.5.0.

Source checkpoint: litert-community/Qwen2.5-0.5B-Instruct f32 release.

Validation: inference-validated on Windows x86_64 (CPU and GPU) and Android arm64 (Snapdragon 865-class device): 35.5 tok/s decode on device CPU — +38 % vs the official q8 file — with correct Korean QA output.

Runtime tested: LiteRT-LM v0.14.0, Windows x86_64 + Android arm64 (Snapdragon 865).

Known caveat: the metadata carries lowercase model_type (tf_lite_prefill_decode) vs uppercase in the official files — loads fine on LiteRT-LM v0.14.0, noted for completeness.

Produced by the LiteRT-LM-Unity project (https://github.com/Leuconoe/LiteRT-LM-Unity, release v0.14.0-unity).