Qwen2.5-Coder-7B-Instruct LiteRT-LM
This repository contains a LiteRT-LM converted version of Qwen2.5-Coder-7B-Instruct for efficient on-device code generation and text generation.
The model was converted using the official LiteRT-LM exporter and has been verified to run successfully with LiteRT-LM.
Base Model
- Model: Qwen/Qwen2.5-Coder-7B-Instruct
- Architecture: Qwen2.5
- Task: Text Generation / Code Generation
- Original License: Apache-2.0 :contentReference[oaicite:0]{index=0}
Conversion Environment
| Item | Value |
|---|---|
| Conversion Date | 2026-07-19 |
| Transformers | 5.14.1 |
| LiteRT-LM Runtime (tested) | 0.14.0 |
Conversion Configuration
| Item | Value |
|---|---|
| Quantization | dynamic_wi4_afp32 |
| Batch Size | 1 |
| Cache Implementation | LiteRTLMCache |
| Cache Length | 4096 |
| Prefill Length | 128 |
| Dynamic Shape | Disabled |
| GPU Dynamic Cache | Disabled |
| GPU Dynamic Prefill | Disabled |
| Mixed Precision | Disabled |
| FP16 | Disabled |
| Externalized Embedding | Enabled |
| Externalized RoPE | Disabled |
| Jinja Chat Template | Enabled |
| Stateful After | -1 |
| Bundle LiteRT-LM | Enabled |
Export Configuration
task : text_generation
quantization_recipe : dynamic_wi4_afp32
batch_size : 1
cache_length : 4096
prefill_lengths : [128]
cache_implementation : LiteRTLMCache
bundle_litert_lm : True
externalize_embedder : True
externalize_rope : False
enable_dynamic_shape : False
enable_gpu_dynamic_cache : False
enable_gpu_dynamic_prefill : False
experimental_use_fp16 : False
experimental_use_mixed_precision : False
use_jinja_template : True
stateful_after : -1
Features
- Official Qwen2.5-Coder-7B-Instruct converted for LiteRT-LM
- Dynamic 4-bit weight quantization (
dynamic_wi4_afp32) - FP32 activation precision
- LiteRT-LM optimized
- Externalized embedding layer
- Original tokenizer and chat template preserved
- GPU backend supported by LiteRT-LM
- Ready for edge-device deployment
Usage
import litert_lm
engine = litert_lm.Engine(
"PATH_TO_MODEL",
backend=litert_lm.Backend.GPU(),
)
response = engine.generate(
"Write a Python implementation of quicksort."
)
print(response)
Performance
The model uses:
- Weight Quantization: Dynamic Int4 (
dynamic_wi4_afp32) - Activation Precision: FP32
- KV Cache: 4096 tokens
- Prefill Length: 128 tokens
Actual inference speed and memory usage depend on the LiteRT-LM backend and target hardware.
Compatibility
This model has been tested with:
- LiteRT-LM 0.14.0
- Transformers 5.14.1
Other LiteRT-LM versions may also work but have not been verified.
Notes
This repository contains only the LiteRT-LM converted model.
It is not intended to be loaded using transformers or AutoModelForCausalLM.from_pretrained(). Instead, use the LiteRT-LM runtime.
Acknowledgements
- Alibaba Cloud Qwen Team
- Google LiteRT-LM
License
This repository contains a converted version of the original model and does not modify the model weights beyond the LiteRT-LM conversion process.
Please refer to the original model repository for licensing details:
- Downloads last month
- 12