Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,125 @@
+---
+license: other
+license_name: qwen-research
+license_link: LICENSE
+base_model: Qwen/Qwen2.5-3B-Instruct
+tags:
+  - litert
+  - litert-lm
+  - litertlm
+  - on-device
+  - edge
+  - qwen2
+  - gptq
+pipeline_tag: text-generation
+library_name: litert-lm
+---
+
+# Qwen2.5-3B-Instruct — LiteRT-LM (GPTQ-calibrated int4, block 128)
+
+**Built with Qwen.**
+
+[Qwen/Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct) converted to the
+**LiteRT-LM** (`.litertlm`) format for on-device inference with Google's
+[LiteRT-LM](https://github.com/google-ai-edge/litert-lm) runtime (the engine behind the
+`litert-community/*` models).
+
+What makes this build different: the int4 weights are **not re-quantized from scratch** — they carry
+**Qwen's official GPTQ calibration**
+([Qwen/Qwen2.5-3B-Instruct-GPTQ-Int4](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct-GPTQ-Int4)),
+transported **losslessly** into the LiteRT bundle via `ai-edge-quantizer`'s
+`dequantized_weight_recovery` (blockwise support, nightly ≥ 0.8.0.dev20260703). You get
+calibrated-int4 quality at block-128 speed, with no calibration step in the conversion.
+
+| | |
+|---|---|
+| **File** | `model.litertlm` — int4 **block 128** (~1.75 GB) |
+| **Quantization** | int4 weights (symmetric, blockwise-128) on **Qwen's official GPTQ grid**; embeddings + lm_head INT8 |
+| **Compute** | integer (dynamic int8 activations) |
+| **Context (KV cache)** | 4096 |
+| **Base model** | Qwen/Qwen2.5-3B-Instruct (36 layers, `Qwen2ForCausalLM`) |
+| **Decode speed** | ~74 tok/s (Mac M-series, GPU) |
+
+## Quality — GSM8K parity
+
+Measured on GSM8K (n=100, greedy, 0-shot chain-of-thought, max_tokens 512, identical prompt and
+answer-extraction for every row).
+
+| Configuration | GSM8K |
+|---|---|
+| bf16 (reference) | 81.0% |
+| Qwen official GPTQ-Int4, dequantized in PyTorch (n=50) | 82.0% |
+| **LiteRT int4 — block 128 (this file)** | **75.0%** (−6 pt vs bf16) |
+
+The official GPTQ calibration itself is lossless on GSM8K (82.0 vs 81.0 = noise), so the −6 pt is
+the cost of the on-device execution format (integer compute), not of the 4-bit weights. The 8-question
+smoke gate reads **8/8** (arithmetic, factual, translation — all correct, terse clean answers, no
+degeneration).
+
+## Usage
+
+```bash
+# build litert-lm from https://github.com/google-ai-edge/litert-lm, then:
+litert_lm_main \
+  --model_path model.litertlm \
+  --backend gpu \
+  --input_prompt "Natalia sold clips to 48 friends in April, and half as many in May. How many altogether?"
+```
+
+The `.litertlm` bundle carries the tokenizer and prompt template (Qwen2 ChatML —
+`<|im_start|>role\n…<|im_end|>`), so no separate tokenizer files are needed.
+
+## Run on Android
+
+> **Update (July 2026):** [Google AI Edge Gallery](https://github.com/google-ai-edge/gallery)
+> **v1.0.16+** can import litert-lm models **directly from Hugging Face** inside the app (tap **+**)
+> — no computer or `adb` needed. The manual steps below are only required on older builds or for
+> sideloading a local file.
+
+The official **[Google AI Edge Gallery](https://github.com/google-ai-edge/gallery)** app runs
+`.litertlm` models on-device:
+
+1. Install a **recent** Gallery (package `com.google.ai.edge.gallery`, 1.0.15+ supports `.litertlm`).
+2. Download `model.litertlm` and push it: `adb push model.litertlm /sdcard/Download/`
+3. In the app tap **+**, pick the file, and choose the **GPU** backend.
+4. Chat — the bundle already carries the tokenizer and Qwen2 chat template.
+
+## Conversion — GPTQ grid pass-through
+
+Converted with the official [`litert-torch`](https://github.com/google-ai-edge/litert-torch)
+converter. Instead of a data-free int4 recipe, the quantization stage uses
+`ai-edge-quantizer`'s **`dequantized_weight_recovery`** algorithm (blockwise support landed
+2026-06-11, nightly-only at the time of conversion): the official GPTQ checkpoint is dequantized to
+fp32 (exact — fp16 scale × int4 is exactly representable in fp32), and recovery re-derives the
+per-block scales bit-exactly, so the deployed int4 grid **is** Qwen's calibrated grid.
+
+```json
+[
+  {"regex": ".*", "operation": "*",
+   "algorithm_key": "dequantized_weight_recovery",
+   "op_config": {"weight_tensor_config": {"num_bits": 4, "symmetric": true,
+                 "granularity": "BLOCKWISE_128", "dtype": "INT"},
+                 "compute_precision": "INTEGER"}},
+  {"regex": ".*", "operation": "EMBEDDING_LOOKUP",
+   "algorithm_key": "min_max_uniform_quantize",
+   "op_config": {"weight_tensor_config": {"num_bits": 8, "symmetric": true,
+                 "granularity": "CHANNELWISE", "dtype": "INT"}}},
+  {"regex": ".*(logits_output|Linear_lm_head).*", "operation": "FULLY_CONNECTED",
+   "algorithm_key": "min_max_uniform_quantize",
+   "op_config": {"weight_tensor_config": {"num_bits": 8, "symmetric": true,
+                 "granularity": "CHANNELWISE", "dtype": "INT"}}}
+]
+```
+
+(The embedding / tied lm_head is not GPTQ-quantized upstream, so it goes to INT8. KV cache 4096.)
+
+## License
+
+**Qwen Research License** (see `LICENSE`), inherited from the base model
+[Qwen/Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct). **Non-commercial
+(research/evaluation) use only** — for commercial use, request a license from Alibaba Cloud.
+
+This repository is a **modified distribution** of the Qwen materials: the model weights were
+quantized (official GPTQ int4 grid, transported via `dequantized_weight_recovery`) and repackaged
+into the LiteRT-LM `.litertlm` format as described in the Conversion section above. Attribution
+notice is in `NOTICE`. Built with Qwen.