Hugging Face's logo

mlboydaisuke
/
DeepSeek-R1-Distill-Qwen-1.5B-LiteRT

Text Generation
LiteRT-LM
LiteRT
litertlm
on-device
edge
reasoning
deepseek-r1
Model card Files
xet
 Community
DeepSeek-R1-Distill-Qwen-1.5B-LiteRT / README.md
mlboydaisuke's picture
mlboydaisuke
Document Gallery 1.0.16 direct Hugging Face import + desktop LiteRT-LM CLI (serve/run)
3c8f42f verified
preview code
|
Raw
History Blame Contribute Delete
3.72 kB
metadata 
license: mit
base_model: deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
tags:
  - litert
  - litert-lm
  - litertlm
  - on-device
  - edge
  - reasoning
  - deepseek-r1
pipeline_tag: text-generation
library_name: litert-lm

DeepSeek-R1-Distill-Qwen-1.5B — LiteRT-LM (blockwise int4)

deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime.

A mobile-size reasoning model: it emits a <think> … </think> chain before the answer, and at ~1 GB it runs on a phone. MIT-licensed (Apache-2.0 Qwen2.5 base). Converted with the official upstream litert-torch — no fork.

File model.litertlm (~1.0 GB)
Quantization int4 weights — blockwise (block 32) + OCTAV optimal-clipping, symmetric; embedding INT8
Compute integer
Context (KV cache) 4096
Base model deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
Decode speed ~116 tok/s (Mac M-series, Metal GPU, greedy); runs on 8 GB phones (iPhone / Android)

Run on Android

Update (July 2026): Google AI Edge Gallery v1.0.16+ can import litert-lm models directly from Hugging Face inside the app (tap +) — no computer or adb needed. The manual steps below are only required on older builds or for sideloading a local file.

The easiest way to try this on a phone is the official Google AI Edge Gallery app:

  1. Install a recent Gallery (com.google.ai.edge.gallery, 1.0.15+ supports .litertlm).
  2. adb push model.litertlm /sdcard/Download/
  3. In the app: + → pick the file → CPU or GPU. At ~1 GB this fits comfortably.
  4. Chat — the bundle carries the tokenizer and DeepSeek prompt template (<｜User｜> / <｜Assistant｜>, stop <｜end▁of▁sentence｜>). The model opens a <think> block, reasons, then answers.

To embed it in your own app, use the LiteRT-LM Kotlin API (com.google.ai.edge.litertlm:litertlm-android).

Run on desktop (LiteRT-LM CLI)

The same .litertlm bundle runs on macOS / Linux / Windows with the official LiteRT-LM CLI — including as a local OpenAI-compatible API server:

pip install litert-lm
litert-lm import --from-huggingface-repo mlboydaisuke/DeepSeek-R1-Distill-Qwen-1.5B-LiteRT model.litertlm deepseek-r1-distill-qwen-1.5b-litert
litert-lm run deepseek-r1-distill-qwen-1.5b-litert     # interactive chat in the terminal
litert-lm serve           # local OpenAI-compatible API server

Quality — GSM8K

GSM8K (n=100, greedy, 0-shot, identical prompt + extraction; max_new_tokens=2048).

Configuration GSM8K
bf16 (reference) 81.0%
This model — LiteRT int4 (BOCTAV4) 73.0%

73 % is a strong, coherent, non-degenerate score for a 1.5B reasoning model that fits on a phone; the <think> reasoning is preserved through 4-bit. At 1.5B, int4 costs ~8 pt vs bf16 (small-model 4-bit sensitivity — a 1.5B has less redundancy than the 7B sibling, which is at −1 pt parity). Shipped as int4 for the best on-device size/speed.

Conversion

Official upstream litert-torch export_hf (clean worktree at upstream/main, no fork). Qwen2ForCausalLM, no custom code. int4 = blockwise-32 + OCTAV, INT8 embedding, KV cache 4096.

License

MIT (model weights); Qwen2.5 base is Apache-2.0. Commercial use and derivatives permitted.