mlboydaisuke/SmolLM3-3B-LiteRT

SmolLM3-3B — LiteRT-LM (blockwise int4)

HuggingFaceTB/SmolLM3-3B converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime (the engine behind the official litert-community/* models).

SmolLM3 is a fully-open 3B decoder (Apache-2.0) with GQA, a NoPE attention schedule, multilingual support, and long-context training — a strong small reasoner.

File	model.litertlm (~1.9 GB)
Quantization	int4 weights — blockwise (block 32) + OCTAV optimal-clipping, symmetric; embedding INT8
Compute	integer
Context (KV cache)	4096
Base model	HuggingFaceTB/SmolLM3-3B
Decode speed	~22.5 tok/s (iPhone 17 Pro, Metal GPU; loads 7.7 s, ~1.24 GB footprint) · ~93 tok/s (Mac M-series, LiteRT-LM, Metal GPU, greedy)

Usage

Run with the LiteRT-LM runtime:

# build litert-lm from https://github.com/google-ai-edge/litert-lm, then:
litert_lm_main \
  --model_path model.litertlm \
  --backend gpu \
  --input_prompt "Explain on-device AI in one sentence."

The .litertlm bundle carries the tokenizer and the prompt template (ChatML — <|im_start|>role / <|im_end|>, stop token <|im_end|>), so no separate tokenizer files are needed.

Run on Android

Update (July 2026): Google AI Edge Gallery v1.0.16+ can import litert-lm models directly from Hugging Face inside the app (tap +) — no computer or adb needed. The manual steps below are only required on older builds or for sideloading a local file.

The easiest way to try this model on a phone is the official Google AI Edge Gallery app — it runs .litertlm models fully on-device and can import your own:

1. Install a recent Gallery (package com.google.ai.edge.gallery, APK from the repo's releases — 1.0.15+ supports .litertlm). Older 1.0.x builds (package com.google.aiedge.gallery) only accept the legacy MediaPipe .task format and reject .litertlm.
2. Download model.litertlm from this repo and push it to the device:

   adb push model.litertlm /sdcard/Download/
   

3. In the app, tap the + button (bottom-right), pick the file, and choose the GPU backend (CPU also works).
4. Chat. Nothing else to configure — the .litertlm bundle already carries the tokenizer and ChatML prompt template.

See the Gallery Importing Local Models guide for details. To embed the model in your own Android app instead, use the LiteRT-LM Kotlin API (Gradle artifact com.google.ai.edge.litertlm:litertlm-android, getting started).

Run on desktop (LiteRT-LM CLI)

The same .litertlm bundle runs on macOS / Linux / Windows with the official LiteRT-LM CLI — including as a local OpenAI-compatible API server:

pip install litert-lm
litert-lm import --from-huggingface-repo mlboydaisuke/SmolLM3-3B-LiteRT model.litertlm smollm3-3b-litert
litert-lm run smollm3-3b-litert     # interactive chat in the terminal
litert-lm serve           # local OpenAI-compatible API server

Quality — GSM8K parity

Measured on GSM8K (n=100, greedy, 0-shot chain-of-thought asking for #### <n>, identical prompt and answer-extraction for both rows — only the quantization differs).

Configuration	GSM8K
bf16 (reference)	81.0%
This model — LiteRT int4 (BOCTAV4)	81.0%

LiteRT int4 is fully at parity — 0.0 pt vs the bf16 reference. The blockwise-32 + OCTAV recipe with a 4096 KV cache preserves reasoning accuracy exactly at n=100. The model produces visible step-by-step chain-of-thought in the answer body and terminates cleanly at <|im_end|> (no rambling).

Conversion

Converted with litert-torch via its generic export_hf path. SmolLM3ForCausalLM rides the existing converter with no custom code: the NoPE attention schedule (rotary disabled on every 4th layer, no_rope_layer_interval=4) lowers to generic ops with no custom kernel. The int4 recipe is blockwise (block 32) + OCTAV optimal-clipping with the embedding kept at INT8; the embedding is externalized into its own bundle section so the main weights section stays under the iOS ~2 GiB single-mmap limit. Blockwise (not channelwise) int4 plus OCTAV is what holds reasoning accuracy at parity.

2026-08-29 — default system prompt restored (weights unchanged)

The upstream chat template emits a default system turn whenever the caller sends no system message — for this model: the ## Metadata header (knowledge cutoff, today's date, Reasoning Mode: /think) followed by the instructions You are a helpful AI assistant named SmolLM, trained by Hugging Face…. The converter's template probe renders the template with a system message already present, so that block was never seen and never reached the bundle: with no system message the model was running without the default system turn it was tuned with. The chat template in model.litertlm now emits the block exactly once when no system message is given. In model.litertlm, the block is not emitted when you pass a system message; the upstream template also wraps a caller's system message in its own preamble, and this file passes it through unchanged, exactly as it did before. The restored block adds 240 prefill tokens to a conversation that sends no system message, so time-to-first-token grows by that much; per-token speed is unchanged.

Metadata-only change: every section of the bundle except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file per token — only the file's own sha256 differs. What changed is the input: with no system message, the prompt now renders byte-identical to the upstream chat template's output, verified on the LiteRT-LM runtime. A system message you pass yourself renders as before. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before 2026-08-29, re-download.

License

Apache-2.0, inherited from the base model HuggingFaceTB/SmolLM3-3B.

---

Want a different model on-device? Open a request — free, open weights only; the export and its measured numbers get published publicly.