OLMo-2-1B-Instruct — LiteRT-LM (blockwise int4)

allenai/OLMo-2-0425-1B-Instruct converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime.

OLMo-2 is AllenAI's fully-open model family — open weights, open data, open training code, Apache-2.0. This 1B variant is small enough to run on a phone, and is converted with the official upstream litert-torch with no fork patches.

Model description

File	OLMo-2-1B-Instruct_q4_block32_ekv4096.litertlm (0.93 GB)
Quantization	int4 weights — blockwise (block 32) + OCTAV optimal-clipping, symmetric; embedding int8
Compute	integer
Context (KV cache)	4096
Base model	allenai/OLMo-2-0425-1B-Instruct

The bundle carries the tokenizer and OLMo-2's native Tülu prompt template (<|user|> / <|assistant|>, stop token <|endoftext|>), so no separate tokenizer files are needed.

How to use

1. Install the runtime

pip install litert-lm

2. Run it in one command — this downloads the model and answers the prompt:

litert-lm run --from-huggingface-repo litert-community/OLMo-2-1B-Instruct \
  OLMo-2-1B-Instruct_q4_block32_ekv4096.litertlm \
  --prompt "Explain on-device AI in one short sentence."

3. Keep it for repeated use — import once, then refer to it by name:

litert-lm import --from-huggingface-repo litert-community/OLMo-2-1B-Instruct \
  OLMo-2-1B-Instruct_q4_block32_ekv4096.litertlm olmo-2-1b-instruct

litert-lm run olmo-2-1b-instruct     # interactive chat
litert-lm serve                      # local OpenAI-compatible API server

The same bundle runs on macOS, Linux and Windows.

4. Run it on Android

Google AI Edge Gallery 1.0.16+ imports .litertlm models directly from Hugging Face — tap + (bottom-right) inside the app and pick this repository; no computer or adb needed. To sideload a local file instead:

adb push OLMo-2-1B-Instruct_q4_block32_ekv4096.litertlm /sdcard/Download/

then tap + and select the file. At 0.93 GB it fits comfortably on an 8 GB phone. To embed the model in your own Android app, use the LiteRT-LM Kotlin API (com.google.ai.edge.litertlm:litertlm-android).

Performance

litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine:

Device	Backend	Prefill (256)	Decode	TTFT
Apple M4 Max (macOS)	CPU	347 tok/s	33.3 tok/s	0.87 s
Apple M4 Max (macOS)	GPU (Metal)	3223 tok/s	150.1 tok/s	0.10 s

Reproducibility: the GPU figures repeat to within about 1% across invocations; the CPU figures are noisier — six invocations of this model spread its CPU decode over 29.0–33.3 tok/s, so treat the CPU row as accurate to roughly ±7%.

On device — iPhone 17 Pro (iOS 27.0), GPU/Metal backend, single cold run with no warm-up turn, prompt "Explain on-device AI in one short sentence.", max 512 tokens:

Device	Backend	Prefill	Decode	TTFT	Load	Peak footprint
iPhone 17 Pro	GPU (Metal)	27.6 tok/s	24.4 tok/s	0.95 s	5.2 s	1330 MB

The iPhone prefill figure is measured on a short chat prompt, so it is dominated by fixed per-turn overhead rather than prefill throughput — it is not comparable to the 256-token desktop prefill column. The model also fits and runs on a Pixel 8a (8 GB) via Gallery; no timing was taken there, so none is quoted.

Accuracy note

GSM8K, n=100, greedy, 0-shot chain-of-thought, identical prompt and answer extraction for every row. The bf16 reference runs in PyTorch on CPU; the quantized rows run as .litertlm through the LiteRT-LM runtime.

Configuration	GSM8K	Size
bf16 (reference)	72.0%	—
int4, blockwise-32 + OCTAV — this model	63.0%	0.93 GB
int8 (dynamic_wi8_afp32)	65.0%	1.50 GB

63% is a coherent, non-degenerate score for a 1B: answers terminate cleanly at <|endoftext|> and do not degenerate. At this size 4-bit quantization costs about 9 points against bf16 — a 1B has less redundancy to absorb int4 rounding than a 3B+, where the same recipe lands at parity. int8 recovers only about 2 points for 60% more size (and the 2-point gap is within n=100 noise), so int4 is shipped as the size/quality trade-off for on-device use.

Galaxy S26 — GPU backend

The published bundle runs on the Android GPU backend: LiteRT takes the whole graph and the model generates.

file	GPU backend	delegation	peak
OLMo-2-1B-Instruct_q4_block32_ekv4096.litertlm	runs	1375 / 1375 ops across 2 subgraphs on LiteRT GPU	931 MB

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-24.

No speed rows, on purpose. On this handset the GPU backend wins prefill and does not win decode, so a GPU throughput figure only means something beside a CPU row from the same handset, and no S26 CPU row exists for this model yet.

GPU wiring, including the Gallery import toggle: GPU guide.

Conversion

Converted with the official upstream litert-torch export_hf (clean git worktree at upstream/main, dev-fork patches excluded). Olmo2ForCausalLM rides the stock converter with no custom code: QK-norm and OLMo-2's reordered post-norm lower to generic ops. The int4 recipe is blockwise (block 32) + OCTAV with the embedding at int8. At 0.93 GB the whole bundle is a single sub-2 GiB section, so it memory-maps on iOS without needing an externalized embedder.

Training data & PII

This is a weights-exact format conversion of allenai/OLMo-2-0425-1B-Instruct; no new training was performed. OLMo-2 is fully open: pretrained on AllenAI's public Dolma-family corpus (web text, code, academic papers, encyclopedic and Q&A data) and post-trained with the public Tülu 3 SFT + preference mix. Because the pretraining corpus is web-derived it may incidentally contain names and other PII; none was deliberately collected and this format conversion adds none. Apply your own content/PII filtering before deployment. See the base model card for full dataset and safety details.

2026-08-30 — tokenizer section replaced (weights unchanged)

The tokenizer.json embedded in OLMo-2-1B-Instruct_q4_block32_ekv4096.litertlm carried the GPT-2 default pre-tokenizer instead of the model's own regex, so digit groups and punctuation followed by a newline were split differently from the upstream tokenizer on every turn (the role markers <|user|>\n / <|assistant|>\n alone differed by 3 tokens per turn). OLMo-2-1B-Instruct_q4_block32_ekv4096.litertlm now embeds the upstream tokenizer.json byte for byte.

Tokenizer-only change: every section of the bundle except the tokenizer is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the chat template are unchanged and the speed and memory numbers on this card still describe this file per token — only the file's own sha256 differs. Verified on the LiteRT-LM runtime: the default turn, 7 probe strings, the 223 standalone characters U+00A1–U+017F and every special token now tokenize identically to the upstream tokenizer, and the four ASCII-only test questions answer byte-identically to the previous file (same ids in, same tokens out). Prompts containing accented letters, symbols or emoji reach the model differently from before, so individual answers to such prompts can change. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file, and any on-device rows were measured on the previous file too — the on-device gate has not been re-run on this one (the runtime's tokenizer code is the same on macOS and on device; the weights and graph are byte-identical). If you downloaded before 2026-08-30, re-download.

License and changes

Apache-2.0, inherited from the base model allenai/OLMo-2-0425-1B-Instruct. Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and prompt template repackaged unmodified into the .litertlm bundle. This repository is a community conversion and is not affiliated with AllenAI.