Olmo-3-7B-Think — LiteRT-LM

allenai/Olmo-3-7B-Think converted to the LiteRT-LM (.litertlm) format for Google's LiteRT-LM runtime: an int8 file, an int4 file and, for phones, the int4 weights with a 2,048-token cache, built with the conversion script in hf-to-litertlm (olmo3_work/). Converted with litert-torch 0.9.4 and tested on litert-lm 0.18.0 on a Mac Studio M4 Max, where the int4 file decodes at 34.8 tok/s on the GPU (measured 2026-10-16); on a Galaxy S26 of the 12 GB class (a platform device-lab unit) the 2,048-token-cache file built its GPU engine and decoded at 7.0 tok/s while the 4,096-token-cache files did not build (Performance). On litert-lm 0.17.1 both files answered a one-word question on the Mac CPU ("Paris", one run each, the thought printed first).

Ai2 describes Olmo 3 as "a new family of 7B and 32B models both Instruct and Think variants. Long chain-of-thought thinking improves reasoning tasks like math and coding", pre-trained on the Dolma 3 dataset and post-trained on the Dolci datasets, with the whole model flow released. Olmo-3-7B-Think is the final RLVR model of the 7B Think line (base Olmo-3-1025-7B, then Think-SFT, then Think-DPO, then this checkpoint), developed by the Allen Institute for AI, English, date cutoff December 2024, described in the Olmo 3 paper (arXiv 2512.13961). Ai2's table reports, with its settings, MATH 95.1, AIME 2024 71.6, AIME 2025 64.6, BBH 86.6, HumanEval+ 89.9, LiveCodeBench v3 75.2, IFEval 88.2, MMLU 77.8 and GPQA 46.2, and recommends temperature 0.6, top_p 0.95 and max_tokens 32,768 for generation; the context is 65,536 positions (4,096 in these files).

The architecture is a dense transformer: 32 layers, 24 with sliding-window attention (window 4,096) and 8 with full attention (every fourth layer), hidden size 4096, intermediate size 11008, 32 attention heads and 32 KV heads of 128 (multi-head attention, no grouping), QK-norm over the full projection width, a post-norm block, RoPE theta 500,000 with YaRN (factor 8 over 8,192 original positions) on the full-attention layers, a 100,278-token vocabulary with untied embeddings; 7,298,011,136 parameters. In these files every layer keeps a full-length KV cache of 4,096 positions, which equals the sliding window, so the sliding layers see every earlier token, as the source model does for any conversation of up to 4,096 tokens.

Olmo-3-7B-Think is a reasoning model that always thinks: Ai2's template ends the prompt with <|im_start|>assistant and an opened <think>, reads no enable_thinking variable, and the model closes the thought with </think> before its answer. The bundle declares the thinking channel (thought, <think> / </think>), so the runtime returns the reasoning separately (the CLI prints it as [thought] … [/thought]) and the answer text carries the answer only. Thinking cannot be switched off: the CLI's --thinking false and --thinking-budget 0 leave the output byte for byte the same (Usage), and a positive budget closes the channel at the budget while the model goes on reasoning in the visible text. The reasoning runs long: on GSM8K the mean reply was about 1,600 tokens and a third of the bf16 model's replies used the whole 2,048-token budget of our runs, against Ai2's max_tokens of 32,768; prompt, thought and answer share the 4,096-token context of these files.

File	Recipe	Size	sha256
Olmo-3-7B-Think_int8.litertlm	int8 dynamic on every linear layer, int8 embedding table; activations float (the runtime's default fp16 on the GPU)	7.43 GB (7,434,672,480 bytes)	e075c5c92582fa1446723003225239b4dd131a07195ca49e4a7117388641bfbb
Olmo-3-7B-Think_int4.litertlm	int4 blockwise-32 (OCTAV) on every linear layer (block scales floored at 2^-14, Conversion notes), int8 embedding table; activations float	4.31 GB (4,309,457,248 bytes)	133ef829dcb98c1ad28f700cd3cd5f18e5fb60ae9bfb6f4efe52d60ef21969be
Olmo-3-7B-Think_int4_c2048.litertlm	the int4 file with a 2,048-token KV cache (the same weights and recipe; contexts of up to 2,048 tokens): the form that runs on a 12 GB Galaxy S26	4.31 GB (4,309,440,864 bytes)	47faae994b95e1ff7ab3b28a86cefcb8cec90cd96f987c7f865c7ace2f5eb0cc

On GSM8K with a 2,048-token output budget (answers scored after </think>, table in Accuracy) the int8 file scored 81 and the int4 file 80 on the Mac GPU, against 84 for the bf16 model in transformers on the same questions and budget. The int4 file is 42 % smaller. Time for a 1024-token prefill plus 256 decoded tokens at context 2048, from the rates in Performance:

Seconds	Mac M4 Max	Galaxy S26	Pixel 8a
int4 GPU	8.59	GPU engine does not build (Performance)	not run (Performance)
int4, 2,048-token cache, GPU	6.58	41.32 (platform device-lab unit, Performance)	not run (Performance)
int8 GPU	9.64	not run (Performance)	not run (Performance)
int8 CPU	46.04	not run (Performance)	not run (Performance)
int4 CPU	51.97	not run (Performance)	not run (Performance)

On the Mac the int4 file is the faster file on the GPU (10.9 % less time, decode 34.8 vs 30.4 tok/s at the same prefill rate) and the int8 file on the CPU (the int4 file's blockwise prefill runs at 64 tok/s against 131). The third file, the int4 weights with a 2,048-token cache, took 6.58 s on the Mac GPU (decode 46.6 tok/s), serves conversations of up to 2,048 tokens only, and is the form that runs on a Galaxy S26 (Performance). Not measured on iOS: the decoder graph of each file is one TFLite section (6.53 GiB int8, 3.62 GiB int4; the embedding table is a 400 MiB section of its own), over the ~2 GiB single-section limit the Ministral-3-3B-Instruct-2512 card describes.

Correctness

8-question check on the Mac M4 Max (the questions with "Answer briefly." appended, one fresh engine each, greedy, up to 2,048 output tokens, answers scored after </think>): int8 8/8 on CPU and GPU (every thought closed, 226–848 decode tokens on the GPU), int4 8/8 on the GPU (302–1,542 tokens) and 7/8 on the CPU, where the rhyme question's thought was still open at the 2,048-token cap (no line repeated, top 8-gram share 0.015) and closed at 2,125 tokens with a 4,096-token cap, answering blue; the bf16 model in transformers 8/8. On the same file the CPU path reasoned longer than the GPU path on that question (2,125 vs 1,542 decode tokens); the cause was not measured on this model.

All 7 signatures of both files are fully delegated on the Mac GPU (1436 / 1436 / 1436 / 1436 / 1436 / 1307 / 1336 nodes, one partition each, no validation error); the embedding lookup runs on the CPU. Every GPU run logs TopK requires src tensor C dimension to be divisible by 4.: the vocabulary has 100,278 entries, so the GPU sampler's top-k reads the first 100,276 logits and the last two ids (<|endofprompt|>, <|pad|>) can never be sampled on the GPU; neither is a stop token or a token a chat reply writes. No start token: the engine's prefill count equals transformers' token count on three prompts (Ai2's tokenizer has no post-processor, and the template's render starts with <|im_start|>). A three-turn conversation ("My name is Kenji." / "What is my name? Answer with the name only." / "What is 17 times 23? Answer with the number only.") on the int8 file, both backends, answered Kenji and 391 with every thought closed, and the KV the engine held before each reply equalled the token count of transformers' render of the conversation with each earlier reply as its answer text (the runtime keeps earlier thoughts out of the history, as Ai2's template does); on the int4 file the counts matched the same way, the GPU answered 391 after 1,969 tokens, and on the CPU the third turn's thought closed but the answer was cut at the 2,048-token cap. A prompt-length check (17 prompt lengths between 44 and 60 tokens, a fresh engine each, both files, CPU and GPU): the prefill count equal to transformers' at every length and no garbage start; the expected word in 67 of 68 replies, the one miss still thinking at that check's 512-token cap (int4, GPU).

Multi-step arithmetic on the Mac CPU (six two-digit multiplications, three fresh processes each, 2,048-token cap): int8 right in 18 of 18 runs, every run using the whole budget and 9 closing the thought inside it (12 × 34 with a 4,096-token cap closed by itself at 2,205 tokens with 408); int4 right in 15 of 18 (3 closed; the three misses are 9 × 87, cut before the number); no run looped. Long-output check (four prompts, 2,048-token cap, Mac CPU): int8 closed 3 of 4 (the refrigerator explanation reached the cap), int4 1 of 4; no line repeated in any of the eight. Engine reuse (two conversations, a pirate persona and then one told to answer in one word, three questions each, on one Engine and on a fresh Engine; int8, CPU, greedy): the same six answers and the same decode counts on both.

Accuracy

GSM8K: test questions 0–99 with a step-by-step instruction and a final #### <number> line, the bundle's template (its default system turn, the opened <think>), greedy, at most 2,048 output tokens, a 4096-token KV budget, a fresh process per question (litert-lm 0.18.0 Python API, Mac GPU). Scoring rule for a reasoner: a reply that closed its thought is scored on the text after </think>; a reply that never closed it is scored on its whole text and counted as unclosed. The bf16 reference is transformers 5.14.1 on MPS with Ai2's template, the same prompts, greedy, the same budget and the same scoring. IFEval was not run; Ai2's table reports IFEval 88.2 with its settings and has no GSM8K row.

Configuration	GSM8K	unclosed thoughts	replies using the whole budget
bf16 reference (transformers, MPS)	84 / 100	26	35
int8, litert-lm Mac GPU	81 / 100	32	37
int4, litert-lm Mac GPU	80 / 100	37	44
int4, 2,048-token cache (the third file), 1,900-token budget	77 / 100	47	12 (37 more ended at the 2,048-token context)

On the closed replies alone: bf16 68 of 74, int8 64 of 68, int4 57 of 63. Both files pass our bars (int8 within 5 points of bf16, int4 within 8). The gaps sit inside the spread of greedy runs: paired against bf16 on the same questions the int8 file is right on 9 that bf16 misses and misses 12 that bf16 gets, the int4 file 6 and 10; the 2,048-token-cache file and the int4 file (the same weights) differ on 21 questions and give the same final number on 71. The mean reply was 1,595 tokens in bf16, 1,582 for int8 and 1,684 for int4.

Usage

litert-lm run ./Olmo-3-7B-Think_int8.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17."
litert-lm run ./Olmo-3-7B-Think_int8.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17." --backend gpu --cache no
litert-lm run ./Olmo-3-7B-Think_int4.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17." --backend gpu --cache no
litert-lm run ./Olmo-3-7B-Think_int4.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17."
litert-lm benchmark ./Olmo-3-7B-Think_int8.litertlm -p 1024 -d 256 --max-num-tokens 2048 --runs 3 --cache no --backend cpu
litert-lm run ./Olmo-3-7B-Think_int8.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17." --backend gpu --cache no --thinking-budget 64

The four plain run commands printed a [thought] … [/thought] block of 3,000–4,600 characters and then the dates ("The dates extracted from the sentence are: May 3 and May 17." and a list form of it), with no control token in the output, in 38–45 s on the GPU and 128–154 s on the CPU including engine init. The last command caps the thought at 64 tokens: the channel closed mid-sentence after 286 characters, the model carried on its reasoning in the visible reply (3,385 characters), wrote </think> as text and then gave the same two dates; --thinking false and --thinking-budget 0 printed the same output as the second command, byte for byte. Without --backend, run uses the CPU; without --cache, it writes a compiled-weights cache next to the model on its first CPU run (6.90 GB for int8, 3.89 GB for int4). The 0.18.0 CLI has no output-length flag and no system-message flag (a system message can come from a --preset file); a reply ends at a stop token or when the KV budget is full (4096 in these bundles, --max-num-tokens to change it), so a long thought can reach the end of the context before the answer. The third file runs with the same commands and holds a 2,048-token KV budget. The bundle sets no sampler; run passes one only when you give --top-k, --top-p, --temperature or --seed (Ai2 recommends temperature 0.6 and top_p 0.95; every number on this card is greedy). For Android, see android-gpu.md.

The chat template is Ai2's chat_template.jinja with three mechanical edits for the runtime's template engine: a content-parts macro, every message content read through it (content parts as well as strings), and message.get('x', none) is not none written as is defined tests (the engine has no .get); everything else is verbatim, including Ai2's default system message ("You are OLMo, a helpful function-calling AI assistant built by Ai2. Your date cutoff is November 2024, …", written when you pass no system message; the README's "Default System Message" section shows a different text, and the bundles follow the template file, as transformers does). A system message you pass replaces it, and the template appends "You do not currently have access to any functions. " to it unless the message carries a functions key. Its renders equal transformers' render of Ai2's template in 9 conversation shapes × thinking unset / false / true × both history forms (<think>…</think> inline and the runtime's reply form), as strings and as content parts. <think> and </think> are plain text in this vocabulary (three tokens each), not special tokens. No start token; stop tokens <|im_end|> (id 100265) and <|endoftext|> (id 100257) from Ai2's generation config. The tokenizer section is the repo's tokenizer.json byte for byte.

Tool calling: Ai2's template takes the function list as the system message's functions key (rendered inside <functions>…</functions>) and a result as an environment turn; the runtime's tools= argument is not rendered (the template has no tools branch), and the bundle's generic model type has no tool-call parser. With a get_weather tool in Ai2's form the int8 file (CPU) answered in text that it has no real-time access and how the function would be called; the int4 file (GPU) announced "I'll use the get_weather function with "Paris" as the location" and then wrote a made-up result (22 °C, sunny) as if the call had happened, with no <function_calls> block and no call parsed; the same file without the function list answered that it has no real-time access. A result handed back in the environment turn got no answer inside 2,048 tokens (the thought kept weighing whether the block belonged to the question). An app that needs tool calls on this model parses the text itself and should expect both behaviours.

Performance

Mac Studio M4 Max, litert-lm 0.18.0: the benchmark command above with --backend cpu or gpu (1024-token prefill, 256-token decode, context 2048), one warmup, then the mean of 3 iterations, two passes per backend (each pass one held measurement window with the files in opposite orders, no other heavy job on the machine), the median of the two; GPU is WebGPU on Metal. Init is the CLI's printed init time of the first iteration; in litert-lm 0.18.0 it sums overlapping init phases and reads about twice the engine build measured from the log's timestamps (on two other files converted the same way: a 9B Mixed INT4 GPU run printed 89.2 s for a 44.5 s build, a granite-4.1-8b int4 GPU run 7.38 s for 3.66 s); the Python API built this model's GPU engines in 3.3 s (int8) and 4.8 s (int4), medians of 100 fresh processes.

Mac M4 Max	Prefill 1024 (tok/s)	Decode 256 (tok/s)	TTFT (s)	Init (s)
int8 GPU	838.81	30.40	1.25	4.58
int4 GPU	836.55	34.76	1.25	6.00
int8 CPU	131.36	6.70	7.96	50.62
int4 CPU	64.28	7.11	16.07	13.59

Process memory during the CPU benchmark (--cache no): int8 50.5 GB, int4 31.1 GB; a GPU run up to 12.9 GB. One variant was measured and is not shipped: the int8 weights with fp32 GPU activations declared took 41 % more time (13.60 vs 9.64 s; prefill 648 tok/s, decode 21.3) and gave the same eight check answers, so the files keep the runtime's fp16 default. The third file, the int4 weights with a 2,048-token cache, took 23 % less time than the int4 file (6.58 vs 8.59 s; prefill 942 tok/s, decode 46.6, init 6.13 s) at a context of 2,048 tokens and scored 77 on GSM8K at a 1,900-token budget (Accuracy).

Phones (Android 16, the prebuilt litert_lm_advanced_main of 2026-09-18 from LiteRT-LM's public binary bucket, sha256 adac974b…):

• Galaxy S26 (SM-S942Q, Qualcomm SM8850, Adreno 840, MemTotal 11,389,756 kB): on a platform device-lab unit of the same model line (SM-S942U1, Android 16, MemTotal 11,389,764 kB; 2026-10-16) the 2,048-token-cache file built its GPU engine and ran the benchmark workload in 41.3 s (prefill 211.8 tok/s, decode 7.03 tok/s, time to first token 5.03 s, init 23.7 s and 28.5 s cold; the median of two measured cycles after a warm-up; peak 2.33 GB RSS; all 7 decoder signatures on the OpenCL delegate, the embedding lookup on the CPU), and in a second session answered the prompt check after a closed thought. The unit started with its GPU and big CPU cluster clock-capped (the GPU at 902 MHz, the big cluster at 3.40 of 4.74 GHz) and the GPU clock fell further over the cycles, so these are the numbers of a platform session, not a headline for a phone in hand. The 4,096-token-cache int4 file did not build on that unit: its warm-up engine build delegated all 7 signatures, then free memory fell from 2.59 GB to 0.67 GB in 11 s while the low-memory killer removed 95 apps, and the system server restarted. The int8 file was not started on the phone's GPU: after a sibling 8B-class int8 file restarted this phone during its GPU engine warm-up on 2026-10-14, no 8B-class int8 engine is started on it. The CPU was not run: by the earlier granite-4.0-micro measurements on the same phone (a CPU peak of 1.5 × the int8 file and 2.0 × the int4 file) these files would need 11.2 GB and 8.6 GB against the phone's 11.66 GB, and a 7.9 GB LiteRT-LM process has already restarted this phone's framework once.
• Pixel 8a (Tensor G3, Mali-G715, MemTotal 7,754,700 kB): not run, and GPU execution on a Mali phone is not verified for these files. The int4 file (4.31 GB) plus a KV cache of 1–2 GiB is more than the phone's free memory at the start of a session (3.9–4.5 GB of 7.75 GB): the two 7–9B int4 GPU tries on this phone, granite-4.1-8b and a 4.69 GB file of a 7B sibling, were stopped by our memory rule (a run ends at the first MemAvailable reading under 400,000 kB; 327,604 and 307,456 kB, from 4,112,928 and 3,908,916 kB at the start) after full delegation and with no output. The int8 file alone is 94 % of the phone's RAM.
• Not run on an NPU: these dynamic-range int8 and blockwise int4 graphs have no HTP path (the NPU path takes static-range or float graphs).

Conversion notes

• Source and stack: allenai/Olmo-3-7B-Think at revision d97e442d (the main branch as of 2026-10-16; every file checked against the Hub's record); litert-torch 0.9.4, litert-converter 0.4.0, ai-edge-quantizer 0.9.0, transformers 5.14.1, torch 2.13.0 (released wheels, no patch); packed and checked with litert-lm 0.18.0. olmo3_work/convert_olmo3_7b_think.sh in hf-to-litertlm is the one command; the conversion run behind these files (the same steps, with the local paths and memory gates of a shared machine) was rebuilt from the pinned checkpoint on 2026-10-16: the int8 file and the 2,048-token-cache file came back with all five sections byte for byte, the int4 file with four of five and a decoder section of the same size that differs in 17,917 bytes (the two RoPE inverse-frequency constants stored in the other order and the operator order of the seven signature subgraphs; a structural comparison that ignores tensor indices and names finds the seven signatures equal, and the 879 decomposition subgraphs are identical). Compare sections, not files: the packer writes a new uuid and timestamp each run. Run as published on 2026-10-16 (one command, the environment of its header, 34 minutes on a 128 GB Mac, peak 55.9 GB), the script rebuilt the 2,048-token-cache file with all five sections byte for byte and the int8 and int4 files with four of five, their decoder sections the same size and equal by the same structural comparison: an export writes the two RoPE constants and the signature subgraphs in either of two orders, and this run produced the other order for both (the int4 one bit for bit as the 2026-10-16 recipe re-run had).
• Export: Olmo3ForCausalLM through litert-torch's stock export_hf path, no custom model code: 6 prefill signatures (1024, 256, 64, 16, 4, 1) plus decode over a 4096-position KV budget (2048 in the third file), 64 cache buffers (a key and a value per layer, [1, 32, 4096, 128]), the sliding-window layers' mask built over the full-length cache, YaRN cos / sin on the full-attention layers; the lightweight conversion (constants of 1e8 bytes and more stay lazy) with the embedding table in a TFLite section of its own; each export peaked at 52.6–56.9 GB on a 128 GiB Mac, one at a time. The GPU-composites graph rewrite that gave two dense 8B files of the same converter (granite-4.1-8b, Ministral-3-8B-Instruct-2512) a 13–17 % faster Mac GPU graph refuses this graph (the attention-mask input also feeds the sliding-window mask construction), so the published files are the exporter's graphs.
• Quantization after the export: int8 = dynamic int8 on every FULLY_CONNECTED (219 per prefill signature, 225 in decode) plus an int8 embedding table; int4 = int4 blockwise-32 (OCTAV) on every FULLY_CONNECTED, the embedding table int8. The plain int4 build wrote 1,574,919 zero and 2,031,784 subnormal fp16 block scales (of 215,210,752; most in the late layers' q_proj), and XNNPACK prepares a blockwise int4 layer only when every scale is a normal float, so the published int4 file floors every block scale at 2^-14 (float16's smallest normal) before the weights are rounded: 3,606,703 scales in 55 tensors, and a scan of the result finds every scale normal (3,611,960 at exactly 2^-14).
• Metadata: the bundle template above; no start token; stop ids 100265 and 100257 (the exporter had written 16 stop entries, the two ids and 14 string copies; the two ids are kept); the thought channel; max_num_tokens 4096 (2048 in the third file); no sampler (litert-torch 0.9.4 turns do_sample off before the packer reads the generation config); the tokenizer section replaced by the repo's tokenizer.json after the export (the exporter's re-serialization differs from the repo file).

License and changes

Apache-2.0, inherited from allenai/Olmo-3-7B-Think (its card metadata and its README: "This model is licensed under Apache 2.0. It is intended for research and educational use in accordance with Ai2's Responsible Use Guidelines"); the source repository carries no LICENSE file at the pinned revision, so this repository carries the Apache License 2.0 text as LICENSE. Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above, the int4 block scales floored at 2^-14; the chat template edited for the runtime's template engine (content parts, .get) with Ai2's default system message verbatim; the tokenizer repackaged unmodified; no start token, stop ids 100265 and 100257, a thought channel and a 4096-token KV budget (2048 in the third file) declared. This repository is a community conversion and is not affiliated with Ai2.