granite-4.0-micro — LiteRT-LM

ibm-granite/granite-4.0-micro converted to the LiteRT-LM (.litertlm) format for Google's LiteRT-LM runtime. Both files carry the GPU-composite graph since 2026-10-16 (Conversion notes); the weights are those of the 2026-10-13 files. Tested on litert-lm 0.18.0 on a Mac; the Pixel 8a rows ran the prebuilt Android binary named below on our own phone, the Galaxy S26 rows ran the same binary, LiteRT-LM's current Android benchmark binary, on Google's Developer Device Platform.

IBM describes Granite-4.0-Micro as a 3B-parameter long-context instruct model finetuned from Granite-4.0-Micro-Base, with improved instruction following and tool calling, for assistants and tool-using agents. It is a dense decoder-only transformer (GQA, RoPE, SwiGLU, RMSNorm): 3,402,836,480 parameters, 40 layers, hidden size 2560, 40 attention heads with 8 KV heads, a 100,352-token vocabulary with tied embeddings, a 128K context (4096 in these files); IBM lists twelve languages, from English to Chinese. The checkpoint is published under the hybrid-MoE class with 40 attention layers and no experts; it was rewritten to the plain Granite class before export, weights unchanged (Conversion notes).

File	Recipe	Size	sha256
granite-4.0-micro_int8.litertlm	int8 dynamic on linears, float activations; the embedding table in its own section; GPU-composite graph	3.77 GB	04ee6ad9e905448f02fa58ad3757cd3a32a773ce7b908fafb73cf8d9b5d84df5
granite-4.0-micro_int4.litertlm	int4 blockwise-32 + OCTAV on linears, int8 embedding; the embedding table in its own section; GPU-composite graph	2.20 GB	b931889ac060fb563f5ae20b3f4ebc772297f15f77cdb9b0f55b1b25a160b325

On GSM8K (n=100) bf16 scored 86, the int8 file 84 on the CPU and 90 on the GPU, the int4 file 84 on the GPU (table in Accuracy); the int4 file is 41.7 % smaller. Time for a 1024-token prefill plus 256 decoded tokens at context 2048, from the rates in Performance:

Seconds	Mac M4 Max	Galaxy S26	Pixel 8a
int4 GPU	3.04	16.70	47.92
int8 GPU	3.61	21.83	no row (Performance)
int8 CPU	17.30	42.25	157.45
int4 CPU	20.84	46.03	167.05

So int4 is the faster file on every GPU, and int8 is the faster file on every CPU. The composite graph helps two of the three GPUs: against the 2026-10-13 files, the same workload on the Mac GPU takes 17 % less time (int4) and 16 % less (int8), on the Galaxy S26 GPU 13 % less (int4, two holds on our own phone, both orders) — with a longer time to the first token there (Performance) — and on the Pixel 8a GPU about the same (two same-hold pairs of the two int4 files, one in each order, within 2 % on their mean). Not measured on iOS: the int8 file's main weights section is over 2 GiB; the embedding table sits in a separate 265,343,328-byte section, as in the granite-4.1-3b conversion, whose card explains the iOS reason.

Correctness

8-question check on the Mac M4 Max (the questions with "Answer briefly." appended, --temperature 0, one litert-lm run process each): int8 8/8 on CPU and GPU, int4 8/8 on CPU and GPU, bf16 (transformers) 8/8; the texts are those of the 2026-10-13 files. Three-turn conversations on CPU and GPU, and the token count of the first prompt against transformers (37 / 32 / 37 tokens, no start token), passed for both files.

All 7 decoder signatures of both files were fully delegated on the Mac GPU with no validation error (each 4-op embedding signature runs on the CPU, one op through XNNPACK); the Pixel 8a GPU legs below ran every decoder graph on the OpenCL accelerator, and the Galaxy S26 GPU sessions loaded the OpenCL accelerator and initialized all 7 decoder graphs from their serialized programs. On the Pixel 8a, both files answered "What is the capital of France? Answer in one sentence." on the CPU and the int4 file on the GPU: "The capital of France is Paris."; on the Galaxy S26 the int4 file gave the same answer on the GPU in the prompt process of each of its timing legs on our own phone, and on the platform's Galaxy S26 one prompt session per file and backend (the same binary, two processes each) answered it with "The capital of France is Paris." for both files on the CPU and on the GPU.

Accuracy

GSM8K: test questions 0–99 with a step-by-step instruction and a final #### <number> line, the bundle's own template, greedy, at most 1024 output tokens, a fresh process per question (litert-lm 0.18.0 Python API); bf16 in transformers on MPS with the same prompts. IFEval-lite was not run for this model. Scores are comparable only within this table, not with IBM's 85.45 (8-shot, a different protocol).

Configuration	GSM8K (n=100)
bf16 reference (transformers, MPS)	86
int8, litert-lm CPU	84
int8, litert-lm Mac GPU	90
int4, litert-lm Mac GPU	84

The 2026-10-13 files scored 83 / 90 / 84 on the same questions; on the GPU the int8 file got the same 90 questions right as before (0 discordant), the int4 file the same 84. The files run the GPU with the runtime's default fp16 activations: the 2026-10-13 int4 file scored the same (84) with fp32 activations declared, and the int8 file's GPU score is not below its CPU score.

Usage

litert-lm run ./granite-4.0-micro_int8.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17."
litert-lm run ./granite-4.0-micro_int8.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17." --backend gpu --cache no
litert-lm run ./granite-4.0-micro_int4.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17."
litert-lm run ./granite-4.0-micro_int4.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17." --backend gpu --cache no
litert-lm benchmark ./granite-4.0-micro_int8.litertlm -p 1024 -d 256 --max-num-tokens 2048 --runs 3 --cache no --backend cpu

All four run commands answered with the two dates ("The dates extracted from the text are May 3 and May 17."; the int4 file on the CPU: "The dates mentioned in the text are May 3 and May 17."), with no chat marker and no thought block. Without --backend, run uses the CPU; without --cache, it writes a compiled-weights cache next to the model (3.42 GB for int8, 1.92 GB for int4). The bundle sets no sampler; run passes one only when you give --top-k, --top-p, --temperature or --seed. For Android, see android-gpu.md.

The chat template is IBM's chat_template.jinja byte for byte, with its default system message ("You are a helpful assistant. Please ensure responses are professional, accurate, and safe."), replaced when you pass one; no start token (the prompt begins with <|start_of_role|>), the only stop token <|end_of_text|> (id 100257), no thinking declared. The runtime renders nine of ten conversation shapes exactly as transformers does. The exception: a tool result passed as a Python-API content part renders empty and the model invented a weather report; passed as the tool message's text it was used ("22 degrees Celsius and sunny" in both files).

Tool calling: with a get_weather tool both files wrote <tool_call>{"name": "get_weather", "arguments": {"city": "Paris"}}</tool_call> as text, which the app has to parse (the bundle's generic model type has no tool-call parser), and IBM's documents turn renders for documents passed as context. Engine reuse (int8 file, Python API, CPU, greedy): a second conversation on an Engine that had served another gave the same replies as on a fresh Engine, including with the shared-prefix condition of LiteRT-LM #3165 (3 of 3 runs); this model keeps only a KV cache, no recurrent state. The template, tool-call and engine-reuse probes ran on the 2026-10-13 files; the re-shipped files carry the same metadata sections (template included) and tokenizer section byte for byte.

Performance

Mac Studio M4 Max, litert-lm 0.18.0: the benchmark command above with --backend cpu or gpu (1024-token prefill, 256-token decode, context 2048), one warmup, then the mean of 3 iterations, two passes per file in alternating order with the 2026-10-13 file in the same windows, the median of the two; GPU is WebGPU on Metal. Init is the first iteration's Init time as the CLI prints it; that value includes both Init Total and the phases inside it, so it counts engine creation twice (LiteRT-LM #4057).

Mac M4 Max	Prefill 1024 (tok/s)	Decode 256 (tok/s)	TTFT (s)	Init (s)
int8 GPU	1380.94	89.33	0.753	3.41
int4 GPU	1391.10	111.25	0.745	4.47
int8 CPU	192.69	21.36	5.362	25.44
int4 CPU	102.42	23.61	10.051	6.93

Galaxy S26 (the platform's device m1q-36, Android 16; the farm's logs do not name the SoC) on Google's Developer Device Platform: one session per row through the litert CLI (litert benchmark <file> --ddp --device m1q-36 --cpu|--gpu --prefill-tokens 1024 --decode-tokens 256 --max-num-tokens 2048 --num-iterations 3, 2026-10-16), which runs LiteRT-LM's current Android litert_lm_advanced_main from the project's public binary bucket (binaries/latest/android_arm64/litert_lm/): a warm-up process writes the compiled-program caches, then the measured process runs three prefill-and-decode cycles. Rates and TTFT are the median of the three cycles; Init cold = the warm-up process's engine init, warm = the measured process's; peak = the measured process's peak RSS. The farm's phone is allocated per session, so these rows are not same-hold pairs; our own Galaxy S26 (SM-S942Q) measured the int4 file's GPU row within 3 % of these numbers (two holds, both orders, the same binary).

Galaxy S26	Prefill 1024 (tok/s)	Decode 256 (tok/s)	TTFT (s)	Init cold / warm (s)	Peak (MB)
int8 GPU	225.8	14.80	4.60	12.57 / 6.25	969
int4 GPU	232.7	20.82	4.45	15.38 / 11.53	1382
int8 CPU	88.1	8.36	11.74	6.59 / 0.84	5649
int4 CPU	47.3	10.50	21.74	6.30 / 8.16	4336

On the Adreno GPU the composite graph trades prefill for decode: against the 2026-10-13 int4 file in the same two holds on our own phone (prefill 335.6 → 235.6 tok/s, decode 15.7 → 20.2, TTFT 3.19 → 4.41 s), decode rose by about 30 % and prefill fell by about 30 %, so the 1024 + 256 workload takes 13 % less time while the first token arrives about 1.2 s later. The CPU rates fell inside the sessions (the 2026-10-13 int4 file's three cycles ran 54, 48 and 47 tok/s of prefill), as on our own phone, which lowers its clocks within seconds of a sustained CPU run (the farm's logs carry no clock reading); the rows are the sustained values.

Pixel 8a (Tensor G3, Mali-G715, MemTotal 7.4 GiB, Android 16), our own phone, the same workload through the prebuilt litert_lm_advanced_main (sha256 adac974b…, 40,382,152 bytes) published 2026-09-18 in LiteRT-LM's public binary bucket, with --benchmark=true --benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256 --max_num_tokens=2048 --num_iterations=3 --report_peak_memory_footprint=true, one leg per hold on a phone rested for at least 300 s. Rates are the mean of iterations 2 and 3 (GPU: LITERT_CL); Init cold = the first process after the compiled-program caches were removed, warm = the benchmark process; peak = the largest of the benchmark process's Peak system ram usage lines.

Pixel 8a	Prefill 1024 (tok/s)	Decode 256 (tok/s)	TTFT (s)	Init cold / warm (s)	Peak (MB)
int8 GPU	—	—	—	—	—
int4 GPU	91.5	6.97	11.34	25.28 / 14.55	2210.2
int8 CPU	18.2	2.53	56.81	12.99 / 1.26	4983.0
int4 CPU	12.2	3.08	84.43	10.94 / 7.98	3658.7

The int8 file has no Pixel 8a GPU row: the harness stops a process when the phone's MemAvailable falls below 400,000 kB, and it did so in every process of the 2026-10-13 int8 file's one try (307,820 kB in the first process, 381,800 kB in the benchmark, 106,656 kB in the prompt process); the re-shipped file has the same weights and nearly the same size (884,736 bytes more), and the try was not repeated. On the Mali GPU the composite graph gives no gain: two same-hold pairs of the two int4 files, one in each order, were within 2 % on their mean, and this row (a different day and hold) reads 15 % slower than the 2026-10-13 file's. The int8 CPU row's benchmark process started with the clocks uncapped, unlike the 2026-10-13 rows and the int4 CPU row; the sampler reads only Adreno GPU clocks, so no Pixel 8a GPU clock was recorded. Not run on the NPU.

Conversion notes

• Source and stack: ibm-granite/granite-4.0-micro at revision 56111ae1 (every file checked against the Hub's record); litert-torch 0.9.4, litert-converter 0.4.0, ai-edge-quantizer 0.9.0, transformers 5.14.1, torch 2.13.0; packed by the litert-lm 0.18.0 CLI. granite_work/convert_granite40_micro.sh in hf-to-litertlm rebuilt the 2026-10-13 files, all five sections byte-identical (only the packer's uuid and timestamp differ).
• GPU-composite graph (2026-10-16): the two files were rebuilt from those files without re-exporting, by tools/gpu_graph/gpu_graph_retrofit.py --decomp exporter --mask add_bcast in the same repository: the two KV-cache writes per layer become one odml.cache_update composite, each attention matmul an odml.runtime_bmm composite, and a param_tensor input is added to every signature — the shape litert-torch's --apply_gpu_composites exports. All 281 weight tensors, the metadata, tokenizer and embedder sections are byte-identical to the 2026-10-13 files; the main graph section grew by 884,736 bytes (int4). The attention mask stays a float input added by broadcast: the bool-mask form the tool also offers ran the Mac GPU 3 % and the Galaxy S26 GPU 10 % faster still, but cost the CPU 8 % (Mac) to 9–23 % (Galaxy S26) against the 2026-10-13 int4 file and 1.2 GB of RSS, so it was not shipped.
• Dense rewrite: the checkpoint's class is GraniteMoeHybrid with 40 attention layers and 0 experts, which transformers 5.14.1 cannot run through generate(). The recipe copies it into the GraniteForCausalLM layout (each layer's shared_mlp.input_linear rows 0–8191 become gate_proj, rows 8192–16383 up_proj, output_linear becomes down_proj; the other 242 tensors unchanged) and proves it on three prompts of 37, 77 and 839 tokens in fp32: max |logit delta| 1.9e-4, 3.3e-4 and 9.1e-4, the same argmax at every position, the same 50 greedy tokens; in float64 the delta is 0.0.
• Direct export: exporting the original class also worked (a fused gate|up matmul plus two slices, 3,771,027,808 bytes) and ran within 3 % of the rewrite on the Mac; the rewrite shipped.
• Order: export of the rewritten checkpoint (6 prefill signatures, 1024 down to 1, plus decode, over a 4096-position KV cache; the embedding table as its own section; no start token; IBM's template embedded) with the int8 recipe or the int4 recipe → int4 block-scale scan (281 blockwise tensors, none zero or subnormal) → metadata edit (stop token id 100257 only, max_num_tokens 4096, no thinking or function-calling flag; the exporter's 14 string stops dropped) with every other section proven unchanged → the GPU-composite retrofit.

License and changes

Distributed under the Apache License 2.0 inherited from the base model (the model repo carries the license in its README metadata and no separate LICENSE file). Changes from the original work: rewritten from the GraniteMoeHybrid class to the GraniteForCausalLM class with the weights unchanged, exported to the LiteRT-LM format and quantized to int8 and int4 as described above, and the prefill and decode graphs rewritten into LiteRT-LM's GPU-composite form (2026-10-16, weights unchanged); chat template IBM's own, stop token reduced to <|end_of_text|>, max_num_tokens 4096. This conversion was made and is published independently of IBM.