Spark-X2.6-1.7B — LiteRT-LM

This repository is a conversion of XHToken/Spark-X2.6-1.7B to LiteRT-LM (.litertlm), made with litert-torch 0.9.3 for LiteRT-LM 0.19.0: on a Galaxy S26 GPU the int4-mlp file decodes at 27.4 tok/s (1,024-token prompt, 256 generated tokens, 2026-10-31).

Spark-X2.6-1.7B is the SparkLLM Team's 1.71B-parameter dense decoder, the second model of the Spark-X2.6 series after Spark-X2.6-4B. Its model card says it keeps the architecture of Spark-X2.5-1.7B and gains its capability from improved post-training. That architecture has 28 layers, hidden size 2048, 8 query and 2 KV heads of 256, three sliding-window (512) layers per full-attention layer, per-head sigmoid gates on the attention output, exact-GELU gated MLPs and a 131,072-entry tied vocabulary. It is a reasoning model: it thinks inside <think>…</think> before it answers. These files carry that: the prompt template opens the thought the way the vendor's chat template does, and the files declare a thought channel, so LiteRT-LM keeps the reasoning out of the answer. Apache-2.0.

File	Recipe	Size
Spark-X2.6-1.7B_int4-mlp.litertlm	int8 attention; int4 (blockwise-32, OCTAV) MLP gate and up projections; int8 MLP down projection and output layer; int8 embedding table in its own section	1706 MB
Spark-X2.6-1.7B_int8.litertlm	int8 (dynamic, per channel) on every linear layer and the embedding	1836 MB

Both files declare fp32 GPU activations and use LiteRT-LM's GPU composite ops for the KV-cache writes and the attention products (see Conversion notes); their weights are those of the int8 / int4-mlp export, unchanged. The numbers for these files were measured with LiteRT-LM 0.19.0 (the iPhone rows with its nightly iOS build 0.19.0-20261029); earlier releases were not tested with these files.

On a phone, use Spark-X2.6-1.7B_int4-mlp.litertlm on the GPU. On the Galaxy S26 GPU it decodes 27.4 tok/s, the int8 file 25.8 tok/s; it is also 130 MB smaller and scores GSM8K 82 against the int8 file's 80 (on the iPhone 18 Pro GPU the two decode at the same speed, 51.4 and 51.5 tok/s). On a Mac the int8 file is 3 % faster on the GPU (151.5 vs 147.0 tok/s, measured back to back). Both files score within 5 points of the bf16 model on GSM8K (int4-mlp 82, int8 80, bf16 84). Use the GPU rather than the CPU: on one of the two free-form prompts we tried, the int8 file on the CPU ended its answer with a literal </think> and then wrote the answer again (see Known runtime issues).

Correctness

Where a CPU result below says "int8 (CPU)", it was measured on the int8 file before the fp32 GPU declaration was added; that declaration is the only difference between the two files.

• 8 short questions with one right answer each (two arithmetic questions, a capital, a translation, a decimal comparison, an opposite, the days of a week and a rhyme; Mac, LiteRT-LM 0.19.0 Python API, one process per question, greedy, thinking on, 2,048 output tokens): int4-mlp CPU 8/8, GPU 8/8; int8 GPU 8/8, int8 (CPU) 8/8. Every thought closed, every answer was a clean final answer.
• On phones, the same 8 questions with the same settings (the LiteRT-LM 0.19.0 Android wheel from PyPI): Galaxy S26 GPU int4-mlp 8/8, int8 8/8; Galaxy S26 int8 (CPU) 8/8. Pixel 11 Pro GPU int4-mlp 8/8, int8 8/8; Pixel 11 Pro CPU int4-mlp 8/8, int8 8/8. iPhone 18 Pro (an app on the nightly CLiteRTLM 0.19.0-20261029, not a 0.19.0 release): GPU int4-mlp 8/8, int8 8/8; CPU int4-mlp 8/8, int8 8/8. On every phone each prompt rendered to the same bytes as on the Mac.
• With reasoning_effort low (Mac GPU, 1,792 output tokens, otherwise as the Mac questions above): int8 8/8 (context 4,096); int4-mlp 7/8 at a context of 2,048 and of 4,096 tokens, both times answering the rhyme question with "purple".
• Multi-turn (3 turns, greedy: plant a fact, do arithmetic, recall the fact; the same conversation run twice per process on one engine, both times at the runtime's default, which drops each turn's thought from the KV cache before the next turn): pass on both files on the GPU, int4-mlp on the CPU (4 processes) and int8 (CPU).
• Tokenizer: the tokenizer section is the upstream tokenizer.json, byte for byte. With these bytes LiteRT-LM encodes 234 of 234 probe rows (every added token alone and mid-string, a role-marker turn, Latin-1 / Extended-A, emoji, CJK, digits, whitespace) to the same ids as the tokenizers library.
• Chat template: for system, user and assistant messages the files' template renders byte-identically to the vendor's chat_template.jinja: 54 renders (three conversation shapes × generation prompt on / off × thinking unset / on / off × reasoning_effort unset / high / low), in transformers and in LiteRT-LM 0.19.0.

Accuracy — GSM8K

GSM8K, the first 100 test questions, greedy, thinking on, a 0-shot chain-of-thought prompt that asks for the answer as #### <number>, 3,584 output tokens (the files' KV cache holds 4,096), one conversation per question. The LiteRT-LM rows ran on the Mac GPU (LiteRT-LM 0.19.0 Python API), the reference with transformers on the Mac (MPS) and the vendor chat template. The answer is read from the text after the thought, or from the thought when the model did not finish within the 3,584 tokens; the same rule for every row.

Configuration	GSM8K
PyTorch bf16 (reference)	84 (14 unfinished)
LiteRT-LM int4-mlp	82 (21 unfinished)
LiteRT-LM int8	80 (14 unfinished)

Question by question against bf16: int4-mlp 77 both right, 7 right only in bf16, 5 right only in int4-mlp; int8 78 / 6 / 2.

With reasoning_effort low and 1,792 output tokens (otherwise as above): bf16 82 (7 unfinished), int4-mlp 77 (8), int8 79 (6).

Usage

pip install litert-lm==0.19.0
litert-lm run ./Spark-X2.6-1.7B_int4-mlp.litertlm --backend gpu --thinking true --prompt "What is the capital of France? Answer in one word."

Python, with the vendor's thinking levels (reasoning_effort high is the default, low asks for a shorter thought, and thinking off renders the vendor's no-think prompt):

import litert_lm

engine = litert_lm.Engine("Spark-X2.6-1.7B_int4-mlp.litertlm", backend=litert_lm.Backend.GPU(), max_num_tokens=4096)
conversation = engine.create_conversation(extra_context={"reasoning_effort": "low"})
reply = conversation.send_message("How many prime numbers are smaller than 30?", max_output_tokens=3584)
print(reply["channels"]["thought"])
print(reply["content"][0]["text"])

• Give it a generous output budget (3,584 tokens for math). The model thinks before it answers; cut off mid-thought it gives no final answer.
• litert-lm run writes its compile caches next to the model file by default: 1.36 GB in two files beside the int4-mlp file after the GPU command above (a 1.33 GB weight cache and a 31.5 MB program cache). --cache no turns that off.
• create_conversation(thinking_config=litert_lm.ThinkingConfig(enable_thinking=False)) renders the vendor's no-think prompt (<｜start▁of▁sentence｜><|Bot|></think>).
• The template is the vendor's plain-chat part: a default system block (<|System|>, "you are a helpful assistant.", the thinking_effort instruction while thinking is on, then your system prompt), every message wrapped in <｜start▁of▁sentence｜> … <｜end▁of▁sentence｜>, generation prompt <｜start▁of▁sentence｜><|Bot|><think>; stop token <｜end▁of▁sentence｜>. Tool-call formatting is not carried. KV cache 4,096 tokens (the original's 1M context does not apply here), eleven prefill sizes (1, 2, 4, …, 1024).
• The vendor's recommended sampling is temperature 1.0 / top-p 0.95; the numbers on this card are greedy.

Performance

Every row: prefill of 1,024 tokens, then 256 decoded tokens, with the context set to 2,048 tokens; 3 runs per process, at least two processes per file, and the value is the mean over processes. LiteRT-LM 0.19.0 from PyPI on the Mac and on the Android phones.

Mac Studio (Apple M4 Max, 128 GB, macOS 27.0), litert-lm benchmark <file> -p 1024 -d 256 --max-num-tokens 2048 --runs 3 --backend <cpu|gpu> --cache no (one warm-up run, then the mean of 3); the two GPU rows were measured back to back in one session:

File	Backend	Prefill	Decode	TTFT	Init
int4-mlp	GPU (WebGPU)	3,025 tok/s	147.0 tok/s	0.35 s	3.2 s
int8	GPU (WebGPU)	3,013 tok/s	151.5 tok/s	0.35 s	2.9 s
int8	CPU	910 tok/s	40.8 tok/s	1.15 s	20.7 s
int4-mlp	CPU	415 tok/s	38.0 tok/s	2.50 s	15.2 s

The int4-mlp CPU row comes from one session in which the two files alternated (int8, int4-mlp, int4-mlp, int8); the int8 file, as published, read 941 tok/s prefill and 38.8 tok/s decode in that session. Each file's two processes decoded at 40.5 / 35.4 tok/s (int4-mlp) and 41.0 / 36.6 tok/s (int8), the second process slower for both, so on the Mac CPU the two files decode at about the same speed; the int4-mlp file prefills at less than half the int8 file's speed.

The int8 CPU row was measured on the file before the fp32 GPU declaration was added (the only difference between the two). Init is the Init time the CLI prints; that value includes both Init Total and the phases inside it, so it counts engine creation twice (LiteRT-LM #4057).

Galaxy S26 (SM-S942Q, Snapdragon, Adreno; Android 16) and Pixel 8a (Mali; Android 16): the litert-lm-api 0.19.0 Android wheel from PyPI, Benchmark(model, backend, prefill_tokens=1024, decode_tokens=256, max_num_tokens=2048), 3 runs per process on one engine; a process's value is the mean of runs 2 and 3 (run 1 is the first use); CPU runs started from an unthrottled phone:

Device	File	Backend	Prefill	Decode	TTFT	Processes
Galaxy S26	int4-mlp	GPU (OpenCL)	305 tok/s	27.4 tok/s	3.4 s	2
Galaxy S26	int8	GPU (OpenCL)	295 tok/s	25.8 tok/s	3.5 s	4
Galaxy S26	int8	CPU	242 tok/s	13.8 tok/s	4.3 s	2
Galaxy S26	int4-mlp	CPU	not measured
Pixel 8a	int8	GPU (OpenCL)	145 tok/s	10.0 tok/s	7.2 s	2

The Galaxy S26 int8 CPU row was measured on the file before the fp32 GPU declaration was added. Our test stops a run at the first reading of MemAvailable (the kernel's estimate, in /proc/meminfo, of the memory available to new work without swapping) under a set level (read every 0.25 s): 1,000,000 KiB on the Galaxy S26, which no run here reached, and 400,000 KiB on the Pixel 8a, where the int8 file's lowest reading was 493,272 KiB. The int4-mlp file was not run on the Pixel 8a.

iPhone 18 Pro (iOS 27.2): an app of ours that drives LiteRT-LM through its C API, on the nightly CLiteRTLM 0.19.0-20261029 framework (Google's nightly iOS build; there was no 0.19.0 iOS release framework when we measured), in its benchmark mode with the same 1,024 / 256 / 2,048, one warm-up run and then 2 runs per process on one engine, 60 s between processes; a process's value is the mean of its 2 runs. The phone slows down after about ten minutes of back-to-back runs (iOS thermal state 2, "serious"), so each row is the mean over the processes whose runs stayed at thermal state 1 or below. The two GPU files were also measured alternately (int8, int4-mlp, int4-mlp, int8) from a cool phone, every run at thermal state 0: int4-mlp 51.4 tok/s, int8 51.5 tok/s.

File	Backend	Prefill	Decode	TTFT	Processes
int4-mlp	GPU (Metal)	652 tok/s	51.4 tok/s	1.6 s	2
int8	GPU (Metal)	648 tok/s	51.5 tok/s	1.6 s	4
int4-mlp	CPU	304 tok/s	28.0 tok/s	3.4 s	2
int8	CPU	609 tok/s	22.0 tok/s	1.7 s	3

The app is signed without the increased-memory-limit entitlement; neither file was stopped by the system.

Not in this repository: a Galaxy S26 NPU file. We also compiled this model for the Galaxy S26 NPU (Qualcomm HTP, ahead of time for SM8850, cache 3,968 tokens so the 3,584-token thinking budget fits). It decoded 20.4 tok/s there (the litert_lm_advanced_main benchmark at the same 1,024 / 256 sizes, built from LiteRT-LM main at 3dbb23e1e with QAIRT 2.47; first run of each of 2 processes), slower than the int4-mlp file on the same phone's GPU, and the NPU executor refuses the second turn of a conversation (NPU executor's SetCurrentStep only supports rolling back one token at the end of decode.), so that file is not published.

Conversion notes

Converted with litert-torch 0.9.3 / transformers 5.14.1 / ai-edge-quantizer 0.8.0, released wheels; files packed with the litert-lm 0.19.0 CLI (reproduction scripts: hf-to-litertlm).

• The vendor modeling code is patched for export, not re-implemented (the patch of the Spark-X2.5 conversion; X2.6 ships the same modeling_spark.py). modeling_spark.py computes attention through its own eager function and ignores config._attn_implementation; the export copy dispatches through the registered attention interface (so litert-torch's transposed KV cache is used), threads the per-call kwargs, declares the attention-backend capability flags, and applies the per-head sigmoid output gate in the interface's layout. In eager mode the patched file is bit-identical to the vendor file (max |Δlogit| 0.0 on 24 random tokens).
• The think opener is pre-filled by the template and the thought channel is declared in the metadata. No start token in the metadata: the tokenizer declares <｜start▁of▁sentence｜> as BOS but never prepends it (add_bos_token: false), and the template writes its own.
• Why int4-mlp and not all-int4. An all-int4 (blockwise-32) file left the thought open past 2,048 tokens on short prompts that the bf16 model and the int8 file close: "How do you say 'thank you' in French?" stayed open in 6 of 6 repeated CPU runs and on the GPU. Keeping the attention, the MLP down projection and the output layer at int8 closed them (3 fresh processes per backend). Block scales are floored at 2^-14 (none needed it here).
• GPU graph. After the export, the KV-cache writes and the attention products of every signature are rewritten as LiteRT-LM's GPU composite ops (odml.cache_update, odml.runtime_bmm) with tools/gpu_graph/gpu_graph_retrofit.py from hf-to-litertlm (--decomp exporter --mask add_bcast, plus the band-mask extension the sliding-window layers need, dory_work/retrofit_gpu.py in the same repository); every weight buffer is byte-equal to the export's. For the int8 file with fp32 activations, GPU decode against the same file without the rewrite went up ×1.52 on the Galaxy S26 and ×1.69 on the Mac; its CPU speed did not change measurably.
• GPU activations are declared fp32 in the files (prefer_activation_type = "fp32", a metadata-only repack). LiteRT-LM runs the text decoder with fp16 activations on the GPU by default, and on this model that costs reasoning accuracy: the same int8 weights scored GSM8K 73 with the default fp16 GPU activations and 81 with fp32 declared (bf16 84); with fp16 more thoughts ran past the 3,584-token budget (26 unfinished against 14).
• Tokenizer: the upstream tokenizer.json, byte for byte (the exporter's own copy marks <unk> as special, which the upstream file does not; replaced).
• Quantization at export time; no other post-processing of the weights.

Known runtime issues

On the CPU, the int8 file can end an answer with a literal </think> and then write the same answer again. We saw it on one of two free-form prompts ("In three sentences, explain why the sky is blue.", greedy, thinking on): int8 (CPU) in 3 of 3 processes on a Galaxy S26, the same text every time, and in 7 of 20 on the Mac; an earlier build of the same int8 weights without the GPU graph rewrite (not in this repository) in 3 of 3 on the Galaxy S26 and 0 of 20 on the Mac. The other prompt ("In two sentences, explain why leaves change color in autumn.") showed it in none of 10 Mac processes per file. No GPU run showed it. The int4-mlp file's CPU path was not tested this way (one clean run). Use the GPU.

Later answers in a conversation can differ from the answer the model gives to the same messages in one prompt. With thinking off, from the second turn of a conversation on, LiteRT-LM puts the previous answer's stop token into the KV cache with a zero embedding, because Spark-X2.6-1.7B_int4-mlp.litertlm holds the embedding table in its own section and the decoder takes embeddings (read in the source of LiteRT-LM 0.16.0 to 0.18.1; measured with 0.18.1 on WeVisOne-VL-2B and Qwen2.5-Coder-1.5B-Instruct, not on this file). With thinking on, the runtime rebuilds the cache from the re-rendered history at each turn, so the second turn is free of it when the conversation opens with a user message; the same source shows a zero embedding at another token from the third turn on (not measured). Spark-X2.6-1.7B_int8.litertlm has no embedder section (the decoder looks the tokens up itself) and is not affected. One question per conversation avoids both, as does a new conversation that starts with the earlier messages and the new question.

An answer whose last punctuation mark is a token of its own, such as a . or ? after a word, can arrive without that mark. On LiteRT-LM releases before 0.17.0 (read in the source of 0.10.1 to 0.16.1) the cause is this: these files carry <｜end▁of▁sentence｜> with each of 13 characters in front (12 punctuation marks and a space) as stop sequences next to <｜end▁of▁sentence｜> alone, so the runtime holds the mark back to see whether <｜end▁of▁sentence｜> follows, and when the answer ends there the held mark is never written. From LiteRT-LM 0.17.0 on (read in the source of 0.17.0 to 0.18.1) Spark-X2.6-1.7B_int4-mlp.litertlm still drops 12 of the 13 marks and keeps only ., and Spark-X2.6-1.7B_int8.litertlm drops all 13: the mark and <｜end▁of▁sentence｜> complete two stop sequences at the same token, <｜end▁of▁sentence｜> alone and the one with the mark, the runtime records the one that comes later in the file's stop list (for <｜end▁of▁sentence｜> alone, its first entry when the file lists it twice), and it writes the held mark only when that is <｜end▁of▁sentence｜> alone. Not measured on these files: this follows from the runtime's source read with these files' stop list and tokenizer; the drop on releases before 0.17.0 was seen with LiteRT-LM 0.16.1 on a Galaxy S26, on another bundle with punctuated stop sequences.

License and changes

Distributed under Apache-2.0 (inherited from the base model; the upstream LICENSE file is included). Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; the attention's KV-cache writes and products rewritten as LiteRT-LM GPU composite ops (weights unchanged); fp32 GPU activations declared; tokenizer and chat template repackaged into the .litertlm files (thinking pre-fill and thought-channel metadata as described); tool-calling template branches not carried. No training or fine-tuning. This conversion was made independently of the SparkLLM Team.