Qwen3.5-9B — LiteRT-LM

Qwen/Qwen3.5-9B converted to the LiteRT-LM (.litertlm) format for Google's LiteRT-LM runtime: an int8 file and a Mixed INT4 file, off the same conversion path as the Qwen3.5-4B, 2B and 0.8B conversions, with the 4B files' chat template and thinking switch. Converted with litert-torch (commit 115a1360 with the Qwen3.5 hybrid patch) and tested on litert-lm 0.18.0 on a Mac Studio M4 Max, where the Mixed INT4 file decodes at 46.1 tok/s on the GPU (measured 2026-10-15); on a Galaxy S26 the GPU engine of the Mixed INT4 file did not build (Performance). On litert-lm 0.17.1 both files answered a plain run ("Paris", Mac CPU, one run each).

Qwen describes Qwen3.5 as "integrating breakthroughs in multimodal learning, architectural efficiency, reinforcement learning scale, and global accessibility", with early-fusion vision-language training and "201 languages and dialects"; Qwen3.5-9B is the post-trained model of the release (base Qwen/Qwen3.5-9B-Base), a "Causal Language Model with Vision Encoder" of 9B parameters in the language model, with a 262,144-token native context (4,096 in these files). Qwen's card reports MMLU-Pro 82.5 and IFEval 91.5 for it (vendor-reported, not run here) and recommends sampling of temperature 0.7, top_p 0.8 and top_k 20 for the non-thinking mode and temperature 1.0, top_p 0.95 and top_k 20 for thinking.

The architecture is Qwen3.5's hybrid: 32 layers, 24 of them GatedDeltaNet linear attention and 8 full attention (one in every four), hidden size 4096, intermediate size 12288 (a dense FFN in every layer), 16 attention heads with 4 KV heads of 256, linear attention with 16 key and 32 value heads of 128, a 248,320-token vocabulary with untied embeddings. The upstream checkpoint is multimodal; these files hold the text decoder only (8,953,803,264 of the release's 9,653,104,368 parameters; the vision tower and the multi-token-prediction head are not in them, the same text-only load as upstream's Qwen3_5ForCausalLM), so the image and video input of Qwen3.5-9B is not available here. Only the 8 attention layers keep a growing KV cache (a 4096-token budget here); the 24 linear-attention layers carry a constant-size conv and recurrent state.

Qwen3.5 thinks by default (its template opens <think> at the generation prompt unless enable_thinking is false). These files run with thinking off unless asked, as the Qwen3.5-4B files do: the bundle's chat template is the 4B files' (the LiteRT-LM models/qwen3_5 template at 87ab9334 with enable_thinking defaulting to false and a content-parts macro), it writes the empty <think>\n\n</think> block at the generation prompt when thinking is unset or off, and the replies are the answers alone. With ThinkingConfig(enable_thinking=True) (the CLI's --thinking true) the prompt opens <think> instead, and the runtime returns the reasoning on the declared thought channel (<think>\n / \n</think>) and the answer as the text. Thinking on is an opt-in with a budget: the measured facts are under Correctness and Usage.

File	Recipe	Size	sha256
Qwen3.5-9B_int8.litertlm	int8 dynamic on every linear layer, int8 embedding; the convs, the delta rule and the 48 GatedDeltaNet decay/gate projections (in_proj_a / in_proj_b) float; fp32 GPU activations declared	9.20 GB (9,200,832,896 bytes)	82d23246c88d957ab316960147be5e04adc7b3f4492548406dcd6540fbeb92f1
Qwen3.5-9B_mixed_int4.litertlm	Mixed INT4: int4 blockwise-32 (min-max) on the linear layers, int8 on the lm_head and the embedding, the same 48 projections float; fp32 GPU activations declared	6.06 GB (6,060,357,536 bytes)	61042ebd3c473763972d22310c055f63fab5a29bc2976ab533d6a59c262cac65

On GSM8K (thinking off, 2,048 output tokens, table in Accuracy) the int8 file scored 98 and the Mixed INT4 file 95 on the Mac GPU against 97 for the bf16 model in transformers. The Mixed INT4 file is 34 % smaller. Time for a 1024-token prefill plus 256 decoded tokens at context 2048, from the rates in Performance:

Seconds	Mac M4 Max	Galaxy S26	Pixel 8a
Mixed INT4 GPU	7.25	GPU engine does not build (Performance)	not run (Performance)
int8 GPU	8.43	not run (Performance)	not run (Performance)
int8 CPU	21.49	not run (Performance)	not run (Performance)
Mixed INT4 CPU	33.43	not run (Performance)	not run (Performance)

On the Mac the Mixed INT4 file is the faster file on the GPU (13.9 % less time, decode 46.1 vs 38.4 tok/s at about the same prefill rate) and the int8 file on the CPU (the int4 file's blockwise prefill runs at 66 tok/s against 228). Not measured on iOS: each file is one TFLite section (8.57 GiB int8, 5.64 GiB Mixed INT4, the embedding table inside it), over the ~2 GiB single-section limit the Ministral-3-3B-Instruct-2512 card describes.

Correctness

8-question check on the Mac M4 Max (the questions with "Answer briefly." appended, --temperature 0, one litert-lm run process each, thinking unset): int8 8/8 on CPU and GPU, Mixed INT4 8/8 on CPU and GPU, degenerate 0; the bf16 model in transformers 8/8. The int8 weights with the runtime's default fp16 GPU activations answered 0 of 8 (every reply a run of !), which is why both files declare fp32 activations.

All 7 signatures of both files are fully delegated on the Mac GPU (34,373 / 25,157 / 22,469 / 22,589 / 22,589 / 2,866 / 2,920 nodes, one partition each, no validation error). No start token: the engine adds nothing before the template's text, and at every swept prompt length its prefill count equals the transformers token count of the render. A three-turn conversation with thinking off ("Give me one word that means 'very big'." / "Use it in a short sentence." / "Now make that sentence a question.") on both files and both backends kept the word across the turns with no chat marker in the text; the int8 file gave the bf16 model's three replies on both backends ("Colossal" / "The colossal statue dominated the entire city square." / "Did the colossal statue dominate the entire city square?", the word in bold), the Mixed INT4 file a mountain instead of a statue. A prompt-length check (17 prompt lengths between 21 and 60 tokens, a fresh engine each, both files, CPU and GPU): the expected word every time, 68 of 68.

Multi-step arithmetic on the Mac CPU (six multiplication questions, three fresh processes each, thinking off, 400-token cap): the right product in 18 of 18 runs per file, every run ending by itself with no repeated line; on the Mac GPU 6 of 6 per file. Long-output check (four prompts, greedy and the runtime's default sampling, up to 512 tokens, Mac CPU): no reply of either file looped; 2 of 8 int8 replies and 3 of 8 Mixed INT4 replies reached the cap, none repeating a sentence. Greedy CPU runs still differ between processes: the same thinking-on prompt, sampler and file gave two thoughts that part after 101 characters, one closing at 797 tokens and the other running to a 2,048-token cap, so CPU results on this card are counted over processes.

Thinking on (ThinkingConfig(enable_thinking=True), Mac CPU, greedy, 2,048-token cap, 3 fresh processes per question): with no budget, "What is 17 times 23?" closed the thought and answered 391 in 3 of 3 runs per file (379–712 tokens), and a 12-pencil word problem closed and answered 100 in 3 of 3 int8 runs (1,095–1,201 tokens) and 0 of 3 Mixed INT4 runs (the thought still open at the cap, one of them repeating a sentence); with thinking_token_budget=300 both questions closed and answered right in 6 of 6 runs per file. A three-turn conversation with thinking on ("My name is Kenji." / "What is 17 times 23?" / "What is my name?", 2,048 output tokens per turn): with the 300-token budget all four file-and-backend combinations answered 391 and Kenji with every thought closed (the KV before each decode equal to the token count of the template's render); with no budget the GPU runs passed (the Mixed INT4 file's second thought took 1,427 tokens) and the CPU runs did not (int8's first thought and Mixed INT4's second ran to the 2,048-token cap without closing), and at 1,024 tokens per turn the Mixed INT4 file left turns empty on both backends. Between turns the runtime takes each reply's thought out of the KV and prefills the history again without it, as Qwen's template renders past turns.

Accuracy

GSM8K: test questions 0–99 with a step-by-step instruction and a final #### <number> line, the bundle's template (thinking off), greedy, at most 2,048 output tokens, a 4096-token KV budget, a fresh process per question (litert-lm 0.18.0 Python API, Mac GPU). The bf16 reference is transformers 5.14.1 on MPS with Qwen's template at enable_thinking=False (byte-identical to the bundle's render), the same prompts, greedy, 2,048 new tokens. IFEval was not run; Qwen's own table reports IFEval 91.5.

Configuration	GSM8K	wrong questions
bf16 reference (transformers, MPS)	97 / 100	12, 87, 93
int8, litert-lm Mac GPU	98 / 100	12, 93
Mixed INT4, litert-lm Mac GPU	95 / 100	38, 41, 62, 75, 87

Question by question, the int8 file agrees with bf16 on 99 of 100 (right only in the int8 file: 87); the Mixed INT4 file agrees on 94 (right only in bf16: 38, 41, 62, 75; right only in the int4 file: 12, 93), and its three replies to questions 38, 41 and 62 reached the 2,048-token cap without a final line and count as wrong. The GSM8K runs and the benchmark below ran on files whose TFLite section is byte-identical to the shipped files'; the tokenizer section was replaced and the metadata re-packed afterwards (Conversion notes), and every evaluation prompt tokenizes to the same ids before and after.

Usage

litert-lm run ./Qwen3.5-9B_int8.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17."
litert-lm run ./Qwen3.5-9B_int8.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17." --backend gpu --cache no
litert-lm run ./Qwen3.5-9B_mixed_int4.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17."
litert-lm run ./Qwen3.5-9B_mixed_int4.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17." --backend gpu --cache no
litert-lm benchmark ./Qwen3.5-9B_int8.litertlm -p 1024 -d 256 --max-num-tokens 2048 --runs 3 --cache no --backend cpu
litert-lm run ./Qwen3.5-9B_int8.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17." --backend gpu --cache no --thinking true

The four plain run commands listed the two dates ("The dates extracted from the sentence are: … May 3 … May 17" and a "Based on the sentence provided" form of it), with no think block and no control token in the output, in 43–62 s including engine init. The last command turns thinking on: the CLI printed the reasoning as a [thought] … [/thought] block and then the answer (a numbered analysis ending with the two dates). The 0.18.0 CLI's --thinking-budget N caps the reasoning; on the CPU a budget is what keeps a thought from using the whole output budget (Correctness), and 300 is the value measured here. Without --backend, run uses the CPU; without --cache, it writes a compiled-weights cache next to the model on its first CPU run (7.99 GB for int8, 4.96 GB for Mixed INT4). The bundle sets no sampler; run passes one only when you give --top-k, --top-p, --temperature or --seed (every number on this card is greedy). For Android, see android-gpu.md.

The chat template (4,250 bytes) is the one the Qwen3.5-4B files carry: LiteRT-LM's models/qwen3_5/chat_template.jinja at 87ab9334 with two edits (its content macro reads a string or typed parts, text and tool_response; enable_thinking defaults to false). It writes ChatML turns, a # Tools block in Qwen3.5's format when tools are given, a tool message as a <tool_response> inside a user turn, past assistant turns as their reply text without a think block (Qwen's "No Thinking Content in History"), and no default system message. The runtime's template engine renders it byte for byte as transformers renders Qwen's template in six conversation shapes (a single question, with a system message, two and three turns, with a system message and three turns, with a tool list) in all three thinking states; the tool-response shape differs (the assistant turn that calls a tool carries no empty think block here). No start token; stop tokens <|endoftext|> (id 248044, config.json's eos) and <|im_end|> (id 248046, the chat turn end; the exporter declares only the former). The tokenizer section is the repo's tokenizer.json byte for byte: the exporter's bundle had carried the tokenizer as transformers writes it for the Qwen2Tokenizer class (a pre-tokenizer without the \p{M} class), which tokenizes Hindi, Thai, Tamil and Arabic with harakat differently (49 vs 36, 24 vs 12, 29 vs 16, 30 vs 29 tokens on four test strings); the runtime's tokenizer now matches the tokenizers library on the repo file in 346 of 346 strings.

Tool calling: the template renders a tool list passed through the API (a get_weather(city) tool made the prompt 285 tokens instead of 19) and the model writes the call in Qwen3.5's XML form as text, <tool_call><function=get_weather><parameter=city>Paris</parameter></function></tool_call>, which the runtime does not parse into a tool call and does not run (0 parsed calls with automatic tool calling off and on, and with Qwen's own tool block as the system message; the bundle's generic model type has no parser for this form, LiteRT-LM issue 3927); a tool result sent back as a tool message was used in the answer ("The weather in Paris is sunny with a temperature of 22°C."). An app that needs the calls parses that text itself.

• One Engine per conversation. On litert-lm 0.18.0 the runtime starts a new Conversation on an Engine that already ran another conversation from the earlier conversation's recurrent state: it reuses the prompt tokens the two conversations share and does not reset the state (the open report). On these files the probe (three questions in a pirate-persona conversation, then in a conversation told to answer in one word, on the same Engine and on a fresh Engine; CPU, greedy, 3 runs per file) kept the one-word rule in 3 of 3 runs on a fresh Engine, and on the shared Engine the second conversation broke in 2 of 3 int8 runs and 1 of 3 Mixed INT4 runs (every reply a repeated </think>). Create a new Engine for each conversation.

Performance

Mac Studio M4 Max, litert-lm 0.18.0: the benchmark command above with --backend cpu or gpu (1024-token prefill, 256-token decode, context 2048), one warmup, then the mean of 3 iterations, two processes per file and backend (each in its own held measurement window), the median of the two; GPU is WebGPU on Metal. Init is the CLI's printed init time of the first iteration; in litert-lm 0.18.0 it sums overlapping init phases and reads about twice the engine build measured from the log's timestamps (the Mixed INT4 file's GPU run printed 89.2 s for a 44.5 s build; a granite-4.1-8b int4 GPU run 7.38 s for 3.66 s; the Python API built these GPU engines in 42.7 s int8 and 45.1 s Mixed INT4, medians of 100 fresh processes). The GPU build is long because each of the seven signature graphs is compiled on one thread (34,373 nodes for the 1024-token prefill, against 1,823 for a dense 8B).

Mac M4 Max	Prefill 1024 (tok/s)	Decode 256 (tok/s)	TTFT (s)	Init (s)
int8 GPU	593.06	38.37	1.754	96.04
Mixed INT4 GPU	603.60	46.07	1.718	88.66
int8 CPU	228.09	15.06	4.556	96.08
Mixed INT4 CPU	66.25	14.25	15.527	57.03

Memory during the benchmark on the CPU (--cache no): int8 50.5 GB, Mixed INT4 30.1 GB; during a GPU run: 10.2–10.7 GB and 8.4–9.0 GB. The files declare fp32 activations for the GPU: the int8 weights with the runtime's fp16 default took 7.8 % less time on the same workload and answered none of the 8 check questions. A three-signature variant of the Mixed INT4 file (prefill 1024 / 256 / 1, the same weights, 50,244,960 bytes smaller) also answers the 8 questions 8/8 on the Mac GPU; on a Galaxy S26 neither it nor the shipped file builds a GPU engine (below).

Phones (Android 16, the prebuilt litert_lm_advanced_main of 2026-09-18 from LiteRT-LM's public binary bucket):

• Galaxy S26 (SM-S942Q, Qualcomm SM8850, Adreno 840, MemTotal 11,389,756 kB): the GPU engine of the Mixed INT4 file does not build on a phone of this memory class. On a platform device-lab Galaxy S26 (SM-S942U1, MemTotal 11,389,764 kB; 2026-10-16) the Ornith-1.0-9B files of the same architecture and recipe (that Mixed INT4 file is 294,912 bytes larger than this one) failed four sessions, the six-signature file and a three-signature variant, each in a benchmark run and in a prompt run, during the engine build: while the OpenCL delegate was taking the signatures the low-memory killer removed other apps 376 to 473 times, the kernel logged thrashing, and each job failed 3 to 6 minutes after the engine creation started, with no output. The benchmark session of this file on that unit (session-ef58b2db) ended the same way: the delegate took prefill_1024 and prefill_256, the runtime log stopped at the OpenCL initialization of the next signature, and the low-memory killer had removed other apps 276 times. Our own phone was not tried with these files (a 7.9 GB LiteRT-LM process has already restarted its framework once). The int8 file was not started on the phone's GPU: after a sibling 8B-class int8 file restarted this phone during its GPU engine warm-up on 2026-10-14, no 8B-class int8 engine is started on it. The CPU was not run: by the earlier granite-4.0-micro measurements on the same phone (a CPU peak of 1.5 × the int8 file and 2.0 × the int4 file) these files would need 13.8 GB and 12.1 GB against the phone's 11.66 GB, and a 7.9 GB LiteRT-LM process has already restarted this phone's framework once.
• Pixel 8a (Tensor G3, Mali-G715, MemTotal 7,754,700 kB): not run, and GPU execution on a Mali phone is not verified for these files. No Qwen3.5 file on the Hub, the 0.8B included, built a GPU engine on this 7.75 GB Mali phone under our memory guard (a run ends at the first MemAvailable reading under 400,000 kB; 0 of 5 engines on 2026-10-12, each file accepted whole by the OpenCL delegate while the engine build exhausts memory), and these files are the same architecture at 2–9 × the size; on the CPU the same multipliers give 13.8 GB and 12.1 GB against 7.94 GB.
• Not run on an NPU: these dynamic-range int8 and blockwise int4 graphs have no HTP path (the NPU path takes static-range or float graphs; the GatedDeltaNet layers' ops come on top).

Conversion notes

• Source and stack: Qwen/Qwen3.5-9B at revision c2022362 (every file of the revision checked against the Hub's record); litert-torch at commit 115a1360 with the Qwen3.5 hybrid patch of hf-to-litertlm (the 2026-08-21 version: the GatedDeltaNet cache layers, the rank-4 chunk kernel and head interleave of the 4B's notes), transformers 5.14.1, torch 2.12.1, ai-edge-quantizer 0.8.0; packed and checked with litert-lm 0.18.0. qwen35_work/convert_qwen35_9b.sh in hf-to-litertlm is the conversion command, and qwen35_work/use_switch_template.sh the metadata step that puts the 4B files' template and thinking switch on its output. The conversion run of 2026-10-14 behind these files rebuilt all four sections of both files byte for byte from the pinned checkpoint, the tokenizer step reproduced the shipped tokenizer section from that run's output, and the switch step reproduced the shipped metadata section from the files before it; run as published on 2026-10-16 for Ornith-1.0-9B (the same script bytes on that repo's id, one command, 18 minutes on a 128 GB Mac), the script rebuilt that repo's files byte for byte. The file sha256 differs from run to run (the packer writes a new uuid and timestamp), so compare sections.
• Export: the text decoder through litert-torch's lightweight conversion (constants of 1e8 bytes and more stay lazy), 6 prefill signatures (1024, 256, 64, 16, 4, 1) plus decode over a 4096-position KV budget, 64 state buffers (8 KV pairs and 48 linear-attention states); 32 minutes and a 49.4 GB peak on a 128 GiB Mac, each quantize step 35.4 GB. Run one export at a time.
• Quantization after the export: int8 = dynamic int8 on every FULLY_CONNECTED plus an int8 embedding, with the 48 decay/gate projections of the 24 linear-attention layers kept float (the CPU backend quantizes each int8 layer's input at run time, and those heads do not take that noise: the 2026-10-13 note on the Qwen3.5-4B card; the multi-step arithmetic check above is the test for it); Mixed INT4 = int4 blockwise-32 min-max on the linear layers, the lm_head and the embedding int8, the same 48 float. The int4 file's 216,006,656 block scales are all normal floats (none zero or subnormal), so no scale floor was needed.
• Metadata: the template and thinking switch above, the thought channel, supports_thinking true, supports_function_calling false (the 4B files' LlmMetadata section byte for byte); no start token; stop ids 248044 and 248046; generic_model; fp32 GPU activations declared; max_num_tokens 4096; the tokenizer section replaced by the repo's tokenizer.json.

License and changes

Apache-2.0, inherited from Qwen/Qwen3.5-9B (the repo's metadata and its LICENSE file, "Copyright 2026 Alibaba Cloud"); this repository carries a copy of that LICENSE. Changes from the original work: the text decoder of the multimodal checkpoint exported (the vision tower and the multi-token-prediction head not loaded), weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above with the 48 decay/gate projections float; the chat template replaced by the Qwen3.5-4B files' template with a thinking switch (off by default) and a thought channel; the tokenizer repackaged unmodified; stop ids 248044 and 248046, fp32 GPU activations and a 4096-token KV budget declared. This repository is a community conversion and is not affiliated with Qwen or Alibaba Cloud.