Qwen3.5-4B — LiteRT-LM

Qwen/Qwen3.5-4B converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15 (both backends gated on 0.15.0 and 0.16.0). Same conversion rail as our Qwen3.5-0.8B — the GPU-delegable rank-≤4 gated-delta kernel — plus two 4B-specific fixes described in the conversion notes.

Qwen3.5 is Alibaba's hybrid architecture: GatedDeltaNet (gated delta rule linear attention) blocks interleaved with a few gated full-attention blocks (this 4B model has 24 linear-attention + 8 attention layers). The linear-attention blocks carry constant-size per-layer conv + recurrent state instead of a growing KV cache, so memory stays nearly flat with context length — only the 8 attention layers keep KV (4096-token budget here). The upstream checkpoint is multimodal; this package is the text decoder only (vision tower and MTP heads dropped exactly as upstream's own Qwen3_5ForCausalLM text-only load contract does).

File	Recipe	Size
Qwen3.5-4B_int8.litertlm	int8 dynamic on linears + embedding (convs and the delta rule stay float), fp32 activations declared	4.10 GB

Correctness

All gates below were run on this exact file (or its float parent, same graph rail):

• Logits-level parity vs PyTorch: the float export matches the HF model teacher-forced across 48 positions — top-1 and top-5 agreement 100%, Pearson 1.0000, KL ≈ 0.
• 8-question sanity gate: 8/8 on CPU and GPU, on both litert-lm 0.15.0 and 0.16.0 (Mac).
• iPhone 17 Pro (Metal GPU): the composite 8-question probe (all 8 questions in one 138-token prompt) answers 8/8, word-for-word identical to the HF fp32 reference run greedy with the same template.
• Prompt-length robustness: first-token check against the runtime's real prefill chunk plans, fresh engine per length — 40/40 lengths clean on CPU, 20/20 on GPU.
• Multi-turn conversations carry state correctly across turns.

Usage

litert-lm run ./Qwen3.5-4B_int8.litertlm --prompt "What is the capital of France? Answer in one word."

# GPU
litert-lm run ./Qwen3.5-4B_int8.litertlm --backend gpu --cache no --prompt "..."

Prefill signatures: 1024, 256, 64, 16, 4, 1 (not the 0.8B's full 1–1024 ladder). Every exported signature is charged engine memory even if never called, and with this model's 248k vocabulary the full ladder's Metal programs push a 12 GB iPhone past its jetsam ceiling during engine creation. The reduced ladder keeps GPU peak at ~4.9 GB on iPhone 17 Pro with no correctness cost (the runtime just plans slightly coarser chunks).

Chat template note: the bundle ships a simplified ChatML template rather than the stock Qwen3.5 template. Thinking is disabled the way the stock template's non-thinking mode does it (an empty <think>\n\n</think> block opens each assistant turn), and — deliberately — that block is kept in history renders too: the stock template strips it from past turns, which breaks LiteRT-LM's incremental conversation rendering (the engine requires each turn's render to be a string-extension of the previous one) and kills multi-turn on turn 2. Tool-calling and vision sections are not included.

Performance

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:

Backend	Prefill (256)	Decode	TTFT
GPU	672 tok/s	62.0 tok/s	0.40 s
CPU	243 tok/s	19.9 tok/s	1.11 s

On device (cold start, unplugged, thermal-nominal start, single runs, 138-token prompt, quality-gate harness):

Device	Backend	Prefill	Decode	TTFT	Peak memory
iPhone 17 Pro	GPU (Metal)	87.1 tok/s	9.3 tok/s	1.93 s	4.98 GB
iPhone 17 Pro	CPU	24.0 tok/s	5.4 tok/s	6.88 s	1.70 GB

(iPhone GPU: model load 66 s on first launch.)

Honest notes:

• GPU inference runs with fp32 activations (declared in the bundle; the fp16-activation formulation is unfinished for this family). That is where the GPU memory multiple over CPU comes from.
• Pixel 8a (and 8 GB-class Android phones): not attempted, and not expected to fit. The 0.8B's fp32-expanded GPU programs already exceeded this phone's ~3.8 GB available memory at 963 MB on disk; this file is 4.3× larger, and its 4.1 GB of int8 weights alone leave no CPU headroom on an 8 GB phone either. Treat this release as Apple-hardware-first; an Android row can follow on 12 GB+ devices.
• On low-end Android GPUs, decode is memory-bandwidth-bound and does not beat the CPU anyway; the GPU win is on Apple hardware (and, generally, prefill/TTFT).
• On quality: individual questions gate 8/8 on every backend. On the harder composite probe (8 questions in one prompt), GPU matches HF fp32 word-for-word; the CPU int8 path answers 6 of the 8 (every answer it gives is correct — it skips two). If you need maximum fidelity on complex prompts on CPU, ask for a float variant.

Conversion notes

Converted with litert-torch plus a hybrid-cache patch (reproduction script + patch: hf-to-litertlm qwen35_work/):

• Rank-4 chunk kernel (shared with the 0.8B): the reference chunked delta rule spells its contractions as broadcast-multiply-reduce over high-rank intermediates. The vendored kernel re-expresses them as batched matmuls with chunk and head axes folded into the batch axis (all tensors rank ≤ 4, no BROADCAST_TO, no int64 index math), and writes every tail-pad as a concat with a zeros constant rather than a PAD op (the GPU delegate miscomputes rank-3 non-final-axis PAD — LiteRT#9272).
• Rank-4 head interleave (4B-specific): the 4B is the first Qwen3.5 with grouped value heads (32 value vs 16 key heads). Upstream expands q/k with repeat_interleave, whose rank-5 lowering the GPU delegate rejects outright — the 0.8B (ratio 1) never traces that branch. The patch re-expresses it as concat + rank-4 reshapes, bitwise-identical to the stock op.
• Reduced prefill ladder (4B-specific): see the signature note above.
• Export cache for GatedDeltaNet layers: conv [B, conv_dim, K] + recurrent [B, heads, k_dim, v_dim] cache layers registered for layer_types == "linear_attention", so torch.export traces the model's own state contract.