Qwen3.5-0.8B — LiteRT-LM

Qwen/Qwen3.5-0.8B converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15. To our knowledge this is the first Qwen3.5 in LiteRT form, and the first gated-delta-net hybrid served by the released LiteRT-LM runtime — it depends on 0.15's generalized state binding (ExecutorMetadata section); the 0.14 engine cannot bind the linear-attention conv/recurrent state buffers.

Qwen3.5 is Alibaba's hybrid architecture: GatedDeltaNet (gated delta rule linear attention) blocks interleaved with a few gated full-attention blocks (this 0.8B model has 18 linear-attention + 6 attention layers). The linear-attention blocks carry constant-size per-layer conv + recurrent state instead of a growing KV cache, so memory stays nearly flat with context length — only the 6 attention layers keep KV (4096-token budget here). The upstream 0.8B checkpoint is multimodal; this package is the text decoder only (the vision tower and MTP heads are dropped exactly as upstream's own Qwen3_5ForCausalLM text-only load contract does).

File	Recipe	Size
Qwen3.5-0.8B_int8.litertlm	int8 dynamic on linears + embedding (convs and the delta rule stay float)	978 MB

Correctness

The converted graph is verified against the PyTorch reference at the logits level: a float export matches the HF model exactly at every decode position (teacher-forced 8-step comparison: per-position max|logit diff| ≤ 3.7e-5, correlation 1.000000, top-1 identical), covering both the chunked prefill path (with cross-chunk state continuation) and the single-step decode path (in-place rolled conv window + delta-rule recurrence). On an 8-question sanity gate, both the float engine build and this published int8 file answer 8/8 — word-for-word identical to the HF fp32 reference run greedy with the same template. A prompt-length robustness sweep (first-token check at every prompt length 12–60 against the runtime's real prefill chunk plans) is all clean, and multi-turn conversations carry state correctly across turns.

Usage

litert-lm run ./Qwen3.5-0.8B_int8.litertlm --prompt "What is the capital of France? Answer in one word."

Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks.

Chat template note: the bundle ships a simplified ChatML template rather than the stock Qwen3.5 template. Thinking is disabled the way the stock template's non-thinking mode does it (an empty <think>\n\n</think> block opens each assistant turn), and — deliberately — that block is kept in history renders too: the stock template strips it from past turns, which breaks LiteRT-LM's incremental conversation rendering (the engine requires each turn's render to be a string-extension of the previous one) and kills multi-turn on turn 2. Tool-calling and vision sections are not included.

Speed

litert-lm benchmark, CPU backend, Mac M4 Max, max-num-tokens 1024:

Variant	Backend	Prefill (256)	Decode	TTFT
int8	CPU	501 tok/s	48.0 tok/s	0.53 s

Honest note: current GPU delegates reject the graph, so this release is CPU-only and the delta-rule scan runs in float on generic ops. Treat it as a correctness-first, first-of-its-architecture release; speed has clear headroom (scan kernels, GPU support) on the runtime side. iPhone / Android on-device numbers: coming — the table will be updated with measurements.

Conversion notes

Converted with litert-torch plus a hybrid-cache patch (reproduction script + patch: hf-to-litertlm qwen35_work/):

• Export cache for GatedDeltaNet layers: conv [B, conv_dim, K] + recurrent [B, heads, k_dim, v_dim] cache layers registered for layer_types == "linear_attention", so torch.export traces the model's own state contract.
• State continuation tracing: prefill graphs trace the chunk-continuation branch (previous conv/recurrent state consumed, so multi-chunk prefill composes) and the decode graph traces the fused single-step branch (conv window rolled in place by causal_conv1d_update).
• Prefill-pad guard: the runtime's chunk planner runs partially-filled prefill chunks; pad positions are made identity steps for the delta rule (per-token decay forced to ~1, zeroed k/v injection) and the stored conv window is gathered at the last valid column via an in-graph one-hot matmul. Without the guard, generation corrupts at chunk-plan-dependent prompt lengths.
• Constant-eye chunk kernel: the reference chunked delta rule builds torch.eye inside the traced function, which lowers to a STABLEHLO_IOTA op no released TFLite kernel set registers (the file would not even load). The kernel is vendored with the identity matrix lifted as a graph constant.
• Runtime state binding: litert-lm ≥ 0.15 binds per-layer states through an ExecutorMetadata section listing each state tensor; it is appended at package time.
• Quantization: post-hoc dynamic int8 over linears + embedding only; the convs and the delta rule stay float (export-time conv-int8 measurably costs quality on state-carrying hybrids).

License and changes

Distributed under Apache-2.0 (inherited from the base model). Changes from the original work: text-decoder weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; vision tower and MTP weights omitted; tokenizer repackaged unmodified; chat template replaced with the simplified ChatML template described above. This repository is a community conversion and is not affiliated with Alibaba / the Qwen team.