mlboydaisuke/Qwen3-0.6B-LiteRT-gpu-composites

Qwen3-0.6B — LiteRT-LM bundles exported with litert-torch's GPU composite flags

Test exports made for the LiteRT team's GPU-composite work (2026-09-10; re-exported 2026-09-11 with the team's fixed-prefill recipe, see Export). Not a release: on LiteRT-LM 0.17.0's macOS GPU paths the full-flag file does not produce correct output yet (see Status below). Weights int4 blockwise-32, activations fp32 (dynamic_wi4b32_afp32).

Files

file	flags	bytes	sha256
Qwen3-0.6B_dynamic_wi4b32_afp32_11flags.litertlm	all eleven flags of the export command below	341,736,912	1048ab332e235e0f84018bbf1eca27b880ff00a57deeeb690a4657647cc61913
Qwen3-0.6B_dynamic_wi4b32_afp32_9flags.litertlm	the same minus --use_sdpa_composite and --use_qkv_norm_rope_composite	341,424,096	81e06e8b6e06ca8d56a448e51c3f6bd606635d089c6eaee2a4970b84d642cf52
Qwen3-0.6B_dynamic_wi4b32_afp32_9flags_rope.litertlm	the nine flags plus --use_rope_composite=True	342,313,632	0910960e5d6bb1294924e9d95c433d4f6c357ffebba1088ee1f5ede6691ba93f

2026-09-21: chat template updated to accept the 0.18 content-parts form. It is the v0.17.0 pbtext's template plus a format_content macro (dual/4155a0aa5616.jinja), swapped in after the export below with swap_template.py. The prompt rendered from string content, the weights, the tokenizer and the executor metadata are byte-identical; the sha256 column is the swapped files'. The pre-swap files are this repo's previous commit.

Signatures: prefill_1024 (fixed) and decode; cache 32771 (from --cache_length=32768 with --enable_gpu_dynamic_cache; the exporter rounds GPU-dynamic sizes up to the next prime). Bundle metadata from LiteRT-LM's models/qwen3/LlmMetadataProto.pbtext (max_num_tokens 4096, sampler TOP_P k=20 p=0.95 t=0.6, stop <|im_end|>). The 2026-09-10 versions of these files (prefill_1031 from --enable_gpu_dynamic_prefill, exporter-generated metadata with max_num_tokens 32771) are in this repo's history at commit 0c5e31dd0281e19d544c853cea37771c30a368d1. Composites in the 11-flag graph: odml.qkv_norm_rope, odml.swiglu, odml.cache_update, odml.sdpa_transposed (decode), with fused QKV / gate-up projections and a bool mask.

Export

litert-torch upstream/main 6d4c622 (2026-09-08; use_swiglu_composite is not in the 0.9.4 wheel), litert-lm-builder 0.17.0 (main's builder needs it), ai-edge-litert 2.2.0, ai-edge-quantizer 0.9.0, torch 2.13.0, transformers 5.14.1.

python -m litert_torch.generative.export_hf \
  --model=Qwen/Qwen3-0.6B --output_dir=out/p1024 \
  --quantization_recipe=dynamic_wi4b32_afp32 \
  --litert_lm_llm_metadata_override=<LiteRT-LM checkout>/models/qwen3/LlmMetadataProto.pbtext \
  --enable_gpu_dynamic_cache=True \
  --apply_gpu_composites=True --fuse_gate_up=True --fuse_qkv=True \
  --use_qkv_norm_rope_composite=True --use_swiglu_composite=True \
  --use_sdpa_composite=True --use_bool_mask=True \
  --bundle_litert_lm=True --prefill_lengths=1024 --cache_length=32768

The 9-flag files drop the two --use_*_composite flags named in the table (the rope variant adds --use_rope_composite=True).

Status on LiteRT-LM 0.17.0 (macOS, Mac Studio M4 Max)

The speeds below were measured 2026-09-10/11 on the previous versions of these files (dynamic prefill prefill_1031, exporter-generated metadata; graphs otherwise identical). The current fixed-1024 files were re-checked 2026-09-11: 9-flag files 8/8 on the 8-question check, 11-flag failure modes unchanged, and an interleaved A/B against the previous 9-flag file (three passes each, same host) read the same speed within 3 %.

file	WebGPU path — PyPI litert-lm 0.17.0 wheel	Metal path — litert_lm_main built from the v0.17.0 tag with --//runtime/executor:gpu_accelerator=metal, plus the release's prebuilt libLiteRtMetalAccelerator.dylib
11 flags	does not run: odml.sdpa_transposed and odml.qkv_norm_rope fail WGSL validation, output is !!	compiles, output wrong (a repetition loop); 28× GpuModelBuilder::UpdateOutputTensors(output of flash_decode_sdpa) shape mismatch ({1,1,1,1,2048} vs {1,16,1,1,128})
9 flags	decode 293 tok/s, prefill 6.6k tok/s (litert-lm benchmark --backend gpu -p 1024 -d 128 --cache no, medians of 3 processes); 8-question sanity check 8/8	decode 243 tok/s, prefill 6.8k (996-token prompt, context 4096, decode to EOS, medians of 3); 7/8
9 flags + rope	decode 310, prefill 6.7k; 8/8	decode 252, prefill 6.9k

Android and iOS: not measured.

Run (macOS)

pip install litert-lm==0.17.0
litert-lm run Qwen3-0.6B_dynamic_wi4b32_afp32_9flags.litertlm --backend gpu --thinking false --cache no --prompt "What is the capital of Japan? Answer briefly."
litert-lm benchmark Qwen3-0.6B_dynamic_wi4b32_afp32_9flags.litertlm --backend gpu -p 1024 -d 128 --cache no

Known runtime issues

From a conversation preface (the system message, initial messages, tools and extra context a conversation is created with) that has tools and no messages, LiteRT-LM renders these files' chat template with an empty message list, and the template reads messages[0] without checking that the list is non-empty, so the render fails with Failed to apply template: undefined value. This happens in two places: when the preface is prefilled at creation (prefillPrefaceOnInit = true in Kotlin, prefillPrefaceOnInit: true on the web, SetPrefillPrefaceOnInit(true) in C++; the default is false), where creating the conversation fails with that error, and in the preface render (renderPrefaceIntoString() in Kotlin and Swift, RenderPrefaceIntoString() in C++, litert_lm_conversation_render_preface_to_string in the C API). The same happens with extra context (such as enable_thinking) and no messages, and in the preface render with an empty preface (run in the runtime on one Qwen3 template; the other templates rendered with minijinja 2.14 and jinja2). Read in the source of LiteRT-LM 0.17.0, 0.18.1 and main; measured with LiteRT-LM 0.18.1 (the litert-lm package from PyPI) through its C API on these files' template text. The default path, where the first message is rendered together with the preface, is not affected: the Python API and the CLI of 0.18.1 expose neither the flag nor the preface render, and Google AI Edge Gallery sets neither (its source as read on 2026-10-25). A system message or any initial message in the conversation config avoids it.