Qwen3-32B LiteRT-LM Models

This repository holds two LiteRT-LM conversions of Qwen/Qwen3-32B for the GPU backend, an int4 file and an int8 file, for LiteRT-LM 0.19.1. On an Apple M4 Max (128 GB) with WebGPU, the int4 file decodes 15.88 tokens/s and prefills 159.58 tokens/s (2026-11-14); the int8 file decodes 13.23 tokens/s and prefills 157.80 tokens/s (2026-11-15; Performance). Both passed the accuracy check. Speed is still being tuned (updates follow). The int4 file picks the same next token as the original model about as often as the MLX community's 4-bit conversion does (4,571 against 4,548 of 5,100 positions, Accuracy).

Both files were exported with litert-torch 0.10.0 and quantized with ai-edge-quantizer 0.10.1, one to int8 and one to int4; the block scales and codes of the int4 file's decoder layers were then recomputed by a clip search. The weights of both files were then moved out of the model graph into a TFLiteWeights section at the end of each file (LiteRT's external weights; Conversion Notes). On the int8 file, fp16 GPU activations in place of fp32 gave a decode median 1.2 % higher, inside the spread between runs; this was not tried on the int4 file.

Qwen/Qwen3-32B is Qwen's dense 32.8B model: 64 layers, 64 query and 8 key-value heads, 32,768 tokens of context natively. It thinks before it answers unless thinking is switched off (source model card).

Use the GPU: on LiteRT-LM's CPU backend the int8 file, run before its weights were moved, drifted in its token choices from the original model's (Runtime notes); the int4 file was not run on the CPU. The released litert-lm 0.19.1 and 0.19.2 from PyPI run both files on the GPU (asked "What is 17 + 25? Answer briefly.", each answered 42). The numbers below come from LiteRT-LM v0.19.1 built from its tag, whose command-line tool has the benchmark and sampler flags (Build the runtime).

Available Artifacts

File Quantization Backend Context Size
qwen3_32b_mixed_int4_ekv4096.litertlm int4 blockwise 32 for every linear layer of the decoder (block scales from a clip search); int8 per channel lm_head and token table; fp32 activations declared; weights in a TFLiteWeights section GPU 4096 19.6 GB
qwen3_32b_int8_ekv4096.litertlm int8 per channel for every linear layer, lm_head and token table; fp32 activations declared; weights in a TFLiteWeights section GPU 4096 32.9 GB

Conversion Notes

  • Exported from the bf16 checkpoint (revision 9216db57) with litert-torch's export_hf and GPU composites: one prefill and one decode signature, whose lengths LiteRT-LM sets at load time (prefill 1,024 and KV 4,096 by default).
  • The int4 file: the weights of the decoder's linear layers are quantized in blocks of 32, each block with its own scale. For each block, 11 candidate scales are tried (max|w| / 7 times 1.0, 0.95, โ€ฆ, 0.5) and the one with the least squared error is kept; the scale is then refit three times by least squares and stored as fp16 (never below 2^-14). With plain min-max scales (max|w| / 7, no search) the file failed the accuracy check on the GPU: with blocks of 128, 3 of the 12 questions and one of the 3 turns ran to the 2,048-token cap; with blocks of 32, one question did, repeating one sentence. The block-128 file was also run on the CPU: 12 of 24 next tokens matched the bf16 checkpoint's.
  • The int4 file's weights: after the clip search, the int4 and int8 weights of its fully connected and embedding ops (17,547,591,680 bytes) were moved from the model graph into one TFLiteWeights section at the end of the file with tools/externalize of hf-to-litertlm, which runs LiteRT's two external-weight tools (commit 8761012b). The tool's checks passed: every section other than the model graph (the LLM metadata with the chat template, the executor metadata and the tokenizer) is byte-identical to the file's before the move, and every moved tensor's bytes equal the buffer it came from. The fp16 block scales stay in the graph.
  • The int8 file's weights: its int8 weights of the same ops (32,761,446,400 bytes) were moved the same way, with the tool at the same commit, and the same checks passed.
  • Metadata: LiteRT-LM 0.19.1's own Qwen3 configuration, byte for byte (chat template with enable_thinking, a thought channel, stop token <|im_end|>, sampler top-k 20 / top-p 0.95 / temperature 0.6). Tokenizer: the checkpoint's tokenizer.json, byte for byte.
  • The int8 conversion, one command: john-rocky/hf-to-litertlm: qwen3_32b_work. The int4 file's clip-search step has no public script yet.

Performance

Apple M4 Max (128 GB, macOS 27.0), litert_lm_advanced_main built as below, 1,024 prefill tokens and 256 decode tokens with a 2,048-token KV cache. The int8 row was measured on 2026-11-15 with the file as uploaded; it replaces the row of 2026-11-11, measured with the same file before its weights were moved into the TFLiteWeights section (prefill 160.54, decode 13.42 tokens/s), which had replaced the row of 2026-11-09 (decode 11.86 tokens/s, peak footprint read with vmmap). The int8 file before the move was not run in the session of 2026-11-15. The int4 row was measured on 2026-11-14 with the file as uploaded; it replaces the row of 2026-11-11, measured with the same file before its weights were moved into the TFLiteWeights section (prefill 163.35, decode 15.77 tokens/s). In the same session of 2026-11-14, the file before the move prefilled 158.24 and decoded 15.88 tokens/s (2 processes).

File Backend Prefill (tok/s) Decode (tok/s) Peak footprint
qwen3_32b_mixed_int4_ekv4096.litertlm GPU (WebGPU) 159.58 15.88 29 GB
qwen3_32b_int8_ekv4096.litertlm GPU (WebGPU) 157.80 13.23 35 GB
  • Command: --backend=gpu --benchmark --benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256 --num_iterations=4 --max_num_tokens=2048 --disable_cache=true --report_peak_memory_footprint --sampler_handles_input=false.
  • The median of 3 processes per file, each the mean of iterations 2 to 4. A process was left out and run again when, during its iterations 2 to 4, the Mac showed git work (10 or more git processes or a full core between them) or other jobs at 3 cores together, or when, while it ran, other jobs used half a core (one job) or 1.5 cores (together) in two 5 s readings in a row; macOS's own processes, the terminal app and the coding-assistant sessions were not counted as jobs. The int8 file: three processes between 13:27 and 13:40 JST (2026-11-15). The int4 file: three processes between 19:07 and 19:24 JST (2026-11-14); during the first and the third, another job copied a file to a phone over USB (for 16 s and 29 s), which was not counted as a job.
  • Peak footprint: the largest physical footprint top showed, read about every 17 s; at this size top prints whole GiB (27 GiB and 33 GiB). The CLI's own peak line (printed because of --report_peak_memory_footprint) is a different measure and is not the number in the table.
  • No phone was tried: the files are 19.6 and 32.9 GB.

Accuracy

12 questions and a 3-turn conversation on the GPU, greedy (--sampler_top_k=1), up to 2,048 tokens per answer with thinking on, judged against Transformers (the bf16 checkpoint on the Mac's GPU through MPS, its token choices checked against a float32 CPU run). Questions: arithmetic, facts, two translations, a rhyme, 0.9 vs 0.11, a word problem and fib(10) in Python; the 3 turns ask the model to recall a number given in turn 1. The source card advises against greedy decoding in thinking mode (endless repetitions); greedy is used here to compare token by token. After their weights were moved into a TFLiteWeights section, both files gave the same text as before, thought and answer, on the 12 questions and in each of the 3 turns.

File Backend Same verdict as Transformers Ends with the stop token 3 turns
qwen3_32b_mixed_int4_ekv4096.litertlm GPU (WebGPU) 12 / 12 12 / 12 pass
qwen3_32b_int8_ekv4096.litertlm GPU (WebGPU) 12 / 12 12 / 12 pass

Next-token agreement: each model was given Transformers' own 15 replies (the 12 questions and the 3 turns, 5,100 reply tokens), and its top-1 next token was compared with Transformers' at every position. Matches: the int4 file 4,571, the int8 file 5,030, mlx-community/Qwen3-32B-4bit (mlx_lm 0.31.3) 4,548. The two LiteRT numbers come from a float32 PyTorch forward pass with each file's quantized weight values, not from a run of the LiteRT-LM runtime.

Token by token over 24 tokens of a math question, the top-1 token computed in float32 from the int8 file's weights (the GPU path's arithmetic, in PyTorch with the file's values) matched the bf16 checkpoint's (also computed in float32) at 23 of 24 positions; at the one miss the checkpoint's two best tokens are 0.02 logits apart. Computed the same way from the int4 file's values, 13 of 24 (mlx-community/Qwen3-32B-4bit run with mlx_lm: 13 of 24). These 24 positions lie in the prompt, where the checkpoint is often undecided (at 8 of them its two best tokens are under 1 logit apart), so the 5,100-position count above is the better measure.

Runtime notes

  • The CPU backend. On the CPU, LiteRT-LM 0.19.1 always quantizes each int8 layer's input to int8 (run-time int8 activation quantization). Before its weights were moved, the int8 file's top-1 token matched the bf16 checkpoint's at 18 of 24 positions of the test prompt (3 runs, identical), one miss at a 5.1-logit margin; without that run-time quantization (the float32 computation under Accuracy), 23 of 24. The CPU was not checked on the 12 questions; the int8 file was not run on the CPU after its weights were moved, and the int4 file was not run on the CPU at all. Use the GPU.
  • External weights. Both files hold their weights in a TFLiteWeights section at the end of the file instead of inside the model graph. They were run with LiteRT-LM 0.19.1 (the tag build and the PyPI package) and 0.19.2 (PyPI); earlier releases were not tried with them. On Android, the GPU path needs LiteRT-LM 0.19.0 or later: the GPU engine of the v0.18.0 and v0.18.1 Android builds crashes (SIGSEGV) on such a file (LiteRT-LM#3986).
  • Long replies. With LiteRT-LM 0.19.1 a blocking send_message still generating after 10 minutes fails with DEADLINE_EXCEEDED. The M4 Max GPU decodes about 16 tokens/s with the int4 file (as measured under Performance) and about 13 with the int8 file; on a slower machine a long thought can hit that limit. Use send_message_async, or cap max_output_tokens.
  • A message that does not fit. On LiteRT-LM 0.18.1 and 0.19.1, after a message fails because it does not fit in the context window, every later message in that conversation fails too (Task failed with state: 7); start a new conversation.

Build the runtime

git clone https://github.com/google-ai-edge/LiteRT-LM.git && cd LiteRT-LM
git checkout v0.19.1   # 7a03724cece417f1724c923b3d527b4319730841
git lfs pull --include "prebuilt/macos_arm64/*"
bazelisk build --config=macos_arm64 -c opt //runtime/engine:litert_lm_advanced_main

Put bazel-bin/runtime/engine/litert_lm_advanced_main in one directory with libwebgpu_dawn.dylib and libGemmaModelConstraintProvider.dylib from prebuilt/macos_arm64/.

Run

./litert_lm_advanced_main --backend=gpu --model_path=qwen3_32b_int8_ekv4096.litertlm \
  --max_output_tokens=2048 --min_log_severity=2 --sampler_top_k=1 --sampler_handles_input=false \
  --disable_cache=true --input_prompt="What is 17 + 25? Answer briefly."

For the int4 file, use --model_path=qwen3_32b_mixed_int4_ekv4096.litertlm. Without --sampler_top_k=1, the CLI samples with the file's TOP_P settings.

License

Apache License 2.0, the license of Qwen/Qwen3-32B.

Citation

@misc{qwen3technicalreport,
      title={Qwen3 Technical Report},
      author={Qwen Team},
      year={2025},
      eprint={2505.09388},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.09388},
}
Downloads last month
149
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for litert-community/Qwen3-32B

Base model

Qwen/Qwen3-32B
Quantized
(191)
this model

Paper for litert-community/Qwen3-32B