Instructions to use litert-community/Qwen3-32B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LiteRT-LM
How to use litert-community/Qwen3-32B with LiteRT-LM:
# LiteRT-LM runs on various platforms (Android, iOS, Windows, Linux, macOS, IoT, Web/WASM) # and supports many APIs (C++, Python, Kotlin, Swift, JavaScript, Flutter). # For platform-specific integration guides, please refer to the official developer website: # https://ai.google.dev/edge/litert-lm # To try LiteRT-LM, the easiest way is to use our CLI tool. # 1. Install the LiteRT-LM CLI tool: pip install -U litert-lm # 2. Download and run this model locally: # See: https://ai.google.dev/edge/litert-lm/cli # A single .litertlm file in the repo is picked automatically; otherwise the CLI asks which one to run # (or pass its name right after the repo id). litert-lm run \ --from-huggingface-repo=litert-community/Qwen3-32B \ --prompt="Write me a poem"
- Notebooks
- Google Colab
- Kaggle
Qwen3-32B LiteRT-LM Models
This repository holds two LiteRT-LM conversions of Qwen/Qwen3-32B for the GPU backend, an int4 file and an int8 file, for LiteRT-LM 0.19.1. On an Apple M4 Max (128 GB) with WebGPU, the int4 file decodes 15.88 tokens/s and prefills 159.58 tokens/s (2026-11-14); the int8 file decodes 13.23 tokens/s and prefills 157.80 tokens/s (2026-11-15; Performance). Both passed the accuracy check. Speed is still being tuned (updates follow). The int4 file picks the same next token as the original model about as often as the MLX community's 4-bit conversion does (4,571 against 4,548 of 5,100 positions, Accuracy).
Both files were exported with litert-torch 0.10.0 and quantized with ai-edge-quantizer 0.10.1, one to int8 and one to int4; the block scales and codes of the int4 file's decoder layers were then recomputed by a clip search. The weights of both files were then moved out of the model graph into a TFLiteWeights section at the end of each file (LiteRT's external weights; Conversion Notes). On the int8 file, fp16 GPU activations in place of fp32 gave a decode median 1.2 % higher, inside the spread between runs; this was not tried on the int4 file.
Qwen/Qwen3-32B is Qwen's dense 32.8B model: 64 layers, 64 query and 8 key-value heads, 32,768 tokens of context natively. It thinks before it answers unless thinking is switched off (source model card).
Use the GPU: on LiteRT-LM's CPU backend the int8 file, run before its weights were moved, drifted in its token choices from the original model's (Runtime notes); the int4 file was not run on the CPU. The released litert-lm 0.19.1 and 0.19.2 from PyPI run both files on the GPU (asked "What is 17 + 25? Answer briefly.", each answered 42). The numbers below come from LiteRT-LM v0.19.1 built from its tag, whose command-line tool has the benchmark and sampler flags (Build the runtime).
Available Artifacts
| File | Quantization | Backend | Context | Size |
|---|---|---|---|---|
qwen3_32b_mixed_int4_ekv4096.litertlm |
int4 blockwise 32 for every linear layer of the decoder (block scales from a clip search); int8 per channel lm_head and token table; fp32 activations declared; weights in a TFLiteWeights section | GPU | 4096 | 19.6 GB |
qwen3_32b_int8_ekv4096.litertlm |
int8 per channel for every linear layer, lm_head and token table; fp32 activations declared; weights in a TFLiteWeights section | GPU | 4096 | 32.9 GB |
Conversion Notes
- Exported from the bf16 checkpoint (revision
9216db57) with litert-torch'sexport_hfand GPU composites: one prefill and one decode signature, whose lengths LiteRT-LM sets at load time (prefill 1,024 and KV 4,096 by default). - The int4 file: the weights of the decoder's linear layers are quantized in blocks of 32, each block with its own scale. For each block, 11 candidate scales are tried (max|w| / 7 times 1.0, 0.95, โฆ, 0.5) and the one with the least squared error is kept; the scale is then refit three times by least squares and stored as fp16 (never below 2^-14). With plain min-max scales (max|w| / 7, no search) the file failed the accuracy check on the GPU: with blocks of 128, 3 of the 12 questions and one of the 3 turns ran to the 2,048-token cap; with blocks of 32, one question did, repeating one sentence. The block-128 file was also run on the CPU: 12 of 24 next tokens matched the bf16 checkpoint's.
- The int4 file's weights: after the clip search, the int4 and int8 weights of its fully connected and embedding ops (17,547,591,680 bytes) were moved from the model graph into one TFLiteWeights section at the end of the file with
tools/externalizeof hf-to-litertlm, which runs LiteRT's two external-weight tools (commit 8761012b). The tool's checks passed: every section other than the model graph (the LLM metadata with the chat template, the executor metadata and the tokenizer) is byte-identical to the file's before the move, and every moved tensor's bytes equal the buffer it came from. The fp16 block scales stay in the graph. - The int8 file's weights: its int8 weights of the same ops (32,761,446,400 bytes) were moved the same way, with the tool at the same commit, and the same checks passed.
- Metadata: LiteRT-LM 0.19.1's own Qwen3 configuration, byte for byte (chat template with
enable_thinking, athoughtchannel, stop token<|im_end|>, sampler top-k 20 / top-p 0.95 / temperature 0.6). Tokenizer: the checkpoint'stokenizer.json, byte for byte. - The int8 conversion, one command: john-rocky/hf-to-litertlm: qwen3_32b_work. The int4 file's clip-search step has no public script yet.
Performance
Apple M4 Max (128 GB, macOS 27.0), litert_lm_advanced_main built as below, 1,024 prefill tokens and 256 decode tokens with a 2,048-token KV cache. The int8 row was measured on 2026-11-15 with the file as uploaded; it replaces the row of 2026-11-11, measured with the same file before its weights were moved into the TFLiteWeights section (prefill 160.54, decode 13.42 tokens/s), which had replaced the row of 2026-11-09 (decode 11.86 tokens/s, peak footprint read with vmmap). The int8 file before the move was not run in the session of 2026-11-15. The int4 row was measured on 2026-11-14 with the file as uploaded; it replaces the row of 2026-11-11, measured with the same file before its weights were moved into the TFLiteWeights section (prefill 163.35, decode 15.77 tokens/s). In the same session of 2026-11-14, the file before the move prefilled 158.24 and decoded 15.88 tokens/s (2 processes).
| File | Backend | Prefill (tok/s) | Decode (tok/s) | Peak footprint |
|---|---|---|---|---|
qwen3_32b_mixed_int4_ekv4096.litertlm |
GPU (WebGPU) | 159.58 | 15.88 | 29 GB |
qwen3_32b_int8_ekv4096.litertlm |
GPU (WebGPU) | 157.80 | 13.23 | 35 GB |
- Command:
--backend=gpu --benchmark --benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256 --num_iterations=4 --max_num_tokens=2048 --disable_cache=true --report_peak_memory_footprint --sampler_handles_input=false. - The median of 3 processes per file, each the mean of iterations 2 to 4. A process was left out and run again when, during its iterations 2 to 4, the Mac showed
gitwork (10 or moregitprocesses or a full core between them) or other jobs at 3 cores together, or when, while it ran, other jobs used half a core (one job) or 1.5 cores (together) in two 5 s readings in a row; macOS's own processes, the terminal app and the coding-assistant sessions were not counted as jobs. The int8 file: three processes between 13:27 and 13:40 JST (2026-11-15). The int4 file: three processes between 19:07 and 19:24 JST (2026-11-14); during the first and the third, another job copied a file to a phone over USB (for 16 s and 29 s), which was not counted as a job. - Peak footprint: the largest physical footprint
topshowed, read about every 17 s; at this sizetopprints whole GiB (27 GiB and 33 GiB). The CLI's own peak line (printed because of--report_peak_memory_footprint) is a different measure and is not the number in the table. - No phone was tried: the files are 19.6 and 32.9 GB.
Accuracy
12 questions and a 3-turn conversation on the GPU, greedy (--sampler_top_k=1), up to 2,048 tokens per answer with thinking on, judged against Transformers (the bf16 checkpoint on the Mac's GPU through MPS, its token choices checked against a float32 CPU run). Questions: arithmetic, facts, two translations, a rhyme, 0.9 vs 0.11, a word problem and fib(10) in Python; the 3 turns ask the model to recall a number given in turn 1. The source card advises against greedy decoding in thinking mode (endless repetitions); greedy is used here to compare token by token. After their weights were moved into a TFLiteWeights section, both files gave the same text as before, thought and answer, on the 12 questions and in each of the 3 turns.
| File | Backend | Same verdict as Transformers | Ends with the stop token | 3 turns |
|---|---|---|---|---|
qwen3_32b_mixed_int4_ekv4096.litertlm |
GPU (WebGPU) | 12 / 12 | 12 / 12 | pass |
qwen3_32b_int8_ekv4096.litertlm |
GPU (WebGPU) | 12 / 12 | 12 / 12 | pass |
Next-token agreement: each model was given Transformers' own 15 replies (the 12 questions and the 3 turns, 5,100 reply tokens), and its top-1 next token was compared with Transformers' at every position. Matches: the int4 file 4,571, the int8 file 5,030, mlx-community/Qwen3-32B-4bit (mlx_lm 0.31.3) 4,548. The two LiteRT numbers come from a float32 PyTorch forward pass with each file's quantized weight values, not from a run of the LiteRT-LM runtime.
Token by token over 24 tokens of a math question, the top-1 token computed in float32 from the int8 file's weights (the GPU path's arithmetic, in PyTorch with the file's values) matched the bf16 checkpoint's (also computed in float32) at 23 of 24 positions; at the one miss the checkpoint's two best tokens are 0.02 logits apart. Computed the same way from the int4 file's values, 13 of 24 (mlx-community/Qwen3-32B-4bit run with mlx_lm: 13 of 24). These 24 positions lie in the prompt, where the checkpoint is often undecided (at 8 of them its two best tokens are under 1 logit apart), so the 5,100-position count above is the better measure.
Runtime notes
- The CPU backend. On the CPU, LiteRT-LM 0.19.1 always quantizes each int8 layer's input to int8 (run-time int8 activation quantization). Before its weights were moved, the int8 file's top-1 token matched the bf16 checkpoint's at 18 of 24 positions of the test prompt (3 runs, identical), one miss at a 5.1-logit margin; without that run-time quantization (the float32 computation under Accuracy), 23 of 24. The CPU was not checked on the 12 questions; the int8 file was not run on the CPU after its weights were moved, and the int4 file was not run on the CPU at all. Use the GPU.
- External weights. Both files hold their weights in a TFLiteWeights section at the end of the file instead of inside the model graph. They were run with LiteRT-LM 0.19.1 (the tag build and the PyPI package) and 0.19.2 (PyPI); earlier releases were not tried with them. On Android, the GPU path needs LiteRT-LM 0.19.0 or later: the GPU engine of the v0.18.0 and v0.18.1 Android builds crashes (SIGSEGV) on such a file (LiteRT-LM#3986).
- Long replies. With LiteRT-LM 0.19.1 a blocking
send_messagestill generating after 10 minutes fails withDEADLINE_EXCEEDED. The M4 Max GPU decodes about 16 tokens/s with the int4 file (as measured under Performance) and about 13 with the int8 file; on a slower machine a long thought can hit that limit. Usesend_message_async, or capmax_output_tokens. - A message that does not fit. On LiteRT-LM 0.18.1 and 0.19.1, after a message fails because it does not fit in the context window, every later message in that conversation fails too (
Task failed with state: 7); start a new conversation.
Build the runtime
git clone https://github.com/google-ai-edge/LiteRT-LM.git && cd LiteRT-LM
git checkout v0.19.1 # 7a03724cece417f1724c923b3d527b4319730841
git lfs pull --include "prebuilt/macos_arm64/*"
bazelisk build --config=macos_arm64 -c opt //runtime/engine:litert_lm_advanced_main
Put bazel-bin/runtime/engine/litert_lm_advanced_main in one directory with libwebgpu_dawn.dylib and libGemmaModelConstraintProvider.dylib from prebuilt/macos_arm64/.
Run
./litert_lm_advanced_main --backend=gpu --model_path=qwen3_32b_int8_ekv4096.litertlm \
--max_output_tokens=2048 --min_log_severity=2 --sampler_top_k=1 --sampler_handles_input=false \
--disable_cache=true --input_prompt="What is 17 + 25? Answer briefly."
For the int4 file, use --model_path=qwen3_32b_mixed_int4_ekv4096.litertlm. Without --sampler_top_k=1, the CLI samples with the file's TOP_P settings.
License
Apache License 2.0, the license of Qwen/Qwen3-32B.
Citation
@misc{qwen3technicalreport,
title={Qwen3 Technical Report},
author={Qwen Team},
year={2025},
eprint={2505.09388},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2505.09388},
}
- Downloads last month
- 149
Model tree for litert-community/Qwen3-32B
Base model
Qwen/Qwen3-32B