Qwen3.5-4B-3D-Coder — LiteRT-LM

This repository is a conversion of ccore/Qwen3.5-4B-3D-Coder to LiteRT-LM (.litertlm), made with litert-torch (commit 115a1360 with the Qwen3.5 patch of hf-to-litertlm) for LiteRT-LM 0.19.0: on a Mac Studio (Apple M4 Max) CPU the 16K file decodes at 14.8 tok/s with the litert-lm 0.19.0 package (1,024-token prompt, 256 generated tokens, 2026-11-02).

Qwen3.5-4B-3D-Coder is CCORE's early-preview fine-tune of Qwen/Qwen3.5-4B for code, with a focus on interactive 3D scenes in Three.js: from a scene description it writes one self-contained HTML file. Its technical report describes full fine-tuning of the language parameters, with the vision encoder frozen, plus one short LoRA stage, on ccore/ThreeJS-3D-Scenes: 3,933 training scenes, each a prompt, a full thinking trace and the complete HTML answer, and 213 validation scenes kept out of the weight updates. It is not an official Qwen release. The architecture is Qwen3.5-4B's: 32 decoder layers that repeat three Gated DeltaNet (linear attention) layers and one full-attention layer, hidden size 2560, 16 query and 4 KV heads of 256 in the full-attention layers, and a 248,320-entry tied vocabulary. Apache-2.0.

These files hold the text decoder only (the vision tower is not converted): text in, text out. They run with thinking off: the template always writes the empty thought of the vendor's template (<think>\n\n</think>\n\n) after the assistant opener, the files declare supports_thinking: false, and enable_thinking changes nothing (why: Conversion notes).

File	KV cache	Prefill sizes	Size
Qwen3.5-4B-3D-Coder_mixed_int4_ekv16384.litertlm (the 16K file)	16,384 tokens	256, 16, 1	2,711 MB
Qwen3.5-4B-3D-Coder_mixed_int4_ekv32768.litertlm (the 32K file)	32,768 tokens	1024, 256, 64, 16, 4, 1	2,768 MB

2026-11-03: both files' LLM metadata now carries the tool-call format of LiteRT-LM v0.19.0's models/qwen3_5 configuration (tool_call_config, supports_function_calling: true), so LiteRT-LM 0.19.0 returns the model's tool calls as tool calls. Nothing else in the LLM metadata changed (the thinking-off template, stop tokens, KV cache length, model type), and the TFLite, executor-metadata and tokenizer sections are byte-identical. sha256: the 16K file e86142bf… → 9c3c197e…, the 32K file defefa86… → 8b0414d5… (sizes unchanged). Checked with litert-lm 0.19.0 on a Mac CPU (Python API, greedy, a fresh process per prompt): 3 of 3 tool prompts per file come back as tool calls (the previous files returned the call as text), 3 plain replies are byte-identical to the previous files', thinking on and off behave as before (no reasoning either way). Procedure: qwen35_work/use_tool_call_config.sh in hf-to-litertlm. The other results on this card were measured on the earlier packs (Conversion notes).

Both files have the same weights recipe: int4 blockwise-32 (min-max) on the linear layers, int8 per-channel embedding and output layer (tied), and the 48 Gated DeltaNet in_proj_a / in_proj_b projections kept float32. Both declare fp32 GPU activations. They differ in the KV cache length and the prefill sizes. The numbers on this card were measured with LiteRT-LM 0.19.0 unless a row names another build; no earlier release was run for generation with these files.

Which file. The KV cache has to hold the prompt and the whole answer. Of the 213 validation scenes of the dataset, 172 fit 16,384 tokens with the source card's system prompt and their reference answer, and all 213 fit 32,768 (the longest needs 26,147). Use the 16K file unless prompt plus answer may pass 16,384 tokens (41 of the 213 validation scenes do): with the litert-lm package it decodes about 1.35 times as fast as the 32K file, on the Mac CPU and on the Mac GPU alike (Performance). In our probe (the 16K file, Mac CPU, litert-lm 0.19.0, max_num_tokens=600, 2026-11-02), an answer that reached the end of the KV cache stopped there, without an error and without </html>.

On a Mac, run the GPU only through LiteRT-LM's C++ CLI with --sampler_handles_input=false (litert_lm_advanced_main, built from the LiteRT-LM source; the release downloads do not include it). With that setting both files passed our scene check on the Mac GPU (4 of 7 scenes, the same 4 for both files; the PyTorch fp16 model passed 4 of the same 7), and the 16K file decodes 48.2 tok/s. Through the litert-lm package from PyPI, the Mac GPU runs the default path, which breaks long code: 6 of the 32K file's 10 scene replies had a syntax error in their script. On that PyPI GPU path the 16K file breaks the same way: all 3 of its scene replies we ran had a syntax error, and the same 3 scenes passed with the C++ CLI and the flag. The Swift package runs the same default path on the Mac GPU (we did not run these files through it), and no setting of those APIs changes the path (Known runtime issues).

With the litert-lm package on a Mac, use the CPU. The 32K file passed our scene check there (3 of 6 scenes; the PyTorch fp16 model 4 of the same 6) at 10.9 tok/s; the 16K file decodes faster (14.8 tok/s), answered 7 of the 8 short questions on the CPU, and its one scene there, the Usage example below, passed our check; we did not run its other scenes on the CPU.

Memory on the Mac CPU. In our benchmark without the weight cache (--cache no) the CPU engine's peak resident set was 14.0 GB with the 16K file and 25.9 GB with the 32K file (Performance); with the default cache setting, the Usage example's scene, run after the CLI line had written that cache, peaked at 4.9 GB with the 16K file.

On an iPhone 18 Pro, use the 16K file on the GPU (our benchmark app on the C API of the LiteRT-LM v0.19.0 release xcframework, Metal: 10.53 tok/s decode, 5.06 on the CPU). We ran the app with the com.apple.developer.kernel.increased-memory-limit entitlement; its memory limit was 6 GiB (6.44 GB), and the GPU engine's peak footprint was 5.92 GB (0.52 GB under the limit). The one scene we ran there (7,925 tokens in 928 s) came out byte-identical to the Mac C++ CLI's reply with --sampler_handles_input=false, although the iPhone runs the default GPU sampler path.

On Android we have no form to recommend yet. On a Pixel 11 Pro (16 GB) the 16K file's GPU engine builds and answers, but in the two runs where decoding slowed (a scene at 0.66 tok/s and the second benchmark run) the process's anonymous memory grew 6–7 MB/s and our test's memory guard stopped both, the scene after 105 tokens (the guard's rule: Performance); in the two faster decodes (3.7–3.9 tok/s) that memory stayed flat. On the CPU the 16K file decodes 2.98 tok/s, so a scene of 9,337 tokens (the median of our reference answers) takes about 52 minutes. On a Galaxy S26 (12 GB) we did not start it: in two tries the phone's free memory stayed under the 7,000,000 KiB (7.2 GB) our test waits for (Performance). No Mali phone ran these files (the Pixel 11 Pro's GPU is a PowerVR), and on 8 GB phones the GPU engine is not expected to fit: on the Pixel 11 Pro its creation took about 8.8–9.3 GB of MemAvailable (8.61–9.07 million KiB).

Correctness

• Scene check. 10 Three.js prompts: the 4 example prompts of the source card and 6 scenes of the dataset's validation split (one each at complexity levels 1, 3, 5, 6, 8 and 10), each with the source card's system prompt; greedy, thinking off, one fresh process per scene. A reply passes when it stops by itself with </html>, every <script type="module"> parses (node --check), and headless Chromium loads the page, with three.js 0.160.0 served locally, without a page error and draws a canvas that is not one flat color. The PyTorch fp16 model (transformers 5.14.1, Mac MPS) passed 7 of 10.

  Runtime and backend (Mac)	16K file	32K file
  C++ CLI litert_lm_advanced_main (LiteRT-LM main dab2e708), GPU, --sampler_handles_input=false	4 of 7, byte-identical to the 32K file's 7 replies	4 of 7 (fp16 model: 4 of the same 7)
  litert-lm 0.19.0 (PyPI), GPU, default path	0 of 3 (3 with a syntax error; the same 3 passed with the C++ CLI and the flag)	1 of 10 (6 with a syntax error, 1 repeating loop, 1 without </html>, 1 failed in the browser)
  litert-lm 0.19.0 (PyPI), CPU	1 of 1 (the Usage example's scene; the other 9 not run)	3 of 6 (fp16 model: 4 of the same 6)

  On the default GPU path the 32K file's six syntax errors were missing ) after argument list and Unexpected token ')' (two replies each), Invalid destructuring assignment target and Invalid or unexpected token. On the CPU the two replies with a syntax error declared one name twice, the kind of bug the source card names ("duplicate variable names").

• 8 short questions with one right answer each (two arithmetic questions, a capital, a translation, a decimal comparison, an opposite, the days of a week and a rhyme; one fresh process per question, greedy, 512 output tokens): both files 8/8 on the Mac GPU with the C++ CLI and --sampler_handles_input=false; with litert-lm 0.19.0 the 32K file 8/8 on the Mac GPU and CPU, the 16K file 7/8 on the Mac CPU ("thank you" in French: "Vous m'excusez"); on a Pixel 11 Pro CPU (the litert-lm-api 0.19.0 Android wheel) the 32K file 7/8. Every reply ended on a stop token.

• Multi-turn (3 turns: plant a name, do arithmetic, recall the name): the 32K file passes with litert-lm 0.19.0 on the Mac CPU and GPU (3 processes for each of the two KV-cache history settings, filter_channel_content_from_kv_cache true and false), both files pass with the C++ CLI on the Mac GPU (3 processes, the default history setting), and the 32K file passes on the Pixel 11 Pro CPU (6 of 6).

• Tokenizer: the tokenizer section of both files is the source repository's tokenizer.json (revision 196c03ff), byte for byte (sha256 a5cd9732…). With it LiteRT-LM 0.19.0 and 0.17.0 encode 22 of 22 probe texts (English, code, HTML, CJK, Hindi, Thai, Arabic with harakat, Tamil, emoji, special tokens, whitespace, the 10 prompts of the scene check and their system prompt) to the same ids as the tokenizers library, and decode them back.

• Chat template: for system, user and assistant messages the files' template renders byte-identically to the vendor's chat_template.jinja with enable_thinking=False in 7 of 8 conversation shapes, in jinja2 and in minijinja 2.14.0 (the template engine of the runtime), the three multi-turn shapes with a generation prompt included. The eighth shape is a history that ends with an assistant message and has no generation prompt: there the vendor's template writes the empty thought into that last message and these files' template does not.

Usage

pip install litert-lm==0.19.0
litert-lm run ./Qwen3.5-4B-3D-Coder_mixed_int4_ekv16384.litertlm --backend cpu --top-k 1 --prompt "Write a JavaScript function that returns the n-th Fibonacci number."

Python, with the source card's system prompt for Three.js scenes; the reply streams, which matters because a scene can take longer than 10 minutes (Known runtime issues):

import litert_lm

SYSTEM = """Use only the Three.js core ES module from
https://cdnjs.cloudflare.com/ajax/libs/three.js/0.160.0/three.module.min.js.
Do not import addons or legacy examples/js paths;
implement any camera controls directly inside the HTML."""

PROMPT = """Please create one self-contained HTML file with Three.js from cdnjs showing
a canal lock and weir complex with a central island."""

engine = litert_lm.Engine("Qwen3.5-4B-3D-Coder_mixed_int4_ekv16384.litertlm",
                          backend=litert_lm.Backend.CPU(), max_num_tokens=16384)
conversation = engine.create_conversation(system_message=SYSTEM,
                                          sampler_config=litert_lm.SamplerConfig(top_k=1))
reply = ""
for chunk in conversation.send_message_async(PROMPT):
    reply += str(chunk)  # the text of each streamed piece
start, end = reply.find("<!DOCTYPE html"), reply.rfind("</html>")
if start < 0 or end < start:
    raise SystemExit("The reply holds no complete HTML document.")
with open("scene.html", "w") as f:
    f.write(reply[start:end + len("</html>")])

• By default the runtime caches next to the model file. On the CPU the first run writes a 2.68 GB XNNPACK weight cache (Qwen3.5-4B-3D-Coder_mixed_int4_ekv16384.litertlm_<mtime>_<size>.xnnpack_cache); the Python example above, run after the CLI line, wrote no second file, and a later CLI run logged XNNPack weight cache loaded from that file (--verbose). On the GPU the 16K file got a 2.42 GB weight cache and a program cache (Known runtime issues). --cache no (CLI) or cache_dir=":nocache" (litert_lm.Engine) turns caching off.
• The source model's generation defaults are temperature 0.6, top-k 20 and top-p 0.95; every reply on this card was decoded greedily (top_k=1).
• The template carries the vendor's tool-calling branches, and since 2026-11-03 the files declare supports_function_calling: true and the tool-call format of LiteRT-LM v0.19.0's models/qwen3_5 configuration (llm_model_type is still generic_model), so LiteRT-LM 0.19.0 returns the model's tool calls as tool calls, with the name and the arguments (Python API, Mac CPU; the 2026-11-03 paragraph at the top). With tools in the conversation, LiteRT-LM 0.19.0 returns an error (INVALID_ARGUMENT, "Failed to parse tool calls from code block") instead of the reply when the model writes a closed <tool_call>…</tool_call> block without a <function=…> call; in 0.19.0 only the C++ API can turn this off (SetReturnErrorOnParseFailure). Seen on 2026-11-03 (Mac CPU, Python API) with Qwen3.5-4B's Mixed INT4 file, which carries the same tool-call configuration.

Performance

Every row: a 1,024-token prompt, then 256 decoded tokens, with the context set to 2,048 tokens. Mac Studio (Apple M4 Max, 128 GB, macOS 27.0).

litert-lm 0.19.0 from PyPI, litert-lm benchmark <file> -p 1024 -d 256 --max-num-tokens 2048 --runs 3 --backend <cpu|gpu> --cache no (one warm-up run, then the mean of 3); two processes per row, and the value is the mean of the two:

File	Backend	Prefill	Decode	TTFT	Init	Peak RSS
16K	CPU	93.2 tok/s	14.8 tok/s	11.1 s	12.7–16.9 s	14.0 GB
16K	GPU (WebGPU)	633 tok/s	48.6 tok/s	1.64 s	20.4–27.0 s	4.6 GB
32K	CPU	95.0 tok/s	10.9 tok/s	10.9 s	18.7–22.1 s	25.9 GB
32K	GPU (WebGPU)	651 tok/s	35.9 tok/s	1.60 s	55.0–59.0 s	6.6 GB

Init is the Init time the CLI prints, given as the range of the two processes; that value includes both Init Total and the phases inside it, so it counts engine creation twice (LiteRT-LM #4057). Peak RSS is the maximum resident set size /usr/bin/time -l reported for the process. The GPU rows run the default GPU path, the one that breaks long code (see the recommendation above).

LiteRT-LM's C++ CLI litert_lm_advanced_main, built from LiteRT-LM main at dab2e708 (2026-10-26, WebGPU), --backend=gpu --benchmark --benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256 --max_num_tokens=2048 --num_iterations=4 --disable_cache=true --use_session=true --async=false --input_prompt="How are you" --sampler_handles_input=false (iteration 1 is the warm-up, the value is the mean of iterations 2–4); one process per row:

File	Backend	Prefill	Decode	TTFT	Init Total
16K	GPU (WebGPU)	637 tok/s	48.2 tok/s	1.63 s	14.0 s
32K	GPU (WebGPU)	646 tok/s	30.4 tok/s	1.62 s	35.5 s

The 32K row's decode is the mean of 34.4, 34.7 and 22.3 tok/s: its last iteration dipped.

iPhone 18 Pro (iOS 27.2), the 16K file, a benchmark app on the C API of the LiteRT-LM v0.19.0 release xcframework (CLiteRTLM.xcframework, C API 1.0.0) signed with the increased-memory-limit and extended-virtual-addressing entitlements; 3 runs on one engine, the value is the mean of runs 2 and 3:

Backend	Prefill	Decode	TTFT	Peak footprint	Processes
GPU (Metal)	154.8 tok/s	10.53 tok/s	5.33, 9.07 s	5.92 GB	1
CPU	65.4 tok/s	5.06 tok/s	15.54, 16.18 s	2.99 GB	1 (started cool; an earlier process: 4.96 tok/s)

The GPU's decode went down run by run (14.55 in the warm-up, then 11.42 and 9.64 tok/s) while the phone reported no thermal pressure; the CPU's runs reached thermal state 2 (serious).

Pixel 11 Pro (Tensor G6, 16 GB, Android 17), the litert-lm-api 0.19.0 Android wheel from PyPI, Benchmark(model, backend, prefill_tokens=1024, decode_tokens=256, max_num_tokens=2048), 3 runs on one engine, the value is the mean of runs 2 and 3; one process per row; the phone capped its CPU for heat in most of the samples we took during each process (118 of 137 for the 16K file, 278 of 320 for the 32K file), so these are mostly capped speeds:

File	Backend	Prefill	Decode	TTFT
16K	CPU	23.6 tok/s	2.98 tok/s	40.2, 47.8 s
32K	CPU	19.5 tok/s	1.09 tok/s	50.9, 56.0 s

No Pixel 11 Pro GPU row: our test stops a run at the first reading of MemAvailable (the kernel's estimate, in /proc/meminfo, of the memory available to new work without swapping) under 400,000 KiB (read every 0.25 s). The 16K file's GPU engine built (it took about 8.8–9.3 GB of MemAvailable) and its first benchmark run finished (prefill 23.9 tok/s, decode 3.73 tok/s); in the second run decoding slowed, the process's anonymous memory grew 6.2 MB/s, and the test stopped it at 398,088 KiB. A scene stopped the same way after 105 tokens (decoding at 0.66 tok/s, anonymous memory +7.0 MB/s); during the faster decodes (3.73 and 3.90 tok/s) that memory stayed flat. The 32K file's GPU engine did not build: one GPU buffer of 2,147,483,648 bytes is over the phone's 1,073,741,824-byte limit.

No Galaxy S26 row: our test starts a run only once MemAvailable is at least 7,000,000 KiB, and in two tries on 2026-11-02 the phone stayed at 6,763,428–6,955,212 KiB, so no engine was started.

No NPU file. These files keep the weights int4 and int8 with float activations, which the Qualcomm NPU path of LiteRT-LM does not take, and an NPU form would need an int16 KV cache of about 0.54 GB at 16,384 tokens on top of the weights, over what one NPU session can address (on a Galaxy S26, the NPU form of another 4B model did not load even with a 2,048-token cache).

Known runtime issues

The Mac GPU's default path breaks long code. By default LiteRT-LM's GPU sampler also writes the next step's decode inputs (tokens, positions and mask; the log line Sampler will handle decode input tensors.). On the Mac GPU (WebGPU) that path breaks long replies of these files: with litert-lm 0.19.0 from PyPI, scene replies of both files had syntax errors in their module scripts, and of the 32K file's, one fell into a repeating loop and one ended without </html> (counts: Correctness). The C++ CLI litert_lm_advanced_main built from LiteRT-LM main at dab2e708 has the same default, and its flag --sampler_handles_input=false turns the sampler's writes off: with it, no scene reply of either file had a syntax error. In the v0.19.0 sources that setting is reachable only from that CLI and from the C++ engine settings; the litert-lm package (CLI and Python), the Swift package, the C API and the release's litert_lm_main have no setting that changes this path. The CPU does not use this path. The one scene we ran on an iPhone 18 Pro's GPU (Metal) went through the same default path and came out byte-identical to the Mac C++ CLI's reply with the flag.

On litert-lm 0.18.1 and 0.19.0, a blocking send_message that is still generating after 10 minutes fails with DEADLINE_EXCEEDED (a fixed timeout with no setting), and the generation it leaves running can crash the process. A scene reply of this model is often longer than that on a CPU or a phone (the iPhone 18 Pro's GPU took 928 s for one scene), so stream with send_message_async, which has no such limit, as the Python example does; the CLI's litert-lm run streams too.

Create a new Engine for each conversation. In Qwen3.5 files, LiteRT-LM reuses the prompt tokens a new conversation shares with the previous conversation on the same Engine, and does not reset the recurrent state of the Gated DeltaNet layers (LiteRT-LM #3165). We did not measure it on these files. A process that runs one conversation is unaffected.

GPU engine creation does not get faster after the first. On an iPhone 18 Pro (Metal), a Pixel 11 Pro (OpenCL) and the Mac (WebGPU), every engine creation of the 16K file that found a GPU program cache loaded 2 of its 4 programs from it and compiled the other 2 again (Deserialization failed: DATA_LOSS; LiteRT-LM 0.19.0 on all three: the iOS release xcframework through its C API, the litert-lm-api Android wheel and the litert-lm package; one such creation on each phone and two on the Mac, 2026-11-02; LiteRT-LM #4178). The 32K file was not checked. The GPU program cache file grows at each such creation: LiteRT-LM appends the rebuilt programs to the end of the file and never reclaims the old bytes (#4178); on the Mac, the 16K file's cache file was 440 MB after three engine creations. With one Engine per conversation, each new conversation pays the recompile and grows the cache file.

Conversion notes

Converted with litert-torch at commit 115a1360 with the Qwen3.5 patch of hf-to-litertlm (qwen35_work/qwen35_hybrid_litert_torch.patch, the patch of the litert-community Qwen3.5 files), transformers 5.14.1, torch 2.12.1 and ai-edge-quantizer 0.8.0; files packed with the litert-lm 0.18.0 CLI and re-packed on 2026-11-03 with the litert-lm 0.19.0 CLI (qwen35_work/use_tool_call_config.sh; reproduction: hf-to-litertlm).

• Thinking off. The vendor's template thinks by default, and every training scene carries a thinking trace. With thinking on, the PyTorch fp16 model (greedy) fell into a repeating loop inside its thought, without closing it, in both scenes we ran to the end (stopped at 2,304 and 2,048 tokens); the source model's report also counts 8 of its 18 calls cut off by reasoning loops before any HTML. So these files ship the thinking-off form only. The template is LiteRT-LM's models/qwen3_5/chat_template.jinja at 87ab9334 with two parts of its v0.19.0 copy, the tool_response content branch and the check that messages is not empty before it reads messages[0] (so a conversation preface without messages renders), and with its generation prompt fixed to the empty thought. The measurements on this card ran on an earlier pack of these files without that check, except the Usage example (its two commands, the CPU cache file they wrote, the scene's memory peak and the later run that read that cache) and the KV-end probe, which ran on the files of 2026-11-02 (the pack before the 2026-11-03 tool-call format). The weights, tokenizer and executor sections of the three packs are byte-identical, and for a conversation with at least one message the two templates render the same text.
• Two KV lengths. The 32K file holds every validation scene; the 16K file is a separate export of the same weights at 16,384 tokens, with the prefill sizes 256, 16 and 1 (on a Pixel 11 Pro the 32K file's 1,024-token prefill needs one GPU buffer of 2,147,483,648 bytes, over that GPU's 1,073,741,824-byte limit). Its greedy replies on the Mac GPU with --sampler_handles_input=false were byte-identical to the 32K file's on all 7 scenes, the 8 short questions and the 3-turn check.
• Stop tokens: <|endoftext|> (248044) and <|im_end|> (248046), the two ids of the source model's generation_config.json; no start token (the source tokenizer has none).
• GPU activations are declared fp32 in the files (prefer_activation_type = "fp32"). LiteRT-LM runs the text decoder with fp16 activations on the GPU by default, and on these files that answers nothing: with fp16 activations forced, the Mac GPU wrote ! up to the 512-token cap for both questions we ran.
• Quantization: int4 block scales floored at 2^-14 (none needed it here: the smallest of 111,411,200 scales is 1.06e-4); the 48 Gated DeltaNet in_proj_a / in_proj_b projections stay float32, as in the litert-community Qwen3.5 files since 2026-10-13. No other post-processing of the weights.
• Tokenizer: the source repository's tokenizer.json, byte for byte.
• The source checkpoint has no multi-token-prediction weights (its config.json still names one MTP layer), and none are converted.

License and changes

Distributed under Apache-2.0 (inherited from the source model; the upstream LICENSE file is included). Changes from the original work: the text decoder converted from safetensors fp16 to LiteRT flatbuffers and quantized as described above (vision tower not converted); fp32 GPU activations declared; tokenizer and a thinking-off chat template packed into the .litertlm files, with the tool-call format of LiteRT-LM's models/qwen3_5 configuration declared. No training or fine-tuning. This conversion was made independently of CCORE.