aufklarer/Qwen3.5-0.8B-Chat-CoreML

Qwen3.5-0.8B Chat — Core ML

On-device chat generation for Apple Silicon. Stateful Core ML export of Qwen/Qwen3.5-0.8B, a hybrid model that interleaves DeltaNet linear attention with full gated attention. Requires iOS 18+ / macOS 15+.

Ships precompiled .mlmodelc bundles only. On-device MLModel.compileModel() has been observed to produce different results per runtime — Mac, simulator and device — so compilation happens ahead of time and the compiled bundle is what is verified and published.

Earlier revisions of this repository were defective and should not be used. Two separate faults, both since fixed and both measured:

1. Final RMSNorm offset. Qwen3.5's last norm.weight is a zero-centred norm applied as x * (1.0 + weight). The exporter folded that +1 into the per-layer norms but not the final one, so the hidden state feeding the LM head was scaled by roughly zero-centred weights. Worth +19.0% perplexity on its own.
2. Core ML state write. Every MLState buffer was replaced with mul_(0) then add_(new), which emits two coreml_update_state ops and zeroes the buffer mid-graph while a value read from it is still live. The CPU backend tolerates this; the GPU backend stores a different tensor than the graph computed, corrupting the DeltaNet recurrent state so that agreement with the reference decayed as generation went on.

If you have a cached copy from before this revision, delete it and re-download.

Model

Detail	Value
Architecture	Qwen3.5 hybrid — 18 DeltaNet + 6 gated-attention layers, 24 total
Layer pattern	3 × DeltaNet then 1 × gated attention, repeated 6 times
Hidden size	1024
Vocabulary	248,320 (tied embedding / LM head)
Temporal state	DeltaNet conv1d + recurrent state and the attention KV cache, all carried as Core ML MLState
KV cache bound	2048 positions (fixed at export)
Weights	8-bit, per-block linear symmetric, block size 64
Precision	float16 compute
Minimum target	iOS 18 / macOS 15

Two Core ML models. embedding.mlmodelc is a stateless token-ID lookup; decoder.mlmodelc is a single-step decoder that owns all recurrent state, so the caller advances one token per prediction and never manages a cache.

Files

File	Size	Description
int8/decoder.mlmodelc	777.8 MB	Single-step decoder, 24 layers + LM head, all MLState
int8/embedding.mlmodelc	262.2 MB	Token-ID → embedding lookup
int8/tokenizer.json	20.0 MB	Tokenizer
int8/chat_template.jinja	7.8 kB	Chat template
int8/chat_config.json	1.1 kB	Architecture, quantization mode and special-token IDs
int8/tokenizer_config.json	1.1 kB	Tokenizer configuration
config.json	—	Repository-level metadata

Total download for the one variant: 1.06 GB. Earlier revisions were 2.04 GB because they shipped an uncompiled .mlpackage copy alongside the .mlmodelc; that copy is gone, and nothing downloaded it.

The directory is named int8/ because that is the path the Swift loader resolves and because the weights are 8-bit. The exact mode is per-block-64 linear quantization rather than per-tensor palettization; int8/chat_config.json records it as "quantization": "int8-linear".

Fidelity

These numbers say how closely the Core ML export reproduces the PyTorch reference it was converted from. They are an agreement measurement, not a quality claim — they say nothing about how good the model is, only that converting it did not change it.

Both models are fed the same five tokens rather than their own predictions, so a disagreement is a conversion fault and never sampling drift.

Check	Measured	Gate
Embedding, max (1 − cosine)	0.000017	≤ 0.001
Decoder, min per-step logit cosine	0.998305	≥ 0.99
Decoder, cosine change across steps	−0.000676 (improves)	must not fall > 0.02
Decoder, argmax agreement	5/5	≥ 4/5

Per step, logit cosine against the reference: 0.998305, 0.999888, 0.999711, 0.999856, 0.999689, with all five argmax predictions matching.

The compiled .mlmodelc published here was verified separately from the intermediate package it was compiled from, and agreed with it to every digit. That second pass exists because compilation is precisely the step that motivates shipping precompiled in the first place.

Measured with coremltools 9.0 and torch 2.13. The exporter's pinned pair, coremltools 8.3.0 and torch 2.3.1, reproduces the same result.

Why there is no 4-bit variant

4-bit was measured and rejected, not omitted for space. It scored 0/5 on the same check, with a minimum logit cosine of 0.448 and an embedding error 270× worse than 8-bit. It fails on the very first token, before any recurrent state has accumulated, so this is plain weight-compression loss on a model whose 248,320 × 1024 tied embedding dominates its parameter count.

There is no 5-bit variant either, for a different reason: Core ML has no 5-bit path. Linear quantization accepts int4/int8 and their unsigned variants, and palettization accepts 1, 2, 3, 4, 6 or 8 bits. The MLX build of this model does default to 5-bit, which is why the two differ.

Usage

import Qwen3Chat

let chat = try await Qwen35CoreMLChat.fromPretrained()
let reply = try chat.generate(messages: [
    ChatMessage(role: .user, content: "Explain linear attention in two sentences.")
])
print(reply)

fromPretrained() downloads only the .mlmodelc bundles and the tokenizer, then holds the decoder's MLState across the generation loop.

Source

Converted from Qwen/Qwen3.5-0.8B. Weights are Apache-2.0, as is this export.

Links

• speech-swift — Apple SDK
• Docs — install and CLI docs
• soniqo.audio — website
• blog — blog