mlboydaisuke/Qwen3-Embedding-0.6B-ExecuTorch

Qwen3-Embedding-0.6B — ExecuTorch

Multilingual text embeddings from a causal LM: text in, one 1024-dimensional vector out. The largest and most capable embedding model on this shelf, and the only one that is a decoder rather than an encoder.

• Source: Qwen/Qwen3-Embedding-0.6B — 595.8M parameters, 28 Qwen3 layers, hidden 1024, 151669 vocabulary
• License: apache-2.0
• Input: input_ids and attention_mask, both [1, 256] int64
• Output: [1, 1024], last-token pooled over the mask and L2-normalised inside the graph

The pooling is in the graph, and it is a different pooling

1_Pooling/config.json says pooling_mode_lasttoken. Every other embedding model on this shelf averages over the sequence or takes CLS; this one takes the hidden state of the last real token, because a causal model's last position is the only one that has attended to the whole input.

	pooling	normalised
all-MiniLM-L6-v2	mean	yes
all-MiniLM-L12-v2	mean	yes
bge-small-en-v1.5	CLS	yes
multilingual-e5-base	mean	yes
multilingual-e5-large	mean	yes
Qwen3-Embedding-0.6B	last token	yes
paraphrase-multilingual-L12	mean	no

Pad on either side. Qwen's reference last_token_pool branches on attention_mask[:, -1] to tell left padding from right. That branch is data-dependent, so torch.export would freeze whichever way the example input happened to be padded and then silently return a pad token's hidden state for the other. This graph instead locates the last set position from the mask itself:

idx = (attention_mask * torch.arange(L)).argmax(dim=1)

Measured against the reference implementation on the same input, both padding sides, max_abs_diff 0.000e+00 — it selects the same token either way.

And the padding is free on both sides. Right padding cannot matter to a causal model: a pad at the end attends to nothing earlier. Left padding shifts every real token's RoPE position, which sounds like it should matter and does not — RoPE encodes differences, so a constant offset cancels. Measured: position_ids = arange + 229 and + 1000 both leave the vector unchanged to 5e-7, while a non-uniform shift (a gap inserted at position 10) moves it to cosine 0.9957. The null result is real, not a disconnected knob.

The prefix is not in the graph, and it is asymmetric

config_sentence_transformers.json gives the query a whole task instruction and the document the empty string:

query:    Instruct: Given a web search query, retrieve relevant passages that answer the query
          Query:{your query}
document: {your document, with nothing in front of it}

Unlike E5's query: /passage: , there is no symmetric mode documented. Putting the instruction on both sides, or on neither, is not what the model was trained for. It does not throw and does not look wrong — it retrieves worse.

Verification

build	file	size (MB)	Mac median (ms)*	XNNPACK takes	worst cosine vs eager	retrieval budget spent
XNNPACK fp32	embed_qwen3_embedding_0_6b_xnnpack_fp32.pte	2383.7	184.0	80.5%	1.000000	0%
XNNPACK fp16	embed_qwen3_embedding_0_6b_xnnpack_fp16.pte	1192.4	521.5	70.2%	0.999992	2%
Core ML fp32	embed_qwen3_embedding_0_6b_coreml_all.pte	1195.1	30.0	100%	0.999910	17%

*Mac arm64, median of 10, one 256-token sequence — a reference point for relative cost, not a device number. Eager fp32 on the same input is 179.7 ms, so the Core ML build is 6.0x eager, 100% delegated in one subgraph.

The XNNPACK column moved because the export stopped going through F.scaled_dot_product_attention. It decomposes to _safe_softmax, whose guard for fully-masked rows is six operations XNNPACK cannot take, once per attention block. This model is right-padded, so no query row can lose every key and the guard can never fire: with 249 of 256 positions masked, attn_implementation="eager" agrees with the sdpa graph at cosine 1.00000000 while taking delegation from 71.8% to 80.5% and fp32 from 203.9 ms to 184.0 ms.

Agreement with eager is the first check and it separates nothing here. The one that decides is the last column: rank all eight test sentences against each other, then ask whether this build's score error is smaller than the gap between the document a query retrieves and the runner-up. Every shipped build keeps all 8 top-1 results and spends well under half the available room.

The vectors also have to be useful, which agreement cannot show:

same language:    0.596 same meaning  vs  0.123 unrelated
across languages: 0.521 same meaning  vs  0.157 unrelated

Note how much wider that is than the E5 models on this shelf, which compress everything into 0.66–0.85. Both work; this one is easier to threshold against.

python convert/check_embed.py qwen3_embedding_0_6b fp32     # or fp16, int8, coreml

Not shipped: int8, and this is the first time it was close

int8 is 1063.9 MB — smaller than fp16's 1192.4 MB. No other embedding model on this shelf manages that. Dynamic int8 quantises the linear weights and leaves the token embedding table in fp32, so it only wins when that table is a small share of the file, and here it is: 151669 x 1024 = 621 MB of the 2384 MB model, 26%. Every other embedding model on the shelf is 46–69% table and loses to fp16 on size alone.

It is still not published, because the size question and the quality question are different questions:

worst retrieval score shift          0.0258
closest fp32 top-1 decision          0.0234

The error is 110% of the room available. On these eight sentences it happens not to change any result — but eight deliberately dissimilar sentences are the friendliest corpus a retrieval model will ever see, and that 0.0234 is an upper bound on what a real corpus leaves. For scale, the same eight sentences give multilingual-e5-large a top-1 decision of 0.0014, and its int8 build — whose score error is smaller than this one's — does change what one of the eight queries retrieves.

So the bar here is half the decision distance rather than all of it, and int8 misses it. Correlation reads 0.990459, which would also have flagged it; the retrieval measurement is what says by how much and why.

The attention rewrite above made int8 13% faster (174.8 ms to 151.7 ms) and 7 points more delegated (77.9% to 85.1%) and did not change this verdict — it moved the score error from 100% of the room to 110%. Speed and quality are separate questions here, and only one of them was ever the reason this build is unpublished.

torch.export -> to_edge_transform_and_lower(partitioner) -> .pte (conversion scripts: executorch-models)