papr-embed-code-1-0.6b ยท CoreML (Apple Neural Engine)
papr-embed-code-1-0.6b, a code-search embedding model, converted to CoreML so it runs on the Apple Neural Engine. A search query embeds in 17 ms on an M2 Pro, and the vectors match the original PyTorch model to cosine 0.9998.
Use it to give an on-device coding agent or editor semantic code search without a GPU, a server or network access. The vectors are interchangeable with the PyTorch, sentence-transformers and ONNX builds, so you can index in the cloud and search on device, or the other way around.
| Base model | papr-ai/papr-embed-code-1-0.6b (Qwen3-Embedding-0.6B, fine-tuned for code search) |
| Output | 1024-d, L2-normalized (pooling and normalization happen inside the model) |
| Precision | fp16 weights and compute, 1.2 GB |
| Lengths | three fixed-length functions: len_64, len_256, len_1024 |
| Requires | Apple Silicon (M1 or later), macOS 15+ (multifunction models) |
Speed and accuracy
Measured on a MacBook Pro M2 Pro (16 GB), macOS 15.6, ComputeUnit.CPU_AND_NE. Cosine is against PyTorch fp32 on the same token ids. Lowest cosine over the samples shown.
| Function | Use for | Latency p50 | p95 | Cosine vs fp32 (min) |
|---|---|---|---|---|
len_64 |
search queries | 16.7 ms | 17 ms | 0.9998 |
len_256 |
short files, snippets | 45 ms | 50 ms | 0.9997 |
len_1024 |
files up to ~1k tokens | 306 ms | 309 ms | 0.9997 |
For comparison, on the same Mac the int8 ONNX build on CPU takes about 350 ms per query (cosine 0.9998). 2,079 of the 2,083 ops run on the Neural Engine; the other 4 are the embedding lookup. Raw numbers are in results/coreml_parity.json.
Quick start (Python)
pip install coremltools tokenizers huggingface_hub numpy
import numpy as np, coremltools as ct
from huggingface_hub import snapshot_download
from tokenizers import Tokenizer
d = snapshot_download("papr-ai/papr-embed-code-1-0.6b-CoreML", local_dir="papr-embed-code-1-0.6b-CoreML")
tok = Tokenizer.from_file(f"{d}/tokenizer.json")
m = ct.models.MLModel(f"{d}/papr-embed-code-1-0.6b.mlpackage",
compute_units=ct.ComputeUnit.CPU_AND_NE, function_name="len_64")
INSTRUCT = "Given a code search query about a repository, retrieve the source code file that must be read or edited"
ids = tok.encode(f"Instruct: {INSTRUCT}\nQuery:retry an upload with backoff", add_special_tokens=False).ids + [151643]
x = np.zeros((1, 64), np.int32); x[0, :len(ids)] = ids
v = m.predict({"input_ids": x, "last_index": np.array([len(ids) - 1], np.int32)})["embedding"][0] # unit vector
examples/embed.py handles all three lengths, queries and documents.
How to call it
| Input | Type | Value |
|---|---|---|
input_ids |
int32 [1, L] |
token ids, `< |
last_index |
int32 [1] |
index of the `< |
Output: embedding, float [1, 1024], already unit-length. Cosine similarity is a dot product.
- Pick the smallest function whose
Lfits your token count plus one. - Queries get the instruction prefix
Instruct: <task>\nQuery:<text>. Documents (code files, chunks) are embedded bare. Any task sentence works, but use the same one for indexing and search tests. The one above is what we evaluate with. - For text over 1,023 tokens, either chunk it or embed it with the PyTorch / ONNX build. The vectors are compatible.
Swift:
let cfg = MLModelConfiguration()
cfg.computeUnits = .cpuAndNeuralEngine
cfg.functionName = "len_64"
let model = try MLModel(contentsOf: compiledURL, configuration: cfg) // compile once with MLModel.compileModel(at:)
Things that silently break it
- The end token. Append
<|endoftext|>(151643). With recenttransformers,tokenizer.eos_tokenresolves to<|im_end|>(151645). Those vectors look normal but land at cosine 0.5โ0.8 from the real ones, and retrieval quietly gets worse. - The Hugging Face cache. Download with
local_dir=(orhf download ... --local-dir). CoreML can't compile an.mlpackagethrough the symlinks in the default cache and fails withweight.bin doesn't exist. - The first load. The first load of each function compiles for the Neural Engine: about 30โ60 s per length, around 50 s for all three. macOS caches the compiled model, so later loads take seconds. Load the lengths you need once, in the background.
- Padding side. Pad on the right.
last_indexmust point at the real last token, not at padding.
How it was converted
convert_coreml.py (coremltools 9, PyTorch 2.x). Running Qwen3 on the Neural Engine correctly took three changes:
- Fixed input shapes. With enumerated or flexible shapes, CoreML planned every op on the CPU, even with
CPU_AND_NE. With one function per fixed length, the Neural Engine runs the model. - RMSNorm as
layer_norm([x, -x]). Qwen3's residual stream reaches about 6,000, soxยฒoverflows fp16 (max 65,504) inside a plain RMSNorm, and the output collapses to cosine 0.05. Concatenatingxand-xgives a zero-mean input, so LayerNorm's variance equalsmean(xยฒ). The two forms are mathematically identical, and the Neural Engine's fused LayerNorm stays in range. - Pooling and normalization inside the model. The model gathers the last token's hidden state and applies a pre-scaled L2 norm, so callers can't pool the wrong token.
We also built and tested an 8-bit-weight version but didn't ship it: 10.7 ms per query and half the size, but its minimum cosine dropped to 0.9986.
Files
| File | |
|---|---|
papr-embed-code-1-0.6b.mlpackage |
the model: multifunction, fp16 |
tokenizer.json, tokenizer_config.json |
Qwen3 tokenizer, same as the base model |
examples/embed.py |
runnable example |
convert_coreml.py |
conversion script |
results/coreml_parity.json |
latency and parity measurements |
Other builds
- papr-ai/papr-embed-code-1-0.6b: PyTorch, sentence-transformers, ONNX (int8, FastEmbed / Qdrant). Evaluation, training details and the full model card are there.
- papr-ai/papr-embed-code-1-4b: the larger model, for server-side indexing.
Used in Papr for local-first code search. Apache-2.0, same as the base model.
- Downloads last month
- 16
Model tree for papr-ai/papr-embed-code-1-0.6b-CoreML
Base model
Qwen/Qwen3-0.6B-Base