papr-embed-code-1-0.6b ยท CoreML (Apple Neural Engine)

papr-embed-code-1-0.6b, a code-search embedding model, converted to CoreML so it runs on the Apple Neural Engine. A search query embeds in 17 ms on an M2 Pro, and the vectors match the original PyTorch model to cosine 0.9998.

Use it to give an on-device coding agent or editor semantic code search without a GPU, a server or network access. The vectors are interchangeable with the PyTorch, sentence-transformers and ONNX builds, so you can index in the cloud and search on device, or the other way around.

Base model papr-ai/papr-embed-code-1-0.6b (Qwen3-Embedding-0.6B, fine-tuned for code search)
Output 1024-d, L2-normalized (pooling and normalization happen inside the model)
Precision fp16 weights and compute, 1.2 GB
Lengths three fixed-length functions: len_64, len_256, len_1024
Requires Apple Silicon (M1 or later), macOS 15+ (multifunction models)

Speed and accuracy

Measured on a MacBook Pro M2 Pro (16 GB), macOS 15.6, ComputeUnit.CPU_AND_NE. Cosine is against PyTorch fp32 on the same token ids. Lowest cosine over the samples shown.

Function Use for Latency p50 p95 Cosine vs fp32 (min)
len_64 search queries 16.7 ms 17 ms 0.9998
len_256 short files, snippets 45 ms 50 ms 0.9997
len_1024 files up to ~1k tokens 306 ms 309 ms 0.9997

For comparison, on the same Mac the int8 ONNX build on CPU takes about 350 ms per query (cosine 0.9998). 2,079 of the 2,083 ops run on the Neural Engine; the other 4 are the embedding lookup. Raw numbers are in results/coreml_parity.json.

Quick start (Python)

pip install coremltools tokenizers huggingface_hub numpy
import numpy as np, coremltools as ct
from huggingface_hub import snapshot_download
from tokenizers import Tokenizer

d = snapshot_download("papr-ai/papr-embed-code-1-0.6b-CoreML", local_dir="papr-embed-code-1-0.6b-CoreML")
tok = Tokenizer.from_file(f"{d}/tokenizer.json")
m = ct.models.MLModel(f"{d}/papr-embed-code-1-0.6b.mlpackage",
                      compute_units=ct.ComputeUnit.CPU_AND_NE, function_name="len_64")

INSTRUCT = "Given a code search query about a repository, retrieve the source code file that must be read or edited"
ids = tok.encode(f"Instruct: {INSTRUCT}\nQuery:retry an upload with backoff", add_special_tokens=False).ids + [151643]
x = np.zeros((1, 64), np.int32); x[0, :len(ids)] = ids
v = m.predict({"input_ids": x, "last_index": np.array([len(ids) - 1], np.int32)})["embedding"][0]  # unit vector

examples/embed.py handles all three lengths, queries and documents.

How to call it

Input Type Value
input_ids int32 [1, L] token ids, `<
last_index int32 [1] index of the `<

Output: embedding, float [1, 1024], already unit-length. Cosine similarity is a dot product.

  1. Pick the smallest function whose L fits your token count plus one.
  2. Queries get the instruction prefix Instruct: <task>\nQuery:<text>. Documents (code files, chunks) are embedded bare. Any task sentence works, but use the same one for indexing and search tests. The one above is what we evaluate with.
  3. For text over 1,023 tokens, either chunk it or embed it with the PyTorch / ONNX build. The vectors are compatible.

Swift:

let cfg = MLModelConfiguration()
cfg.computeUnits = .cpuAndNeuralEngine
cfg.functionName = "len_64"
let model = try MLModel(contentsOf: compiledURL, configuration: cfg)  // compile once with MLModel.compileModel(at:)

Things that silently break it

  • The end token. Append <|endoftext|> (151643). With recent transformers, tokenizer.eos_token resolves to <|im_end|> (151645). Those vectors look normal but land at cosine 0.5โ€“0.8 from the real ones, and retrieval quietly gets worse.
  • The Hugging Face cache. Download with local_dir= (or hf download ... --local-dir). CoreML can't compile an .mlpackage through the symlinks in the default cache and fails with weight.bin doesn't exist.
  • The first load. The first load of each function compiles for the Neural Engine: about 30โ€“60 s per length, around 50 s for all three. macOS caches the compiled model, so later loads take seconds. Load the lengths you need once, in the background.
  • Padding side. Pad on the right. last_index must point at the real last token, not at padding.

How it was converted

convert_coreml.py (coremltools 9, PyTorch 2.x). Running Qwen3 on the Neural Engine correctly took three changes:

  • Fixed input shapes. With enumerated or flexible shapes, CoreML planned every op on the CPU, even with CPU_AND_NE. With one function per fixed length, the Neural Engine runs the model.
  • RMSNorm as layer_norm([x, -x]). Qwen3's residual stream reaches about 6,000, so xยฒ overflows fp16 (max 65,504) inside a plain RMSNorm, and the output collapses to cosine 0.05. Concatenating x and -x gives a zero-mean input, so LayerNorm's variance equals mean(xยฒ). The two forms are mathematically identical, and the Neural Engine's fused LayerNorm stays in range.
  • Pooling and normalization inside the model. The model gathers the last token's hidden state and applies a pre-scaled L2 norm, so callers can't pool the wrong token.

We also built and tested an 8-bit-weight version but didn't ship it: 10.7 ms per query and half the size, but its minimum cosine dropped to 0.9986.

Files

File
papr-embed-code-1-0.6b.mlpackage the model: multifunction, fp16
tokenizer.json, tokenizer_config.json Qwen3 tokenizer, same as the base model
examples/embed.py runnable example
convert_coreml.py conversion script
results/coreml_parity.json latency and parity measurements

Other builds

Used in Papr for local-first code search. Apache-2.0, same as the base model.

Downloads last month
16
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for papr-ai/papr-embed-code-1-0.6b-CoreML

Quantized
(1)
this model