reranker / README.md

Usage card: run on the Apple Neural Engine via ANEForge

ddf5b84 verified about 19 hours ago

2.2 kB

library_name: aneforge
pipeline_tag: text-ranking
tags:
  - apple-neural-engine
  - ane
  - coreml-free
  - on-device
  - apple-silicon
  - reranker
  - cross-encoder
  - rag
license: mit

Reranking on the Apple Neural Engine (via ANEForge)

ANEForge runs computation on the Apple Neural Engine (ANE) directly, without CoreML. Its CrossEncoder loads a BERT-family cross-encoder / reranker from the Hub by repo id and runs the transformer on the engine, matching the sentence_transformers.CrossEncoder API.

This is a usage card, not a re-hosted model: it points at the upstream weights and shows how to run them on the ANE.

Install

pip install aneforge

Apple Silicon, macOS 14+.

Use

from aneforge.sentence_transformers import CrossEncoder

ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")   # any BERT-family cross-encoder
query = "How many people live in Berlin?"
passages = [
    "Berlin has about 3.85 million residents.",
    "Paris is the capital of France.",
]
scores = ce.predict([(query, p) for p in passages])   # higher = more relevant; transformer on the ANE
ranked = sorted(zip(scores, passages), reverse=True)

Measured

On an M5 Pro, cross-encoder/ms-marco-MiniLM-L-6-v2 scores a (query, passage) pair in ~0.8 ms, matching the Hugging Face reference ranking (relerr 5e-4).

Scope

BERT-family cross-encoders with a pooler + classifier head (e.g. cross-encoder/ms-marco-MiniLM-L-6-v2, -L-12-v2). RoBERTa/XLM-R rerankers (bge-reranker) are a work in progress -- see the repo issues.

Why the ANE

The ANE is the fixed-function accelerator on every recent Apple device. In production it is reachable only through CoreML, which can silently fall back to CPU/GPU; ANEForge compiles the transformer to a single ANE program and dispatches it through the same daemon and kernel-driver stack Apple's own frameworks use.

Cite

Bryngelson, S. H. ANEForge: Python for direct computation on the Apple Neural Engine. arXiv:2606.17090 (2026).

aneforge
/

reranker