README.md

---
library_name: aneforge
pipeline_tag: text-ranking
tags:
- apple-neural-engine
- ane
- coreml-free
- on-device
- apple-silicon
- reranker
- cross-encoder
- rag
license: mit
---

# Reranking on the Apple Neural Engine (via ANEForge)

[ANEForge](https://github.com/sbryngelson/ANEForge) runs computation on the Apple Neural
Engine (ANE) directly, without CoreML. Its `CrossEncoder` loads a BERT-family cross-encoder /
reranker from the Hub by repo id and runs the transformer on the engine, matching the
`sentence_transformers.CrossEncoder` API.

This is a usage card, not a re-hosted model: it points at the upstream weights and shows how
to run them on the ANE.

## Install

```sh
pip install aneforge
```

Apple Silicon, macOS 14+.

## Use

```python
from aneforge.sentence_transformers import CrossEncoder

ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")   # any BERT-family cross-encoder
query = "How many people live in Berlin?"
passages = [
    "Berlin has about 3.85 million residents.",
    "Paris is the capital of France.",
]
scores = ce.predict([(query, p) for p in passages])   # higher = more relevant; transformer on the ANE
ranked = sorted(zip(scores, passages), reverse=True)
```

## Measured

On an M5 Pro, `cross-encoder/ms-marco-MiniLM-L-6-v2` scores a (query, passage) pair in
**~0.8 ms**, matching the Hugging Face reference ranking (relerr 5e-4).

## Scope

BERT-family cross-encoders with a pooler + classifier head (e.g. `cross-encoder/ms-marco-MiniLM-L-6-v2`,
`-L-12-v2`). RoBERTa/XLM-R rerankers (bge-reranker) are a work in progress -- see the repo issues.

## Why the ANE

The ANE is the fixed-function accelerator on every recent Apple device. In production it is
reachable only through CoreML, which can silently fall back to CPU/GPU; ANEForge compiles the
transformer to a single ANE program and dispatches it through the same daemon and kernel-driver
stack Apple's own frameworks use.

## Links

- Code: https://github.com/sbryngelson/ANEForge
- Package: https://pypi.org/project/aneforge/
- Paper: https://arxiv.org/abs/2606.17090

## Cite

> Bryngelson, S. H. *ANEForge: Python for direct computation on the Apple Neural Engine.* arXiv:2606.17090 (2026).