README.md

---
library_name: aneforge
pipeline_tag: text-generation
tags:
- apple-neural-engine
- ane
- coreml-free
- on-device
- apple-silicon
- llama
- qwen
license: mit
---

# LLM text generation on the Apple Neural Engine (via ANEForge)

[ANEForge](https://github.com/sbryngelson/ANEForge) runs computation on the Apple Neural
Engine (ANE) directly, without CoreML. `aneforge.llm.from_pretrained` loads a **Llama- or
Qwen-family causal LM** from the Hub by repo id and runs prefill + resident-KV-cache decode
on the engine.

This is a usage card, not a re-hosted model: it points at the upstream weights and shows
how to run them on the ANE.

## Install

```sh
pip install aneforge
```

Apple Silicon, macOS 14+. `import aneforge` works anywhere; compiling and dispatching to the
ANE needs the hardware.

## Use

```python
from transformers import AutoTokenizer
import aneforge.llm as llm

name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"      # any Llama/Qwen-family causal LM by repo id
tok = AutoTokenizer.from_pretrained(name)
model = llm.from_pretrained(name)                 # compress="int8" / "int4" to stream quantized weights

prompt = "The Apple Neural Engine is"
ids = tok(prompt)["input_ids"]
out = model.generate(ids, max_new_tokens=32, eos_id=tok.eos_token_id)
print(prompt + tok.decode(out))                   # decode runs on the ANE, KV cache resident across steps
```

`compress="int8"` / `"int4"` streams quantized weights from the engine's dequant path
(~4x smaller for int4, accuracy-gated). Larger models are bounded by the ANE program size;
small (~1B) models fit comfortably.

**Measured:** on an M5 Pro, `TinyLlama-1.1B-Chat-v1.0` (fp16) decodes at ~62 tok/s with the
KV cache resident on the engine, generating coherent text end to end. Decode throughput is
latency-bound and varies by chip; see the [ane-rooflines dataset](https://huggingface.co/datasets/aneforge/ane-rooflines).

## Why the ANE

The ANE is the fixed-function accelerator on every recent Apple device. ANEForge compiles the
decoder to a single ANE program and dispatches it through the same daemon and kernel-driver
stack Apple's own frameworks use, keeping the KV cache and weights resident across steps so
each decode step is one on-engine dispatch.

Cross-chip decode throughput is tracked in the
[ane-rooflines dataset](https://huggingface.co/datasets/aneforge/ane-rooflines).

## Links

- Code: https://github.com/sbryngelson/ANEForge
- Package: https://pypi.org/project/aneforge/
- Paper: https://arxiv.org/abs/2606.17090

## Cite

> Bryngelson, S. H. *ANEForge: Python for direct computation on the Apple Neural Engine.* arXiv:2606.17090 (2026).