File size: 2,636 Bytes
f18bcfb afe3ae5 f18bcfb | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 | ---
library_name: aneforge
pipeline_tag: text-generation
tags:
- apple-neural-engine
- ane
- coreml-free
- on-device
- apple-silicon
- llama
- qwen
license: mit
---
# LLM text generation on the Apple Neural Engine (via ANEForge)
[ANEForge](https://github.com/sbryngelson/ANEForge) runs computation on the Apple Neural
Engine (ANE) directly, without CoreML. `aneforge.llm.from_pretrained` loads a **Llama- or
Qwen-family causal LM** from the Hub by repo id and runs prefill + resident-KV-cache decode
on the engine.
This is a usage card, not a re-hosted model: it points at the upstream weights and shows
how to run them on the ANE.
## Install
```sh
pip install aneforge
```
Apple Silicon, macOS 14+. `import aneforge` works anywhere; compiling and dispatching to the
ANE needs the hardware.
## Use
```python
from transformers import AutoTokenizer
import aneforge.llm as llm
name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0" # any Llama/Qwen-family causal LM by repo id
tok = AutoTokenizer.from_pretrained(name)
model = llm.from_pretrained(name) # compress="int8" / "int4" to stream quantized weights
prompt = "The Apple Neural Engine is"
ids = tok(prompt)["input_ids"]
out = model.generate(ids, max_new_tokens=32, eos_id=tok.eos_token_id)
print(prompt + tok.decode(out)) # decode runs on the ANE, KV cache resident across steps
```
`compress="int8"` / `"int4"` streams quantized weights from the engine's dequant path
(~4x smaller for int4, accuracy-gated). Larger models are bounded by the ANE program size;
small (~1B) models fit comfortably.
**Measured:** on an M5 Pro, `TinyLlama-1.1B-Chat-v1.0` (fp16) decodes at ~62 tok/s with the
KV cache resident on the engine, generating coherent text end to end. Decode throughput is
latency-bound and varies by chip; see the [ane-rooflines dataset](https://huggingface.co/datasets/aneforge/ane-rooflines).
## Why the ANE
The ANE is the fixed-function accelerator on every recent Apple device. ANEForge compiles the
decoder to a single ANE program and dispatches it through the same daemon and kernel-driver
stack Apple's own frameworks use, keeping the KV cache and weights resident across steps so
each decode step is one on-engine dispatch.
Cross-chip decode throughput is tracked in the
[ane-rooflines dataset](https://huggingface.co/datasets/aneforge/ane-rooflines).
## Links
- Code: https://github.com/sbryngelson/ANEForge
- Package: https://pypi.org/project/aneforge/
- Paper: https://arxiv.org/abs/2606.17090
## Cite
> Bryngelson, S. H. *ANEForge: Python for direct computation on the Apple Neural Engine.* arXiv:2606.17090 (2026).
|