aneforge/llm-text-generation

LLM text generation on the Apple Neural Engine (via ANEForge)

ANEForge runs computation on the Apple Neural Engine (ANE) directly, without CoreML. aneforge.llm.from_pretrained loads a Llama- or Qwen-family causal LM from the Hub by repo id and runs prefill + resident-KV-cache decode on the engine.

This is a usage card, not a re-hosted model: it points at the upstream weights and shows how to run them on the ANE.

Install

pip install aneforge

Apple Silicon, macOS 14+. import aneforge works anywhere; compiling and dispatching to the ANE needs the hardware.

Use

from transformers import AutoTokenizer
import aneforge.llm as llm

name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"      # any Llama/Qwen-family causal LM by repo id
tok = AutoTokenizer.from_pretrained(name)
model = llm.from_pretrained(name)                 # compress="int8" / "int4" to stream quantized weights

prompt = "The Apple Neural Engine is"
ids = tok(prompt)["input_ids"]
out = model.generate(ids, max_new_tokens=32, eos_id=tok.eos_token_id)
print(prompt + tok.decode(out))                   # decode runs on the ANE, KV cache resident across steps

compress="int8" / "int4" streams quantized weights from the engine's dequant path (4x smaller for int4, accuracy-gated). Larger models are bounded by the ANE program size; small (1B) models fit comfortably.

Measured: on an M5 Pro, TinyLlama-1.1B-Chat-v1.0 (fp16) decodes at ~62 tok/s with the KV cache resident on the engine, generating coherent text end to end. Decode throughput is latency-bound and varies by chip; see the ane-rooflines dataset.

Why the ANE

The ANE is the fixed-function accelerator on every recent Apple device. ANEForge compiles the decoder to a single ANE program and dispatches it through the same daemon and kernel-driver stack Apple's own frameworks use, keeping the KV cache and weights resident across steps so each decode step is one on-engine dispatch.

Cross-chip decode throughput is tracked in the ane-rooflines dataset.

Links

• Code: https://github.com/sbryngelson/ANEForge
• Package: https://pypi.org/project/aneforge/
• Paper: https://arxiv.org/abs/2606.17090

Cite

Bryngelson, S. H. ANEForge: Python for direct computation on the Apple Neural Engine. arXiv:2606.17090 (2026).