card: Neural Engine bundle section
Browse files
README.md
CHANGED
|
@@ -86,6 +86,21 @@ conversation history; `streamResponse(to:)` yields tokens as they decode.
|
|
| 86 |
- Measure in Release — Debug is ~3× slower on per-token host work
|
| 87 |
<!-- gen-cards:use-it end -->
|
| 88 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 89 |
## Measured
|
| 90 |
|
| 91 |
| | decode | prefill | numerics | size |
|
|
|
|
| 86 |
- Measure in Release — Debug is ~3× slower on per-token host work
|
| 87 |
<!-- gen-cards:use-it end -->
|
| 88 |
|
| 89 |
+
## Neural Engine bundle (iOS static export, AOT h18p) — 2026-09-15
|
| 90 |
+
|
| 91 |
+
`ios-ane-h18p/` is Apple's stock `coreai.llm.export --platform iOS` static export of this checkpoint
|
| 92 |
+
(k-means **8-bit palettized, group 32** (`conversion/minicpm5_pal8_g32.yaml`); embeddings int8; static graphs `prompt_opt`/`extend` ×
|
| 93 |
+
contexts {256, 512, 1024, 2048, 4096} × query {8, 16, 64}), AOT-compiled with
|
| 94 |
+
`xcrun coreai-build compile --platform iOS --preferred-compute neural-engine --architecture h18p`
|
| 95 |
+
(31/31 ANE regions, 1.3 GB). It loads through Apple's `EngineFactory` → `StaticShapeEngine` unchanged
|
| 96 |
+
(iPhone 17-class devices). `ios-static/` is the same export before AOT — the portable IR; compile it
|
| 97 |
+
for another chip yourself. Apple's default 4-bit preset **fails** the same gate on this checkpoint (two margin-clear flips on a chat turn, unchanged with fp16 embeddings; `gate-minicpm5-1b-ane-device-4bit-FAIL.json`), so the 1B ships at 8 bits.
|
| 98 |
+
|
| 99 |
+
Gated on an iPhone 17 Pro (iOS 27.0) against the fp32 HF oracle: teacher-forced single-step sweep 24/24 + 10/10 + 16/16 and free-run greedy token-exact including the stop — transcript in the zoo,
|
| 100 |
+
`models/minicpm5-1b/gate-minicpm5-1b-ane-device.json`. Speed on the same phone (Apple llm-benchmark
|
| 101 |
+
method, 512-token prompt, 1024 generated, 5 trials, two back-to-back runs): decode **69.6 / 58.6 tok/s**, prefill 2710 / 2364 tok/s; footprint 1.3 GB (p128 / g256: decode 62.3). The Neural Engine is the power-efficient lane; these are
|
| 102 |
+
not a GPU comparison (different protocol from the `int8/` row above).
|
| 103 |
+
|
| 104 |
## Measured
|
| 105 |
|
| 106 |
| | decode | prefill | numerics | size |
|