d1-2B gestures on Core AI (GPU and Neural Engine)

A Core AI conversion of LiquidAI/d1-2B by Liquid AI, for the d1 gestures webcam demo: one frame in, hand gesture, facial expression, hand box and face box out, in one forward pass, on the GPU or the Neural Engine through Apple's Core AI (macOS 27, iOS / iPadOS 27).

The questions are inputs, not part of the graph. The functions take the frame plus the prompt (its token embeddings, its attention mask, the answer positions and the answer tokens' readout rows), so other questions need no re-export, as long as the prompt fits: 640 tokens with the image, 16 answer positions, 1,152 answer tokens (the demo's four questions use 578, 12 and 1,021). code/d1_common.py (prompt_inputs) builds them from code/questions.py and embed_tokens_fp16.npy.

File What it is
d1_gpu_prompt_fp16.aimodel/ the function for the GPU, fp16, 3.1 GB
d1_ane_prompt_fp16.aimodel/ the same function re-authored for the Neural Engine, fp16, 3.1 GB (Mac)
d1_ane_prompt_split_fp16/ the Neural Engine function in 4 pieces (part0…3.aimodel, run in order), for iPhone and iPad
embed_tokens_fp16.npy d1's token embedding table, fp16 [128000, 2048], to build the prompt inputs (500 MB)
calibration.json Liquid AI's calibration, unchanged from LiquidAI/d1-2B
code/ the export (d1_torch.py, d1_ane.py, export_*.py), the parity and speed checks, the host readout, the Swift worker
LICENSE the LFM Open License v1.0, as distributed with the original weights
d1_gpu_fp16.aimodel/, d1_ane_fp16.aimodel/, prompt.json the first version, with the four questions baked in; superseded, kept for old checkouts

Function

image   float32 [1, 3, 352, 640]   the 640x360 frame after the d1 image processor (resize to 640x352, (x/255 - 0.5)/0.5)
text    fp16 [420, 2048]           the prompt's token embeddings, without the 220 image positions (tokens 5…224)
attend  fp16 [640, 640]            1 where query token i may attend to token j (d1's decision mask, padding masked out)
slots   fp16 [16, 640]             one-hot: which token each answer position reads
head    fp16 [1152, 2048]          the embedding rows of the answer tokens, the readout
logits  [16, 1152]                 one row per answer position, one column per answer token (pad rows/columns ignored)

The demo's 12 rows are the answer positions: hand gesture (1), facial expression (1), hand box (present + 4 edges), face box (present + 4 edges). Turning them into answers (softmax over each option's tokens, Liquid AI's calibration, box medians) is d1's own readout; code/d1_common.py runs it through mlx-vlm's d1 code, so the answers have exactly the format of model.system_one(...). The iOS app has the same readout in Swift.

Run it with the Swift worker (cd code/swift && swift build -c release), the prompt inputs written as raw fp16 files plus a shapes.json (code/coreai_backend.py does this):

.build/release/d1worker --model ../../d1_ane_prompt_fp16.aimodel --compute ane --prompt /path/prompt   # or the GPU model, --compute gpu
# stdin, one JSON a line:  {"pixels": "/path/frame.f32"}   (raw float32 [1,3,352,640])  ->  {"logits": [...], "ms": ...}

or from Python with coreai.runtime (code/coreai_backend.py is the backend the demo uses).

Why 4 pieces on iOS

iOS's Neural Engine compiler fails on the whole 3.1 GB function ("Could not retrieve data from weight file") and Core AI then quietly runs it on the GPU. The split runs layers 0–6 with the vision tower, 7–12, 13–19, and 20–23 with the readout, passing the LM stream from one to the next; each piece takes attend, the first also image and text, the last also slots and head. On the Mac the pieces are as fast as the whole.

Measured (MacBook Pro M5 Max, macOS 27.3)

Parity with the MLX reference (LiquidAI/d1-2B-MLX-8bit, system_one) on 9 frames; a frame passes with the same top choice for hand and face, that choice's probability within 0.05, and every box edge within 3% of the frame:

GPU fp16 Neural Engine fp16 Neural Engine, 4 pieces
frames passing 9 / 9 8 / 9 8 / 9

The Neural Engine's one miss is a synthetic uniform mid-gray frame: same answers, a probability about 0.05 off. Every real frame passes.

Speed (per decision, preprocessing + model + readout): GPU about 50 ms, Neural Engine about 180 ms (whole or in pieces). Measured on the first, baked-prompt export (the prompt-input one runs at the same speed), with power for the whole Mac (macmon, 20 s of back-to-back decisions):

MLX 8-bit Core AI GPU Core AI Neural Engine
per decision 59 ms 55 ms 182 ms
first load (specialization, once) 12 s 56 s
cached load 0.6 s 0.3 s
Mac power above idle 131 W 98 W 17 W
energy per decision 8.4 J 6.0 J 3.2 J

The Neural Engine build runs entirely on the Neural Engine: its compiled plan is marked mps.fullyPlacedOnANE and mps.noGPUActivity. It is about 3Γ— slower than the GPU for this 2B model and uses about half the energy per decision.

Changes from the original

Modified by Ivan Fioravanti (notice under section 4(b) of the LFM Open License). The .aimodel files and embed_tokens_fp16.npy are derived from the LiquidAI/d1-2B weights:

  • re-implemented d1's decision pass as one fixed-shape PyTorch function (the prompt as inputs) and exported it with Apple's Core AI tools (coreai-torch 0.4.2, coreai-models export path); weights cast from bf16 to fp16;
  • the Neural Engine variant is re-authored for the Neural Engine (1Γ—1 convolutions, per-head attention, fp16 constants) and, to keep fp16 precision there, carries exact rescalings (the vision and text residual streams, the patch embedding and the branch outputs before scale-invariant norms are carried at larger scale), computes sigmoid through tanh, and computes each MLP in 8 chunks of its hidden channels (1.4Γ— faster there). The function is mathematically the same; numbers differ only by fp16 rounding; the split variant is the same function cut between decoder layers;
  • the token embedding table is shipped separately (embed_tokens_fp16.npy) to build the prompt inputs; d1's LM head is tied to it, so the readout rows come from the same table.

No retraining or fine-tuning. Int8 weights were tried and are not shipped: no GPU speedup; on the Neural Engine, per-channel int8 brought no speedup and per-block int8 does not compile for it (Core AI then silently runs it on the GPU).

License

Same license as the original weights: LFM Open License v1.0 (LICENSE, also at LiquidAI/d1-2B). As there, commercial use is licensed only for entities below US$10 million in annual revenue (section 5). "d1", "LFM" and "Liquid AI" are Liquid AI's names, used here only to describe where this comes from.

Citation

Cite the original model:

@article{liquidAI2026opend1,
  author  = {Liquid AI},
  title   = {Open d1: Edge decision models for text, vision, and audio},
  journal = {Liquid AI Blog},
  year    = {2026},
  note    = {https://www.liquid.ai/blog/open-d1},
}
@article{liquidAI2026LFM3,
  author  = {Liquid AI},
  title   = {Introducing LFM3: Every Modality for Specialized Intelligence},
  journal = {Liquid AI Blog},
  year    = {2026},
  note    = {https://www.liquid.ai/blog/lfm3},
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for ivanfioravanti/d1-2B-gestures-CoreAI

Finetuned
LiquidAI/d1-2B
Finetuned
(12)
this model