File size: 3,674 Bytes
78a1f66 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 | ---
license: apache-2.0
base_model: vllm-sr/d3-edge
library_name: coreml
pipeline_tag: zero-shot-classification
tags:
- coreml
- mlx
- apple-silicon
- decision-model
- vision
- video
---
# d3-edge — Core ML and MLX
[vllm-sr/d3-edge](https://huggingface.co/vllm-sr/d3-edge) (0.6B, the camera-stream model of [Decision 3.0](https://huggingface.co/collections/vllm-sr/decision-30) by the vLLM Semantic Router team) for Apple silicon: text, image and video requests on device, as Core ML packages and as MLX weights. Same contract as upstream: one forward pass per question, the 255-way FP32 readout at the last prompt token, a probability for every option, no text generated.
## Files
| File | What |
|---|---|
| `d3_E256` / `d3_E640` / `d3_E1536` / `d3_E4096.mlpackage` | Core ML text graph (fp16 compute, int8 weights) per prompt-length bucket (prompts up to 4,096 tokens); input = token embeddings with image/video tokens spliced in |
| `vision_P320` / `vision_P1536` / `vision_P3072.mlpackage` | Core ML vision tower (fp32) per patch bucket; video frames go through it one frame pair at a time (block mask) |
| `embeddings.f16`, `pos_embed.f32`, `readout.f32`, `tokenizer.json`, `config.json`, `coreml_config.json` | host-side tables and config |
| `mlx/` | the upstream bf16 weights in the layout [mlx-vlm](https://github.com/Blaizzy/mlx-vlm) loads (`model.safetensors`, `config.json`) |
| `python/` | Python hosts: `d3_coreml_text.py`, `d3_coreml_image.py`, `d3_coreml_video.py` (coremltools) and `d3_mlx_video.py` (MLX, text and video) |
## Checked against upstream
Same weights as upstream (revision `fe71f8e1`). Answers that differ from the upstream PyTorch runtime (bf16, MPS), same requests:
| | Core ML | MLX |
|---|---|---|
| Text (typed-decisions TEST, 20 requests, 100 questions) | 1 | 0 |
| Images (8 head-camera parkour frames × 4 questions) | 0 / 32 | — |
| Video (2-second parkour windows × 8 yes/no questions, 192 answers) | 3 | 3 |
Speed on an M5 Pro (24 GB), one request at a time:
| | Core ML (`CPU_AND_GPU`) | MLX |
|---|---|---|
| per text question | ~20 ms | ~16 ms |
| per 2-second video look, 8 questions | 440 ms | **112 ms** |
| upstream PyTorch (MPS), same look | 651 ms | |
MLX is faster on video because it runs all of a request's questions as one batched forward at their exact lengths; the Core ML graph runs one padded pass per question. Neither places ops on the Neural Engine: with `CPU_AND_NE`, Core ML falls back to the CPU for every op of this architecture.
## Python
```python
# Core ML
from d3_coreml_video import D3CoreMLVideo # d3_coreml_text.D3CoreMLText, d3_coreml_image.D3CoreMLImage
m = D3CoreMLVideo("<vllm-sr/d3-edge snapshot>", "<this folder>", max_pixels=100_352)
m.system_one(state="A short clip from a head camera.", videos=[frames_2fps], questions={"high": {"type": "noul", "instructions": "Is the camera high above the ground?"}})
# MLX (needs mlx-vlm >= 0.7.8)
from d3_mlx_video import D3MLXVideo
m = D3MLXVideo("<vllm-sr/d3-edge snapshot>", "<this folder>/mlx", max_pixels=100_352)
m.system_one(state="...", questions={...}, videos=[frames_2fps]) # or videos=None for text
```
Both hosts build prompts with upstream's own runtime (`d3_runtime.py` in the upstream snapshot, run without weights), so prompts match upstream exactly. Note for MLX: mlx-vlm's Qwen3.5 builds layer kinds from `full_attention_interval`, but d3-edge's full-attention layers are 2, 5, 8, 9, 10 and 11 (`layer_types` in `config.json`); `d3_mlx_video.py` patches the layer constructor from `layer_types` before loading, and loads strictly. Without that, half the layers load no weights.
License: Apache-2.0, as upstream.
|