Add 768x480 vision towers (Qwen3.5, Gemma 4); update 512x320 Qwen tower
Browse filesNew Core ML model packages:
- qwen3_5_mlx_4b_vision_vid_768x480_w8 Qwen3.5-4B vision tower @ 768x480
- gemma4_mlx_e4b_vision_768x480_w8 Gemma 4 (e4b) vision tower @ 768x480
Re-convert the existing Qwen3.5-4B vision tower @ 512x320 to the
tensor_video_rgb variant: inputs are now two planar RGB fp16 tensors
(1,3,320,512) in 0..255 with x/127.5-1 normalisation baked into the graph,
replacing the previous BGRA CVPixelBuffer image inputs. The new 768x480
Qwen build shares that I/O contract.
All new/updated towers are Apache-2.0 (Gemma 4 weights included -- unlike
earlier Gemma releases). Update README.md (I/O, shapes, precision,
conversion toolchain, SHA-256 checksums) and add a Gemma 4 upstream
component entry to THIRD_PARTY_LICENSES.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- README.md +47 -11
- THIRD_PARTY_LICENSES.md +42 -7
- gemma4_mlx_e4b_vision_768x480_w8.tar +3 -0
- qwen3_5_mlx_4b_vision_vid_512x320_w8.tar +2 -2
- qwen3_5_mlx_4b_vision_vid_768x480_w8.tar +3 -0
|
@@ -21,7 +21,7 @@ VPIPE runtime can fetch a pinned, ready-to-run artifact instead of
|
|
| 21 |
re-running the conversion.
|
| 22 |
|
| 23 |
> **Licensing note:** the bundled models are **not** all under a single
|
| 24 |
-
> license.
|
| 25 |
> [THIRD_PARTY_LICENSES.md](THIRD_PARTY_LICENSES.md) for the per-model
|
| 26 |
> attribution and verbatim license texts.
|
| 27 |
|
|
@@ -45,26 +45,60 @@ written `WxH`.
|
|
| 45 |
|
| 46 |
| Archive | Model | Task | License |
|
| 47 |
| --- | --- | --- | --- |
|
| 48 |
-
| `qwen3_5_mlx_4b_vision_vid_512x320_w8.tar` | Qwen3.5-4B vision tower (Core ML) | Vision encoder for a multimodal LLM | Apache-2.0 |
|
|
|
|
|
|
|
| 49 |
| `yolox_l_1024x640_w8.tar` | YOLOX-L (Core ML) | Object detection | Apache-2.0 |
|
| 50 |
| `silero-vad-unified-v6.tar` | Silero VAD v6, unified (Core ML) | Voice activity detection | MIT |
|
| 51 |
|
| 52 |
### `qwen3_5_mlx_4b_vision_vid_512x320_w8`
|
| 53 |
|
| 54 |
-
Qwen3.5-4B vision tower (`variant=
|
| 55 |
-
turns video frames into the embedding sequence consumed by
|
| 56 |
-
language model.
|
| 57 |
|
| 58 |
-
- **Task:**
|
| 59 |
- **Inputs:** `image0`, `image1` — the two consecutive video frames that
|
| 60 |
-
form one temporal patch, each
|
| 61 |
-
`
|
| 62 |
-
`-1`
|
| 63 |
-
- **Output:** `image_features` — vision features of shape `(160, 2560)`.
|
| 64 |
- **Precision:** 8-bit weight quantization, FP16 compute.
|
| 65 |
- **Upstream:** [Qwen3.5](https://huggingface.co/Qwen) (Qwen team, Alibaba Cloud) — Apache-2.0.
|
| 66 |
- **Conversion:** coremltools 9.0 from TorchScript (`torch==2.12.0`).
|
| 67 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 68 |
### `yolox_l_1024x640_w8`
|
| 69 |
|
| 70 |
YOLOX-L anchor-free object detector, converted to Core ML at a fixed
|
|
@@ -100,7 +134,9 @@ Core ML conversion by FluidInference.
|
|
| 100 |
SHA-256 checksums of the archives:
|
| 101 |
|
| 102 |
```
|
| 103 |
-
|
|
|
|
|
|
|
| 104 |
7f7bf85947bb228f978470284e16a9d00c906a6cb07359f9a51c68806fa9c609 yolox_l_1024x640_w8.tar
|
| 105 |
ee14d006d1cea87f15a4f0ac5f3e1e00b80a10c40eca855e8ff6741f389b4423 silero-vad-unified-v6.tar
|
| 106 |
```
|
|
|
|
| 21 |
re-running the conversion.
|
| 22 |
|
| 23 |
> **Licensing note:** the bundled models are **not** all under a single
|
| 24 |
+
> license. Four are Apache-2.0 and one (Silero VAD) is MIT. See
|
| 25 |
> [THIRD_PARTY_LICENSES.md](THIRD_PARTY_LICENSES.md) for the per-model
|
| 26 |
> attribution and verbatim license texts.
|
| 27 |
|
|
|
|
| 45 |
|
| 46 |
| Archive | Model | Task | License |
|
| 47 |
| --- | --- | --- | --- |
|
| 48 |
+
| `qwen3_5_mlx_4b_vision_vid_512x320_w8.tar` | Qwen3.5-4B vision tower @ 512×320 (Core ML) | Vision encoder for a multimodal LLM | Apache-2.0 |
|
| 49 |
+
| `qwen3_5_mlx_4b_vision_vid_768x480_w8.tar` | Qwen3.5-4B vision tower @ 768×480 (Core ML) | Vision encoder for a multimodal LLM | Apache-2.0 |
|
| 50 |
+
| `gemma4_mlx_e4b_vision_768x480_w8.tar` | Gemma 4 (e4b) vision tower @ 768×480 (Core ML) | Vision encoder for a multimodal LLM | Apache-2.0 |
|
| 51 |
| `yolox_l_1024x640_w8.tar` | YOLOX-L (Core ML) | Object detection | Apache-2.0 |
|
| 52 |
| `silero-vad-unified-v6.tar` | Silero VAD v6, unified (Core ML) | Voice activity detection | MIT |
|
| 53 |
|
| 54 |
### `qwen3_5_mlx_4b_vision_vid_512x320_w8`
|
| 55 |
|
| 56 |
+
Qwen3.5-4B vision tower (`variant=tensor_video_rgb`), the vision encoder
|
| 57 |
+
that turns a pair of video frames into the embedding sequence consumed by
|
| 58 |
+
the Qwen3.5-4B language model.
|
| 59 |
|
| 60 |
+
- **Task:** video → vision feature embeddings (multimodal LLM front-end).
|
| 61 |
- **Inputs:** `image0`, `image1` — the two consecutive video frames that
|
| 62 |
+
form one temporal patch, each a planar RGB fp16 tensor `(1, 3, 320, 512)`
|
| 63 |
+
(`1×3×H×W`, i.e. 512×320 W×H) with raw pixels in `0..255`. The
|
| 64 |
+
`x/127.5 - 1` normalisation is baked into the graph (applied on the ANE).
|
| 65 |
+
- **Output:** `image_features` — vision features of shape `(160, 2560)`, fp16.
|
| 66 |
- **Precision:** 8-bit weight quantization, FP16 compute.
|
| 67 |
- **Upstream:** [Qwen3.5](https://huggingface.co/Qwen) (Qwen team, Alibaba Cloud) — Apache-2.0.
|
| 68 |
- **Conversion:** coremltools 9.0 from TorchScript (`torch==2.12.0`).
|
| 69 |
|
| 70 |
+
### `qwen3_5_mlx_4b_vision_vid_768x480_w8`
|
| 71 |
+
|
| 72 |
+
The same Qwen3.5-4B vision tower (`variant=tensor_video_rgb`) at a higher
|
| 73 |
+
768×480 input — identical I/O contract to the 512×320 build, with a longer
|
| 74 |
+
output token sequence.
|
| 75 |
+
|
| 76 |
+
- **Task:** video → vision feature embeddings (multimodal LLM front-end).
|
| 77 |
+
- **Inputs:** `image0`, `image1` — the two consecutive video frames that
|
| 78 |
+
form one temporal patch, each a planar RGB fp16 tensor `(1, 3, 480, 768)`
|
| 79 |
+
(`1×3×H×W`, i.e. 768×480 W×H) with raw pixels in `0..255`. The
|
| 80 |
+
`x/127.5 - 1` normalisation is baked into the graph (applied on the ANE).
|
| 81 |
+
- **Output:** `image_features` — vision features of shape `(360, 2560)`, fp16.
|
| 82 |
+
- **Precision:** 8-bit weight quantization, FP16 compute.
|
| 83 |
+
- **Upstream:** [Qwen3.5](https://huggingface.co/Qwen) (Qwen team, Alibaba Cloud) — Apache-2.0.
|
| 84 |
+
- **Conversion:** coremltools 9.0 from TorchScript (`torch==2.12.0`).
|
| 85 |
+
|
| 86 |
+
### `gemma4_mlx_e4b_vision_768x480_w8`
|
| 87 |
+
|
| 88 |
+
Gemma 4 (e4b) vision tower, converted to Core ML. It pools and projects the
|
| 89 |
+
visual features into the language model's 2560-dim text embedding space,
|
| 90 |
+
emitting "soft tokens" ready to splice into the Gemma 4 prompt.
|
| 91 |
+
|
| 92 |
+
- **Task:** image → LLM soft tokens (multimodal LLM front-end).
|
| 93 |
+
- **Input:** `image` — a single planar RGB fp16 MLMultiArray
|
| 94 |
+
`[1, 3, 480, 768]` (`1×3×H×W`, i.e. 768×480 W×H), raw pixels in `0..255`
|
| 95 |
+
(channel order R, G, B). Preprocessing is baked in: `x/255` then
|
| 96 |
+
`2*(x - 0.5)`.
|
| 97 |
+
- **Output:** `soft_tokens` — shape `[1, 160, 2560]`, fp16.
|
| 98 |
+
- **Precision:** int8 weight-only quantization (per-block), FP16 compute.
|
| 99 |
+
- **Upstream:** [Gemma 4 (e4b)](https://huggingface.co/google/gemma-4-E4B) (Google DeepMind) — Apache-2.0.
|
| 100 |
+
- **Conversion:** coremltools 9.0 from TorchScript (`torch==2.12.1`).
|
| 101 |
+
|
| 102 |
### `yolox_l_1024x640_w8`
|
| 103 |
|
| 104 |
YOLOX-L anchor-free object detector, converted to Core ML at a fixed
|
|
|
|
| 134 |
SHA-256 checksums of the archives:
|
| 135 |
|
| 136 |
```
|
| 137 |
+
ed27924a97a873a019d1c8a1e1f9826bd30da2588683843d6c59cfb0b2289f30 qwen3_5_mlx_4b_vision_vid_512x320_w8.tar
|
| 138 |
+
3e928ea0c3e6848896278b68436d46f1aa306d95d8ca4a1faadfad354d705b87 qwen3_5_mlx_4b_vision_vid_768x480_w8.tar
|
| 139 |
+
17bebe9fdb416a760df4d00ed39c7edf296ba768d54139c8183c1323da646d3e gemma4_mlx_e4b_vision_768x480_w8.tar
|
| 140 |
7f7bf85947bb228f978470284e16a9d00c906a6cb07359f9a51c68806fa9c609 yolox_l_1024x640_w8.tar
|
| 141 |
ee14d006d1cea87f15a4f0ac5f3e1e00b80a10c40eca855e8ff6741f389b4423 silero-vad-unified-v6.tar
|
| 142 |
```
|
|
@@ -6,13 +6,16 @@ component and the archive it ships in, identifies the upstream source,
|
|
| 6 |
reproduces the upstream copyright statement, and points to the verbatim
|
| 7 |
license text required by that component's license.
|
| 8 |
|
| 9 |
-
|
| 10 |
-
the
|
| 11 |
-
license texts are reproduced in the appendices at the end of this
|
|
|
|
| 12 |
|
| 13 |
| Archive | Component | License |
|
| 14 |
| --- | --- | --- |
|
| 15 |
| `qwen3_5_mlx_4b_vision_vid_512x320_w8.tar` | Qwen3.5 vision tower | Apache-2.0 ([Appendix A](#appendix-a-apache-license-version-20)) |
|
|
|
|
|
|
|
| 16 |
| `yolox_l_1024x640_w8.tar` | YOLOX | Apache-2.0 ([Appendix A](#appendix-a-apache-license-version-20)) |
|
| 17 |
| `silero-vad-unified-v6.tar` | Silero VAD (FluidInference Core ML conversion) | MIT ([Appendix B](#appendix-b-mit-license)) |
|
| 18 |
|
|
@@ -20,10 +23,11 @@ license texts are reproduced in the appendices at the end of this file.
|
|
| 20 |
|
| 21 |
## Qwen3.5 (vision tower)
|
| 22 |
|
| 23 |
-
The
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
|
|
|
| 27 |
8-bit-quantized and the graph is exported to Core ML; the model is otherwise
|
| 28 |
a derivative of the upstream Qwen3.5 weights.
|
| 29 |
|
|
@@ -46,6 +50,37 @@ upstream at
|
|
| 46 |
|
| 47 |
---
|
| 48 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 49 |
## YOLOX
|
| 50 |
|
| 51 |
The archive `yolox_l_1024x640_w8.tar` contains a Core ML conversion of
|
|
|
|
| 6 |
reproduces the upstream copyright statement, and points to the verbatim
|
| 7 |
license text required by that component's license.
|
| 8 |
|
| 9 |
+
Every model except one is licensed under the Apache License, Version 2.0;
|
| 10 |
+
the remaining one (Silero VAD) is licensed under the MIT License. The
|
| 11 |
+
verbatim license texts are reproduced in the appendices at the end of this
|
| 12 |
+
file.
|
| 13 |
|
| 14 |
| Archive | Component | License |
|
| 15 |
| --- | --- | --- |
|
| 16 |
| `qwen3_5_mlx_4b_vision_vid_512x320_w8.tar` | Qwen3.5 vision tower | Apache-2.0 ([Appendix A](#appendix-a-apache-license-version-20)) |
|
| 17 |
+
| `qwen3_5_mlx_4b_vision_vid_768x480_w8.tar` | Qwen3.5 vision tower | Apache-2.0 ([Appendix A](#appendix-a-apache-license-version-20)) |
|
| 18 |
+
| `gemma4_mlx_e4b_vision_768x480_w8.tar` | Gemma 4 vision tower | Apache-2.0 ([Appendix A](#appendix-a-apache-license-version-20)) |
|
| 19 |
| `yolox_l_1024x640_w8.tar` | YOLOX | Apache-2.0 ([Appendix A](#appendix-a-apache-license-version-20)) |
|
| 20 |
| `silero-vad-unified-v6.tar` | Silero VAD (FluidInference Core ML conversion) | MIT ([Appendix B](#appendix-b-mit-license)) |
|
| 21 |
|
|
|
|
| 23 |
|
| 24 |
## Qwen3.5 (vision tower)
|
| 25 |
|
| 26 |
+
The archives `qwen3_5_mlx_4b_vision_vid_512x320_w8.tar` and
|
| 27 |
+
`qwen3_5_mlx_4b_vision_vid_768x480_w8.tar` contain Core ML conversions of
|
| 28 |
+
the **Qwen3.5-4B** vision tower (the multimodal vision encoder) at two input
|
| 29 |
+
resolutions, derived from the Qwen3.5 model series published by the Qwen
|
| 30 |
+
team at Alibaba Cloud (<https://huggingface.co/Qwen>). The weights are
|
| 31 |
8-bit-quantized and the graph is exported to Core ML; the model is otherwise
|
| 32 |
a derivative of the upstream Qwen3.5 weights.
|
| 33 |
|
|
|
|
| 50 |
|
| 51 |
---
|
| 52 |
|
| 53 |
+
## Gemma 4 (vision tower)
|
| 54 |
+
|
| 55 |
+
The archive `gemma4_mlx_e4b_vision_768x480_w8.tar` contains a Core ML
|
| 56 |
+
conversion of the **Gemma 4 (e4b)** vision tower (the multimodal vision
|
| 57 |
+
encoder that projects image features into the language model's text
|
| 58 |
+
embedding space), derived from the Gemma 4 model series published by Google
|
| 59 |
+
DeepMind (<https://huggingface.co/google/gemma-4-E4B>). The weights are
|
| 60 |
+
int8 weight-only quantized and the graph is exported to Core ML; the model
|
| 61 |
+
is otherwise a derivative of the upstream Gemma 4 weights.
|
| 62 |
+
|
| 63 |
+
Gemma 4 open-weight models are distributed under the Apache License,
|
| 64 |
+
Version 2.0. (This differs from earlier Gemma releases, which used the
|
| 65 |
+
custom Gemma Terms of Use.)
|
| 66 |
+
|
| 67 |
+
### Copyright notice
|
| 68 |
+
|
| 69 |
+
> Copyright Google LLC
|
| 70 |
+
|
| 71 |
+
The upstream model card declares the license as Apache-2.0; Google does not
|
| 72 |
+
publish a per-file copyright line for the Gemma 4 weights.
|
| 73 |
+
|
| 74 |
+
### License
|
| 75 |
+
|
| 76 |
+
Apache License, Version 2.0. The verbatim text is reproduced in
|
| 77 |
+
[Appendix A](#appendix-a-apache-license-version-20), and is available
|
| 78 |
+
upstream at
|
| 79 |
+
<https://ai.google.dev/gemma/docs/gemma_4_license> and
|
| 80 |
+
<https://www.apache.org/licenses/LICENSE-2.0>.
|
| 81 |
+
|
| 82 |
+
---
|
| 83 |
+
|
| 84 |
## YOLOX
|
| 85 |
|
| 86 |
The archive `yolox_l_1024x640_w8.tar` contains a Core ML conversion of
|
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:17bebe9fdb416a760df4d00ed39c7edf296ba768d54139c8183c1323da646d3e
|
| 3 |
+
size 165332992
|
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ed27924a97a873a019d1c8a1e1f9826bd30da2588683843d6c59cfb0b2289f30
|
| 3 |
+
size 333078016
|
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3e928ea0c3e6848896278b68436d46f1aa306d95d8ca4a1faadfad354d705b87
|
| 3 |
+
size 334001152
|