tgo-app-dev Claude Opus 4.8 commited on
Commit
14f26cd
·
1 Parent(s): 776766b

Add 768x480 vision towers (Qwen3.5, Gemma 4); update 512x320 Qwen tower

Browse files

New Core ML model packages:
- qwen3_5_mlx_4b_vision_vid_768x480_w8 Qwen3.5-4B vision tower @ 768x480
- gemma4_mlx_e4b_vision_768x480_w8 Gemma 4 (e4b) vision tower @ 768x480

Re-convert the existing Qwen3.5-4B vision tower @ 512x320 to the
tensor_video_rgb variant: inputs are now two planar RGB fp16 tensors
(1,3,320,512) in 0..255 with x/127.5-1 normalisation baked into the graph,
replacing the previous BGRA CVPixelBuffer image inputs. The new 768x480
Qwen build shares that I/O contract.

All new/updated towers are Apache-2.0 (Gemma 4 weights included -- unlike
earlier Gemma releases). Update README.md (I/O, shapes, precision,
conversion toolchain, SHA-256 checksums) and add a Gemma 4 upstream
component entry to THIRD_PARTY_LICENSES.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

README.md CHANGED
@@ -21,7 +21,7 @@ VPIPE runtime can fetch a pinned, ready-to-run artifact instead of
21
  re-running the conversion.
22
 
23
  > **Licensing note:** the bundled models are **not** all under a single
24
- > license. Two are Apache-2.0 and one (Silero VAD) is MIT. See
25
  > [THIRD_PARTY_LICENSES.md](THIRD_PARTY_LICENSES.md) for the per-model
26
  > attribution and verbatim license texts.
27
 
@@ -45,26 +45,60 @@ written `WxH`.
45
 
46
  | Archive | Model | Task | License |
47
  | --- | --- | --- | --- |
48
- | `qwen3_5_mlx_4b_vision_vid_512x320_w8.tar` | Qwen3.5-4B vision tower (Core ML) | Vision encoder for a multimodal LLM | Apache-2.0 |
 
 
49
  | `yolox_l_1024x640_w8.tar` | YOLOX-L (Core ML) | Object detection | Apache-2.0 |
50
  | `silero-vad-unified-v6.tar` | Silero VAD v6, unified (Core ML) | Voice activity detection | MIT |
51
 
52
  ### `qwen3_5_mlx_4b_vision_vid_512x320_w8`
53
 
54
- Qwen3.5-4B vision tower (`variant=ane_v1_video`), the vision encoder that
55
- turns video frames into the embedding sequence consumed by the Qwen3.5-4B
56
- language model.
57
 
58
- - **Task:** image/video → vision feature embeddings (multimodal LLM front-end).
59
  - **Inputs:** `image0`, `image1` — the two consecutive video frames that
60
- form one temporal patch, each **512×320 (W×H)**. Fed as a BGRA8888
61
- `CVPixelBuffer`; the graph drops alpha and applies a `1/127.5` scale and
62
- `-1` bias, yielding a BGR tensor in `[-1, 1]`.
63
- - **Output:** `image_features` — vision features of shape `(160, 2560)`.
64
  - **Precision:** 8-bit weight quantization, FP16 compute.
65
  - **Upstream:** [Qwen3.5](https://huggingface.co/Qwen) (Qwen team, Alibaba Cloud) — Apache-2.0.
66
  - **Conversion:** coremltools 9.0 from TorchScript (`torch==2.12.0`).
67
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
68
  ### `yolox_l_1024x640_w8`
69
 
70
  YOLOX-L anchor-free object detector, converted to Core ML at a fixed
@@ -100,7 +134,9 @@ Core ML conversion by FluidInference.
100
  SHA-256 checksums of the archives:
101
 
102
  ```
103
- 3940398c2518ac5f01c09f5f5eab868c2b217ab53c7d9a145d2387e76251b5d5 qwen3_5_mlx_4b_vision_vid_512x320_w8.tar
 
 
104
  7f7bf85947bb228f978470284e16a9d00c906a6cb07359f9a51c68806fa9c609 yolox_l_1024x640_w8.tar
105
  ee14d006d1cea87f15a4f0ac5f3e1e00b80a10c40eca855e8ff6741f389b4423 silero-vad-unified-v6.tar
106
  ```
 
21
  re-running the conversion.
22
 
23
  > **Licensing note:** the bundled models are **not** all under a single
24
+ > license. Four are Apache-2.0 and one (Silero VAD) is MIT. See
25
  > [THIRD_PARTY_LICENSES.md](THIRD_PARTY_LICENSES.md) for the per-model
26
  > attribution and verbatim license texts.
27
 
 
45
 
46
  | Archive | Model | Task | License |
47
  | --- | --- | --- | --- |
48
+ | `qwen3_5_mlx_4b_vision_vid_512x320_w8.tar` | Qwen3.5-4B vision tower @ 512×320 (Core ML) | Vision encoder for a multimodal LLM | Apache-2.0 |
49
+ | `qwen3_5_mlx_4b_vision_vid_768x480_w8.tar` | Qwen3.5-4B vision tower @ 768×480 (Core ML) | Vision encoder for a multimodal LLM | Apache-2.0 |
50
+ | `gemma4_mlx_e4b_vision_768x480_w8.tar` | Gemma 4 (e4b) vision tower @ 768×480 (Core ML) | Vision encoder for a multimodal LLM | Apache-2.0 |
51
  | `yolox_l_1024x640_w8.tar` | YOLOX-L (Core ML) | Object detection | Apache-2.0 |
52
  | `silero-vad-unified-v6.tar` | Silero VAD v6, unified (Core ML) | Voice activity detection | MIT |
53
 
54
  ### `qwen3_5_mlx_4b_vision_vid_512x320_w8`
55
 
56
+ Qwen3.5-4B vision tower (`variant=tensor_video_rgb`), the vision encoder
57
+ that turns a pair of video frames into the embedding sequence consumed by
58
+ the Qwen3.5-4B language model.
59
 
60
+ - **Task:** video → vision feature embeddings (multimodal LLM front-end).
61
  - **Inputs:** `image0`, `image1` — the two consecutive video frames that
62
+ form one temporal patch, each a planar RGB fp16 tensor `(1, 3, 320, 512)`
63
+ (`1×3×H×W`, i.e. 512×320 W×H) with raw pixels in `0..255`. The
64
+ `x/127.5 - 1` normalisation is baked into the graph (applied on the ANE).
65
+ - **Output:** `image_features` — vision features of shape `(160, 2560)`, fp16.
66
  - **Precision:** 8-bit weight quantization, FP16 compute.
67
  - **Upstream:** [Qwen3.5](https://huggingface.co/Qwen) (Qwen team, Alibaba Cloud) — Apache-2.0.
68
  - **Conversion:** coremltools 9.0 from TorchScript (`torch==2.12.0`).
69
 
70
+ ### `qwen3_5_mlx_4b_vision_vid_768x480_w8`
71
+
72
+ The same Qwen3.5-4B vision tower (`variant=tensor_video_rgb`) at a higher
73
+ 768×480 input — identical I/O contract to the 512×320 build, with a longer
74
+ output token sequence.
75
+
76
+ - **Task:** video → vision feature embeddings (multimodal LLM front-end).
77
+ - **Inputs:** `image0`, `image1` — the two consecutive video frames that
78
+ form one temporal patch, each a planar RGB fp16 tensor `(1, 3, 480, 768)`
79
+ (`1×3×H×W`, i.e. 768×480 W×H) with raw pixels in `0..255`. The
80
+ `x/127.5 - 1` normalisation is baked into the graph (applied on the ANE).
81
+ - **Output:** `image_features` — vision features of shape `(360, 2560)`, fp16.
82
+ - **Precision:** 8-bit weight quantization, FP16 compute.
83
+ - **Upstream:** [Qwen3.5](https://huggingface.co/Qwen) (Qwen team, Alibaba Cloud) — Apache-2.0.
84
+ - **Conversion:** coremltools 9.0 from TorchScript (`torch==2.12.0`).
85
+
86
+ ### `gemma4_mlx_e4b_vision_768x480_w8`
87
+
88
+ Gemma 4 (e4b) vision tower, converted to Core ML. It pools and projects the
89
+ visual features into the language model's 2560-dim text embedding space,
90
+ emitting "soft tokens" ready to splice into the Gemma 4 prompt.
91
+
92
+ - **Task:** image → LLM soft tokens (multimodal LLM front-end).
93
+ - **Input:** `image` — a single planar RGB fp16 MLMultiArray
94
+ `[1, 3, 480, 768]` (`1×3×H×W`, i.e. 768×480 W×H), raw pixels in `0..255`
95
+ (channel order R, G, B). Preprocessing is baked in: `x/255` then
96
+ `2*(x - 0.5)`.
97
+ - **Output:** `soft_tokens` — shape `[1, 160, 2560]`, fp16.
98
+ - **Precision:** int8 weight-only quantization (per-block), FP16 compute.
99
+ - **Upstream:** [Gemma 4 (e4b)](https://huggingface.co/google/gemma-4-E4B) (Google DeepMind) — Apache-2.0.
100
+ - **Conversion:** coremltools 9.0 from TorchScript (`torch==2.12.1`).
101
+
102
  ### `yolox_l_1024x640_w8`
103
 
104
  YOLOX-L anchor-free object detector, converted to Core ML at a fixed
 
134
  SHA-256 checksums of the archives:
135
 
136
  ```
137
+ ed27924a97a873a019d1c8a1e1f9826bd30da2588683843d6c59cfb0b2289f30 qwen3_5_mlx_4b_vision_vid_512x320_w8.tar
138
+ 3e928ea0c3e6848896278b68436d46f1aa306d95d8ca4a1faadfad354d705b87 qwen3_5_mlx_4b_vision_vid_768x480_w8.tar
139
+ 17bebe9fdb416a760df4d00ed39c7edf296ba768d54139c8183c1323da646d3e gemma4_mlx_e4b_vision_768x480_w8.tar
140
  7f7bf85947bb228f978470284e16a9d00c906a6cb07359f9a51c68806fa9c609 yolox_l_1024x640_w8.tar
141
  ee14d006d1cea87f15a4f0ac5f3e1e00b80a10c40eca855e8ff6741f389b4423 silero-vad-unified-v6.tar
142
  ```
THIRD_PARTY_LICENSES.md CHANGED
@@ -6,13 +6,16 @@ component and the archive it ships in, identifies the upstream source,
6
  reproduces the upstream copyright statement, and points to the verbatim
7
  license text required by that component's license.
8
 
9
- Two of the three models are licensed under the Apache License, Version 2.0;
10
- the third (Silero VAD) is licensed under the MIT License. The verbatim
11
- license texts are reproduced in the appendices at the end of this file.
 
12
 
13
  | Archive | Component | License |
14
  | --- | --- | --- |
15
  | `qwen3_5_mlx_4b_vision_vid_512x320_w8.tar` | Qwen3.5 vision tower | Apache-2.0 ([Appendix A](#appendix-a-apache-license-version-20)) |
 
 
16
  | `yolox_l_1024x640_w8.tar` | YOLOX | Apache-2.0 ([Appendix A](#appendix-a-apache-license-version-20)) |
17
  | `silero-vad-unified-v6.tar` | Silero VAD (FluidInference Core ML conversion) | MIT ([Appendix B](#appendix-b-mit-license)) |
18
 
@@ -20,10 +23,11 @@ license texts are reproduced in the appendices at the end of this file.
20
 
21
  ## Qwen3.5 (vision tower)
22
 
23
- The archive `qwen3_5_mlx_4b_vision_vid_512x320_w8.tar` contains a Core ML
24
- conversion of the **Qwen3.5-4B** vision tower (the multimodal vision
25
- encoder), derived from the Qwen3.5 model series published by the Qwen team
26
- at Alibaba Cloud (<https://huggingface.co/Qwen>). The weights are
 
27
  8-bit-quantized and the graph is exported to Core ML; the model is otherwise
28
  a derivative of the upstream Qwen3.5 weights.
29
 
@@ -46,6 +50,37 @@ upstream at
46
 
47
  ---
48
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
49
  ## YOLOX
50
 
51
  The archive `yolox_l_1024x640_w8.tar` contains a Core ML conversion of
 
6
  reproduces the upstream copyright statement, and points to the verbatim
7
  license text required by that component's license.
8
 
9
+ Every model except one is licensed under the Apache License, Version 2.0;
10
+ the remaining one (Silero VAD) is licensed under the MIT License. The
11
+ verbatim license texts are reproduced in the appendices at the end of this
12
+ file.
13
 
14
  | Archive | Component | License |
15
  | --- | --- | --- |
16
  | `qwen3_5_mlx_4b_vision_vid_512x320_w8.tar` | Qwen3.5 vision tower | Apache-2.0 ([Appendix A](#appendix-a-apache-license-version-20)) |
17
+ | `qwen3_5_mlx_4b_vision_vid_768x480_w8.tar` | Qwen3.5 vision tower | Apache-2.0 ([Appendix A](#appendix-a-apache-license-version-20)) |
18
+ | `gemma4_mlx_e4b_vision_768x480_w8.tar` | Gemma 4 vision tower | Apache-2.0 ([Appendix A](#appendix-a-apache-license-version-20)) |
19
  | `yolox_l_1024x640_w8.tar` | YOLOX | Apache-2.0 ([Appendix A](#appendix-a-apache-license-version-20)) |
20
  | `silero-vad-unified-v6.tar` | Silero VAD (FluidInference Core ML conversion) | MIT ([Appendix B](#appendix-b-mit-license)) |
21
 
 
23
 
24
  ## Qwen3.5 (vision tower)
25
 
26
+ The archives `qwen3_5_mlx_4b_vision_vid_512x320_w8.tar` and
27
+ `qwen3_5_mlx_4b_vision_vid_768x480_w8.tar` contain Core ML conversions of
28
+ the **Qwen3.5-4B** vision tower (the multimodal vision encoder) at two input
29
+ resolutions, derived from the Qwen3.5 model series published by the Qwen
30
+ team at Alibaba Cloud (<https://huggingface.co/Qwen>). The weights are
31
  8-bit-quantized and the graph is exported to Core ML; the model is otherwise
32
  a derivative of the upstream Qwen3.5 weights.
33
 
 
50
 
51
  ---
52
 
53
+ ## Gemma 4 (vision tower)
54
+
55
+ The archive `gemma4_mlx_e4b_vision_768x480_w8.tar` contains a Core ML
56
+ conversion of the **Gemma 4 (e4b)** vision tower (the multimodal vision
57
+ encoder that projects image features into the language model's text
58
+ embedding space), derived from the Gemma 4 model series published by Google
59
+ DeepMind (<https://huggingface.co/google/gemma-4-E4B>). The weights are
60
+ int8 weight-only quantized and the graph is exported to Core ML; the model
61
+ is otherwise a derivative of the upstream Gemma 4 weights.
62
+
63
+ Gemma 4 open-weight models are distributed under the Apache License,
64
+ Version 2.0. (This differs from earlier Gemma releases, which used the
65
+ custom Gemma Terms of Use.)
66
+
67
+ ### Copyright notice
68
+
69
+ > Copyright Google LLC
70
+
71
+ The upstream model card declares the license as Apache-2.0; Google does not
72
+ publish a per-file copyright line for the Gemma 4 weights.
73
+
74
+ ### License
75
+
76
+ Apache License, Version 2.0. The verbatim text is reproduced in
77
+ [Appendix A](#appendix-a-apache-license-version-20), and is available
78
+ upstream at
79
+ <https://ai.google.dev/gemma/docs/gemma_4_license> and
80
+ <https://www.apache.org/licenses/LICENSE-2.0>.
81
+
82
+ ---
83
+
84
  ## YOLOX
85
 
86
  The archive `yolox_l_1024x640_w8.tar` contains a Core ML conversion of
gemma4_mlx_e4b_vision_768x480_w8.tar ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:17bebe9fdb416a760df4d00ed39c7edf296ba768d54139c8183c1323da646d3e
3
+ size 165332992
qwen3_5_mlx_4b_vision_vid_512x320_w8.tar CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:3940398c2518ac5f01c09f5f5eab868c2b217ab53c7d9a145d2387e76251b5d5
3
- size 333077504
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ed27924a97a873a019d1c8a1e1f9826bd30da2588683843d6c59cfb0b2289f30
3
+ size 333078016
qwen3_5_mlx_4b_vision_vid_768x480_w8.tar ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3e928ea0c3e6848896278b68436d46f1aa306d95d8ca4a1faadfad354d705b87
3
+ size 334001152