Keep vision/audio towers dense (fix image understanding)

README.md

@@ -35,20 +35,24 @@ tokenizer needed by MLX-VLM.
 
 ## What is quantized
 
-- **527 linear and embedding layers** (`q/k/v/o` projections, MLP gate/up/down,
-  multimodal projectors, and the large embeddings) are 6-bit affine quantized:
-  packed `uint32` `weight` (4 values per 3 bytes, low bits first) + float16
-  `scales`/`biases`.
-- **Layer norms, convolutions, and biases stay float16** — matching MLX-VLM's
-  standard affine quantization. The dense per-layer input embedding
-  (`embed_tokens_per_layer`) **is** quantized here, so the package stays under
-  ~4.2 GB rather than the ~7 GB a dense embedding would force.
+- **280 language-model layers** (`q/k/v/o` projections, MLP gate/up/down,
+  the multimodal embedding projections, and the large embeddings) are 6-bit
+  affine quantized: packed `uint32` `weight` (4 values per 3 bytes, low bits
+  first) + float16 `scales`/`biases`.
+- **The vision tower and audio tower are left in float16 (dense)** — matching
+  MLX-VLM's `convert --quantize`, which skips multimodal modules. Their QAT
+  `ClippableLinear` layers carry input/output clipping parameters
+  (`input_max`/`input_min`/`output_max`/`output_min`) that must not be
+  affine-quantized, so they stay dense and are loaded as regular `nn.Linear`.
+- The dense per-layer input embedding (`embed_tokens_per_layer`) **is**
+  quantized here, so the language model stays compact without exceeding the
+  Metal buffer cap.
 
 ## Package contents
 
-- `model-00001-of-00002.safetensors` (4,000,099,030 bytes) and
-  `model-00002-of-00002.safetensors` (166,857,424 bytes): the 6-bit MLX model
-  (~4.2 GB total).
+- `model-00001-of-00002.safetensors` (3,993,096,556 bytes) and
+  `model-00002-of-00002.safetensors` (715,527,930 bytes): the 6-bit MLX model
+  (~4.71 GB total).
 - `model.safetensors.index.json`: complete shard map.
 - `config.json` (with `quantization` + `quantization_config`), `generation_config.json`,
   `processor_config.json`, tokenizer files, and `chat_template.jinja`.
@@ -58,14 +62,13 @@ tokenizer needed by MLX-VLM.
 The conversion host has no Apple-Silicon MLX runtime, so the quantized package
 was structurally validated before upload:
 
-- 1,951 source tensors mapped with no missing or extra keys; 527 linear +
-  embedding layers quantized.
+- 1,951 source tensors mapped with no missing or extra keys; 280 language-model
+  layers quantized; vision/audio towers left dense.
 - Quantized weight format matches the MLX affine contract: 6-bit values packed
   4-per-3-bytes (24-bit little-endian word), dequantization `scale * q + bias`,
   group 64.
-- Round-trip dequantization of sampled layers (attention projections + the
-  2.35B-param `embed_tokens_per_layer`) reproduces the source weights to within
-  6-bit precision (max relative error ≈ 1.4%).
+- Round-trip dequantization of sampled layers reproduces the source weights to
+  within 6-bit precision.
 
 **Apple-Silicon MLX-VLM inference has not been run.** Treat this as a
 structurally validated quantization pending a real Apple-Silicon text / image /

model-00001-of-00002.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8336e228e847c64f20cd601798c394902c0f32609adc8e9446777a50775ae6a9
-size 4000099030
+oid sha256:7904e77f1eaefe819fe83087f49aa1a7fe10f7d03e98d8a4d2d30c1b65b11591
+size 3993096556

model-00002-of-00002.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f344f9aab6609b5e1b29c97f6f268105292224c0a78dcc06a62b5e658d08cfab
-size 166857424
+oid sha256:9f61f16dabae23782fd868f6c8ac00ccbb6df864e3bb48f12410b20991eaefd6
+size 715527930