Keep vision/audio towers dense (fix image understanding)

README.md

@@ -35,19 +35,23 @@ tokenizer needed by MLX-VLM.
 
 ## What is quantized
 
-- **527 linear and embedding layers** (`q/k/v/o` projections, MLP gate/up/down,
-  multimodal projectors, and the large embeddings) are 4-bit affine quantized:
-  packed `uint32` `weight` (8 values per word, low nibble first) + float16
-  `scales`/`biases`.
-- **Layer norms, convolutions, and biases stay float16** — matching MLX-VLM's
-  standard affine quantization. The dense per-layer input embedding
-  (`embed_tokens_per_layer`) **is** quantized here, so the package stays under
-  ~2.9 GB rather than the ~7 GB a dense embedding would force.
+- **280 language-model layers** (`q/k/v/o` projections, MLP gate/up/down,
+  the multimodal embedding projections, and the large embeddings) are 4-bit
+  affine quantized: packed `uint32` `weight` (8 values per word, low nibble
+  first) + float16 `scales`/`biases`.
+- **The vision tower and audio tower are left in float16 (dense)** — matching
+  MLX-VLM's `convert --quantize`, which skips multimodal modules. Their QAT
+  `ClippableLinear` layers carry input/output clipping parameters
+  (`input_max`/`input_min`/`output_max`/`output_min`) that must not be
+  affine-quantized, so they stay dense and are loaded as regular `nn.Linear`.
+- The dense per-layer input embedding (`embed_tokens_per_layer`) **is**
+  quantized here, so the language model stays compact without exceeding the
+  Metal buffer cap.
 
 ## Package contents
 
-- `model-00001-of-00001.safetensors` (2,894,953,598 bytes): the 4-bit MLX model
-  in a single shard (~2.9 GB total).
+- `model-00001-of-00001.safetensors` (3,550,670,830 bytes): the 4-bit MLX model
+  in a single shard (~3.55 GB total).
 - `model.safetensors.index.json`: complete shard map.
 - `config.json` (with `quantization` + `quantization_config`), `generation_config.json`,
   `processor_config.json`, tokenizer files, and `chat_template.jinja`.
@@ -57,13 +61,12 @@ tokenizer needed by MLX-VLM.
 The conversion host has no Apple-Silicon MLX runtime, so the quantized package
 was structurally validated before upload:
 
-- 1,951 source tensors mapped with no missing or extra keys; 527 linear +
-  embedding layers quantized.
+- 1,951 source tensors mapped with no missing or extra keys; 280 language-model
+  layers quantized; vision/audio towers left dense.
 - Quantized weight format matches the MLX affine contract: 4-bit values packed
   8-per-`uint32` (low nibble first), dequantization `scale * q + bias`, group 64.
-- Round-trip dequantization of sampled layers (attention projections + the
-  2.35B-param `embed_tokens_per_layer`) reproduces the source weights to within
-  4-bit precision (max relative error ≈ 5%).
+- Round-trip dequantization of sampled layers reproduces the source weights to
+  within 4-bit precision.
 
 **Apple-Silicon MLX-VLM inference has not been run.** Treat this as a
 structurally validated quantization pending a real Apple-Silicon text / image /

model-00001-of-00001.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:48ce7e8977c31fba22b0920b8462e6a7b7d7f94b76902e10f723430c9a88bcd8
-size 2894953598
+oid sha256:38de6d49e4d9cf80c90e5d739705f7bcebeeb29f851a3e30c8e78d7982eaeb13
+size 3550670830