d3-lite mixed-bit MLX
Unofficial MLX post-training quantization of vllm-sr/d3-lite (bf16, Apache-2.0). The source is a 24-layer Qwen3.5 text stack plus a vision tower and a 255-way decision readout. This package is not an official vLLM Semantic Router release.
Suggested repository id: Jakevin/d3-lite-ternary-mlx.
Method
The original bfloat16 checkpoint is the reference. For each language-model projection
(attention Q/K/V/O, DeltaNet in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj,
and MLP gate/up/down) one candidate replaces that projection alone and the KL of the decision
readout's option distribution is measured. Candidates are ternary (one scale per group of 64,
values -s, 0, +s), and MLX affine 2-bit, 3-bit and 4-bit with group size 64. Ternary is stored
in MLX's built-in 2-bit affine tensors so mlx-lm / mlx-vlm quantized matmul can run it.
No custom Metal kernel is included.
Layer kinds are read from text_config.layer_types: 18 DeltaNet layers and 6
full-attention layers. Full attention is non-causal, as in the source checkpoint. DeltaNet stays causal.
Bits start from the assignment that minimizes the sum of those one-at-a-time KL values with the
average storage width of the language projections about 3.5 bits. Ternary counts as 2 storage bits
because that is the container it uses; it is selected only when its KL is lower than affine 2-bit.
That additive minimum sums to 3.7363.
No joint bit-swap refinement was applied to this build.
The shipped assignment's one-at-a-time sum is 3.7363.
The decision readout is a separate matrix, last_hidden @ readout.T. It does not use the tied
lm_head, so the token embedding is input-only and stays 8-bit affine (group 64).
The decision readout, RMSNorms, DeltaNet convolution / A_log / dt_bias, and the entire vision
tower stay at the original precision.
Calibration: 64 questions from the author's decision-v7 development split
(seed 20261011, one question from each drawn record) plus the three text questions in the
source model card. 67 questions total. hard-v1 was not used.
Size
| bytes | |
|---|---|
model.safetensors (quantized text + full-precision vision tower) |
721,266,092 |
readout.safetensors (fp32 decision head, unchanged) |
1,044,560 |
| this folder | 745,359,678 |
| source bf16 shards + readout | 1,707,072,248 |
| tied embedding tensors, 8-bit (used) | 270,172,160 |
Language-projection storage mix (ternary counted as 2 bits). Average 3.500 bits
over 497,614,848 projection parameters. The tied embedding is 8-bit and is not in that average.
The vision tower inside model.safetensors is uncompressed bf16.
| assignment | parameters | share | storage bits |
|---|---|---|---|
| ternary | 17,334,272 | 3.48% | 2 |
| 2 | 63,995,904 | 12.86% | 2 |
| 3 | 86,163,456 | 17.32% | 3 |
| 4 | 330,121,216 | 66.34% | 4 |
Evaluation
Single run, no confidence interval. 150 questions: 50 holdout from decision-v7 (seed 20261013,
disjoint from calibration and the 8 smoke questions) plus 50 decision-v7 and 50 transfer-v9
(same seed, disjoint from calibration). Agreement is argmax vs the source bf16 model on the same
MLX decision path. Accuracy is argmax vs the dataset label.
Overall agreement 130/150 (86.7%). Mean KL(bf16 || quantized) 0.0407, median 0.0097. Compressed accuracy 99/150, bf16 accuracy 98/150.
Gate note. This build does not reach 90% agreement with bf16. For models under 1B the release gate is: compressed accuracy not below bf16 on these 150 questions and mean KL <= 0.05. This build has 99 vs 98 correct and mean KL 0.0407.
The 20 disagreements are mostly questions where bf16 itself was unsure: bf16 top-1 probability median 0.50 (top-1 minus top-2 margin median 0.10) vs 0.96 (margin 0.92) on the 130 agreeing questions. Of the 20, 7 went correct->wrong, 8 wrong->correct and 5 wrong->wrong.
The 150 questions are not fully independent: the holdout and decision-v7 sets share 49 of their 50 questions.
| suite | compressed correct | bf16 correct | agree | mean KL | median KL |
|---|---|---|---|---|---|
| holdout | 35/50 | 34/50 | 43/50 | 0.0324 | 0.0073 |
| decision-v7 | 36/50 | 35/50 | 43/50 | 0.0324 | 0.0073 |
| transfer-v9 | 28/50 | 29/50 | 44/50 | 0.0574 | 0.0217 |
Reload check: eight other questions scored from the in-memory quantized model and again after
loading out/release/ from disk. Agreement with memory 8/8.
The vision tower was not quantized and is untested in this package; it was not part of this evaluation.
Official source-card scores, quoted from vllm-sr/d3-lite (Jev Decision Index 0.3.1, source evaluation, not reproduced here): Decision Index 29.0, public suite 36.9, same-skill tests 31.6, new-domain tasks 21.6. Vision Index 46.9. Perception Test (internal) 59.3.
Use
from pathlib import Path
from transformers import AutoTokenizer
import decide
import d3_mlx
root = Path(".")
model, config, decision, readout = d3_mlx.load_model(root)
tokenizer = AutoTokenizer.from_pretrained(root)
answers = decide.predict(model, readout, tokenizer, decision["codes"], state, questions)
From a shell, in this directory:
python decide.py
That runs the three text questions from the source model card. python decide.py request.json
reads {"state": ..., "questions": ...}.
mlx_lm can load the language tower because config.json sets model_file to d3_mlx.py,
which builds layers from layer_types and uses stock MLX quantized linears. The vision weights
use mlx-vlm's vision_tower.* names and stay bf16. Text decisions use decide.py.
License
Apache-2.0. Source: vllm-sr/d3-lite,
itself built on Qwen/Qwen3.5-0.8B.
See LICENSE and NOTICE.
繁體中文摘要
這是 vllm-sr/d3-lite 的非官方 MLX 量化包。
語言層的每個投影各自試過三值、2、3、4 bit，先用一次換一層的 KL 在平均約 3.50 bit 的預算內分配，
未做聯合對調。三值借 MLX 內建 2-bit affine 儲存。決策頭是獨立 readout，不經過 tied 的輸出頭，
所以嵌入維持 8 bit。決策頭、norm 與視覺塔維持原精度，視覺部分未壓縮。
model.safetensors 為 721.3 MB，整個資料夾 745.4 MB。
150 題（單次、無信賴區間）與 bf16 一致 130/150，
壓縮版正確 99/150，平均 KL 0.0407。
與 bf16 的不一致多半發生在 bf16 本身沒把握的題目（20 題中 7 題對→錯、8 題錯→對、5 題錯→錯）。holdout 與 decision-v7 有 49/50 題相同。視覺部分未壓縮、未測試。
- Downloads last month
- 27
4-bit