pplx-decider-flash-v1.2-mlx

MLX quantizations of the text backbone of pplx-decider-flash-v1.2, a multi-question decision/classification model. Text-only.

path	quantization	size
/ (repo root, recommended)	4-bit affine, group size 32	2.45 GiB
4bit-g64/	4-bit affine, group size 64	2.20 GiB
3bit/	3-bit affine, group size 64	1.72 GiB
8bit/	8-bit affine, group size 64	4.16 GiB
d3-ternary-2.30g/	D3 mixed precision (ternary/2/3/4/8-bit per layer group)	2.30 GiB
d3-ternary-2.20g/	D3 mixed precision (ternary/2/3/4/8-bit per layer group)	2.20 GiB

The uniform variant folders are self-contained (weights, config.json, readout.safetensors, decision_config.json, joint_config.json, tokenizer). The d3-ternary-* folders contain weights, config.json, tokenizer, decision_config.json, joint_config.json and d3_allocation.json; they use the repo-root readout.safetensors (identical BF16 file). The runner code is in runner/ at the repo root.

Provenance and license

This is a derivative work of perplexity-ai/pplx-decider-flash-v1.2 (revision 6d771ad7b5ecff5e93b8fc7b90d4c5ccc341954d), itself derived from Qwen/Qwen3.5-4B. It is distributed under the same Apache License 2.0; the original LICENSE and NOTICE files are included unchanged. This is not an official Perplexity release.

Modifications from the original checkpoint

• Quantization: MLX affine quantization (root: 4 bits / group 32; 4bit-g64/: 4 bits / group 64; 3bit/: 3 bits / group 64; 8bit/: 8 bits / group 64), applied to every Linear layer of the text backbone and to the token embedding (embed_tokens, same bits as the variant). Conv1d, norms, A_log/dt_bias are not quantized.
• Vision tower dropped: all visual.* tensors removed; image inputs are not supported.
• Weight renaming / layout: language_model.* → language_model.model.* (mlx-lm qwen3_5 layout); conv1d weights transposed and RMSNorm weights converted to the 1 + w form by mlx-lm's sanitize (numerically equivalent).
• Readout head kept BF16: readout.safetensors ([255, 2560]) is the original file, unquantized; it is applied in fp32.
• No lm_head: as in the original, there is no vocabulary head; mlx_lm.generate is not meaningful for this model.
• Custom non-causal runner required: runner/ reproduces the original protocol (joint prompt, <|fim_suffix|> marker per question, non-causal full-attention layers, readout at each marker, calibrated temperature from decision_config.json). Stock mlx-lm uses causal attention, which does not reproduce this checkpoint.
• The original model card is included as ORIGINAL_MODEL_CARD.md.

Usage

import sys
from huggingface_hub import snapshot_download
repo = snapshot_download("Jakevin/pplx-decider-flash-v1.2-mlx")     # or allow_patterns=["runner/*", "4bit-g64/*"] etc.
sys.path.insert(0, f"{repo}/runner")
import mlx_decider as D
from decider_common import Encoder, answers
path = repo                     # root = 4-bit g32; use f"{repo}/4bit-g64", f"{repo}/3bit" or f"{repo}/8bit" for the other variants
enc = Encoder(path); model, _ = D.load_mlx_dir(path); W = D.load_readout(path)
req = {"state": "My integration keeps failing with a 401 since we rotated our API key.",
        "questions": {"team": {"type": "choice", "criteria": {"billing": None, "support": None, "sales": None}}}}
ids, markers, counts = enc.encode(req)
print(answers(req, D.readout_logits(model, W, ids, markers, counts), enc.temperature))

Requires mlx, mlx-lm (tested 0.32.4 / 0.32.2) and transformers (tokenizer only). runner/run_mlx.py is the evaluation / quantization script used for the table below (it expects a local copy of the original checkpoint and an eval_set.json).

Evaluation (agreement with the original BF16 model)

Reference: the unquantized BF16 weights run through the same non-causal runner (validated against the original PyTorch pplx_decider non-causal hook: 14/14 identical answers, max |Δlogit| 0.06; independent llama.cpp BF16 path: 246/246 identical). Held-out set: 244 requests / 246 questions (model-card example; validation splits of SST-2, MNLI-matched, BoolQ, dair-ai/emotion, STS-B). "Confident flips" = answer changed where the BF16 top-1 margin was > 0.2. KL is on calibrated (T=1.554) probabilities.

variant	size GiB	peak mem GiB (M4)	label agree % (all)	agree % (choice+noul)	confident flips	KL mean	KL max	gold acc %	STS-B ρ
original BF16 (reference)	7.83	8.06	100.0	100.0	0	0.0000	0.0000	86.0	0.935
root: 4-bit g32	2.45	3.03	95.1	97.5	0	0.0092	0.3414	87.0	0.934
4bit-g64/	2.20	2.79	93.1	94.6	3	0.0155	0.4157	84.5	0.928
3bit/	1.72	2.32	91.5	93.1	4	0.0592	0.4355	89.0	0.917
8bit/ ¹	4.16	4.73	99.2	99.0	0	0.0001	0.0011	87.0	0.932

¹ 8bit/ was built with runner/run_mlx.py q8_g64 --save (8 bits / group 64, embedding 8-bit) from the same original checkpoint and evaluated on the same 246 questions against the same BF16 reference results as the other rows, but on an Apple M4 Pro with mlx 0.32.3 / mlx-lm 0.32.0 (peak memory is from that machine). A BF16 rerun on that machine matched the reference on 246/246 answers (KL max 0.0010). The saved folder reloaded with mlx_lm.utils.load_model gives 246/246 identical answers to the in-memory run. This row replaces the earlier unpublished 8-bit g64 row.

D3 mixed-precision variants (d3-ternary-2.30g/, d3-ternary-2.20g/)

Bits are allocated per layer group by output sensitivity: each group (token embedding; per layer the token mixer — Gated-DeltaNet or attention projections —, MLP gate+up, MLP down; 97 groups) was quantized alone and the KL of the calibrated readout probabilities vs BF16 was measured; a size-constrained knapsack then picked one format per group. Formats: ternary g64, 2-bit g64, 3-bit g64/g32, 4-bit g64/g32, 8-bit g64 (round-to-nearest, same as the uniform variants); readout stays BF16. Calibration: 54 examples from the train splits of SST-2, MNLI, BoolQ, dair-ai/emotion and STS-B, deduplicated against the 246 evaluation questions (no overlap); no model-card examples or validation/test data.

Same 246 questions and BF16 reference as the table above:

variant	size GiB	agree % (choice+noul)	label agree % (all)	flips	confident flips	KL mean	KL max
root: uniform 4-bit g32	2.45	97.5	95.1	12	0	0.0092	0.3414
4bit-g64/ (uniform)	2.20	94.6	93.1	17	3	0.0155	0.4157
3bit/ (uniform)	1.72	93.1	91.5	21	4	0.0592	0.4355
8bit/ (uniform) ¹	4.16	99.0	99.2	2	0	0.0001	0.0011
d3-ternary-2.30g/	2.30	98.0	96.7	8	0	0.0066	0.0865
d3-ternary-2.20g/	2.20	97.1	95.5	11	0	0.0080	0.1546

• Noise: the choice+noul subset has 204 questions, so one question ≈ 0.5 points; agreement differences of ±1–2 questions are within noise. KL and confident flips are the more stable signals (both improve over the uniform builds of similar size).
• Ternary is used in only one group: the layer-29 Gated-DeltaNet token mixer (~1% of parameters). It is stored in MLX's standard 2-bit affine format (codes 0/1/2 → −s/0/+s), so stock MLX kernels run it; no custom kernel is needed.
• Bit mix (share of quantized parameters, incl. embedding):
  • 2.30g: ternary 1.0%, 2-bit 5.9%, 3-bit g64 20.0%, 3-bit g32 0.6%, 4-bit g64 39.7%, 4-bit g32 22.2%, 8-bit 10.6% (embedding 4-bit g64).
  • 2.20g: ternary 1.0%, 2-bit 7.6%, 3-bit g64 18.4%, 3-bit g32 15.7%, 4-bit g64 28.9%, 4-bit g32 20.3%, 8-bit 8.1% (embedding 3-bit g32).
• Per-group formats are in each folder's d3_allocation.json and in config.json → quantization (loaded by mlx_lm.utils.load_model).
• Usage: as above with path = f"{repo}/d3-ternary-2.30g", but load the readout from the repo root: W = D.load_readout(repo).

Measured on an Apple M4 / 16 GB (except the 8bit/ rows, see ¹). Small sample; validate on your own task. For a GGUF / llama.cpp build see Jakevin/pplx-decider-flash-v1.2-GGUF. The two D3 ternary builds are also published as a standalone repo: Jakevin/pplx-decider-flash-v1.2-ternary-mlx.