Instructions to use mlx-community/pplx-decider-v1.3-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use mlx-community/pplx-decider-v1.3-4bit with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download mlx-community/pplx-decider-v1.3-4bit --local-dir pplx-decider-v1.3-4bit
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
mlx-community/pplx-decider-v1.3-4bit
An MLX 4-bit conversion of
perplexity-ai/pplx-decider-v1.3 for Apple Silicon, with a
standalone inference script and a small HTTP server. It supports everything in the original release except the
CUDA-only fast path: decisions (choice, noul, score), images, Locate span extraction, and both question modes.
This is a classifier, not a chat model. Give it a state (text, JSON and/or images) and one or more questions. One forward pass returns calibrated probabilities over each question's options. There is no generation step.
| Question type | criteria |
Answer |
|---|---|---|
choice |
{option_name: description or None}, 1 to 255 options |
choice, probabilities, confidence |
noul |
optional {"true": description, "false": description} |
noul = P(true) |
score |
list of 2 to 255 level descriptions, lowest first | score (expected level), probabilities, legend, confidence |
locate |
not used; the question key is the label, or instructions if given |
spans: start, end, text, score |
Why it needs the included code
mlx_lm.load / mlx_vlm.load will not give correct results with this checkpoint:
- The 16 full-attention layers are noncausal: every token sees the whole sequence, and only padding is masked. The 48 Gated DeltaNet layers stay causal. With a causal mask the top answer changes on 16 of 115 of our test questions and the probabilities are far off (mean KL 0.076, against 0.005 for this model).
- The output is a 255-row decision readout (stored as
language_model.lm_head), not a vocabulary head. It is read at each question's<|fim_suffix|>marker, and the softmax runs over that question's options only, at the saved calibration temperature (1.2264). - The prompt is the v1.3 chunked protocol, with the state and every question tokenized separately.
decider_mlx.py implements all of this. It is a port of the official pplx_decider package to MLX and uses the same
request and response format.
Quick start
Requires an Apple Silicon Mac and Python 3.12+. The model uses 15.4 GB at rest and more for long inputs or Locate (see Speed and memory), so 32 GB of memory is recommended; 24 GB is enough for short text inputs.
hf download mlx-community/pplx-decider-v1.3-4bit --local-dir pplx-decider-v1.3-4bit
cd pplx-decider-v1.3-4bit
pip install -r requirements.txt # mlx, mlx-vlm, transformers (tokenizer only), pillow: no torch needed
from decider_mlx import Decider
decider = Decider.from_pretrained(".") # or "mlx-community/pplx-decider-v1.3-4bit"
response = decider.decide({
"state": "My integration keeps failing with a 401 since we rotated our API key. Please help.",
"questions": {
"team": {
"type": "choice",
"instructions": "Which team should handle this request?",
"criteria": {
"billing": "Charges and refunds",
"support": "Technical integration errors",
"sales": "Questions about buying a product",
},
},
"urgent": {"type": "noul", "instructions": "Is the customer blocked right now?"},
"frustration": {"type": "score", "criteria": ["calm", "mildly annoyed", "frustrated", "angry"]},
"error": {"type": "locate", "instructions": "Error codes"},
},
# "images": ["chart.png"], # optional: file paths, data:image/... URLs or PIL images (not with locate)
})
print(response["answers"]["team"]["choice"], response["answers"]["urgent"]["noul"])
print(response["answers"]["error"]["spans"]) # [{"start": 36, "end": 39, "text": "401", "score": 0.99...}]
decide takes one request and returns one response, or a list and returns a list in the same order. Each response is
{"answers": {"<id>": {...}}, "usage": {"input_tokens", "output_tokens", "sequences"}}. decider.logits(requests)
returns the raw readout logits of each question's options, before temperature. See the
original model card for the full request schema,
calibration advice and limits; they apply unchanged.
CLI:
python decider_mlx.py --state "Win a FREE iPhone, click now!!!" \
--questions '{"spam": {"type": "noul", "instructions": "Is this spam?"}}'
Several questions in one request
Decider.from_pretrained(..., mode=...) selects how the questions of a request are run, as in the original:
mode="separate"(default): each question is its own sequence (state + that question). Questions do not see each other. This is the computation Perplexity evaluated.mode="isolated": one sequence per request. The state is read once and each question attends the state and itself. It is much cheaper when several questions share a state, but the state also attends the questions, so answers can shift slightly with the set of questions (Perplexity reports 1.5-2.2% of top answers differing).
Time for one request on an M5 Max, this model:
| State | Questions | separate |
isolated |
|---|---|---|---|
| 200 tokens | 8 | 3.9 s | 0.9 s |
| 1,000 tokens | 8 | 15.2 s | 2.5 s |
| 4,000 tokens | 8 | 58 s | 7.5 s |
| 4,000 tokens | 32 | 232 s | 9.3 s |
Locate
"type": "locate" extracts verbatim spans of the state, with character offsets (state[start:end] == text) and a
score. The adapter (an unmerged rank-64 LoRA, a trained <|locate|> embedding and a span head, about 1.9 GB) is
loaded on the first Locate request and is switched off for decision questions, so decisions are unaffected. All
Locate questions of a request run together in one isolated sequence, as in the original. Locate needs a nonempty text
state and no images. Decider.from_pretrained(..., locate=False) disables it.
Images
Images go in "images". They are placed before the state, and every question can see them. Each image is resized to
at most 1,638,400 pixels (the checkpoint's max_pixels); a 1.6 MP image uses about 1,600 sequence positions.
Server
python server.py --model . --port 8092 # add --mode isolated to run each request as one sequence
curl localhost:8092/v1/systemone -H 'Content-Type: application/json' -d '{
"state": "My integration keeps failing with a 401 since we rotated our API key. Please help.",
"questions": {
"urgent": {"type": "noul", "instructions": "Is the customer blocked right now?"},
"error": {"type": "locate", "instructions": "Error codes"}
}}'
The request takes state (a string, JSON, or chat messages with image_url parts), questions and optional images
(data URLs, up to 8), the same shape as llama.cpp's llama-server /v1/systemone. The response has answers and
usage. Also available: GET /v1/models, /health and /stats. Set DECIDER_API_KEY to require a bearer token.
One GPU worker runs MLX; concurrent requests are batched together by similar length.
Accuracy
Checked against the official PyTorch implementation (pplx_decider 0.5.0, transformers 5.17.0), run in fp32 with one
request per forward pass. The test set has 77 requests: one row from each of 50 public benchmark
families (MMLU-Pro, BANKING77, ContractNLI, BFCL, RAGTruth, ...) plus hand-written edge cases: 255 options, a single
option, JSON and chat states, special tokens in user text, whitespace-only and empty states, several languages,
images of different sizes, and Locate. Sequences run from 66 to 7,300 tokens.
Tokenization: the token ids of every sequence are identical to the reference's.
Decisions (probabilities over each question's options, at the saved temperature):
| questions | top answer agrees | mean KL | mean abs 螖p | max abs 螖p | |
|---|---|---|---|---|---|
| MLX bf16 (the port without quantization), separate | 115 | 115/115 | 0.00008 | 0.002 | 0.015 |
| this model (4-bit), separate | 115 | 112/115 | 0.0054 | 0.018 | 0.26 |
| this model (4-bit), isolated | 19 | 19/19 | 0.0015 | 0.006 | 0.031 |
| control: this model with causal attention (the wrong mode) | 115 | 99/115 | 0.076 | 0.085 | 0.70 |
The three top answers that changed were close calls in the reference: its top two options were 0.02, 0.14 and 0.16 apart. Isolated mode was compared on 6 multi-question requests only, because the reference run on CPU was stopped at a time limit.
Against the unquantized MLX port on a larger set (127 requests, 303 questions, both modes), this model agrees on the top answer for 192/197 questions in separate mode and 103/106 in isolated mode (mean KL 0.007 and 0.003). On the 100 benchmark rows of that set, accuracy against the benchmark labels is 115/140 for this model and 112/140 unquantized, a difference within noise.
Locate: on 17 Locate questions the unquantized port returns the same 72 spans as the reference (scores within 0.023). This 4-bit model returns 70 of them. It misses two spans that the reference scored 0.55 and 0.73, adds one spurious long span at 0.54, and moves one score from 0.92 to 0.63; the other scores differ by 0.007 on average. Locate is more sensitive to quantization than decisions, mostly near the 0.5 cutoff.
Port correctness was checked separately on the smaller
pplx-decider-flash-v1.3, which uses the same code and
fits in memory in fp32 on both sides. MLX fp32 against PyTorch fp32: 46/46 top answers in separate mode and 62/62 in
isolated mode, mean KL 5e-7 and 7e-7, and 71/71 Locate spans identical with scores within 0.0014.
Speed and memory
Measured on an Apple M5 Max (128 GB), text input, one request at a time unless noted.
| Input | Time | Tokens/s | Peak memory |
|---|---|---|---|
| 80 tokens | 0.15 s | 545 | 15.6 GB |
| 1,000 tokens | 1.1 s | 890 | 16.3 GB |
| 4,000 tokens | 6.1 s | 660 | 17.2 GB |
| 8,000 tokens | 13.2 s | 610 | 17.6 GB |
| 16,000 tokens | 31 s | 510 | 19.8 GB |
| 32,000 tokens | 78 s | 410 | 24.3 GB |
- Load: about 1.5 s once the files are in the page cache; 15.4 GB at rest.
- 64 short decisions (about 115 tokens each) in one
decide([...])call: 14.4 s, 4.4 decisions/s. - Locate, 3 questions on a 1,100-token state: 2.3 s. The adapter adds about 1.9 GB.
- One 1.6 MP image (1,645 positions): 3.3 s.
These were measured in the default power mode.
Tips:
- Use
mode="isolated"when several questions share a state. - Batch requests of similar length (
decide([...])does this for you). Padding is the main waste. - Full-attention layers need every key at once, so memory grows with input length and inputs cannot be chunked.
The default limit is 32,768 tokens per sequence (
max_length=); the server defaults to 8,192. - On a MacBook, High Power mode can help:
sudo pmset -a powermode 2(it roughly doubled the speed of the v1.1 conversion; not measured here).
Conversion details
- Source:
perplexity-ai/pplx-decider-v1.3@9c2ebb27, all weight files verified againstrelease-manifest.json. - The
Qwen3_5Modelbackbone is mapped to themlx_vlm.models.qwen3_5layout. The readout (readout.safetensors,[255, 5120]) is stored aslanguage_model.lm_head. - Quantization: affine, 4 bits, group size 64, on the linear layers and the token embedding of the language backbone.
Kept unquantized: the vision tower, the readout, the convolutions,
A_log/dt_bias, and the two small gate projections of each Gated DeltaNet layer (in_proj_a,in_proj_b; 40 MB in total, they drive the recurrent decay directly and keeping them lowered the error by about a sixth). - RMSNorm weights are stored in fp32 as
1 + wand applied in fp32, as the reference does. A stock mlx-vlm conversion adds the 1 in bf16, which rounds the larger weights. locate-span/is copied unchanged (fp32).convert.pyreproduces the weights and configs of this folder bit for bit from the original checkpoint:python convert.py <original_dir> <output_dir> --bits 4 --keep in_proj_a,in_proj_b.- Size: 15.4 GB of backbone weights in three shards, plus 1.9 GB for
locate-span/: 17.3 GB in total. - Tested with mlx 0.32.4, mlx-vlm 0.8.0 and transformers 5.23.0 (see
requirements.txt).decider_mlx.pyuses mlx-vlm internals, so keep the pinned mlx-vlm version.
Limitations
- The CUDA fast path (
fast=True) of the original is not ported. Video input is not supported. - One
Deciderinstance runs on one thread at a time;server.pyhandles concurrency with a single GPU worker. - 4-bit quantization moves probabilities a little (see the table above). If your application depends on probability values or thresholds, calibrate on your own data, as the original card also recommends.
- This is an unofficial community conversion. See the original model card for intended use and limits.
License and credits
Apache-2.0, the same license as the original model. See LICENSE and NOTICE. All credit for the model goes to
Perplexity (model card,
blog post). It is built on
Qwen3.8-27B, and much of its training data comes from
tasksource:
@inproceedings{sileo-2024-tasksource,
title = "tasksource: A Large Collection of {NLP} tasks with a Structured Dataset Preprocessing Framework",
author = "Sileo, Damien",
booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
month = may,
year = "2024",
address = "Torino, Italia",
publisher = "ELRA and ICCL",
url = "https://aclanthology.org/2024.lrec-main.1361",
pages = "15655--15684",
}
- Downloads last month
- 95
4-bit