mlx-community/pplx-decider-v1.3-4bit

An MLX 4-bit conversion of perplexity-ai/pplx-decider-v1.3 for Apple Silicon, with a standalone inference script and a small HTTP server. It supports everything in the original release except the CUDA-only fast path: decisions (choice, noul, score), images, Locate span extraction, and both question modes.

This is a classifier, not a chat model. Give it a state (text, JSON and/or images) and one or more questions. One forward pass returns calibrated probabilities over each question's options. There is no generation step.

Question type criteria Answer
choice {option_name: description or None}, 1 to 255 options choice, probabilities, confidence
noul optional {"true": description, "false": description} noul = P(true)
score list of 2 to 255 level descriptions, lowest first score (expected level), probabilities, legend, confidence
locate not used; the question key is the label, or instructions if given spans: start, end, text, score

Why it needs the included code

mlx_lm.load / mlx_vlm.load will not give correct results with this checkpoint:

  • The 16 full-attention layers are noncausal: every token sees the whole sequence, and only padding is masked. The 48 Gated DeltaNet layers stay causal. With a causal mask the top answer changes on 16 of 115 of our test questions and the probabilities are far off (mean KL 0.076, against 0.005 for this model).
  • The output is a 255-row decision readout (stored as language_model.lm_head), not a vocabulary head. It is read at each question's <|fim_suffix|> marker, and the softmax runs over that question's options only, at the saved calibration temperature (1.2264).
  • The prompt is the v1.3 chunked protocol, with the state and every question tokenized separately.

decider_mlx.py implements all of this. It is a port of the official pplx_decider package to MLX and uses the same request and response format.

Quick start

Requires an Apple Silicon Mac and Python 3.12+. The model uses 15.4 GB at rest and more for long inputs or Locate (see Speed and memory), so 32 GB of memory is recommended; 24 GB is enough for short text inputs.

hf download mlx-community/pplx-decider-v1.3-4bit --local-dir pplx-decider-v1.3-4bit
cd pplx-decider-v1.3-4bit
pip install -r requirements.txt     # mlx, mlx-vlm, transformers (tokenizer only), pillow: no torch needed
from decider_mlx import Decider

decider = Decider.from_pretrained(".")   # or "mlx-community/pplx-decider-v1.3-4bit"
response = decider.decide({
    "state": "My integration keeps failing with a 401 since we rotated our API key. Please help.",
    "questions": {
        "team": {
            "type": "choice",
            "instructions": "Which team should handle this request?",
            "criteria": {
                "billing": "Charges and refunds",
                "support": "Technical integration errors",
                "sales": "Questions about buying a product",
            },
        },
        "urgent": {"type": "noul", "instructions": "Is the customer blocked right now?"},
        "frustration": {"type": "score", "criteria": ["calm", "mildly annoyed", "frustrated", "angry"]},
        "error": {"type": "locate", "instructions": "Error codes"},
    },
    # "images": ["chart.png"],   # optional: file paths, data:image/... URLs or PIL images (not with locate)
})
print(response["answers"]["team"]["choice"], response["answers"]["urgent"]["noul"])
print(response["answers"]["error"]["spans"])   # [{"start": 36, "end": 39, "text": "401", "score": 0.99...}]

decide takes one request and returns one response, or a list and returns a list in the same order. Each response is {"answers": {"<id>": {...}}, "usage": {"input_tokens", "output_tokens", "sequences"}}. decider.logits(requests) returns the raw readout logits of each question's options, before temperature. See the original model card for the full request schema, calibration advice and limits; they apply unchanged.

CLI:

python decider_mlx.py --state "Win a FREE iPhone, click now!!!" \
  --questions '{"spam": {"type": "noul", "instructions": "Is this spam?"}}'

Several questions in one request

Decider.from_pretrained(..., mode=...) selects how the questions of a request are run, as in the original:

  • mode="separate" (default): each question is its own sequence (state + that question). Questions do not see each other. This is the computation Perplexity evaluated.
  • mode="isolated": one sequence per request. The state is read once and each question attends the state and itself. It is much cheaper when several questions share a state, but the state also attends the questions, so answers can shift slightly with the set of questions (Perplexity reports 1.5-2.2% of top answers differing).

Time for one request on an M5 Max, this model:

State Questions separate isolated
200 tokens 8 3.9 s 0.9 s
1,000 tokens 8 15.2 s 2.5 s
4,000 tokens 8 58 s 7.5 s
4,000 tokens 32 232 s 9.3 s

Locate

"type": "locate" extracts verbatim spans of the state, with character offsets (state[start:end] == text) and a score. The adapter (an unmerged rank-64 LoRA, a trained <|locate|> embedding and a span head, about 1.9 GB) is loaded on the first Locate request and is switched off for decision questions, so decisions are unaffected. All Locate questions of a request run together in one isolated sequence, as in the original. Locate needs a nonempty text state and no images. Decider.from_pretrained(..., locate=False) disables it.

Images

Images go in "images". They are placed before the state, and every question can see them. Each image is resized to at most 1,638,400 pixels (the checkpoint's max_pixels); a 1.6 MP image uses about 1,600 sequence positions.

Server

python server.py --model . --port 8092        # add --mode isolated to run each request as one sequence
curl localhost:8092/v1/systemone -H 'Content-Type: application/json' -d '{
  "state": "My integration keeps failing with a 401 since we rotated our API key. Please help.",
  "questions": {
    "urgent": {"type": "noul", "instructions": "Is the customer blocked right now?"},
    "error":  {"type": "locate", "instructions": "Error codes"}
  }}'

The request takes state (a string, JSON, or chat messages with image_url parts), questions and optional images (data URLs, up to 8), the same shape as llama.cpp's llama-server /v1/systemone. The response has answers and usage. Also available: GET /v1/models, /health and /stats. Set DECIDER_API_KEY to require a bearer token. One GPU worker runs MLX; concurrent requests are batched together by similar length.

Accuracy

Checked against the official PyTorch implementation (pplx_decider 0.5.0, transformers 5.17.0), run in fp32 with one request per forward pass. The test set has 77 requests: one row from each of 50 public benchmark families (MMLU-Pro, BANKING77, ContractNLI, BFCL, RAGTruth, ...) plus hand-written edge cases: 255 options, a single option, JSON and chat states, special tokens in user text, whitespace-only and empty states, several languages, images of different sizes, and Locate. Sequences run from 66 to 7,300 tokens.

Tokenization: the token ids of every sequence are identical to the reference's.

Decisions (probabilities over each question's options, at the saved temperature):

questions top answer agrees mean KL mean abs 螖p max abs 螖p
MLX bf16 (the port without quantization), separate 115 115/115 0.00008 0.002 0.015
this model (4-bit), separate 115 112/115 0.0054 0.018 0.26
this model (4-bit), isolated 19 19/19 0.0015 0.006 0.031
control: this model with causal attention (the wrong mode) 115 99/115 0.076 0.085 0.70

The three top answers that changed were close calls in the reference: its top two options were 0.02, 0.14 and 0.16 apart. Isolated mode was compared on 6 multi-question requests only, because the reference run on CPU was stopped at a time limit.

Against the unquantized MLX port on a larger set (127 requests, 303 questions, both modes), this model agrees on the top answer for 192/197 questions in separate mode and 103/106 in isolated mode (mean KL 0.007 and 0.003). On the 100 benchmark rows of that set, accuracy against the benchmark labels is 115/140 for this model and 112/140 unquantized, a difference within noise.

Locate: on 17 Locate questions the unquantized port returns the same 72 spans as the reference (scores within 0.023). This 4-bit model returns 70 of them. It misses two spans that the reference scored 0.55 and 0.73, adds one spurious long span at 0.54, and moves one score from 0.92 to 0.63; the other scores differ by 0.007 on average. Locate is more sensitive to quantization than decisions, mostly near the 0.5 cutoff.

Port correctness was checked separately on the smaller pplx-decider-flash-v1.3, which uses the same code and fits in memory in fp32 on both sides. MLX fp32 against PyTorch fp32: 46/46 top answers in separate mode and 62/62 in isolated mode, mean KL 5e-7 and 7e-7, and 71/71 Locate spans identical with scores within 0.0014.

Speed and memory

Measured on an Apple M5 Max (128 GB), text input, one request at a time unless noted.

Input Time Tokens/s Peak memory
80 tokens 0.15 s 545 15.6 GB
1,000 tokens 1.1 s 890 16.3 GB
4,000 tokens 6.1 s 660 17.2 GB
8,000 tokens 13.2 s 610 17.6 GB
16,000 tokens 31 s 510 19.8 GB
32,000 tokens 78 s 410 24.3 GB
  • Load: about 1.5 s once the files are in the page cache; 15.4 GB at rest.
  • 64 short decisions (about 115 tokens each) in one decide([...]) call: 14.4 s, 4.4 decisions/s.
  • Locate, 3 questions on a 1,100-token state: 2.3 s. The adapter adds about 1.9 GB.
  • One 1.6 MP image (1,645 positions): 3.3 s.

These were measured in the default power mode.

Tips:

  • Use mode="isolated" when several questions share a state.
  • Batch requests of similar length (decide([...]) does this for you). Padding is the main waste.
  • Full-attention layers need every key at once, so memory grows with input length and inputs cannot be chunked. The default limit is 32,768 tokens per sequence (max_length=); the server defaults to 8,192.
  • On a MacBook, High Power mode can help: sudo pmset -a powermode 2 (it roughly doubled the speed of the v1.1 conversion; not measured here).

Conversion details

  • Source: perplexity-ai/pplx-decider-v1.3 @ 9c2ebb27, all weight files verified against release-manifest.json.
  • The Qwen3_5Model backbone is mapped to the mlx_vlm.models.qwen3_5 layout. The readout (readout.safetensors, [255, 5120]) is stored as language_model.lm_head.
  • Quantization: affine, 4 bits, group size 64, on the linear layers and the token embedding of the language backbone. Kept unquantized: the vision tower, the readout, the convolutions, A_log / dt_bias, and the two small gate projections of each Gated DeltaNet layer (in_proj_a, in_proj_b; 40 MB in total, they drive the recurrent decay directly and keeping them lowered the error by about a sixth).
  • RMSNorm weights are stored in fp32 as 1 + w and applied in fp32, as the reference does. A stock mlx-vlm conversion adds the 1 in bf16, which rounds the larger weights.
  • locate-span/ is copied unchanged (fp32).
  • convert.py reproduces the weights and configs of this folder bit for bit from the original checkpoint: python convert.py <original_dir> <output_dir> --bits 4 --keep in_proj_a,in_proj_b.
  • Size: 15.4 GB of backbone weights in three shards, plus 1.9 GB for locate-span/: 17.3 GB in total.
  • Tested with mlx 0.32.4, mlx-vlm 0.8.0 and transformers 5.23.0 (see requirements.txt). decider_mlx.py uses mlx-vlm internals, so keep the pinned mlx-vlm version.

Limitations

  • The CUDA fast path (fast=True) of the original is not ported. Video input is not supported.
  • One Decider instance runs on one thread at a time; server.py handles concurrency with a single GPU worker.
  • 4-bit quantization moves probabilities a little (see the table above). If your application depends on probability values or thresholds, calibrate on your own data, as the original card also recommends.
  • This is an unofficial community conversion. See the original model card for intended use and limits.

License and credits

Apache-2.0, the same license as the original model. See LICENSE and NOTICE. All credit for the model goes to Perplexity (model card, blog post). It is built on Qwen3.8-27B, and much of its training data comes from tasksource:

@inproceedings{sileo-2024-tasksource,
    title = "tasksource: A Large Collection of {NLP} tasks with a Structured Dataset Preprocessing Framework",
    author = "Sileo, Damien",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.1361",
    pages = "15655--15684",
}
Downloads last month
95
Safetensors
Model size
4B params
Tensor type
BF16
路
U32
路
F32
路
MLX
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 馃檵 Ask for provider support

Model tree for mlx-community/pplx-decider-v1.3-4bit

Base model

Qwen/Qwen3.8-27B
Quantized
(5)
this model