mlx-community/AREX-Turbo-8bit

BAAI/AREX-Turbo converted to MLX and quantized to 8-bit, for inference on Apple Silicon.

Converted with mlx-vlm 0.6.8 (mlx 0.32.0).

Quantization

Requested bits	8
Group size	64
Mode	affine
Effective bits per weight	9.053
On-disk size	4.8 GB

The effective figure exceeds 8 because mlx-vlm leaves the vision tower in bf16 by design and quantizes only the language model — the vision encoder is a small share of the weights but disproportionately sensitive to quantization error. Verified on this conversion:

language_model: QuantizedLinear x186, QuantizedEmbedding x1   <- 8-bit
vision_tower  : Linear x50, LayerNorm x25, Conv3d x1          <- bf16

Evaluation

All numbers below were measured against the unquantized bf16 source on an M2 Pro (32 GB), greedy decoding throughout.

Distributional fidelity (teacher-forced, deterministic)

Perplexity over 204 tokens of held-out text spanning prose, code, legal and scientific registers; top-1 agreement and KL are computed per position over the full next-token distribution. These are the numbers to judge quantization by — they involve no sampling and no decoding choices.

Metric	This model (8-bit)	bf16 reference
Perplexity	3.008	3.0171
Perplexity ratio	0.997	1.000
Top-1 agreement with bf16	0.9951	—
KL(bf16 ‖ quant)	0.000691 nats/token	0

Generation agreement vs bf16

Greedy (temperature=0.0) continuations, scored against the bf16 output as reference.

Metric	Score
BLEU	82.65
chrF	87.05
ROUGE-1 / ROUGE-L	0.9122 / 0.9122
Exact match	4/6

Read these as agreement, not quality. bf16 is the reference here, not ground truth, so a divergence is only a defect if the quantized answer is worse. It is not always: on 17 * 23 the bf16 model started a long derivation while the 8-bit model answered 391 directly — a mismatch that counts against BLEU while being the better response. That is why the two sections below exist.

Task accuracy (ground truth, no judge)

Pass rate on 8 short prompts with verifiable answers (arithmetic, factual recall, sorting). Deterministic — this is the only layer that measures correctness rather than similarity.

	Accuracy
bf16	7/8
8-bit	7/8

Judged quality (LLM-as-judge, blind)

18 open-ended prompts graded by claude-sonnet-5. The judge never sees which model produced which answer. Pairwise comparisons are run twice with positions swapped; a verdict counts only if the judge picks the same model both times, and disagreements are reported as inconsistent rather than resolved silently. Absolute grades are averaged over 2 repeats per answer.

Metric	bf16	8-bit
Absolute quality (1-5)	4.722 ± 0.101	4.694 ± 0.115
Pairwise wins	1	1
Ties	12
Inconsistent (judge flipped)	4

Gap of 0.028 against combined judge noise of 0.216 → indistinguishable from bf16.

Judge noise is not negligible and is reported rather than hidden: grading the same bf16 model across separate runs varied by ~0.17 on this scale. Differences smaller than the combined SEM should not be read as a ranking.

Across all variants

Variant	bpw	PPL ratio	Top-1 agree	KL	BLEU	Accuracy	Judge (1-5)	Size
4-bit	5.347	1.1158	0.9265	0.052755	49.52	8/8	4.583	2.9 GB
6-bit	7.2	1.0014	0.9902	0.002364	65.02	7/8	4.583	3.8 GB
8-bit	9.053	0.997	0.9951	0.000691	82.65	7/8	4.694	4.8 GB

BLEU against bf16 falls from 82.7 (8-bit) to 49.5 (4-bit), while task accuracy and judged quality stay flat. Judged quality is statistically indistinguishable from bf16 at every bit width tested. KL divergence from bf16 drops ~76x across that same range.

Throughput (M2 Pro, 32 GB, 128-token decode)

Hardware-specific; will differ on other chips.

Variant	Decode tok/s	Peak RAM
bf16 (source)	18.2	9.229 GB
4-bit	60.9	3.677 GB
6-bit	44.5	4.917 GB
8-bit	34.7	6.152 GB

What was not measured

No standard task benchmarks (MMLU, GSM8K, agentic/tool-use evals) were run. The accuracy layer above is 8 short prompts, not a benchmark. The vision path was checked for coherence on sample images but not scored. The source model is agent/deep-research oriented, and none of its agentic capabilities were evaluated here — if that is your use case, measure on your own data.

Usage

pip install mlx-vlm

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

model, processor = load("mlx-community/AREX-Turbo-8bit")
config = load_config("mlx-community/AREX-Turbo-8bit")

prompt = apply_chat_template(processor, config, "What can you do?", num_images=0)
print(generate(model, processor, prompt, max_tokens=256, verbose=False).text)

With an image:

prompt = apply_chat_template(processor, config, "Describe this image.", num_images=1)
out = generate(model, processor, prompt, image=["<path-or-url>"], max_tokens=256)
print(out.text)

See the original model card for capabilities, intended use and limitations. All credit for the model belongs to its authors.