README.md

---
library_name: mlx
license: apache-2.0
pipeline_tag: text-generation
base_model: BAAI/AREX-Turbo
language:
- en
tags:
- agent
- deep-research
- reasoning
- tool-use
- long-context
- qwen3.5
- dense
- image-text-to-text
- mlx
---

# mlx-community/AREX-Turbo-8bit

[BAAI/AREX-Turbo](https://huggingface.co/BAAI/AREX-Turbo) converted to MLX and quantized to
**8-bit**, for inference on Apple Silicon.

Converted with `mlx-vlm` 0.6.8 (`mlx` 0.32.0).

## Quantization

| | |
|---|---|
| Requested bits | 8 |
| Group size | 64 |
| Mode | affine |
| **Effective bits per weight** | **9.053** |
| On-disk size | 4.8 GB |

The effective figure exceeds 8 because `mlx-vlm` **leaves the vision tower
in bf16 by design** and quantizes only the language model — the vision encoder is
a small share of the weights but disproportionately sensitive to quantization
error. Verified on this conversion:

```
language_model: QuantizedLinear x186, QuantizedEmbedding x1   <- 8-bit
vision_tower  : Linear x50, LayerNorm x25, Conv3d x1          <- bf16
```

## Evaluation

All numbers below were measured against the **unquantized bf16 source** on an
M2 Pro (32 GB), greedy decoding throughout.

### Distributional fidelity (teacher-forced, deterministic)

Perplexity over 204 tokens of held-out text spanning prose, code, legal and
scientific registers; top-1 agreement and KL are computed per position over the
full next-token distribution. These are the numbers to judge quantization by —
they involve no sampling and no decoding choices.

| Metric | This model (8-bit) | bf16 reference |
|---|---|---|
| Perplexity | 3.008 | 3.0171 |
| Perplexity ratio | **0.997** | 1.000 |
| Top-1 agreement with bf16 | **0.9951** | — |
| KL(bf16 ‖ quant) | **0.000691** nats/token | 0 |

### Generation agreement vs bf16

Greedy (`temperature=0.0`) continuations, scored against the bf16 output as
reference.

| Metric | Score |
|---|---|
| BLEU | 82.65 |
| chrF | 87.05 |
| ROUGE-1 / ROUGE-L | 0.9122 / 0.9122 |
| Exact match | 4/6 |

**Read these as agreement, not quality.** bf16 is the reference here, not ground
truth, so a divergence is only a defect if the quantized answer is *worse*. It
is not always: on `17 * 23` the bf16 model started a long derivation while the
8-bit model answered `391` directly — a mismatch that counts against BLEU
while being the better response. That is why the two sections below exist.

### Task accuracy (ground truth, no judge)

Pass rate on 8 short prompts with verifiable answers (arithmetic, factual
recall, sorting). Deterministic — this is the only layer that measures
correctness rather than similarity.

| | Accuracy |
|---|---|
| bf16 | 7/8 |
| **8-bit** | **7/8** |

### Judged quality (LLM-as-judge, blind)

18 open-ended prompts graded by `claude-sonnet-5`. The judge never sees which
model produced which answer. Pairwise comparisons are run **twice with positions
swapped**; a verdict counts only if the judge picks the same model both times,
and disagreements are reported as `inconsistent` rather than resolved silently.
Absolute grades are averaged over 2 repeats per answer.

| Metric | bf16 | 8-bit |
|---|---|---|
| Absolute quality (1-5) | 4.722 ± 0.101 | 4.694 ± 0.115 |
| Pairwise wins | 1 | 1 |
| Ties | 12 | |
| Inconsistent (judge flipped) | 4 | |

Gap of **0.028** against combined judge noise of **0.216** →
**indistinguishable from bf16**.

Judge noise is not negligible and is reported rather than hidden: grading the
same bf16 model across separate runs varied by ~0.17 on this scale. Differences
smaller than the combined SEM should not be read as a ranking.

### Across all variants

| Variant | bpw | PPL ratio | Top-1 agree | KL | BLEU | Accuracy | Judge (1-5) | Size |
|---|---|---|---|---|---|---|---|---|
| 4-bit | 5.347 | 1.1158 | 0.9265 | 0.052755 | 49.52 | 8/8 | 4.583 | 2.9 GB |
| 6-bit | 7.2 | 1.0014 | 0.9902 | 0.002364 | 65.02 | 7/8 | 4.583 | 3.8 GB |
| 8-bit | 9.053 | 0.997 | 0.9951 | 0.000691 | 82.65 | 7/8 | 4.694 | 4.8 GB |

BLEU against bf16 falls from 82.7 (8-bit) to 49.5 (4-bit), while task accuracy
and judged quality stay flat. Judged quality is statistically indistinguishable
from bf16 at every bit width tested. KL divergence from bf16 drops ~76x across
that same range.

### Throughput (M2 Pro, 32 GB, 128-token decode)

Hardware-specific; will differ on other chips.

| Variant | Decode tok/s | Peak RAM |
|---|---|---|
| bf16 (source) | 18.2 | 9.229 GB |
| 4-bit | 60.9 | 3.677 GB |
| 6-bit | 44.5 | 4.917 GB |
| 8-bit | 34.7 | 6.152 GB |

### What was not measured

No standard task benchmarks (MMLU, GSM8K, agentic/tool-use evals) were run. The
accuracy layer above is 8 short prompts, not a benchmark. The vision path
was checked for coherence on sample images but not scored. The source model is
agent/deep-research oriented, and none of its agentic capabilities were
evaluated here — if that is your use case, measure on your own data.

## Usage

```bash
pip install mlx-vlm
```

```python
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

model, processor = load("mlx-community/AREX-Turbo-8bit")
config = load_config("mlx-community/AREX-Turbo-8bit")

prompt = apply_chat_template(processor, config, "What can you do?", num_images=0)
print(generate(model, processor, prompt, max_tokens=256, verbose=False).text)
```

With an image:

```python
prompt = apply_chat_template(processor, config, "Describe this image.", num_images=1)
out = generate(model, processor, prompt, image=["<path-or-url>"], max_tokens=256)
print(out.text)
```

See the [original model card](https://huggingface.co/BAAI/AREX-Turbo) for capabilities,
intended use and limitations. All credit for the model belongs to its authors.