README.md

---
language:
- en
library_name: mlx
license: mit
pipeline_tag: image-text-to-text
tags:
- multimodal
- computer-use
- cua
- web-agent
- agent
- vision-language
- mlx
base_model: microsoft/Fara-7B
---

# mlx-community/Fara-7B-8bit

[microsoft/Fara-7B](https://huggingface.co/microsoft/Fara-7B) converted to MLX and
quantized to **8-bit**, for inference on Apple Silicon.

Fara-7B is a computer-use agent model built on Qwen2.5-VL — it reads screenshots
and acts on interfaces. **The vision path is preserved** in this conversion, which
for this model class is the point.

See also [Fara-7B-4bit](https://huggingface.co/mlx-community/Fara-7B-4bit) for the
smaller variant, and [Fara1.5-9B-8bit](https://huggingface.co/mlx-community/Fara1.5-9B-8bit)
for the newer generation of the same family.

## Quantization

| | |
|---|---|
| Requested bits | 8 |
| Group size | 64 |
| Mode | affine |
| **Effective bits per weight** | **9.11** |
| On-disk size | 8.8 GB |
| Shards | 2 |

Effective bits exceed the requested value because `mlx-vlm` quantizes only the
language model and leaves the **vision tower in bf16** by design — 390 vision
tensors, none of them quantized. The vision encoder is a small share of the
weights but disproportionately sensitive to quantization error.

```
language_model : 198 tensors quantized (8-bit, group size 64)
vision_tower   : 390 tensors, bf16    <- unquantized
```

## Fidelity vs the original weights

Measured against the **bf16 source**, tensor by tensor, over all 198 quantized
tensors (7,615,283,200 parameters). No prompts or sampling involved — this is a
direct measurement of how much numerical information the quantization discarded,
and it is exact and hardware-independent.

| Metric | 8-bit | 4-bit |
|---|---|---|
| Relative L2 error | **0.74%** | 9.38% |
| Cosine similarity | **0.999973** | 0.995603 |
| Signal-to-quantization-noise | **42.66 dB** | 20.55 dB |
| Worst single-element error | 0.007812 | 0.089844 |

Highest-error tensors at 8-bit — `v_proj` and early-layer `down_proj` are
consistently the most sensitive:

```
rel_l2=0.00870  snr= 41.21 dB  language_model.model.layers.1.mlp.down_proj
rel_l2=0.00868  snr= 41.23 dB  language_model.model.layers.23.self_attn.v_proj
rel_l2=0.00848  snr= 41.44 dB  language_model.model.layers.22.self_attn.v_proj
rel_l2=0.00847  snr= 41.44 dB  language_model.model.layers.25.self_attn.v_proj
rel_l2=0.00824  snr= 41.68 dB  language_model.lm_head
```

## Throughput

Measured on an M2 Pro / 32 GB, 64 generated tokens, greedy.

| Variant | Decode tok/s | Prompt tok/s | Peak RAM |
|---|---|---|---|
| 8-bit | 18.8 | 105.5 | 9.57 GB |
| 4-bit | 36.0 | 111.6 | 5.80 GB |

The 4-bit variant decodes **1.9x faster** at 1.7x less memory, at the cost of the
fidelity difference shown above (20.55 dB vs 42.66 dB). Numbers do not transfer
across chips.

## Why there is no behavioural evaluation

Other conversions in this series report perplexity ratio, top-1 agreement and KL
divergence against the bf16 source — see
[Fara1.5-9B-8bit](https://huggingface.co/mlx-community/Fara1.5-9B-8bit), which
reaches top-1 agreement of 1.000 that way. **That protocol does not work for
Fara-7B, and the reason is worth stating rather than quietly omitting.**

Fara-7B is a computer-use model: it expects a screenshot plus an action space, not
prose. Scored on plain text it is out of distribution *before* any quantization —
the unquantized bf16 source itself has a perplexity of **12.30** on the same
passages where Fara1.5-9B scores 3.23. With a distribution that flat, the metric
stops discriminating. Measured that way, the 4-bit variant came out **better** than
the 8-bit one:

| Text-only teacher forcing | 8-bit | 4-bit |
|---|---|---|
| Perplexity ratio | 1.4897 | 1.2312 |
| KL (nats/token) | 0.4809 | 0.2768 |

That ordering is impossible — a 4-bit quantization cannot be more faithful than an
8-bit one of the same model. The weight-level numbers above confirm the correct
ordering (42.66 dB vs 20.55 dB), so the anomaly is in the measurement, not in the
weights. Reporting those behavioural figures would have been misleading, so they
are excluded and the exact weight-level comparison is used instead.

A meaningful behavioural benchmark for this model would need screenshots and a
verifiable action space — a computer-use harness, which was not available here.

## What was not measured

No standard benchmarks: no ScreenSpot, WebArena, OSWorld, or any agentic
evaluation. No judged quality. The vision path was verified to load and run, not
scored on a dataset. **If your use case is the full computer-use loop, evaluate on
your own tasks.**

## Usage

```bash
pip install mlx-vlm
```

```python
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("mlx-community/Fara-7B-8bit")

prompt = apply_chat_template(
    processor, model.config,
    "Describe this screenshot. What buttons do you see?",
    num_images=1,
)
out = generate(model, processor, prompt, image=["screenshot.png"], max_tokens=256)
print(out.text)
```

Text-only works too — pass `num_images=0` and omit `image`.

Note that stock `mlx-lm` loads the **text path only**; use `mlx-vlm` for image
input.

## Credits

All credit for the model belongs to Microsoft. This is a format conversion and
quantization; no training or fine-tuning was performed. Licensed MIT, as the
original. See the [original card](https://huggingface.co/microsoft/Fara-7B) for
intended use and limitations.