README.md

---
language:
- en
library_name: mlx
license: mit
pipeline_tag: image-text-to-text
tags:
- multimodal
- computer-use
- cua
- web-agent
- agent
- vision-language
- mlx
base_model: microsoft/Fara-7B
---

# mlx-community/Fara-7B-4bit

[microsoft/Fara-7B](https://huggingface.co/microsoft/Fara-7B) converted to MLX and
quantized to **4-bit**, for inference on Apple Silicon.

Fara-7B is a computer-use agent model built on Qwen2.5-VL — it reads screenshots
and acts on interfaces. **The vision path is preserved** in this conversion, which
for this model class is the point.

See also [Fara-7B-8bit](https://huggingface.co/mlx-community/Fara-7B-8bit) for the
higher-fidelity variant, and [Fara1.5-9B-8bit](https://huggingface.co/mlx-community/Fara1.5-9B-8bit)
for the newer generation of the same family.

## Quantization

| | |
|---|---|
| Requested bits | 4 |
| Group size | 64 |
| Mode | affine |
| **Effective bits per weight** | **5.44** |
| On-disk size | 5.3 GB |

Effective bits exceed the requested value because `mlx-vlm` quantizes only the
language model and leaves the **vision tower in bf16** by design — 390 vision
tensors, none of them quantized. The vision encoder is a small share of the
weights but disproportionately sensitive to quantization error.

```
language_model : 198 tensors quantized (4-bit, group size 64)
vision_tower   : 390 tensors, bf16    <- unquantized
```

## Fidelity vs the original weights

Measured against the **bf16 source**, tensor by tensor, over all 198 quantized
tensors (7,615,283,200 parameters). No prompts or sampling involved — this is a
direct measurement of how much numerical information the quantization discarded,
and it is exact and hardware-independent.

| Metric | 4-bit | 8-bit |
|---|---|---|
| Relative L2 error | **9.38%** | 0.74% |
| Cosine similarity | **0.995603** | 0.999973 |
| Signal-to-quantization-noise | **20.55 dB** | 42.66 dB |
| Worst single-element error | 0.089844 | 0.007812 |

Highest-error tensors at 4-bit — `v_proj` and early-layer `down_proj` are
consistently the most sensitive:

```
rel_l2=0.10771  snr= 19.36 dB  language_model.model.layers.1.mlp.down_proj
rel_l2=0.10725  snr= 19.39 dB  language_model.model.layers.23.self_attn.v_proj
rel_l2=0.10547  snr= 19.54 dB  language_model.model.layers.22.self_attn.v_proj
rel_l2=0.10419  snr= 19.64 dB  language_model.model.layers.25.self_attn.v_proj
rel_l2=0.10355  snr= 19.70 dB  language_model.model.layers.27.self_attn.v_proj
```

**Going from 4-bit to 8-bit improves the signal-to-quantization-noise ratio by
22 dB** — a factor of roughly 160 in noise power. If your machine has the memory,
prefer the 8-bit variant; use this one when 5.3 GB versus 8.8 GB is the deciding
constraint.

## Throughput

Measured on an M2 Pro / 32 GB, 64 generated tokens, greedy.

| Variant | Decode tok/s | Prompt tok/s | Peak RAM |
|---|---|---|---|
| 4-bit | 36.0 | 111.6 | 5.80 GB |
| 8-bit | 18.8 | 105.5 | 9.57 GB |

This variant decodes **1.9x faster** at 1.7x less memory. Numbers do not transfer
across chips.

## Why there is no behavioural evaluation

Other conversions in this series report perplexity ratio, top-1 agreement and KL
divergence against the bf16 source — see
[Fara1.5-9B-8bit](https://huggingface.co/mlx-community/Fara1.5-9B-8bit), which
reaches top-1 agreement of 1.000 that way. **That protocol does not work for
Fara-7B, and the reason is worth stating rather than quietly omitting.**

Fara-7B is a computer-use model: it expects a screenshot plus an action space, not
prose. Scored on plain text it is out of distribution *before* any quantization —
the unquantized bf16 source itself has a perplexity of **12.30** on the same
passages where Fara1.5-9B scores 3.23. With a distribution that flat, the metric
stops discriminating: measured that way this 4-bit variant scored a *better*
perplexity ratio (1.2312) and KL (0.2768) than the 8-bit one (1.4897 / 0.4809),
which is impossible — a 4-bit quantization cannot be more faithful than an 8-bit
one of the same model.

The weight-level numbers above give the correct ordering, so the anomaly is in the
measurement, not in the weights. Those behavioural figures are therefore excluded
rather than reported.

A meaningful behavioural benchmark for this model would need screenshots and a
verifiable action space — a computer-use harness, which was not available here.

## What was not measured

No standard benchmarks: no ScreenSpot, WebArena, OSWorld, or any agentic
evaluation. No judged quality. The vision path was verified to load and run, not
scored on a dataset. **If your use case is the full computer-use loop, evaluate on
your own tasks.**

## Usage

```bash
pip install mlx-vlm
```

```python
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("mlx-community/Fara-7B-4bit")

prompt = apply_chat_template(
    processor, model.config,
    "Describe this screenshot. What buttons do you see?",
    num_images=1,
)
out = generate(model, processor, prompt, image=["screenshot.png"], max_tokens=256)
print(out.text)
```

Text-only works too — pass `num_images=0` and omit `image`.

Note that stock `mlx-lm` loads the **text path only**; use `mlx-vlm` for image
input.

## Credits

All credit for the model belongs to Microsoft. This is a format conversion and
quantization; no training or fine-tuning was performed. Licensed MIT, as the
original. See the [original card](https://huggingface.co/microsoft/Fara-7B) for
intended use and limitations.