Instructions to use mlx-community/Fara1.5-4B-8bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mlx-community/Fara1.5-4B-8bit with MLX:

# Make sure mlx-vlm is installed
# pip install --upgrade mlx-vlm

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

# Load the model
model, processor = load("mlx-community/Fara1.5-4B-8bit")
config = load_config("mlx-community/Fara1.5-4B-8bit")

# Prepare input
image = ["http://images.cocodataset.org/val2017/000000039769.jpg"]
prompt = "Describe this image."

# Apply chat template
formatted_prompt = apply_chat_template(
    processor, config, prompt, num_images=1
)

# Generate output
output = generate(model, processor, formatted_prompt, image)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps Settings
LM Studio

How to use mlx-community/Fara1.5-4B-8bit with Pi:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "mlx-community/Fara1.5-4B-8bit"

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "mlx-lm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "mlx-community/Fara1.5-4B-8bit"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

OpenClaw new

How to use mlx-community/Fara1.5-4B-8bit with OpenClaw:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "mlx-community/Fara1.5-4B-8bit"

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "mlx-community/Fara1.5-4B-8bit" \
  --custom-provider-id mlx-lm \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Hermes Agent

How to use mlx-community/Fara1.5-4B-8bit with Hermes Agent:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "mlx-community/Fara1.5-4B-8bit"

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default mlx-community/Fara1.5-4B-8bit

Run Hermes

hermes

Fara1.5-4B-8bit / README.md

axiom-of-choice

Add 8-bit MLX quantization with measured fidelity vs bf16

e9d0e3a verified 6 days ago

preview code

Raw

History Blame Contribute Delete

6.38 kB

	---
	license: mit
	library_name: mlx
	pipeline_tag: image-text-to-text
	tags:
	- mlx
	- computer-use
	- cua
	- web-agent
	- multimodal
	- vision-language
	- agent
	base_model: microsoft/Fara1.5-4B
	---

	# Fara1.5-4B-8bit

	8-bit MLX quantization of
	[`microsoft/Fara1.5-4B`](https://huggingface.co/microsoft/Fara1.5-4B), a 4.5B
	computer-use / web-agent VLM built on Qwen3.5-4B. It reads screenshots and acts on
	interfaces. Converted with `mlx-vlm` 0.6.8 (`mlx` 0.32.0) for inference on Apple
	Silicon.

	Every prior MLX conversion of this model is 4-bit, so 8-bit was the open gap.
	Fidelity was measured against the unquantized bf16 source, not assumed.

	```bash
	pip install mlx-vlm
	python -m mlx_vlm generate \
	--model mlx-community/Fara1.5-4B-8bit \
	--image screenshot.png \
	--prompt "Describe the buttons and the total shown in this screenshot." \
	--max-tokens 256
	```

	Image input needs `mlx-vlm`. Stock `mlx-lm` loads the text path only, which for a
	computer-use model is not a smaller model but a broken one.

	## Quantization

	\| \| \|
	\|---\|---\|
	\| Bits \| 8 \|
	\| Group size \| 64 \|
	\| Mode \| affine \|
	\| Effective bits/weight \| 9.053 \|
	\| Size on disk \| 4.8 GB \|
	\| Quantized tensors \| 249 (language model) \|
	\| Unquantized tensors \| 297 (vision tower, bf16) \|

	Effective bits per weight exceeds the nominal 8 because `mlx-vlm` quantizes only
	the language model and leaves the vision tower in bf16 by design. **The vision
	path is preserved**, which for a computer-use model is the whole point: verified
	at tensor level, of 297 vision tensors zero carry a `.scales` entry.

	### A conversion trap worth knowing about

	This model declares `dtype: bfloat16` at the top level of its config but
	`float32` in both `text_config` and `vision_config`. `mlx_vlm.convert` honours
	the sub-configs, so converting without an explicit `--dtype bfloat16` silently
	keeps the scales, biases and the entire vision tower in fp32:

	\| \| default \| `--dtype bfloat16` \|
	\|---\|---\|---\|
	\| Effective bpw \| 10.695 \| 9.053 \|
	\| Size on disk \| 5.7 GB \| 4.8 GB \|
	\| Peak RAM \| 7.44 GB \| 6.23 GB \|
	\| Weight SNR vs bf16 \| 45.22 dB \| 42.70 dB \|

	This repo is the `--dtype bfloat16` build. The higher SNR of the fp32 build is
	not better fidelity: it is a float32 reconstruction being compared against a
	bf16 source, bought with 0.9 GB of precision that has no business in an 8-bit
	repo. Rounding those scales to bf16 accounts for the entire 2.5 dB difference.

	## Measured fidelity

	Against the unquantized bf16 source, greedy decoding, on an M2 Pro / 32 GB.

	\| Metric \| bf16 (source) \| 8-bit \|
	\|---\|---\|---\|
	\| Weight SNR \| n/a \| 42.70 dB \|
	\| Relative L2 \| n/a \| 0.73% \|
	\| Cosine similarity \| n/a \| 0.999973 \|
	\| Worst single element \| n/a \| 0.003906 \|
	\| Perplexity \| 3.4974 \| 3.4775 \|
	\| Top-1 agreement \| n/a \| 0.9853 \|
	\| KL (nats/token) \| 0 \| 0.00086 \|
	\| Task accuracy \| 7/8 \| 7/8 \|
	\| BLEU vs bf16 \| n/a \| 81.30 \|
	\| Decode tok/s \| 18.5 \| 35.4 \|
	\| Peak RAM \| 9.23 GB \| 6.15 GB \|

	Weight-level metrics stream the source tensor by tensor from disk over all
	4,204,789,760 quantized parameters. No prompts, no sampling, hardware-independent.

	42.70 dB is the expected value, and that is the point. Six 8-bit affine
	conversions at group size 64 across six different architectures (AREX-Turbo,
	Fara-7B, Fara1.5-4B, GLM-4.1V, MagenticBrain, OvisOCR2) all land on
	42.66-42.70 dB. A reading outside that band is a defect signal, which is
	exactly how the fp32-scales issue above was caught.

	Highest-error tensors, all in the linear-attention input projections:

	```
	rel_l2=0.00841 snr=41.50 dB language_model.model.layers.25.linear_attn.in_proj_a
	rel_l2=0.00839 snr=41.53 dB language_model.model.layers.28.linear_attn.in_proj_a
	rel_l2=0.00834 snr=41.57 dB language_model.model.layers.26.linear_attn.in_proj_a
	```

	1.9x faster, 1.5x less memory than bf16.

	### Vision path, verified working rather than merely present

	Given a synthetic UI screenshot, the 8-bit model read both button labels
	(`SubmitOrder`, `Cancel`) and the total (`42.50 USD`) correctly.

	### On the perplexity aggregate

	The harness reports aggregate perplexity as `exp(sum_nll/sum_tokens)` over five
	passages, and flags it as unreliable here: the bf16 source itself spans 10.3x
	across those passages, so a single out-of-domain passage can dominate. The
	per-passage ratios are reported instead, and they agree with the aggregate rather
	than contradict it, all within 1.8% of the bf16 reference:

	\| Passage \| bf16 PPL \| 8-bit PPL \| Ratio \|
	\|---\|---\|---\|---\|
	\| prose \| 4.66 \| 4.69 \| 1.007 \|
	\| python-code \| 1.23 \| 1.23 \| 0.999 \|
	\| legal \| 8.82 \| 8.70 \| 0.986 \|
	\| biology \| 6.62 \| 6.64 \| 1.002 \|
	\| meta \| 12.74 \| 12.51 \| 0.982 \|

	BLEU of 81.30 measures drift, not quality: it treats the bf16 output as ground
	truth, so it penalises valid paraphrase and would penalise a better answer.

	## What was not measured

	- No judged quality. In the open-ended set 11/18 (bf16) and 14/18 (8-bit)
	answers hit the 640-token cap, so grading them would have measured truncation
	rather than quality. This model emits a `<think>` block, which consumes budget.
	- No agentic evaluation. No WebArena, no ScreenSpot, no WebVoyager, no
	end-to-end computer-use task. This model's actual job is multi-step browser
	automation and none of that is scored here.
	- No standard benchmarks (MMLU, GSM8K). The accuracy layer is 8 short
	verifiable prompts, not a benchmark.
	- The vision check is one screenshot, not a dataset, and it was scored for
	correctness by reading it, not by a metric.
	- Throughput figures are for one M2 Pro and do not transfer to other chips.

	## Related conversions

	\| Model \| Variant \| Note \|
	\|---\|---\|---\|
	\| [Fara1.5-9B-8bit](https://huggingface.co/mlx-community/Fara1.5-9B-8bit) \| 8.86 bpw \| larger sibling, top-1 agreement 1.000 \|
	\| [Fara-7B-8bit](https://huggingface.co/mlx-community/Fara-7B-8bit) \| 9.11 bpw \| previous generation \|
	\| [Fara-7B-4bit](https://huggingface.co/mlx-community/Fara-7B-4bit) \| 5.44 bpw \| previous generation \|

	## Credits

	All credit for the underlying model belongs to
	[Microsoft](https://huggingface.co/microsoft). This is a format conversion and
	quantization; no training or fine-tuning was performed. See the
	[source model card](https://huggingface.co/microsoft/Fara1.5-4B) for license,
	intended use and limitations.