README.md

---
library_name: mlx
license: mit
pipeline_tag: text-generation
base_model: microsoft/MagenticBrain
language:
- en
tags:
- agent
- agentic
- tool-use
- function-calling
- orchestration
- magentic
- qwen3
- mlx
---

# mlx-community/MagenticBrain-4bit

[microsoft/MagenticBrain](https://huggingface.co/microsoft/MagenticBrain) converted to MLX and quantized to
**4-bit**, for inference on Apple Silicon.

MagenticBrain is a 14.8B orchestration model from Microsoft Research AI
Frontiers, supervised fine-tuned from Qwen3-14B for planning, tool selection,
multi-turn tool chaining and sub-agent delegation. It is **not a general-purpose
chat model** — see the [original card](https://huggingface.co/microsoft/MagenticBrain).

## Quantization

| | |
|---|---|
| Requested bits | 4 |
| Group size | 64 |
| Mode | affine |
| **Effective bits per weight** | **4.5** |
| On-disk size | 7.8 GB |
| Shards | 8 |

The source is stored in **float32** (14.8B params × 4 bytes = 59 GB on disk),
which is why the repo is roughly twice the size of a typical bf16 release. Weights
are cast to bf16 before quantizing.

## Fidelity vs the original weights

Measured against the **fp32 source**, streamed tensor-by-tensor from disk, over
all **14,767,882,240** parameters. No prompts or sampling involved — this is a direct
measurement of how much numerical information the quantization discarded.

| Metric | 4-bit |
|---|---|
| Relative L2 error | **9.30%** |
| Cosine similarity | **0.995684** |
| Signal-to-quantization-noise | **20.63 dB** |
| Worst single-element error | 0.098161 |

Both variants, for comparison:

| Variant | bpw | Relative L2 | Cosine | SNR | Size |
|---|---|---|---|---|---|
| 4-bit | 4.5 | 9.30% | 0.995684 | 20.63 dB | 7.8 GB |
| 8-bit | 8.5 | 0.73% | 0.999973 | 42.68 dB | 15 GB |

Highest-error tensors in this variant (the early layers are consistently the
most sensitive):

```
rel_l2=0.11013  snr= 19.16 dB  model.layers.1.self_attn.q_proj
rel_l2=0.10583  snr= 19.51 dB  model.layers.1.self_attn.k_proj
rel_l2=0.10436  snr= 19.63 dB  model.layers.1.self_attn.v_proj
rel_l2=0.10272  snr= 19.77 dB  model.layers.1.mlp.gate_proj
rel_l2=0.10245  snr= 19.79 dB  model.layers.4.mlp.down_proj
rel_l2=0.09928  snr= 20.06 dB  model.layers.3.mlp.down_proj
```

## Why there is no bf16 behavioural control

The methodology used for these conversions compares a quantized model's
*outputs* against the unquantized original. **That was not possible here**, and
the reason is worth stating rather than omitting:

At 14.8B parameters, bf16 weights are ~28 GB. On the 32 GB machine used for this
conversion, loading them drove the system into swap — measured at 34.9 GB of
35.8 GB swap in use, with 50 tokens taking over 10 minutes. Any benchmark run
under those conditions would measure paging, not the model.

So the comparison against the original is done at the **weight level** (above),
which is exact and hardware-independent, and behavioural benchmarks are run on
the variants that actually fit in memory. What is *not* claimed anywhere here is
"indistinguishable from bf16 in behaviour" — that would require a control this
hardware cannot run.

## Tool calling (BFCL)

[Berkeley Function-Calling Leaderboard](https://gorilla.cs.berkeley.edu/leaderboard.html)
v4, scored with **AST checking** against BFCL's ground truth: correct function
selected, all required parameters present, each argument matching BFCL's list of
accepted values, types normalised, no invented parameters. Deterministic — no
judge involved.

| Category | Accuracy | Parse rate | n |
|---|---|---|---|
| `live_simple` | 0.900 | 0.925 | 40 |
| `live_multiple` | 0.675 | 0.925 | 40 |
| `multiple` | 0.775 | 0.925 | 40 |
| `parallel` | 0.625 | 0.825 | 40 |
| **Overall** | **0.744** | | 160 |

Categories: `live_simple` (one function, real user queries), `live_multiple` and
`multiple` (must select among several), `parallel` (several calls in one turn).
Accuracy degrading toward `parallel` is expected — it is the hardest category.

## Throughput

Decode speed and memory on the machine used for conversion (M2 Pro, 32 GB) are
reported in the project notes rather than here, since they do not transfer across
chips. The practical point: at 4-bit the model needs roughly 7.8 GB of
weights, which fits comfortably in 32 GB alongside a working KV cache.

## Usage

```bash
pip install mlx-lm
```

```python
from mlx_lm import load, generate

model, tokenizer = load("mlx-community/MagenticBrain-4bit")

tools = [{
    "type": "function",
    "function": {
        "name": "web_search",
        "description": "Search the web",
        "parameters": {
            "type": "object",
            "properties": {"query": {"type": "string"}},
            "required": ["query"],
        },
    },
}]

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Find the 2026 Turing Award winner."}],
    tools=tools, tokenize=False, add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt, max_tokens=256, verbose=False))
```

The model emits structured JSON tool calls and selects only from the tools you
declare. Your harness is responsible for parsing the calls, executing them, and
handling the `submit` terminator — see the original card for the protocol.

## What was not measured

No IFEval or general-knowledge benchmarks were run. No agentic end-to-end
evaluation inside MagenticLite (Microsoft's harness, which the model was
co-designed with) was performed. If your use case is the full orchestration loop,
evaluate on your own tasks.

## Credits

All credit for the model belongs to Microsoft Research AI Frontiers. This is a
format conversion and quantization; no training or fine-tuning was performed.
Licensed MIT, as the original.