mlx-community/MagenticBrain-4bit

microsoft/MagenticBrain converted to MLX and quantized to 4-bit, for inference on Apple Silicon.

MagenticBrain is a 14.8B orchestration model from Microsoft Research AI Frontiers, supervised fine-tuned from Qwen3-14B for planning, tool selection, multi-turn tool chaining and sub-agent delegation. It is not a general-purpose chat model — see the original card.

Quantization

Requested bits	4
Group size	64
Mode	affine
Effective bits per weight	4.5
On-disk size	7.8 GB
Shards	8

The source is stored in float32 (14.8B params × 4 bytes = 59 GB on disk), which is why the repo is roughly twice the size of a typical bf16 release. Weights are cast to bf16 before quantizing.

Fidelity vs the original weights

Measured against the fp32 source, streamed tensor-by-tensor from disk, over all 14,767,882,240 parameters. No prompts or sampling involved — this is a direct measurement of how much numerical information the quantization discarded.

Metric	4-bit
Relative L2 error	9.30%
Cosine similarity	0.995684
Signal-to-quantization-noise	20.63 dB
Worst single-element error	0.098161

Both variants, for comparison:

Variant	bpw	Relative L2	Cosine	SNR	Size
4-bit	4.5	9.30%	0.995684	20.63 dB	7.8 GB
8-bit	8.5	0.73%	0.999973	42.68 dB	15 GB

Highest-error tensors in this variant (the early layers are consistently the most sensitive):

rel_l2=0.11013  snr= 19.16 dB  model.layers.1.self_attn.q_proj
rel_l2=0.10583  snr= 19.51 dB  model.layers.1.self_attn.k_proj
rel_l2=0.10436  snr= 19.63 dB  model.layers.1.self_attn.v_proj
rel_l2=0.10272  snr= 19.77 dB  model.layers.1.mlp.gate_proj
rel_l2=0.10245  snr= 19.79 dB  model.layers.4.mlp.down_proj
rel_l2=0.09928  snr= 20.06 dB  model.layers.3.mlp.down_proj

Why there is no bf16 behavioural control

The methodology used for these conversions compares a quantized model's outputs against the unquantized original. That was not possible here, and the reason is worth stating rather than omitting:

At 14.8B parameters, bf16 weights are ~28 GB. On the 32 GB machine used for this conversion, loading them drove the system into swap — measured at 34.9 GB of 35.8 GB swap in use, with 50 tokens taking over 10 minutes. Any benchmark run under those conditions would measure paging, not the model.

So the comparison against the original is done at the weight level (above), which is exact and hardware-independent, and behavioural benchmarks are run on the variants that actually fit in memory. What is not claimed anywhere here is "indistinguishable from bf16 in behaviour" — that would require a control this hardware cannot run.

Tool calling (BFCL)

Berkeley Function-Calling Leaderboard v4, scored with AST checking against BFCL's ground truth: correct function selected, all required parameters present, each argument matching BFCL's list of accepted values, types normalised, no invented parameters. Deterministic — no judge involved.

Category	Accuracy	Parse rate	n
live_simple	0.900	0.925	40
live_multiple	0.675	0.925	40
multiple	0.775	0.925	40
parallel	0.625	0.825	40
Overall	0.744		160

Categories: live_simple (one function, real user queries), live_multiple and multiple (must select among several), parallel (several calls in one turn). Accuracy degrading toward parallel is expected — it is the hardest category.

Throughput

Decode speed and memory on the machine used for conversion (M2 Pro, 32 GB) are reported in the project notes rather than here, since they do not transfer across chips. The practical point: at 4-bit the model needs roughly 7.8 GB of weights, which fits comfortably in 32 GB alongside a working KV cache.

Usage

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/MagenticBrain-4bit")

tools = [{
    "type": "function",
    "function": {
        "name": "web_search",
        "description": "Search the web",
        "parameters": {
            "type": "object",
            "properties": {"query": {"type": "string"}},
            "required": ["query"],
        },
    },
}]

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Find the 2026 Turing Award winner."}],
    tools=tools, tokenize=False, add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt, max_tokens=256, verbose=False))

The model emits structured JSON tool calls and selects only from the tools you declare. Your harness is responsible for parsing the calls, executing them, and handling the submit terminator — see the original card for the protocol.

What was not measured

No IFEval or general-knowledge benchmarks were run. No agentic end-to-end evaluation inside MagenticLite (Microsoft's harness, which the model was co-designed with) was performed. If your use case is the full orchestration loop, evaluate on your own tasks.

Credits

All credit for the model belongs to Microsoft Research AI Frontiers. This is a format conversion and quantization; no training or fine-tuning was performed. Licensed MIT, as the original.