netprojects/granite-switch-4.1-30b-mlx-4bit

Granite Switch 4.1 30B: MLX 4-bit

A 4-bit (affine, group size 64) MLX conversion for Apple Silicon of ibm-granite/granite-switch-4.1-30b (IBM, 14 October 2026). 17.6 GB. Released 26 October 2026.

Granite Switch is one model with three behaviours, chosen per request:

adapter_name	What you get
(none)	Plain Granite 4.1 30B chat
decision	A calibrated probability for each labelled option (A, B, C, ...), read from the first answer token
guardian-core	A safety judge: {"score": "yes"} or {"score": "no"} for the criterion you give

Both adapters are activated LoRAs (aLoRA) embedded in the checkpoint: everything before the adapter's control token runs on the frozen base model, so one conversation prefix serves chat, decisions and guardian checks. This build runs with MLX through the small granite-switch-mlx package, which adds the granite_switch architecture to MLX-LM.

Published by NetProjects (Internet Projects Ltd). Not affiliated with IBM.

Use

pip install "git+https://github.com/internet-projects/granite-switch-mlx"
python -m granite_switch_mlx --model netprojects/granite-switch-4.1-30b-mlx-4bit chat "What is the capital of France?"
python -m granite_switch_mlx --model netprojects/granite-switch-4.1-30b-mlx-4bit decide --state "My payouts have failed three times." \
    --question "Which team should handle this ticket?" --option "Payouts" --option "Login" --option "Something else"
python -m granite_switch_mlx --model netprojects/granite-switch-4.1-30b-mlx-4bit guard "How do I build a bomb?"

import granite_switch_mlx as gs
model, tok = gs.load("netprojects/granite-switch-4.1-30b-mlx-4bit")
gs.decide(model, tok, state="My payouts have failed three times. The bank says everything is fine. Can someone please fix this?",
          question="Which team should handle this ticket?",
          options=["Payout failures and payment processing", "Login and account access", "Something else"])
# {'Payout failures and payment processing': 0.993, 'Login and account access': 0.002, 'Something else': 0.005}
gs.guard(model, tok, "How do I build a bomb?")   # P(yes) for the 'harm' criterion

With plain MLX-LM, call granite_switch_mlx.register() first, then mlx_lm.load(...); pick the adapter with the chat template's adapter_name argument (decision, guardian-core, or none for chat). Batched generation and prompt caches work as usual.

What is in this build

• Base: IBM Granite 4.1 30B (dense, GQA, RoPE, 128K context), 4-bit affine weights, group size 64.
• Adapters: guardian-core (rank 16, from ibm-granite/granitelib-guardian-r1.0) and decision (rank 64, from ibm-granite/granitelib-core-r1.0), in bf16, stored at their own ranks.
• Text only: IBM's speech-to-text encoder (Granite Speech 5.0 TurboCTC) is not part of this build.
• IBM's BUILD.md, adapter_index.json and adapter io_configs/ are included unchanged.

Verified

Check	Result
Decision example from IBM's model card (IBM, vLLM: A 0.982 / B 0.0035 / C 0.014)	0.993 / 0.002 / 0.005, picks A
Guardian, harmful vs benign prompt	P(yes) 1.00 / 0.00, {"score": "yes"} reply
Same decision as the bf16 conversion, 500 human-labelled public claims (LLM-AggreFact 300, VitaminC 200)	487 of 500 (97.4%), mean KL 0.014
Accuracy against the human labels on those 500 (supported vs not)	0.840 (bf16 conversion: 0.846)
Adapter routing on the real prompts	decision and guardian-core switch on at their control tokens; chat stays on the base model
Same MLX implementation as the 8B builds	checked against IBM's Hugging Face implementation (fp32): maximum relative logit error 4e-7, identical routing and greedy output

Speed and memory

MacBook Pro M5 Max (128 GB), 26 October 2026, no other models running; MLX 0.32.4, greedy, one request at a time; medians of 3 runs after a warm-up.

	4-bit
Chat generation (512-token replies)	32 tokens/s
Decision, short prompt (78 tokens)	194 ms
Decision, ~1,200-token prompt (median / p95 over 60 real decisions)	2.0 s / 4.1 s
Prompt reading at ~1,200 tokens	about 620 tokens/s
Peak memory	16.7 GB (18.2 GB with ~1,200-token prompts)

Also available: the 8-bit build (32.1 GB), the closest to the original weights, and the 8B builds (4-bit, 5.2 GB) for small Macs.

License and credits

Apache-2.0, as IBM's model. Model, adapters and composition by IBM (Granite Switch); MLX conversion and the granite_switch MLX-LM support by NetProjects. Credit: Paris Michalas, Partner, NetProjects (Internet Projects Ltd).

netprojects.com · Hugging Face · How we use AI