MLX builds for Apple Silicon (6-bit, 8-bit, 4-bit, bf16)

#7
by Paris99 - opened

Hi, and thank you for releasing SenseNova-Loop-9B.

We converted the release to MLX for Apple Silicon, from revision 329a65f. The port runs the model the way your code does: layers 4 to 27 run twice with shared weights, with the two-stream mHC routers kept in float32 and never quantized, and your YaRN setting for 512K context.

MLX builds (stock mlx-lm 0.32+, Python 3.11+, trust_remote_code):

Code, parity script and tests: https://github.com/internet-projects/sensenova-loop-mlx (v0.2.0)

from mlx_lm import load, generate
model, tokenizer = load("netprojects/SenseNova-Loop-9B-MLX-6bit", trust_remote_code=True)

What we checked on a MacBook Pro M5 Max:

  • Agreement with the original model (same next token as the PyTorch reference): bf16 98.6%, 8-bit 97.6%, 6-bit 96.7%, 4-bit 84.0%.
  • Chat and thinking mode: every MLX build answered short chat and thinking-mode prompts correctly.
  • loop_gate: the release adds a loop_gate.logits tensor that the reference code never reads, so the builds leave it out. Please tell us if that changes.
  • Long context: the 6-bit build found a pass code hidden in a 64K-token prompt.

The cards quote only your own benchmark numbers. The builds are Apache-2.0, with your licence and a NOTICE crediting SenseTime / SenseNova. They are published by NetProjects, and we are not affiliated with SenseTime or SenseNova. If you would like anything changed on the cards, please tell us here.

Paris Michalas, NetProjects

Update (18 Nov): the four MLX builds now support image input and your MTP head for faster decoding.

  • Image input: the vision tower runs in MLX and matches your PyTorch code (same top token on every image-prompt position, identical image positions).
  • MTP: about 1.7x faster decoding on the 6-bit build (19.3 to 33.5 tok/s on an M5 Max, 3 drafts). Greedy output matches plain decoding except at near-exact ties.
  • Norm weights are now kept in float32 as in your code, so agreement goes up to 99.1% (bf16) and 98.6% (8-bit).

Text still loads with stock mlx-lm. Images and MTP use v0.3.0: https://github.com/internet-projects/sensenova-loop-mlx/releases/tag/v0.3.0

Thanks again for adding the MTP weights.

Someone get rapid-mlx to also support this

Sign up or log in to comment