SenseNova-Loop-9B MLX 4-bit (TensorFold, Apple Silicon)

The tree the recipe serves on Apple Silicon: SenseNova-Loop-9B's text weights as 4-bit MLX affine codes, group 64, with the checkpoint's vision tower carried in BF16 (loaded only with --vision). It is the recipe's loop9b-mlx4: download it instead of converting the checkpoint yourself. The text codes are byte-identical, tensor by tensor, to the 4-bit tree the recipe ran its CUDA task benchmarks on.

Experimental. This is a looped model (physical layers 4-27 run twice, 56 executions from 32 weight sets, wrapped in a two-stream mHC). Stock transformers, vLLM, llama.cpp, mlx-lm and exllamav3 cannot serve these files; they need TensorFold with this recipe's patch series. Read the recipe's known issues before you rely on it.

Tested on

device engine status
Mac Studio M5 Ultra (96 GB), macOS 27.2 Metal: TensorFold 6d0a128b0 + patches/metal 0001-0044 (engine tree d33e2d93ea37) tested: text, images, tools, copy drafts
16 GB Macs (M-series) the same supported; measured on the M5 Ultra under a 16 GB Mac's Metal working-set cap
MacBook Pro M4 Pro (24 GB) the same tested: decode

8 GB Macs are not supported. On a DGX Spark, serve the 8-bit tree, SenseNova-Loop-9B-MLX-8bit-vision: 4-bit keeps top-1 on 220 of 249 positions there against 246 for 8-bit.

Serve

Needs Xcode with the macOS SDK and Metal toolchain, and Zig 0.17.0:

git clone https://github.com/jayleaton/sensenova-loop9b-tensorfold.git && cd sensenova-loop9b-tensorfold
scripts/build-engine.sh metal engine-metal                       # fetches the pin, applies patches/metal, builds
hf download jayleaton/SenseNova-Loop-9B-MLX-4bit --local-dir loop9b-mlx4
scripts/serve.sh metal engine-metal loop9b-mlx4                  # --context 33024 --parallel 4, copy drafts on
PROFILE=vision scripts/serve.sh metal engine-metal loop9b-mlx4   # image input: --context 4096 --no-drafts --vision

tool_choice: "required", a named function and response_format need libtfgrammar.dylib beside the server (xgrammar v0.2.8, zig build grammar); see the recipe's METAL page. Thinking is on by default; send "chat_template_kwargs": {"enable_thinking": false} to turn it off.

Measured

Mac Studio M5 Ultra, from the recipe's METAL page:

decode, drafts off 59.5-60.0 tok/s one chat
cold prefill, 512 / 2k / 8k / 32k tokens 2,709 / 2,752 / 2,717 / 2,461 tok/s; TTFT 0.019 s for a 14-token prompt
copy drafts, one chat re-quote 362 tok/s, code edit 87, prose 94; 16/16 reply pairs token-identical to drafts off
task gate, 64 greedy tokens against transformers on the same 4-bit weights code 64/64, reasoning 64/64, prose 19/64 (the reference's own bf16/fp32 noise floor)
tools (with libtfgrammar) 10/10 cases, 4/4 with four chats at once
images vision tower within bf16 noise of the reference (cosine 0.99869); one image a request
Mac decode prefill peak memory
16 GB Mac (working-set cap on the M5 Ultra) 58 tok/s 2,719 tok/s at 8k 7.60 / 7.82 / 9.13 GiB at 4k / 8k / 32k
MacBook Pro M4 Pro, 24 GB 26.7 tok/s about 230 tok/s

The 16 GB runs used the text shards alone; the vision tower adds 0.85 GiB only when served with --vision. Teacher- forced top-1 over 64 positions against BF16 (prose / code / reasoning): 57 / 62 / 61.

Task benchmarks of the same 4-bit codes on a DGX Spark (served CUDA v5 engine, --parallel 4 --context 33024, copy drafts on), next to the 8-bit tree under the same harness:

benchmark 4-bit (these codes) 8-bit
HumanEval+ 126/164 (76.8%) 130/164 (79.3%)
MBPP+ 239/378 (63.2%) 253/378 (66.9%)
GSM8K, thinking off / on 282/300 / 283/300 284/300 / 285/300
MATH, thinking off / on 171/200 / 182/200 175/200 / 183/200
BFCL 116/200 (58.0%) 127/200 (63.5%)

Sizes

This repository: 13 model files, 5,982,934,074 bytes (5.98 GB, 5.57 GiB), plus this card and LICENSE. The weight files and their sha256:

file bytes sha256
model-00001.safetensors 1,822,297,144 cdf270f6fd2925c7439b5ea7b35104cb783a07ca6701849789ed56f60b9f006e
model-00002.safetensors 1,826,655,872 38267bc2bdecc70102d15af0896f576d6981b72b3e17a61fcbe33f9b7e104b51
model-00003.safetensors 1,800,704,712 7de04179b40988b6edfb25fc0dc966013286f200254c7e336ab4f97794e54127
model-00004.safetensors 513,156,744 92cd2f7273c28ee554bbeecca9babf6fd49242009d3d46ddc29c4812cbc4cee6

How it was made

From the BF16 checkpoint at revision 329a65f701ba8c48b73c30d0cbaa6f0449733673, with the converter carried in the Metal series:

python3 engine-metal/tools/sensenova_loop_mlx.py --source SenseNova-Loop-9B --out loop9b-mlx4

4-bit MLX affine codes with bf16 scales and biases per group of 64, the zero-centred RMSNorm weights folded to 1 + w, the mHC routers in fp32, and the 333 vision tensors in bf16 as vision_tower.*. tensorfold-convert.json records a round-trip check. Details: CONVERT.

Credits

  • Model: SenseNova-Loop-9B by SenseNova (GitHub). The looped decoder, the two-stream mHC, the vision tower and the reference modeling_sensenova_loop.py are theirs.
  • Engine: TensorFold by Ash Hart (ashhart) and the TensorFold contributors, Apache License 2.0. The recipe's patches extend it; they are not upstream changes.
  • Shared-round drafter pricing: upstream TensorFold PR #2475 by jrb-ai.
  • EXL3: exllamav3 by turboderp, the format and quantizer, built on QTIP (Cornell RelaxML) trellis codebooks.
  • Recipe, conversion and measurements: jayleaton/sensenova-loop9b-tensorfold.

Licence

Apache License 2.0, the base model's licence. This repository is a quantized derivative of sensenova/SenseNova-Loop-9B (revision 329a65f701ba8c48b73c30d0cbaa6f0449733673) by SenseNova: the weights were converted to MLX affine codes and are otherwise SenseNova's model. The licence text is in LICENSE. The tokenizer, chat template, processor and generation configuration come from the base model.

Downloads last month
-
Safetensors
Model size
9B params
Tensor type
F32
路
U32
路
BF16
路
Inference Providers NEW
This model isn't deployed by any Inference Provider. 馃檵 Ask for provider support

Model tree for jayleaton/SenseNova-Loop-9B-MLX-4bit

Quantized
(42)
this model