SenseNova-Loop-9B MLX 4-bit (TensorFold, Apple Silicon)
The tree the recipe serves on Apple Silicon: SenseNova-Loop-9B's text weights as 4-bit MLX affine
codes, group 64, with the checkpoint's vision tower carried in BF16 (loaded only with --vision). It is the
recipe's loop9b-mlx4: download it instead of converting the checkpoint yourself. The text codes are byte-identical,
tensor by tensor, to the 4-bit tree the recipe ran its CUDA task benchmarks on.
Experimental. This is a looped model (physical layers 4-27 run twice, 56 executions from 32 weight sets, wrapped in a two-stream mHC). Stock transformers, vLLM, llama.cpp, mlx-lm and exllamav3 cannot serve these files; they need TensorFold with this recipe's patch series. Read the recipe's known issues before you rely on it.
Tested on
| device | engine | status |
|---|---|---|
| Mac Studio M5 Ultra (96 GB), macOS 27.2 | Metal: TensorFold 6d0a128b0 + patches/metal 0001-0044 (engine tree d33e2d93ea37) |
tested: text, images, tools, copy drafts |
| 16 GB Macs (M-series) | the same | supported; measured on the M5 Ultra under a 16 GB Mac's Metal working-set cap |
| MacBook Pro M4 Pro (24 GB) | the same | tested: decode |
8 GB Macs are not supported. On a DGX Spark, serve the 8-bit tree, SenseNova-Loop-9B-MLX-8bit-vision: 4-bit keeps top-1 on 220 of 249 positions there against 246 for 8-bit.
Serve
Needs Xcode with the macOS SDK and Metal toolchain, and Zig 0.17.0:
git clone https://github.com/jayleaton/sensenova-loop9b-tensorfold.git && cd sensenova-loop9b-tensorfold
scripts/build-engine.sh metal engine-metal # fetches the pin, applies patches/metal, builds
hf download jayleaton/SenseNova-Loop-9B-MLX-4bit --local-dir loop9b-mlx4
scripts/serve.sh metal engine-metal loop9b-mlx4 # --context 33024 --parallel 4, copy drafts on
PROFILE=vision scripts/serve.sh metal engine-metal loop9b-mlx4 # image input: --context 4096 --no-drafts --vision
tool_choice: "required", a named function and response_format need libtfgrammar.dylib beside the server (xgrammar
v0.2.8, zig build grammar); see the recipe's METAL page. Thinking is on by
default; send "chat_template_kwargs": {"enable_thinking": false} to turn it off.
Measured
Mac Studio M5 Ultra, from the recipe's METAL page:
| decode, drafts off | 59.5-60.0 tok/s one chat |
| cold prefill, 512 / 2k / 8k / 32k tokens | 2,709 / 2,752 / 2,717 / 2,461 tok/s; TTFT 0.019 s for a 14-token prompt |
| copy drafts, one chat | re-quote 362 tok/s, code edit 87, prose 94; 16/16 reply pairs token-identical to drafts off |
| task gate, 64 greedy tokens against transformers on the same 4-bit weights | code 64/64, reasoning 64/64, prose 19/64 (the reference's own bf16/fp32 noise floor) |
tools (with libtfgrammar) |
10/10 cases, 4/4 with four chats at once |
| images | vision tower within bf16 noise of the reference (cosine 0.99869); one image a request |
| Mac | decode | prefill | peak memory |
|---|---|---|---|
| 16 GB Mac (working-set cap on the M5 Ultra) | 58 tok/s | 2,719 tok/s at 8k | 7.60 / 7.82 / 9.13 GiB at 4k / 8k / 32k |
| MacBook Pro M4 Pro, 24 GB | 26.7 tok/s | about 230 tok/s |
The 16 GB runs used the text shards alone; the vision tower adds 0.85 GiB only when served with --vision. Teacher-
forced top-1 over 64 positions against BF16 (prose / code / reasoning): 57 / 62 / 61.
Task benchmarks of the same 4-bit codes on a DGX Spark (served CUDA v5 engine, --parallel 4 --context 33024,
copy drafts on), next to the 8-bit tree under the same harness:
| benchmark | 4-bit (these codes) | 8-bit |
|---|---|---|
| HumanEval+ | 126/164 (76.8%) | 130/164 (79.3%) |
| MBPP+ | 239/378 (63.2%) | 253/378 (66.9%) |
| GSM8K, thinking off / on | 282/300 / 283/300 | 284/300 / 285/300 |
| MATH, thinking off / on | 171/200 / 182/200 | 175/200 / 183/200 |
| BFCL | 116/200 (58.0%) | 127/200 (63.5%) |
Sizes
This repository: 13 model files, 5,982,934,074 bytes (5.98 GB, 5.57 GiB), plus this card and LICENSE. The weight files and their sha256:
| file | bytes | sha256 |
|---|---|---|
model-00001.safetensors |
1,822,297,144 | cdf270f6fd2925c7439b5ea7b35104cb783a07ca6701849789ed56f60b9f006e |
model-00002.safetensors |
1,826,655,872 | 38267bc2bdecc70102d15af0896f576d6981b72b3e17a61fcbe33f9b7e104b51 |
model-00003.safetensors |
1,800,704,712 | 7de04179b40988b6edfb25fc0dc966013286f200254c7e336ab4f97794e54127 |
model-00004.safetensors |
513,156,744 | 92cd2f7273c28ee554bbeecca9babf6fd49242009d3d46ddc29c4812cbc4cee6 |
How it was made
From the BF16 checkpoint at revision 329a65f701ba8c48b73c30d0cbaa6f0449733673, with the converter carried in the Metal series:
python3 engine-metal/tools/sensenova_loop_mlx.py --source SenseNova-Loop-9B --out loop9b-mlx4
4-bit MLX affine codes with bf16 scales and biases per group of 64, the zero-centred RMSNorm weights folded to 1 + w,
the mHC routers in fp32, and the 333 vision tensors in bf16 as vision_tower.*. tensorfold-convert.json records a
round-trip check. Details: CONVERT.
Credits
- Model: SenseNova-Loop-9B by SenseNova
(GitHub). The looped decoder, the two-stream mHC, the vision
tower and the reference
modeling_sensenova_loop.pyare theirs. - Engine: TensorFold by Ash Hart (ashhart) and the TensorFold contributors, Apache License 2.0. The recipe's patches extend it; they are not upstream changes.
- Shared-round drafter pricing: upstream TensorFold PR #2475 by jrb-ai.
- EXL3: exllamav3 by turboderp, the format and quantizer, built on QTIP (Cornell RelaxML) trellis codebooks.
- Recipe, conversion and measurements: jayleaton/sensenova-loop9b-tensorfold.
Licence
Apache License 2.0, the base model's licence. This repository is a quantized derivative of
sensenova/SenseNova-Loop-9B (revision 329a65f701ba8c48b73c30d0cbaa6f0449733673) by SenseNova: the weights were converted to MLX affine
codes and are otherwise SenseNova's model. The licence text is in LICENSE. The tokenizer, chat template,
processor and generation configuration come from the base model.
- Downloads last month
- -
Model tree for jayleaton/SenseNova-Loop-9B-MLX-4bit
Base model
sensenova/SenseNova-Loop-9B