--- license: apache-2.0 base_model: sensenova/SenseNova-Loop-9B base_model_relation: quantized pipeline_tag: image-text-to-text tags: - tensorfold - mlx-affine - 4-bit - quantized - sensenova-loop - looped-transformer - mhc - multimodal - apple-silicon - metal --- # SenseNova-Loop-9B MLX 4-bit (TensorFold, Apple Silicon) The tree the [recipe](https://github.com/jayleaton/sensenova-loop9b-tensorfold) serves on **Apple Silicon**: SenseNova-Loop-9B's text weights as **4-bit MLX affine codes, group 64**, with the checkpoint's **vision tower carried in BF16** (loaded only with `--vision`). It is the recipe's `loop9b-mlx4`: download it instead of converting the checkpoint yourself. The text codes are byte-identical, tensor by tensor, to the 4-bit tree the recipe ran its CUDA task benchmarks on. > **Experimental.** This is a looped model (physical layers 4-27 run twice, 56 executions from 32 weight sets, > wrapped in a two-stream mHC). Stock transformers, vLLM, llama.cpp, mlx-lm and exllamav3 cannot serve these files; > they need TensorFold with this recipe's patch series. Read the recipe's > [known issues](https://github.com/jayleaton/sensenova-loop9b-tensorfold/blob/main/docs/KNOWN-ISSUES.md) before you rely on it. ## Tested on | device | engine | status | | --- | --- | --- | | Mac Studio M5 Ultra (96 GB), macOS 27.2 | Metal: TensorFold [`6d0a128b0`](https://github.com/ashhart/TensorFold/commit/6d0a128b0cde35aa091bda933577cfced3519b8a) + [`patches/metal`](https://github.com/jayleaton/sensenova-loop9b-tensorfold/tree/main/patches/metal) `0001`-`0044` (engine tree `d33e2d93ea37`) | tested: text, images, tools, copy drafts | | 16 GB Macs (M-series) | the same | supported; measured on the M5 Ultra under a 16 GB Mac's Metal working-set cap | | MacBook Pro M4 Pro (24 GB) | the same | tested: decode | 8 GB Macs are not supported. On a DGX Spark, serve the 8-bit tree, [SenseNova-Loop-9B-MLX-8bit-vision](https://huggingface.co/jayleaton/SenseNova-Loop-9B-MLX-8bit-vision): 4-bit keeps top-1 on 220 of 249 positions there against 246 for 8-bit. ## Serve Needs Xcode with the macOS SDK and Metal toolchain, and Zig 0.17.0: ```bash git clone https://github.com/jayleaton/sensenova-loop9b-tensorfold.git && cd sensenova-loop9b-tensorfold scripts/build-engine.sh metal engine-metal # fetches the pin, applies patches/metal, builds hf download jayleaton/SenseNova-Loop-9B-MLX-4bit --local-dir loop9b-mlx4 scripts/serve.sh metal engine-metal loop9b-mlx4 # --context 33024 --parallel 4, copy drafts on PROFILE=vision scripts/serve.sh metal engine-metal loop9b-mlx4 # image input: --context 4096 --no-drafts --vision ``` `tool_choice: "required"`, a named function and `response_format` need `libtfgrammar.dylib` beside the server (xgrammar v0.2.8, `zig build grammar`); see the recipe's [METAL page](https://github.com/jayleaton/sensenova-loop9b-tensorfold/blob/main/docs/METAL.md). Thinking is on by default; send `"chat_template_kwargs": {"enable_thinking": false}` to turn it off. ## Measured Mac Studio M5 Ultra, from the recipe's [METAL page](https://github.com/jayleaton/sensenova-loop9b-tensorfold/blob/main/docs/METAL.md): | | | | --- | --- | | decode, drafts off | 59.5-60.0 tok/s one chat | | cold prefill, 512 / 2k / 8k / 32k tokens | 2,709 / 2,752 / 2,717 / 2,461 tok/s; TTFT 0.019 s for a 14-token prompt | | copy drafts, one chat | re-quote 362 tok/s, code edit 87, prose 94; 16/16 reply pairs token-identical to drafts off | | task gate, 64 greedy tokens against transformers on the same 4-bit weights | code 64/64, reasoning 64/64, prose 19/64 (the reference's own bf16/fp32 noise floor) | | tools (with `libtfgrammar`) | 10/10 cases, 4/4 with four chats at once | | images | vision tower within bf16 noise of the reference (cosine 0.99869); one image a request | | Mac | decode | prefill | peak memory | | --- | --- | --- | --- | | 16 GB Mac (working-set cap on the M5 Ultra) | 58 tok/s | 2,719 tok/s at 8k | 7.60 / 7.82 / 9.13 GiB at 4k / 8k / 32k | | MacBook Pro M4 Pro, 24 GB | 26.7 tok/s | about 230 tok/s | | The 16 GB runs used the text shards alone; the vision tower adds 0.85 GiB only when served with `--vision`. Teacher- forced top-1 over 64 positions against BF16 (prose / code / reasoning): 57 / 62 / 61. **Task benchmarks** of the same 4-bit codes on a DGX Spark (served CUDA v5 engine, `--parallel 4 --context 33024`, copy drafts on), next to the 8-bit tree under the same harness: | benchmark | 4-bit (these codes) | 8-bit | | --- | --- | --- | | HumanEval+ | 126/164 (76.8%) | 130/164 (79.3%) | | MBPP+ | 239/378 (63.2%) | 253/378 (66.9%) | | GSM8K, thinking off / on | 282/300 / 283/300 | 284/300 / 285/300 | | MATH, thinking off / on | 171/200 / 182/200 | 175/200 / 183/200 | | BFCL | 116/200 (58.0%) | 127/200 (63.5%) | ## Sizes This repository: 13 model files, 5,982,934,074 bytes (5.98 GB, 5.57 GiB), plus this card and `LICENSE`. The weight files and their sha256: | file | bytes | sha256 | | --- | ---: | --- | | `model-00001.safetensors` | 1,822,297,144 | `cdf270f6fd2925c7439b5ea7b35104cb783a07ca6701849789ed56f60b9f006e` | | `model-00002.safetensors` | 1,826,655,872 | `38267bc2bdecc70102d15af0896f576d6981b72b3e17a61fcbe33f9b7e104b51` | | `model-00003.safetensors` | 1,800,704,712 | `7de04179b40988b6edfb25fc0dc966013286f200254c7e336ab4f97794e54127` | | `model-00004.safetensors` | 513,156,744 | `92cd2f7273c28ee554bbeecca9babf6fd49242009d3d46ddc29c4812cbc4cee6` | ## How it was made From the BF16 checkpoint at revision `329a65f701ba8c48b73c30d0cbaa6f0449733673`, with the converter carried in the Metal series: ```bash python3 engine-metal/tools/sensenova_loop_mlx.py --source SenseNova-Loop-9B --out loop9b-mlx4 ``` 4-bit MLX affine codes with bf16 scales and biases per group of 64, the zero-centred RMSNorm weights folded to `1 + w`, the mHC routers in fp32, and the 333 vision tensors in bf16 as `vision_tower.*`. `tensorfold-convert.json` records a round-trip check. Details: [CONVERT](https://github.com/jayleaton/sensenova-loop9b-tensorfold/blob/main/docs/CONVERT.md). ## Credits - **Model: [SenseNova-Loop-9B](https://huggingface.co/sensenova/SenseNova-Loop-9B) by [SenseNova](https://huggingface.co/sensenova)** ([GitHub](https://github.com/OpenSenseNova/SenseNova-Loop)). The looped decoder, the two-stream mHC, the vision tower and the reference `modeling_sensenova_loop.py` are theirs. - **Engine: [TensorFold](https://github.com/ashhart/TensorFold) by Ash Hart (ashhart) and the TensorFold contributors**, Apache License 2.0. The recipe's patches extend it; they are not upstream changes. - **Shared-round drafter pricing:** upstream TensorFold PR [#2475](https://github.com/ashhart/TensorFold/pull/2475) by [jrb-ai](https://github.com/jrb-ai). - **EXL3:** [exllamav3](https://github.com/turboderp-org/exllamav3) by turboderp, the format and quantizer, built on [QTIP](https://github.com/Cornell-RelaxML/qtip) (Cornell RelaxML) trellis codebooks. - **Recipe, conversion and measurements:** [jayleaton/sensenova-loop9b-tensorfold](https://github.com/jayleaton/sensenova-loop9b-tensorfold). ## Licence Apache License 2.0, the base model's licence. This repository is a quantized derivative of [sensenova/SenseNova-Loop-9B](https://huggingface.co/sensenova/SenseNova-Loop-9B) (revision `329a65f701ba8c48b73c30d0cbaa6f0449733673`) by SenseNova: the weights were converted to MLX affine codes and are otherwise SenseNova's model. The licence text is in [LICENSE](LICENSE). The tokenizer, chat template, processor and generation configuration come from the base model.