jayleaton's picture
Upload SenseNova-Loop-9B-MLX-4bit
0afa03c verified
|
Raw History Blame Contribute Delete
7.63 kB
---
license: apache-2.0
base_model: sensenova/SenseNova-Loop-9B
base_model_relation: quantized
pipeline_tag: image-text-to-text
tags:
- tensorfold
- mlx-affine
- 4-bit
- quantized
- sensenova-loop
- looped-transformer
- mhc
- multimodal
- apple-silicon
- metal
---
# SenseNova-Loop-9B MLX 4-bit (TensorFold, Apple Silicon)
The tree the [recipe](https://github.com/jayleaton/sensenova-loop9b-tensorfold) serves on **Apple Silicon**: SenseNova-Loop-9B's text weights as **4-bit MLX affine
codes, group 64**, with the checkpoint's **vision tower carried in BF16** (loaded only with `--vision`). It is the
recipe's `loop9b-mlx4`: download it instead of converting the checkpoint yourself. The text codes are byte-identical,
tensor by tensor, to the 4-bit tree the recipe ran its CUDA task benchmarks on.
> **Experimental.** This is a looped model (physical layers 4-27 run twice, 56 executions from 32 weight sets,
> wrapped in a two-stream mHC). Stock transformers, vLLM, llama.cpp, mlx-lm and exllamav3 cannot serve these files;
> they need TensorFold with this recipe's patch series. Read the recipe's
> [known issues](https://github.com/jayleaton/sensenova-loop9b-tensorfold/blob/main/docs/KNOWN-ISSUES.md) before you rely on it.
## Tested on
| device | engine | status |
| --- | --- | --- |
| Mac Studio M5 Ultra (96 GB), macOS 27.2 | Metal: TensorFold [`6d0a128b0`](https://github.com/ashhart/TensorFold/commit/6d0a128b0cde35aa091bda933577cfced3519b8a) + [`patches/metal`](https://github.com/jayleaton/sensenova-loop9b-tensorfold/tree/main/patches/metal) `0001`-`0044` (engine tree `d33e2d93ea37`) | tested: text, images, tools, copy drafts |
| 16 GB Macs (M-series) | the same | supported; measured on the M5 Ultra under a 16 GB Mac's Metal working-set cap |
| MacBook Pro M4 Pro (24 GB) | the same | tested: decode |
8 GB Macs are not supported. On a DGX Spark, serve the 8-bit tree,
[SenseNova-Loop-9B-MLX-8bit-vision](https://huggingface.co/jayleaton/SenseNova-Loop-9B-MLX-8bit-vision): 4-bit keeps
top-1 on 220 of 249 positions there against 246 for 8-bit.
## Serve
Needs Xcode with the macOS SDK and Metal toolchain, and Zig 0.17.0:
```bash
git clone https://github.com/jayleaton/sensenova-loop9b-tensorfold.git && cd sensenova-loop9b-tensorfold
scripts/build-engine.sh metal engine-metal # fetches the pin, applies patches/metal, builds
hf download jayleaton/SenseNova-Loop-9B-MLX-4bit --local-dir loop9b-mlx4
scripts/serve.sh metal engine-metal loop9b-mlx4 # --context 33024 --parallel 4, copy drafts on
PROFILE=vision scripts/serve.sh metal engine-metal loop9b-mlx4 # image input: --context 4096 --no-drafts --vision
```
`tool_choice: "required"`, a named function and `response_format` need `libtfgrammar.dylib` beside the server (xgrammar
v0.2.8, `zig build grammar`); see the recipe's [METAL page](https://github.com/jayleaton/sensenova-loop9b-tensorfold/blob/main/docs/METAL.md). Thinking is on by
default; send `"chat_template_kwargs": {"enable_thinking": false}` to turn it off.
## Measured
Mac Studio M5 Ultra, from the recipe's [METAL page](https://github.com/jayleaton/sensenova-loop9b-tensorfold/blob/main/docs/METAL.md):
| | |
| --- | --- |
| decode, drafts off | 59.5-60.0 tok/s one chat |
| cold prefill, 512 / 2k / 8k / 32k tokens | 2,709 / 2,752 / 2,717 / 2,461 tok/s; TTFT 0.019 s for a 14-token prompt |
| copy drafts, one chat | re-quote 362 tok/s, code edit 87, prose 94; 16/16 reply pairs token-identical to drafts off |
| task gate, 64 greedy tokens against transformers on the same 4-bit weights | code 64/64, reasoning 64/64, prose 19/64 (the reference's own bf16/fp32 noise floor) |
| tools (with `libtfgrammar`) | 10/10 cases, 4/4 with four chats at once |
| images | vision tower within bf16 noise of the reference (cosine 0.99869); one image a request |
| Mac | decode | prefill | peak memory |
| --- | --- | --- | --- |
| 16 GB Mac (working-set cap on the M5 Ultra) | 58 tok/s | 2,719 tok/s at 8k | 7.60 / 7.82 / 9.13 GiB at 4k / 8k / 32k |
| MacBook Pro M4 Pro, 24 GB | 26.7 tok/s | about 230 tok/s | |
The 16 GB runs used the text shards alone; the vision tower adds 0.85 GiB only when served with `--vision`. Teacher-
forced top-1 over 64 positions against BF16 (prose / code / reasoning): 57 / 62 / 61.
**Task benchmarks** of the same 4-bit codes on a DGX Spark (served CUDA v5 engine, `--parallel 4 --context 33024`,
copy drafts on), next to the 8-bit tree under the same harness:
| benchmark | 4-bit (these codes) | 8-bit |
| --- | --- | --- |
| HumanEval+ | 126/164 (76.8%) | 130/164 (79.3%) |
| MBPP+ | 239/378 (63.2%) | 253/378 (66.9%) |
| GSM8K, thinking off / on | 282/300 / 283/300 | 284/300 / 285/300 |
| MATH, thinking off / on | 171/200 / 182/200 | 175/200 / 183/200 |
| BFCL | 116/200 (58.0%) | 127/200 (63.5%) |
## Sizes
This repository: 13 model files, 5,982,934,074 bytes (5.98 GB, 5.57 GiB), plus this card and `LICENSE`. The weight files and their sha256:
| file | bytes | sha256 |
| --- | ---: | --- |
| `model-00001.safetensors` | 1,822,297,144 | `cdf270f6fd2925c7439b5ea7b35104cb783a07ca6701849789ed56f60b9f006e` |
| `model-00002.safetensors` | 1,826,655,872 | `38267bc2bdecc70102d15af0896f576d6981b72b3e17a61fcbe33f9b7e104b51` |
| `model-00003.safetensors` | 1,800,704,712 | `7de04179b40988b6edfb25fc0dc966013286f200254c7e336ab4f97794e54127` |
| `model-00004.safetensors` | 513,156,744 | `92cd2f7273c28ee554bbeecca9babf6fd49242009d3d46ddc29c4812cbc4cee6` |
## How it was made
From the BF16 checkpoint at revision `329a65f701ba8c48b73c30d0cbaa6f0449733673`, with the converter carried in the Metal series:
```bash
python3 engine-metal/tools/sensenova_loop_mlx.py --source SenseNova-Loop-9B --out loop9b-mlx4
```
4-bit MLX affine codes with bf16 scales and biases per group of 64, the zero-centred RMSNorm weights folded to `1 + w`,
the mHC routers in fp32, and the 333 vision tensors in bf16 as `vision_tower.*`. `tensorfold-convert.json` records a
round-trip check. Details: [CONVERT](https://github.com/jayleaton/sensenova-loop9b-tensorfold/blob/main/docs/CONVERT.md).
## Credits
- **Model: [SenseNova-Loop-9B](https://huggingface.co/sensenova/SenseNova-Loop-9B) by [SenseNova](https://huggingface.co/sensenova)**
([GitHub](https://github.com/OpenSenseNova/SenseNova-Loop)). The looped decoder, the two-stream mHC, the vision
tower and the reference `modeling_sensenova_loop.py` are theirs.
- **Engine: [TensorFold](https://github.com/ashhart/TensorFold) by Ash Hart (ashhart) and the TensorFold contributors**, Apache License 2.0. The
recipe's patches extend it; they are not upstream changes.
- **Shared-round drafter pricing:** upstream TensorFold PR [#2475](https://github.com/ashhart/TensorFold/pull/2475) by
[jrb-ai](https://github.com/jrb-ai).
- **EXL3:** [exllamav3](https://github.com/turboderp-org/exllamav3) by turboderp, the format and quantizer, built on
[QTIP](https://github.com/Cornell-RelaxML/qtip) (Cornell RelaxML) trellis codebooks.
- **Recipe, conversion and measurements:** [jayleaton/sensenova-loop9b-tensorfold](https://github.com/jayleaton/sensenova-loop9b-tensorfold).
## Licence
Apache License 2.0, the base model's licence. This repository is a quantized derivative of
[sensenova/SenseNova-Loop-9B](https://huggingface.co/sensenova/SenseNova-Loop-9B) (revision `329a65f701ba8c48b73c30d0cbaa6f0449733673`) by SenseNova: the weights were converted to MLX affine
codes and are otherwise SenseNova's model. The licence text is in [LICENSE](LICENSE). The tokenizer, chat template,
processor and generation configuration come from the base model.