|
Download README.md from jayleaton/SenseNova-Loop-9B-MLX-4bit: direct link, hf CLI and curl.
- Browser
- Download file 7.63 kB
-
https://huggingface.co/jayleaton/SenseNova-Loop-9B-MLX-4bit/resolve/main/README.md
- Command line
-
hf download hf://jayleaton/SenseNova-Loop-9B-MLX-4bit/README.md
-
curl -L -o README.md https://huggingface.co/jayleaton/SenseNova-Loop-9B-MLX-4bit/resolve/main/README.md
7.63 kB
| license: apache-2.0 | |
| base_model: sensenova/SenseNova-Loop-9B | |
| base_model_relation: quantized | |
| pipeline_tag: image-text-to-text | |
| tags: | |
| - tensorfold | |
| - mlx-affine | |
| - 4-bit | |
| - quantized | |
| - sensenova-loop | |
| - looped-transformer | |
| - mhc | |
| - multimodal | |
| - apple-silicon | |
| - metal | |
| # SenseNova-Loop-9B MLX 4-bit (TensorFold, Apple Silicon) | |
| The tree the [recipe](https://github.com/jayleaton/sensenova-loop9b-tensorfold) serves on **Apple Silicon**: SenseNova-Loop-9B's text weights as **4-bit MLX affine | |
| codes, group 64**, with the checkpoint's **vision tower carried in BF16** (loaded only with `--vision`). It is the | |
| recipe's `loop9b-mlx4`: download it instead of converting the checkpoint yourself. The text codes are byte-identical, | |
| tensor by tensor, to the 4-bit tree the recipe ran its CUDA task benchmarks on. | |
| > **Experimental.** This is a looped model (physical layers 4-27 run twice, 56 executions from 32 weight sets, | |
| > wrapped in a two-stream mHC). Stock transformers, vLLM, llama.cpp, mlx-lm and exllamav3 cannot serve these files; | |
| > they need TensorFold with this recipe's patch series. Read the recipe's | |
| > [known issues](https://github.com/jayleaton/sensenova-loop9b-tensorfold/blob/main/docs/KNOWN-ISSUES.md) before you rely on it. | |
| ## Tested on | |
| | device | engine | status | | |
| | --- | --- | --- | | |
| | Mac Studio M5 Ultra (96 GB), macOS 27.2 | Metal: TensorFold [`6d0a128b0`](https://github.com/ashhart/TensorFold/commit/6d0a128b0cde35aa091bda933577cfced3519b8a) + [`patches/metal`](https://github.com/jayleaton/sensenova-loop9b-tensorfold/tree/main/patches/metal) `0001`-`0044` (engine tree `d33e2d93ea37`) | tested: text, images, tools, copy drafts | | |
| | 16 GB Macs (M-series) | the same | supported; measured on the M5 Ultra under a 16 GB Mac's Metal working-set cap | | |
| | MacBook Pro M4 Pro (24 GB) | the same | tested: decode | | |
| 8 GB Macs are not supported. On a DGX Spark, serve the 8-bit tree, | |
| [SenseNova-Loop-9B-MLX-8bit-vision](https://huggingface.co/jayleaton/SenseNova-Loop-9B-MLX-8bit-vision): 4-bit keeps | |
| top-1 on 220 of 249 positions there against 246 for 8-bit. | |
| ## Serve | |
| Needs Xcode with the macOS SDK and Metal toolchain, and Zig 0.17.0: | |
| ```bash | |
| git clone https://github.com/jayleaton/sensenova-loop9b-tensorfold.git && cd sensenova-loop9b-tensorfold | |
| scripts/build-engine.sh metal engine-metal # fetches the pin, applies patches/metal, builds | |
| hf download jayleaton/SenseNova-Loop-9B-MLX-4bit --local-dir loop9b-mlx4 | |
| scripts/serve.sh metal engine-metal loop9b-mlx4 # --context 33024 --parallel 4, copy drafts on | |
| PROFILE=vision scripts/serve.sh metal engine-metal loop9b-mlx4 # image input: --context 4096 --no-drafts --vision | |
| ``` | |
| `tool_choice: "required"`, a named function and `response_format` need `libtfgrammar.dylib` beside the server (xgrammar | |
| v0.2.8, `zig build grammar`); see the recipe's [METAL page](https://github.com/jayleaton/sensenova-loop9b-tensorfold/blob/main/docs/METAL.md). Thinking is on by | |
| default; send `"chat_template_kwargs": {"enable_thinking": false}` to turn it off. | |
| ## Measured | |
| Mac Studio M5 Ultra, from the recipe's [METAL page](https://github.com/jayleaton/sensenova-loop9b-tensorfold/blob/main/docs/METAL.md): | |
| | | | | |
| | --- | --- | | |
| | decode, drafts off | 59.5-60.0 tok/s one chat | | |
| | cold prefill, 512 / 2k / 8k / 32k tokens | 2,709 / 2,752 / 2,717 / 2,461 tok/s; TTFT 0.019 s for a 14-token prompt | | |
| | copy drafts, one chat | re-quote 362 tok/s, code edit 87, prose 94; 16/16 reply pairs token-identical to drafts off | | |
| | task gate, 64 greedy tokens against transformers on the same 4-bit weights | code 64/64, reasoning 64/64, prose 19/64 (the reference's own bf16/fp32 noise floor) | | |
| | tools (with `libtfgrammar`) | 10/10 cases, 4/4 with four chats at once | | |
| | images | vision tower within bf16 noise of the reference (cosine 0.99869); one image a request | | |
| | Mac | decode | prefill | peak memory | | |
| | --- | --- | --- | --- | | |
| | 16 GB Mac (working-set cap on the M5 Ultra) | 58 tok/s | 2,719 tok/s at 8k | 7.60 / 7.82 / 9.13 GiB at 4k / 8k / 32k | | |
| | MacBook Pro M4 Pro, 24 GB | 26.7 tok/s | about 230 tok/s | | | |
| The 16 GB runs used the text shards alone; the vision tower adds 0.85 GiB only when served with `--vision`. Teacher- | |
| forced top-1 over 64 positions against BF16 (prose / code / reasoning): 57 / 62 / 61. | |
| **Task benchmarks** of the same 4-bit codes on a DGX Spark (served CUDA v5 engine, `--parallel 4 --context 33024`, | |
| copy drafts on), next to the 8-bit tree under the same harness: | |
| | benchmark | 4-bit (these codes) | 8-bit | | |
| | --- | --- | --- | | |
| | HumanEval+ | 126/164 (76.8%) | 130/164 (79.3%) | | |
| | MBPP+ | 239/378 (63.2%) | 253/378 (66.9%) | | |
| | GSM8K, thinking off / on | 282/300 / 283/300 | 284/300 / 285/300 | | |
| | MATH, thinking off / on | 171/200 / 182/200 | 175/200 / 183/200 | | |
| | BFCL | 116/200 (58.0%) | 127/200 (63.5%) | | |
| ## Sizes | |
| This repository: 13 model files, 5,982,934,074 bytes (5.98 GB, 5.57 GiB), plus this card and `LICENSE`. The weight files and their sha256: | |
| | file | bytes | sha256 | | |
| | --- | ---: | --- | | |
| | `model-00001.safetensors` | 1,822,297,144 | `cdf270f6fd2925c7439b5ea7b35104cb783a07ca6701849789ed56f60b9f006e` | | |
| | `model-00002.safetensors` | 1,826,655,872 | `38267bc2bdecc70102d15af0896f576d6981b72b3e17a61fcbe33f9b7e104b51` | | |
| | `model-00003.safetensors` | 1,800,704,712 | `7de04179b40988b6edfb25fc0dc966013286f200254c7e336ab4f97794e54127` | | |
| | `model-00004.safetensors` | 513,156,744 | `92cd2f7273c28ee554bbeecca9babf6fd49242009d3d46ddc29c4812cbc4cee6` | | |
| ## How it was made | |
| From the BF16 checkpoint at revision `329a65f701ba8c48b73c30d0cbaa6f0449733673`, with the converter carried in the Metal series: | |
| ```bash | |
| python3 engine-metal/tools/sensenova_loop_mlx.py --source SenseNova-Loop-9B --out loop9b-mlx4 | |
| ``` | |
| 4-bit MLX affine codes with bf16 scales and biases per group of 64, the zero-centred RMSNorm weights folded to `1 + w`, | |
| the mHC routers in fp32, and the 333 vision tensors in bf16 as `vision_tower.*`. `tensorfold-convert.json` records a | |
| round-trip check. Details: [CONVERT](https://github.com/jayleaton/sensenova-loop9b-tensorfold/blob/main/docs/CONVERT.md). | |
| ## Credits | |
| - **Model: [SenseNova-Loop-9B](https://huggingface.co/sensenova/SenseNova-Loop-9B) by [SenseNova](https://huggingface.co/sensenova)** | |
| ([GitHub](https://github.com/OpenSenseNova/SenseNova-Loop)). The looped decoder, the two-stream mHC, the vision | |
| tower and the reference `modeling_sensenova_loop.py` are theirs. | |
| - **Engine: [TensorFold](https://github.com/ashhart/TensorFold) by Ash Hart (ashhart) and the TensorFold contributors**, Apache License 2.0. The | |
| recipe's patches extend it; they are not upstream changes. | |
| - **Shared-round drafter pricing:** upstream TensorFold PR [#2475](https://github.com/ashhart/TensorFold/pull/2475) by | |
| [jrb-ai](https://github.com/jrb-ai). | |
| - **EXL3:** [exllamav3](https://github.com/turboderp-org/exllamav3) by turboderp, the format and quantizer, built on | |
| [QTIP](https://github.com/Cornell-RelaxML/qtip) (Cornell RelaxML) trellis codebooks. | |
| - **Recipe, conversion and measurements:** [jayleaton/sensenova-loop9b-tensorfold](https://github.com/jayleaton/sensenova-loop9b-tensorfold). | |
| ## Licence | |
| Apache License 2.0, the base model's licence. This repository is a quantized derivative of | |
| [sensenova/SenseNova-Loop-9B](https://huggingface.co/sensenova/SenseNova-Loop-9B) (revision `329a65f701ba8c48b73c30d0cbaa6f0449733673`) by SenseNova: the weights were converted to MLX affine | |
| codes and are otherwise SenseNova's model. The licence text is in [LICENSE](LICENSE). The tokenizer, chat template, | |
| processor and generation configuration come from the base model. | |