SenseNova-Loop-9B MLX 8-bit + BF16 vision (TensorFold, DGX Spark)
The tree the recipe serves on one NVIDIA DGX Spark (GB10, 128 GB, TP = 1): SenseNova-Loop-9B's text
weights as 8-bit MLX affine codes, group 64, the checkpoint's vision tower in BF16 as one more shard, and a
config with YaRN stretched to factor 4 (1,048,576 positions over the trained 262,144). It is the recipe's
loop9b-q8-vis, byte for byte: download it instead of converting the checkpoint yourself.
Experimental. This is a looped model (physical layers 4-27 run twice, 56 executions from 32 weight sets, wrapped in a two-stream mHC). Stock transformers, vLLM, llama.cpp, mlx-lm and exllamav3 cannot serve these files; they need TensorFold with this recipe's patch series. Read the recipe's known issues before you rely on it.
Tested on
| device | engine | status |
|---|---|---|
| DGX Spark (GB10, 128 GB) | CUDA v5: TensorFold 038e4a278 + patches/cuda 0001-0022 (engine tree d5938b12bbcc) |
served: 1M context, 4 chats, images |
| DGX Spark | CUDA v6: the same + 0023-0027 (engine tree 9bac6ecbdef1) |
validated on this tree, not yet served |
Serve
Needs the CUDA 13 toolkit and Zig 0.17.0. Build the engine from the recipe, then serve this repository's directory with the measured profile:
git clone https://github.com/jayleaton/sensenova-loop9b-tensorfold.git && cd sensenova-loop9b-tensorfold
scripts/build-engine.sh cuda engine-cuda # fetches the pin, applies patches/cuda, checks the tree, builds
hf download jayleaton/SenseNova-Loop-9B-MLX-8bit-vision --local-dir loop9b-q8-vis
scripts/serve.sh cuda engine-cuda loop9b-q8-vis # 1M context, 4 chats, fp8 KV, images, port 8090
which runs:
TENSORFOLD_MEMORY_RESERVE_GIB=36 TF_QWEN_KV_PAGES=256 \
engine-cuda/zig-out/native/bin/tensorfold-native serve loop9b-q8-vis --name loop9b --port 8090 \
--parallel 4 --context 1048576 --kv-cache fp8 --max-tokens 32768 --thinking-budget 4096 --decode-share 0.5 \
--prompt-cache-gib 24 --spill-gib 256 --learn-daily-gib 256 --checkpoint-slots 8 --vision
tool_choice: "required", a named function and response_format need libtfgrammar.so beside the server (xgrammar
v0.2.8, zig build grammar); see the recipe's CUDA page. Thinking is on by default;
send "chat_template_kwargs": {"enable_thinking": false} to turn it off.
Measured
On one DGX Spark, the served v5 engine, from the recipe's CUDA page:
| decode | 14.4-18.0 tok/s one chat; 13.2-14.6 tok/s each with four chats |
| copy drafts, code chats, aggregate | 15.3 / 33.3 / 54.5 tok/s at 1 / 2 / 4 chats, against 14.5 / 28.0 / 54.0 without; token ids identical |
| cold prefill | 63,716 / 223,827 / 492,251 tokens in 42.5 / 227.0 / 785.3 s |
| prefix reuse | a 74K prompt's second turn: TTFT 0.66 s against 52.4 s cold |
| long context | needle recall 4/4 at 231K-362K tokens and at 1M |
| fidelity | top-1 equal to BF16 HF; 8-bit top-1 246/249 positions against BF16; token check 6/6 |
| tools | strict tool gate pass (required and auto) |
| soak | 45 requests over 12 rotating conversations, 0 errors |
v6 on this tree (the loop-seam fix, 0023-0027): mean KL to the BF16 reference 0.0244 to 0.0186 over 736
positions, top-1 707 to 717/736; prefill 1.04x / 1.09x / 1.13x at 64K / 224K / 492K; decode unchanged.
Task benchmarks on the served v5 engine, --parallel 4 --context 33024, copy drafts on. Same harness and settings
for each column; the 4-bit column is SenseNova-Loop-9B-MLX-4bit's
text codes, and the control is stock Qwen3.5-9B through the same converter at 4 bits:
| benchmark | this tree (8-bit) | 4-bit | stock Qwen3.5-9B, 4-bit |
|---|---|---|---|
| HumanEval+ | 130/164 (79.3%) | 126/164 (76.8%) | 129/164 |
| MBPP+ | 253/378 (66.9%) | 239/378 (63.2%) | 233/378 |
| GSM8K, thinking off / on | 284/300 / 285/300 | 282/300 / 283/300 | 274/300 / 281/300 |
| MATH, thinking off / on | 175/200 / 183/200 | 171/200 / 182/200 | 166/200 / 160/200 |
| BFCL | 127/200 (63.5%) | 116/200 (58.0%) | 183/200 |
Under tool_choice: "auto" the model often answers in prose instead of calling a tool (62 of this tree's 73 BFCL
misses); required or a named function forces the call. It is the model, not the quantization.
Sizes
This repository: 16 model files, 10,459,302,765 bytes (10.46 GB, 9.74 GiB), plus this card and LICENSE. The weight files and their sha256:
| file | bytes | sha256 |
|---|---|---|
model-00001.safetensors |
2,161,378,000 | 65f5186a76c01325cae0e86f11d38d7b81fc2433d36bb3af56baa81aaaa6bc61 |
model-00002.safetensors |
1,801,325,816 | 7a8c8b502ede334796aad2eeb690ee65222391faf8978e04ca750adcf58f32f5 |
model-00003.safetensors |
1,837,840,504 | 53f97c225148f14a58816a6dcffbd4ce187911fb3baa2dff6e257340e867c385 |
model-00004.safetensors |
1,822,738,152 | 8d493ed478f8a12f451f518e3f507bae2710b71cbc246a477bac4ca6ecfc95f0 |
model-00005.safetensors |
1,850,350,488 | cfc2edceec024545dfdeb81e0bae8a8050e26c24c67b9c895393af3e854bd3b7 |
model-00006.safetensors |
53,494,400 | 842660d6512207eb6bfcf96ea5ca6ac05f6556320ce5348ffeccef7bef63e679 |
model-visual.safetensors |
912,057,992 | 04efa7bb15041331f4cce8e28d5cd5fff2ed8f5387305da51e43c78e2921cf32 |
How it was made
From the BF16 checkpoint at revision 329a65f701ba8c48b73c30d0cbaa6f0449733673:
python3 engine-cuda/tools/sensenova_loop_mlx.py --source SenseNova-Loop-9B --out loop9b-mlx8 --bits 8
python3 scripts/serve-tree.py engine-cuda loop9b-mlx8 SenseNova-Loop-9B loop9b-q8-vis 4
The converter (tools/sensenova_loop_mlx.py, carried in patches/cuda/0008) writes 8-bit MLX affine codes with bf16
scales and biases per group of 64, folds the zero-centred RMSNorm weights to 1 + w, keeps the mHC routers in fp32
and records a round-trip check in tensorfold-convert.json. serve-tree.py adds the checkpoint's BF16 vision tower
as model-visual.safetensors and writes the YaRN factor-4 config. Details: CONVERT.
Credits
- Model: SenseNova-Loop-9B by SenseNova
(GitHub). The looped decoder, the two-stream mHC, the vision
tower and the reference
modeling_sensenova_loop.pyare theirs. - Engine: TensorFold by Ash Hart (ashhart) and the TensorFold contributors, Apache License 2.0. The recipe's patches extend it; they are not upstream changes.
- Shared-round drafter pricing: upstream TensorFold PR #2475 by jrb-ai.
- EXL3: exllamav3 by turboderp, the format and quantizer, built on QTIP (Cornell RelaxML) trellis codebooks.
- Recipe, conversion and measurements: jayleaton/sensenova-loop9b-tensorfold.
Licence
Apache License 2.0, the base model's licence. This repository is a quantized derivative of
sensenova/SenseNova-Loop-9B (revision 329a65f701ba8c48b73c30d0cbaa6f0449733673) by SenseNova: the weights were converted to MLX affine
codes and are otherwise SenseNova's model. The licence text is in LICENSE. The tokenizer, chat template,
processor and generation configuration come from the base model.
- Downloads last month
- -
Model tree for jayleaton/SenseNova-Loop-9B-MLX-8bit-vision
Base model
sensenova/SenseNova-Loop-9B