SenseNova-Loop-9B MLX 8-bit + BF16 vision (TensorFold, DGX Spark)

The tree the recipe serves on one NVIDIA DGX Spark (GB10, 128 GB, TP = 1): SenseNova-Loop-9B's text weights as 8-bit MLX affine codes, group 64, the checkpoint's vision tower in BF16 as one more shard, and a config with YaRN stretched to factor 4 (1,048,576 positions over the trained 262,144). It is the recipe's loop9b-q8-vis, byte for byte: download it instead of converting the checkpoint yourself.

Experimental. This is a looped model (physical layers 4-27 run twice, 56 executions from 32 weight sets, wrapped in a two-stream mHC). Stock transformers, vLLM, llama.cpp, mlx-lm and exllamav3 cannot serve these files; they need TensorFold with this recipe's patch series. Read the recipe's known issues before you rely on it.

Tested on

device engine status
DGX Spark (GB10, 128 GB) CUDA v5: TensorFold 038e4a278 + patches/cuda 0001-0022 (engine tree d5938b12bbcc) served: 1M context, 4 chats, images
DGX Spark CUDA v6: the same + 0023-0027 (engine tree 9bac6ecbdef1) validated on this tree, not yet served

Serve

Needs the CUDA 13 toolkit and Zig 0.17.0. Build the engine from the recipe, then serve this repository's directory with the measured profile:

git clone https://github.com/jayleaton/sensenova-loop9b-tensorfold.git && cd sensenova-loop9b-tensorfold
scripts/build-engine.sh cuda engine-cuda          # fetches the pin, applies patches/cuda, checks the tree, builds
hf download jayleaton/SenseNova-Loop-9B-MLX-8bit-vision --local-dir loop9b-q8-vis
scripts/serve.sh cuda engine-cuda loop9b-q8-vis   # 1M context, 4 chats, fp8 KV, images, port 8090

which runs:

TENSORFOLD_MEMORY_RESERVE_GIB=36 TF_QWEN_KV_PAGES=256 \
engine-cuda/zig-out/native/bin/tensorfold-native serve loop9b-q8-vis --name loop9b --port 8090 \
  --parallel 4 --context 1048576 --kv-cache fp8 --max-tokens 32768 --thinking-budget 4096 --decode-share 0.5 \
  --prompt-cache-gib 24 --spill-gib 256 --learn-daily-gib 256 --checkpoint-slots 8 --vision

tool_choice: "required", a named function and response_format need libtfgrammar.so beside the server (xgrammar v0.2.8, zig build grammar); see the recipe's CUDA page. Thinking is on by default; send "chat_template_kwargs": {"enable_thinking": false} to turn it off.

Measured

On one DGX Spark, the served v5 engine, from the recipe's CUDA page:

decode 14.4-18.0 tok/s one chat; 13.2-14.6 tok/s each with four chats
copy drafts, code chats, aggregate 15.3 / 33.3 / 54.5 tok/s at 1 / 2 / 4 chats, against 14.5 / 28.0 / 54.0 without; token ids identical
cold prefill 63,716 / 223,827 / 492,251 tokens in 42.5 / 227.0 / 785.3 s
prefix reuse a 74K prompt's second turn: TTFT 0.66 s against 52.4 s cold
long context needle recall 4/4 at 231K-362K tokens and at 1M
fidelity top-1 equal to BF16 HF; 8-bit top-1 246/249 positions against BF16; token check 6/6
tools strict tool gate pass (required and auto)
soak 45 requests over 12 rotating conversations, 0 errors

v6 on this tree (the loop-seam fix, 0023-0027): mean KL to the BF16 reference 0.0244 to 0.0186 over 736 positions, top-1 707 to 717/736; prefill 1.04x / 1.09x / 1.13x at 64K / 224K / 492K; decode unchanged.

Task benchmarks on the served v5 engine, --parallel 4 --context 33024, copy drafts on. Same harness and settings for each column; the 4-bit column is SenseNova-Loop-9B-MLX-4bit's text codes, and the control is stock Qwen3.5-9B through the same converter at 4 bits:

benchmark this tree (8-bit) 4-bit stock Qwen3.5-9B, 4-bit
HumanEval+ 130/164 (79.3%) 126/164 (76.8%) 129/164
MBPP+ 253/378 (66.9%) 239/378 (63.2%) 233/378
GSM8K, thinking off / on 284/300 / 285/300 282/300 / 283/300 274/300 / 281/300
MATH, thinking off / on 175/200 / 183/200 171/200 / 182/200 166/200 / 160/200
BFCL 127/200 (63.5%) 116/200 (58.0%) 183/200

Under tool_choice: "auto" the model often answers in prose instead of calling a tool (62 of this tree's 73 BFCL misses); required or a named function forces the call. It is the model, not the quantization.

Sizes

This repository: 16 model files, 10,459,302,765 bytes (10.46 GB, 9.74 GiB), plus this card and LICENSE. The weight files and their sha256:

file bytes sha256
model-00001.safetensors 2,161,378,000 65f5186a76c01325cae0e86f11d38d7b81fc2433d36bb3af56baa81aaaa6bc61
model-00002.safetensors 1,801,325,816 7a8c8b502ede334796aad2eeb690ee65222391faf8978e04ca750adcf58f32f5
model-00003.safetensors 1,837,840,504 53f97c225148f14a58816a6dcffbd4ce187911fb3baa2dff6e257340e867c385
model-00004.safetensors 1,822,738,152 8d493ed478f8a12f451f518e3f507bae2710b71cbc246a477bac4ca6ecfc95f0
model-00005.safetensors 1,850,350,488 cfc2edceec024545dfdeb81e0bae8a8050e26c24c67b9c895393af3e854bd3b7
model-00006.safetensors 53,494,400 842660d6512207eb6bfcf96ea5ca6ac05f6556320ce5348ffeccef7bef63e679
model-visual.safetensors 912,057,992 04efa7bb15041331f4cce8e28d5cd5fff2ed8f5387305da51e43c78e2921cf32

How it was made

From the BF16 checkpoint at revision 329a65f701ba8c48b73c30d0cbaa6f0449733673:

python3 engine-cuda/tools/sensenova_loop_mlx.py --source SenseNova-Loop-9B --out loop9b-mlx8 --bits 8
python3 scripts/serve-tree.py engine-cuda loop9b-mlx8 SenseNova-Loop-9B loop9b-q8-vis 4

The converter (tools/sensenova_loop_mlx.py, carried in patches/cuda/0008) writes 8-bit MLX affine codes with bf16 scales and biases per group of 64, folds the zero-centred RMSNorm weights to 1 + w, keeps the mHC routers in fp32 and records a round-trip check in tensorfold-convert.json. serve-tree.py adds the checkpoint's BF16 vision tower as model-visual.safetensors and writes the YaRN factor-4 config. Details: CONVERT.

Credits

  • Model: SenseNova-Loop-9B by SenseNova (GitHub). The looped decoder, the two-stream mHC, the vision tower and the reference modeling_sensenova_loop.py are theirs.
  • Engine: TensorFold by Ash Hart (ashhart) and the TensorFold contributors, Apache License 2.0. The recipe's patches extend it; they are not upstream changes.
  • Shared-round drafter pricing: upstream TensorFold PR #2475 by jrb-ai.
  • EXL3: exllamav3 by turboderp, the format and quantizer, built on QTIP (Cornell RelaxML) trellis codebooks.
  • Recipe, conversion and measurements: jayleaton/sensenova-loop9b-tensorfold.

Licence

Apache License 2.0, the base model's licence. This repository is a quantized derivative of sensenova/SenseNova-Loop-9B (revision 329a65f701ba8c48b73c30d0cbaa6f0449733673) by SenseNova: the weights were converted to MLX affine codes and are otherwise SenseNova's model. The licence text is in LICENSE. The tokenizer, chat template, processor and generation configuration come from the base model.

Downloads last month
-
Safetensors
Model size
9B params
Tensor type
U32
路
F32
路
BF16
路
Inference Providers NEW
This model isn't deployed by any Inference Provider. 馃檵 Ask for provider support

Model tree for jayleaton/SenseNova-Loop-9B-MLX-8bit-vision

Quantized
(42)
this model