SenseNova-Loop-9B EXL3 5.0 bpw (TensorFold, 16 GB RTX)
An EXL3 quantization of sensenova/SenseNova-Loop-9B at 5.0 bits per weight (6-bit output
head, BF16 embedding), sized for a 16 GB NVIDIA RTX card. It is the recipe's sn-5.0 pack, byte for
byte: download it instead of running the quantize chain yourself. The other packs: 4.0 / 6.0. This one is
between the two.
Experimental. This is a looped model (physical layers 4-27 run twice, 56 executions from 32 weight sets, wrapped in a two-stream mHC). Stock transformers, vLLM, llama.cpp, mlx-lm and exllamav3 cannot serve these files; they need TensorFold with this recipe's patch series. Read the recipe's known issues before you rely on it.
Tested on
| device | engine | status |
|---|---|---|
RTX 5070 Ti 16 GB (Blackwell, sm_120), WSL2 Ubuntu 24.04 |
RTX: TensorFold 038e4a278 + patches/cuda 0001-0022 + patches/rtx 0001-0002 (engine tree 0f61f4723db8) |
tested |
| DGX Spark (GB10) under a 16 GB memory cap | the same EXL3 code | sizing and quality |
Other 16 GB cards and native Windows are untested. Text only: the packs do not carry the vision tower.
Serve
Needs the CUDA 13 toolkit and Zig 0.17.0 (on Windows, inside WSL2; see the recipe's WSL2 steps):
git clone https://github.com/jayleaton/sensenova-loop9b-tensorfold.git && cd sensenova-loop9b-tensorfold
scripts/build-engine.sh rtx engine-rtx # SM=120 (RTX 50) by default; set SM= for another card
hf download jayleaton/SenseNova-Loop-9B-EXL3-5.0bpw --local-dir sn-5.0
scripts/serve.sh rtx engine-rtx sn-5.0 # --parallel 1 --kv-cache fp8 --context 212992 --prompt-cache-gib 0
In WSL2, keep the pack on ext4 under your home directory, never under /mnt/c. tool_choice: "required", a named
function and response_format need libtfgrammar.so beside the server. More fits (several chats, the host prompt
cache, bf16 KV) are in the recipe's RTX page.
Measured
On the RTX 5070 Ti 16 GB (WSL2), drafts off, from the recipe's RTX page:
| largest fp8-KV window on the card | 286,720 tokens |
| gate cosine / top-1 against BF16 (298 positions) | 0.99779 / 290, equal to the Spark's run |
| decode at 4k / 8k / 32k | 50.5 / 51.2 / 44.0 tok/s |
| prefill at 4k / 8k / 32k | 1,950 / 1,885 / 1,789 tok/s |
| tool calls | not run |
| other | peak 9.32 GiB at a 32k bf16-KV window |
On a DGX Spark, against BF16 over 249 positions (the three packs and 8-bit MLX for scale):
| pack | size | cosine / top-1 | device peak, 32k, bf16 KV | largest fp8-KV window, 15.0 / 14.5 GiB cap |
|---|---|---|---|---|
sn-4.0 |
6.31 GB | 0.99393 / 236 | 8.51 GiB | 294,912 / 278,528 |
sn-5.0 |
7.17 GB | 0.99807 / 244 | 9.31 GiB | 270,336 / 253,952 |
sn-6.0 |
8.29 GB | 0.99937 / 244 | 10.35 GiB | 229,376 / 212,992 |
| MLX 8-bit | 0.99975 / 246 |
15.0 GiB is what a 16 GB card leaves after the driver context; 14.5 GiB allows for a Windows desktop on the same card. Decode varies with desktop load and the card's clock under sustained load; quote a quiet, cool card.
Sizes
This repository: 11 model files, 7,170,609,346 bytes (7.17 GB, 6.68 GiB), plus this card and LICENSE. The weight files and their sha256:
| file | bytes | sha256 |
|---|---|---|
model-sensenova-extras.safetensors |
12,611,592 | f9ee50391ab6d77f00daaa59e79995465b17b14e0df8d279ccb6020bd29d11a4 |
model.safetensors |
7,137,567,490 | b57d62fa6b0cb1b70356d167af8e9194be875762f982574bb618c9a575be67c6 |
How it was made
exllamav3 cannot trace this model (sensenova_loop is not a registered architecture, and its qwen3_5 forward has
no loop or mHC). The recipe's RTX series (patches/rtx/0001) carries the chain: Hessians captured from the real looped
model with transformers, one per unique weight summed over both visits (250 rows x 2,048 tokens of exllamav3's default
calibration set), then exllamav3's own quantizer from those Hessians. exllamav3 1.6.7
(turboderp-org/exllamav3@8c593e4e5), mul1 codebook, output scales always, body 5.0 bpw, head 6 bpw,
embedding BF16. The mHC routers and loop gates (97 fp32 tensors) are attached unquantized in
model-sensenova-extras.safetensors. Steps and recipes: docs/exl3-sensenova-quantize.md and docs/exl3-recipes/ in
the patched engine checkout.
Credits
- Model: SenseNova-Loop-9B by SenseNova
(GitHub). The looped decoder, the two-stream mHC, the vision
tower and the reference
modeling_sensenova_loop.pyare theirs. - Engine: TensorFold by Ash Hart (ashhart) and the TensorFold contributors, Apache License 2.0. The recipe's patches extend it; they are not upstream changes.
- Shared-round drafter pricing: upstream TensorFold PR #2475 by jrb-ai.
- EXL3: exllamav3 by turboderp, the format and quantizer, built on QTIP (Cornell RelaxML) trellis codebooks.
- Recipe, conversion and measurements: jayleaton/sensenova-loop9b-tensorfold.
Licence
Apache License 2.0, the base model's licence. This repository is a quantized derivative of
sensenova/SenseNova-Loop-9B (revision 329a65f701ba8c48b73c30d0cbaa6f0449733673) by SenseNova: the weights were quantized to EXL3 and are
otherwise SenseNova's model. The licence text is in LICENSE. The tokenizer, chat template, processor and
generation configuration and modeling_sensenova_loop.py come from the base model unchanged.
- Downloads last month
- 23
Model tree for jayleaton/SenseNova-Loop-9B-EXL3-5.0bpw
Base model
sensenova/SenseNova-Loop-9B