SenseNova-Loop-9B on cortiq (CMF): CUDA, Vulkan, Metal, CPU β€” a standard file and a Turbo file for long documents

#13
by infosave - opened

SenseNova-Loop-9B now runs in cortiq, an open-source inference engine in Rust with no Python and no ML framework: NVIDIA cards through CUDA, NVIDIA, AMD and Intel GPUs through Vulkan, Apple silicon through Metal, and the CPU. The model comes as single CMF files in two versions, from cortiq 0.9.5 on.

cargo install cortiq-cli --features cuda        # without --features cuda: Vulkan, DX12, Metal, CPU
hf download infosave/SenseNova-Loop-9B-cmf sensenova-loop-9b-q4tp.cmf --local-dir .
CMF_GPU=cuda cortiq run sensenova-loop-9b-q4tp.cmf --prompt "Write a Python function that merges two sorted lists." --greedy --no-think

1. Standard: infosave/SenseNova-Loop-9B-cmf, for chat, code and images. q4tp (5.7 GB, GPTQ calibrated over both visits of the loop) and q8_2f (10.2 GB). The files keep the loop (24 layers run twice), the two mHC streams, the vision encoder and the MTP head, whose drafts the card checks in one batch. Greedy text of q8_2f matches Transformers float32 on the reference prompts. Code, 256-token answers, the same text with drafts and without:

with drafts, tok/s without, tok/s
RTX A5000, CUDA 88.1 51.0
RTX A5000, Vulkan 57.5 49.3
MacBook Air M4 (24 GB), Metal 14.4 9.5

2. Turbo: infosave/SenseNova-Loop-9B-Turbo-cmf, for long documents. The same 4-bit weights plus a header that holds card memory in place: full-attention passes keep the first rows, the last rows and rows chosen as important exact, and the rest in a fixed far field. The loop makes this matter more, since each of its passes keeps its own keys and values.

RTX A5000 standard turbo
CUDA, 128 000 tokens: card memory / decode 16.5 GB / 30.5 tok/s 8.3 GB / 49.9 tok/s
CUDA, 250 000 tokens β€” 8.3 GB / 50.4 tok/s
Vulkan, 128 000 tokens 19.4 GB / 23.5 tok/s 5.8 GB / 46.2 tok/s
facts planted in 60 000 / 120 000 / 240 000 characters 4/4, 4/4, β€” 4/4 at each

Standard and turbo through CUDA

Past the exact rows the turbo file's text is approximate: what lies far back fades. Short prompts (under 8192 tokens) run exact attention and the drafts, as in the standard file.

Two notes from the conversion, in case they help others: the MTP head ships in mtp.safetensors, outside model.safetensors.index.json, so tools that follow the index alone miss it; and YaRN applies at every length here, so in cortiq its cos/sin factor is folded into the q/k norms once at load.

The far field is the subject of U.S. patent application No. 19/738,763. Thanks for releasing the model and its looped design. Feedback is welcome.

Sign up or log in to comment