Q4_K_M: ssm_alpha / ssm_beta are Q4_K — possible missing F32 pin; plus a metadata/layout question

#1
by wepiqx - opened

Hi AtomicChat team!!!
Thanks for publishing these -- the README is unusually clear, and the mHC routing precision note is exactly the right thing to document. I downloaded SenseNova-Loop-9B-Preview-Q4_K_M.gguf (SHA-256 f3b321f001b04fdc8915b4067f0592f2b7cf55676b584080bc3129b964bf1d4b, 5,628,723,104 bytes) and inspected the actual GGUF tensor types so I have two observations

  1. The SSM alpha / beta gates are not F32 in Q4_K_M
    The model has 24 SSM layers, and each has ssm_alpha.weight and ssm_beta.weight, shape [4096, 32] — 48 tensors total. In your Q4_K_M:
    ssm_alpha: 24 × Q4_K
    ssm_beta: 24 × Q4_K

By contrast, the 96 mHC tensors are correctly F32, and all 24 ssm_conv1d tensors are F32. So the precision intent is clearly there; the SSM gates look like a narrow omission rather than a general “quantize everything” choice.
Why this may be happening: in the upstream qwen35 path, only ssm_conv1d is special-cased to F32. There is no equivalent rule for ssm_alpha / ssm_beta, so unless the patched build adds an explicit rule, they land at Q4_K. A regex that matches blk.N.ssm_alpha.weight / blk.N.ssm_beta.weight and forces F32 should be enough.
The cost is tiny: 48 × ~0.43 MiB extra versus Q4_K = ~20.6 MiB on a 5.2 GB file. That is cheap enough that I would pin them even if the benefit were only suspected.
For context: in our own patched sensenova_loop builds, pinning these 48 gates to F32 (together with our allocator) produced a qualitative generation improvement, including under MTP; we are still isolating the exact contribution, but the direction and the price both point the same way. Your mHC result suggests you already found the same class of issue for routing — this looks like the same “small tensor, large behavioral effect” pattern.
Could you add explicit F32 rules for ssm_alpha and ssm_beta in the next quant revision? A small type dump per quant would also make this easy to verify.

  1. Metadata/layout: qwen35 + num_loops vs sensenova_loop
    This is the part that puzzles me. Your Q4_K_M reports:
    general.architecture = qwen35
    tensor count = 523
    qwen35.block_count = 32
    qwen35.num_loops = 2
    qwen35.loop_start = 4
    qwen35.loop_end = 28
    A sensenova_loop GGUF from our patched build reports:

general.architecture = sensenova_loop
tensor count = 538
sensenova_loop.block_count = 57
block names = blk.0 … blk.27, blk.52 … blk.56

Your README says the decoder stores 32 layers and executes layers 4–27 twice, and that MTP/vision are omitted. That explains the 15-tensor difference (our build includes the MTP block), but it does not explain the different architecture key or the different block numbering. Is qwen35 + num_loops the intended GGUF representation produced by your patch, or is it an artifact of the converter? Which runtime/build is expected to load it — only your patched engine, or any sensenova_loop-aware build?
Could you share the exact patched llama.cpp fork/PR (you mention base commit 5047748f11)? I would like to reproduce your path and compare the same graph rather than comparing two different tensor layouts. If the qwen35 metadata is intentional, that is fine — we just want to know whether the files are cross-compatible or engine-specific.

  1. Things I verified as correct
  • 96 mHC tensors are F32.
  • 24 ssm_conv1d tensors are F32.
  • No vision tower / mmproj, as documented.
  • No MTP block, as documented.

So this is not a complaint about the text-only scope; it is specifically about the SSM gates and the architecture/layout metadata
4. Small verification snippet
If useful, this is the check I used:

from collections import Counter
from gguf import GGUFReader

r = GGUFReader("SenseNova-Loop-9B-Preview-Q4_K_M.gguf")
gates = [t for t in r.tensors
if "ssm_alpha" in t.name or "ssm_beta" in t.name]
print(len(gates), Counter(t.tensor_type.name for t in gates))

48 Counter({'Q4_K': 48})

Happy to send the full tensor/type dump or test a revised quant thanks again for the release - the architecture notes made this much easier to audit

Sign up or log in to comment