SenseNova-Loop-9B-FP8

Block-wise FP8 weight quantization of sensenova/SenseNova-Loop-9B, the 9B-parameter looped multimodal decoder (32 physical layers executed as 56 layer passes, two-stream mHC, hybrid linear/full attention). The checkpoint layout matches the official Qwen FP8 releases (quant_method: fp8, block-wise weight_scale_inv), so FP8-aware engines can load it without renames or extra files.

Quantization recipe

  • Weights: FP8 e4m3fn, one BF16 scale per 128x128 block (<name>.weight_scale_inv, shape [ceil(out/128), ceil(in/128)]), per-block absmax scaling. Weight-only, no calibration data.
  • quantization_config: quant_method: fp8, fmt: e4m3, weight_block_size: [128, 128], activation_scheme: dynamic — same schema as Qwen/Qwen3-32B-FP8.
  • Quantized to FP8 (344 tensors):
    • Decoder full-attention q/k/v/o_proj
    • Gated-DeltaNet linear-attention in_proj_a/b/qkv/z and out_proj
    • MLP gate/up/down_proj
    • Vision blocks 3-26 (attn.qkv, attn.proj, mlp.linear_fc1/2)
  • Kept at original precision:
    • embed_tokens, lm_head, all norms (BF16)
    • conv1d, A_log, dt_bias (BF16)
    • mHC routing mhc_attn / mhc_ffn and loop_gate (FP32, as in the original)
    • Vision blocks 0-2, merger, patch_embed, pos_embed (BF16) — same ignore policy as official Qwen3-VL-*-FP8
  • Size: 18.83 GB (BF16) -> 11.55 GB (-38.7%).

Verification

Single RTX 4090, transformers 5.18.0 / torch 2.11.0+cu130 (the upstream-pinned environment). BF16 original vs dequantized FP8, both loaded through the identical transformers code path:

  • Weight error (dequantized FP8 vs original BF16): mean relative L2 2.64%, max 2.66% across the 344 quantized tensors (expected magnitude for e4m3 block quantization).
  • Greedy generation, 4 prompts (math, code, Chinese, reasoning), thinking off:
    • math: 3/3 tokens identical ("91")
    • code: 160/160 tokens identical
    • Chinese: identical for 20 tokens, then an equivalent paraphrase
    • reasoning: identical for 59 tokens, then equivalent steps; both end with the correct answer (9)
  • Fixed-input logits: cosine similarity 0.9996 (last position) / 0.9998 (mean over positions), top-1 agreement.
  • Multimodal smoke test (synthetic image with shapes + text): both models correctly describe the red circle, blue square and "HELLO FP8" text, with near-identical wording.

Usage

The checkpoint targets FP8-aware inference engines. SenseNova-Loop engine support is available via vLLM PR #64832 and SGLang PR #47281; FP8 kernel coverage for this architecture in those branches is not yet verified upstream — BF16 is the documented path there.

For transformers (the custom modeling code loads BF16), dequantize at load time with the verified helper shipped in this repo:

hf download letechlead/SenseNova-Loop-9B-FP8 --local-dir SenseNova-Loop-9B-FP8
python dequant_to_bf16.py --src SenseNova-Loop-9B-FP8 --dst SenseNova-Loop-9B-FP8-bf16
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SenseNova-Loop-9B-FP8-bf16"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True,
    dtype=torch.bfloat16, device_map={"": "cuda:0"}).eval()

The dequantized copy is numerically the FP8 model (same weights, BF16 container) and needs ~19 GB of disk plus the usual BF16 runtime memory. For native BF16 or 4-bit deployment use the upstream releases: sensenova/SenseNova-Loop-9B and the official AWQ variant.

License

Apache-2.0, inherited from the original model. Original weights (c) SenseNova; quantization by LeTechLead.

Downloads last month
52
Safetensors
Model size
9B params
Tensor type
BF16
·
F32
·
F8_E4M3
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for letechlead/SenseNova-Loop-9B-FP8

Quantized
(25)
this model