SenseNova-Loop-9B-Preview-W4A16-AutoRound

4-bit (W4A16) AutoRound quantization of sensenova/SenseNova-Loop-9B-Preview.

Quantization Summary

  • Method: AutoRound 0.16.1 (auto-round, auto_round:auto_gptq packing)
  • Precision: 4-bit symmetric weights, 16-bit activations (W4A16), group size 128, 100 tuning iterations (256 calibration samples, seqlen=2048)
  • Scope: 248/359 layers quantized across the 32 decoder blocks (model.language_model.layers); lm_head, model.language_model.embed_tokens, model.visual.* (vision tower & merger), and mtp.* kept in BF16
  • Artifact size: ~8.5 GB on disk (model-00001-of-00002.safetensors, model-00002-of-00002.safetensors, model_extra_tensors.safetensors)

Evaluation Status

Check Result Notes
Canary suite (5 prompts) 5/5 passed Coding, math, architecture, strict instruction-following, factual extraction (load_time_sec: 8.02s, vram_budget_passed: true)
WikiText-2 PPL (W4A16) 22.068 100,000 tokens, seqlen=2048, stride=512, bfloat16 (316.12s, loaded in 3.13s)
BF16 PPL baseline / ratio Pending Single 24 GB GPU OOMd on full BF16 at seqlen=2048
Fleet / downstream bench Pending —
Serving engine check Pending Base architecture uses trust_remote_code=True (modeling_sensenova_loop.py) and targets SGLang / LightLLM / transformers

Note on local speed: Canary generation averaged 0.71 tok/s under unoptimized Hugging Face transformers (auto_round:tritonv2_zp) with the PyTorch reference causal_conv1d fallback. This is not representative of SGLang/LightLLM serving throughput.

Usage (transformers)

from transformers import AutoModelForImageTextToText, AutoTokenizer

model_id = "stuchapin/SenseNova-Loop-9B-Preview-W4A16-AutoRound"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype="auto",
    trust_remote_code=True,
)
Downloads last month
17
Safetensors
Model size
3B params
Tensor type
F32
·
I32
·
BF16
·
F16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for stuchapin/SenseNova-Loop-9B-Preview-W4A16-AutoRound

Quantized
(13)
this model