Instructions to use letechlead/SenseNova-Loop-9B-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use letechlead/SenseNova-Loop-9B-FP8 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="letechlead/SenseNova-Loop-9B-FP8", trust_remote_code=True) messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("letechlead/SenseNova-Loop-9B-FP8", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use letechlead/SenseNova-Loop-9B-FP8 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "letechlead/SenseNova-Loop-9B-FP8" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "letechlead/SenseNova-Loop-9B-FP8", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/letechlead/SenseNova-Loop-9B-FP8
- SGLang
How to use letechlead/SenseNova-Loop-9B-FP8 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "letechlead/SenseNova-Loop-9B-FP8" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "letechlead/SenseNova-Loop-9B-FP8", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "letechlead/SenseNova-Loop-9B-FP8" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "letechlead/SenseNova-Loop-9B-FP8", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use letechlead/SenseNova-Loop-9B-FP8 with Docker Model Runner:
docker model run hf.co/letechlead/SenseNova-Loop-9B-FP8
SenseNova-Loop-9B-FP8
Block-wise FP8 weight quantization of sensenova/SenseNova-Loop-9B,
the 9B-parameter looped multimodal decoder (32 physical layers executed as 56 layer passes, two-stream mHC,
hybrid linear/full attention). The checkpoint layout matches the official Qwen FP8 releases
(quant_method: fp8, block-wise weight_scale_inv), so FP8-aware engines can load it without renames or extra files.
Quantization recipe
- Weights: FP8
e4m3fn, one BF16 scale per 128x128 block (<name>.weight_scale_inv, shape[ceil(out/128), ceil(in/128)]), per-block absmax scaling. Weight-only, no calibration data. quantization_config:quant_method: fp8,fmt: e4m3,weight_block_size: [128, 128],activation_scheme: dynamic— same schema asQwen/Qwen3-32B-FP8.- Quantized to FP8 (344 tensors):
- Decoder full-attention
q/k/v/o_proj - Gated-DeltaNet linear-attention
in_proj_a/b/qkv/zandout_proj - MLP
gate/up/down_proj - Vision blocks 3-26 (
attn.qkv,attn.proj,mlp.linear_fc1/2)
- Decoder full-attention
- Kept at original precision:
embed_tokens,lm_head, all norms (BF16)conv1d,A_log,dt_bias(BF16)- mHC routing
mhc_attn/mhc_ffnandloop_gate(FP32, as in the original) - Vision blocks 0-2, merger, patch_embed, pos_embed (BF16) — same ignore policy as official
Qwen3-VL-*-FP8
- Size: 18.83 GB (BF16) -> 11.55 GB (-38.7%).
Verification
Single RTX 4090, transformers 5.18.0 / torch 2.11.0+cu130 (the upstream-pinned environment). BF16 original vs dequantized FP8, both loaded through the identical transformers code path:
- Weight error (dequantized FP8 vs original BF16): mean relative L2 2.64%, max 2.66% across the 344 quantized tensors (expected magnitude for e4m3 block quantization).
- Greedy generation, 4 prompts (math, code, Chinese, reasoning), thinking off:
- math: 3/3 tokens identical ("91")
- code: 160/160 tokens identical
- Chinese: identical for 20 tokens, then an equivalent paraphrase
- reasoning: identical for 59 tokens, then equivalent steps; both end with the correct answer (9)
- Fixed-input logits: cosine similarity 0.9996 (last position) / 0.9998 (mean over positions), top-1 agreement.
- Multimodal smoke test (synthetic image with shapes + text): both models correctly describe the red circle, blue square and "HELLO FP8" text, with near-identical wording.
Usage
The checkpoint targets FP8-aware inference engines. SenseNova-Loop engine support is available via vLLM PR #64832 and SGLang PR #47281; FP8 kernel coverage for this architecture in those branches is not yet verified upstream — BF16 is the documented path there.
For transformers (the custom modeling code loads BF16), dequantize at load time with the
verified helper shipped in this repo:
hf download letechlead/SenseNova-Loop-9B-FP8 --local-dir SenseNova-Loop-9B-FP8
python dequant_to_bf16.py --src SenseNova-Loop-9B-FP8 --dst SenseNova-Loop-9B-FP8-bf16
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "SenseNova-Loop-9B-FP8-bf16"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id, trust_remote_code=True,
dtype=torch.bfloat16, device_map={"": "cuda:0"}).eval()
The dequantized copy is numerically the FP8 model (same weights, BF16 container) and needs ~19 GB of disk plus the usual BF16 runtime memory. For native BF16 or 4-bit deployment use the upstream releases: sensenova/SenseNova-Loop-9B and the official AWQ variant.
License
Apache-2.0, inherited from the original model. Original weights (c) SenseNova; quantization by LeTechLead.
- Downloads last month
- 52
Model tree for letechlead/SenseNova-Loop-9B-FP8
Base model
sensenova/SenseNova-Loop-9B