Instructions to use sensenova/SenseNova-Loop-9B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use sensenova/SenseNova-Loop-9B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="sensenova/SenseNova-Loop-9B", trust_remote_code=True) messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("sensenova/SenseNova-Loop-9B", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use sensenova/SenseNova-Loop-9B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "sensenova/SenseNova-Loop-9B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "sensenova/SenseNova-Loop-9B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/sensenova/SenseNova-Loop-9B
- SGLang
How to use sensenova/SenseNova-Loop-9B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "sensenova/SenseNova-Loop-9B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "sensenova/SenseNova-Loop-9B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "sensenova/SenseNova-Loop-9B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "sensenova/SenseNova-Loop-9B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use sensenova/SenseNova-Loop-9B with Docker Model Runner:
docker model run hf.co/sensenova/SenseNova-Loop-9B
Recommendation for 8gb vram?
Amazing, can't wait to use it in llama.cpp
Any recommendations for 8gb vram perhaps, or is 12gb vram due to looping really needed?
Thanks for your interest! We expect to release the quantized versions of the model along with the corresponding llama.cpp support tomorrow.
We're also exploring KV cache rewriting strategies for the looped architecture to further reduce inference memory requirements, with the goal of bringing VRAM usage below 8 GB. We'll share more updates soon!
Thanks for your interest and patience! The GGUF version is now available for llama.cpp: https://huggingface.co/sensenova/SenseNova-Loop-9B-GGUF/
We've also run some preliminary VRAM usage tests with different context lengths and KV cache quantization settings:
| Context length | 8-bit KV cache | 4-bit KV cache |
|---|---|---|
| 64K | ~10 GiB | ~9 GiB |
| 128K | ~12 GiB | ~10 GiB |
| 256K | ~15 GiB | ~12 GiB |
Based on these results, 8 GB VRAM is still a bit tight for the tested long-context configurations. We're continuing to explore KV cache optimizations to further reduce memory usage. Stay tuned!