Recommendation for 8gb vram?

#5
by idkgoogleit - opened

Amazing, can't wait to use it in llama.cpp

Any recommendations for 8gb vram perhaps, or is 12gb vram due to looping really needed?

idkgoogleit changed discussion title from Recommendation for ?8gb vram? to Recommendation for 8gb vram?

Thanks for your interest! We expect to release the quantized versions of the model along with the corresponding llama.cpp support tomorrow.

We're also exploring KV cache rewriting strategies for the looped architecture to further reduce inference memory requirements, with the goal of bringing VRAM usage below 8 GB. We'll share more updates soon!

Thanks for your interest and patience! The GGUF version is now available for llama.cpp: https://huggingface.co/sensenova/SenseNova-Loop-9B-GGUF/

We've also run some preliminary VRAM usage tests with different context lengths and KV cache quantization settings:

Context length 8-bit KV cache 4-bit KV cache
64K ~10 GiB ~9 GiB
128K ~12 GiB ~10 GiB
256K ~15 GiB ~12 GiB

Based on these results, 8 GB VRAM is still a bit tight for the tested long-context configurations. We're continuing to explore KV cache optimizations to further reduce memory usage. Stay tuned!

Sign up or log in to comment