Any advice for using llm-compressor? Trying to get a W4A8
I want to try this with a hopper card.
I have 2xH100 and about 400 GB of RAM available. But I just keep hitting OOM and other errors. Would you all be willing to share you llm compressor formula? Thanks!
Since Hopper GPUs do not support FP4 activations, the model will likely not run with FP4 activation quantization.
A more practical option would be to reduce max_len to lower the KV cache memory usage, or load the KV cache in FP8.
Alternatively, I strongly recommend trying one of the more aggressively optimized versions below:
https://huggingface.co/nota-ai/Solar-Open2-250B-Nota-INT4-GlobalPruned
https://huggingface.co/nota-ai/Solar-Open2-250B-Nota-NVFP4-GlobalPruned
Thanks!
Im running the nvfp4 model on 4x h100 presently. It works fine, using the marlin kernel which basically turns into W4A16
But that's why I was curious about W4A8 which would utilize the FP8 tensor cores in my H100 gpus.
However after about 4 hours of trying I had to give up. From the model card and files it appears you all used llm-compressor to create your quants. I was hoping you all could share your llm-compressor recipe!
That sounds like a promising approach and a worthwhile direction to explore. We would also be interested in investigating how W4A8 support could be implemented effectively.
To clarify, we did not use llm-compressor to perform the NVFP4 quantization itself. We quantized the model using our own quantization pipeline and then packed and serialized the resulting NVFP4 weights in the compressed-tensors format supported by vLLM.
Therefore, we do not have an llm-compressor quantization recipe to share. We only used the compressed-tensors format for model storage and loading, rather than using llm-compressor as the actual quantization method.
Thank you for the guidance! That makes sense!
Ill continue to explore options for W4A8 quants for my hopper cards.