as far as I'm aware, NVFP4 doesn't perform nearly as well if the model wasn't trained in NVFP4 sadly
it's possible that if a model were to be compressed using the right amount of compute to nvfp4 it could then be converted to GGUF format, but I'm not sure there's actually any benefit as of yet
if you have any counter information I'd love to see it !
I tested Qwen3.6-27B-NVFP4-Q8_0.gguf vs Qwen3.6-27B-Q8_0(3).gguf using a modded RTX 2080ti 22GB, and its about 6x faster (I know, its the size not the architecture, but that is EXACTLY the point). For those using blackwell GPUs, they would feel a much better benefit. But my point goes beyond speed. We have to be fair when comparing NVFP4 vs GGUF "traditional quants" (Q4_K_M, etc). NVFP4 has the best efficiency-to-accuracy ratio. The Accuracy Drop for a NVFP4 is >1%, something that can only be achieved by a Q6_K_M or more clearly a Q8_0. The difference in size for small models is meaningless, but for sizes above 30GB, this could be the difference from running a very accurate model 100% in GPU, or running a subpar quantization AND partially in CPU. Once the model is converted to NVFP4 (https://github.com/vllm-project/llm-compressor), we do not "convert" to GGUF, we just "store" (containerization) it in GGUF using "convert_hf_to_gguf.py" of a compiled llama.cpp, so llama.cpp users (as me) can run it directly. And since llama.cpp now support NVFP4 (since release B9080: https://github.com/ggml-org/llama.cpp/releases/tag/b9080), having a NVFP4 option would be a great adition, specially for the bigger models (minimax, mimo, stepfun, etc).
