Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
inflatebot 
posted an update about 9 hours ago
Post
175
i am BEGGING you people to stop shoving everything in the GGUF Hole and ship w4a8/w6a8 models for ComfyUI. Comfy has bespoke kernels for these! They're fast everywhere!
4 bits should be treated as the floor for DiT weights, if you really need 3 bits or lower you should just use a different model!

Also stop shilling nvfp4 you animals!!!
inflatebot

Thank you for coming to my TED talk
dipankarsarkar

The 4-bit floor for DiT weights is the claim I'd want a number on.

Where does 3-bit break for you: a metric (FID, or CLIP score against the bf16 image at a fixed seed), or by eye?

The a8 half worries me more than the w4.
DiT activations shift a lot across denoising steps. PTQ4DiT and Q-DiT both build that timestep variation into their activation quantization for exactly this reason.

Do Comfy's w4a8 kernels use one static activation scale for every step, or a dynamic one?

·
inflatebot

The 4-bit floor for DiT weights is the claim I'd want a number on.

I honestly consider 4 bits to be the floor for most things. For image models, especially step-distilled ones, it's possible lower precisions work decently well. I'm a bit privileged in that I've never had to bother with it before. Where low precision really starts to sting is with video models. (Which is a shame because that's exactly where you'd want it to work well!)

Comfy Kitchen's AsymInt8 layouts use fp8 group scales and a per-tensor Lloyd-Max codebook to dequantize to grouped INT8, which is accelerated on everything going back to Turing, where NVFP4 is only fast on Blackwell.

The a8 half worries me more than the w4... Do Comfy's w4a8 kernels use one static activation scale for every step, or a dynamic one?

It's handled dynamically in the kernel, yes. It's not calibrated, but neither is NVFP4, and the point is that it's easy to work with and fast on the cards people actually own.

In my personal experience, NVFP4 makes worse videos than w4a8 and takes way longer to do it. int8_convrot was rapidly taken up by the ComfyUI crowd for good reason, it's just superior to fp8_scaled in every way unless you're on a truly ancient GPU.

There's almost definitely better quantization strategies out there (I've seen some uptake of GPTQ which is interesting!) but AsymInt8 is some excellent work for what ComfyUI in particular needs.

Using GGUF in Comfy, on the other hand, bypasses all of that kernel work, doesn't work with Aimdo (at least not when I last tried it), complicates workflows and is overall just a worse user experience. I'm not sure anything really stops anybody from shipping AsymInt8-type layouts in the GGUF container format, but nobody does, so it's moot, and it wouldn't run through Comfy Kitchen's kernel.

I could probably get some updated numbers, though. I just remember using GGUF for LTX-2.3, being aggravated by the fact that it turned my PC into sludge, and then getting the FP8 on a lark and being like "what the hell, it actually works properly now." Maybe the situation's improved with GGUF since. I just don't see the point.
dipankarsarkar

Dynamic scales settle my worry. Timestep drift is a static-scale problem, and you don't have one.

I suspect your NVFP4 vs w4a8 gap is the activation half, not the weights.
The usual NVFP4 recipe runs W4A4: activations drop to FP4 too, in 16-value blocks with one FP8 scale each.
So both sides hold weights at 4 bits. The real comparison is a8 vs a4.

If you pull fresh numbers, a third arm would settle it: NVFP4 weights with bf16 activations, same seed, same clip.
If that one looks like w4a8, the weight format was never the problem.

When NVFP4 wrecks a video, is it flicker between frames, or lost detail inside each frame?
ItzPingCat

tldr; ONLY USE GGUF on llama.cpp and NOT comfyui
In this post