INT8 Convrot

#1
by Talanca - opened

Is an INT8 Convrot version possible?
I would do the quantization myself, but my quant node doesn't have the MOSS profile to select the proper quantization.

uh, bruh, if you want to use int8 convrot it will just works isn't it? bcuz int8 convrot make matmul happen in int8, and activation natively sit in fp32?
fp16 fix ckpt is for fp16 fix usage somehow

Is an INT8 Convrot version possible?
I would do the quantization myself, but my quant node doesn't have the MOSS profile to select the proper quantization.

https://huggingface.co/mkrgg/MOSS-Image-Base-INT8-FP16-Turing

uh, bruh, if you want to use int8 convrot it will just works isn't it? bcuz int8 convrot make matmul happen in int8, and activation natively sit in fp32?
fp16 fix ckpt is for fp16 fix usage somehow

On turing gpus, bf16 (including activations and everything else) fallback to fp32, slowing down the inference.

on turing GPUs, you can just use fp16 (and the fp16fix ckpt)

I know, but for fp16 weights you need more vram, the majority of cards from rtx 2000 series have 8gb, only the 2080ti have 11gb and some 2060 have 12gb

Then I can't understand what you are talking at all
why not just use int8convrot?
https://huggingface.co/KBlueLeaf/MOSS-Image-heal/blob/main/comfyui/moss_image_heal_int8_convrot.safetensors
the dir have it

The only problem is that version of int8 don't have fp16, just check the tensors plz :(

image

image

you can direclty cast them to fp16

yep, I understand what you mean. Changing how comfyui load the model we can cast them to fp16 but idk if i can do that in another platform/system and having a "native" safetensors is better for people with no knowledge, is only that :D

Sign up or log in to comment