BF16 x FP8_E5M2 vs KJ's FP8_E5M2

#1
by NeoHuggingF - opened

Hey, there.

Thank you for your work and the model weights.

I did some tests trying to find out which model would give the result closets to the BF16 model from Comfy-Org repo.

Setup
ComfyUI 0.10.0 running with just --low-vram toggle (latest version as of today)
32GB DDR5 + Nvidia 3050 6GB (this is why I chose fp8_e5m2)
Each image was generated without clearing the cache/memory, so the same CLIP embedding is used

Results

  1. Closest to BF16 is KJ's FP8_E5M2 with weight_dtype=default (at least regarding the motorcycle)
  2. Using KJ's FP8 yields different results if weight_dtype is set to default or fp8_e5m2 , while there is no change doing the same with this model
  3. Using BF16 with wight_dtype=fp8_e5m2 generates an image that is very similar to the one generated by this model (FP8_E5M2)
  4. Generation time was more or less the same for all combinations, around 96s per image
  5. I always thought "superbike" was used for powerful motorcycles, not bicycles (without a motor) πŸ˜…

bf16 - default
bf16_default

bf16 - fp8_e5m2
bf16_e5m2

fp8_e5m2 - default
fp8_e5m2_default

fp8_e5m2 - fp8_e5m2
fp8_e5m2_e5m2

KJ's fp8_e5m2 - default
fp8_e5m2_kj_default

KJ's fp8_e5m2 - fp8_e5m2
fp8_e5m2_kj_e5m2

Sign up or log in to comment