Compare fp16 without the fix, fp16 with the fix, bf16 and fp32 in every panel

.gitattributes

@@ -158,3 +158,24 @@ images/sft_c_sweep_p0_1.webp filter=lfs diff=lfs merge=lfs -text
 images/sft_plain_fp16_p1.webp filter=lfs diff=lfs merge=lfs -text
 images/sft_stream_per_block.webp filter=lfs diff=lfs merge=lfs -text
 images/base_fp16_break_2560.webp filter=lfs diff=lfs merge=lfs -text
+images/gallery/base_comfyui_lora0_p1_s11.webp filter=lfs diff=lfs merge=lfs -text
+images/gallery/base_comfyui_lora1_p0_s11.webp filter=lfs diff=lfs merge=lfs -text
+images/gallery/base_comfyui_p1_s11.webp filter=lfs diff=lfs merge=lfs -text
+images/gallery/base_comfyui_p3_s11.webp filter=lfs diff=lfs merge=lfs -text
+images/gallery/base_diffusers_p0_s12.webp filter=lfs diff=lfs merge=lfs -text
+images/gallery/base_diffusers_p1_s12.webp filter=lfs diff=lfs merge=lfs -text
+images/gallery/base_diffusers_p3_s12.webp filter=lfs diff=lfs merge=lfs -text
+images/gallery/base_diffusers_p6_s11.webp filter=lfs diff=lfs merge=lfs -text
+images/gallery/base_kohakumoss_lora0_p1_s11.webp filter=lfs diff=lfs merge=lfs -text
+images/gallery/base_kohakumoss_p0_s12.webp filter=lfs diff=lfs merge=lfs -text
+images/gallery/base_kohakumoss_p1_s12.webp filter=lfs diff=lfs merge=lfs -text
+images/gallery/base_kohakumoss_p3_s11.webp filter=lfs diff=lfs merge=lfs -text
+images/gallery/base_kohakumoss_p3_s12.webp filter=lfs diff=lfs merge=lfs -text
+images/gallery/base_kohakumoss_p6_s11.webp filter=lfs diff=lfs merge=lfs -text
+images/gallery/sft_comfyui_lora0_p1_s11.webp filter=lfs diff=lfs merge=lfs -text
+images/gallery/sft_comfyui_lora1_p4_s11.webp filter=lfs diff=lfs merge=lfs -text
+images/gallery/sft_comfyui_p1_s11.webp filter=lfs diff=lfs merge=lfs -text
+images/gallery/sft_diffusers_p0_s12.webp filter=lfs diff=lfs merge=lfs -text
+images/gallery/sft_diffusers_p4_s12.webp filter=lfs diff=lfs merge=lfs -text
+images/gallery/sft_kohakumoss_p4_s12.webp filter=lfs diff=lfs merge=lfs -text
+images/sft_plain_fp16_p0.webp filter=lfs diff=lfs merge=lfs -text

README.md

@@ -60,7 +60,9 @@ are the official ones.
 | `comfyui/moss_image_sft_fp16_fixed.safetensors` | ComfyUI single file (fp16) |
 | `reports/{base,sft}_fp16_fix.json` | every number on this page, per image |
 
-<img src="images/gallery/base_comfyui_p1_s11.webp" alt="fp16-fixed vs official bf16 vs fp32, ComfyUI">
+<img src="images/gallery/base_comfyui_p1_s11.webp" alt="fp16 without the fix | fp16 with the fix | bf16 | fp32, ComfyUI">
+
+fp16 without the fix | fp16 with the fix | bf16 | fp32 (ComfyUI, 1024^2, seed 11).
 
 ## Why: the stream sits at 95% of the fp16 limit
 
@@ -79,22 +81,25 @@ block.
 
 ![plain fp16 and the margin per block, Base](images/base_plain_fp16.webp)
 
-<img src="images/base_plain_fp16_p0.webp" alt="official weights fp16 / bf16 / fixed fp16">
+<img src="images/base_plain_fp16_p0.webp" alt="fp16 without the fix | fp16 with the fix | bf16 | fp32, 1024^2">
+
+fp16 without the fix | fp16 with the fix | bf16 | fp32 (KohakuMoss, 1024^2, prompt 0).
 
 **Where plain fp16 breaks.** ComfyUI with `--fp16-unet` on the official Base file, 50
 steps, CFG 3, euler / simple: all 8 prompts at 1024^2 finish (stream max 62,080), and so
 does 2048^2 (64,096, 2.2% below the limit). **2560^2 is the first size that gives a
 black image**: on step 29 of 50 the stream reaches 65,504 at block 26, every later
-forward is NaN, and the image is black. The same run in fp32 peaks at 67,048 (a single
-fp32 forward at sigma 0.7: 67,681), 2.4-3.3% over the fp16 limit. The fixed weights peak
-at 4,188 (x 16 = 67,008, 15.6x headroom):
+forward is NaN, and the image is black. KohakuMoss in plain fp16 goes non-finite at the
+same step and block. The same run in fp32 peaks at 67,048 (a single fp32 forward at sigma
+0.7: 67,681), 2.4-3.3% over the fp16 limit; bf16 peaks at 67,072 and stays
+finite. The fixed weights peak at 4,188 (x 16 = 67,008, 15.6x headroom):
 
 <img src="images/base_fp16_break_2560.webp" alt="2560x2560: fp16 without the fix (NaN) | fp16 with the fix | bf16 | fp32">
 
 fp16 without the fix | fp16 with the fix | bf16 | fp32 (ComfyUI, 2560x2560, prompt 0
-"tabby cat", seed 11; the bf16-compute image is still rendering). In single forwards over 7 sigmas, 2560^2, 2048x3072, 3072x2048
-and 3072^2 overflow; 1536^2, 2048^2, 768x2048 and 1024x4096 do not. A 226-token prompt
-moves the stream by under 1%.
+"tabby cat", seed 11; `LPIPS*` is the distance to fp32). In single forwards over 7
+sigmas, 2560^2, 2048x3072, 3072x2048 and 3072^2 overflow; 1536^2, 2048^2, 768x2048 and
+1024x4096 do not. A 226-token prompt moves the stream by under 1%.
 
 <details><summary>SFT charts</summary>
 
@@ -165,20 +170,29 @@ steps) against fp32.
 1024^2, 50 steps, CFG 3, 8 prompts x seeds 11 / 12 (16 images), each runtime against its
 own fp32 run. LPIPS (AlexNet) / PSNR, lower / higher is closer.
 
-| runtime | variant | fp16-fixed | official bf16 | fixed vs bf16 |
-| --- | --- | --- | --- | --- |
-| ComfyUI | Base | **0.0062** / 44.8 dB | 0.0501 / 29.3 dB | 0.047 |
-| ComfyUI | SFT | **0.0042** / 43.2 dB | 0.0413 / 29.3 dB | 0.042 |
-| diffusers | Base | **0.0111** / 42.8 dB | 0.0900 / 26.8 dB | 0.089 |
-| diffusers | SFT | **0.0094** / 42.2 dB | 0.0655 / 28.1 dB | 0.065 |
-| KohakuMoss | Base | **0.0145** / 42.7 dB | 0.0934 / 26.7 dB | 0.090 |
-| KohakuMoss | SFT | **0.0121** / 42.9 dB | 0.0683 / 27.4 dB | 0.072 |
+| runtime | variant | fp16, no fix | fp16-fixed | official bf16 | fixed vs bf16 |
+| --- | --- | --- | --- | --- | --- |
+| ComfyUI | Base | 0.0062 / 44.7 dB | **0.0062** / 44.8 dB | 0.0501 / 29.3 dB | 0.047 |
+| ComfyUI | SFT | 0.0035 / 42.6 dB | **0.0042** / 43.2 dB | 0.0413 / 29.3 dB | 0.042 |
+| diffusers | Base | 0.0202 / 36.9 dB | **0.0111** / 42.8 dB | 0.0900 / 26.8 dB | 0.089 |
+| diffusers | SFT | 0.0151 / 37.0 dB | **0.0094** / 42.2 dB | 0.0655 / 28.1 dB | 0.065 |
+| KohakuMoss | Base | 0.0171 / 42.0 dB | **0.0145** / 42.7 dB | 0.0934 / 26.7 dB | 0.090 |
+| KohakuMoss | SFT | 0.0053 / 42.2 dB | **0.0121** / 42.9 dB | 0.0683 / 27.4 dB | 0.072 |
 
 fp16 carries 3 more mantissa bits than bf16; with the stream in range that is what the
 fixed weights give back. "fixed vs bf16" is the size of bf16's own rounding drift, not an
-error of the fix. No 1024^2 run produced a non-finite latent or image.
-
-Each panel: fp16-fixed | official bf16 | fp32 (ComfyUI, seed 11).
+error of the fix. No 1024^2 run produced a non-finite latent or image: at this size the
+official weights still fit fp16 and plain fp16 is about as close to fp32 as the fixed
+weights. The difference is headroom (1.05x vs 16.9x), which is what the 2560^2 example
+above uses up.
+
+"fp16, no fix" ran on a V100, the other columns on an RTX 5090, with each seed's initial
+noise as the RTX 5090 draws it. The same plain-fp16 input on the two GPUs differs by LPIPS
+0.039 / 0.012 (ComfyUI, Base / SFT), 0.015 / 0.025 (diffusers), 0.040 / 0.008
+(KohakuMoss) over prompt 0 and both seeds; that is part of this column's distance. In
+fp32 the two GPUs agree to 0.0003.
+
+Each panel: fp16 without the fix | fp16 with the fix | bf16 | fp32 (ComfyUI, seed 11).
 `LPIPS*` is the distance to the fp32 panel.
 
 ### Base
@@ -224,7 +238,8 @@ Each panel: fp16-fixed | official bf16 | fp32 (ComfyUI, seed 11).
 </div>
 
 Every runtime, both seeds and the adapter panels are in `images/gallery/`
-(`<variant>_<runtime>_p<prompt>_s<seed>.webp`).
+(`<variant>_<runtime>_p<prompt>_s<seed>.webp`, `<variant>_<runtime>_lora<i>_p<prompt>_s11.webp`),
+all in the same four columns.
 
 ## Whole pipeline in fp16
 
@@ -239,33 +254,36 @@ transformer + fp16 VAE) is what the diffusers snippet below runs.
 ## Adapters
 
 A LoRA (rank 8) and a LoKr (factor 16), trained on the official Base weights in
-KohakuMoss, applied as-is to the fixed weights. Distance between fixed + adapter and
-official bf16 + adapter, next to the adapter's own effect:
-
-| runtime | variant | adapter | fixed + adapter vs bf16 + adapter | without adapter | adapter effect |
-| --- | --- | --- | --- | --- | --- |
-| ComfyUI | Base | LoRA | 0.056 | 0.047 | 0.287 |
-| ComfyUI | Base | LoKr | 0.064 | 0.047 | 0.389 |
-| ComfyUI | SFT | LoRA | 0.039 | 0.042 | 0.284 |
-| ComfyUI | SFT | LoKr | 0.039 | 0.042 | 0.395 |
-| KohakuMoss | Base | LoRA | 0.105 | 0.090 | 0.316 |
-| KohakuMoss | Base | LoKr | 0.053 | 0.090 | 0.406 |
-| KohakuMoss | SFT | LoRA | 0.053 | 0.072 | 0.288 |
-| KohakuMoss | SFT | LoKr | 0.042 | 0.072 | 0.417 |
-
-(LPIPS, 16 images each.) The adapter changes the image 3-10x more
-than the fix does, and fixed + adapter stays about as close to bf16 + adapter as the two
-bases are without it: it transfers.
+KohakuMoss, applied as-is to the fixed weights. Distance of each base + adapter to fp32 +
+adapter, then between fixed + adapter and official bf16 + adapter, next to the adapter's
+own effect:
+
+| runtime | variant | adapter | fp16, no fix | fp16-fixed | bf16 | fixed + adapter vs bf16 + adapter | without adapter | adapter effect |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| ComfyUI | Base | LoRA | 0.003 | 0.006 | 0.068 | 0.056 | 0.047 | 0.287 |
+| ComfyUI | Base | LoKr | 0.011 | 0.005 | 0.049 | 0.064 | 0.047 | 0.389 |
+| ComfyUI | SFT | LoRA | 0.001 | 0.001 | 0.034 | 0.039 | 0.042 | 0.284 |
+| ComfyUI | SFT | LoKr | 0.001 | 0.001 | 0.024 | 0.039 | 0.042 | 0.395 |
+| KohakuMoss | Base | LoRA | 0.006 | 0.008 | 0.120 | 0.105 | 0.090 | 0.316 |
+| KohakuMoss | Base | LoKr | 0.004 | 0.004 | 0.046 | 0.053 | 0.090 | 0.406 |
+| KohakuMoss | SFT | LoRA | 0.003 | 0.002 | 0.038 | 0.053 | 0.072 | 0.288 |
+| KohakuMoss | SFT | LoKr | 0.002 | 0.001 | 0.037 | 0.042 | 0.072 | 0.417 |
+
+(LPIPS. The first three columns: 8 images each, seed 11, against fp32 + adapter run on a
+V100, with plain fp16 + adapter also on the V100; the last three: 16 images each, RTX
+5090.) The adapter changes the image 3-10x more than the fix does, fixed + adapter is as
+close to fp32 + adapter as plain fp16 is at 1024^2, and stays about as close to bf16 +
+adapter as the two bases are without it: it transfers.
 
 <div class="mf-gal">
 <input type="radio" name="gl" id="l0" checked><label for="l0">Base LoRA</label>
 <input type="radio" name="gl" id="l1"><label for="l1">Base LoKr</label>
 <input type="radio" name="gl" id="l2"><label for="l2">SFT LoRA</label>
 <input type="radio" name="gl" id="l3"><label for="l3">SFT LoKr</label>
-<div class="mf-panel" id="pl0"><img src="images/gallery/base_comfyui_lora0_p3_s11.webp" alt=""><p>fp16-fixed + adapter | official bf16 + adapter | official bf16 without it (ComfyUI)</p></div>
-<div class="mf-panel" id="pl1"><img src="images/gallery/base_comfyui_lora1_p3_s11.webp" alt=""><p>fp16-fixed + adapter | official bf16 + adapter | official bf16 without it (ComfyUI)</p></div>
-<div class="mf-panel" id="pl2"><img src="images/gallery/sft_comfyui_lora0_p3_s11.webp" alt=""><p>fp16-fixed + adapter | official bf16 + adapter | official bf16 without it (ComfyUI)</p></div>
-<div class="mf-panel" id="pl3"><img src="images/gallery/sft_comfyui_lora1_p3_s11.webp" alt=""><p>fp16-fixed + adapter | official bf16 + adapter | official bf16 without it (ComfyUI)</p></div>
+<div class="mf-panel" id="pl0"><img src="images/gallery/base_comfyui_lora0_p3_s11.webp" alt=""><p>fp16 without the fix + adapter | fp16 with the fix + adapter | bf16 + adapter | fp32 + adapter (ComfyUI)</p></div>
+<div class="mf-panel" id="pl1"><img src="images/gallery/base_comfyui_lora1_p3_s11.webp" alt=""><p>fp16 without the fix + adapter | fp16 with the fix + adapter | bf16 + adapter | fp32 + adapter (ComfyUI)</p></div>
+<div class="mf-panel" id="pl2"><img src="images/gallery/sft_comfyui_lora0_p3_s11.webp" alt=""><p>fp16 without the fix + adapter | fp16 with the fix + adapter | bf16 + adapter | fp32 + adapter (ComfyUI)</p></div>
+<div class="mf-panel" id="pl3"><img src="images/gallery/sft_comfyui_lora1_p3_s11.webp" alt=""><p>fp16 without the fix + adapter | fp16 with the fix + adapter | bf16 + adapter | fp32 + adapter (ComfyUI)</p></div>
 </div>
 
 Adapters on the joint blocks' q / k / v / o / `ffn_up_gate` / `ffn_down` (the usual
@@ -329,7 +347,12 @@ See [KohakuMoss](https://github.com/KohakuBlueleaf/KohakuMossTrainer).
 ## Notes
 
 - Measured on an RTX 5090 (fp16 and bf16 matmuls with fp32 accumulation); the fp32
-  reference is the official weights in fp32. The 2560^2 example ran in ComfyUI on a V100.
+  reference is the official weights in fp32. The 2560^2 example, the "fp16, no fix"
+  column and the fp32 + adapter references ran on a V100 (bf16 there is emulated by torch;
+  ComfyUI's own default on that GPU casts bf16 weights to fp32, so the 2560^2 bf16 run
+  disables that cast). KohakuMoss and diffusers draw their initial noise with torch's CUDA
+  generator, which gives different numbers on different GPUs; the V100 runs use the RTX
+  5090's noise for each seed.
 - The headroom figures cover the prompts, resolutions and sigmas above, and 2560^2 for
   the fixed Base weights (15.6x). Larger images, very long prompts and finetuned weights
   were not measured.