HelloSun commited on
Commit
8466782
·
1 Parent(s): 26151cf

stage3: record BF16 conv GEMM negative result (slower than the F16 path)

Browse files

- tried converting the VAE conv kernel to BF16 to reach llamafile's 32-wide
AVX512-BF16 tinyBLAS: im2col 851->1323 ms, GEMM 2196->2481 ms, VAE 3.8->4.6 s
- reverted; the VAE weights are already F16 and the generic F16 vec-dot GEMM is
the faster path on this host

Files changed (1) hide show
  1. stages/stage3_vae_conv/README.md +17 -1
stages/stage3_vae_conv/README.md CHANGED
@@ -40,7 +40,23 @@ The same patch adds two diagnostic env knobs (no effect when unset):
40
  `GGML_CONV_PROFILE=1` (phase timing) and `GGML_CONV_PATCH_MB=<n>` (cap the im2col
41
  batch buffer; capping to L3 size made no difference, so it is left unlimited).
42
 
43
- ## 3. Remaining VAE cost
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
44
 
45
  The GEMM is now the largest phase (~2.2 s, ~0.7 TFLOPS effective) and runs through the
46
  F32 llamafile sgemm. im2col is ~0.9 s. OpenVINO still decodes this VAE in ~0.3 s
 
40
  `GGML_CONV_PROFILE=1` (phase timing) and `GGML_CONV_PATCH_MB=<n>` (cap the im2col
41
  batch buffer; capping to L3 size made no difference, so it is left unlimited).
42
 
43
+ ## 3. Negative result: BF16 conv GEMM
44
+
45
+ The VAE conv weights are loaded as **F16**, so `ggml_call_mul_mat` takes the generic
46
+ F16 vec-dot path. llamafile has a 32-wide AVX512-BF16 tinyBLAS path, so a BF16 variant
47
+ was tried (convert the kernel to BF16 once per call, let im2col emit BF16, call the GEMM
48
+ as BF16):
49
+
50
+ | conv GEMM dtype | im2col | GEMM | VAE decode | pixel diff vs F16 |
51
+ |---|---|---|---|---|
52
+ | F16 (current) | 851 ms | 2196 ms | ~3.8 s | ref |
53
+ | BF16 (tinyBLAS) | 1323 ms | 2481 ms | ~4.6 s | 0.10 / 11 |
54
+
55
+ BF16 is **slower** here, both in im2col (per-run bf16 conversion) and in the GEMM, so it
56
+ is not used. The BF16 `tinyBLAS` path is not a win on this machine; the F16 generic path
57
+ is already competitive. Not shipped.
58
+
59
+ ## 4. Remaining VAE cost
60
 
61
  The GEMM is now the largest phase (~2.2 s, ~0.7 TFLOPS effective) and runs through the
62
  F32 llamafile sgemm. im2col is ~0.9 s. OpenVINO still decodes this VAE in ~0.3 s