stage3: record BF16 conv GEMM negative result (slower than the F16 path)
Browse files- tried converting the VAE conv kernel to BF16 to reach llamafile's 32-wide
AVX512-BF16 tinyBLAS: im2col 851->1323 ms, GEMM 2196->2481 ms, VAE 3.8->4.6 s
- reverted; the VAE weights are already F16 and the generic F16 vec-dot GEMM is
the faster path on this host
stages/stage3_vae_conv/README.md
CHANGED
|
@@ -40,7 +40,23 @@ The same patch adds two diagnostic env knobs (no effect when unset):
|
|
| 40 |
`GGML_CONV_PROFILE=1` (phase timing) and `GGML_CONV_PATCH_MB=<n>` (cap the im2col
|
| 41 |
batch buffer; capping to L3 size made no difference, so it is left unlimited).
|
| 42 |
|
| 43 |
-
## 3.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 44 |
|
| 45 |
The GEMM is now the largest phase (~2.2 s, ~0.7 TFLOPS effective) and runs through the
|
| 46 |
F32 llamafile sgemm. im2col is ~0.9 s. OpenVINO still decodes this VAE in ~0.3 s
|
|
|
|
| 40 |
`GGML_CONV_PROFILE=1` (phase timing) and `GGML_CONV_PATCH_MB=<n>` (cap the im2col
|
| 41 |
batch buffer; capping to L3 size made no difference, so it is left unlimited).
|
| 42 |
|
| 43 |
+
## 3. Negative result: BF16 conv GEMM
|
| 44 |
+
|
| 45 |
+
The VAE conv weights are loaded as **F16**, so `ggml_call_mul_mat` takes the generic
|
| 46 |
+
F16 vec-dot path. llamafile has a 32-wide AVX512-BF16 tinyBLAS path, so a BF16 variant
|
| 47 |
+
was tried (convert the kernel to BF16 once per call, let im2col emit BF16, call the GEMM
|
| 48 |
+
as BF16):
|
| 49 |
+
|
| 50 |
+
| conv GEMM dtype | im2col | GEMM | VAE decode | pixel diff vs F16 |
|
| 51 |
+
|---|---|---|---|---|
|
| 52 |
+
| F16 (current) | 851 ms | 2196 ms | ~3.8 s | ref |
|
| 53 |
+
| BF16 (tinyBLAS) | 1323 ms | 2481 ms | ~4.6 s | 0.10 / 11 |
|
| 54 |
+
|
| 55 |
+
BF16 is **slower** here, both in im2col (per-run bf16 conversion) and in the GEMM, so it
|
| 56 |
+
is not used. The BF16 `tinyBLAS` path is not a win on this machine; the F16 generic path
|
| 57 |
+
is already competitive. Not shipped.
|
| 58 |
+
|
| 59 |
+
## 4. Remaining VAE cost
|
| 60 |
|
| 61 |
The GEMM is now the largest phase (~2.2 s, ~0.7 TFLOPS effective) and runs through the
|
| 62 |
F32 llamafile sgemm. im2col is ~0.9 s. OpenVINO still decodes this VAE in ~0.3 s
|