Pipeline stage P2: explain and fix refiner Q8_0 slowdown, V-pack hoist + 128x128 flash-attn tiles

- root cause: the reference Q4_K GGUF stores the 2 noise + 2 context refiner
blocks as Q8_0; on a memory-bound CPU GEMM the same M=1024 w13 matmul is
89.4 ms (Q8_0) vs 42.9 ms (Q4_K), ~-1.5 s per 8-step run
- tools/requant_refiner.py + tools/q4k.c rewrite only the 16 refiner
attention/FFN matrices as Q4_K (adaLN kept at Q8_0); build.sh now produces
z_image_turbo-Q4_K-ffn13-rq.gguf (opt-in, pixel diff mean 6.4)
- flash-attn: hoist V packing to once per head and raise tiles to 128x128;
FLASH_ATTN_EXT 4663 -> 4451 ms (-4.5%), pixel-identical
- add stages/stage_p2_refiner notes, SmartTest rq row, README findings

README.md

@@ -26,10 +26,13 @@ stable-diffusion.cpp 在 **純 CPU** 上跑 Z-Image-Turbo 的優化實驗。這
 - 單獨啟用 AMX 權重佈局（`SD_CPU_EXTRA_BUFT=1`，Q4_K）相對 `amx_lto` 約快 15–20%，是 sd.cpp 內最主要的收益來源。
 - AVX512 相對 AVX2 約快 2 倍（118.5 s → 58.1 s）。
 - FFN gate/up 融合（w1 + w3 → w13）輸出與原版逐位元相同，但在這台機器上沒有可量測的加速。
+- **Refiner 慢 2 倍的根因**：參考 Q4_K GGUF 把 2 個 noise-refiner 與 2 個 context-refiner 存成 **Q8_0**（主層是 Q4_K）。在記憶體受限的 CPU GEMM 上，同形狀 w13 每次 89.4 ms（Q8_0）→ 42.9 ms（Q4_K）。用 `tools/requant_refiner.py` 只把這 16 個 refiner 的 attention/FFN 權重改成 Q4_K，每 8 步約省 **1.5 s**（`-rq` 模型，屬 opt-in，像素差 mean 6.4）。
+- **flash-attn tiling**：tile 由 64x64 改為 128x128 並把 V 打包提升為每頭一次，`FLASH_ATTN_EXT` 4663 → 4451 ms（-4.5%），輸出逐像素相同。
 - Q4_K 是 sd.cpp 在此 CPU 上最快的量化格式。Q8_0 即使開 AMX 也比 Q4_K 慢。
 - 峰值 RSS：sd.cpp 7.7–11.1 GiB，OpenVINO 12.6 GiB，皆遠低於記憶體守衛上限。
 
 **雜訊說明**：同一 stock 設定在不同批次的總生成時間為 56.8 s（前次）與 59.7 s（本次）。排名相近的列（例如 Q4_K-ffn13+AMXw 與 Q4_K+AMXw）差距在 ±5% 內，不宜過度解讀。
+**負載說明**：主機是共用環境，co-tenant 負載會讓**所有**絕對時間一起上下移動（本 session 觀察到 OpenVINO 從 37.5 s 到 25 s、sd.cpp 從 46 s 到 36 s）。因此跨批次只能比**比值**或在同一批交錯量測；表中的絕對秒數是記錄當時負載下的結果，不是跨批次可比的定值。
 
 ## 比較表（中位數，n = 2）
 
@@ -68,6 +71,13 @@ OpenVINO 的 VAE 解碼約 0.55 s，sd.cpp 約 10 s。第一階段嘗試了 conv
 llamafile 版本把 VAE 解碼從 10.7 s 降到 6.6 s（-38%），加上 conv-direct 後為 5.0 s（-53%）。
 詳細數據與重現步驟見 `stages/stage1_vae/README.md`。
 
+## Pipeline 階段 P2：refiner 重新量化 + flash-attn tile（已完成）
+
+參考 GGUF 的 refiner 區塊是 Q8_0，是 P1「refiner w13 慢 1.7x」未解之謎的根因。`tools/requant_refiner.py`
+只改 16 個 refiner attention/FFN 權重為 Q4_K（adaLN 保持 Q8_0），同形狀每次 89.4 → 42.9 ms，每 8 步省約 1.5 s。
+flash-attn 的 V 打包提升為每頭一次、tile 改 128x128，`FLASH_ATTN_EXT` 再降 4.5%，且輸出逐像素相同。
+詳見 `stages/stage_p2_refiner/README.md`。
+
 ## 各版本的意義
 
 | variant | CMake 設定 | 說明 |
@@ -84,6 +94,7 @@ llamafile 版本把 VAE 解碼從 10.7 s 降到 6.6 s（-38%），加上 conv-di
 
 - `SD_CPU_EXTRA_BUFT=1`：擴散模型的 2 維權重改用 CPU 的 AMX 佈局（實驗性質，預設關閉）。
 - `z_image_turbo-Q4_K-ffn13.gguf`：由 `tools/fuse_ffn.py` 把 FFN 的 w1、w3 串接成 w13，模型會自動偵測並使用融合路徑。
+- `z_image_turbo-Q4_K-ffn13-rq.gguf`：再由 `tools/requant_refiner.py` 把 refiner 的 16 個 attention/FFN 權重從 Q8_0 改成 Q4_K（速度優先、opt-in，輸出與 base 略有差異）。
 - `GGML_CPU_PROFILE=1`：逐算子計時，輸出到 stderr（僅供分析，預設關閉）。
 
 ## Profile（Q4_K，優化前）
@@ -133,7 +144,11 @@ REPEAT=2 ./SmartTest.sh          # 每列跑 2 次取中位數
 - `patches/`
   - `sdcpp-0001-zimage-ffn-w13-and-cpu-extra-buft.patch`：`z_image.hpp`（FFN w13 融合）、`model_manager.cpp`（AMX 權重佈局）
   - `ggml-0001-cpu-op-profile.patch`：`ggml-cpu.c`（逐算子計時）
+  - `ggml-0002-flash-attn-kpack.patch`：`ops.cpp`（flash-attn K/V 打包提升為每頭一次）
+  - `ggml-0003-flash-attn-tiles.patch`：`common.h`（flash-attn tile 128x128）
 - `tools/fuse_ffn.py`：把 GGUF 的 FFN w1、w3 串接成 w13（Q4_K/Q8_0 的位元組串接是合法的）。
+- `tools/requant_refiner.py` + `tools/q4k.c`：把 refiner 區塊的 Q8_0 權重重新量化成 Q4_K。
+- `stages/stage_p2_refiner/`：refiner 根因分析、requant 與 flash-attn tile 的數據。
 - `tools/measure_rss.py`：執行指令並回報行程樹的峰值 RSS 與 wall time。
 - `results/`：`results.csv`、`results.md`、`hardware.txt`、`images/`（每次執行的 512x512 輸出）、`logs/`。
 

SmartTest.sh

@@ -48,12 +48,15 @@ MATRIX=(
     # stage 1 (VAE decode): llamafile sgemm build, with and without conv-direct for the VAE
     "llamafile-Q4_K-ffn13+AMXw|llamafile|Q4_K-ffn13|SD_CPU_EXTRA_BUFT=1|"
     "llamafile-Q4_K-ffn13+AMXw+convdirect|llamafile|Q4_K-ffn13|SD_CPU_EXTRA_BUFT=1|--conv-direct auto,vae=on"
+    # stage P2: refiner weights requantized Q8_0 -> Q4_K (speed-first, opt-in)
+    "llamafile-Q4_K-ffn13-rq+AMXw+convdirect|llamafile|Q4_K-ffn13-rq|SD_CPU_EXTRA_BUFT=1|--conv-direct auto,vae=on"
 )
 
 model_file() {
     case "$1" in
         Q4_K)       echo "$MODELS/z_image_turbo-Q4_K.gguf" ;;
         Q4_K-ffn13) echo "$MODELS/z_image_turbo-Q4_K-ffn13.gguf" ;;
+        Q4_K-ffn13-rq) echo "$MODELS/z_image_turbo-Q4_K-ffn13-rq.gguf" ;;
         Q4_0)       echo "$MODELS/z_image_turbo-Q4_0.gguf" ;;
         Q8_0)       echo "$MODELS/z_image_turbo-Q8_0.gguf" ;;
         *) return 1 ;;

build.sh

@@ -140,6 +140,7 @@ if [[ ${#SELECTED[@]} -gt 0 ]]; then
     apply_patch "$SRC"        "$ROOT/patches/sdcpp-0001-zimage-ffn-w13-and-cpu-extra-buft.patch"
     apply_patch "$SRC/ggml"   "$ROOT/patches/ggml-0001-cpu-op-profile.patch"
     apply_patch "$SRC/ggml"   "$ROOT/patches/ggml-0002-flash-attn-kpack.patch"
+    apply_patch "$SRC/ggml"   "$ROOT/patches/ggml-0003-flash-attn-tiles.patch"
 
     # ------------------------------------------------------------ build
     for v in "${SELECTED[@]}"; do
@@ -154,6 +155,17 @@ if [[ ${#SELECTED[@]} -gt 0 ]]; then
         cp "$build_dir/bin/sd-cli" "$BIN/$v/sd-cli"
         log "-> $BIN/$v/sd-cli"
     done
+
+    # q4k helper: wraps ggml's own Q4_K quantizer for tools/requant_refiner.py
+    ref_build="$SRC/build-llamafile"
+    [[ -f "$ref_build/ggml/src/libggml-base.a" ]] || ref_build="$SRC/build-${SELECTED[0]}"
+    if [[ -f "$ref_build/ggml/src/libggml-base.a" ]]; then
+        log "building q4k quantizer helper"
+        gcc -O2 -fopenmp -I "$SRC/include" -I "$SRC/ggml/include" -I "$SRC/ggml/src" \
+            "$ROOT/tools/q4k.c" "$ref_build/ggml/src/libggml-base.a" \
+            -o "$WORK/q4k" -lpthread -lm -lstdc++ \
+            || die "failed to build q4k helper"
+    fi
 fi
 
 # ---------------------------------------------------------------- models
@@ -200,6 +212,14 @@ PY
         log "fusing FFN w1/w3 -> w13 (Q4_K)"
         "$PY" "$ROOT/tools/fuse_ffn.py" "$MODELS/z_image_turbo-Q4_K.gguf" "$MODELS/z_image_turbo-Q4_K-ffn13.gguf"
     fi
+    # refiner requant: the reference GGUF stores the 2 noise + 2 context refiner
+    # blocks as Q8_0, which is ~2x slower than Q4_K on a memory-bound CPU GEMM.
+    # Rewrite just those 16 attention/FFN matrices as Q4_K (adaLN left at Q8_0).
+    if [[ -x "$WORK/q4k" && ! -f "$MODELS/z_image_turbo-Q4_K-ffn13-rq.gguf" ]]; then
+        log "requantizing refiner blocks Q8_0 -> Q4_K"
+        Q4K="$WORK/q4k" "$PY" "$ROOT/tools/requant_refiner.py" \
+            "$MODELS/z_image_turbo-Q4_K-ffn13.gguf" "$MODELS/z_image_turbo-Q4_K-ffn13-rq.gguf"
+    fi
 fi
 
 log "done"

patches/ggml-0002-flash-attn-kpack.patch

@@ -1,8 +1,8 @@
 diff --git a/src/ggml-cpu/ops.cpp b/src/ggml-cpu/ops.cpp
-index e8ef14f2..8a155984 100644
+index e8ef14f2..bc6918ca 100644
 --- a/src/ggml-cpu/ops.cpp
 +++ b/src/ggml-cpu/ops.cpp
-@@ -9238,10 +9238,20 @@ static void ggml_compute_forward_flash_attn_ext_f16_one_chunk(
+@@ -9238,10 +9238,27 @@ static void ggml_compute_forward_flash_attn_ext_f16_one_chunk(
      }
  }
  
@@ -14,16 +14,23 @@ index e8ef14f2..8a155984 100644
 +static thread_local std::vector<float> fa_kpack_buf;
 +static thread_local fa_kpack_key       fa_kpack_cur;
 +static thread_local bool               fa_kpack_valid = false;
++// Stage P1b: same idea for V - pack [nek1][DV] F32 for the current head once instead of
++// re-converting the whole V for every Q tile of that head.
++struct fa_vpack_key { const void * data; int64_t iv2, iv3, nek1, nbv1, nbv2, nbv3, dv; };
++static thread_local std::vector<float> fa_vpack_buf;
++static thread_local fa_vpack_key       fa_vpack_cur;
++static thread_local bool               fa_vpack_valid = false;
 +
  static void ggml_compute_forward_flash_attn_ext_tiled(
          const ggml_compute_params * params,
          ggml_tensor * dst,
          int ir0, int ir1) {
 +    fa_kpack_valid = false;  // new chunk: K contents may differ from the previous call
++    fa_vpack_valid = false;
      const ggml_tensor * q     = dst->src[0];
      const ggml_tensor * k     = dst->src[1];
      const ggml_tensor * v     = dst->src[2];
-@@ -9381,6 +9391,43 @@ static void ggml_compute_forward_flash_attn_ext_tiled(
+@@ -9381,6 +9398,69 @@ static void ggml_compute_forward_flash_attn_ext_tiled(
          memset(K_f32, 0, DK * KV_TILE_SZ * sizeof(float));
          memset(V32,   0, KV_TILE_SZ * DV * sizeof(float));
  
@@ -63,11 +70,37 @@ index e8ef14f2..8a155984 100644
 +            }
 +            kpack = fa_kpack_buf.data();
 +        }
++
++        // packed V for this head: [n_kv_tiles*KV_TILE_SZ][DV], zero-padded last tile
++        const float * vpack = nullptr;
++        {
++            const fa_vpack_key key{ v->data, iv2, iv3, nek1, nbv1, nbv2, nbv3, DV };
++            const int64_t n_kv_tiles = (nek1 + KV_TILE_SZ - 1) / KV_TILE_SZ;
++            const size_t  need       = (size_t) (n_kv_tiles * KV_TILE_SZ * DV);
++            if (!fa_vpack_valid || memcmp(&key, &fa_vpack_cur, sizeof(key)) != 0 || fa_vpack_buf.size() < need) {
++                if (fa_vpack_buf.size() < need) {
++                    fa_vpack_buf.resize(need);
++                }
++                float * dst_pack = fa_vpack_buf.data();
++                memset(dst_pack, 0, need * sizeof(float));
++                for (int64_t tk = 0; tk < nek1; tk++) {
++                    const char * v_data = (const char *)v->data + tk*nbv1 + iv2*nbv2 + iv3*nbv3;
++                    if (kv_type == GGML_TYPE_F16) {
++                        ggml_cpu_fp16_to_fp32((const ggml_fp16_t *)v_data, dst_pack + tk * DV, DV);
++                    } else {
++                        memcpy(dst_pack + tk * DV, v_data, DV * sizeof(float));
++                    }
++                }
++                fa_vpack_cur   = key;
++                fa_vpack_valid = true;
++            }
++            vpack = fa_vpack_buf.data();
++        }
 +
          for (int64_t ic = 0; ic < nek1; ic += KV_TILE_SZ) {
              const int kv_tile = (int)std::min((int64_t)KV_TILE_SZ, nek1 - ic);
  
-@@ -9406,24 +9453,10 @@ static void ggml_compute_forward_flash_attn_ext_tiled(
+@@ -9406,24 +9486,10 @@ static void ggml_compute_forward_flash_attn_ext_tiled(
                  }
              }
  
@@ -95,3 +128,28 @@ index e8ef14f2..8a155984 100644
              ggml_vec_scale_f32(Q_TILE_SZ * KV_TILE_SZ, KQ, scale);
  
              if (logit_softcap != 0.0f) {
+@@ -9472,21 +9538,14 @@ static void ggml_compute_forward_flash_attn_ext_tiled(
+             }
+ 
+             // V accumulation: VKQ32 += softmax(KQ) * V
+-            // Pack V tile to contiguous F32, zero-padded
+-            for (int tk = 0; tk < kv_tile; tk++) {
+-                const char * v_data = (const char *)v->data + (ic + tk)*nbv1 + iv2*nbv2 + iv3*nbv3;
+-                if (kv_type == GGML_TYPE_F16) {
+-                    ggml_cpu_fp16_to_fp32((const ggml_fp16_t *)v_data, V32 + tk * DV, DV);
+-                } else {
+-                    memcpy(V32 + tk * DV, v_data, DV * sizeof(float));
+-                }
+-            }
++            // V tile comes from the per-head packed buffer (see above)
++            const float * V_tile = vpack + ic * DV;
+             for (int tq = 0; tq < Q_TILE_SZ; tq++) {
+                 if (skip[tq]) {
+                     memset(KQ + tq * KV_TILE_SZ, 0, KV_TILE_SZ * sizeof(float));
+                 }
+             }
+-            simd_gemm(VKQ32, KQ, V32, Q_TILE_SZ, KV_TILE_SZ, DV);
++            simd_gemm(VKQ32, KQ, V_tile, Q_TILE_SZ, KV_TILE_SZ, DV);
+         }
+ 
+         // sinks (apply only to valid rows in the tile)

patches/ggml-0003-flash-attn-tiles.patch

@@ -0,0 +1,15 @@
+diff --git a/src/ggml-cpu/common.h b/src/ggml-cpu/common.h
+index abbadc35..55ed9826 100644
+--- a/src/ggml-cpu/common.h
++++ b/src/ggml-cpu/common.h
+@@ -6,8 +6,8 @@
+ #include "ggml-impl.h"
+ #include "simd-mappings.h"
+ 
+-#define GGML_FA_TILE_Q  64
+-#define GGML_FA_TILE_KV 64
++#define GGML_FA_TILE_Q  128
++#define GGML_FA_TILE_KV 128
+ 
+ #ifdef __cplusplus
+ 

stages/stage_p2_refiner/README.md

@@ -0,0 +1,109 @@
+# Stage P2 — noise-refiner slowdown, refiner requant, flash-attn tiling
+
+Continues stage P1. Three things:
+
+1. Explains the stage-P1 open question: *"noise-refiner w13 is ~1.7x slower per call than the
+   main-layer w13 at the same shape; not explained yet."*
+2. Fixes it with a weight-only requant (`tools/requant_refiner.py`).
+3. Tunes the flash-attn tiled kernel tile sizes and hoists the V packing.
+
+All numbers below are from the same `llamafile` build, `SD_CPU_EXTRA_BUFT=1`,
+`--conv-direct auto,vae=on`, Q4_K-ffn13, 512x512, 8 steps, seed 42.
+
+## 1. Root cause of the refiner w13 "2x" anomaly
+
+The noise-refiner and context-refiner blocks in the reference `leejet/Z-Image-Turbo-GGUF`
+Q4_K file are **Q8_0**, while every main-layer transformer block is **Q4_K**
+(`ggml` type `8` = `Q8_0`, `12` = `Q4_K`). On a memory-bound CPU GEMM the 8-bit
+weights move twice the bytes, so the refiner matmuls run at roughly half the
+throughput of the Q4_K main layers even though the refiner token count is smaller
+(M=1024 vs M=1088).
+
+Shape-level profile (`GGML_CPU_PROFILE=2`), same M=1024 `w13` matmul (K=3840, N=20480):
+
+| refiner w13 weights | per call |
+|---|---|
+| Q8_0 (reference GGUF) | 89.4 ms |
+| Q4_K (after requant) | 42.9 ms |
+
+Per 8-step run the four refiner blocks (2 noise + 2 context) went from ~2985 ms to
+~1410 ms, i.e. about **-1.5 s** of sampling.
+
+## 2. Requant tool (`tools/requant_refiner.py`)
+
+Rewrites only the 16 refiner **attention** (`qkv`, `out`) and **feed-forward**
+(`w13`, `w2`) Q8_0 tensors as Q4_K. Everything else — including the small
+`adaLN_modulation` tensors, whose M=1 GEMV cost is irrelevant — is copied
+byte-for-byte. Uses ggml's own Q4_K quantizer through the tiny `tools/q4k.c`
+helper, so the requantized values match what ggml produces natively.
+
+Output: `z_image_turbo-Q4_K-ffn13-rq.gguf` (built automatically by `build.sh`).
+
+Quality trade-off (8 steps, seed 42, vs the un-requantized model):
+
+| model | pixel diff vs base (mean / max) |
+|---|---|
+| base Q4_K-ffn13 (refiner Q8_0) | ref |
+| **Q4_K-ffn13-rq** (refiner Q4_K) | 6.40 / 252 |
+
+Small but visible, so it is an **opt-in** speed variant; keep using the base model
+when refiner fidelity matters.
+
+## 3. Flash-attn tile sizes + V-pack hoist
+
+Stage P1 hoisted the K packing to once per head. This stage does the same for V
+(previously re-converted for every Q tile), zero-padding the tail tile, and raises
+the tile size from 64x64 to 128x128 (`patches/ggml-0003-flash-attn-tiles.patch`).
+
+| build | FLASH_ATTN_EXT (8 steps) |
+|---|---|
+| P1 (K hoist, 64x64) | 4663 ms |
+| + V hoist, 64x64 | 4671 ms (no change: V repack was already cheap) |
+| + V hoist, 128x128 | 4451 ms (-4.5%) |
+| + V hoist, 128x256 | 4427 ms (~-5%, more scratch; not used) |
+
+Output is **pixel-identical** to the P1 build (diff mean 0, max 0) at every tile
+size tested, so the tiling change is safe. The V hoist alone shows the "measure,
+don't assume" point: it is correct but buys nothing, because the F32 tile copy was
+never the bottleneck — the `simd_gemm` F32 math is.
+
+## 4. Where the time still goes (and the remaining gap to OpenVINO)
+
+On a lightly loaded host, OpenVINO INT4 finishes the whole 8-step 512x512 job in
+~25-27 s (VAE decode ~0.3 s). sd.cpp (base or rq) lands around ~36-40 s. Wall-clock
+swings by several seconds with co-tenant load, so compare **within a batch** or use
+the profile totals.
+
+Per-op split of the ~31-32 s sd.cpp sampling (Q4_K-ffn13-rq, conv-direct):
+
+| op | time | note |
+|---|---|---|
+| MUL_MAT | ~24 s | main-layer Q4_K AMX; ~4 TFLOPS effective |
+| FLASH_ATTN_EXT | ~4.4 s | F32 tiled kernel |
+| element-wise | ~2.5 s | RMS norm, gate residuals, etc. |
+| VAE decode | ~4.3 s | OpenVINO is ~0.3 s — biggest relative gap |
+
+The sampling gap to OpenVINO is only ~15-20%, but the VAE is ~14x slower, so the
+VAE now dominates the remaining distance to parity.
+
+## Next steps
+
+- **VAE**: the 1.5 TFLOP F32 decode cannot beat INT8 AMX at F32. The realistic fix is
+  an INT8/Q8_0 conv path (implicit-GEMM over the 9 kernel offsets) reusing the AMX
+  int8 GEMM, which needs a 2-D flattened conv-weight layout at load time. This is the
+  main piece of work left to approach OpenVINO's 0.3 s.
+- **flash-attn**: F32 `simd_gemm` tops out near 0.5 TFLOPS here; a bf16 AVX512
+  attention path is the next step.
+- **matmul**: try AMX cache blocking (nb-outer / mb-inner) for the tall-skinny
+  main-layer shapes.
+
+## Reproduce
+
+```bash
+./build.sh llamafile                 # builds q4k, ffn13 and the -rq model
+SD_CPU_EXTRA_BUFT=1 GGML_CPU_PROFILE=1 work/bin/llamafile/sd-cli \
+  --diffusion-model work/models/z_image_turbo-Q4_K-ffn13-rq.gguf \
+  --vae work/models/ae.safetensors --llm work/models/Qwen3-4B-Instruct-2507-Q4_K_M.gguf \
+  -p "a cute cat" -W 512 -H 512 --steps 8 --seed 42 -t 16 \
+  --conv-direct auto,vae=on -o out.png
+```

tools/q4k.c

@@ -0,0 +1,17 @@
+#include "ggml.h"
+#include <stdio.h>
+#include <stdlib.h>
+int main(int argc,char**argv){
+    if(argc<3) return 2;
+    long nrows=atol(argv[1]), np=atol(argv[2]);
+    long n=nrows*np;
+    float *src=malloc(n*sizeof(float));
+    if(fread(src,sizeof(float),n,stdin)!=(size_t)n){fprintf(stderr,"read fail\n");return 3;}
+    size_t nb=ggml_row_size(GGML_TYPE_Q4_K,np);
+    unsigned char *dst=malloc(nb*nrows);
+    ggml_quantize_init(GGML_TYPE_Q4_K);
+    size_t got=ggml_quantize_chunk(GGML_TYPE_Q4_K,src,dst,0,nrows,np,NULL);
+    if(got!=nb*nrows){fprintf(stderr,"quant size mismatch %zu %zu\n",got,nb*nrows);return 4;}
+    fwrite(dst,1,got,stdout);
+    return 0;
+}

tools/requant_refiner.py

@@ -0,0 +1,91 @@
+#!/usr/bin/env python3
+"""Requantize selected GGUF tensors (default: refiner FFN/attention weights) to Q4_K.
+
+The reference Z-Image-Turbo Q4_K GGUF stores the two noise-refiner and two
+context-refiner blocks as Q8_0. On a memory-bound CPU GEMM those 8-bit weights
+make the refiner matmuls ~2x slower than the Q4_K main layers even though the
+token count is smaller. This tool rewrites just the matching Q8_0 tensors as
+Q4_K, leaving every other tensor byte-for-byte untouched.
+
+Requires the small `q4k` helper (build with build.sh or see README) which wraps
+ggml's own Q4_K quantizer.
+"""
+import argparse
+import os
+import subprocess
+import sys
+import tempfile
+
+import numpy as np
+import gguf
+from gguf import GGUFReader, GGUFWriter, GGUFValueType
+
+Q4K = os.environ.get("Q4K", "/tmp/q4k")
+
+
+def q4k_quantize(f32: np.ndarray, helper: str) -> np.ndarray:
+    nrows, n_per_row = f32.shape
+    p = subprocess.run([helper, str(nrows), str(n_per_row)],
+                       input=np.ascontiguousarray(f32, dtype=np.float32).tobytes(),
+                       stdout=subprocess.PIPE, check=True)
+    return np.frombuffer(p.stdout, dtype=np.uint8)
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("src")
+    ap.add_argument("dst")
+    ap.add_argument("--pattern", default="refiner")
+    ap.add_argument("--only-kinds", default="attention,feed_forward",
+                    help="comma list of substrings a tensor name must contain (empty = any)")
+    ap.add_argument("--helper", default=Q4K)
+    ap.add_argument("--dry-run", action="store_true")
+    args = ap.parse_args()
+
+    r = GGUFReader(args.src)
+    arch = str(r.fields["general.architecture"].contents()) if "general.architecture" in r.fields else "zimage"
+
+    # pick targets
+    kinds = [k for k in args.only_kinds.split(",") if k]
+    targets = []
+    for t in r.tensors:
+        if args.pattern in t.name and t.tensor_type == gguf.GGMLQuantizationType.Q8_0:
+            if kinds and not any(k in t.name for k in kinds):
+                continue
+            targets.append(t.name)
+    print(f"targets ({len(targets)}):")
+    for n in targets:
+        print("  ", n)
+    if args.dry_run or not targets:
+        return 0
+
+    w = GGUFWriter(args.dst, arch, use_temp_file=True)
+    for name, field in r.fields.items():
+        if name == "general.architecture" or name.startswith("GGUF."):
+            continue
+        v = field.contents()
+        w.add_key_value(name, v, field.types[0],
+                        sub_type=field.types[-1] if len(field.types) > 1 else None)
+
+    tset = set(targets)
+    for t in r.tensors:
+        if t.name in tset:
+            f32 = gguf.quants.dequantize(t.data, t.tensor_type)
+            raw = q4k_quantize(f32, args.helper)
+            tgt = int(t.shape[0]) // 256 * 144  # Q4_K bytes per row (ne0 = input width)
+            w.add_tensor(t.name, raw.reshape(-1, tgt),
+                         raw_dtype=gguf.GGMLQuantizationType.Q4_K)
+            print(f"  requantized {t.name} {tuple(t.shape)} Q8_0->Q4_K "
+                  f"{t.n_bytes} -> {raw.nbytes} bytes")
+        else:
+            w.add_tensor(t.name, t.data, raw_dtype=t.tensor_type)
+    w.write_header_to_file()
+    w.write_kv_data_to_file()
+    w.write_tensors_to_file()
+    w.close()
+    print("wrote", args.dst)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())