add gguf/README.md from AngelSlim/Hy4-preview-GGUF

gguf/README.md

@@ -0,0 +1,249 @@
+# Hy4-preview GGUF
+
+Two GGUF builds of HY4-Preview: https://huggingface.co/tencent/Hy4-preview
+
+**Language / 语言:** [English](#english) · [中文](#中文)
+
+| file | size | bpw | notes |
+|---|---:|---:|---|
+| `Hy4-preview-Q4_K_M.gguf` | 435.20 GiB | 4.86 | standard 4-bit, safe default |
+| `Hy4-preview-STQ1_0.gguf` | 213.66 GiB | 2.38 | mixed 1-2 bit, half the size |
+
+**Neither file runs on stock llama.cpp.** The `hyv4` architecture is not upstream. Apply the
+patches in `hy4-preview-patch/`
+
+---
+
+<a name="english"></a>
+## English
+
+### 1. What these are
+
+**`Hy4-preview-Q4_K_M.gguf`** — a conventional Q4_K_M. Most tensors are Q4_K; `ffn_down_exps`
+gets Q6_K on 37 layers via llama.cpp's own logic. Use this unless you are memory-constrained.
+
+**`Hy4-preview-STQ1_0.gguf`** — mixed precision at ~2.38 bpw, roughly **half the size** for the
+same model. The routed-expert `gate`/`up` projections run at 1.3125 bpw (STQ1_0) on 29 layers and
+2.0625 bpw (IQ2_XXS) on the other 48. See section 3.
+
+Type histograms:
+
+```
+Q4_K_M:   F32 1080 / Q4_K 901 / Q8_0 78 / Q6_K 75
+STQ1_0:   F32 1080 / Q8_0 354 / Q5_K 234 / Q6_K 234 / IQ2_XXS 96 / IQ3_XXS 74 / STQ1_0 58 / IQ4_XS 3 / Q4_K 1
+```
+
+### 2. Running them
+
+Build a patched llama.cpp 
+
+
+```bash
+git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
+git checkout 0cea36222
+
+git apply hy4-preview-patch/0001-hyv4-architecture.patch
+git apply hy4-preview-patch/0002-stq1_0-quant-and-cuda.patch   # skip if only using Q4_K_M
+
+export PATH=/usr/local/cuda-13.0/bin:$PATH CUDACXX=/usr/local/cuda-13.0/bin/nvcc
+cmake -B build-cuda -DGGML_CUDA=ON -DLLAMA_CURL=OFF -DGGML_NATIVE=OFF \
+      -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=90 \
+      -DLLAMA_BUILD_UI=OFF -DLLAMA_USE_PREBUILT_UI=OFF
+cmake --build build-cuda --target llama-cli llama-bench llama-quantize -j 48
+```
+
+Set `-DCMAKE_CUDA_ARCHITECTURES` for your GPU (90 = H20/H100). Both `-DLLAMA_BUILD_UI=OFF` and
+`-DLLAMA_USE_PREBUILT_UI=OFF` are needed for an offline build; the first alone still downloads
+prebuilt assets.
+
+Then
+
+```bash
+# single prompt
+build-cuda/bin/llama-cli -m Hy4-preview-Q4_K_M.gguf -ngl 99 -c 8192 \
+    --temp 0 -n 512 --no-warmup --jinja -st -f prompt.txt
+
+# throughput
+build-cuda/bin/llama-bench -m Hy4-preview-STQ1_0.gguf -ngl 99 -p 512 -n 128 -r 3
+```
+
+- **`--jinja` is required for chat.** The HY4 chat template matches no llama.cpp built-in family.
+- **Keep the GGUF on local disk.** llama.cpp mmaps weights; over NFS random page faults run at
+  ~12 MB/s, turning a 1-minute load into hours.
+- **Use `-st -f prompt.txt` for a single prompt.** `-no-cnv` is ignored in this build and it will
+  spin printing `>` on EOF.
+- VRAM for full residency: ~435 GiB (Q4_K_M) or ~214 GiB (STQ1_0). With less, lower `-ngl`.
+
+Measured on 8x H20:
+
+| | prefill (pp512) | decode (tg128) |
+|---|---:|---:|
+| STQ1_0 | 204.56 ± 1.42 t/s | 19.52 ± 0.01 t/s |
+
+
+
+Python tools reading these files must use the patched `gguf-py` with an **absolute** path:
+`sys.path.insert(0, '/path/to/llama.cpp/gguf-py')`. 
+### 3. STQ1_0 and the mixed-precision strategy
+
+**The format.** STQ1_0 comes from llama.cpp PR #22836. Weights are ternary `{-d, 0, +d}`, with
+**exactly one of every four lanes forced to zero** (3:4 sparsity). Each group of 4 weights is a
+4-bit code plus a 1-bit table-select, indexing a 32-entry codebook; one fp16 scale covers 256
+weights. That is `2 + 32 + 8 = 42` bytes per 256 weights = **1.3125 bpw**.
+
+**Our encoder.** Upstream's quantizer targets QAT inputs already on the ternary grid: it ignores
+the imatrix, sets `d = amax`, and zeroes `argmin |x|`. That is weak for post-training
+quantization. We keep the format byte-identical and change only two decisions:
+
+1. **Weighted least-squares scale**, `d = sum(w*sel*x) / sum(w*sel^2)` instead of `d = amax`.
+2. **Imatrix-aware zero placement** — zero the lane minimising `w[j]*(x[j]^2 - (|x[j]|-d)^2)`,
+   the *incremental* cost rather than the smallest magnitude.
+
+alternating for 3 rounds. Measured on 1200 real expert rows: the LS scale alone gives **-89.7%**
+weighted SSD, and the imatrix terms a further **-4.1%** of the remainder. The headline win is the
+scale — `amax` pins `d` to the single largest outlier among 256 weights.
+
+**Where the bits go.** The three routed-expert families are 97.7% of all parameters, so the
+recipe spends freely on everything else:
+
+| family | STQ1_0 build | why |
+|---|---|---|
+| `ffn_gate_exps` / `ffn_up_exps` | STQ1_0 (29 layers) / IQ2_XXS (48 layers) | the bulk; layer choice is imatrix-derived |
+| `ffn_down_exps` | IQ3_XXS, IQ4_XS on last 3 | **writes straight into the residual stream**, so its error is not attenuated by a later gate — deliberately 2 levels higher |
+| attention out / gate / q_a | Q5_K | llama.cpp only auto-bumps these when `n_expert == 8`; HY4 has 256 |
+| MLA `q_b`/`k_b`/`v_b`/`kv_a_mqa` | Q8_0 | HY4's *split* names miss llama.cpp's substring match, so they get no automatic bump |
+| DSA indexer | Q8_0 / F32 | 105 tensors, 0.21 GiB total, gates which 2048 tokens each query sees |
+| iHC `*_fn`, router, norms, sink | F32 | mirrors the reference's `_keep_in_fp32_modules` |
+| `output` (lm_head) | F32 | via `--leave-output-tensor` |
+
+### 4. Building a runtime
+
+#### Re-quantizing from bf16
+
+The recipe files are included. **An imatrix is mandatory for STQ1_0** — its encoder uses it for
+the scale solve and zero placement.
+
+```bash
+build-cuda/bin/llama-quantize --dry-run --imatrix imatrix.gguf \
+    --tensor-type-file Hy4-preview-STQ1_0.tensortypes --leave-output-tensor \
+    HY4.bf16.gguf out.gguf IQ1_M          # Q4_K_M build: use Q4_K_M as the base ftype
+```
+
+---
+
+<a name="中文"></a>
+## 中文
+
+### 1. 这是什么
+
+**`Hy4-preview-Q4_K_M.gguf`** —— 常规 Q4_K_M。多数张量为 Q4_K，`ffn_down_exps` 由 llama.cpp
+自身逻辑提到 Q6_K（37 层）。**没有显存压力就用这个。**
+
+**`Hy4-preview-STQ1_0.gguf`** —— 约 2.38 bpw 的混合精度，同一个模型**体积减半**。路由专家的
+`gate`/`up` 在 29 层用 1.3125 bpw（STQ1_0），另 48 层用 2.0625 bpw（IQ2_XXS）。见第 3 节。
+
+类型直方图：
+
+```
+Q4_K_M:   F32 1080 / Q4_K 901 / Q8_0 78 / Q6_K 75
+STQ1_0:   F32 1080 / Q8_0 354 / Q5_K 234 / Q6_K 234 / IQ2_XXS 96 / IQ3_XXS 74 / STQ1_0 58 / IQ4_XS 3 / Q4_K 1
+```
+
+### 2. 如何使用
+
+
+```bash
+git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
+git checkout 0cea36222
+
+git apply hy4-preview-patch/0001-hyv4-architecture.patch
+git apply hy4-preview-patch/0002-stq1_0-quant-and-cuda.patch   # 只用 Q4_K_M 可跳过
+
+export PATH=/usr/local/cuda-13.0/bin:$PATH CUDACXX=/usr/local/cuda-13.0/bin/nvcc
+cmake -B build-cuda -DGGML_CUDA=ON -DLLAMA_CURL=OFF -DGGML_NATIVE=OFF \
+      -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=90 \
+      -DLLAMA_BUILD_UI=OFF -DLLAMA_USE_PREBUILT_UI=OFF
+cmake --build build-cuda --target llama-cli llama-bench llama-quantize -j 48
+```
+
+`-DCMAKE_CUDA_ARCHITECTURES` 按自己的 GPU 设置（90 = H20/H100）。离线构建**同时**需要
+`-DLLAMA_BUILD_UI=OFF` 与 `-DLLAMA_USE_PREBUILT_UI=OFF`，只给前者仍会去下载预构建资源。
+
+
+```bash
+# 单条 prompt
+build-cuda/bin/llama-cli -m Hy4-preview-Q4_K_M.gguf -ngl 99 -c 8192 \
+    --temp 0 -n 512 --no-warmup --jinja -st -f prompt.txt
+
+# 测速
+build-cuda/bin/llama-bench -m Hy4-preview-STQ1_0.gguf -ngl 99 -p 512 -n 128 -r 3
+```
+
+- **chat 必须加 `--jinja`。** HY4 的 chat template 不匹配 llama.cpp 任何内置模板家族。
+- **GGUF 必须放本地盘。** llama.cpp 用 mmap，NFS 随机页错误约 12 MB/s，本来 1 分钟的加载会变
+  成几小时。
+- **单条 prompt 用 `-st -f prompt.txt`。** 本 build 忽略 `-no-cnv`，遇 EOF 会一直打印 `>`。
+- 全量驻留显存需求：约 435 GiB（Q4_K_M）或约 214 GiB（STQ1_0）。不够就降低 `-ngl`。
+
+在 8 x H20 上实测（已确认 GPU 空闲、权重全驻显存）：
+
+| | 预填充 (pp512) | 解码 (tg128) |
+|---|---:|---:|
+| STQ1_0 | 204.56 ± 1.42 t/s | 19.52 ± 0.01 t/s |
+
+
+读这些文件的 Python 工具必须用打过补丁的 `gguf-py`，且用**绝对路径**：
+`sys.path.insert(0, '/path/to/llama.cpp/gguf-py')`。
+
+### 3. STQ1_0 与混合精度策略
+
+**格式。** STQ1_0 来自 llama.cpp PR #22836。权重为三值 `{-d, 0, +d}`，且**每 4 个 lane 强制
+一个为零**（3:4 稀疏）。每 4 个权重存成 4-bit code 加 1-bit 选表位，索引一张 32 项码本；每
+256 个权重共用一个 fp16 scale。即每 256 权重 `2 + 32 + 8 = 42` 字节 = **1.3125 bpw**。
+
+**我们的编码器。** 上游的量化器面向已落在三值网格上的 QAT 输入：直接忽略 imatrix，取
+`d = amax`，并把零放在 `argmin |x|`。这对训练后量化（PTQ）很弱。我们保持格式**逐字节一致**，
+只改两个决策：
+
+1. **加权最小二乘 scale**：`d = sum(w*sel*x) / sum(w*sel^2)`，取代 `d = amax`。
+2. **imatrix-aware 零位置**：零掉使 `w[j]*(x[j]^2 - (|x[j]|-d)^2)` 最小的 lane，即比较**增量**
+   代价，而非单纯的最小幅值。
+
+两者交替 3 轮。在 1200 行真实专家权重上实测：仅最小二乘 scale 就带来 **-89.7%** 加权 SSD，
+imatrix 项在残差上再补 **-4.1%**。**主要收益来自 scale**——`amax` 会把 `d` 钉在 256 个权重里
+的单个最大离群值上。
+
+**bit 花在哪。** 三个路由专家族占全部参数的 97.7%，所以配方在其余张量上舍得花：
+
+| 张量族 | STQ1_0 产物 | 原因 |
+|---|---|---|
+| `ffn_gate_exps` / `ffn_up_exps` | STQ1_0（29 层）/ IQ2_XXS（48 层）| 体积主体；选层由 imatrix 推导 |
+| `ffn_down_exps` | IQ3_XXS，最后 3 层 IQ4_XS | **直接写回残差流**，误差不会被后续 gate 衰减，故刻意高两档 |
+| attention out / gate / q_a | Q5_K | llama.cpp 只在 `n_expert == 8` 时自动提档，而 HY4 有 256 个专家 |
+| MLA `q_b`/`k_b`/`v_b`/`kv_a_mqa` | Q8_0 | HY4 的**拆分**命名匹配不上 llama.cpp 的子串规则，完全拿不到自动提档 |
+| DSA indexer | Q8_0 / F32 | 105 个张量共 0.21 GiB，却是决定每个 query 能看到哪 2048 个 token 的闸门 |
+| iHC `*_fn`、router、norms、sink | F32 | 对齐参考实现的 `_keep_in_fp32_modules` |
+| `output`（lm_head）| F32 | 通过 `--leave-output-tensor` |
+
+
+
+#### 从 bf16 重新量化
+
+配方文件已随附。**STQ1_0 强制需要 imatrix**——它的编码器要用 imatrix 做 scale 求解与零位置选择。
+
+```bash
+build-cuda/bin/llama-quantize --dry-run --imatrix imatrix.gguf \
+    --tensor-type-file Hy4-preview-STQ1_0.tensortypes --leave-output-tensor \
+    HY4.bf16.gguf out.gguf IQ1_M          # Q4_K_M 产物：基础 ftype 用 Q4_K_M
+```
+---
+
+## Files
+
+```
+hy4-preview-patch/
+  0001-hyv4-architecture.patch        18 files, +1632/-3   both GGUFs need this
+  0002-stq1_0-quant-and-cuda.patch    25 files, +683/-4    STQ1_0 only
+  Hy4-preview-STQ1_0.tensortypes      the STQ1_0 recipe
+  Hy4-preview-Q4_K_M.tensortypes      the Q4_K_M recipe
+```