Translate model card to Chinese

README.md

@@ -15,115 +15,96 @@ tags:
 - nvfp4
 ---
 
-# flint CPU optimization (llama.cpp patch + GGUF files)
+# flint CPU 優化（llama.cpp patch + GGUF 檔案）
 
-[Recursive Flint](https://huggingface.co/RecursiveSI/flint) is a 39.8B-parameter mixture-of-experts coding model with about 2.5B
-active parameters per token (256 experts, top-6, 49 layers). This repo holds a **llama.cpp CPU patch** that speeds it up on
-Intel Xeon (AVX512-VNNI / AMX) and the GGUF files it was tuned and measured on. **The GGUF files are unmodified**; every gain
-comes from code (`patches/flint-cpu-opt.patch`, base commit `de5122d87`). It is in the style of
-[HelloSun/Qwen3.6-35B-A3B-GGUF_CPU_OPT](https://huggingface.co/HelloSun/Qwen3.6-35B-A3B-GGUF_CPU_OPT).
+[Recursive Flint](https://huggingface.co/RecursiveSI/flint) 是一個 39.8B 參數的 mixture-of-experts 程式碼模型，每個 token 約啟用 2.5B 參數（256 個 expert，top-6，49 層）。本 repo 提供一個 **llama.cpp CPU patch**，可在 Intel Xeon（AVX512-VNNI / AMX）上加速，並附上調校與量測所用的 GGUF 檔案。**GGUF 檔案未經修改**，所有加速都來自程式碼（`patches/flint-cpu-opt.patch`，base commit `de5122d87`）。寫法沿用 [HelloSun/Qwen3.6-35B-A3B-GGUF_CPU_OPT](https://huggingface.co/HelloSun/Qwen3.6-35B-A3B-GGUF_CPU_OPT)。
 
-| file | source | quant | size | in this repo |
+| 檔案 | 來源 | 量化 | 大小 | 是否在本 repo |
 |---|---|---|---|---|
-| `flint-UD-Q4_K_XL.gguf` | [unsloth/flint-GGUF](https://huggingface.co/unsloth/flint-GGUF) | Q5_K 11.6 GiB + IQ4_XS 9.0 GiB + Q8_0 2.0 GiB (+ small Q4_K/Q6_K/IQ3_S) | 23.3 GiB | yes |
-| `flint-int4-q4_1.gguf` | [RecursiveSI/flint-int4-gguf](https://huggingface.co/RecursiveSI/flint-int4-gguf) | Q4_1 for every weight (token_embd and router F32) | 24.2 GiB | yes |
-| `flint-nvfp4.gguf` | [RecursiveSI/flint-nvfp4-gguf](https://huggingface.co/RecursiveSI/flint-nvfp4-gguf) | NVFP4 weights (485 tensors, 20.7 GiB), F32 per-tensor `.scale`, BF16 token_embd | 22.9 GB | **no**, download from the link |
+| `flint-UD-Q4_K_XL.gguf` | [unsloth/flint-GGUF](https://huggingface.co/unsloth/flint-GGUF) | Q5_K 11.6 GiB + IQ4_XS 9.0 GiB + Q8_0 2.0 GiB（另有少量 Q4_K/Q6_K/IQ3_S） | 23.3 GiB | 是 |
+| `flint-int4-q4_1.gguf` | [RecursiveSI/flint-int4-gguf](https://huggingface.co/RecursiveSI/flint-int4-gguf) | 所有權重皆為 Q4_1（token_embd 與 router 為 F32） | 24.2 GiB | 是 |
+| `flint-nvfp4.gguf` | [RecursiveSI/flint-nvfp4-gguf](https://huggingface.co/RecursiveSI/flint-nvfp4-gguf) | NVFP4 權重（485 個 tensor，20.7 GiB），F32 的 per-tensor `.scale`，BF16 token_embd | 22.9 GB | **否**，請從連結下載 |
 
-## Which file to use
+## 選擇哪個檔案
 
-Measured on Intel Xeon Platinum 8559C, 16 threads, with the patch (median of 3 interleaved rounds, tokens/s).
-Q4_K_XL and Q4_1 rows are from the earlier measurement on a quieter host; the NVFP4 rows are from a later run on a busier
-host (see "Measurement notes"), so compare NVFP4 only with the NVFP4 rows.
+量測環境：Intel Xeon Platinum 8559C，16 threads，套用 patch，數值為三輪交錯量測的中位數（tokens/s）。Q4_K_XL 與 Q4_1 的數據來自較安靜的主機，NVFP4 的數據來自之後較忙碌的主機（見「量測注意事項」），因此 NVFP4 只能與 NVFP4 的數據比較。
 
-| file | prefill pp256 | prefill pp2048 | decode tg128 | PPL (wikitext-2, 32x512) |
+| 檔案 | prefill pp256 | prefill pp2048 | decode tg128 | PPL（wikitext-2, 32x512） |
 |---|---|---|---|---|
-| Q4_K_XL, stock llama.cpp | 231 | 274 | 38.2 | 18.363 +- 0.795 |
-| **Q4_K_XL, patched** | **296** | **307** | 42.8 | 18.277 +- 0.787 |
-| Q4_1, stock llama.cpp | 183 | 189 | 48.3 | 18.260 +- 0.780 |
-| **Q4_1, patched** | 266 | 297 | **55.7** | 18.240 +- 0.779 |
-| NVFP4, stock llama.cpp | 83.7 | 83.6 | 30.6 | 24.417 +- 1.566 (wikitext-2, 16x512) |
-| **NVFP4, patched** | **178.5** | **211.1** | 32.1 | 24.414 +- 1.568 (wikitext-2, 16x512) |
-
-- **Long prompts / large contexts (prefill-heavy): Q4_K_XL** still prefills fastest of the three in the earlier measurement,
-  and patched Q4_1 is within about 10 %.
-- **Chat and agent loops with a cached prefix (decode-heavy): Q4_1**. It decodes about 30 % faster than Q4_K_XL.
-- **NVFP4 on CPU**: prefill is about 2.1x (pp256) to 2.5x (pp2048) faster than stock after the patch. Decode is unchanged
-  within noise. NVFP4 is primarily intended for CUDA; on this CPU it is still slower than Q4_1 for both phases.
-- Quality is equivalent within measurement error: PPL differences are much smaller than the standard errors.
-
-## Gain of the patch (vs stock llama.cpp, same file)
-
-| file | pp256 | pp2048 | tg128 |
+| Q4_K_XL，stock llama.cpp | 231 | 274 | 38.2 | 18.363 ± 0.795 |
+| **Q4_K_XL，patched** | **296** | **307** | 42.8 | 18.277 ± 0.787 |
+| Q4_1，stock llama.cpp | 183 | 189 | 48.3 | 18.260 ± 0.780 |
+| **Q4_1，patched** | 266 | 297 | **55.7** | 18.240 ± 0.779 |
+| NVFP4，stock llama.cpp | 83.7 | 83.6 | 30.6 | 24.417 ± 1.566（wikitext-2, 16x512） |
+| **NVFP4，patched** | **178.5** | **211.1** | 32.1 | 24.414 ± 1.568（wikitext-2, 16x512） |
+
+- **長 prompt / 大 context（prefill 為主）：Q4_K_XL** 在三者中 prefill 最快（早期量測），patched Q4_1 約差 10% 以內。
+- **聊天與 agent 循環、有快取前綴（decode 為主）：Q4_1**，decode 比 Q4_K_XL 快約 30%。
+- **NVFP4 在 CPU 上**：套用 patch 後，prefill 比 stock 快約 2.1 倍（pp256）到 2.5 倍（pp2048）。decode 在雜訊範圍內沒有改善。NVFP4 主要為 CUDA 設計，在此 CPU 上 prefill 與 decode 仍都慢於 Q4_1。
+- 品質在量測誤差內相當：PPL 差異遠小於標準誤差。
+
+## patch 的加速效果（相對於 stock llama.cpp，同一檔案）
+
+| 檔案 | pp256 | pp2048 | tg128 |
 |---|---|---|---|
 | Q4_K_XL | 1.28x | 1.12x | 1.12x |
 | Q4_1 | 1.46x | 1.57x | 1.15x |
-| NVFP4 | 2.13x | 2.52x | 1.05x (noise) |
+| NVFP4 | 2.13x | 2.52x | 1.05x（雜訊） |
 
-## What the patch changes
+## patch 的修改內容
 
-For Q4_K_XL most of the gain comes from the Qwen3.6 patch kernels (Q4_K/Q5_K/Q6_K VNNI, AMX tweaks, MoE tile grouping) also
-applying to flint. Against the Qwen3.6 patch alone the flint-specific part is small (pp256 +12 %, decode +2 %, pp2048 equal).
-The flint-specific changes:
+對 Q4_K_XL 而言，大部分加速來自 Qwen3.6 patch 的 kernel（Q4_K/Q5_K/Q6_K VNNI、AMX 調整、MoE tile 分組），同樣適用於 flint。若只比較 flint 特有的部分（相對於單獨的 Qwen3.6 patch），差異很小（pp256 +12%、decode +2%、pp2048 持平）。flint 特有的修改如下：
 
-1. **IQ4_XS x Q8_K `vec_dot` in AVX512-VNNI** (x86 had AVX2 only). Unsigned LUT (+128) with `vpdpbusd`, bias removed through `bsums`, L2 prefetch.
-2. **Q4_1 x Q8_1 `vec_dot` in AVX-512VL VNNI**. Two blocks per iteration, packed into one ymm with memory-source `vinserti128`, low/high nibbles each through one `vpdpbusd`, fp16 headers converted together. A 512-bit version using cross-lane shuffles was slower than stock and dropped.
-3. **Q4_1 in the tiled VNNI GEMM** for prefill. Q4_1 has no super-block scale, so the tiles gain per-32-block f32 scale/min arrays (`sd`/`sm`) and a `Q41` kernel flag; activations reuse the existing Q8_1 `vec_dot_type`. Per 32-block: `d_a * (d_w * dot(q, code) + m_w * sum(code))`, integer dot/sum exact. This lifts pp256/pp2048 by ~34 %/42 % over the `vec_dot` path.
-4. Q5_K 5th-bit merge with `vptestmb` + masked add (bit-identical to the previous version, no speed change).
-5. **NVFP4 x Q8_0 `vec_dot` in AVX512-VNNI** (decode). Codes `|w|` (0..12) form the unsigned operand, the weight sign is moved onto the int8 activations. Each 64-element block has four 16-element subblocks with UE4M3 scales.
-6. **NVFP4 in the tiled VNNI GEMM** (prefill). Weights are stored as `LUT + 12` (unsigned 0..24) with one f32 scale per 16-wide subblock (`sd16`). The bias is removed exactly through the per-16 activation code sums. Activations are Q8_0 with one scale per 32-block. Enabled only in VNNI builds.
+1. **IQ4_XS x Q8_K 的 AVX512-VNNI `vec_dot`**（x86 原本只有 AVX2）。使用無號 LUT（+128）搭配 `vpdpbusd`，透過 `bsums` 移除偏移，並加入 L2 prefetch。
+2. **Q4_1 x Q8_1 的 AVX-512VL VNNI `vec_dot`**。每次迭代處理兩個 block，以記憶體來源的 `vinserti128` 打包進一個 ymm，低位與高位 nibble 各用一次 `vpdpbusd`，兩個 block 的 fp16 header 一起轉換。使用跨 lane shuffle 的 512 位元版本比 stock 慢，已捨棄。
+3. **Q4_1 進入 prefill 用的 tiled VNNI GEMM**。Q4_1 沒有 super-block scale，因此 tile 增加每 32-block 的 f32 scale/min 陣列（`sd`/`sm`）與 `Q41` kernel 旗標；activation 重用既有的 Q8_1 `vec_dot_type`。每個 32-block 計算 `d_a * (d_w * dot(q, code) + m_w * sum(code))`，整數的 dot 與 sum 皆精確。這使 pp256/pp2048 相對於 `vec_dot` 路徑分別提升約 34%/42%。
+4. Q5_K 第 5 個 bit 的合併改用 `vptestmb` + 遮罩加法（與前一版位元相同，速度無變化）。
+5. **NVFP4 x Q8_0 的 AVX512-VNNI `vec_dot`**（decode）。權重碼 `|w|`（0..12）作為無號運算元，權重的正負號移到 int8 activation 上。每 64 元素的 block 分為四個 16 元素的 subblock，各有 UE4M3 scale。
+6. **NVFP4 進入 tiled VNNI GEMM**（prefill）。權重儲存為 `LUT + 12`（無號 0..24），每 16 寬 subblock 一個 f32 scale（`sd16`）。偏移透過每 16 個 activation 碼的和精確移除。activation 為 Q8_0，每 32-block 一個 scale。僅在 VNNI 建置中啟用。
 
-## Accuracy
+## 準確度
 
-Wikitext-2 test, KL divergence against the stated baseline logits of the same file (Q4_K_XL / Q4_1):
+Wikitext-2 test，與同一檔案的基準 logits 比較 KL divergence（Q4_K_XL / Q4_1）：
 
-| comparison | KLD | same top-1 | noise floor: stock vs stock with `-fa off` |
+| 比較項目 | KLD | top-1 相同比例 | 雜訊底線：stock 與 stock 加 `-fa off` |
 |---|---|---|---|
-| Q4_K_XL, patched vs stock | 0.0666 +- 0.0049 | ~92 % | 0.0762 (also `-ub 128`: 0.0700) |
-| Q4_1, `vec_dot` patch vs stock | 0.0364 +- 0.0029 | 93.8 % | 0.0385 |
-| Q4_1, tiled GEMM vs `vec_dot` patch | 0.0372 +- 0.0026 | 94.2 % | 0.0385 |
+| Q4_K_XL，patched vs stock | 0.0666 ± 0.0049 | ~92% | 0.0762（另 `-ub 128`：0.0700） |
+| Q4_1，`vec_dot` patch vs stock | 0.0364 ± 0.0029 | 93.8% | 0.0385 |
+| Q4_1，tiled GEMM vs `vec_dot` patch | 0.0372 ± 0.0026 | 94.2% | 0.0385 |
 
-The KLD is **inside the noise floor of stock itself**: flint is very sensitive to tiny numeric differences (changing only the
-attention path of stock gives the same KLD), so it does not indicate a defect.
+KLD **落在 stock 本身的雜訊底線之內**：flint 對微小的數值差異很敏感（只改 stock 的 attention 路徑也會得到相同的 KLD），因此這不代表有缺陷。
 
-NVFP4 (not KLD-tested): perplexity on wikitext-2 test, 16 chunks x 512 tokens, `llama-perplexity`: stock 24.4165 +- 1.566,
-patched 24.4135 +- 1.568. Kernel-level checks: AVX512 `vec_dot` vs generic reference, relative error <= 1.3e-6; tiled vs
-`vec_dot`, relative L2 1.4e-7 .. 2.5e-7 over several shapes including ragged row counts and the narrow path.
-Integer dot products are exact; only float accumulation order changes.
+NVFP4（未做 KLD 測試）：`llama-perplexity` 在 wikitext-2 test 上，16 chunks x 512 tokens：stock 24.4165 ± 1.566，patched 24.4135 ± 1.568。kernel 層級的檢查：AVX512 `vec_dot` 與 generic 參考實作的相對誤差 ≤ 1.3e-6；tiled 與 `vec_dot` 的相對 L2 誤差為 1.4e-7 到 2.5e-7，涵蓋多種形狀，包括不整齊的 row 數與 narrow 路徑。整數點積是精確的，只有浮點累加順序會改變。
 
-## Where the time goes
+## 時間花在哪裡
 
-Measured DRAM read bandwidth on this host is 125-175 GB/s with 16 threads (single thread 15 GB/s). Decode of the Q4_K_XL / Q4_1
-files is mostly memory-bound (about 90 GB/s effective). Prefill is compute-bound. For NVFP4, `GGML_PROF` per-op timing showed
-that prefill `MUL_MAT` / `MUL_MAT_ID` with 256 rows dominated the run time before the tiled kernel was added; decode spends its
-time spread over many small ops.
+在此主機上，16 threads 的 DRAM 讀取頻寬為 125–175 GB/s（單 thread 為 15 GB/s）。Q4_K_XL / Q4_1 的 decode 大多受記憶體頻寬限制（實際約 90 GB/s）。prefill 則受計算限制。對 NVFP4，`GGML_PROF` 逐 op 計時顯示：加入 tiled kernel 前，256 rows 的 prefill `MUL_MAT` / `MUL_MAT_ID` 佔去大部分執行時間；decode 的時間則分散在許多小 op 上。
 
-## Build and run
+## 建置與執行
 
 ```bash
-./build.sh        # clones llama.cpp at patches/BASE_COMMIT, applies patches/flint-cpu-opt.patch, builds with -DGGML_NATIVE=ON
+./build.sh        # 會 clone llama.cpp 到 patches/BASE_COMMIT，套用 patches/flint-cpu-opt.patch，並以 -DGGML_NATIVE=ON 建置
 taskset -c 0-15 llama.cpp/build/bin/llama-cli -m flint-UD-Q4_K_XL.gguf -t 16 -fa on -lm mlock
 ```
 
-For NVFP4, download `flint-nvfp4.gguf` from RecursiveSI and run the same binary with `-m flint-nvfp4.gguf`.
+使用 NVFP4 時，請從 RecursiveSI 下載 `flint-nvfp4.gguf`，並以同一個 binary 搭配 `-m flint-nvfp4.gguf` 執行。
 
-`GGML_IQ4XS_PF`, `GGML_Q41_PF`, `GGML_K4_PF`, `GGML_K5_PF`, `GGML_Q6_PF` tune the prefetch distances (defaults are the tuned
-values). `patches/optional-profiler.patch` adds `GGML_PROF=1` per-op timing. `GGML_CPU_TILED_MM=0` disables the tiled GEMM.
+`GGML_IQ4XS_PF`、`GGML_Q41_PF`、`GGML_K4_PF`、`GGML_K5_PF`、`GGML_Q6_PF` 用來調整 prefetch 距離（預設值即為調校後的數值）。`patches/optional-profiler.patch` 提供 `GGML_PROF=1` 的逐 op 計時。`GGML_CPU_TILED_MM=0` 可關閉 tiled GEMM。
 
-## Measurement notes
+## 量測注意事項
 
-- Validated only on Intel Xeon Platinum 8559C (AVX512-VNNI + AMX), 16 threads, on a shared host. Other CPUs are untested; the new kernels need AVX512-VNNI (the Q4_1 one also AVX512VL/F16C).
-- Single runs on this shared host vary widely. The NVFP4 round-to-round spread was up to 2x for decode while the host load average was 55-63. Use medians of interleaved rounds; the raw data is in `benchmarks/nvfp4/`.
-- The Q4_K_XL numbers were measured before the Q4_1 kernels were added; the K-quant code path is unchanged except for removing unused constants, and was not re-benchmarked.
-- PPL and KLD use short tests (16 or 32 chunks of 512 tokens); no downstream task evaluation was run.
+- 僅在 Intel Xeon Platinum 8559C（AVX512-VNNI + AMX）、16 threads、共享主機上驗證。其他 CPU 尚未測試；新 kernel 需要 AVX512-VNNI（Q4_1 的版本另需 AVX512VL/F16C）。
+- 共享主機上的單次量測波動很大。NVFP4 在主機 load average 55–63 時，decode 的輪與輪之間差異最多達 2 倍。請使用交錯量測的中位數；原始資料在 `benchmarks/nvfp4/`。
+- Q4_K_XL 的數據是在加入 Q4_1 kernel 之前量測的；K-quant 的程式路徑除了移除未使用的常數外沒有變動，且未重新量測。
+- PPL 與 KLD 使用的是短測試（16 或 32 chunks，每 chunk 512 tokens）；未進行下游任務評估。
 
-## Files
+## 檔案說���
 
-- `patches/flint-cpu-opt.patch` (base `patches/BASE_COMMIT`), `patches/optional-profiler.patch`, `build.sh`, `machine.txt`
-- `benchmarks/summary.md` (Q4_K_XL), `benchmarks/q41-summary.md` (Q4_1), raw A/B CSVs (`ab-interleaved-flint.csv`, `q41-ab-interleaved.csv`, `q41-tiled-ab-interleaved.csv`), KLD logs (`kld_vs_stock.log`, `q41-kld.log`, `q41-tiled-kld.log`), op profiles, KLD bisect, `graph_accuracy_test.cpp`
-- `benchmarks/nvfp4/` (NVFP4 summary `README.md`, raw llama-bench CSV per round `bin_*_r*.csv`, flat `summary.csv`, PPL logs `ppl_*.log`, run script, kernel test sources)
-- `handoff/HANDOFF.md`: state of the work and next steps for the next agent
+- `patches/flint-cpu-opt.patch`（base 為 `patches/BASE_COMMIT`）、`patches/optional-profiler.patch`、`build.sh`、`machine.txt`
+- `benchmarks/summary.md`（Q4_K_XL）、`benchmarks/q41-summary.md`（Q4_1）、原始 A/B CSV（`ab-interleaved-flint.csv`、`q41-ab-interleaved.csv`、`q41-tiled-ab-interleaved.csv`）、KLD log（`kld_vs_stock.log`、`q41-kld.log`、`q41-tiled-kld.log`）、op profile、KLD bisect、`graph_accuracy_test.cpp`
+- `benchmarks/nvfp4/`（NVFP4 摘要 `README.md`、每輪原始 llama-bench CSV `bin_*_r*.csv`、扁平化的 `summary.csv`、PPL log `ppl_*.log`、執行腳本、kernel 測試原始碼）
+- `handoff/HANDOFF.md`：工作狀態與下一位 agent 的待辦事項
 
-## Credits and license
+## 致謝與授權
 
-Model: Recursive Flint (Apache-2.0), GGUF quantizations by Unsloth and RecursiveSI, unmodified. Patch builds on llama.cpp (MIT)
-and reuses ideas from the Qwen3.6 CPU-OPT patch. This repo is Apache-2.0 for the model files; the patch follows llama.cpp's license.
+模型：Recursive Flint（Apache-2.0），GGUF 量化由 Unsloth 與 RecursiveSI 提供，未經修改。patch 建立於 llama.cpp（MIT），並沿用 Qwen3.6 CPU-OPT patch 的部分想法。本 repo 的模型檔案採 Apache-2.0；patch 遵循 llama.cpp 的授權。