Integrate oneDNN/OpenVINO CPU backend study; merge docs; point build script at Experiment_1

da11688 verified 7 days ago

8.93 kB

---
license: other
license_name: openpangu-model-license-agreement-1.0
license_link: LICENSE
base_model: openpangu/openPangu-R-7B-2512
library_name: gguf
---
# openPangu-R-7B-2512-Q4_K_M (GGUF)

Powered by openPangu. openPangu is a trademark of Huawei Technologies Co., Ltd.

Q4_K_M GGUF conversion of [openpangu/openPangu-R-7B-2512](https://huggingface.co/openpangu/openPangu-R-7B-2512),
together with a CPU inference backend study (llama.cpp vs oneDNN vs OpenVINO).
This is an unofficial community conversion, not released or endorsed by Huawei.

## Repository layout

| path | contents |
|---|---|
| `openPangu-R-7B-2512-Q4_K_M.gguf` | the Q4_K_M model (4.72 GiB) |
| `openpangu-r-llama.cpp-v0.7.0.patch` | llama.cpp v0.7.0 patch adding the `openpangu-r` architecture |
| `build_openpangu_r.sh` | one-click clone + patch + build + model download |
| `onednn/` | CPU backend comparison: oneDNN backend patch, OpenVINO programs, benchmarks, raw logs |
| `LICENSE` | OpenPangu Model License Agreement 1.0 |

## Important

- **Upstream llama.cpp does not support this model.** The GGUF uses the architecture name `openpangu-r`,
  which exists only in the patch in this repository. The patch targets **llama.cpp v0.7.0** (latest stable release).
- Tested on CPU only (x86-64, AVX-512). GPU backends were not tested.

## Quick start

```bash
bash build_openpangu_r.sh          # clones llama.cpp v0.7.0, applies the patch, builds, downloads the GGUF
```

Options: `SKIP_MODEL=1` skips the download; `HF_TOKEN` is only needed if this repository is private.
The script downloads from this repository (`HelloSun/Experiment_1`) and checks the sha256 of the patch and the GGUF.

Run (raw prompt, Chinese chat template of the model):

```bash
./llama.cpp/build/bin/llama-completion -m models/openPangu-R-7B-2512-Q4_K_M.gguf -no-cnv --temp 0 -n 128 --no-display-prompt \
  -p "[unused9]系统：你是一个有帮助的助手。[unused10][unused9]用户：用一句话介绍什么是大语言模型。 /no_think[unused10][unused9]助手：[unused16]"
```

Prompt format: `[unused9]系统：…[unused10][unused9]用户：…[unused10][unused9]助手：[unused16]`.
Append ` /no_think` to a user message for fast (non-thinking) answers. The answer follows `[unused17]`; generation ends with `[unused10]`.

## Conversion

1. `convert_hf_to_gguf.py` (patched) with `--outtype f16` from the BF16 safetensors. The MTP module (layer 27) and the second final norm are dropped.
2. `llama-quantize <f16.gguf> <out.gguf> Q4_K_M`. Sinks and the conv weight are kept in F32.

## Verification (done in this environment)

- **Reference check:** a CPU fp32 reference was written from the HF modeling code. For a 33-token prompt, the top-10 logits
  match llama.cpp F16 (max |Δlogit| ≈ 0.01). For a 484-token prompt (sliding-window layers active), the top-10 match with
  max |Δlogit| ≈ 0.03. Turning the sliding window off in the reference changes the logits by about 6, so the window is verified.
- **Greedy decoding:** llama.cpp F16 matches the fp32 reference token by token (11 tokens checked).
- **Batching:** F16 gives the same results with prompt decoding in one batch, in chunks of 4, and token by token.
- **Long context:** a fact placed about 400 tokens before the question was answered correctly (`7312`).
- **Multi-turn:** a second user turn appended to the same context is answered in context.

Sample outputs (Q4_K_M):

| prompt | output |
|---|---|
| 用一句话介绍什么是大语言模型（快思考） | 大语言模型是一种基于大量文本数据训练而成的人工智能模型，能够理解和生成人类语言，广泛应用于自然语言处理任务。 |
| 写一首关于秋天的四行短诗（快思考） | 秋风轻拂叶纷飞，/ 金黄铺满小径长。/ 寒蝉低唱夕阳下，/ 一缕炊烟入梦乡。 |
| 一个班有30名学生，其中2/5是女生，女生有多少人？（慢思考） | 12人 |
| 123 + 456 等于多少？（慢思考） | 579 |

Note: in fast mode the model once answered `123 + 456 = 679`. The fp32 reference gives the same answer for that prompt, so this is a model
behavior, not a conversion or implementation error. Thinking mode answers 579.

## CPU backend comparison (Q4_K_M)

Measured on an Intel Xeon Platinum 8559C (192 threads host, 16-CPU container, AMX BF16/INT8, AVX-512),
80 GiB memory guard, no GPU. `pp512` / `tg128` are steady-state throughput; OpenVINO `tg128` is the
median of 128 decode steps. Full tables, accuracy checks and methodology: [`onednn/README.md`](onednn/README.md),
raw numbers in [`onednn/results.csv`](onednn/results.csv).

| backend | weights | pp512 tok/s | tg128 tok/s | ppl512 | argmax vs f32 |
|---|---|---:|---:|---:|---:|
| llama.cpp (plain AVX512) | Q4_K_M | **180.50** | **13.39** | - | - |
| llama.cpp (default AMX+repack) | Q4_K_M | 112.77 | 12.19 | - | - |
| oneDNN | int8 | 90.09 | 7.67 | - | - |
| oneDNN | bf16 | 75.52 | 4.79 | - | - |
| OpenVINO | f32 | 125.60 | 2.22 | **28.82** | 1.000 |
| OpenVINO | bf16-hint | 209.43 | 3.93 | 28.85 | 0.975 |
| OpenVINO | int8-asym | 219.17 | 4.70 | 29.11 | 0.951 |
| OpenVINO | int8-sym | 226.68 | - | 29.34 | 0.941 |
| OpenVINO | int4-asym | 223.65 | 5.20 | 32.68 | 0.703 |
| OpenVINO | int4-sym | 222.48 | - | 42.95 | 0.701 |

Takeaways:

- llama.cpp Q4_K_M is the fastest overall on this 16-CPU container: `13.39 tg128`, `180.50 pp512`.
- The oneDNN hook (in `onednn/onednn.patch`) is bandwidth-bound: bf16 weights are 16.6 GB/token vs
  ~5 GB/token for Q4_K, int8 8.3 GB/token. The AMX int8 kernel itself is fast (~7.1 TFLOPS at M=512);
  the loss comes from the weight format and per-call activation quantization.
- OpenVINO has the fastest prefill but the slowest decode here, because its decode graph runs attention
  over the full 640-slot KV cache and moves all KV tensors across the host/plugin boundary each step.
  int8/int4 are NNCF weight-only compression of the f32 IR: int8 stays close to f32 (PPL +0.3, ~95% argmax),
  int4 keeps short factual answers but loses distributional fidelity (PPL +3.9, ~70% argmax).
Reproduce: build the oneDNN backend with `onednn/build.sh <llama.cpp dir>`, benchmark with
`onednn/bench.sh <Q4_K_M.gguf>` (needs a plain AVX512 build and an oneDNN build alongside a default build).
The OpenVINO route is a direct `ov.convert_model` trace of the HF model (stateless prefill + decode IRs);
`optimum-cli`, `openvino-genai` `LLMPipeline` and `torch.compile(backend="openvino")` all reject this
architecture — see [`onednn/README.md`](onednn/README.md) and [`onednn/openvino_quantization.md`](onednn/openvino_quantization.md).

## Known limitations

- **One sequence per batch.** A batch with more than one sequence aborts. The conv history of the merge layers is kept inside the model
  and follows one sequence decoded contiguously. Prompt-cache reuse, `seq_rm`, and multi-slot serving are not supported.
- The MTP (next-token) head is not included and not used.
- Hadamard-rotated (quantized) KV cache is not supported; use the default f16 KV cache.
- `llama-server` was not validated.
- The oneDNN backend handles only 2D quantized-weight x 2D activation `MUL_MAT` (projections and `lm_head`);
  batched attention matmuls stay on the ggml path. int4 has no optimized kernel in oneDNN 3.7.

## License and notices

The model and derivative works are licensed under the **OPENPANGU MODEL LICENSE AGREEMENT VERSION 1.0** (see `LICENSE`).

- **EU restriction (Section 3):** you must not access, download, install, run, deploy, integrate, modify, or otherwise use this model within the European Union.
- **Acknowledgment (Section 4.2):** "Powered by openPangu" and the trademark notice "openPangu is a trademark of Huawei Technologies Co., Ltd." must appear on pages, manuals, or materials that describe features of the model.
- The model is provided "AS IS" without warranty. Outputs are generated automatically and may be inaccurate.

## Files and checksums

| file | sha256 |
|---|---|
| `openPangu-R-7B-2512-Q4_K_M.gguf` | `efeb81edbf64ad9d7a80254332f3128d86deec589ee39315a3c1f7ec1aa5f10a` |
| `openpangu-r-llama.cpp-v0.7.0.patch` | `b5f302da0cece9c6bc717982d820f8b3e6ca0970f8f32b549efb41ef0d6b8773` |
| `LICENSE` | `747083c8c24995c75ed641d5c031a95a859b1ec37cd3deeb033425a75249a3cb` |

Backend-study material (all small text files):

| path | contents |
|---|---|
| `onednn/README.md` | full comparison, environment, accuracy and per-backend notes |
| `onednn/results.csv` | every measured configuration |
| `onednn/onednn.{cpp,h}`, `onednn/onednn.patch` | oneDNN `MUL_MAT` backend for ggml-cpu |
| `onednn/build.sh`, `onednn/bench.sh` | build and benchmark helpers |
| `onednn/openvino/` | prefill/decode IR builders, NNCF compression, PPL/argmax and isolation tests |
| `onednn/logs/` | raw `llama-bench`, OpenVINO and NNCF logs (incl. `logs/quant/`) |
| `onednn/openvino_quantization.md` | why int8/int4 needed NNCF weight compression |

	---
	license: other
	license_name: openpangu-model-license-agreement-1.0
	license_link: LICENSE
	base_model: openpangu/openPangu-R-7B-2512
	library_name: gguf
	---

	# openPangu-R-7B-2512-Q4_K_M (GGUF)

	Powered by openPangu. openPangu is a trademark of Huawei Technologies Co., Ltd.

	Q4_K_M GGUF conversion of [openpangu/openPangu-R-7B-2512](https://huggingface.co/openpangu/openPangu-R-7B-2512),
	together with a CPU inference backend study (llama.cpp vs oneDNN vs OpenVINO).
	This is an unofficial community conversion, not released or endorsed by Huawei.

	## Repository layout

	\| path \| contents \|
	\|---\|---\|
	\| `openPangu-R-7B-2512-Q4_K_M.gguf` \| the Q4_K_M model (4.72 GiB) \|
	\| `openpangu-r-llama.cpp-v0.7.0.patch` \| llama.cpp v0.7.0 patch adding the `openpangu-r` architecture \|
	\| `build_openpangu_r.sh` \| one-click clone + patch + build + model download \|
	\| `onednn/` \| CPU backend comparison: oneDNN backend patch, OpenVINO programs, benchmarks, raw logs \|
	\| `LICENSE` \| OpenPangu Model License Agreement 1.0 \|

	## Important

	- Upstream llama.cpp does not support this model. The GGUF uses the architecture name `openpangu-r`,
	which exists only in the patch in this repository. The patch targets llama.cpp v0.7.0 (latest stable release).
	- Tested on CPU only (x86-64, AVX-512). GPU backends were not tested.

	## Quick start

	```bash
	bash build_openpangu_r.sh # clones llama.cpp v0.7.0, applies the patch, builds, downloads the GGUF
	```

	Options: `SKIP_MODEL=1` skips the download; `HF_TOKEN` is only needed if this repository is private.
	The script downloads from this repository (`HelloSun/Experiment_1`) and checks the sha256 of the patch and the GGUF.

	Run (raw prompt, Chinese chat template of the model):

	```bash
	./llama.cpp/build/bin/llama-completion -m models/openPangu-R-7B-2512-Q4_K_M.gguf -no-cnv --temp 0 -n 128 --no-display-prompt \
	-p "[unused9]系统：你是一个有帮助的助手。[unused10][unused9]用户：用一句话介绍什么是大语言模型。 /no_think[unused10][unused9]助手：[unused16]"
	```

	Prompt format: `[unused9]系统：…[unused10][unused9]用户：…[unused10][unused9]助手：[unused16]`.
	Append ` /no_think` to a user message for fast (non-thinking) answers. The answer follows `[unused17]`; generation ends with `[unused10]`.

	## Conversion

	1. `convert_hf_to_gguf.py` (patched) with `--outtype f16` from the BF16 safetensors. The MTP module (layer 27) and the second final norm are dropped.
	2. `llama-quantize <f16.gguf> <out.gguf> Q4_K_M`. Sinks and the conv weight are kept in F32.

	## Verification (done in this environment)

	- Reference check: a CPU fp32 reference was written from the HF modeling code. For a 33-token prompt, the top-10 logits
	match llama.cpp F16 (max \|Δlogit\| ≈ 0.01). For a 484-token prompt (sliding-window layers active), the top-10 match with
	max \|Δlogit\| ≈ 0.03. Turning the sliding window off in the reference changes the logits by about 6, so the window is verified.
	- Greedy decoding: llama.cpp F16 matches the fp32 reference token by token (11 tokens checked).
	- Batching: F16 gives the same results with prompt decoding in one batch, in chunks of 4, and token by token.
	- Long context: a fact placed about 400 tokens before the question was answered correctly (`7312`).
	- Multi-turn: a second user turn appended to the same context is answered in context.

	Sample outputs (Q4_K_M):

	\| prompt \| output \|
	\|---\|---\|
	\| 用一句话介绍什么是大语言模型（快思考） \| 大语言模型是一种基于大量文本数据训练而成的人工智能模型，能够理解和生成人类语言，广泛应用于自然语言处理任务。 \|
	\| 写一首关于秋天的四行短诗（快思考） \| 秋风轻拂叶纷飞，/ 金黄铺满小径长。/ 寒蝉低唱夕阳下，/ 一缕炊烟入梦乡。 \|
	\| 一个班有30名学生，其中2/5是女生，女生有多少人？（慢思考） \| 12人 \|
	\| 123 + 456 等于多少？（慢思考） \| 579 \|

	Note: in fast mode the model once answered `123 + 456 = 679`. The fp32 reference gives the same answer for that prompt, so this is a model
	behavior, not a conversion or implementation error. Thinking mode answers 579.

	## CPU backend comparison (Q4_K_M)

	Measured on an Intel Xeon Platinum 8559C (192 threads host, 16-CPU container, AMX BF16/INT8, AVX-512),
	80 GiB memory guard, no GPU. `pp512` / `tg128` are steady-state throughput; OpenVINO `tg128` is the
	median of 128 decode steps. Full tables, accuracy checks and methodology: [`onednn/README.md`](onednn/README.md),
	raw numbers in [`onednn/results.csv`](onednn/results.csv).

	\| backend \| weights \| pp512 tok/s \| tg128 tok/s \| ppl512 \| argmax vs f32 \|
	\|---\|---\|---:\|---:\|---:\|---:\|
	\| llama.cpp (plain AVX512) \| Q4_K_M \| 180.50 \| 13.39 \| - \| - \|
	\| llama.cpp (default AMX+repack) \| Q4_K_M \| 112.77 \| 12.19 \| - \| - \|
	\| oneDNN \| int8 \| 90.09 \| 7.67 \| - \| - \|
	\| oneDNN \| bf16 \| 75.52 \| 4.79 \| - \| - \|
	\| OpenVINO \| f32 \| 125.60 \| 2.22 \| 28.82 \| 1.000 \|
	\| OpenVINO \| bf16-hint \| 209.43 \| 3.93 \| 28.85 \| 0.975 \|
	\| OpenVINO \| int8-asym \| 219.17 \| 4.70 \| 29.11 \| 0.951 \|
	\| OpenVINO \| int8-sym \| 226.68 \| - \| 29.34 \| 0.941 \|
	\| OpenVINO \| int4-asym \| 223.65 \| 5.20 \| 32.68 \| 0.703 \|
	\| OpenVINO \| int4-sym \| 222.48 \| - \| 42.95 \| 0.701 \|

	Takeaways:

	- llama.cpp Q4_K_M is the fastest overall on this 16-CPU container: `13.39 tg128`, `180.50 pp512`.
	- The oneDNN hook (in `onednn/onednn.patch`) is bandwidth-bound: bf16 weights are 16.6 GB/token vs
	~5 GB/token for Q4_K, int8 8.3 GB/token. The AMX int8 kernel itself is fast (~7.1 TFLOPS at M=512);
	the loss comes from the weight format and per-call activation quantization.
	- OpenVINO has the fastest prefill but the slowest decode here, because its decode graph runs attention
	over the full 640-slot KV cache and moves all KV tensors across the host/plugin boundary each step.
	int8/int4 are NNCF weight-only compression of the f32 IR: int8 stays close to f32 (PPL +0.3, ~95% argmax),
	int4 keeps short factual answers but loses distributional fidelity (PPL +3.9, ~70% argmax).

	Reproduce: build the oneDNN backend with `onednn/build.sh <llama.cpp dir>`, benchmark with
	`onednn/bench.sh <Q4_K_M.gguf>` (needs a plain AVX512 build and an oneDNN build alongside a default build).
	The OpenVINO route is a direct `ov.convert_model` trace of the HF model (stateless prefill + decode IRs);
	`optimum-cli`, `openvino-genai` `LLMPipeline` and `torch.compile(backend="openvino")` all reject this
	architecture — see [`onednn/README.md`](onednn/README.md) and [`onednn/openvino_quantization.md`](onednn/openvino_quantization.md).

	## Known limitations

	- One sequence per batch. A batch with more than one sequence aborts. The conv history of the merge layers is kept inside the model
	and follows one sequence decoded contiguously. Prompt-cache reuse, `seq_rm`, and multi-slot serving are not supported.
	- The MTP (next-token) head is not included and not used.
	- Hadamard-rotated (quantized) KV cache is not supported; use the default f16 KV cache.
	- `llama-server` was not validated.
	- The oneDNN backend handles only 2D quantized-weight x 2D activation `MUL_MAT` (projections and `lm_head`);
	batched attention matmuls stay on the ggml path. int4 has no optimized kernel in oneDNN 3.7.

	## License and notices

	The model and derivative works are licensed under the OPENPANGU MODEL LICENSE AGREEMENT VERSION 1.0 (see `LICENSE`).

	- EU restriction (Section 3): you must not access, download, install, run, deploy, integrate, modify, or otherwise use this model within the European Union.
	- Acknowledgment (Section 4.2): "Powered by openPangu" and the trademark notice "openPangu is a trademark of Huawei Technologies Co., Ltd." must appear on pages, manuals, or materials that describe features of the model.
	- The model is provided "AS IS" without warranty. Outputs are generated automatically and may be inaccurate.

	## Files and checksums

	\| file \| sha256 \|
	\|---\|---\|
	\| `openPangu-R-7B-2512-Q4_K_M.gguf` \| `efeb81edbf64ad9d7a80254332f3128d86deec589ee39315a3c1f7ec1aa5f10a` \|
	\| `openpangu-r-llama.cpp-v0.7.0.patch` \| `b5f302da0cece9c6bc717982d820f8b3e6ca0970f8f32b549efb41ef0d6b8773` \|
	\| `LICENSE` \| `747083c8c24995c75ed641d5c031a95a859b1ec37cd3deeb033425a75249a3cb` \|

	Backend-study material (all small text files):

	\| path \| contents \|
	\|---\|---\|
	\| `onednn/README.md` \| full comparison, environment, accuracy and per-backend notes \|
	\| `onednn/results.csv` \| every measured configuration \|
	\| `onednn/onednn.{cpp,h}`, `onednn/onednn.patch` \| oneDNN `MUL_MAT` backend for ggml-cpu \|
	\| `onednn/build.sh`, `onednn/bench.sh` \| build and benchmark helpers \|
	\| `onednn/openvino/` \| prefill/decode IR builders, NNCF compression, PPL/argmax and isolation tests \|
	\| `onednn/logs/` \| raw `llama-bench`, OpenVINO and NNCF logs (incl. `logs/quant/`) \|
	\| `onednn/openvino_quantization.md` \| why int8/int4 needed NNCF weight compression \|