README.md

---
language:
  - en
  - zh
base_model: openpangu/openPangu-R-7B-Diffusion
tags:
  - gguf
  - llama.cpp
  - diffusion
  - block-diffusion
  - openPangu
license: other
---

# openPangu-R-7B-Diffusion — Q4_K_M GGUF (llama.cpp)

Quantized **Q4_K_M** GGUF of [`openpangu/openPangu-R-7B-Diffusion`](https://huggingface.co/openpangu/openPangu-R-7B-Diffusion),
together with a llama.cpp patch that adds native support for the model's
**context-causal block-diffusion** decoding.

> `openPangu-R-7B-Diffusion` is a 7B block-diffusion LLM. Generation is *not* plain
> next-token autoregression: a block of tokens (default length **32**) is denoised in
> parallel with **full (bi-directional) attention inside the block** and **causal
> attention for the preceding context**, revealing the most confident tokens first.
> The model can also be run in ordinary autoregressive mode.

Upstream llama.cpp (as of tag `b11719`) ships the `PanguEmbedded` architecture
(used by the autoregressive `openPangu-Embedded`) but does **not** implement the
diffusion decoding this model needs, hence the patch.

## Files

| File | Description |
|---|---|
| `pangu-diffusion-Q4_K_M.gguf` | Q4_K_M quantized weights (~4.6 GB), ready to run |
| `pangu-diffusion-llamacpp.patch` | llama.cpp patch adding openPangu-R-Diffusion support |
| `build_llamacpp_pangu_diffusion.sh` | one-click script: clone llama.cpp → apply patch → build |

## Quick start

### 1. Build llama.cpp with the patch (one-click)

```bash
bash build_llamacpp_pangu_diffusion.sh
# binaries end up in ./pangu-diffusion-build/llama.cpp/build/bin/
```

Or manually:

```bash
git clone --depth 1 --branch b11719 https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git apply /path/to/pangu-diffusion-llamacpp.patch
cmake -B build -DGGML_NATIVE=ON -DLLAMA_CURL=OFF
cmake --build build -j --target llama-diffusion-cli llama-cli llama-quantize
```

### 2. Diffusion decoding (recommended)

`llama-diffusion-cli` applies the model's chat template automatically: pass the **raw
user message**, not a pre-formatted prompt. The rendered prompt is

```
[unused9]系统：<system>[unused10][unused9]用户：<user>[unused10][unused9]助手：
```

```bash
./build/bin/llama-diffusion-cli \
  -m pangu-diffusion-Q4_K_M.gguf \
  -p "Which is bigger, 9.11 or 9.9?" \
  -ub 256 \
  --diffusion-block-length 32 \
  --diffusion-small-block 4 \
  --diffusion-algorithm 1 \
  --temp 0 -t 16 --seed 42
```

Recommended sampling (from the official model card):
`alg = entropy`, `num_small_blocks = 8` → `--diffusion-small-block 4` (with block length 32),
`top_p = 1`, `temperature = 0`. For coding tasks the authors suggest
`alg = entropy, num_small_blocks = 32 (→ --diffusion-small-block 1), top_p = 0.8, temperature = 1`.

**Fast mode** (higher throughput): reveal every masked token whose confidence exceeds a
threshold in a single denoising step, with one small block per block:

```bash
./build/bin/llama-diffusion-cli \
  -m pangu-diffusion-Q4_K_M.gguf \
  -p "<question>" \
  -ub 256 --diffusion-block-length 32 --diffusion-small-block 32 \
  --diffusion-confidence-threshold 0.9 --diffusion-algorithm 4 \
  --temp 0 -t 16 --seed 42
```

`--diffusion-n-seq N` runs N sequences through the block decoder in a single
(micro-)batch (the prompt is replicated; see *Batched decoding* below).

`--diffusion-algorithm` values: `0` origin, `1` entropy, `2` margin, `3` random, `4` confidence.
`-ub N` is the maximum sequence length; the tail that does not fill a whole block is dropped.

### 3. Autoregressive decoding

```bash
./build/bin/llama-cli -m pangu-diffusion-Q4_K_M.gguf \
  -p "[unused9]系统：[unused10][unused9]用户：Hi![unused10][unused9]助手：" -n 128 -t 16
```

## Example

Prompt (translated): *“Which is bigger, 9.11 or 9.9?”* — command as in §2, fast mode
(`-ub 256`, `--diffusion-confidence-threshold 0.9`, Q4_K_M):

```
I need to figure out which is bigger between 9.11 and 9.9. Both are decimals, so I should compare them directly.

First, let's write them with the same number of decimal places to make it easier. 9.11 has two decimal places, and 9.9 has one. So, I can write 9.9 as 9.90 to make it two decimal places.

So, 9.11 vs 9.90.

Now, comparing the tenths place: both have 9 in the tenths place. So, that's the same.

Next, the hundredths place: 9.11 has 1 in the hundredths place, and 9.90 has 0 in the hundredths place. Since 1 is greater than 0, ...

I can think of them as fractions. 9.11 is 911/100, and 9.9 is 990/100.

Since the denominator is the same, I can compare the numerators: 911 vs 990.
911 is less than 990, so 911/100 is less than 990/100, which ...
```

For a short prompt the output is reproduced **token-for-token identically to the Hugging
Face reference** (see *Numerical parity* below), e.g. `-p "hello"` (f16 GGUF):
`[unused16]Okay, the user just said "hello". That's a simple greeting. I should respond
politely and warmly. Maybe they're just starting a conversation`.

## Speed when you actually use it (measured)

Test environment — the resources **actually usable inside the container (docker)**:

| Resource | Value |
|---|---|
| CPU | Intel Xeon Platinum 8559C |
| Effective cores | **16 vCPU** (cgroup `cpu.max` = `1600000 100000`) |
| Memory (cgroup limit) | **≈ 97 GiB** |
| GPU | **none usable** — `/dev/nvidia*` nodes exist but `cuInit()` returns `CUDA_ERROR_NO_DEVICE` and NVML init fails |
| Build / runtime | CPU-only, `-t 16` |

These are **end-user speeds** measured from real runs of the CLI on
`pangu-diffusion-Q4_K_M.gguf` (4.59 GiB, 8.03 B params), i.e. what you actually
experience while using it:

| What the user experiences | Mode | Speed |
|---|---|---|
| **Input** — waiting for the prompt to be read before output starts | both | **≈ 78 t/s** short prompt; **≈ 132 t/s** for longer prompts (`llama-bench` pp512) |
| **Output** — text generated (autoregressive, streamed live) | `llama-cli` | **11.9 t/s** (llama-bench tg128: 11.1 t/s) |
| **Output** — block diffusion (quality: block 32 / small-block 4, `alg=entropy`, `temp=0`) | `llama-diffusion-cli` | **≈ 4.4 t/s** |
| **Output** — block diffusion (fast: `--diffusion-confidence-threshold 0.9`, small-block 32, `alg=confidence` — the official throughput setting) | `llama-diffusion-cli` | **≈ 5 t/s** |

Notes:
- Input (prompt processing) is the same causal prefill for both paths; it is what you wait
  for before the answer appears.
- The AR output speed (≈ 12 t/s) is what `llama-cli` prints as `Generation: X t/s`; the
  diffused text is only flushed at the end of a run (unless `--diffusion-visual`).
- Block diffusion normally reveals one token per denoising step, i.e. up to `block_length`
  (= 32) forward passes per block. With `--diffusion-confidence-threshold T` every masked
  token whose confidence exceeds `T` is revealed in the same step (`transfer = argmax(conf)
  | (conf > T)`), which raises throughput — the same mechanism the authors report as giving
  up to 2.5× over autoregressive decoding on Ascend NPUs. A lower `T` is faster but less
  accurate; measured on this CPU (235 tokens, block 32 / small-block 32, `alg=confidence`,
  `temp=0`):

  | `--diffusion-confidence-threshold` | speed |
  |---|---|
  | `0.9` (default) | 4.9 t/s |
  | `0.7` | 7.4 t/s |
  | `0.5` | 8.7 t/s |

- The AR figures are the best reference for raw model throughput on this hardware.

## What the patch does

1. **`src/llama-arch.cpp`** — mark `LLM_ARCH_PANGU_EMBED` as a diffusion architecture
   (`llm_arch_is_diffusion()`), so `llama-diffusion-cli` accepts the model.
2. **`examples/diffusion/diffusion.{h,cpp}`** — add a *context-causal block-diffusion*
   decoder (`diffusion.block_attention`):
   - prefill the prompt with a causal forward (kept in the KV cache);
   - denoise each block with `causal_attn = false` → full attention over the cached
     context **plus** the current block (the KV-cache kq mask keeps every non-empty
     cached cell when non-causal);
   - re-decode the block each step, dropping its stale keys (`llama_memory_seq_rm`),
     reveal the most-confident masked token of the current small block, left to right;
   - commit the block with a causal forward so the KV cache holds correct causal
     context, then sample the first token of the next block from the last position;
   - `diffusion_generate_nseq()` generalises this to `n_seq` sequences sharing one
     micro-batch (see `--diffusion-n-seq`).
3. **`examples/diffusion/diffusion-cli.cpp`** — read `diffusion.block_attention`, add
   `--diffusion-small-block/-confidence-threshold/-n-seq`, size the context for the
   batched prefill, and use greedy sampling when `temperature == 0`.
4. **`gguf-py` + `conversion/pangu.py`** — write the new `diffusion.block_attention`
   metadata key for diffusion checkpoints (alongside `diffusion.shift_logits` and
   `tokenizer.ggml.mask_token_id`), and set `tokenizer.ggml.add_space_prefix = false`
   (see *Tokenizer fix* below).

### Model metadata in the GGUF

```
general.architecture            = pangu-embedded
diffusion.shift_logits          = true
diffusion.block_attention       = true
tokenizer.ggml.mask_token_id    = 45830
tokenizer.ggml.add_space_prefix = false
```

### Tokenizer fix

The upstream conversion honours `add_prefix_space` from `tokenizer_config.json`, but the
slow `PanguTokenizer` (sentencepiece, `legacy: true`) does **not** actually prepend a space.
With the flag left on, llama.cpp inserted spurious space tokens after every special token
(`[unused9]系统：…` became `[unused9]␠系统：…`), which changed the input and the generation.
The patch forces `tokenizer.ggml.add_space_prefix = false`; the tokenizer output then
matches the reference exactly (a 22-token prompt becomes 22 tokens, was 37).

### Numerical parity

Validated against the Hugging Face reference (`inference/generation_utils.py`,
`alg=confidence`, `num_small_blocks=1`, `threshold=0.9`, `temperature=0`) on the **f16**
GGUF: for `-p "hello"` the generated token IDs are **identical, token for token**, to the
reference. (The earlier mismatch was entirely the tokenizer issue above, not the
decoder.)

## Batched decoding (`--diffusion-n-seq`)

`--diffusion-n-seq N` runs N independent sequences through the block decoder inside a
single (micro-)batch. The example CLI replicates the prompt, so this is mainly a way to
measure batching; a real deployment would fill the rows with different prompts.

All sequences produce **byte-identical output** to the single-sequence run (verified for
N = 4 and N = 8), i.e. the batching is correct.

Measured on the CPU above (Q4_K_M, block 32 / small-block 32, `threshold 0.9`, greedy,
235 generated tokens):

| `--diffusion-n-seq` | wall time | aggregate | per sequence |
|---|---|---|---|
| 1 | 44.5 s | 5.28 t/s | 5.28 t/s |
| 4 | 162.4 s | 5.79 t/s | 1.45 t/s |
| 8 | 321.2 s | 5.85 t/s | 0.73 t/s |

**Review / honest verdict:** on this **CPU** batching gives essentially *no* throughput
gain — aggregate speed stays flat (≈5.3–5.9 t/s) and the time grows ~linearly with `N`,
so per-sequence speed drops 1/N. That is expected: a CPU decode step is compute-bound and
widens ~linearly with `n_seq` (there are no tensor cores to fill and no memory-bandwidth
headroom to exploit), so the fixed per-step work simply multiplies. Batching is where the
official 2.5× over autoregressive decoding comes from **on Ascend NPUs/GPUs**, not here.
We kept it because it is correct, cheap to carry, and a prerequisite for any GPU
deployment — but do not expect a CPU speed-up from it.

## Notes / limitations

- CPU-only, `-t 16`, ≈ 4.6 GB file: a 256-token block-diffusion run takes tens of seconds
  to a few minutes (`block_length` forwards per block). Throughput can be improved with
  `alg = confidence_threshold, num_small_blocks = 1` (see the official README).
- The decoder reproduces the Hugging Face reference **token-for-token** on the f16 GGUF
  (see *Numerical parity*), and the block-diffusion path was cross-checked against the
  reference logits (`cos ≈ 0.996`, f16 vs bf16).
- **Quantization + the greedy fast mode can change the reasoning**: on the adversarial
  `9.11 vs 9.9` prompt the Q4_K_M fast mode first states the wrong intermediate answer and
  only corrects itself from the fraction comparison above, whereas the f16 model matches
  the reference. Use the f16 GGUF, or `alg = entropy`, `temp = 0`, when a case must be
  exactly reproduced.

## Supported features

Implemented (matching the reference `inference/generation_utils.py`):

| Feature | Status |
|---|---|
| Context-causal block diffusion — full attention inside a block, causal attention over the preceding context | ✓ |
| Block-by-block decoding (block length 32) | ✓ `--diffusion-block-length` |
| Committed-block KV cache with a causal mask; first token of the next block decoded from the last position | ✓ |
| Small blocks inside a block (`num_small_blocks`, optimum 4 or 8) | ✓ `--diffusion-small-block` |
| Variable-length inference / flexible context | ✓ |
| Logit shift (`diffusion.shift_logits`) | ✓ |
| Dedicated mask token (`tokenizer.ggml.mask_token_id = 45830`) | ✓ |
| Reveal order: `confidence`, `entropy`, `topk_margin` | ✓ `--diffusion-algorithm 4/1/2` |
| `confidence_threshold` fast mode (multi-token reveal per step) | ✓ `--diffusion-confidence-threshold` |
| EOS early stop | ✓ |
| Batched (multi-sequence) block diffusion | ✓ `--diffusion-n-seq` |
| Autoregressive decoding | ✓ `llama-cli` |

Not implemented:
- Vendor NPU kernels — this uses whatever backend llama.cpp was built with (CPU here).

## Original model

- Weights: [`openpangu/openPangu-R-7B-Diffusion`](https://huggingface.co/openpangu/openPangu-R-7B-Diffusion)
- Architecture details, benchmarks and citation: see the original model card.