--- language: - en - zh base_model: openpangu/openPangu-R-7B-Diffusion tags: - gguf - llama.cpp - diffusion - block-diffusion - openPangu license: other --- # openPangu-R-7B-Diffusion — Q4_K_M GGUF (llama.cpp) Quantized **Q4_K_M** GGUF of [`openpangu/openPangu-R-7B-Diffusion`](https://huggingface.co/openpangu/openPangu-R-7B-Diffusion), together with a llama.cpp patch that adds native support for the model's **context-causal block-diffusion** decoding. > `openPangu-R-7B-Diffusion` is a 7B block-diffusion LLM. Generation is *not* plain > next-token autoregression: a block of tokens (default length **32**) is denoised in > parallel with **full (bi-directional) attention inside the block** and **causal > attention for the preceding context**, revealing the most confident tokens first. > The model can also be run in ordinary autoregressive mode. Upstream llama.cpp (as of tag `b11719`) ships the `PanguEmbedded` architecture (used by the autoregressive `openPangu-Embedded`) but does **not** implement the diffusion decoding this model needs, hence the patch. ## Files | File | Description | |---|---| | `pangu-diffusion-Q4_K_M.gguf` | Q4_K_M quantized weights (~4.6 GB), ready to run | | `pangu-diffusion-llamacpp.patch` | llama.cpp patch adding openPangu-R-Diffusion support | | `build_llamacpp_pangu_diffusion.sh` | one-click script: clone llama.cpp → apply patch → build | ## Quick start ### 1. Build llama.cpp with the patch (one-click) ```bash bash build_llamacpp_pangu_diffusion.sh # binaries end up in ./pangu-diffusion-build/llama.cpp/build/bin/ ``` Or manually: ```bash git clone --depth 1 --branch b11719 https://github.com/ggml-org/llama.cpp.git cd llama.cpp git apply /path/to/pangu-diffusion-llamacpp.patch cmake -B build -DGGML_NATIVE=ON -DLLAMA_CURL=OFF cmake --build build -j --target llama-diffusion-cli llama-cli llama-quantize ``` ### 2. Diffusion decoding (recommended) `llama-diffusion-cli` applies the model's chat template automatically: pass the **raw user message**, not a pre-formatted prompt. The rendered prompt is ``` [unused9]系统：[unused10][unused9]用户：[unused10][unused9]助手： ``` ```bash ./build/bin/llama-diffusion-cli \ -m pangu-diffusion-Q4_K_M.gguf \ -p "Which is bigger, 9.11 or 9.9?" \ -ub 256 \ --diffusion-block-length 32 \ --diffusion-small-block 4 \ --diffusion-algorithm 1 \ --temp 0 -t 16 --seed 42 ``` Recommended sampling (from the official model card): `alg = entropy`, `num_small_blocks = 8` → `--diffusion-small-block 4` (with block length 32), `top_p = 1`, `temperature = 0`. For coding tasks the authors suggest `alg = entropy, num_small_blocks = 32 (→ --diffusion-small-block 1), top_p = 0.8, temperature = 1`. **Fast mode** (higher throughput): reveal every masked token whose confidence exceeds a threshold in a single denoising step, with one small block per block: ```bash ./build/bin/llama-diffusion-cli \ -m pangu-diffusion-Q4_K_M.gguf \ -p "" \ -ub 256 --diffusion-block-length 32 --diffusion-small-block 32 \ --diffusion-confidence-threshold 0.9 --diffusion-algorithm 4 \ --temp 0 -t 16 --seed 42 ``` `--diffusion-n-seq N` runs N sequences through the block decoder in a single (micro-)batch (the prompt is replicated; see *Batched decoding* below). `--diffusion-algorithm` values: `0` origin, `1` entropy, `2` margin, `3` random, `4` confidence. `-ub N` is the maximum sequence length; the tail that does not fill a whole block is dropped. ### 3. Autoregressive decoding ```bash ./build/bin/llama-cli -m pangu-diffusion-Q4_K_M.gguf \ -p "[unused9]系统：[unused10][unused9]用户：Hi![unused10][unused9]助手：" -n 128 -t 16 ``` ## Example Prompt (translated): *“Which is bigger, 9.11 or 9.9?”* — command as in §2, fast mode (`-ub 256`, `--diffusion-confidence-threshold 0.9`, Q4_K_M): ``` I need to figure out which is bigger between 9.11 and 9.9. Both are decimals, so I should compare them directly. First, let's write them with the same number of decimal places to make it easier. 9.11 has two decimal places, and 9.9 has one. So, I can write 9.9 as 9.90 to make it two decimal places. So, 9.11 vs 9.90. Now, comparing the tenths place: both have 9 in the tenths place. So, that's the same. Next, the hundredths place: 9.11 has 1 in the hundredths place, and 9.90 has 0 in the hundredths place. Since 1 is greater than 0, ... I can think of them as fractions. 9.11 is 911/100, and 9.9 is 990/100. Since the denominator is the same, I can compare the numerators: 911 vs 990. 911 is less than 990, so 911/100 is less than 990/100, which ... ``` For a short prompt the output is reproduced **token-for-token identically to the Hugging Face reference** (see *Numerical parity* below), e.g. `-p "hello"` (f16 GGUF): `[unused16]Okay, the user just said "hello". That's a simple greeting. I should respond politely and warmly. Maybe they're just starting a conversation`. ## Speed when you actually use it (measured) Test environment — the resources **actually usable inside the container (docker)**: | Resource | Value | |---|---| | CPU | Intel Xeon Platinum 8559C | | Effective cores | **16 vCPU** (cgroup `cpu.max` = `1600000 100000`) | | Memory (cgroup limit) | **≈ 97 GiB** | | GPU | **none usable** — `/dev/nvidia*` nodes exist but `cuInit()` returns `CUDA_ERROR_NO_DEVICE` and NVML init fails | | Build / runtime | CPU-only, `-t 16` | These are **end-user speeds** measured from real runs of the CLI on `pangu-diffusion-Q4_K_M.gguf` (4.59 GiB, 8.03 B params), i.e. what you actually experience while using it: | What the user experiences | Mode | Speed | |---|---|---| | **Input** — waiting for the prompt to be read before output starts | both | **≈ 78 t/s** short prompt; **≈ 132 t/s** for longer prompts (`llama-bench` pp512) | | **Output** — text generated (autoregressive, streamed live) | `llama-cli` | **11.9 t/s** (llama-bench tg128: 11.1 t/s) | | **Output** — block diffusion (quality: block 32 / small-block 4, `alg=entropy`, `temp=0`) | `llama-diffusion-cli` | **≈ 4.4 t/s** | | **Output** — block diffusion (fast: `--diffusion-confidence-threshold 0.9`, small-block 32, `alg=confidence` — the official throughput setting) | `llama-diffusion-cli` | **≈ 5 t/s** | Notes: - Input (prompt processing) is the same causal prefill for both paths; it is what you wait for before the answer appears. - The AR output speed (≈ 12 t/s) is what `llama-cli` prints as `Generation: X t/s`; the diffused text is only flushed at the end of a run (unless `--diffusion-visual`). - Block diffusion normally reveals one token per denoising step, i.e. up to `block_length` (= 32) forward passes per block. With `--diffusion-confidence-threshold T` every masked token whose confidence exceeds `T` is revealed in the same step (`transfer = argmax(conf) | (conf > T)`), which raises throughput — the same mechanism the authors report as giving up to 2.5× over autoregressive decoding on Ascend NPUs. A lower `T` is faster but less accurate; measured on this CPU (235 tokens, block 32 / small-block 32, `alg=confidence`, `temp=0`): | `--diffusion-confidence-threshold` | speed | |---|---| | `0.9` (default) | 4.9 t/s | | `0.7` | 7.4 t/s | | `0.5` | 8.7 t/s | - The AR figures are the best reference for raw model throughput on this hardware. ## What the patch does 1. **`src/llama-arch.cpp`** — mark `LLM_ARCH_PANGU_EMBED` as a diffusion architecture (`llm_arch_is_diffusion()`), so `llama-diffusion-cli` accepts the model. 2. **`examples/diffusion/diffusion.{h,cpp}`** — add a *context-causal block-diffusion* decoder (`diffusion.block_attention`): - prefill the prompt with a causal forward (kept in the KV cache); - denoise each block with `causal_attn = false` → full attention over the cached context **plus** the current block (the KV-cache kq mask keeps every non-empty cached cell when non-causal); - re-decode the block each step, dropping its stale keys (`llama_memory_seq_rm`), reveal the most-confident masked token of the current small block, left to right; - commit the block with a causal forward so the KV cache holds correct causal context, then sample the first token of the next block from the last position; - `diffusion_generate_nseq()` generalises this to `n_seq` sequences sharing one micro-batch (see `--diffusion-n-seq`). 3. **`examples/diffusion/diffusion-cli.cpp`** — read `diffusion.block_attention`, add `--diffusion-small-block/-confidence-threshold/-n-seq`, size the context for the batched prefill, and use greedy sampling when `temperature == 0`. 4. **`gguf-py` + `conversion/pangu.py`** — write the new `diffusion.block_attention` metadata key for diffusion checkpoints (alongside `diffusion.shift_logits` and `tokenizer.ggml.mask_token_id`), and set `tokenizer.ggml.add_space_prefix = false` (see *Tokenizer fix* below). ### Model metadata in the GGUF ``` general.architecture = pangu-embedded diffusion.shift_logits = true diffusion.block_attention = true tokenizer.ggml.mask_token_id = 45830 tokenizer.ggml.add_space_prefix = false ``` ### Tokenizer fix The upstream conversion honours `add_prefix_space` from `tokenizer_config.json`, but the slow `PanguTokenizer` (sentencepiece, `legacy: true`) does **not** actually prepend a space. With the flag left on, llama.cpp inserted spurious space tokens after every special token (`[unused9]系统：…` became `[unused9]␠系统：…`), which changed the input and the generation. The patch forces `tokenizer.ggml.add_space_prefix = false`; the tokenizer output then matches the reference exactly (a 22-token prompt becomes 22 tokens, was 37). ### Numerical parity Validated against the Hugging Face reference (`inference/generation_utils.py`, `alg=confidence`, `num_small_blocks=1`, `threshold=0.9`, `temperature=0`) on the **f16** GGUF: for `-p "hello"` the generated token IDs are **identical, token for token**, to the reference. (The earlier mismatch was entirely the tokenizer issue above, not the decoder.) ## Batched decoding (`--diffusion-n-seq`) `--diffusion-n-seq N` runs N independent sequences through the block decoder inside a single (micro-)batch. The example CLI replicates the prompt, so this is mainly a way to measure batching; a real deployment would fill the rows with different prompts. All sequences produce **byte-identical output** to the single-sequence run (verified for N = 4 and N = 8), i.e. the batching is correct. Measured on the CPU above (Q4_K_M, block 32 / small-block 32, `threshold 0.9`, greedy, 235 generated tokens): | `--diffusion-n-seq` | wall time | aggregate | per sequence | |---|---|---|---| | 1 | 44.5 s | 5.28 t/s | 5.28 t/s | | 4 | 162.4 s | 5.79 t/s | 1.45 t/s | | 8 | 321.2 s | 5.85 t/s | 0.73 t/s | **Review / honest verdict:** on this **CPU** batching gives essentially *no* throughput gain — aggregate speed stays flat (≈5.3–5.9 t/s) and the time grows ~linearly with `N`, so per-sequence speed drops 1/N. That is expected: a CPU decode step is compute-bound and widens ~linearly with `n_seq` (there are no tensor cores to fill and no memory-bandwidth headroom to exploit), so the fixed per-step work simply multiplies. Batching is where the official 2.5× over autoregressive decoding comes from **on Ascend NPUs/GPUs**, not here. We kept it because it is correct, cheap to carry, and a prerequisite for any GPU deployment — but do not expect a CPU speed-up from it. ## Notes / limitations - CPU-only, `-t 16`, ≈ 4.6 GB file: a 256-token block-diffusion run takes tens of seconds to a few minutes (`block_length` forwards per block). Throughput can be improved with `alg = confidence_threshold, num_small_blocks = 1` (see the official README). - The decoder reproduces the Hugging Face reference **token-for-token** on the f16 GGUF (see *Numerical parity*), and the block-diffusion path was cross-checked against the reference logits (`cos ≈ 0.996`, f16 vs bf16). - **Quantization + the greedy fast mode can change the reasoning**: on the adversarial `9.11 vs 9.9` prompt the Q4_K_M fast mode first states the wrong intermediate answer and only corrects itself from the fraction comparison above, whereas the f16 model matches the reference. Use the f16 GGUF, or `alg = entropy`, `temp = 0`, when a case must be exactly reproduced. ## Supported features Implemented (matching the reference `inference/generation_utils.py`): | Feature | Status | |---|---| | Context-causal block diffusion — full attention inside a block, causal attention over the preceding context | ✓ | | Block-by-block decoding (block length 32) | ✓ `--diffusion-block-length` | | Committed-block KV cache with a causal mask; first token of the next block decoded from the last position | ✓ | | Small blocks inside a block (`num_small_blocks`, optimum 4 or 8) | ✓ `--diffusion-small-block` | | Variable-length inference / flexible context | ✓ | | Logit shift (`diffusion.shift_logits`) | ✓ | | Dedicated mask token (`tokenizer.ggml.mask_token_id = 45830`) | ✓ | | Reveal order: `confidence`, `entropy`, `topk_margin` | ✓ `--diffusion-algorithm 4/1/2` | | `confidence_threshold` fast mode (multi-token reveal per step) | ✓ `--diffusion-confidence-threshold` | | EOS early stop | ✓ | | Batched (multi-sequence) block diffusion | ✓ `--diffusion-n-seq` | | Autoregressive decoding | ✓ `llama-cli` | Not implemented: - Vendor NPU kernels — this uses whatever backend llama.cpp was built with (CPU here). ## Original model - Weights: [`openpangu/openPangu-R-7B-Diffusion`](https://huggingface.co/openpangu/openPangu-R-7B-Diffusion) - Architecture details, benchmarks and citation: see the original model card.