Upload README.md with huggingface_hub

README.md

@@ -57,7 +57,8 @@ cmake --build build -j --target llama-diffusion-cli llama-cli llama-quantize
 
 ### 2. Diffusion decoding (recommended)
 
-The chat format is
+`llama-diffusion-cli` applies the model's chat template automatically: pass the **raw
+user message**, not a pre-formatted prompt. The rendered prompt is
 
 ```
 [unused9]系统：<system>[unused10][unused9]用户：<user>[unused10][unused9]助手：
@@ -66,7 +67,7 @@ The chat format is
 ```bash
 ./build/bin/llama-diffusion-cli \
   -m pangu-diffusion-Q4_K_M.gguf \
-  -p "[unused9]系统：[unused10][unused9]用户：Which is bigger, 9.11 or 9.9?[unused10][unused9]助手：" \
+  -p "Which is bigger, 9.11 or 9.9?" \
   -ub 256 \
   --diffusion-block-length 32 \
   --diffusion-small-block 4 \
@@ -85,12 +86,15 @@ threshold in a single denoising step, with one small block per block:
 ```bash
 ./build/bin/llama-diffusion-cli \
   -m pangu-diffusion-Q4_K_M.gguf \
-  -p "[unused9]系统：[unused10][unused9]用户：<question>[unused10][unused9]助手：" \
+  -p "<question>" \
   -ub 256 --diffusion-block-length 32 --diffusion-small-block 32 \
   --diffusion-confidence-threshold 0.9 --diffusion-algorithm 4 \
   --temp 0 -t 16 --seed 42
 ```
 
+`--diffusion-n-seq N` runs N sequences through the block decoder in a single
+(micro-)batch (the prompt is replicated; see *Batched decoding* below).
+
 `--diffusion-algorithm` values: `0` origin, `1` entropy, `2` margin, `3` random, `4` confidence.
 `-ub N` is the maximum sequence length; the tail that does not fill a whole block is dropped.
 
@@ -103,34 +107,31 @@ threshold in a single denoising step, with one small block per block:
 
 ## Example
 
-Prompt (translated): *“Which is bigger, 9.11 or 9.9?”* — command as in §2:
+Prompt (translated): *“Which is bigger, 9.11 or 9.9?”* — command as in §2, fast mode
+(`-ub 256`, `--diffusion-confidence-threshold 0.9`, Q4_K_M):
 
 ```
-# Which is bigger, 9.11 or 9.9?
-
-I need to compare 9.11 and 9.9 to see which one is bigger. Both numbers are
-between 9 and 10, so they're both greater than 9 but less than 10.
+I need to figure out which is bigger between 9.11 and 9.9. Both are decimals, so I should compare them directly.
 
-Let me think about this carefully. 9.11 means 9 and 11 hundredths, which is
-9 + 11/100. Similarly, 9.9 means 9 and 9 tenths, which is 9 + 9/10.
+First, let's write them with the same number of decimal places to make it easier. 9.11 has two decimal places, and 9.9 has one. So, I can write 9.9 as 9.90 to make it two decimal places.
 
-To compare them, I should convert them to the same format. Let me convert both
-to hundredths.
+So, 9.11 vs 9.90.
 
-9.9 = 9.90 (adding a zero to make it 9.90)
+Now, comparing the tenths place: both have 9 in the tenths place. So, that's the same.
 
-Now I have:
-- 9.11
-- 9.90
+Next, the hundredths place: 9.11 has 1 in the hundredths place, and 9.90 has 0 in the hundredths place. Since 1 is greater than 0, ...
 
-Since both have the same number of decimal places, I can compare them directly.
+I can think of them as fractions. 9.11 is 911/100, and 9.9 is 990/100.
 
-The first digit after the decimal is 1 for 9.11 and 9 for 9.90. Since 9 > 1,
-9.90 is larger than 9.11.
-
-So, 9.9 is bigger than 9.11.
+Since the denominator is the same, I can compare the numerators: 911 vs 990.
+911 is less than 990, so 911/100 is less than 990/100, which ...
 ```
 
+For a short prompt the output is reproduced **token-for-token identically to the Hugging
+Face reference** (see *Numerical parity* below), e.g. `-p "hello"` (f16 GGUF):
+`[unused16]Okay, the user just said "hello". That's a simple greeting. I should respond
+politely and warmly. Maybe they're just starting a conversation`.
+
 ## Speed when you actually use it (measured)
 
 Test environment — the resources **actually usable inside the container (docker)**:
@@ -164,7 +165,7 @@ Notes:
   token whose confidence exceeds `T` is revealed in the same step (`transfer = argmax(conf)
   | (conf > T)`), which raises throughput — the same mechanism the authors report as giving
   up to 2.5× over autoregressive decoding on Ascend NPUs. A lower `T` is faster but less
-  accurate; measured on this CPU (224 tokens, block 32 / small-block 32, `alg=confidence`,
+  accurate; measured on this CPU (235 tokens, block 32 / small-block 32, `alg=confidence`,
   `temp=0`):
 
   | `--diffusion-confidence-threshold` | speed |
@@ -188,12 +189,16 @@ Notes:
    - re-decode the block each step, dropping its stale keys (`llama_memory_seq_rm`),
      reveal the most-confident masked token of the current small block, left to right;
    - commit the block with a causal forward so the KV cache holds correct causal
-     context, then sample the first token of the next block from the last position.
-3. **`examples/diffusion/diffusion-cli.cpp`** — read `diffusion.block_attention`,
-   add `--diffusion-small-block`, use greedy sampling when `temperature == 0`.
+     context, then sample the first token of the next block from the last position;
+   - `diffusion_generate_nseq()` generalises this to `n_seq` sequences sharing one
+     micro-batch (see `--diffusion-n-seq`).
+3. **`examples/diffusion/diffusion-cli.cpp`** — read `diffusion.block_attention`, add
+   `--diffusion-small-block/-confidence-threshold/-n-seq`, size the context for the
+   batched prefill, and use greedy sampling when `temperature == 0`.
 4. **`gguf-py` + `conversion/pangu.py`** — write the new `diffusion.block_attention`
    metadata key for diffusion checkpoints (alongside `diffusion.shift_logits` and
-   `tokenizer.ggml.mask_token_id`).
+   `tokenizer.ggml.mask_token_id`), and set `tokenizer.ggml.add_space_prefix = false`
+   (see *Tokenizer fix* below).
 
 ### Model metadata in the GGUF
 
@@ -202,18 +207,66 @@ general.architecture            = pangu-embedded
 diffusion.shift_logits          = true
 diffusion.block_attention       = true
 tokenizer.ggml.mask_token_id    = 45830
+tokenizer.ggml.add_space_prefix = false
 ```
 
+### Tokenizer fix
+
+The upstream conversion honours `add_prefix_space` from `tokenizer_config.json`, but the
+slow `PanguTokenizer` (sentencepiece, `legacy: true`) does **not** actually prepend a space.
+With the flag left on, llama.cpp inserted spurious space tokens after every special token
+(`[unused9]系统：…` became `[unused9]␠系统：…`), which changed the input and the generation.
+The patch forces `tokenizer.ggml.add_space_prefix = false`; the tokenizer output then
+matches the reference exactly (a 22-token prompt becomes 22 tokens, was 37).
+
+### Numerical parity
+
+Validated against the Hugging Face reference (`inference/generation_utils.py`,
+`alg=confidence`, `num_small_blocks=1`, `threshold=0.9`, `temperature=0`) on the **f16**
+GGUF: for `-p "hello"` the generated token IDs are **identical, token for token**, to the
+reference. (The earlier mismatch was entirely the tokenizer issue above, not the
+decoder.)
+
+## Batched decoding (`--diffusion-n-seq`)
+
+`--diffusion-n-seq N` runs N independent sequences through the block decoder inside a
+single (micro-)batch. The example CLI replicates the prompt, so this is mainly a way to
+measure batching; a real deployment would fill the rows with different prompts.
+
+All sequences produce **byte-identical output** to the single-sequence run (verified for
+N = 4 and N = 8), i.e. the batching is correct.
+
+Measured on the CPU above (Q4_K_M, block 32 / small-block 32, `threshold 0.9`, greedy,
+235 generated tokens):
+
+| `--diffusion-n-seq` | wall time | aggregate | per sequence |
+|---|---|---|---|
+| 1 | 44.5 s | 5.28 t/s | 5.28 t/s |
+| 4 | 162.4 s | 5.79 t/s | 1.45 t/s |
+| 8 | 321.2 s | 5.85 t/s | 0.73 t/s |
+
+**Review / honest verdict:** on this **CPU** batching gives essentially *no* throughput
+gain — aggregate speed stays flat (≈5.3–5.9 t/s) and the time grows ~linearly with `N`,
+so per-sequence speed drops 1/N. That is expected: a CPU decode step is compute-bound and
+widens ~linearly with `n_seq` (there are no tensor cores to fill and no memory-bandwidth
+headroom to exploit), so the fixed per-step work simply multiplies. Batching is where the
+official 2.5× over autoregressive decoding comes from **on Ascend NPUs/GPUs**, not here.
+We kept it because it is correct, cheap to carry, and a prerequisite for any GPU
+deployment — but do not expect a CPU speed-up from it.
+
 ## Notes / limitations
 
-- CPU-only, `-t 16`, ≈ 4.6 GB file: a 256-token block-diffusion run takes a few minutes
-  (`block_length` forwards per block). Throughput can be improved with
+- CPU-only, `-t 16`, ≈ 4.6 GB file: a 256-token block-diffusion run takes tens of seconds
+  to a few minutes (`block_length` forwards per block). Throughput can be improved with
   `alg = confidence_threshold, num_small_blocks = 1` (see the official README).
-- The committed-block KV-cache path was numerically cross-checked against the
-  Hugging Face reference (`cos ≈ 0.996`, f16 vs bf16), and the same binaries reproduce
-  the model's reference-quality output above.
-- Quantization may slightly affect the tricky reasoning cases; use `alg = entropy`,
-  `temp = 0` for the most stable results.
+- The decoder reproduces the Hugging Face reference **token-for-token** on the f16 GGUF
+  (see *Numerical parity*), and the block-diffusion path was cross-checked against the
+  reference logits (`cos ≈ 0.996`, f16 vs bf16).
+- **Quantization + the greedy fast mode can change the reasoning**: on the adversarial
+  `9.11 vs 9.9` prompt the Q4_K_M fast mode first states the wrong intermediate answer and
+  only corrects itself from the fraction comparison above, whereas the f16 model matches
+  the reference. Use the f16 GGUF, or `alg = entropy`, `temp = 0`, when a case must be
+  exactly reproduced.
 
 ## Supported features
 
@@ -231,10 +284,10 @@ Implemented (matching the reference `inference/generation_utils.py`):
 | Reveal order: `confidence`, `entropy`, `topk_margin` | ✓ `--diffusion-algorithm 4/1/2` |
 | `confidence_threshold` fast mode (multi-token reveal per step) | ✓ `--diffusion-confidence-threshold` |
 | EOS early stop | ✓ |
+| Batched (multi-sequence) block diffusion | ✓ `--diffusion-n-seq` |
 | Autoregressive decoding | ✓ `llama-cli` |
 
 Not implemented:
-- Batched (multi-sequence) block diffusion — the example CLI runs a single sequence.
 - Vendor NPU kernels — this uses whatever backend llama.cpp was built with (CPU here).
 
 ## Original model