Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -152,7 +152,7 @@ experience while using it:
|
|
| 152 |
| **Input** — waiting for the prompt to be read before output starts | both | **≈ 78 t/s** short prompt; **≈ 132 t/s** for longer prompts (`llama-bench` pp512) |
|
| 153 |
| **Output** — text generated (autoregressive, streamed live) | `llama-cli` | **11.9 t/s** (llama-bench tg128: 11.1 t/s) |
|
| 154 |
| **Output** — block diffusion (quality: block 32 / small-block 4, `alg=entropy`, `temp=0`) | `llama-diffusion-cli` | **≈ 4.4 t/s** |
|
| 155 |
-
| **Output** — block diffusion (fast: `--diffusion-confidence-threshold 0.9`, small-block 32, `alg=confidence`) | `llama-diffusion-cli` | **≈ 5
|
| 156 |
|
| 157 |
Notes:
|
| 158 |
- Input (prompt processing) is the same causal prefill for both paths; it is what you wait
|
|
@@ -161,8 +161,18 @@ Notes:
|
|
| 161 |
diffused text is only flushed at the end of a run (unless `--diffusion-visual`).
|
| 162 |
- Block diffusion normally reveals one token per denoising step, i.e. up to `block_length`
|
| 163 |
(= 32) forward passes per block. With `--diffusion-confidence-threshold T` every masked
|
| 164 |
-
token whose confidence exceeds `T` is revealed in the same step
|
| 165 |
-
(
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 166 |
- The AR figures are the best reference for raw model throughput on this hardware.
|
| 167 |
|
| 168 |
## What the patch does
|
|
|
|
| 152 |
| **Input** — waiting for the prompt to be read before output starts | both | **≈ 78 t/s** short prompt; **≈ 132 t/s** for longer prompts (`llama-bench` pp512) |
|
| 153 |
| **Output** — text generated (autoregressive, streamed live) | `llama-cli` | **11.9 t/s** (llama-bench tg128: 11.1 t/s) |
|
| 154 |
| **Output** — block diffusion (quality: block 32 / small-block 4, `alg=entropy`, `temp=0`) | `llama-diffusion-cli` | **≈ 4.4 t/s** |
|
| 155 |
+
| **Output** — block diffusion (fast: `--diffusion-confidence-threshold 0.9`, small-block 32, `alg=confidence` — the official throughput setting) | `llama-diffusion-cli` | **≈ 5 t/s** |
|
| 156 |
|
| 157 |
Notes:
|
| 158 |
- Input (prompt processing) is the same causal prefill for both paths; it is what you wait
|
|
|
|
| 161 |
diffused text is only flushed at the end of a run (unless `--diffusion-visual`).
|
| 162 |
- Block diffusion normally reveals one token per denoising step, i.e. up to `block_length`
|
| 163 |
(= 32) forward passes per block. With `--diffusion-confidence-threshold T` every masked
|
| 164 |
+
token whose confidence exceeds `T` is revealed in the same step (`transfer = argmax(conf)
|
| 165 |
+
| (conf > T)`), which raises throughput — the same mechanism the authors report as giving
|
| 166 |
+
up to 2.5× over autoregressive decoding on Ascend NPUs. A lower `T` is faster but less
|
| 167 |
+
accurate; measured on this CPU (224 tokens, block 32 / small-block 32, `alg=confidence`,
|
| 168 |
+
`temp=0`):
|
| 169 |
+
|
| 170 |
+
| `--diffusion-confidence-threshold` | speed |
|
| 171 |
+
|---|---|
|
| 172 |
+
| `0.9` (default) | 4.9 t/s |
|
| 173 |
+
| `0.7` | 7.4 t/s |
|
| 174 |
+
| `0.5` | 8.7 t/s |
|
| 175 |
+
|
| 176 |
- The AR figures are the best reference for raw model throughput on this hardware.
|
| 177 |
|
| 178 |
## What the patch does
|