Instructions to use HelloSun/openPangu-R-7B-Diffusion-Q4KM with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use HelloSun/openPangu-R-7B-Diffusion-Q4KM with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf HelloSun/openPangu-R-7B-Diffusion-Q4KM:Q4_K_M # Run inference directly in the terminal: llama cli -hf HelloSun/openPangu-R-7B-Diffusion-Q4KM:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf HelloSun/openPangu-R-7B-Diffusion-Q4KM:Q4_K_M # Run inference directly in the terminal: llama cli -hf HelloSun/openPangu-R-7B-Diffusion-Q4KM:Q4_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf HelloSun/openPangu-R-7B-Diffusion-Q4KM:Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf HelloSun/openPangu-R-7B-Diffusion-Q4KM:Q4_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf HelloSun/openPangu-R-7B-Diffusion-Q4KM:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf HelloSun/openPangu-R-7B-Diffusion-Q4KM:Q4_K_M
Use Docker
docker model run hf.co/HelloSun/openPangu-R-7B-Diffusion-Q4KM:Q4_K_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use HelloSun/openPangu-R-7B-Diffusion-Q4KM with Ollama:
ollama run hf.co/HelloSun/openPangu-R-7B-Diffusion-Q4KM:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use HelloSun/openPangu-R-7B-Diffusion-Q4KM with Docker Model Runner:
docker model run hf.co/HelloSun/openPangu-R-7B-Diffusion-Q4KM:Q4_K_M
- Lemonade
How to use HelloSun/openPangu-R-7B-Diffusion-Q4KM with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull HelloSun/openPangu-R-7B-Diffusion-Q4KM:Q4_K_M
Run and chat with the model
lemonade run user.openPangu-R-7B-Diffusion-Q4KM-Q4_K_M
List all available models
lemonade list
- Atomic Chat
Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -119,7 +119,7 @@ The first digit after the decimal is 1 for 9.11 and 9 for 9.90. Since 9 > 1,
|
|
| 119 |
So, 9.9 is bigger than 9.11.
|
| 120 |
```
|
| 121 |
|
| 122 |
-
##
|
| 123 |
|
| 124 |
Test environment — the resources **actually usable inside the container (docker)**:
|
| 125 |
|
|
@@ -131,21 +131,25 @@ Test environment — the resources **actually usable inside the container (docke
|
|
| 131 |
| GPU | **none usable** — `/dev/nvidia*` nodes exist but `cuInit()` returns `CUDA_ERROR_NO_DEVICE` and NVML init fails |
|
| 132 |
| Build / runtime | CPU-only, `-t 16` |
|
| 133 |
|
| 134 |
-
|
|
|
|
|
|
|
| 135 |
|
| 136 |
-
|
|
| 137 |
|---|---|---|
|
| 138 |
-
| **Input** — prompt
|
| 139 |
-
| **Output** — autoregressive
|
| 140 |
-
| **Output** — block diffusion
|
| 141 |
|
| 142 |
Notes:
|
| 143 |
-
-
|
| 144 |
-
|
| 145 |
-
-
|
| 146 |
-
|
| 147 |
-
-
|
| 148 |
-
|
|
|
|
|
|
|
| 149 |
|
| 150 |
## What the patch does
|
| 151 |
|
|
|
|
| 119 |
So, 9.9 is bigger than 9.11.
|
| 120 |
```
|
| 121 |
|
| 122 |
+
## Speed when you actually use it (measured)
|
| 123 |
|
| 124 |
Test environment — the resources **actually usable inside the container (docker)**:
|
| 125 |
|
|
|
|
| 131 |
| GPU | **none usable** — `/dev/nvidia*` nodes exist but `cuInit()` returns `CUDA_ERROR_NO_DEVICE` and NVML init fails |
|
| 132 |
| Build / runtime | CPU-only, `-t 16` |
|
| 133 |
|
| 134 |
+
These are **end-user speeds** measured from real runs of the CLI on
|
| 135 |
+
`pangu-diffusion-Q4_K_M.gguf` (4.59 GiB, 8.03 B params), i.e. what you actually
|
| 136 |
+
experience while using it:
|
| 137 |
|
| 138 |
+
| What the user experiences | Mode | Speed |
|
| 139 |
|---|---|---|
|
| 140 |
+
| **Input** — waiting for the prompt to be read before output starts | both | **≈ 78 t/s** short prompt; **≈ 132 t/s** for longer prompts (`llama-bench` pp512) |
|
| 141 |
+
| **Output** — text generated (autoregressive, streamed live) | `llama-cli` | **11.9 t/s** (llama-bench tg128: 11.1 t/s) |
|
| 142 |
+
| **Output** — text generated (block diffusion, block 32 / small-block 4, `alg=entropy`, `temp=0`) | `llama-diffusion-cli` | **≈ 4.4 t/s** |
|
| 143 |
|
| 144 |
Notes:
|
| 145 |
+
- Input (prompt processing) is the same causal prefill for both paths; it is what you wait
|
| 146 |
+
for before the answer appears.
|
| 147 |
+
- The AR output speed (≈ 12 t/s) is what `llama-cli` prints as `Generation: X t/s`; the
|
| 148 |
+
diffused text is only flushed at the end of a run (unless `--diffusion-visual`).
|
| 149 |
+
- Block diffusion reveals one token per denoising step, i.e. up to `block_length` (= 32)
|
| 150 |
+
forward passes per block, so its tokens/s is inherently lower. The authors' faster
|
| 151 |
+
`alg=confidence_threshold` mode is not implemented here (fixed 1-token steps).
|
| 152 |
+
- The AR figures are the best reference for raw model throughput on this hardware.
|
| 153 |
|
| 154 |
## What the patch does
|
| 155 |
|