Instructions to use HelloSun/openPangu-R-7B-Diffusion-Q4KM with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use HelloSun/openPangu-R-7B-Diffusion-Q4KM with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf HelloSun/openPangu-R-7B-Diffusion-Q4KM:Q4_K_M # Run inference directly in the terminal: llama cli -hf HelloSun/openPangu-R-7B-Diffusion-Q4KM:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf HelloSun/openPangu-R-7B-Diffusion-Q4KM:Q4_K_M # Run inference directly in the terminal: llama cli -hf HelloSun/openPangu-R-7B-Diffusion-Q4KM:Q4_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf HelloSun/openPangu-R-7B-Diffusion-Q4KM:Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf HelloSun/openPangu-R-7B-Diffusion-Q4KM:Q4_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf HelloSun/openPangu-R-7B-Diffusion-Q4KM:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf HelloSun/openPangu-R-7B-Diffusion-Q4KM:Q4_K_M
Use Docker
docker model run hf.co/HelloSun/openPangu-R-7B-Diffusion-Q4KM:Q4_K_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use HelloSun/openPangu-R-7B-Diffusion-Q4KM with Ollama:
ollama run hf.co/HelloSun/openPangu-R-7B-Diffusion-Q4KM:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use HelloSun/openPangu-R-7B-Diffusion-Q4KM with Docker Model Runner:
docker model run hf.co/HelloSun/openPangu-R-7B-Diffusion-Q4KM:Q4_K_M
- Lemonade
How to use HelloSun/openPangu-R-7B-Diffusion-Q4KM with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull HelloSun/openPangu-R-7B-Diffusion-Q4KM:Q4_K_M
Run and chat with the model
lemonade run user.openPangu-R-7B-Diffusion-Q4KM-Q4_K_M
List all available models
lemonade list
- Atomic Chat
Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -119,6 +119,34 @@ The first digit after the decimal is 1 for 9.11 and 9 for 9.90. Since 9 > 1,
|
|
| 119 |
So, 9.9 is bigger than 9.11.
|
| 120 |
```
|
| 121 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 122 |
## What the patch does
|
| 123 |
|
| 124 |
1. **`src/llama-arch.cpp`** — mark `LLM_ARCH_PANGU_EMBED` as a diffusion architecture
|
|
|
|
| 119 |
So, 9.9 is bigger than 9.11.
|
| 120 |
```
|
| 121 |
|
| 122 |
+
## Performance (measured)
|
| 123 |
+
|
| 124 |
+
Test environment — the resources **actually usable inside the container (docker)**:
|
| 125 |
+
|
| 126 |
+
| Resource | Value |
|
| 127 |
+
|---|---|
|
| 128 |
+
| CPU | Intel Xeon Platinum 8559C |
|
| 129 |
+
| Effective cores | **16 vCPU** (cgroup `cpu.max` = `1600000 100000`) |
|
| 130 |
+
| Memory (cgroup limit) | **≈ 97 GiB** |
|
| 131 |
+
| GPU | **none usable** — `/dev/nvidia*` nodes exist but `cuInit()` returns `CUDA_ERROR_NO_DEVICE` and NVML init fails |
|
| 132 |
+
| Build / runtime | CPU-only, `-t 16` |
|
| 133 |
+
|
| 134 |
+
Measured on `pangu-diffusion-Q4_K_M.gguf` (4.59 GiB, 8.03 B params):
|
| 135 |
+
|
| 136 |
+
| Mode | Benchmark | Speed |
|
| 137 |
+
|---|---|---|
|
| 138 |
+
| **Input** — prompt processing (causal prefill) | `llama-bench` pp512 | **131.8 t/s** (131.77 ± 3.34) |
|
| 139 |
+
| **Output** — autoregressive decoding | `llama-bench` tg128 | **11.1 t/s** (11.11 ± 0.35) |
|
| 140 |
+
| **Output** — block diffusion (block 32 / small-block 4, `alg=entropy`, `temp=0`) | 224 tokens | **≈ 4.4 t/s** (≈ 55 s end-to-end) |
|
| 141 |
+
|
| 142 |
+
Notes:
|
| 143 |
+
- The input/prompt-processing number is the same causal prefill used by both the AR and the
|
| 144 |
+
diffusion path.
|
| 145 |
+
- Block diffusion reveals one token per denoising step, i.e. it runs up to `block_length`
|
| 146 |
+
(= 32) extra forward passes per block, so its tokens/s is inherently lower than AR.
|
| 147 |
+
- The AR figures (`pp512` / `tg128`) are the best reference for raw model throughput on
|
| 148 |
+
this hardware.
|
| 149 |
+
|
| 150 |
## What the patch does
|
| 151 |
|
| 152 |
1. **`src/llama-arch.cpp`** — mark `LLM_ARCH_PANGU_EMBED` as a diffusion architecture
|