File size: 1,592 Bytes
f17c411 6823459 7d073cb 6823459 0dc6c0d 6823459 0dc6c0d 6823459 756baca 21f3da7 756baca 6823459 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 | ---
license: mit
---
# gguf-cpp
One package for working with GGUF models locally: an OpenAI-compatible LLM
server, a diffusion image/video generator and a GGUF metadata/tensor editor
with a built-in quantizer — three panels on one GUI, powered by one unified
**gguf.cpp** engine compiled in a single build with a shared set of ggml
kernels.
## Install
```bash
pip install gguf-cpp
```
The build compiles the bundled engine (CPU by default, Metal on macOS).
GPU backends are opt-in at install time:
```bash
GGUF_CPP_CUDA=1 pip install gguf-cpp # NVIDIA
GGUF_CPP_HIP=1 pip install gguf-cpp # AMD ROCm
GGUF_CPP_VULKAN=1 pip install gguf-cpp # Vulkan
```
## Run
```bash
gguf-cpp # unified GUI — Server / Diffuser / Editor panels
python -m gguf_cpp # same thing
```
Each panel also runs on its own, exactly like the standalone
gguf-server / gguf-diffusion / gguf-editor packages did:
```bash
gguf-cpp server # LLM server GUI
gguf-cpp diffuser # image/video generation GUI
gguf-cpp editor # GGUF editor GUI
```
And the engines are directly scriptable from the CLI:
```bash
gguf-cpp server engine -- --model model.gguf --port 8888
gguf-cpp diffuser engine -- -m sd.gguf -p "a lighthouse at dusk" -o out.png
gguf-cpp editor quantize -m in.gguf -o out-q4_k.gguf --type q4_k
gguf-cpp editor devices
```
or run it with `gguf-connector`
```
ggc gp
```
 |