| --- |
| license: mit |
| --- |
| # gguf-cpp |
|
|
| One package for working with GGUF models locally: an OpenAI-compatible LLM |
| server, a diffusion image/video generator and a GGUF metadata/tensor editor |
| with a built-in quantizer — three panels on one GUI, powered by one unified |
| **gguf.cpp** engine compiled in a single build with a shared set of ggml |
| kernels. |
|
|
| ## Install |
|
|
| ```bash |
| pip install gguf-cpp |
| ``` |
|
|
| The build compiles the bundled engine (CPU by default, Metal on macOS). |
| GPU backends are opt-in at install time: |
|
|
| ```bash |
| GGUF_CPP_CUDA=1 pip install gguf-cpp # NVIDIA |
| GGUF_CPP_HIP=1 pip install gguf-cpp # AMD ROCm |
| GGUF_CPP_VULKAN=1 pip install gguf-cpp # Vulkan |
| ``` |
|
|
| ## Run |
|
|
| ```bash |
| gguf-cpp # unified GUI — Server / Diffuser / Editor panels |
| python -m gguf_cpp # same thing |
| ``` |
|
|
|  |
|
|
| Each panel also runs on its own, exactly like the standalone |
| gguf-server / gguf-diffusion / gguf-editor packages did: |
|
|
| ```bash |
| gguf-cpp server # LLM server GUI |
| gguf-cpp diffuser # image/video generation GUI |
| gguf-cpp editor # GGUF editor GUI |
| ``` |
|
|
| And the engines are directly scriptable from the CLI: |
|
|
| ```bash |
| gguf-cpp server engine -- --model model.gguf --port 8888 |
| gguf-cpp diffuser engine -- -m sd.gguf -p "a lighthouse at dusk" -o out.png |
| gguf-cpp editor quantize -m in.gguf -o out-q4_k.gguf --type q4_k |
| gguf-cpp editor devices |
| ``` |
|
|
| or run it with `gguf-connector` |
| ``` |
| ggc gp |
| ``` |
|
|
|  |
