| # llama.cpp |
|
|
| llama.cpp is the best backend in two important scenarios: |
|
|
| 1) You don't have a GPU. |
| 2) You want to run a model that doesn't fit into your GPU. |
|
|
| ## Setting up the models |
|
|
| #### Pre-converted |
|
|
| Download the GGUF or GGML models directly into your `text-generation-webui/models` folder. It will be a single file. |
|
|
| * For GGUF models, make sure its name contains `.gguf`. |
| * For GGML models, make sure its name contains `ggml` and ends in `.bin`. |
|
|
| `q4_K_M` quantization is recommended. |
|
|
| #### Convert Llama yourself |
|
|
| Follow the instructions in the llama.cpp README to generate a ggml: https://github.com/ggerganov/llama.cpp#prepare-data--run |
|
|
| ## GPU acceleration |
|
|
| Enabled with the `--n-gpu-layers` parameter. |
|
|
| * If you have enough VRAM, use a high number like `--n-gpu-layers 1000` to offload all layers to the GPU. |
| * Otherwise, start with a low number like `--n-gpu-layers 10` and then gradually increase it until you run out of memory. |
|
|
| This feature works out of the box for NVIDIA GPUs on Linux (amd64) or Windows. For other GPUs, you need to uninstall `llama-cpp-python` with |
|
|
| ``` |
| pip uninstall -y llama-cpp-python |
| ``` |
|
|
| and then recompile it using the commands here: https://pypi.org/project/llama-cpp-python/ |
|
|
| #### macOS |
|
|
| For macOS, these are the commands: |
|
|
| ``` |
| pip uninstall -y llama-cpp-python |
| CMAKE_ARGS="-DLLAMA_METAL=on" FORCE_CMAKE=1 pip install llama-cpp-python --no-cache-dir |
| ``` |
|
|
