inference-endpoints/pr_151/en/engines/llama_cpp.md

llama.cpp

llama.cpp is a high-performance inference engine written in C/C++, tailored for running Llama and compatible models in the GGUF format.

Core features:

• GGUF Model Support: Native compatibility with the GGUF format and all quantization types that comes with it.
• Multi-Platform: Optimized for both CPU and GPU execution, with support for AVX, AVX2, AVX512, and CUDA acceleration.
• OpenAI-Compatible API: Provides endpoints for chat, completion, embedding, and more, enabling seamless integration with existing tools and workflows.
• Active Community and Ecosystem: Rapid development and a rich ecosystem of tools, extensions, and integrations

When you create an endpoint with a GGUF model, a llama.cpp container is automatically selected using the latest image built from the master branch of the llama.cpp repository. Upon successful deployment, a server with an OpenAI-compatible endpoint becomes available.

llama.cpp supports multiple endpoints like /tokenize, /health, /embedding, and many more. For a comprehensive list of available endpoints, please refer to the API documentation.

Deployment Steps

To deploy an endpoint with a llama.cpp container, follow these steps:

1. Create a new endpoint and select a repository containing a GGUF model. The llama.cpp container will be automatically selected.

2. Choose the desired GGUF file, noting that memory requirements will vary depending on the selected file. For example, an F16 model requires more memory than a Q4_K_M model.

3. Select your desired hardware configuration.

4. Optionally, you can customize the container's configuration settings like Max Tokens, Number of Concurrent Requests. For more information on those, please refer to the Configurations section below.

5. Click the Create Endpoint button to complete the deployment.

Alternatively, you can follow the video tutorial below for a step-by-step guide on deploying an endpoint with a llama.cpp container: