Buckets:
| Metadata-Version: 2.1 | |
| Name: llama_cpp_python | |
| Version: 0.3.28 | |
| Summary: Python bindings for the llama.cpp library | |
| Author-Email: Andrei Betlen <abetlen@gmail.com> | |
| License: MIT | |
| Classifier: Programming Language :: Python :: 3 | |
| Classifier: Programming Language :: Python :: 3.8 | |
| Classifier: Programming Language :: Python :: 3.9 | |
| Classifier: Programming Language :: Python :: 3.10 | |
| Classifier: Programming Language :: Python :: 3.11 | |
| Classifier: Programming Language :: Python :: 3.12 | |
| Classifier: Programming Language :: Python :: 3.13 | |
| Classifier: Programming Language :: Python :: 3.14 | |
| Project-URL: Homepage, https://github.com/abetlen/llama-cpp-python | |
| Project-URL: Issues, https://github.com/abetlen/llama-cpp-python/issues | |
| Project-URL: Documentation, https://llama-cpp-python.readthedocs.io/en/latest/ | |
| Project-URL: Changelog, https://llama-cpp-python.readthedocs.io/en/latest/changelog/ | |
| Requires-Python: >=3.8 | |
| Requires-Dist: typing-extensions>=4.5.0 | |
| Requires-Dist: numpy>=1.20.0 | |
| Requires-Dist: diskcache>=5.6.1 | |
| Requires-Dist: jinja2>=2.11.3 | |
| Provides-Extra: server | |
| Requires-Dist: uvicorn>=0.22.0; extra == "server" | |
| Requires-Dist: fastapi>=0.100.0; extra == "server" | |
| Requires-Dist: pydantic-settings>=2.0.1; extra == "server" | |
| Requires-Dist: sse-starlette>=1.6.1; extra == "server" | |
| Requires-Dist: starlette-context<0.4,>=0.3.6; extra == "server" | |
| Requires-Dist: PyYAML>=5.1; extra == "server" | |
| Provides-Extra: test | |
| Requires-Dist: pytest>=7.4.0; extra == "test" | |
| Requires-Dist: httpx>=0.24.1; extra == "test" | |
| Requires-Dist: scipy>=1.10; extra == "test" | |
| Requires-Dist: fastapi>=0.100.0; extra == "test" | |
| Requires-Dist: sse-starlette>=1.6.1; extra == "test" | |
| Requires-Dist: starlette-context<0.4,>=0.3.6; extra == "test" | |
| Requires-Dist: pydantic-settings>=2.0.1; extra == "test" | |
| Requires-Dist: huggingface-hub>=0.23.0; extra == "test" | |
| Provides-Extra: dev | |
| Requires-Dist: ruff>=0.15.7; extra == "dev" | |
| Requires-Dist: twine>=4.0.2; extra == "dev" | |
| Requires-Dist: mkdocs>=1.4.3; extra == "dev" | |
| Requires-Dist: mkdocstrings[python]>=0.22.0; extra == "dev" | |
| Requires-Dist: mkdocs-material>=9.1.18; extra == "dev" | |
| Requires-Dist: pytest>=7.4.0; extra == "dev" | |
| Requires-Dist: httpx>=0.24.1; extra == "dev" | |
| Provides-Extra: all | |
| Requires-Dist: llama_cpp_python[dev,server,test]; extra == "all" | |
| Description-Content-Type: text/markdown | |
| <p align="center"> | |
| <img src="https://raw.githubusercontent.com/abetlen/llama-cpp-python/main/docs/icon.svg" style="height: 5rem; width: 5rem"> | |
| </p> | |
| # Python Bindings for [`llama.cpp`](https://github.com/ggerganov/llama.cpp) | |
| [](https://llama-cpp-python.readthedocs.io/en/latest/?badge=latest) | |
| ) | |
| ``` | |
| Chat completion is available through the | |
| } | |
| } | |
| }], | |
| tool_choice={ | |
| "type": "function", | |
| "function": { | |
| "name": "UserDetail" | |
| } | |
| } | |
| ) | |
| ``` | |
| <details> | |
| <summary>Functionary v2</summary> | |
| The various gguf-converted files for this set of models can be found [here](https://huggingface.co/meetkai). Functionary is able to intelligently call functions and also analyze any provided function outputs to generate coherent responses. All v2 models of functionary supports **parallel function calling**. You can provide either `functionary-v1` or `functionary-v2` for the `chat_format` when initializing the Llama class. | |
| Due to discrepancies between llama.cpp and HuggingFace's tokenizers, it is required to provide HF Tokenizer for functionary. The `LlamaHFTokenizer` class can be initialized and passed into the Llama class. This will override the default llama.cpp tokenizer used in Llama class. The tokenizer files are already included in the respective HF repositories hosting the gguf files. | |
| ```python | |
| from llama_cpp import Llama | |
| from llama_cpp.llama_tokenizer import LlamaHFTokenizer | |
| llm = Llama.from_pretrained( | |
| repo_id="meetkai/functionary-small-v2.2-GGUF", | |
| filename="functionary-small-v2.2.q4_0.gguf", | |
| chat_format="functionary-v2", | |
| tokenizer=LlamaHFTokenizer.from_pretrained("meetkai/functionary-small-v2.2-GGUF") | |
| ) | |
| ``` | |
| **NOTE**: There is no need to provide the default system messages used in Functionary as they are added automatically in the Functionary chat handler. Thus, the messages should contain just the chat messages and/or system messages that provide additional context for the model (e.g.: datetime, etc.). | |
| </details> | |
| ### Multi-modal Models | |
| `llama-cpp-python` supports such as llava1.5 which allow the language model to read information from both text and images. | |
| Below are the supported multi-modal models and their respective chat handlers (Python API) and chat formats (Server API). | |
| | Model | `LlamaChatHandler` | `chat_format` | | |
| |:--- |:--- |:--- | | |
| | [llava-v1.5-7b](https://huggingface.co/mys/ggml_llava-v1.5-7b) | `Llava15ChatHandler` | `llava-1-5` | | |
| | [llava-v1.5-13b](https://huggingface.co/mys/ggml_llava-v1.5-13b) | `Llava15ChatHandler` | `llava-1-5` | | |
| | [llava-v1.6-34b](https://huggingface.co/cjpais/llava-v1.6-34B-gguf) | `Llava16ChatHandler` | `llava-1-6` | | |
| | [moondream2](https://huggingface.co/vikhyatk/moondream2) | `MoondreamChatHandler` | `moondream2` | | |
| | [nanollava](https://huggingface.co/abetlen/nanollava-gguf) | `NanoLlavaChatHandler` | `nanollava` | | |
| | [llama-3-vision-alpha](https://huggingface.co/abetlen/llama-3-vision-alpha-gguf) | `Llama3VisionAlphaChatHandler` | `llama-3-vision-alpha` | | |
| | [minicpm-v-2.6](https://huggingface.co/openbmb/MiniCPM-V-2_6-gguf) | `MiniCPMv26ChatHandler` | `minicpm-v-2.6` | | |
| | [qwen2.5-vl](https://huggingface.co/unsloth/Qwen2.5-VL-3B-Instruct-GGUF) | `Qwen25VLChatHandler` | `qwen2.5-vl` | | |
| | [gemma-4](https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF) | `Gemma4ChatHandler` | `gemma4` | | |
| | GGUF models with an mtmd projector and embedded chat template | `MTMDChatHandler` | `mtmd` | | |
| Try Gemma 4 12B in Google Colab -> [](https://colab.research.google.com/github/abetlen/llama-cpp-python/blob/main/examples/colab/notebook.ipynb) | |
| Try Gemma 4 12B QAT in Google Colab -> [](https://colab.research.google.com/github/abetlen/llama-cpp-python/blob/main/examples/colab/Gemma4-12B-QAT.ipynb) | |
| Then you'll need to use a custom chat handler to load the clip model and process the chat messages and images. | |
| ```python | |
| from llama_cpp import Llama | |
| from llama_cpp.llama_chat_format import Llava15ChatHandler | |
| chat_handler = Llava15ChatHandler(clip_model_path="path/to/llava/mmproj.bin") | |
| llm = Llama( | |
| model_path="./path/to/llava/llama-model.gguf", | |
| chat_handler=chat_handler, | |
| n_ctx=2048, # n_ctx should be increased to accommodate the image embedding | |
| ) | |
| llm.create_chat_completion( | |
| messages = [ | |
| {"role": "system", "content": "You are an assistant who perfectly describes images."}, | |
| { | |
| "role": "user", | |
| "content": [ | |
| {"type" : "text", "text": "What's in this image?"}, | |
| {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" } } | |
| ] | |
| } | |
| ] | |
| ) | |
| ``` | |
| You can also pull the model from the Hugging Face Hub using the `from_pretrained` method. | |
| ```python | |
| from llama_cpp import Llama | |
| from llama_cpp.llama_chat_format import MoondreamChatHandler | |
| chat_handler = MoondreamChatHandler.from_pretrained( | |
| repo_id="vikhyatk/moondream2", | |
| filename="*mmproj*", | |
| ) | |
| llm = Llama.from_pretrained( | |
| repo_id="vikhyatk/moondream2", | |
| filename="*text-model*", | |
| chat_handler=chat_handler, | |
| n_ctx=2048, # n_ctx should be increased to accommodate the image embedding | |
| ) | |
| response = llm.create_chat_completion( | |
| messages = [ | |
| { | |
| "role": "user", | |
| "content": [ | |
| {"type" : "text", "text": "What's in this image?"}, | |
| {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" } } | |
| ] | |
| } | |
| ] | |
| ) | |
| print(response["choices"][0]["text"]) | |
| ``` | |
| **Note**: Multi-modal models also support tool calling and JSON mode. | |
| <details> | |
| <summary>Loading a Local Image</summary> | |
| Images can be passed as base64 encoded data URIs. The following example demonstrates how to do this. | |
| ```python | |
| import base64 | |
| def image_to_base64_data_uri(file_path): | |
| with open(file_path, "rb") as img_file: | |
| base64_data = base64.b64encode(img_file.read()).decode('utf-8') | |
| return f"data:image/png;base64,{base64_data}" | |
| # Replace 'file_path.png' with the actual path to your PNG file | |
| file_path = 'file_path.png' | |
| data_uri = image_to_base64_data_uri(file_path) | |
| messages = [ | |
| {"role": "system", "content": "You are an assistant who perfectly describes images."}, | |
| { | |
| "role": "user", | |
| "content": [ | |
| {"type": "image_url", "image_url": {"url": data_uri }}, | |
| {"type" : "text", "text": "Describe this image in detail please."} | |
| ] | |
| } | |
| ] | |
| ``` | |
| </details> | |
| ### Speculative Decoding | |
| `llama-cpp-python` supports speculative decoding which allows the model to generate completions based on a draft model. | |
| The fastest way to use speculative decoding is through the `LlamaPromptLookupDecoding` class. | |
| Just pass this as a draft model to the `Llama` class during initialization. | |
| ```python | |
| from llama_cpp import Llama | |
| from llama_cpp.llama_speculative import LlamaPromptLookupDecoding | |
| llama = Llama( | |
| model_path="path/to/model.gguf", | |
| draft_model=LlamaPromptLookupDecoding(num_pred_tokens=10) # num_pred_tokens is the number of tokens to predict 10 is the default and generally good for gpu, 2 performs better for cpu-only machines. | |
| ) | |
| ``` | |
| ### Embeddings | |
| To generate text embeddings use [`create_embedding`](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.create_embedding) or [`embed`](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.embed). Note that you must pass `embedding=True` to the constructor upon model creation for these to work properly. | |
| ```python | |
| import llama_cpp | |
| llm = llama_cpp.Llama(model_path="path/to/model.gguf", embedding=True) | |
| embeddings = llm.create_embedding("Hello, world!") | |
| # or create multiple embeddings at once | |
| embeddings = llm.create_embedding(["Hello, world!", "Goodbye, world!"]) | |
| ``` | |
| There are two primary notions of embeddings in a Transformer-style model: *token level* and *sequence level*. Sequence level embeddings are produced by "pooling" token level embeddings together, usually by averaging them or using the first token. | |
| Models that are explicitly geared towards embeddings will usually return sequence level embeddings by default, one for each input string. Non-embedding models such as those designed for text generation will typically return only token level embeddings, one for each token in each sequence. Thus the dimensionality of the return type will be one higher for token level embeddings. | |
| It is possible to control pooling behavior in some cases using the `pooling_type` flag on model creation. You can ensure token level embeddings from any model using `LLAMA_POOLING_TYPE_NONE`. The reverse, getting a generation oriented model to yield sequence level embeddings is currently not possible, but you can always do the pooling manually. | |
| ### Adjusting the Context Window | |
| The context window of the Llama models determines the maximum number of tokens that can be processed at once. By default, this is set to 512 tokens, but can be adjusted based on your requirements. | |
| For instance, if you want to work with larger contexts, you can expand the context window by setting the n_ctx parameter when initializing the Llama object: | |
| ```python | |
| llm = Llama(model_path="./models/7B/llama-model.gguf", n_ctx=2048) | |
| ``` | |
| ## OpenAI Compatible Web Server | |
| `llama-cpp-python` offers a web server which aims to act as a drop-in replacement for the OpenAI API. | |
| This allows you to use llama.cpp compatible models with any OpenAI compatible client (language libraries, services, etc). | |
| To install the server package and get started: | |
| ```bash | |
| pip install 'llama-cpp-python[server]' | |
| python3 -m llama_cpp.server --model models/7B/llama-model.gguf | |
| ``` | |
| Similar to Hardware Acceleration section above, you can also install with GPU (cuBLAS) support like this: | |
| ```bash | |
| CMAKE_ARGS="-DGGML_CUDA=on" FORCE_CMAKE=1 pip install 'llama-cpp-python[server]' | |
| python3 -m llama_cpp.server --model models/7B/llama-model.gguf --n_gpu_layers 35 | |
| ``` | |
| Navigate to [http://localhost:8000/docs](http://localhost:8000/docs) to see the OpenAPI documentation. | |
| To bind to `0.0.0.0` to enable remote connections, use `python3 -m llama_cpp.server --host 0.0.0.0`. | |
| Similarly, to change the port (default is 8000), use `--port`. | |
| You probably also want to set the prompt format. For chatml, use | |
| ```bash | |
| python3 -m llama_cpp.server --model models/7B/llama-model.gguf --chat_format chatml | |
| ``` | |
| That will format the prompt according to how model expects it. You can find the prompt format in the model card. | |
| For possible options, see [llama_cpp/llama_chat_format.py](llama_cpp/llama_chat_format.py) and look for lines starting with "@register_chat_format". | |
| If you have `huggingface-hub` installed, you can also use the `--hf_model_repo_id` flag to load a model from the Hugging Face Hub. | |
| ```bash | |
| python3 -m llama_cpp.server --hf_model_repo_id lmstudio-community/Qwen3.5-0.8B-GGUF --model '*Q8_0.gguf' | |
| ``` | |
| ### Web Server Features | |
| - [Local Copilot replacement](https://llama-cpp-python.readthedocs.io/en/latest/server/#code-completion) | |
| - [Function Calling support](https://llama-cpp-python.readthedocs.io/en/latest/server/#function-calling) | |
| - [Vision API support](https://llama-cpp-python.readthedocs.io/en/latest/server/#multimodal-models) | |
| - [Multiple Models](https://llama-cpp-python.readthedocs.io/en/latest/server/#configuration-and-multi-model-support) | |
| ## Docker image | |
| A Docker image is available on [GHCR](https://ghcr.io/abetlen/llama-cpp-python). To run the server: | |
| ```bash | |
| docker run --rm -it -p 8000:8000 -v /path/to/models:/models -e MODEL=/models/llama-model.gguf ghcr.io/abetlen/llama-cpp-python:latest | |
| ``` | |
| [Docker on termux (requires root)](https://gist.github.com/FreddieOliveira/efe850df7ff3951cb62d74bd770dce27) is currently the only known way to run this on phones, see [termux support issue](https://github.com/abetlen/llama-cpp-python/issues/389) | |
| ## Low-level API | |
| [API Reference](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#low-level-api) | |
| The low-level API is a direct [`ctypes`](https://docs.python.org/3/library/ctypes.html) binding to the C API provided by `llama.cpp`. | |
| The entire low-level API can be found in [llama_cpp/llama_cpp.py](https://github.com/abetlen/llama-cpp-python/blob/master/llama_cpp/llama_cpp.py) and directly mirrors the C API in [llama.h](https://github.com/ggerganov/llama.cpp/blob/master/llama.h). | |
| Below is a short example demonstrating how to use the low-level API to tokenize a prompt: | |
| ```python | |
| import llama_cpp | |
| import ctypes | |
| llama_cpp.llama_backend_init() # Must be called once at the start of each program | |
| model_params = llama_cpp.llama_model_default_params() | |
| ctx_params = llama_cpp.llama_context_default_params() | |
| prompt = b"Q: Name the planets in the solar system? A: " | |
| # use bytes for char * params | |
| model = llama_cpp.llama_model_load_from_file(b"./models/7b/llama-model.gguf", model_params) | |
| ctx = llama_cpp.llama_init_from_model(model, ctx_params) | |
| vocab = llama_cpp.llama_model_get_vocab(model) | |
| max_tokens = ctx_params.n_ctx | |
| # use ctypes arrays for array params | |
| tokens = (llama_cpp.llama_token * int(max_tokens))() | |
| n_tokens = llama_cpp.llama_tokenize(vocab, prompt, len(prompt), tokens, max_tokens, True, False) | |
| llama_cpp.llama_free(ctx) | |
| llama_cpp.llama_model_free(model) | |
| ``` | |
| Check out the [examples folder](examples/low_level_api) for more examples of using the low-level API. | |
| ## Documentation | |
| Documentation is available via [https://llama-cpp-python.readthedocs.io/](https://llama-cpp-python.readthedocs.io/). | |
| If you find any issues with the documentation, please open an issue or submit a PR. | |
| ## Development | |
| This package is under active development and I welcome any contributions. | |
| See [CONTRIBUTING.md](CONTRIBUTING.md) for contribution workflow, PR title, changelog, testing, and style guidelines. | |
| To get started, clone the repository and install the package in editable / development mode: | |
| ```bash | |
| git clone --recurse-submodules https://github.com/abetlen/llama-cpp-python.git | |
| cd llama-cpp-python | |
| # Upgrade pip (required for editable mode) | |
| pip install --upgrade pip | |
| # Install with pip | |
| pip install -e . | |
| # install development tooling (tests, docs, ruff) | |
| pip install -e '.[dev]' | |
| # if you want to use the fastapi / openapi server | |
| pip install -e '.[server]' | |
| # to install all optional dependencies | |
| pip install -e '.[all]' | |
| # to clear the local build cache | |
| make clean | |
| ``` | |
| Now try running the tests | |
| ```bash | |
| pytest | |
| ``` | |
| And check formatting / linting before opening a PR: | |
| ```bash | |
| python -m ruff check llama_cpp tests | |
| python -m ruff format --check llama_cpp tests | |
| # or use the Makefile targets | |
| make lint | |
| make format | |
| ``` | |
| There's a `Makefile` available with useful targets. | |
| A typical workflow would look like this: | |
| ```bash | |
| make build | |
| make test | |
| ``` | |
| You can also test out specific commits of `llama.cpp` by checking out the desired commit in the `vendor/llama.cpp` submodule and then running `make clean` and `pip install -e .` again. Any changes in the `llama.h` API will require | |
| changes to the `llama_cpp/llama_cpp.py` file to match the new API (additional changes may be required elsewhere). | |
| ## FAQ | |
| ### Are there pre-built binaries / binary wheels available? | |
| The recommended installation method is to install from source as described above. | |
| The reason for this is that `llama.cpp` is built with compiler optimizations that are specific to your system. | |
| Using pre-built binaries would require disabling these optimizations or supporting a large number of pre-built binaries for each platform. | |
| That being said there are some pre-built binaries available through the Releases as well as some community provided wheels. | |
| In the future, I would like to provide pre-built binaries and wheels for common platforms and I'm happy to accept any useful contributions in this area. | |
| This is currently being tracked in [#741](https://github.com/abetlen/llama-cpp-python/issues/741) | |
| ### How does this compare to other Python bindings of `llama.cpp`? | |
| I originally wrote this package for my own use with two goals in mind: | |
| - Provide a simple process to install `llama.cpp` and access the full C API in `llama.h` from Python | |
| - Provide a high-level Python API that can be used as a drop-in replacement for the OpenAI API so existing apps can be easily ported to use `llama.cpp` | |
| Any contributions and changes to this package will be made with these goals in mind. | |
| ## License | |
| This project is licensed under the terms of the MIT license. | |
Xet Storage Details
- Size:
- 35.7 kB
- Xet hash:
- 721502e7a18e5cb8da6b526299e079a7f5999490f803bb94f3e194f3b8439ec4
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.