Instructions to use TheBloke/Llama-2-7B-GGML with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use TheBloke/Llama-2-7B-GGML with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="TheBloke/Llama-2-7B-GGML")# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("TheBloke/Llama-2-7B-GGML", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use TheBloke/Llama-2-7B-GGML with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "TheBloke/Llama-2-7B-GGML" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "TheBloke/Llama-2-7B-GGML", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/TheBloke/Llama-2-7B-GGML
- SGLang
How to use TheBloke/Llama-2-7B-GGML with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "TheBloke/Llama-2-7B-GGML" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "TheBloke/Llama-2-7B-GGML", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "TheBloke/Llama-2-7B-GGML" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "TheBloke/Llama-2-7B-GGML", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use TheBloke/Llama-2-7B-GGML with Docker Model Runner:
docker model run hf.co/TheBloke/Llama-2-7B-GGML
Assertion Error due to which Model is not being Loaded
The AssertionError is related to how Llama's constructor __init__ method tries to set self.model, but received the value None. Assertions are used to check if conditions are true. Try to check if model_path is correctly set, either by debugging or just looking at the value of model_path before it is passed to the Llama class. Alternatively, post more of your code here so it would be easier to try to see what the problem is. My guess is that you might've changed model_path to a None value somewhere.
error is due to using ggml with llama cpp python which isn't supported anymore.
