Instructions to use TheBloke/CodeLlama-7B-Python-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use TheBloke/CodeLlama-7B-Python-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="TheBloke/CodeLlama-7B-Python-GGUF")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("TheBloke/CodeLlama-7B-Python-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use TheBloke/CodeLlama-7B-Python-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf TheBloke/CodeLlama-7B-Python-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf TheBloke/CodeLlama-7B-Python-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf TheBloke/CodeLlama-7B-Python-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf TheBloke/CodeLlama-7B-Python-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf TheBloke/CodeLlama-7B-Python-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama cli -hf TheBloke/CodeLlama-7B-Python-GGUF:Q4_K_M

Use Docker

docker model run hf.co/TheBloke/CodeLlama-7B-Python-GGUF:Q4_K_M

Llama new
LM Studio
Jan

vLLM

How to use TheBloke/CodeLlama-7B-Python-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "TheBloke/CodeLlama-7B-Python-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/CodeLlama-7B-Python-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/TheBloke/CodeLlama-7B-Python-GGUF:Q4_K_M

SGLang

How to use TheBloke/CodeLlama-7B-Python-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "TheBloke/CodeLlama-7B-Python-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/CodeLlama-7B-Python-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "TheBloke/CodeLlama-7B-Python-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/CodeLlama-7B-Python-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Ollama
How to use TheBloke/CodeLlama-7B-Python-GGUF with Ollama:
```
ollama run hf.co/TheBloke/CodeLlama-7B-Python-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use TheBloke/CodeLlama-7B-Python-GGUF with Docker Model Runner:
```
docker model run hf.co/TheBloke/CodeLlama-7B-Python-GGUF:Q4_K_M
```

Lemonade

How to use TheBloke/CodeLlama-7B-Python-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull TheBloke/CodeLlama-7B-Python-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.CodeLlama-7B-Python-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

CodeLlama-7B-Python-GGUF

Commit History

Sep 27, 2023

12:46
Update base_model formatting

TheBloke

3c6970f

Sep 19, 2023

00:51
Upload README.md

TheBloke

b09c67c

Sep 7, 2023

11:13
Upload README.md

TheBloke

dcd7825

Sep 5, 2023

14:13
Upload README.md

TheBloke

f848c2c
02:34
Upload README.md

TheBloke

b3633fd
02:34
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

27dbdf0
02:34
Add Llama 2 license files

TheBloke

335af58
02:34
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

1b06c06
02:34
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

b18e28d
02:33
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

bbd726f
02:33
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

bc85d3e
02:32
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

365e5f8
02:32
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

7bcc1dd
02:31
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

f3e10d8
02:30
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

5dcadd7
02:30
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

6c71f21
02:30
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

68ee8f4
02:29
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

90bd2dc
02:29
Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

TheBloke

d687884

Aug 26, 2023

15:46
Initial GGUF model commit

TheBloke

1e85574
15:46
Initial GGUF model commit

TheBloke

f2927b5
15:46
Add Llama 2 license files

TheBloke

8c709d1
15:46
Add Llama 2 license files

TheBloke

08437cd
15:46
Add Llama 2 license files

TheBloke

c3c91ef
11:14
Initial GGUF model commit

TheBloke

cb7ee1b
11:14
Initial GGUF model commit

TheBloke

a855e41
11:14
Add Llama 2 license files

TheBloke

b372d5a
11:14
Add Llama 2 license files

TheBloke

b4476f9
11:14
Add Llama 2 license files

TheBloke

93a8d97
11:09
Initial GGUF model commit

TheBloke

0f0db35
11:09
Initial GGUF model commit

TheBloke

cc31bf6
11:09
Add Llama 2 license files

TheBloke

04549d8
11:09
Add Llama 2 license files

TheBloke

bd86951
11:09
Add Llama 2 license files

TheBloke

2130841
11:09
Initial GGUF model commit

TheBloke

8626746
11:09
Add Llama 2 license files

TheBloke

a6c5b7f
11:09
Add Llama 2 license files

TheBloke

19caaed
11:09
Add Llama 2 license files

TheBloke

e9ee1d7

Aug 25, 2023

15:34
Initial GGUF model commit

TheBloke

3a607fa
15:34
Initial GGUF model commit

TheBloke

9c9cb36
15:34
Add Llama 2 license files

TheBloke

7fa9c90
15:34
Add Llama 2 license files

TheBloke

4f75ac4
15:34
Add Llama 2 license files

TheBloke

cb20025
15:31
Initial GGUF model commit

TheBloke

c0e4377
15:31
Initial GGUF model commit

TheBloke

ca80b1b
15:31
Add Llama 2 license files

TheBloke

b9c5b2c
15:31
Add Llama 2 license files

TheBloke

d480687
15:31
Add Llama 2 license files

TheBloke

ed20e49
12:03
Update README.md

TheBloke

bad2278

Aug 24, 2023

20:17
Initial GGUF model commit

TheBloke

1f853a1

Commit History

Sep 27, 2023

Update base_model formatting

Sep 19, 2023

Upload README.md

Sep 7, 2023

Upload README.md

Sep 5, 2023

Upload README.md

Upload README.md

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Add Llama 2 license files

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Initial GGUF model commit (model made with llama.cpp commit 2ba85c8)

Aug 26, 2023

Initial GGUF model commit

Initial GGUF model commit

Add Llama 2 license files

Add Llama 2 license files

Add Llama 2 license files

Initial GGUF model commit

Initial GGUF model commit

Add Llama 2 license files

Add Llama 2 license files

Add Llama 2 license files

Initial GGUF model commit

Initial GGUF model commit

Add Llama 2 license files

Add Llama 2 license files

Add Llama 2 license files

Initial GGUF model commit

Add Llama 2 license files

Add Llama 2 license files

Add Llama 2 license files

Aug 25, 2023

Initial GGUF model commit

Initial GGUF model commit

Add Llama 2 license files

Add Llama 2 license files

Add Llama 2 license files

Initial GGUF model commit

Initial GGUF model commit

Add Llama 2 license files

Add Llama 2 license files

Add Llama 2 license files

Update README.md

Aug 24, 2023

Initial GGUF model commit