Instructions to use nuprl/MultiPL-T-CodeLlama_34b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nuprl/MultiPL-T-CodeLlama_34b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="nuprl/MultiPL-T-CodeLlama_34b")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("nuprl/MultiPL-T-CodeLlama_34b")
model = AutoModelForCausalLM.from_pretrained("nuprl/MultiPL-T-CodeLlama_34b")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use nuprl/MultiPL-T-CodeLlama_34b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nuprl/MultiPL-T-CodeLlama_34b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nuprl/MultiPL-T-CodeLlama_34b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/nuprl/MultiPL-T-CodeLlama_34b

SGLang

How to use nuprl/MultiPL-T-CodeLlama_34b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nuprl/MultiPL-T-CodeLlama_34b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nuprl/MultiPL-T-CodeLlama_34b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nuprl/MultiPL-T-CodeLlama_34b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nuprl/MultiPL-T-CodeLlama_34b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use nuprl/MultiPL-T-CodeLlama_34b with Docker Model Runner:
```
docker model run hf.co/nuprl/MultiPL-T-CodeLlama_34b
```

cassanof commited on Oct 2, 2023

Commit

284286c

1 Parent(s): ba339d0

model_codellama_34b_multiplt_racket-epoch5

Browse files

Files changed (8) hide show

config.json +1 -1
pytorch_model-00001-of-00007.bin +1 -1
pytorch_model-00002-of-00007.bin +1 -1
pytorch_model-00003-of-00007.bin +1 -1
pytorch_model-00004-of-00007.bin +1 -1
pytorch_model-00005-of-00007.bin +1 -1
pytorch_model-00006-of-00007.bin +1 -1
pytorch_model-00007-of-00007.bin +1 -1

config.json CHANGED Viewed

@@ -1,5 +1,5 @@
 {
-  "_name_or_path": "model_codellama_34b_multiplt_racket/checkpoint-328",
   "architectures": [
     "LlamaForCausalLM"
   ],

 {
+  "_name_or_path": "model_codellama_34b_multiplt_racket/checkpoint-410",
   "architectures": [
     "LlamaForCausalLM"
   ],

pytorch_model-00001-of-00007.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:96a572affa160b68a14c1102d596dac9afc551cc9ac43f20d1fa01add6a0c9f3
 size 9852637497

 version https://git-lfs.github.com/spec/v1
+oid sha256:b26055a90e428e41497f6ff5f0c17e18f8d8468e4eee3272c16d91acd1e6d201
 size 9852637497

pytorch_model-00002-of-00007.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:528a05bb3e2baffb18e8367ee346baa9c5aeb250cdad5a8053ff4c3ef2338427
 size 9689093137

 version https://git-lfs.github.com/spec/v1
+oid sha256:2adadaec8c76a5aa8f3f87734e2a67fd4eb1fb8e11b0177674641b3c8fd81840
 size 9689093137

pytorch_model-00003-of-00007.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d554e5b49dddf76f1843f462bfa905d88ee68e682de02234d226dd91caf2e218
 size 9689093137

 version https://git-lfs.github.com/spec/v1
+oid sha256:52204045c3be796dc4080fb5604cff7434c6da99abff446a75206e6c29c47f07
 size 9689093137

pytorch_model-00004-of-00007.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:42ad6518023d5b37ea6fb24a09f080a68abd20e91f987df13f2bcd272b0a1cf5
 size 9689093137

 version https://git-lfs.github.com/spec/v1
+oid sha256:62a1a79bb79b4f5eeec4a0fe0aed3aac4bd2e5f1f859b07ca9a8d6001f37d2fa
 size 9689093137

pytorch_model-00005-of-00007.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b7b29e6b5d648f4a966001d5519644bf742857834d1a1fd0032ed1b0b9955ef2
 size 9689093137

 version https://git-lfs.github.com/spec/v1
+oid sha256:52ad966b6c8bbf71929997b3536a853da67b6ca2d4023e8120735a86b7a3249d
 size 9689093137

pytorch_model-00006-of-00007.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f605ae09d03356858df7608d2d5641a7a258c11ed44b291a6005d9a13847e018
 size 9689093137

 version https://git-lfs.github.com/spec/v1
+oid sha256:b951b55afd9b9f1c576d0057daec10cc842e4ea3b7f6e1f792e014531ef2dc04
 size 9689093137

pytorch_model-00007-of-00007.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a556378336a841d4b70afcbc0042578747049f8be0bd1f326139080449d1d99e
 size 9189985945

 version https://git-lfs.github.com/spec/v1
+oid sha256:df1f2fb9003cf145b8fef62029361ac37b6fb9202c52c7daad205e0f4196ef1f
 size 9189985945