Instructions to use nuprl/MultiPL-T-StarCoder2_15B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nuprl/MultiPL-T-StarCoder2_15B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="nuprl/MultiPL-T-StarCoder2_15B")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("nuprl/MultiPL-T-StarCoder2_15B")
model = AutoModelForCausalLM.from_pretrained("nuprl/MultiPL-T-StarCoder2_15B")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use nuprl/MultiPL-T-StarCoder2_15B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nuprl/MultiPL-T-StarCoder2_15B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nuprl/MultiPL-T-StarCoder2_15B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/nuprl/MultiPL-T-StarCoder2_15B

SGLang

How to use nuprl/MultiPL-T-StarCoder2_15B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nuprl/MultiPL-T-StarCoder2_15B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nuprl/MultiPL-T-StarCoder2_15B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nuprl/MultiPL-T-StarCoder2_15B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nuprl/MultiPL-T-StarCoder2_15B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use nuprl/MultiPL-T-StarCoder2_15B with Docker Model Runner:
```
docker model run hf.co/nuprl/MultiPL-T-StarCoder2_15B
```

arjunguha commited on Mar 26, 2024

Commit

62ce674

verified ·

1 Parent(s): 3a8dce7

StarCoder2_15b_ml_checkpoint_950

Browse files

Files changed (8) hide show

config.json +1 -1
model-00001-of-00007.safetensors +1 -1
model-00002-of-00007.safetensors +1 -1
model-00003-of-00007.safetensors +1 -1
model-00004-of-00007.safetensors +1 -1
model-00005-of-00007.safetensors +1 -1
model-00006-of-00007.safetensors +1 -1
model-00007-of-00007.safetensors +1 -1

config.json CHANGED Viewed

@@ -1,5 +1,5 @@
 {
-  "_name_or_path": "/scratch/a.guha/MutiPL-T/multiplt_ocaml-starcoder2_15b/output/checkpoint-95",
   "architectures": [
     "Starcoder2ForCausalLM"
   ],

 {
+  "_name_or_path": "/scratch/a.guha/MutiPL-T/multiplt_ocaml-starcoder2_15b/output/checkpoint-950",
   "architectures": [
     "Starcoder2ForCausalLM"
   ],

model-00001-of-00007.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:20ed532f8e54e6ff05525be5d944933b4e73f8c53f3a1f165bd9b703123957b7
 size 4908107736

 version https://git-lfs.github.com/spec/v1
+oid sha256:409f6740e2c245558c6e1d2a8f9bfeb2dbd8fdda96137be20ba2c0a11ac510b3
 size 4908107736

model-00002-of-00007.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cb061beccb64659a8e60d4687ccbc35dca0a1836d0da6564577feadb6dc4820c
 size 4996327240

 version https://git-lfs.github.com/spec/v1
+oid sha256:6dd7febe6011122bbfeebc63d108a93e3c14e65a2a5f9ff43bdd6c19666e309b
 size 4996327240

model-00003-of-00007.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:710a6c700a776dbbe6d558e7531175bbd020daee919b7cc79fd74168a0dc5c46
 size 4983729152

 version https://git-lfs.github.com/spec/v1
+oid sha256:ef7890468498b4c24c0fa14de5a7869989bb216a05a34388bb93bdab14a01cf2
 size 4983729152

model-00004-of-00007.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8f8c29232245171097e148a816b7236a2163d6cf906481227b38732188f84b34
 size 4996327312

 version https://git-lfs.github.com/spec/v1
+oid sha256:b8595f350b9e31b62e85efef0ef9540e0ffc99fd6936fd2206758a2330baafc8
 size 4996327312

model-00005-of-00007.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8856668eeacf611727571ccd942429dda2f6f906ba0b3a4a6a843780c4a6e8ee
 size 4983729152

 version https://git-lfs.github.com/spec/v1
+oid sha256:a1007231e54d8b84138ea2bf7baf41e13773328a1778860561512df6a0509331
 size 4983729152

model-00006-of-00007.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a1a623098252b3b42fff10a14fb8e6939774d513437f4aadac5dcf4baf430fc0
 size 4996327312

 version https://git-lfs.github.com/spec/v1
+oid sha256:41b4820d7c9a825d9539c33963eb4f974e4a8c2d2ea41c8a1be7a7675ea23b05
 size 4996327312

model-00007-of-00007.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6f721e2ecd420b59097c9becbbe6aabc78cf7a85602e28cb11156c5fba95df42
 size 2051302552

 version https://git-lfs.github.com/spec/v1
+oid sha256:f98fe265b44f12ee77b5e0e272a3f4a71fd2a5cadd29f776efe7199eaaed8609
 size 2051302552