Instructions to use patched-codes/patched-coder-34b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use patched-codes/patched-coder-34b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="patched-codes/patched-coder-34b")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("patched-codes/patched-coder-34b")
model = AutoModelForCausalLM.from_pretrained("patched-codes/patched-coder-34b")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use patched-codes/patched-coder-34b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "patched-codes/patched-coder-34b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "patched-codes/patched-coder-34b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/patched-codes/patched-coder-34b

SGLang

How to use patched-codes/patched-coder-34b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "patched-codes/patched-coder-34b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "patched-codes/patched-coder-34b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "patched-codes/patched-coder-34b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "patched-codes/patched-coder-34b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use patched-codes/patched-coder-34b with Docker Model Runner:
```
docker model run hf.co/patched-codes/patched-coder-34b
```

codelion commited on Sep 22, 2023

Commit

650aede

1 Parent(s): a1e52c4

Upload LlamaForCausalLM

Browse files

Files changed (7) hide show

pytorch_model-00001-of-00007.bin +1 -1
pytorch_model-00002-of-00007.bin +1 -1
pytorch_model-00003-of-00007.bin +1 -1
pytorch_model-00004-of-00007.bin +1 -1
pytorch_model-00005-of-00007.bin +1 -1
pytorch_model-00006-of-00007.bin +1 -1
pytorch_model-00007-of-00007.bin +1 -1

pytorch_model-00001-of-00007.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:31bc0eee2229df0bd32b8af8d73d39df715a3495f9d8d249251cb66ab338c67f
 size 9852637497

 version https://git-lfs.github.com/spec/v1
+oid sha256:51dfa72906c1387819fc154cef5c48f2a3900fb1e693ce4fff1d3a273dc799c5
 size 9852637497

pytorch_model-00002-of-00007.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:71de359b59ae88a103ebbe23128703b45055a2fb3cfc2b199355d63866b093b3
 size 9689093137

 version https://git-lfs.github.com/spec/v1
+oid sha256:e9f508a6cab68f251349bb4bb03b14ebbb8f16e7175010ebf304e191d50db2d2
 size 9689093137

pytorch_model-00003-of-00007.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bab35c203537db2565e6e275bfc17795bba4a868dab4ad4dbd3fdd4fa07c530e
 size 9689093137

 version https://git-lfs.github.com/spec/v1
+oid sha256:0f6429d75e483cf7412f8a8fc321667a674dbd7c45ad4966f02e4d662ccbde58
 size 9689093137

pytorch_model-00004-of-00007.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a8622cf4c2ad31f218f40ba8e884b8532b6b5c3501204b9909012ab18024e232
 size 9689093137

 version https://git-lfs.github.com/spec/v1
+oid sha256:dac549fdc2ad06ddb32f1b127f48cfaab2ee6faa51b437176986b3c46b510906
 size 9689093137

pytorch_model-00005-of-00007.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:449bdec04ee6d972488a9d618f934753af6614c43f68e57abbb9d71284be9793
 size 9689093137

 version https://git-lfs.github.com/spec/v1
+oid sha256:a3d47b408b90e4baf59497c4ad0282b3b0c3be530e5cc98875398c40b8d92dcf
 size 9689093137

pytorch_model-00006-of-00007.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0e57754ec5420aa26573311f914194400204b2316ed461ef864fcb3c30c1fa45
 size 9689093137

 version https://git-lfs.github.com/spec/v1
+oid sha256:949d70547d2ce8d88cefcee5b4e0c16c56785483bf276bf93dcc6fe6ea8594fe
 size 9689093137

pytorch_model-00007-of-00007.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4dbf973a09ad6308ddb3a6e3f5e6b5707ac342377888d3813d9048383a927bde
 size 9189985945

 version https://git-lfs.github.com/spec/v1
+oid sha256:e1ad98f40441d27c5e107eb7932f17ad38ece4952a3b703a2309c0821a6c3116
 size 9189985945