Instructions to use upstage/llama-30b-instruct-2048 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use upstage/llama-30b-instruct-2048 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="upstage/llama-30b-instruct-2048")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("upstage/llama-30b-instruct-2048")
model = AutoModelForCausalLM.from_pretrained("upstage/llama-30b-instruct-2048", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use upstage/llama-30b-instruct-2048 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "upstage/llama-30b-instruct-2048"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "upstage/llama-30b-instruct-2048",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/upstage/llama-30b-instruct-2048

SGLang

How to use upstage/llama-30b-instruct-2048 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "upstage/llama-30b-instruct-2048" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "upstage/llama-30b-instruct-2048",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "upstage/llama-30b-instruct-2048" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "upstage/llama-30b-instruct-2048",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use upstage/llama-30b-instruct-2048 with Docker Model Runner:
```
docker model run hf.co/upstage/llama-30b-instruct-2048
```

llama-30b-instruct-2048

Commit History

Update README.md

9f246be

Limerobot commited on Aug 3, 2023

Update README.md

52d240f

wonhosong commited on Aug 3, 2023

Update README.md

b76245f

Chanjun commited on Aug 1, 2023

Update README.md

1f3ef2b

wonhosong commited on Aug 1, 2023

Update README.md

5a8d3e8

wonhosong commited on Aug 1, 2023

Update README.md

936cf81

wonhosong commited on Aug 1, 2023

Update README.md

020b885

wonhosong commited on Aug 1, 2023

Update README.md

1176c64

Chanjun commited on Aug 1, 2023

Update README.md

1eda685

Chanjun commited on Aug 1, 2023

Update README.md

c301698

Chanjun commited on Aug 1, 2023

Update README.md

9599e13

Chanjun commited on Aug 1, 2023

Update README.md

0205635

Chanjun commited on Aug 1, 2023

Update README.md

f191b77

Chanjun commited on Aug 1, 2023

Update README.md

3e0a210

wonhosong commited on Aug 1, 2023

Update README.md

b1b163e

Limerobot commited on Jul 30, 2023

Update README.md

3b060bc

Limerobot commited on Jul 25, 2023

Update README.md

abb2842

Chanjun commited on Jul 25, 2023

Update config.json

16b254d

wonhosong commited on Jul 25, 2023

Update README.md

b0f7012

Chanjun commited on Jul 22, 2023

Update README.md

7bdac3d

Chanjun commited on Jul 22, 2023

Update README.md

5187bf2

Chanjun commited on Jul 22, 2023

link datasets (#3)

93365af

wonhosong

lhoestq HF Staff commited on Jul 21, 2023