Instructions to use VMware/open-llama-7b-open-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use VMware/open-llama-7b-open-instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="VMware/open-llama-7b-open-instruct")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("VMware/open-llama-7b-open-instruct")
model = AutoModelForCausalLM.from_pretrained("VMware/open-llama-7b-open-instruct")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use VMware/open-llama-7b-open-instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "VMware/open-llama-7b-open-instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "VMware/open-llama-7b-open-instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/VMware/open-llama-7b-open-instruct

SGLang

How to use VMware/open-llama-7b-open-instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "VMware/open-llama-7b-open-instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "VMware/open-llama-7b-open-instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "VMware/open-llama-7b-open-instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "VMware/open-llama-7b-open-instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use VMware/open-llama-7b-open-instruct with Docker Model Runner:
```
docker model run hf.co/VMware/open-llama-7b-open-instruct
```

open-llama-7b-open-instruct

Commit History

Adding `safetensors` variant of this model (#3)

6c005d1
verified

rbattle

SFconvertbot commited on Dec 16, 2024

Update README.md

8af0414

rbattle commited on Dec 1, 2023

Adding Evaluation Results (#2)

eebd7e4

rbattle

leaderboard-pr-bot commited on Nov 17, 2023

Update README.md

fdf9f03

Teja-Gollapudi commited on Jul 11, 2023

Update README.md

a4fbf07

Teja-Gollapudi commited on Jul 10, 2023

Update README.md

0278fdb

rbattle commited on Jun 12, 2023

Update README.md

6a94432

Teja-Gollapudi commited on Jun 9, 2023

Update README.md

e7910ae

Teja-Gollapudi commited on Jun 9, 2023

Update README.md

9df23a7

Teja-Gollapudi commited on Jun 9, 2023

Update README.md

65f3931

Teja-Gollapudi commited on Jun 9, 2023

Update README.md

d6197e8

Teja-Gollapudi commited on Jun 9, 2023

Update README.md

599bdd9

Teja-Gollapudi commited on Jun 8, 2023

Update README.md

9d35e5c

Teja-Gollapudi commited on Jun 8, 2023

Update README.md

05ff50d

Teja-Gollapudi commited on Jun 8, 2023

Update README.md

7c3316d

Teja-Gollapudi commited on Jun 8, 2023

Update tokenizer_config.json

a4ae248

Teja-Gollapudi commited on Jun 8, 2023

Update README.md

518a136

Teja-Gollapudi commited on Jun 8, 2023

Update README.md

880b48d

Teja-Gollapudi commited on Jun 8, 2023

Update README.md

5e15ece

Teja-Gollapudi commited on Jun 8, 2023

Update README.md

db61bb3

Teja-Gollapudi commited on Jun 8, 2023

Create README.md

e249fd0

Teja-Gollapudi commited on Jun 8, 2023

Upload tokenizer

5ef8730

Teja-Gollapudi commited on Jun 8, 2023

Upload LlamaForCausalLM

05d0cd3

Teja-Gollapudi commited on Jun 8, 2023

initial commit

e1d15b5

Teja-Gollapudi commited on Jun 8, 2023

Commit History

Adding `safetensors` variant of this model (#3) 6c005d1 verified

Update README.md 8af0414

Adding Evaluation Results (#2) eebd7e4

Update README.md fdf9f03

Update README.md a4fbf07

Update README.md 0278fdb

Update README.md 6a94432

Update README.md e7910ae

Update README.md 9df23a7

Update README.md 65f3931

Update README.md d6197e8

Update README.md 599bdd9

Update README.md 9d35e5c

Update README.md 05ff50d

Update README.md 7c3316d

Update tokenizer_config.json a4ae248

Update README.md 518a136

Update README.md 880b48d

Update README.md 5e15ece

Update README.md db61bb3

Create README.md e249fd0

Upload tokenizer 5ef8730

Upload LlamaForCausalLM 05d0cd3

initial commit e1d15b5

Adding `safetensors` variant of this model (#3)

6c005d1
verified

Update README.md

8af0414

Adding Evaluation Results (#2)

eebd7e4

Update README.md

fdf9f03

Update README.md

a4fbf07

Update README.md

0278fdb

Update README.md

6a94432

Update README.md

e7910ae

Update README.md

9df23a7

Update README.md

65f3931

Update README.md

d6197e8

Update README.md

599bdd9

Update README.md

9d35e5c

Update README.md

05ff50d

Update README.md

7c3316d

Update tokenizer_config.json

a4ae248

Update README.md

518a136

Update README.md

880b48d

Update README.md

5e15ece

Update README.md

db61bb3

Create README.md

e249fd0

Upload tokenizer

5ef8730

Upload LlamaForCausalLM

05d0cd3

initial commit

e1d15b5