Instructions to use mattshumer/Hermes-2-Pro-11B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mattshumer/Hermes-2-Pro-11B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mattshumer/Hermes-2-Pro-11B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("mattshumer/Hermes-2-Pro-11B")
model = AutoModelForCausalLM.from_pretrained("mattshumer/Hermes-2-Pro-11B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use mattshumer/Hermes-2-Pro-11B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "mattshumer/Hermes-2-Pro-11B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mattshumer/Hermes-2-Pro-11B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/mattshumer/Hermes-2-Pro-11B

SGLang

How to use mattshumer/Hermes-2-Pro-11B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mattshumer/Hermes-2-Pro-11B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mattshumer/Hermes-2-Pro-11B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mattshumer/Hermes-2-Pro-11B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mattshumer/Hermes-2-Pro-11B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use mattshumer/Hermes-2-Pro-11B with Docker Model Runner:
```
docker model run hf.co/mattshumer/Hermes-2-Pro-11B
```

Missing tokens

by pdevine - opened Mar 26, 2024

Discussion

pdevine

Mar 26, 2024

When converting the model it seems like there are some missing tokens; the config.json lists the vocab size as 32,032, but the tokens.json only has 32,000 tokens in it, and the added_tokens.json file only defines two tokens. Similarly the tokenizer_config.json doesn't have the missing tokens in it either.

I've tried padding out the vocabulary when I'm running it through Ollama (i.e. in GGUF format) and get output which looks like:

>>> tell me the cool stuff
I'd be glad to! Here are some more interesting facts:
1. The largest known<dummy00001> spider web is 2,8<dummy00002> a whopping 46<dummy00001> feet wide and was made by a<dummy00018>a goat-faced ornspider in Japan.
2. The tallest wooden structure in the world, the 568-foot-tall pagoda, is located in Tallinn, Estonia. It's called the "Hydro-Cube."
3. If you were to stack all the grains of sand on Earth end-to-end, it would reach the Sun 1,900,000 times and still have plenty left over. And if you tried
to fill up a large stadium like Wembley Stadium in London with all that sand, it wouldn't even come close to how much sand is on our planet!
4. Speaking of large structures, the Great Pyramid of Giza in Egypt is so old and so well-preserved that it's still usable as a parking garage for about
102,000 cars, or 153,000 people on a crowded day.

Also, I'm not really sure how you would park 102k cars in the Great Pyramid of Giza :-D

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment