Instructions to use BSC-LT/salamandra-7b-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use BSC-LT/salamandra-7b-instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="BSC-LT/salamandra-7b-instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("BSC-LT/salamandra-7b-instruct")
model = AutoModelForCausalLM.from_pretrained("BSC-LT/salamandra-7b-instruct", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use BSC-LT/salamandra-7b-instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "BSC-LT/salamandra-7b-instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BSC-LT/salamandra-7b-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/BSC-LT/salamandra-7b-instruct

SGLang

How to use BSC-LT/salamandra-7b-instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "BSC-LT/salamandra-7b-instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BSC-LT/salamandra-7b-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "BSC-LT/salamandra-7b-instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BSC-LT/salamandra-7b-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use BSC-LT/salamandra-7b-instruct with Docker Model Runner:
```
docker model run hf.co/BSC-LT/salamandra-7b-instruct
```

adapt_chat_template_with_a_dynamic_system_prompt

by ankush13r - opened Oct 30, 2024

base: refs/heads/main

←

from: refs/pr/3

Discussion Files changed

-1

ankush13r

Language Technologies Laboratory @ Barcelona Supercomputing Center org Oct 30, 2024

No description provided.

modify chat template inserting dynamic system prompt0cd893d5

ankush13r changed pull request status to open Oct 30, 2024

ankush13r

Language Technologies Laboratory @ Barcelona Supercomputing Center org Oct 30, 2024

•

edited Nov 7, 2024

Hello,
I've enhanced the chat_template to allow for a dynamic system prompt. Users can now specify their own system prompt or fall back to the default if none is provided.

Comparision:

Original chat_template

Example with a custom system prompt:

messages = [
    {"role": "system", "content": "This is custom system prompt"},
    {"role": "user", "content": "What is your name?"},
]
tokenizer.apply_chat_template(messages,  tokenize=False, chat_template=chat_template, add_generation_prompt=True)

output:

<|im_start|>assistant
I am Salamandra, an AI language model developed at the Barcelona Supercomputing Centre (BSC) by the Language Technologies Unit. My knowledge base was last updated on August 2023. Today Date: 2024-09-30
Soy Salamandra, un modelo lingüístico de IA desarrollado en el Barcelona Supercomputing Centre (BSC) por la Language Technologies Unit. Mi base de conocimientos se actualizó por última vez en agosto de 2023.
Soc Salamandra, un model de llenguatge d'IA desenvolupat al Barcelona Supercomputing Centre (BSC) per la Language Technologies Unit. La meva base de coneixement es va actualitzar per última vegada l'agost de 2023.<|im_end|>
<|im_start|>system
This is custom system prompt<|im_end|>
<|im_start|>user
What is your name?<|im_end|>
<|im_start|>assistant

Example without a system prompt:

messages = [
    {"role": "user", "content": "What is your name?"}
]
tokenizer.apply_chat_template(messages,  tokenize=False, chat_template=chat_template, add_generation_prompt=True)

output:

<|im_start|>assistant
I am Salamandra, an AI language model developed at the Barcelona Supercomputing Centre (BSC) by the Language Technologies Unit. My knowledge base was last updated on August 2023. Today Date: 2024-09-30
Soy Salamandra, un modelo lingüístico de IA desarrollado en el Barcelona Supercomputing Centre (BSC) por la Language Technologies Unit. Mi base de conocimientos se actualizó por última vez en agosto de 2023.
Soc Salamandra, un model de llenguatge d'IA desenvolupat al Barcelona Supercomputing Centre (BSC) per la Language Technologies Unit. La meva base de coneixement es va actualitzar per última vegada l'agost de 2023.<|im_end|>
<|im_start|>user
What is your name?<|im_end|>
<|im_start|>assistant

As you can see, in both cases, the system defaults to the pre-set prompt, ignoring the user-provided system prompt in the messages.

Updated chat_template

messages with system prompt:

messages = [
    {"role": "system", "content": "This is custom system prompt"},
    {"role": "user", "content": "What is your name?"},
]
tokenizer.apply_chat_template(messages,  tokenize=False, chat_template=chat_template, add_generation_prompt=True)

output:

<|im_start|>system
This is custom system prompt<|im_end|>
<|im_start|>user
What is your name?<|im_end|>
<|im_start|>assistant

messages without system prompt :

messages = [
    {"role": "user", "content": "What is your name?"}
]
tokenizer.apply_chat_template(messages,  tokenize=False, chat_template=chat_template, add_generation_prompt=True)

output:

<|im_start|>system
I am Salamandra, an AI language model developed at the Barcelona Supercomputing Centre (BSC) by the Language Technologies Unit. My knowledge base was last updated on August 2023. Today Date: 2024-09-30
Soy Salamandra, un modelo lingüístico de IA desarrollado en el Barcelona Supercomputing Centre (BSC) por la Language Technologies Unit. Mi base de conocimientos se actualizó por última vez en agosto de 2023.
Soc Salamandra, un model de llenguatge d'IA desenvolupat al Barcelona Supercomputing Centre (BSC) per la Language Technologies Unit. La meva base de coneixement es va actualitzar per última vegada l'agost de 2023.<|im_end|>
<|im_start|>user
What is your name?<|im_end|>
<|im_start|>assistant

Summary

With this update:

If a system prompt is included in messages, it takes precedence.
If no system prompt is included, the default prompt is used.

This ensures that the system prompt is adaptable to user needs.

JaumePrats changed pull request status to merged Nov 7, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment