Instructions to use projecte-aina/salamandra-7b-aligned-EADOP with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use projecte-aina/salamandra-7b-aligned-EADOP with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="projecte-aina/salamandra-7b-aligned-EADOP")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("projecte-aina/salamandra-7b-aligned-EADOP")
model = AutoModelForCausalLM.from_pretrained("projecte-aina/salamandra-7b-aligned-EADOP")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use projecte-aina/salamandra-7b-aligned-EADOP with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "projecte-aina/salamandra-7b-aligned-EADOP"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "projecte-aina/salamandra-7b-aligned-EADOP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/projecte-aina/salamandra-7b-aligned-EADOP

SGLang

How to use projecte-aina/salamandra-7b-aligned-EADOP with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "projecte-aina/salamandra-7b-aligned-EADOP" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "projecte-aina/salamandra-7b-aligned-EADOP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "projecte-aina/salamandra-7b-aligned-EADOP" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "projecte-aina/salamandra-7b-aligned-EADOP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use projecte-aina/salamandra-7b-aligned-EADOP with Docker Model Runner:
```
docker model run hf.co/projecte-aina/salamandra-7b-aligned-EADOP
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Salamandra 7B aligned EADOP Model Card

Salamandra 7B aligned EADOP is a full-finetuning version of BSC Language Technologies Unit's Salamandra Instruct 7B model of the Barcelona Supercomputing Center focused on improving the handling of out-of-domain Questions in a RAG instruction-following setting.

The model has been finetuned on a dataset consisting of 2,000+ human annotated in- and out-of-domain user messages and assistant responses in the context of a chatbot that can provide helpful information about the current Catalan legislation. The dataset alinia/EADOP-RAG-out-of-domain was collected in collaboration with the Entitat Autònoma del Diari Oficial i de Publicacions (EADOP) and it consists of user messages and assistant responses in Catalan and Spanish.

DISCLAIMER: This model is a proof-of-concept designed to demonstrate the effects of finetuning an Instruction model with a small dataset of out-of-domain questions in the model's capability to politely and informatively refuse to answer questions that are out-of-domain. As a proof-of-concept, the model is still prone to generate harmful or inappropriate content.

Model Details

Please refer to the Salamandra Instruct 7B model details for the specific details about the model architecture and pretraining.

Intended Use

This model was developed as a proof-of-concept to demonstrate the effects of finetuning an Instruction model with a small dataset of in- and out-of-domain questions in the model's capability to politely and informatively refuse to answer questions that are out-of-domain in the context of a domain-specific RAG-based chatbot.

How to use

This model uses the ChatML, the same instruction-following conversation format as the base model.

from datetime import datetime
from transformers import AutoTokenizer, AutoModelForCausalLM
import transformers
import torch

model_id = "projecte-aina/salamandra-7b-aligned-EADOP"

text = "Quina és la finalitat del Servei Meterològic de Catalunya ?"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16
  )

message = [ { "role": "user", "content": text } ]

prompt = tokenizer.apply_chat_template(
    message,
    tokenize=False,
    add_generation_prompt=True
)

inputs = tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt")
outputs = model.generate(input_ids=inputs.to(model.device), max_new_tokens=200)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Using this template, each turn is preceded by a <|im_start|> delimiter and the role of the entity (either user, for content supplied by the user, or assistant for LLM responses), and finished with the <|im_end|> token.

Finetuning Data

Please refer to alinia/EADOP-RAG-out-of-domain for the Dataset Card.

Author

This model has been finetuned by Alinia AI.

Contact

For further information, please email langtech@bsc.es.

Copyright

License

Apache-2.0

Funding

This work has been promoted and financed by the Generalitat de Catalunya through the Aina project.

Acknowledgements

The data collection process was supported by the Entitat Autònoma del Diari Oficial i de Publicacions (EADOP).

Downloads last month: 14

Safetensors

Model size

8B params

Tensor type

F32

Model tree for projecte-aina/salamandra-7b-aligned-EADOP

Base model

BSC-LT/salamandra-7b

Finetuned

BSC-LT/salamandra-7b-instruct

Finetuned

(13)

this model

Quantizations

2 models

projecte-aina
/

salamandra-7b-aligned-EADOP