Instructions to use dphn/dolphin-2.2.1-mistral-7b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use dphn/dolphin-2.2.1-mistral-7b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="dphn/dolphin-2.2.1-mistral-7b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("dphn/dolphin-2.2.1-mistral-7b")
model = AutoModelForCausalLM.from_pretrained("dphn/dolphin-2.2.1-mistral-7b", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use dphn/dolphin-2.2.1-mistral-7b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "dphn/dolphin-2.2.1-mistral-7b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "dphn/dolphin-2.2.1-mistral-7b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/dphn/dolphin-2.2.1-mistral-7b

SGLang

How to use dphn/dolphin-2.2.1-mistral-7b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "dphn/dolphin-2.2.1-mistral-7b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "dphn/dolphin-2.2.1-mistral-7b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "dphn/dolphin-2.2.1-mistral-7b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "dphn/dolphin-2.2.1-mistral-7b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use dphn/dolphin-2.2.1-mistral-7b with Docker Model Runner:
```
docker model run hf.co/dphn/dolphin-2.2.1-mistral-7b
```

How to further fine-tune Samantha onto dolphin mistral

by unshadow - opened Nov 2, 2023

Discussion

unshadow

Nov 2, 2023

How would I, or what configuration would I need, to be able to use this dolphin mistral model as the starting chat model and then add your Samantha dataset to further fine-tune to add personality and therapist SME?

What settings would you recommend to combine both of your great projects? I like the idea of using this as an instruct model to add depth instead of the other dolphin chat model based on mistral chat that has the openai branding in its dataset or the other one that is from the base model that doesn't have as in-depth chat fine-tuning.

ehartford

Dolphin org Nov 2, 2023

There's an Axolotl config included in the model repo

You can tweak that, point it at dolphin as the base model, and point it at Samantha-1.1.jsonl as the dataset

unshadow

Nov 2, 2023

There's an Axolotl config included in the model repo

You can tweak that, point it at dolphin as the base model, and point it at Samantha-1.1.jsonl as the dataset

Thank you for the response!

Would you recommend I use this config file from the Samantha repo, https://huggingface.co/ehartford/samantha-1.2-mistral-7b/blob/main/configs/samantha-mistral-7b.yml , or the config file from this repo?

ehartford

Dolphin org Nov 2, 2023

Either file will work fine;
just make sure to change the base model and the dataset, according to what you want to train.

ehartford

Dolphin org Nov 2, 2023

if you aren't using 4x A100 80gb, then you will probably need to update the batch settings

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment