Instructions to use dphn/dolphin-2.1-70b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use dphn/dolphin-2.1-70b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="dphn/dolphin-2.1-70b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("dphn/dolphin-2.1-70b")
model = AutoModelForCausalLM.from_pretrained("dphn/dolphin-2.1-70b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use dphn/dolphin-2.1-70b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "dphn/dolphin-2.1-70b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "dphn/dolphin-2.1-70b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/dphn/dolphin-2.1-70b

SGLang

How to use dphn/dolphin-2.1-70b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "dphn/dolphin-2.1-70b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "dphn/dolphin-2.1-70b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "dphn/dolphin-2.1-70b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "dphn/dolphin-2.1-70b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use dphn/dolphin-2.1-70b with Docker Model Runner:
```
docker model run hf.co/dphn/dolphin-2.1-70b
```

eos token seems not working

by ehartford - opened Oct 23, 2023

Discussion

ehartford

Dolphin org Oct 23, 2023

maybe there's a quick fix? I'm digging.

ehartford

Dolphin org Oct 24, 2023

I have discovered a fix that requires retraining the model. In progress.
This model still completely blows me away with its intelligence.

CosmicCat

Oct 24, 2023

How do you mean EOS token does not work?

ehartford

Dolphin org Oct 24, 2023

It just keeps talking and doesn't stop

mayeaux

Oct 24, 2023

I have discovered a fix that requires retraining the model. In progress.
This model still completely blows me away with its intelligence.

Looking forward to it! Though I will have to wait for a TheBloke quantization.

Congrats on the model!

ehartford

Dolphin org Oct 26, 2023

There is a workaround:

In your system message, tell the model to write <|FINISHED|> when it is finished

Then have your client stop generating when it sees that string.

gotzmann

Oct 26, 2023

Argh, that's the known problem :) Yeah, adding <|FINISHED|> might work as a temporary workaround. But I'd prefer to see this model (trained on StellarBright again) with correct EOS handling too

ehartford

Dolphin org Oct 26, 2023

Yeah I decided to go ahead and use StellarBright since the OpenAI stuff only comes out when you deviate from ChatML

ehartford

Dolphin org Oct 26, 2023

I am training it again but - suddenly Azure is WAY more popular so my spot instance keeps getting evicted. It's gotten much worse in the last week.

ehartford

Dolphin org Oct 31, 2023

dolphin-2.2-70b will be finished in about a week.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment