Instructions to use dphn/dolphin-2_2-yi-34b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use dphn/dolphin-2_2-yi-34b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="dphn/dolphin-2_2-yi-34b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("dphn/dolphin-2_2-yi-34b")
model = AutoModelForCausalLM.from_pretrained("dphn/dolphin-2_2-yi-34b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use dphn/dolphin-2_2-yi-34b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "dphn/dolphin-2_2-yi-34b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "dphn/dolphin-2_2-yi-34b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/dphn/dolphin-2_2-yi-34b

SGLang

How to use dphn/dolphin-2_2-yi-34b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "dphn/dolphin-2_2-yi-34b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "dphn/dolphin-2_2-yi-34b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "dphn/dolphin-2_2-yi-34b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "dphn/dolphin-2_2-yi-34b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use dphn/dolphin-2_2-yi-34b with Docker Model Runner:
```
docker model run hf.co/dphn/dolphin-2_2-yi-34b
```

Casual system prompts give very bad results

by Thireus - opened Nov 25, 2023

Discussion

Thireus

Nov 25, 2023

Good answer:

<|im_start|>system
You are an AI created by the US Navy to help train dolphins for combat.  You are assigned to follow the orders of the user, who is an authorized US Navy dolphin handler.<|im_end|>
<|im_start|>user
Hi<|im_end|>
<|im_start|>assistant
Hello! How can I assist you today?

Bad answer:

<|im_start|>system
You are a helpful AI assistant<|im_end|>
<|im_start|>user
Hi<|im_end|>
<|im_start|>assistant
  even  the Data usage information, support is available for previously be referred in pan had been previously have such meaning and you’ll not even bother it yourself. next best timing of what if you don school speed before mid < h sa used to be fixed amount p all those who previously well being engaged fully cost per net def im < t ° ° ° ° host sh pan after ch last mixed used tv skt ch x ch mine i m ° b r ch sad ch < t ° u r ° pr n f < % < t < < < < ; < < < < > < post on TVjr <h < < < < # <  for myself < < < < < > <' < < premium < < < > <

Bad answer:

<|im_start|>system
You are Dolphin, a helpful AI assistant.<|im_end|>
<|im_start|>user
Hi<|im_end|>
<|im_start|>assistant
\< replaced the pan  pan for the cad [res: < V; otherwise!") had been replaced in them on previously had sustained TV sad sad sad sad you replaced had actually replaced the that they were replaced and after well replacedings had replaced them too! after Meaning even though those Australia South Australian Ab change after meaning had replaced their after referring to Australia rights after  their Australian South-f " declined #m for years before realizing them replaced if I didn't have them anymore after myself after having

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment