Instructions to use mlabonne/NeuralDaredevil-7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mlabonne/NeuralDaredevil-7B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mlabonne/NeuralDaredevil-7B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("mlabonne/NeuralDaredevil-7B")
model = AutoModelForCausalLM.from_pretrained("mlabonne/NeuralDaredevil-7B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use mlabonne/NeuralDaredevil-7B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "mlabonne/NeuralDaredevil-7B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mlabonne/NeuralDaredevil-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/mlabonne/NeuralDaredevil-7B

SGLang

How to use mlabonne/NeuralDaredevil-7B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mlabonne/NeuralDaredevil-7B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mlabonne/NeuralDaredevil-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mlabonne/NeuralDaredevil-7B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mlabonne/NeuralDaredevil-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use mlabonne/NeuralDaredevil-7B with Docker Model Runner:
```
docker model run hf.co/mlabonne/NeuralDaredevil-7B
```

Prompt Template?

by amgadhasan - opened Jul 13, 2024

Discussion

amgadhasan

Jul 13, 2024

Thanks for sharing this model.

What is the correct prompt template for chatting with this model?

I tried using tokenizer.apply_chat_template but it says there is no bundled chat template:

$ from transformers import AutoTokenizer
$ import transformers
$ import torch

$ model = "mlabonne/NeuralDaredevil-7B"
$messages = [{"role": "user", "content": "What is Deep Learning?"}]

$ tokenizer = AutoTokenizer.from_pretrained(model)
$ prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
No chat template is set for this tokenizer, falling back to a default class-level template. This is very error-prone, because models are often trained with templates different from the class default! Default chat templates are a legacy feature and will be removed in Transformers v4.43, at which point any code depending on them will stop working. We recommend setting a valid chat template before then to ensure that this model continues working without issues.

Thanks!

mlabonne

Owner Jul 13, 2024

Oops sorry about that! Mistral Instruct is the correct prompt template for this model.

amgadhasan

Jul 13, 2024

Oops sorry about that! Mistral Instruct is the correct prompt template for this model.

Thanks for answering.
I opened 2 PRs to solve this.

mlabonne

Owner Jul 14, 2024

Thanks a lot @amgadhasan , it's now merged!

amgadhasan changed discussion status to closed Jul 14, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment