Instructions to use Equall/Saul-7B-Base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Equall/Saul-7B-Base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Equall/Saul-7B-Base")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Equall/Saul-7B-Base")
model = AutoModelForCausalLM.from_pretrained("Equall/Saul-7B-Base")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Local Apps Settings

vLLM

How to use Equall/Saul-7B-Base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Equall/Saul-7B-Base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Equall/Saul-7B-Base",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Equall/Saul-7B-Base

SGLang

How to use Equall/Saul-7B-Base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Equall/Saul-7B-Base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Equall/Saul-7B-Base",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Equall/Saul-7B-Base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Equall/Saul-7B-Base",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Equall/Saul-7B-Base with Docker Model Runner:
```
docker model run hf.co/Equall/Saul-7B-Base
```

Easily Accesible?

by aibasics - opened Mar 23, 2024

Discussion

aibasics

Mar 23, 2024

Non-technical here, sorry. Is there any way you can just straight up chat with it without knowing how to code? Thanks.

PierreColombo

Equall org Mar 23, 2024

Ask maybe the HF people. @lhoestq ?

sirrosendo

Mar 24, 2024

From my understanding if you utilize the right AI coding if you know how to leverage several of them is something you don't really need it's not the coding that matters anymore it's the prompt engineering explaining to the AI what you need precisely the AI does all the work for you that's why though you have to leverage more than one AI you can't just use one you have to leverage multiple eyes that have been trained on multiple data sets this way each one of them double checks the work of the other so you have a seamless workflow. Now I apologies I'm no expert at this I've only been learning since the first day that these AI came onto the market I've been training and constantly leveraging them trying to understand how they think and trying to understand how to solve the Black box problem.

lhoestq

Mar 25, 2024

This model is not enabled to get a Demo widget, but you can still build a gradio Demo app and host it in HF Spaces using this guide @PierreColombo : https://www.gradio.app/guides/creating-a-chatbot-fast
It should make it easier for people to play with this model.

bencser

Apr 3, 2024

install https://lmstudio.ai/

sirrosendo

May 1, 2024

You can create a chat interface for Saul-Base using Gradio and host it on Hugging Face Spaces. This will make it easier for people to interact with the model. While some coding knowledge is required, this guide should help you get started: https://www.gradio.app/guides/creating-a-chatbot-fast

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment