Instructions to use sambanovasystems/SambaLingo-Turkish-Chat with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use sambanovasystems/SambaLingo-Turkish-Chat with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="sambanovasystems/SambaLingo-Turkish-Chat")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("sambanovasystems/SambaLingo-Turkish-Chat")
model = AutoModelForCausalLM.from_pretrained("sambanovasystems/SambaLingo-Turkish-Chat", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use sambanovasystems/SambaLingo-Turkish-Chat with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "sambanovasystems/SambaLingo-Turkish-Chat"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sambanovasystems/SambaLingo-Turkish-Chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/sambanovasystems/SambaLingo-Turkish-Chat

SGLang

How to use sambanovasystems/SambaLingo-Turkish-Chat with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "sambanovasystems/SambaLingo-Turkish-Chat" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sambanovasystems/SambaLingo-Turkish-Chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "sambanovasystems/SambaLingo-Turkish-Chat" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sambanovasystems/SambaLingo-Turkish-Chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use sambanovasystems/SambaLingo-Turkish-Chat with Docker Model Runner:
```
docker model run hf.co/sambanovasystems/SambaLingo-Turkish-Chat
```

How can I run it with 8GB of RAM

by whatnext - opened Mar 21, 2024

Discussion

whatnext

Mar 21, 2024

I have a poor Ryzen 5 8GB RAM laptop. I want to make a Turkish chatting AI.
I tried to run by following the example in readme file. I got an error message at first such as

  ValueError: Input length of input_ids is 21, but `max_length` is set to 20. This can lead to unexpected behavior. You should 
 consider increasing `max_length` or, better yet, setting `max_new_tokens`.

Chat-GPT helped me handle it.
Though the process continued for dozens of minutes it failed to output any result.
I tried tokens way but process kept killing itself after a while I ran it.

I am a noob regarding LLM.
I need your guidance.
How can I run it properly?

zolicsaki

SambaNova org Mar 21, 2024

•

edited Mar 21, 2024

Hey @whatnext , it looks like you do not have the proper hardware setup to run this model, the model weights are 13GB so 8GB RAM will not be enough to run this.

I have heard of llama cpp, which can help you run on a laptop, but I am not too familiar with this and not sure if it would work https://github.com/ggerganov/llama.cpp.

If I were you I would

Use our API we provide for free https://sambaverse.sambanova.net/
documentation: https://docs.sambanova.ai/sambaverse/latest/use-sambaverse.html#:~:text=Task%2Dspecific%20model-,Your%20API%20key,%27https%3A//sambaverse.sambanova.net/api/predict%27,-Rate%20limits
Run the model on the cloud, for example you can use some free GPU/TPU time on google colab, or some money to access more compute https://colab.google/. There are probably many other cloud offerings, I am not familiar with what the best options are.

We hope you enjoy our model and would appreciate if like and share our project for more visibility, thank you!

whatnext

Mar 23, 2024

@zolicsaki Thank you for your help. I want an offline chat model. I 'll try quantization technique of llama.cpp, though I don't know if it works.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment