Instructions to use myrkur/shotor with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use myrkur/shotor with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="myrkur/shotor")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("myrkur/shotor")
model = AutoModelForCausalLM.from_pretrained("myrkur/shotor")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use myrkur/shotor with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "myrkur/shotor"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "myrkur/shotor",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/myrkur/shotor

SGLang

How to use myrkur/shotor with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "myrkur/shotor" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "myrkur/shotor",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "myrkur/shotor" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "myrkur/shotor",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use myrkur/shotor with Docker Model Runner:
```
docker model run hf.co/myrkur/shotor
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Shotor (Llama 3 8B Instruction Tuned on Farsi)

Shotor is a Persian language model built upon the llama 3 8B architecture, a multilingual Large Language Model (LLM). It has been fine-tuned using supervised learning techniques and the Dora method for efficient fine-tuning. The model has been specifically tailored and trained on Persian datasets, particularly leveraging the dataset provided by persian-alpaca-deep-clean.

Usage

Here's a sample Python code snippet demonstrating how to use Shotor for text generation:

import transformers
import torch

# Load the Shotor model
model_id = "myrkur/shotor"
pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

# Define user messages
messages = [
    {"role": "user", "content": "علم بهتر است یا ثروت؟"},
]

# Apply chat template and generate text
prompt = pipeline.tokenizer.apply_chat_template(
        messages, 
        tokenize=False, 
        add_generation_prompt=True
)

terminators = [
    pipeline.tokenizer.eos_token_id,
    pipeline.tokenizer.convert_tokens_to_ids("<|eot_id|>")
]

outputs = pipeline(
    prompt,
    max_new_tokens=512,
    eos_token_id=terminators,
    do_sample=True,
    temperature=0.5,
    top_p=0.9,
    repetition_penalty=1.1
)
print(outputs[0]["generated_text"][len(prompt):])

Contributions

Contributions to Shotor are welcome! Whether it's enhancing the model's capabilities, improving its performance on specific tasks, or evaluating its performance, your contributions can help advance Persian natural language processing.