Instructions to use SherlockAssistant/Mistral-7B-Instruct-Ukrainian with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use SherlockAssistant/Mistral-7B-Instruct-Ukrainian with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="SherlockAssistant/Mistral-7B-Instruct-Ukrainian")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("SherlockAssistant/Mistral-7B-Instruct-Ukrainian")
model = AutoModelForCausalLM.from_pretrained("SherlockAssistant/Mistral-7B-Instruct-Ukrainian")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use SherlockAssistant/Mistral-7B-Instruct-Ukrainian with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "SherlockAssistant/Mistral-7B-Instruct-Ukrainian"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SherlockAssistant/Mistral-7B-Instruct-Ukrainian",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/SherlockAssistant/Mistral-7B-Instruct-Ukrainian

SGLang

How to use SherlockAssistant/Mistral-7B-Instruct-Ukrainian with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "SherlockAssistant/Mistral-7B-Instruct-Ukrainian" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SherlockAssistant/Mistral-7B-Instruct-Ukrainian",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "SherlockAssistant/Mistral-7B-Instruct-Ukrainian" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SherlockAssistant/Mistral-7B-Instruct-Ukrainian",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use SherlockAssistant/Mistral-7B-Instruct-Ukrainian with Docker Model Runner:
```
docker model run hf.co/SherlockAssistant/Mistral-7B-Instruct-Ukrainian
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Model card for Mistral-7B-Instruct-Ukrainian

Mistral-7B-UK is a Large Language Model finetuned for the Ukrainian language.

Mistral-7B-UK is trained using the following formula:

Initial finetuning of Mistral-7B-v0.2 using structured and unstructured datasets.
SLERP merge of the finetuned model with a model that performs better than Mistral-7B-v0.2 on OpenLLM benchmark: NeuralTrix-7B
DPO of the final model.

Instruction format

In order to leverage instruction fine-tuning, your prompt should be surrounded by [INST] and [/INST] tokens.

E.g.

text = "[INST]Відповідайте лише буквою правильної відповіді: Елементи експресіонізму наявні у творі: A. «Камінний хрест», B. «Інститутка», C. «Маруся», D. «Людина»[/INST]"

This format is available as a chat template via the apply_chat_template() method:

Model Architecture

This instruction model is based on Mistral-7B-v0.2, a transformer model with the following architecture choices:

Grouped-Query Attention
Sliding-Window Attention
Byte-fallback BPE tokenizer

Datasets - Structured

Datasets - Unstructured

Ukrainian Wiki

Datasets - DPO

Ukrainian translation of distilabel-indel-orca-dpo-pairs

💻 Usage

!pip install -qU transformers accelerate

from transformers import AutoTokenizer
import transformers
import torch

model = "SherlockAssistant/Mistral-7B-Instruct-Ukrainian"
messages = [{"role": "user", "content": "What is a large language model?"}]

tokenizer = AutoTokenizer.from_pretrained(model)
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])

Citation

If you are using this model in your research and publishing a paper, please help by citing our paper:

BIB

@inproceedings{boros-chivereanu-dumitrescu-purcaru-2024-llm-uk,
    title = "Fine-tuning and Retrieval Augmented Generation for Question Answering using affordable Large Language Models",
    author = "Boros, Tiberiu and Chivereanu, Radu and Dumitrescu, Stefan Daniel and Purcaru, Octavian",
    booktitle = "Proceedings of the Third Ukrainian Natural Language Processing Workshop, LREC-COLING",
    month = may,
    year = "2024",
    address = "Torino, Italy",
    publisher = "European Language Resources Association",
}

APA

Boros, T., Chivereanu, R., Dumitrescu, S., & Purcaru, O. (2024). Fine-tuning and Retrieval Augmented Generation for Question Answering using affordable Large Language Models. In Proceedings of the Third Ukrainian Natural Language Processing Workshop, LREC-COLING. European Language Resources Association.

MLA

Boros, Tiberiu, Radu, Chivereanu, Stefan Daniel, Dumitrescu, Octavian, Purcaru. "Fine-tuning and Retrieval Augmented Generation for Question Answering using affordable Large Language Models." Proceedings of the Third Ukrainian Natural Language Processing Workshop, LREC-COLING. European Language Resources Association, 2024.

Chicago

Boros, Tiberiu, Radu, Chivereanu, Stefan Daniel, Dumitrescu, and Octavian, Purcaru. "Fine-tuning and Retrieval Augmented Generation for Question Answering using affordable Large Language Models." . In Proceedings of the Third Ukrainian Natural Language Processing Workshop, LREC-COLING. European Language Resources Association, 2024.