Instructions to use QuantFactory/Sailor2-L-8B-Chat-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use QuantFactory/Sailor2-L-8B-Chat-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="QuantFactory/Sailor2-L-8B-Chat-GGUF")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("QuantFactory/Sailor2-L-8B-Chat-GGUF", dtype="auto")

llama-cpp-python

How to use QuantFactory/Sailor2-L-8B-Chat-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="QuantFactory/Sailor2-L-8B-Chat-GGUF",
	filename="Sailor2-L-8B-Chat.Q2_K.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use QuantFactory/Sailor2-L-8B-Chat-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf QuantFactory/Sailor2-L-8B-Chat-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf QuantFactory/Sailor2-L-8B-Chat-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf QuantFactory/Sailor2-L-8B-Chat-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf QuantFactory/Sailor2-L-8B-Chat-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf QuantFactory/Sailor2-L-8B-Chat-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf QuantFactory/Sailor2-L-8B-Chat-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf QuantFactory/Sailor2-L-8B-Chat-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf QuantFactory/Sailor2-L-8B-Chat-GGUF:Q4_K_M

Use Docker

docker model run hf.co/QuantFactory/Sailor2-L-8B-Chat-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use QuantFactory/Sailor2-L-8B-Chat-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "QuantFactory/Sailor2-L-8B-Chat-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantFactory/Sailor2-L-8B-Chat-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/QuantFactory/Sailor2-L-8B-Chat-GGUF:Q4_K_M

SGLang

How to use QuantFactory/Sailor2-L-8B-Chat-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "QuantFactory/Sailor2-L-8B-Chat-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantFactory/Sailor2-L-8B-Chat-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "QuantFactory/Sailor2-L-8B-Chat-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantFactory/Sailor2-L-8B-Chat-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use QuantFactory/Sailor2-L-8B-Chat-GGUF with Ollama:
```
ollama run hf.co/QuantFactory/Sailor2-L-8B-Chat-GGUF:Q4_K_M
```

Unsloth Studio new

How to use QuantFactory/Sailor2-L-8B-Chat-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for QuantFactory/Sailor2-L-8B-Chat-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for QuantFactory/Sailor2-L-8B-Chat-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for QuantFactory/Sailor2-L-8B-Chat-GGUF to start chatting

Docker Model Runner
How to use QuantFactory/Sailor2-L-8B-Chat-GGUF with Docker Model Runner:
```
docker model run hf.co/QuantFactory/Sailor2-L-8B-Chat-GGUF:Q4_K_M
```

Lemonade

How to use QuantFactory/Sailor2-L-8B-Chat-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull QuantFactory/Sailor2-L-8B-Chat-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Sailor2-L-8B-Chat-GGUF-Q4_K_M

List all available models

lemonade list

QuantFactory/Sailor2-L-8B-Chat-GGUF

This is quantized version of sail/Sailor2-L-8B-Chat created using llama.cpp

Original Model Card

The logo was generated by MidJourney

Sailor2 is a community-driven initiative that brings cutting-edge multilingual language models to South-East Asia (SEA). Our research highlights a strong demand for models in the 8B and 20B parameter range for production use, alongside 1B models for specialized applications, such as speculative decoding and research purposes. These models, released under the Apache 2.0 license, provide enhanced accessibility to advanced language technologies across the region.

Sailor2 builds upon the foundation of the awesome multilingual model Qwen 2.5 and is continuously pre-trained on 500B tokens to support 15 languages better with a unified model. These languages include English, Chinese, Burmese, Cebuano, Ilocano, Indonesian, Javanese, Khmer, Lao, Malay, Sundanese, Tagalog, Thai, Vietnamese, and Waray. By addressing the growing demand for diverse, robust, and accessible language models, Sailor2 seeks to serve the underserved in SEA areas with open, inclusive, and accessible multilingual LLMs. The Sailor2 model comes in three sizes, 1B, 8B, and 20B, which are expanded from the Qwen2.5 base models of 0.5B, 7B, and 14B, respectively.

Model Summary

Model Collections: Base Model & Chat Model
Project Website: sea-sailor.github.io/blog/sailor2/
Codebase: github.com/sail-sg/sailor2
Technical Report: Sailor2 Report

Training details

During development, we employ a range of advanced technologies to ensure top-tier performance and efficiency:

model expansion
optimized data mixing strategies
multi-stage pre-training protocols
advanced multilingual post-training

Please refer to Sailor2 Blog for more training details.

Requirements

The code of Sailor2 has been in the latest Hugging face transformers and we advise you to install transformers==4.46.3.

Quickstart

Here provides a code snippet to show you how to load the tokenizer and model and how to generate contents.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda"

model = AutoModelForCausalLM.from_pretrained(
    'sail/Sailor2-20B-Chat',
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained('sail/Sailor2-20B-Chat')
system_prompt= \
'You are an AI assistant named Sailor2, created by Sea AI Lab. \
As an AI assistant, you can answer questions in English, Chinese, and Southeast Asian languages \
such as Burmese, Cebuano, Ilocano, Indonesian, Javanese, Khmer, Lao, Malay, Sundanese, Tagalog, Thai, Vietnamese, and Waray. \
Your responses should be friendly, unbiased, informative, detailed, and faithful.'

prompt = "Beri saya pengenalan singkat tentang model bahasa besar."
# prompt = "Hãy cho tôi một giới thiệu ngắn gọn về mô hình ngôn ngữ lớn."
# prompt = "ให้ฉันแนะนำสั้น ๆ เกี่ยวกับโมเดลภาษาขนาดใหญ่"

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

model_inputs = tokenizer([text], return_tensors="pt").to(device)
input_ids = model_inputs.input_ids.to(device)

generated_ids = model.generate(
    input_ids,
    max_new_tokens=512,
)

generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)

License

Sailor2 is distributed under the terms of the Apache License 2.0. No restrict on the research and the commercial use.

Citation

If you find Sailor2 useful, please cite our work as follows:

@article{sailor2report,
  title  = {Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLM},
  author = {Longxu Dou and Qian Liu and Fan Zhou and Changyu Chen and Zili Wang and Ziqi Jin and Zichen Liu and Tongyao Zhu and Cunxiao Du and Penghui Yang and Haonan Wang and Jiaheng Liu and Yongchi Zhao and Xiachong Feng and Xin Mao and Man Tsung Yeung and Kunat Pipatanakul and Fajri Koto and Min Si Thu and Hynek Kydl{\'\i}{\v{c}}ek and Zeyi Liu and Qunshu Lin and Sittipong Sripaisarnmongkol and Kridtaphad Sae-Khow and Nirattisai Thongchim and Taechawat Konkaew and Narong Borijindargoon and Anh Dao and Matichon Maneegard and Phakphum Artkaew and Zheng-Xin Yong and Quan Nguyen and Wannaphong Phatthiyaphaibun and Hoang H. Tran and Mike Zhang and Shiqi Chen and Tianyu Pang and Chao Du and Xinyi Wan and Wei Lu and Min Lin},
  journal={arXiv preprint arXiv:2502.12982},
  year   = {2025}
}

Contact Us

If you have any questions, please raise an issue or contact us at doulx@sea.com or liuqian.sea@gmail.com.

Downloads last month: 80

GGUF

Model size

9B params

Architecture

qwen2

Hardware compatibility

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

Model tree for QuantFactory/Sailor2-L-8B-Chat-GGUF

Base model

Qwen/Qwen2.5-7B

Finetuned

sail/Sailor2-8B

Finetuned

sail/Sailor2-L-8B

Quantized

(2)

this model

Paper for QuantFactory/Sailor2-L-8B-Chat-GGUF

Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs

Paper • 2502.12982 • Published Feb 18, 2025 • 19