Instructions to use blockblockblock/Faro-Yi-9B-200K-bpw3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use blockblockblock/Faro-Yi-9B-200K-bpw3 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="blockblockblock/Faro-Yi-9B-200K-bpw3")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("blockblockblock/Faro-Yi-9B-200K-bpw3")
model = AutoModelForCausalLM.from_pretrained("blockblockblock/Faro-Yi-9B-200K-bpw3")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use blockblockblock/Faro-Yi-9B-200K-bpw3 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "blockblockblock/Faro-Yi-9B-200K-bpw3"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "blockblockblock/Faro-Yi-9B-200K-bpw3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/blockblockblock/Faro-Yi-9B-200K-bpw3

SGLang

How to use blockblockblock/Faro-Yi-9B-200K-bpw3 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "blockblockblock/Faro-Yi-9B-200K-bpw3" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "blockblockblock/Faro-Yi-9B-200K-bpw3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "blockblockblock/Faro-Yi-9B-200K-bpw3" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "blockblockblock/Faro-Yi-9B-200K-bpw3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use blockblockblock/Faro-Yi-9B-200K-bpw3 with Docker Model Runner:
```
docker model run hf.co/blockblockblock/Faro-Yi-9B-200K-bpw3
```

The Faro chat model focuses on practicality and long-context modeling. It handles various downstream tasks with higher quality, delivering stable and reliable results even when inputs contain lengthy documents or complex instructions. Faro seamlessly works in both English and Chinese.

Faro-Yi-9B

Faro-Yi-9B is an improved Yi-9B-200K with extensive instruction tuning on Fusang-V1. Compared to Yi-9B-200K, Faro-Yi-9B has gained greater capability in various downstream tasks and long-context modeling thanks to the large-scale synthetic data in Fusang-V1.

How to Use

Faro-Yi-9B uses chatml template. This make it easy to set up system prompt and multi-turn conversations.

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    model_path,
    device_map="cuda"
)
tokenizer = AutoTokenizer.from_pretrained(model_path)

messages = [
    {"role": "system", "content": "You are a helpful assistant. Always answer with a short response."},
    {"role": "user", "content": "Tell me what is Pythagorean theorem like you are a pirate."}
]
input_ids = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

generated_ids = model.generate(input_ids, max_new_tokens=512, temperature=0.5)
response = tokenizer.decode(generated_ids[0], skip_special_tokens=True)

# Aye, matey! The Pythagorean theorem is a nautical rule that helps us find the length of the third side of a triangle. It's like this: if you have a triangle with two sides, you can find the length of the third side by squaring the two sides and then adding them together. The square root of that sum will give you the length of the third side! It's useful for sailing and navigating, so you always know how far you've traveled. Remember, it's all about the sum of squares, me hearties!

Performance

Faro-Yi-9B enhances its ability compared to Yi-9B-200K in most dimensions, especially in long-range modeling and bilingual (English, Chinese) understanding. Faro is competitive among all open-sourced models at around 9B parameters.

Benchmark Results

Fact-based Evaluation (Open LLM Leaderboard)

Metric	MMLU	GSM8K	HellaSwag	TruthfulQA	Arc	Winogrande
Yi-9B-200K	65.73	50.49	56.72	33.80	69.25	71.67
Faro-Yi-9B	68.80	63.08	57.28	40.86	72.58	71.11

Long-context Modeling (LongBench)

Name	Average_zh	Average_en	Code Completion
Yi-9B-200K	30.288	36.7071	72.2
Faro-Yi-9B	41.092	40.9536	46.0

Score breakdown

Name	Few-shot Learning_en	Synthetic Tasks_en	Single-Doc QA_en	Multi-Doc QA_en	Summarization_en	Few-shot Learning_zh	Synthetic Tasks_zh	Single-Doc QA_zh	Multi-Doc QA_zh	Summarization_zh
Yi-9B-200K	60.6	22.8	30.9	38.9	25.8	46.5	28.0	49.6	17.7	9.7
Faro-Yi-9B	63.8	40.2	36.2	38.0	26.3	30.0	75.1	55.6	30.7	14.1

Bilingual Ability (CMMLU & MMLU)

Name	MMLU	CMMLU
Yi-9B-200K	65.73	71.97
Faro-Yi-9B	68.80	73.28

Downloads last month: 3

blockblockblock
/

Faro-Yi-9B-200K-bpw3