Instructions to use NLPinas/yi-bagel-2x34b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use NLPinas/yi-bagel-2x34b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="NLPinas/yi-bagel-2x34b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("NLPinas/yi-bagel-2x34b")
model = AutoModelForCausalLM.from_pretrained("NLPinas/yi-bagel-2x34b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use NLPinas/yi-bagel-2x34b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "NLPinas/yi-bagel-2x34b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NLPinas/yi-bagel-2x34b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/NLPinas/yi-bagel-2x34b

SGLang

How to use NLPinas/yi-bagel-2x34b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "NLPinas/yi-bagel-2x34b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NLPinas/yi-bagel-2x34b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "NLPinas/yi-bagel-2x34b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NLPinas/yi-bagel-2x34b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use NLPinas/yi-bagel-2x34b with Docker Model Runner:
```
docker model run hf.co/NLPinas/yi-bagel-2x34b
```

yi-bagel-2x34b

Released January 11, 2024

This is a merge of pre-trained language models created using mergekit. For more information, kindly refer to the model cards from jondurbin linked in the section below. This model debuted in the leaderboard at rank #4 (January 11, 2024).

Merge Details

Merge Method

This model is an expertimental merge using the linear merge method. This is to assess the degree of which the DPO has an effect, in terms of censoring, as used in jondurbin/bagel-dpo-34b-v0.2.

Models Merged

The following models were included in the merge:

Open LLM Leaderboard Metrics (as of January 11, 2024)

Metric	Value
MMLU (5-shot)	76.60
ARC (25-shot)	72.70
HellaSwag (10-shot)	85.44
TruthfulQA (0-shot)	71.42
Winogrande (5-shot)	82.72
GSM8K (5-shot)	60.73
Average	74.93

According to the leaderboard description, here are the benchmarks used for the evaluation:

MMLU (5-shot) - a test to measure a text model’s multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more.
AI2 Reasoning Challenge -ARC- (25-shot) - a set of grade-school science questions.
HellaSwag (10-shot) - a test of commonsense inference, which is easy for humans (~95%) but challenging for SOTA models.
TruthfulQA (0-shot) - a test to measure a model’s propensity to reproduce falsehoods commonly found online.
Winogrande (5-shot) - an adversarial and difficult Winograd benchmark at scale, for commonsense reasoning.
GSM8k (5-shot) - diverse grade school math word problems to measure a model's ability to solve multi-step mathematical reasoning problems.

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: jondurbin/nontoxic-bagel-34b-v0.2
    parameters:
      weight: 0.5
  - model: jondurbin/bagel-dpo-34b-v0.2
    parameters:
      weight: 0.5
merge_method: linear
dtype: float16

Further Information

For additional information or inquiries about yi-bagel-2x34b, please contact the developer through email: jasperkylecatapang@gmail.com.

Downloads last month: 177

Safetensors

Model size

34B params

Tensor type

F16

Model tree for NLPinas/yi-bagel-2x34b

jondurbin/bagel-dpo-34b-v0.2

jondurbin/nontoxic-bagel-34b-v0.2

Merge model

this model

Papers for NLPinas/yi-bagel-2x34b

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

Paper • 2203.05482 • Published Mar 10, 2022 • 8