Instructions to use CarrotAI/OpenCarrot-Mix-7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use CarrotAI/OpenCarrot-Mix-7B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="CarrotAI/OpenCarrot-Mix-7B")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("CarrotAI/OpenCarrot-Mix-7B")
model = AutoModelForCausalLM.from_pretrained("CarrotAI/OpenCarrot-Mix-7B")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use CarrotAI/OpenCarrot-Mix-7B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "CarrotAI/OpenCarrot-Mix-7B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CarrotAI/OpenCarrot-Mix-7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/CarrotAI/OpenCarrot-Mix-7B

SGLang

How to use CarrotAI/OpenCarrot-Mix-7B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "CarrotAI/OpenCarrot-Mix-7B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CarrotAI/OpenCarrot-Mix-7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "CarrotAI/OpenCarrot-Mix-7B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CarrotAI/OpenCarrot-Mix-7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use CarrotAI/OpenCarrot-Mix-7B with Docker Model Runner:
```
docker model run hf.co/CarrotAI/OpenCarrot-Mix-7B
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

output

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the linear merge method.

Models Merged

The following models were included in the merge:

Score

openai/gpt-4 : 0.6158
gemini-pro: 0.515
OpenCarrot-Mix-7B (this) : 0.4425
mistralai/Mixtral-8x7B-Instruct-v0.1 : 0.4304
openai/gpt-3.5-turbo : 0.4217

평가 지표	점수
AVG_llm_kr_eval	0.4425
EL	0.0522
FA	0.0865
NLI	0.6700
QA	0.5100
RC	0.8937
klue_ner_set_f1	0.0944
klue_re_exact_match	0.0100
kmmlu_preview_exact_match	0.4000
kobest_copa_exact_match	0.8200
kobest_hs_exact_match	0.5500
kobest_sn_exact_match	0.9800
kobest_wic_exact_match	0.6200
korea_cg_bleu	0.0865
kornli_exact_match	0.6400
korsts_pearson	0.8547
korsts_spearman	0.8464

LogicKor

카테고리	싱글 점수 평균	멀티 점수 평균
코딩(Coding)	7.71	7.71
수학(Math)	5.57	3.86
이해(Understanding)	6.86	8.14
추론(Reasoning)	8.14	6.43
글쓰기(Writing)	8.71	6.86
문법(Grammar)	5.29	2.29

카테고리	싱글 점수 평균	멀티 점수 평균
전체 싱글	7.05	5.88

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: amazingvince/Not-WizardLM-2-7B
    parameters:
      weight: 1.0
  - model: CarrotAI/OpenCarrot-Mistral-7B-Instruct-v0.2
    parameters:
      weight: 0.5
merge_method: linear
dtype: float16

Downloads last month: 7

Safetensors

Model size

7B params

Tensor type

F16

Model tree for CarrotAI/OpenCarrot-Mix-7B

Base model

amazingvince/Not-WizardLM-2-7B

Finetuned

(3)

this model

Quantizations

1 model

Paper for CarrotAI/OpenCarrot-Mix-7B

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

Paper • 2203.05482 • Published Mar 10, 2022 • 8