Instructions to use jondurbin/airoboros-13b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use jondurbin/airoboros-13b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="jondurbin/airoboros-13b", device_map="auto")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("jondurbin/airoboros-13b")
model = AutoModelForCausalLM.from_pretrained("jondurbin/airoboros-13b", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use jondurbin/airoboros-13b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jondurbin/airoboros-13b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jondurbin/airoboros-13b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/jondurbin/airoboros-13b

SGLang

How to use jondurbin/airoboros-13b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "jondurbin/airoboros-13b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jondurbin/airoboros-13b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "jondurbin/airoboros-13b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jondurbin/airoboros-13b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use jondurbin/airoboros-13b with Docker Model Runner:
```
docker model run hf.co/jondurbin/airoboros-13b
```

prompt format

by underlines - opened May 22, 2023

Discussion

underlines

May 22, 2023

Thanks for the dataset release and the great model!

Is there any specific prompt format to be followed?

jondurbin

Owner May 22, 2023

•

edited May 22, 2023

I used the default template in fastchat to fine-tune the model, so it should be the same as vicuna, etc., something like:

A chat between a curious user and an artificial intelligence assistant.  The assistant gives helpful, detailed and polite answers to the user's instructions.

USER: [prompt]
<\s>

ASSISTANT:

underlines

May 22, 2023

Ok, because in your Repo, the example was:

Human: Write an email introducing a new instruction-tuned AI model named airoboros.  The key features about airoboros are: 1. it was fine-tuned with entirely machine-generated synthetic instruction/response pairs  2. the 7 billion parameter model performs very well in comparison to 13b models  3. the synthetic prompt generation tool is free, open-source, and highly customizable.
Assistant: Subject: Introducing airoboros - a new instruction-tuned AI model

Dear [Recipient],

So I guess the model is smart enough to deduce the prefixes, no matter if it's USER/ASSISTANT or HUMAN/ASSISTANT etc...

jondurbin

Owner May 22, 2023

Ah, yeah that's the older version that I trained using a previous version of FastChat (original vicuna, not 1.1 with updated stop token). I'll add it to the model card.

Qubitium

May 24, 2023

@jondurbin To clarify, was this model trained with 1.1 of vicuna token format or 1.0? Thanks.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment