Instructions to use OpenAssistant/falcon-40b-sft-mix-1226 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use OpenAssistant/falcon-40b-sft-mix-1226 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="OpenAssistant/falcon-40b-sft-mix-1226", trust_remote_code=True)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("OpenAssistant/falcon-40b-sft-mix-1226", trust_remote_code=True, dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use OpenAssistant/falcon-40b-sft-mix-1226 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "OpenAssistant/falcon-40b-sft-mix-1226"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenAssistant/falcon-40b-sft-mix-1226",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/OpenAssistant/falcon-40b-sft-mix-1226

SGLang

How to use OpenAssistant/falcon-40b-sft-mix-1226 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "OpenAssistant/falcon-40b-sft-mix-1226" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenAssistant/falcon-40b-sft-mix-1226",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "OpenAssistant/falcon-40b-sft-mix-1226" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenAssistant/falcon-40b-sft-mix-1226",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use OpenAssistant/falcon-40b-sft-mix-1226 with Docker Model Runner:
```
docker model run hf.co/OpenAssistant/falcon-40b-sft-mix-1226
```

Open-Assistant Falcon 40B SFT MIX Model

This model is a fine-tuning of TII's Falcon 40B LLM. It was trained on a mixture of OASST top-2 threads (exported on June 2, 2023), Dolly-15k and synthetic instruction datasets (see dataset configuration below).

Model Details

Finetuned from: tiiuae/falcon-40b
Model type: Causal decoder-only transformer language model
Language: English, German, Spanish, French (and limited capabilities in Italian, Portuguese, Polish, Dutch, Romanian, Czech, Swedish);
Demo: Continuations for 250 random prompts, multiligual-60
Eval results: ilm-eval
Weights & Biases: Training log (checkpoint: 1226 steps)
License: Apache 2.0
Contact: Open-Assistant Discord

Prompting

Input prompt example:

<|prompter|>What is a meme, and what's the history behind this word?<|endoftext|><|assistant|>

The input ends with the <|assistant|> token to signal that the model should start generating the assistant reply.

Configuration Details

Model:

falcon-40b:
  dtype: bf16
  learning_rate: 1e-5
  model_name: "tiiuae/falcon-40b"
  deepspeed_config: configs/zero3_config_falcon.json
  weight_decay: 0.0
  max_length: 2048
  warmup_steps: 20
  gradient_checkpointing: true
  gradient_accumulation_steps: 1
  per_device_train_batch_size: 18
  per_device_eval_batch_size: 10
  eval_steps: 120
  save_strategy: steps
  save_steps: 613
  num_train_epochs: 8
  save_total_limit: 4
  use_flash_attention: false
  residual_dropout: 0.3
  residual_dropout_lima: true

Dataset:

sft9-stage2:
  # oasst_export: 100.00% (29899)
  # vicuna: 50.00% (16963)
  # code_alpaca: 50.00% (9510)
  # oa_wiki_qa_bart_10000row: 100.00% (9434)
  # grade_school_math_instructions: 100.00% (8351)
  # dolly15k: 100.00% (14250)

  use_custom_sampler: true
  datasets:
    - oasst_export:
        lang: "bg,ca,cs,da,de,en,es,fr,hr,hu,it,nl,pl,pt,ro,ru,sl,sr,sv,uk" # sft-8.0
        input_file_path: 2023-06-02_oasst_all_labels.jsonl.gz
        val_split: 0.05
        top_k: 2
    - vicuna:
        fraction: 0.5
        val_split: 0.025
        max_val_set: 250
    - code_alpaca:
        fraction: 0.5
        val_split: 0.05
        max_val_set: 250
    - oa_wiki_qa_bart_10000row:
        val_split: 0.05
        max_val_set: 250
    - grade_school_math_instructions:
        val_split: 0.05
    - dolly15k:
        val_split: 0.05
        max_val_set: 300

Downloads last month: 21

OpenAssistant
/

falcon-40b-sft-mix-1226

Open-Assistant Falcon 40B SFT MIX Model

Model Details

Prompting

Configuration Details

Datasets used to train OpenAssistant/falcon-40b-sft-mix-1226

Spaces using OpenAssistant/falcon-40b-sft-mix-1226 6