Instructions to use tiiuae/falcon-7b-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tiiuae/falcon-7b-instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="tiiuae/falcon-7b-instruct", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-7b-instruct", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-7b-instruct", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tiiuae/falcon-7b-instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tiiuae/falcon-7b-instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiiuae/falcon-7b-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/tiiuae/falcon-7b-instruct

SGLang

How to use tiiuae/falcon-7b-instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tiiuae/falcon-7b-instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiiuae/falcon-7b-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tiiuae/falcon-7b-instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiiuae/falcon-7b-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use tiiuae/falcon-7b-instruct with Docker Model Runner:
```
docker model run hf.co/tiiuae/falcon-7b-instruct
```

falcon-7b-instruct

Commit History

Adding `safetensors` variant of this model (#126)

8782b5c
verified

ybelkada

SFconvertbot commited on Oct 12, 2024

enh: add link to new model

b22819d
verified

ybelkada commited on Sep 28, 2024

add chat template to tokenizer_config.json (#111)

0b9684d
verified

nilabhra

epignatelli commited on Sep 16, 2024

Move to in-libary checkpoint (for real this time) (#88)

cf4b3c4

Rocketknight1 HF Staff commited on Sep 29, 2023

Revert in-library PR (#63)

eb410fb

sgugger commited on Jul 13, 2023

Move to in-library checkpoint (#56)

f8dac3f

FalconLLM commited on Jul 12, 2023

Add widget examples

c7f670a

FalconLLM commited on Jun 26, 2023

Add recommendations for use

9f16e66

FalconLLM commited on Jun 9, 2023

Update citation info

8281cc0

FalconLLM commited on Jun 5, 2023

Add hf endpoint handler.py (#24)

7f5eb0f

FalconLLM

olivierdehaene commited on Jun 5, 2023

Update metadata

a1d11a0

FalconLLM commited on Jun 5, 2023

Remove TII Falcon LLM license

22225c3

FalconLLM commited on Jun 1, 2023

Update license information to Apache 2.0

43b757f

FalconLLM commited on May 31, 2023

Update README.md (#14)

a8b5565

victor HF Staff commited on May 31, 2023

license: tii-falcon-llm (#11)

6677bc9

FalconLLM

julien-c HF Staff commited on May 30, 2023

Update config.json

9cbb161

FalconLLM commited on May 30, 2023

Update modelling_RW.py

93b55b6

FalconLLM commited on May 30, 2023

coreml-weights (#8)

983c105

FalconLLM

pcuenq HF Staff commited on May 30, 2023

Add PyTorch 2.0 notice

b6efaea

slippylolo commited on May 26, 2023

Add evaluation disclaimer

6940d5f

slippylolo commited on May 26, 2023

Update README.md

5aad858

slippylolo commited on May 26, 2023

Update configuration_RW.py

69601d5

Daniel Hesslow commited on May 25, 2023

Update modelling_RW.py

8d630c5

Daniel Hesslow commited on May 25, 2023

Update README.md

54b3714

slippylolo commited on May 25, 2023

Create LICENSE.txt

7ea4d00

slippylolo commited on May 25, 2023

Update modelling_RW.py

c147e5d

Daniel Hesslow commited on May 25, 2023

Upload tokenizer

bacaa8a

Daniel Hesslow commited on May 24, 2023

Update configuration_RW.py

19299bf

Daniel Hesslow commited on May 24, 2023

Update README.md

e0e1228

Daniel Hesslow commited on Apr 26, 2023

Create README.md

e251c35

Daniel Hesslow commited on Apr 26, 2023

Upload RWForCausalLM

06337a9

Daniel Hesslow commited on Apr 25, 2023

initial commit

7972440

DanielHesslow commited on Apr 25, 2023

Commit History

Adding `safetensors` variant of this model (#126) 8782b5c verified

enh: add link to new model b22819d verified

add chat template to tokenizer_config.json (#111) 0b9684d verified

Move to in-libary checkpoint (for real this time) (#88) cf4b3c4

Revert in-library PR (#63) eb410fb

Move to in-library checkpoint (#56) f8dac3f

Add widget examples c7f670a

Add recommendations for use 9f16e66

Update citation info 8281cc0

Add hf endpoint handler.py (#24) 7f5eb0f

Update metadata a1d11a0

Remove TII Falcon LLM license 22225c3

Update license information to Apache 2.0 43b757f

Update README.md (#14) a8b5565

license: tii-falcon-llm (#11) 6677bc9

Update config.json 9cbb161

Update modelling_RW.py 93b55b6

coreml-weights (#8) 983c105

Add PyTorch 2.0 notice b6efaea

Add evaluation disclaimer 6940d5f

Update README.md 5aad858

Update configuration_RW.py 69601d5

Update modelling_RW.py 8d630c5

Update README.md 54b3714

Create LICENSE.txt 7ea4d00

Update modelling_RW.py c147e5d

Upload tokenizer bacaa8a

Update configuration_RW.py 19299bf

Update README.md e0e1228

Create README.md e251c35

Upload RWForCausalLM 06337a9

initial commit 7972440

Adding `safetensors` variant of this model (#126)

8782b5c
verified

enh: add link to new model

b22819d
verified

add chat template to tokenizer_config.json (#111)

0b9684d
verified

Move to in-libary checkpoint (for real this time) (#88)

cf4b3c4

Revert in-library PR (#63)

eb410fb

Move to in-library checkpoint (#56)

f8dac3f

Add widget examples

c7f670a

Add recommendations for use

9f16e66

Update citation info

8281cc0

Add hf endpoint handler.py (#24)

7f5eb0f

Update metadata

a1d11a0

Remove TII Falcon LLM license

22225c3

Update license information to Apache 2.0

43b757f

Update README.md (#14)

a8b5565

license: tii-falcon-llm (#11)

6677bc9

Update config.json

9cbb161

Update modelling_RW.py

93b55b6

coreml-weights (#8)

983c105

Add PyTorch 2.0 notice

b6efaea

Add evaluation disclaimer

6940d5f

Update README.md

5aad858

Update configuration_RW.py

69601d5

Update modelling_RW.py

8d630c5

Update README.md

54b3714

Create LICENSE.txt

7ea4d00

Update modelling_RW.py

c147e5d

Upload tokenizer

bacaa8a

Update configuration_RW.py

19299bf

Update README.md

e0e1228

Create README.md

e251c35

Upload RWForCausalLM

06337a9

initial commit

7972440