Instructions to use tiiuae/falcon-180B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use tiiuae/falcon-180B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="tiiuae/falcon-180B")# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-180B") model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-180B") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use tiiuae/falcon-180B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "tiiuae/falcon-180B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "tiiuae/falcon-180B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/tiiuae/falcon-180B
- SGLang
How to use tiiuae/falcon-180B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "tiiuae/falcon-180B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "tiiuae/falcon-180B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "tiiuae/falcon-180B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "tiiuae/falcon-180B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use tiiuae/falcon-180B with Docker Model Runner:
docker model run hf.co/tiiuae/falcon-180B
Thank you!! Artificial Intelligence for all instead of for the few is the only way forward. This is exciting for open source!
I just wanted to share my appreciation for this release. In a time where regulatory capture and censorship within corporate models has reduced capabilities in the name of 'alignment', releases like this give me hope for a better future. The only way forward is for AI to be in the hands of everyone, as the alternative is AI in the hands of the few.
Falcon 180B is a major turning point for open source. This model is an icebreaker as the open source community can now tinker and innovate with a base model that is competitive with the best proprietary ones. This model will have an impact on personalized tutoring, specialized field specific tutoring, personalized assistants, personalized lawyers, and so much more without the censorship and privacy violations we are seeing from companies.
The net positive value this release will provide cannot be understated so thank you! You guys are amazing and I feel like I had to write this to show my gratitude.