Instructions to use tiiuae/falcon-180B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use tiiuae/falcon-180B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="tiiuae/falcon-180B")# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-180B") model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-180B") - Notebooks
- Google Colab
- Kaggle
- Local Apps
- vLLM
How to use tiiuae/falcon-180B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "tiiuae/falcon-180B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "tiiuae/falcon-180B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/tiiuae/falcon-180B
- SGLang
How to use tiiuae/falcon-180B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "tiiuae/falcon-180B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "tiiuae/falcon-180B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "tiiuae/falcon-180B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "tiiuae/falcon-180B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use tiiuae/falcon-180B with Docker Model Runner:
docker model run hf.co/tiiuae/falcon-180B
Zain
#37 opened 12 months ago
by
zkauser1984
Request: DOI
#35 opened over 1 year ago
by
Andrew285
Request: DOI
#33 opened over 1 year ago
by
balla1243
When it says 400GB of memory, does it mean RAM or GPU memory?
1
#32 opened almost 2 years ago
by
rahulsanjay18
The model is not presented in Open LLM Leaderboard
#31 opened about 2 years ago
by
rodion-m
๐ฉ Report
#30 opened over 2 years ago
by
RaulAf1
Update README.md
#29 opened over 2 years ago
by
luiscruzn1511
Create Oshen chat
1
#28 opened over 2 years ago
by
Oshenwithanage
Update README.md
#27 opened over 2 years ago
by
galakfyyar
I think the Falcon-180B web app is having a stroke
1
#26 opened over 2 years ago
by
andreped
I keep getting "gated" error even though I am logged in and registered.
๐ 1
4
#25 opened over 2 years ago
by
ehartford
Update README.md
#24 opened over 2 years ago
by
stereo-smarty
tiiuae/falcon-180B which tokenisation is used?
#23 opened over 2 years ago
by
Danny5050
Start an API for falcon-180B
๐ 10
6
#22 opened over 2 years ago
by
DrLuttapi
Fix model ID in code sample
๐ 1
1
#20 opened over 2 years ago
by
ei-grad
access token?
#18 opened over 2 years ago
by
Parasharadithya
This will be a great release, thank you!
#16 opened over 2 years ago
by
ybeetle
Base model "tiiuae/falcon-180B" was not found on hub!
#15 opened over 2 years ago
by
dfurman
Update README.md
2
#14 opened over 2 years ago
by
Abduali
Can I use Falcon 180B to generate a large training dataset of JSON prompts and answers, then train and distribute a new model with Apache 2.0?
1
#12 opened over 2 years ago
by
nib12345
CPU + GPU usage
๐ 1
1
#11 opened over 2 years ago
by
Cosmos911
Connected open interpreter
๐ 2
1
#10 opened over 2 years ago
by
Cosmos911
Want to use a Particular safetensor
#8 opened over 2 years ago
by
Kunjesh07
Thank you!! Artificial Intelligence for all instead of for the few is the only way forward. This is exciting for open source!
๐ 7
#7 opened over 2 years ago
by
tts2902
Falcon 40B license vs 180B - conflict ?
3
#4 opened over 2 years ago
by
cmp-nct
The email from the license seems unreachable
๐ 7
#3 opened over 2 years ago
by
justheuristic
What are the key differences between "Falcon-180B TII license" and Apache 2.0 License?
3
#2 opened over 2 years ago
by
wrh00