Instructions to use helloollel/vicuna-13b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use helloollel/vicuna-13b with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="helloollel/vicuna-13b")# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("helloollel/vicuna-13b") model = AutoModelForCausalLM.from_pretrained("helloollel/vicuna-13b") - Inference
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use helloollel/vicuna-13b with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "helloollel/vicuna-13b" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "helloollel/vicuna-13b", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/helloollel/vicuna-13b
- SGLang
How to use helloollel/vicuna-13b with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "helloollel/vicuna-13b" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "helloollel/vicuna-13b", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "helloollel/vicuna-13b" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "helloollel/vicuna-13b", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use helloollel/vicuna-13b with Docker Model Runner:
docker model run hf.co/helloollel/vicuna-13b
Commit ·
0977b85
1
Parent(s): f52107b
Upload vicuna.ipynb
Browse files- vicuna.ipynb +70 -0
vicuna.ipynb
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"cells": [
|
| 3 |
+
{
|
| 4 |
+
"cell_type": "code",
|
| 5 |
+
"execution_count": null,
|
| 6 |
+
"id": "9531c2e1",
|
| 7 |
+
"metadata": {},
|
| 8 |
+
"outputs": [],
|
| 9 |
+
"source": [
|
| 10 |
+
"!git clone https://github.com/lm-sys/FastChat.git\n",
|
| 11 |
+
"%cd FastChat\n",
|
| 12 |
+
"!python3 -m pip install -e .\n",
|
| 13 |
+
"%cd .."
|
| 14 |
+
]
|
| 15 |
+
},
|
| 16 |
+
{
|
| 17 |
+
"cell_type": "code",
|
| 18 |
+
"execution_count": null,
|
| 19 |
+
"id": "9d3f37d3",
|
| 20 |
+
"metadata": {},
|
| 21 |
+
"outputs": [],
|
| 22 |
+
"source": [
|
| 23 |
+
"!curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash\n",
|
| 24 |
+
"!sudo apt-get install git-lfs\n",
|
| 25 |
+
"!git lfs install"
|
| 26 |
+
]
|
| 27 |
+
},
|
| 28 |
+
{
|
| 29 |
+
"cell_type": "code",
|
| 30 |
+
"execution_count": null,
|
| 31 |
+
"id": "66c11bb4",
|
| 32 |
+
"metadata": {},
|
| 33 |
+
"outputs": [],
|
| 34 |
+
"source": [
|
| 35 |
+
"!git clone https://huggingface.co/helloollel/vicuna-13b"
|
| 36 |
+
]
|
| 37 |
+
},
|
| 38 |
+
{
|
| 39 |
+
"cell_type": "code",
|
| 40 |
+
"execution_count": null,
|
| 41 |
+
"id": "dbe18cb7",
|
| 42 |
+
"metadata": {},
|
| 43 |
+
"outputs": [],
|
| 44 |
+
"source": [
|
| 45 |
+
"!python3 -m fastchat.serve.cli --model-name ./vicuna-13b"
|
| 46 |
+
]
|
| 47 |
+
}
|
| 48 |
+
],
|
| 49 |
+
"metadata": {
|
| 50 |
+
"kernelspec": {
|
| 51 |
+
"display_name": "Python 3 (ipykernel)",
|
| 52 |
+
"language": "python",
|
| 53 |
+
"name": "python3"
|
| 54 |
+
},
|
| 55 |
+
"language_info": {
|
| 56 |
+
"codemirror_mode": {
|
| 57 |
+
"name": "ipython",
|
| 58 |
+
"version": 3
|
| 59 |
+
},
|
| 60 |
+
"file_extension": ".py",
|
| 61 |
+
"mimetype": "text/x-python",
|
| 62 |
+
"name": "python",
|
| 63 |
+
"nbconvert_exporter": "python",
|
| 64 |
+
"pygments_lexer": "ipython3",
|
| 65 |
+
"version": "3.10.8"
|
| 66 |
+
}
|
| 67 |
+
},
|
| 68 |
+
"nbformat": 4,
|
| 69 |
+
"nbformat_minor": 5
|
| 70 |
+
}
|