luotingdan
commited on
Commit
·
6fa640f
1
Parent(s):
303e8de
add vllm image
Browse files
README.md
CHANGED
|
@@ -180,15 +180,22 @@ print(decoded)
|
|
| 180 |
|
| 181 |
For deployment, you can use vllm to create an OpenAI-compatible API endpoint.
|
| 182 |
|
| 183 |
-
1. Install vLLM nightly:
|
|
|
|
| 184 |
|
| 185 |
-
|
| 186 |
-
|
| 187 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 188 |
|
| 189 |
-
|
|
|
|
|
|
|
| 190 |
|
| 191 |
-
|
| 192 |
|
| 193 |
2. Launch the server:
|
| 194 |
|
|
|
|
| 180 |
|
| 181 |
For deployment, you can use vllm to create an OpenAI-compatible API endpoint.
|
| 182 |
|
| 183 |
+
1. Install vLLM nightly (choose one):
|
| 184 |
+
- **Python / pip**
|
| 185 |
|
| 186 |
+
```bash
|
| 187 |
+
pip install vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
|
| 188 |
+
```
|
| 189 |
+
|
| 190 |
+
Python ≥3.10 is required. Please ensure vLLM version >= 0.14.0rc2.dev143+gc0a350ca7.
|
| 191 |
+
|
| 192 |
+
- **Docker (nightly image)**
|
| 193 |
|
| 194 |
+
```bash
|
| 195 |
+
docker pull vllm/vllm-openai:nightly-963dc0b865a3b6011fde7e0d938f86245dccbfac
|
| 196 |
+
```
|
| 197 |
|
| 198 |
+
The tag above pins the nightly build we validated; update to the latest nightly tag if needed.
|
| 199 |
|
| 200 |
2. Launch the server:
|
| 201 |
|