File size: 1,091 Bytes
6e668dc | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 | # `vllm-rs` CLI Quick Start
Start Qwen3 with one managed `vllm-rs serve` command from the repo root:
```bash
HF_HUB_OFFLINE=1 \
VLLM_CPU_KVCACHE_SPACE=2 \
VLLM_HOST_IP=127.0.0.1 \
VLLM_LOOPBACK_IP=127.0.0.1 \
cargo run --bin vllm-rs -- serve \
Qwen/Qwen3-0.6B \
--python ../vllm/.venv/bin/python \
--max-model-len 512 \
-- \
--dtype float16
```
This launches:
- a managed headless Python `vllm` engine
- the Rust OpenAI-compatible frontend on `127.0.0.1:8000`
All Python engine arguments must be placed after `--`. Arguments before `--` are parsed by the Rust
frontend itself.
You can then send OpenAI-style requests to the Rust frontend:
```bash
curl http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-0.6B",
"messages": [{"role": "user", "content": "What is the capital of France?"}],
"stream": true
}'
```
If you already started headless `vllm` yourself, use `frontend` instead:
```bash
cargo run --bin vllm-rs -- frontend \
--handshake-address tcp://127.0.0.1:62100 \
Qwen/Qwen3-0.6B
```
|