issues to deploy on 2x48GB hardware
vllm:latest keep running into issues. logs:
(EngineCore pid=684) INFO 07-27 14:28:25 [distributed/device_communicators/shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(APIServer pid=106) DEBUG 07-27 14:28:27 [v1/engine/utils.py:1262] Waiting for 1 local, 0 remote core engine proc(s) to start.
VLLM workers at 100% cpu, but it doesn't seem to finish whatever its doing (compiling?) even after 1h. Tried some ampere cards like A40, but seeing a similar issue with L40S.
vllm serve poolside/Laguna-S-2.1-INT4 --host 0.0.0.0 --port 8000 --gpu-memory-utilization 0.90 --max-num-batched-tokens 8192 --enable-prefix-caching --trust-remote-code
--max-model-len 4096
--tensor-parallel-size 2
--tool-call-parser poolside_v1
--reasoning-parser poolside_v1
--enable-auto-tool-choice
--default-chat-template-kwargs '{"enable_thinking": true}'
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
NCCL_P2P_DISABLE=1 <- this was necessary to get it past previous issues
got it to work with the config below, getting 60-70 tok/s with speculative decoding, pretty nice, will test more.
vllm/vllm-openai:v0.26.0
vllm serve poolside/Laguna-S-2.1-INT4 --host 0.0.0.0 --port 8000 --gpu-memory-utilization 0.94 --max-num-batched-tokens 4096 --enable-prefix-caching --trust-remote-code --max-num-seqs 1
--max-model-len 300000 # good enough on 2x A40
--tensor-parallel-size 2
--tool-call-parser poolside_v1
--reasoning-parser poolside_v1
--enable-auto-tool-choice
--default-chat-template-kwargs '{"enable_thinking": true}'
--safetensors-load-strategy=prefetch
--compilation-config '{"pass_config":{"fuse_norm_quant":false}}' # simple compilation, reduces performance
--override-generation-config '{"temperature":0.7,"top_p":0.95,"top_k":20}'
--disable-custom-all-reduce # this did the trick for me
--speculative-config '{"model":"poolside/Laguna-S-2.1-DFlash-INT4","num_speculative_tokens":15,"method":"dflash"}'
disk 5gb
volume 78gb
VLLM_API_KEY=sk-*********
HF_HOME=/workspace/.huggingface
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
HF_TOKEN="{{ RUNPOD_SECRET_HF_TOKEN }}"
HF_HUB_VERBOSITY=info # change to debug if needed
VLLM_LOGGING_LEVEL=info # change to debug if needed
TRANSFORMERS_VERBOSITY=info # change to debug if needed
XDG_CACHE_HOME=/workspace/.cache
VLLM_CACHE_ROOT=/workspace/.cache/vllm
TORCH_HOME=/workspace/.cache/torch
TRITON_CACHE_DIR=/workspace/.cache/triton
FLASHINFER_WORKSPACE_DIR=/workspace/.cache/flashinfer
NCCL_P2P_DISABLE=1 # fixed some crashes related to tensor size = 2
NCCL_PROTO=Simple # might have helped too
TORCH_LOGS=+inductor # very verbose use only for debug
TORCHINDUCTOR_CACHE_DIR=/workspace/.cache/torchinductor
PYTORCH_ALLOC_CONF="expandable_segments:True"