Card: serve with --max-model-len 32768 in Run with vLLM
#7
by subin - opened
At the default 262,144 positions, vLLM's startup profiling pass for d3 does not fit on an 80 GB GPU, even with d3-FP8 (discussion #2). The serve command now passes --max-model-len 32768, which covers d3's requests, and the section says why.
Xunzhuo changed pull request status to merged