Translation
Safetensors
mistral
8-bit precision
compressed-tensors

config.json 可能有问题

#2
by dogvane - opened

config.json 文件里 "head_dim": null,
在 vllm 0.8.5 显卡为 2080ti 22g,启动时会出错,改为
"head_dim": 128,
后正常。

Sign up or log in to comment