Hugging Face's logo

Qwen
/
Qwen-7B-Chat

Text Generation
Transformers
Safetensors
Chinese
English
qwen
custom_code
Model card Files
xet
 Community
58

BF16是不是依赖CUDA 11.7，我的机器是12.2

#7
by Weiguo - opened
Discussion
Weiguo

下面编译不过，现在只能用8位量化，4090应该是可以跑半精度的

model = AutoModelForCausalLM.from_pretrained("Qwen-7B-Chat", device_map="auto", trust_remote_code=True, use_bf16=True).eval()

有没有可能支持

model = AutoModelForCausalLM.from_pretrained("Qwen-7B-Chat", device_map="cuda:0", trust_remote_code=True).half().cuda()
itkingtao

16bit量化都不行
TypeError: QWenLMHeadModel.init() got an unexpected keyword argument 'use_bf16'

TypeError: QWenLMHeadModel.init() got an unexpected keyword argument 'use_fp16'
Weiguo

16bit量化都不行
TypeError: QWenLMHeadModel.init() got an unexpected keyword argument 'use_bf16'

TypeError: QWenLMHeadModel.init() got an unexpected keyword argument 'use_fp16'

要装某个依赖CUDA 11.7的东西
logicwong

@Weiguo 把use_bf16=True改成bf16=True就行，Readme已修改
Weiguo changed discussion status to closed

Sign up or log in to comment