fine-tuning with autotrain-advanced
Hi I was trying to fine tune this model with autotrain advanced, but I face this exception:
out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = flash_attn_cuda.fwd(
RuntimeError: FlashAttention only support fp16 and bf16 data type
this is my command:
autotrain llm --train --project_name deacon-34b --model /run/media/knut/HD2/Yi-34B/ --data_path . --train_batch_size 1 --num_train_epochs 5 --trainer sft --use_int4 --use_peft --merge_adapter --target_modules "q_proj,k_proj,v_proj,o_proj"
I tried to reinstall a few libs, but could not resolve it. Is it already possible to fine tune your model with hf autotrain-advanced?
When loading the model, considering add torch_type='auto' in the model_class.from_pretrained API
I'm trying this with 4bit or 8bit peft fine tuning. I hacked around both in the models python files and in the main.py of autotrain-advanced, tried to add toch_dtype='auto' where I could see it might work. So far I had no luck. https://github.com/huggingface/autotrain-advanced/blob/main/src/autotrain/trainers/clm/__main__.py
I will close this.Please help make peft fine tuning with hf transformers work.
Please help make peft fine tuning with hf transformers work.
Will do!
And please watch our github repo for the latest progress. 🤗