Error in Lora Config of traget modules not found
#33
by rawalba - opened
ValueError: Target modules ['query_key_valuedense', 'dense_h_to_4h', 'dense_4h_to_h', 'q_proj', 'v_proj'] not found in the base model. Please check the target modules and try again.
@gugarosa I've trained qlora with no problem using transformer and bitsandbytes. Edit* I used the Alpaca instructions to fine tune on a closed domain dataset and I am astonished at how well it performs. It did take me more epochs +50 for it to finally learn the content since it never seen it before though.
@rawalba use Target modules [ "Wqkv", "out_proj", "fc1", "fc2" ]
@Ermarrero thanks for the explanation! I wasn't aware that LoRA would work in the Phi models.
gugarosa changed discussion status to closed
