Why weight avg instead of lora merge?
Airoboros is a qlora, why not just merge the lora into chronos?
Partially the tools I am familair with and me not noticing the qlora. But in this case the merge ratio is 75% chronos. So it is not just applying the lora. Its applying varing percentages and settling on one I liked.
Partially the tools I am familair with and me not noticing the qlora. But in this case the merge ratio is 75% chronos. So it is not just applying the lora. Its applying varing percentages and settling on one I liked.
Hijacking this thread a bit, but speaking of merging models can you share your method or script how you accomplished this? I'm trying to do something similar and just can't get it working right for some reason.
Scripts are here : https://github.com/ontocord/MDEL/tree/main/Model%20Merge%20And%20Analysis%20Tools
For this model I used the Enhanced Merger not the more advanced ones that let you do individual layers. Script variable was edited to merge it with 0.75, airoboros was selected as the first model.
Scripts are here : https://github.com/ontocord/MDEL/tree/main/Model%20Merge%20And%20Analysis%20Tools
For this model I used the Enhanced Merger not the more advanced ones that let you do individual layers. Script variable was edited to merge it with 0.75, airoboros was selected as the first model.
Thank you, that is very helpful.