Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
Paper • 2311.03099 • Published • 35
merged_dare_3models
This is a merge of pre-trained language models created using mergekit.
Merge Details
Merge Method
This model was merged using the DARE TIES merge method using downloaded_models/unconst_Teutonic-I-base3 as a base.
Models Merged
The following models were included in the merge:
- downloaded_models/best26_Teutonic-I-best2
- downloaded_models/ClarenceDan_Teutonic-I-Clarence-A5504
- downloaded_models/ChrisJackieChan_Teutonic-I-7
Configuration
The following YAML configuration was used to produce this model:
merge_method: dare_ties
base_model: downloaded_models/unconst_Teutonic-I-base3
dtype: bfloat16
parameters:
int_inv_lambda: 1.0
normalize: true
models:
- model: downloaded_models/best26_Teutonic-I-best2
parameters:
weight: 0.40
density: 0.90
- model: downloaded_models/ClarenceDan_Teutonic-I-Clarence-A5504
parameters:
weight: 0.33
density: 0.90
- model: downloaded_models/ChrisJackieChan_Teutonic-I-7
parameters:
weight: 0.27
density: 0.90
- Downloads last month
- 8