ClaudioItaly/Memo-2024

merge

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the TIES merge method using SillyTilly/mistralai_Mistral-Nemo-Base-2407 as a base.

Models Merged

The following models were included in the merge:

• SillyTilly/mistralai_Mistral-Nemo-Instruct-2407

Configuration

The following YAML configuration was used to produce this model:

base_model: SillyTilly/mistralai_Mistral-Nemo-Base-2407
dtype: bfloat16
merge_method: ties
parameters:
  normalize: true
  int8_mask: true
  lambda: 0.8
slices:
  - sources:
    - model: SillyTilly/mistralai_Mistral-Nemo-Base-2407
      layer_range: [0, -1]
      parameters:
        weight: 1.0
        density: 0.6
    - model: SillyTilly/mistralai_Mistral-Nemo-Instruct-2407
      layer_range: [0, -1]
      parameters:
        weight: 0.7
        density: 0.6
tie_alpha: 0.3
tie_norm: L2
merge_verbose: true