bruhzair/ignore-base12

base12

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the DARE TIES merge method using /workspace/cache/models--bruhzair--ignore-merge-17/snapshots/bd0af76a6bc4d9ae4bab5fa6b50e6545e6f3fd4f as a base.

Models Merged

The following models were included in the merge:

• /workspace/cache/models--bruhzair--ignore-merge-19/snapshots/57e2fc7118091e0706844a2e88d4c911d94e1e52
• /workspace/cache/models--bruhzair--ignore-merge-10-nemo/snapshots/bc2bc1ac38a2d05d5d32a31dd24d109bcc37c64c

Configuration

The following YAML configuration was used to produce this model:

base_model: /workspace/cache/models--bruhzair--ignore-merge-17/snapshots/bd0af76a6bc4d9ae4bab5fa6b50e6545e6f3fd4f
dtype: float32
merge_method: dare_ties
modules:
  default:
    slices:
    - sources:
      - layer_range: [0, 120]
        model: /workspace/cache/models--bruhzair--ignore-merge-19/snapshots/57e2fc7118091e0706844a2e88d4c911d94e1e52
        parameters:
          density: 0.2
          weight: 0.5
      - layer_range: [0, 120]
        model: /workspace/cache/models--bruhzair--ignore-merge-10-nemo/snapshots/bc2bc1ac38a2d05d5d32a31dd24d109bcc37c64c
        parameters:
          density: 0.2
          weight: 0.5
      - layer_range: [0, 120]
        model: /workspace/cache/models--bruhzair--ignore-merge-17/snapshots/bd0af76a6bc4d9ae4bab5fa6b50e6545e6f3fd4f
out_dtype: bfloat16
parameters:
  normalize: 1.0
tokenizer:
  source: base