| | --- |
| | base_model: |
| | - unsloth/gemma-2-9b |
| | - crestf411/gemma2-9B-sunfall-v0.5.2 |
| | - anthracite-org/magnum-v4-9b |
| | - unsloth/gemma-2-9b-it |
| | - nbeerbower/Gemma2-Gutenberg-Doppel-9B |
| | library_name: transformers |
| | tags: |
| | - mergekit |
| | - merge |
| |
|
| | --- |
| | # Gemma2-Eclipse-9B |
| |
|
| | The merge script was taken from [MarinaraSpaghetti/NemoMix-Unleashed-12B](https://huggingface.co/MarinaraSpaghetti/NemoMix-Unleashed-12B). All I did was replace the models (and I couldn't find a replacement for one of the models so I just increased the weight on one) and it seems to work pretty well. |
| |
|
| | ChatML prompt format seems to work fine. Gemma 2 format seems to also work. |
| |
|
| | ChatML seems cause it to incorrectly spew this at the end |im_end|> so Gemma 2 format might work better. |
| | |
| | This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit). |
| | |
| | ## Merge Details |
| | ### Merge Method |
| | |
| | This model was merged using the della_linear merge method using [unsloth/gemma-2-9b](https://huggingface.co/unsloth/gemma-2-9b) as a base. |
| |
|
| | ### Models Merged |
| |
|
| | The following models were included in the merge: |
| | * [crestf411/gemma2-9B-sunfall-v0.5.2](https://huggingface.co/crestf411/gemma2-9B-sunfall-v0.5.2) |
| | * [anthracite-org/magnum-v4-9b](https://huggingface.co/anthracite-org/magnum-v4-9b) |
| | * [unsloth/gemma-2-9b-it](https://huggingface.co/unsloth/gemma-2-9b-it) |
| | * [nbeerbower/Gemma2-Gutenberg-Doppel-9B](https://huggingface.co/nbeerbower/Gemma2-Gutenberg-Doppel-9B) |
| |
|
| | ### Configuration |
| |
|
| | The following YAML configuration was used to produce this model: |
| |
|
| | ```yaml |
| | models: |
| | - model: unsloth/gemma-2-9b-it |
| | parameters: |
| | weight: 0.22 |
| | density: 0.5 |
| | - model: nbeerbower/Gemma2-Gutenberg-Doppel-9B |
| | parameters: |
| | weight: 0.2 |
| | density: 0.6 |
| | - model: crestf411/gemma2-9B-sunfall-v0.5.2 |
| | parameters: |
| | weight: 0.25 |
| | density: 0.7 |
| | - model: anthracite-org/magnum-v4-9b |
| | parameters: |
| | weight: 0.33 |
| | density: 0.8 |
| | merge_method: della_linear |
| | base_model: unsloth/gemma-2-9b |
| | parameters: |
| | epsilon: 0.05 |
| | lambda: 1 |
| | dtype: bfloat16 |
| | tokenizer_source: base |
| | |
| | ``` |
| |
|
