README.md

metadata

base_model:
  - Vortex5/Vermilion-Sage-12B
  - SicariusSicariiStuff/Impish_Nemo_12B
library_name: transformers
tags:
  - mergekit
  - merge
  - roleplay

Harmony Bird 12B

01 // Overview

Harmony-Bird-12B is a merged model intended for roleplay and storytelling.

Built with MergeKit.

02 // Merge Step 1

Vermilion-Sage-12B merges with Impish-Nemo-12B using the harmony_forge method (focus=8.0, blend=0.7).

Show YAML

merge_method: harmony_forge
models:
  - model: Vortex5/Vermilion-Sage-12B
  - model: SicariusSicariiStuff/Impish_Nemo_12B
parameters:
  focus: 8.0
  blend: 0.7
tokenizer:
  source: union
dtype: bfloat16

03 // Custom merge method

It is a custom adaptive merge method that blends models through consensus weighting across both **spatial** (parameter alignment) and **frequency** (spectral structure) domains. It is designed for stable, noise-resistant merges that preserve the shared strengths of multiple models while reducing conflicts and outlier effects. It supports standard and task-vector merging (when a `base_model` is provided). **How it works:** The method first centers all model weights (relative to either a `base_model` or their median) and normalizes their scales to ensure balance. It then analyzes correlations in parameter space (spatial features) and in the frequency domain (via FFT) to measure similarity and coherence between models. Each model receives a *goodness score* based on how well it aligns with others, adjusted by stability and outlier suppression terms.

These scores are converted into normalized merge weights using a softmax function, which smoothly scales scores so that higher values receive more weight while all weights sum to 1. The focus parameter controls how sharply these weights are distributed — low focus blends models evenly, while high focus concentrates more weight on the most consistent ones. The blend parameter mixes how much spatial versus frequency information influences the final weighting. The merged parameters are then computed as a weighted sum across models.

Key parameters:

• focus: Controls decisiveness of weighting (higher = more selective). Default: 1.0
• blend: Uses a 0–1 scale to control merge emphasis — 0 represents full reliance on spatial similarity, meaning model weights are compared directly in parameter space. 1 represents full reliance on frequency-domain similarity, where weights are compared by their spectral (FFT) patterns. Default: 0.5 blends both equally for balanced structural and behavioral alignment.

By weighting models through adaptive consensus across spatial and frequency domains, Harmony Forge emphasizes aligned, stable patterns—encouraging coherent, balanced merges that often inherit the strongest traits of each source model.

05 // Acknowledgments

• Team Mradermacher — Static & imatrix quantizations
• DeathGodlike — EXL3 quants
• Original model authors and contributors whose work made this model possible.