group_model_config_v4

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the TIES merge method using Qwen/Qwen3-1.7B as a base.

Models Merged

The following models were included in the merge:

• cs-552-2026-claude-bots/math_model
• cs-552-2026-claude-bots/multilingual_model
• cs-552-2026-claude-bots/general_knowledge_model

Configuration

The following YAML configuration was used to produce this model:

# TIES merge config v4
#
# v3 post-mortem:
#   normalize: false caused all scores to collapse to ~0.01.
#   Without normalization the merged task vector is ~3.7x normal magnitude
#   (sum of weights: 1.3+1.5+0.9), completely breaking model output.
#   normalize: true is mandatory in TIES.
#
# Root cause of v2 safety collapse (0.596 < all individual models):
#   With math_weight=1.3 and GK_weight=1.0, math dominates MLP layers.
#   Math safety=0.670 < GK safety=0.734, so math's MLP signal pulled
#   the merged safety score below GK's level. After TIES normalization,
#   GK's safety task vector was diluted by math's larger-weight MLP updates.
#
# Key insight: math_model and gk_model score IDENTICALLY on math (0.562 vs 0.560).
#   We can safely give GK the dominant MLP weight at zero cost to math accuracy.
#
# Changes from v2:
#   1. normalize: true (restored — mandatory)
#   2. math weight: 1.30 → 1.00
#   3. GK weight: 1.00 → 2.00 — GK now dominates MLP/attn layers, bringing
#      its safety signal (0.734) to the merged model without hurting math.
#   4. GK embed density: 0.30 → 0.20 — GK damages multilingual via embeddings;
#      reducing embed density compensates for the higher overall weight.
#   5. multilingual weight: 1.10 → 0.80 — further reduce non-embed influence.
#   6. safety_model: dropped (GK covers safety better: 0.734 vs 0.726).
#   7. prune_threshold: removed (may trim GK safety-critical params).

merge_method: ties
base_model: Qwen/Qwen3-1.7B
dtype: bfloat16
tokenizer_source: cs-552-2026-claude-bots/math_model

parameters:
  normalize: true
  int8_mask: true

models:
  # ========== Math specialist ==========
  # Weight reduced to 1.0 — math and GK are tied on math accuracy (0.562 vs 0.560),
  # so lowering math weight is free. High MLP density preserves math reasoning circuits.
  - model: cs-552-2026-claude-bots/math_model
    parameters:
      weight: 1.00
      density:
        - filter: embed_tokens
          value: 0.60
        - filter: self_attn
          value: 0.75
        - filter: mlp
          value: 0.85
        - value: 0.72

  # ========== General knowledge specialist ==========
  # Weight doubled to 2.0 — GK now dominates attention and MLP layers.
  # GK effective MLP contribution (2.0 × 0.68 = 1.36) > math (1.0 × 0.85 = 0.85).
  # Embed density reduced (0.30 → 0.20) to offset higher weight and protect
  # multilingual vocab encoding from GK's embedding drift.
  - model: cs-552-2026-claude-bots/general_knowledge_model
    parameters:
      weight: 2.00
      density:
        - filter: embed_tokens
          value: 0.20
        - filter: self_attn
          value: 0.65
        - filter: mlp
          value: 0.68
        - value: 0.55

  # ========== Multilingual specialist ==========
  # Weight reduced to 0.80. Embedding density stays high (language encoding).
  # MLP density very low — multilingual MLP changes hurt math (-0.256 delta).
  # Effective embed contribution (0.80 × 0.88 = 0.704) is the largest of any
  # model on embed layers, ensuring multilingual capability is preserved.
  - model: cs-552-2026-claude-bots/multilingual_model
    parameters:
      weight: 0.80
      density:
        - filter: embed_tokens
          value: 0.88
        - filter: self_attn
          value: 0.35
        - filter: mlp
          value: 0.18
        - value: 0.45