| --- |
| library_name: transformers |
| tags: |
| - model-merging |
| - mergeability |
| - research-artifact |
| - do-not-use-aligned-variant |
| --- |
| |
| # beetle-humanscale-deu-eng |
|
|
| Merged checkpoints for one model pair. **24 variant(s)** in this repository: `average__aligned`, `average__naive`, `average__transport`, `csp__aligned`, `csp__naive`, `csp__transport`, `dare__aligned`, `dare__naive`, `dare__transport`, `opcm__aligned`, `opcm__naive`, `opcm__transport`, `ortho_merge__aligned`, `ortho_merge__naive`, `ortho_merge__transport`, `svc__aligned`, `svc__naive`, `svc__transport`, `task_arithmetic__aligned`, `task_arithmetic__naive`, `task_arithmetic__transport`, `ties__aligned`, `ties__naive`, `ties__transport`. |
|
|
| - Operators present: `average`, `csp`, `dare`, `opcm`, `ortho_merge`, `svc`, `task_arithmetic`, `ties` |
| - Arms present: `aligned`, `naive`, `transport` |
|
|
| ## ⚠️ The `__aligned` variants in this repository are not usable |
| |
| They were merged from models damaged by a bug in our alignment code. The role table that classifies a |
| parameter by its relation to the residual stream did not recognise this architecture's tensor names, |
| so **90.8% of the residual-facing parameter mass was left unpermuted** while attention and both norm |
| tensors were permuted. The result is not an aligned model; it is a broken one. Under a map that must |
| be exactly function-preserving, the logits moved by a relative 0.32 (it must be ~1e-7). |
| |
| **Do not use, cite, or benchmark against the `__aligned` folders here.** The `__naive` and |
| `__transport` folders are unaffected and remain valid: the bug only ever touched the aligned arm. |
|
|
| The bug is fixed (`common/alignment.py` now classifies these names, `residual_coverage()` reports the |
| fraction classified, and `align_state_dict(..., strict=True)` raises rather than silently skipping |
| tensors when it cannot classify ≥99% of the residual-facing mass). Corrected aligned merges are being |
| regenerated from the re-run sweep. |
|
|
|
|
| ## What this is |
|
|
| A **model-merging artefact**, not a model anyone should deploy. Each folder is one merge of two parent |
| models, published so that the merging results in the accompanying work are inspectable and |
| reproducible rather than taken on trust. |
|
|
| Folders are named `<recipe>__<arm>`: |
|
|
| | part | meaning | |
| |---|---| |
| | `average`, `task_arithmetic`, `ties`, `qmd_guided` | the merge operator | |
| | `__naive` | parents merged as-is | |
| | `__aligned` | parents merged after a function-preserving alignment map | |
| | `__transport` | parents merged after vocabulary transport (cross-tokenizer pairs) | |
|
|
| ## Regime warning, which matters more than usual here |
|
|
| These are **sub-1B models pretrained from scratch**, not fine-tunes of a shared checkpoint. In this |
| regime merging is *destructive*: across 2,320 merges the mean change in NLL against the better parent |
| is about **-1.4 nats**, and a merge that beats its better parent is rare. That is a finding, not a |
| defect of these artefacts, and it is the opposite of what the model-soup literature reports for the |
| regime it studies. Both can be true; the regimes differ. Do not read a merge here as a capable model. |
|
|
| ## Provenance of the numbers |
|
|
| Merge quality for every artefact in this org is recorded in the accompanying repository's sweep |
| tables, with `delta_floor` (nats against the better parent) as the outcome. Alignment-dependent |
| columns from sweeps predating the fix are void; see the note above where it applies. |
|
|
