suchirsalhan commited on
Commit
e2c08bd
·
verified ·
1 Parent(s): 6cccd7c

consolidate: drop the duplicated root copy

Browse files
README.md DELETED
@@ -1,43 +0,0 @@
1
- ---
2
- library_name: transformers
3
- tags: [model-merging, mergeability, training-free, quotient-merge-distance]
4
- ---
5
-
6
- # goldfish-en-spa_latn__average__naive
7
-
8
- Training-free merged checkpoint from the **Mergeability** sweep
9
- (`benchmark/emit_lm.py --real`), produced by weight-space merging of two independently
10
- trained parents. No gradient steps were taken.
11
-
12
- | field | value |
13
- |---|---|
14
- | pair_id | `goldfish-en-spa_latn` |
15
- | parent_a | `goldfish-models/eng_latn_1000mb` |
16
- | parent_b | `goldfish-models/spa_latn_1000mb` |
17
- | ceiling | `catherinearnett/B-GPT_en_es_simultaneous` |
18
- | operator | `average` |
19
- | alignment | `naive` |
20
- | align_method | `permutation` |
21
- | regime | `different_corpora` |
22
- | eval_langs | `eng_latn+spa_latn` |
23
- | nll_merge | `8.6857` |
24
- | nll_floor | `4.5501` |
25
- | param_coverage | `1.0` |
26
- | MS | `-6.081` |
27
-
28
- ## How it was made
29
-
30
- Parents were loaded, activations extracted on a shared calibration corpus, and the merge applied
31
- either **naive** (parents combined in their own coordinates) or **aligned** (parent B carried into
32
- parent A's residual-stream basis via `common.alignment.residual_basis_map` before merging —
33
- permutation for same-width pairs, orthogonal/rectangular for cross-width).
34
-
35
- `MS` is the recovery score from `common.eval.mergeability_score` (merged vs. floor vs. ceiling), the
36
- same normalisation used by Zhou et al., so it is comparable across rows of the sweep.
37
-
38
- ## Caveats
39
-
40
- Sub-1B merges are noisy; an aligned signal where the naive one is noise is the finding, not a bug.
41
- Rows without a joint ceiling are floor-relative and must not be read as absolute recovery.
42
-
43
- Generated automatically — see the [mergeability repo](https://github.com/suchirsalhan/mergeability).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
config.json DELETED
@@ -1,36 +0,0 @@
1
- {
2
- "activation_function": "gelu",
3
- "add_cross_attention": false,
4
- "architectures": [
5
- "GPT2LMHeadModel"
6
- ],
7
- "attn_pdrop": 0.1,
8
- "bos_token_id": 50000,
9
- "dtype": "float32",
10
- "embd_pdrop": 0.1,
11
- "eos_token_id": 50001,
12
- "initializer_range": 0.02,
13
- "layer_norm_epsilon": 1e-05,
14
- "model_type": "gpt2",
15
- "n_ctx": 512,
16
- "n_embd": 768,
17
- "n_head": 12,
18
- "n_inner": 3072,
19
- "n_layer": 12,
20
- "n_positions": 512,
21
- "pad_token_id": 50002,
22
- "prefix": "[CLS]",
23
- "reorder_and_upcast_attn": false,
24
- "resid_pdrop": 0.1,
25
- "scale_attn_by_inverse_layer_idx": false,
26
- "scale_attn_weights": true,
27
- "summary_activation": null,
28
- "summary_first_dropout": 0.1,
29
- "summary_proj_to_labels": true,
30
- "summary_type": "cls_index",
31
- "summary_use_proj": true,
32
- "tie_word_embeddings": true,
33
- "transformers_version": "5.14.1",
34
- "use_cache": true,
35
- "vocab_size": 51200
36
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
generation_config.json DELETED
@@ -1,8 +0,0 @@
1
- {
2
- "_from_model_config": true,
3
- "bos_token_id": 50000,
4
- "eos_token_id": 50001,
5
- "pad_token_id": 50002,
6
- "transformers_version": "5.14.1",
7
- "use_cache": true
8
- }
 
 
 
 
 
 
 
 
 
model.safetensors DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:d9d491eccd8fb47af1e57fdc593a225945c411e84ba7bc502d6e8f24bab08dc7
3
- size 499098240
 
 
 
 
tokenizer.json DELETED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json DELETED
@@ -1,12 +0,0 @@
1
- {
2
- "backend": "tokenizers",
3
- "bos_token": "[CLS]",
4
- "cls_token": "[CLS]",
5
- "eos_token": "[SEP]",
6
- "mask_token": "[MASK]",
7
- "model_max_length": 1000000000000000019884624838656,
8
- "pad_token": "<pad>",
9
- "sep_token": "[SEP]",
10
- "tokenizer_class": "TokenizersBackend",
11
- "unk_token": "<unk>"
12
- }