Hugging Face's logo

eworojoshua
/
vas-01

Text Generation
Transformers
Safetensors
qwen2
mergekit
Merge
conversational
text-generation-inference
Model card Files
xet
 Community
1
eworojoshua commited on
Commit
f8252fa
·
verified ·
1 Parent(s): 0eb818a

Upload folder using huggingface_hub

Browse files
Files changed (6) hide show
  1. README.md +9 -9
  2. mergekit_config.yml +7 -7
  3. model-00001-of-00004.safetensors +1 -1
  4. model-00002-of-00004.safetensors +1 -1
  5. model-00003-of-00004.safetensors +1 -1
  6. model-00004-of-00004.safetensors +1 -1
README.md CHANGED
@@ -1,7 +1,7 @@
1
  ---
2
  base_model:
3
- - Xiaojian9992024/Qwen2.5-THREADRIPPER-Small
4
  - jeffmeloy/Qwen2.5-7B-nerd-uncensored-v1.0
 
5
  library_name: transformers
6
  tags:
7
  - mergekit
@@ -20,8 +20,8 @@ This model was merged using the [SLERP](https://en.wikipedia.org/wiki/Slerp) mer
20
  ### Models Merged
21
 
22
  The following models were included in the merge:
23
- * [Xiaojian9992024/Qwen2.5-THREADRIPPER-Small](https://huggingface.co/Xiaojian9992024/Qwen2.5-THREADRIPPER-Small)
24
  * [jeffmeloy/Qwen2.5-7B-nerd-uncensored-v1.0](https://huggingface.co/jeffmeloy/Qwen2.5-7B-nerd-uncensored-v1.0)
 
25
 
26
  ### Configuration
27
 
@@ -29,22 +29,22 @@ The following YAML configuration was used to produce this model:
29
 
30
  ```yaml
31
  model_name: "vas-01-7b"
 
 
 
32
  slices:
33
  - sources:
34
  - model: Xiaojian9992024/Qwen2.5-THREADRIPPER-Small
35
  layer_range: [0, 28]
36
  - model: jeffmeloy/Qwen2.5-7B-nerd-uncensored-v1.0
37
  layer_range: [0, 28]
38
- merge_method: slerp
39
- base_model: Xiaojian9992024/Qwen2.5-THREADRIPPER-Small
40
  parameters:
41
  t:
42
  - filter: self_attn
43
- value: [0.5, 0.5, 0.4, 0.4, 0.3] # More balanced attention weights
44
  - filter: mlp
45
- value: [0.6, 0.6, 0.7, 0.7, 0.8] # Stronger preservation of Nerd's MLP
46
  - filter: input_layernorm|post_attention_layernorm
47
- value: 0.5 # Equal weight for normalization layers
48
- - value: 0.5 # Default for remaining components
49
- dtype: bfloat16
50
  ```
 
1
  ---
2
  base_model:
 
3
  - jeffmeloy/Qwen2.5-7B-nerd-uncensored-v1.0
4
+ - Xiaojian9992024/Qwen2.5-THREADRIPPER-Small
5
  library_name: transformers
6
  tags:
7
  - mergekit
 
20
  ### Models Merged
21
 
22
  The following models were included in the merge:
 
23
  * [jeffmeloy/Qwen2.5-7B-nerd-uncensored-v1.0](https://huggingface.co/jeffmeloy/Qwen2.5-7B-nerd-uncensored-v1.0)
24
+ * [Xiaojian9992024/Qwen2.5-THREADRIPPER-Small](https://huggingface.co/Xiaojian9992024/Qwen2.5-THREADRIPPER-Small)
25
 
26
  ### Configuration
27
 
 
29
 
30
  ```yaml
31
  model_name: "vas-01-7b"
32
+ base_model: Xiaojian9992024/Qwen2.5-THREADRIPPER-Small
33
+ merge_method: slerp
34
+ dtype: bfloat16
35
  slices:
36
  - sources:
37
  - model: Xiaojian9992024/Qwen2.5-THREADRIPPER-Small
38
  layer_range: [0, 28]
39
  - model: jeffmeloy/Qwen2.5-7B-nerd-uncensored-v1.0
40
  layer_range: [0, 28]
 
 
41
  parameters:
42
  t:
43
  - filter: self_attn
44
+ value: [0.0, 0.3, 0.5, 0.7, 1.0]
45
  - filter: mlp
46
+ value: [1.0, 0.7, 0.5, 0.3, 0.0]
47
  - filter: input_layernorm|post_attention_layernorm
48
+ value: 0.5
49
+ - value: 0.5
 
50
  ```
mergekit_config.yml CHANGED
@@ -1,19 +1,19 @@
1
  model_name: "vas-01-7b"
 
 
 
2
  slices:
3
  - sources:
4
  - model: Xiaojian9992024/Qwen2.5-THREADRIPPER-Small
5
  layer_range: [0, 28]
6
  - model: jeffmeloy/Qwen2.5-7B-nerd-uncensored-v1.0
7
  layer_range: [0, 28]
8
- merge_method: slerp
9
- base_model: Xiaojian9992024/Qwen2.5-THREADRIPPER-Small
10
  parameters:
11
  t:
12
  - filter: self_attn
13
- value: [0.5, 0.5, 0.4, 0.4, 0.3] # More balanced attention weights
14
  - filter: mlp
15
- value: [0.6, 0.6, 0.7, 0.7, 0.8] # Stronger preservation of Nerd's MLP
16
  - filter: input_layernorm|post_attention_layernorm
17
- value: 0.5 # Equal weight for normalization layers
18
- - value: 0.5 # Default for remaining components
19
- dtype: bfloat16
 
1
  model_name: "vas-01-7b"
2
+ base_model: Xiaojian9992024/Qwen2.5-THREADRIPPER-Small
3
+ merge_method: slerp
4
+ dtype: bfloat16
5
  slices:
6
  - sources:
7
  - model: Xiaojian9992024/Qwen2.5-THREADRIPPER-Small
8
  layer_range: [0, 28]
9
  - model: jeffmeloy/Qwen2.5-7B-nerd-uncensored-v1.0
10
  layer_range: [0, 28]
 
 
11
  parameters:
12
  t:
13
  - filter: self_attn
14
+ value: [0.0, 0.3, 0.5, 0.7, 1.0]
15
  - filter: mlp
16
+ value: [1.0, 0.7, 0.5, 0.3, 0.0]
17
  - filter: input_layernorm|post_attention_layernorm
18
+ value: 0.5
19
+ - value: 0.5
 
model-00001-of-00004.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:95340dfd0059ffcb15a38f7e7b9695c01ea5019da3e9a5c4aca6aaf7470316fb
3
  size 4976698776
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b2e9b53831d85a7f95d6687e41863cbc3c49007c28016a1856a3101e4a1abe01
3
  size 4976698776
model-00002-of-00004.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:1d1432b2eaae5a386a22bf62b91ac6372895f658787a06f5f20f74bcc2a0adc5
3
  size 4932751032
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fc4ea59c2afb81ff4936c121053dbca6aec37d6d838d831fc8292414342f6f27
3
  size 4932751032
model-00003-of-00004.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:9253eba3b52e42f9ae30b932534df3799b0b1505f758e175c3b9f0a3205f2b6f
3
  size 4991495808
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2e63c61767f1571145af318fd332ce38c58c3a433c208bcdb8b9c153bc550cbe
3
  size 4991495808
model-00004-of-00004.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:5a4811121f25397ef90999538020985f691e3945ef13f871132f1f69513abf2d
3
  size 330326240
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ca161d09be6665935d5b5f6c6cc55c74180902264efb8800deec008923695cdd
3
  size 330326240