Update README.md
Browse files
README.md
CHANGED
|
@@ -14,24 +14,16 @@ library_name: transformers
|
|
| 14 |
|
| 15 |
Model created by analyzing and selecting the optimal layers from other Qwen2.5-7B models based on their dimensional utilization efficiency, measured by the Normalized Effective Rank (NER). Computed like:
|
| 16 |
|
| 17 |
-
|
| 18 |
-
- Input: Weight matrix A ∈ R^(m×n) # m = number of output features, n = number of input features
|
| 19 |
- Compute singular values σᵢ where σᵢ ≥ 0 # σᵢ represents the importance of each dimension
|
| 20 |
-
- Filter values above numerical threshold (>1e-12)
|
| 21 |
-
|
| 22 |
-
Distribution Normalization:
|
| 23 |
- Sum all singular values: S = Σσᵢ # S acts as normalization factor
|
| 24 |
- Create probability distribution: pᵢ = σᵢ/S # converts singular values to probabilities summing to 1
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
- Compute Shannon entropy: H = -Σ(pᵢ * log₂(pᵢ)) # measures information content of distribution
|
| 28 |
-
- Calculate maximum possible entropy: H_max = log₂(n) # n = number of singular values
|
| 29 |
-
where n is the number of singular values # maximum entropy occurs when all dimensions contribute equally
|
| 30 |
-
|
| 31 |
-
Normalization:
|
| 32 |
- Final NER score = H/H_max # normalizes score to [0,1] range
|
| 33 |
-
- Results in value between 0 and 1
|
| 34 |
-
|
| 35 |
|
| 36 |
## Creating Composite Model
|
| 37 |
|
|
@@ -63,6 +55,8 @@ fine_tuned_models: # uncomment the models you want to merge
|
|
| 63 |
|
| 64 |
#- "Qwen/Qwen2.5-7B-Instruct"
|
| 65 |
|
|
|
|
|
|
|
| 66 |
#- "FourOhFour/Vapor_v2_7B"
|
| 67 |
|
| 68 |
#- "Goekdeniz-Guelmez/Josiefied-Qwen2.5-7B-Instruct-abliterated-v2"
|
|
@@ -77,21 +71,41 @@ fine_tuned_models: # uncomment the models you want to merge
|
|
| 77 |
|
| 78 |
#- "Orion-zhen/Meissa-Qwen2.5-7B-Instruct"
|
| 79 |
|
|
|
|
|
|
|
| 80 |
#- "jeffmeloy/Qwen2.5-7B-nerd-uncensored-v1.0"
|
| 81 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 82 |
#- "rombodawg/Rombos-LLM-V2.5-Qwen-7b"
|
| 83 |
|
| 84 |
#- "Cran-May/T.E-8.1"
|
| 85 |
|
| 86 |
-
#- "thomas-yanxin/XinYuan-Qwen2.5-7B-0917"
|
| 87 |
-
|
| 88 |
#- "beomi/Qwen2.5-7B-Instruct-kowiki-qa"
|
| 89 |
|
| 90 |
#- "Orion-zhen/Qwen2.5-7B-Gutenberg-KTO"
|
| 91 |
|
| 92 |
-
#-
|
|
|
|
|
|
|
| 93 |
|
| 94 |
-
#-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 95 |
|
| 96 |
#- "Qwen/Qwen2.5-Coder-7B-Instruct"
|
| 97 |
|
|
@@ -101,11 +115,13 @@ fine_tuned_models: # uncomment the models you want to merge
|
|
| 101 |
|
| 102 |
#- "Qwen/Qwen2.5-Math-7B"
|
| 103 |
|
| 104 |
-
#- "
|
| 105 |
|
| 106 |
-
#- "
|
| 107 |
|
| 108 |
-
#- "
|
|
|
|
|
|
|
| 109 |
|
| 110 |
models_dir: "./input_models/"
|
| 111 |
|
|
|
|
| 14 |
|
| 15 |
Model created by analyzing and selecting the optimal layers from other Qwen2.5-7B models based on their dimensional utilization efficiency, measured by the Normalized Effective Rank (NER). Computed like:
|
| 16 |
|
| 17 |
+
- Input: Weight matrix for each model layer
|
|
|
|
| 18 |
- Compute singular values σᵢ where σᵢ ≥ 0 # σᵢ represents the importance of each dimension
|
| 19 |
+
- Filter values above numerical threshold (>1e-12)
|
|
|
|
|
|
|
| 20 |
- Sum all singular values: S = Σσᵢ # S acts as normalization factor
|
| 21 |
- Create probability distribution: pᵢ = σᵢ/S # converts singular values to probabilities summing to 1
|
| 22 |
+
- Compute Shannon entropy: H = -Σ(pᵢ * log₂(pᵢ)) # measures information content
|
| 23 |
+
- Calculate maximum possible entropy: H_max = log₂(n)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 24 |
- Final NER score = H/H_max # normalizes score to [0,1] range
|
| 25 |
+
- Results in value between 0 and 1
|
| 26 |
+
# 0 = single dimension dominance, 1 = uniform dimensional utilization
|
| 27 |
|
| 28 |
## Creating Composite Model
|
| 29 |
|
|
|
|
| 55 |
|
| 56 |
#- "Qwen/Qwen2.5-7B-Instruct"
|
| 57 |
|
| 58 |
+
#- "EVA-UNIT-01/EVA-Qwen2.5-7B-v0.1"
|
| 59 |
+
|
| 60 |
#- "FourOhFour/Vapor_v2_7B"
|
| 61 |
|
| 62 |
#- "Goekdeniz-Guelmez/Josiefied-Qwen2.5-7B-Instruct-abliterated-v2"
|
|
|
|
| 71 |
|
| 72 |
#- "Orion-zhen/Meissa-Qwen2.5-7B-Instruct"
|
| 73 |
|
| 74 |
+
#- "jeffmeloy/Qwen2.5-7B-nerd-uncensored-v0.9"
|
| 75 |
+
|
| 76 |
#- "jeffmeloy/Qwen2.5-7B-nerd-uncensored-v1.0"
|
| 77 |
|
| 78 |
+
#- "jeffmeloy/Qwen2.5-7B-nerd-uncensored-v1.1"
|
| 79 |
+
|
| 80 |
+
#- "jeffmeloy/Qwen2.5-7B-nerd-uncensored-v1.2"
|
| 81 |
+
|
| 82 |
+
#- "AmberYifan/Qwen2.5-7B-dpo-2k"
|
| 83 |
+
|
| 84 |
+
#- "sethuiyer/Qwen2.5-7B-Anvita"
|
| 85 |
+
|
| 86 |
#- "rombodawg/Rombos-LLM-V2.5-Qwen-7b"
|
| 87 |
|
| 88 |
#- "Cran-May/T.E-8.1"
|
| 89 |
|
|
|
|
|
|
|
| 90 |
#- "beomi/Qwen2.5-7B-Instruct-kowiki-qa"
|
| 91 |
|
| 92 |
#- "Orion-zhen/Qwen2.5-7B-Gutenberg-KTO"
|
| 93 |
|
| 94 |
+
#- "fblgit/cybertron-v4-qw7B-MGS"
|
| 95 |
+
|
| 96 |
+
#- "nguyentd/FinancialAdvice-Qwen2.5-7B"
|
| 97 |
|
| 98 |
+
#- "WhiteRabbitNeo/WhiteRabbitNeo-2.5-Qwen-2.5-Coder-7B"
|
| 99 |
+
|
| 100 |
+
#- "edgerunner-ai/EdgeRunner-Command-Nested"
|
| 101 |
+
|
| 102 |
+
#- "katanemo/Arch-Function-7B"
|
| 103 |
+
|
| 104 |
+
#- "DeepGlint-AI/llava-mlcd-qwen2.5-7b"
|
| 105 |
+
|
| 106 |
+
#- "mergekit-community/mergekit-slerp-aflqaqy"
|
| 107 |
+
|
| 108 |
+
#- "mergekit-community/mergekit-ties-inxwsfo"
|
| 109 |
|
| 110 |
#- "Qwen/Qwen2.5-Coder-7B-Instruct"
|
| 111 |
|
|
|
|
| 115 |
|
| 116 |
#- "Qwen/Qwen2.5-Math-7B"
|
| 117 |
|
| 118 |
+
#- "thomas-yanxin/XinYuan-Qwen2.5-7B-0917"
|
| 119 |
|
| 120 |
+
#- "jbjeong91/Qwen2.5_7B_IST_StoryGen_vanilla"
|
| 121 |
|
| 122 |
+
#- "AmberYifan/Qwen2.5-7B-dpo-2k-hhrlhf"
|
| 123 |
+
|
| 124 |
+
#- "jbjeong91/Qwen2.5_7B_IST_StoryGen_test2"
|
| 125 |
|
| 126 |
models_dir: "./input_models/"
|
| 127 |
|