Hugging Face's logo

casperhansen
/
Mistral-Small-3.1-24B-Base-2503-Text-Only

Safetensors
vllm
mistral
Model card Files
xet
 Community
casperhansen commited on
Commit
b3e3307
·
verified ·
1 Parent(s): 0c3d5ac

Add files using upload-large-folder tool

Browse files
Files changed (19) hide show
  1. .gitattributes +1 -0
  2. README.md +220 -0
  3. config.json +26 -0
  4. convert.py +430 -0
  5. generation_config.json +6 -0
  6. model-00001-of-00010.safetensors +3 -0
  7. model-00002-of-00010.safetensors +3 -0
  8. model-00003-of-00010.safetensors +3 -0
  9. model-00004-of-00010.safetensors +3 -0
  10. model-00005-of-00010.safetensors +3 -0
  11. model-00006-of-00010.safetensors +3 -0
  12. model-00007-of-00010.safetensors +3 -0
  13. model-00008-of-00010.safetensors +3 -0
  14. model-00009-of-00010.safetensors +3 -0
  15. model-00010-of-00010.safetensors +3 -0
  16. model.safetensors.index.json +370 -0
  17. special_tokens_map.json +0 -0
  18. tokenizer.json +3 -0
  19. tokenizer_config.json +0 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,220 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ - fr
5
+ - de
6
+ - es
7
+ - pt
8
+ - it
9
+ - ja
10
+ - ko
11
+ - ru
12
+ - zh
13
+ - ar
14
+ - fa
15
+ - id
16
+ - ms
17
+ - ne
18
+ - pl
19
+ - ro
20
+ - sr
21
+ - sv
22
+ - tr
23
+ - uk
24
+ - vi
25
+ - hi
26
+ - bn
27
+ license: apache-2.0
28
+ library_name: vllm
29
+ inference: false
30
+ ---
31
+
32
+ # Model Card for Mistral-Small-3.1-24B-Base-2503 (TEXT ONLY)
33
+
34
+ This is the text-only variant of [mistralai/Mistral-Small-3.1-24B-Base-2503](https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Base-2503).
35
+ This also serves as the base-model for [mistralai/Devstral-Small-2505](https://huggingface.co/mistralai/Devstral-Small-2505), which had no official base model released.
36
+
37
+ Features:
38
+ - Text-only, no multimodality.
39
+ - 128k context length.
40
+
41
+ How was a text-only model achieved? The vision encoder was removed and the model architecture was converted from mistral3 to mistral. The tokenizer was not modified.
42
+
43
+ ## Reproduced eval
44
+
45
+ Serve with vLLM:
46
+
47
+ ```
48
+ vllm serve casperhansen/Mistral-Small-3.1-24B-Base-2503-Text-Only
49
+ ```
50
+
51
+ The reproduced results can be seen below.
52
+
53
+ | Model | MMLU (0-shot) |
54
+ |------------------------------------|-----------------|
55
+ | Small 3.1 24B Base (Text Only) | 77.25% ± 0.0033 |
56
+ | Small 3.1 24B Base (Multimodal) | 77.34% ± 0.0033 |
57
+
58
+ ### Original Multimodal: Full MMLU (Reproduced)
59
+
60
+ ```
61
+ lm_eval --model local-completions \
62
+ --model_args "base_url=http://localhost:8000/v1/completions,model=mistralai/Mistral-Small-3.1-24B-Base-2503" \
63
+ --tasks mmlu \
64
+ --batch_size 128
65
+ ```
66
+
67
+ | Tasks |Version|Filter|n-shot|Metric| |Value | |Stderr|
68
+ |---------------------------------------|------:|------|-----:|------|---|-----:|---|-----:|
69
+ |mmlu | 2|none | |acc |↑ |0.7734|± |0.0033|
70
+ | - humanities | 2|none | |acc |↑ |0.6820|± |0.0062|
71
+ | - formal_logic | 1|none | 0|acc |↑ |0.5714|± |0.0443|
72
+ | - high_school_european_history | 1|none | 0|acc |↑ |0.8303|± |0.0293|
73
+ | - high_school_us_history | 1|none | 0|acc |↑ |0.9363|± |0.0171|
74
+ | - high_school_world_history | 1|none | 0|acc |↑ |0.9241|± |0.0172|
75
+ | - international_law | 1|none | 0|acc |↑ |0.9091|± |0.0262|
76
+ | - jurisprudence | 1|none | 0|acc |↑ |0.8148|± |0.0376|
77
+ | - logical_fallacies | 1|none | 0|acc |↑ |0.8589|± |0.0274|
78
+ | - moral_disputes | 1|none | 0|acc |↑ |0.8208|± |0.0206|
79
+ | - moral_scenarios | 1|none | 0|acc |↑ |0.3844|± |0.0163|
80
+ | - philosophy | 1|none | 0|acc |↑ |0.8296|± |0.0214|
81
+ | - prehistory | 1|none | 0|acc |↑ |0.8704|± |0.0187|
82
+ | - professional_law | 1|none | 0|acc |↑ |0.6095|± |0.0125|
83
+ | - world_religions | 1|none | 0|acc |↑ |0.8713|± |0.0257|
84
+ | - other | 2|none | |acc |↑ |0.8317|± |0.0064|
85
+ | - business_ethics | 1|none | 0|acc |↑ |0.8200|± |0.0386|
86
+ | - clinical_knowledge | 1|none | 0|acc |↑ |0.8679|± |0.0208|
87
+ | - college_medicine | 1|none | 0|acc |↑ |0.7803|± |0.0316|
88
+ | - global_facts | 1|none | 0|acc |↑ |0.6600|± |0.0476|
89
+ | - human_aging | 1|none | 0|acc |↑ |0.7982|± |0.0269|
90
+ | - management | 1|none | 0|acc |↑ |0.9029|± |0.0293|
91
+ | - marketing | 1|none | 0|acc |↑ |0.9359|± |0.0160|
92
+ | - medical_genetics | 1|none | 0|acc |↑ |0.8900|± |0.0314|
93
+ | - miscellaneous | 1|none | 0|acc |↑ |0.9183|± |0.0098|
94
+ | - nutrition | 1|none | 0|acc |↑ |0.8791|± |0.0187|
95
+ | - professional_accounting | 1|none | 0|acc |↑ |0.6277|± |0.0288|
96
+ | - professional_medicine | 1|none | 0|acc |↑ |0.8603|± |0.0211|
97
+ | - virology | 1|none | 0|acc |↑ |0.5602|± |0.0386|
98
+ | - social sciences | 2|none | |acc |↑ |0.8736|± |0.0059|
99
+ | - econometrics | 1|none | 0|acc |↑ |0.6491|± |0.0449|
100
+ | - high_school_geography | 1|none | 0|acc |↑ |0.8990|± |0.0215|
101
+ | - high_school_government_and_politics| 1|none | 0|acc |↑ |0.9637|± |0.0135|
102
+ | - high_school_macroeconomics | 1|none | 0|acc |↑ |0.8103|± |0.0199|
103
+ | - high_school_microeconomics | 1|none | 0|acc |↑ |0.9034|± |0.0192|
104
+ | - high_school_psychology | 1|none | 0|acc |↑ |0.9358|± |0.0105|
105
+ | - human_sexuality | 1|none | 0|acc |↑ |0.8855|± |0.0279|
106
+ | - professional_psychology | 1|none | 0|acc |↑ |0.8578|± |0.0141|
107
+ | - public_relations | 1|none | 0|acc |↑ |0.7909|± |0.0390|
108
+ | - security_studies | 1|none | 0|acc |↑ |0.8327|± |0.0239|
109
+ | - sociology | 1|none | 0|acc |↑ |0.9154|± |0.0197|
110
+ | - us_foreign_policy | 1|none | 0|acc |↑ |0.9300|± |0.0256|
111
+ | - stem | 2|none | |acc |↑ |0.7545|± |0.0073|
112
+ | - abstract_algebra | 1|none | 0|acc |↑ |0.4600|± |0.0501|
113
+ | - anatomy | 1|none | 0|acc |↑ |0.8148|± |0.0336|
114
+ | - astronomy | 1|none | 0|acc |↑ |0.9211|± |0.0219|
115
+ | - college_biology | 1|none | 0|acc |↑ |0.9444|± |0.0192|
116
+ | - college_chemistry | 1|none | 0|acc |↑ |0.5700|± |0.0498|
117
+ | - college_computer_science | 1|none | 0|acc |↑ |0.7100|± |0.0456|
118
+ | - college_mathematics | 1|none | 0|acc |↑ |0.6200|± |0.0488|
119
+ | - college_physics | 1|none | 0|acc |↑ |0.6569|± |0.0472|
120
+ | - computer_security | 1|none | 0|acc |↑ |0.8300|± |0.0378|
121
+ | - conceptual_physics | 1|none | 0|acc |↑ |0.8170|± |0.0253|
122
+ | - electrical_engineering | 1|none | 0|acc |↑ |0.7931|± |0.0338|
123
+ | - elementary_mathematics | 1|none | 0|acc |↑ |0.7910|± |0.0209|
124
+ | - high_school_biology | 1|none | 0|acc |↑ |0.9323|± |0.0143|
125
+ | - high_school_chemistry | 1|none | 0|acc |↑ |0.7586|± |0.0301|
126
+ | - high_school_computer_science | 1|none | 0|acc |↑ |0.8900|± |0.0314|
127
+ | - high_school_mathematics | 1|none | 0|acc |↑ |0.5185|± |0.0305|
128
+ | - high_school_physics | 1|none | 0|acc |↑ |0.6291|± |0.0394|
129
+ | - high_school_statistics | 1|none | 0|acc |↑ |0.7593|± |0.0292|
130
+ | - machine_learning | 1|none | 0|acc |↑ |0.6250|± |0.0460|
131
+
132
+ | Groups |Version|Filter|n-shot|Metric| |Value | |Stderr|
133
+ |------------------|------:|------|------|------|---|-----:|---|-----:|
134
+ |mmlu | 2|none | |acc |↑ |0.7734|± |0.0033|
135
+ | - humanities | 2|none | |acc |↑ |0.6820|± |0.0062|
136
+ | - other | 2|none | |acc |↑ |0.8317|± |0.0064|
137
+ | - social sciences| 2|none | |acc |↑ |0.8736|± |0.0059|
138
+ | - stem | 2|none | |acc |↑ |0.7545|± |0.0073|
139
+
140
+ ### Text Only: Full MMLU
141
+
142
+ ```
143
+ lm_eval --model local-completions \
144
+ --model_args "base_url=http://localhost:8000/v1/completions,model=casperhansen/Mistral-Small-3.1-24B-Base-2503-Text-Only" \
145
+ --tasks mmlu \
146
+ --batch_size 128
147
+ ```
148
+
149
+ | Tasks |Version|Filter|n-shot|Metric| |Value | |Stderr|
150
+ |---------------------------------------|------:|------|-----:|------|---|-----:|---|-----:|
151
+ |mmlu | 2|none | |acc |↑ |0.7725|± |0.0033|
152
+ | - humanities | 2|none | |acc |↑ |0.6793|± |0.0062|
153
+ | - formal_logic | 1|none | 0|acc |↑ |0.5397|± |0.0446|
154
+ | - high_school_european_history | 1|none | 0|acc |↑ |0.8364|± |0.0289|
155
+ | - high_school_us_history | 1|none | 0|acc |↑ |0.9363|± |0.0171|
156
+ | - high_school_world_history | 1|none | 0|acc |↑ |0.9198|± |0.0177|
157
+ | - international_law | 1|none | 0|acc |↑ |0.9008|± |0.0273|
158
+ | - jurisprudence | 1|none | 0|acc |↑ |0.8148|± |0.0376|
159
+ | - logical_fallacies | 1|none | 0|acc |↑ |0.8405|± |0.0288|
160
+ | - moral_disputes | 1|none | 0|acc |↑ |0.8237|± |0.0205|
161
+ | - moral_scenarios | 1|none | 0|acc |↑ |0.3765|± |0.0162|
162
+ | - philosophy | 1|none | 0|acc |↑ |0.8264|± |0.0215|
163
+ | - prehistory | 1|none | 0|acc |↑ |0.8704|± |0.0187|
164
+ | - professional_law | 1|none | 0|acc |↑ |0.6108|± |0.0125|
165
+ | - world_religions | 1|none | 0|acc |↑ |0.8713|± |0.0257|
166
+ | - other | 2|none | |acc |↑ |0.8339|± |0.0064|
167
+ | - business_ethics | 1|none | 0|acc |↑ |0.8300|± |0.0378|
168
+ | - clinical_knowledge | 1|none | 0|acc |↑ |0.8679|± |0.0208|
169
+ | - college_medicine | 1|none | 0|acc |↑ |0.7746|± |0.0319|
170
+ | - global_facts | 1|none | 0|acc |↑ |0.6800|± |0.0469|
171
+ | - human_aging | 1|none | 0|acc |↑ |0.8027|± |0.0267|
172
+ | - management | 1|none | 0|acc |↑ |0.9029|± |0.0293|
173
+ | - marketing | 1|none | 0|acc |↑ |0.9402|± |0.0155|
174
+ | - medical_genetics | 1|none | 0|acc |↑ |0.8900|± |0.0314|
175
+ | - miscellaneous | 1|none | 0|acc |↑ |0.9208|± |0.0097|
176
+ | - nutrition | 1|none | 0|acc |↑ |0.8791|± |0.0187|
177
+ | - professional_accounting | 1|none | 0|acc |↑ |0.6312|± |0.0288|
178
+ | - professional_medicine | 1|none | 0|acc |↑ |0.8603|± |0.0211|
179
+ | - virology | 1|none | 0|acc |↑ |0.5602|± |0.0386|
180
+ | - social sciences | 2|none | |acc |↑ |0.8739|± |0.0059|
181
+ | - econometrics | 1|none | 0|acc |↑ |0.6667|± |0.0443|
182
+ | - high_school_geography | 1|none | 0|acc |↑ |0.8939|± |0.0219|
183
+ | - high_school_government_and_politics| 1|none | 0|acc |↑ |0.9585|± |0.0144|
184
+ | - high_school_macroeconomics | 1|none | 0|acc |↑ |0.8103|± |0.0199|
185
+ | - high_school_microeconomics | 1|none | 0|acc |↑ |0.9076|± |0.0188|
186
+ | - high_school_psychology | 1|none | 0|acc |↑ |0.9358|± |0.0105|
187
+ | - human_sexuality | 1|none | 0|acc |↑ |0.8855|± |0.0279|
188
+ | - professional_psychology | 1|none | 0|acc |↑ |0.8578|± |0.0141|
189
+ | - public_relations | 1|none | 0|acc |↑ |0.7909|± |0.0390|
190
+ | - security_studies | 1|none | 0|acc |↑ |0.8327|± |0.0239|
191
+ | - sociology | 1|none | 0|acc |↑ |0.9104|± |0.0202|
192
+ | - us_foreign_policy | 1|none | 0|acc |↑ |0.9400|± |0.0239|
193
+ | - stem | 2|none | |acc |↑ |0.7520|± |0.0073|
194
+ | - abstract_algebra | 1|none | 0|acc |↑ |0.4500|± |0.0500|
195
+ | - anatomy | 1|none | 0|acc |↑ |0.8296|± |0.0325|
196
+ | - astronomy | 1|none | 0|acc |↑ |0.9211|± |0.0219|
197
+ | - college_biology | 1|none | 0|acc |↑ |0.9444|± |0.0192|
198
+ | - college_chemistry | 1|none | 0|acc |↑ |0.5600|± |0.0499|
199
+ | - college_computer_science | 1|none | 0|acc |↑ |0.7100|± |0.0456|
200
+ | - college_mathematics | 1|none | 0|acc |↑ |0.6200|± |0.0488|
201
+ | - college_physics | 1|none | 0|acc |↑ |0.6569|± |0.0472|
202
+ | - computer_security | 1|none | 0|acc |↑ |0.8300|± |0.0378|
203
+ | - conceptual_physics | 1|none | 0|acc |↑ |0.8213|± |0.0250|
204
+ | - electrical_engineering | 1|none | 0|acc |↑ |0.7862|± |0.0342|
205
+ | - elementary_mathematics | 1|none | 0|acc |↑ |0.7804|± |0.0213|
206
+ | - high_school_biology | 1|none | 0|acc |↑ |0.9290|± |0.0146|
207
+ | - high_school_chemistry | 1|none | 0|acc |↑ |0.7488|± |0.0305|
208
+ | - high_school_computer_science | 1|none | 0|acc |↑ |0.8900|± |0.0314|
209
+ | - high_school_mathematics | 1|none | 0|acc |↑ |0.5222|± |0.0305|
210
+ | - high_school_physics | 1|none | 0|acc |↑ |0.6225|± |0.0396|
211
+ | - high_school_statistics | 1|none | 0|acc |↑ |0.7500|± |0.0295|
212
+ | - machine_learning | 1|none | 0|acc |↑ |0.6339|± |0.0457|
213
+
214
+ | Groups |Version|Filter|n-shot|Metric| |Value | |Stderr|
215
+ |------------------|------:|------|------|------|---|-----:|---|-----:|
216
+ |mmlu | 2|none | |acc |↑ |0.7725|± |0.0033|
217
+ | - humanities | 2|none | |acc |↑ |0.6793|± |0.0062|
218
+ | - other | 2|none | |acc |↑ |0.8339|± |0.0064|
219
+ | - social sciences| 2|none | |acc |↑ |0.8739|± |0.0059|
220
+ | - stem | 2|none | |acc |↑ |0.7520|± |0.0073|
config.json ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "MistralForCausalLM"
4
+ ],
5
+ "attention_dropout": 0.0,
6
+ "bos_token_id": 1,
7
+ "eos_token_id": 2,
8
+ "head_dim": 128,
9
+ "hidden_act": "silu",
10
+ "hidden_size": 5120,
11
+ "initializer_range": 0.02,
12
+ "intermediate_size": 32768,
13
+ "max_position_embeddings": 131072,
14
+ "model_type": "mistral",
15
+ "num_attention_heads": 32,
16
+ "num_hidden_layers": 40,
17
+ "num_key_value_heads": 8,
18
+ "rms_norm_eps": 1e-05,
19
+ "rope_theta": 1000000000.0,
20
+ "sliding_window": null,
21
+ "tie_word_embeddings": false,
22
+ "torch_dtype": "bfloat16",
23
+ "transformers_version": "4.51.3",
24
+ "use_cache": true,
25
+ "vocab_size": 131072
26
+ }
convert.py ADDED
@@ -0,0 +1,430 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """
3
+ Mistral Model Transformer
4
+
5
+ This script transforms Mistral-Small-3.1-24B-Base-2503 into a text-only model by:
6
+ 1. Removing multimodality features
7
+ 2. Removing the vision encoder
8
+ 3. Changing the architecture from "mistral3" to "mistral"
9
+ 4. Ensuring weight mapping structure matches Devstral-Small-2505 exactly
10
+
11
+ Usage:
12
+ python convert.py --input-model mistralai/Mistral-Small-3.1-24B-Base-2503 --output-path ./mistral-small-text-only --reference-model mistralai/Devstral-Small-2505
13
+
14
+ Note:
15
+ This script requires significant disk space to download and process the full model.
16
+ """
17
+
18
+ import argparse
19
+ import json
20
+ import os
21
+ import shutil
22
+ from pathlib import Path
23
+ import logging
24
+
25
+ from huggingface_hub import snapshot_download, hf_hub_download
26
+ from safetensors.torch import load_file, save_file
27
+ from transformers import AutoConfig, AutoModelForCausalLM
28
+
29
+ logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
30
+ logger = logging.getLogger(__name__)
31
+ def parse_args():
32
+ parser = argparse.ArgumentParser(description="Transform Mistral model to text-only version")
33
+ parser.add_argument(
34
+ "--input-model",
35
+ type=str,
36
+ default="mistralai/Mistral-Small-3.1-24B-Base-2503",
37
+ help="Path or HF repo id of the input model"
38
+ )
39
+ parser.add_argument(
40
+ "--output-path",
41
+ type=str,
42
+ required=True,
43
+ help="Path to save the transformed model"
44
+ )
45
+ parser.add_argument(
46
+ "--cache-dir",
47
+ type=str,
48
+ default=None,
49
+ help="Cache directory for downloading models"
50
+ )
51
+ parser.add_argument(
52
+ "--reference-model",
53
+ type=str,
54
+ default="mistralai/Devstral-Small-2505",
55
+ help="Path or HF repo id of the reference model for weight mapping"
56
+ )
57
+ return parser.parse_args()
58
+
59
+ def transform_config(config_path, output_path, reference_config=None):
60
+ """
61
+ Transform the model config by:
62
+ 1. Changing model_type from "mistral3" to "mistral"
63
+ 2. Removing vision_config
64
+ 3. Removing multimodal parameters
65
+ 4. Updating architectures to match Devstral exactly
66
+ 5. Ensuring all parameters match Devstral's config exactly
67
+ """
68
+ logger.info(f"Transforming config at {config_path}")
69
+
70
+ with open(config_path, "r") as f:
71
+ config = json.load(f)
72
+
73
+ if reference_config:
74
+ logger.info("Using reference config as template")
75
+ new_config = reference_config.copy()
76
+
77
+ text_config = config.get("text_config", config)
78
+
79
+ for key, value in text_config.items():
80
+ if key not in new_config and key != "model_type":
81
+ new_config[key] = value
82
+ logger.info(f"Added parameter from original config: {key}")
83
+ else:
84
+ logger.info("No reference config available, using basic transformation")
85
+ new_config = config.copy()
86
+
87
+ # Change model_type from mistral3 to mistral
88
+ if new_config.get("model_type") == "mistral3":
89
+ new_config["model_type"] = "mistral"
90
+ logger.info("Changed model_type from 'mistral3' to 'mistral'")
91
+
92
+ # Update architectures to use MistralForCausalLM
93
+ if "architectures" in new_config:
94
+ new_config["architectures"] = ["MistralForCausalLM"]
95
+ logger.info("Changed architecture to 'MistralForCausalLM'")
96
+
97
+ # Remove vision_config
98
+ if "vision_config" in new_config:
99
+ del new_config["vision_config"]
100
+ logger.info("Removed vision_config")
101
+
102
+ # Remove multimodal-related parameters
103
+ multimodal_params = [
104
+ "image_token_index",
105
+ "multimodal_projector_bias",
106
+ "projector_hidden_act",
107
+ "spatial_merge_size",
108
+ "vision_tower_layer_list",
109
+ "vision_feature_layer"
110
+ ]
111
+
112
+ for param in multimodal_params:
113
+ if param in new_config:
114
+ del new_config[param]
115
+ logger.info(f"Removed multimodal parameter: {param}")
116
+
117
+ if "text_config" in new_config:
118
+ text_config = new_config.pop("text_config")
119
+ for key, value in text_config.items():
120
+ if key != "model_type": # Don't overwrite the model_type
121
+ new_config[key] = value
122
+ logger.info("Moved text_config parameters to top level")
123
+
124
+ if "bos_token_id" not in new_config:
125
+ new_config["bos_token_id"] = 1
126
+ logger.info("Added bos_token_id: 1")
127
+
128
+ if "eos_token_id" not in new_config:
129
+ new_config["eos_token_id"] = 2
130
+ logger.info("Added eos_token_id: 2")
131
+
132
+ if "tie_word_embeddings" not in new_config:
133
+ new_config["tie_word_embeddings"] = False
134
+ logger.info("Added tie_word_embeddings: false")
135
+
136
+ new_config["transformers_version"] = "4.51.3"
137
+ logger.info("Updated transformers_version to 4.51.3")
138
+
139
+ os_output_path = Path(output_path) / "config.json"
140
+ with open(os_output_path, "w") as f:
141
+ json.dump(new_config, f, indent=2)
142
+
143
+ logger.info(f"Saved transformed config to {os_output_path}")
144
+ return new_config
145
+
146
+ def is_vision_weight(weight_name):
147
+ """Check if a weight is related to vision functionality"""
148
+ vision_patterns = ["vision_tower", "multi_modal_projector"]
149
+ return any(pattern in weight_name for pattern in vision_patterns)
150
+
151
+ def transform_weights(model_path, output_path, safetensors_index_path, reference_weight_map=None):
152
+ """
153
+ Transform model weights by:
154
+ 1. Loading the weight map from safetensors index
155
+ 2. Filtering out vision-related weights
156
+ 3. Removing the "language_model." prefix from weight names
157
+ 4. Ensuring the exact same partitioning as Devstral
158
+ 5. Saving the filtered weights to the output path
159
+ """
160
+ logger.info(f"Transforming weights using index at {safetensors_index_path}")
161
+
162
+ with open(safetensors_index_path, "r") as f:
163
+ index_data = json.load(f)
164
+
165
+ original_weight_map = index_data.get("weight_map", {})
166
+
167
+ # Count vision and non-vision weights
168
+ vision_weights = [name for name in original_weight_map if is_vision_weight(name)]
169
+ non_vision_weights = [name for name in original_weight_map if not is_vision_weight(name)]
170
+
171
+ logger.info(f"Found {len(vision_weights)} vision-related weights to remove")
172
+ logger.info(f"Found {len(non_vision_weights)} non-vision weights to keep")
173
+
174
+ # Create a mapping from original weight names to Devstral-style weight names
175
+ weight_name_mapping = {}
176
+ for original_name in non_vision_weights:
177
+ if original_name.startswith("language_model."):
178
+ new_name = original_name[len("language_model."):]
179
+ weight_name_mapping[original_name] = new_name
180
+ else:
181
+ weight_name_mapping[original_name] = original_name
182
+
183
+ logger.info(f"Created mapping for {len(weight_name_mapping)} weight names")
184
+
185
+ new_weight_map = {}
186
+
187
+ if reference_weight_map and "weight_map" in reference_weight_map:
188
+ devstral_weight_map = reference_weight_map["weight_map"]
189
+ logger.info(f"Using Devstral reference weight map with {len(devstral_weight_map)} entries")
190
+
191
+ for original_name, new_name in weight_name_mapping.items():
192
+ if new_name in devstral_weight_map:
193
+ new_weight_map[new_name] = devstral_weight_map[new_name]
194
+ else:
195
+ logger.warning(f"Weight {new_name} not found in Devstral reference map")
196
+ else:
197
+ logger.warning("No Devstral reference map available, using original partitioning")
198
+ for original_name, new_name in weight_name_mapping.items():
199
+ new_weight_map[new_name] = original_weight_map[original_name]
200
+
201
+ # Group weights by their safetensor file for the actual transformation
202
+ file_to_weights = {}
203
+ for new_name, file_name in new_weight_map.items():
204
+ if file_name not in file_to_weights:
205
+ file_to_weights[file_name] = []
206
+
207
+ original_names = [orig for orig, new in weight_name_mapping.items() if new == new_name]
208
+ if original_names:
209
+ file_to_weights[file_name].append((original_names[0], new_name))
210
+
211
+ os.makedirs(Path(output_path), exist_ok=True)
212
+
213
+ # Process each safetensor file
214
+ for file_name, weight_pairs in file_to_weights.items():
215
+ logger.info(f"Processing {file_name} with {len(weight_pairs)} weights")
216
+
217
+ tensors_to_save = {}
218
+
219
+ for original_name, new_name in weight_pairs:
220
+ original_file = original_weight_map.get(original_name)
221
+ if not original_file:
222
+ logger.warning(f"Original file not found for weight {original_name}")
223
+ continue
224
+
225
+ input_file_path = Path(model_path) / original_file
226
+ if not input_file_path.exists():
227
+ logger.warning(f"File {input_file_path} does not exist, skipping")
228
+ continue
229
+
230
+ try:
231
+ original_tensors = load_file(input_file_path)
232
+ if original_name in original_tensors:
233
+ tensors_to_save[new_name] = original_tensors[original_name]
234
+ else:
235
+ logger.warning(f"Weight {original_name} not found in {original_file}")
236
+ except Exception as e:
237
+ logger.error(f"Error loading {original_file}: {e}")
238
+
239
+ if tensors_to_save:
240
+ output_file_path = Path(output_path) / file_name
241
+ try:
242
+ save_file(tensors_to_save, output_file_path)
243
+ logger.info(f"Saved {len(tensors_to_save)} weights to {file_name}")
244
+ except Exception as e:
245
+ logger.error(f"Error saving {file_name}: {e}")
246
+
247
+ # Save the new safetensors index
248
+ new_index = {
249
+ "metadata": {"total_size": reference_weight_map.get("metadata", {}).get("total_size", 0)}
250
+ if reference_weight_map else index_data.get("metadata", {}),
251
+ "weight_map": new_weight_map
252
+ }
253
+
254
+ output_index_path = Path(output_path) / "model.safetensors.index.json"
255
+ with open(output_index_path, "w") as f:
256
+ json.dump(new_index, f, indent=2)
257
+
258
+ logger.info(f"Saved transformed safetensors index to {output_index_path}")
259
+
260
+ def copy_additional_files(model_path, output_path):
261
+ """Copy additional model files like tokenizer, generation config, etc."""
262
+ additional_files = [
263
+ "tokenizer.json",
264
+ "tokenizer_config.json",
265
+ "special_tokens_map.json",
266
+ "generation_config.json"
267
+ ]
268
+
269
+ for filename in additional_files:
270
+ src_path = Path(model_path) / filename
271
+ if src_path.exists():
272
+ dst_path = Path(output_path) / filename
273
+ shutil.copy(src_path, dst_path)
274
+ logger.info(f"Copied {filename} to output directory")
275
+ else:
276
+ logger.warning(f"File {filename} not found in model directory")
277
+
278
+ def download_minimal_files(repo_id, output_dir, cache_dir=None):
279
+ """Download only the necessary files for transformation without the full model"""
280
+ logger.info(f"Downloading minimal files from {repo_id}")
281
+
282
+ # List of files to download
283
+ files_to_download = [
284
+ "config.json",
285
+ "model.safetensors.index.json",
286
+ "tokenizer_config.json",
287
+ "special_tokens_map.json",
288
+ "generation_config.json"
289
+ ]
290
+
291
+ downloaded_files = {}
292
+
293
+ for filename in files_to_download:
294
+ try:
295
+ file_path = hf_hub_download(
296
+ repo_id=repo_id,
297
+ filename=filename,
298
+ cache_dir=cache_dir,
299
+ local_files_only=False
300
+ )
301
+ downloaded_files[filename] = file_path
302
+ logger.info(f"Downloaded {filename} to {file_path}")
303
+ except Exception as e:
304
+ logger.warning(f"Failed to download {filename}: {e}")
305
+
306
+ return downloaded_files
307
+
308
+ def download_reference_weight_map(reference_model, cache_dir=None):
309
+ """Download reference model's weight map to use as a reference"""
310
+ logger.info(f"Downloading reference weight map from {reference_model}")
311
+
312
+ try:
313
+ file_path = hf_hub_download(
314
+ repo_id=reference_model,
315
+ filename="model.safetensors.index.json",
316
+ cache_dir=cache_dir,
317
+ local_files_only=False
318
+ )
319
+
320
+ with open(file_path, "r") as f:
321
+ reference_map = json.load(f)
322
+
323
+ logger.info(f"Successfully loaded reference weight map with {len(reference_map.get('weight_map', {}))} weights")
324
+ return reference_map
325
+ except Exception as e:
326
+ logger.error(f"Failed to download reference weight map: {e}")
327
+ return None
328
+
329
+ def download_reference_config(reference_model, cache_dir=None):
330
+ """Download reference model's config.json to use as a reference"""
331
+ logger.info(f"Downloading reference config from {reference_model}")
332
+
333
+ try:
334
+ file_path = hf_hub_download(
335
+ repo_id=reference_model,
336
+ filename="config.json",
337
+ cache_dir=cache_dir,
338
+ local_files_only=False
339
+ )
340
+
341
+ with open(file_path, "r") as f:
342
+ reference_config = json.load(f)
343
+
344
+ logger.info(f"Successfully loaded reference config")
345
+ return reference_config
346
+ except Exception as e:
347
+ logger.error(f"Failed to download reference config: {e}")
348
+ return None
349
+
350
+ def verify_model(output_path):
351
+ """Verify that the transformed model can be loaded without errors"""
352
+ logger.info(f"Verifying transformed model at {output_path}")
353
+ try:
354
+ config = AutoConfig.from_pretrained(output_path)
355
+ logger.info(f"Successfully loaded config with model_type={config.model_type}")
356
+
357
+ # Attempt to load just the model architecture (without weights)
358
+ # This verifies the configuration is valid
359
+ AutoModelForCausalLM.from_config(config)
360
+ logger.info("Successfully loaded model architecture from config")
361
+
362
+ return True
363
+ except Exception as e:
364
+ logger.error(f"Error verifying model: {e}")
365
+ return False
366
+
367
+ def main():
368
+ args = parse_args()
369
+
370
+ input_model = args.input_model
371
+ output_path = args.output_path
372
+ cache_dir = args.cache_dir
373
+ reference_model = args.reference_model
374
+
375
+ # Download reference weight map and config
376
+ reference_weight_map = download_reference_weight_map(reference_model, cache_dir)
377
+ if not reference_weight_map:
378
+ logger.warning("Could not download reference weight map. The weight partitioning may not match exactly.")
379
+
380
+ reference_config = download_reference_config(reference_model, cache_dir)
381
+ if not reference_config:
382
+ logger.warning("Could not download reference config. The config may not match exactly.")
383
+
384
+ # Create output directory
385
+ os.makedirs(output_path, exist_ok=True)
386
+
387
+ # Download the full model
388
+ if not os.path.exists(input_model) or not os.path.isdir(input_model):
389
+ logger.info(f"Downloading model from {input_model}")
390
+ try:
391
+ model_path = snapshot_download(
392
+ repo_id=input_model,
393
+ cache_dir=cache_dir,
394
+ local_files_only=False,
395
+ ignore_patterns=["*consolidated*"]
396
+ )
397
+ except Exception as e:
398
+ logger.error(f"Error downloading model: {e}")
399
+ return
400
+ else:
401
+ model_path = input_model
402
+
403
+ logger.info(f"Model path: {model_path}")
404
+
405
+ # Transform config
406
+ config_path = os.path.join(model_path, "config.json")
407
+ transform_config(config_path, output_path, reference_config)
408
+
409
+ # Transform weights
410
+ safetensors_index_path = os.path.join(model_path, "model.safetensors.index.json")
411
+ transform_weights(
412
+ model_path,
413
+ output_path,
414
+ safetensors_index_path,
415
+ reference_weight_map=reference_weight_map
416
+ )
417
+
418
+ # Copy additional files
419
+ copy_additional_files(model_path, output_path)
420
+
421
+ # Verify the transformed model
422
+ success = verify_model(output_path)
423
+
424
+ if success:
425
+ logger.info(f"Successfully transformed model to {output_path}")
426
+ else:
427
+ logger.error(f"Failed to transform model properly")
428
+
429
+ if __name__ == "__main__":
430
+ main()
generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "transformers_version": "4.50.0.dev0"
6
+ }
model-00001-of-00010.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d2a76fd51ca4d1842da1814eb6793722d583bda92679b92025924ec7a859cc70
3
+ size 4781571704
model-00002-of-00010.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:87df53abceb14a4f8aa758554c681e6184f2f5e3eabe2e0e74e2356c524b733c
3
+ size 4781592752
model-00003-of-00010.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e532e79e000bf988b96beaead472ba4184a1eb4c4d8b5e054ee4889337c42061
3
+ size 4781592768
model-00004-of-00010.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1193aef9a3d4b739cceb8381190a2ebf9a80f3122e67559c27873bd8ce432be4
3
+ size 4886471568
model-00005-of-00010.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c8dd57b2263c028892bbf0d1a932f05b127ea3bf3d0f652fdd145641c9ec6031
3
+ size 4781592792
model-00006-of-00010.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:690684d541a1acd8daaff26857cc5938e1da3431865ebcb894cdcb60e41f3ef5
3
+ size 4781592784
model-00007-of-00010.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:70b3262c332eb7735bf5316f577ef267178bbbf80485e9ded6a6278d7ee0dc1b
3
+ size 4886471568
model-00008-of-00010.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d716948f528122126056ad58dd14d1487d8f9fce29e9f6986bf5146fe568087e
3
+ size 4781592792
model-00009-of-00010.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:05a38f3aad6f204a1af9d31d69eddaeb4fb26257aef912040c0c4bb5387900e3
3
+ size 4781592784
model-00010-of-00010.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:51d899b348647d9d6762d0bf1b2205b6b4f1bf3590c4c4c1619b7acf682d17a8
3
+ size 3900777040
model.safetensors.index.json ADDED
@@ -0,0 +1,370 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "metadata": {
3
+ "total_size": 47144806400
4
+ },
5
+ "weight_map": {
6
+ "lm_head.weight": "model-00010-of-00010.safetensors",
7
+ "model.embed_tokens.weight": "model-00001-of-00010.safetensors",
8
+ "model.layers.0.input_layernorm.weight": "model-00001-of-00010.safetensors",
9
+ "model.layers.0.mlp.down_proj.weight": "model-00001-of-00010.safetensors",
10
+ "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00010.safetensors",
11
+ "model.layers.0.mlp.up_proj.weight": "model-00001-of-00010.safetensors",
12
+ "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00010.safetensors",
13
+ "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00010.safetensors",
14
+ "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00010.safetensors",
15
+ "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00010.safetensors",
16
+ "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00010.safetensors",
17
+ "model.layers.1.input_layernorm.weight": "model-00001-of-00010.safetensors",
18
+ "model.layers.1.mlp.down_proj.weight": "model-00001-of-00010.safetensors",
19
+ "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00010.safetensors",
20
+ "model.layers.1.mlp.up_proj.weight": "model-00001-of-00010.safetensors",
21
+ "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00010.safetensors",
22
+ "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00010.safetensors",
23
+ "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00010.safetensors",
24
+ "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00010.safetensors",
25
+ "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00010.safetensors",
26
+ "model.layers.10.input_layernorm.weight": "model-00003-of-00010.safetensors",
27
+ "model.layers.10.mlp.down_proj.weight": "model-00003-of-00010.safetensors",
28
+ "model.layers.10.mlp.gate_proj.weight": "model-00003-of-00010.safetensors",
29
+ "model.layers.10.mlp.up_proj.weight": "model-00003-of-00010.safetensors",
30
+ "model.layers.10.post_attention_layernorm.weight": "model-00003-of-00010.safetensors",
31
+ "model.layers.10.self_attn.k_proj.weight": "model-00003-of-00010.safetensors",
32
+ "model.layers.10.self_attn.o_proj.weight": "model-00003-of-00010.safetensors",
33
+ "model.layers.10.self_attn.q_proj.weight": "model-00003-of-00010.safetensors",
34
+ "model.layers.10.self_attn.v_proj.weight": "model-00003-of-00010.safetensors",
35
+ "model.layers.11.input_layernorm.weight": "model-00004-of-00010.safetensors",
36
+ "model.layers.11.mlp.down_proj.weight": "model-00004-of-00010.safetensors",
37
+ "model.layers.11.mlp.gate_proj.weight": "model-00003-of-00010.safetensors",
38
+ "model.layers.11.mlp.up_proj.weight": "model-00003-of-00010.safetensors",
39
+ "model.layers.11.post_attention_layernorm.weight": "model-00004-of-00010.safetensors",
40
+ "model.layers.11.self_attn.k_proj.weight": "model-00003-of-00010.safetensors",
41
+ "model.layers.11.self_attn.o_proj.weight": "model-00003-of-00010.safetensors",
42
+ "model.layers.11.self_attn.q_proj.weight": "model-00003-of-00010.safetensors",
43
+ "model.layers.11.self_attn.v_proj.weight": "model-00003-of-00010.safetensors",
44
+ "model.layers.12.input_layernorm.weight": "model-00004-of-00010.safetensors",
45
+ "model.layers.12.mlp.down_proj.weight": "model-00004-of-00010.safetensors",
46
+ "model.layers.12.mlp.gate_proj.weight": "model-00004-of-00010.safetensors",
47
+ "model.layers.12.mlp.up_proj.weight": "model-00004-of-00010.safetensors",
48
+ "model.layers.12.post_attention_layernorm.weight": "model-00004-of-00010.safetensors",
49
+ "model.layers.12.self_attn.k_proj.weight": "model-00004-of-00010.safetensors",
50
+ "model.layers.12.self_attn.o_proj.weight": "model-00004-of-00010.safetensors",
51
+ "model.layers.12.self_attn.q_proj.weight": "model-00004-of-00010.safetensors",
52
+ "model.layers.12.self_attn.v_proj.weight": "model-00004-of-00010.safetensors",
53
+ "model.layers.13.input_layernorm.weight": "model-00004-of-00010.safetensors",
54
+ "model.layers.13.mlp.down_proj.weight": "model-00004-of-00010.safetensors",
55
+ "model.layers.13.mlp.gate_proj.weight": "model-00004-of-00010.safetensors",
56
+ "model.layers.13.mlp.up_proj.weight": "model-00004-of-00010.safetensors",
57
+ "model.layers.13.post_attention_layernorm.weight": "model-00004-of-00010.safetensors",
58
+ "model.layers.13.self_attn.k_proj.weight": "model-00004-of-00010.safetensors",
59
+ "model.layers.13.self_attn.o_proj.weight": "model-00004-of-00010.safetensors",
60
+ "model.layers.13.self_attn.q_proj.weight": "model-00004-of-00010.safetensors",
61
+ "model.layers.13.self_attn.v_proj.weight": "model-00004-of-00010.safetensors",
62
+ "model.layers.14.input_layernorm.weight": "model-00004-of-00010.safetensors",
63
+ "model.layers.14.mlp.down_proj.weight": "model-00004-of-00010.safetensors",
64
+ "model.layers.14.mlp.gate_proj.weight": "model-00004-of-00010.safetensors",
65
+ "model.layers.14.mlp.up_proj.weight": "model-00004-of-00010.safetensors",
66
+ "model.layers.14.post_attention_layernorm.weight": "model-00004-of-00010.safetensors",
67
+ "model.layers.14.self_attn.k_proj.weight": "model-00004-of-00010.safetensors",
68
+ "model.layers.14.self_attn.o_proj.weight": "model-00004-of-00010.safetensors",
69
+ "model.layers.14.self_attn.q_proj.weight": "model-00004-of-00010.safetensors",
70
+ "model.layers.14.self_attn.v_proj.weight": "model-00004-of-00010.safetensors",
71
+ "model.layers.15.input_layernorm.weight": "model-00004-of-00010.safetensors",
72
+ "model.layers.15.mlp.down_proj.weight": "model-00004-of-00010.safetensors",
73
+ "model.layers.15.mlp.gate_proj.weight": "model-00004-of-00010.safetensors",
74
+ "model.layers.15.mlp.up_proj.weight": "model-00004-of-00010.safetensors",
75
+ "model.layers.15.post_attention_layernorm.weight": "model-00004-of-00010.safetensors",
76
+ "model.layers.15.self_attn.k_proj.weight": "model-00004-of-00010.safetensors",
77
+ "model.layers.15.self_attn.o_proj.weight": "model-00004-of-00010.safetensors",
78
+ "model.layers.15.self_attn.q_proj.weight": "model-00004-of-00010.safetensors",
79
+ "model.layers.15.self_attn.v_proj.weight": "model-00004-of-00010.safetensors",
80
+ "model.layers.16.input_layernorm.weight": "model-00005-of-00010.safetensors",
81
+ "model.layers.16.mlp.down_proj.weight": "model-00005-of-00010.safetensors",
82
+ "model.layers.16.mlp.gate_proj.weight": "model-00005-of-00010.safetensors",
83
+ "model.layers.16.mlp.up_proj.weight": "model-00005-of-00010.safetensors",
84
+ "model.layers.16.post_attention_layernorm.weight": "model-00005-of-00010.safetensors",
85
+ "model.layers.16.self_attn.k_proj.weight": "model-00004-of-00010.safetensors",
86
+ "model.layers.16.self_attn.o_proj.weight": "model-00004-of-00010.safetensors",
87
+ "model.layers.16.self_attn.q_proj.weight": "model-00004-of-00010.safetensors",
88
+ "model.layers.16.self_attn.v_proj.weight": "model-00004-of-00010.safetensors",
89
+ "model.layers.17.input_layernorm.weight": "model-00005-of-00010.safetensors",
90
+ "model.layers.17.mlp.down_proj.weight": "model-00005-of-00010.safetensors",
91
+ "model.layers.17.mlp.gate_proj.weight": "model-00005-of-00010.safetensors",
92
+ "model.layers.17.mlp.up_proj.weight": "model-00005-of-00010.safetensors",
93
+ "model.layers.17.post_attention_layernorm.weight": "model-00005-of-00010.safetensors",
94
+ "model.layers.17.self_attn.k_proj.weight": "model-00005-of-00010.safetensors",
95
+ "model.layers.17.self_attn.o_proj.weight": "model-00005-of-00010.safetensors",
96
+ "model.layers.17.self_attn.q_proj.weight": "model-00005-of-00010.safetensors",
97
+ "model.layers.17.self_attn.v_proj.weight": "model-00005-of-00010.safetensors",
98
+ "model.layers.18.input_layernorm.weight": "model-00005-of-00010.safetensors",
99
+ "model.layers.18.mlp.down_proj.weight": "model-00005-of-00010.safetensors",
100
+ "model.layers.18.mlp.gate_proj.weight": "model-00005-of-00010.safetensors",
101
+ "model.layers.18.mlp.up_proj.weight": "model-00005-of-00010.safetensors",
102
+ "model.layers.18.post_attention_layernorm.weight": "model-00005-of-00010.safetensors",
103
+ "model.layers.18.self_attn.k_proj.weight": "model-00005-of-00010.safetensors",
104
+ "model.layers.18.self_attn.o_proj.weight": "model-00005-of-00010.safetensors",
105
+ "model.layers.18.self_attn.q_proj.weight": "model-00005-of-00010.safetensors",
106
+ "model.layers.18.self_attn.v_proj.weight": "model-00005-of-00010.safetensors",
107
+ "model.layers.19.input_layernorm.weight": "model-00005-of-00010.safetensors",
108
+ "model.layers.19.mlp.down_proj.weight": "model-00005-of-00010.safetensors",
109
+ "model.layers.19.mlp.gate_proj.weight": "model-00005-of-00010.safetensors",
110
+ "model.layers.19.mlp.up_proj.weight": "model-00005-of-00010.safetensors",
111
+ "model.layers.19.post_attention_layernorm.weight": "model-00005-of-00010.safetensors",
112
+ "model.layers.19.self_attn.k_proj.weight": "model-00005-of-00010.safetensors",
113
+ "model.layers.19.self_attn.o_proj.weight": "model-00005-of-00010.safetensors",
114
+ "model.layers.19.self_attn.q_proj.weight": "model-00005-of-00010.safetensors",
115
+ "model.layers.19.self_attn.v_proj.weight": "model-00005-of-00010.safetensors",
116
+ "model.layers.2.input_layernorm.weight": "model-00001-of-00010.safetensors",
117
+ "model.layers.2.mlp.down_proj.weight": "model-00001-of-00010.safetensors",
118
+ "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00010.safetensors",
119
+ "model.layers.2.mlp.up_proj.weight": "model-00001-of-00010.safetensors",
120
+ "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00010.safetensors",
121
+ "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00010.safetensors",
122
+ "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00010.safetensors",
123
+ "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00010.safetensors",
124
+ "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00010.safetensors",
125
+ "model.layers.20.input_layernorm.weight": "model-00006-of-00010.safetensors",
126
+ "model.layers.20.mlp.down_proj.weight": "model-00006-of-00010.safetensors",
127
+ "model.layers.20.mlp.gate_proj.weight": "model-00005-of-00010.safetensors",
128
+ "model.layers.20.mlp.up_proj.weight": "model-00006-of-00010.safetensors",
129
+ "model.layers.20.post_attention_layernorm.weight": "model-00006-of-00010.safetensors",
130
+ "model.layers.20.self_attn.k_proj.weight": "model-00005-of-00010.safetensors",
131
+ "model.layers.20.self_attn.o_proj.weight": "model-00005-of-00010.safetensors",
132
+ "model.layers.20.self_attn.q_proj.weight": "model-00005-of-00010.safetensors",
133
+ "model.layers.20.self_attn.v_proj.weight": "model-00005-of-00010.safetensors",
134
+ "model.layers.21.input_layernorm.weight": "model-00006-of-00010.safetensors",
135
+ "model.layers.21.mlp.down_proj.weight": "model-00006-of-00010.safetensors",
136
+ "model.layers.21.mlp.gate_proj.weight": "model-00006-of-00010.safetensors",
137
+ "model.layers.21.mlp.up_proj.weight": "model-00006-of-00010.safetensors",
138
+ "model.layers.21.post_attention_layernorm.weight": "model-00006-of-00010.safetensors",
139
+ "model.layers.21.self_attn.k_proj.weight": "model-00006-of-00010.safetensors",
140
+ "model.layers.21.self_attn.o_proj.weight": "model-00006-of-00010.safetensors",
141
+ "model.layers.21.self_attn.q_proj.weight": "model-00006-of-00010.safetensors",
142
+ "model.layers.21.self_attn.v_proj.weight": "model-00006-of-00010.safetensors",
143
+ "model.layers.22.input_layernorm.weight": "model-00006-of-00010.safetensors",
144
+ "model.layers.22.mlp.down_proj.weight": "model-00006-of-00010.safetensors",
145
+ "model.layers.22.mlp.gate_proj.weight": "model-00006-of-00010.safetensors",
146
+ "model.layers.22.mlp.up_proj.weight": "model-00006-of-00010.safetensors",
147
+ "model.layers.22.post_attention_layernorm.weight": "model-00006-of-00010.safetensors",
148
+ "model.layers.22.self_attn.k_proj.weight": "model-00006-of-00010.safetensors",
149
+ "model.layers.22.self_attn.o_proj.weight": "model-00006-of-00010.safetensors",
150
+ "model.layers.22.self_attn.q_proj.weight": "model-00006-of-00010.safetensors",
151
+ "model.layers.22.self_attn.v_proj.weight": "model-00006-of-00010.safetensors",
152
+ "model.layers.23.input_layernorm.weight": "model-00006-of-00010.safetensors",
153
+ "model.layers.23.mlp.down_proj.weight": "model-00006-of-00010.safetensors",
154
+ "model.layers.23.mlp.gate_proj.weight": "model-00006-of-00010.safetensors",
155
+ "model.layers.23.mlp.up_proj.weight": "model-00006-of-00010.safetensors",
156
+ "model.layers.23.post_attention_layernorm.weight": "model-00006-of-00010.safetensors",
157
+ "model.layers.23.self_attn.k_proj.weight": "model-00006-of-00010.safetensors",
158
+ "model.layers.23.self_attn.o_proj.weight": "model-00006-of-00010.safetensors",
159
+ "model.layers.23.self_attn.q_proj.weight": "model-00006-of-00010.safetensors",
160
+ "model.layers.23.self_attn.v_proj.weight": "model-00006-of-00010.safetensors",
161
+ "model.layers.24.input_layernorm.weight": "model-00007-of-00010.safetensors",
162
+ "model.layers.24.mlp.down_proj.weight": "model-00007-of-00010.safetensors",
163
+ "model.layers.24.mlp.gate_proj.weight": "model-00006-of-00010.safetensors",
164
+ "model.layers.24.mlp.up_proj.weight": "model-00006-of-00010.safetensors",
165
+ "model.layers.24.post_attention_layernorm.weight": "model-00007-of-00010.safetensors",
166
+ "model.layers.24.self_attn.k_proj.weight": "model-00006-of-00010.safetensors",
167
+ "model.layers.24.self_attn.o_proj.weight": "model-00006-of-00010.safetensors",
168
+ "model.layers.24.self_attn.q_proj.weight": "model-00006-of-00010.safetensors",
169
+ "model.layers.24.self_attn.v_proj.weight": "model-00006-of-00010.safetensors",
170
+ "model.layers.25.input_layernorm.weight": "model-00007-of-00010.safetensors",
171
+ "model.layers.25.mlp.down_proj.weight": "model-00007-of-00010.safetensors",
172
+ "model.layers.25.mlp.gate_proj.weight": "model-00007-of-00010.safetensors",
173
+ "model.layers.25.mlp.up_proj.weight": "model-00007-of-00010.safetensors",
174
+ "model.layers.25.post_attention_layernorm.weight": "model-00007-of-00010.safetensors",
175
+ "model.layers.25.self_attn.k_proj.weight": "model-00007-of-00010.safetensors",
176
+ "model.layers.25.self_attn.o_proj.weight": "model-00007-of-00010.safetensors",
177
+ "model.layers.25.self_attn.q_proj.weight": "model-00007-of-00010.safetensors",
178
+ "model.layers.25.self_attn.v_proj.weight": "model-00007-of-00010.safetensors",
179
+ "model.layers.26.input_layernorm.weight": "model-00007-of-00010.safetensors",
180
+ "model.layers.26.mlp.down_proj.weight": "model-00007-of-00010.safetensors",
181
+ "model.layers.26.mlp.gate_proj.weight": "model-00007-of-00010.safetensors",
182
+ "model.layers.26.mlp.up_proj.weight": "model-00007-of-00010.safetensors",
183
+ "model.layers.26.post_attention_layernorm.weight": "model-00007-of-00010.safetensors",
184
+ "model.layers.26.self_attn.k_proj.weight": "model-00007-of-00010.safetensors",
185
+ "model.layers.26.self_attn.o_proj.weight": "model-00007-of-00010.safetensors",
186
+ "model.layers.26.self_attn.q_proj.weight": "model-00007-of-00010.safetensors",
187
+ "model.layers.26.self_attn.v_proj.weight": "model-00007-of-00010.safetensors",
188
+ "model.layers.27.input_layernorm.weight": "model-00007-of-00010.safetensors",
189
+ "model.layers.27.mlp.down_proj.weight": "model-00007-of-00010.safetensors",
190
+ "model.layers.27.mlp.gate_proj.weight": "model-00007-of-00010.safetensors",
191
+ "model.layers.27.mlp.up_proj.weight": "model-00007-of-00010.safetensors",
192
+ "model.layers.27.post_attention_layernorm.weight": "model-00007-of-00010.safetensors",
193
+ "model.layers.27.self_attn.k_proj.weight": "model-00007-of-00010.safetensors",
194
+ "model.layers.27.self_attn.o_proj.weight": "model-00007-of-00010.safetensors",
195
+ "model.layers.27.self_attn.q_proj.weight": "model-00007-of-00010.safetensors",
196
+ "model.layers.27.self_attn.v_proj.weight": "model-00007-of-00010.safetensors",
197
+ "model.layers.28.input_layernorm.weight": "model-00007-of-00010.safetensors",
198
+ "model.layers.28.mlp.down_proj.weight": "model-00007-of-00010.safetensors",
199
+ "model.layers.28.mlp.gate_proj.weight": "model-00007-of-00010.safetensors",
200
+ "model.layers.28.mlp.up_proj.weight": "model-00007-of-00010.safetensors",
201
+ "model.layers.28.post_attention_layernorm.weight": "model-00007-of-00010.safetensors",
202
+ "model.layers.28.self_attn.k_proj.weight": "model-00007-of-00010.safetensors",
203
+ "model.layers.28.self_attn.o_proj.weight": "model-00007-of-00010.safetensors",
204
+ "model.layers.28.self_attn.q_proj.weight": "model-00007-of-00010.safetensors",
205
+ "model.layers.28.self_attn.v_proj.weight": "model-00007-of-00010.safetensors",
206
+ "model.layers.29.input_layernorm.weight": "model-00008-of-00010.safetensors",
207
+ "model.layers.29.mlp.down_proj.weight": "model-00008-of-00010.safetensors",
208
+ "model.layers.29.mlp.gate_proj.weight": "model-00008-of-00010.safetensors",
209
+ "model.layers.29.mlp.up_proj.weight": "model-00008-of-00010.safetensors",
210
+ "model.layers.29.post_attention_layernorm.weight": "model-00008-of-00010.safetensors",
211
+ "model.layers.29.self_attn.k_proj.weight": "model-00007-of-00010.safetensors",
212
+ "model.layers.29.self_attn.o_proj.weight": "model-00007-of-00010.safetensors",
213
+ "model.layers.29.self_attn.q_proj.weight": "model-00007-of-00010.safetensors",
214
+ "model.layers.29.self_attn.v_proj.weight": "model-00007-of-00010.safetensors",
215
+ "model.layers.3.input_layernorm.weight": "model-00002-of-00010.safetensors",
216
+ "model.layers.3.mlp.down_proj.weight": "model-00002-of-00010.safetensors",
217
+ "model.layers.3.mlp.gate_proj.weight": "model-00002-of-00010.safetensors",
218
+ "model.layers.3.mlp.up_proj.weight": "model-00002-of-00010.safetensors",
219
+ "model.layers.3.post_attention_layernorm.weight": "model-00002-of-00010.safetensors",
220
+ "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00010.safetensors",
221
+ "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00010.safetensors",
222
+ "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00010.safetensors",
223
+ "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00010.safetensors",
224
+ "model.layers.30.input_layernorm.weight": "model-00008-of-00010.safetensors",
225
+ "model.layers.30.mlp.down_proj.weight": "model-00008-of-00010.safetensors",
226
+ "model.layers.30.mlp.gate_proj.weight": "model-00008-of-00010.safetensors",
227
+ "model.layers.30.mlp.up_proj.weight": "model-00008-of-00010.safetensors",
228
+ "model.layers.30.post_attention_layernorm.weight": "model-00008-of-00010.safetensors",
229
+ "model.layers.30.self_attn.k_proj.weight": "model-00008-of-00010.safetensors",
230
+ "model.layers.30.self_attn.o_proj.weight": "model-00008-of-00010.safetensors",
231
+ "model.layers.30.self_attn.q_proj.weight": "model-00008-of-00010.safetensors",
232
+ "model.layers.30.self_attn.v_proj.weight": "model-00008-of-00010.safetensors",
233
+ "model.layers.31.input_layernorm.weight": "model-00008-of-00010.safetensors",
234
+ "model.layers.31.mlp.down_proj.weight": "model-00008-of-00010.safetensors",
235
+ "model.layers.31.mlp.gate_proj.weight": "model-00008-of-00010.safetensors",
236
+ "model.layers.31.mlp.up_proj.weight": "model-00008-of-00010.safetensors",
237
+ "model.layers.31.post_attention_layernorm.weight": "model-00008-of-00010.safetensors",
238
+ "model.layers.31.self_attn.k_proj.weight": "model-00008-of-00010.safetensors",
239
+ "model.layers.31.self_attn.o_proj.weight": "model-00008-of-00010.safetensors",
240
+ "model.layers.31.self_attn.q_proj.weight": "model-00008-of-00010.safetensors",
241
+ "model.layers.31.self_attn.v_proj.weight": "model-00008-of-00010.safetensors",
242
+ "model.layers.32.input_layernorm.weight": "model-00008-of-00010.safetensors",
243
+ "model.layers.32.mlp.down_proj.weight": "model-00008-of-00010.safetensors",
244
+ "model.layers.32.mlp.gate_proj.weight": "model-00008-of-00010.safetensors",
245
+ "model.layers.32.mlp.up_proj.weight": "model-00008-of-00010.safetensors",
246
+ "model.layers.32.post_attention_layernorm.weight": "model-00008-of-00010.safetensors",
247
+ "model.layers.32.self_attn.k_proj.weight": "model-00008-of-00010.safetensors",
248
+ "model.layers.32.self_attn.o_proj.weight": "model-00008-of-00010.safetensors",
249
+ "model.layers.32.self_attn.q_proj.weight": "model-00008-of-00010.safetensors",
250
+ "model.layers.32.self_attn.v_proj.weight": "model-00008-of-00010.safetensors",
251
+ "model.layers.33.input_layernorm.weight": "model-00009-of-00010.safetensors",
252
+ "model.layers.33.mlp.down_proj.weight": "model-00009-of-00010.safetensors",
253
+ "model.layers.33.mlp.gate_proj.weight": "model-00008-of-00010.safetensors",
254
+ "model.layers.33.mlp.up_proj.weight": "model-00009-of-00010.safetensors",
255
+ "model.layers.33.post_attention_layernorm.weight": "model-00009-of-00010.safetensors",
256
+ "model.layers.33.self_attn.k_proj.weight": "model-00008-of-00010.safetensors",
257
+ "model.layers.33.self_attn.o_proj.weight": "model-00008-of-00010.safetensors",
258
+ "model.layers.33.self_attn.q_proj.weight": "model-00008-of-00010.safetensors",
259
+ "model.layers.33.self_attn.v_proj.weight": "model-00008-of-00010.safetensors",
260
+ "model.layers.34.input_layernorm.weight": "model-00009-of-00010.safetensors",
261
+ "model.layers.34.mlp.down_proj.weight": "model-00009-of-00010.safetensors",
262
+ "model.layers.34.mlp.gate_proj.weight": "model-00009-of-00010.safetensors",
263
+ "model.layers.34.mlp.up_proj.weight": "model-00009-of-00010.safetensors",
264
+ "model.layers.34.post_attention_layernorm.weight": "model-00009-of-00010.safetensors",
265
+ "model.layers.34.self_attn.k_proj.weight": "model-00009-of-00010.safetensors",
266
+ "model.layers.34.self_attn.o_proj.weight": "model-00009-of-00010.safetensors",
267
+ "model.layers.34.self_attn.q_proj.weight": "model-00009-of-00010.safetensors",
268
+ "model.layers.34.self_attn.v_proj.weight": "model-00009-of-00010.safetensors",
269
+ "model.layers.35.input_layernorm.weight": "model-00009-of-00010.safetensors",
270
+ "model.layers.35.mlp.down_proj.weight": "model-00009-of-00010.safetensors",
271
+ "model.layers.35.mlp.gate_proj.weight": "model-00009-of-00010.safetensors",
272
+ "model.layers.35.mlp.up_proj.weight": "model-00009-of-00010.safetensors",
273
+ "model.layers.35.post_attention_layernorm.weight": "model-00009-of-00010.safetensors",
274
+ "model.layers.35.self_attn.k_proj.weight": "model-00009-of-00010.safetensors",
275
+ "model.layers.35.self_attn.o_proj.weight": "model-00009-of-00010.safetensors",
276
+ "model.layers.35.self_attn.q_proj.weight": "model-00009-of-00010.safetensors",
277
+ "model.layers.35.self_attn.v_proj.weight": "model-00009-of-00010.safetensors",
278
+ "model.layers.36.input_layernorm.weight": "model-00009-of-00010.safetensors",
279
+ "model.layers.36.mlp.down_proj.weight": "model-00009-of-00010.safetensors",
280
+ "model.layers.36.mlp.gate_proj.weight": "model-00009-of-00010.safetensors",
281
+ "model.layers.36.mlp.up_proj.weight": "model-00009-of-00010.safetensors",
282
+ "model.layers.36.post_attention_layernorm.weight": "model-00009-of-00010.safetensors",
283
+ "model.layers.36.self_attn.k_proj.weight": "model-00009-of-00010.safetensors",
284
+ "model.layers.36.self_attn.o_proj.weight": "model-00009-of-00010.safetensors",
285
+ "model.layers.36.self_attn.q_proj.weight": "model-00009-of-00010.safetensors",
286
+ "model.layers.36.self_attn.v_proj.weight": "model-00009-of-00010.safetensors",
287
+ "model.layers.37.input_layernorm.weight": "model-00010-of-00010.safetensors",
288
+ "model.layers.37.mlp.down_proj.weight": "model-00010-of-00010.safetensors",
289
+ "model.layers.37.mlp.gate_proj.weight": "model-00009-of-00010.safetensors",
290
+ "model.layers.37.mlp.up_proj.weight": "model-00009-of-00010.safetensors",
291
+ "model.layers.37.post_attention_layernorm.weight": "model-00010-of-00010.safetensors",
292
+ "model.layers.37.self_attn.k_proj.weight": "model-00009-of-00010.safetensors",
293
+ "model.layers.37.self_attn.o_proj.weight": "model-00009-of-00010.safetensors",
294
+ "model.layers.37.self_attn.q_proj.weight": "model-00009-of-00010.safetensors",
295
+ "model.layers.37.self_attn.v_proj.weight": "model-00009-of-00010.safetensors",
296
+ "model.layers.38.input_layernorm.weight": "model-00010-of-00010.safetensors",
297
+ "model.layers.38.mlp.down_proj.weight": "model-00010-of-00010.safetensors",
298
+ "model.layers.38.mlp.gate_proj.weight": "model-00010-of-00010.safetensors",
299
+ "model.layers.38.mlp.up_proj.weight": "model-00010-of-00010.safetensors",
300
+ "model.layers.38.post_attention_layernorm.weight": "model-00010-of-00010.safetensors",
301
+ "model.layers.38.self_attn.k_proj.weight": "model-00010-of-00010.safetensors",
302
+ "model.layers.38.self_attn.o_proj.weight": "model-00010-of-00010.safetensors",
303
+ "model.layers.38.self_attn.q_proj.weight": "model-00010-of-00010.safetensors",
304
+ "model.layers.38.self_attn.v_proj.weight": "model-00010-of-00010.safetensors",
305
+ "model.layers.39.input_layernorm.weight": "model-00010-of-00010.safetensors",
306
+ "model.layers.39.mlp.down_proj.weight": "model-00010-of-00010.safetensors",
307
+ "model.layers.39.mlp.gate_proj.weight": "model-00010-of-00010.safetensors",
308
+ "model.layers.39.mlp.up_proj.weight": "model-00010-of-00010.safetensors",
309
+ "model.layers.39.post_attention_layernorm.weight": "model-00010-of-00010.safetensors",
310
+ "model.layers.39.self_attn.k_proj.weight": "model-00010-of-00010.safetensors",
311
+ "model.layers.39.self_attn.o_proj.weight": "model-00010-of-00010.safetensors",
312
+ "model.layers.39.self_attn.q_proj.weight": "model-00010-of-00010.safetensors",
313
+ "model.layers.39.self_attn.v_proj.weight": "model-00010-of-00010.safetensors",
314
+ "model.layers.4.input_layernorm.weight": "model-00002-of-00010.safetensors",
315
+ "model.layers.4.mlp.down_proj.weight": "model-00002-of-00010.safetensors",
316
+ "model.layers.4.mlp.gate_proj.weight": "model-00002-of-00010.safetensors",
317
+ "model.layers.4.mlp.up_proj.weight": "model-00002-of-00010.safetensors",
318
+ "model.layers.4.post_attention_layernorm.weight": "model-00002-of-00010.safetensors",
319
+ "model.layers.4.self_attn.k_proj.weight": "model-00002-of-00010.safetensors",
320
+ "model.layers.4.self_attn.o_proj.weight": "model-00002-of-00010.safetensors",
321
+ "model.layers.4.self_attn.q_proj.weight": "model-00002-of-00010.safetensors",
322
+ "model.layers.4.self_attn.v_proj.weight": "model-00002-of-00010.safetensors",
323
+ "model.layers.5.input_layernorm.weight": "model-00002-of-00010.safetensors",
324
+ "model.layers.5.mlp.down_proj.weight": "model-00002-of-00010.safetensors",
325
+ "model.layers.5.mlp.gate_proj.weight": "model-00002-of-00010.safetensors",
326
+ "model.layers.5.mlp.up_proj.weight": "model-00002-of-00010.safetensors",
327
+ "model.layers.5.post_attention_layernorm.weight": "model-00002-of-00010.safetensors",
328
+ "model.layers.5.self_attn.k_proj.weight": "model-00002-of-00010.safetensors",
329
+ "model.layers.5.self_attn.o_proj.weight": "model-00002-of-00010.safetensors",
330
+ "model.layers.5.self_attn.q_proj.weight": "model-00002-of-00010.safetensors",
331
+ "model.layers.5.self_attn.v_proj.weight": "model-00002-of-00010.safetensors",
332
+ "model.layers.6.input_layernorm.weight": "model-00002-of-00010.safetensors",
333
+ "model.layers.6.mlp.down_proj.weight": "model-00002-of-00010.safetensors",
334
+ "model.layers.6.mlp.gate_proj.weight": "model-00002-of-00010.safetensors",
335
+ "model.layers.6.mlp.up_proj.weight": "model-00002-of-00010.safetensors",
336
+ "model.layers.6.post_attention_layernorm.weight": "model-00002-of-00010.safetensors",
337
+ "model.layers.6.self_attn.k_proj.weight": "model-00002-of-00010.safetensors",
338
+ "model.layers.6.self_attn.o_proj.weight": "model-00002-of-00010.safetensors",
339
+ "model.layers.6.self_attn.q_proj.weight": "model-00002-of-00010.safetensors",
340
+ "model.layers.6.self_attn.v_proj.weight": "model-00002-of-00010.safetensors",
341
+ "model.layers.7.input_layernorm.weight": "model-00003-of-00010.safetensors",
342
+ "model.layers.7.mlp.down_proj.weight": "model-00003-of-00010.safetensors",
343
+ "model.layers.7.mlp.gate_proj.weight": "model-00002-of-00010.safetensors",
344
+ "model.layers.7.mlp.up_proj.weight": "model-00003-of-00010.safetensors",
345
+ "model.layers.7.post_attention_layernorm.weight": "model-00003-of-00010.safetensors",
346
+ "model.layers.7.self_attn.k_proj.weight": "model-00002-of-00010.safetensors",
347
+ "model.layers.7.self_attn.o_proj.weight": "model-00002-of-00010.safetensors",
348
+ "model.layers.7.self_attn.q_proj.weight": "model-00002-of-00010.safetensors",
349
+ "model.layers.7.self_attn.v_proj.weight": "model-00002-of-00010.safetensors",
350
+ "model.layers.8.input_layernorm.weight": "model-00003-of-00010.safetensors",
351
+ "model.layers.8.mlp.down_proj.weight": "model-00003-of-00010.safetensors",
352
+ "model.layers.8.mlp.gate_proj.weight": "model-00003-of-00010.safetensors",
353
+ "model.layers.8.mlp.up_proj.weight": "model-00003-of-00010.safetensors",
354
+ "model.layers.8.post_attention_layernorm.weight": "model-00003-of-00010.safetensors",
355
+ "model.layers.8.self_attn.k_proj.weight": "model-00003-of-00010.safetensors",
356
+ "model.layers.8.self_attn.o_proj.weight": "model-00003-of-00010.safetensors",
357
+ "model.layers.8.self_attn.q_proj.weight": "model-00003-of-00010.safetensors",
358
+ "model.layers.8.self_attn.v_proj.weight": "model-00003-of-00010.safetensors",
359
+ "model.layers.9.input_layernorm.weight": "model-00003-of-00010.safetensors",
360
+ "model.layers.9.mlp.down_proj.weight": "model-00003-of-00010.safetensors",
361
+ "model.layers.9.mlp.gate_proj.weight": "model-00003-of-00010.safetensors",
362
+ "model.layers.9.mlp.up_proj.weight": "model-00003-of-00010.safetensors",
363
+ "model.layers.9.post_attention_layernorm.weight": "model-00003-of-00010.safetensors",
364
+ "model.layers.9.self_attn.k_proj.weight": "model-00003-of-00010.safetensors",
365
+ "model.layers.9.self_attn.o_proj.weight": "model-00003-of-00010.safetensors",
366
+ "model.layers.9.self_attn.q_proj.weight": "model-00003-of-00010.safetensors",
367
+ "model.layers.9.self_attn.v_proj.weight": "model-00003-of-00010.safetensors",
368
+ "model.norm.weight": "model-00010-of-00010.safetensors"
369
+ }
370
+ }
special_tokens_map.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b76085f9923309d873994d444989f7eb6ec074b06f25b58f1e8d7b7741070949
3
+ size 17078037
tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff