Hugging Face's logo

JayceAnova
/
Benchmark-Singel-v2

TensorBoard
Safetensors
Model card Files
xet
Metrics Community
JayceAnova commited on
Commit
bd2cf44
·
verified ·
1 Parent(s): 52c9a61

Add files using upload-large-folder tool

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. Ins/added_tokens.json +1026 -0
  2. Ins/checkpoint-9678/added_tokens.json +1026 -0
  3. Ins/checkpoint-9678/latest +1 -0
  4. Ins/checkpoint-9678/model.safetensors.index.json +780 -0
  5. Ins/checkpoint-9678/trainer_state.json +0 -0
  6. Ins/checkpoint-9678/zero_to_fp32.py +674 -0
  7. Ins/config.json +111 -0
  8. Ins/finetune/README.md +202 -0
  9. Ins/finetune/adapter_config.json +42 -0
  10. Ins/finetune/added_tokens.json +1026 -0
  11. Ins/finetune/eval_result.json +47 -0
  12. Ins/finetune/log.txt +0 -0
  13. Ins/finetune/special_tokens_map.json +24 -0
  14. Ins/finetune/tokenizer_config.json +0 -0
  15. Ins/finetune/trainer_state.json +3682 -0
  16. Ins/indices.json +0 -0
  17. Ins/log.txt +0 -0
  18. Ins/model.safetensors.index.json +780 -0
  19. Ins/special_tokens_map.json +24 -0
  20. Ins/tokenizer_config.json +0 -0
  21. Ins/trainer_state.json +0 -0
  22. __pycache__/collator.cpython-312.pyc +0 -0
  23. __pycache__/data.cpython-312.pyc +0 -0
  24. __pycache__/data_finetune.cpython-312.pyc +0 -0
  25. __pycache__/evaluate.cpython-312.pyc +0 -0
  26. __pycache__/prompt.cpython-312.pyc +0 -0
  27. __pycache__/prompt_finetune.cpython-312.pyc +0 -0
  28. __pycache__/rq_llama.cpython-312.pyc +0 -0
  29. __pycache__/utils.cpython-312.pyc +0 -0
  30. collator.py +272 -0
  31. config/ds_z2_bf16.json +28 -0
  32. config/ds_z2_fp16.json +34 -0
  33. config/ds_z3_bf16.json +31 -0
  34. config/ds_z3_bf16_save16bit.json +31 -0
  35. config/ds_z3_fp16.json +37 -0
  36. config/ds_z3_fp16_save16bit.json +37 -0
  37. continue_finetune.py +108 -0
  38. continue_pretrain.py +126 -0
  39. convert/convert.log +1 -0
  40. convert/convert.py +16 -0
  41. convert/convert.sh +18 -0
  42. convert/convert_fp16.py +23 -0
  43. convert/make_delta.py +46 -0
  44. convert/merge_delta.py +167 -0
  45. convert/zero_to_fp32.py +600 -0
  46. data_finetune.py +852 -0
  47. data_process/amazon18_data_process.py +299 -0
  48. data_process/amazon18_recbole_data_process.py +226 -0
  49. data_process/amazon_text_emb.py +139 -0
  50. data_process/get_llm_output.py +374 -0
Ins/added_tokens.json ADDED
@@ -0,0 +1,1026 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "<a-0>": 32000,
3
+ "<a-100>": 32100,
4
+ "<a-101>": 32101,
5
+ "<a-102>": 32102,
6
+ "<a-103>": 32103,
7
+ "<a-104>": 32104,
8
+ "<a-105>": 32105,
9
+ "<a-106>": 32106,
10
+ "<a-107>": 32107,
11
+ "<a-108>": 32108,
12
+ "<a-109>": 32109,
13
+ "<a-10>": 32010,
14
+ "<a-110>": 32110,
15
+ "<a-111>": 32111,
16
+ "<a-112>": 32112,
17
+ "<a-113>": 32113,
18
+ "<a-114>": 32114,
19
+ "<a-115>": 32115,
20
+ "<a-116>": 32116,
21
+ "<a-117>": 32117,
22
+ "<a-118>": 32118,
23
+ "<a-119>": 32119,
24
+ "<a-11>": 32011,
25
+ "<a-120>": 32120,
26
+ "<a-121>": 32121,
27
+ "<a-122>": 32122,
28
+ "<a-123>": 32123,
29
+ "<a-124>": 32124,
30
+ "<a-125>": 32125,
31
+ "<a-126>": 32126,
32
+ "<a-127>": 32127,
33
+ "<a-128>": 32128,
34
+ "<a-129>": 32129,
35
+ "<a-12>": 32012,
36
+ "<a-130>": 32130,
37
+ "<a-131>": 32131,
38
+ "<a-132>": 32132,
39
+ "<a-133>": 32133,
40
+ "<a-134>": 32134,
41
+ "<a-135>": 32135,
42
+ "<a-136>": 32136,
43
+ "<a-137>": 32137,
44
+ "<a-138>": 32138,
45
+ "<a-139>": 32139,
46
+ "<a-13>": 32013,
47
+ "<a-140>": 32140,
48
+ "<a-141>": 32141,
49
+ "<a-142>": 32142,
50
+ "<a-143>": 32143,
51
+ "<a-144>": 32144,
52
+ "<a-145>": 32145,
53
+ "<a-146>": 32146,
54
+ "<a-147>": 32147,
55
+ "<a-148>": 32148,
56
+ "<a-149>": 32149,
57
+ "<a-14>": 32014,
58
+ "<a-150>": 32150,
59
+ "<a-151>": 32151,
60
+ "<a-152>": 32152,
61
+ "<a-153>": 32153,
62
+ "<a-154>": 32154,
63
+ "<a-155>": 32155,
64
+ "<a-156>": 32156,
65
+ "<a-157>": 32157,
66
+ "<a-158>": 32158,
67
+ "<a-159>": 32159,
68
+ "<a-15>": 32015,
69
+ "<a-160>": 32160,
70
+ "<a-161>": 32161,
71
+ "<a-162>": 32162,
72
+ "<a-163>": 32163,
73
+ "<a-164>": 32164,
74
+ "<a-165>": 32165,
75
+ "<a-166>": 32166,
76
+ "<a-167>": 32167,
77
+ "<a-168>": 32168,
78
+ "<a-169>": 32169,
79
+ "<a-16>": 32016,
80
+ "<a-170>": 32170,
81
+ "<a-171>": 32171,
82
+ "<a-172>": 32172,
83
+ "<a-173>": 32173,
84
+ "<a-174>": 32174,
85
+ "<a-175>": 32175,
86
+ "<a-176>": 32176,
87
+ "<a-177>": 32177,
88
+ "<a-178>": 32178,
89
+ "<a-179>": 32179,
90
+ "<a-17>": 32017,
91
+ "<a-180>": 32180,
92
+ "<a-181>": 32181,
93
+ "<a-182>": 32182,
94
+ "<a-183>": 32183,
95
+ "<a-184>": 32184,
96
+ "<a-185>": 32185,
97
+ "<a-186>": 32186,
98
+ "<a-187>": 32187,
99
+ "<a-188>": 32188,
100
+ "<a-189>": 32189,
101
+ "<a-18>": 32018,
102
+ "<a-190>": 32190,
103
+ "<a-191>": 32191,
104
+ "<a-192>": 32192,
105
+ "<a-193>": 32193,
106
+ "<a-194>": 32194,
107
+ "<a-195>": 32195,
108
+ "<a-196>": 32196,
109
+ "<a-197>": 32197,
110
+ "<a-198>": 32198,
111
+ "<a-199>": 32199,
112
+ "<a-19>": 32019,
113
+ "<a-1>": 32001,
114
+ "<a-200>": 32200,
115
+ "<a-201>": 32201,
116
+ "<a-202>": 32202,
117
+ "<a-203>": 32203,
118
+ "<a-204>": 32204,
119
+ "<a-205>": 32205,
120
+ "<a-206>": 32206,
121
+ "<a-207>": 32207,
122
+ "<a-208>": 32208,
123
+ "<a-209>": 32209,
124
+ "<a-20>": 32020,
125
+ "<a-210>": 32210,
126
+ "<a-211>": 32211,
127
+ "<a-212>": 32212,
128
+ "<a-213>": 32213,
129
+ "<a-214>": 32214,
130
+ "<a-215>": 32215,
131
+ "<a-216>": 32216,
132
+ "<a-217>": 32217,
133
+ "<a-218>": 32218,
134
+ "<a-219>": 32219,
135
+ "<a-21>": 32021,
136
+ "<a-220>": 32220,
137
+ "<a-221>": 32221,
138
+ "<a-222>": 32222,
139
+ "<a-223>": 32223,
140
+ "<a-224>": 32224,
141
+ "<a-225>": 32225,
142
+ "<a-226>": 32226,
143
+ "<a-227>": 32227,
144
+ "<a-228>": 32228,
145
+ "<a-229>": 32229,
146
+ "<a-22>": 32022,
147
+ "<a-230>": 32230,
148
+ "<a-231>": 32231,
149
+ "<a-232>": 32232,
150
+ "<a-233>": 32233,
151
+ "<a-234>": 32234,
152
+ "<a-235>": 32235,
153
+ "<a-236>": 32236,
154
+ "<a-237>": 32237,
155
+ "<a-238>": 32238,
156
+ "<a-239>": 32239,
157
+ "<a-23>": 32023,
158
+ "<a-240>": 32240,
159
+ "<a-241>": 32241,
160
+ "<a-242>": 32242,
161
+ "<a-243>": 32243,
162
+ "<a-244>": 32244,
163
+ "<a-245>": 32245,
164
+ "<a-246>": 32246,
165
+ "<a-247>": 32247,
166
+ "<a-248>": 32248,
167
+ "<a-249>": 32249,
168
+ "<a-24>": 32024,
169
+ "<a-250>": 32250,
170
+ "<a-251>": 32251,
171
+ "<a-252>": 32252,
172
+ "<a-253>": 32253,
173
+ "<a-254>": 32254,
174
+ "<a-255>": 32255,
175
+ "<a-25>": 32025,
176
+ "<a-26>": 32026,
177
+ "<a-27>": 32027,
178
+ "<a-28>": 32028,
179
+ "<a-29>": 32029,
180
+ "<a-2>": 32002,
181
+ "<a-30>": 32030,
182
+ "<a-31>": 32031,
183
+ "<a-32>": 32032,
184
+ "<a-33>": 32033,
185
+ "<a-34>": 32034,
186
+ "<a-35>": 32035,
187
+ "<a-36>": 32036,
188
+ "<a-37>": 32037,
189
+ "<a-38>": 32038,
190
+ "<a-39>": 32039,
191
+ "<a-3>": 32003,
192
+ "<a-40>": 32040,
193
+ "<a-41>": 32041,
194
+ "<a-42>": 32042,
195
+ "<a-43>": 32043,
196
+ "<a-44>": 32044,
197
+ "<a-45>": 32045,
198
+ "<a-46>": 32046,
199
+ "<a-47>": 32047,
200
+ "<a-48>": 32048,
201
+ "<a-49>": 32049,
202
+ "<a-4>": 32004,
203
+ "<a-50>": 32050,
204
+ "<a-51>": 32051,
205
+ "<a-52>": 32052,
206
+ "<a-53>": 32053,
207
+ "<a-54>": 32054,
208
+ "<a-55>": 32055,
209
+ "<a-56>": 32056,
210
+ "<a-57>": 32057,
211
+ "<a-58>": 32058,
212
+ "<a-59>": 32059,
213
+ "<a-5>": 32005,
214
+ "<a-60>": 32060,
215
+ "<a-61>": 32061,
216
+ "<a-62>": 32062,
217
+ "<a-63>": 32063,
218
+ "<a-64>": 32064,
219
+ "<a-65>": 32065,
220
+ "<a-66>": 32066,
221
+ "<a-67>": 32067,
222
+ "<a-68>": 32068,
223
+ "<a-69>": 32069,
224
+ "<a-6>": 32006,
225
+ "<a-70>": 32070,
226
+ "<a-71>": 32071,
227
+ "<a-72>": 32072,
228
+ "<a-73>": 32073,
229
+ "<a-74>": 32074,
230
+ "<a-75>": 32075,
231
+ "<a-76>": 32076,
232
+ "<a-77>": 32077,
233
+ "<a-78>": 32078,
234
+ "<a-79>": 32079,
235
+ "<a-7>": 32007,
236
+ "<a-80>": 32080,
237
+ "<a-81>": 32081,
238
+ "<a-82>": 32082,
239
+ "<a-83>": 32083,
240
+ "<a-84>": 32084,
241
+ "<a-85>": 32085,
242
+ "<a-86>": 32086,
243
+ "<a-87>": 32087,
244
+ "<a-88>": 32088,
245
+ "<a-89>": 32089,
246
+ "<a-8>": 32008,
247
+ "<a-90>": 32090,
248
+ "<a-91>": 32091,
249
+ "<a-92>": 32092,
250
+ "<a-93>": 32093,
251
+ "<a-94>": 32094,
252
+ "<a-95>": 32095,
253
+ "<a-96>": 32096,
254
+ "<a-97>": 32097,
255
+ "<a-98>": 32098,
256
+ "<a-99>": 32099,
257
+ "<a-9>": 32009,
258
+ "<b-0>": 32256,
259
+ "<b-100>": 32356,
260
+ "<b-101>": 32357,
261
+ "<b-102>": 32358,
262
+ "<b-103>": 32359,
263
+ "<b-104>": 32360,
264
+ "<b-105>": 32361,
265
+ "<b-106>": 32362,
266
+ "<b-107>": 32363,
267
+ "<b-108>": 32364,
268
+ "<b-109>": 32365,
269
+ "<b-10>": 32266,
270
+ "<b-110>": 32366,
271
+ "<b-111>": 32367,
272
+ "<b-112>": 32368,
273
+ "<b-113>": 32369,
274
+ "<b-114>": 32370,
275
+ "<b-115>": 32371,
276
+ "<b-116>": 32372,
277
+ "<b-117>": 32373,
278
+ "<b-118>": 32374,
279
+ "<b-119>": 32375,
280
+ "<b-11>": 32267,
281
+ "<b-120>": 32376,
282
+ "<b-121>": 32377,
283
+ "<b-122>": 32378,
284
+ "<b-123>": 32379,
285
+ "<b-124>": 32380,
286
+ "<b-125>": 32381,
287
+ "<b-126>": 32382,
288
+ "<b-127>": 32383,
289
+ "<b-128>": 32384,
290
+ "<b-129>": 32385,
291
+ "<b-12>": 32268,
292
+ "<b-130>": 32386,
293
+ "<b-131>": 32387,
294
+ "<b-132>": 32388,
295
+ "<b-133>": 32389,
296
+ "<b-134>": 32390,
297
+ "<b-135>": 32391,
298
+ "<b-136>": 32392,
299
+ "<b-137>": 32393,
300
+ "<b-138>": 32394,
301
+ "<b-139>": 32395,
302
+ "<b-13>": 32269,
303
+ "<b-140>": 32396,
304
+ "<b-141>": 32397,
305
+ "<b-142>": 32398,
306
+ "<b-143>": 32399,
307
+ "<b-144>": 32400,
308
+ "<b-145>": 32401,
309
+ "<b-146>": 32402,
310
+ "<b-147>": 32403,
311
+ "<b-148>": 32404,
312
+ "<b-149>": 32405,
313
+ "<b-14>": 32270,
314
+ "<b-150>": 32406,
315
+ "<b-151>": 32407,
316
+ "<b-152>": 32408,
317
+ "<b-153>": 32409,
318
+ "<b-154>": 32410,
319
+ "<b-155>": 32411,
320
+ "<b-156>": 32412,
321
+ "<b-157>": 32413,
322
+ "<b-158>": 32414,
323
+ "<b-159>": 32415,
324
+ "<b-15>": 32271,
325
+ "<b-160>": 32416,
326
+ "<b-161>": 32417,
327
+ "<b-162>": 32418,
328
+ "<b-163>": 32419,
329
+ "<b-164>": 32420,
330
+ "<b-165>": 32421,
331
+ "<b-166>": 32422,
332
+ "<b-167>": 32423,
333
+ "<b-168>": 32424,
334
+ "<b-169>": 32425,
335
+ "<b-16>": 32272,
336
+ "<b-170>": 32426,
337
+ "<b-171>": 32427,
338
+ "<b-172>": 32428,
339
+ "<b-173>": 32429,
340
+ "<b-174>": 32430,
341
+ "<b-175>": 32431,
342
+ "<b-176>": 32432,
343
+ "<b-177>": 32433,
344
+ "<b-178>": 32434,
345
+ "<b-179>": 32435,
346
+ "<b-17>": 32273,
347
+ "<b-180>": 32436,
348
+ "<b-181>": 32437,
349
+ "<b-182>": 32438,
350
+ "<b-183>": 32439,
351
+ "<b-184>": 32440,
352
+ "<b-185>": 32441,
353
+ "<b-186>": 32442,
354
+ "<b-187>": 32443,
355
+ "<b-188>": 32444,
356
+ "<b-189>": 32445,
357
+ "<b-18>": 32274,
358
+ "<b-190>": 32446,
359
+ "<b-191>": 32447,
360
+ "<b-192>": 32448,
361
+ "<b-193>": 32449,
362
+ "<b-194>": 32450,
363
+ "<b-195>": 32451,
364
+ "<b-196>": 32452,
365
+ "<b-197>": 32453,
366
+ "<b-198>": 32454,
367
+ "<b-199>": 32455,
368
+ "<b-19>": 32275,
369
+ "<b-1>": 32257,
370
+ "<b-200>": 32456,
371
+ "<b-201>": 32457,
372
+ "<b-202>": 32458,
373
+ "<b-203>": 32459,
374
+ "<b-204>": 32460,
375
+ "<b-205>": 32461,
376
+ "<b-206>": 32462,
377
+ "<b-207>": 32463,
378
+ "<b-208>": 32464,
379
+ "<b-209>": 32465,
380
+ "<b-20>": 32276,
381
+ "<b-210>": 32466,
382
+ "<b-211>": 32467,
383
+ "<b-212>": 32468,
384
+ "<b-213>": 32469,
385
+ "<b-214>": 32470,
386
+ "<b-215>": 32471,
387
+ "<b-216>": 32472,
388
+ "<b-217>": 32473,
389
+ "<b-218>": 32474,
390
+ "<b-219>": 32475,
391
+ "<b-21>": 32277,
392
+ "<b-220>": 32476,
393
+ "<b-221>": 32477,
394
+ "<b-222>": 32478,
395
+ "<b-223>": 32479,
396
+ "<b-224>": 32480,
397
+ "<b-225>": 32481,
398
+ "<b-226>": 32482,
399
+ "<b-227>": 32483,
400
+ "<b-228>": 32484,
401
+ "<b-229>": 32485,
402
+ "<b-22>": 32278,
403
+ "<b-230>": 32486,
404
+ "<b-231>": 32487,
405
+ "<b-232>": 32488,
406
+ "<b-233>": 32489,
407
+ "<b-234>": 32490,
408
+ "<b-235>": 32491,
409
+ "<b-236>": 32492,
410
+ "<b-237>": 32493,
411
+ "<b-238>": 32494,
412
+ "<b-239>": 32495,
413
+ "<b-23>": 32279,
414
+ "<b-240>": 32496,
415
+ "<b-241>": 32497,
416
+ "<b-242>": 32498,
417
+ "<b-243>": 32499,
418
+ "<b-244>": 32500,
419
+ "<b-245>": 32501,
420
+ "<b-246>": 32502,
421
+ "<b-247>": 32503,
422
+ "<b-248>": 32504,
423
+ "<b-249>": 32505,
424
+ "<b-24>": 32280,
425
+ "<b-250>": 32506,
426
+ "<b-251>": 32507,
427
+ "<b-252>": 32508,
428
+ "<b-253>": 32509,
429
+ "<b-254>": 32510,
430
+ "<b-255>": 32511,
431
+ "<b-25>": 32281,
432
+ "<b-26>": 32282,
433
+ "<b-27>": 32283,
434
+ "<b-28>": 32284,
435
+ "<b-29>": 32285,
436
+ "<b-2>": 32258,
437
+ "<b-30>": 32286,
438
+ "<b-31>": 32287,
439
+ "<b-32>": 32288,
440
+ "<b-33>": 32289,
441
+ "<b-34>": 32290,
442
+ "<b-35>": 32291,
443
+ "<b-36>": 32292,
444
+ "<b-37>": 32293,
445
+ "<b-38>": 32294,
446
+ "<b-39>": 32295,
447
+ "<b-3>": 32259,
448
+ "<b-40>": 32296,
449
+ "<b-41>": 32297,
450
+ "<b-42>": 32298,
451
+ "<b-43>": 32299,
452
+ "<b-44>": 32300,
453
+ "<b-45>": 32301,
454
+ "<b-46>": 32302,
455
+ "<b-47>": 32303,
456
+ "<b-48>": 32304,
457
+ "<b-49>": 32305,
458
+ "<b-4>": 32260,
459
+ "<b-50>": 32306,
460
+ "<b-51>": 32307,
461
+ "<b-52>": 32308,
462
+ "<b-53>": 32309,
463
+ "<b-54>": 32310,
464
+ "<b-55>": 32311,
465
+ "<b-56>": 32312,
466
+ "<b-57>": 32313,
467
+ "<b-58>": 32314,
468
+ "<b-59>": 32315,
469
+ "<b-5>": 32261,
470
+ "<b-60>": 32316,
471
+ "<b-61>": 32317,
472
+ "<b-62>": 32318,
473
+ "<b-63>": 32319,
474
+ "<b-64>": 32320,
475
+ "<b-65>": 32321,
476
+ "<b-66>": 32322,
477
+ "<b-67>": 32323,
478
+ "<b-68>": 32324,
479
+ "<b-69>": 32325,
480
+ "<b-6>": 32262,
481
+ "<b-70>": 32326,
482
+ "<b-71>": 32327,
483
+ "<b-72>": 32328,
484
+ "<b-73>": 32329,
485
+ "<b-74>": 32330,
486
+ "<b-75>": 32331,
487
+ "<b-76>": 32332,
488
+ "<b-77>": 32333,
489
+ "<b-78>": 32334,
490
+ "<b-79>": 32335,
491
+ "<b-7>": 32263,
492
+ "<b-80>": 32336,
493
+ "<b-81>": 32337,
494
+ "<b-82>": 32338,
495
+ "<b-83>": 32339,
496
+ "<b-84>": 32340,
497
+ "<b-85>": 32341,
498
+ "<b-86>": 32342,
499
+ "<b-87>": 32343,
500
+ "<b-88>": 32344,
501
+ "<b-89>": 32345,
502
+ "<b-8>": 32264,
503
+ "<b-90>": 32346,
504
+ "<b-91>": 32347,
505
+ "<b-92>": 32348,
506
+ "<b-93>": 32349,
507
+ "<b-94>": 32350,
508
+ "<b-95>": 32351,
509
+ "<b-96>": 32352,
510
+ "<b-97>": 32353,
511
+ "<b-98>": 32354,
512
+ "<b-99>": 32355,
513
+ "<b-9>": 32265,
514
+ "<c-0>": 32512,
515
+ "<c-100>": 32612,
516
+ "<c-101>": 32613,
517
+ "<c-102>": 32614,
518
+ "<c-103>": 32615,
519
+ "<c-104>": 32616,
520
+ "<c-105>": 32617,
521
+ "<c-106>": 32618,
522
+ "<c-107>": 32619,
523
+ "<c-108>": 32620,
524
+ "<c-109>": 32621,
525
+ "<c-10>": 32522,
526
+ "<c-110>": 32622,
527
+ "<c-111>": 32623,
528
+ "<c-112>": 32624,
529
+ "<c-113>": 32625,
530
+ "<c-114>": 32626,
531
+ "<c-115>": 32627,
532
+ "<c-116>": 32628,
533
+ "<c-117>": 32629,
534
+ "<c-118>": 32630,
535
+ "<c-119>": 32631,
536
+ "<c-11>": 32523,
537
+ "<c-120>": 32632,
538
+ "<c-121>": 32633,
539
+ "<c-122>": 32634,
540
+ "<c-123>": 32635,
541
+ "<c-124>": 32636,
542
+ "<c-125>": 32637,
543
+ "<c-126>": 32638,
544
+ "<c-127>": 32639,
545
+ "<c-128>": 32640,
546
+ "<c-129>": 32641,
547
+ "<c-12>": 32524,
548
+ "<c-130>": 32642,
549
+ "<c-131>": 32643,
550
+ "<c-132>": 32644,
551
+ "<c-133>": 32645,
552
+ "<c-134>": 32646,
553
+ "<c-135>": 32647,
554
+ "<c-136>": 32648,
555
+ "<c-137>": 32649,
556
+ "<c-138>": 32650,
557
+ "<c-139>": 32651,
558
+ "<c-13>": 32525,
559
+ "<c-140>": 32652,
560
+ "<c-141>": 32653,
561
+ "<c-142>": 32654,
562
+ "<c-143>": 32655,
563
+ "<c-144>": 32656,
564
+ "<c-145>": 32657,
565
+ "<c-146>": 32658,
566
+ "<c-147>": 32659,
567
+ "<c-148>": 32660,
568
+ "<c-149>": 32661,
569
+ "<c-14>": 32526,
570
+ "<c-150>": 32662,
571
+ "<c-151>": 32663,
572
+ "<c-152>": 32664,
573
+ "<c-153>": 32665,
574
+ "<c-154>": 32666,
575
+ "<c-155>": 32667,
576
+ "<c-156>": 32668,
577
+ "<c-157>": 32669,
578
+ "<c-158>": 32670,
579
+ "<c-159>": 32671,
580
+ "<c-15>": 32527,
581
+ "<c-160>": 32672,
582
+ "<c-161>": 32673,
583
+ "<c-162>": 32674,
584
+ "<c-163>": 32675,
585
+ "<c-164>": 32676,
586
+ "<c-165>": 32677,
587
+ "<c-166>": 32678,
588
+ "<c-167>": 32679,
589
+ "<c-168>": 32680,
590
+ "<c-169>": 32681,
591
+ "<c-16>": 32528,
592
+ "<c-170>": 32682,
593
+ "<c-171>": 32683,
594
+ "<c-172>": 32684,
595
+ "<c-173>": 32685,
596
+ "<c-174>": 32686,
597
+ "<c-175>": 32687,
598
+ "<c-176>": 32688,
599
+ "<c-177>": 32689,
600
+ "<c-178>": 32690,
601
+ "<c-179>": 32691,
602
+ "<c-17>": 32529,
603
+ "<c-180>": 32692,
604
+ "<c-181>": 32693,
605
+ "<c-182>": 32694,
606
+ "<c-183>": 32695,
607
+ "<c-184>": 32696,
608
+ "<c-185>": 32697,
609
+ "<c-186>": 32698,
610
+ "<c-187>": 32699,
611
+ "<c-188>": 32700,
612
+ "<c-189>": 32701,
613
+ "<c-18>": 32530,
614
+ "<c-190>": 32702,
615
+ "<c-191>": 32703,
616
+ "<c-192>": 32704,
617
+ "<c-193>": 32705,
618
+ "<c-194>": 32706,
619
+ "<c-195>": 32707,
620
+ "<c-196>": 32708,
621
+ "<c-197>": 32709,
622
+ "<c-198>": 32710,
623
+ "<c-199>": 32711,
624
+ "<c-19>": 32531,
625
+ "<c-1>": 32513,
626
+ "<c-200>": 32712,
627
+ "<c-201>": 32713,
628
+ "<c-202>": 32714,
629
+ "<c-203>": 32715,
630
+ "<c-204>": 32716,
631
+ "<c-205>": 32717,
632
+ "<c-206>": 32718,
633
+ "<c-207>": 32719,
634
+ "<c-208>": 32720,
635
+ "<c-209>": 32721,
636
+ "<c-20>": 32532,
637
+ "<c-210>": 32722,
638
+ "<c-211>": 32723,
639
+ "<c-212>": 32724,
640
+ "<c-213>": 32725,
641
+ "<c-214>": 32726,
642
+ "<c-215>": 32727,
643
+ "<c-216>": 32728,
644
+ "<c-217>": 32729,
645
+ "<c-218>": 32730,
646
+ "<c-219>": 32731,
647
+ "<c-21>": 32533,
648
+ "<c-220>": 32732,
649
+ "<c-221>": 32733,
650
+ "<c-222>": 32734,
651
+ "<c-223>": 32735,
652
+ "<c-224>": 32736,
653
+ "<c-225>": 32737,
654
+ "<c-226>": 32738,
655
+ "<c-227>": 32739,
656
+ "<c-228>": 32740,
657
+ "<c-229>": 32741,
658
+ "<c-22>": 32534,
659
+ "<c-230>": 32742,
660
+ "<c-231>": 32743,
661
+ "<c-232>": 32744,
662
+ "<c-233>": 32745,
663
+ "<c-234>": 32746,
664
+ "<c-235>": 32747,
665
+ "<c-236>": 32748,
666
+ "<c-237>": 32749,
667
+ "<c-238>": 32750,
668
+ "<c-239>": 32751,
669
+ "<c-23>": 32535,
670
+ "<c-240>": 32752,
671
+ "<c-241>": 32753,
672
+ "<c-242>": 32754,
673
+ "<c-243>": 32755,
674
+ "<c-244>": 32756,
675
+ "<c-245>": 32757,
676
+ "<c-246>": 32758,
677
+ "<c-247>": 32759,
678
+ "<c-248>": 32760,
679
+ "<c-249>": 32761,
680
+ "<c-24>": 32536,
681
+ "<c-250>": 32762,
682
+ "<c-251>": 32763,
683
+ "<c-252>": 32764,
684
+ "<c-253>": 32765,
685
+ "<c-254>": 32766,
686
+ "<c-255>": 32767,
687
+ "<c-25>": 32537,
688
+ "<c-26>": 32538,
689
+ "<c-27>": 32539,
690
+ "<c-28>": 32540,
691
+ "<c-29>": 32541,
692
+ "<c-2>": 32514,
693
+ "<c-30>": 32542,
694
+ "<c-31>": 32543,
695
+ "<c-32>": 32544,
696
+ "<c-33>": 32545,
697
+ "<c-34>": 32546,
698
+ "<c-35>": 32547,
699
+ "<c-36>": 32548,
700
+ "<c-37>": 32549,
701
+ "<c-38>": 32550,
702
+ "<c-39>": 32551,
703
+ "<c-3>": 32515,
704
+ "<c-40>": 32552,
705
+ "<c-41>": 32553,
706
+ "<c-42>": 32554,
707
+ "<c-43>": 32555,
708
+ "<c-44>": 32556,
709
+ "<c-45>": 32557,
710
+ "<c-46>": 32558,
711
+ "<c-47>": 32559,
712
+ "<c-48>": 32560,
713
+ "<c-49>": 32561,
714
+ "<c-4>": 32516,
715
+ "<c-50>": 32562,
716
+ "<c-51>": 32563,
717
+ "<c-52>": 32564,
718
+ "<c-53>": 32565,
719
+ "<c-54>": 32566,
720
+ "<c-55>": 32567,
721
+ "<c-56>": 32568,
722
+ "<c-57>": 32569,
723
+ "<c-58>": 32570,
724
+ "<c-59>": 32571,
725
+ "<c-5>": 32517,
726
+ "<c-60>": 32572,
727
+ "<c-61>": 32573,
728
+ "<c-62>": 32574,
729
+ "<c-63>": 32575,
730
+ "<c-64>": 32576,
731
+ "<c-65>": 32577,
732
+ "<c-66>": 32578,
733
+ "<c-67>": 32579,
734
+ "<c-68>": 32580,
735
+ "<c-69>": 32581,
736
+ "<c-6>": 32518,
737
+ "<c-70>": 32582,
738
+ "<c-71>": 32583,
739
+ "<c-72>": 32584,
740
+ "<c-73>": 32585,
741
+ "<c-74>": 32586,
742
+ "<c-75>": 32587,
743
+ "<c-76>": 32588,
744
+ "<c-77>": 32589,
745
+ "<c-78>": 32590,
746
+ "<c-79>": 32591,
747
+ "<c-7>": 32519,
748
+ "<c-80>": 32592,
749
+ "<c-81>": 32593,
750
+ "<c-82>": 32594,
751
+ "<c-83>": 32595,
752
+ "<c-84>": 32596,
753
+ "<c-85>": 32597,
754
+ "<c-86>": 32598,
755
+ "<c-87>": 32599,
756
+ "<c-88>": 32600,
757
+ "<c-89>": 32601,
758
+ "<c-8>": 32520,
759
+ "<c-90>": 32602,
760
+ "<c-91>": 32603,
761
+ "<c-92>": 32604,
762
+ "<c-93>": 32605,
763
+ "<c-94>": 32606,
764
+ "<c-95>": 32607,
765
+ "<c-96>": 32608,
766
+ "<c-97>": 32609,
767
+ "<c-98>": 32610,
768
+ "<c-99>": 32611,
769
+ "<c-9>": 32521,
770
+ "<d-0>": 32768,
771
+ "<d-100>": 32868,
772
+ "<d-101>": 32869,
773
+ "<d-102>": 32870,
774
+ "<d-103>": 32871,
775
+ "<d-104>": 32872,
776
+ "<d-105>": 32873,
777
+ "<d-106>": 32874,
778
+ "<d-107>": 32875,
779
+ "<d-108>": 32876,
780
+ "<d-109>": 32877,
781
+ "<d-10>": 32778,
782
+ "<d-110>": 32878,
783
+ "<d-111>": 32879,
784
+ "<d-112>": 32880,
785
+ "<d-113>": 32881,
786
+ "<d-114>": 32882,
787
+ "<d-115>": 32883,
788
+ "<d-116>": 32884,
789
+ "<d-117>": 32885,
790
+ "<d-118>": 32886,
791
+ "<d-119>": 32887,
792
+ "<d-11>": 32779,
793
+ "<d-120>": 32888,
794
+ "<d-121>": 32889,
795
+ "<d-122>": 32890,
796
+ "<d-123>": 32891,
797
+ "<d-124>": 32892,
798
+ "<d-125>": 32893,
799
+ "<d-126>": 32894,
800
+ "<d-127>": 32895,
801
+ "<d-128>": 32896,
802
+ "<d-129>": 32897,
803
+ "<d-12>": 32780,
804
+ "<d-130>": 32898,
805
+ "<d-131>": 32899,
806
+ "<d-132>": 32900,
807
+ "<d-133>": 32901,
808
+ "<d-134>": 32902,
809
+ "<d-135>": 32903,
810
+ "<d-136>": 32904,
811
+ "<d-137>": 32905,
812
+ "<d-138>": 32906,
813
+ "<d-139>": 32907,
814
+ "<d-13>": 32781,
815
+ "<d-140>": 32908,
816
+ "<d-141>": 32909,
817
+ "<d-142>": 32910,
818
+ "<d-143>": 32911,
819
+ "<d-144>": 32912,
820
+ "<d-145>": 32913,
821
+ "<d-146>": 32914,
822
+ "<d-147>": 32915,
823
+ "<d-148>": 32916,
824
+ "<d-149>": 32917,
825
+ "<d-14>": 32782,
826
+ "<d-150>": 32918,
827
+ "<d-151>": 32919,
828
+ "<d-152>": 32920,
829
+ "<d-153>": 32921,
830
+ "<d-154>": 32922,
831
+ "<d-155>": 32923,
832
+ "<d-156>": 32924,
833
+ "<d-157>": 32925,
834
+ "<d-158>": 32926,
835
+ "<d-159>": 32927,
836
+ "<d-15>": 32783,
837
+ "<d-160>": 32928,
838
+ "<d-161>": 32929,
839
+ "<d-162>": 32930,
840
+ "<d-163>": 32931,
841
+ "<d-164>": 32932,
842
+ "<d-165>": 32933,
843
+ "<d-166>": 32934,
844
+ "<d-167>": 32935,
845
+ "<d-168>": 32936,
846
+ "<d-169>": 32937,
847
+ "<d-16>": 32784,
848
+ "<d-170>": 32938,
849
+ "<d-171>": 32939,
850
+ "<d-172>": 32940,
851
+ "<d-173>": 32941,
852
+ "<d-174>": 32942,
853
+ "<d-175>": 32943,
854
+ "<d-176>": 32944,
855
+ "<d-177>": 32945,
856
+ "<d-178>": 32946,
857
+ "<d-179>": 32947,
858
+ "<d-17>": 32785,
859
+ "<d-180>": 32948,
860
+ "<d-181>": 32949,
861
+ "<d-182>": 32950,
862
+ "<d-183>": 32951,
863
+ "<d-184>": 32952,
864
+ "<d-185>": 32953,
865
+ "<d-186>": 32954,
866
+ "<d-187>": 32955,
867
+ "<d-188>": 32956,
868
+ "<d-189>": 32957,
869
+ "<d-18>": 32786,
870
+ "<d-190>": 32958,
871
+ "<d-191>": 32959,
872
+ "<d-192>": 32960,
873
+ "<d-193>": 32961,
874
+ "<d-194>": 32962,
875
+ "<d-195>": 32963,
876
+ "<d-196>": 32964,
877
+ "<d-197>": 32965,
878
+ "<d-198>": 32966,
879
+ "<d-199>": 32967,
880
+ "<d-19>": 32787,
881
+ "<d-1>": 32769,
882
+ "<d-200>": 32968,
883
+ "<d-201>": 32969,
884
+ "<d-202>": 32970,
885
+ "<d-203>": 32971,
886
+ "<d-204>": 32972,
887
+ "<d-205>": 32973,
888
+ "<d-206>": 32974,
889
+ "<d-207>": 32975,
890
+ "<d-208>": 32976,
891
+ "<d-209>": 32977,
892
+ "<d-20>": 32788,
893
+ "<d-210>": 32978,
894
+ "<d-211>": 32979,
895
+ "<d-212>": 32980,
896
+ "<d-213>": 32981,
897
+ "<d-214>": 32982,
898
+ "<d-215>": 32983,
899
+ "<d-216>": 32984,
900
+ "<d-217>": 32985,
901
+ "<d-218>": 32986,
902
+ "<d-219>": 32987,
903
+ "<d-21>": 32789,
904
+ "<d-220>": 32988,
905
+ "<d-221>": 32989,
906
+ "<d-222>": 32990,
907
+ "<d-223>": 32991,
908
+ "<d-224>": 32992,
909
+ "<d-225>": 32993,
910
+ "<d-226>": 32994,
911
+ "<d-227>": 32995,
912
+ "<d-228>": 32996,
913
+ "<d-229>": 32997,
914
+ "<d-22>": 32790,
915
+ "<d-230>": 32998,
916
+ "<d-231>": 32999,
917
+ "<d-232>": 33000,
918
+ "<d-233>": 33001,
919
+ "<d-234>": 33002,
920
+ "<d-235>": 33003,
921
+ "<d-236>": 33004,
922
+ "<d-237>": 33005,
923
+ "<d-238>": 33006,
924
+ "<d-239>": 33007,
925
+ "<d-23>": 32791,
926
+ "<d-240>": 33008,
927
+ "<d-241>": 33009,
928
+ "<d-242>": 33010,
929
+ "<d-243>": 33011,
930
+ "<d-244>": 33012,
931
+ "<d-245>": 33013,
932
+ "<d-246>": 33014,
933
+ "<d-247>": 33015,
934
+ "<d-248>": 33016,
935
+ "<d-249>": 33017,
936
+ "<d-24>": 32792,
937
+ "<d-250>": 33018,
938
+ "<d-251>": 33019,
939
+ "<d-252>": 33020,
940
+ "<d-253>": 33021,
941
+ "<d-254>": 33022,
942
+ "<d-255>": 33023,
943
+ "<d-25>": 32793,
944
+ "<d-26>": 32794,
945
+ "<d-27>": 32795,
946
+ "<d-28>": 32796,
947
+ "<d-29>": 32797,
948
+ "<d-2>": 32770,
949
+ "<d-30>": 32798,
950
+ "<d-31>": 32799,
951
+ "<d-32>": 32800,
952
+ "<d-33>": 32801,
953
+ "<d-34>": 32802,
954
+ "<d-35>": 32803,
955
+ "<d-36>": 32804,
956
+ "<d-37>": 32805,
957
+ "<d-38>": 32806,
958
+ "<d-39>": 32807,
959
+ "<d-3>": 32771,
960
+ "<d-40>": 32808,
961
+ "<d-41>": 32809,
962
+ "<d-42>": 32810,
963
+ "<d-43>": 32811,
964
+ "<d-44>": 32812,
965
+ "<d-45>": 32813,
966
+ "<d-46>": 32814,
967
+ "<d-47>": 32815,
968
+ "<d-48>": 32816,
969
+ "<d-49>": 32817,
970
+ "<d-4>": 32772,
971
+ "<d-50>": 32818,
972
+ "<d-51>": 32819,
973
+ "<d-52>": 32820,
974
+ "<d-53>": 32821,
975
+ "<d-54>": 32822,
976
+ "<d-55>": 32823,
977
+ "<d-56>": 32824,
978
+ "<d-57>": 32825,
979
+ "<d-58>": 32826,
980
+ "<d-59>": 32827,
981
+ "<d-5>": 32773,
982
+ "<d-60>": 32828,
983
+ "<d-61>": 32829,
984
+ "<d-62>": 32830,
985
+ "<d-63>": 32831,
986
+ "<d-64>": 32832,
987
+ "<d-65>": 32833,
988
+ "<d-66>": 32834,
989
+ "<d-67>": 32835,
990
+ "<d-68>": 32836,
991
+ "<d-69>": 32837,
992
+ "<d-6>": 32774,
993
+ "<d-70>": 32838,
994
+ "<d-71>": 32839,
995
+ "<d-72>": 32840,
996
+ "<d-73>": 32841,
997
+ "<d-74>": 32842,
998
+ "<d-75>": 32843,
999
+ "<d-76>": 32844,
1000
+ "<d-77>": 32845,
1001
+ "<d-78>": 32846,
1002
+ "<d-79>": 32847,
1003
+ "<d-7>": 32775,
1004
+ "<d-80>": 32848,
1005
+ "<d-81>": 32849,
1006
+ "<d-82>": 32850,
1007
+ "<d-83>": 32851,
1008
+ "<d-84>": 32852,
1009
+ "<d-85>": 32853,
1010
+ "<d-86>": 32854,
1011
+ "<d-87>": 32855,
1012
+ "<d-88>": 32856,
1013
+ "<d-89>": 32857,
1014
+ "<d-8>": 32776,
1015
+ "<d-90>": 32858,
1016
+ "<d-91>": 32859,
1017
+ "<d-92>": 32860,
1018
+ "<d-93>": 32861,
1019
+ "<d-94>": 32862,
1020
+ "<d-95>": 32863,
1021
+ "<d-96>": 32864,
1022
+ "<d-97>": 32865,
1023
+ "<d-98>": 32866,
1024
+ "<d-99>": 32867,
1025
+ "<d-9>": 32777
1026
+ }
Ins/checkpoint-9678/added_tokens.json ADDED
@@ -0,0 +1,1026 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "<a-0>": 32000,
3
+ "<a-100>": 32100,
4
+ "<a-101>": 32101,
5
+ "<a-102>": 32102,
6
+ "<a-103>": 32103,
7
+ "<a-104>": 32104,
8
+ "<a-105>": 32105,
9
+ "<a-106>": 32106,
10
+ "<a-107>": 32107,
11
+ "<a-108>": 32108,
12
+ "<a-109>": 32109,
13
+ "<a-10>": 32010,
14
+ "<a-110>": 32110,
15
+ "<a-111>": 32111,
16
+ "<a-112>": 32112,
17
+ "<a-113>": 32113,
18
+ "<a-114>": 32114,
19
+ "<a-115>": 32115,
20
+ "<a-116>": 32116,
21
+ "<a-117>": 32117,
22
+ "<a-118>": 32118,
23
+ "<a-119>": 32119,
24
+ "<a-11>": 32011,
25
+ "<a-120>": 32120,
26
+ "<a-121>": 32121,
27
+ "<a-122>": 32122,
28
+ "<a-123>": 32123,
29
+ "<a-124>": 32124,
30
+ "<a-125>": 32125,
31
+ "<a-126>": 32126,
32
+ "<a-127>": 32127,
33
+ "<a-128>": 32128,
34
+ "<a-129>": 32129,
35
+ "<a-12>": 32012,
36
+ "<a-130>": 32130,
37
+ "<a-131>": 32131,
38
+ "<a-132>": 32132,
39
+ "<a-133>": 32133,
40
+ "<a-134>": 32134,
41
+ "<a-135>": 32135,
42
+ "<a-136>": 32136,
43
+ "<a-137>": 32137,
44
+ "<a-138>": 32138,
45
+ "<a-139>": 32139,
46
+ "<a-13>": 32013,
47
+ "<a-140>": 32140,
48
+ "<a-141>": 32141,
49
+ "<a-142>": 32142,
50
+ "<a-143>": 32143,
51
+ "<a-144>": 32144,
52
+ "<a-145>": 32145,
53
+ "<a-146>": 32146,
54
+ "<a-147>": 32147,
55
+ "<a-148>": 32148,
56
+ "<a-149>": 32149,
57
+ "<a-14>": 32014,
58
+ "<a-150>": 32150,
59
+ "<a-151>": 32151,
60
+ "<a-152>": 32152,
61
+ "<a-153>": 32153,
62
+ "<a-154>": 32154,
63
+ "<a-155>": 32155,
64
+ "<a-156>": 32156,
65
+ "<a-157>": 32157,
66
+ "<a-158>": 32158,
67
+ "<a-159>": 32159,
68
+ "<a-15>": 32015,
69
+ "<a-160>": 32160,
70
+ "<a-161>": 32161,
71
+ "<a-162>": 32162,
72
+ "<a-163>": 32163,
73
+ "<a-164>": 32164,
74
+ "<a-165>": 32165,
75
+ "<a-166>": 32166,
76
+ "<a-167>": 32167,
77
+ "<a-168>": 32168,
78
+ "<a-169>": 32169,
79
+ "<a-16>": 32016,
80
+ "<a-170>": 32170,
81
+ "<a-171>": 32171,
82
+ "<a-172>": 32172,
83
+ "<a-173>": 32173,
84
+ "<a-174>": 32174,
85
+ "<a-175>": 32175,
86
+ "<a-176>": 32176,
87
+ "<a-177>": 32177,
88
+ "<a-178>": 32178,
89
+ "<a-179>": 32179,
90
+ "<a-17>": 32017,
91
+ "<a-180>": 32180,
92
+ "<a-181>": 32181,
93
+ "<a-182>": 32182,
94
+ "<a-183>": 32183,
95
+ "<a-184>": 32184,
96
+ "<a-185>": 32185,
97
+ "<a-186>": 32186,
98
+ "<a-187>": 32187,
99
+ "<a-188>": 32188,
100
+ "<a-189>": 32189,
101
+ "<a-18>": 32018,
102
+ "<a-190>": 32190,
103
+ "<a-191>": 32191,
104
+ "<a-192>": 32192,
105
+ "<a-193>": 32193,
106
+ "<a-194>": 32194,
107
+ "<a-195>": 32195,
108
+ "<a-196>": 32196,
109
+ "<a-197>": 32197,
110
+ "<a-198>": 32198,
111
+ "<a-199>": 32199,
112
+ "<a-19>": 32019,
113
+ "<a-1>": 32001,
114
+ "<a-200>": 32200,
115
+ "<a-201>": 32201,
116
+ "<a-202>": 32202,
117
+ "<a-203>": 32203,
118
+ "<a-204>": 32204,
119
+ "<a-205>": 32205,
120
+ "<a-206>": 32206,
121
+ "<a-207>": 32207,
122
+ "<a-208>": 32208,
123
+ "<a-209>": 32209,
124
+ "<a-20>": 32020,
125
+ "<a-210>": 32210,
126
+ "<a-211>": 32211,
127
+ "<a-212>": 32212,
128
+ "<a-213>": 32213,
129
+ "<a-214>": 32214,
130
+ "<a-215>": 32215,
131
+ "<a-216>": 32216,
132
+ "<a-217>": 32217,
133
+ "<a-218>": 32218,
134
+ "<a-219>": 32219,
135
+ "<a-21>": 32021,
136
+ "<a-220>": 32220,
137
+ "<a-221>": 32221,
138
+ "<a-222>": 32222,
139
+ "<a-223>": 32223,
140
+ "<a-224>": 32224,
141
+ "<a-225>": 32225,
142
+ "<a-226>": 32226,
143
+ "<a-227>": 32227,
144
+ "<a-228>": 32228,
145
+ "<a-229>": 32229,
146
+ "<a-22>": 32022,
147
+ "<a-230>": 32230,
148
+ "<a-231>": 32231,
149
+ "<a-232>": 32232,
150
+ "<a-233>": 32233,
151
+ "<a-234>": 32234,
152
+ "<a-235>": 32235,
153
+ "<a-236>": 32236,
154
+ "<a-237>": 32237,
155
+ "<a-238>": 32238,
156
+ "<a-239>": 32239,
157
+ "<a-23>": 32023,
158
+ "<a-240>": 32240,
159
+ "<a-241>": 32241,
160
+ "<a-242>": 32242,
161
+ "<a-243>": 32243,
162
+ "<a-244>": 32244,
163
+ "<a-245>": 32245,
164
+ "<a-246>": 32246,
165
+ "<a-247>": 32247,
166
+ "<a-248>": 32248,
167
+ "<a-249>": 32249,
168
+ "<a-24>": 32024,
169
+ "<a-250>": 32250,
170
+ "<a-251>": 32251,
171
+ "<a-252>": 32252,
172
+ "<a-253>": 32253,
173
+ "<a-254>": 32254,
174
+ "<a-255>": 32255,
175
+ "<a-25>": 32025,
176
+ "<a-26>": 32026,
177
+ "<a-27>": 32027,
178
+ "<a-28>": 32028,
179
+ "<a-29>": 32029,
180
+ "<a-2>": 32002,
181
+ "<a-30>": 32030,
182
+ "<a-31>": 32031,
183
+ "<a-32>": 32032,
184
+ "<a-33>": 32033,
185
+ "<a-34>": 32034,
186
+ "<a-35>": 32035,
187
+ "<a-36>": 32036,
188
+ "<a-37>": 32037,
189
+ "<a-38>": 32038,
190
+ "<a-39>": 32039,
191
+ "<a-3>": 32003,
192
+ "<a-40>": 32040,
193
+ "<a-41>": 32041,
194
+ "<a-42>": 32042,
195
+ "<a-43>": 32043,
196
+ "<a-44>": 32044,
197
+ "<a-45>": 32045,
198
+ "<a-46>": 32046,
199
+ "<a-47>": 32047,
200
+ "<a-48>": 32048,
201
+ "<a-49>": 32049,
202
+ "<a-4>": 32004,
203
+ "<a-50>": 32050,
204
+ "<a-51>": 32051,
205
+ "<a-52>": 32052,
206
+ "<a-53>": 32053,
207
+ "<a-54>": 32054,
208
+ "<a-55>": 32055,
209
+ "<a-56>": 32056,
210
+ "<a-57>": 32057,
211
+ "<a-58>": 32058,
212
+ "<a-59>": 32059,
213
+ "<a-5>": 32005,
214
+ "<a-60>": 32060,
215
+ "<a-61>": 32061,
216
+ "<a-62>": 32062,
217
+ "<a-63>": 32063,
218
+ "<a-64>": 32064,
219
+ "<a-65>": 32065,
220
+ "<a-66>": 32066,
221
+ "<a-67>": 32067,
222
+ "<a-68>": 32068,
223
+ "<a-69>": 32069,
224
+ "<a-6>": 32006,
225
+ "<a-70>": 32070,
226
+ "<a-71>": 32071,
227
+ "<a-72>": 32072,
228
+ "<a-73>": 32073,
229
+ "<a-74>": 32074,
230
+ "<a-75>": 32075,
231
+ "<a-76>": 32076,
232
+ "<a-77>": 32077,
233
+ "<a-78>": 32078,
234
+ "<a-79>": 32079,
235
+ "<a-7>": 32007,
236
+ "<a-80>": 32080,
237
+ "<a-81>": 32081,
238
+ "<a-82>": 32082,
239
+ "<a-83>": 32083,
240
+ "<a-84>": 32084,
241
+ "<a-85>": 32085,
242
+ "<a-86>": 32086,
243
+ "<a-87>": 32087,
244
+ "<a-88>": 32088,
245
+ "<a-89>": 32089,
246
+ "<a-8>": 32008,
247
+ "<a-90>": 32090,
248
+ "<a-91>": 32091,
249
+ "<a-92>": 32092,
250
+ "<a-93>": 32093,
251
+ "<a-94>": 32094,
252
+ "<a-95>": 32095,
253
+ "<a-96>": 32096,
254
+ "<a-97>": 32097,
255
+ "<a-98>": 32098,
256
+ "<a-99>": 32099,
257
+ "<a-9>": 32009,
258
+ "<b-0>": 32256,
259
+ "<b-100>": 32356,
260
+ "<b-101>": 32357,
261
+ "<b-102>": 32358,
262
+ "<b-103>": 32359,
263
+ "<b-104>": 32360,
264
+ "<b-105>": 32361,
265
+ "<b-106>": 32362,
266
+ "<b-107>": 32363,
267
+ "<b-108>": 32364,
268
+ "<b-109>": 32365,
269
+ "<b-10>": 32266,
270
+ "<b-110>": 32366,
271
+ "<b-111>": 32367,
272
+ "<b-112>": 32368,
273
+ "<b-113>": 32369,
274
+ "<b-114>": 32370,
275
+ "<b-115>": 32371,
276
+ "<b-116>": 32372,
277
+ "<b-117>": 32373,
278
+ "<b-118>": 32374,
279
+ "<b-119>": 32375,
280
+ "<b-11>": 32267,
281
+ "<b-120>": 32376,
282
+ "<b-121>": 32377,
283
+ "<b-122>": 32378,
284
+ "<b-123>": 32379,
285
+ "<b-124>": 32380,
286
+ "<b-125>": 32381,
287
+ "<b-126>": 32382,
288
+ "<b-127>": 32383,
289
+ "<b-128>": 32384,
290
+ "<b-129>": 32385,
291
+ "<b-12>": 32268,
292
+ "<b-130>": 32386,
293
+ "<b-131>": 32387,
294
+ "<b-132>": 32388,
295
+ "<b-133>": 32389,
296
+ "<b-134>": 32390,
297
+ "<b-135>": 32391,
298
+ "<b-136>": 32392,
299
+ "<b-137>": 32393,
300
+ "<b-138>": 32394,
301
+ "<b-139>": 32395,
302
+ "<b-13>": 32269,
303
+ "<b-140>": 32396,
304
+ "<b-141>": 32397,
305
+ "<b-142>": 32398,
306
+ "<b-143>": 32399,
307
+ "<b-144>": 32400,
308
+ "<b-145>": 32401,
309
+ "<b-146>": 32402,
310
+ "<b-147>": 32403,
311
+ "<b-148>": 32404,
312
+ "<b-149>": 32405,
313
+ "<b-14>": 32270,
314
+ "<b-150>": 32406,
315
+ "<b-151>": 32407,
316
+ "<b-152>": 32408,
317
+ "<b-153>": 32409,
318
+ "<b-154>": 32410,
319
+ "<b-155>": 32411,
320
+ "<b-156>": 32412,
321
+ "<b-157>": 32413,
322
+ "<b-158>": 32414,
323
+ "<b-159>": 32415,
324
+ "<b-15>": 32271,
325
+ "<b-160>": 32416,
326
+ "<b-161>": 32417,
327
+ "<b-162>": 32418,
328
+ "<b-163>": 32419,
329
+ "<b-164>": 32420,
330
+ "<b-165>": 32421,
331
+ "<b-166>": 32422,
332
+ "<b-167>": 32423,
333
+ "<b-168>": 32424,
334
+ "<b-169>": 32425,
335
+ "<b-16>": 32272,
336
+ "<b-170>": 32426,
337
+ "<b-171>": 32427,
338
+ "<b-172>": 32428,
339
+ "<b-173>": 32429,
340
+ "<b-174>": 32430,
341
+ "<b-175>": 32431,
342
+ "<b-176>": 32432,
343
+ "<b-177>": 32433,
344
+ "<b-178>": 32434,
345
+ "<b-179>": 32435,
346
+ "<b-17>": 32273,
347
+ "<b-180>": 32436,
348
+ "<b-181>": 32437,
349
+ "<b-182>": 32438,
350
+ "<b-183>": 32439,
351
+ "<b-184>": 32440,
352
+ "<b-185>": 32441,
353
+ "<b-186>": 32442,
354
+ "<b-187>": 32443,
355
+ "<b-188>": 32444,
356
+ "<b-189>": 32445,
357
+ "<b-18>": 32274,
358
+ "<b-190>": 32446,
359
+ "<b-191>": 32447,
360
+ "<b-192>": 32448,
361
+ "<b-193>": 32449,
362
+ "<b-194>": 32450,
363
+ "<b-195>": 32451,
364
+ "<b-196>": 32452,
365
+ "<b-197>": 32453,
366
+ "<b-198>": 32454,
367
+ "<b-199>": 32455,
368
+ "<b-19>": 32275,
369
+ "<b-1>": 32257,
370
+ "<b-200>": 32456,
371
+ "<b-201>": 32457,
372
+ "<b-202>": 32458,
373
+ "<b-203>": 32459,
374
+ "<b-204>": 32460,
375
+ "<b-205>": 32461,
376
+ "<b-206>": 32462,
377
+ "<b-207>": 32463,
378
+ "<b-208>": 32464,
379
+ "<b-209>": 32465,
380
+ "<b-20>": 32276,
381
+ "<b-210>": 32466,
382
+ "<b-211>": 32467,
383
+ "<b-212>": 32468,
384
+ "<b-213>": 32469,
385
+ "<b-214>": 32470,
386
+ "<b-215>": 32471,
387
+ "<b-216>": 32472,
388
+ "<b-217>": 32473,
389
+ "<b-218>": 32474,
390
+ "<b-219>": 32475,
391
+ "<b-21>": 32277,
392
+ "<b-220>": 32476,
393
+ "<b-221>": 32477,
394
+ "<b-222>": 32478,
395
+ "<b-223>": 32479,
396
+ "<b-224>": 32480,
397
+ "<b-225>": 32481,
398
+ "<b-226>": 32482,
399
+ "<b-227>": 32483,
400
+ "<b-228>": 32484,
401
+ "<b-229>": 32485,
402
+ "<b-22>": 32278,
403
+ "<b-230>": 32486,
404
+ "<b-231>": 32487,
405
+ "<b-232>": 32488,
406
+ "<b-233>": 32489,
407
+ "<b-234>": 32490,
408
+ "<b-235>": 32491,
409
+ "<b-236>": 32492,
410
+ "<b-237>": 32493,
411
+ "<b-238>": 32494,
412
+ "<b-239>": 32495,
413
+ "<b-23>": 32279,
414
+ "<b-240>": 32496,
415
+ "<b-241>": 32497,
416
+ "<b-242>": 32498,
417
+ "<b-243>": 32499,
418
+ "<b-244>": 32500,
419
+ "<b-245>": 32501,
420
+ "<b-246>": 32502,
421
+ "<b-247>": 32503,
422
+ "<b-248>": 32504,
423
+ "<b-249>": 32505,
424
+ "<b-24>": 32280,
425
+ "<b-250>": 32506,
426
+ "<b-251>": 32507,
427
+ "<b-252>": 32508,
428
+ "<b-253>": 32509,
429
+ "<b-254>": 32510,
430
+ "<b-255>": 32511,
431
+ "<b-25>": 32281,
432
+ "<b-26>": 32282,
433
+ "<b-27>": 32283,
434
+ "<b-28>": 32284,
435
+ "<b-29>": 32285,
436
+ "<b-2>": 32258,
437
+ "<b-30>": 32286,
438
+ "<b-31>": 32287,
439
+ "<b-32>": 32288,
440
+ "<b-33>": 32289,
441
+ "<b-34>": 32290,
442
+ "<b-35>": 32291,
443
+ "<b-36>": 32292,
444
+ "<b-37>": 32293,
445
+ "<b-38>": 32294,
446
+ "<b-39>": 32295,
447
+ "<b-3>": 32259,
448
+ "<b-40>": 32296,
449
+ "<b-41>": 32297,
450
+ "<b-42>": 32298,
451
+ "<b-43>": 32299,
452
+ "<b-44>": 32300,
453
+ "<b-45>": 32301,
454
+ "<b-46>": 32302,
455
+ "<b-47>": 32303,
456
+ "<b-48>": 32304,
457
+ "<b-49>": 32305,
458
+ "<b-4>": 32260,
459
+ "<b-50>": 32306,
460
+ "<b-51>": 32307,
461
+ "<b-52>": 32308,
462
+ "<b-53>": 32309,
463
+ "<b-54>": 32310,
464
+ "<b-55>": 32311,
465
+ "<b-56>": 32312,
466
+ "<b-57>": 32313,
467
+ "<b-58>": 32314,
468
+ "<b-59>": 32315,
469
+ "<b-5>": 32261,
470
+ "<b-60>": 32316,
471
+ "<b-61>": 32317,
472
+ "<b-62>": 32318,
473
+ "<b-63>": 32319,
474
+ "<b-64>": 32320,
475
+ "<b-65>": 32321,
476
+ "<b-66>": 32322,
477
+ "<b-67>": 32323,
478
+ "<b-68>": 32324,
479
+ "<b-69>": 32325,
480
+ "<b-6>": 32262,
481
+ "<b-70>": 32326,
482
+ "<b-71>": 32327,
483
+ "<b-72>": 32328,
484
+ "<b-73>": 32329,
485
+ "<b-74>": 32330,
486
+ "<b-75>": 32331,
487
+ "<b-76>": 32332,
488
+ "<b-77>": 32333,
489
+ "<b-78>": 32334,
490
+ "<b-79>": 32335,
491
+ "<b-7>": 32263,
492
+ "<b-80>": 32336,
493
+ "<b-81>": 32337,
494
+ "<b-82>": 32338,
495
+ "<b-83>": 32339,
496
+ "<b-84>": 32340,
497
+ "<b-85>": 32341,
498
+ "<b-86>": 32342,
499
+ "<b-87>": 32343,
500
+ "<b-88>": 32344,
501
+ "<b-89>": 32345,
502
+ "<b-8>": 32264,
503
+ "<b-90>": 32346,
504
+ "<b-91>": 32347,
505
+ "<b-92>": 32348,
506
+ "<b-93>": 32349,
507
+ "<b-94>": 32350,
508
+ "<b-95>": 32351,
509
+ "<b-96>": 32352,
510
+ "<b-97>": 32353,
511
+ "<b-98>": 32354,
512
+ "<b-99>": 32355,
513
+ "<b-9>": 32265,
514
+ "<c-0>": 32512,
515
+ "<c-100>": 32612,
516
+ "<c-101>": 32613,
517
+ "<c-102>": 32614,
518
+ "<c-103>": 32615,
519
+ "<c-104>": 32616,
520
+ "<c-105>": 32617,
521
+ "<c-106>": 32618,
522
+ "<c-107>": 32619,
523
+ "<c-108>": 32620,
524
+ "<c-109>": 32621,
525
+ "<c-10>": 32522,
526
+ "<c-110>": 32622,
527
+ "<c-111>": 32623,
528
+ "<c-112>": 32624,
529
+ "<c-113>": 32625,
530
+ "<c-114>": 32626,
531
+ "<c-115>": 32627,
532
+ "<c-116>": 32628,
533
+ "<c-117>": 32629,
534
+ "<c-118>": 32630,
535
+ "<c-119>": 32631,
536
+ "<c-11>": 32523,
537
+ "<c-120>": 32632,
538
+ "<c-121>": 32633,
539
+ "<c-122>": 32634,
540
+ "<c-123>": 32635,
541
+ "<c-124>": 32636,
542
+ "<c-125>": 32637,
543
+ "<c-126>": 32638,
544
+ "<c-127>": 32639,
545
+ "<c-128>": 32640,
546
+ "<c-129>": 32641,
547
+ "<c-12>": 32524,
548
+ "<c-130>": 32642,
549
+ "<c-131>": 32643,
550
+ "<c-132>": 32644,
551
+ "<c-133>": 32645,
552
+ "<c-134>": 32646,
553
+ "<c-135>": 32647,
554
+ "<c-136>": 32648,
555
+ "<c-137>": 32649,
556
+ "<c-138>": 32650,
557
+ "<c-139>": 32651,
558
+ "<c-13>": 32525,
559
+ "<c-140>": 32652,
560
+ "<c-141>": 32653,
561
+ "<c-142>": 32654,
562
+ "<c-143>": 32655,
563
+ "<c-144>": 32656,
564
+ "<c-145>": 32657,
565
+ "<c-146>": 32658,
566
+ "<c-147>": 32659,
567
+ "<c-148>": 32660,
568
+ "<c-149>": 32661,
569
+ "<c-14>": 32526,
570
+ "<c-150>": 32662,
571
+ "<c-151>": 32663,
572
+ "<c-152>": 32664,
573
+ "<c-153>": 32665,
574
+ "<c-154>": 32666,
575
+ "<c-155>": 32667,
576
+ "<c-156>": 32668,
577
+ "<c-157>": 32669,
578
+ "<c-158>": 32670,
579
+ "<c-159>": 32671,
580
+ "<c-15>": 32527,
581
+ "<c-160>": 32672,
582
+ "<c-161>": 32673,
583
+ "<c-162>": 32674,
584
+ "<c-163>": 32675,
585
+ "<c-164>": 32676,
586
+ "<c-165>": 32677,
587
+ "<c-166>": 32678,
588
+ "<c-167>": 32679,
589
+ "<c-168>": 32680,
590
+ "<c-169>": 32681,
591
+ "<c-16>": 32528,
592
+ "<c-170>": 32682,
593
+ "<c-171>": 32683,
594
+ "<c-172>": 32684,
595
+ "<c-173>": 32685,
596
+ "<c-174>": 32686,
597
+ "<c-175>": 32687,
598
+ "<c-176>": 32688,
599
+ "<c-177>": 32689,
600
+ "<c-178>": 32690,
601
+ "<c-179>": 32691,
602
+ "<c-17>": 32529,
603
+ "<c-180>": 32692,
604
+ "<c-181>": 32693,
605
+ "<c-182>": 32694,
606
+ "<c-183>": 32695,
607
+ "<c-184>": 32696,
608
+ "<c-185>": 32697,
609
+ "<c-186>": 32698,
610
+ "<c-187>": 32699,
611
+ "<c-188>": 32700,
612
+ "<c-189>": 32701,
613
+ "<c-18>": 32530,
614
+ "<c-190>": 32702,
615
+ "<c-191>": 32703,
616
+ "<c-192>": 32704,
617
+ "<c-193>": 32705,
618
+ "<c-194>": 32706,
619
+ "<c-195>": 32707,
620
+ "<c-196>": 32708,
621
+ "<c-197>": 32709,
622
+ "<c-198>": 32710,
623
+ "<c-199>": 32711,
624
+ "<c-19>": 32531,
625
+ "<c-1>": 32513,
626
+ "<c-200>": 32712,
627
+ "<c-201>": 32713,
628
+ "<c-202>": 32714,
629
+ "<c-203>": 32715,
630
+ "<c-204>": 32716,
631
+ "<c-205>": 32717,
632
+ "<c-206>": 32718,
633
+ "<c-207>": 32719,
634
+ "<c-208>": 32720,
635
+ "<c-209>": 32721,
636
+ "<c-20>": 32532,
637
+ "<c-210>": 32722,
638
+ "<c-211>": 32723,
639
+ "<c-212>": 32724,
640
+ "<c-213>": 32725,
641
+ "<c-214>": 32726,
642
+ "<c-215>": 32727,
643
+ "<c-216>": 32728,
644
+ "<c-217>": 32729,
645
+ "<c-218>": 32730,
646
+ "<c-219>": 32731,
647
+ "<c-21>": 32533,
648
+ "<c-220>": 32732,
649
+ "<c-221>": 32733,
650
+ "<c-222>": 32734,
651
+ "<c-223>": 32735,
652
+ "<c-224>": 32736,
653
+ "<c-225>": 32737,
654
+ "<c-226>": 32738,
655
+ "<c-227>": 32739,
656
+ "<c-228>": 32740,
657
+ "<c-229>": 32741,
658
+ "<c-22>": 32534,
659
+ "<c-230>": 32742,
660
+ "<c-231>": 32743,
661
+ "<c-232>": 32744,
662
+ "<c-233>": 32745,
663
+ "<c-234>": 32746,
664
+ "<c-235>": 32747,
665
+ "<c-236>": 32748,
666
+ "<c-237>": 32749,
667
+ "<c-238>": 32750,
668
+ "<c-239>": 32751,
669
+ "<c-23>": 32535,
670
+ "<c-240>": 32752,
671
+ "<c-241>": 32753,
672
+ "<c-242>": 32754,
673
+ "<c-243>": 32755,
674
+ "<c-244>": 32756,
675
+ "<c-245>": 32757,
676
+ "<c-246>": 32758,
677
+ "<c-247>": 32759,
678
+ "<c-248>": 32760,
679
+ "<c-249>": 32761,
680
+ "<c-24>": 32536,
681
+ "<c-250>": 32762,
682
+ "<c-251>": 32763,
683
+ "<c-252>": 32764,
684
+ "<c-253>": 32765,
685
+ "<c-254>": 32766,
686
+ "<c-255>": 32767,
687
+ "<c-25>": 32537,
688
+ "<c-26>": 32538,
689
+ "<c-27>": 32539,
690
+ "<c-28>": 32540,
691
+ "<c-29>": 32541,
692
+ "<c-2>": 32514,
693
+ "<c-30>": 32542,
694
+ "<c-31>": 32543,
695
+ "<c-32>": 32544,
696
+ "<c-33>": 32545,
697
+ "<c-34>": 32546,
698
+ "<c-35>": 32547,
699
+ "<c-36>": 32548,
700
+ "<c-37>": 32549,
701
+ "<c-38>": 32550,
702
+ "<c-39>": 32551,
703
+ "<c-3>": 32515,
704
+ "<c-40>": 32552,
705
+ "<c-41>": 32553,
706
+ "<c-42>": 32554,
707
+ "<c-43>": 32555,
708
+ "<c-44>": 32556,
709
+ "<c-45>": 32557,
710
+ "<c-46>": 32558,
711
+ "<c-47>": 32559,
712
+ "<c-48>": 32560,
713
+ "<c-49>": 32561,
714
+ "<c-4>": 32516,
715
+ "<c-50>": 32562,
716
+ "<c-51>": 32563,
717
+ "<c-52>": 32564,
718
+ "<c-53>": 32565,
719
+ "<c-54>": 32566,
720
+ "<c-55>": 32567,
721
+ "<c-56>": 32568,
722
+ "<c-57>": 32569,
723
+ "<c-58>": 32570,
724
+ "<c-59>": 32571,
725
+ "<c-5>": 32517,
726
+ "<c-60>": 32572,
727
+ "<c-61>": 32573,
728
+ "<c-62>": 32574,
729
+ "<c-63>": 32575,
730
+ "<c-64>": 32576,
731
+ "<c-65>": 32577,
732
+ "<c-66>": 32578,
733
+ "<c-67>": 32579,
734
+ "<c-68>": 32580,
735
+ "<c-69>": 32581,
736
+ "<c-6>": 32518,
737
+ "<c-70>": 32582,
738
+ "<c-71>": 32583,
739
+ "<c-72>": 32584,
740
+ "<c-73>": 32585,
741
+ "<c-74>": 32586,
742
+ "<c-75>": 32587,
743
+ "<c-76>": 32588,
744
+ "<c-77>": 32589,
745
+ "<c-78>": 32590,
746
+ "<c-79>": 32591,
747
+ "<c-7>": 32519,
748
+ "<c-80>": 32592,
749
+ "<c-81>": 32593,
750
+ "<c-82>": 32594,
751
+ "<c-83>": 32595,
752
+ "<c-84>": 32596,
753
+ "<c-85>": 32597,
754
+ "<c-86>": 32598,
755
+ "<c-87>": 32599,
756
+ "<c-88>": 32600,
757
+ "<c-89>": 32601,
758
+ "<c-8>": 32520,
759
+ "<c-90>": 32602,
760
+ "<c-91>": 32603,
761
+ "<c-92>": 32604,
762
+ "<c-93>": 32605,
763
+ "<c-94>": 32606,
764
+ "<c-95>": 32607,
765
+ "<c-96>": 32608,
766
+ "<c-97>": 32609,
767
+ "<c-98>": 32610,
768
+ "<c-99>": 32611,
769
+ "<c-9>": 32521,
770
+ "<d-0>": 32768,
771
+ "<d-100>": 32868,
772
+ "<d-101>": 32869,
773
+ "<d-102>": 32870,
774
+ "<d-103>": 32871,
775
+ "<d-104>": 32872,
776
+ "<d-105>": 32873,
777
+ "<d-106>": 32874,
778
+ "<d-107>": 32875,
779
+ "<d-108>": 32876,
780
+ "<d-109>": 32877,
781
+ "<d-10>": 32778,
782
+ "<d-110>": 32878,
783
+ "<d-111>": 32879,
784
+ "<d-112>": 32880,
785
+ "<d-113>": 32881,
786
+ "<d-114>": 32882,
787
+ "<d-115>": 32883,
788
+ "<d-116>": 32884,
789
+ "<d-117>": 32885,
790
+ "<d-118>": 32886,
791
+ "<d-119>": 32887,
792
+ "<d-11>": 32779,
793
+ "<d-120>": 32888,
794
+ "<d-121>": 32889,
795
+ "<d-122>": 32890,
796
+ "<d-123>": 32891,
797
+ "<d-124>": 32892,
798
+ "<d-125>": 32893,
799
+ "<d-126>": 32894,
800
+ "<d-127>": 32895,
801
+ "<d-128>": 32896,
802
+ "<d-129>": 32897,
803
+ "<d-12>": 32780,
804
+ "<d-130>": 32898,
805
+ "<d-131>": 32899,
806
+ "<d-132>": 32900,
807
+ "<d-133>": 32901,
808
+ "<d-134>": 32902,
809
+ "<d-135>": 32903,
810
+ "<d-136>": 32904,
811
+ "<d-137>": 32905,
812
+ "<d-138>": 32906,
813
+ "<d-139>": 32907,
814
+ "<d-13>": 32781,
815
+ "<d-140>": 32908,
816
+ "<d-141>": 32909,
817
+ "<d-142>": 32910,
818
+ "<d-143>": 32911,
819
+ "<d-144>": 32912,
820
+ "<d-145>": 32913,
821
+ "<d-146>": 32914,
822
+ "<d-147>": 32915,
823
+ "<d-148>": 32916,
824
+ "<d-149>": 32917,
825
+ "<d-14>": 32782,
826
+ "<d-150>": 32918,
827
+ "<d-151>": 32919,
828
+ "<d-152>": 32920,
829
+ "<d-153>": 32921,
830
+ "<d-154>": 32922,
831
+ "<d-155>": 32923,
832
+ "<d-156>": 32924,
833
+ "<d-157>": 32925,
834
+ "<d-158>": 32926,
835
+ "<d-159>": 32927,
836
+ "<d-15>": 32783,
837
+ "<d-160>": 32928,
838
+ "<d-161>": 32929,
839
+ "<d-162>": 32930,
840
+ "<d-163>": 32931,
841
+ "<d-164>": 32932,
842
+ "<d-165>": 32933,
843
+ "<d-166>": 32934,
844
+ "<d-167>": 32935,
845
+ "<d-168>": 32936,
846
+ "<d-169>": 32937,
847
+ "<d-16>": 32784,
848
+ "<d-170>": 32938,
849
+ "<d-171>": 32939,
850
+ "<d-172>": 32940,
851
+ "<d-173>": 32941,
852
+ "<d-174>": 32942,
853
+ "<d-175>": 32943,
854
+ "<d-176>": 32944,
855
+ "<d-177>": 32945,
856
+ "<d-178>": 32946,
857
+ "<d-179>": 32947,
858
+ "<d-17>": 32785,
859
+ "<d-180>": 32948,
860
+ "<d-181>": 32949,
861
+ "<d-182>": 32950,
862
+ "<d-183>": 32951,
863
+ "<d-184>": 32952,
864
+ "<d-185>": 32953,
865
+ "<d-186>": 32954,
866
+ "<d-187>": 32955,
867
+ "<d-188>": 32956,
868
+ "<d-189>": 32957,
869
+ "<d-18>": 32786,
870
+ "<d-190>": 32958,
871
+ "<d-191>": 32959,
872
+ "<d-192>": 32960,
873
+ "<d-193>": 32961,
874
+ "<d-194>": 32962,
875
+ "<d-195>": 32963,
876
+ "<d-196>": 32964,
877
+ "<d-197>": 32965,
878
+ "<d-198>": 32966,
879
+ "<d-199>": 32967,
880
+ "<d-19>": 32787,
881
+ "<d-1>": 32769,
882
+ "<d-200>": 32968,
883
+ "<d-201>": 32969,
884
+ "<d-202>": 32970,
885
+ "<d-203>": 32971,
886
+ "<d-204>": 32972,
887
+ "<d-205>": 32973,
888
+ "<d-206>": 32974,
889
+ "<d-207>": 32975,
890
+ "<d-208>": 32976,
891
+ "<d-209>": 32977,
892
+ "<d-20>": 32788,
893
+ "<d-210>": 32978,
894
+ "<d-211>": 32979,
895
+ "<d-212>": 32980,
896
+ "<d-213>": 32981,
897
+ "<d-214>": 32982,
898
+ "<d-215>": 32983,
899
+ "<d-216>": 32984,
900
+ "<d-217>": 32985,
901
+ "<d-218>": 32986,
902
+ "<d-219>": 32987,
903
+ "<d-21>": 32789,
904
+ "<d-220>": 32988,
905
+ "<d-221>": 32989,
906
+ "<d-222>": 32990,
907
+ "<d-223>": 32991,
908
+ "<d-224>": 32992,
909
+ "<d-225>": 32993,
910
+ "<d-226>": 32994,
911
+ "<d-227>": 32995,
912
+ "<d-228>": 32996,
913
+ "<d-229>": 32997,
914
+ "<d-22>": 32790,
915
+ "<d-230>": 32998,
916
+ "<d-231>": 32999,
917
+ "<d-232>": 33000,
918
+ "<d-233>": 33001,
919
+ "<d-234>": 33002,
920
+ "<d-235>": 33003,
921
+ "<d-236>": 33004,
922
+ "<d-237>": 33005,
923
+ "<d-238>": 33006,
924
+ "<d-239>": 33007,
925
+ "<d-23>": 32791,
926
+ "<d-240>": 33008,
927
+ "<d-241>": 33009,
928
+ "<d-242>": 33010,
929
+ "<d-243>": 33011,
930
+ "<d-244>": 33012,
931
+ "<d-245>": 33013,
932
+ "<d-246>": 33014,
933
+ "<d-247>": 33015,
934
+ "<d-248>": 33016,
935
+ "<d-249>": 33017,
936
+ "<d-24>": 32792,
937
+ "<d-250>": 33018,
938
+ "<d-251>": 33019,
939
+ "<d-252>": 33020,
940
+ "<d-253>": 33021,
941
+ "<d-254>": 33022,
942
+ "<d-255>": 33023,
943
+ "<d-25>": 32793,
944
+ "<d-26>": 32794,
945
+ "<d-27>": 32795,
946
+ "<d-28>": 32796,
947
+ "<d-29>": 32797,
948
+ "<d-2>": 32770,
949
+ "<d-30>": 32798,
950
+ "<d-31>": 32799,
951
+ "<d-32>": 32800,
952
+ "<d-33>": 32801,
953
+ "<d-34>": 32802,
954
+ "<d-35>": 32803,
955
+ "<d-36>": 32804,
956
+ "<d-37>": 32805,
957
+ "<d-38>": 32806,
958
+ "<d-39>": 32807,
959
+ "<d-3>": 32771,
960
+ "<d-40>": 32808,
961
+ "<d-41>": 32809,
962
+ "<d-42>": 32810,
963
+ "<d-43>": 32811,
964
+ "<d-44>": 32812,
965
+ "<d-45>": 32813,
966
+ "<d-46>": 32814,
967
+ "<d-47>": 32815,
968
+ "<d-48>": 32816,
969
+ "<d-49>": 32817,
970
+ "<d-4>": 32772,
971
+ "<d-50>": 32818,
972
+ "<d-51>": 32819,
973
+ "<d-52>": 32820,
974
+ "<d-53>": 32821,
975
+ "<d-54>": 32822,
976
+ "<d-55>": 32823,
977
+ "<d-56>": 32824,
978
+ "<d-57>": 32825,
979
+ "<d-58>": 32826,
980
+ "<d-59>": 32827,
981
+ "<d-5>": 32773,
982
+ "<d-60>": 32828,
983
+ "<d-61>": 32829,
984
+ "<d-62>": 32830,
985
+ "<d-63>": 32831,
986
+ "<d-64>": 32832,
987
+ "<d-65>": 32833,
988
+ "<d-66>": 32834,
989
+ "<d-67>": 32835,
990
+ "<d-68>": 32836,
991
+ "<d-69>": 32837,
992
+ "<d-6>": 32774,
993
+ "<d-70>": 32838,
994
+ "<d-71>": 32839,
995
+ "<d-72>": 32840,
996
+ "<d-73>": 32841,
997
+ "<d-74>": 32842,
998
+ "<d-75>": 32843,
999
+ "<d-76>": 32844,
1000
+ "<d-77>": 32845,
1001
+ "<d-78>": 32846,
1002
+ "<d-79>": 32847,
1003
+ "<d-7>": 32775,
1004
+ "<d-80>": 32848,
1005
+ "<d-81>": 32849,
1006
+ "<d-82>": 32850,
1007
+ "<d-83>": 32851,
1008
+ "<d-84>": 32852,
1009
+ "<d-85>": 32853,
1010
+ "<d-86>": 32854,
1011
+ "<d-87>": 32855,
1012
+ "<d-88>": 32856,
1013
+ "<d-89>": 32857,
1014
+ "<d-8>": 32776,
1015
+ "<d-90>": 32858,
1016
+ "<d-91>": 32859,
1017
+ "<d-92>": 32860,
1018
+ "<d-93>": 32861,
1019
+ "<d-94>": 32862,
1020
+ "<d-95>": 32863,
1021
+ "<d-96>": 32864,
1022
+ "<d-97>": 32865,
1023
+ "<d-98>": 32866,
1024
+ "<d-99>": 32867,
1025
+ "<d-9>": 32777
1026
+ }
Ins/checkpoint-9678/latest ADDED
@@ -0,0 +1 @@
 
 
1
+ global_step9678
Ins/checkpoint-9678/model.safetensors.index.json ADDED
@@ -0,0 +1,780 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "metadata": {
3
+ "total_size": 14119477056
4
+ },
5
+ "weight_map": {
6
+ "model.base_model.model.lm_head.modules_to_save.default.weight": "model-00003-of-00003.safetensors",
7
+ "model.base_model.model.lm_head.original_module.weight": "model-00003-of-00003.safetensors",
8
+ "model.base_model.model.model.embed_tokens.modules_to_save.default.weight": "model-00001-of-00003.safetensors",
9
+ "model.base_model.model.model.embed_tokens.original_module.weight": "model-00001-of-00003.safetensors",
10
+ "model.base_model.model.model.layers.0.input_layernorm.weight": "model-00001-of-00003.safetensors",
11
+ "model.base_model.model.model.layers.0.mlp.down_proj.base_layer.weight": "model-00001-of-00003.safetensors",
12
+ "model.base_model.model.model.layers.0.mlp.down_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
13
+ "model.base_model.model.model.layers.0.mlp.down_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
14
+ "model.base_model.model.model.layers.0.mlp.gate_proj.base_layer.weight": "model-00001-of-00003.safetensors",
15
+ "model.base_model.model.model.layers.0.mlp.gate_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
16
+ "model.base_model.model.model.layers.0.mlp.gate_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
17
+ "model.base_model.model.model.layers.0.mlp.up_proj.base_layer.weight": "model-00001-of-00003.safetensors",
18
+ "model.base_model.model.model.layers.0.mlp.up_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
19
+ "model.base_model.model.model.layers.0.mlp.up_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
20
+ "model.base_model.model.model.layers.0.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
21
+ "model.base_model.model.model.layers.0.self_attn.k_proj.base_layer.weight": "model-00001-of-00003.safetensors",
22
+ "model.base_model.model.model.layers.0.self_attn.k_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
23
+ "model.base_model.model.model.layers.0.self_attn.k_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
24
+ "model.base_model.model.model.layers.0.self_attn.o_proj.base_layer.weight": "model-00001-of-00003.safetensors",
25
+ "model.base_model.model.model.layers.0.self_attn.o_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
26
+ "model.base_model.model.model.layers.0.self_attn.o_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
27
+ "model.base_model.model.model.layers.0.self_attn.q_proj.base_layer.weight": "model-00001-of-00003.safetensors",
28
+ "model.base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
29
+ "model.base_model.model.model.layers.0.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
30
+ "model.base_model.model.model.layers.0.self_attn.v_proj.base_layer.weight": "model-00001-of-00003.safetensors",
31
+ "model.base_model.model.model.layers.0.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
32
+ "model.base_model.model.model.layers.0.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
33
+ "model.base_model.model.model.layers.1.input_layernorm.weight": "model-00001-of-00003.safetensors",
34
+ "model.base_model.model.model.layers.1.mlp.down_proj.base_layer.weight": "model-00001-of-00003.safetensors",
35
+ "model.base_model.model.model.layers.1.mlp.down_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
36
+ "model.base_model.model.model.layers.1.mlp.down_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
37
+ "model.base_model.model.model.layers.1.mlp.gate_proj.base_layer.weight": "model-00001-of-00003.safetensors",
38
+ "model.base_model.model.model.layers.1.mlp.gate_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
39
+ "model.base_model.model.model.layers.1.mlp.gate_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
40
+ "model.base_model.model.model.layers.1.mlp.up_proj.base_layer.weight": "model-00001-of-00003.safetensors",
41
+ "model.base_model.model.model.layers.1.mlp.up_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
42
+ "model.base_model.model.model.layers.1.mlp.up_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
43
+ "model.base_model.model.model.layers.1.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
44
+ "model.base_model.model.model.layers.1.self_attn.k_proj.base_layer.weight": "model-00001-of-00003.safetensors",
45
+ "model.base_model.model.model.layers.1.self_attn.k_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
46
+ "model.base_model.model.model.layers.1.self_attn.k_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
47
+ "model.base_model.model.model.layers.1.self_attn.o_proj.base_layer.weight": "model-00001-of-00003.safetensors",
48
+ "model.base_model.model.model.layers.1.self_attn.o_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
49
+ "model.base_model.model.model.layers.1.self_attn.o_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
50
+ "model.base_model.model.model.layers.1.self_attn.q_proj.base_layer.weight": "model-00001-of-00003.safetensors",
51
+ "model.base_model.model.model.layers.1.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
52
+ "model.base_model.model.model.layers.1.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
53
+ "model.base_model.model.model.layers.1.self_attn.v_proj.base_layer.weight": "model-00001-of-00003.safetensors",
54
+ "model.base_model.model.model.layers.1.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
55
+ "model.base_model.model.model.layers.1.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
56
+ "model.base_model.model.model.layers.10.input_layernorm.weight": "model-00002-of-00003.safetensors",
57
+ "model.base_model.model.model.layers.10.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
58
+ "model.base_model.model.model.layers.10.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
59
+ "model.base_model.model.model.layers.10.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
60
+ "model.base_model.model.model.layers.10.mlp.gate_proj.base_layer.weight": "model-00001-of-00003.safetensors",
61
+ "model.base_model.model.model.layers.10.mlp.gate_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
62
+ "model.base_model.model.model.layers.10.mlp.gate_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
63
+ "model.base_model.model.model.layers.10.mlp.up_proj.base_layer.weight": "model-00001-of-00003.safetensors",
64
+ "model.base_model.model.model.layers.10.mlp.up_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
65
+ "model.base_model.model.model.layers.10.mlp.up_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
66
+ "model.base_model.model.model.layers.10.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
67
+ "model.base_model.model.model.layers.10.self_attn.k_proj.base_layer.weight": "model-00001-of-00003.safetensors",
68
+ "model.base_model.model.model.layers.10.self_attn.k_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
69
+ "model.base_model.model.model.layers.10.self_attn.k_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
70
+ "model.base_model.model.model.layers.10.self_attn.o_proj.base_layer.weight": "model-00001-of-00003.safetensors",
71
+ "model.base_model.model.model.layers.10.self_attn.o_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
72
+ "model.base_model.model.model.layers.10.self_attn.o_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
73
+ "model.base_model.model.model.layers.10.self_attn.q_proj.base_layer.weight": "model-00001-of-00003.safetensors",
74
+ "model.base_model.model.model.layers.10.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
75
+ "model.base_model.model.model.layers.10.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
76
+ "model.base_model.model.model.layers.10.self_attn.v_proj.base_layer.weight": "model-00001-of-00003.safetensors",
77
+ "model.base_model.model.model.layers.10.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
78
+ "model.base_model.model.model.layers.10.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
79
+ "model.base_model.model.model.layers.11.input_layernorm.weight": "model-00002-of-00003.safetensors",
80
+ "model.base_model.model.model.layers.11.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
81
+ "model.base_model.model.model.layers.11.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
82
+ "model.base_model.model.model.layers.11.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
83
+ "model.base_model.model.model.layers.11.mlp.gate_proj.base_layer.weight": "model-00002-of-00003.safetensors",
84
+ "model.base_model.model.model.layers.11.mlp.gate_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
85
+ "model.base_model.model.model.layers.11.mlp.gate_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
86
+ "model.base_model.model.model.layers.11.mlp.up_proj.base_layer.weight": "model-00002-of-00003.safetensors",
87
+ "model.base_model.model.model.layers.11.mlp.up_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
88
+ "model.base_model.model.model.layers.11.mlp.up_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
89
+ "model.base_model.model.model.layers.11.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
90
+ "model.base_model.model.model.layers.11.self_attn.k_proj.base_layer.weight": "model-00002-of-00003.safetensors",
91
+ "model.base_model.model.model.layers.11.self_attn.k_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
92
+ "model.base_model.model.model.layers.11.self_attn.k_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
93
+ "model.base_model.model.model.layers.11.self_attn.o_proj.base_layer.weight": "model-00002-of-00003.safetensors",
94
+ "model.base_model.model.model.layers.11.self_attn.o_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
95
+ "model.base_model.model.model.layers.11.self_attn.o_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
96
+ "model.base_model.model.model.layers.11.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
97
+ "model.base_model.model.model.layers.11.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
98
+ "model.base_model.model.model.layers.11.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
99
+ "model.base_model.model.model.layers.11.self_attn.v_proj.base_layer.weight": "model-00002-of-00003.safetensors",
100
+ "model.base_model.model.model.layers.11.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
101
+ "model.base_model.model.model.layers.11.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
102
+ "model.base_model.model.model.layers.12.input_layernorm.weight": "model-00002-of-00003.safetensors",
103
+ "model.base_model.model.model.layers.12.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
104
+ "model.base_model.model.model.layers.12.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
105
+ "model.base_model.model.model.layers.12.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
106
+ "model.base_model.model.model.layers.12.mlp.gate_proj.base_layer.weight": "model-00002-of-00003.safetensors",
107
+ "model.base_model.model.model.layers.12.mlp.gate_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
108
+ "model.base_model.model.model.layers.12.mlp.gate_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
109
+ "model.base_model.model.model.layers.12.mlp.up_proj.base_layer.weight": "model-00002-of-00003.safetensors",
110
+ "model.base_model.model.model.layers.12.mlp.up_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
111
+ "model.base_model.model.model.layers.12.mlp.up_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
112
+ "model.base_model.model.model.layers.12.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
113
+ "model.base_model.model.model.layers.12.self_attn.k_proj.base_layer.weight": "model-00002-of-00003.safetensors",
114
+ "model.base_model.model.model.layers.12.self_attn.k_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
115
+ "model.base_model.model.model.layers.12.self_attn.k_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
116
+ "model.base_model.model.model.layers.12.self_attn.o_proj.base_layer.weight": "model-00002-of-00003.safetensors",
117
+ "model.base_model.model.model.layers.12.self_attn.o_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
118
+ "model.base_model.model.model.layers.12.self_attn.o_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
119
+ "model.base_model.model.model.layers.12.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
120
+ "model.base_model.model.model.layers.12.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
121
+ "model.base_model.model.model.layers.12.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
122
+ "model.base_model.model.model.layers.12.self_attn.v_proj.base_layer.weight": "model-00002-of-00003.safetensors",
123
+ "model.base_model.model.model.layers.12.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
124
+ "model.base_model.model.model.layers.12.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
125
+ "model.base_model.model.model.layers.13.input_layernorm.weight": "model-00002-of-00003.safetensors",
126
+ "model.base_model.model.model.layers.13.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
127
+ "model.base_model.model.model.layers.13.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
128
+ "model.base_model.model.model.layers.13.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
129
+ "model.base_model.model.model.layers.13.mlp.gate_proj.base_layer.weight": "model-00002-of-00003.safetensors",
130
+ "model.base_model.model.model.layers.13.mlp.gate_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
131
+ "model.base_model.model.model.layers.13.mlp.gate_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
132
+ "model.base_model.model.model.layers.13.mlp.up_proj.base_layer.weight": "model-00002-of-00003.safetensors",
133
+ "model.base_model.model.model.layers.13.mlp.up_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
134
+ "model.base_model.model.model.layers.13.mlp.up_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
135
+ "model.base_model.model.model.layers.13.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
136
+ "model.base_model.model.model.layers.13.self_attn.k_proj.base_layer.weight": "model-00002-of-00003.safetensors",
137
+ "model.base_model.model.model.layers.13.self_attn.k_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
138
+ "model.base_model.model.model.layers.13.self_attn.k_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
139
+ "model.base_model.model.model.layers.13.self_attn.o_proj.base_layer.weight": "model-00002-of-00003.safetensors",
140
+ "model.base_model.model.model.layers.13.self_attn.o_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
141
+ "model.base_model.model.model.layers.13.self_attn.o_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
142
+ "model.base_model.model.model.layers.13.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
143
+ "model.base_model.model.model.layers.13.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
144
+ "model.base_model.model.model.layers.13.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
145
+ "model.base_model.model.model.layers.13.self_attn.v_proj.base_layer.weight": "model-00002-of-00003.safetensors",
146
+ "model.base_model.model.model.layers.13.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
147
+ "model.base_model.model.model.layers.13.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
148
+ "model.base_model.model.model.layers.14.input_layernorm.weight": "model-00002-of-00003.safetensors",
149
+ "model.base_model.model.model.layers.14.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
150
+ "model.base_model.model.model.layers.14.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
151
+ "model.base_model.model.model.layers.14.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
152
+ "model.base_model.model.model.layers.14.mlp.gate_proj.base_layer.weight": "model-00002-of-00003.safetensors",
153
+ "model.base_model.model.model.layers.14.mlp.gate_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
154
+ "model.base_model.model.model.layers.14.mlp.gate_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
155
+ "model.base_model.model.model.layers.14.mlp.up_proj.base_layer.weight": "model-00002-of-00003.safetensors",
156
+ "model.base_model.model.model.layers.14.mlp.up_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
157
+ "model.base_model.model.model.layers.14.mlp.up_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
158
+ "model.base_model.model.model.layers.14.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
159
+ "model.base_model.model.model.layers.14.self_attn.k_proj.base_layer.weight": "model-00002-of-00003.safetensors",
160
+ "model.base_model.model.model.layers.14.self_attn.k_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
161
+ "model.base_model.model.model.layers.14.self_attn.k_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
162
+ "model.base_model.model.model.layers.14.self_attn.o_proj.base_layer.weight": "model-00002-of-00003.safetensors",
163
+ "model.base_model.model.model.layers.14.self_attn.o_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
164
+ "model.base_model.model.model.layers.14.self_attn.o_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
165
+ "model.base_model.model.model.layers.14.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
166
+ "model.base_model.model.model.layers.14.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
167
+ "model.base_model.model.model.layers.14.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
168
+ "model.base_model.model.model.layers.14.self_attn.v_proj.base_layer.weight": "model-00002-of-00003.safetensors",
169
+ "model.base_model.model.model.layers.14.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
170
+ "model.base_model.model.model.layers.14.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
171
+ "model.base_model.model.model.layers.15.input_layernorm.weight": "model-00002-of-00003.safetensors",
172
+ "model.base_model.model.model.layers.15.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
173
+ "model.base_model.model.model.layers.15.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
174
+ "model.base_model.model.model.layers.15.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
175
+ "model.base_model.model.model.layers.15.mlp.gate_proj.base_layer.weight": "model-00002-of-00003.safetensors",
176
+ "model.base_model.model.model.layers.15.mlp.gate_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
177
+ "model.base_model.model.model.layers.15.mlp.gate_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
178
+ "model.base_model.model.model.layers.15.mlp.up_proj.base_layer.weight": "model-00002-of-00003.safetensors",
179
+ "model.base_model.model.model.layers.15.mlp.up_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
180
+ "model.base_model.model.model.layers.15.mlp.up_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
181
+ "model.base_model.model.model.layers.15.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
182
+ "model.base_model.model.model.layers.15.self_attn.k_proj.base_layer.weight": "model-00002-of-00003.safetensors",
183
+ "model.base_model.model.model.layers.15.self_attn.k_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
184
+ "model.base_model.model.model.layers.15.self_attn.k_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
185
+ "model.base_model.model.model.layers.15.self_attn.o_proj.base_layer.weight": "model-00002-of-00003.safetensors",
186
+ "model.base_model.model.model.layers.15.self_attn.o_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
187
+ "model.base_model.model.model.layers.15.self_attn.o_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
188
+ "model.base_model.model.model.layers.15.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
189
+ "model.base_model.model.model.layers.15.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
190
+ "model.base_model.model.model.layers.15.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
191
+ "model.base_model.model.model.layers.15.self_attn.v_proj.base_layer.weight": "model-00002-of-00003.safetensors",
192
+ "model.base_model.model.model.layers.15.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
193
+ "model.base_model.model.model.layers.15.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
194
+ "model.base_model.model.model.layers.16.input_layernorm.weight": "model-00002-of-00003.safetensors",
195
+ "model.base_model.model.model.layers.16.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
196
+ "model.base_model.model.model.layers.16.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
197
+ "model.base_model.model.model.layers.16.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
198
+ "model.base_model.model.model.layers.16.mlp.gate_proj.base_layer.weight": "model-00002-of-00003.safetensors",
199
+ "model.base_model.model.model.layers.16.mlp.gate_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
200
+ "model.base_model.model.model.layers.16.mlp.gate_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
201
+ "model.base_model.model.model.layers.16.mlp.up_proj.base_layer.weight": "model-00002-of-00003.safetensors",
202
+ "model.base_model.model.model.layers.16.mlp.up_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
203
+ "model.base_model.model.model.layers.16.mlp.up_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
204
+ "model.base_model.model.model.layers.16.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
205
+ "model.base_model.model.model.layers.16.self_attn.k_proj.base_layer.weight": "model-00002-of-00003.safetensors",
206
+ "model.base_model.model.model.layers.16.self_attn.k_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
207
+ "model.base_model.model.model.layers.16.self_attn.k_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
208
+ "model.base_model.model.model.layers.16.self_attn.o_proj.base_layer.weight": "model-00002-of-00003.safetensors",
209
+ "model.base_model.model.model.layers.16.self_attn.o_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
210
+ "model.base_model.model.model.layers.16.self_attn.o_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
211
+ "model.base_model.model.model.layers.16.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
212
+ "model.base_model.model.model.layers.16.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
213
+ "model.base_model.model.model.layers.16.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
214
+ "model.base_model.model.model.layers.16.self_attn.v_proj.base_layer.weight": "model-00002-of-00003.safetensors",
215
+ "model.base_model.model.model.layers.16.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
216
+ "model.base_model.model.model.layers.16.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
217
+ "model.base_model.model.model.layers.17.input_layernorm.weight": "model-00002-of-00003.safetensors",
218
+ "model.base_model.model.model.layers.17.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
219
+ "model.base_model.model.model.layers.17.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
220
+ "model.base_model.model.model.layers.17.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
221
+ "model.base_model.model.model.layers.17.mlp.gate_proj.base_layer.weight": "model-00002-of-00003.safetensors",
222
+ "model.base_model.model.model.layers.17.mlp.gate_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
223
+ "model.base_model.model.model.layers.17.mlp.gate_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
224
+ "model.base_model.model.model.layers.17.mlp.up_proj.base_layer.weight": "model-00002-of-00003.safetensors",
225
+ "model.base_model.model.model.layers.17.mlp.up_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
226
+ "model.base_model.model.model.layers.17.mlp.up_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
227
+ "model.base_model.model.model.layers.17.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
228
+ "model.base_model.model.model.layers.17.self_attn.k_proj.base_layer.weight": "model-00002-of-00003.safetensors",
229
+ "model.base_model.model.model.layers.17.self_attn.k_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
230
+ "model.base_model.model.model.layers.17.self_attn.k_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
231
+ "model.base_model.model.model.layers.17.self_attn.o_proj.base_layer.weight": "model-00002-of-00003.safetensors",
232
+ "model.base_model.model.model.layers.17.self_attn.o_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
233
+ "model.base_model.model.model.layers.17.self_attn.o_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
234
+ "model.base_model.model.model.layers.17.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
235
+ "model.base_model.model.model.layers.17.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
236
+ "model.base_model.model.model.layers.17.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
237
+ "model.base_model.model.model.layers.17.self_attn.v_proj.base_layer.weight": "model-00002-of-00003.safetensors",
238
+ "model.base_model.model.model.layers.17.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
239
+ "model.base_model.model.model.layers.17.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
240
+ "model.base_model.model.model.layers.18.input_layernorm.weight": "model-00002-of-00003.safetensors",
241
+ "model.base_model.model.model.layers.18.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
242
+ "model.base_model.model.model.layers.18.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
243
+ "model.base_model.model.model.layers.18.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
244
+ "model.base_model.model.model.layers.18.mlp.gate_proj.base_layer.weight": "model-00002-of-00003.safetensors",
245
+ "model.base_model.model.model.layers.18.mlp.gate_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
246
+ "model.base_model.model.model.layers.18.mlp.gate_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
247
+ "model.base_model.model.model.layers.18.mlp.up_proj.base_layer.weight": "model-00002-of-00003.safetensors",
248
+ "model.base_model.model.model.layers.18.mlp.up_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
249
+ "model.base_model.model.model.layers.18.mlp.up_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
250
+ "model.base_model.model.model.layers.18.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
251
+ "model.base_model.model.model.layers.18.self_attn.k_proj.base_layer.weight": "model-00002-of-00003.safetensors",
252
+ "model.base_model.model.model.layers.18.self_attn.k_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
253
+ "model.base_model.model.model.layers.18.self_attn.k_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
254
+ "model.base_model.model.model.layers.18.self_attn.o_proj.base_layer.weight": "model-00002-of-00003.safetensors",
255
+ "model.base_model.model.model.layers.18.self_attn.o_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
256
+ "model.base_model.model.model.layers.18.self_attn.o_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
257
+ "model.base_model.model.model.layers.18.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
258
+ "model.base_model.model.model.layers.18.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
259
+ "model.base_model.model.model.layers.18.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
260
+ "model.base_model.model.model.layers.18.self_attn.v_proj.base_layer.weight": "model-00002-of-00003.safetensors",
261
+ "model.base_model.model.model.layers.18.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
262
+ "model.base_model.model.model.layers.18.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
263
+ "model.base_model.model.model.layers.19.input_layernorm.weight": "model-00002-of-00003.safetensors",
264
+ "model.base_model.model.model.layers.19.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
265
+ "model.base_model.model.model.layers.19.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
266
+ "model.base_model.model.model.layers.19.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
267
+ "model.base_model.model.model.layers.19.mlp.gate_proj.base_layer.weight": "model-00002-of-00003.safetensors",
268
+ "model.base_model.model.model.layers.19.mlp.gate_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
269
+ "model.base_model.model.model.layers.19.mlp.gate_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
270
+ "model.base_model.model.model.layers.19.mlp.up_proj.base_layer.weight": "model-00002-of-00003.safetensors",
271
+ "model.base_model.model.model.layers.19.mlp.up_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
272
+ "model.base_model.model.model.layers.19.mlp.up_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
273
+ "model.base_model.model.model.layers.19.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
274
+ "model.base_model.model.model.layers.19.self_attn.k_proj.base_layer.weight": "model-00002-of-00003.safetensors",
275
+ "model.base_model.model.model.layers.19.self_attn.k_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
276
+ "model.base_model.model.model.layers.19.self_attn.k_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
277
+ "model.base_model.model.model.layers.19.self_attn.o_proj.base_layer.weight": "model-00002-of-00003.safetensors",
278
+ "model.base_model.model.model.layers.19.self_attn.o_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
279
+ "model.base_model.model.model.layers.19.self_attn.o_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
280
+ "model.base_model.model.model.layers.19.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
281
+ "model.base_model.model.model.layers.19.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
282
+ "model.base_model.model.model.layers.19.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
283
+ "model.base_model.model.model.layers.19.self_attn.v_proj.base_layer.weight": "model-00002-of-00003.safetensors",
284
+ "model.base_model.model.model.layers.19.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
285
+ "model.base_model.model.model.layers.19.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
286
+ "model.base_model.model.model.layers.2.input_layernorm.weight": "model-00001-of-00003.safetensors",
287
+ "model.base_model.model.model.layers.2.mlp.down_proj.base_layer.weight": "model-00001-of-00003.safetensors",
288
+ "model.base_model.model.model.layers.2.mlp.down_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
289
+ "model.base_model.model.model.layers.2.mlp.down_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
290
+ "model.base_model.model.model.layers.2.mlp.gate_proj.base_layer.weight": "model-00001-of-00003.safetensors",
291
+ "model.base_model.model.model.layers.2.mlp.gate_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
292
+ "model.base_model.model.model.layers.2.mlp.gate_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
293
+ "model.base_model.model.model.layers.2.mlp.up_proj.base_layer.weight": "model-00001-of-00003.safetensors",
294
+ "model.base_model.model.model.layers.2.mlp.up_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
295
+ "model.base_model.model.model.layers.2.mlp.up_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
296
+ "model.base_model.model.model.layers.2.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
297
+ "model.base_model.model.model.layers.2.self_attn.k_proj.base_layer.weight": "model-00001-of-00003.safetensors",
298
+ "model.base_model.model.model.layers.2.self_attn.k_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
299
+ "model.base_model.model.model.layers.2.self_attn.k_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
300
+ "model.base_model.model.model.layers.2.self_attn.o_proj.base_layer.weight": "model-00001-of-00003.safetensors",
301
+ "model.base_model.model.model.layers.2.self_attn.o_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
302
+ "model.base_model.model.model.layers.2.self_attn.o_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
303
+ "model.base_model.model.model.layers.2.self_attn.q_proj.base_layer.weight": "model-00001-of-00003.safetensors",
304
+ "model.base_model.model.model.layers.2.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
305
+ "model.base_model.model.model.layers.2.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
306
+ "model.base_model.model.model.layers.2.self_attn.v_proj.base_layer.weight": "model-00001-of-00003.safetensors",
307
+ "model.base_model.model.model.layers.2.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
308
+ "model.base_model.model.model.layers.2.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
309
+ "model.base_model.model.model.layers.20.input_layernorm.weight": "model-00002-of-00003.safetensors",
310
+ "model.base_model.model.model.layers.20.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
311
+ "model.base_model.model.model.layers.20.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
312
+ "model.base_model.model.model.layers.20.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
313
+ "model.base_model.model.model.layers.20.mlp.gate_proj.base_layer.weight": "model-00002-of-00003.safetensors",
314
+ "model.base_model.model.model.layers.20.mlp.gate_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
315
+ "model.base_model.model.model.layers.20.mlp.gate_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
316
+ "model.base_model.model.model.layers.20.mlp.up_proj.base_layer.weight": "model-00002-of-00003.safetensors",
317
+ "model.base_model.model.model.layers.20.mlp.up_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
318
+ "model.base_model.model.model.layers.20.mlp.up_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
319
+ "model.base_model.model.model.layers.20.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
320
+ "model.base_model.model.model.layers.20.self_attn.k_proj.base_layer.weight": "model-00002-of-00003.safetensors",
321
+ "model.base_model.model.model.layers.20.self_attn.k_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
322
+ "model.base_model.model.model.layers.20.self_attn.k_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
323
+ "model.base_model.model.model.layers.20.self_attn.o_proj.base_layer.weight": "model-00002-of-00003.safetensors",
324
+ "model.base_model.model.model.layers.20.self_attn.o_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
325
+ "model.base_model.model.model.layers.20.self_attn.o_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
326
+ "model.base_model.model.model.layers.20.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
327
+ "model.base_model.model.model.layers.20.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
328
+ "model.base_model.model.model.layers.20.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
329
+ "model.base_model.model.model.layers.20.self_attn.v_proj.base_layer.weight": "model-00002-of-00003.safetensors",
330
+ "model.base_model.model.model.layers.20.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
331
+ "model.base_model.model.model.layers.20.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
332
+ "model.base_model.model.model.layers.21.input_layernorm.weight": "model-00002-of-00003.safetensors",
333
+ "model.base_model.model.model.layers.21.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
334
+ "model.base_model.model.model.layers.21.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
335
+ "model.base_model.model.model.layers.21.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
336
+ "model.base_model.model.model.layers.21.mlp.gate_proj.base_layer.weight": "model-00002-of-00003.safetensors",
337
+ "model.base_model.model.model.layers.21.mlp.gate_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
338
+ "model.base_model.model.model.layers.21.mlp.gate_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
339
+ "model.base_model.model.model.layers.21.mlp.up_proj.base_layer.weight": "model-00002-of-00003.safetensors",
340
+ "model.base_model.model.model.layers.21.mlp.up_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
341
+ "model.base_model.model.model.layers.21.mlp.up_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
342
+ "model.base_model.model.model.layers.21.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
343
+ "model.base_model.model.model.layers.21.self_attn.k_proj.base_layer.weight": "model-00002-of-00003.safetensors",
344
+ "model.base_model.model.model.layers.21.self_attn.k_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
345
+ "model.base_model.model.model.layers.21.self_attn.k_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
346
+ "model.base_model.model.model.layers.21.self_attn.o_proj.base_layer.weight": "model-00002-of-00003.safetensors",
347
+ "model.base_model.model.model.layers.21.self_attn.o_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
348
+ "model.base_model.model.model.layers.21.self_attn.o_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
349
+ "model.base_model.model.model.layers.21.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
350
+ "model.base_model.model.model.layers.21.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
351
+ "model.base_model.model.model.layers.21.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
352
+ "model.base_model.model.model.layers.21.self_attn.v_proj.base_layer.weight": "model-00002-of-00003.safetensors",
353
+ "model.base_model.model.model.layers.21.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
354
+ "model.base_model.model.model.layers.21.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
355
+ "model.base_model.model.model.layers.22.input_layernorm.weight": "model-00002-of-00003.safetensors",
356
+ "model.base_model.model.model.layers.22.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
357
+ "model.base_model.model.model.layers.22.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
358
+ "model.base_model.model.model.layers.22.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
359
+ "model.base_model.model.model.layers.22.mlp.gate_proj.base_layer.weight": "model-00002-of-00003.safetensors",
360
+ "model.base_model.model.model.layers.22.mlp.gate_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
361
+ "model.base_model.model.model.layers.22.mlp.gate_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
362
+ "model.base_model.model.model.layers.22.mlp.up_proj.base_layer.weight": "model-00002-of-00003.safetensors",
363
+ "model.base_model.model.model.layers.22.mlp.up_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
364
+ "model.base_model.model.model.layers.22.mlp.up_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
365
+ "model.base_model.model.model.layers.22.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
366
+ "model.base_model.model.model.layers.22.self_attn.k_proj.base_layer.weight": "model-00002-of-00003.safetensors",
367
+ "model.base_model.model.model.layers.22.self_attn.k_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
368
+ "model.base_model.model.model.layers.22.self_attn.k_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
369
+ "model.base_model.model.model.layers.22.self_attn.o_proj.base_layer.weight": "model-00002-of-00003.safetensors",
370
+ "model.base_model.model.model.layers.22.self_attn.o_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
371
+ "model.base_model.model.model.layers.22.self_attn.o_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
372
+ "model.base_model.model.model.layers.22.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
373
+ "model.base_model.model.model.layers.22.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
374
+ "model.base_model.model.model.layers.22.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
375
+ "model.base_model.model.model.layers.22.self_attn.v_proj.base_layer.weight": "model-00002-of-00003.safetensors",
376
+ "model.base_model.model.model.layers.22.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
377
+ "model.base_model.model.model.layers.22.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
378
+ "model.base_model.model.model.layers.23.input_layernorm.weight": "model-00003-of-00003.safetensors",
379
+ "model.base_model.model.model.layers.23.mlp.down_proj.base_layer.weight": "model-00003-of-00003.safetensors",
380
+ "model.base_model.model.model.layers.23.mlp.down_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
381
+ "model.base_model.model.model.layers.23.mlp.down_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
382
+ "model.base_model.model.model.layers.23.mlp.gate_proj.base_layer.weight": "model-00003-of-00003.safetensors",
383
+ "model.base_model.model.model.layers.23.mlp.gate_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
384
+ "model.base_model.model.model.layers.23.mlp.gate_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
385
+ "model.base_model.model.model.layers.23.mlp.up_proj.base_layer.weight": "model-00003-of-00003.safetensors",
386
+ "model.base_model.model.model.layers.23.mlp.up_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
387
+ "model.base_model.model.model.layers.23.mlp.up_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
388
+ "model.base_model.model.model.layers.23.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
389
+ "model.base_model.model.model.layers.23.self_attn.k_proj.base_layer.weight": "model-00003-of-00003.safetensors",
390
+ "model.base_model.model.model.layers.23.self_attn.k_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
391
+ "model.base_model.model.model.layers.23.self_attn.k_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
392
+ "model.base_model.model.model.layers.23.self_attn.o_proj.base_layer.weight": "model-00003-of-00003.safetensors",
393
+ "model.base_model.model.model.layers.23.self_attn.o_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
394
+ "model.base_model.model.model.layers.23.self_attn.o_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
395
+ "model.base_model.model.model.layers.23.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
396
+ "model.base_model.model.model.layers.23.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
397
+ "model.base_model.model.model.layers.23.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
398
+ "model.base_model.model.model.layers.23.self_attn.v_proj.base_layer.weight": "model-00003-of-00003.safetensors",
399
+ "model.base_model.model.model.layers.23.self_attn.v_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
400
+ "model.base_model.model.model.layers.23.self_attn.v_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
401
+ "model.base_model.model.model.layers.24.input_layernorm.weight": "model-00003-of-00003.safetensors",
402
+ "model.base_model.model.model.layers.24.mlp.down_proj.base_layer.weight": "model-00003-of-00003.safetensors",
403
+ "model.base_model.model.model.layers.24.mlp.down_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
404
+ "model.base_model.model.model.layers.24.mlp.down_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
405
+ "model.base_model.model.model.layers.24.mlp.gate_proj.base_layer.weight": "model-00003-of-00003.safetensors",
406
+ "model.base_model.model.model.layers.24.mlp.gate_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
407
+ "model.base_model.model.model.layers.24.mlp.gate_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
408
+ "model.base_model.model.model.layers.24.mlp.up_proj.base_layer.weight": "model-00003-of-00003.safetensors",
409
+ "model.base_model.model.model.layers.24.mlp.up_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
410
+ "model.base_model.model.model.layers.24.mlp.up_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
411
+ "model.base_model.model.model.layers.24.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
412
+ "model.base_model.model.model.layers.24.self_attn.k_proj.base_layer.weight": "model-00003-of-00003.safetensors",
413
+ "model.base_model.model.model.layers.24.self_attn.k_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
414
+ "model.base_model.model.model.layers.24.self_attn.k_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
415
+ "model.base_model.model.model.layers.24.self_attn.o_proj.base_layer.weight": "model-00003-of-00003.safetensors",
416
+ "model.base_model.model.model.layers.24.self_attn.o_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
417
+ "model.base_model.model.model.layers.24.self_attn.o_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
418
+ "model.base_model.model.model.layers.24.self_attn.q_proj.base_layer.weight": "model-00003-of-00003.safetensors",
419
+ "model.base_model.model.model.layers.24.self_attn.q_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
420
+ "model.base_model.model.model.layers.24.self_attn.q_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
421
+ "model.base_model.model.model.layers.24.self_attn.v_proj.base_layer.weight": "model-00003-of-00003.safetensors",
422
+ "model.base_model.model.model.layers.24.self_attn.v_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
423
+ "model.base_model.model.model.layers.24.self_attn.v_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
424
+ "model.base_model.model.model.layers.25.input_layernorm.weight": "model-00003-of-00003.safetensors",
425
+ "model.base_model.model.model.layers.25.mlp.down_proj.base_layer.weight": "model-00003-of-00003.safetensors",
426
+ "model.base_model.model.model.layers.25.mlp.down_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
427
+ "model.base_model.model.model.layers.25.mlp.down_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
428
+ "model.base_model.model.model.layers.25.mlp.gate_proj.base_layer.weight": "model-00003-of-00003.safetensors",
429
+ "model.base_model.model.model.layers.25.mlp.gate_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
430
+ "model.base_model.model.model.layers.25.mlp.gate_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
431
+ "model.base_model.model.model.layers.25.mlp.up_proj.base_layer.weight": "model-00003-of-00003.safetensors",
432
+ "model.base_model.model.model.layers.25.mlp.up_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
433
+ "model.base_model.model.model.layers.25.mlp.up_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
434
+ "model.base_model.model.model.layers.25.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
435
+ "model.base_model.model.model.layers.25.self_attn.k_proj.base_layer.weight": "model-00003-of-00003.safetensors",
436
+ "model.base_model.model.model.layers.25.self_attn.k_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
437
+ "model.base_model.model.model.layers.25.self_attn.k_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
438
+ "model.base_model.model.model.layers.25.self_attn.o_proj.base_layer.weight": "model-00003-of-00003.safetensors",
439
+ "model.base_model.model.model.layers.25.self_attn.o_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
440
+ "model.base_model.model.model.layers.25.self_attn.o_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
441
+ "model.base_model.model.model.layers.25.self_attn.q_proj.base_layer.weight": "model-00003-of-00003.safetensors",
442
+ "model.base_model.model.model.layers.25.self_attn.q_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
443
+ "model.base_model.model.model.layers.25.self_attn.q_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
444
+ "model.base_model.model.model.layers.25.self_attn.v_proj.base_layer.weight": "model-00003-of-00003.safetensors",
445
+ "model.base_model.model.model.layers.25.self_attn.v_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
446
+ "model.base_model.model.model.layers.25.self_attn.v_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
447
+ "model.base_model.model.model.layers.26.input_layernorm.weight": "model-00003-of-00003.safetensors",
448
+ "model.base_model.model.model.layers.26.mlp.down_proj.base_layer.weight": "model-00003-of-00003.safetensors",
449
+ "model.base_model.model.model.layers.26.mlp.down_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
450
+ "model.base_model.model.model.layers.26.mlp.down_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
451
+ "model.base_model.model.model.layers.26.mlp.gate_proj.base_layer.weight": "model-00003-of-00003.safetensors",
452
+ "model.base_model.model.model.layers.26.mlp.gate_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
453
+ "model.base_model.model.model.layers.26.mlp.gate_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
454
+ "model.base_model.model.model.layers.26.mlp.up_proj.base_layer.weight": "model-00003-of-00003.safetensors",
455
+ "model.base_model.model.model.layers.26.mlp.up_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
456
+ "model.base_model.model.model.layers.26.mlp.up_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
457
+ "model.base_model.model.model.layers.26.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
458
+ "model.base_model.model.model.layers.26.self_attn.k_proj.base_layer.weight": "model-00003-of-00003.safetensors",
459
+ "model.base_model.model.model.layers.26.self_attn.k_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
460
+ "model.base_model.model.model.layers.26.self_attn.k_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
461
+ "model.base_model.model.model.layers.26.self_attn.o_proj.base_layer.weight": "model-00003-of-00003.safetensors",
462
+ "model.base_model.model.model.layers.26.self_attn.o_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
463
+ "model.base_model.model.model.layers.26.self_attn.o_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
464
+ "model.base_model.model.model.layers.26.self_attn.q_proj.base_layer.weight": "model-00003-of-00003.safetensors",
465
+ "model.base_model.model.model.layers.26.self_attn.q_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
466
+ "model.base_model.model.model.layers.26.self_attn.q_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
467
+ "model.base_model.model.model.layers.26.self_attn.v_proj.base_layer.weight": "model-00003-of-00003.safetensors",
468
+ "model.base_model.model.model.layers.26.self_attn.v_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
469
+ "model.base_model.model.model.layers.26.self_attn.v_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
470
+ "model.base_model.model.model.layers.27.input_layernorm.weight": "model-00003-of-00003.safetensors",
471
+ "model.base_model.model.model.layers.27.mlp.down_proj.base_layer.weight": "model-00003-of-00003.safetensors",
472
+ "model.base_model.model.model.layers.27.mlp.down_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
473
+ "model.base_model.model.model.layers.27.mlp.down_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
474
+ "model.base_model.model.model.layers.27.mlp.gate_proj.base_layer.weight": "model-00003-of-00003.safetensors",
475
+ "model.base_model.model.model.layers.27.mlp.gate_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
476
+ "model.base_model.model.model.layers.27.mlp.gate_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
477
+ "model.base_model.model.model.layers.27.mlp.up_proj.base_layer.weight": "model-00003-of-00003.safetensors",
478
+ "model.base_model.model.model.layers.27.mlp.up_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
479
+ "model.base_model.model.model.layers.27.mlp.up_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
480
+ "model.base_model.model.model.layers.27.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
481
+ "model.base_model.model.model.layers.27.self_attn.k_proj.base_layer.weight": "model-00003-of-00003.safetensors",
482
+ "model.base_model.model.model.layers.27.self_attn.k_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
483
+ "model.base_model.model.model.layers.27.self_attn.k_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
484
+ "model.base_model.model.model.layers.27.self_attn.o_proj.base_layer.weight": "model-00003-of-00003.safetensors",
485
+ "model.base_model.model.model.layers.27.self_attn.o_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
486
+ "model.base_model.model.model.layers.27.self_attn.o_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
487
+ "model.base_model.model.model.layers.27.self_attn.q_proj.base_layer.weight": "model-00003-of-00003.safetensors",
488
+ "model.base_model.model.model.layers.27.self_attn.q_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
489
+ "model.base_model.model.model.layers.27.self_attn.q_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
490
+ "model.base_model.model.model.layers.27.self_attn.v_proj.base_layer.weight": "model-00003-of-00003.safetensors",
491
+ "model.base_model.model.model.layers.27.self_attn.v_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
492
+ "model.base_model.model.model.layers.27.self_attn.v_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
493
+ "model.base_model.model.model.layers.28.input_layernorm.weight": "model-00003-of-00003.safetensors",
494
+ "model.base_model.model.model.layers.28.mlp.down_proj.base_layer.weight": "model-00003-of-00003.safetensors",
495
+ "model.base_model.model.model.layers.28.mlp.down_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
496
+ "model.base_model.model.model.layers.28.mlp.down_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
497
+ "model.base_model.model.model.layers.28.mlp.gate_proj.base_layer.weight": "model-00003-of-00003.safetensors",
498
+ "model.base_model.model.model.layers.28.mlp.gate_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
499
+ "model.base_model.model.model.layers.28.mlp.gate_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
500
+ "model.base_model.model.model.layers.28.mlp.up_proj.base_layer.weight": "model-00003-of-00003.safetensors",
501
+ "model.base_model.model.model.layers.28.mlp.up_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
502
+ "model.base_model.model.model.layers.28.mlp.up_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
503
+ "model.base_model.model.model.layers.28.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
504
+ "model.base_model.model.model.layers.28.self_attn.k_proj.base_layer.weight": "model-00003-of-00003.safetensors",
505
+ "model.base_model.model.model.layers.28.self_attn.k_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
506
+ "model.base_model.model.model.layers.28.self_attn.k_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
507
+ "model.base_model.model.model.layers.28.self_attn.o_proj.base_layer.weight": "model-00003-of-00003.safetensors",
508
+ "model.base_model.model.model.layers.28.self_attn.o_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
509
+ "model.base_model.model.model.layers.28.self_attn.o_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
510
+ "model.base_model.model.model.layers.28.self_attn.q_proj.base_layer.weight": "model-00003-of-00003.safetensors",
511
+ "model.base_model.model.model.layers.28.self_attn.q_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
512
+ "model.base_model.model.model.layers.28.self_attn.q_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
513
+ "model.base_model.model.model.layers.28.self_attn.v_proj.base_layer.weight": "model-00003-of-00003.safetensors",
514
+ "model.base_model.model.model.layers.28.self_attn.v_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
515
+ "model.base_model.model.model.layers.28.self_attn.v_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
516
+ "model.base_model.model.model.layers.29.input_layernorm.weight": "model-00003-of-00003.safetensors",
517
+ "model.base_model.model.model.layers.29.mlp.down_proj.base_layer.weight": "model-00003-of-00003.safetensors",
518
+ "model.base_model.model.model.layers.29.mlp.down_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
519
+ "model.base_model.model.model.layers.29.mlp.down_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
520
+ "model.base_model.model.model.layers.29.mlp.gate_proj.base_layer.weight": "model-00003-of-00003.safetensors",
521
+ "model.base_model.model.model.layers.29.mlp.gate_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
522
+ "model.base_model.model.model.layers.29.mlp.gate_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
523
+ "model.base_model.model.model.layers.29.mlp.up_proj.base_layer.weight": "model-00003-of-00003.safetensors",
524
+ "model.base_model.model.model.layers.29.mlp.up_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
525
+ "model.base_model.model.model.layers.29.mlp.up_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
526
+ "model.base_model.model.model.layers.29.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
527
+ "model.base_model.model.model.layers.29.self_attn.k_proj.base_layer.weight": "model-00003-of-00003.safetensors",
528
+ "model.base_model.model.model.layers.29.self_attn.k_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
529
+ "model.base_model.model.model.layers.29.self_attn.k_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
530
+ "model.base_model.model.model.layers.29.self_attn.o_proj.base_layer.weight": "model-00003-of-00003.safetensors",
531
+ "model.base_model.model.model.layers.29.self_attn.o_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
532
+ "model.base_model.model.model.layers.29.self_attn.o_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
533
+ "model.base_model.model.model.layers.29.self_attn.q_proj.base_layer.weight": "model-00003-of-00003.safetensors",
534
+ "model.base_model.model.model.layers.29.self_attn.q_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
535
+ "model.base_model.model.model.layers.29.self_attn.q_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
536
+ "model.base_model.model.model.layers.29.self_attn.v_proj.base_layer.weight": "model-00003-of-00003.safetensors",
537
+ "model.base_model.model.model.layers.29.self_attn.v_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
538
+ "model.base_model.model.model.layers.29.self_attn.v_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
539
+ "model.base_model.model.model.layers.3.input_layernorm.weight": "model-00001-of-00003.safetensors",
540
+ "model.base_model.model.model.layers.3.mlp.down_proj.base_layer.weight": "model-00001-of-00003.safetensors",
541
+ "model.base_model.model.model.layers.3.mlp.down_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
542
+ "model.base_model.model.model.layers.3.mlp.down_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
543
+ "model.base_model.model.model.layers.3.mlp.gate_proj.base_layer.weight": "model-00001-of-00003.safetensors",
544
+ "model.base_model.model.model.layers.3.mlp.gate_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
545
+ "model.base_model.model.model.layers.3.mlp.gate_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
546
+ "model.base_model.model.model.layers.3.mlp.up_proj.base_layer.weight": "model-00001-of-00003.safetensors",
547
+ "model.base_model.model.model.layers.3.mlp.up_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
548
+ "model.base_model.model.model.layers.3.mlp.up_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
549
+ "model.base_model.model.model.layers.3.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
550
+ "model.base_model.model.model.layers.3.self_attn.k_proj.base_layer.weight": "model-00001-of-00003.safetensors",
551
+ "model.base_model.model.model.layers.3.self_attn.k_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
552
+ "model.base_model.model.model.layers.3.self_attn.k_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
553
+ "model.base_model.model.model.layers.3.self_attn.o_proj.base_layer.weight": "model-00001-of-00003.safetensors",
554
+ "model.base_model.model.model.layers.3.self_attn.o_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
555
+ "model.base_model.model.model.layers.3.self_attn.o_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
556
+ "model.base_model.model.model.layers.3.self_attn.q_proj.base_layer.weight": "model-00001-of-00003.safetensors",
557
+ "model.base_model.model.model.layers.3.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
558
+ "model.base_model.model.model.layers.3.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
559
+ "model.base_model.model.model.layers.3.self_attn.v_proj.base_layer.weight": "model-00001-of-00003.safetensors",
560
+ "model.base_model.model.model.layers.3.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
561
+ "model.base_model.model.model.layers.3.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
562
+ "model.base_model.model.model.layers.30.input_layernorm.weight": "model-00003-of-00003.safetensors",
563
+ "model.base_model.model.model.layers.30.mlp.down_proj.base_layer.weight": "model-00003-of-00003.safetensors",
564
+ "model.base_model.model.model.layers.30.mlp.down_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
565
+ "model.base_model.model.model.layers.30.mlp.down_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
566
+ "model.base_model.model.model.layers.30.mlp.gate_proj.base_layer.weight": "model-00003-of-00003.safetensors",
567
+ "model.base_model.model.model.layers.30.mlp.gate_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
568
+ "model.base_model.model.model.layers.30.mlp.gate_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
569
+ "model.base_model.model.model.layers.30.mlp.up_proj.base_layer.weight": "model-00003-of-00003.safetensors",
570
+ "model.base_model.model.model.layers.30.mlp.up_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
571
+ "model.base_model.model.model.layers.30.mlp.up_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
572
+ "model.base_model.model.model.layers.30.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
573
+ "model.base_model.model.model.layers.30.self_attn.k_proj.base_layer.weight": "model-00003-of-00003.safetensors",
574
+ "model.base_model.model.model.layers.30.self_attn.k_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
575
+ "model.base_model.model.model.layers.30.self_attn.k_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
576
+ "model.base_model.model.model.layers.30.self_attn.o_proj.base_layer.weight": "model-00003-of-00003.safetensors",
577
+ "model.base_model.model.model.layers.30.self_attn.o_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
578
+ "model.base_model.model.model.layers.30.self_attn.o_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
579
+ "model.base_model.model.model.layers.30.self_attn.q_proj.base_layer.weight": "model-00003-of-00003.safetensors",
580
+ "model.base_model.model.model.layers.30.self_attn.q_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
581
+ "model.base_model.model.model.layers.30.self_attn.q_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
582
+ "model.base_model.model.model.layers.30.self_attn.v_proj.base_layer.weight": "model-00003-of-00003.safetensors",
583
+ "model.base_model.model.model.layers.30.self_attn.v_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
584
+ "model.base_model.model.model.layers.30.self_attn.v_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
585
+ "model.base_model.model.model.layers.31.input_layernorm.weight": "model-00003-of-00003.safetensors",
586
+ "model.base_model.model.model.layers.31.mlp.down_proj.base_layer.weight": "model-00003-of-00003.safetensors",
587
+ "model.base_model.model.model.layers.31.mlp.down_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
588
+ "model.base_model.model.model.layers.31.mlp.down_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
589
+ "model.base_model.model.model.layers.31.mlp.gate_proj.base_layer.weight": "model-00003-of-00003.safetensors",
590
+ "model.base_model.model.model.layers.31.mlp.gate_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
591
+ "model.base_model.model.model.layers.31.mlp.gate_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
592
+ "model.base_model.model.model.layers.31.mlp.up_proj.base_layer.weight": "model-00003-of-00003.safetensors",
593
+ "model.base_model.model.model.layers.31.mlp.up_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
594
+ "model.base_model.model.model.layers.31.mlp.up_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
595
+ "model.base_model.model.model.layers.31.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
596
+ "model.base_model.model.model.layers.31.self_attn.k_proj.base_layer.weight": "model-00003-of-00003.safetensors",
597
+ "model.base_model.model.model.layers.31.self_attn.k_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
598
+ "model.base_model.model.model.layers.31.self_attn.k_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
599
+ "model.base_model.model.model.layers.31.self_attn.o_proj.base_layer.weight": "model-00003-of-00003.safetensors",
600
+ "model.base_model.model.model.layers.31.self_attn.o_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
601
+ "model.base_model.model.model.layers.31.self_attn.o_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
602
+ "model.base_model.model.model.layers.31.self_attn.q_proj.base_layer.weight": "model-00003-of-00003.safetensors",
603
+ "model.base_model.model.model.layers.31.self_attn.q_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
604
+ "model.base_model.model.model.layers.31.self_attn.q_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
605
+ "model.base_model.model.model.layers.31.self_attn.v_proj.base_layer.weight": "model-00003-of-00003.safetensors",
606
+ "model.base_model.model.model.layers.31.self_attn.v_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
607
+ "model.base_model.model.model.layers.31.self_attn.v_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
608
+ "model.base_model.model.model.layers.4.input_layernorm.weight": "model-00001-of-00003.safetensors",
609
+ "model.base_model.model.model.layers.4.mlp.down_proj.base_layer.weight": "model-00001-of-00003.safetensors",
610
+ "model.base_model.model.model.layers.4.mlp.down_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
611
+ "model.base_model.model.model.layers.4.mlp.down_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
612
+ "model.base_model.model.model.layers.4.mlp.gate_proj.base_layer.weight": "model-00001-of-00003.safetensors",
613
+ "model.base_model.model.model.layers.4.mlp.gate_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
614
+ "model.base_model.model.model.layers.4.mlp.gate_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
615
+ "model.base_model.model.model.layers.4.mlp.up_proj.base_layer.weight": "model-00001-of-00003.safetensors",
616
+ "model.base_model.model.model.layers.4.mlp.up_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
617
+ "model.base_model.model.model.layers.4.mlp.up_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
618
+ "model.base_model.model.model.layers.4.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
619
+ "model.base_model.model.model.layers.4.self_attn.k_proj.base_layer.weight": "model-00001-of-00003.safetensors",
620
+ "model.base_model.model.model.layers.4.self_attn.k_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
621
+ "model.base_model.model.model.layers.4.self_attn.k_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
622
+ "model.base_model.model.model.layers.4.self_attn.o_proj.base_layer.weight": "model-00001-of-00003.safetensors",
623
+ "model.base_model.model.model.layers.4.self_attn.o_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
624
+ "model.base_model.model.model.layers.4.self_attn.o_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
625
+ "model.base_model.model.model.layers.4.self_attn.q_proj.base_layer.weight": "model-00001-of-00003.safetensors",
626
+ "model.base_model.model.model.layers.4.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
627
+ "model.base_model.model.model.layers.4.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
628
+ "model.base_model.model.model.layers.4.self_attn.v_proj.base_layer.weight": "model-00001-of-00003.safetensors",
629
+ "model.base_model.model.model.layers.4.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
630
+ "model.base_model.model.model.layers.4.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
631
+ "model.base_model.model.model.layers.5.input_layernorm.weight": "model-00001-of-00003.safetensors",
632
+ "model.base_model.model.model.layers.5.mlp.down_proj.base_layer.weight": "model-00001-of-00003.safetensors",
633
+ "model.base_model.model.model.layers.5.mlp.down_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
634
+ "model.base_model.model.model.layers.5.mlp.down_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
635
+ "model.base_model.model.model.layers.5.mlp.gate_proj.base_layer.weight": "model-00001-of-00003.safetensors",
636
+ "model.base_model.model.model.layers.5.mlp.gate_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
637
+ "model.base_model.model.model.layers.5.mlp.gate_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
638
+ "model.base_model.model.model.layers.5.mlp.up_proj.base_layer.weight": "model-00001-of-00003.safetensors",
639
+ "model.base_model.model.model.layers.5.mlp.up_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
640
+ "model.base_model.model.model.layers.5.mlp.up_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
641
+ "model.base_model.model.model.layers.5.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
642
+ "model.base_model.model.model.layers.5.self_attn.k_proj.base_layer.weight": "model-00001-of-00003.safetensors",
643
+ "model.base_model.model.model.layers.5.self_attn.k_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
644
+ "model.base_model.model.model.layers.5.self_attn.k_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
645
+ "model.base_model.model.model.layers.5.self_attn.o_proj.base_layer.weight": "model-00001-of-00003.safetensors",
646
+ "model.base_model.model.model.layers.5.self_attn.o_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
647
+ "model.base_model.model.model.layers.5.self_attn.o_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
648
+ "model.base_model.model.model.layers.5.self_attn.q_proj.base_layer.weight": "model-00001-of-00003.safetensors",
649
+ "model.base_model.model.model.layers.5.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
650
+ "model.base_model.model.model.layers.5.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
651
+ "model.base_model.model.model.layers.5.self_attn.v_proj.base_layer.weight": "model-00001-of-00003.safetensors",
652
+ "model.base_model.model.model.layers.5.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
653
+ "model.base_model.model.model.layers.5.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
654
+ "model.base_model.model.model.layers.6.input_layernorm.weight": "model-00001-of-00003.safetensors",
655
+ "model.base_model.model.model.layers.6.mlp.down_proj.base_layer.weight": "model-00001-of-00003.safetensors",
656
+ "model.base_model.model.model.layers.6.mlp.down_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
657
+ "model.base_model.model.model.layers.6.mlp.down_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
658
+ "model.base_model.model.model.layers.6.mlp.gate_proj.base_layer.weight": "model-00001-of-00003.safetensors",
659
+ "model.base_model.model.model.layers.6.mlp.gate_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
660
+ "model.base_model.model.model.layers.6.mlp.gate_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
661
+ "model.base_model.model.model.layers.6.mlp.up_proj.base_layer.weight": "model-00001-of-00003.safetensors",
662
+ "model.base_model.model.model.layers.6.mlp.up_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
663
+ "model.base_model.model.model.layers.6.mlp.up_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
664
+ "model.base_model.model.model.layers.6.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
665
+ "model.base_model.model.model.layers.6.self_attn.k_proj.base_layer.weight": "model-00001-of-00003.safetensors",
666
+ "model.base_model.model.model.layers.6.self_attn.k_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
667
+ "model.base_model.model.model.layers.6.self_attn.k_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
668
+ "model.base_model.model.model.layers.6.self_attn.o_proj.base_layer.weight": "model-00001-of-00003.safetensors",
669
+ "model.base_model.model.model.layers.6.self_attn.o_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
670
+ "model.base_model.model.model.layers.6.self_attn.o_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
671
+ "model.base_model.model.model.layers.6.self_attn.q_proj.base_layer.weight": "model-00001-of-00003.safetensors",
672
+ "model.base_model.model.model.layers.6.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
673
+ "model.base_model.model.model.layers.6.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
674
+ "model.base_model.model.model.layers.6.self_attn.v_proj.base_layer.weight": "model-00001-of-00003.safetensors",
675
+ "model.base_model.model.model.layers.6.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
676
+ "model.base_model.model.model.layers.6.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
677
+ "model.base_model.model.model.layers.7.input_layernorm.weight": "model-00001-of-00003.safetensors",
678
+ "model.base_model.model.model.layers.7.mlp.down_proj.base_layer.weight": "model-00001-of-00003.safetensors",
679
+ "model.base_model.model.model.layers.7.mlp.down_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
680
+ "model.base_model.model.model.layers.7.mlp.down_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
681
+ "model.base_model.model.model.layers.7.mlp.gate_proj.base_layer.weight": "model-00001-of-00003.safetensors",
682
+ "model.base_model.model.model.layers.7.mlp.gate_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
683
+ "model.base_model.model.model.layers.7.mlp.gate_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
684
+ "model.base_model.model.model.layers.7.mlp.up_proj.base_layer.weight": "model-00001-of-00003.safetensors",
685
+ "model.base_model.model.model.layers.7.mlp.up_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
686
+ "model.base_model.model.model.layers.7.mlp.up_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
687
+ "model.base_model.model.model.layers.7.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
688
+ "model.base_model.model.model.layers.7.self_attn.k_proj.base_layer.weight": "model-00001-of-00003.safetensors",
689
+ "model.base_model.model.model.layers.7.self_attn.k_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
690
+ "model.base_model.model.model.layers.7.self_attn.k_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
691
+ "model.base_model.model.model.layers.7.self_attn.o_proj.base_layer.weight": "model-00001-of-00003.safetensors",
692
+ "model.base_model.model.model.layers.7.self_attn.o_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
693
+ "model.base_model.model.model.layers.7.self_attn.o_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
694
+ "model.base_model.model.model.layers.7.self_attn.q_proj.base_layer.weight": "model-00001-of-00003.safetensors",
695
+ "model.base_model.model.model.layers.7.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
696
+ "model.base_model.model.model.layers.7.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
697
+ "model.base_model.model.model.layers.7.self_attn.v_proj.base_layer.weight": "model-00001-of-00003.safetensors",
698
+ "model.base_model.model.model.layers.7.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
699
+ "model.base_model.model.model.layers.7.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
700
+ "model.base_model.model.model.layers.8.input_layernorm.weight": "model-00001-of-00003.safetensors",
701
+ "model.base_model.model.model.layers.8.mlp.down_proj.base_layer.weight": "model-00001-of-00003.safetensors",
702
+ "model.base_model.model.model.layers.8.mlp.down_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
703
+ "model.base_model.model.model.layers.8.mlp.down_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
704
+ "model.base_model.model.model.layers.8.mlp.gate_proj.base_layer.weight": "model-00001-of-00003.safetensors",
705
+ "model.base_model.model.model.layers.8.mlp.gate_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
706
+ "model.base_model.model.model.layers.8.mlp.gate_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
707
+ "model.base_model.model.model.layers.8.mlp.up_proj.base_layer.weight": "model-00001-of-00003.safetensors",
708
+ "model.base_model.model.model.layers.8.mlp.up_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
709
+ "model.base_model.model.model.layers.8.mlp.up_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
710
+ "model.base_model.model.model.layers.8.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
711
+ "model.base_model.model.model.layers.8.self_attn.k_proj.base_layer.weight": "model-00001-of-00003.safetensors",
712
+ "model.base_model.model.model.layers.8.self_attn.k_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
713
+ "model.base_model.model.model.layers.8.self_attn.k_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
714
+ "model.base_model.model.model.layers.8.self_attn.o_proj.base_layer.weight": "model-00001-of-00003.safetensors",
715
+ "model.base_model.model.model.layers.8.self_attn.o_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
716
+ "model.base_model.model.model.layers.8.self_attn.o_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
717
+ "model.base_model.model.model.layers.8.self_attn.q_proj.base_layer.weight": "model-00001-of-00003.safetensors",
718
+ "model.base_model.model.model.layers.8.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
719
+ "model.base_model.model.model.layers.8.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
720
+ "model.base_model.model.model.layers.8.self_attn.v_proj.base_layer.weight": "model-00001-of-00003.safetensors",
721
+ "model.base_model.model.model.layers.8.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
722
+ "model.base_model.model.model.layers.8.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
723
+ "model.base_model.model.model.layers.9.input_layernorm.weight": "model-00001-of-00003.safetensors",
724
+ "model.base_model.model.model.layers.9.mlp.down_proj.base_layer.weight": "model-00001-of-00003.safetensors",
725
+ "model.base_model.model.model.layers.9.mlp.down_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
726
+ "model.base_model.model.model.layers.9.mlp.down_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
727
+ "model.base_model.model.model.layers.9.mlp.gate_proj.base_layer.weight": "model-00001-of-00003.safetensors",
728
+ "model.base_model.model.model.layers.9.mlp.gate_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
729
+ "model.base_model.model.model.layers.9.mlp.gate_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
730
+ "model.base_model.model.model.layers.9.mlp.up_proj.base_layer.weight": "model-00001-of-00003.safetensors",
731
+ "model.base_model.model.model.layers.9.mlp.up_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
732
+ "model.base_model.model.model.layers.9.mlp.up_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
733
+ "model.base_model.model.model.layers.9.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
734
+ "model.base_model.model.model.layers.9.self_attn.k_proj.base_layer.weight": "model-00001-of-00003.safetensors",
735
+ "model.base_model.model.model.layers.9.self_attn.k_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
736
+ "model.base_model.model.model.layers.9.self_attn.k_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
737
+ "model.base_model.model.model.layers.9.self_attn.o_proj.base_layer.weight": "model-00001-of-00003.safetensors",
738
+ "model.base_model.model.model.layers.9.self_attn.o_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
739
+ "model.base_model.model.model.layers.9.self_attn.o_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
740
+ "model.base_model.model.model.layers.9.self_attn.q_proj.base_layer.weight": "model-00001-of-00003.safetensors",
741
+ "model.base_model.model.model.layers.9.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
742
+ "model.base_model.model.model.layers.9.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
743
+ "model.base_model.model.model.layers.9.self_attn.v_proj.base_layer.weight": "model-00001-of-00003.safetensors",
744
+ "model.base_model.model.model.layers.9.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
745
+ "model.base_model.model.model.layers.9.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
746
+ "model.base_model.model.model.norm.weight": "model-00003-of-00003.safetensors",
747
+ "rqvae.decoder.mlp_layers.1.bias": "model-00003-of-00003.safetensors",
748
+ "rqvae.decoder.mlp_layers.1.weight": "model-00003-of-00003.safetensors",
749
+ "rqvae.decoder.mlp_layers.10.bias": "model-00003-of-00003.safetensors",
750
+ "rqvae.decoder.mlp_layers.10.weight": "model-00003-of-00003.safetensors",
751
+ "rqvae.decoder.mlp_layers.13.bias": "model-00003-of-00003.safetensors",
752
+ "rqvae.decoder.mlp_layers.13.weight": "model-00003-of-00003.safetensors",
753
+ "rqvae.decoder.mlp_layers.16.bias": "model-00003-of-00003.safetensors",
754
+ "rqvae.decoder.mlp_layers.16.weight": "model-00003-of-00003.safetensors",
755
+ "rqvae.decoder.mlp_layers.19.bias": "model-00003-of-00003.safetensors",
756
+ "rqvae.decoder.mlp_layers.19.weight": "model-00003-of-00003.safetensors",
757
+ "rqvae.decoder.mlp_layers.4.bias": "model-00003-of-00003.safetensors",
758
+ "rqvae.decoder.mlp_layers.4.weight": "model-00003-of-00003.safetensors",
759
+ "rqvae.decoder.mlp_layers.7.bias": "model-00003-of-00003.safetensors",
760
+ "rqvae.decoder.mlp_layers.7.weight": "model-00003-of-00003.safetensors",
761
+ "rqvae.encoder.mlp_layers.1.bias": "model-00003-of-00003.safetensors",
762
+ "rqvae.encoder.mlp_layers.1.weight": "model-00003-of-00003.safetensors",
763
+ "rqvae.encoder.mlp_layers.10.bias": "model-00003-of-00003.safetensors",
764
+ "rqvae.encoder.mlp_layers.10.weight": "model-00003-of-00003.safetensors",
765
+ "rqvae.encoder.mlp_layers.13.bias": "model-00003-of-00003.safetensors",
766
+ "rqvae.encoder.mlp_layers.13.weight": "model-00003-of-00003.safetensors",
767
+ "rqvae.encoder.mlp_layers.16.bias": "model-00003-of-00003.safetensors",
768
+ "rqvae.encoder.mlp_layers.16.weight": "model-00003-of-00003.safetensors",
769
+ "rqvae.encoder.mlp_layers.19.bias": "model-00003-of-00003.safetensors",
770
+ "rqvae.encoder.mlp_layers.19.weight": "model-00003-of-00003.safetensors",
771
+ "rqvae.encoder.mlp_layers.4.bias": "model-00003-of-00003.safetensors",
772
+ "rqvae.encoder.mlp_layers.4.weight": "model-00003-of-00003.safetensors",
773
+ "rqvae.encoder.mlp_layers.7.bias": "model-00003-of-00003.safetensors",
774
+ "rqvae.encoder.mlp_layers.7.weight": "model-00003-of-00003.safetensors",
775
+ "rqvae.rq.vq_layers.0.embedding.weight": "model-00003-of-00003.safetensors",
776
+ "rqvae.rq.vq_layers.1.embedding.weight": "model-00003-of-00003.safetensors",
777
+ "rqvae.rq.vq_layers.2.embedding.weight": "model-00003-of-00003.safetensors",
778
+ "rqvae.rq.vq_layers.3.embedding.weight": "model-00003-of-00003.safetensors"
779
+ }
780
+ }
Ins/checkpoint-9678/trainer_state.json ADDED
The diff for this file is too large to render. See raw diff
 
Ins/checkpoint-9678/zero_to_fp32.py ADDED
@@ -0,0 +1,674 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python
2
+
3
+ # Copyright (c) Microsoft Corporation.
4
+ # SPDX-License-Identifier: Apache-2.0
5
+
6
+ # DeepSpeed Team
7
+
8
+ # This script extracts fp32 consolidated weights from a zero 1, 2 and 3 DeepSpeed checkpoints. It gets
9
+ # copied into the top level checkpoint dir, so the user can easily do the conversion at any point in
10
+ # the future. Once extracted, the weights don't require DeepSpeed and can be used in any
11
+ # application.
12
+ #
13
+ # example:
14
+ # python zero_to_fp32.py . output_dir/
15
+ # or
16
+ # python zero_to_fp32.py . output_dir/ --safe_serialization
17
+
18
+ import argparse
19
+ import torch
20
+ import glob
21
+ import math
22
+ import os
23
+ import re
24
+ import json
25
+ from tqdm import tqdm
26
+ from collections import OrderedDict
27
+ from dataclasses import dataclass
28
+
29
+ # while this script doesn't use deepspeed to recover data, since the checkpoints are pickled with
30
+ # DeepSpeed data structures it has to be available in the current python environment.
31
+ from deepspeed.utils import logger
32
+ from deepspeed.checkpoint.constants import (DS_VERSION, OPTIMIZER_STATE_DICT, SINGLE_PARTITION_OF_FP32_GROUPS,
33
+ FP32_FLAT_GROUPS, ZERO_STAGE, PARTITION_COUNT, PARAM_SHAPES, BUFFER_NAMES,
34
+ FROZEN_PARAM_SHAPES, FROZEN_PARAM_FRAGMENTS)
35
+
36
+
37
+ @dataclass
38
+ class zero_model_state:
39
+ buffers: dict()
40
+ param_shapes: dict()
41
+ shared_params: list
42
+ ds_version: int
43
+ frozen_param_shapes: dict()
44
+ frozen_param_fragments: dict()
45
+
46
+
47
+ debug = 0
48
+
49
+ # load to cpu
50
+ device = torch.device('cpu')
51
+
52
+
53
+ def atoi(text):
54
+ return int(text) if text.isdigit() else text
55
+
56
+
57
+ def natural_keys(text):
58
+ '''
59
+ alist.sort(key=natural_keys) sorts in human order
60
+ http://nedbatchelder.com/blog/200712/human_sorting.html
61
+ (See Toothy's implementation in the comments)
62
+ '''
63
+ return [atoi(c) for c in re.split(r'(\d+)', text)]
64
+
65
+
66
+ def get_model_state_file(checkpoint_dir, zero_stage):
67
+ if not os.path.isdir(checkpoint_dir):
68
+ raise FileNotFoundError(f"Directory '{checkpoint_dir}' doesn't exist")
69
+
70
+ # there should be only one file
71
+ if zero_stage <= 2:
72
+ file = os.path.join(checkpoint_dir, "mp_rank_00_model_states.pt")
73
+ elif zero_stage == 3:
74
+ file = os.path.join(checkpoint_dir, "zero_pp_rank_0_mp_rank_00_model_states.pt")
75
+
76
+ if not os.path.exists(file):
77
+ raise FileNotFoundError(f"can't find model states file at '{file}'")
78
+
79
+ return file
80
+
81
+
82
+ def get_checkpoint_files(checkpoint_dir, glob_pattern):
83
+ # XXX: need to test that this simple glob rule works for multi-node setup too
84
+ ckpt_files = sorted(glob.glob(os.path.join(checkpoint_dir, glob_pattern)), key=natural_keys)
85
+
86
+ if len(ckpt_files) == 0:
87
+ raise FileNotFoundError(f"can't find {glob_pattern} files in directory '{checkpoint_dir}'")
88
+
89
+ return ckpt_files
90
+
91
+
92
+ def get_optim_files(checkpoint_dir):
93
+ return get_checkpoint_files(checkpoint_dir, "*_optim_states.pt")
94
+
95
+
96
+ def get_model_state_files(checkpoint_dir):
97
+ return get_checkpoint_files(checkpoint_dir, "*_model_states.pt")
98
+
99
+
100
+ def parse_model_states(files):
101
+ zero_model_states = []
102
+ for file in files:
103
+ state_dict = torch.load(file, map_location=device)
104
+
105
+ if BUFFER_NAMES not in state_dict:
106
+ raise ValueError(f"{file} is not a model state checkpoint")
107
+ buffer_names = state_dict[BUFFER_NAMES]
108
+ if debug:
109
+ print("Found buffers:", buffer_names)
110
+
111
+ # recover just the buffers while restoring them to fp32 if they were saved in fp16
112
+ buffers = {k: v.float() for k, v in state_dict["module"].items() if k in buffer_names}
113
+ param_shapes = state_dict[PARAM_SHAPES]
114
+
115
+ # collect parameters that are included in param_shapes
116
+ param_names = []
117
+ for s in param_shapes:
118
+ for name in s.keys():
119
+ param_names.append(name)
120
+
121
+ # update with frozen parameters
122
+ frozen_param_shapes = state_dict.get(FROZEN_PARAM_SHAPES, None)
123
+ if frozen_param_shapes is not None:
124
+ if debug:
125
+ print(f"Found frozen_param_shapes: {frozen_param_shapes}")
126
+ param_names += list(frozen_param_shapes.keys())
127
+
128
+ # handle shared params
129
+ shared_params = [[k, v] for k, v in state_dict["shared_params"].items()]
130
+
131
+ ds_version = state_dict.get(DS_VERSION, None)
132
+
133
+ frozen_param_fragments = state_dict.get(FROZEN_PARAM_FRAGMENTS, None)
134
+
135
+ z_model_state = zero_model_state(buffers=buffers,
136
+ param_shapes=param_shapes,
137
+ shared_params=shared_params,
138
+ ds_version=ds_version,
139
+ frozen_param_shapes=frozen_param_shapes,
140
+ frozen_param_fragments=frozen_param_fragments)
141
+ zero_model_states.append(z_model_state)
142
+
143
+ return zero_model_states
144
+
145
+
146
+ def parse_optim_states(files, ds_checkpoint_dir):
147
+ total_files = len(files)
148
+ state_dicts = []
149
+ for f in files:
150
+ state_dict = torch.load(f, map_location=device)
151
+ # immediately discard the potentially huge 2 optimizer states as we only care for fp32 master weights
152
+ # and also handle the case where it was already removed by another helper script
153
+ state_dict["optimizer_state_dict"].pop("optimizer_state_dict", None)
154
+ state_dicts.append(state_dict)
155
+
156
+ if not ZERO_STAGE in state_dicts[0][OPTIMIZER_STATE_DICT]:
157
+ raise ValueError(f"{files[0]} is not a zero checkpoint")
158
+ zero_stage = state_dicts[0][OPTIMIZER_STATE_DICT][ZERO_STAGE]
159
+ world_size = state_dicts[0][OPTIMIZER_STATE_DICT][PARTITION_COUNT]
160
+
161
+ # For ZeRO-2 each param group can have different partition_count as data parallelism for expert
162
+ # parameters can be different from data parallelism for non-expert parameters. So we can just
163
+ # use the max of the partition_count to get the dp world_size.
164
+
165
+ if type(world_size) is list:
166
+ world_size = max(world_size)
167
+
168
+ if world_size != total_files:
169
+ raise ValueError(
170
+ f"Expected {world_size} of '*_optim_states.pt' under '{ds_checkpoint_dir}' but found {total_files} files. "
171
+ "Possibly due to an overwrite of an old checkpoint, or a checkpoint didn't get saved by one or more processes."
172
+ )
173
+
174
+ # the groups are named differently in each stage
175
+ if zero_stage <= 2:
176
+ fp32_groups_key = SINGLE_PARTITION_OF_FP32_GROUPS
177
+ elif zero_stage == 3:
178
+ fp32_groups_key = FP32_FLAT_GROUPS
179
+ else:
180
+ raise ValueError(f"unknown zero stage {zero_stage}")
181
+
182
+ if zero_stage <= 2:
183
+ fp32_flat_groups = [state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key] for i in range(len(state_dicts))]
184
+ elif zero_stage == 3:
185
+ # if there is more than one param group, there will be multiple flattened tensors - one
186
+ # flattened tensor per group - for simplicity merge them into a single tensor
187
+ #
188
+ # XXX: could make the script more memory efficient for when there are multiple groups - it
189
+ # will require matching the sub-lists of param_shapes for each param group flattened tensor
190
+
191
+ fp32_flat_groups = [
192
+ torch.cat(state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key], 0) for i in range(len(state_dicts))
193
+ ]
194
+
195
+ return zero_stage, world_size, fp32_flat_groups
196
+
197
+
198
+ def _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir, exclude_frozen_parameters):
199
+ """
200
+ Returns fp32 state_dict reconstructed from ds checkpoint
201
+
202
+ Args:
203
+ - ``ds_checkpoint_dir``: path to the deepspeed checkpoint folder (where the optimizer files are)
204
+
205
+ """
206
+ print(f"Processing zero checkpoint '{ds_checkpoint_dir}'")
207
+
208
+ optim_files = get_optim_files(ds_checkpoint_dir)
209
+ zero_stage, world_size, fp32_flat_groups = parse_optim_states(optim_files, ds_checkpoint_dir)
210
+ print(f"Detected checkpoint of type zero stage {zero_stage}, world_size: {world_size}")
211
+
212
+ model_files = get_model_state_files(ds_checkpoint_dir)
213
+
214
+ zero_model_states = parse_model_states(model_files)
215
+ print(f'Parsing checkpoint created by deepspeed=={zero_model_states[0].ds_version}')
216
+
217
+ if zero_stage <= 2:
218
+ return _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states,
219
+ exclude_frozen_parameters)
220
+ elif zero_stage == 3:
221
+ return _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states,
222
+ exclude_frozen_parameters)
223
+
224
+
225
+ def _zero2_merge_frozen_params(state_dict, zero_model_states):
226
+ if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
227
+ return
228
+
229
+ frozen_param_shapes = zero_model_states[0].frozen_param_shapes
230
+ frozen_param_fragments = zero_model_states[0].frozen_param_fragments
231
+
232
+ if debug:
233
+ num_elem = sum(s.numel() for s in frozen_param_shapes.values())
234
+ print(f'rank 0: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
235
+
236
+ wanted_params = len(frozen_param_shapes)
237
+ wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
238
+ avail_numel = sum([p.numel() for p in frozen_param_fragments.values()])
239
+ print(f'Frozen params: Have {avail_numel} numels to process.')
240
+ print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
241
+
242
+ total_params = 0
243
+ total_numel = 0
244
+ for name, shape in frozen_param_shapes.items():
245
+ total_params += 1
246
+ unpartitioned_numel = shape.numel()
247
+ total_numel += unpartitioned_numel
248
+
249
+ state_dict[name] = frozen_param_fragments[name]
250
+
251
+ if debug:
252
+ print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
253
+
254
+ print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
255
+
256
+
257
+ def _has_callable(obj, fn):
258
+ attr = getattr(obj, fn, None)
259
+ return callable(attr)
260
+
261
+
262
+ def _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
263
+ param_shapes = zero_model_states[0].param_shapes
264
+
265
+ # Reconstruction protocol:
266
+ #
267
+ # XXX: document this
268
+
269
+ if debug:
270
+ for i in range(world_size):
271
+ for j in range(len(fp32_flat_groups[0])):
272
+ print(f"{FP32_FLAT_GROUPS}[{i}][{j}].shape={fp32_flat_groups[i][j].shape}")
273
+
274
+ # XXX: memory usage doubles here (zero2)
275
+ num_param_groups = len(fp32_flat_groups[0])
276
+ merged_single_partition_of_fp32_groups = []
277
+ for i in range(num_param_groups):
278
+ merged_partitions = [sd[i] for sd in fp32_flat_groups]
279
+ full_single_fp32_vector = torch.cat(merged_partitions, 0)
280
+ merged_single_partition_of_fp32_groups.append(full_single_fp32_vector)
281
+ avail_numel = sum(
282
+ [full_single_fp32_vector.numel() for full_single_fp32_vector in merged_single_partition_of_fp32_groups])
283
+
284
+ if debug:
285
+ wanted_params = sum([len(shapes) for shapes in param_shapes])
286
+ wanted_numel = sum([sum(shape.numel() for shape in shapes.values()) for shapes in param_shapes])
287
+ # not asserting if there is a mismatch due to possible padding
288
+ print(f"Have {avail_numel} numels to process.")
289
+ print(f"Need {wanted_numel} numels in {wanted_params} params.")
290
+
291
+ # params
292
+ # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
293
+ # out-of-core computing solution
294
+ total_numel = 0
295
+ total_params = 0
296
+ for shapes, full_single_fp32_vector in zip(param_shapes, merged_single_partition_of_fp32_groups):
297
+ offset = 0
298
+ avail_numel = full_single_fp32_vector.numel()
299
+ for name, shape in shapes.items():
300
+
301
+ unpartitioned_numel = shape.numel() if _has_callable(shape, 'numel') else math.prod(shape)
302
+ total_numel += unpartitioned_numel
303
+ total_params += 1
304
+
305
+ if debug:
306
+ print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
307
+ state_dict[name] = full_single_fp32_vector.narrow(0, offset, unpartitioned_numel).view(shape)
308
+ offset += unpartitioned_numel
309
+
310
+ # Z2 started to align to 2*world_size to improve nccl performance. Therefore both offset and
311
+ # avail_numel can differ by anywhere between 0..2*world_size. Due to two unrelated complex
312
+ # paddings performed in the code it's almost impossible to predict the exact numbers w/o the
313
+ # live optimizer object, so we are checking that the numbers are within the right range
314
+ align_to = 2 * world_size
315
+
316
+ def zero2_align(x):
317
+ return align_to * math.ceil(x / align_to)
318
+
319
+ if debug:
320
+ print(f"original offset={offset}, avail_numel={avail_numel}")
321
+
322
+ offset = zero2_align(offset)
323
+ avail_numel = zero2_align(avail_numel)
324
+
325
+ if debug:
326
+ print(f"aligned offset={offset}, avail_numel={avail_numel}")
327
+
328
+ # Sanity check
329
+ if offset != avail_numel:
330
+ raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
331
+
332
+ print(f"Reconstructed fp32 state dict with {total_params} params {total_numel} elements")
333
+
334
+
335
+ def _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states,
336
+ exclude_frozen_parameters):
337
+ state_dict = OrderedDict()
338
+
339
+ # buffers
340
+ buffers = zero_model_states[0].buffers
341
+ state_dict.update(buffers)
342
+ if debug:
343
+ print(f"added {len(buffers)} buffers")
344
+
345
+ if not exclude_frozen_parameters:
346
+ _zero2_merge_frozen_params(state_dict, zero_model_states)
347
+
348
+ _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
349
+
350
+ # recover shared parameters
351
+ for pair in zero_model_states[0].shared_params:
352
+ if pair[1] in state_dict:
353
+ state_dict[pair[0]] = state_dict[pair[1]]
354
+
355
+ return state_dict
356
+
357
+
358
+ def zero3_partitioned_param_info(unpartitioned_numel, world_size):
359
+ remainder = unpartitioned_numel % world_size
360
+ padding_numel = (world_size - remainder) if remainder else 0
361
+ partitioned_numel = math.ceil(unpartitioned_numel / world_size)
362
+ return partitioned_numel, padding_numel
363
+
364
+
365
+ def _zero3_merge_frozen_params(state_dict, world_size, zero_model_states):
366
+ if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
367
+ return
368
+
369
+ if debug:
370
+ for i in range(world_size):
371
+ num_elem = sum(s.numel() for s in zero_model_states[i].frozen_param_fragments.values())
372
+ print(f'rank {i}: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
373
+
374
+ frozen_param_shapes = zero_model_states[0].frozen_param_shapes
375
+ wanted_params = len(frozen_param_shapes)
376
+ wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
377
+ avail_numel = sum([p.numel() for p in zero_model_states[0].frozen_param_fragments.values()]) * world_size
378
+ print(f'Frozen params: Have {avail_numel} numels to process.')
379
+ print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
380
+
381
+ total_params = 0
382
+ total_numel = 0
383
+ for name, shape in zero_model_states[0].frozen_param_shapes.items():
384
+ total_params += 1
385
+ unpartitioned_numel = shape.numel()
386
+ total_numel += unpartitioned_numel
387
+
388
+ param_frags = tuple(model_state.frozen_param_fragments[name] for model_state in zero_model_states)
389
+ state_dict[name] = torch.cat(param_frags, 0).narrow(0, 0, unpartitioned_numel).view(shape)
390
+
391
+ partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
392
+
393
+ if debug:
394
+ print(
395
+ f"Frozen params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
396
+ )
397
+
398
+ print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
399
+
400
+
401
+ def _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
402
+ param_shapes = zero_model_states[0].param_shapes
403
+ avail_numel = fp32_flat_groups[0].numel() * world_size
404
+ # Reconstruction protocol: For zero3 we need to zip the partitions together at boundary of each
405
+ # param, re-consolidating each param, while dealing with padding if any
406
+
407
+ # merge list of dicts, preserving order
408
+ param_shapes = {k: v for d in param_shapes for k, v in d.items()}
409
+
410
+ if debug:
411
+ for i in range(world_size):
412
+ print(f"{FP32_FLAT_GROUPS}[{i}].shape={fp32_flat_groups[i].shape}")
413
+
414
+ wanted_params = len(param_shapes)
415
+ wanted_numel = sum(shape.numel() for shape in param_shapes.values())
416
+ # not asserting if there is a mismatch due to possible padding
417
+ avail_numel = fp32_flat_groups[0].numel() * world_size
418
+ print(f"Trainable params: Have {avail_numel} numels to process.")
419
+ print(f"Trainable params: Need {wanted_numel} numels in {wanted_params} params.")
420
+
421
+ # params
422
+ # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
423
+ # out-of-core computing solution
424
+ offset = 0
425
+ total_numel = 0
426
+ total_params = 0
427
+ for name, shape in tqdm(param_shapes.items(), desc='Gathering Sharded Weights'):
428
+ unpartitioned_numel = shape.numel()
429
+ total_numel += unpartitioned_numel
430
+ total_params += 1
431
+ partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
432
+
433
+ if debug:
434
+ print(
435
+ f"Trainable params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
436
+ )
437
+
438
+ # XXX: memory usage doubles here
439
+ state_dict[name] = torch.cat(
440
+ tuple(fp32_flat_groups[i].narrow(0, offset, partitioned_numel) for i in range(world_size)),
441
+ 0).narrow(0, 0, unpartitioned_numel).view(shape)
442
+ offset += partitioned_numel
443
+
444
+ offset *= world_size
445
+
446
+ # Sanity check
447
+ if offset != avail_numel:
448
+ raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
449
+
450
+ print(f"Reconstructed Trainable fp32 state dict with {total_params} params {total_numel} elements")
451
+
452
+
453
+ def _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states,
454
+ exclude_frozen_parameters):
455
+ state_dict = OrderedDict()
456
+
457
+ # buffers
458
+ buffers = zero_model_states[0].buffers
459
+ state_dict.update(buffers)
460
+ if debug:
461
+ print(f"added {len(buffers)} buffers")
462
+
463
+ if not exclude_frozen_parameters:
464
+ _zero3_merge_frozen_params(state_dict, world_size, zero_model_states)
465
+
466
+ _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
467
+
468
+ # recover shared parameters
469
+ for pair in zero_model_states[0].shared_params:
470
+ if pair[1] in state_dict:
471
+ state_dict[pair[0]] = state_dict[pair[1]]
472
+
473
+ return state_dict
474
+
475
+
476
+ def get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag=None, exclude_frozen_parameters=False):
477
+ """
478
+ Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated state_dict that can be loaded with
479
+ ``load_state_dict()`` and used for training without DeepSpeed or shared with others, for example
480
+ via a model hub.
481
+
482
+ Args:
483
+ - ``checkpoint_dir``: path to the desired checkpoint folder
484
+ - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in 'latest' file. e.g., ``global_step14``
485
+ - ``exclude_frozen_parameters``: exclude frozen parameters
486
+
487
+ Returns:
488
+ - pytorch ``state_dict``
489
+
490
+ Note: this approach may not work if your application doesn't have sufficient free CPU memory and
491
+ you may need to use the offline approach using the ``zero_to_fp32.py`` script that is saved with
492
+ the checkpoint.
493
+
494
+ A typical usage might be ::
495
+
496
+ from deepspeed.utils.zero_to_fp32 import get_fp32_state_dict_from_zero_checkpoint
497
+ # do the training and checkpoint saving
498
+ state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir) # already on cpu
499
+ model = model.cpu() # move to cpu
500
+ model.load_state_dict(state_dict)
501
+ # submit to model hub or save the model to share with others
502
+
503
+ In this example the ``model`` will no longer be usable in the deepspeed context of the same
504
+ application. i.e. you will need to re-initialize the deepspeed engine, since
505
+ ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
506
+
507
+ If you want it all done for you, use ``load_state_dict_from_zero_checkpoint`` instead.
508
+
509
+ """
510
+ if tag is None:
511
+ latest_path = os.path.join(checkpoint_dir, 'latest')
512
+ if os.path.isfile(latest_path):
513
+ with open(latest_path, 'r') as fd:
514
+ tag = fd.read().strip()
515
+ else:
516
+ raise ValueError(f"Unable to find 'latest' file at {latest_path}")
517
+
518
+ ds_checkpoint_dir = os.path.join(checkpoint_dir, tag)
519
+
520
+ if not os.path.isdir(ds_checkpoint_dir):
521
+ raise FileNotFoundError(f"Directory '{ds_checkpoint_dir}' doesn't exist")
522
+
523
+ return _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir, exclude_frozen_parameters)
524
+
525
+
526
+ def convert_zero_checkpoint_to_fp32_state_dict(checkpoint_dir,
527
+ output_dir,
528
+ max_shard_size="5GB",
529
+ safe_serialization=False,
530
+ tag=None,
531
+ exclude_frozen_parameters=False):
532
+ """
533
+ Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict`` file that can be
534
+ loaded with ``torch.load(file)`` + ``load_state_dict()`` and used for training without DeepSpeed.
535
+
536
+ Args:
537
+ - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
538
+ - ``output_dir``: directory to the pytorch fp32 state_dict output files
539
+ - ``max_shard_size``: the maximum size for a checkpoint before being sharded, default value is 5GB
540
+ - ``safe_serialization``: whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).
541
+ - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
542
+ - ``exclude_frozen_parameters``: exclude frozen parameters
543
+ """
544
+ # Dependency pre-check
545
+ if safe_serialization:
546
+ try:
547
+ from safetensors.torch import save_file
548
+ except ImportError:
549
+ print('If you want to use `safe_serialization`, please `pip install safetensors`')
550
+ raise
551
+ if max_shard_size is not None:
552
+ try:
553
+ from huggingface_hub import split_torch_state_dict_into_shards
554
+ except ImportError:
555
+ print('If you want to use `max_shard_size`, please `pip install huggingface_hub`')
556
+ raise
557
+
558
+ # Convert zero checkpoint to state_dict
559
+ state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag, exclude_frozen_parameters)
560
+
561
+ # Shard the model if it is too big.
562
+ weights_name = "model.safetensors" if safe_serialization else "pytorch_model.bin"
563
+ if max_shard_size is not None:
564
+ filename_pattern = weights_name.replace(".bin", "{suffix}.bin").replace(".safetensors", "{suffix}.safetensors")
565
+ state_dict_split = split_torch_state_dict_into_shards(state_dict,
566
+ filename_pattern=filename_pattern,
567
+ max_shard_size=max_shard_size)
568
+ else:
569
+ from collections import namedtuple
570
+ StateDictSplit = namedtuple("StateDictSplit", ["is_sharded", "filename_to_tensors"])
571
+ state_dict_split = StateDictSplit(is_sharded=False,
572
+ filename_to_tensors={weights_name: list(state_dict.keys())})
573
+
574
+ # Save the model
575
+ filename_to_tensors = state_dict_split.filename_to_tensors.items()
576
+ for shard_file, tensors in tqdm(filename_to_tensors, desc="Saving checkpoint shards"):
577
+ shard = {tensor: state_dict[tensor].contiguous() for tensor in tensors}
578
+ output_path = os.path.join(output_dir, shard_file)
579
+ if safe_serialization:
580
+ save_file(shard, output_path, metadata={"format": "pt"})
581
+ else:
582
+ torch.save(shard, output_path)
583
+
584
+ # Save index if sharded
585
+ if state_dict_split.is_sharded:
586
+ index = {
587
+ "metadata": state_dict_split.metadata,
588
+ "weight_map": state_dict_split.tensor_to_filename,
589
+ }
590
+ save_index_file = "model.safetensors.index.json" if safe_serialization else "pytorch_model.bin.index.json"
591
+ save_index_file = os.path.join(output_dir, save_index_file)
592
+ with open(save_index_file, "w", encoding="utf-8") as f:
593
+ content = json.dumps(index, indent=2, sort_keys=True) + "\n"
594
+ f.write(content)
595
+
596
+
597
+ def load_state_dict_from_zero_checkpoint(model, checkpoint_dir, tag=None):
598
+ """
599
+ 1. Put the provided model to cpu
600
+ 2. Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict``
601
+ 3. Load it into the provided model
602
+
603
+ Args:
604
+ - ``model``: the model object to update
605
+ - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
606
+ - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
607
+
608
+ Returns:
609
+ - ``model`: modified model
610
+
611
+ Make sure you have plenty of CPU memory available before you call this function. If you don't
612
+ have enough use the ``zero_to_fp32.py`` utility to do the conversion. You will find it
613
+ conveniently placed for you in the checkpoint folder.
614
+
615
+ A typical usage might be ::
616
+
617
+ from deepspeed.utils.zero_to_fp32 import load_state_dict_from_zero_checkpoint
618
+ model = load_state_dict_from_zero_checkpoint(trainer.model, checkpoint_dir)
619
+ # submit to model hub or save the model to share with others
620
+
621
+ Note, that once this was run, the ``model`` will no longer be usable in the deepspeed context
622
+ of the same application. i.e. you will need to re-initialize the deepspeed engine, since
623
+ ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
624
+
625
+ """
626
+ logger.info(f"Extracting fp32 weights")
627
+ state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag)
628
+
629
+ logger.info(f"Overwriting model with fp32 weights")
630
+ model = model.cpu()
631
+ model.load_state_dict(state_dict, strict=False)
632
+
633
+ return model
634
+
635
+
636
+ if __name__ == "__main__":
637
+ parser = argparse.ArgumentParser()
638
+ parser.add_argument("checkpoint_dir",
639
+ type=str,
640
+ help="path to the desired checkpoint folder, e.g., path/checkpoint-12")
641
+ parser.add_argument("output_dir",
642
+ type=str,
643
+ help="directory to the pytorch fp32 state_dict output files"
644
+ "(e.g. path/checkpoint-12-output/)")
645
+ parser.add_argument(
646
+ "--max_shard_size",
647
+ type=str,
648
+ default="5GB",
649
+ help="The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size"
650
+ "lower than this size. If expressed as a string, needs to be digits followed by a unit (like `5MB`"
651
+ "We default it to 5GB in order for models to be able to run easily on free-tier google colab instances"
652
+ "without CPU OOM issues.")
653
+ parser.add_argument(
654
+ "--safe_serialization",
655
+ default=False,
656
+ action='store_true',
657
+ help="Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).")
658
+ parser.add_argument("-t",
659
+ "--tag",
660
+ type=str,
661
+ default=None,
662
+ help="checkpoint tag used as a unique identifier for checkpoint. e.g., global_step1")
663
+ parser.add_argument("--exclude_frozen_parameters", action='store_true', help="exclude frozen parameters")
664
+ parser.add_argument("-d", "--debug", action='store_true', help="enable debug")
665
+ args = parser.parse_args()
666
+
667
+ debug = args.debug
668
+
669
+ convert_zero_checkpoint_to_fp32_state_dict(args.checkpoint_dir,
670
+ args.output_dir,
671
+ max_shard_size=args.max_shard_size,
672
+ safe_serialization=args.safe_serialization,
673
+ tag=args.tag,
674
+ exclude_frozen_parameters=args.exclude_frozen_parameters)
Ins/config.json ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "/home/sgugger/tmp/llama/llama-7b/",
3
+ "architectures": [
4
+ "LlamaWithRQ"
5
+ ],
6
+ "args": {
7
+ "add_prefix": false,
8
+ "base_model": "/home/jovyan/workspace/Llama-7b",
9
+ "batch_size": 1024,
10
+ "bf16": true,
11
+ "bn": false,
12
+ "ckpt_dir": "",
13
+ "data_path": "/home/jovyan/workspace",
14
+ "dataloader_num_workers": 4,
15
+ "dataloader_prefetch_factor": 2,
16
+ "dataset": "Instruments",
17
+ "deepspeed": "./config/ds_z2_bf16.json",
18
+ "device": "cuda:1",
19
+ "dropout_prob": 0.0,
20
+ "e_dim": 32,
21
+ "epochs": 1,
22
+ "eval_step": 50,
23
+ "fp16": false,
24
+ "gradient_accumulation_steps": 2,
25
+ "his_sep": ", ",
26
+ "index_file": ".index.json",
27
+ "kmeans_init": false,
28
+ "kmeans_iters": 100,
29
+ "layers": [
30
+ 2048,
31
+ 1024,
32
+ 512,
33
+ 256,
34
+ 128,
35
+ 64
36
+ ],
37
+ "learner": "AdamW",
38
+ "learning_rate": 0.0005,
39
+ "logging_step": 10,
40
+ "lora_alpha": 32,
41
+ "lora_dropout": 0.05,
42
+ "lora_modules_to_save": "embed_tokens,lm_head",
43
+ "lora_r": 8,
44
+ "lora_target_modules": "q_proj,v_proj,k_proj,o_proj,gate_proj,down_proj,up_proj",
45
+ "loss_type": "mse",
46
+ "lr": 0.001,
47
+ "lr_scheduler_type": "cosine",
48
+ "max_his_len": 20,
49
+ "model_max_length": 1024,
50
+ "num_emb_list": [
51
+ 256,
52
+ 256,
53
+ 256,
54
+ 256
55
+ ],
56
+ "num_workers": 4,
57
+ "only_train_response": true,
58
+ "optim": "adamw_torch",
59
+ "output_dir": "./Ins",
60
+ "per_device_batch_size": 8,
61
+ "quant_loss_weight": 1.0,
62
+ "remove_unused_columns": false,
63
+ "resume_from_checkpoint": null,
64
+ "rqvae_model": "/home/jovyan/workspace/LC-Rec/index/Ins/Apr-04-2025_07-12-04/best_collision_model.pth",
65
+ "sample_valid": true,
66
+ "save_and_eval_steps": 1000,
67
+ "save_and_eval_strategy": "epoch",
68
+ "seed": 42,
69
+ "sk_epsilons": [
70
+ 0.0,
71
+ 0.0,
72
+ 0.0,
73
+ 0.0
74
+ ],
75
+ "sk_iters": 50,
76
+ "tasks": "seqrec,itemsearch,inters2title,inters2description,preferenceobtain,item2index,index2item,intertitles2item,query2item",
77
+ "train_data_sample_num": "0,0,0,0,0,0,0,0,0",
78
+ "train_prompt_sample_num": "1,1,1,1,1,1,1,1,1",
79
+ "valid_prompt_id": 0,
80
+ "valid_prompt_sample_num": 2,
81
+ "warmup": 5,
82
+ "warmup_ratio": 0.01,
83
+ "weight_decay": 0.01
84
+ },
85
+ "attention_bias": false,
86
+ "attention_dropout": 0.0,
87
+ "bos_token_id": 1,
88
+ "eos_token_id": 2,
89
+ "head_dim": 128,
90
+ "hidden_act": "silu",
91
+ "hidden_size": 4096,
92
+ "initializer_range": 0.02,
93
+ "intermediate_size": 11008,
94
+ "max_position_embeddings": 2048,
95
+ "max_sequence_length": 2048,
96
+ "mlp_bias": false,
97
+ "model_type": "llama",
98
+ "num_attention_heads": 32,
99
+ "num_hidden_layers": 32,
100
+ "num_key_value_heads": 32,
101
+ "pad_token_id": 0,
102
+ "pretraining_tp": 1,
103
+ "rms_norm_eps": 1e-06,
104
+ "rope_scaling": null,
105
+ "rope_theta": 10000.0,
106
+ "tie_word_embeddings": false,
107
+ "torch_dtype": "bfloat16",
108
+ "transformers_version": "4.45.2",
109
+ "use_cache": false,
110
+ "vocab_size": 33024
111
+ }
Ins/finetune/README.md ADDED
@@ -0,0 +1,202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: /home/jovyan/workspace/Llama-7b
3
+ library_name: peft
4
+ ---
5
+
6
+ # Model Card for Model ID
7
+
8
+ <!-- Provide a quick summary of what the model is/does. -->
9
+
10
+
11
+
12
+ ## Model Details
13
+
14
+ ### Model Description
15
+
16
+ <!-- Provide a longer summary of what this model is. -->
17
+
18
+
19
+
20
+ - **Developed by:** [More Information Needed]
21
+ - **Funded by [optional]:** [More Information Needed]
22
+ - **Shared by [optional]:** [More Information Needed]
23
+ - **Model type:** [More Information Needed]
24
+ - **Language(s) (NLP):** [More Information Needed]
25
+ - **License:** [More Information Needed]
26
+ - **Finetuned from model [optional]:** [More Information Needed]
27
+
28
+ ### Model Sources [optional]
29
+
30
+ <!-- Provide the basic links for the model. -->
31
+
32
+ - **Repository:** [More Information Needed]
33
+ - **Paper [optional]:** [More Information Needed]
34
+ - **Demo [optional]:** [More Information Needed]
35
+
36
+ ## Uses
37
+
38
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
39
+
40
+ ### Direct Use
41
+
42
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
43
+
44
+ [More Information Needed]
45
+
46
+ ### Downstream Use [optional]
47
+
48
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
49
+
50
+ [More Information Needed]
51
+
52
+ ### Out-of-Scope Use
53
+
54
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
55
+
56
+ [More Information Needed]
57
+
58
+ ## Bias, Risks, and Limitations
59
+
60
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
61
+
62
+ [More Information Needed]
63
+
64
+ ### Recommendations
65
+
66
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
67
+
68
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
69
+
70
+ ## How to Get Started with the Model
71
+
72
+ Use the code below to get started with the model.
73
+
74
+ [More Information Needed]
75
+
76
+ ## Training Details
77
+
78
+ ### Training Data
79
+
80
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
81
+
82
+ [More Information Needed]
83
+
84
+ ### Training Procedure
85
+
86
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
87
+
88
+ #### Preprocessing [optional]
89
+
90
+ [More Information Needed]
91
+
92
+
93
+ #### Training Hyperparameters
94
+
95
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
96
+
97
+ #### Speeds, Sizes, Times [optional]
98
+
99
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
100
+
101
+ [More Information Needed]
102
+
103
+ ## Evaluation
104
+
105
+ <!-- This section describes the evaluation protocols and provides the results. -->
106
+
107
+ ### Testing Data, Factors & Metrics
108
+
109
+ #### Testing Data
110
+
111
+ <!-- This should link to a Dataset Card if possible. -->
112
+
113
+ [More Information Needed]
114
+
115
+ #### Factors
116
+
117
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
118
+
119
+ [More Information Needed]
120
+
121
+ #### Metrics
122
+
123
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
124
+
125
+ [More Information Needed]
126
+
127
+ ### Results
128
+
129
+ [More Information Needed]
130
+
131
+ #### Summary
132
+
133
+
134
+
135
+ ## Model Examination [optional]
136
+
137
+ <!-- Relevant interpretability work for the model goes here -->
138
+
139
+ [More Information Needed]
140
+
141
+ ## Environmental Impact
142
+
143
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
144
+
145
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
146
+
147
+ - **Hardware Type:** [More Information Needed]
148
+ - **Hours used:** [More Information Needed]
149
+ - **Cloud Provider:** [More Information Needed]
150
+ - **Compute Region:** [More Information Needed]
151
+ - **Carbon Emitted:** [More Information Needed]
152
+
153
+ ## Technical Specifications [optional]
154
+
155
+ ### Model Architecture and Objective
156
+
157
+ [More Information Needed]
158
+
159
+ ### Compute Infrastructure
160
+
161
+ [More Information Needed]
162
+
163
+ #### Hardware
164
+
165
+ [More Information Needed]
166
+
167
+ #### Software
168
+
169
+ [More Information Needed]
170
+
171
+ ## Citation [optional]
172
+
173
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
174
+
175
+ **BibTeX:**
176
+
177
+ [More Information Needed]
178
+
179
+ **APA:**
180
+
181
+ [More Information Needed]
182
+
183
+ ## Glossary [optional]
184
+
185
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
186
+
187
+ [More Information Needed]
188
+
189
+ ## More Information [optional]
190
+
191
+ [More Information Needed]
192
+
193
+ ## Model Card Authors [optional]
194
+
195
+ [More Information Needed]
196
+
197
+ ## Model Card Contact
198
+
199
+ [More Information Needed]
200
+ ### Framework versions
201
+
202
+ - PEFT 0.15.1
Ins/finetune/adapter_config.json ADDED
@@ -0,0 +1,42 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alpha_pattern": {},
3
+ "auto_mapping": null,
4
+ "base_model_name_or_path": "/home/jovyan/workspace/Llama-7b",
5
+ "bias": "none",
6
+ "corda_config": null,
7
+ "eva_config": null,
8
+ "exclude_modules": null,
9
+ "fan_in_fan_out": false,
10
+ "inference_mode": true,
11
+ "init_lora_weights": true,
12
+ "layer_replication": null,
13
+ "layers_pattern": null,
14
+ "layers_to_transform": null,
15
+ "loftq_config": {},
16
+ "lora_alpha": 32,
17
+ "lora_bias": false,
18
+ "lora_dropout": 0.05,
19
+ "megatron_config": null,
20
+ "megatron_core": "megatron.core",
21
+ "modules_to_save": [
22
+ "embed_tokens",
23
+ "lm_head"
24
+ ],
25
+ "peft_type": "LORA",
26
+ "r": 8,
27
+ "rank_pattern": {},
28
+ "revision": null,
29
+ "target_modules": [
30
+ "up_proj",
31
+ "v_proj",
32
+ "q_proj",
33
+ "k_proj",
34
+ "down_proj",
35
+ "gate_proj",
36
+ "o_proj"
37
+ ],
38
+ "task_type": "CAUSAL_LM",
39
+ "trainable_token_indices": null,
40
+ "use_dora": false,
41
+ "use_rslora": false
42
+ }
Ins/finetune/added_tokens.json ADDED
@@ -0,0 +1,1026 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "<a-0>": 32000,
3
+ "<a-100>": 32100,
4
+ "<a-101>": 32101,
5
+ "<a-102>": 32102,
6
+ "<a-103>": 32103,
7
+ "<a-104>": 32104,
8
+ "<a-105>": 32105,
9
+ "<a-106>": 32106,
10
+ "<a-107>": 32107,
11
+ "<a-108>": 32108,
12
+ "<a-109>": 32109,
13
+ "<a-10>": 32010,
14
+ "<a-110>": 32110,
15
+ "<a-111>": 32111,
16
+ "<a-112>": 32112,
17
+ "<a-113>": 32113,
18
+ "<a-114>": 32114,
19
+ "<a-115>": 32115,
20
+ "<a-116>": 32116,
21
+ "<a-117>": 32117,
22
+ "<a-118>": 32118,
23
+ "<a-119>": 32119,
24
+ "<a-11>": 32011,
25
+ "<a-120>": 32120,
26
+ "<a-121>": 32121,
27
+ "<a-122>": 32122,
28
+ "<a-123>": 32123,
29
+ "<a-124>": 32124,
30
+ "<a-125>": 32125,
31
+ "<a-126>": 32126,
32
+ "<a-127>": 32127,
33
+ "<a-128>": 32128,
34
+ "<a-129>": 32129,
35
+ "<a-12>": 32012,
36
+ "<a-130>": 32130,
37
+ "<a-131>": 32131,
38
+ "<a-132>": 32132,
39
+ "<a-133>": 32133,
40
+ "<a-134>": 32134,
41
+ "<a-135>": 32135,
42
+ "<a-136>": 32136,
43
+ "<a-137>": 32137,
44
+ "<a-138>": 32138,
45
+ "<a-139>": 32139,
46
+ "<a-13>": 32013,
47
+ "<a-140>": 32140,
48
+ "<a-141>": 32141,
49
+ "<a-142>": 32142,
50
+ "<a-143>": 32143,
51
+ "<a-144>": 32144,
52
+ "<a-145>": 32145,
53
+ "<a-146>": 32146,
54
+ "<a-147>": 32147,
55
+ "<a-148>": 32148,
56
+ "<a-149>": 32149,
57
+ "<a-14>": 32014,
58
+ "<a-150>": 32150,
59
+ "<a-151>": 32151,
60
+ "<a-152>": 32152,
61
+ "<a-153>": 32153,
62
+ "<a-154>": 32154,
63
+ "<a-155>": 32155,
64
+ "<a-156>": 32156,
65
+ "<a-157>": 32157,
66
+ "<a-158>": 32158,
67
+ "<a-159>": 32159,
68
+ "<a-15>": 32015,
69
+ "<a-160>": 32160,
70
+ "<a-161>": 32161,
71
+ "<a-162>": 32162,
72
+ "<a-163>": 32163,
73
+ "<a-164>": 32164,
74
+ "<a-165>": 32165,
75
+ "<a-166>": 32166,
76
+ "<a-167>": 32167,
77
+ "<a-168>": 32168,
78
+ "<a-169>": 32169,
79
+ "<a-16>": 32016,
80
+ "<a-170>": 32170,
81
+ "<a-171>": 32171,
82
+ "<a-172>": 32172,
83
+ "<a-173>": 32173,
84
+ "<a-174>": 32174,
85
+ "<a-175>": 32175,
86
+ "<a-176>": 32176,
87
+ "<a-177>": 32177,
88
+ "<a-178>": 32178,
89
+ "<a-179>": 32179,
90
+ "<a-17>": 32017,
91
+ "<a-180>": 32180,
92
+ "<a-181>": 32181,
93
+ "<a-182>": 32182,
94
+ "<a-183>": 32183,
95
+ "<a-184>": 32184,
96
+ "<a-185>": 32185,
97
+ "<a-186>": 32186,
98
+ "<a-187>": 32187,
99
+ "<a-188>": 32188,
100
+ "<a-189>": 32189,
101
+ "<a-18>": 32018,
102
+ "<a-190>": 32190,
103
+ "<a-191>": 32191,
104
+ "<a-192>": 32192,
105
+ "<a-193>": 32193,
106
+ "<a-194>": 32194,
107
+ "<a-195>": 32195,
108
+ "<a-196>": 32196,
109
+ "<a-197>": 32197,
110
+ "<a-198>": 32198,
111
+ "<a-199>": 32199,
112
+ "<a-19>": 32019,
113
+ "<a-1>": 32001,
114
+ "<a-200>": 32200,
115
+ "<a-201>": 32201,
116
+ "<a-202>": 32202,
117
+ "<a-203>": 32203,
118
+ "<a-204>": 32204,
119
+ "<a-205>": 32205,
120
+ "<a-206>": 32206,
121
+ "<a-207>": 32207,
122
+ "<a-208>": 32208,
123
+ "<a-209>": 32209,
124
+ "<a-20>": 32020,
125
+ "<a-210>": 32210,
126
+ "<a-211>": 32211,
127
+ "<a-212>": 32212,
128
+ "<a-213>": 32213,
129
+ "<a-214>": 32214,
130
+ "<a-215>": 32215,
131
+ "<a-216>": 32216,
132
+ "<a-217>": 32217,
133
+ "<a-218>": 32218,
134
+ "<a-219>": 32219,
135
+ "<a-21>": 32021,
136
+ "<a-220>": 32220,
137
+ "<a-221>": 32221,
138
+ "<a-222>": 32222,
139
+ "<a-223>": 32223,
140
+ "<a-224>": 32224,
141
+ "<a-225>": 32225,
142
+ "<a-226>": 32226,
143
+ "<a-227>": 32227,
144
+ "<a-228>": 32228,
145
+ "<a-229>": 32229,
146
+ "<a-22>": 32022,
147
+ "<a-230>": 32230,
148
+ "<a-231>": 32231,
149
+ "<a-232>": 32232,
150
+ "<a-233>": 32233,
151
+ "<a-234>": 32234,
152
+ "<a-235>": 32235,
153
+ "<a-236>": 32236,
154
+ "<a-237>": 32237,
155
+ "<a-238>": 32238,
156
+ "<a-239>": 32239,
157
+ "<a-23>": 32023,
158
+ "<a-240>": 32240,
159
+ "<a-241>": 32241,
160
+ "<a-242>": 32242,
161
+ "<a-243>": 32243,
162
+ "<a-244>": 32244,
163
+ "<a-245>": 32245,
164
+ "<a-246>": 32246,
165
+ "<a-247>": 32247,
166
+ "<a-248>": 32248,
167
+ "<a-249>": 32249,
168
+ "<a-24>": 32024,
169
+ "<a-250>": 32250,
170
+ "<a-251>": 32251,
171
+ "<a-252>": 32252,
172
+ "<a-253>": 32253,
173
+ "<a-254>": 32254,
174
+ "<a-255>": 32255,
175
+ "<a-25>": 32025,
176
+ "<a-26>": 32026,
177
+ "<a-27>": 32027,
178
+ "<a-28>": 32028,
179
+ "<a-29>": 32029,
180
+ "<a-2>": 32002,
181
+ "<a-30>": 32030,
182
+ "<a-31>": 32031,
183
+ "<a-32>": 32032,
184
+ "<a-33>": 32033,
185
+ "<a-34>": 32034,
186
+ "<a-35>": 32035,
187
+ "<a-36>": 32036,
188
+ "<a-37>": 32037,
189
+ "<a-38>": 32038,
190
+ "<a-39>": 32039,
191
+ "<a-3>": 32003,
192
+ "<a-40>": 32040,
193
+ "<a-41>": 32041,
194
+ "<a-42>": 32042,
195
+ "<a-43>": 32043,
196
+ "<a-44>": 32044,
197
+ "<a-45>": 32045,
198
+ "<a-46>": 32046,
199
+ "<a-47>": 32047,
200
+ "<a-48>": 32048,
201
+ "<a-49>": 32049,
202
+ "<a-4>": 32004,
203
+ "<a-50>": 32050,
204
+ "<a-51>": 32051,
205
+ "<a-52>": 32052,
206
+ "<a-53>": 32053,
207
+ "<a-54>": 32054,
208
+ "<a-55>": 32055,
209
+ "<a-56>": 32056,
210
+ "<a-57>": 32057,
211
+ "<a-58>": 32058,
212
+ "<a-59>": 32059,
213
+ "<a-5>": 32005,
214
+ "<a-60>": 32060,
215
+ "<a-61>": 32061,
216
+ "<a-62>": 32062,
217
+ "<a-63>": 32063,
218
+ "<a-64>": 32064,
219
+ "<a-65>": 32065,
220
+ "<a-66>": 32066,
221
+ "<a-67>": 32067,
222
+ "<a-68>": 32068,
223
+ "<a-69>": 32069,
224
+ "<a-6>": 32006,
225
+ "<a-70>": 32070,
226
+ "<a-71>": 32071,
227
+ "<a-72>": 32072,
228
+ "<a-73>": 32073,
229
+ "<a-74>": 32074,
230
+ "<a-75>": 32075,
231
+ "<a-76>": 32076,
232
+ "<a-77>": 32077,
233
+ "<a-78>": 32078,
234
+ "<a-79>": 32079,
235
+ "<a-7>": 32007,
236
+ "<a-80>": 32080,
237
+ "<a-81>": 32081,
238
+ "<a-82>": 32082,
239
+ "<a-83>": 32083,
240
+ "<a-84>": 32084,
241
+ "<a-85>": 32085,
242
+ "<a-86>": 32086,
243
+ "<a-87>": 32087,
244
+ "<a-88>": 32088,
245
+ "<a-89>": 32089,
246
+ "<a-8>": 32008,
247
+ "<a-90>": 32090,
248
+ "<a-91>": 32091,
249
+ "<a-92>": 32092,
250
+ "<a-93>": 32093,
251
+ "<a-94>": 32094,
252
+ "<a-95>": 32095,
253
+ "<a-96>": 32096,
254
+ "<a-97>": 32097,
255
+ "<a-98>": 32098,
256
+ "<a-99>": 32099,
257
+ "<a-9>": 32009,
258
+ "<b-0>": 32256,
259
+ "<b-100>": 32356,
260
+ "<b-101>": 32357,
261
+ "<b-102>": 32358,
262
+ "<b-103>": 32359,
263
+ "<b-104>": 32360,
264
+ "<b-105>": 32361,
265
+ "<b-106>": 32362,
266
+ "<b-107>": 32363,
267
+ "<b-108>": 32364,
268
+ "<b-109>": 32365,
269
+ "<b-10>": 32266,
270
+ "<b-110>": 32366,
271
+ "<b-111>": 32367,
272
+ "<b-112>": 32368,
273
+ "<b-113>": 32369,
274
+ "<b-114>": 32370,
275
+ "<b-115>": 32371,
276
+ "<b-116>": 32372,
277
+ "<b-117>": 32373,
278
+ "<b-118>": 32374,
279
+ "<b-119>": 32375,
280
+ "<b-11>": 32267,
281
+ "<b-120>": 32376,
282
+ "<b-121>": 32377,
283
+ "<b-122>": 32378,
284
+ "<b-123>": 32379,
285
+ "<b-124>": 32380,
286
+ "<b-125>": 32381,
287
+ "<b-126>": 32382,
288
+ "<b-127>": 32383,
289
+ "<b-128>": 32384,
290
+ "<b-129>": 32385,
291
+ "<b-12>": 32268,
292
+ "<b-130>": 32386,
293
+ "<b-131>": 32387,
294
+ "<b-132>": 32388,
295
+ "<b-133>": 32389,
296
+ "<b-134>": 32390,
297
+ "<b-135>": 32391,
298
+ "<b-136>": 32392,
299
+ "<b-137>": 32393,
300
+ "<b-138>": 32394,
301
+ "<b-139>": 32395,
302
+ "<b-13>": 32269,
303
+ "<b-140>": 32396,
304
+ "<b-141>": 32397,
305
+ "<b-142>": 32398,
306
+ "<b-143>": 32399,
307
+ "<b-144>": 32400,
308
+ "<b-145>": 32401,
309
+ "<b-146>": 32402,
310
+ "<b-147>": 32403,
311
+ "<b-148>": 32404,
312
+ "<b-149>": 32405,
313
+ "<b-14>": 32270,
314
+ "<b-150>": 32406,
315
+ "<b-151>": 32407,
316
+ "<b-152>": 32408,
317
+ "<b-153>": 32409,
318
+ "<b-154>": 32410,
319
+ "<b-155>": 32411,
320
+ "<b-156>": 32412,
321
+ "<b-157>": 32413,
322
+ "<b-158>": 32414,
323
+ "<b-159>": 32415,
324
+ "<b-15>": 32271,
325
+ "<b-160>": 32416,
326
+ "<b-161>": 32417,
327
+ "<b-162>": 32418,
328
+ "<b-163>": 32419,
329
+ "<b-164>": 32420,
330
+ "<b-165>": 32421,
331
+ "<b-166>": 32422,
332
+ "<b-167>": 32423,
333
+ "<b-168>": 32424,
334
+ "<b-169>": 32425,
335
+ "<b-16>": 32272,
336
+ "<b-170>": 32426,
337
+ "<b-171>": 32427,
338
+ "<b-172>": 32428,
339
+ "<b-173>": 32429,
340
+ "<b-174>": 32430,
341
+ "<b-175>": 32431,
342
+ "<b-176>": 32432,
343
+ "<b-177>": 32433,
344
+ "<b-178>": 32434,
345
+ "<b-179>": 32435,
346
+ "<b-17>": 32273,
347
+ "<b-180>": 32436,
348
+ "<b-181>": 32437,
349
+ "<b-182>": 32438,
350
+ "<b-183>": 32439,
351
+ "<b-184>": 32440,
352
+ "<b-185>": 32441,
353
+ "<b-186>": 32442,
354
+ "<b-187>": 32443,
355
+ "<b-188>": 32444,
356
+ "<b-189>": 32445,
357
+ "<b-18>": 32274,
358
+ "<b-190>": 32446,
359
+ "<b-191>": 32447,
360
+ "<b-192>": 32448,
361
+ "<b-193>": 32449,
362
+ "<b-194>": 32450,
363
+ "<b-195>": 32451,
364
+ "<b-196>": 32452,
365
+ "<b-197>": 32453,
366
+ "<b-198>": 32454,
367
+ "<b-199>": 32455,
368
+ "<b-19>": 32275,
369
+ "<b-1>": 32257,
370
+ "<b-200>": 32456,
371
+ "<b-201>": 32457,
372
+ "<b-202>": 32458,
373
+ "<b-203>": 32459,
374
+ "<b-204>": 32460,
375
+ "<b-205>": 32461,
376
+ "<b-206>": 32462,
377
+ "<b-207>": 32463,
378
+ "<b-208>": 32464,
379
+ "<b-209>": 32465,
380
+ "<b-20>": 32276,
381
+ "<b-210>": 32466,
382
+ "<b-211>": 32467,
383
+ "<b-212>": 32468,
384
+ "<b-213>": 32469,
385
+ "<b-214>": 32470,
386
+ "<b-215>": 32471,
387
+ "<b-216>": 32472,
388
+ "<b-217>": 32473,
389
+ "<b-218>": 32474,
390
+ "<b-219>": 32475,
391
+ "<b-21>": 32277,
392
+ "<b-220>": 32476,
393
+ "<b-221>": 32477,
394
+ "<b-222>": 32478,
395
+ "<b-223>": 32479,
396
+ "<b-224>": 32480,
397
+ "<b-225>": 32481,
398
+ "<b-226>": 32482,
399
+ "<b-227>": 32483,
400
+ "<b-228>": 32484,
401
+ "<b-229>": 32485,
402
+ "<b-22>": 32278,
403
+ "<b-230>": 32486,
404
+ "<b-231>": 32487,
405
+ "<b-232>": 32488,
406
+ "<b-233>": 32489,
407
+ "<b-234>": 32490,
408
+ "<b-235>": 32491,
409
+ "<b-236>": 32492,
410
+ "<b-237>": 32493,
411
+ "<b-238>": 32494,
412
+ "<b-239>": 32495,
413
+ "<b-23>": 32279,
414
+ "<b-240>": 32496,
415
+ "<b-241>": 32497,
416
+ "<b-242>": 32498,
417
+ "<b-243>": 32499,
418
+ "<b-244>": 32500,
419
+ "<b-245>": 32501,
420
+ "<b-246>": 32502,
421
+ "<b-247>": 32503,
422
+ "<b-248>": 32504,
423
+ "<b-249>": 32505,
424
+ "<b-24>": 32280,
425
+ "<b-250>": 32506,
426
+ "<b-251>": 32507,
427
+ "<b-252>": 32508,
428
+ "<b-253>": 32509,
429
+ "<b-254>": 32510,
430
+ "<b-255>": 32511,
431
+ "<b-25>": 32281,
432
+ "<b-26>": 32282,
433
+ "<b-27>": 32283,
434
+ "<b-28>": 32284,
435
+ "<b-29>": 32285,
436
+ "<b-2>": 32258,
437
+ "<b-30>": 32286,
438
+ "<b-31>": 32287,
439
+ "<b-32>": 32288,
440
+ "<b-33>": 32289,
441
+ "<b-34>": 32290,
442
+ "<b-35>": 32291,
443
+ "<b-36>": 32292,
444
+ "<b-37>": 32293,
445
+ "<b-38>": 32294,
446
+ "<b-39>": 32295,
447
+ "<b-3>": 32259,
448
+ "<b-40>": 32296,
449
+ "<b-41>": 32297,
450
+ "<b-42>": 32298,
451
+ "<b-43>": 32299,
452
+ "<b-44>": 32300,
453
+ "<b-45>": 32301,
454
+ "<b-46>": 32302,
455
+ "<b-47>": 32303,
456
+ "<b-48>": 32304,
457
+ "<b-49>": 32305,
458
+ "<b-4>": 32260,
459
+ "<b-50>": 32306,
460
+ "<b-51>": 32307,
461
+ "<b-52>": 32308,
462
+ "<b-53>": 32309,
463
+ "<b-54>": 32310,
464
+ "<b-55>": 32311,
465
+ "<b-56>": 32312,
466
+ "<b-57>": 32313,
467
+ "<b-58>": 32314,
468
+ "<b-59>": 32315,
469
+ "<b-5>": 32261,
470
+ "<b-60>": 32316,
471
+ "<b-61>": 32317,
472
+ "<b-62>": 32318,
473
+ "<b-63>": 32319,
474
+ "<b-64>": 32320,
475
+ "<b-65>": 32321,
476
+ "<b-66>": 32322,
477
+ "<b-67>": 32323,
478
+ "<b-68>": 32324,
479
+ "<b-69>": 32325,
480
+ "<b-6>": 32262,
481
+ "<b-70>": 32326,
482
+ "<b-71>": 32327,
483
+ "<b-72>": 32328,
484
+ "<b-73>": 32329,
485
+ "<b-74>": 32330,
486
+ "<b-75>": 32331,
487
+ "<b-76>": 32332,
488
+ "<b-77>": 32333,
489
+ "<b-78>": 32334,
490
+ "<b-79>": 32335,
491
+ "<b-7>": 32263,
492
+ "<b-80>": 32336,
493
+ "<b-81>": 32337,
494
+ "<b-82>": 32338,
495
+ "<b-83>": 32339,
496
+ "<b-84>": 32340,
497
+ "<b-85>": 32341,
498
+ "<b-86>": 32342,
499
+ "<b-87>": 32343,
500
+ "<b-88>": 32344,
501
+ "<b-89>": 32345,
502
+ "<b-8>": 32264,
503
+ "<b-90>": 32346,
504
+ "<b-91>": 32347,
505
+ "<b-92>": 32348,
506
+ "<b-93>": 32349,
507
+ "<b-94>": 32350,
508
+ "<b-95>": 32351,
509
+ "<b-96>": 32352,
510
+ "<b-97>": 32353,
511
+ "<b-98>": 32354,
512
+ "<b-99>": 32355,
513
+ "<b-9>": 32265,
514
+ "<c-0>": 32512,
515
+ "<c-100>": 32612,
516
+ "<c-101>": 32613,
517
+ "<c-102>": 32614,
518
+ "<c-103>": 32615,
519
+ "<c-104>": 32616,
520
+ "<c-105>": 32617,
521
+ "<c-106>": 32618,
522
+ "<c-107>": 32619,
523
+ "<c-108>": 32620,
524
+ "<c-109>": 32621,
525
+ "<c-10>": 32522,
526
+ "<c-110>": 32622,
527
+ "<c-111>": 32623,
528
+ "<c-112>": 32624,
529
+ "<c-113>": 32625,
530
+ "<c-114>": 32626,
531
+ "<c-115>": 32627,
532
+ "<c-116>": 32628,
533
+ "<c-117>": 32629,
534
+ "<c-118>": 32630,
535
+ "<c-119>": 32631,
536
+ "<c-11>": 32523,
537
+ "<c-120>": 32632,
538
+ "<c-121>": 32633,
539
+ "<c-122>": 32634,
540
+ "<c-123>": 32635,
541
+ "<c-124>": 32636,
542
+ "<c-125>": 32637,
543
+ "<c-126>": 32638,
544
+ "<c-127>": 32639,
545
+ "<c-128>": 32640,
546
+ "<c-129>": 32641,
547
+ "<c-12>": 32524,
548
+ "<c-130>": 32642,
549
+ "<c-131>": 32643,
550
+ "<c-132>": 32644,
551
+ "<c-133>": 32645,
552
+ "<c-134>": 32646,
553
+ "<c-135>": 32647,
554
+ "<c-136>": 32648,
555
+ "<c-137>": 32649,
556
+ "<c-138>": 32650,
557
+ "<c-139>": 32651,
558
+ "<c-13>": 32525,
559
+ "<c-140>": 32652,
560
+ "<c-141>": 32653,
561
+ "<c-142>": 32654,
562
+ "<c-143>": 32655,
563
+ "<c-144>": 32656,
564
+ "<c-145>": 32657,
565
+ "<c-146>": 32658,
566
+ "<c-147>": 32659,
567
+ "<c-148>": 32660,
568
+ "<c-149>": 32661,
569
+ "<c-14>": 32526,
570
+ "<c-150>": 32662,
571
+ "<c-151>": 32663,
572
+ "<c-152>": 32664,
573
+ "<c-153>": 32665,
574
+ "<c-154>": 32666,
575
+ "<c-155>": 32667,
576
+ "<c-156>": 32668,
577
+ "<c-157>": 32669,
578
+ "<c-158>": 32670,
579
+ "<c-159>": 32671,
580
+ "<c-15>": 32527,
581
+ "<c-160>": 32672,
582
+ "<c-161>": 32673,
583
+ "<c-162>": 32674,
584
+ "<c-163>": 32675,
585
+ "<c-164>": 32676,
586
+ "<c-165>": 32677,
587
+ "<c-166>": 32678,
588
+ "<c-167>": 32679,
589
+ "<c-168>": 32680,
590
+ "<c-169>": 32681,
591
+ "<c-16>": 32528,
592
+ "<c-170>": 32682,
593
+ "<c-171>": 32683,
594
+ "<c-172>": 32684,
595
+ "<c-173>": 32685,
596
+ "<c-174>": 32686,
597
+ "<c-175>": 32687,
598
+ "<c-176>": 32688,
599
+ "<c-177>": 32689,
600
+ "<c-178>": 32690,
601
+ "<c-179>": 32691,
602
+ "<c-17>": 32529,
603
+ "<c-180>": 32692,
604
+ "<c-181>": 32693,
605
+ "<c-182>": 32694,
606
+ "<c-183>": 32695,
607
+ "<c-184>": 32696,
608
+ "<c-185>": 32697,
609
+ "<c-186>": 32698,
610
+ "<c-187>": 32699,
611
+ "<c-188>": 32700,
612
+ "<c-189>": 32701,
613
+ "<c-18>": 32530,
614
+ "<c-190>": 32702,
615
+ "<c-191>": 32703,
616
+ "<c-192>": 32704,
617
+ "<c-193>": 32705,
618
+ "<c-194>": 32706,
619
+ "<c-195>": 32707,
620
+ "<c-196>": 32708,
621
+ "<c-197>": 32709,
622
+ "<c-198>": 32710,
623
+ "<c-199>": 32711,
624
+ "<c-19>": 32531,
625
+ "<c-1>": 32513,
626
+ "<c-200>": 32712,
627
+ "<c-201>": 32713,
628
+ "<c-202>": 32714,
629
+ "<c-203>": 32715,
630
+ "<c-204>": 32716,
631
+ "<c-205>": 32717,
632
+ "<c-206>": 32718,
633
+ "<c-207>": 32719,
634
+ "<c-208>": 32720,
635
+ "<c-209>": 32721,
636
+ "<c-20>": 32532,
637
+ "<c-210>": 32722,
638
+ "<c-211>": 32723,
639
+ "<c-212>": 32724,
640
+ "<c-213>": 32725,
641
+ "<c-214>": 32726,
642
+ "<c-215>": 32727,
643
+ "<c-216>": 32728,
644
+ "<c-217>": 32729,
645
+ "<c-218>": 32730,
646
+ "<c-219>": 32731,
647
+ "<c-21>": 32533,
648
+ "<c-220>": 32732,
649
+ "<c-221>": 32733,
650
+ "<c-222>": 32734,
651
+ "<c-223>": 32735,
652
+ "<c-224>": 32736,
653
+ "<c-225>": 32737,
654
+ "<c-226>": 32738,
655
+ "<c-227>": 32739,
656
+ "<c-228>": 32740,
657
+ "<c-229>": 32741,
658
+ "<c-22>": 32534,
659
+ "<c-230>": 32742,
660
+ "<c-231>": 32743,
661
+ "<c-232>": 32744,
662
+ "<c-233>": 32745,
663
+ "<c-234>": 32746,
664
+ "<c-235>": 32747,
665
+ "<c-236>": 32748,
666
+ "<c-237>": 32749,
667
+ "<c-238>": 32750,
668
+ "<c-239>": 32751,
669
+ "<c-23>": 32535,
670
+ "<c-240>": 32752,
671
+ "<c-241>": 32753,
672
+ "<c-242>": 32754,
673
+ "<c-243>": 32755,
674
+ "<c-244>": 32756,
675
+ "<c-245>": 32757,
676
+ "<c-246>": 32758,
677
+ "<c-247>": 32759,
678
+ "<c-248>": 32760,
679
+ "<c-249>": 32761,
680
+ "<c-24>": 32536,
681
+ "<c-250>": 32762,
682
+ "<c-251>": 32763,
683
+ "<c-252>": 32764,
684
+ "<c-253>": 32765,
685
+ "<c-254>": 32766,
686
+ "<c-255>": 32767,
687
+ "<c-25>": 32537,
688
+ "<c-26>": 32538,
689
+ "<c-27>": 32539,
690
+ "<c-28>": 32540,
691
+ "<c-29>": 32541,
692
+ "<c-2>": 32514,
693
+ "<c-30>": 32542,
694
+ "<c-31>": 32543,
695
+ "<c-32>": 32544,
696
+ "<c-33>": 32545,
697
+ "<c-34>": 32546,
698
+ "<c-35>": 32547,
699
+ "<c-36>": 32548,
700
+ "<c-37>": 32549,
701
+ "<c-38>": 32550,
702
+ "<c-39>": 32551,
703
+ "<c-3>": 32515,
704
+ "<c-40>": 32552,
705
+ "<c-41>": 32553,
706
+ "<c-42>": 32554,
707
+ "<c-43>": 32555,
708
+ "<c-44>": 32556,
709
+ "<c-45>": 32557,
710
+ "<c-46>": 32558,
711
+ "<c-47>": 32559,
712
+ "<c-48>": 32560,
713
+ "<c-49>": 32561,
714
+ "<c-4>": 32516,
715
+ "<c-50>": 32562,
716
+ "<c-51>": 32563,
717
+ "<c-52>": 32564,
718
+ "<c-53>": 32565,
719
+ "<c-54>": 32566,
720
+ "<c-55>": 32567,
721
+ "<c-56>": 32568,
722
+ "<c-57>": 32569,
723
+ "<c-58>": 32570,
724
+ "<c-59>": 32571,
725
+ "<c-5>": 32517,
726
+ "<c-60>": 32572,
727
+ "<c-61>": 32573,
728
+ "<c-62>": 32574,
729
+ "<c-63>": 32575,
730
+ "<c-64>": 32576,
731
+ "<c-65>": 32577,
732
+ "<c-66>": 32578,
733
+ "<c-67>": 32579,
734
+ "<c-68>": 32580,
735
+ "<c-69>": 32581,
736
+ "<c-6>": 32518,
737
+ "<c-70>": 32582,
738
+ "<c-71>": 32583,
739
+ "<c-72>": 32584,
740
+ "<c-73>": 32585,
741
+ "<c-74>": 32586,
742
+ "<c-75>": 32587,
743
+ "<c-76>": 32588,
744
+ "<c-77>": 32589,
745
+ "<c-78>": 32590,
746
+ "<c-79>": 32591,
747
+ "<c-7>": 32519,
748
+ "<c-80>": 32592,
749
+ "<c-81>": 32593,
750
+ "<c-82>": 32594,
751
+ "<c-83>": 32595,
752
+ "<c-84>": 32596,
753
+ "<c-85>": 32597,
754
+ "<c-86>": 32598,
755
+ "<c-87>": 32599,
756
+ "<c-88>": 32600,
757
+ "<c-89>": 32601,
758
+ "<c-8>": 32520,
759
+ "<c-90>": 32602,
760
+ "<c-91>": 32603,
761
+ "<c-92>": 32604,
762
+ "<c-93>": 32605,
763
+ "<c-94>": 32606,
764
+ "<c-95>": 32607,
765
+ "<c-96>": 32608,
766
+ "<c-97>": 32609,
767
+ "<c-98>": 32610,
768
+ "<c-99>": 32611,
769
+ "<c-9>": 32521,
770
+ "<d-0>": 32768,
771
+ "<d-100>": 32868,
772
+ "<d-101>": 32869,
773
+ "<d-102>": 32870,
774
+ "<d-103>": 32871,
775
+ "<d-104>": 32872,
776
+ "<d-105>": 32873,
777
+ "<d-106>": 32874,
778
+ "<d-107>": 32875,
779
+ "<d-108>": 32876,
780
+ "<d-109>": 32877,
781
+ "<d-10>": 32778,
782
+ "<d-110>": 32878,
783
+ "<d-111>": 32879,
784
+ "<d-112>": 32880,
785
+ "<d-113>": 32881,
786
+ "<d-114>": 32882,
787
+ "<d-115>": 32883,
788
+ "<d-116>": 32884,
789
+ "<d-117>": 32885,
790
+ "<d-118>": 32886,
791
+ "<d-119>": 32887,
792
+ "<d-11>": 32779,
793
+ "<d-120>": 32888,
794
+ "<d-121>": 32889,
795
+ "<d-122>": 32890,
796
+ "<d-123>": 32891,
797
+ "<d-124>": 32892,
798
+ "<d-125>": 32893,
799
+ "<d-126>": 32894,
800
+ "<d-127>": 32895,
801
+ "<d-128>": 32896,
802
+ "<d-129>": 32897,
803
+ "<d-12>": 32780,
804
+ "<d-130>": 32898,
805
+ "<d-131>": 32899,
806
+ "<d-132>": 32900,
807
+ "<d-133>": 32901,
808
+ "<d-134>": 32902,
809
+ "<d-135>": 32903,
810
+ "<d-136>": 32904,
811
+ "<d-137>": 32905,
812
+ "<d-138>": 32906,
813
+ "<d-139>": 32907,
814
+ "<d-13>": 32781,
815
+ "<d-140>": 32908,
816
+ "<d-141>": 32909,
817
+ "<d-142>": 32910,
818
+ "<d-143>": 32911,
819
+ "<d-144>": 32912,
820
+ "<d-145>": 32913,
821
+ "<d-146>": 32914,
822
+ "<d-147>": 32915,
823
+ "<d-148>": 32916,
824
+ "<d-149>": 32917,
825
+ "<d-14>": 32782,
826
+ "<d-150>": 32918,
827
+ "<d-151>": 32919,
828
+ "<d-152>": 32920,
829
+ "<d-153>": 32921,
830
+ "<d-154>": 32922,
831
+ "<d-155>": 32923,
832
+ "<d-156>": 32924,
833
+ "<d-157>": 32925,
834
+ "<d-158>": 32926,
835
+ "<d-159>": 32927,
836
+ "<d-15>": 32783,
837
+ "<d-160>": 32928,
838
+ "<d-161>": 32929,
839
+ "<d-162>": 32930,
840
+ "<d-163>": 32931,
841
+ "<d-164>": 32932,
842
+ "<d-165>": 32933,
843
+ "<d-166>": 32934,
844
+ "<d-167>": 32935,
845
+ "<d-168>": 32936,
846
+ "<d-169>": 32937,
847
+ "<d-16>": 32784,
848
+ "<d-170>": 32938,
849
+ "<d-171>": 32939,
850
+ "<d-172>": 32940,
851
+ "<d-173>": 32941,
852
+ "<d-174>": 32942,
853
+ "<d-175>": 32943,
854
+ "<d-176>": 32944,
855
+ "<d-177>": 32945,
856
+ "<d-178>": 32946,
857
+ "<d-179>": 32947,
858
+ "<d-17>": 32785,
859
+ "<d-180>": 32948,
860
+ "<d-181>": 32949,
861
+ "<d-182>": 32950,
862
+ "<d-183>": 32951,
863
+ "<d-184>": 32952,
864
+ "<d-185>": 32953,
865
+ "<d-186>": 32954,
866
+ "<d-187>": 32955,
867
+ "<d-188>": 32956,
868
+ "<d-189>": 32957,
869
+ "<d-18>": 32786,
870
+ "<d-190>": 32958,
871
+ "<d-191>": 32959,
872
+ "<d-192>": 32960,
873
+ "<d-193>": 32961,
874
+ "<d-194>": 32962,
875
+ "<d-195>": 32963,
876
+ "<d-196>": 32964,
877
+ "<d-197>": 32965,
878
+ "<d-198>": 32966,
879
+ "<d-199>": 32967,
880
+ "<d-19>": 32787,
881
+ "<d-1>": 32769,
882
+ "<d-200>": 32968,
883
+ "<d-201>": 32969,
884
+ "<d-202>": 32970,
885
+ "<d-203>": 32971,
886
+ "<d-204>": 32972,
887
+ "<d-205>": 32973,
888
+ "<d-206>": 32974,
889
+ "<d-207>": 32975,
890
+ "<d-208>": 32976,
891
+ "<d-209>": 32977,
892
+ "<d-20>": 32788,
893
+ "<d-210>": 32978,
894
+ "<d-211>": 32979,
895
+ "<d-212>": 32980,
896
+ "<d-213>": 32981,
897
+ "<d-214>": 32982,
898
+ "<d-215>": 32983,
899
+ "<d-216>": 32984,
900
+ "<d-217>": 32985,
901
+ "<d-218>": 32986,
902
+ "<d-219>": 32987,
903
+ "<d-21>": 32789,
904
+ "<d-220>": 32988,
905
+ "<d-221>": 32989,
906
+ "<d-222>": 32990,
907
+ "<d-223>": 32991,
908
+ "<d-224>": 32992,
909
+ "<d-225>": 32993,
910
+ "<d-226>": 32994,
911
+ "<d-227>": 32995,
912
+ "<d-228>": 32996,
913
+ "<d-229>": 32997,
914
+ "<d-22>": 32790,
915
+ "<d-230>": 32998,
916
+ "<d-231>": 32999,
917
+ "<d-232>": 33000,
918
+ "<d-233>": 33001,
919
+ "<d-234>": 33002,
920
+ "<d-235>": 33003,
921
+ "<d-236>": 33004,
922
+ "<d-237>": 33005,
923
+ "<d-238>": 33006,
924
+ "<d-239>": 33007,
925
+ "<d-23>": 32791,
926
+ "<d-240>": 33008,
927
+ "<d-241>": 33009,
928
+ "<d-242>": 33010,
929
+ "<d-243>": 33011,
930
+ "<d-244>": 33012,
931
+ "<d-245>": 33013,
932
+ "<d-246>": 33014,
933
+ "<d-247>": 33015,
934
+ "<d-248>": 33016,
935
+ "<d-249>": 33017,
936
+ "<d-24>": 32792,
937
+ "<d-250>": 33018,
938
+ "<d-251>": 33019,
939
+ "<d-252>": 33020,
940
+ "<d-253>": 33021,
941
+ "<d-254>": 33022,
942
+ "<d-255>": 33023,
943
+ "<d-25>": 32793,
944
+ "<d-26>": 32794,
945
+ "<d-27>": 32795,
946
+ "<d-28>": 32796,
947
+ "<d-29>": 32797,
948
+ "<d-2>": 32770,
949
+ "<d-30>": 32798,
950
+ "<d-31>": 32799,
951
+ "<d-32>": 32800,
952
+ "<d-33>": 32801,
953
+ "<d-34>": 32802,
954
+ "<d-35>": 32803,
955
+ "<d-36>": 32804,
956
+ "<d-37>": 32805,
957
+ "<d-38>": 32806,
958
+ "<d-39>": 32807,
959
+ "<d-3>": 32771,
960
+ "<d-40>": 32808,
961
+ "<d-41>": 32809,
962
+ "<d-42>": 32810,
963
+ "<d-43>": 32811,
964
+ "<d-44>": 32812,
965
+ "<d-45>": 32813,
966
+ "<d-46>": 32814,
967
+ "<d-47>": 32815,
968
+ "<d-48>": 32816,
969
+ "<d-49>": 32817,
970
+ "<d-4>": 32772,
971
+ "<d-50>": 32818,
972
+ "<d-51>": 32819,
973
+ "<d-52>": 32820,
974
+ "<d-53>": 32821,
975
+ "<d-54>": 32822,
976
+ "<d-55>": 32823,
977
+ "<d-56>": 32824,
978
+ "<d-57>": 32825,
979
+ "<d-58>": 32826,
980
+ "<d-59>": 32827,
981
+ "<d-5>": 32773,
982
+ "<d-60>": 32828,
983
+ "<d-61>": 32829,
984
+ "<d-62>": 32830,
985
+ "<d-63>": 32831,
986
+ "<d-64>": 32832,
987
+ "<d-65>": 32833,
988
+ "<d-66>": 32834,
989
+ "<d-67>": 32835,
990
+ "<d-68>": 32836,
991
+ "<d-69>": 32837,
992
+ "<d-6>": 32774,
993
+ "<d-70>": 32838,
994
+ "<d-71>": 32839,
995
+ "<d-72>": 32840,
996
+ "<d-73>": 32841,
997
+ "<d-74>": 32842,
998
+ "<d-75>": 32843,
999
+ "<d-76>": 32844,
1000
+ "<d-77>": 32845,
1001
+ "<d-78>": 32846,
1002
+ "<d-79>": 32847,
1003
+ "<d-7>": 32775,
1004
+ "<d-80>": 32848,
1005
+ "<d-81>": 32849,
1006
+ "<d-82>": 32850,
1007
+ "<d-83>": 32851,
1008
+ "<d-84>": 32852,
1009
+ "<d-85>": 32853,
1010
+ "<d-86>": 32854,
1011
+ "<d-87>": 32855,
1012
+ "<d-88>": 32856,
1013
+ "<d-89>": 32857,
1014
+ "<d-8>": 32776,
1015
+ "<d-90>": 32858,
1016
+ "<d-91>": 32859,
1017
+ "<d-92>": 32860,
1018
+ "<d-93>": 32861,
1019
+ "<d-94>": 32862,
1020
+ "<d-95>": 32863,
1021
+ "<d-96>": 32864,
1022
+ "<d-97>": 32865,
1023
+ "<d-98>": 32866,
1024
+ "<d-99>": 32867,
1025
+ "<d-9>": 32777
1026
+ }
Ins/finetune/eval_result.json ADDED
@@ -0,0 +1,47 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "test_prompt_ids": "all",
3
+ "mean_results": {
4
+ "hit@1": 0.06059260455352818,
5
+ "hit@5": 0.08478658700683567,
6
+ "hit@10": 0.10541471553904946,
7
+ "ndcg@5": 0.07265244992899023,
8
+ "ndcg@10": 0.07924828323692125
9
+ },
10
+ "min_results": {
11
+ "hit@1": 0.06055223639593089,
12
+ "hit@5": 0.08436944937833037,
13
+ "hit@10": 0.10528015501372517,
14
+ "ndcg@5": 0.07247135835974591,
15
+ "ndcg@10": 0.07913786546067342
16
+ },
17
+ "max_results": {
18
+ "hit@1": 0.06067334086872275,
19
+ "hit@5": 0.0852171806878734,
20
+ "hit@10": 0.10560310027450347,
21
+ "ndcg@5": 0.07292103606730017,
22
+ "ndcg@10": 0.079352714069041
23
+ },
24
+ "all_prompt_results": [
25
+ {
26
+ "hit@1": 0.06055223639593089,
27
+ "hit@5": 0.08477313095430325,
28
+ "hit@10": 0.10528015501372517,
29
+ "ndcg@5": 0.07256495535992459,
30
+ "ndcg@10": 0.07913786546067342
31
+ },
32
+ {
33
+ "hit@1": 0.06055223639593089,
34
+ "hit@5": 0.08436944937833037,
35
+ "hit@10": 0.10560310027450347,
36
+ "ndcg@5": 0.07247135835974591,
37
+ "ndcg@10": 0.07925427018104932
38
+ },
39
+ {
40
+ "hit@1": 0.06067334086872275,
41
+ "hit@5": 0.0852171806878734,
42
+ "hit@10": 0.10536089132891975,
43
+ "ndcg@5": 0.07292103606730017,
44
+ "ndcg@10": 0.079352714069041
45
+ }
46
+ ]
47
+ }
Ins/finetune/log.txt ADDED
The diff for this file is too large to render. See raw diff
 
Ins/finetune/special_tokens_map.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<s>",
4
+ "lstrip": false,
5
+ "normalized": true,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "eos_token": {
10
+ "content": "</s>",
11
+ "lstrip": false,
12
+ "normalized": true,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "pad_token": "<unk>",
17
+ "unk_token": {
18
+ "content": "<unk>",
19
+ "lstrip": false,
20
+ "normalized": true,
21
+ "rstrip": false,
22
+ "single_word": false
23
+ }
24
+ }
Ins/finetune/tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff
 
Ins/finetune/trainer_state.json ADDED
@@ -0,0 +1,3682 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 0.9999039292919589,
5
+ "eval_steps": 1000,
6
+ "global_step": 5204,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "epoch": 0.0019214141608223652,
13
+ "grad_norm": 2.4305918216705322,
14
+ "learning_rate": 9.433962264150944e-05,
15
+ "loss": 0.837,
16
+ "step": 10
17
+ },
18
+ {
19
+ "epoch": 0.0038428283216447303,
20
+ "grad_norm": 1.3495147228240967,
21
+ "learning_rate": 0.00018867924528301889,
22
+ "loss": 0.7681,
23
+ "step": 20
24
+ },
25
+ {
26
+ "epoch": 0.005764242482467096,
27
+ "grad_norm": 0.7792263627052307,
28
+ "learning_rate": 0.0002830188679245283,
29
+ "loss": 0.7964,
30
+ "step": 30
31
+ },
32
+ {
33
+ "epoch": 0.007685656643289461,
34
+ "grad_norm": 2.0215744972229004,
35
+ "learning_rate": 0.00037735849056603777,
36
+ "loss": 0.8555,
37
+ "step": 40
38
+ },
39
+ {
40
+ "epoch": 0.009607070804111826,
41
+ "grad_norm": 0.9997712969779968,
42
+ "learning_rate": 0.0004716981132075472,
43
+ "loss": 0.973,
44
+ "step": 50
45
+ },
46
+ {
47
+ "epoch": 0.011528484964934192,
48
+ "grad_norm": 1.6805115938186646,
49
+ "learning_rate": 0.0004999977216414076,
50
+ "loss": 0.8087,
51
+ "step": 60
52
+ },
53
+ {
54
+ "epoch": 0.013449899125756557,
55
+ "grad_norm": 1.888655424118042,
56
+ "learning_rate": 0.0004999865624339867,
57
+ "loss": 0.8988,
58
+ "step": 70
59
+ },
60
+ {
61
+ "epoch": 0.015371313286578921,
62
+ "grad_norm": 1.7796673774719238,
63
+ "learning_rate": 0.0004999661043182869,
64
+ "loss": 0.9576,
65
+ "step": 80
66
+ },
67
+ {
68
+ "epoch": 0.017292727447401287,
69
+ "grad_norm": 1.4261183738708496,
70
+ "learning_rate": 0.000499936348055302,
71
+ "loss": 0.9808,
72
+ "step": 90
73
+ },
74
+ {
75
+ "epoch": 0.019214141608223653,
76
+ "grad_norm": 1.119388461112976,
77
+ "learning_rate": 0.0004998972947518943,
78
+ "loss": 0.9937,
79
+ "step": 100
80
+ },
81
+ {
82
+ "epoch": 0.021135555769046018,
83
+ "grad_norm": 1.7162309885025024,
84
+ "learning_rate": 0.0004998489458607546,
85
+ "loss": 0.8548,
86
+ "step": 110
87
+ },
88
+ {
89
+ "epoch": 0.023056969929868384,
90
+ "grad_norm": 2.1326487064361572,
91
+ "learning_rate": 0.0004997913031803468,
92
+ "loss": 1.0711,
93
+ "step": 120
94
+ },
95
+ {
96
+ "epoch": 0.02497838409069075,
97
+ "grad_norm": 2.0284371376037598,
98
+ "learning_rate": 0.0004997243688548423,
99
+ "loss": 1.1614,
100
+ "step": 130
101
+ },
102
+ {
103
+ "epoch": 0.026899798251513115,
104
+ "grad_norm": 1.6004656553268433,
105
+ "learning_rate": 0.0004996481453740396,
106
+ "loss": 0.9902,
107
+ "step": 140
108
+ },
109
+ {
110
+ "epoch": 0.02882121241233548,
111
+ "grad_norm": 2.1148738861083984,
112
+ "learning_rate": 0.0004995626355732716,
113
+ "loss": 0.9928,
114
+ "step": 150
115
+ },
116
+ {
117
+ "epoch": 0.030742626573157843,
118
+ "grad_norm": 1.4066400527954102,
119
+ "learning_rate": 0.0004994678426333004,
120
+ "loss": 1.0203,
121
+ "step": 160
122
+ },
123
+ {
124
+ "epoch": 0.03266404073398021,
125
+ "grad_norm": 2.228640079498291,
126
+ "learning_rate": 0.0004993637700801992,
127
+ "loss": 0.8989,
128
+ "step": 170
129
+ },
130
+ {
131
+ "epoch": 0.034585454894802574,
132
+ "grad_norm": 1.9775664806365967,
133
+ "learning_rate": 0.0004992504217852204,
134
+ "loss": 0.9383,
135
+ "step": 180
136
+ },
137
+ {
138
+ "epoch": 0.03650686905562494,
139
+ "grad_norm": 1.725915551185608,
140
+ "learning_rate": 0.0004991278019646523,
141
+ "loss": 0.8669,
142
+ "step": 190
143
+ },
144
+ {
145
+ "epoch": 0.038428283216447305,
146
+ "grad_norm": 1.2805640697479248,
147
+ "learning_rate": 0.0004989959151796617,
148
+ "loss": 0.9812,
149
+ "step": 200
150
+ },
151
+ {
152
+ "epoch": 0.04034969737726967,
153
+ "grad_norm": 1.4963750839233398,
154
+ "learning_rate": 0.0004988547663361251,
155
+ "loss": 1.0042,
156
+ "step": 210
157
+ },
158
+ {
159
+ "epoch": 0.042271111538092036,
160
+ "grad_norm": 2.081295967102051,
161
+ "learning_rate": 0.000498704360684445,
162
+ "loss": 0.9171,
163
+ "step": 220
164
+ },
165
+ {
166
+ "epoch": 0.0441925256989144,
167
+ "grad_norm": 1.5910736322402954,
168
+ "learning_rate": 0.0004985447038193558,
169
+ "loss": 1.0254,
170
+ "step": 230
171
+ },
172
+ {
173
+ "epoch": 0.04611393985973677,
174
+ "grad_norm": 1.3932862281799316,
175
+ "learning_rate": 0.0004983758016797147,
176
+ "loss": 0.9257,
177
+ "step": 240
178
+ },
179
+ {
180
+ "epoch": 0.04803535402055913,
181
+ "grad_norm": 1.7471644878387451,
182
+ "learning_rate": 0.0004981976605482817,
183
+ "loss": 0.95,
184
+ "step": 250
185
+ },
186
+ {
187
+ "epoch": 0.0499567681813815,
188
+ "grad_norm": 1.6283310651779175,
189
+ "learning_rate": 0.0004980102870514847,
190
+ "loss": 1.0308,
191
+ "step": 260
192
+ },
193
+ {
194
+ "epoch": 0.051878182342203864,
195
+ "grad_norm": 3.03869366645813,
196
+ "learning_rate": 0.0004978136881591746,
197
+ "loss": 1.118,
198
+ "step": 270
199
+ },
200
+ {
201
+ "epoch": 0.05379959650302623,
202
+ "grad_norm": 1.8459982872009277,
203
+ "learning_rate": 0.0004976078711843645,
204
+ "loss": 1.0017,
205
+ "step": 280
206
+ },
207
+ {
208
+ "epoch": 0.055721010663848596,
209
+ "grad_norm": 2.3289153575897217,
210
+ "learning_rate": 0.0004973928437829586,
211
+ "loss": 1.0451,
212
+ "step": 290
213
+ },
214
+ {
215
+ "epoch": 0.05764242482467096,
216
+ "grad_norm": 2.016827344894409,
217
+ "learning_rate": 0.0004971686139534673,
218
+ "loss": 1.0557,
219
+ "step": 300
220
+ },
221
+ {
222
+ "epoch": 0.05956383898549332,
223
+ "grad_norm": 2.2310760021209717,
224
+ "learning_rate": 0.0004969351900367092,
225
+ "loss": 1.0835,
226
+ "step": 310
227
+ },
228
+ {
229
+ "epoch": 0.061485253146315685,
230
+ "grad_norm": 2.030564308166504,
231
+ "learning_rate": 0.0004966925807155016,
232
+ "loss": 0.984,
233
+ "step": 320
234
+ },
235
+ {
236
+ "epoch": 0.06340666730713805,
237
+ "grad_norm": 2.2611382007598877,
238
+ "learning_rate": 0.0004964407950143367,
239
+ "loss": 0.9892,
240
+ "step": 330
241
+ },
242
+ {
243
+ "epoch": 0.06532808146796042,
244
+ "grad_norm": 1.6385694742202759,
245
+ "learning_rate": 0.0004961798422990465,
246
+ "loss": 1.0289,
247
+ "step": 340
248
+ },
249
+ {
250
+ "epoch": 0.06724949562878278,
251
+ "grad_norm": 2.0164542198181152,
252
+ "learning_rate": 0.0004959097322764543,
253
+ "loss": 1.0196,
254
+ "step": 350
255
+ },
256
+ {
257
+ "epoch": 0.06917090978960515,
258
+ "grad_norm": 2.2265095710754395,
259
+ "learning_rate": 0.0004956304749940134,
260
+ "loss": 1.009,
261
+ "step": 360
262
+ },
263
+ {
264
+ "epoch": 0.07109232395042751,
265
+ "grad_norm": 2.171071767807007,
266
+ "learning_rate": 0.0004953420808394334,
267
+ "loss": 1.0981,
268
+ "step": 370
269
+ },
270
+ {
271
+ "epoch": 0.07301373811124988,
272
+ "grad_norm": 2.4532387256622314,
273
+ "learning_rate": 0.0004950445605402943,
274
+ "loss": 1.1436,
275
+ "step": 380
276
+ },
277
+ {
278
+ "epoch": 0.07493515227207224,
279
+ "grad_norm": 1.9529705047607422,
280
+ "learning_rate": 0.0004947379251636468,
281
+ "loss": 1.0647,
282
+ "step": 390
283
+ },
284
+ {
285
+ "epoch": 0.07685656643289461,
286
+ "grad_norm": 2.0442216396331787,
287
+ "learning_rate": 0.000494422186115601,
288
+ "loss": 0.9467,
289
+ "step": 400
290
+ },
291
+ {
292
+ "epoch": 0.07877798059371698,
293
+ "grad_norm": 1.7892729043960571,
294
+ "learning_rate": 0.0004940973551409018,
295
+ "loss": 1.0112,
296
+ "step": 410
297
+ },
298
+ {
299
+ "epoch": 0.08069939475453934,
300
+ "grad_norm": 1.5844688415527344,
301
+ "learning_rate": 0.0004937634443224925,
302
+ "loss": 0.9157,
303
+ "step": 420
304
+ },
305
+ {
306
+ "epoch": 0.08262080891536171,
307
+ "grad_norm": 1.737804651260376,
308
+ "learning_rate": 0.0004934204660810651,
309
+ "loss": 0.9012,
310
+ "step": 430
311
+ },
312
+ {
313
+ "epoch": 0.08454222307618407,
314
+ "grad_norm": 1.868697166442871,
315
+ "learning_rate": 0.000493068433174598,
316
+ "loss": 0.9935,
317
+ "step": 440
318
+ },
319
+ {
320
+ "epoch": 0.08646363723700644,
321
+ "grad_norm": 2.2820777893066406,
322
+ "learning_rate": 0.000492707358697882,
323
+ "loss": 1.1136,
324
+ "step": 450
325
+ },
326
+ {
327
+ "epoch": 0.0883850513978288,
328
+ "grad_norm": 1.7056925296783447,
329
+ "learning_rate": 0.000492337256082033,
330
+ "loss": 1.096,
331
+ "step": 460
332
+ },
333
+ {
334
+ "epoch": 0.09030646555865117,
335
+ "grad_norm": 1.9547524452209473,
336
+ "learning_rate": 0.0004919581390939917,
337
+ "loss": 0.9342,
338
+ "step": 470
339
+ },
340
+ {
341
+ "epoch": 0.09222787971947354,
342
+ "grad_norm": 1.2121398448944092,
343
+ "learning_rate": 0.0004915700218360126,
344
+ "loss": 1.062,
345
+ "step": 480
346
+ },
347
+ {
348
+ "epoch": 0.0941492938802959,
349
+ "grad_norm": 2.3924214839935303,
350
+ "learning_rate": 0.0004911729187451389,
351
+ "loss": 0.9367,
352
+ "step": 490
353
+ },
354
+ {
355
+ "epoch": 0.09607070804111827,
356
+ "grad_norm": 1.8371268510818481,
357
+ "learning_rate": 0.0004907668445926656,
358
+ "loss": 0.9993,
359
+ "step": 500
360
+ },
361
+ {
362
+ "epoch": 0.09799212220194063,
363
+ "grad_norm": 1.3057355880737305,
364
+ "learning_rate": 0.0004903518144835895,
365
+ "loss": 1.0558,
366
+ "step": 510
367
+ },
368
+ {
369
+ "epoch": 0.099913536362763,
370
+ "grad_norm": 1.72185480594635,
371
+ "learning_rate": 0.0004899278438560481,
372
+ "loss": 1.0392,
373
+ "step": 520
374
+ },
375
+ {
376
+ "epoch": 0.10183495052358536,
377
+ "grad_norm": 2.33323335647583,
378
+ "learning_rate": 0.0004894949484807453,
379
+ "loss": 1.0486,
380
+ "step": 530
381
+ },
382
+ {
383
+ "epoch": 0.10375636468440773,
384
+ "grad_norm": 2.24397349357605,
385
+ "learning_rate": 0.0004890531444603637,
386
+ "loss": 1.1116,
387
+ "step": 540
388
+ },
389
+ {
390
+ "epoch": 0.1056777788452301,
391
+ "grad_norm": 1.5562959909439087,
392
+ "learning_rate": 0.0004886024482289674,
393
+ "loss": 0.9655,
394
+ "step": 550
395
+ },
396
+ {
397
+ "epoch": 0.10759919300605246,
398
+ "grad_norm": 1.6606842279434204,
399
+ "learning_rate": 0.00048814287655138877,
400
+ "loss": 0.9895,
401
+ "step": 560
402
+ },
403
+ {
404
+ "epoch": 0.10952060716687483,
405
+ "grad_norm": 1.591304898262024,
406
+ "learning_rate": 0.00048767444652260644,
407
+ "loss": 1.0343,
408
+ "step": 570
409
+ },
410
+ {
411
+ "epoch": 0.11144202132769719,
412
+ "grad_norm": 1.592618465423584,
413
+ "learning_rate": 0.0004871971755671084,
414
+ "loss": 1.0607,
415
+ "step": 580
416
+ },
417
+ {
418
+ "epoch": 0.11336343548851956,
419
+ "grad_norm": 1.6015795469284058,
420
+ "learning_rate": 0.00048671108143824414,
421
+ "loss": 1.1387,
422
+ "step": 590
423
+ },
424
+ {
425
+ "epoch": 0.11528484964934192,
426
+ "grad_norm": 2.1288461685180664,
427
+ "learning_rate": 0.0004862161822175646,
428
+ "loss": 1.0174,
429
+ "step": 600
430
+ },
431
+ {
432
+ "epoch": 0.11720626381016427,
433
+ "grad_norm": 2.2891428470611572,
434
+ "learning_rate": 0.0004857124963141489,
435
+ "loss": 1.0713,
436
+ "step": 610
437
+ },
438
+ {
439
+ "epoch": 0.11912767797098664,
440
+ "grad_norm": 2.3416545391082764,
441
+ "learning_rate": 0.0004852000424639205,
442
+ "loss": 0.9538,
443
+ "step": 620
444
+ },
445
+ {
446
+ "epoch": 0.121049092131809,
447
+ "grad_norm": 1.7260591983795166,
448
+ "learning_rate": 0.00048467883972894897,
449
+ "loss": 1.0152,
450
+ "step": 630
451
+ },
452
+ {
453
+ "epoch": 0.12297050629263137,
454
+ "grad_norm": 2.208760976791382,
455
+ "learning_rate": 0.00048414890749674247,
456
+ "loss": 0.9158,
457
+ "step": 640
458
+ },
459
+ {
460
+ "epoch": 0.12489192045345374,
461
+ "grad_norm": 1.8179646730422974,
462
+ "learning_rate": 0.0004836102654795252,
463
+ "loss": 0.9743,
464
+ "step": 650
465
+ },
466
+ {
467
+ "epoch": 0.1268133346142761,
468
+ "grad_norm": 1.9145281314849854,
469
+ "learning_rate": 0.0004830629337135051,
470
+ "loss": 1.0373,
471
+ "step": 660
472
+ },
473
+ {
474
+ "epoch": 0.12873474877509847,
475
+ "grad_norm": 1.7451472282409668,
476
+ "learning_rate": 0.00048250693255812795,
477
+ "loss": 1.03,
478
+ "step": 670
479
+ },
480
+ {
481
+ "epoch": 0.13065616293592083,
482
+ "grad_norm": 1.9409160614013672,
483
+ "learning_rate": 0.0004819422826953204,
484
+ "loss": 1.0401,
485
+ "step": 680
486
+ },
487
+ {
488
+ "epoch": 0.1325775770967432,
489
+ "grad_norm": 1.6609559059143066,
490
+ "learning_rate": 0.00048136900512872063,
491
+ "loss": 1.0893,
492
+ "step": 690
493
+ },
494
+ {
495
+ "epoch": 0.13449899125756556,
496
+ "grad_norm": 1.9034476280212402,
497
+ "learning_rate": 0.0004807871211828969,
498
+ "loss": 0.9952,
499
+ "step": 700
500
+ },
501
+ {
502
+ "epoch": 0.13642040541838793,
503
+ "grad_norm": 2.3018155097961426,
504
+ "learning_rate": 0.00048019665250255417,
505
+ "loss": 1.1016,
506
+ "step": 710
507
+ },
508
+ {
509
+ "epoch": 0.1383418195792103,
510
+ "grad_norm": 1.745120644569397,
511
+ "learning_rate": 0.00047959762105172964,
512
+ "loss": 0.9148,
513
+ "step": 720
514
+ },
515
+ {
516
+ "epoch": 0.14026323374003266,
517
+ "grad_norm": 1.6242783069610596,
518
+ "learning_rate": 0.00047899004911297496,
519
+ "loss": 0.9516,
520
+ "step": 730
521
+ },
522
+ {
523
+ "epoch": 0.14218464790085503,
524
+ "grad_norm": 1.867432713508606,
525
+ "learning_rate": 0.00047837395928652785,
526
+ "loss": 1.1345,
527
+ "step": 740
528
+ },
529
+ {
530
+ "epoch": 0.1441060620616774,
531
+ "grad_norm": 2.4577126502990723,
532
+ "learning_rate": 0.00047774937448947125,
533
+ "loss": 1.1109,
534
+ "step": 750
535
+ },
536
+ {
537
+ "epoch": 0.14602747622249976,
538
+ "grad_norm": 2.3080718517303467,
539
+ "learning_rate": 0.00047711631795488093,
540
+ "loss": 1.1036,
541
+ "step": 760
542
+ },
543
+ {
544
+ "epoch": 0.14794889038332212,
545
+ "grad_norm": 1.4562822580337524,
546
+ "learning_rate": 0.0004764748132309612,
547
+ "loss": 1.1009,
548
+ "step": 770
549
+ },
550
+ {
551
+ "epoch": 0.1498703045441445,
552
+ "grad_norm": 2.0178232192993164,
553
+ "learning_rate": 0.000475824884180169,
554
+ "loss": 1.054,
555
+ "step": 780
556
+ },
557
+ {
558
+ "epoch": 0.15179171870496685,
559
+ "grad_norm": 1.6681246757507324,
560
+ "learning_rate": 0.0004751665549783264,
561
+ "loss": 0.9606,
562
+ "step": 790
563
+ },
564
+ {
565
+ "epoch": 0.15371313286578922,
566
+ "grad_norm": 1.3220267295837402,
567
+ "learning_rate": 0.0004744998501137209,
568
+ "loss": 0.9446,
569
+ "step": 800
570
+ },
571
+ {
572
+ "epoch": 0.15563454702661159,
573
+ "grad_norm": 2.040184259414673,
574
+ "learning_rate": 0.0004738247943861953,
575
+ "loss": 1.1417,
576
+ "step": 810
577
+ },
578
+ {
579
+ "epoch": 0.15755596118743395,
580
+ "grad_norm": 2.3848249912261963,
581
+ "learning_rate": 0.00047314141290622435,
582
+ "loss": 1.032,
583
+ "step": 820
584
+ },
585
+ {
586
+ "epoch": 0.15947737534825632,
587
+ "grad_norm": 1.8507578372955322,
588
+ "learning_rate": 0.00047244973109398115,
589
+ "loss": 1.0843,
590
+ "step": 830
591
+ },
592
+ {
593
+ "epoch": 0.16139878950907868,
594
+ "grad_norm": 1.6274956464767456,
595
+ "learning_rate": 0.0004717497746783916,
596
+ "loss": 1.089,
597
+ "step": 840
598
+ },
599
+ {
600
+ "epoch": 0.16332020366990105,
601
+ "grad_norm": 1.817091703414917,
602
+ "learning_rate": 0.0004710415696961773,
603
+ "loss": 1.0112,
604
+ "step": 850
605
+ },
606
+ {
607
+ "epoch": 0.16524161783072341,
608
+ "grad_norm": 2.092291831970215,
609
+ "learning_rate": 0.0004703251424908868,
610
+ "loss": 1.0262,
611
+ "step": 860
612
+ },
613
+ {
614
+ "epoch": 0.16716303199154578,
615
+ "grad_norm": 1.6193962097167969,
616
+ "learning_rate": 0.0004696005197119159,
617
+ "loss": 1.0422,
618
+ "step": 870
619
+ },
620
+ {
621
+ "epoch": 0.16908444615236815,
622
+ "grad_norm": 1.929657220840454,
623
+ "learning_rate": 0.00046886772831351663,
624
+ "loss": 0.9062,
625
+ "step": 880
626
+ },
627
+ {
628
+ "epoch": 0.1710058603131905,
629
+ "grad_norm": 2.1643612384796143,
630
+ "learning_rate": 0.0004681267955537941,
631
+ "loss": 1.0763,
632
+ "step": 890
633
+ },
634
+ {
635
+ "epoch": 0.17292727447401288,
636
+ "grad_norm": 1.4907313585281372,
637
+ "learning_rate": 0.0004673777489936927,
638
+ "loss": 0.9692,
639
+ "step": 900
640
+ },
641
+ {
642
+ "epoch": 0.17484868863483524,
643
+ "grad_norm": 1.7910531759262085,
644
+ "learning_rate": 0.0004666206164959712,
645
+ "loss": 0.9425,
646
+ "step": 910
647
+ },
648
+ {
649
+ "epoch": 0.1767701027956576,
650
+ "grad_norm": 2.4189772605895996,
651
+ "learning_rate": 0.00046585542622416587,
652
+ "loss": 0.9799,
653
+ "step": 920
654
+ },
655
+ {
656
+ "epoch": 0.17869151695647997,
657
+ "grad_norm": 1.827263355255127,
658
+ "learning_rate": 0.0004650822066415434,
659
+ "loss": 0.9834,
660
+ "step": 930
661
+ },
662
+ {
663
+ "epoch": 0.18061293111730234,
664
+ "grad_norm": 0.9529337882995605,
665
+ "learning_rate": 0.0004643009865100415,
666
+ "loss": 0.9683,
667
+ "step": 940
668
+ },
669
+ {
670
+ "epoch": 0.1825343452781247,
671
+ "grad_norm": 1.812353491783142,
672
+ "learning_rate": 0.0004635117948891997,
673
+ "loss": 1.063,
674
+ "step": 950
675
+ },
676
+ {
677
+ "epoch": 0.18445575943894707,
678
+ "grad_norm": 1.5051437616348267,
679
+ "learning_rate": 0.00046271466113507776,
680
+ "loss": 0.9554,
681
+ "step": 960
682
+ },
683
+ {
684
+ "epoch": 0.18637717359976944,
685
+ "grad_norm": 2.0191311836242676,
686
+ "learning_rate": 0.0004619096148991641,
687
+ "loss": 1.1198,
688
+ "step": 970
689
+ },
690
+ {
691
+ "epoch": 0.1882985877605918,
692
+ "grad_norm": 1.7233866453170776,
693
+ "learning_rate": 0.0004610966861272728,
694
+ "loss": 1.0903,
695
+ "step": 980
696
+ },
697
+ {
698
+ "epoch": 0.19022000192141417,
699
+ "grad_norm": 1.3007336854934692,
700
+ "learning_rate": 0.0004602759050584295,
701
+ "loss": 1.0139,
702
+ "step": 990
703
+ },
704
+ {
705
+ "epoch": 0.19214141608223653,
706
+ "grad_norm": 2.3556885719299316,
707
+ "learning_rate": 0.00045944730222374693,
708
+ "loss": 1.0511,
709
+ "step": 1000
710
+ },
711
+ {
712
+ "epoch": 0.1940628302430589,
713
+ "grad_norm": 2.345998525619507,
714
+ "learning_rate": 0.00045861090844528864,
715
+ "loss": 0.9954,
716
+ "step": 1010
717
+ },
718
+ {
719
+ "epoch": 0.19598424440388126,
720
+ "grad_norm": 1.4576616287231445,
721
+ "learning_rate": 0.00045776675483492305,
722
+ "loss": 1.0297,
723
+ "step": 1020
724
+ },
725
+ {
726
+ "epoch": 0.19790565856470363,
727
+ "grad_norm": 2.5236847400665283,
728
+ "learning_rate": 0.000456914872793166,
729
+ "loss": 0.9905,
730
+ "step": 1030
731
+ },
732
+ {
733
+ "epoch": 0.199827072725526,
734
+ "grad_norm": 1.475441575050354,
735
+ "learning_rate": 0.0004560552940080126,
736
+ "loss": 1.1028,
737
+ "step": 1040
738
+ },
739
+ {
740
+ "epoch": 0.20174848688634836,
741
+ "grad_norm": 2.0014760494232178,
742
+ "learning_rate": 0.00045518805045375855,
743
+ "loss": 0.892,
744
+ "step": 1050
745
+ },
746
+ {
747
+ "epoch": 0.20366990104717073,
748
+ "grad_norm": 1.9783899784088135,
749
+ "learning_rate": 0.0004543131743898109,
750
+ "loss": 1.0747,
751
+ "step": 1060
752
+ },
753
+ {
754
+ "epoch": 0.2055913152079931,
755
+ "grad_norm": 2.0599184036254883,
756
+ "learning_rate": 0.0004534306983594879,
757
+ "loss": 1.1359,
758
+ "step": 1070
759
+ },
760
+ {
761
+ "epoch": 0.20751272936881546,
762
+ "grad_norm": 1.8748066425323486,
763
+ "learning_rate": 0.0004525406551888087,
764
+ "loss": 1.0495,
765
+ "step": 1080
766
+ },
767
+ {
768
+ "epoch": 0.20943414352963782,
769
+ "grad_norm": 1.6007455587387085,
770
+ "learning_rate": 0.0004516430779852721,
771
+ "loss": 1.1541,
772
+ "step": 1090
773
+ },
774
+ {
775
+ "epoch": 0.2113555576904602,
776
+ "grad_norm": 1.7732211351394653,
777
+ "learning_rate": 0.00045073800013662493,
778
+ "loss": 0.94,
779
+ "step": 1100
780
+ },
781
+ {
782
+ "epoch": 0.21327697185128255,
783
+ "grad_norm": 1.3590902090072632,
784
+ "learning_rate": 0.00044982545530962046,
785
+ "loss": 1.0359,
786
+ "step": 1110
787
+ },
788
+ {
789
+ "epoch": 0.21519838601210492,
790
+ "grad_norm": 1.4337408542633057,
791
+ "learning_rate": 0.00044890547744876573,
792
+ "loss": 1.0596,
793
+ "step": 1120
794
+ },
795
+ {
796
+ "epoch": 0.21711980017292729,
797
+ "grad_norm": 2.3610267639160156,
798
+ "learning_rate": 0.0004479781007750593,
799
+ "loss": 1.0121,
800
+ "step": 1130
801
+ },
802
+ {
803
+ "epoch": 0.21904121433374965,
804
+ "grad_norm": 1.9422202110290527,
805
+ "learning_rate": 0.0004470433597847176,
806
+ "loss": 1.0459,
807
+ "step": 1140
808
+ },
809
+ {
810
+ "epoch": 0.22096262849457202,
811
+ "grad_norm": 1.4544517993927002,
812
+ "learning_rate": 0.0004461012892478927,
813
+ "loss": 0.9524,
814
+ "step": 1150
815
+ },
816
+ {
817
+ "epoch": 0.22288404265539438,
818
+ "grad_norm": 1.988907814025879,
819
+ "learning_rate": 0.000445151924207378,
820
+ "loss": 0.9969,
821
+ "step": 1160
822
+ },
823
+ {
824
+ "epoch": 0.22480545681621675,
825
+ "grad_norm": 2.1969475746154785,
826
+ "learning_rate": 0.0004441952999773056,
827
+ "loss": 1.0165,
828
+ "step": 1170
829
+ },
830
+ {
831
+ "epoch": 0.2267268709770391,
832
+ "grad_norm": 1.824702501296997,
833
+ "learning_rate": 0.000443231452141832,
834
+ "loss": 0.9324,
835
+ "step": 1180
836
+ },
837
+ {
838
+ "epoch": 0.22864828513786148,
839
+ "grad_norm": 1.3211947679519653,
840
+ "learning_rate": 0.00044226041655381465,
841
+ "loss": 0.9587,
842
+ "step": 1190
843
+ },
844
+ {
845
+ "epoch": 0.23056969929868384,
846
+ "grad_norm": 1.7261728048324585,
847
+ "learning_rate": 0.00044128222933347857,
848
+ "loss": 1.0037,
849
+ "step": 1200
850
+ },
851
+ {
852
+ "epoch": 0.2324911134595062,
853
+ "grad_norm": 1.6653329133987427,
854
+ "learning_rate": 0.0004402969268670725,
855
+ "loss": 1.0027,
856
+ "step": 1210
857
+ },
858
+ {
859
+ "epoch": 0.23441252762032855,
860
+ "grad_norm": 1.6761009693145752,
861
+ "learning_rate": 0.00043930454580551543,
862
+ "loss": 0.9722,
863
+ "step": 1220
864
+ },
865
+ {
866
+ "epoch": 0.2363339417811509,
867
+ "grad_norm": 2.3884928226470947,
868
+ "learning_rate": 0.0004383051230630335,
869
+ "loss": 1.0587,
870
+ "step": 1230
871
+ },
872
+ {
873
+ "epoch": 0.23825535594197328,
874
+ "grad_norm": 1.994282841682434,
875
+ "learning_rate": 0.0004372986958157864,
876
+ "loss": 0.9573,
877
+ "step": 1240
878
+ },
879
+ {
880
+ "epoch": 0.24017677010279564,
881
+ "grad_norm": 1.8947217464447021,
882
+ "learning_rate": 0.0004362853015004853,
883
+ "loss": 0.9498,
884
+ "step": 1250
885
+ },
886
+ {
887
+ "epoch": 0.242098184263618,
888
+ "grad_norm": 1.383719801902771,
889
+ "learning_rate": 0.00043526497781299923,
890
+ "loss": 0.985,
891
+ "step": 1260
892
+ },
893
+ {
894
+ "epoch": 0.24401959842444038,
895
+ "grad_norm": 1.3979140520095825,
896
+ "learning_rate": 0.00043423776270695393,
897
+ "loss": 0.991,
898
+ "step": 1270
899
+ },
900
+ {
901
+ "epoch": 0.24594101258526274,
902
+ "grad_norm": 2.1677207946777344,
903
+ "learning_rate": 0.0004332036943923192,
904
+ "loss": 1.0134,
905
+ "step": 1280
906
+ },
907
+ {
908
+ "epoch": 0.2478624267460851,
909
+ "grad_norm": 2.216585636138916,
910
+ "learning_rate": 0.0004321628113339885,
911
+ "loss": 1.0522,
912
+ "step": 1290
913
+ },
914
+ {
915
+ "epoch": 0.24978384090690747,
916
+ "grad_norm": 1.8316318988800049,
917
+ "learning_rate": 0.000431115152250347,
918
+ "loss": 0.9927,
919
+ "step": 1300
920
+ },
921
+ {
922
+ "epoch": 0.25170525506772984,
923
+ "grad_norm": 1.802751898765564,
924
+ "learning_rate": 0.0004300607561118325,
925
+ "loss": 0.9922,
926
+ "step": 1310
927
+ },
928
+ {
929
+ "epoch": 0.2536266692285522,
930
+ "grad_norm": 1.3108793497085571,
931
+ "learning_rate": 0.00042899966213948496,
932
+ "loss": 1.0054,
933
+ "step": 1320
934
+ },
935
+ {
936
+ "epoch": 0.25554808338937457,
937
+ "grad_norm": 1.4082355499267578,
938
+ "learning_rate": 0.00042793190980348797,
939
+ "loss": 0.906,
940
+ "step": 1330
941
+ },
942
+ {
943
+ "epoch": 0.25746949755019694,
944
+ "grad_norm": 1.4897574186325073,
945
+ "learning_rate": 0.00042685753882170063,
946
+ "loss": 1.0401,
947
+ "step": 1340
948
+ },
949
+ {
950
+ "epoch": 0.2593909117110193,
951
+ "grad_norm": 1.6914223432540894,
952
+ "learning_rate": 0.00042577658915817987,
953
+ "loss": 1.0751,
954
+ "step": 1350
955
+ },
956
+ {
957
+ "epoch": 0.26131232587184167,
958
+ "grad_norm": 1.5143028497695923,
959
+ "learning_rate": 0.0004246891010216939,
960
+ "loss": 0.9363,
961
+ "step": 1360
962
+ },
963
+ {
964
+ "epoch": 0.26323374003266403,
965
+ "grad_norm": 1.7084685564041138,
966
+ "learning_rate": 0.0004235951148642269,
967
+ "loss": 0.8922,
968
+ "step": 1370
969
+ },
970
+ {
971
+ "epoch": 0.2651551541934864,
972
+ "grad_norm": 2.340919256210327,
973
+ "learning_rate": 0.00042249467137947386,
974
+ "loss": 1.0042,
975
+ "step": 1380
976
+ },
977
+ {
978
+ "epoch": 0.26707656835430876,
979
+ "grad_norm": 1.8660237789154053,
980
+ "learning_rate": 0.00042138781150132703,
981
+ "loss": 0.9955,
982
+ "step": 1390
983
+ },
984
+ {
985
+ "epoch": 0.26899798251513113,
986
+ "grad_norm": 1.997502326965332,
987
+ "learning_rate": 0.0004202745764023536,
988
+ "loss": 0.9646,
989
+ "step": 1400
990
+ },
991
+ {
992
+ "epoch": 0.2709193966759535,
993
+ "grad_norm": 1.2619147300720215,
994
+ "learning_rate": 0.0004191550074922634,
995
+ "loss": 0.9677,
996
+ "step": 1410
997
+ },
998
+ {
999
+ "epoch": 0.27284081083677586,
1000
+ "grad_norm": 1.921871542930603,
1001
+ "learning_rate": 0.0004180291464163696,
1002
+ "loss": 0.8853,
1003
+ "step": 1420
1004
+ },
1005
+ {
1006
+ "epoch": 0.2747622249975982,
1007
+ "grad_norm": 1.8054934740066528,
1008
+ "learning_rate": 0.0004168970350540384,
1009
+ "loss": 0.9755,
1010
+ "step": 1430
1011
+ },
1012
+ {
1013
+ "epoch": 0.2766836391584206,
1014
+ "grad_norm": 1.6009109020233154,
1015
+ "learning_rate": 0.00041575871551713254,
1016
+ "loss": 0.8524,
1017
+ "step": 1440
1018
+ },
1019
+ {
1020
+ "epoch": 0.27860505331924296,
1021
+ "grad_norm": 1.9609386920928955,
1022
+ "learning_rate": 0.00041461423014844354,
1023
+ "loss": 1.0005,
1024
+ "step": 1450
1025
+ },
1026
+ {
1027
+ "epoch": 0.2805264674800653,
1028
+ "grad_norm": 1.6398521661758423,
1029
+ "learning_rate": 0.00041346362152011763,
1030
+ "loss": 1.0373,
1031
+ "step": 1460
1032
+ },
1033
+ {
1034
+ "epoch": 0.2824478816408877,
1035
+ "grad_norm": 1.6447991132736206,
1036
+ "learning_rate": 0.00041230693243207185,
1037
+ "loss": 1.0963,
1038
+ "step": 1470
1039
+ },
1040
+ {
1041
+ "epoch": 0.28436929580171005,
1042
+ "grad_norm": 1.813947081565857,
1043
+ "learning_rate": 0.0004111442059104017,
1044
+ "loss": 0.9324,
1045
+ "step": 1480
1046
+ },
1047
+ {
1048
+ "epoch": 0.2862907099625324,
1049
+ "grad_norm": 1.4735851287841797,
1050
+ "learning_rate": 0.00040997548520578097,
1051
+ "loss": 0.9437,
1052
+ "step": 1490
1053
+ },
1054
+ {
1055
+ "epoch": 0.2882121241233548,
1056
+ "grad_norm": 1.5962815284729004,
1057
+ "learning_rate": 0.000408800813791853,
1058
+ "loss": 0.9427,
1059
+ "step": 1500
1060
+ },
1061
+ {
1062
+ "epoch": 0.29013353828417715,
1063
+ "grad_norm": 1.527431845664978,
1064
+ "learning_rate": 0.00040762023536361334,
1065
+ "loss": 0.9776,
1066
+ "step": 1510
1067
+ },
1068
+ {
1069
+ "epoch": 0.2920549524449995,
1070
+ "grad_norm": 1.4146003723144531,
1071
+ "learning_rate": 0.00040643379383578453,
1072
+ "loss": 0.9488,
1073
+ "step": 1520
1074
+ },
1075
+ {
1076
+ "epoch": 0.2939763666058219,
1077
+ "grad_norm": 1.4645968675613403,
1078
+ "learning_rate": 0.0004052415333411824,
1079
+ "loss": 0.8507,
1080
+ "step": 1530
1081
+ },
1082
+ {
1083
+ "epoch": 0.29589778076664425,
1084
+ "grad_norm": 1.2212328910827637,
1085
+ "learning_rate": 0.00040404349822907484,
1086
+ "loss": 0.99,
1087
+ "step": 1540
1088
+ },
1089
+ {
1090
+ "epoch": 0.2978191949274666,
1091
+ "grad_norm": 2.1287434101104736,
1092
+ "learning_rate": 0.0004028397330635315,
1093
+ "loss": 0.9587,
1094
+ "step": 1550
1095
+ },
1096
+ {
1097
+ "epoch": 0.299740609088289,
1098
+ "grad_norm": 1.9199310541152954,
1099
+ "learning_rate": 0.0004016302826217667,
1100
+ "loss": 0.9378,
1101
+ "step": 1560
1102
+ },
1103
+ {
1104
+ "epoch": 0.30166202324911134,
1105
+ "grad_norm": 1.958397626876831,
1106
+ "learning_rate": 0.0004004151918924734,
1107
+ "loss": 1.0496,
1108
+ "step": 1570
1109
+ },
1110
+ {
1111
+ "epoch": 0.3035834374099337,
1112
+ "grad_norm": 1.6783572435379028,
1113
+ "learning_rate": 0.0003991945060741502,
1114
+ "loss": 0.9261,
1115
+ "step": 1580
1116
+ },
1117
+ {
1118
+ "epoch": 0.3055048515707561,
1119
+ "grad_norm": 2.240941047668457,
1120
+ "learning_rate": 0.0003979682705734194,
1121
+ "loss": 0.966,
1122
+ "step": 1590
1123
+ },
1124
+ {
1125
+ "epoch": 0.30742626573157844,
1126
+ "grad_norm": 1.0998966693878174,
1127
+ "learning_rate": 0.0003967365310033385,
1128
+ "loss": 0.9686,
1129
+ "step": 1600
1130
+ },
1131
+ {
1132
+ "epoch": 0.3093476798924008,
1133
+ "grad_norm": 1.5177958011627197,
1134
+ "learning_rate": 0.00039549933318170353,
1135
+ "loss": 0.9887,
1136
+ "step": 1610
1137
+ },
1138
+ {
1139
+ "epoch": 0.31126909405322317,
1140
+ "grad_norm": 1.780063509941101,
1141
+ "learning_rate": 0.0003942567231293442,
1142
+ "loss": 0.8932,
1143
+ "step": 1620
1144
+ },
1145
+ {
1146
+ "epoch": 0.31319050821404554,
1147
+ "grad_norm": 1.9309372901916504,
1148
+ "learning_rate": 0.0003930087470684127,
1149
+ "loss": 0.9206,
1150
+ "step": 1630
1151
+ },
1152
+ {
1153
+ "epoch": 0.3151119223748679,
1154
+ "grad_norm": 1.5048413276672363,
1155
+ "learning_rate": 0.00039175545142066385,
1156
+ "loss": 0.9546,
1157
+ "step": 1640
1158
+ },
1159
+ {
1160
+ "epoch": 0.31703333653569027,
1161
+ "grad_norm": 1.0540335178375244,
1162
+ "learning_rate": 0.00039049688280572847,
1163
+ "loss": 0.8159,
1164
+ "step": 1650
1165
+ },
1166
+ {
1167
+ "epoch": 0.31895475069651263,
1168
+ "grad_norm": 2.461320638656616,
1169
+ "learning_rate": 0.00038923308803937956,
1170
+ "loss": 0.9886,
1171
+ "step": 1660
1172
+ },
1173
+ {
1174
+ "epoch": 0.320876164857335,
1175
+ "grad_norm": 1.532655954360962,
1176
+ "learning_rate": 0.0003879641141317903,
1177
+ "loss": 0.9431,
1178
+ "step": 1670
1179
+ },
1180
+ {
1181
+ "epoch": 0.32279757901815737,
1182
+ "grad_norm": 1.5244719982147217,
1183
+ "learning_rate": 0.0003866900082857857,
1184
+ "loss": 0.943,
1185
+ "step": 1680
1186
+ },
1187
+ {
1188
+ "epoch": 0.32471899317897973,
1189
+ "grad_norm": 1.2937613725662231,
1190
+ "learning_rate": 0.00038541081789508693,
1191
+ "loss": 0.9269,
1192
+ "step": 1690
1193
+ },
1194
+ {
1195
+ "epoch": 0.3266404073398021,
1196
+ "grad_norm": 1.5547471046447754,
1197
+ "learning_rate": 0.00038412659054254796,
1198
+ "loss": 0.8873,
1199
+ "step": 1700
1200
+ },
1201
+ {
1202
+ "epoch": 0.32856182150062446,
1203
+ "grad_norm": 1.1971843242645264,
1204
+ "learning_rate": 0.00038283737399838583,
1205
+ "loss": 0.9211,
1206
+ "step": 1710
1207
+ },
1208
+ {
1209
+ "epoch": 0.33048323566144683,
1210
+ "grad_norm": 1.9691673517227173,
1211
+ "learning_rate": 0.0003815432162184037,
1212
+ "loss": 0.8766,
1213
+ "step": 1720
1214
+ },
1215
+ {
1216
+ "epoch": 0.3324046498222692,
1217
+ "grad_norm": 1.8342334032058716,
1218
+ "learning_rate": 0.0003802441653422073,
1219
+ "loss": 0.9308,
1220
+ "step": 1730
1221
+ },
1222
+ {
1223
+ "epoch": 0.33432606398309156,
1224
+ "grad_norm": 1.1400641202926636,
1225
+ "learning_rate": 0.0003789402696914136,
1226
+ "loss": 0.9256,
1227
+ "step": 1740
1228
+ },
1229
+ {
1230
+ "epoch": 0.3362474781439139,
1231
+ "grad_norm": 1.6318907737731934,
1232
+ "learning_rate": 0.0003776315777678537,
1233
+ "loss": 0.8789,
1234
+ "step": 1750
1235
+ },
1236
+ {
1237
+ "epoch": 0.3381688923047363,
1238
+ "grad_norm": 1.1623560190200806,
1239
+ "learning_rate": 0.00037631813825176913,
1240
+ "loss": 1.022,
1241
+ "step": 1760
1242
+ },
1243
+ {
1244
+ "epoch": 0.34009030646555866,
1245
+ "grad_norm": 1.0850412845611572,
1246
+ "learning_rate": 0.000375,
1247
+ "loss": 0.8747,
1248
+ "step": 1770
1249
+ },
1250
+ {
1251
+ "epoch": 0.342011720626381,
1252
+ "grad_norm": 1.9052928686141968,
1253
+ "learning_rate": 0.00037367721204416873,
1254
+ "loss": 0.8603,
1255
+ "step": 1780
1256
+ },
1257
+ {
1258
+ "epoch": 0.3439331347872034,
1259
+ "grad_norm": 1.469117283821106,
1260
+ "learning_rate": 0.0003723498235888556,
1261
+ "loss": 0.9025,
1262
+ "step": 1790
1263
+ },
1264
+ {
1265
+ "epoch": 0.34585454894802575,
1266
+ "grad_norm": 2.0990898609161377,
1267
+ "learning_rate": 0.0003710178840097685,
1268
+ "loss": 0.8507,
1269
+ "step": 1800
1270
+ },
1271
+ {
1272
+ "epoch": 0.3477759631088481,
1273
+ "grad_norm": 1.6102617979049683,
1274
+ "learning_rate": 0.0003696814428519064,
1275
+ "loss": 0.9469,
1276
+ "step": 1810
1277
+ },
1278
+ {
1279
+ "epoch": 0.3496973772696705,
1280
+ "grad_norm": 1.8937904834747314,
1281
+ "learning_rate": 0.0003683405498277164,
1282
+ "loss": 0.9119,
1283
+ "step": 1820
1284
+ },
1285
+ {
1286
+ "epoch": 0.35161879143049285,
1287
+ "grad_norm": 1.446227788925171,
1288
+ "learning_rate": 0.0003669952548152443,
1289
+ "loss": 0.8413,
1290
+ "step": 1830
1291
+ },
1292
+ {
1293
+ "epoch": 0.3535402055913152,
1294
+ "grad_norm": 1.6752564907073975,
1295
+ "learning_rate": 0.00036564560785627974,
1296
+ "loss": 0.9523,
1297
+ "step": 1840
1298
+ },
1299
+ {
1300
+ "epoch": 0.3554616197521376,
1301
+ "grad_norm": 1.836383581161499,
1302
+ "learning_rate": 0.00036429165915449416,
1303
+ "loss": 0.889,
1304
+ "step": 1850
1305
+ },
1306
+ {
1307
+ "epoch": 0.35738303391295995,
1308
+ "grad_norm": 1.7696865797042847,
1309
+ "learning_rate": 0.0003629334590735738,
1310
+ "loss": 1.0661,
1311
+ "step": 1860
1312
+ },
1313
+ {
1314
+ "epoch": 0.3593044480737823,
1315
+ "grad_norm": 1.602179765701294,
1316
+ "learning_rate": 0.0003615710581353463,
1317
+ "loss": 0.8948,
1318
+ "step": 1870
1319
+ },
1320
+ {
1321
+ "epoch": 0.3612258622346047,
1322
+ "grad_norm": 1.8728015422821045,
1323
+ "learning_rate": 0.0003602045070179009,
1324
+ "loss": 0.9631,
1325
+ "step": 1880
1326
+ },
1327
+ {
1328
+ "epoch": 0.36314727639542704,
1329
+ "grad_norm": 2.071622848510742,
1330
+ "learning_rate": 0.0003588338565537039,
1331
+ "loss": 0.941,
1332
+ "step": 1890
1333
+ },
1334
+ {
1335
+ "epoch": 0.3650686905562494,
1336
+ "grad_norm": 1.5780361890792847,
1337
+ "learning_rate": 0.0003574591577277076,
1338
+ "loss": 0.8537,
1339
+ "step": 1900
1340
+ },
1341
+ {
1342
+ "epoch": 0.3669901047170718,
1343
+ "grad_norm": 1.4528398513793945,
1344
+ "learning_rate": 0.0003560804616754536,
1345
+ "loss": 0.8613,
1346
+ "step": 1910
1347
+ },
1348
+ {
1349
+ "epoch": 0.36891151887789414,
1350
+ "grad_norm": 1.888594627380371,
1351
+ "learning_rate": 0.0003546978196811711,
1352
+ "loss": 0.9791,
1353
+ "step": 1920
1354
+ },
1355
+ {
1356
+ "epoch": 0.3708329330387165,
1357
+ "grad_norm": 1.3231236934661865,
1358
+ "learning_rate": 0.00035331128317586885,
1359
+ "loss": 0.9041,
1360
+ "step": 1930
1361
+ },
1362
+ {
1363
+ "epoch": 0.37275434719953887,
1364
+ "grad_norm": 1.3922632932662964,
1365
+ "learning_rate": 0.0003519209037354222,
1366
+ "loss": 0.9247,
1367
+ "step": 1940
1368
+ },
1369
+ {
1370
+ "epoch": 0.37467576136036124,
1371
+ "grad_norm": 1.7650400400161743,
1372
+ "learning_rate": 0.0003505267330786547,
1373
+ "loss": 0.9474,
1374
+ "step": 1950
1375
+ },
1376
+ {
1377
+ "epoch": 0.3765971755211836,
1378
+ "grad_norm": 1.8358489274978638,
1379
+ "learning_rate": 0.0003491288230654138,
1380
+ "loss": 0.8982,
1381
+ "step": 1960
1382
+ },
1383
+ {
1384
+ "epoch": 0.37851858968200597,
1385
+ "grad_norm": 1.263236165046692,
1386
+ "learning_rate": 0.00034772722569464276,
1387
+ "loss": 0.8903,
1388
+ "step": 1970
1389
+ },
1390
+ {
1391
+ "epoch": 0.38044000384282833,
1392
+ "grad_norm": 1.6004908084869385,
1393
+ "learning_rate": 0.00034632199310244535,
1394
+ "loss": 0.9012,
1395
+ "step": 1980
1396
+ },
1397
+ {
1398
+ "epoch": 0.3823614180036507,
1399
+ "grad_norm": 1.7554082870483398,
1400
+ "learning_rate": 0.00034491317756014706,
1401
+ "loss": 0.9149,
1402
+ "step": 1990
1403
+ },
1404
+ {
1405
+ "epoch": 0.38428283216447306,
1406
+ "grad_norm": 1.0712970495224,
1407
+ "learning_rate": 0.00034350083147235077,
1408
+ "loss": 1.0325,
1409
+ "step": 2000
1410
+ },
1411
+ {
1412
+ "epoch": 0.38620424632529543,
1413
+ "grad_norm": 1.5798851251602173,
1414
+ "learning_rate": 0.0003420850073749872,
1415
+ "loss": 1.0006,
1416
+ "step": 2010
1417
+ },
1418
+ {
1419
+ "epoch": 0.3881256604861178,
1420
+ "grad_norm": 1.782596230506897,
1421
+ "learning_rate": 0.00034066575793336075,
1422
+ "loss": 0.8972,
1423
+ "step": 2020
1424
+ },
1425
+ {
1426
+ "epoch": 0.39004707464694016,
1427
+ "grad_norm": 1.332011342048645,
1428
+ "learning_rate": 0.0003392431359401906,
1429
+ "loss": 0.8255,
1430
+ "step": 2030
1431
+ },
1432
+ {
1433
+ "epoch": 0.3919684888077625,
1434
+ "grad_norm": 1.5429998636245728,
1435
+ "learning_rate": 0.0003378171943136469,
1436
+ "loss": 0.8661,
1437
+ "step": 2040
1438
+ },
1439
+ {
1440
+ "epoch": 0.3938899029685849,
1441
+ "grad_norm": 1.1828477382659912,
1442
+ "learning_rate": 0.0003363879860953822,
1443
+ "loss": 0.9084,
1444
+ "step": 2050
1445
+ },
1446
+ {
1447
+ "epoch": 0.39581131712940726,
1448
+ "grad_norm": 1.411515712738037,
1449
+ "learning_rate": 0.0003349555644485585,
1450
+ "loss": 0.8647,
1451
+ "step": 2060
1452
+ },
1453
+ {
1454
+ "epoch": 0.3977327312902296,
1455
+ "grad_norm": 1.8091164827346802,
1456
+ "learning_rate": 0.00033351998265586987,
1457
+ "loss": 0.9742,
1458
+ "step": 2070
1459
+ },
1460
+ {
1461
+ "epoch": 0.399654145451052,
1462
+ "grad_norm": 1.3836525678634644,
1463
+ "learning_rate": 0.00033208129411756024,
1464
+ "loss": 0.8556,
1465
+ "step": 2080
1466
+ },
1467
+ {
1468
+ "epoch": 0.40157555961187436,
1469
+ "grad_norm": 1.6256176233291626,
1470
+ "learning_rate": 0.00033063955234943705,
1471
+ "loss": 0.8681,
1472
+ "step": 2090
1473
+ },
1474
+ {
1475
+ "epoch": 0.4034969737726967,
1476
+ "grad_norm": 1.7528786659240723,
1477
+ "learning_rate": 0.0003291948109808809,
1478
+ "loss": 0.9599,
1479
+ "step": 2100
1480
+ },
1481
+ {
1482
+ "epoch": 0.4054183879335191,
1483
+ "grad_norm": 1.83562433719635,
1484
+ "learning_rate": 0.0003277471237528502,
1485
+ "loss": 0.8709,
1486
+ "step": 2110
1487
+ },
1488
+ {
1489
+ "epoch": 0.40733980209434145,
1490
+ "grad_norm": 1.7219703197479248,
1491
+ "learning_rate": 0.0003262965445158823,
1492
+ "loss": 0.9668,
1493
+ "step": 2120
1494
+ },
1495
+ {
1496
+ "epoch": 0.4092612162551638,
1497
+ "grad_norm": 1.4686826467514038,
1498
+ "learning_rate": 0.0003248431272280908,
1499
+ "loss": 0.9449,
1500
+ "step": 2130
1501
+ },
1502
+ {
1503
+ "epoch": 0.4111826304159862,
1504
+ "grad_norm": 1.539268970489502,
1505
+ "learning_rate": 0.0003233869259531577,
1506
+ "loss": 0.925,
1507
+ "step": 2140
1508
+ },
1509
+ {
1510
+ "epoch": 0.41310404457680855,
1511
+ "grad_norm": 1.6110196113586426,
1512
+ "learning_rate": 0.0003219279948583229,
1513
+ "loss": 0.8146,
1514
+ "step": 2150
1515
+ },
1516
+ {
1517
+ "epoch": 0.4150254587376309,
1518
+ "grad_norm": 1.3205841779708862,
1519
+ "learning_rate": 0.00032046638821236914,
1520
+ "loss": 0.9522,
1521
+ "step": 2160
1522
+ },
1523
+ {
1524
+ "epoch": 0.4169468728984533,
1525
+ "grad_norm": 2.021388530731201,
1526
+ "learning_rate": 0.00031900216038360313,
1527
+ "loss": 0.8603,
1528
+ "step": 2170
1529
+ },
1530
+ {
1531
+ "epoch": 0.41886828705927565,
1532
+ "grad_norm": 1.2504761219024658,
1533
+ "learning_rate": 0.00031753536583783374,
1534
+ "loss": 0.9003,
1535
+ "step": 2180
1536
+ },
1537
+ {
1538
+ "epoch": 0.420789701220098,
1539
+ "grad_norm": 2.206533432006836,
1540
+ "learning_rate": 0.00031606605913634534,
1541
+ "loss": 0.8686,
1542
+ "step": 2190
1543
+ },
1544
+ {
1545
+ "epoch": 0.4227111153809204,
1546
+ "grad_norm": 1.5957015752792358,
1547
+ "learning_rate": 0.00031459429493386863,
1548
+ "loss": 0.8447,
1549
+ "step": 2200
1550
+ },
1551
+ {
1552
+ "epoch": 0.42463252954174274,
1553
+ "grad_norm": 1.2046681642532349,
1554
+ "learning_rate": 0.00031312012797654756,
1555
+ "loss": 0.826,
1556
+ "step": 2210
1557
+ },
1558
+ {
1559
+ "epoch": 0.4265539437025651,
1560
+ "grad_norm": 2.017876625061035,
1561
+ "learning_rate": 0.00031164361309990283,
1562
+ "loss": 0.8967,
1563
+ "step": 2220
1564
+ },
1565
+ {
1566
+ "epoch": 0.4284753578633875,
1567
+ "grad_norm": 1.3104060888290405,
1568
+ "learning_rate": 0.00031016480522679223,
1569
+ "loss": 0.8396,
1570
+ "step": 2230
1571
+ },
1572
+ {
1573
+ "epoch": 0.43039677202420984,
1574
+ "grad_norm": 1.9322564601898193,
1575
+ "learning_rate": 0.00030868375936536754,
1576
+ "loss": 0.9624,
1577
+ "step": 2240
1578
+ },
1579
+ {
1580
+ "epoch": 0.4323181861850322,
1581
+ "grad_norm": 1.526458501815796,
1582
+ "learning_rate": 0.00030720053060702835,
1583
+ "loss": 0.9021,
1584
+ "step": 2250
1585
+ },
1586
+ {
1587
+ "epoch": 0.43423960034585457,
1588
+ "grad_norm": 1.5850962400436401,
1589
+ "learning_rate": 0.0003057151741243731,
1590
+ "loss": 0.874,
1591
+ "step": 2260
1592
+ },
1593
+ {
1594
+ "epoch": 0.43616101450667694,
1595
+ "grad_norm": 1.2977805137634277,
1596
+ "learning_rate": 0.0003042277451691462,
1597
+ "loss": 0.9072,
1598
+ "step": 2270
1599
+ },
1600
+ {
1601
+ "epoch": 0.4380824286674993,
1602
+ "grad_norm": 2.240307092666626,
1603
+ "learning_rate": 0.0003027382990701833,
1604
+ "loss": 0.8864,
1605
+ "step": 2280
1606
+ },
1607
+ {
1608
+ "epoch": 0.44000384282832167,
1609
+ "grad_norm": 1.734005093574524,
1610
+ "learning_rate": 0.00030124689123135306,
1611
+ "loss": 0.8596,
1612
+ "step": 2290
1613
+ },
1614
+ {
1615
+ "epoch": 0.44192525698914403,
1616
+ "grad_norm": 1.654840350151062,
1617
+ "learning_rate": 0.00029975357712949625,
1618
+ "loss": 0.9219,
1619
+ "step": 2300
1620
+ },
1621
+ {
1622
+ "epoch": 0.4438466711499664,
1623
+ "grad_norm": 1.530916690826416,
1624
+ "learning_rate": 0.0002982584123123619,
1625
+ "loss": 0.976,
1626
+ "step": 2310
1627
+ },
1628
+ {
1629
+ "epoch": 0.44576808531078876,
1630
+ "grad_norm": 1.589181661605835,
1631
+ "learning_rate": 0.00029676145239654144,
1632
+ "loss": 0.8203,
1633
+ "step": 2320
1634
+ },
1635
+ {
1636
+ "epoch": 0.44768949947161113,
1637
+ "grad_norm": 1.8620957136154175,
1638
+ "learning_rate": 0.0002952627530653997,
1639
+ "loss": 0.9294,
1640
+ "step": 2330
1641
+ },
1642
+ {
1643
+ "epoch": 0.4496109136324335,
1644
+ "grad_norm": 1.6283154487609863,
1645
+ "learning_rate": 0.00029376237006700366,
1646
+ "loss": 1.0087,
1647
+ "step": 2340
1648
+ },
1649
+ {
1650
+ "epoch": 0.45153232779325586,
1651
+ "grad_norm": 1.7942941188812256,
1652
+ "learning_rate": 0.00029226035921204864,
1653
+ "loss": 0.9341,
1654
+ "step": 2350
1655
+ },
1656
+ {
1657
+ "epoch": 0.4534537419540782,
1658
+ "grad_norm": 1.7389261722564697,
1659
+ "learning_rate": 0.00029075677637178243,
1660
+ "loss": 0.8171,
1661
+ "step": 2360
1662
+ },
1663
+ {
1664
+ "epoch": 0.4553751561149006,
1665
+ "grad_norm": 1.1926878690719604,
1666
+ "learning_rate": 0.000289251677475927,
1667
+ "loss": 0.8444,
1668
+ "step": 2370
1669
+ },
1670
+ {
1671
+ "epoch": 0.45729657027572296,
1672
+ "grad_norm": 1.444225788116455,
1673
+ "learning_rate": 0.0002877451185105979,
1674
+ "loss": 0.7467,
1675
+ "step": 2380
1676
+ },
1677
+ {
1678
+ "epoch": 0.4592179844365453,
1679
+ "grad_norm": 1.2821637392044067,
1680
+ "learning_rate": 0.00028623715551622187,
1681
+ "loss": 0.7329,
1682
+ "step": 2390
1683
+ },
1684
+ {
1685
+ "epoch": 0.4611393985973677,
1686
+ "grad_norm": 1.3402514457702637,
1687
+ "learning_rate": 0.0002847278445854522,
1688
+ "loss": 0.88,
1689
+ "step": 2400
1690
+ },
1691
+ {
1692
+ "epoch": 0.46306081275819005,
1693
+ "grad_norm": 1.2902830839157104,
1694
+ "learning_rate": 0.00028321724186108226,
1695
+ "loss": 0.927,
1696
+ "step": 2410
1697
+ },
1698
+ {
1699
+ "epoch": 0.4649822269190124,
1700
+ "grad_norm": 1.1409978866577148,
1701
+ "learning_rate": 0.00028170540353395694,
1702
+ "loss": 0.8171,
1703
+ "step": 2420
1704
+ },
1705
+ {
1706
+ "epoch": 0.4669036410798348,
1707
+ "grad_norm": 1.7634387016296387,
1708
+ "learning_rate": 0.00028019238584088286,
1709
+ "loss": 0.8339,
1710
+ "step": 2430
1711
+ },
1712
+ {
1713
+ "epoch": 0.4688250552406571,
1714
+ "grad_norm": 1.3894178867340088,
1715
+ "learning_rate": 0.00027867824506253605,
1716
+ "loss": 0.8551,
1717
+ "step": 2440
1718
+ },
1719
+ {
1720
+ "epoch": 0.47074646940147946,
1721
+ "grad_norm": 1.396325707435608,
1722
+ "learning_rate": 0.00027716303752136864,
1723
+ "loss": 0.8561,
1724
+ "step": 2450
1725
+ },
1726
+ {
1727
+ "epoch": 0.4726678835623018,
1728
+ "grad_norm": 1.8292694091796875,
1729
+ "learning_rate": 0.00027564681957951406,
1730
+ "loss": 0.9106,
1731
+ "step": 2460
1732
+ },
1733
+ {
1734
+ "epoch": 0.4745892977231242,
1735
+ "grad_norm": 1.632660984992981,
1736
+ "learning_rate": 0.00027412964763669006,
1737
+ "loss": 0.7915,
1738
+ "step": 2470
1739
+ },
1740
+ {
1741
+ "epoch": 0.47651071188394656,
1742
+ "grad_norm": 1.5735293626785278,
1743
+ "learning_rate": 0.000272611578128101,
1744
+ "loss": 0.7958,
1745
+ "step": 2480
1746
+ },
1747
+ {
1748
+ "epoch": 0.4784321260447689,
1749
+ "grad_norm": 1.52776038646698,
1750
+ "learning_rate": 0.00027109266752233847,
1751
+ "loss": 0.8285,
1752
+ "step": 2490
1753
+ },
1754
+ {
1755
+ "epoch": 0.4803535402055913,
1756
+ "grad_norm": 1.4657976627349854,
1757
+ "learning_rate": 0.0002695729723192811,
1758
+ "loss": 0.8942,
1759
+ "step": 2500
1760
+ },
1761
+ {
1762
+ "epoch": 0.48227495436641366,
1763
+ "grad_norm": 2.0572874546051025,
1764
+ "learning_rate": 0.0002680525490479925,
1765
+ "loss": 0.8646,
1766
+ "step": 2510
1767
+ },
1768
+ {
1769
+ "epoch": 0.484196368527236,
1770
+ "grad_norm": 1.147052526473999,
1771
+ "learning_rate": 0.0002665314542646188,
1772
+ "loss": 0.8884,
1773
+ "step": 2520
1774
+ },
1775
+ {
1776
+ "epoch": 0.4861177826880584,
1777
+ "grad_norm": 1.3121823072433472,
1778
+ "learning_rate": 0.00026500974455028473,
1779
+ "loss": 0.894,
1780
+ "step": 2530
1781
+ },
1782
+ {
1783
+ "epoch": 0.48803919684888075,
1784
+ "grad_norm": 2.331324577331543,
1785
+ "learning_rate": 0.00026348747650898897,
1786
+ "loss": 0.8319,
1787
+ "step": 2540
1788
+ },
1789
+ {
1790
+ "epoch": 0.4899606110097031,
1791
+ "grad_norm": 1.9230785369873047,
1792
+ "learning_rate": 0.0002619647067654988,
1793
+ "loss": 0.7815,
1794
+ "step": 2550
1795
+ },
1796
+ {
1797
+ "epoch": 0.4918820251705255,
1798
+ "grad_norm": 0.8924723863601685,
1799
+ "learning_rate": 0.00026044149196324324,
1800
+ "loss": 0.8115,
1801
+ "step": 2560
1802
+ },
1803
+ {
1804
+ "epoch": 0.49380343933134785,
1805
+ "grad_norm": 1.117889165878296,
1806
+ "learning_rate": 0.00025891788876220706,
1807
+ "loss": 0.7597,
1808
+ "step": 2570
1809
+ },
1810
+ {
1811
+ "epoch": 0.4957248534921702,
1812
+ "grad_norm": 0.8507145047187805,
1813
+ "learning_rate": 0.00025739395383682205,
1814
+ "loss": 0.765,
1815
+ "step": 2580
1816
+ },
1817
+ {
1818
+ "epoch": 0.4976462676529926,
1819
+ "grad_norm": 1.1917976140975952,
1820
+ "learning_rate": 0.00025586974387385947,
1821
+ "loss": 0.8403,
1822
+ "step": 2590
1823
+ },
1824
+ {
1825
+ "epoch": 0.49956768181381495,
1826
+ "grad_norm": 1.5689092874526978,
1827
+ "learning_rate": 0.0002543453155703214,
1828
+ "loss": 0.8236,
1829
+ "step": 2600
1830
+ },
1831
+ {
1832
+ "epoch": 0.5014890959746373,
1833
+ "grad_norm": 1.2577379941940308,
1834
+ "learning_rate": 0.00025282072563133167,
1835
+ "loss": 0.8568,
1836
+ "step": 2610
1837
+ },
1838
+ {
1839
+ "epoch": 0.5034105101354597,
1840
+ "grad_norm": 1.9359431266784668,
1841
+ "learning_rate": 0.0002512960307680266,
1842
+ "loss": 0.8204,
1843
+ "step": 2620
1844
+ },
1845
+ {
1846
+ "epoch": 0.505331924296282,
1847
+ "grad_norm": 1.4619979858398438,
1848
+ "learning_rate": 0.00024977128769544524,
1849
+ "loss": 0.7493,
1850
+ "step": 2630
1851
+ },
1852
+ {
1853
+ "epoch": 0.5072533384571044,
1854
+ "grad_norm": 1.4376741647720337,
1855
+ "learning_rate": 0.00024824655313042014,
1856
+ "loss": 0.8281,
1857
+ "step": 2640
1858
+ },
1859
+ {
1860
+ "epoch": 0.5091747526179268,
1861
+ "grad_norm": 1.581264853477478,
1862
+ "learning_rate": 0.0002467218837894674,
1863
+ "loss": 0.7787,
1864
+ "step": 2650
1865
+ },
1866
+ {
1867
+ "epoch": 0.5110961667787491,
1868
+ "grad_norm": 1.4558833837509155,
1869
+ "learning_rate": 0.0002451973363866766,
1870
+ "loss": 0.7455,
1871
+ "step": 2660
1872
+ },
1873
+ {
1874
+ "epoch": 0.5130175809395715,
1875
+ "grad_norm": 1.8191595077514648,
1876
+ "learning_rate": 0.000243672967631602,
1877
+ "loss": 0.8622,
1878
+ "step": 2670
1879
+ },
1880
+ {
1881
+ "epoch": 0.5149389951003939,
1882
+ "grad_norm": 1.5446815490722656,
1883
+ "learning_rate": 0.00024214883422715212,
1884
+ "loss": 0.7506,
1885
+ "step": 2680
1886
+ },
1887
+ {
1888
+ "epoch": 0.5168604092612162,
1889
+ "grad_norm": 1.3518351316452026,
1890
+ "learning_rate": 0.00024062499286748142,
1891
+ "loss": 0.7654,
1892
+ "step": 2690
1893
+ },
1894
+ {
1895
+ "epoch": 0.5187818234220386,
1896
+ "grad_norm": 1.2076045274734497,
1897
+ "learning_rate": 0.0002391015002358807,
1898
+ "loss": 0.8667,
1899
+ "step": 2700
1900
+ },
1901
+ {
1902
+ "epoch": 0.520703237582861,
1903
+ "grad_norm": 1.7851399183273315,
1904
+ "learning_rate": 0.00023757841300266896,
1905
+ "loss": 0.8002,
1906
+ "step": 2710
1907
+ },
1908
+ {
1909
+ "epoch": 0.5226246517436833,
1910
+ "grad_norm": 1.0870471000671387,
1911
+ "learning_rate": 0.00023605578782308538,
1912
+ "loss": 0.8146,
1913
+ "step": 2720
1914
+ },
1915
+ {
1916
+ "epoch": 0.5245460659045057,
1917
+ "grad_norm": 1.481876015663147,
1918
+ "learning_rate": 0.0002345336813351819,
1919
+ "loss": 0.8197,
1920
+ "step": 2730
1921
+ },
1922
+ {
1923
+ "epoch": 0.5264674800653281,
1924
+ "grad_norm": 1.550943374633789,
1925
+ "learning_rate": 0.00023301215015771607,
1926
+ "loss": 0.7727,
1927
+ "step": 2740
1928
+ },
1929
+ {
1930
+ "epoch": 0.5283888942261504,
1931
+ "grad_norm": 1.1842530965805054,
1932
+ "learning_rate": 0.0002314912508880456,
1933
+ "loss": 0.7361,
1934
+ "step": 2750
1935
+ },
1936
+ {
1937
+ "epoch": 0.5303103083869728,
1938
+ "grad_norm": 1.2636481523513794,
1939
+ "learning_rate": 0.0002299710401000226,
1940
+ "loss": 0.8723,
1941
+ "step": 2760
1942
+ },
1943
+ {
1944
+ "epoch": 0.5322317225477952,
1945
+ "grad_norm": 0.6428311467170715,
1946
+ "learning_rate": 0.0002284515743418893,
1947
+ "loss": 0.7911,
1948
+ "step": 2770
1949
+ },
1950
+ {
1951
+ "epoch": 0.5341531367086175,
1952
+ "grad_norm": 1.473262906074524,
1953
+ "learning_rate": 0.00022693291013417452,
1954
+ "loss": 0.7811,
1955
+ "step": 2780
1956
+ },
1957
+ {
1958
+ "epoch": 0.5360745508694399,
1959
+ "grad_norm": 1.4524637460708618,
1960
+ "learning_rate": 0.00022541510396759134,
1961
+ "loss": 0.7198,
1962
+ "step": 2790
1963
+ },
1964
+ {
1965
+ "epoch": 0.5379959650302623,
1966
+ "grad_norm": 2.153787851333618,
1967
+ "learning_rate": 0.00022389821230093575,
1968
+ "loss": 0.8221,
1969
+ "step": 2800
1970
+ },
1971
+ {
1972
+ "epoch": 0.5399173791910846,
1973
+ "grad_norm": 1.4184430837631226,
1974
+ "learning_rate": 0.00022238229155898656,
1975
+ "loss": 0.8138,
1976
+ "step": 2810
1977
+ },
1978
+ {
1979
+ "epoch": 0.541838793351907,
1980
+ "grad_norm": 1.007478952407837,
1981
+ "learning_rate": 0.00022086739813040622,
1982
+ "loss": 0.7708,
1983
+ "step": 2820
1984
+ },
1985
+ {
1986
+ "epoch": 0.5437602075127294,
1987
+ "grad_norm": 1.315148949623108,
1988
+ "learning_rate": 0.00021935358836564406,
1989
+ "loss": 0.7335,
1990
+ "step": 2830
1991
+ },
1992
+ {
1993
+ "epoch": 0.5456816216735517,
1994
+ "grad_norm": 1.2237809896469116,
1995
+ "learning_rate": 0.0002178409185748392,
1996
+ "loss": 0.8343,
1997
+ "step": 2840
1998
+ },
1999
+ {
2000
+ "epoch": 0.5476030358343741,
2001
+ "grad_norm": 1.43268883228302,
2002
+ "learning_rate": 0.00021632944502572663,
2003
+ "loss": 0.7771,
2004
+ "step": 2850
2005
+ },
2006
+ {
2007
+ "epoch": 0.5495244499951965,
2008
+ "grad_norm": 1.8583613634109497,
2009
+ "learning_rate": 0.000214819223941544,
2010
+ "loss": 0.8243,
2011
+ "step": 2860
2012
+ },
2013
+ {
2014
+ "epoch": 0.5514458641560188,
2015
+ "grad_norm": 1.0081727504730225,
2016
+ "learning_rate": 0.00021331031149894014,
2017
+ "loss": 0.7548,
2018
+ "step": 2870
2019
+ },
2020
+ {
2021
+ "epoch": 0.5533672783168412,
2022
+ "grad_norm": 1.6709460020065308,
2023
+ "learning_rate": 0.0002118027638258856,
2024
+ "loss": 0.8975,
2025
+ "step": 2880
2026
+ },
2027
+ {
2028
+ "epoch": 0.5552886924776635,
2029
+ "grad_norm": 1.0278853178024292,
2030
+ "learning_rate": 0.0002102966369995847,
2031
+ "loss": 0.7381,
2032
+ "step": 2890
2033
+ },
2034
+ {
2035
+ "epoch": 0.5572101066384859,
2036
+ "grad_norm": 1.155301570892334,
2037
+ "learning_rate": 0.00020879198704438945,
2038
+ "loss": 0.867,
2039
+ "step": 2900
2040
+ },
2041
+ {
2042
+ "epoch": 0.5591315207993083,
2043
+ "grad_norm": 0.615483283996582,
2044
+ "learning_rate": 0.0002072888699297162,
2045
+ "loss": 0.8096,
2046
+ "step": 2910
2047
+ },
2048
+ {
2049
+ "epoch": 0.5610529349601306,
2050
+ "grad_norm": 1.269416093826294,
2051
+ "learning_rate": 0.0002057873415679628,
2052
+ "loss": 0.7184,
2053
+ "step": 2920
2054
+ },
2055
+ {
2056
+ "epoch": 0.562974349120953,
2057
+ "grad_norm": 1.3008477687835693,
2058
+ "learning_rate": 0.0002042874578124295,
2059
+ "loss": 0.834,
2060
+ "step": 2930
2061
+ },
2062
+ {
2063
+ "epoch": 0.5648957632817754,
2064
+ "grad_norm": 1.5526058673858643,
2065
+ "learning_rate": 0.00020278927445524104,
2066
+ "loss": 0.6741,
2067
+ "step": 2940
2068
+ },
2069
+ {
2070
+ "epoch": 0.5668171774425977,
2071
+ "grad_norm": 1.1507304906845093,
2072
+ "learning_rate": 0.00020129284722527127,
2073
+ "loss": 0.7561,
2074
+ "step": 2950
2075
+ },
2076
+ {
2077
+ "epoch": 0.5687385916034201,
2078
+ "grad_norm": 1.2722505331039429,
2079
+ "learning_rate": 0.00019979823178607043,
2080
+ "loss": 0.7212,
2081
+ "step": 2960
2082
+ },
2083
+ {
2084
+ "epoch": 0.5706600057642425,
2085
+ "grad_norm": 1.5021613836288452,
2086
+ "learning_rate": 0.00019830548373379425,
2087
+ "loss": 0.7767,
2088
+ "step": 2970
2089
+ },
2090
+ {
2091
+ "epoch": 0.5725814199250648,
2092
+ "grad_norm": 1.3983266353607178,
2093
+ "learning_rate": 0.00019681465859513632,
2094
+ "loss": 0.7895,
2095
+ "step": 2980
2096
+ },
2097
+ {
2098
+ "epoch": 0.5745028340858872,
2099
+ "grad_norm": 1.1755023002624512,
2100
+ "learning_rate": 0.00019532581182526225,
2101
+ "loss": 0.8512,
2102
+ "step": 2990
2103
+ },
2104
+ {
2105
+ "epoch": 0.5764242482467096,
2106
+ "grad_norm": 1.3724703788757324,
2107
+ "learning_rate": 0.00019383899880574696,
2108
+ "loss": 0.8009,
2109
+ "step": 3000
2110
+ },
2111
+ {
2112
+ "epoch": 0.5783456624075319,
2113
+ "grad_norm": 1.3599048852920532,
2114
+ "learning_rate": 0.00019235427484251474,
2115
+ "loss": 0.828,
2116
+ "step": 3010
2117
+ },
2118
+ {
2119
+ "epoch": 0.5802670765683543,
2120
+ "grad_norm": 1.1706438064575195,
2121
+ "learning_rate": 0.00019087169516378202,
2122
+ "loss": 0.7888,
2123
+ "step": 3020
2124
+ },
2125
+ {
2126
+ "epoch": 0.5821884907291767,
2127
+ "grad_norm": 1.0189845561981201,
2128
+ "learning_rate": 0.00018939131491800277,
2129
+ "loss": 0.7479,
2130
+ "step": 3030
2131
+ },
2132
+ {
2133
+ "epoch": 0.584109904889999,
2134
+ "grad_norm": 1.106514573097229,
2135
+ "learning_rate": 0.00018791318917181726,
2136
+ "loss": 0.8299,
2137
+ "step": 3040
2138
+ },
2139
+ {
2140
+ "epoch": 0.5860313190508214,
2141
+ "grad_norm": 1.3856253623962402,
2142
+ "learning_rate": 0.00018643737290800392,
2143
+ "loss": 0.7903,
2144
+ "step": 3050
2145
+ },
2146
+ {
2147
+ "epoch": 0.5879527332116438,
2148
+ "grad_norm": 1.2991724014282227,
2149
+ "learning_rate": 0.0001849639210234337,
2150
+ "loss": 0.7899,
2151
+ "step": 3060
2152
+ },
2153
+ {
2154
+ "epoch": 0.5898741473724661,
2155
+ "grad_norm": 1.586128830909729,
2156
+ "learning_rate": 0.00018349288832702836,
2157
+ "loss": 0.8614,
2158
+ "step": 3070
2159
+ },
2160
+ {
2161
+ "epoch": 0.5917955615332885,
2162
+ "grad_norm": 1.5094038248062134,
2163
+ "learning_rate": 0.00018202432953772147,
2164
+ "loss": 0.8301,
2165
+ "step": 3080
2166
+ },
2167
+ {
2168
+ "epoch": 0.5937169756941109,
2169
+ "grad_norm": 1.4626994132995605,
2170
+ "learning_rate": 0.00018055829928242316,
2171
+ "loss": 0.7848,
2172
+ "step": 3090
2173
+ },
2174
+ {
2175
+ "epoch": 0.5956383898549332,
2176
+ "grad_norm": 1.445717215538025,
2177
+ "learning_rate": 0.00017909485209398817,
2178
+ "loss": 0.8187,
2179
+ "step": 3100
2180
+ },
2181
+ {
2182
+ "epoch": 0.5975598040157556,
2183
+ "grad_norm": 1.1802334785461426,
2184
+ "learning_rate": 0.00017763404240918706,
2185
+ "loss": 0.8007,
2186
+ "step": 3110
2187
+ },
2188
+ {
2189
+ "epoch": 0.599481218176578,
2190
+ "grad_norm": 1.464058756828308,
2191
+ "learning_rate": 0.00017617592456668177,
2192
+ "loss": 0.7304,
2193
+ "step": 3120
2194
+ },
2195
+ {
2196
+ "epoch": 0.6014026323374003,
2197
+ "grad_norm": 1.331037163734436,
2198
+ "learning_rate": 0.0001747205528050039,
2199
+ "loss": 0.7858,
2200
+ "step": 3130
2201
+ },
2202
+ {
2203
+ "epoch": 0.6033240464982227,
2204
+ "grad_norm": 0.8612129092216492,
2205
+ "learning_rate": 0.00017326798126053738,
2206
+ "loss": 0.7141,
2207
+ "step": 3140
2208
+ },
2209
+ {
2210
+ "epoch": 0.605245460659045,
2211
+ "grad_norm": 1.2344214916229248,
2212
+ "learning_rate": 0.00017181826396550477,
2213
+ "loss": 0.8229,
2214
+ "step": 3150
2215
+ },
2216
+ {
2217
+ "epoch": 0.6071668748198674,
2218
+ "grad_norm": 1.0575777292251587,
2219
+ "learning_rate": 0.00017037145484595712,
2220
+ "loss": 0.6958,
2221
+ "step": 3160
2222
+ },
2223
+ {
2224
+ "epoch": 0.6090882889806898,
2225
+ "grad_norm": 1.1421902179718018,
2226
+ "learning_rate": 0.0001689276077197684,
2227
+ "loss": 0.7866,
2228
+ "step": 3170
2229
+ },
2230
+ {
2231
+ "epoch": 0.6110097031415122,
2232
+ "grad_norm": 1.1171417236328125,
2233
+ "learning_rate": 0.00016748677629463331,
2234
+ "loss": 0.7247,
2235
+ "step": 3180
2236
+ },
2237
+ {
2238
+ "epoch": 0.6129311173023345,
2239
+ "grad_norm": 0.9187817573547363,
2240
+ "learning_rate": 0.00016604901416606988,
2241
+ "loss": 0.7873,
2242
+ "step": 3190
2243
+ },
2244
+ {
2245
+ "epoch": 0.6148525314631569,
2246
+ "grad_norm": 0.9294849038124084,
2247
+ "learning_rate": 0.00016461437481542527,
2248
+ "loss": 0.6915,
2249
+ "step": 3200
2250
+ },
2251
+ {
2252
+ "epoch": 0.6167739456239792,
2253
+ "grad_norm": 1.6183511018753052,
2254
+ "learning_rate": 0.00016318291160788678,
2255
+ "loss": 0.7612,
2256
+ "step": 3210
2257
+ },
2258
+ {
2259
+ "epoch": 0.6186953597848016,
2260
+ "grad_norm": 1.2286678552627563,
2261
+ "learning_rate": 0.00016175467779049683,
2262
+ "loss": 0.788,
2263
+ "step": 3220
2264
+ },
2265
+ {
2266
+ "epoch": 0.620616773945624,
2267
+ "grad_norm": 1.836376428604126,
2268
+ "learning_rate": 0.00016032972649017205,
2269
+ "loss": 0.7777,
2270
+ "step": 3230
2271
+ },
2272
+ {
2273
+ "epoch": 0.6225381881064463,
2274
+ "grad_norm": 1.4563829898834229,
2275
+ "learning_rate": 0.00015890811071172717,
2276
+ "loss": 0.6769,
2277
+ "step": 3240
2278
+ },
2279
+ {
2280
+ "epoch": 0.6244596022672687,
2281
+ "grad_norm": 1.6083195209503174,
2282
+ "learning_rate": 0.00015748988333590347,
2283
+ "loss": 0.8031,
2284
+ "step": 3250
2285
+ },
2286
+ {
2287
+ "epoch": 0.6263810164280911,
2288
+ "grad_norm": 1.3100241422653198,
2289
+ "learning_rate": 0.00015607509711740175,
2290
+ "loss": 0.7412,
2291
+ "step": 3260
2292
+ },
2293
+ {
2294
+ "epoch": 0.6283024305889134,
2295
+ "grad_norm": 1.613335132598877,
2296
+ "learning_rate": 0.00015466380468291986,
2297
+ "loss": 0.6871,
2298
+ "step": 3270
2299
+ },
2300
+ {
2301
+ "epoch": 0.6302238447497358,
2302
+ "grad_norm": 1.3441022634506226,
2303
+ "learning_rate": 0.00015325605852919502,
2304
+ "loss": 0.7887,
2305
+ "step": 3280
2306
+ },
2307
+ {
2308
+ "epoch": 0.6321452589105582,
2309
+ "grad_norm": 1.4960367679595947,
2310
+ "learning_rate": 0.00015185191102105134,
2311
+ "loss": 0.7291,
2312
+ "step": 3290
2313
+ },
2314
+ {
2315
+ "epoch": 0.6340666730713805,
2316
+ "grad_norm": 1.2021230459213257,
2317
+ "learning_rate": 0.00015045141438945186,
2318
+ "loss": 0.7821,
2319
+ "step": 3300
2320
+ },
2321
+ {
2322
+ "epoch": 0.6359880872322029,
2323
+ "grad_norm": 1.384557843208313,
2324
+ "learning_rate": 0.00014905462072955548,
2325
+ "loss": 0.7331,
2326
+ "step": 3310
2327
+ },
2328
+ {
2329
+ "epoch": 0.6379095013930253,
2330
+ "grad_norm": 1.8842626810073853,
2331
+ "learning_rate": 0.00014766158199877943,
2332
+ "loss": 0.7579,
2333
+ "step": 3320
2334
+ },
2335
+ {
2336
+ "epoch": 0.6398309155538476,
2337
+ "grad_norm": 1.3242464065551758,
2338
+ "learning_rate": 0.00014627235001486664,
2339
+ "loss": 0.7771,
2340
+ "step": 3330
2341
+ },
2342
+ {
2343
+ "epoch": 0.64175232971467,
2344
+ "grad_norm": 1.142120599746704,
2345
+ "learning_rate": 0.00014488697645395775,
2346
+ "loss": 0.7195,
2347
+ "step": 3340
2348
+ },
2349
+ {
2350
+ "epoch": 0.6436737438754924,
2351
+ "grad_norm": 1.052718162536621,
2352
+ "learning_rate": 0.00014350551284866942,
2353
+ "loss": 0.7954,
2354
+ "step": 3350
2355
+ },
2356
+ {
2357
+ "epoch": 0.6455951580363147,
2358
+ "grad_norm": 1.222748041152954,
2359
+ "learning_rate": 0.00014212801058617714,
2360
+ "loss": 0.7353,
2361
+ "step": 3360
2362
+ },
2363
+ {
2364
+ "epoch": 0.6475165721971371,
2365
+ "grad_norm": 1.3705120086669922,
2366
+ "learning_rate": 0.00014075452090630392,
2367
+ "loss": 0.7094,
2368
+ "step": 3370
2369
+ },
2370
+ {
2371
+ "epoch": 0.6494379863579595,
2372
+ "grad_norm": 1.1215436458587646,
2373
+ "learning_rate": 0.0001393850948996139,
2374
+ "loss": 0.8098,
2375
+ "step": 3380
2376
+ },
2377
+ {
2378
+ "epoch": 0.6513594005187818,
2379
+ "grad_norm": 1.4095954895019531,
2380
+ "learning_rate": 0.00013801978350551242,
2381
+ "loss": 0.7171,
2382
+ "step": 3390
2383
+ },
2384
+ {
2385
+ "epoch": 0.6532808146796042,
2386
+ "grad_norm": 0.9342811107635498,
2387
+ "learning_rate": 0.00013665863751035118,
2388
+ "loss": 0.7001,
2389
+ "step": 3400
2390
+ },
2391
+ {
2392
+ "epoch": 0.6552022288404266,
2393
+ "grad_norm": 0.9027429223060608,
2394
+ "learning_rate": 0.00013530170754553857,
2395
+ "loss": 0.7636,
2396
+ "step": 3410
2397
+ },
2398
+ {
2399
+ "epoch": 0.6571236430012489,
2400
+ "grad_norm": 1.1059980392456055,
2401
+ "learning_rate": 0.00013394904408565682,
2402
+ "loss": 0.6662,
2403
+ "step": 3420
2404
+ },
2405
+ {
2406
+ "epoch": 0.6590450571620713,
2407
+ "grad_norm": 1.402596116065979,
2408
+ "learning_rate": 0.00013260069744658404,
2409
+ "loss": 0.7067,
2410
+ "step": 3430
2411
+ },
2412
+ {
2413
+ "epoch": 0.6609664713228937,
2414
+ "grad_norm": 1.4740309715270996,
2415
+ "learning_rate": 0.00013125671778362307,
2416
+ "loss": 0.7453,
2417
+ "step": 3440
2418
+ },
2419
+ {
2420
+ "epoch": 0.662887885483716,
2421
+ "grad_norm": 1.6051650047302246,
2422
+ "learning_rate": 0.0001299171550896355,
2423
+ "loss": 0.7181,
2424
+ "step": 3450
2425
+ },
2426
+ {
2427
+ "epoch": 0.6648092996445384,
2428
+ "grad_norm": 1.1679162979125977,
2429
+ "learning_rate": 0.00012858205919318206,
2430
+ "loss": 0.7084,
2431
+ "step": 3460
2432
+ },
2433
+ {
2434
+ "epoch": 0.6667307138053608,
2435
+ "grad_norm": 1.467795729637146,
2436
+ "learning_rate": 0.00012725147975666947,
2437
+ "loss": 0.7222,
2438
+ "step": 3470
2439
+ },
2440
+ {
2441
+ "epoch": 0.6686521279661831,
2442
+ "grad_norm": 1.1850277185440063,
2443
+ "learning_rate": 0.00012592546627450242,
2444
+ "loss": 0.7365,
2445
+ "step": 3480
2446
+ },
2447
+ {
2448
+ "epoch": 0.6705735421270055,
2449
+ "grad_norm": 1.2745028734207153,
2450
+ "learning_rate": 0.00012460406807124314,
2451
+ "loss": 0.7828,
2452
+ "step": 3490
2453
+ },
2454
+ {
2455
+ "epoch": 0.6724949562878278,
2456
+ "grad_norm": 1.2522541284561157,
2457
+ "learning_rate": 0.0001232873342997764,
2458
+ "loss": 0.729,
2459
+ "step": 3500
2460
+ },
2461
+ {
2462
+ "epoch": 0.6744163704486502,
2463
+ "grad_norm": 1.3559709787368774,
2464
+ "learning_rate": 0.00012197531393948111,
2465
+ "loss": 0.7295,
2466
+ "step": 3510
2467
+ },
2468
+ {
2469
+ "epoch": 0.6763377846094726,
2470
+ "grad_norm": 1.270632028579712,
2471
+ "learning_rate": 0.00012066805579440849,
2472
+ "loss": 0.6994,
2473
+ "step": 3520
2474
+ },
2475
+ {
2476
+ "epoch": 0.678259198770295,
2477
+ "grad_norm": 1.3047348260879517,
2478
+ "learning_rate": 0.00011936560849146657,
2479
+ "loss": 0.6652,
2480
+ "step": 3530
2481
+ },
2482
+ {
2483
+ "epoch": 0.6801806129311173,
2484
+ "grad_norm": 1.5254732370376587,
2485
+ "learning_rate": 0.00011806802047861148,
2486
+ "loss": 0.7909,
2487
+ "step": 3540
2488
+ },
2489
+ {
2490
+ "epoch": 0.6821020270919397,
2491
+ "grad_norm": 1.30349600315094,
2492
+ "learning_rate": 0.00011677534002304529,
2493
+ "loss": 0.7628,
2494
+ "step": 3550
2495
+ },
2496
+ {
2497
+ "epoch": 0.684023441252762,
2498
+ "grad_norm": 1.0586496591567993,
2499
+ "learning_rate": 0.00011548761520942048,
2500
+ "loss": 0.6931,
2501
+ "step": 3560
2502
+ },
2503
+ {
2504
+ "epoch": 0.6859448554135844,
2505
+ "grad_norm": 1.3567843437194824,
2506
+ "learning_rate": 0.00011420489393805142,
2507
+ "loss": 0.6769,
2508
+ "step": 3570
2509
+ },
2510
+ {
2511
+ "epoch": 0.6878662695744068,
2512
+ "grad_norm": 1.2144553661346436,
2513
+ "learning_rate": 0.0001129272239231327,
2514
+ "loss": 0.711,
2515
+ "step": 3580
2516
+ },
2517
+ {
2518
+ "epoch": 0.6897876837352291,
2519
+ "grad_norm": 1.1030033826828003,
2520
+ "learning_rate": 0.00011165465269096375,
2521
+ "loss": 0.7395,
2522
+ "step": 3590
2523
+ },
2524
+ {
2525
+ "epoch": 0.6917090978960515,
2526
+ "grad_norm": 1.145440697669983,
2527
+ "learning_rate": 0.00011038722757818163,
2528
+ "loss": 0.6487,
2529
+ "step": 3600
2530
+ },
2531
+ {
2532
+ "epoch": 0.6936305120568739,
2533
+ "grad_norm": 1.0338114500045776,
2534
+ "learning_rate": 0.00010912499573000006,
2535
+ "loss": 0.6389,
2536
+ "step": 3610
2537
+ },
2538
+ {
2539
+ "epoch": 0.6955519262176962,
2540
+ "grad_norm": 1.4889144897460938,
2541
+ "learning_rate": 0.00010786800409845537,
2542
+ "loss": 0.7449,
2543
+ "step": 3620
2544
+ },
2545
+ {
2546
+ "epoch": 0.6974733403785186,
2547
+ "grad_norm": 1.3305447101593018,
2548
+ "learning_rate": 0.00010661629944066043,
2549
+ "loss": 0.6682,
2550
+ "step": 3630
2551
+ },
2552
+ {
2553
+ "epoch": 0.699394754539341,
2554
+ "grad_norm": 0.9120017886161804,
2555
+ "learning_rate": 0.0001053699283170649,
2556
+ "loss": 0.7316,
2557
+ "step": 3640
2558
+ },
2559
+ {
2560
+ "epoch": 0.7013161687001633,
2561
+ "grad_norm": 0.8771267533302307,
2562
+ "learning_rate": 0.00010412893708972387,
2563
+ "loss": 0.726,
2564
+ "step": 3650
2565
+ },
2566
+ {
2567
+ "epoch": 0.7032375828609857,
2568
+ "grad_norm": 0.9893380999565125,
2569
+ "learning_rate": 0.00010289337192057296,
2570
+ "loss": 0.6199,
2571
+ "step": 3660
2572
+ },
2573
+ {
2574
+ "epoch": 0.7051589970218081,
2575
+ "grad_norm": 1.0728754997253418,
2576
+ "learning_rate": 0.00010166327876971126,
2577
+ "loss": 0.7224,
2578
+ "step": 3670
2579
+ },
2580
+ {
2581
+ "epoch": 0.7070804111826304,
2582
+ "grad_norm": 0.4639236330986023,
2583
+ "learning_rate": 0.00010043870339369174,
2584
+ "loss": 0.7264,
2585
+ "step": 3680
2586
+ },
2587
+ {
2588
+ "epoch": 0.7090018253434528,
2589
+ "grad_norm": 1.6407727003097534,
2590
+ "learning_rate": 9.92196913438192e-05,
2591
+ "loss": 0.7552,
2592
+ "step": 3690
2593
+ },
2594
+ {
2595
+ "epoch": 0.7109232395042752,
2596
+ "grad_norm": 1.1802722215652466,
2597
+ "learning_rate": 9.800628796445585e-05,
2598
+ "loss": 0.7481,
2599
+ "step": 3700
2600
+ },
2601
+ {
2602
+ "epoch": 0.7128446536650975,
2603
+ "grad_norm": 1.6073251962661743,
2604
+ "learning_rate": 9.67985383913347e-05,
2605
+ "loss": 0.7225,
2606
+ "step": 3710
2607
+ },
2608
+ {
2609
+ "epoch": 0.7147660678259199,
2610
+ "grad_norm": 1.8788129091262817,
2611
+ "learning_rate": 9.559648754988054e-05,
2612
+ "loss": 0.6767,
2613
+ "step": 3720
2614
+ },
2615
+ {
2616
+ "epoch": 0.7166874819867423,
2617
+ "grad_norm": 1.3577216863632202,
2618
+ "learning_rate": 9.440018015353874e-05,
2619
+ "loss": 0.722,
2620
+ "step": 3730
2621
+ },
2622
+ {
2623
+ "epoch": 0.7186088961475646,
2624
+ "grad_norm": 1.0307360887527466,
2625
+ "learning_rate": 9.320966070211226e-05,
2626
+ "loss": 0.7395,
2627
+ "step": 3740
2628
+ },
2629
+ {
2630
+ "epoch": 0.720530310308387,
2631
+ "grad_norm": 1.6263340711593628,
2632
+ "learning_rate": 9.202497348010608e-05,
2633
+ "loss": 0.7294,
2634
+ "step": 3750
2635
+ },
2636
+ {
2637
+ "epoch": 0.7224517244692094,
2638
+ "grad_norm": 0.6837071776390076,
2639
+ "learning_rate": 9.084616255508013e-05,
2640
+ "loss": 0.6566,
2641
+ "step": 3760
2642
+ },
2643
+ {
2644
+ "epoch": 0.7243731386300317,
2645
+ "grad_norm": 0.9681917428970337,
2646
+ "learning_rate": 8.967327177600997e-05,
2647
+ "loss": 0.7113,
2648
+ "step": 3770
2649
+ },
2650
+ {
2651
+ "epoch": 0.7262945527908541,
2652
+ "grad_norm": 1.1300331354141235,
2653
+ "learning_rate": 8.850634477165581e-05,
2654
+ "loss": 0.7455,
2655
+ "step": 3780
2656
+ },
2657
+ {
2658
+ "epoch": 0.7282159669516765,
2659
+ "grad_norm": 1.0961904525756836,
2660
+ "learning_rate": 8.734542494893955e-05,
2661
+ "loss": 0.6819,
2662
+ "step": 3790
2663
+ },
2664
+ {
2665
+ "epoch": 0.7301373811124988,
2666
+ "grad_norm": 0.8156876564025879,
2667
+ "learning_rate": 8.619055549132992e-05,
2668
+ "loss": 0.6541,
2669
+ "step": 3800
2670
+ },
2671
+ {
2672
+ "epoch": 0.7320587952733212,
2673
+ "grad_norm": 1.1605948209762573,
2674
+ "learning_rate": 8.504177935723672e-05,
2675
+ "loss": 0.6239,
2676
+ "step": 3810
2677
+ },
2678
+ {
2679
+ "epoch": 0.7339802094341435,
2680
+ "grad_norm": 1.2597169876098633,
2681
+ "learning_rate": 8.389913927841231e-05,
2682
+ "loss": 0.6716,
2683
+ "step": 3820
2684
+ },
2685
+ {
2686
+ "epoch": 0.7359016235949659,
2687
+ "grad_norm": 0.7605863213539124,
2688
+ "learning_rate": 8.276267775836266e-05,
2689
+ "loss": 0.6895,
2690
+ "step": 3830
2691
+ },
2692
+ {
2693
+ "epoch": 0.7378230377557883,
2694
+ "grad_norm": 1.0140622854232788,
2695
+ "learning_rate": 8.163243707076548e-05,
2696
+ "loss": 0.64,
2697
+ "step": 3840
2698
+ },
2699
+ {
2700
+ "epoch": 0.7397444519166106,
2701
+ "grad_norm": 0.9782150387763977,
2702
+ "learning_rate": 8.050845925789862e-05,
2703
+ "loss": 0.6633,
2704
+ "step": 3850
2705
+ },
2706
+ {
2707
+ "epoch": 0.741665866077433,
2708
+ "grad_norm": 1.9944454431533813,
2709
+ "learning_rate": 7.939078612907567e-05,
2710
+ "loss": 0.7092,
2711
+ "step": 3860
2712
+ },
2713
+ {
2714
+ "epoch": 0.7435872802382554,
2715
+ "grad_norm": 0.9737130403518677,
2716
+ "learning_rate": 7.827945925909094e-05,
2717
+ "loss": 0.6458,
2718
+ "step": 3870
2719
+ },
2720
+ {
2721
+ "epoch": 0.7455086943990777,
2722
+ "grad_norm": 0.9813506007194519,
2723
+ "learning_rate": 7.71745199866729e-05,
2724
+ "loss": 0.6435,
2725
+ "step": 3880
2726
+ },
2727
+ {
2728
+ "epoch": 0.7474301085599001,
2729
+ "grad_norm": 0.6654618978500366,
2730
+ "learning_rate": 7.607600941294657e-05,
2731
+ "loss": 0.6256,
2732
+ "step": 3890
2733
+ },
2734
+ {
2735
+ "epoch": 0.7493515227207225,
2736
+ "grad_norm": 1.6668603420257568,
2737
+ "learning_rate": 7.498396839990456e-05,
2738
+ "loss": 0.7146,
2739
+ "step": 3900
2740
+ },
2741
+ {
2742
+ "epoch": 0.7512729368815448,
2743
+ "grad_norm": 1.2132741212844849,
2744
+ "learning_rate": 7.389843756888712e-05,
2745
+ "loss": 0.6987,
2746
+ "step": 3910
2747
+ },
2748
+ {
2749
+ "epoch": 0.7531943510423672,
2750
+ "grad_norm": 0.9006101489067078,
2751
+ "learning_rate": 7.281945729907119e-05,
2752
+ "loss": 0.6177,
2753
+ "step": 3920
2754
+ },
2755
+ {
2756
+ "epoch": 0.7551157652031896,
2757
+ "grad_norm": 1.2604845762252808,
2758
+ "learning_rate": 7.174706772596831e-05,
2759
+ "loss": 0.6868,
2760
+ "step": 3930
2761
+ },
2762
+ {
2763
+ "epoch": 0.7570371793640119,
2764
+ "grad_norm": 1.1921509504318237,
2765
+ "learning_rate": 7.068130873993181e-05,
2766
+ "loss": 0.6461,
2767
+ "step": 3940
2768
+ },
2769
+ {
2770
+ "epoch": 0.7589585935248343,
2771
+ "grad_norm": 0.9460396766662598,
2772
+ "learning_rate": 6.962221998467261e-05,
2773
+ "loss": 0.7165,
2774
+ "step": 3950
2775
+ },
2776
+ {
2777
+ "epoch": 0.7608800076856567,
2778
+ "grad_norm": 1.009789228439331,
2779
+ "learning_rate": 6.8569840855785e-05,
2780
+ "loss": 0.7098,
2781
+ "step": 3960
2782
+ },
2783
+ {
2784
+ "epoch": 0.762801421846479,
2785
+ "grad_norm": 1.3344640731811523,
2786
+ "learning_rate": 6.752421049928123e-05,
2787
+ "loss": 0.6595,
2788
+ "step": 3970
2789
+ },
2790
+ {
2791
+ "epoch": 0.7647228360073014,
2792
+ "grad_norm": 1.0439685583114624,
2793
+ "learning_rate": 6.648536781013495e-05,
2794
+ "loss": 0.6576,
2795
+ "step": 3980
2796
+ },
2797
+ {
2798
+ "epoch": 0.7666442501681238,
2799
+ "grad_norm": 0.7947928309440613,
2800
+ "learning_rate": 6.545335143083483e-05,
2801
+ "loss": 0.7006,
2802
+ "step": 3990
2803
+ },
2804
+ {
2805
+ "epoch": 0.7685656643289461,
2806
+ "grad_norm": 1.037701964378357,
2807
+ "learning_rate": 6.442819974994669e-05,
2808
+ "loss": 0.7077,
2809
+ "step": 4000
2810
+ },
2811
+ {
2812
+ "epoch": 0.7704870784897685,
2813
+ "grad_norm": 1.210694670677185,
2814
+ "learning_rate": 6.34099509006861e-05,
2815
+ "loss": 0.7446,
2816
+ "step": 4010
2817
+ },
2818
+ {
2819
+ "epoch": 0.7724084926505909,
2820
+ "grad_norm": 1.6502310037612915,
2821
+ "learning_rate": 6.239864275949958e-05,
2822
+ "loss": 0.6645,
2823
+ "step": 4020
2824
+ },
2825
+ {
2826
+ "epoch": 0.7743299068114132,
2827
+ "grad_norm": 0.899091362953186,
2828
+ "learning_rate": 6.139431294465558e-05,
2829
+ "loss": 0.693,
2830
+ "step": 4030
2831
+ },
2832
+ {
2833
+ "epoch": 0.7762513209722356,
2834
+ "grad_norm": 1.1899176836013794,
2835
+ "learning_rate": 6.03969988148457e-05,
2836
+ "loss": 0.7106,
2837
+ "step": 4040
2838
+ },
2839
+ {
2840
+ "epoch": 0.778172735133058,
2841
+ "grad_norm": 0.8680855631828308,
2842
+ "learning_rate": 5.940673746779421e-05,
2843
+ "loss": 0.6068,
2844
+ "step": 4050
2845
+ },
2846
+ {
2847
+ "epoch": 0.7800941492938803,
2848
+ "grad_norm": 0.9416540265083313,
2849
+ "learning_rate": 5.842356573887886e-05,
2850
+ "loss": 0.6198,
2851
+ "step": 4060
2852
+ },
2853
+ {
2854
+ "epoch": 0.7820155634547027,
2855
+ "grad_norm": 1.4808692932128906,
2856
+ "learning_rate": 5.744752019976027e-05,
2857
+ "loss": 0.6638,
2858
+ "step": 4070
2859
+ },
2860
+ {
2861
+ "epoch": 0.783936977615525,
2862
+ "grad_norm": 0.8431961536407471,
2863
+ "learning_rate": 5.647863715702173e-05,
2864
+ "loss": 0.6449,
2865
+ "step": 4080
2866
+ },
2867
+ {
2868
+ "epoch": 0.7858583917763474,
2869
+ "grad_norm": 0.9810335636138916,
2870
+ "learning_rate": 5.5516952650818605e-05,
2871
+ "loss": 0.6363,
2872
+ "step": 4090
2873
+ },
2874
+ {
2875
+ "epoch": 0.7877798059371698,
2876
+ "grad_norm": 1.1404001712799072,
2877
+ "learning_rate": 5.45625024535377e-05,
2878
+ "loss": 0.6726,
2879
+ "step": 4100
2880
+ },
2881
+ {
2882
+ "epoch": 0.7897012200979922,
2883
+ "grad_norm": 1.1580007076263428,
2884
+ "learning_rate": 5.36153220684667e-05,
2885
+ "loss": 0.6647,
2886
+ "step": 4110
2887
+ },
2888
+ {
2889
+ "epoch": 0.7916226342588145,
2890
+ "grad_norm": 1.1604604721069336,
2891
+ "learning_rate": 5.267544672847352e-05,
2892
+ "loss": 0.7539,
2893
+ "step": 4120
2894
+ },
2895
+ {
2896
+ "epoch": 0.7935440484196369,
2897
+ "grad_norm": 0.8880569338798523,
2898
+ "learning_rate": 5.174291139469559e-05,
2899
+ "loss": 0.6777,
2900
+ "step": 4130
2901
+ },
2902
+ {
2903
+ "epoch": 0.7954654625804592,
2904
+ "grad_norm": 1.15243399143219,
2905
+ "learning_rate": 5.081775075523959e-05,
2906
+ "loss": 0.6044,
2907
+ "step": 4140
2908
+ },
2909
+ {
2910
+ "epoch": 0.7973868767412816,
2911
+ "grad_norm": 1.313981056213379,
2912
+ "learning_rate": 4.989999922389102e-05,
2913
+ "loss": 0.6544,
2914
+ "step": 4150
2915
+ },
2916
+ {
2917
+ "epoch": 0.799308290902104,
2918
+ "grad_norm": 1.1018239259719849,
2919
+ "learning_rate": 4.898969093883396e-05,
2920
+ "loss": 0.6708,
2921
+ "step": 4160
2922
+ },
2923
+ {
2924
+ "epoch": 0.8012297050629263,
2925
+ "grad_norm": 1.0654897689819336,
2926
+ "learning_rate": 4.808685976138147e-05,
2927
+ "loss": 0.7295,
2928
+ "step": 4170
2929
+ },
2930
+ {
2931
+ "epoch": 0.8031511192237487,
2932
+ "grad_norm": 0.6468730568885803,
2933
+ "learning_rate": 4.719153927471598e-05,
2934
+ "loss": 0.6127,
2935
+ "step": 4180
2936
+ },
2937
+ {
2938
+ "epoch": 0.8050725333845711,
2939
+ "grad_norm": 0.9620100259780884,
2940
+ "learning_rate": 4.6303762782639895e-05,
2941
+ "loss": 0.6429,
2942
+ "step": 4190
2943
+ },
2944
+ {
2945
+ "epoch": 0.8069939475453934,
2946
+ "grad_norm": 1.0371164083480835,
2947
+ "learning_rate": 4.542356330833697e-05,
2948
+ "loss": 0.6625,
2949
+ "step": 4200
2950
+ },
2951
+ {
2952
+ "epoch": 0.8089153617062158,
2953
+ "grad_norm": 1.0234382152557373,
2954
+ "learning_rate": 4.455097359314361e-05,
2955
+ "loss": 0.6418,
2956
+ "step": 4210
2957
+ },
2958
+ {
2959
+ "epoch": 0.8108367758670382,
2960
+ "grad_norm": 1.1065011024475098,
2961
+ "learning_rate": 4.368602609533146e-05,
2962
+ "loss": 0.6692,
2963
+ "step": 4220
2964
+ },
2965
+ {
2966
+ "epoch": 0.8127581900278605,
2967
+ "grad_norm": 0.9469298124313354,
2968
+ "learning_rate": 4.282875298889966e-05,
2969
+ "loss": 0.6272,
2970
+ "step": 4230
2971
+ },
2972
+ {
2973
+ "epoch": 0.8146796041886829,
2974
+ "grad_norm": 0.7343427538871765,
2975
+ "learning_rate": 4.1979186162378115e-05,
2976
+ "loss": 0.6458,
2977
+ "step": 4240
2978
+ },
2979
+ {
2980
+ "epoch": 0.8166010183495053,
2981
+ "grad_norm": 1.0217844247817993,
2982
+ "learning_rate": 4.113735721764161e-05,
2983
+ "loss": 0.6489,
2984
+ "step": 4250
2985
+ },
2986
+ {
2987
+ "epoch": 0.8185224325103276,
2988
+ "grad_norm": 0.8793905377388,
2989
+ "learning_rate": 4.030329746873365e-05,
2990
+ "loss": 0.6778,
2991
+ "step": 4260
2992
+ },
2993
+ {
2994
+ "epoch": 0.82044384667115,
2995
+ "grad_norm": 1.3632718324661255,
2996
+ "learning_rate": 3.9477037940702346e-05,
2997
+ "loss": 0.6279,
2998
+ "step": 4270
2999
+ },
3000
+ {
3001
+ "epoch": 0.8223652608319724,
3002
+ "grad_norm": 0.9788404703140259,
3003
+ "learning_rate": 3.865860936844595e-05,
3004
+ "loss": 0.6902,
3005
+ "step": 4280
3006
+ },
3007
+ {
3008
+ "epoch": 0.8242866749927947,
3009
+ "grad_norm": 0.5577342510223389,
3010
+ "learning_rate": 3.7848042195569725e-05,
3011
+ "loss": 0.5732,
3012
+ "step": 4290
3013
+ },
3014
+ {
3015
+ "epoch": 0.8262080891536171,
3016
+ "grad_norm": 1.5902769565582275,
3017
+ "learning_rate": 3.704536657325347e-05,
3018
+ "loss": 0.6999,
3019
+ "step": 4300
3020
+ },
3021
+ {
3022
+ "epoch": 0.8281295033144395,
3023
+ "grad_norm": 1.079472303390503,
3024
+ "learning_rate": 3.625061235913002e-05,
3025
+ "loss": 0.6913,
3026
+ "step": 4310
3027
+ },
3028
+ {
3029
+ "epoch": 0.8300509174752618,
3030
+ "grad_norm": 1.0420955419540405,
3031
+ "learning_rate": 3.5463809116174555e-05,
3032
+ "loss": 0.632,
3033
+ "step": 4320
3034
+ },
3035
+ {
3036
+ "epoch": 0.8319723316360842,
3037
+ "grad_norm": 0.9489886164665222,
3038
+ "learning_rate": 3.468498611160495e-05,
3039
+ "loss": 0.6202,
3040
+ "step": 4330
3041
+ },
3042
+ {
3043
+ "epoch": 0.8338937457969066,
3044
+ "grad_norm": 1.1345503330230713,
3045
+ "learning_rate": 3.391417231579308e-05,
3046
+ "loss": 0.6264,
3047
+ "step": 4340
3048
+ },
3049
+ {
3050
+ "epoch": 0.8358151599577289,
3051
+ "grad_norm": 1.043662667274475,
3052
+ "learning_rate": 3.315139640118728e-05,
3053
+ "loss": 0.5818,
3054
+ "step": 4350
3055
+ },
3056
+ {
3057
+ "epoch": 0.8377365741185513,
3058
+ "grad_norm": 1.3796449899673462,
3059
+ "learning_rate": 3.2396686741245765e-05,
3060
+ "loss": 0.6731,
3061
+ "step": 4360
3062
+ },
3063
+ {
3064
+ "epoch": 0.8396579882793737,
3065
+ "grad_norm": 0.87774258852005,
3066
+ "learning_rate": 3.165007140938095e-05,
3067
+ "loss": 0.7172,
3068
+ "step": 4370
3069
+ },
3070
+ {
3071
+ "epoch": 0.841579402440196,
3072
+ "grad_norm": 0.8624753952026367,
3073
+ "learning_rate": 3.0911578177915595e-05,
3074
+ "loss": 0.6313,
3075
+ "step": 4380
3076
+ },
3077
+ {
3078
+ "epoch": 0.8435008166010184,
3079
+ "grad_norm": 0.7553603053092957,
3080
+ "learning_rate": 3.0181234517049654e-05,
3081
+ "loss": 0.5941,
3082
+ "step": 4390
3083
+ },
3084
+ {
3085
+ "epoch": 0.8454222307618408,
3086
+ "grad_norm": 1.1863903999328613,
3087
+ "learning_rate": 2.945906759383815e-05,
3088
+ "loss": 0.6725,
3089
+ "step": 4400
3090
+ },
3091
+ {
3092
+ "epoch": 0.8473436449226631,
3093
+ "grad_norm": 0.886174201965332,
3094
+ "learning_rate": 2.8745104271180933e-05,
3095
+ "loss": 0.6407,
3096
+ "step": 4410
3097
+ },
3098
+ {
3099
+ "epoch": 0.8492650590834855,
3100
+ "grad_norm": 0.7463008165359497,
3101
+ "learning_rate": 2.8039371106823196e-05,
3102
+ "loss": 0.577,
3103
+ "step": 4420
3104
+ },
3105
+ {
3106
+ "epoch": 0.8511864732443079,
3107
+ "grad_norm": 1.0898914337158203,
3108
+ "learning_rate": 2.734189435236789e-05,
3109
+ "loss": 0.6662,
3110
+ "step": 4430
3111
+ },
3112
+ {
3113
+ "epoch": 0.8531078874051302,
3114
+ "grad_norm": 1.42017662525177,
3115
+ "learning_rate": 2.6652699952298994e-05,
3116
+ "loss": 0.6746,
3117
+ "step": 4440
3118
+ },
3119
+ {
3120
+ "epoch": 0.8550293015659526,
3121
+ "grad_norm": 0.884002149105072,
3122
+ "learning_rate": 2.5971813543016475e-05,
3123
+ "loss": 0.6162,
3124
+ "step": 4450
3125
+ },
3126
+ {
3127
+ "epoch": 0.856950715726775,
3128
+ "grad_norm": 1.4128063917160034,
3129
+ "learning_rate": 2.529926045188291e-05,
3130
+ "loss": 0.6601,
3131
+ "step": 4460
3132
+ },
3133
+ {
3134
+ "epoch": 0.8588721298875973,
3135
+ "grad_norm": 0.7436163425445557,
3136
+ "learning_rate": 2.463506569628085e-05,
3137
+ "loss": 0.6425,
3138
+ "step": 4470
3139
+ },
3140
+ {
3141
+ "epoch": 0.8607935440484197,
3142
+ "grad_norm": 1.0224891901016235,
3143
+ "learning_rate": 2.39792539826828e-05,
3144
+ "loss": 0.6066,
3145
+ "step": 4480
3146
+ },
3147
+ {
3148
+ "epoch": 0.862714958209242,
3149
+ "grad_norm": 1.1273442506790161,
3150
+ "learning_rate": 2.3331849705731876e-05,
3151
+ "loss": 0.5953,
3152
+ "step": 4490
3153
+ },
3154
+ {
3155
+ "epoch": 0.8646363723700644,
3156
+ "grad_norm": 1.1450196504592896,
3157
+ "learning_rate": 2.2692876947334406e-05,
3158
+ "loss": 0.6013,
3159
+ "step": 4500
3160
+ },
3161
+ {
3162
+ "epoch": 0.8665577865308868,
3163
+ "grad_norm": 1.3181530237197876,
3164
+ "learning_rate": 2.2062359475764266e-05,
3165
+ "loss": 0.67,
3166
+ "step": 4510
3167
+ },
3168
+ {
3169
+ "epoch": 0.8684792006917091,
3170
+ "grad_norm": 1.0618128776550293,
3171
+ "learning_rate": 2.144032074477861e-05,
3172
+ "loss": 0.6158,
3173
+ "step": 4520
3174
+ },
3175
+ {
3176
+ "epoch": 0.8704006148525315,
3177
+ "grad_norm": 0.7197682857513428,
3178
+ "learning_rate": 2.0826783892745617e-05,
3179
+ "loss": 0.5484,
3180
+ "step": 4530
3181
+ },
3182
+ {
3183
+ "epoch": 0.8723220290133539,
3184
+ "grad_norm": 1.1397417783737183,
3185
+ "learning_rate": 2.0221771741783578e-05,
3186
+ "loss": 0.5875,
3187
+ "step": 4540
3188
+ },
3189
+ {
3190
+ "epoch": 0.8742434431741762,
3191
+ "grad_norm": 1.1843700408935547,
3192
+ "learning_rate": 1.9625306796912158e-05,
3193
+ "loss": 0.6341,
3194
+ "step": 4550
3195
+ },
3196
+ {
3197
+ "epoch": 0.8761648573349986,
3198
+ "grad_norm": 1.336387038230896,
3199
+ "learning_rate": 1.903741124521516e-05,
3200
+ "loss": 0.6003,
3201
+ "step": 4560
3202
+ },
3203
+ {
3204
+ "epoch": 0.878086271495821,
3205
+ "grad_norm": 0.7944359183311462,
3206
+ "learning_rate": 1.8458106955015318e-05,
3207
+ "loss": 0.6365,
3208
+ "step": 4570
3209
+ },
3210
+ {
3211
+ "epoch": 0.8800076856566433,
3212
+ "grad_norm": 0.6551217436790466,
3213
+ "learning_rate": 1.7887415475060646e-05,
3214
+ "loss": 0.6544,
3215
+ "step": 4580
3216
+ },
3217
+ {
3218
+ "epoch": 0.8819290998174657,
3219
+ "grad_norm": 1.1356467008590698,
3220
+ "learning_rate": 1.7325358033723092e-05,
3221
+ "loss": 0.7436,
3222
+ "step": 4590
3223
+ },
3224
+ {
3225
+ "epoch": 0.8838505139782881,
3226
+ "grad_norm": 0.9559223651885986,
3227
+ "learning_rate": 1.6771955538208843e-05,
3228
+ "loss": 0.6652,
3229
+ "step": 4600
3230
+ },
3231
+ {
3232
+ "epoch": 0.8857719281391104,
3233
+ "grad_norm": 1.0125079154968262,
3234
+ "learning_rate": 1.622722857378056e-05,
3235
+ "loss": 0.6105,
3236
+ "step": 4610
3237
+ },
3238
+ {
3239
+ "epoch": 0.8876933422999328,
3240
+ "grad_norm": 0.899423360824585,
3241
+ "learning_rate": 1.5691197402991684e-05,
3242
+ "loss": 0.5898,
3243
+ "step": 4620
3244
+ },
3245
+ {
3246
+ "epoch": 0.8896147564607552,
3247
+ "grad_norm": 1.1466995477676392,
3248
+ "learning_rate": 1.5163881964932653e-05,
3249
+ "loss": 0.6615,
3250
+ "step": 4630
3251
+ },
3252
+ {
3253
+ "epoch": 0.8915361706215775,
3254
+ "grad_norm": 1.5939006805419922,
3255
+ "learning_rate": 1.4645301874489342e-05,
3256
+ "loss": 0.7415,
3257
+ "step": 4640
3258
+ },
3259
+ {
3260
+ "epoch": 0.8934575847823999,
3261
+ "grad_norm": 0.6204648613929749,
3262
+ "learning_rate": 1.4135476421613419e-05,
3263
+ "loss": 0.6184,
3264
+ "step": 4650
3265
+ },
3266
+ {
3267
+ "epoch": 0.8953789989432223,
3268
+ "grad_norm": 1.0177160501480103,
3269
+ "learning_rate": 1.3634424570604682e-05,
3270
+ "loss": 0.6398,
3271
+ "step": 4660
3272
+ },
3273
+ {
3274
+ "epoch": 0.8973004131040446,
3275
+ "grad_norm": 1.6055784225463867,
3276
+ "learning_rate": 1.3142164959405817e-05,
3277
+ "loss": 0.6313,
3278
+ "step": 4670
3279
+ },
3280
+ {
3281
+ "epoch": 0.899221827264867,
3282
+ "grad_norm": 1.299264669418335,
3283
+ "learning_rate": 1.265871589890885e-05,
3284
+ "loss": 0.7217,
3285
+ "step": 4680
3286
+ },
3287
+ {
3288
+ "epoch": 0.9011432414256894,
3289
+ "grad_norm": 1.453226089477539,
3290
+ "learning_rate": 1.2184095372274301e-05,
3291
+ "loss": 0.6966,
3292
+ "step": 4690
3293
+ },
3294
+ {
3295
+ "epoch": 0.9030646555865117,
3296
+ "grad_norm": 1.0507123470306396,
3297
+ "learning_rate": 1.1718321034262125e-05,
3298
+ "loss": 0.6368,
3299
+ "step": 4700
3300
+ },
3301
+ {
3302
+ "epoch": 0.9049860697473341,
3303
+ "grad_norm": 0.8116253614425659,
3304
+ "learning_rate": 1.1261410210574918e-05,
3305
+ "loss": 0.6051,
3306
+ "step": 4710
3307
+ },
3308
+ {
3309
+ "epoch": 0.9069074839081565,
3310
+ "grad_norm": 1.0800602436065674,
3311
+ "learning_rate": 1.0813379897213593e-05,
3312
+ "loss": 0.628,
3313
+ "step": 4720
3314
+ },
3315
+ {
3316
+ "epoch": 0.9088288980689788,
3317
+ "grad_norm": 1.0864616632461548,
3318
+ "learning_rate": 1.0374246759845134e-05,
3319
+ "loss": 0.5293,
3320
+ "step": 4730
3321
+ },
3322
+ {
3323
+ "epoch": 0.9107503122298012,
3324
+ "grad_norm": 1.0752801895141602,
3325
+ "learning_rate": 9.94402713318257e-06,
3326
+ "loss": 0.6651,
3327
+ "step": 4740
3328
+ },
3329
+ {
3330
+ "epoch": 0.9126717263906235,
3331
+ "grad_norm": 1.0369071960449219,
3332
+ "learning_rate": 9.52273702037748e-06,
3333
+ "loss": 0.6574,
3334
+ "step": 4750
3335
+ },
3336
+ {
3337
+ "epoch": 0.9145931405514459,
3338
+ "grad_norm": 1.087040662765503,
3339
+ "learning_rate": 9.110392092424647e-06,
3340
+ "loss": 0.6523,
3341
+ "step": 4760
3342
+ },
3343
+ {
3344
+ "epoch": 0.9165145547122683,
3345
+ "grad_norm": 0.632030189037323,
3346
+ "learning_rate": 8.707007687579177e-06,
3347
+ "loss": 0.6064,
3348
+ "step": 4770
3349
+ },
3350
+ {
3351
+ "epoch": 0.9184359688730906,
3352
+ "grad_norm": 1.062684178352356,
3353
+ "learning_rate": 8.312598810785943e-06,
3354
+ "loss": 0.6736,
3355
+ "step": 4780
3356
+ },
3357
+ {
3358
+ "epoch": 0.920357383033913,
3359
+ "grad_norm": 1.3037798404693604,
3360
+ "learning_rate": 7.927180133121298e-06,
3361
+ "loss": 0.6999,
3362
+ "step": 4790
3363
+ },
3364
+ {
3365
+ "epoch": 0.9222787971947354,
3366
+ "grad_norm": 0.8168327808380127,
3367
+ "learning_rate": 7.550765991247654e-06,
3368
+ "loss": 0.5788,
3369
+ "step": 4800
3370
+ },
3371
+ {
3372
+ "epoch": 0.9242002113555577,
3373
+ "grad_norm": 0.6602869629859924,
3374
+ "learning_rate": 7.183370386879884e-06,
3375
+ "loss": 0.6263,
3376
+ "step": 4810
3377
+ },
3378
+ {
3379
+ "epoch": 0.9261216255163801,
3380
+ "grad_norm": 0.7275961637496948,
3381
+ "learning_rate": 6.825006986264703e-06,
3382
+ "loss": 0.6584,
3383
+ "step": 4820
3384
+ },
3385
+ {
3386
+ "epoch": 0.9280430396772025,
3387
+ "grad_norm": 1.1119617223739624,
3388
+ "learning_rate": 6.475689119672168e-06,
3389
+ "loss": 0.6648,
3390
+ "step": 4830
3391
+ },
3392
+ {
3393
+ "epoch": 0.9299644538380248,
3394
+ "grad_norm": 0.8867122530937195,
3395
+ "learning_rate": 6.135429780899926e-06,
3396
+ "loss": 0.5792,
3397
+ "step": 4840
3398
+ },
3399
+ {
3400
+ "epoch": 0.9318858679988472,
3401
+ "grad_norm": 1.2685691118240356,
3402
+ "learning_rate": 5.804241626789747e-06,
3403
+ "loss": 0.5848,
3404
+ "step": 4850
3405
+ },
3406
+ {
3407
+ "epoch": 0.9338072821596696,
3408
+ "grad_norm": 1.0925408601760864,
3409
+ "learning_rate": 5.482136976756952e-06,
3410
+ "loss": 0.6466,
3411
+ "step": 4860
3412
+ },
3413
+ {
3414
+ "epoch": 0.9357286963204919,
3415
+ "grad_norm": 0.6285922527313232,
3416
+ "learning_rate": 5.169127812331892e-06,
3417
+ "loss": 0.6179,
3418
+ "step": 4870
3419
+ },
3420
+ {
3421
+ "epoch": 0.9376501104813142,
3422
+ "grad_norm": 2.0445570945739746,
3423
+ "learning_rate": 4.865225776714471e-06,
3424
+ "loss": 0.7558,
3425
+ "step": 4880
3426
+ },
3427
+ {
3428
+ "epoch": 0.9395715246421366,
3429
+ "grad_norm": 0.9207888245582581,
3430
+ "learning_rate": 4.570442174340883e-06,
3431
+ "loss": 0.6298,
3432
+ "step": 4890
3433
+ },
3434
+ {
3435
+ "epoch": 0.9414929388029589,
3436
+ "grad_norm": 0.5049605369567871,
3437
+ "learning_rate": 4.284787970463277e-06,
3438
+ "loss": 0.5918,
3439
+ "step": 4900
3440
+ },
3441
+ {
3442
+ "epoch": 0.9434143529637813,
3443
+ "grad_norm": 0.7330450415611267,
3444
+ "learning_rate": 4.008273790741701e-06,
3445
+ "loss": 0.5832,
3446
+ "step": 4910
3447
+ },
3448
+ {
3449
+ "epoch": 0.9453357671246037,
3450
+ "grad_norm": 1.1495826244354248,
3451
+ "learning_rate": 3.7409099208490506e-06,
3452
+ "loss": 0.6474,
3453
+ "step": 4920
3454
+ },
3455
+ {
3456
+ "epoch": 0.947257181285426,
3457
+ "grad_norm": 1.0758661031723022,
3458
+ "learning_rate": 3.4827063060882404e-06,
3459
+ "loss": 0.6798,
3460
+ "step": 4930
3461
+ },
3462
+ {
3463
+ "epoch": 0.9491785954462484,
3464
+ "grad_norm": 0.7488206624984741,
3465
+ "learning_rate": 3.2336725510224986e-06,
3466
+ "loss": 0.5957,
3467
+ "step": 4940
3468
+ },
3469
+ {
3470
+ "epoch": 0.9511000096070708,
3471
+ "grad_norm": 0.78911954164505,
3472
+ "learning_rate": 2.993817919117875e-06,
3473
+ "loss": 0.6642,
3474
+ "step": 4950
3475
+ },
3476
+ {
3477
+ "epoch": 0.9530214237678931,
3478
+ "grad_norm": 0.4790767729282379,
3479
+ "learning_rate": 2.7631513323988777e-06,
3480
+ "loss": 0.6532,
3481
+ "step": 4960
3482
+ },
3483
+ {
3484
+ "epoch": 0.9549428379287155,
3485
+ "grad_norm": 1.1064733266830444,
3486
+ "learning_rate": 2.5416813711163777e-06,
3487
+ "loss": 0.7227,
3488
+ "step": 4970
3489
+ },
3490
+ {
3491
+ "epoch": 0.9568642520895378,
3492
+ "grad_norm": 1.0712919235229492,
3493
+ "learning_rate": 2.329416273428614e-06,
3494
+ "loss": 0.6789,
3495
+ "step": 4980
3496
+ },
3497
+ {
3498
+ "epoch": 0.9587856662503602,
3499
+ "grad_norm": 0.9066910147666931,
3500
+ "learning_rate": 2.1263639350946884e-06,
3501
+ "loss": 0.6153,
3502
+ "step": 4990
3503
+ },
3504
+ {
3505
+ "epoch": 0.9607070804111826,
3506
+ "grad_norm": 0.6191006898880005,
3507
+ "learning_rate": 1.9325319091808845e-06,
3508
+ "loss": 0.7482,
3509
+ "step": 5000
3510
+ },
3511
+ {
3512
+ "epoch": 0.962628494572005,
3513
+ "grad_norm": 0.6870371103286743,
3514
+ "learning_rate": 1.7479274057796146e-06,
3515
+ "loss": 0.5674,
3516
+ "step": 5010
3517
+ },
3518
+ {
3519
+ "epoch": 0.9645499087328273,
3520
+ "grad_norm": 1.2739887237548828,
3521
+ "learning_rate": 1.572557291741411e-06,
3522
+ "loss": 0.6202,
3523
+ "step": 5020
3524
+ },
3525
+ {
3526
+ "epoch": 0.9664713228936497,
3527
+ "grad_norm": 0.7925472259521484,
3528
+ "learning_rate": 1.4064280904192983e-06,
3529
+ "loss": 0.6017,
3530
+ "step": 5030
3531
+ },
3532
+ {
3533
+ "epoch": 0.968392737054472,
3534
+ "grad_norm": 1.0697252750396729,
3535
+ "learning_rate": 1.2495459814262366e-06,
3536
+ "loss": 0.5934,
3537
+ "step": 5040
3538
+ },
3539
+ {
3540
+ "epoch": 0.9703141512152944,
3541
+ "grad_norm": 0.6690041422843933,
3542
+ "learning_rate": 1.101916800405306e-06,
3543
+ "loss": 0.5856,
3544
+ "step": 5050
3545
+ },
3546
+ {
3547
+ "epoch": 0.9722355653761168,
3548
+ "grad_norm": 1.0088361501693726,
3549
+ "learning_rate": 9.635460388124629e-07,
3550
+ "loss": 0.525,
3551
+ "step": 5060
3552
+ },
3553
+ {
3554
+ "epoch": 0.9741569795369391,
3555
+ "grad_norm": 1.1077866554260254,
3556
+ "learning_rate": 8.344388437125372e-07,
3557
+ "loss": 0.6342,
3558
+ "step": 5070
3559
+ },
3560
+ {
3561
+ "epoch": 0.9760783936977615,
3562
+ "grad_norm": 0.7781296968460083,
3563
+ "learning_rate": 7.146000175874412e-07,
3564
+ "loss": 0.6074,
3565
+ "step": 5080
3566
+ },
3567
+ {
3568
+ "epoch": 0.9779998078585839,
3569
+ "grad_norm": 1.0012695789337158,
3570
+ "learning_rate": 6.040340181578119e-07,
3571
+ "loss": 0.624,
3572
+ "step": 5090
3573
+ },
3574
+ {
3575
+ "epoch": 0.9799212220194062,
3576
+ "grad_norm": 0.6960607767105103,
3577
+ "learning_rate": 5.027449582170884e-07,
3578
+ "loss": 0.6221,
3579
+ "step": 5100
3580
+ },
3581
+ {
3582
+ "epoch": 0.9818426361802286,
3583
+ "grad_norm": 1.2364157438278198,
3584
+ "learning_rate": 4.107366054784956e-07,
3585
+ "loss": 0.7251,
3586
+ "step": 5110
3587
+ },
3588
+ {
3589
+ "epoch": 0.983764050341051,
3590
+ "grad_norm": 1.0493301153182983,
3591
+ "learning_rate": 3.28012382434878e-07,
3592
+ "loss": 0.6961,
3593
+ "step": 5120
3594
+ },
3595
+ {
3596
+ "epoch": 0.9856854645018733,
3597
+ "grad_norm": 0.7855087518692017,
3598
+ "learning_rate": 2.54575366231552e-07,
3599
+ "loss": 0.545,
3600
+ "step": 5130
3601
+ },
3602
+ {
3603
+ "epoch": 0.9876068786626957,
3604
+ "grad_norm": 0.9644796848297119,
3605
+ "learning_rate": 1.9042828855159177e-07,
3606
+ "loss": 0.6529,
3607
+ "step": 5140
3608
+ },
3609
+ {
3610
+ "epoch": 0.9895282928235181,
3611
+ "grad_norm": 1.0050861835479736,
3612
+ "learning_rate": 1.3557353551446605e-07,
3613
+ "loss": 0.6603,
3614
+ "step": 5150
3615
+ },
3616
+ {
3617
+ "epoch": 0.9914497069843404,
3618
+ "grad_norm": 1.5977165699005127,
3619
+ "learning_rate": 9.001314758708135e-08,
3620
+ "loss": 0.6425,
3621
+ "step": 5160
3622
+ },
3623
+ {
3624
+ "epoch": 0.9933711211451628,
3625
+ "grad_norm": 1.4106450080871582,
3626
+ "learning_rate": 5.374881950803712e-08,
3627
+ "loss": 0.6156,
3628
+ "step": 5170
3629
+ },
3630
+ {
3631
+ "epoch": 0.9952925353059852,
3632
+ "grad_norm": 1.0265008211135864,
3633
+ "learning_rate": 2.6781900224481792e-08,
3634
+ "loss": 0.6507,
3635
+ "step": 5180
3636
+ },
3637
+ {
3638
+ "epoch": 0.9972139494668075,
3639
+ "grad_norm": 1.0650451183319092,
3640
+ "learning_rate": 9.113392841958445e-09,
3641
+ "loss": 0.6396,
3642
+ "step": 5190
3643
+ },
3644
+ {
3645
+ "epoch": 0.9991353636276299,
3646
+ "grad_norm": 0.8263037204742432,
3647
+ "learning_rate": 7.439545870735476e-10,
3648
+ "loss": 0.6519,
3649
+ "step": 5200
3650
+ },
3651
+ {
3652
+ "epoch": 0.9999039292919589,
3653
+ "step": 5204,
3654
+ "total_flos": 4.294943192106664e+18,
3655
+ "train_loss": 0.8275534134475567,
3656
+ "train_runtime": 12276.2585,
3657
+ "train_samples_per_second": 27.131,
3658
+ "train_steps_per_second": 0.424
3659
+ }
3660
+ ],
3661
+ "logging_steps": 10,
3662
+ "max_steps": 5204,
3663
+ "num_input_tokens_seen": 0,
3664
+ "num_train_epochs": 1,
3665
+ "save_steps": 1000,
3666
+ "stateful_callbacks": {
3667
+ "TrainerControl": {
3668
+ "args": {
3669
+ "should_epoch_stop": false,
3670
+ "should_evaluate": false,
3671
+ "should_log": false,
3672
+ "should_save": true,
3673
+ "should_training_stop": true
3674
+ },
3675
+ "attributes": {}
3676
+ }
3677
+ },
3678
+ "total_flos": 4.294943192106664e+18,
3679
+ "train_batch_size": 8,
3680
+ "trial_name": null,
3681
+ "trial_params": null
3682
+ }
Ins/indices.json ADDED
The diff for this file is too large to render. See raw diff
 
Ins/log.txt ADDED
The diff for this file is too large to render. See raw diff
 
Ins/model.safetensors.index.json ADDED
@@ -0,0 +1,780 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "metadata": {
3
+ "total_size": 14119477056
4
+ },
5
+ "weight_map": {
6
+ "model.base_model.model.lm_head.modules_to_save.default.weight": "model-00003-of-00003.safetensors",
7
+ "model.base_model.model.lm_head.original_module.weight": "model-00003-of-00003.safetensors",
8
+ "model.base_model.model.model.embed_tokens.modules_to_save.default.weight": "model-00001-of-00003.safetensors",
9
+ "model.base_model.model.model.embed_tokens.original_module.weight": "model-00001-of-00003.safetensors",
10
+ "model.base_model.model.model.layers.0.input_layernorm.weight": "model-00001-of-00003.safetensors",
11
+ "model.base_model.model.model.layers.0.mlp.down_proj.base_layer.weight": "model-00001-of-00003.safetensors",
12
+ "model.base_model.model.model.layers.0.mlp.down_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
13
+ "model.base_model.model.model.layers.0.mlp.down_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
14
+ "model.base_model.model.model.layers.0.mlp.gate_proj.base_layer.weight": "model-00001-of-00003.safetensors",
15
+ "model.base_model.model.model.layers.0.mlp.gate_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
16
+ "model.base_model.model.model.layers.0.mlp.gate_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
17
+ "model.base_model.model.model.layers.0.mlp.up_proj.base_layer.weight": "model-00001-of-00003.safetensors",
18
+ "model.base_model.model.model.layers.0.mlp.up_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
19
+ "model.base_model.model.model.layers.0.mlp.up_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
20
+ "model.base_model.model.model.layers.0.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
21
+ "model.base_model.model.model.layers.0.self_attn.k_proj.base_layer.weight": "model-00001-of-00003.safetensors",
22
+ "model.base_model.model.model.layers.0.self_attn.k_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
23
+ "model.base_model.model.model.layers.0.self_attn.k_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
24
+ "model.base_model.model.model.layers.0.self_attn.o_proj.base_layer.weight": "model-00001-of-00003.safetensors",
25
+ "model.base_model.model.model.layers.0.self_attn.o_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
26
+ "model.base_model.model.model.layers.0.self_attn.o_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
27
+ "model.base_model.model.model.layers.0.self_attn.q_proj.base_layer.weight": "model-00001-of-00003.safetensors",
28
+ "model.base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
29
+ "model.base_model.model.model.layers.0.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
30
+ "model.base_model.model.model.layers.0.self_attn.v_proj.base_layer.weight": "model-00001-of-00003.safetensors",
31
+ "model.base_model.model.model.layers.0.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
32
+ "model.base_model.model.model.layers.0.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
33
+ "model.base_model.model.model.layers.1.input_layernorm.weight": "model-00001-of-00003.safetensors",
34
+ "model.base_model.model.model.layers.1.mlp.down_proj.base_layer.weight": "model-00001-of-00003.safetensors",
35
+ "model.base_model.model.model.layers.1.mlp.down_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
36
+ "model.base_model.model.model.layers.1.mlp.down_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
37
+ "model.base_model.model.model.layers.1.mlp.gate_proj.base_layer.weight": "model-00001-of-00003.safetensors",
38
+ "model.base_model.model.model.layers.1.mlp.gate_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
39
+ "model.base_model.model.model.layers.1.mlp.gate_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
40
+ "model.base_model.model.model.layers.1.mlp.up_proj.base_layer.weight": "model-00001-of-00003.safetensors",
41
+ "model.base_model.model.model.layers.1.mlp.up_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
42
+ "model.base_model.model.model.layers.1.mlp.up_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
43
+ "model.base_model.model.model.layers.1.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
44
+ "model.base_model.model.model.layers.1.self_attn.k_proj.base_layer.weight": "model-00001-of-00003.safetensors",
45
+ "model.base_model.model.model.layers.1.self_attn.k_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
46
+ "model.base_model.model.model.layers.1.self_attn.k_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
47
+ "model.base_model.model.model.layers.1.self_attn.o_proj.base_layer.weight": "model-00001-of-00003.safetensors",
48
+ "model.base_model.model.model.layers.1.self_attn.o_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
49
+ "model.base_model.model.model.layers.1.self_attn.o_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
50
+ "model.base_model.model.model.layers.1.self_attn.q_proj.base_layer.weight": "model-00001-of-00003.safetensors",
51
+ "model.base_model.model.model.layers.1.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
52
+ "model.base_model.model.model.layers.1.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
53
+ "model.base_model.model.model.layers.1.self_attn.v_proj.base_layer.weight": "model-00001-of-00003.safetensors",
54
+ "model.base_model.model.model.layers.1.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
55
+ "model.base_model.model.model.layers.1.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
56
+ "model.base_model.model.model.layers.10.input_layernorm.weight": "model-00002-of-00003.safetensors",
57
+ "model.base_model.model.model.layers.10.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
58
+ "model.base_model.model.model.layers.10.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
59
+ "model.base_model.model.model.layers.10.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
60
+ "model.base_model.model.model.layers.10.mlp.gate_proj.base_layer.weight": "model-00001-of-00003.safetensors",
61
+ "model.base_model.model.model.layers.10.mlp.gate_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
62
+ "model.base_model.model.model.layers.10.mlp.gate_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
63
+ "model.base_model.model.model.layers.10.mlp.up_proj.base_layer.weight": "model-00001-of-00003.safetensors",
64
+ "model.base_model.model.model.layers.10.mlp.up_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
65
+ "model.base_model.model.model.layers.10.mlp.up_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
66
+ "model.base_model.model.model.layers.10.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
67
+ "model.base_model.model.model.layers.10.self_attn.k_proj.base_layer.weight": "model-00001-of-00003.safetensors",
68
+ "model.base_model.model.model.layers.10.self_attn.k_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
69
+ "model.base_model.model.model.layers.10.self_attn.k_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
70
+ "model.base_model.model.model.layers.10.self_attn.o_proj.base_layer.weight": "model-00001-of-00003.safetensors",
71
+ "model.base_model.model.model.layers.10.self_attn.o_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
72
+ "model.base_model.model.model.layers.10.self_attn.o_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
73
+ "model.base_model.model.model.layers.10.self_attn.q_proj.base_layer.weight": "model-00001-of-00003.safetensors",
74
+ "model.base_model.model.model.layers.10.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
75
+ "model.base_model.model.model.layers.10.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
76
+ "model.base_model.model.model.layers.10.self_attn.v_proj.base_layer.weight": "model-00001-of-00003.safetensors",
77
+ "model.base_model.model.model.layers.10.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
78
+ "model.base_model.model.model.layers.10.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
79
+ "model.base_model.model.model.layers.11.input_layernorm.weight": "model-00002-of-00003.safetensors",
80
+ "model.base_model.model.model.layers.11.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
81
+ "model.base_model.model.model.layers.11.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
82
+ "model.base_model.model.model.layers.11.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
83
+ "model.base_model.model.model.layers.11.mlp.gate_proj.base_layer.weight": "model-00002-of-00003.safetensors",
84
+ "model.base_model.model.model.layers.11.mlp.gate_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
85
+ "model.base_model.model.model.layers.11.mlp.gate_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
86
+ "model.base_model.model.model.layers.11.mlp.up_proj.base_layer.weight": "model-00002-of-00003.safetensors",
87
+ "model.base_model.model.model.layers.11.mlp.up_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
88
+ "model.base_model.model.model.layers.11.mlp.up_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
89
+ "model.base_model.model.model.layers.11.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
90
+ "model.base_model.model.model.layers.11.self_attn.k_proj.base_layer.weight": "model-00002-of-00003.safetensors",
91
+ "model.base_model.model.model.layers.11.self_attn.k_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
92
+ "model.base_model.model.model.layers.11.self_attn.k_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
93
+ "model.base_model.model.model.layers.11.self_attn.o_proj.base_layer.weight": "model-00002-of-00003.safetensors",
94
+ "model.base_model.model.model.layers.11.self_attn.o_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
95
+ "model.base_model.model.model.layers.11.self_attn.o_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
96
+ "model.base_model.model.model.layers.11.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
97
+ "model.base_model.model.model.layers.11.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
98
+ "model.base_model.model.model.layers.11.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
99
+ "model.base_model.model.model.layers.11.self_attn.v_proj.base_layer.weight": "model-00002-of-00003.safetensors",
100
+ "model.base_model.model.model.layers.11.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
101
+ "model.base_model.model.model.layers.11.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
102
+ "model.base_model.model.model.layers.12.input_layernorm.weight": "model-00002-of-00003.safetensors",
103
+ "model.base_model.model.model.layers.12.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
104
+ "model.base_model.model.model.layers.12.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
105
+ "model.base_model.model.model.layers.12.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
106
+ "model.base_model.model.model.layers.12.mlp.gate_proj.base_layer.weight": "model-00002-of-00003.safetensors",
107
+ "model.base_model.model.model.layers.12.mlp.gate_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
108
+ "model.base_model.model.model.layers.12.mlp.gate_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
109
+ "model.base_model.model.model.layers.12.mlp.up_proj.base_layer.weight": "model-00002-of-00003.safetensors",
110
+ "model.base_model.model.model.layers.12.mlp.up_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
111
+ "model.base_model.model.model.layers.12.mlp.up_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
112
+ "model.base_model.model.model.layers.12.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
113
+ "model.base_model.model.model.layers.12.self_attn.k_proj.base_layer.weight": "model-00002-of-00003.safetensors",
114
+ "model.base_model.model.model.layers.12.self_attn.k_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
115
+ "model.base_model.model.model.layers.12.self_attn.k_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
116
+ "model.base_model.model.model.layers.12.self_attn.o_proj.base_layer.weight": "model-00002-of-00003.safetensors",
117
+ "model.base_model.model.model.layers.12.self_attn.o_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
118
+ "model.base_model.model.model.layers.12.self_attn.o_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
119
+ "model.base_model.model.model.layers.12.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
120
+ "model.base_model.model.model.layers.12.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
121
+ "model.base_model.model.model.layers.12.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
122
+ "model.base_model.model.model.layers.12.self_attn.v_proj.base_layer.weight": "model-00002-of-00003.safetensors",
123
+ "model.base_model.model.model.layers.12.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
124
+ "model.base_model.model.model.layers.12.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
125
+ "model.base_model.model.model.layers.13.input_layernorm.weight": "model-00002-of-00003.safetensors",
126
+ "model.base_model.model.model.layers.13.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
127
+ "model.base_model.model.model.layers.13.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
128
+ "model.base_model.model.model.layers.13.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
129
+ "model.base_model.model.model.layers.13.mlp.gate_proj.base_layer.weight": "model-00002-of-00003.safetensors",
130
+ "model.base_model.model.model.layers.13.mlp.gate_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
131
+ "model.base_model.model.model.layers.13.mlp.gate_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
132
+ "model.base_model.model.model.layers.13.mlp.up_proj.base_layer.weight": "model-00002-of-00003.safetensors",
133
+ "model.base_model.model.model.layers.13.mlp.up_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
134
+ "model.base_model.model.model.layers.13.mlp.up_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
135
+ "model.base_model.model.model.layers.13.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
136
+ "model.base_model.model.model.layers.13.self_attn.k_proj.base_layer.weight": "model-00002-of-00003.safetensors",
137
+ "model.base_model.model.model.layers.13.self_attn.k_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
138
+ "model.base_model.model.model.layers.13.self_attn.k_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
139
+ "model.base_model.model.model.layers.13.self_attn.o_proj.base_layer.weight": "model-00002-of-00003.safetensors",
140
+ "model.base_model.model.model.layers.13.self_attn.o_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
141
+ "model.base_model.model.model.layers.13.self_attn.o_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
142
+ "model.base_model.model.model.layers.13.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
143
+ "model.base_model.model.model.layers.13.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
144
+ "model.base_model.model.model.layers.13.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
145
+ "model.base_model.model.model.layers.13.self_attn.v_proj.base_layer.weight": "model-00002-of-00003.safetensors",
146
+ "model.base_model.model.model.layers.13.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
147
+ "model.base_model.model.model.layers.13.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
148
+ "model.base_model.model.model.layers.14.input_layernorm.weight": "model-00002-of-00003.safetensors",
149
+ "model.base_model.model.model.layers.14.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
150
+ "model.base_model.model.model.layers.14.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
151
+ "model.base_model.model.model.layers.14.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
152
+ "model.base_model.model.model.layers.14.mlp.gate_proj.base_layer.weight": "model-00002-of-00003.safetensors",
153
+ "model.base_model.model.model.layers.14.mlp.gate_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
154
+ "model.base_model.model.model.layers.14.mlp.gate_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
155
+ "model.base_model.model.model.layers.14.mlp.up_proj.base_layer.weight": "model-00002-of-00003.safetensors",
156
+ "model.base_model.model.model.layers.14.mlp.up_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
157
+ "model.base_model.model.model.layers.14.mlp.up_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
158
+ "model.base_model.model.model.layers.14.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
159
+ "model.base_model.model.model.layers.14.self_attn.k_proj.base_layer.weight": "model-00002-of-00003.safetensors",
160
+ "model.base_model.model.model.layers.14.self_attn.k_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
161
+ "model.base_model.model.model.layers.14.self_attn.k_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
162
+ "model.base_model.model.model.layers.14.self_attn.o_proj.base_layer.weight": "model-00002-of-00003.safetensors",
163
+ "model.base_model.model.model.layers.14.self_attn.o_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
164
+ "model.base_model.model.model.layers.14.self_attn.o_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
165
+ "model.base_model.model.model.layers.14.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
166
+ "model.base_model.model.model.layers.14.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
167
+ "model.base_model.model.model.layers.14.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
168
+ "model.base_model.model.model.layers.14.self_attn.v_proj.base_layer.weight": "model-00002-of-00003.safetensors",
169
+ "model.base_model.model.model.layers.14.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
170
+ "model.base_model.model.model.layers.14.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
171
+ "model.base_model.model.model.layers.15.input_layernorm.weight": "model-00002-of-00003.safetensors",
172
+ "model.base_model.model.model.layers.15.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
173
+ "model.base_model.model.model.layers.15.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
174
+ "model.base_model.model.model.layers.15.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
175
+ "model.base_model.model.model.layers.15.mlp.gate_proj.base_layer.weight": "model-00002-of-00003.safetensors",
176
+ "model.base_model.model.model.layers.15.mlp.gate_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
177
+ "model.base_model.model.model.layers.15.mlp.gate_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
178
+ "model.base_model.model.model.layers.15.mlp.up_proj.base_layer.weight": "model-00002-of-00003.safetensors",
179
+ "model.base_model.model.model.layers.15.mlp.up_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
180
+ "model.base_model.model.model.layers.15.mlp.up_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
181
+ "model.base_model.model.model.layers.15.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
182
+ "model.base_model.model.model.layers.15.self_attn.k_proj.base_layer.weight": "model-00002-of-00003.safetensors",
183
+ "model.base_model.model.model.layers.15.self_attn.k_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
184
+ "model.base_model.model.model.layers.15.self_attn.k_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
185
+ "model.base_model.model.model.layers.15.self_attn.o_proj.base_layer.weight": "model-00002-of-00003.safetensors",
186
+ "model.base_model.model.model.layers.15.self_attn.o_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
187
+ "model.base_model.model.model.layers.15.self_attn.o_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
188
+ "model.base_model.model.model.layers.15.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
189
+ "model.base_model.model.model.layers.15.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
190
+ "model.base_model.model.model.layers.15.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
191
+ "model.base_model.model.model.layers.15.self_attn.v_proj.base_layer.weight": "model-00002-of-00003.safetensors",
192
+ "model.base_model.model.model.layers.15.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
193
+ "model.base_model.model.model.layers.15.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
194
+ "model.base_model.model.model.layers.16.input_layernorm.weight": "model-00002-of-00003.safetensors",
195
+ "model.base_model.model.model.layers.16.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
196
+ "model.base_model.model.model.layers.16.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
197
+ "model.base_model.model.model.layers.16.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
198
+ "model.base_model.model.model.layers.16.mlp.gate_proj.base_layer.weight": "model-00002-of-00003.safetensors",
199
+ "model.base_model.model.model.layers.16.mlp.gate_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
200
+ "model.base_model.model.model.layers.16.mlp.gate_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
201
+ "model.base_model.model.model.layers.16.mlp.up_proj.base_layer.weight": "model-00002-of-00003.safetensors",
202
+ "model.base_model.model.model.layers.16.mlp.up_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
203
+ "model.base_model.model.model.layers.16.mlp.up_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
204
+ "model.base_model.model.model.layers.16.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
205
+ "model.base_model.model.model.layers.16.self_attn.k_proj.base_layer.weight": "model-00002-of-00003.safetensors",
206
+ "model.base_model.model.model.layers.16.self_attn.k_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
207
+ "model.base_model.model.model.layers.16.self_attn.k_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
208
+ "model.base_model.model.model.layers.16.self_attn.o_proj.base_layer.weight": "model-00002-of-00003.safetensors",
209
+ "model.base_model.model.model.layers.16.self_attn.o_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
210
+ "model.base_model.model.model.layers.16.self_attn.o_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
211
+ "model.base_model.model.model.layers.16.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
212
+ "model.base_model.model.model.layers.16.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
213
+ "model.base_model.model.model.layers.16.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
214
+ "model.base_model.model.model.layers.16.self_attn.v_proj.base_layer.weight": "model-00002-of-00003.safetensors",
215
+ "model.base_model.model.model.layers.16.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
216
+ "model.base_model.model.model.layers.16.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
217
+ "model.base_model.model.model.layers.17.input_layernorm.weight": "model-00002-of-00003.safetensors",
218
+ "model.base_model.model.model.layers.17.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
219
+ "model.base_model.model.model.layers.17.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
220
+ "model.base_model.model.model.layers.17.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
221
+ "model.base_model.model.model.layers.17.mlp.gate_proj.base_layer.weight": "model-00002-of-00003.safetensors",
222
+ "model.base_model.model.model.layers.17.mlp.gate_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
223
+ "model.base_model.model.model.layers.17.mlp.gate_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
224
+ "model.base_model.model.model.layers.17.mlp.up_proj.base_layer.weight": "model-00002-of-00003.safetensors",
225
+ "model.base_model.model.model.layers.17.mlp.up_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
226
+ "model.base_model.model.model.layers.17.mlp.up_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
227
+ "model.base_model.model.model.layers.17.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
228
+ "model.base_model.model.model.layers.17.self_attn.k_proj.base_layer.weight": "model-00002-of-00003.safetensors",
229
+ "model.base_model.model.model.layers.17.self_attn.k_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
230
+ "model.base_model.model.model.layers.17.self_attn.k_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
231
+ "model.base_model.model.model.layers.17.self_attn.o_proj.base_layer.weight": "model-00002-of-00003.safetensors",
232
+ "model.base_model.model.model.layers.17.self_attn.o_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
233
+ "model.base_model.model.model.layers.17.self_attn.o_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
234
+ "model.base_model.model.model.layers.17.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
235
+ "model.base_model.model.model.layers.17.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
236
+ "model.base_model.model.model.layers.17.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
237
+ "model.base_model.model.model.layers.17.self_attn.v_proj.base_layer.weight": "model-00002-of-00003.safetensors",
238
+ "model.base_model.model.model.layers.17.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
239
+ "model.base_model.model.model.layers.17.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
240
+ "model.base_model.model.model.layers.18.input_layernorm.weight": "model-00002-of-00003.safetensors",
241
+ "model.base_model.model.model.layers.18.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
242
+ "model.base_model.model.model.layers.18.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
243
+ "model.base_model.model.model.layers.18.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
244
+ "model.base_model.model.model.layers.18.mlp.gate_proj.base_layer.weight": "model-00002-of-00003.safetensors",
245
+ "model.base_model.model.model.layers.18.mlp.gate_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
246
+ "model.base_model.model.model.layers.18.mlp.gate_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
247
+ "model.base_model.model.model.layers.18.mlp.up_proj.base_layer.weight": "model-00002-of-00003.safetensors",
248
+ "model.base_model.model.model.layers.18.mlp.up_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
249
+ "model.base_model.model.model.layers.18.mlp.up_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
250
+ "model.base_model.model.model.layers.18.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
251
+ "model.base_model.model.model.layers.18.self_attn.k_proj.base_layer.weight": "model-00002-of-00003.safetensors",
252
+ "model.base_model.model.model.layers.18.self_attn.k_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
253
+ "model.base_model.model.model.layers.18.self_attn.k_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
254
+ "model.base_model.model.model.layers.18.self_attn.o_proj.base_layer.weight": "model-00002-of-00003.safetensors",
255
+ "model.base_model.model.model.layers.18.self_attn.o_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
256
+ "model.base_model.model.model.layers.18.self_attn.o_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
257
+ "model.base_model.model.model.layers.18.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
258
+ "model.base_model.model.model.layers.18.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
259
+ "model.base_model.model.model.layers.18.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
260
+ "model.base_model.model.model.layers.18.self_attn.v_proj.base_layer.weight": "model-00002-of-00003.safetensors",
261
+ "model.base_model.model.model.layers.18.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
262
+ "model.base_model.model.model.layers.18.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
263
+ "model.base_model.model.model.layers.19.input_layernorm.weight": "model-00002-of-00003.safetensors",
264
+ "model.base_model.model.model.layers.19.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
265
+ "model.base_model.model.model.layers.19.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
266
+ "model.base_model.model.model.layers.19.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
267
+ "model.base_model.model.model.layers.19.mlp.gate_proj.base_layer.weight": "model-00002-of-00003.safetensors",
268
+ "model.base_model.model.model.layers.19.mlp.gate_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
269
+ "model.base_model.model.model.layers.19.mlp.gate_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
270
+ "model.base_model.model.model.layers.19.mlp.up_proj.base_layer.weight": "model-00002-of-00003.safetensors",
271
+ "model.base_model.model.model.layers.19.mlp.up_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
272
+ "model.base_model.model.model.layers.19.mlp.up_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
273
+ "model.base_model.model.model.layers.19.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
274
+ "model.base_model.model.model.layers.19.self_attn.k_proj.base_layer.weight": "model-00002-of-00003.safetensors",
275
+ "model.base_model.model.model.layers.19.self_attn.k_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
276
+ "model.base_model.model.model.layers.19.self_attn.k_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
277
+ "model.base_model.model.model.layers.19.self_attn.o_proj.base_layer.weight": "model-00002-of-00003.safetensors",
278
+ "model.base_model.model.model.layers.19.self_attn.o_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
279
+ "model.base_model.model.model.layers.19.self_attn.o_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
280
+ "model.base_model.model.model.layers.19.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
281
+ "model.base_model.model.model.layers.19.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
282
+ "model.base_model.model.model.layers.19.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
283
+ "model.base_model.model.model.layers.19.self_attn.v_proj.base_layer.weight": "model-00002-of-00003.safetensors",
284
+ "model.base_model.model.model.layers.19.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
285
+ "model.base_model.model.model.layers.19.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
286
+ "model.base_model.model.model.layers.2.input_layernorm.weight": "model-00001-of-00003.safetensors",
287
+ "model.base_model.model.model.layers.2.mlp.down_proj.base_layer.weight": "model-00001-of-00003.safetensors",
288
+ "model.base_model.model.model.layers.2.mlp.down_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
289
+ "model.base_model.model.model.layers.2.mlp.down_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
290
+ "model.base_model.model.model.layers.2.mlp.gate_proj.base_layer.weight": "model-00001-of-00003.safetensors",
291
+ "model.base_model.model.model.layers.2.mlp.gate_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
292
+ "model.base_model.model.model.layers.2.mlp.gate_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
293
+ "model.base_model.model.model.layers.2.mlp.up_proj.base_layer.weight": "model-00001-of-00003.safetensors",
294
+ "model.base_model.model.model.layers.2.mlp.up_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
295
+ "model.base_model.model.model.layers.2.mlp.up_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
296
+ "model.base_model.model.model.layers.2.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
297
+ "model.base_model.model.model.layers.2.self_attn.k_proj.base_layer.weight": "model-00001-of-00003.safetensors",
298
+ "model.base_model.model.model.layers.2.self_attn.k_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
299
+ "model.base_model.model.model.layers.2.self_attn.k_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
300
+ "model.base_model.model.model.layers.2.self_attn.o_proj.base_layer.weight": "model-00001-of-00003.safetensors",
301
+ "model.base_model.model.model.layers.2.self_attn.o_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
302
+ "model.base_model.model.model.layers.2.self_attn.o_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
303
+ "model.base_model.model.model.layers.2.self_attn.q_proj.base_layer.weight": "model-00001-of-00003.safetensors",
304
+ "model.base_model.model.model.layers.2.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
305
+ "model.base_model.model.model.layers.2.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
306
+ "model.base_model.model.model.layers.2.self_attn.v_proj.base_layer.weight": "model-00001-of-00003.safetensors",
307
+ "model.base_model.model.model.layers.2.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
308
+ "model.base_model.model.model.layers.2.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
309
+ "model.base_model.model.model.layers.20.input_layernorm.weight": "model-00002-of-00003.safetensors",
310
+ "model.base_model.model.model.layers.20.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
311
+ "model.base_model.model.model.layers.20.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
312
+ "model.base_model.model.model.layers.20.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
313
+ "model.base_model.model.model.layers.20.mlp.gate_proj.base_layer.weight": "model-00002-of-00003.safetensors",
314
+ "model.base_model.model.model.layers.20.mlp.gate_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
315
+ "model.base_model.model.model.layers.20.mlp.gate_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
316
+ "model.base_model.model.model.layers.20.mlp.up_proj.base_layer.weight": "model-00002-of-00003.safetensors",
317
+ "model.base_model.model.model.layers.20.mlp.up_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
318
+ "model.base_model.model.model.layers.20.mlp.up_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
319
+ "model.base_model.model.model.layers.20.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
320
+ "model.base_model.model.model.layers.20.self_attn.k_proj.base_layer.weight": "model-00002-of-00003.safetensors",
321
+ "model.base_model.model.model.layers.20.self_attn.k_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
322
+ "model.base_model.model.model.layers.20.self_attn.k_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
323
+ "model.base_model.model.model.layers.20.self_attn.o_proj.base_layer.weight": "model-00002-of-00003.safetensors",
324
+ "model.base_model.model.model.layers.20.self_attn.o_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
325
+ "model.base_model.model.model.layers.20.self_attn.o_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
326
+ "model.base_model.model.model.layers.20.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
327
+ "model.base_model.model.model.layers.20.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
328
+ "model.base_model.model.model.layers.20.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
329
+ "model.base_model.model.model.layers.20.self_attn.v_proj.base_layer.weight": "model-00002-of-00003.safetensors",
330
+ "model.base_model.model.model.layers.20.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
331
+ "model.base_model.model.model.layers.20.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
332
+ "model.base_model.model.model.layers.21.input_layernorm.weight": "model-00002-of-00003.safetensors",
333
+ "model.base_model.model.model.layers.21.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
334
+ "model.base_model.model.model.layers.21.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
335
+ "model.base_model.model.model.layers.21.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
336
+ "model.base_model.model.model.layers.21.mlp.gate_proj.base_layer.weight": "model-00002-of-00003.safetensors",
337
+ "model.base_model.model.model.layers.21.mlp.gate_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
338
+ "model.base_model.model.model.layers.21.mlp.gate_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
339
+ "model.base_model.model.model.layers.21.mlp.up_proj.base_layer.weight": "model-00002-of-00003.safetensors",
340
+ "model.base_model.model.model.layers.21.mlp.up_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
341
+ "model.base_model.model.model.layers.21.mlp.up_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
342
+ "model.base_model.model.model.layers.21.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
343
+ "model.base_model.model.model.layers.21.self_attn.k_proj.base_layer.weight": "model-00002-of-00003.safetensors",
344
+ "model.base_model.model.model.layers.21.self_attn.k_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
345
+ "model.base_model.model.model.layers.21.self_attn.k_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
346
+ "model.base_model.model.model.layers.21.self_attn.o_proj.base_layer.weight": "model-00002-of-00003.safetensors",
347
+ "model.base_model.model.model.layers.21.self_attn.o_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
348
+ "model.base_model.model.model.layers.21.self_attn.o_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
349
+ "model.base_model.model.model.layers.21.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
350
+ "model.base_model.model.model.layers.21.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
351
+ "model.base_model.model.model.layers.21.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
352
+ "model.base_model.model.model.layers.21.self_attn.v_proj.base_layer.weight": "model-00002-of-00003.safetensors",
353
+ "model.base_model.model.model.layers.21.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
354
+ "model.base_model.model.model.layers.21.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
355
+ "model.base_model.model.model.layers.22.input_layernorm.weight": "model-00002-of-00003.safetensors",
356
+ "model.base_model.model.model.layers.22.mlp.down_proj.base_layer.weight": "model-00002-of-00003.safetensors",
357
+ "model.base_model.model.model.layers.22.mlp.down_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
358
+ "model.base_model.model.model.layers.22.mlp.down_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
359
+ "model.base_model.model.model.layers.22.mlp.gate_proj.base_layer.weight": "model-00002-of-00003.safetensors",
360
+ "model.base_model.model.model.layers.22.mlp.gate_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
361
+ "model.base_model.model.model.layers.22.mlp.gate_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
362
+ "model.base_model.model.model.layers.22.mlp.up_proj.base_layer.weight": "model-00002-of-00003.safetensors",
363
+ "model.base_model.model.model.layers.22.mlp.up_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
364
+ "model.base_model.model.model.layers.22.mlp.up_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
365
+ "model.base_model.model.model.layers.22.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
366
+ "model.base_model.model.model.layers.22.self_attn.k_proj.base_layer.weight": "model-00002-of-00003.safetensors",
367
+ "model.base_model.model.model.layers.22.self_attn.k_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
368
+ "model.base_model.model.model.layers.22.self_attn.k_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
369
+ "model.base_model.model.model.layers.22.self_attn.o_proj.base_layer.weight": "model-00002-of-00003.safetensors",
370
+ "model.base_model.model.model.layers.22.self_attn.o_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
371
+ "model.base_model.model.model.layers.22.self_attn.o_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
372
+ "model.base_model.model.model.layers.22.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
373
+ "model.base_model.model.model.layers.22.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
374
+ "model.base_model.model.model.layers.22.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
375
+ "model.base_model.model.model.layers.22.self_attn.v_proj.base_layer.weight": "model-00002-of-00003.safetensors",
376
+ "model.base_model.model.model.layers.22.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
377
+ "model.base_model.model.model.layers.22.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
378
+ "model.base_model.model.model.layers.23.input_layernorm.weight": "model-00003-of-00003.safetensors",
379
+ "model.base_model.model.model.layers.23.mlp.down_proj.base_layer.weight": "model-00003-of-00003.safetensors",
380
+ "model.base_model.model.model.layers.23.mlp.down_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
381
+ "model.base_model.model.model.layers.23.mlp.down_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
382
+ "model.base_model.model.model.layers.23.mlp.gate_proj.base_layer.weight": "model-00003-of-00003.safetensors",
383
+ "model.base_model.model.model.layers.23.mlp.gate_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
384
+ "model.base_model.model.model.layers.23.mlp.gate_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
385
+ "model.base_model.model.model.layers.23.mlp.up_proj.base_layer.weight": "model-00003-of-00003.safetensors",
386
+ "model.base_model.model.model.layers.23.mlp.up_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
387
+ "model.base_model.model.model.layers.23.mlp.up_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
388
+ "model.base_model.model.model.layers.23.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
389
+ "model.base_model.model.model.layers.23.self_attn.k_proj.base_layer.weight": "model-00003-of-00003.safetensors",
390
+ "model.base_model.model.model.layers.23.self_attn.k_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
391
+ "model.base_model.model.model.layers.23.self_attn.k_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
392
+ "model.base_model.model.model.layers.23.self_attn.o_proj.base_layer.weight": "model-00003-of-00003.safetensors",
393
+ "model.base_model.model.model.layers.23.self_attn.o_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
394
+ "model.base_model.model.model.layers.23.self_attn.o_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
395
+ "model.base_model.model.model.layers.23.self_attn.q_proj.base_layer.weight": "model-00002-of-00003.safetensors",
396
+ "model.base_model.model.model.layers.23.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00003.safetensors",
397
+ "model.base_model.model.model.layers.23.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00003.safetensors",
398
+ "model.base_model.model.model.layers.23.self_attn.v_proj.base_layer.weight": "model-00003-of-00003.safetensors",
399
+ "model.base_model.model.model.layers.23.self_attn.v_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
400
+ "model.base_model.model.model.layers.23.self_attn.v_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
401
+ "model.base_model.model.model.layers.24.input_layernorm.weight": "model-00003-of-00003.safetensors",
402
+ "model.base_model.model.model.layers.24.mlp.down_proj.base_layer.weight": "model-00003-of-00003.safetensors",
403
+ "model.base_model.model.model.layers.24.mlp.down_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
404
+ "model.base_model.model.model.layers.24.mlp.down_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
405
+ "model.base_model.model.model.layers.24.mlp.gate_proj.base_layer.weight": "model-00003-of-00003.safetensors",
406
+ "model.base_model.model.model.layers.24.mlp.gate_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
407
+ "model.base_model.model.model.layers.24.mlp.gate_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
408
+ "model.base_model.model.model.layers.24.mlp.up_proj.base_layer.weight": "model-00003-of-00003.safetensors",
409
+ "model.base_model.model.model.layers.24.mlp.up_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
410
+ "model.base_model.model.model.layers.24.mlp.up_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
411
+ "model.base_model.model.model.layers.24.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
412
+ "model.base_model.model.model.layers.24.self_attn.k_proj.base_layer.weight": "model-00003-of-00003.safetensors",
413
+ "model.base_model.model.model.layers.24.self_attn.k_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
414
+ "model.base_model.model.model.layers.24.self_attn.k_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
415
+ "model.base_model.model.model.layers.24.self_attn.o_proj.base_layer.weight": "model-00003-of-00003.safetensors",
416
+ "model.base_model.model.model.layers.24.self_attn.o_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
417
+ "model.base_model.model.model.layers.24.self_attn.o_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
418
+ "model.base_model.model.model.layers.24.self_attn.q_proj.base_layer.weight": "model-00003-of-00003.safetensors",
419
+ "model.base_model.model.model.layers.24.self_attn.q_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
420
+ "model.base_model.model.model.layers.24.self_attn.q_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
421
+ "model.base_model.model.model.layers.24.self_attn.v_proj.base_layer.weight": "model-00003-of-00003.safetensors",
422
+ "model.base_model.model.model.layers.24.self_attn.v_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
423
+ "model.base_model.model.model.layers.24.self_attn.v_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
424
+ "model.base_model.model.model.layers.25.input_layernorm.weight": "model-00003-of-00003.safetensors",
425
+ "model.base_model.model.model.layers.25.mlp.down_proj.base_layer.weight": "model-00003-of-00003.safetensors",
426
+ "model.base_model.model.model.layers.25.mlp.down_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
427
+ "model.base_model.model.model.layers.25.mlp.down_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
428
+ "model.base_model.model.model.layers.25.mlp.gate_proj.base_layer.weight": "model-00003-of-00003.safetensors",
429
+ "model.base_model.model.model.layers.25.mlp.gate_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
430
+ "model.base_model.model.model.layers.25.mlp.gate_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
431
+ "model.base_model.model.model.layers.25.mlp.up_proj.base_layer.weight": "model-00003-of-00003.safetensors",
432
+ "model.base_model.model.model.layers.25.mlp.up_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
433
+ "model.base_model.model.model.layers.25.mlp.up_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
434
+ "model.base_model.model.model.layers.25.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
435
+ "model.base_model.model.model.layers.25.self_attn.k_proj.base_layer.weight": "model-00003-of-00003.safetensors",
436
+ "model.base_model.model.model.layers.25.self_attn.k_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
437
+ "model.base_model.model.model.layers.25.self_attn.k_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
438
+ "model.base_model.model.model.layers.25.self_attn.o_proj.base_layer.weight": "model-00003-of-00003.safetensors",
439
+ "model.base_model.model.model.layers.25.self_attn.o_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
440
+ "model.base_model.model.model.layers.25.self_attn.o_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
441
+ "model.base_model.model.model.layers.25.self_attn.q_proj.base_layer.weight": "model-00003-of-00003.safetensors",
442
+ "model.base_model.model.model.layers.25.self_attn.q_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
443
+ "model.base_model.model.model.layers.25.self_attn.q_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
444
+ "model.base_model.model.model.layers.25.self_attn.v_proj.base_layer.weight": "model-00003-of-00003.safetensors",
445
+ "model.base_model.model.model.layers.25.self_attn.v_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
446
+ "model.base_model.model.model.layers.25.self_attn.v_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
447
+ "model.base_model.model.model.layers.26.input_layernorm.weight": "model-00003-of-00003.safetensors",
448
+ "model.base_model.model.model.layers.26.mlp.down_proj.base_layer.weight": "model-00003-of-00003.safetensors",
449
+ "model.base_model.model.model.layers.26.mlp.down_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
450
+ "model.base_model.model.model.layers.26.mlp.down_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
451
+ "model.base_model.model.model.layers.26.mlp.gate_proj.base_layer.weight": "model-00003-of-00003.safetensors",
452
+ "model.base_model.model.model.layers.26.mlp.gate_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
453
+ "model.base_model.model.model.layers.26.mlp.gate_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
454
+ "model.base_model.model.model.layers.26.mlp.up_proj.base_layer.weight": "model-00003-of-00003.safetensors",
455
+ "model.base_model.model.model.layers.26.mlp.up_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
456
+ "model.base_model.model.model.layers.26.mlp.up_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
457
+ "model.base_model.model.model.layers.26.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
458
+ "model.base_model.model.model.layers.26.self_attn.k_proj.base_layer.weight": "model-00003-of-00003.safetensors",
459
+ "model.base_model.model.model.layers.26.self_attn.k_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
460
+ "model.base_model.model.model.layers.26.self_attn.k_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
461
+ "model.base_model.model.model.layers.26.self_attn.o_proj.base_layer.weight": "model-00003-of-00003.safetensors",
462
+ "model.base_model.model.model.layers.26.self_attn.o_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
463
+ "model.base_model.model.model.layers.26.self_attn.o_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
464
+ "model.base_model.model.model.layers.26.self_attn.q_proj.base_layer.weight": "model-00003-of-00003.safetensors",
465
+ "model.base_model.model.model.layers.26.self_attn.q_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
466
+ "model.base_model.model.model.layers.26.self_attn.q_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
467
+ "model.base_model.model.model.layers.26.self_attn.v_proj.base_layer.weight": "model-00003-of-00003.safetensors",
468
+ "model.base_model.model.model.layers.26.self_attn.v_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
469
+ "model.base_model.model.model.layers.26.self_attn.v_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
470
+ "model.base_model.model.model.layers.27.input_layernorm.weight": "model-00003-of-00003.safetensors",
471
+ "model.base_model.model.model.layers.27.mlp.down_proj.base_layer.weight": "model-00003-of-00003.safetensors",
472
+ "model.base_model.model.model.layers.27.mlp.down_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
473
+ "model.base_model.model.model.layers.27.mlp.down_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
474
+ "model.base_model.model.model.layers.27.mlp.gate_proj.base_layer.weight": "model-00003-of-00003.safetensors",
475
+ "model.base_model.model.model.layers.27.mlp.gate_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
476
+ "model.base_model.model.model.layers.27.mlp.gate_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
477
+ "model.base_model.model.model.layers.27.mlp.up_proj.base_layer.weight": "model-00003-of-00003.safetensors",
478
+ "model.base_model.model.model.layers.27.mlp.up_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
479
+ "model.base_model.model.model.layers.27.mlp.up_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
480
+ "model.base_model.model.model.layers.27.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
481
+ "model.base_model.model.model.layers.27.self_attn.k_proj.base_layer.weight": "model-00003-of-00003.safetensors",
482
+ "model.base_model.model.model.layers.27.self_attn.k_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
483
+ "model.base_model.model.model.layers.27.self_attn.k_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
484
+ "model.base_model.model.model.layers.27.self_attn.o_proj.base_layer.weight": "model-00003-of-00003.safetensors",
485
+ "model.base_model.model.model.layers.27.self_attn.o_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
486
+ "model.base_model.model.model.layers.27.self_attn.o_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
487
+ "model.base_model.model.model.layers.27.self_attn.q_proj.base_layer.weight": "model-00003-of-00003.safetensors",
488
+ "model.base_model.model.model.layers.27.self_attn.q_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
489
+ "model.base_model.model.model.layers.27.self_attn.q_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
490
+ "model.base_model.model.model.layers.27.self_attn.v_proj.base_layer.weight": "model-00003-of-00003.safetensors",
491
+ "model.base_model.model.model.layers.27.self_attn.v_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
492
+ "model.base_model.model.model.layers.27.self_attn.v_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
493
+ "model.base_model.model.model.layers.28.input_layernorm.weight": "model-00003-of-00003.safetensors",
494
+ "model.base_model.model.model.layers.28.mlp.down_proj.base_layer.weight": "model-00003-of-00003.safetensors",
495
+ "model.base_model.model.model.layers.28.mlp.down_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
496
+ "model.base_model.model.model.layers.28.mlp.down_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
497
+ "model.base_model.model.model.layers.28.mlp.gate_proj.base_layer.weight": "model-00003-of-00003.safetensors",
498
+ "model.base_model.model.model.layers.28.mlp.gate_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
499
+ "model.base_model.model.model.layers.28.mlp.gate_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
500
+ "model.base_model.model.model.layers.28.mlp.up_proj.base_layer.weight": "model-00003-of-00003.safetensors",
501
+ "model.base_model.model.model.layers.28.mlp.up_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
502
+ "model.base_model.model.model.layers.28.mlp.up_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
503
+ "model.base_model.model.model.layers.28.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
504
+ "model.base_model.model.model.layers.28.self_attn.k_proj.base_layer.weight": "model-00003-of-00003.safetensors",
505
+ "model.base_model.model.model.layers.28.self_attn.k_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
506
+ "model.base_model.model.model.layers.28.self_attn.k_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
507
+ "model.base_model.model.model.layers.28.self_attn.o_proj.base_layer.weight": "model-00003-of-00003.safetensors",
508
+ "model.base_model.model.model.layers.28.self_attn.o_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
509
+ "model.base_model.model.model.layers.28.self_attn.o_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
510
+ "model.base_model.model.model.layers.28.self_attn.q_proj.base_layer.weight": "model-00003-of-00003.safetensors",
511
+ "model.base_model.model.model.layers.28.self_attn.q_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
512
+ "model.base_model.model.model.layers.28.self_attn.q_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
513
+ "model.base_model.model.model.layers.28.self_attn.v_proj.base_layer.weight": "model-00003-of-00003.safetensors",
514
+ "model.base_model.model.model.layers.28.self_attn.v_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
515
+ "model.base_model.model.model.layers.28.self_attn.v_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
516
+ "model.base_model.model.model.layers.29.input_layernorm.weight": "model-00003-of-00003.safetensors",
517
+ "model.base_model.model.model.layers.29.mlp.down_proj.base_layer.weight": "model-00003-of-00003.safetensors",
518
+ "model.base_model.model.model.layers.29.mlp.down_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
519
+ "model.base_model.model.model.layers.29.mlp.down_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
520
+ "model.base_model.model.model.layers.29.mlp.gate_proj.base_layer.weight": "model-00003-of-00003.safetensors",
521
+ "model.base_model.model.model.layers.29.mlp.gate_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
522
+ "model.base_model.model.model.layers.29.mlp.gate_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
523
+ "model.base_model.model.model.layers.29.mlp.up_proj.base_layer.weight": "model-00003-of-00003.safetensors",
524
+ "model.base_model.model.model.layers.29.mlp.up_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
525
+ "model.base_model.model.model.layers.29.mlp.up_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
526
+ "model.base_model.model.model.layers.29.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
527
+ "model.base_model.model.model.layers.29.self_attn.k_proj.base_layer.weight": "model-00003-of-00003.safetensors",
528
+ "model.base_model.model.model.layers.29.self_attn.k_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
529
+ "model.base_model.model.model.layers.29.self_attn.k_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
530
+ "model.base_model.model.model.layers.29.self_attn.o_proj.base_layer.weight": "model-00003-of-00003.safetensors",
531
+ "model.base_model.model.model.layers.29.self_attn.o_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
532
+ "model.base_model.model.model.layers.29.self_attn.o_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
533
+ "model.base_model.model.model.layers.29.self_attn.q_proj.base_layer.weight": "model-00003-of-00003.safetensors",
534
+ "model.base_model.model.model.layers.29.self_attn.q_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
535
+ "model.base_model.model.model.layers.29.self_attn.q_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
536
+ "model.base_model.model.model.layers.29.self_attn.v_proj.base_layer.weight": "model-00003-of-00003.safetensors",
537
+ "model.base_model.model.model.layers.29.self_attn.v_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
538
+ "model.base_model.model.model.layers.29.self_attn.v_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
539
+ "model.base_model.model.model.layers.3.input_layernorm.weight": "model-00001-of-00003.safetensors",
540
+ "model.base_model.model.model.layers.3.mlp.down_proj.base_layer.weight": "model-00001-of-00003.safetensors",
541
+ "model.base_model.model.model.layers.3.mlp.down_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
542
+ "model.base_model.model.model.layers.3.mlp.down_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
543
+ "model.base_model.model.model.layers.3.mlp.gate_proj.base_layer.weight": "model-00001-of-00003.safetensors",
544
+ "model.base_model.model.model.layers.3.mlp.gate_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
545
+ "model.base_model.model.model.layers.3.mlp.gate_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
546
+ "model.base_model.model.model.layers.3.mlp.up_proj.base_layer.weight": "model-00001-of-00003.safetensors",
547
+ "model.base_model.model.model.layers.3.mlp.up_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
548
+ "model.base_model.model.model.layers.3.mlp.up_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
549
+ "model.base_model.model.model.layers.3.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
550
+ "model.base_model.model.model.layers.3.self_attn.k_proj.base_layer.weight": "model-00001-of-00003.safetensors",
551
+ "model.base_model.model.model.layers.3.self_attn.k_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
552
+ "model.base_model.model.model.layers.3.self_attn.k_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
553
+ "model.base_model.model.model.layers.3.self_attn.o_proj.base_layer.weight": "model-00001-of-00003.safetensors",
554
+ "model.base_model.model.model.layers.3.self_attn.o_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
555
+ "model.base_model.model.model.layers.3.self_attn.o_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
556
+ "model.base_model.model.model.layers.3.self_attn.q_proj.base_layer.weight": "model-00001-of-00003.safetensors",
557
+ "model.base_model.model.model.layers.3.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
558
+ "model.base_model.model.model.layers.3.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
559
+ "model.base_model.model.model.layers.3.self_attn.v_proj.base_layer.weight": "model-00001-of-00003.safetensors",
560
+ "model.base_model.model.model.layers.3.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
561
+ "model.base_model.model.model.layers.3.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
562
+ "model.base_model.model.model.layers.30.input_layernorm.weight": "model-00003-of-00003.safetensors",
563
+ "model.base_model.model.model.layers.30.mlp.down_proj.base_layer.weight": "model-00003-of-00003.safetensors",
564
+ "model.base_model.model.model.layers.30.mlp.down_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
565
+ "model.base_model.model.model.layers.30.mlp.down_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
566
+ "model.base_model.model.model.layers.30.mlp.gate_proj.base_layer.weight": "model-00003-of-00003.safetensors",
567
+ "model.base_model.model.model.layers.30.mlp.gate_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
568
+ "model.base_model.model.model.layers.30.mlp.gate_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
569
+ "model.base_model.model.model.layers.30.mlp.up_proj.base_layer.weight": "model-00003-of-00003.safetensors",
570
+ "model.base_model.model.model.layers.30.mlp.up_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
571
+ "model.base_model.model.model.layers.30.mlp.up_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
572
+ "model.base_model.model.model.layers.30.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
573
+ "model.base_model.model.model.layers.30.self_attn.k_proj.base_layer.weight": "model-00003-of-00003.safetensors",
574
+ "model.base_model.model.model.layers.30.self_attn.k_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
575
+ "model.base_model.model.model.layers.30.self_attn.k_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
576
+ "model.base_model.model.model.layers.30.self_attn.o_proj.base_layer.weight": "model-00003-of-00003.safetensors",
577
+ "model.base_model.model.model.layers.30.self_attn.o_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
578
+ "model.base_model.model.model.layers.30.self_attn.o_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
579
+ "model.base_model.model.model.layers.30.self_attn.q_proj.base_layer.weight": "model-00003-of-00003.safetensors",
580
+ "model.base_model.model.model.layers.30.self_attn.q_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
581
+ "model.base_model.model.model.layers.30.self_attn.q_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
582
+ "model.base_model.model.model.layers.30.self_attn.v_proj.base_layer.weight": "model-00003-of-00003.safetensors",
583
+ "model.base_model.model.model.layers.30.self_attn.v_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
584
+ "model.base_model.model.model.layers.30.self_attn.v_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
585
+ "model.base_model.model.model.layers.31.input_layernorm.weight": "model-00003-of-00003.safetensors",
586
+ "model.base_model.model.model.layers.31.mlp.down_proj.base_layer.weight": "model-00003-of-00003.safetensors",
587
+ "model.base_model.model.model.layers.31.mlp.down_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
588
+ "model.base_model.model.model.layers.31.mlp.down_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
589
+ "model.base_model.model.model.layers.31.mlp.gate_proj.base_layer.weight": "model-00003-of-00003.safetensors",
590
+ "model.base_model.model.model.layers.31.mlp.gate_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
591
+ "model.base_model.model.model.layers.31.mlp.gate_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
592
+ "model.base_model.model.model.layers.31.mlp.up_proj.base_layer.weight": "model-00003-of-00003.safetensors",
593
+ "model.base_model.model.model.layers.31.mlp.up_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
594
+ "model.base_model.model.model.layers.31.mlp.up_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
595
+ "model.base_model.model.model.layers.31.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
596
+ "model.base_model.model.model.layers.31.self_attn.k_proj.base_layer.weight": "model-00003-of-00003.safetensors",
597
+ "model.base_model.model.model.layers.31.self_attn.k_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
598
+ "model.base_model.model.model.layers.31.self_attn.k_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
599
+ "model.base_model.model.model.layers.31.self_attn.o_proj.base_layer.weight": "model-00003-of-00003.safetensors",
600
+ "model.base_model.model.model.layers.31.self_attn.o_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
601
+ "model.base_model.model.model.layers.31.self_attn.o_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
602
+ "model.base_model.model.model.layers.31.self_attn.q_proj.base_layer.weight": "model-00003-of-00003.safetensors",
603
+ "model.base_model.model.model.layers.31.self_attn.q_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
604
+ "model.base_model.model.model.layers.31.self_attn.q_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
605
+ "model.base_model.model.model.layers.31.self_attn.v_proj.base_layer.weight": "model-00003-of-00003.safetensors",
606
+ "model.base_model.model.model.layers.31.self_attn.v_proj.lora_A.default.weight": "model-00003-of-00003.safetensors",
607
+ "model.base_model.model.model.layers.31.self_attn.v_proj.lora_B.default.weight": "model-00003-of-00003.safetensors",
608
+ "model.base_model.model.model.layers.4.input_layernorm.weight": "model-00001-of-00003.safetensors",
609
+ "model.base_model.model.model.layers.4.mlp.down_proj.base_layer.weight": "model-00001-of-00003.safetensors",
610
+ "model.base_model.model.model.layers.4.mlp.down_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
611
+ "model.base_model.model.model.layers.4.mlp.down_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
612
+ "model.base_model.model.model.layers.4.mlp.gate_proj.base_layer.weight": "model-00001-of-00003.safetensors",
613
+ "model.base_model.model.model.layers.4.mlp.gate_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
614
+ "model.base_model.model.model.layers.4.mlp.gate_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
615
+ "model.base_model.model.model.layers.4.mlp.up_proj.base_layer.weight": "model-00001-of-00003.safetensors",
616
+ "model.base_model.model.model.layers.4.mlp.up_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
617
+ "model.base_model.model.model.layers.4.mlp.up_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
618
+ "model.base_model.model.model.layers.4.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
619
+ "model.base_model.model.model.layers.4.self_attn.k_proj.base_layer.weight": "model-00001-of-00003.safetensors",
620
+ "model.base_model.model.model.layers.4.self_attn.k_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
621
+ "model.base_model.model.model.layers.4.self_attn.k_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
622
+ "model.base_model.model.model.layers.4.self_attn.o_proj.base_layer.weight": "model-00001-of-00003.safetensors",
623
+ "model.base_model.model.model.layers.4.self_attn.o_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
624
+ "model.base_model.model.model.layers.4.self_attn.o_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
625
+ "model.base_model.model.model.layers.4.self_attn.q_proj.base_layer.weight": "model-00001-of-00003.safetensors",
626
+ "model.base_model.model.model.layers.4.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
627
+ "model.base_model.model.model.layers.4.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
628
+ "model.base_model.model.model.layers.4.self_attn.v_proj.base_layer.weight": "model-00001-of-00003.safetensors",
629
+ "model.base_model.model.model.layers.4.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
630
+ "model.base_model.model.model.layers.4.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
631
+ "model.base_model.model.model.layers.5.input_layernorm.weight": "model-00001-of-00003.safetensors",
632
+ "model.base_model.model.model.layers.5.mlp.down_proj.base_layer.weight": "model-00001-of-00003.safetensors",
633
+ "model.base_model.model.model.layers.5.mlp.down_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
634
+ "model.base_model.model.model.layers.5.mlp.down_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
635
+ "model.base_model.model.model.layers.5.mlp.gate_proj.base_layer.weight": "model-00001-of-00003.safetensors",
636
+ "model.base_model.model.model.layers.5.mlp.gate_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
637
+ "model.base_model.model.model.layers.5.mlp.gate_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
638
+ "model.base_model.model.model.layers.5.mlp.up_proj.base_layer.weight": "model-00001-of-00003.safetensors",
639
+ "model.base_model.model.model.layers.5.mlp.up_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
640
+ "model.base_model.model.model.layers.5.mlp.up_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
641
+ "model.base_model.model.model.layers.5.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
642
+ "model.base_model.model.model.layers.5.self_attn.k_proj.base_layer.weight": "model-00001-of-00003.safetensors",
643
+ "model.base_model.model.model.layers.5.self_attn.k_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
644
+ "model.base_model.model.model.layers.5.self_attn.k_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
645
+ "model.base_model.model.model.layers.5.self_attn.o_proj.base_layer.weight": "model-00001-of-00003.safetensors",
646
+ "model.base_model.model.model.layers.5.self_attn.o_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
647
+ "model.base_model.model.model.layers.5.self_attn.o_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
648
+ "model.base_model.model.model.layers.5.self_attn.q_proj.base_layer.weight": "model-00001-of-00003.safetensors",
649
+ "model.base_model.model.model.layers.5.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
650
+ "model.base_model.model.model.layers.5.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
651
+ "model.base_model.model.model.layers.5.self_attn.v_proj.base_layer.weight": "model-00001-of-00003.safetensors",
652
+ "model.base_model.model.model.layers.5.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
653
+ "model.base_model.model.model.layers.5.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
654
+ "model.base_model.model.model.layers.6.input_layernorm.weight": "model-00001-of-00003.safetensors",
655
+ "model.base_model.model.model.layers.6.mlp.down_proj.base_layer.weight": "model-00001-of-00003.safetensors",
656
+ "model.base_model.model.model.layers.6.mlp.down_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
657
+ "model.base_model.model.model.layers.6.mlp.down_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
658
+ "model.base_model.model.model.layers.6.mlp.gate_proj.base_layer.weight": "model-00001-of-00003.safetensors",
659
+ "model.base_model.model.model.layers.6.mlp.gate_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
660
+ "model.base_model.model.model.layers.6.mlp.gate_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
661
+ "model.base_model.model.model.layers.6.mlp.up_proj.base_layer.weight": "model-00001-of-00003.safetensors",
662
+ "model.base_model.model.model.layers.6.mlp.up_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
663
+ "model.base_model.model.model.layers.6.mlp.up_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
664
+ "model.base_model.model.model.layers.6.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
665
+ "model.base_model.model.model.layers.6.self_attn.k_proj.base_layer.weight": "model-00001-of-00003.safetensors",
666
+ "model.base_model.model.model.layers.6.self_attn.k_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
667
+ "model.base_model.model.model.layers.6.self_attn.k_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
668
+ "model.base_model.model.model.layers.6.self_attn.o_proj.base_layer.weight": "model-00001-of-00003.safetensors",
669
+ "model.base_model.model.model.layers.6.self_attn.o_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
670
+ "model.base_model.model.model.layers.6.self_attn.o_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
671
+ "model.base_model.model.model.layers.6.self_attn.q_proj.base_layer.weight": "model-00001-of-00003.safetensors",
672
+ "model.base_model.model.model.layers.6.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
673
+ "model.base_model.model.model.layers.6.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
674
+ "model.base_model.model.model.layers.6.self_attn.v_proj.base_layer.weight": "model-00001-of-00003.safetensors",
675
+ "model.base_model.model.model.layers.6.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
676
+ "model.base_model.model.model.layers.6.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
677
+ "model.base_model.model.model.layers.7.input_layernorm.weight": "model-00001-of-00003.safetensors",
678
+ "model.base_model.model.model.layers.7.mlp.down_proj.base_layer.weight": "model-00001-of-00003.safetensors",
679
+ "model.base_model.model.model.layers.7.mlp.down_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
680
+ "model.base_model.model.model.layers.7.mlp.down_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
681
+ "model.base_model.model.model.layers.7.mlp.gate_proj.base_layer.weight": "model-00001-of-00003.safetensors",
682
+ "model.base_model.model.model.layers.7.mlp.gate_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
683
+ "model.base_model.model.model.layers.7.mlp.gate_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
684
+ "model.base_model.model.model.layers.7.mlp.up_proj.base_layer.weight": "model-00001-of-00003.safetensors",
685
+ "model.base_model.model.model.layers.7.mlp.up_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
686
+ "model.base_model.model.model.layers.7.mlp.up_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
687
+ "model.base_model.model.model.layers.7.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
688
+ "model.base_model.model.model.layers.7.self_attn.k_proj.base_layer.weight": "model-00001-of-00003.safetensors",
689
+ "model.base_model.model.model.layers.7.self_attn.k_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
690
+ "model.base_model.model.model.layers.7.self_attn.k_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
691
+ "model.base_model.model.model.layers.7.self_attn.o_proj.base_layer.weight": "model-00001-of-00003.safetensors",
692
+ "model.base_model.model.model.layers.7.self_attn.o_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
693
+ "model.base_model.model.model.layers.7.self_attn.o_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
694
+ "model.base_model.model.model.layers.7.self_attn.q_proj.base_layer.weight": "model-00001-of-00003.safetensors",
695
+ "model.base_model.model.model.layers.7.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
696
+ "model.base_model.model.model.layers.7.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
697
+ "model.base_model.model.model.layers.7.self_attn.v_proj.base_layer.weight": "model-00001-of-00003.safetensors",
698
+ "model.base_model.model.model.layers.7.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
699
+ "model.base_model.model.model.layers.7.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
700
+ "model.base_model.model.model.layers.8.input_layernorm.weight": "model-00001-of-00003.safetensors",
701
+ "model.base_model.model.model.layers.8.mlp.down_proj.base_layer.weight": "model-00001-of-00003.safetensors",
702
+ "model.base_model.model.model.layers.8.mlp.down_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
703
+ "model.base_model.model.model.layers.8.mlp.down_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
704
+ "model.base_model.model.model.layers.8.mlp.gate_proj.base_layer.weight": "model-00001-of-00003.safetensors",
705
+ "model.base_model.model.model.layers.8.mlp.gate_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
706
+ "model.base_model.model.model.layers.8.mlp.gate_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
707
+ "model.base_model.model.model.layers.8.mlp.up_proj.base_layer.weight": "model-00001-of-00003.safetensors",
708
+ "model.base_model.model.model.layers.8.mlp.up_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
709
+ "model.base_model.model.model.layers.8.mlp.up_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
710
+ "model.base_model.model.model.layers.8.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
711
+ "model.base_model.model.model.layers.8.self_attn.k_proj.base_layer.weight": "model-00001-of-00003.safetensors",
712
+ "model.base_model.model.model.layers.8.self_attn.k_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
713
+ "model.base_model.model.model.layers.8.self_attn.k_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
714
+ "model.base_model.model.model.layers.8.self_attn.o_proj.base_layer.weight": "model-00001-of-00003.safetensors",
715
+ "model.base_model.model.model.layers.8.self_attn.o_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
716
+ "model.base_model.model.model.layers.8.self_attn.o_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
717
+ "model.base_model.model.model.layers.8.self_attn.q_proj.base_layer.weight": "model-00001-of-00003.safetensors",
718
+ "model.base_model.model.model.layers.8.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
719
+ "model.base_model.model.model.layers.8.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
720
+ "model.base_model.model.model.layers.8.self_attn.v_proj.base_layer.weight": "model-00001-of-00003.safetensors",
721
+ "model.base_model.model.model.layers.8.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
722
+ "model.base_model.model.model.layers.8.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
723
+ "model.base_model.model.model.layers.9.input_layernorm.weight": "model-00001-of-00003.safetensors",
724
+ "model.base_model.model.model.layers.9.mlp.down_proj.base_layer.weight": "model-00001-of-00003.safetensors",
725
+ "model.base_model.model.model.layers.9.mlp.down_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
726
+ "model.base_model.model.model.layers.9.mlp.down_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
727
+ "model.base_model.model.model.layers.9.mlp.gate_proj.base_layer.weight": "model-00001-of-00003.safetensors",
728
+ "model.base_model.model.model.layers.9.mlp.gate_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
729
+ "model.base_model.model.model.layers.9.mlp.gate_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
730
+ "model.base_model.model.model.layers.9.mlp.up_proj.base_layer.weight": "model-00001-of-00003.safetensors",
731
+ "model.base_model.model.model.layers.9.mlp.up_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
732
+ "model.base_model.model.model.layers.9.mlp.up_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
733
+ "model.base_model.model.model.layers.9.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
734
+ "model.base_model.model.model.layers.9.self_attn.k_proj.base_layer.weight": "model-00001-of-00003.safetensors",
735
+ "model.base_model.model.model.layers.9.self_attn.k_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
736
+ "model.base_model.model.model.layers.9.self_attn.k_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
737
+ "model.base_model.model.model.layers.9.self_attn.o_proj.base_layer.weight": "model-00001-of-00003.safetensors",
738
+ "model.base_model.model.model.layers.9.self_attn.o_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
739
+ "model.base_model.model.model.layers.9.self_attn.o_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
740
+ "model.base_model.model.model.layers.9.self_attn.q_proj.base_layer.weight": "model-00001-of-00003.safetensors",
741
+ "model.base_model.model.model.layers.9.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
742
+ "model.base_model.model.model.layers.9.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
743
+ "model.base_model.model.model.layers.9.self_attn.v_proj.base_layer.weight": "model-00001-of-00003.safetensors",
744
+ "model.base_model.model.model.layers.9.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00003.safetensors",
745
+ "model.base_model.model.model.layers.9.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00003.safetensors",
746
+ "model.base_model.model.model.norm.weight": "model-00003-of-00003.safetensors",
747
+ "rqvae.decoder.mlp_layers.1.bias": "model-00003-of-00003.safetensors",
748
+ "rqvae.decoder.mlp_layers.1.weight": "model-00003-of-00003.safetensors",
749
+ "rqvae.decoder.mlp_layers.10.bias": "model-00003-of-00003.safetensors",
750
+ "rqvae.decoder.mlp_layers.10.weight": "model-00003-of-00003.safetensors",
751
+ "rqvae.decoder.mlp_layers.13.bias": "model-00003-of-00003.safetensors",
752
+ "rqvae.decoder.mlp_layers.13.weight": "model-00003-of-00003.safetensors",
753
+ "rqvae.decoder.mlp_layers.16.bias": "model-00003-of-00003.safetensors",
754
+ "rqvae.decoder.mlp_layers.16.weight": "model-00003-of-00003.safetensors",
755
+ "rqvae.decoder.mlp_layers.19.bias": "model-00003-of-00003.safetensors",
756
+ "rqvae.decoder.mlp_layers.19.weight": "model-00003-of-00003.safetensors",
757
+ "rqvae.decoder.mlp_layers.4.bias": "model-00003-of-00003.safetensors",
758
+ "rqvae.decoder.mlp_layers.4.weight": "model-00003-of-00003.safetensors",
759
+ "rqvae.decoder.mlp_layers.7.bias": "model-00003-of-00003.safetensors",
760
+ "rqvae.decoder.mlp_layers.7.weight": "model-00003-of-00003.safetensors",
761
+ "rqvae.encoder.mlp_layers.1.bias": "model-00003-of-00003.safetensors",
762
+ "rqvae.encoder.mlp_layers.1.weight": "model-00003-of-00003.safetensors",
763
+ "rqvae.encoder.mlp_layers.10.bias": "model-00003-of-00003.safetensors",
764
+ "rqvae.encoder.mlp_layers.10.weight": "model-00003-of-00003.safetensors",
765
+ "rqvae.encoder.mlp_layers.13.bias": "model-00003-of-00003.safetensors",
766
+ "rqvae.encoder.mlp_layers.13.weight": "model-00003-of-00003.safetensors",
767
+ "rqvae.encoder.mlp_layers.16.bias": "model-00003-of-00003.safetensors",
768
+ "rqvae.encoder.mlp_layers.16.weight": "model-00003-of-00003.safetensors",
769
+ "rqvae.encoder.mlp_layers.19.bias": "model-00003-of-00003.safetensors",
770
+ "rqvae.encoder.mlp_layers.19.weight": "model-00003-of-00003.safetensors",
771
+ "rqvae.encoder.mlp_layers.4.bias": "model-00003-of-00003.safetensors",
772
+ "rqvae.encoder.mlp_layers.4.weight": "model-00003-of-00003.safetensors",
773
+ "rqvae.encoder.mlp_layers.7.bias": "model-00003-of-00003.safetensors",
774
+ "rqvae.encoder.mlp_layers.7.weight": "model-00003-of-00003.safetensors",
775
+ "rqvae.rq.vq_layers.0.embedding.weight": "model-00003-of-00003.safetensors",
776
+ "rqvae.rq.vq_layers.1.embedding.weight": "model-00003-of-00003.safetensors",
777
+ "rqvae.rq.vq_layers.2.embedding.weight": "model-00003-of-00003.safetensors",
778
+ "rqvae.rq.vq_layers.3.embedding.weight": "model-00003-of-00003.safetensors"
779
+ }
780
+ }
Ins/special_tokens_map.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<s>",
4
+ "lstrip": false,
5
+ "normalized": true,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "eos_token": {
10
+ "content": "</s>",
11
+ "lstrip": false,
12
+ "normalized": true,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "pad_token": "<unk>",
17
+ "unk_token": {
18
+ "content": "<unk>",
19
+ "lstrip": false,
20
+ "normalized": true,
21
+ "rstrip": false,
22
+ "single_word": false
23
+ }
24
+ }
Ins/tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff
 
Ins/trainer_state.json ADDED
The diff for this file is too large to render. See raw diff
 
__pycache__/collator.cpython-312.pyc ADDED
Binary file (7.31 kB). View file
 
__pycache__/data.cpython-312.pyc ADDED
Binary file (48.2 kB). View file
 
__pycache__/data_finetune.cpython-312.pyc ADDED
Binary file (43.4 kB). View file
 
__pycache__/evaluate.cpython-312.pyc ADDED
Binary file (3.07 kB). View file
 
__pycache__/prompt.cpython-312.pyc ADDED
Binary file (20.1 kB). View file
 
__pycache__/prompt_finetune.cpython-312.pyc ADDED
Binary file (19.8 kB). View file
 
__pycache__/rq_llama.cpython-312.pyc ADDED
Binary file (15.2 kB). View file
 
__pycache__/utils.cpython-312.pyc ADDED
Binary file (22.7 kB). View file
 
collator.py ADDED
@@ -0,0 +1,272 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import torch
2
+ import copy
3
+ import argparse
4
+ from dataclasses import dataclass
5
+
6
+ import transformers
7
+ import math
8
+ from torch.utils.data import Sampler
9
+ import torch.distributed as dist
10
+ from transformers import LlamaForCausalLM, LlamaTokenizer, LlamaConfig, T5Tokenizer, T5Config, T5ForConditionalGeneration
11
+
12
+ class VanillaCollator(object):
13
+ def __init__(self, args, tokenizer):
14
+ self.args = args
15
+ self.tokenizer = tokenizer
16
+ def __call__(self, data):
17
+ # print('collator data:',data)
18
+ '''
19
+ [{
20
+ 'input_ids':
21
+ "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n
22
+ ### Instruction:\n
23
+ Access the user's historical item interaction records: {inters}.
24
+ Your objective is to describe the next potential item for him, taking into account his past interactions.\n\n
25
+ ### Response:",
26
+ 'labels':
27
+ "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n
28
+ ### Instruction:\n
29
+ Access the user's historical item interaction records: {inters}.
30
+ Your objective is to describe the next potential item for him, taking into account his past interactions.\n\n
31
+ ### Response:
32
+ Dunlop guitar picks are a top choice of today's pro musician! Dunlop's wide variety of gauges, shapes, sizes and materials
33
+ allows the player to select the exact pick for his/her own particular style of playing. From classic country to nu-metal,
34
+ every great player knows that their pick is an integral part of their tone, and Dunlop guitar picks are the picks that more
35
+ pros rely on in the studio or on stage. Picks are a grossly underrated accessory. Don't sacrifice your tone...pick Dunlop guitar picks!.",
36
+ 'inters': '341,2804,3895,3893,7064',
37
+ 'item': 'placeholder',
38
+ 'task': 'inters2description'
39
+ },
40
+ {
41
+ 'input_ids':
42
+ 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n
43
+ ### Instruction:\n
44
+ Based on the user\'s historical interactions with the following items: {inters}.
45
+ You can infer his preference by observing the historical interactions: "The user\'s short-term preferences have shift to heavier picks,
46
+ suggesting that He is looking for a heavier sound.". Now the user wants a new item and searches for: "I like the durability and
47
+ effectiveness of the picks.". Please select a suitable item that matches his preference and search intent.\n\n
48
+ ### Response:',
49
+ 'labels':
50
+ 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n
51
+ ### Instruction:\n
52
+ Based on the user\'s historical interactions with the following items: {inters}.
53
+ You can infer his preference by observing the historical interactions: "The user\'s short-term preferences have shift to heavier picks,
54
+ suggesting that He is looking for a heavier sound.". Now the user wants a new item and searches for: "I like the durability and
55
+ effectiveness of the picks.". Please select a suitable item that matches his preference and search intent.\n\n
56
+ ### Response:{item}',
57
+ 'inters': '122,469,8918',
58
+ 'item': '7140',
59
+ 'task': 'itemsearch'
60
+ }]
61
+ '''
62
+ dict_data = {
63
+ 'input_ids': [],
64
+ 'labels': [],
65
+ 'inters': [],
66
+ 'item': [],
67
+ 'task': []
68
+ }
69
+
70
+ for d in data:
71
+ for k in dict_data.keys():
72
+ if k == 'labels':
73
+ dict_data[k].append(d[k] + self.tokenizer.eos_token)
74
+ else:
75
+ dict_data[k].append(d[k])
76
+
77
+ return dict_data
78
+
79
+ class TestCollator(object):
80
+ def __init__(self, args, tokenizer):
81
+ self.args = args
82
+ self.tokenizer = tokenizer
83
+ if self.tokenizer.pad_token_id is None:
84
+ self.tokenizer.pad_token_id = 0
85
+
86
+ if isinstance(self.tokenizer, LlamaTokenizer):
87
+ self.tokenizer.padding_side = "left"
88
+
89
+ def __call__(self, batch):
90
+ input_texts = [d["input_ids"] for d in batch]
91
+ targets = [d["labels"] for d in batch]
92
+ inputs = self.tokenizer(
93
+ text = input_texts,
94
+ return_tensors ="pt",
95
+ padding = "longest",
96
+ max_length = self.tokenizer.model_max_length,
97
+ truncation = True,
98
+ return_attention_mask = True,
99
+ )
100
+
101
+ return (inputs, targets)
102
+
103
+ class Collator(object):
104
+
105
+ def __init__(self, args, tokenizer):
106
+ self.args = args
107
+ self.only_train_response = args.only_train_response
108
+ self.tokenizer = tokenizer
109
+ if self.tokenizer.pad_token_id is None:
110
+ self.tokenizer.pad_token_id = self.tokenizer.unk_token_id
111
+ # print(self.tokenizer.model_max_length)
112
+
113
+ def __call__(self, batch):
114
+
115
+ input_texts = [d["input_ids"] for d in batch]
116
+ full_texts = [d["labels"] + self.tokenizer.eos_token for d in batch]
117
+
118
+ inputs = self.tokenizer(
119
+ text = full_texts,
120
+ text_target = input_texts,
121
+ return_tensors="pt",
122
+ padding="longest",
123
+ max_length=self.tokenizer.model_max_length,
124
+ truncation=True,
125
+ return_attention_mask=True,
126
+ )
127
+ labels = copy.deepcopy(inputs["input_ids"])
128
+ if self.only_train_response:
129
+ # ignore padding
130
+ labels[labels == self.tokenizer.pad_token_id] = -100
131
+ # ignore input text
132
+ labels[torch.where(inputs["labels"] != self.tokenizer.pad_token_id)] = -100
133
+
134
+ inputs["labels"] = labels
135
+
136
+
137
+ return inputs
138
+
139
+ # RuntimeError: Cannot re-initialize CUDA in forked subprocess.
140
+ # To use CUDA with multiprocessing, you must use the 'spawn' start method.
141
+ # class ValidCollator(object):
142
+ # def __init__(self, args, model):
143
+ # self.args = args
144
+ # self.model = model
145
+ # self.only_train_response = args.only_train_response
146
+ # self.tokenizer = model.tokenizer
147
+ # def __call__(self, data):
148
+ # llama_model = self.model.model.get_decoder()
149
+ # for d in data:
150
+ # inter_emb_list = []
151
+ # inter_item_list = d['inters'].split(',')
152
+ # for inter_item in inter_item_list:
153
+ # inter_feature = self.model.item_texts[inter_item]['title'] + ' ' + self.model.item_texts[inter_item]['description']
154
+ # inter_id = self.tokenizer(inter_feature, return_tensors = 'pt', padding=True, truncation=True).to(self.model.device)
155
+ # inter_emb = llama_model(input_ids = inter_id.input_ids, attention_mask = inter_id.attention_mask)
156
+ # inter_emb = inter_emb.last_hidden_state * inter_id.attention_mask.unsqueeze(-1)
157
+ # inter_emb = inter_emb.sum(dim=1) / inter_id.attention_mask.sum(dim = -1, keepdim = True)
158
+ # inter_emb_list.append(inter_emb.detach())
159
+ # inter_embs = torch.cat(inter_emb_list, dim = 0)
160
+ # item_feature = self.model.item_texts[d['item']]['title'] + ' ' + self.model.item_texts[d['item']]['description']
161
+ # item_ids = self.tokenizer(item_feature, return_tensors = 'pt', padding=True, truncation=True).to(self.model.device)
162
+ # item_emb = llama_model(input_ids = item_ids.input_ids, attention_mask = item_ids.attention_mask)
163
+ # item_emb = item_emb.last_hidden_state * item_ids.attention_mask.unsqueeze(-1)
164
+ # item_emb = item_emb.sum(dim=1) / item_ids.attention_mask.sum(dim = -1, keepdim = True)
165
+ # item_emb = item_emb.detach()
166
+
167
+ # rqids = self.model.rqvae.get_indices(torch.cat([inter_embs, item_emb], dim = 0))
168
+
169
+ # inters_rqids = rqids.view(-1, rqids.shape[-1]).cpu().numpy().tolist()[:-1]
170
+ # item_rqid = rqids.view(-1, rqids.shape[-1]).cpu().numpy().tolist()[-1]
171
+
172
+ # text_rqids = {}
173
+ # code = ''
174
+ # for rqid in inters_rqids:
175
+ # for k, idx in enumerate(rqid):
176
+ # code = code + self.model.prefix[k].format(idx)
177
+ # code = code + ', '
178
+ # text_rqids['inters'] = code[:-2]
179
+ # code = ''
180
+ # for k, idx in enumerate(item_rqid):
181
+ # code = code + self.model.prefix[k].format(idx)
182
+ # text_rqids['item'] = code
183
+
184
+ # d['input_ids'] = d['input_ids'].format(inters = text_rqids['inters'])
185
+ # d['labels'] = d['labels'].format(inters = text_rqids['inters'], item = text_rqids['item'])
186
+
187
+ # input_texts = [d["input_ids"] for d in data]
188
+ # full_texts = [d["labels"] + self.tokenizer.eos_token for d in data]
189
+
190
+ # inputs = self.tokenizer(
191
+ # text = full_texts,
192
+ # text_target = input_texts,
193
+ # return_tensors="pt",
194
+ # padding="longest",
195
+ # max_length=self.tokenizer.model_max_length,
196
+ # truncation=True,
197
+ # return_attention_mask=True,
198
+ # )
199
+
200
+ # labels = copy.deepcopy(inputs["input_ids"])
201
+ # if self.only_train_response:
202
+ # labels[labels == self.tokenizer.pad_token_id] = -100
203
+ # labels[torch.where(inputs["labels"] != self.tokenizer.pad_token_id)] = -100
204
+ # inputs["labels"] = labels
205
+
206
+ # return inputs
207
+
208
+ # RuntimeError: Cannot re-initialize CUDA in forked subprocess.
209
+ # To use CUDA with multiprocessing, you must use the 'spawn' start method.
210
+ # class TestCollator(object):
211
+ # def __init__(self, args, model):
212
+ # self.args = args
213
+ # self.model = model
214
+ # self.tokenizer = model.tokenizer
215
+ # if self.tokenizer.pad_token_id is None:
216
+ # self.tokenizer.pad_token_id = 0
217
+ # if isinstance(self.tokenizer, LlamaTokenizer):
218
+ # self.tokenizer.padding_side = "left"
219
+
220
+ # def __call__(self, data):
221
+ # llama_model = self.model.model.get_decoder()
222
+ # for d in data:
223
+ # inter_emb_list = []
224
+ # inter_item_list = d['inters'].split(',')
225
+ # for inter_item in inter_item_list:
226
+ # inter_feature = self.model.item_texts[inter_item]['title'] + ' ' + self.model.item_texts[inter_item]['description']
227
+ # inter_id = self.tokenizer(inter_feature, return_tensors = 'pt', padding=True, truncation=True).to(self.model.device)
228
+ # inter_emb = llama_model(input_ids = inter_id.input_ids, attention_mask = inter_id.attention_mask)
229
+ # inter_emb = inter_emb.last_hidden_state * inter_id.attention_mask.unsqueeze(-1)
230
+ # inter_emb = inter_emb.sum(dim=1) / inter_id.attention_mask.sum(dim = -1, keepdim = True)
231
+ # inter_emb_list.append(inter_emb.detach())
232
+ # inter_embs = torch.cat(inter_emb_list, dim = 0)
233
+ # item_feature = self.model.item_texts[d['item']]['title'] + ' ' + self.model.item_texts[d['item']]['description']
234
+ # item_ids = self.tokenizer(item_feature, return_tensors = 'pt', padding=True, truncation=True).to(self.model.device)
235
+ # item_emb = llama_model(input_ids = item_ids.input_ids, attention_mask = item_ids.attention_mask)
236
+ # item_emb = item_emb.last_hidden_state * item_ids.attention_mask.unsqueeze(-1)
237
+ # item_emb = item_emb.sum(dim=1) / item_ids.attention_mask.sum(dim = -1, keepdim = True)
238
+ # item_emb = item_emb.detach()
239
+
240
+ # rqids = self.model.rqvae.get_indices(torch.cat([inter_embs, item_emb], dim = 0))
241
+
242
+ # inters_rqids = rqids.view(-1, rqids.shape[-1]).cpu().numpy().tolist()[:-1]
243
+ # item_rqid = rqids.view(-1, rqids.shape[-1]).cpu().numpy().tolist()[-1]
244
+
245
+ # text_rqids = {}
246
+ # code = ''
247
+ # for rqid in inters_rqids:
248
+ # for k, idx in enumerate(rqid):
249
+ # code = code + self.model.prefix[k].format(idx)
250
+ # code = code + ', '
251
+ # text_rqids['inters'] = code[:-2]
252
+ # code = ''
253
+ # for k, idx in enumerate(item_rqid):
254
+ # code = code + self.model.prefix[k].format(idx)
255
+ # text_rqids['item'] = code
256
+
257
+ # d['input_ids'] = d['input_ids'].format(inters = text_rqids['inters'])
258
+ # d['labels'] = d['labels'].format(inters = text_rqids['inters'], item = text_rqids['item'])
259
+
260
+ # input_texts = [d["input_ids"] for d in data]
261
+ # targets = [d["labels"] for d in data]
262
+
263
+ # inputs = self.tokenizer(
264
+ # text=input_texts,
265
+ # return_tensors="pt",
266
+ # padding="longest",
267
+ # max_length=self.tokenizer.model_max_length,
268
+ # truncation=True,
269
+ # return_attention_mask=True,
270
+ # )
271
+
272
+ # return (inputs, targets)
config/ds_z2_bf16.json ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bf16": {
3
+ "enabled": "auto"
4
+ },
5
+ "zero_optimization": {
6
+ "stage": 2,
7
+ "allgather_partitions": true,
8
+ "allgather_bucket_size": 5e8,
9
+ "overlap_comm": true,
10
+ "reduce_scatter": true,
11
+ "reduce_bucket_size": 5e8,
12
+ "contiguous_gradients": true
13
+ },
14
+ "gradient_accumulation_steps": "auto",
15
+ "gradient_clipping": "auto",
16
+ "steps_per_print": 2000,
17
+ "train_batch_size": "auto",
18
+ "train_micro_batch_size_per_gpu": "auto",
19
+ "wall_clock_breakdown": false,
20
+ "flops_profiler": {
21
+ "enabled": true,
22
+ "profile_step": 10,
23
+ "module_depth": -1,
24
+ "top_modules": 3,
25
+ "detailed": true,
26
+ "output_file": "flops_profiler.out"
27
+ }
28
+ }
config/ds_z2_fp16.json ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "fp16": {
3
+ "enabled": "auto",
4
+ "auto_cast": false,
5
+ "loss_scale": 0,
6
+ "initial_scale_power": 16,
7
+ "loss_scale_window": 1000,
8
+ "hysteresis": 2,
9
+ "min_loss_scale": 1
10
+ },
11
+ "zero_optimization": {
12
+ "stage": 2,
13
+ "allgather_partitions": true,
14
+ "allgather_bucket_size": 5e8,
15
+ "overlap_comm": true,
16
+ "reduce_scatter": true,
17
+ "reduce_bucket_size": 5e8,
18
+ "contiguous_gradients": true
19
+ },
20
+ "gradient_accumulation_steps": "auto",
21
+ "gradient_clipping": "auto",
22
+ "steps_per_print": 2000,
23
+ "train_batch_size": "auto",
24
+ "train_micro_batch_size_per_gpu": "auto",
25
+ "wall_clock_breakdown": false,
26
+ "flops_profiler": {
27
+ "enabled": true,
28
+ "profile_step": 10,
29
+ "module_depth": -1,
30
+ "top_modules": 3,
31
+ "detailed": true,
32
+ "output_file": "flops_profiler.out"
33
+ }
34
+ }
config/ds_z3_bf16.json ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bf16": {
3
+ "enabled": "auto"
4
+ },
5
+ "zero_optimization": {
6
+ "stage": 3,
7
+ "overlap_comm": true,
8
+ "contiguous_gradients": true,
9
+ "sub_group_size": 1e9,
10
+ "reduce_bucket_size": "auto",
11
+ "stage3_prefetch_bucket_size": "auto",
12
+ "stage3_param_persistence_threshold": "auto",
13
+ "stage3_max_live_parameters": 1e9,
14
+ "stage3_max_reuse_distance": 1e9,
15
+ "stage3_gather_16bit_weights_on_model_save": false
16
+ },
17
+ "gradient_accumulation_steps": "auto",
18
+ "gradient_clipping": "auto",
19
+ "steps_per_print": 2000,
20
+ "train_batch_size": "auto",
21
+ "train_micro_batch_size_per_gpu": "auto",
22
+ "wall_clock_breakdown": false,
23
+ "flops_profiler": {
24
+ "enabled": true,
25
+ "profile_step": 10,
26
+ "module_depth": -1,
27
+ "top_modules": 3,
28
+ "detailed": true,
29
+ "output_file": "flops_profiler.out"
30
+ }
31
+ }
config/ds_z3_bf16_save16bit.json ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bf16": {
3
+ "enabled": "auto"
4
+ },
5
+ "zero_optimization": {
6
+ "stage": 3,
7
+ "overlap_comm": true,
8
+ "contiguous_gradients": true,
9
+ "sub_group_size": 1e9,
10
+ "reduce_bucket_size": "auto",
11
+ "stage3_prefetch_bucket_size": "auto",
12
+ "stage3_param_persistence_threshold": "auto",
13
+ "stage3_max_live_parameters": 1e9,
14
+ "stage3_max_reuse_distance": 1e9,
15
+ "stage3_gather_16bit_weights_on_model_save": true
16
+ },
17
+ "gradient_accumulation_steps": "auto",
18
+ "gradient_clipping": "auto",
19
+ "steps_per_print": 2000,
20
+ "train_batch_size": "auto",
21
+ "train_micro_batch_size_per_gpu": "auto",
22
+ "wall_clock_breakdown": false,
23
+ "flops_profiler": {
24
+ "enabled": true,
25
+ "profile_step": 10,
26
+ "module_depth": -1,
27
+ "top_modules": 3,
28
+ "detailed": true,
29
+ "output_file": "flops_profiler.out"
30
+ }
31
+ }
config/ds_z3_fp16.json ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "fp16": {
3
+ "enabled": "auto",
4
+ "auto_cast": false,
5
+ "loss_scale": 0,
6
+ "initial_scale_power": 16,
7
+ "loss_scale_window": 1000,
8
+ "hysteresis": 2,
9
+ "min_loss_scale": 1
10
+ },
11
+ "zero_optimization": {
12
+ "stage": 3,
13
+ "overlap_comm": true,
14
+ "contiguous_gradients": true,
15
+ "sub_group_size": 1e9,
16
+ "reduce_bucket_size": "auto",
17
+ "stage3_prefetch_bucket_size": "auto",
18
+ "stage3_param_persistence_threshold": "auto",
19
+ "stage3_max_live_parameters": 1e9,
20
+ "stage3_max_reuse_distance": 1e9,
21
+ "stage3_gather_16bit_weights_on_model_save": false
22
+ },
23
+ "gradient_accumulation_steps": "auto",
24
+ "gradient_clipping": "auto",
25
+ "steps_per_print": 2000,
26
+ "train_batch_size": "auto",
27
+ "train_micro_batch_size_per_gpu": "auto",
28
+ "wall_clock_breakdown": false,
29
+ "flops_profiler": {
30
+ "enabled": true,
31
+ "profile_step": 10,
32
+ "module_depth": -1,
33
+ "top_modules": 3,
34
+ "detailed": true,
35
+ "output_file": "flops_profiler.out"
36
+ }
37
+ }
config/ds_z3_fp16_save16bit.json ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "fp16": {
3
+ "enabled": "auto",
4
+ "auto_cast": false,
5
+ "loss_scale": 0,
6
+ "initial_scale_power": 16,
7
+ "loss_scale_window": 1000,
8
+ "hysteresis": 2,
9
+ "min_loss_scale": 1
10
+ },
11
+ "zero_optimization": {
12
+ "stage": 3,
13
+ "overlap_comm": true,
14
+ "contiguous_gradients": true,
15
+ "sub_group_size": 1e9,
16
+ "reduce_bucket_size": "auto",
17
+ "stage3_prefetch_bucket_size": "auto",
18
+ "stage3_param_persistence_threshold": "auto",
19
+ "stage3_max_live_parameters": 1e9,
20
+ "stage3_max_reuse_distance": 1e9,
21
+ "stage3_gather_16bit_weights_on_model_save": true
22
+ },
23
+ "gradient_accumulation_steps": "auto",
24
+ "gradient_clipping": "auto",
25
+ "steps_per_print": 2000,
26
+ "train_batch_size": "auto",
27
+ "train_micro_batch_size_per_gpu": "auto",
28
+ "wall_clock_breakdown": false,
29
+ "flops_profiler": {
30
+ "enabled": true,
31
+ "profile_step": 10,
32
+ "module_depth": -1,
33
+ "top_modules": 3,
34
+ "detailed": true,
35
+ "output_file": "flops_profiler.out"
36
+ }
37
+ }
continue_finetune.py ADDED
@@ -0,0 +1,108 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import argparse
2
+ import os
3
+ import sys
4
+ from typing import List
5
+
6
+ import torch
7
+ import transformers
8
+ from peft import PeftModel
9
+ from peft import (
10
+ TaskType,
11
+ LoraConfig,
12
+ get_peft_model,
13
+ get_peft_model_state_dict,
14
+ set_peft_model_state_dict,
15
+ )
16
+ from transformers import LlamaForCausalLM, LlamaTokenizer, LlamaConfig
17
+
18
+ from utils import *
19
+ from collator import Collator
20
+
21
+ import argparse
22
+ from utils import *
23
+ from rq_llama import *
24
+
25
+ parser = argparse.ArgumentParser(description = 'rqllama-finetune')
26
+ parser = parse_finetune_args(parser)
27
+ args = parser.parse_args()
28
+
29
+ set_seed(args.seed)
30
+ ensure_dir(args.output_dir)
31
+
32
+ device_map = "auto"
33
+ world_size = int(os.environ.get("WORLD_SIZE", 1))
34
+ ddp = world_size != 1
35
+ local_rank = int(os.environ.get("LOCAL_RANK") or 0)
36
+ if local_rank == 0:
37
+ print(vars(args))
38
+
39
+ if ddp:
40
+ device_map = {"": local_rank}
41
+
42
+ train_data, valid_data = load_finetune_datasets(args)
43
+
44
+ tokenizer = LlamaTokenizer.from_pretrained(args.ckpt_path)
45
+ base_model = LlamaForCausalLM.from_pretrained(args.base_model, torch_dtype=torch.float16, low_cpu_mem_usage = True, device_map = device_map)
46
+ base_model.resize_token_embeddings(len(tokenizer))
47
+ rqllama = PeftModel.from_pretrained(base_model, args.ckpt_path, torch_dtype = torch.float16, device_map = device_map)
48
+
49
+ if local_rank == 0:
50
+ print("token num:", len(tokenizer))
51
+ print("data num:", len(train_data))
52
+
53
+ collator = Collator(args, tokenizer)
54
+
55
+ rqllama.train()
56
+
57
+ if local_rank == 0:
58
+ rqllama.print_trainable_parameters()
59
+
60
+ trainer = transformers.Trainer(
61
+ model = rqllama,
62
+ train_dataset = train_data,
63
+ eval_dataset = valid_data,
64
+ args = transformers.TrainingArguments(
65
+ seed = args.seed,
66
+ per_device_train_batch_size = args.per_device_batch_size,
67
+ per_device_eval_batch_size = args.per_device_batch_size,
68
+ gradient_accumulation_steps = args.gradient_accumulation_steps,
69
+ warmup_ratio = args.warmup_ratio,
70
+ num_train_epochs = args.epochs,
71
+ learning_rate = args.learning_rate,
72
+ weight_decay = args.weight_decay,
73
+ lr_scheduler_type = args.lr_scheduler_type,
74
+ fp16 = args.fp16,
75
+ bf16 = args.bf16,
76
+ logging_steps = args.logging_step,
77
+ optim = args.optim,
78
+ gradient_checkpointing = True,
79
+ evaluation_strategy = args.save_and_eval_strategy,
80
+ save_strategy = args.save_and_eval_strategy,
81
+ eval_steps = args.save_and_eval_steps,
82
+ save_steps = args.save_and_eval_steps,
83
+ output_dir = args.output_dir,
84
+ save_total_limit = 5,
85
+ load_best_model_at_end = True,
86
+ deepspeed = args.deepspeed,
87
+ ddp_find_unused_parameters = False if ddp else None,
88
+ report_to = None,
89
+ eval_delay = 1 if args.save_and_eval_strategy=="epoch" else 2000,
90
+ dataloader_num_workers = args.dataloader_num_workers,
91
+ dataloader_prefetch_factor = args.dataloader_prefetch_factor,
92
+ remove_unused_columns = args.remove_unused_columns,
93
+ ),
94
+ tokenizer = tokenizer,
95
+ data_collator = collator,
96
+ )
97
+ rqllama.config.use_cache = False
98
+
99
+ if torch.__version__ >= "2" and sys.platform != "win32":
100
+ rqllama = torch.compile(rqllama)
101
+
102
+ trainer.train(resume_from_checkpoint = args.resume_from_checkpoint)
103
+
104
+ trainer.save_state()
105
+ trainer.save_model(output_dir = args.output_dir)
106
+
107
+ if local_rank == 0:
108
+ print('rqllama fine-tune finished.')
continue_pretrain.py ADDED
@@ -0,0 +1,126 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import os
2
+ import sys
3
+ from typing import List
4
+ import argparse
5
+
6
+ import wandb
7
+ import torch
8
+ import transformers
9
+ from transformers import LlamaForCausalLM, LlamaTokenizer, LlamaConfig
10
+
11
+ from peft import (
12
+ TaskType,
13
+ LoraConfig,
14
+ get_peft_model,
15
+ get_peft_model_state_dict,
16
+ set_peft_model_state_dict,
17
+ )
18
+
19
+ from collator import VanillaCollator
20
+ from rq_llama import *
21
+ from utils import *
22
+
23
+ parser = argparse.ArgumentParser(description = 'rqllama-pretrain-more')
24
+ parser = parse_global_args(parser)
25
+ parser = parse_train_args(parser)
26
+ parser = parse_dataset_args(parser)
27
+ parser = parse_rqvae_args(parser)
28
+ parser = parse_pretrain_args(parser)
29
+ args = parser.parse_args()
30
+ wandb.init(config = args, reinit = True)
31
+
32
+ set_seed(args.seed)
33
+ ensure_dir(args.output_dir)
34
+
35
+ device_map = "auto"
36
+ world_size = int(os.environ.get("WORLD_SIZE", 1))
37
+ ddp = world_size != 1
38
+ local_rank = int(os.environ.get("LOCAL_RANK") or 0)
39
+ if local_rank == 0:
40
+ print(vars(args))
41
+ if ddp:
42
+ device_map = {"": local_rank}
43
+
44
+ train_data, valid_data = load_datasets(args)
45
+
46
+ rqllama = LlamaWithRQ.from_pretrained(args.ckpt_path, torch_dtype = torch.float16, low_cpu_mem_usage = True, device_map = device_map)
47
+
48
+ for i in range(len(args.num_emb_list)):
49
+ rqllama.rqvae.rq.vq_layers[i].initted = True
50
+
51
+ if local_rank == 0:
52
+ print("token num:", len(rqllama.tokenizer))
53
+ print("data num:", len(train_data))
54
+ rqllama.tokenizer.save_pretrained(args.output_dir)
55
+ rqllama.config.save_pretrained(args.output_dir)
56
+
57
+ if args.resume_from_checkpoint:
58
+ checkpoint_name = os.path.join(args.resume_from_checkpoint, "adapter_model.bin")
59
+ args.resume_from_checkpoint = False
60
+ if os.path.exists(checkpoint_name):
61
+ if local_rank == 0:
62
+ print(f"Restarting from {checkpoint_name}")
63
+ adapters_weights = torch.load(checkpoint_name)
64
+ rqllama.model = set_peft_model_state_dict(rqllama.model, adapters_weights)
65
+ else:
66
+ if local_rank == 0:
67
+ print(f"Checkpoint {checkpoint_name} not found")
68
+
69
+ if local_rank == 0:
70
+ rqllama.model.print_trainable_parameters()
71
+
72
+ if not ddp and torch.cuda.device_count() > 1:
73
+ rqllama.is_parallelizable = True
74
+ rqllama.model_parallel = True
75
+
76
+ collator = VanillaCollator(args, rqllama.tokenizer)
77
+
78
+ trainer = transformers.Trainer(
79
+ model = rqllama,
80
+ train_dataset = train_data,
81
+ eval_dataset = valid_data,
82
+ args = transformers.TrainingArguments(
83
+ seed = args.seed,
84
+ per_device_train_batch_size = args.per_device_batch_size,
85
+ per_device_eval_batch_size = args.per_device_batch_size,
86
+ gradient_accumulation_steps = args.gradient_accumulation_steps,
87
+ warmup_ratio = args.warmup_ratio,
88
+ num_train_epochs = args.epochs,
89
+ learning_rate = args.learning_rate,
90
+ weight_decay = args.weight_decay,
91
+ lr_scheduler_type = args.lr_scheduler_type,
92
+ fp16 = args.fp16,
93
+ bf16 = args.bf16,
94
+ logging_steps = args.logging_step,
95
+ optim = args.optim,
96
+ gradient_checkpointing = True,
97
+ evaluation_strategy = args.save_and_eval_strategy,
98
+ save_strategy = args.save_and_eval_strategy,
99
+ eval_steps = args.save_and_eval_steps,
100
+ save_steps = args.save_and_eval_steps,
101
+ output_dir = args.output_dir,
102
+ save_total_limit = 5,
103
+ load_best_model_at_end = True,
104
+ deepspeed = args.deepspeed,
105
+ ddp_find_unused_parameters = False if ddp else None,
106
+ report_to = None,
107
+ eval_delay = 1 if args.save_and_eval_strategy=="epoch" else 2000,
108
+ dataloader_num_workers = args.dataloader_num_workers,
109
+ dataloader_prefetch_factor = args.dataloader_prefetch_factor,
110
+ remove_unused_columns = args.remove_unused_columns,
111
+ ),
112
+ tokenizer = rqllama.tokenizer,
113
+ data_collator = collator,
114
+ )
115
+ rqllama.config.use_cache = False
116
+
117
+ if torch.__version__ >= "2" and sys.platform != "win32":
118
+ rqllama = torch.compile(rqllama)
119
+
120
+ trainer.train(resume_from_checkpoint = args.resume_from_checkpoint)
121
+
122
+ trainer.save_state()
123
+ trainer.save_model(output_dir = args.output_dir)
124
+
125
+ if local_rank == 0:
126
+ print('rqllama pre-train finished.')
convert/convert.log ADDED
@@ -0,0 +1 @@
 
 
1
+ nohup: failed to run command './convert.sh': Permission denied
convert/convert.py ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import transformers
2
+ import argparse
3
+ import os
4
+
5
+ if __name__ == '__main__':
6
+ parser = argparse.ArgumentParser()
7
+ parser.add_argument("--source", "-s", type=str, default="", help="source path of models")
8
+ parser.add_argument("--target", "-t", type=str, default="", help="target path of models")
9
+
10
+ args, _ = parser.parse_known_args()
11
+
12
+ assert os.path.exists(args.source)
13
+ assert args.target != ""
14
+
15
+ model = transformers.AutoModelForCausalLM.from_pretrained(args.source)
16
+ model.save_pretrained(args.target, state_dict=model.state_dict())
convert/convert.sh ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ model=$1
2
+
3
+ set -x
4
+
5
+ for step in `ls ${model} | grep checkpoint | awk -F'-' '{ print $2 }'`
6
+ do
7
+ mkdir ${model}/tmp-checkpoint-${step}
8
+ mkdir ${model}/final-checkpoint-${step}
9
+ python ./zero_to_fp32.py ${model}/checkpoint-${step}/ ${model}/tmp-checkpoint-${step}/pytorch_model.bin
10
+ cp ${model}/*.json ${model}/tmp-checkpoint-${step}
11
+ python ./convert.py -s ${model}/tmp-checkpoint-${step} -t ${model}/final-checkpoint-${step}
12
+ cp ${model}/checkpoint-${step}/*.json ${model}/final-checkpoint-${step}
13
+ cp ${model}/*.json ${model}/final-checkpoint-${step}
14
+ cp ${model}/tokenizer* ${model}/final-checkpoint-${step}
15
+ cp ${model}/train* ${model}/final-checkpoint-${step}
16
+ #rm -rf ${model}/tmp-checkpoint-${step} ${model}/checkpoint-${step} ${model}/global_step${step}
17
+ #mv ${model}/final-checkpoint-${step} ${model}/checkpoint-${step}
18
+ done
convert/convert_fp16.py ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ import argparse
3
+
4
+ from transformers import AutoTokenizer, AutoModelForCausalLM
5
+ import torch
6
+
7
+
8
+ def convert_fp16(in_checkpoint, out_checkpoint):
9
+ tokenizer = AutoTokenizer.from_pretrained(in_checkpoint, use_fast=False)
10
+ model = AutoModelForCausalLM.from_pretrained(
11
+ in_checkpoint, torch_dtype=torch.float16, low_cpu_mem_usage=True
12
+ )
13
+ model.save_pretrained(out_checkpoint)
14
+ tokenizer.save_pretrained(out_checkpoint)
15
+
16
+
17
+ if __name__ == "__main__":
18
+ parser = argparse.ArgumentParser()
19
+ parser.add_argument("--in-checkpoint", type=str, help="Path to the model")
20
+ parser.add_argument("--out-checkpoint", type=str, help="Path to the output model")
21
+ args = parser.parse_args()
22
+
23
+ convert_fp16(args.in_checkpoint, args.out_checkpoint)
convert/make_delta.py ADDED
@@ -0,0 +1,46 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ import argparse
3
+
4
+ import torch
5
+ from tqdm import tqdm
6
+ from transformers import AutoTokenizer, AutoModelForCausalLM
7
+
8
+
9
+ def make_delta(base_model_path, target_model_path, delta_path):
10
+ print(f"Loading the base model from {base_model_path}")
11
+ base = AutoModelForCausalLM.from_pretrained(
12
+ base_model_path, torch_dtype=torch.float16, low_cpu_mem_usage=True
13
+ )
14
+
15
+ print(f"Loading the target model from {target_model_path}")
16
+ target = AutoModelForCausalLM.from_pretrained(
17
+ target_model_path, torch_dtype=torch.float16, low_cpu_mem_usage=True
18
+ )
19
+ target_tokenizer = AutoTokenizer.from_pretrained(target_model_path, use_fast=False)
20
+
21
+ print("Calculating the delta")
22
+ for name, param in tqdm(target.state_dict().items(), desc="Calculating delta"):
23
+ assert name in base.state_dict()
24
+ if param.shape == base.state_dict()[name].shape:
25
+ param.data -= base.state_dict()[name]
26
+ else:
27
+ print(name)
28
+
29
+ print(f"Saving the delta to {delta_path}")
30
+ if args.hub_repo_id:
31
+ kwargs = {"push_to_hub": True, "repo_id": args.hub_repo_id}
32
+ else:
33
+ kwargs = {}
34
+ target.save_pretrained(delta_path, **kwargs)
35
+ target_tokenizer.save_pretrained(delta_path, **kwargs)
36
+
37
+
38
+ if __name__ == "__main__":
39
+ parser = argparse.ArgumentParser()
40
+ parser.add_argument("--base-model-path", type=str, required=True)
41
+ parser.add_argument("--target-model-path", type=str, required=True)
42
+ parser.add_argument("--delta-path", type=str, required=True)
43
+ parser.add_argument("--hub-repo-id", type=str)
44
+ args = parser.parse_args()
45
+
46
+ make_delta(args.base_model_path, args.target_model_path, args.delta_path)
convert/merge_delta.py ADDED
@@ -0,0 +1,167 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ import argparse
3
+ import gc
4
+ import glob
5
+ import json
6
+ import os
7
+ import shutil
8
+ import tempfile
9
+
10
+ from huggingface_hub import snapshot_download
11
+ import torch
12
+ from torch import nn
13
+ from tqdm import tqdm
14
+ from transformers import AutoTokenizer, AutoModelForCausalLM, AutoConfig
15
+
16
+
17
+ GB = 1 << 30
18
+
19
+
20
+ def split_files(model_path, tmp_path, split_size):
21
+ if not os.path.exists(model_path):
22
+ model_path = snapshot_download(repo_id=model_path)
23
+ if not os.path.exists(tmp_path):
24
+ os.makedirs(tmp_path)
25
+
26
+ file_pattern = os.path.join(model_path, "pytorch_model-*.bin")
27
+ files = glob.glob(file_pattern)
28
+
29
+ part = 0
30
+ try:
31
+ for file_path in tqdm(files):
32
+ state_dict = torch.load(file_path)
33
+ new_state_dict = {}
34
+
35
+ current_size = 0
36
+ for name, param in state_dict.items():
37
+ param_size = param.numel() * param.element_size()
38
+
39
+ if current_size + param_size > split_size:
40
+ new_file_name = f"pytorch_model-{part}.bin"
41
+ new_file_path = os.path.join(tmp_path, new_file_name)
42
+ torch.save(new_state_dict, new_file_path)
43
+ current_size = 0
44
+ new_state_dict = None
45
+ gc.collect()
46
+ new_state_dict = {}
47
+ part += 1
48
+
49
+ new_state_dict[name] = param
50
+ current_size += param_size
51
+
52
+ new_file_name = f"pytorch_model-{part}.bin"
53
+ new_file_path = os.path.join(tmp_path, new_file_name)
54
+ torch.save(new_state_dict, new_file_path)
55
+ new_state_dict = None
56
+ gc.collect()
57
+ new_state_dict = {}
58
+ part += 1
59
+ except Exception as e:
60
+ print(f"An error occurred during split_files: {e}")
61
+ shutil.rmtree(tmp_path)
62
+ raise
63
+
64
+
65
+ def apply_delta_low_cpu_mem(base_model_path, target_model_path, delta_path):
66
+ delta_tokenizer = AutoTokenizer.from_pretrained(delta_path, use_fast=False)
67
+ delta_config = AutoConfig.from_pretrained(delta_path)
68
+
69
+ if os.path.exists(target_model_path):
70
+ shutil.rmtree(target_model_path)
71
+ os.makedirs(target_model_path)
72
+
73
+ split_size = 4 * GB
74
+
75
+ with tempfile.TemporaryDirectory() as tmp_base_path, tempfile.TemporaryDirectory() as tmp_delta_path:
76
+ print(f"Split files for the base model to {tmp_base_path}")
77
+ split_files(base_model_path, tmp_base_path, split_size)
78
+ print(f"Split files for the delta weights to {tmp_delta_path}")
79
+ split_files(delta_path, tmp_delta_path, split_size)
80
+
81
+ base_pattern = os.path.join(tmp_base_path, "pytorch_model-*.bin")
82
+ base_files = glob.glob(base_pattern)
83
+ base_state_dict = torch.load(base_files[0])
84
+ delta_pattern = os.path.join(tmp_delta_path, "pytorch_model-*.bin")
85
+ delta_files = glob.glob(delta_pattern)
86
+ # delta_state_dict = torch.load(delta_files[0])
87
+
88
+ print("Applying the delta")
89
+ weight_map = {}
90
+ total_size = 0
91
+
92
+ for i, delta_file in tqdm(enumerate(delta_files)):
93
+ state_dict = torch.load(delta_file)
94
+ file_name = f"pytorch_model-{i}.bin"
95
+ for name, param in state_dict.items():
96
+ if name not in base_state_dict:
97
+ for base_file in base_files:
98
+ base_state_dict = torch.load(base_file)
99
+ gc.collect()
100
+ if name in base_state_dict:
101
+ break
102
+ if state_dict[name].shape == base_state_dict[name].shape:
103
+ state_dict[name] += base_state_dict[name]
104
+ else:
105
+ print(name)
106
+ weight_map[name] = file_name
107
+ total_size += param.numel() * param.element_size()
108
+ gc.collect()
109
+ torch.save(state_dict, os.path.join(target_model_path, file_name))
110
+
111
+ with open(
112
+ os.path.join(target_model_path, "pytorch_model.bin.index.json"), "w"
113
+ ) as f:
114
+ json.dump(
115
+ {"weight_map": weight_map, "metadata": {"total_size": total_size}}, f
116
+ )
117
+
118
+ print(f"Saving the target model to {target_model_path}")
119
+ delta_tokenizer.save_pretrained(target_model_path)
120
+ delta_config.save_pretrained(target_model_path)
121
+
122
+
123
+ def apply_delta(base_model_path, target_model_path, delta_path):
124
+ print(f"Loading the delta weights from {delta_path}")
125
+ delta_tokenizer = AutoTokenizer.from_pretrained(delta_path, use_fast=False)
126
+ delta = AutoModelForCausalLM.from_pretrained(
127
+ delta_path, torch_dtype=torch.float16, low_cpu_mem_usage=True
128
+ )
129
+
130
+ print(f"Loading the base model from {base_model_path}")
131
+ base = AutoModelForCausalLM.from_pretrained(
132
+ base_model_path, torch_dtype=torch.float16, low_cpu_mem_usage=True
133
+ )
134
+
135
+ print("Applying the delta")
136
+ for name, param in tqdm(delta.state_dict().items(), desc="Applying delta"):
137
+ assert name in base.state_dict()
138
+ if param.shape == base.state_dict()[name].shape:
139
+ param.data += base.state_dict()[name]
140
+ else:
141
+ print(name)
142
+
143
+
144
+ print(f"Saving the target model to {target_model_path}")
145
+ delta.save_pretrained(target_model_path)
146
+ delta_tokenizer.save_pretrained(target_model_path)
147
+
148
+
149
+ if __name__ == "__main__":
150
+ parser = argparse.ArgumentParser()
151
+ parser.add_argument("--base-model-path", type=str, required=True)
152
+ parser.add_argument("--target-model-path", type=str, required=True)
153
+ parser.add_argument("--delta-path", type=str, required=True)
154
+ parser.add_argument(
155
+ "--low-cpu-mem",
156
+ action="store_true",
157
+ help="Lower the cpu memory usage. This will split large files and use "
158
+ "disk as swap to reduce the memory usage below 10GB.",
159
+ )
160
+ args = parser.parse_args()
161
+
162
+ if args.low_cpu_mem:
163
+ apply_delta_low_cpu_mem(
164
+ args.base_model_path, args.target_model_path, args.delta_path
165
+ )
166
+ else:
167
+ apply_delta(args.base_model_path, args.target_model_path, args.delta_path)
convert/zero_to_fp32.py ADDED
@@ -0,0 +1,600 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python
2
+
3
+ # Copyright (c) Microsoft Corporation.
4
+ # SPDX-License-Identifier: Apache-2.0
5
+
6
+ # DeepSpeed Team
7
+
8
+ # This script extracts fp32 consolidated weights from a zero 2 and 3 DeepSpeed checkpoints. It gets
9
+ # copied into the top level checkpoint dir, so the user can easily do the conversion at any point in
10
+ # the future. Once extracted, the weights don't require DeepSpeed and can be used in any
11
+ # application.
12
+ #
13
+ # example: python zero_to_fp32.py . pytorch_model.bin
14
+
15
+ import argparse
16
+ import torch
17
+ import glob
18
+ import math
19
+ import os
20
+ import re
21
+ from collections import OrderedDict
22
+ from dataclasses import dataclass
23
+ from tqdm import tqdm
24
+
25
+ # while this script doesn't use deepspeed to recover data, since the checkpoints are pickled with
26
+ # DeepSpeed data structures it has to be available in the current python environment.
27
+ from deepspeed.utils import logger
28
+ from deepspeed.checkpoint.constants import (DS_VERSION, OPTIMIZER_STATE_DICT, SINGLE_PARTITION_OF_FP32_GROUPS,
29
+ FP32_FLAT_GROUPS, ZERO_STAGE, PARTITION_COUNT, PARAM_SHAPES, BUFFER_NAMES,
30
+ FROZEN_PARAM_SHAPES, FROZEN_PARAM_FRAGMENTS)
31
+
32
+
33
+ @dataclass
34
+ class zero_model_state:
35
+ buffers: dict()
36
+ param_shapes: dict()
37
+ shared_params: list
38
+ ds_version: int
39
+ frozen_param_shapes: dict()
40
+ frozen_param_fragments: dict()
41
+
42
+
43
+ debug = 0
44
+
45
+ # load to cpu
46
+ device = torch.device('cpu')
47
+
48
+
49
+ def atoi(text):
50
+ return int(text) if text.isdigit() else text
51
+
52
+
53
+ def natural_keys(text):
54
+ '''
55
+ alist.sort(key=natural_keys) sorts in human order
56
+ http://nedbatchelder.com/blog/200712/human_sorting.html
57
+ (See Toothy's implementation in the comments)
58
+ '''
59
+ return [atoi(c) for c in re.split(r'(\d+)', text)]
60
+
61
+
62
+ def get_model_state_file(checkpoint_dir, zero_stage):
63
+ if not os.path.isdir(checkpoint_dir):
64
+ raise FileNotFoundError(f"Directory '{checkpoint_dir}' doesn't exist")
65
+
66
+ # there should be only one file
67
+ if zero_stage == 2:
68
+ file = os.path.join(checkpoint_dir, "mp_rank_00_model_states.pt")
69
+ elif zero_stage == 3:
70
+ file = os.path.join(checkpoint_dir, "zero_pp_rank_0_mp_rank_00_model_states.pt")
71
+
72
+ if not os.path.exists(file):
73
+ raise FileNotFoundError(f"can't find model states file at '{file}'")
74
+
75
+ return file
76
+
77
+
78
+ def get_checkpoint_files(checkpoint_dir, glob_pattern):
79
+ # XXX: need to test that this simple glob rule works for multi-node setup too
80
+ ckpt_files = sorted(glob.glob(os.path.join(checkpoint_dir, glob_pattern)), key=natural_keys)
81
+
82
+ if len(ckpt_files) == 0:
83
+ raise FileNotFoundError(f"can't find {glob_pattern} files in directory '{checkpoint_dir}'")
84
+
85
+ return ckpt_files
86
+
87
+
88
+ def get_optim_files(checkpoint_dir):
89
+ return get_checkpoint_files(checkpoint_dir, "*_optim_states.pt")
90
+
91
+
92
+ def get_model_state_files(checkpoint_dir):
93
+ return get_checkpoint_files(checkpoint_dir, "*_model_states.pt")
94
+
95
+
96
+ def parse_model_states(files):
97
+ zero_model_states = []
98
+ for file in files:
99
+ state_dict = torch.load(file, map_location=device)
100
+
101
+ if BUFFER_NAMES not in state_dict:
102
+ raise ValueError(f"{file} is not a model state checkpoint")
103
+ buffer_names = state_dict[BUFFER_NAMES]
104
+ if debug:
105
+ print("Found buffers:", buffer_names)
106
+
107
+ # recover just the buffers while restoring them to fp32 if they were saved in fp16
108
+ buffers = {k: v.float() for k, v in state_dict["module"].items() if k in buffer_names}
109
+ param_shapes = state_dict[PARAM_SHAPES]
110
+
111
+ # collect parameters that are included in param_shapes
112
+ param_names = []
113
+ for s in param_shapes:
114
+ for name in s.keys():
115
+ param_names.append(name)
116
+
117
+ # update with frozen parameters
118
+ frozen_param_shapes = state_dict.get(FROZEN_PARAM_SHAPES, None)
119
+ if frozen_param_shapes is not None:
120
+ if debug:
121
+ print(f"Found frozen_param_shapes: {frozen_param_shapes}")
122
+ param_names += list(frozen_param_shapes.keys())
123
+
124
+ # record shared parameters so that they can be recovered based on partners
125
+ # this is because such parameters holding reference only are not saved by optimizer
126
+ shared_params = []
127
+ for param in state_dict["module"]:
128
+ if param not in [*param_names, *buffer_names]:
129
+ for share_param in state_dict["module"]:
130
+ if (state_dict["module"][share_param].data_ptr() == state_dict["module"][param].data_ptr()
131
+ and share_param != param):
132
+ shared_params.append([param, share_param])
133
+ break
134
+
135
+ ds_version = state_dict.get(DS_VERSION, None)
136
+
137
+ frozen_param_fragments = state_dict.get(FROZEN_PARAM_FRAGMENTS, None)
138
+
139
+ z_model_state = zero_model_state(buffers=buffers,
140
+ param_shapes=param_shapes,
141
+ shared_params=shared_params,
142
+ ds_version=ds_version,
143
+ frozen_param_shapes=frozen_param_shapes,
144
+ frozen_param_fragments=frozen_param_fragments)
145
+ zero_model_states.append(z_model_state)
146
+
147
+ return zero_model_states
148
+
149
+
150
+ def parse_optim_states(files, ds_checkpoint_dir):
151
+
152
+ total_files = len(files)
153
+ state_dicts = []
154
+ for i, f in enumerate(tqdm(files)):
155
+ state_dicts.append(torch.load(f, map_location=device))
156
+ if i == 0:
157
+ if not ZERO_STAGE in state_dicts[0][OPTIMIZER_STATE_DICT]:
158
+ raise ValueError(f"{files[0]} is not a zero checkpoint")
159
+ zero_stage = state_dicts[0][OPTIMIZER_STATE_DICT][ZERO_STAGE]
160
+ world_size = state_dicts[0][OPTIMIZER_STATE_DICT][PARTITION_COUNT]
161
+
162
+ # For ZeRO-2 each param group can have different partition_count as data parallelism for expert
163
+ # parameters can be different from data parallelism for non-expert parameters. So we can just
164
+ # use the max of the partition_count to get the dp world_size.
165
+
166
+ if type(world_size) is list:
167
+ world_size = max(world_size)
168
+
169
+ if world_size != total_files:
170
+ raise ValueError(
171
+ f"Expected {world_size} of '*_optim_states.pt' under '{ds_checkpoint_dir}' but found {total_files} files. "
172
+ "Possibly due to an overwrite of an old checkpoint, or a checkpoint didn't get saved by one or more processes."
173
+ )
174
+
175
+ # the groups are named differently in each stage
176
+ if zero_stage == 2:
177
+ fp32_groups_key = SINGLE_PARTITION_OF_FP32_GROUPS
178
+ elif zero_stage == 3:
179
+ fp32_groups_key = FP32_FLAT_GROUPS
180
+ else:
181
+ raise ValueError(f"unknown zero stage {zero_stage}")
182
+
183
+ key_list = list(state_dicts[-1][OPTIMIZER_STATE_DICT].keys())
184
+ for key in key_list:
185
+ if zero_stage == 2:
186
+ if key != fp32_groups_key:
187
+ del state_dicts[-1][OPTIMIZER_STATE_DICT][key]
188
+ elif zero_stage == 3:
189
+ if key == fp32_groups_key:
190
+ value = torch.cat(state_dicts[-1][OPTIMIZER_STATE_DICT][fp32_groups_key], 0)
191
+ del state_dicts[-1][OPTIMIZER_STATE_DICT][key]
192
+ if key == fp32_groups_key:
193
+ state_dicts[-1][OPTIMIZER_STATE_DICT][key] = value
194
+
195
+ print('zero_stage:', zero_stage)
196
+ fp32_flat_groups = [state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key] for i in range(len(state_dicts))]
197
+ # if zero_stage == 2:
198
+ # # fp32_flat_groups = [state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key] for i in range(len(state_dicts))]
199
+ # elif zero_stage == 3:
200
+ # # if there is more than one param group, there will be multiple flattened tensors - one
201
+ # # flattened tensor per group - for simplicity merge them into a single tensor
202
+ # #
203
+ # # XXX: could make the script more memory efficient for when there are multiple groups - it
204
+ # # will require matching the sub-lists of param_shapes for each param group flattened tensor
205
+
206
+ # print('start!')
207
+ # # fp32_flat_groups = [
208
+ # # torch.cat(state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key], 0) for i in range(len(state_dicts))
209
+ # # ]
210
+
211
+ return zero_stage, world_size, fp32_flat_groups
212
+
213
+
214
+ def _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir):
215
+ """
216
+ Returns fp32 state_dict reconstructed from ds checkpoint
217
+
218
+ Args:
219
+ - ``ds_checkpoint_dir``: path to the deepspeed checkpoint folder (where the optimizer files are)
220
+
221
+ """
222
+ print(f"Processing zero checkpoint '{ds_checkpoint_dir}'")
223
+
224
+ optim_files = get_optim_files(ds_checkpoint_dir)
225
+ zero_stage, world_size, fp32_flat_groups = parse_optim_states(optim_files, ds_checkpoint_dir)
226
+ print(f"Detected checkpoint of type zero stage {zero_stage}, world_size: {world_size}")
227
+
228
+ model_files = get_model_state_files(ds_checkpoint_dir)
229
+
230
+ zero_model_states = parse_model_states(model_files)
231
+ print(f'Parsing checkpoint created by deepspeed=={zero_model_states[0].ds_version}')
232
+
233
+ if zero_stage == 2:
234
+ return _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states)
235
+ elif zero_stage == 3:
236
+ return _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states)
237
+
238
+
239
+ def _zero2_merge_frozen_params(state_dict, zero_model_states):
240
+ if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
241
+ return
242
+
243
+ frozen_param_shapes = zero_model_states[0].frozen_param_shapes
244
+ frozen_param_fragments = zero_model_states[0].frozen_param_fragments
245
+
246
+ if debug:
247
+ num_elem = sum(s.numel() for s in frozen_param_shapes.values())
248
+ print(f'rank 0: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
249
+
250
+ wanted_params = len(frozen_param_shapes)
251
+ wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
252
+ avail_numel = sum([p.numel() for p in frozen_param_fragments.values()])
253
+ print(f'Frozen params: Have {avail_numel} numels to process.')
254
+ print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
255
+
256
+ total_params = 0
257
+ total_numel = 0
258
+ for name, shape in frozen_param_shapes.items():
259
+ total_params += 1
260
+ unpartitioned_numel = shape.numel()
261
+ total_numel += unpartitioned_numel
262
+
263
+ state_dict[name] = frozen_param_fragments[name]
264
+
265
+ if debug:
266
+ print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
267
+
268
+ print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
269
+
270
+
271
+ def _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
272
+ param_shapes = zero_model_states[0].param_shapes
273
+
274
+ # Reconstruction protocol:
275
+ #
276
+ # XXX: document this
277
+
278
+ if debug:
279
+ for i in range(world_size):
280
+ for j in range(len(fp32_flat_groups[0])):
281
+ print(f"{FP32_FLAT_GROUPS}[{i}][{j}].shape={fp32_flat_groups[i][j].shape}")
282
+
283
+ # XXX: memory usage doubles here (zero2)
284
+ num_param_groups = len(fp32_flat_groups[0])
285
+ merged_single_partition_of_fp32_groups = []
286
+ for i in range(num_param_groups):
287
+ merged_partitions = [sd[i] for sd in fp32_flat_groups]
288
+ full_single_fp32_vector = torch.cat(merged_partitions, 0)
289
+ merged_single_partition_of_fp32_groups.append(full_single_fp32_vector)
290
+ avail_numel = sum(
291
+ [full_single_fp32_vector.numel() for full_single_fp32_vector in merged_single_partition_of_fp32_groups])
292
+
293
+ if debug:
294
+ wanted_params = sum([len(shapes) for shapes in param_shapes])
295
+ wanted_numel = sum([sum(shape.numel() for shape in shapes.values()) for shapes in param_shapes])
296
+ # not asserting if there is a mismatch due to possible padding
297
+ print(f"Have {avail_numel} numels to process.")
298
+ print(f"Need {wanted_numel} numels in {wanted_params} params.")
299
+
300
+ # params
301
+ # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
302
+ # out-of-core computing solution
303
+ total_numel = 0
304
+ total_params = 0
305
+ for shapes, full_single_fp32_vector in zip(param_shapes, merged_single_partition_of_fp32_groups):
306
+ offset = 0
307
+ avail_numel = full_single_fp32_vector.numel()
308
+ for name, shape in shapes.items():
309
+
310
+ unpartitioned_numel = shape.numel()
311
+ total_numel += unpartitioned_numel
312
+ total_params += 1
313
+
314
+ if debug:
315
+ print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
316
+ state_dict[name] = full_single_fp32_vector.narrow(0, offset, unpartitioned_numel).view(shape)
317
+ offset += unpartitioned_numel
318
+
319
+ # Z2 started to align to 2*world_size to improve nccl performance. Therefore both offset and
320
+ # avail_numel can differ by anywhere between 0..2*world_size. Due to two unrelated complex
321
+ # paddings performed in the code it's almost impossible to predict the exact numbers w/o the
322
+ # live optimizer object, so we are checking that the numbers are within the right range
323
+ align_to = 2 * world_size
324
+
325
+ def zero2_align(x):
326
+ return align_to * math.ceil(x / align_to)
327
+
328
+ if debug:
329
+ print(f"original offset={offset}, avail_numel={avail_numel}")
330
+
331
+ offset = zero2_align(offset)
332
+ avail_numel = zero2_align(avail_numel)
333
+
334
+ if debug:
335
+ print(f"aligned offset={offset}, avail_numel={avail_numel}")
336
+
337
+ # Sanity check
338
+ if offset != avail_numel:
339
+ raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
340
+
341
+ print(f"Reconstructed fp32 state dict with {total_params} params {total_numel} elements")
342
+
343
+
344
+ def _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states):
345
+ state_dict = OrderedDict()
346
+
347
+ # buffers
348
+ buffers = zero_model_states[0].buffers
349
+ state_dict.update(buffers)
350
+ if debug:
351
+ print(f"added {len(buffers)} buffers")
352
+
353
+ _zero2_merge_frozen_params(state_dict, zero_model_states)
354
+
355
+ _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
356
+
357
+ # recover shared parameters
358
+ for pair in zero_model_states[0].shared_params:
359
+ state_dict[pair[0]] = state_dict[pair[1]]
360
+
361
+ return state_dict
362
+
363
+
364
+ def zero3_partitioned_param_info(unpartitioned_numel, world_size):
365
+ remainder = unpartitioned_numel % world_size
366
+ padding_numel = (world_size - remainder) if remainder else 0
367
+ partitioned_numel = math.ceil(unpartitioned_numel / world_size)
368
+ return partitioned_numel, padding_numel
369
+
370
+
371
+ def _zero3_merge_frozen_params(state_dict, world_size, zero_model_states):
372
+ if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
373
+ return
374
+
375
+ if debug:
376
+ for i in range(world_size):
377
+ num_elem = sum(s.numel() for s in zero_model_states[i].frozen_param_fragments.values())
378
+ print(f'rank {i}: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
379
+
380
+ frozen_param_shapes = zero_model_states[0].frozen_param_shapes
381
+ wanted_params = len(frozen_param_shapes)
382
+ wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
383
+ avail_numel = sum([p.numel() for p in zero_model_states[0].frozen_param_fragments.values()]) * world_size
384
+ print(f'Frozen params: Have {avail_numel} numels to process.')
385
+ print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
386
+
387
+ total_params = 0
388
+ total_numel = 0
389
+ for name, shape in tqdm(zero_model_states[0].frozen_param_shapes.items()):
390
+ total_params += 1
391
+ unpartitioned_numel = shape.numel()
392
+ total_numel += unpartitioned_numel
393
+
394
+ param_frags = tuple(model_state.frozen_param_fragments[name] for model_state in zero_model_states)
395
+ state_dict[name] = torch.cat(param_frags, 0).narrow(0, 0, unpartitioned_numel).view(shape)
396
+
397
+ partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
398
+
399
+ if debug:
400
+ print(
401
+ f"Frozen params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
402
+ )
403
+
404
+ print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
405
+
406
+
407
+ def _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
408
+ param_shapes = zero_model_states[0].param_shapes
409
+ avail_numel = fp32_flat_groups[0].numel() * world_size
410
+ # Reconstruction protocol: For zero3 we need to zip the partitions together at boundary of each
411
+ # param, re-consolidating each param, while dealing with padding if any
412
+
413
+ # merge list of dicts, preserving order
414
+ param_shapes = {k: v for d in param_shapes for k, v in d.items()}
415
+
416
+ if debug:
417
+ for i in range(world_size):
418
+ print(f"{FP32_FLAT_GROUPS}[{i}].shape={fp32_flat_groups[i].shape}")
419
+
420
+ wanted_params = len(param_shapes)
421
+ wanted_numel = sum(shape.numel() for shape in param_shapes.values())
422
+ # not asserting if there is a mismatch due to possible padding
423
+ avail_numel = fp32_flat_groups[0].numel() * world_size
424
+ print(f"Trainable params: Have {avail_numel} numels to process.")
425
+ print(f"Trainable params: Need {wanted_numel} numels in {wanted_params} params.")
426
+
427
+ # params
428
+ # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
429
+ # out-of-core computing solution
430
+ offset = 0
431
+ total_numel = 0
432
+ total_params = 0
433
+ for name, shape in tqdm(param_shapes.items()):
434
+
435
+ unpartitioned_numel = shape.numel()
436
+ total_numel += unpartitioned_numel
437
+ total_params += 1
438
+
439
+ partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
440
+
441
+ if debug:
442
+ print(
443
+ f"Trainable params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
444
+ )
445
+
446
+ # XXX: memory usage doubles here
447
+ state_dict[name] = torch.cat(
448
+ tuple(fp32_flat_groups[i].narrow(0, offset, partitioned_numel) for i in range(world_size)),
449
+ 0).narrow(0, 0, unpartitioned_numel).view(shape)
450
+ offset += partitioned_numel
451
+
452
+ offset *= world_size
453
+
454
+ # Sanity check
455
+ if offset != avail_numel:
456
+ raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
457
+
458
+ print(f"Reconstructed Trainable fp32 state dict with {total_params} params {total_numel} elements")
459
+
460
+
461
+ def _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states):
462
+ state_dict = OrderedDict()
463
+
464
+ # buffers
465
+ buffers = zero_model_states[0].buffers
466
+ state_dict.update(buffers)
467
+ if debug:
468
+ print(f"added {len(buffers)} buffers")
469
+
470
+ _zero3_merge_frozen_params(state_dict, world_size, zero_model_states)
471
+
472
+ _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
473
+
474
+ # recover shared parameters
475
+ for pair in zero_model_states[0].shared_params:
476
+ state_dict[pair[0]] = state_dict[pair[1]]
477
+
478
+ return state_dict
479
+
480
+
481
+ def get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag=None):
482
+ """
483
+ Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated state_dict that can be loaded with
484
+ ``load_state_dict()`` and used for training without DeepSpeed or shared with others, for example
485
+ via a model hub.
486
+
487
+ Args:
488
+ - ``checkpoint_dir``: path to the desired checkpoint folder
489
+ - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in 'latest' file. e.g., ``global_step14``
490
+
491
+ Returns:
492
+ - pytorch ``state_dict``
493
+
494
+ Note: this approach may not work if your application doesn't have sufficient free CPU memory and
495
+ you may need to use the offline approach using the ``zero_to_fp32.py`` script that is saved with
496
+ the checkpoint.
497
+
498
+ A typical usage might be ::
499
+
500
+ from deepspeed.utils.zero_to_fp32 import get_fp32_state_dict_from_zero_checkpoint
501
+ # do the training and checkpoint saving
502
+ state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir) # already on cpu
503
+ model = model.cpu() # move to cpu
504
+ model.load_state_dict(state_dict)
505
+ # submit to model hub or save the model to share with others
506
+
507
+ In this example the ``model`` will no longer be usable in the deepspeed context of the same
508
+ application. i.e. you will need to re-initialize the deepspeed engine, since
509
+ ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
510
+
511
+ If you want it all done for you, use ``load_state_dict_from_zero_checkpoint`` instead.
512
+
513
+ """
514
+ if tag is None:
515
+ latest_path = os.path.join(checkpoint_dir, 'latest')
516
+ if os.path.isfile(latest_path):
517
+ with open(latest_path, 'r') as fd:
518
+ tag = fd.read().strip()
519
+ else:
520
+ raise ValueError(f"Unable to find 'latest' file at {latest_path}")
521
+
522
+ ds_checkpoint_dir = os.path.join(checkpoint_dir, tag)
523
+
524
+ if not os.path.isdir(ds_checkpoint_dir):
525
+ raise FileNotFoundError(f"Directory '{ds_checkpoint_dir}' doesn't exist")
526
+
527
+ return _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir)
528
+
529
+
530
+ def convert_zero_checkpoint_to_fp32_state_dict(checkpoint_dir, output_file, tag=None):
531
+ """
532
+ Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict`` file that can be
533
+ loaded with ``torch.load(file)`` + ``load_state_dict()`` and used for training without DeepSpeed.
534
+
535
+ Args:
536
+ - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
537
+ - ``output_file``: path to the pytorch fp32 state_dict output file (e.g. path/pytorch_model.bin)
538
+ - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
539
+ """
540
+
541
+ state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag)
542
+ print(f"Saving fp32 state dict to {output_file}")
543
+ torch.save(state_dict, output_file)
544
+
545
+
546
+ def load_state_dict_from_zero_checkpoint(model, checkpoint_dir, tag=None):
547
+ """
548
+ 1. Put the provided model to cpu
549
+ 2. Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict``
550
+ 3. Load it into the provided model
551
+
552
+ Args:
553
+ - ``model``: the model object to update
554
+ - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
555
+ - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
556
+
557
+ Returns:
558
+ - ``model`: modified model
559
+
560
+ Make sure you have plenty of CPU memory available before you call this function. If you don't
561
+ have enough use the ``zero_to_fp32.py`` utility to do the conversion. You will find it
562
+ conveniently placed for you in the checkpoint folder.
563
+
564
+ A typical usage might be ::
565
+
566
+ from deepspeed.utils.zero_to_fp32 import load_state_dict_from_zero_checkpoint
567
+ model = load_state_dict_from_zero_checkpoint(trainer.model, checkpoint_dir)
568
+ # submit to model hub or save the model to share with others
569
+
570
+ Note, that once this was run, the ``model`` will no longer be usable in the deepspeed context
571
+ of the same application. i.e. you will need to re-initialize the deepspeed engine, since
572
+ ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
573
+
574
+ """
575
+ logger.info(f"Extracting fp32 weights")
576
+ state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag)
577
+
578
+ logger.info(f"Overwriting model with fp32 weights")
579
+ model = model.cpu()
580
+ model.load_state_dict(state_dict, strict=False)
581
+
582
+ return model
583
+
584
+
585
+ if __name__ == "__main__":
586
+
587
+ parser = argparse.ArgumentParser()
588
+ parser.add_argument("checkpoint_dir",
589
+ type=str,
590
+ help="path to the desired checkpoint folder, e.g., path/checkpoint-12")
591
+ parser.add_argument(
592
+ "output_file",
593
+ type=str,
594
+ help="path to the pytorch fp32 state_dict output file (e.g. path/checkpoint-12/pytorch_model.bin)")
595
+ parser.add_argument("-d", "--debug", action='store_true', help="enable debug")
596
+ args = parser.parse_args()
597
+
598
+ debug = args.debug
599
+
600
+ convert_zero_checkpoint_to_fp32_state_dict(args.checkpoint_dir, args.output_file)
data_finetune.py ADDED
@@ -0,0 +1,852 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import copy
2
+ import random
3
+ import argparse
4
+ import os
5
+ import torch
6
+ import torch.nn as nn
7
+ from torch.utils.data import Dataset
8
+ from tqdm import tqdm
9
+ from collections import defaultdict
10
+ import torch.distributed as dist
11
+ import logging
12
+ import re
13
+ import pdb
14
+ import json
15
+ from prompt_finetune import sft_prompt, all_prompt
16
+ import numpy as np
17
+
18
+
19
+ class BaseDataset(Dataset):
20
+
21
+ def __init__(self, args):
22
+ super().__init__()
23
+
24
+ self.args = args
25
+ self.dataset = args.dataset
26
+ self.data_path = os.path.join(args.data_path, self.dataset)
27
+
28
+ self.max_his_len = args.max_his_len
29
+ self.his_sep = args.his_sep
30
+ self.index_file = args.index_file
31
+ self.add_prefix = args.add_prefix
32
+
33
+ self.new_tokens = None
34
+ self.allowed_tokens = None
35
+ self.all_items = None
36
+
37
+
38
+ def _load_data(self):
39
+
40
+ with open(os.path.join(self.data_path, self.dataset + self.index_file), 'r') as f:
41
+ self.indices = json.load(f)
42
+
43
+ def get_new_tokens(self):
44
+
45
+ if self.new_tokens is not None:
46
+ return self.new_tokens
47
+
48
+ self.new_tokens = set()
49
+ for index in self.indices.values():
50
+ for token in index:
51
+ self.new_tokens.add(token)
52
+ self.new_tokens = sorted(list(self.new_tokens))
53
+
54
+ return self.new_tokens
55
+
56
+ def get_all_items(self):
57
+
58
+ if self.all_items is not None:
59
+ return self.all_items
60
+
61
+ self.all_items = set()
62
+ for index in self.indices.values():
63
+ self.all_items.add("".join(index))
64
+
65
+ return self.all_items
66
+
67
+ def get_prefix_allowed_tokens_fn(self, tokenizer):
68
+
69
+
70
+ if self.allowed_tokens is None:
71
+ self.allowed_tokens = {}
72
+ for index in self.indices.values():
73
+ for i, token in enumerate(index):
74
+ token_id = tokenizer(token)["input_ids"][1]
75
+ if i not in self.allowed_tokens.keys():
76
+ self.allowed_tokens[i] = set()
77
+ self.allowed_tokens[i].add(token_id)
78
+ self.allowed_tokens[len(self.allowed_tokens.keys())] = set([tokenizer.eos_token_id])
79
+ sep = tokenizer("Response:")["input_ids"][1:]
80
+
81
+ def prefix_allowed_tokens_fn(batch_id, sentence):
82
+ sentence = sentence.tolist()
83
+ reversed_sent = sentence[::-1]
84
+ for i in range(len(reversed_sent)):
85
+ if reversed_sent[i:i + len(sep)] == sep[::-1]:
86
+ # print(list(self.allowed_tokens[i]))
87
+ return list(self.allowed_tokens[i])
88
+
89
+ return prefix_allowed_tokens_fn
90
+
91
+ def _process_data(self):
92
+
93
+ raise NotImplementedError
94
+
95
+
96
+
97
+ class SeqRecFinetune(BaseDataset):
98
+
99
+ def __init__(self, args, mode="train",
100
+ prompt_sample_num=1, prompt_id=0, sample_num=-1):
101
+ super().__init__(args)
102
+
103
+ self.mode = mode
104
+ self.prompt_sample_num = prompt_sample_num
105
+ self.prompt_id = prompt_id
106
+ self.sample_num = sample_num
107
+
108
+ self.prompts = all_prompt["seqrec"]
109
+
110
+
111
+ # load data
112
+ self._load_data()
113
+ self._remap_items()
114
+
115
+ # load data
116
+ if self.mode == 'train':
117
+ self.inter_data = self._process_train_data()
118
+ elif self.mode == 'valid':
119
+ self.sample_valid = args.sample_valid
120
+ self.valid_prompt_id = args.valid_prompt_id
121
+ self.inter_data = self._process_valid_data()
122
+ self._construct_valid_text()
123
+ elif self.mode == 'test':
124
+ self.inter_data = self._process_test_data()
125
+ else:
126
+ raise NotImplementedError
127
+
128
+
129
+
130
+ def _load_data(self):
131
+
132
+ with open(os.path.join(self.data_path, self.dataset + ".inter.json"), 'r') as f:
133
+ self.inters = json.load(f)
134
+ with open(self.index_file, 'r') as f:
135
+ self.indices = json.load(f)
136
+
137
+
138
+ def _remap_items(self):
139
+
140
+ self.remapped_inters = dict()
141
+ for uid, items in self.inters.items():
142
+ new_items = ["".join(self.indices[str(i)]) for i in items]
143
+ self.remapped_inters[uid] = new_items
144
+
145
+
146
+ def _process_train_data(self):
147
+
148
+ inter_data = []
149
+ for uid in self.remapped_inters:
150
+ items = self.remapped_inters[uid][:-2]
151
+ for i in range(1, len(items)):
152
+ one_data = dict()
153
+ # one_data["user"] = uid
154
+ one_data["item"] = items[i]
155
+ history = items[:i]
156
+ if self.max_his_len > 0:
157
+ history = history[-self.max_his_len:]
158
+ if self.add_prefix:
159
+ history = [str(k+1) + ". " + item_idx for k, item_idx in enumerate(history)]
160
+ one_data["inters"] = self.his_sep.join(history)
161
+ inter_data.append(one_data)
162
+
163
+ return inter_data
164
+
165
+ def _process_valid_data(self):
166
+
167
+ inter_data = []
168
+ for uid in self.remapped_inters:
169
+ items = self.remapped_inters[uid]
170
+ one_data = dict()
171
+ # one_data["user"] = uid
172
+ one_data["item"] = items[-2]
173
+ history = items[:-2]
174
+ if self.max_his_len > 0:
175
+ history = history[-self.max_his_len:]
176
+ if self.add_prefix:
177
+ history = [str(k + 1) + ". " + item_idx for k, item_idx in enumerate(history)]
178
+ one_data["inters"] = self.his_sep.join(history)
179
+ inter_data.append(one_data)
180
+
181
+ return inter_data
182
+
183
+ def _process_test_data(self):
184
+
185
+ inter_data = []
186
+ for uid in self.remapped_inters:
187
+ items = self.remapped_inters[uid]
188
+ one_data = dict()
189
+ # one_data["user"] = uid
190
+ one_data["item"] = items[-1]
191
+ history = items[:-1]
192
+ if self.max_his_len > 0:
193
+ history = history[-self.max_his_len:]
194
+ if self.add_prefix:
195
+ history = [str(k + 1) + ". " + item_idx for k, item_idx in enumerate(history)]
196
+ one_data["inters"] = self.his_sep.join(history)
197
+ inter_data.append(one_data)
198
+
199
+ if self.sample_num > 0:
200
+ all_inter_idx = range(len(inter_data))
201
+ sample_idx = np.random.choice(all_inter_idx, self.sample_num, replace=False)
202
+ inter_data = np.array(inter_data)[sample_idx].tolist()
203
+
204
+ return inter_data
205
+
206
+ def set_prompt(self, prompt_id):
207
+
208
+ self.prompt_id = prompt_id
209
+
210
+ def __len__(self):
211
+ if self.mode == 'train':
212
+ return len(self.inter_data) * self.prompt_sample_num
213
+ elif self.mode == 'valid':
214
+ return len(self.valid_text_data)
215
+ elif self.mode == 'test':
216
+ return len(self.inter_data)
217
+ else:
218
+ raise NotImplementedError
219
+
220
+ def _construct_valid_text(self):
221
+ self.valid_text_data = []
222
+ if self.sample_valid:
223
+ all_prompt_ids = range(len(self.prompts))
224
+ for i in range(len(self.inter_data)):
225
+ d = self.inter_data[i]
226
+ prompt_ids = np.random.choice(all_prompt_ids, self.prompt_sample_num, replace=False)
227
+ for prompt_id in prompt_ids:
228
+ prompt = self.prompts[prompt_id]
229
+ input, output = self._get_text_data(d, prompt)
230
+ self.valid_text_data.append({"input_ids": input, "labels": output})
231
+ else:
232
+ self.prompt_sample_num = 1
233
+ prompt = self.prompts[self.valid_prompt_id]
234
+ for i in range(len(self.inter_data)):
235
+ d = self.inter_data[i]
236
+ input, output = self._get_text_data(d, prompt)
237
+ self.valid_text_data.append({"input_ids": input, "labels": output})
238
+
239
+ def _get_text_data(self, data, prompt):
240
+
241
+ instruction = prompt["instruction"].format(**data)
242
+ response = prompt["response"].format(**data)
243
+
244
+ input = sft_prompt.format(instruction = instruction, response = "")
245
+ output = sft_prompt.format(instruction = instruction, response = response)
246
+
247
+ if self.mode == 'test':
248
+ return input, response
249
+
250
+ return input, output
251
+
252
+ def __getitem__(self, index):
253
+
254
+ if self.mode == 'valid':
255
+ return self.valid_text_data[index]
256
+
257
+ idx = index // self.prompt_sample_num
258
+ d = self.inter_data[idx]
259
+ # print(index, idx)
260
+
261
+ if self.mode == 'train':
262
+ prompt_id = random.randint(0, len(self.prompts) - 1)
263
+ elif self.mode == 'test':
264
+ prompt_id = self.prompt_id
265
+
266
+ prompt = self.prompts[prompt_id]
267
+
268
+ input, output = self._get_text_data(d, prompt)
269
+
270
+ # print({"input": input, "output": output})
271
+
272
+ return dict(input_ids=input, labels=output)
273
+
274
+
275
+ class FusionSeqRecFinetune(BaseDataset):
276
+
277
+ def __init__(self, args, mode="train",
278
+ prompt_sample_num=1, prompt_id=0, sample_num=-1):
279
+ super().__init__(args)
280
+
281
+ self.mode = mode
282
+ self.prompt_sample_num = prompt_sample_num
283
+ self.prompt_id = prompt_id
284
+ self.sample_num = sample_num
285
+
286
+ self.prompts = all_prompt["fusionseqrec"]
287
+
288
+ # load data
289
+ self._load_data()
290
+ # self._remap_items()
291
+
292
+ # load data
293
+ if self.mode == 'train':
294
+ self.inter_data = self._process_train_data()
295
+ elif self.mode == 'valid':
296
+ self.sample_valid = args.sample_valid
297
+ self.valid_prompt_id = args.valid_prompt_id
298
+ self.inter_data = self._process_valid_data()
299
+ self._construct_valid_text()
300
+ elif self.mode == 'test':
301
+ self.inter_data = self._process_test_data()
302
+ else:
303
+ raise NotImplementedError
304
+
305
+
306
+ def _load_data(self):
307
+
308
+ with open(os.path.join(self.data_path, self.dataset + ".inter.json"), 'r') as f:
309
+ self.inters = json.load(f)
310
+ with open(self.index_file, 'r') as f:
311
+ self.indices = json.load(f)
312
+ # with open(os.path.join(self.data_path, self.dataset + self.index_file), 'r') as f:
313
+ # self.indices = json.load(f)
314
+ with open(os.path.join(self.data_path, self.dataset + ".item.json"), 'r') as f:
315
+ self.item_feat = json.load(f)
316
+
317
+ def _process_train_data(self):
318
+
319
+ inter_data = []
320
+ for uid in self.inters:
321
+ items = self.inters[uid][:-2]
322
+ for i in range(1, len(items)):
323
+ one_data = dict()
324
+ # one_data["user"] = uid
325
+ one_data["item"] = "".join(self.indices[str(items[i])])
326
+ one_data["title"] = self.item_feat[str(items[i])]["title"].strip().strip(".!?,;:`")
327
+ one_data["description"] = self.item_feat[str(items[i])]["description"]
328
+ history = items[:i]
329
+ if self.max_his_len > 0:
330
+ history = history[-self.max_his_len:]
331
+ inters = ["".join(self.indices[str(j)]) for j in history]
332
+ inter_titles = ["\"" + self.item_feat[str(j)]["title"].strip().strip(".!?,;:`") + "\"" for j in history]
333
+
334
+
335
+ if self.add_prefix:
336
+ inters = [str(k + 1) + ". " + item_idx for k, item_idx in enumerate(inters)]
337
+ inter_titles = [str(k + 1) + ". " + item_title for k, item_title in enumerate(inter_titles)]
338
+
339
+ one_data["inters"] = self.his_sep.join(inters)
340
+ one_data["inter_titles"] = self.his_sep.join(inter_titles)
341
+ inter_data.append(one_data)
342
+
343
+ if self.sample_num > 0:
344
+ all_inter_idx = range(len(inter_data))
345
+ sample_idx = np.random.choice(all_inter_idx, self.sample_num, replace=False)
346
+ inter_data = np.array(inter_data)[sample_idx].tolist()
347
+
348
+ return inter_data
349
+
350
+ def _process_valid_data(self):
351
+
352
+ inter_data = []
353
+ for uid in self.inters:
354
+ items = self.inters[uid]
355
+ one_data = dict()
356
+ one_data["item"] = "".join(self.indices[str(items[-2])])
357
+ one_data["title"] = self.item_feat[str(items[-2])]["title"].strip().strip(".!?,;:`")
358
+ one_data["description"] = self.item_feat[str(items[-2])]["description"]
359
+
360
+
361
+ history = items[:-2]
362
+ if self.max_his_len > 0:
363
+ history = history[-self.max_his_len:]
364
+ inters = ["".join(self.indices[str(j)]) for j in history]
365
+ inter_titles = ["\"" + self.item_feat[str(j)]["title"].strip().strip(".!?,;:`") + "\"" for j in history]
366
+
367
+ if self.add_prefix:
368
+ inters = [str(k + 1) + ". " + item_idx for k, item_idx in enumerate(inters)]
369
+ inter_titles = [str(k + 1) + ". " + item_title for k, item_title in enumerate(inter_titles)]
370
+
371
+ one_data["inters"] = self.his_sep.join(inters)
372
+ one_data["inter_titles"] = self.his_sep.join(inter_titles)
373
+ inter_data.append(one_data)
374
+
375
+ if self.sample_num > 0:
376
+ all_inter_idx = range(len(inter_data))
377
+ sample_idx = np.random.choice(all_inter_idx, self.sample_num, replace=False)
378
+ inter_data = np.array(inter_data)[sample_idx].tolist()
379
+
380
+ return inter_data
381
+
382
+ def _process_test_data(self):
383
+
384
+ inter_data = []
385
+ for uid in self.inters:
386
+ items = self.inters[uid]
387
+ one_data = dict()
388
+ one_data["item"] = "".join(self.indices[str(items[-1])])
389
+ one_data["title"] = self.item_feat[str(items[-1])]["title"].strip().strip(".!?,;:`")
390
+ one_data["description"] = self.item_feat[str(items[-1])]["description"]
391
+
392
+ history = items[:-1]
393
+ if self.max_his_len > 0:
394
+ history = history[-self.max_his_len:]
395
+ inters = ["".join(self.indices[str(j)]) for j in history]
396
+ inter_titles = ["\"" + self.item_feat[str(j)]["title"].strip().strip(".!?,;:`") + "\"" for j in history]
397
+
398
+ if self.add_prefix:
399
+ inters = [str(k + 1) + ". " + item_idx for k, item_idx in enumerate(inters)]
400
+ inter_titles = [str(k + 1) + ". " + item_title for k, item_title in enumerate(inter_titles)]
401
+
402
+ one_data["inters"] = self.his_sep.join(inters)
403
+ one_data["inter_titles"] = self.his_sep.join(inter_titles)
404
+ inter_data.append(one_data)
405
+
406
+ if self.sample_num > 0:
407
+ all_inter_idx = range(len(inter_data))
408
+ sample_idx = np.random.choice(all_inter_idx, self.sample_num, replace=False)
409
+ inter_data = np.array(inter_data)[sample_idx].tolist()
410
+
411
+ return inter_data
412
+
413
+ def set_prompt(self, prompt_id):
414
+
415
+ self.prompt_id = prompt_id
416
+
417
+ def __len__(self):
418
+ if self.mode == 'train':
419
+ return len(self.inter_data) * self.prompt_sample_num
420
+ elif self.mode == 'valid':
421
+ return len(self.valid_text_data)
422
+ elif self.mode == 'test':
423
+ return len(self.inter_data)
424
+ else:
425
+ raise NotImplementedError
426
+
427
+ def _construct_valid_text(self):
428
+ self.valid_text_data = []
429
+ if self.sample_valid:
430
+ all_prompt_ids = range(len(self.prompts))
431
+ for i in range(len(self.inter_data)):
432
+ d = self.inter_data[i]
433
+ prompt_ids = np.random.choice(all_prompt_ids, self.prompt_sample_num, replace=False)
434
+ for prompt_id in prompt_ids:
435
+ prompt = self.prompts[prompt_id]
436
+ input, output = self._get_text_data(d, prompt)
437
+ self.valid_text_data.append({"input_ids": input, "labels": output})
438
+ else:
439
+ self.prompt_sample_num = 1
440
+ prompt = self.prompts[self.valid_prompt_id]
441
+ for i in range(len(self.inter_data)):
442
+ d = self.inter_data[i]
443
+ input, output = self._get_text_data(d, prompt)
444
+ self.valid_text_data.append({"input_ids": input, "labels": output})
445
+
446
+ def _get_text_data(self, data, prompt):
447
+
448
+ instruction = prompt["instruction"].format(**data)
449
+ response = prompt["response"].format(**data)
450
+
451
+ input = sft_prompt.format(instruction=instruction, response="")
452
+ output = sft_prompt.format(instruction=instruction, response=response)
453
+
454
+ if self.mode == 'test':
455
+ return input, response
456
+
457
+ return input, output
458
+
459
+ def __getitem__(self, index):
460
+
461
+ if self.mode == 'valid':
462
+ return self.valid_text_data[index]
463
+
464
+ idx = index // self.prompt_sample_num
465
+ d = self.inter_data[idx]
466
+
467
+ if self.mode == 'train':
468
+ prompt_id = random.randint(0, len(self.prompts) - 1)
469
+ elif self.mode == 'test':
470
+ prompt_id = self.prompt_id
471
+
472
+ prompt = self.prompts[prompt_id]
473
+
474
+ input, output = self._get_text_data(d, prompt)
475
+
476
+
477
+ return dict(input_ids=input, labels=output)
478
+
479
+
480
+ class ItemFeatFinetune(BaseDataset):
481
+
482
+ def __init__(self, args, task="item2index", prompt_sample_num=1, sample_num=-1):
483
+ super().__init__(args)
484
+
485
+ self.task = task.lower()
486
+ self.prompt_sample_num = prompt_sample_num
487
+ self.sample_num = sample_num
488
+
489
+ self.prompts = all_prompt[self.task]
490
+
491
+ # load data
492
+ self._load_data()
493
+ self.feat_data = self._process_data()
494
+
495
+
496
+
497
+ def _load_data(self):
498
+
499
+ # with open(os.path.join(self.data_path, self.dataset + self.index_file), 'r') as f:
500
+ # self.indices = json.load(f)
501
+ with open(self.index_file, 'r') as f:
502
+ self.indices = json.load(f)
503
+ with open(os.path.join(self.data_path, self.dataset + ".item.json"), 'r') as f:
504
+ self.item_feat = json.load(f)
505
+
506
+
507
+ def _process_data(self):
508
+
509
+ feat_data = []
510
+ for iid in self.item_feat:
511
+ feat = self.item_feat[iid]
512
+ index = "".join(self.indices[iid])
513
+ feat["item"] = index
514
+ feat["title"] = feat["title"].strip().strip(".!?,;:`")
515
+ feat_data.append(feat)
516
+
517
+ if self.sample_num > 0:
518
+ all_idx = range(len(feat_data))
519
+ sample_idx = np.random.choice(all_idx, self.sample_num, replace=False)
520
+
521
+ feat_data = np.array(feat_data)[sample_idx].tolist()
522
+
523
+ return feat_data
524
+
525
+
526
+ def __len__(self):
527
+ return len(self.feat_data) * self.prompt_sample_num
528
+
529
+ def _get_text_data(self, data, prompt):
530
+
531
+ instruction = prompt["instruction"].format(**data)
532
+ response = prompt["response"].format(**data)
533
+
534
+ input = sft_prompt.format(instruction = instruction, response = "")
535
+ output = sft_prompt.format(instruction = instruction, response = response)
536
+
537
+ return input, output
538
+
539
+ def __getitem__(self, index):
540
+
541
+ idx = index // self.prompt_sample_num
542
+ d = self.feat_data[idx]
543
+
544
+ prompt_id = random.randint(0, len(self.prompts) - 1)
545
+
546
+ prompt = self.prompts[prompt_id]
547
+
548
+ input, output = self._get_text_data(d, prompt)
549
+
550
+ return dict(input_ids=input, labels=output)
551
+
552
+
553
+ class ItemSearchFinetune(BaseDataset):
554
+
555
+ def __init__(self, args, mode="train",
556
+ prompt_sample_num=1, prompt_id=0, sample_num=-1):
557
+ super().__init__(args)
558
+
559
+ self.mode = mode
560
+ self.prompt_sample_num = prompt_sample_num
561
+ self.prompt_id = prompt_id
562
+ self.sample_num = sample_num
563
+
564
+ self.prompts = all_prompt["itemsearch"]
565
+
566
+ # load data
567
+ self._load_data()
568
+ self.search_data = self._process_data()
569
+
570
+
571
+
572
+ def _load_data(self):
573
+
574
+ # with open(os.path.join(self.data_path, self.dataset + self.index_file), 'r') as f:
575
+ # self.indices = json.load(f)
576
+ with open(self.index_file, 'r') as f:
577
+ self.indices = json.load(f)
578
+ with open(os.path.join(self.data_path, self.dataset + ".user.json"), 'r') as f:
579
+ self.user_info = json.load(f)
580
+
581
+
582
+ def _process_data(self):
583
+
584
+ search_data = []
585
+ user_explicit_preference = self.user_info["user_explicit_preference"]
586
+ user_vague_intention = self.user_info["user_vague_intention"]
587
+ if self.mode == 'train':
588
+ user_vague_intention = user_vague_intention["train"]
589
+ elif self.mode == 'test':
590
+ user_vague_intention = user_vague_intention["test"]
591
+ else:
592
+ raise NotImplementedError
593
+
594
+ for uid in user_explicit_preference.keys():
595
+ one_data = {}
596
+ user_ep = user_explicit_preference[uid]
597
+ user_vi = user_vague_intention[uid]["querys"]
598
+ one_data["explicit_preferences"] = user_ep
599
+ one_data["user_related_intention"] = user_vi[0]
600
+ one_data["item_related_intention"] = user_vi[1]
601
+
602
+ iid = user_vague_intention[uid]["item"]
603
+ inters = user_vague_intention[uid]["inters"]
604
+
605
+ index = "".join(self.indices[str(iid)])
606
+ one_data["item"] = index
607
+
608
+ if self.max_his_len > 0:
609
+ inters = inters[-self.max_his_len:]
610
+ inters = ["".join(self.indices[str(i)]) for i in inters]
611
+ if self.add_prefix:
612
+ inters = [str(k + 1) + ". " + item_idx for k, item_idx in enumerate(inters)]
613
+
614
+ one_data["inters"] = self.his_sep.join(inters)
615
+
616
+ search_data.append(one_data)
617
+
618
+ if self.sample_num > 0:
619
+ all_idx = range(len(search_data))
620
+ sample_idx = np.random.choice(all_idx, self.sample_num, replace=False)
621
+
622
+ search_data = np.array(search_data)[sample_idx].tolist()
623
+
624
+ return search_data
625
+
626
+ def set_prompt(self, prompt_id):
627
+ self.prompt_id = prompt_id
628
+
629
+ def __len__(self):
630
+ if self.mode == 'train':
631
+ return len(self.search_data) * self.prompt_sample_num
632
+ elif self.mode == 'test':
633
+ return len(self.search_data)
634
+ else:
635
+ return len(self.search_data)
636
+
637
+
638
+ def _get_text_data(self, data, prompt):
639
+
640
+ instruction = prompt["instruction"].format(**data)
641
+ response = prompt["response"].format(**data)
642
+
643
+ input = sft_prompt.format(instruction = instruction, response = "")
644
+ output = sft_prompt.format(instruction = instruction, response = response)
645
+
646
+ if self.mode == 'test':
647
+ return input, response
648
+
649
+ return input, output
650
+
651
+ def __getitem__(self, index):
652
+
653
+ idx = index // self.prompt_sample_num
654
+
655
+ d = self.search_data[idx]
656
+ if self.mode == 'train':
657
+ prompt_id = random.randint(0, len(self.prompts) - 1)
658
+ elif self.mode == 'test':
659
+ prompt_id = self.prompt_id
660
+
661
+ prompt = self.prompts[prompt_id]
662
+
663
+ d["explicit_preference"] = copy.deepcopy(random.choice(d["explicit_preferences"]))
664
+ all_querys = [d["user_related_intention"], d["item_related_intention"]]
665
+ d["query"] = random.choice(all_querys)
666
+
667
+ input, output = self._get_text_data(d, prompt)
668
+
669
+ return dict(input_ids=input, labels=output)
670
+
671
+
672
+
673
+ class PreferenceObtainFinetune(BaseDataset):
674
+
675
+ def __init__(self, args, prompt_sample_num=1, sample_num=-1):
676
+ super().__init__(args)
677
+
678
+ self.prompt_sample_num = prompt_sample_num
679
+ self.sample_num = sample_num
680
+
681
+ self.prompts = all_prompt["preferenceobtain"]
682
+
683
+ # load data
684
+ self._load_data()
685
+ self._remap_items()
686
+
687
+ self.preference_data = self._process_data()
688
+
689
+
690
+
691
+ def _load_data(self):
692
+
693
+ with open(os.path.join(self.data_path, self.dataset + ".user.json"), 'r') as f:
694
+ self.user_info = json.load(f)
695
+ with open(os.path.join(self.data_path, self.dataset + ".inter.json"), 'r') as f:
696
+ self.inters = json.load(f)
697
+ # with open(os.path.join(self.data_path, self.dataset + self.index_file), 'r') as f:
698
+ # self.indices = json.load(f)
699
+ with open(self.index_file, 'r') as f:
700
+ self.indices = json.load(f)
701
+
702
+
703
+ def _remap_items(self):
704
+
705
+ self.remapped_inters = dict()
706
+ for uid, items in self.inters.items():
707
+ new_items = ["".join(self.indices[str(i)]) for i in items]
708
+ self.remapped_inters[uid] = new_items
709
+
710
+ def _process_data(self):
711
+
712
+ preference_data = []
713
+ user_explicit_preference = self.user_info["user_explicit_preference"]
714
+
715
+ for uid in user_explicit_preference.keys():
716
+ one_data = {}
717
+ inters = self.remapped_inters[uid][:-3]
718
+ user_ep = user_explicit_preference[uid]
719
+
720
+ if self.max_his_len > 0:
721
+ inters = inters[-self.max_his_len:]
722
+ if self.add_prefix:
723
+ inters = [str(k + 1) + ". " + item_idx for k, item_idx in enumerate(inters)]
724
+
725
+ one_data["explicit_preferences"] = user_ep
726
+ one_data["inters"] = self.his_sep.join(inters)
727
+
728
+ preference_data.append(one_data)
729
+
730
+ if self.sample_num > 0:
731
+ all_idx = range(len(preference_data))
732
+ sample_idx = np.random.choice(all_idx, self.sample_num, replace=False)
733
+
734
+ preference_data = np.array(preference_data)[sample_idx].tolist()
735
+
736
+ return preference_data
737
+
738
+ def set_prompt(self, prompt_id):
739
+ self.prompt_id = prompt_id
740
+
741
+ def __len__(self):
742
+ return len(self.preference_data) * self.prompt_sample_num
743
+
744
+
745
+ def _get_text_data(self, data, prompt):
746
+
747
+ instruction = prompt["instruction"].format(**data)
748
+ response = prompt["response"].format(**data)
749
+
750
+ input = sft_prompt.format(instruction = instruction, response = "")
751
+ output = sft_prompt.format(instruction = instruction, response = response)
752
+
753
+ return input, output
754
+
755
+ def __getitem__(self, index):
756
+
757
+ idx = index // self.prompt_sample_num
758
+
759
+ d = self.preference_data[idx]
760
+ prompt_id = random.randint(0, len(self.prompts) - 1)
761
+
762
+ prompt = self.prompts[prompt_id]
763
+
764
+ d["explicit_preference"] = copy.deepcopy(random.choice(d["explicit_preferences"]))
765
+
766
+ input, output = self._get_text_data(d, prompt)
767
+
768
+ return dict(input_ids=input, labels=output)
769
+
770
+
771
+
772
+
773
+
774
+ class SeqRecTestDataset(BaseDataset):
775
+
776
+ def __init__(self, args, prompt_id=0, sample_num=-1):
777
+ super().__init__(args)
778
+
779
+ self.prompt_id = prompt_id
780
+ self.sample_num = sample_num
781
+
782
+ self.prompt = all_prompt["seqrec"][self.prompt_id]
783
+
784
+ # load data
785
+ self._load_data()
786
+ self._remap_items()
787
+
788
+ self.inter_data = self._process_test_data()
789
+
790
+ def _load_data(self):
791
+
792
+ with open(os.path.join(self.data_path, self.dataset + ".inter.json"), 'r') as f:
793
+ self.inters = json.load(f)
794
+ with open(os.path.join(self.data_path, self.dataset + self.index_file), 'r') as f:
795
+ self.indices = json.load(f)
796
+
797
+
798
+ def _remap_items(self):
799
+
800
+ self.remapped_inters = dict()
801
+ for uid, items in self.inters.items():
802
+ new_items = ["".join(self.indices[str(i)]) for i in items]
803
+ self.remapped_inters[uid] = new_items
804
+
805
+ def _process_test_data(self):
806
+
807
+ inter_data = []
808
+ for uid in self.remapped_inters:
809
+ items = self.remapped_inters[uid]
810
+ one_data = dict()
811
+ # one_data["user"] = uid
812
+ one_data["item"] = items[-1]
813
+ history = items[:-1]
814
+ if self.max_his_len > 0:
815
+ history = history[-self.max_his_len:]
816
+ if self.add_prefix:
817
+ history = [str(k + 1) + ". " + item_idx for k, item_idx in enumerate(history)]
818
+ one_data["inters"] = self.his_sep.join(history)
819
+ inter_data.append(one_data)
820
+
821
+ if self.sample_num > 0:
822
+ all_inter_idx = range(len(inter_data))
823
+ sample_idx = np.random.choice(all_inter_idx, self.sample_num, replace=False)
824
+
825
+ inter_data = np.array(inter_data)[sample_idx].tolist()
826
+
827
+ return inter_data
828
+
829
+ def set_prompt(self, prompt_id):
830
+ self.prompt_id = prompt_id
831
+
832
+ self.prompt = all_prompt["seqrec"][self.prompt_id]
833
+
834
+ def __len__(self):
835
+
836
+ return len(self.inter_data)
837
+
838
+ def _get_text_data(self, data, prompt):
839
+
840
+ instruction = prompt["instruction"].format(**data)
841
+ response = prompt["response"].format(**data)
842
+
843
+ input = sft_prompt.format(instruction=instruction, response="")
844
+
845
+ return input, response
846
+
847
+ def __getitem__(self, index):
848
+
849
+ d = self.inter_data[index]
850
+ input, target = self._get_text_data(d, self.prompt)
851
+
852
+ return dict(input_ids=input, labels=target)
data_process/amazon18_data_process.py ADDED
@@ -0,0 +1,299 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import argparse
2
+ import collections
3
+ import gzip
4
+ import html
5
+ import json
6
+ import os
7
+ import random
8
+ import re
9
+ import torch
10
+ from tqdm import tqdm
11
+ import numpy as np
12
+ from utils import check_path, clean_text, amazon18_dataset2fullname, write_json_file, write_remap_index
13
+
14
+ def load_ratings(file):
15
+ users, items, inters = set(), set(), set()
16
+ with open(file, 'r') as fp:
17
+ for line in tqdm(fp, desc='Load ratings'):
18
+ try:
19
+ item, user, rating, time = line.strip().split(',')
20
+ users.add(user)
21
+ items.add(item)
22
+ inters.add((user, item, float(rating), int(time)))
23
+ except ValueError:
24
+ print(line)
25
+ return users, items, inters
26
+
27
+
28
+ def load_meta_items(file):
29
+ items = {}
30
+ with gzip.open(file, "r") as fp:
31
+ for line in tqdm(fp, desc="Load metas"):
32
+ data = json.loads(line)
33
+ item = data["asin"]
34
+ title = clean_text(data["title"])
35
+
36
+ descriptions = data["description"]
37
+ descriptions = clean_text(descriptions)
38
+
39
+ brand = data["brand"].replace("by\n", "").strip()
40
+
41
+ categories = data["category"]
42
+ new_categories = []
43
+ for category in categories:
44
+ if "</span>" in category:
45
+ break
46
+ new_categories.append(category.strip())
47
+ categories = ",".join(new_categories).strip()
48
+
49
+ items[item] = {"title": title, "description": descriptions, "brand": brand, "categories": categories}
50
+ # print(items[item])
51
+ return items
52
+
53
+
54
+ def load_review_data(args, user2id, item2id):
55
+
56
+ dataset_full_name = amazon18_dataset2fullname[args.dataset]
57
+ review_file_path = os.path.join(args.input_path, 'Review', dataset_full_name + '.json.gz')
58
+
59
+ reviews = {}
60
+
61
+ with gzip.open(review_file_path, "r") as fp:
62
+
63
+ for line in tqdm(fp,desc='Load reviews'):
64
+ inter = json.loads(line)
65
+ try:
66
+ user = inter['reviewerID']
67
+ item = inter['asin']
68
+ if user in user2id and item in item2id:
69
+ uid = user2id[user]
70
+ iid = item2id[item]
71
+ else:
72
+ continue
73
+ if 'reviewText' in inter:
74
+ review = clean_text(inter['reviewText'])
75
+ else:
76
+ review = ''
77
+ if 'summary' in inter:
78
+ summary = clean_text(inter['summary'])
79
+ else:
80
+ summary = ''
81
+ reviews[str((uid,iid))]={"review":review, "summary":summary}
82
+
83
+ except ValueError:
84
+ print(line)
85
+
86
+ return reviews
87
+
88
+
89
+ def get_user2count(inters):
90
+ user2count = collections.defaultdict(int)
91
+ for unit in inters:
92
+ user2count[unit[0]] += 1
93
+ return user2count
94
+
95
+
96
+ def get_item2count(inters):
97
+ item2count = collections.defaultdict(int)
98
+ for unit in inters:
99
+ item2count[unit[1]] += 1
100
+ return item2count
101
+
102
+
103
+ def generate_candidates(unit2count, threshold):
104
+ cans = set()
105
+ for unit, count in unit2count.items():
106
+ if count >= threshold:
107
+ cans.add(unit)
108
+ return cans, len(unit2count) - len(cans)
109
+
110
+
111
+ def filter_inters(inters, can_items=None,
112
+ user_k_core_threshold=0, item_k_core_threshold=0):
113
+ new_inters = []
114
+
115
+ # filter by meta items
116
+ if can_items:
117
+ print('\nFiltering by meta items: ')
118
+ for unit in inters:
119
+ if unit[1] in can_items.keys():
120
+ new_inters.append(unit)
121
+ inters, new_inters = new_inters, []
122
+ print(' The number of inters: ', len(inters))
123
+
124
+ # filter by k-core
125
+ if user_k_core_threshold or item_k_core_threshold:
126
+ print('\nFiltering by k-core:')
127
+ idx = 0
128
+ user2count = get_user2count(inters)
129
+ item2count = get_item2count(inters)
130
+
131
+ while True:
132
+ new_user2count = collections.defaultdict(int)
133
+ new_item2count = collections.defaultdict(int)
134
+ users, n_filtered_users = generate_candidates( # users is set
135
+ user2count, user_k_core_threshold)
136
+ items, n_filtered_items = generate_candidates(
137
+ item2count, item_k_core_threshold)
138
+ if n_filtered_users == 0 and n_filtered_items == 0:
139
+ break
140
+ for unit in inters:
141
+ if unit[0] in users and unit[1] in items:
142
+ new_inters.append(unit)
143
+ new_user2count[unit[0]] += 1
144
+ new_item2count[unit[1]] += 1
145
+ idx += 1
146
+ inters, new_inters = new_inters, []
147
+ user2count, item2count = new_user2count, new_item2count
148
+ print(' Epoch %d The number of inters: %d, users: %d, items: %d'
149
+ % (idx, len(inters), len(user2count), len(item2count)))
150
+ return inters
151
+
152
+
153
+ def make_inters_in_order(inters):
154
+ user2inters, new_inters = collections.defaultdict(list), list()
155
+ for inter in inters:
156
+ user, item, rating, timestamp = inter
157
+ user2inters[user].append((user, item, rating, timestamp))
158
+ for user in user2inters:
159
+ user_inters = user2inters[user]
160
+ user_inters.sort(key=lambda d: d[3])
161
+ interacted_item = set()
162
+ for inter in user_inters:
163
+ if inter[1] in interacted_item: # 过滤重复交互
164
+ continue
165
+ interacted_item.add(inter[1])
166
+ new_inters.append(inter)
167
+ return new_inters
168
+
169
+
170
+ def preprocess_rating(args):
171
+ dataset_full_name = amazon18_dataset2fullname[args.dataset]
172
+
173
+ print('Process rating data: ')
174
+ print(' Dataset: ', args.dataset)
175
+
176
+ # load ratings
177
+ rating_file_path = os.path.join(args.input_path, 'Ratings', dataset_full_name + '.csv')
178
+ rating_users, rating_items, rating_inters = load_ratings(rating_file_path)
179
+
180
+ # load item IDs with meta data
181
+ meta_file_path = os.path.join(args.input_path, 'Metadata', f'meta_{dataset_full_name}.json.gz')
182
+ meta_items = load_meta_items(meta_file_path)
183
+
184
+ # 1. Filter items w/o meta data;
185
+ # 2. K-core filtering;
186
+ print('The number of raw inters: ', len(rating_inters))
187
+
188
+ rating_inters = make_inters_in_order(rating_inters)
189
+
190
+ rating_inters = filter_inters(rating_inters, can_items=meta_items,
191
+ user_k_core_threshold=args.user_k,
192
+ item_k_core_threshold=args.item_k)
193
+
194
+ # sort interactions chronologically for each user
195
+ rating_inters = make_inters_in_order(rating_inters)
196
+ print('\n')
197
+
198
+ # return: list of (user_ID, item_ID, rating, timestamp)
199
+ return rating_inters, meta_items
200
+
201
+ def convert_inters2dict(inters):
202
+ user2items = collections.defaultdict(list)
203
+ user2index, item2index = dict(), dict()
204
+ for inter in inters:
205
+ user, item, rating, timestamp = inter
206
+ if user not in user2index:
207
+ user2index[user] = len(user2index)
208
+ if item not in item2index:
209
+ item2index[item] = len(item2index)
210
+ user2items[user2index[user]].append(item2index[item])
211
+ return user2items, user2index, item2index
212
+
213
+ def generate_data(args, rating_inters):
214
+ print('Split dataset: ')
215
+ print(' Dataset: ', args.dataset)
216
+
217
+ # generate train valid temp
218
+ user2items, user2index, item2index = convert_inters2dict(rating_inters)
219
+ train_inters, valid_inters, test_inters = dict(), dict(), dict()
220
+ for u_index in range(len(user2index)):
221
+ inters = user2items[u_index]
222
+ # leave one out
223
+ train_inters[u_index] = [str(i_index) for i_index in inters[:-2]]
224
+ valid_inters[u_index] = [str(inters[-2])]
225
+ test_inters[u_index] = [str(inters[-1])]
226
+ assert len(user2items[u_index]) == len(train_inters[u_index]) + \
227
+ len(valid_inters[u_index]) + len(test_inters[u_index])
228
+ return user2items, train_inters, valid_inters, test_inters, user2index, item2index
229
+
230
+ def convert_to_atomic_files(args, train_data, valid_data, test_data):
231
+ print('Convert dataset: ')
232
+ print(' Dataset: ', args.dataset)
233
+ uid_list = list(train_data.keys())
234
+ uid_list.sort(key=lambda t: int(t))
235
+
236
+ with open(os.path.join(args.output_path, args.dataset, f'{args.dataset}.train.inter'), 'w') as file:
237
+ file.write('user_id:token\titem_id_list:token_seq\titem_id:token\n')
238
+ for uid in uid_list:
239
+ item_seq = train_data[uid]
240
+ seq_len = len(item_seq)
241
+ for target_idx in range(1, seq_len):
242
+ target_item = item_seq[-target_idx]
243
+ seq = item_seq[:-target_idx][-50:]
244
+ file.write(f'{uid}\t{" ".join(seq)}\t{target_item}\n')
245
+
246
+ with open(os.path.join(args.output_path, args.dataset, f'{args.dataset}.valid.inter'), 'w') as file:
247
+ file.write('user_id:token\titem_id_list:token_seq\titem_id:token\n')
248
+ for uid in uid_list:
249
+ item_seq = train_data[uid][-50:]
250
+ target_item = valid_data[uid][0]
251
+ file.write(f'{uid}\t{" ".join(item_seq)}\t{target_item}\n')
252
+
253
+ with open(os.path.join(args.output_path, args.dataset, f'{args.dataset}.test.inter'), 'w') as file:
254
+ file.write('user_id:token\titem_id_list:token_seq\titem_id:token\n')
255
+ for uid in uid_list:
256
+ item_seq = (train_data[uid] + valid_data[uid])[-50:]
257
+ target_item = test_data[uid][0]
258
+ file.write(f'{uid}\t{" ".join(item_seq)}\t{target_item}\n')
259
+
260
+ def parse_args():
261
+ parser = argparse.ArgumentParser()
262
+ parser.add_argument('--dataset', type=str, default='Arts', help='Instruments / Arts / Games')
263
+ parser.add_argument('--user_k', type=int, default=5, help='user k-core filtering')
264
+ parser.add_argument('--item_k', type=int, default=5, help='item k-core filtering')
265
+ parser.add_argument('--input_path', type=str, default='')
266
+ parser.add_argument('--output_path', type=str, default='')
267
+ return parser.parse_args()
268
+
269
+
270
+ if __name__ == '__main__':
271
+ args = parse_args()
272
+
273
+ # load interactions from raw rating file
274
+ rating_inters, meta_items = preprocess_rating(args)
275
+
276
+
277
+ # split train/valid/temp
278
+ all_inters,train_inters, valid_inters, test_inters, user2index, item2index = generate_data(args, rating_inters)
279
+
280
+ check_path(os.path.join(args.output_path, args.dataset))
281
+
282
+ write_json_file(all_inters, os.path.join(args.output_path, args.dataset, f'{args.dataset}.inter.json'))
283
+ convert_to_atomic_files(args, train_inters, valid_inters, test_inters)
284
+
285
+ item2feature = collections.defaultdict(dict)
286
+ for item, item_id in item2index.items():
287
+ item2feature[item_id] = meta_items[item]
288
+
289
+ # reviews = load_review_data(args, user2index, item2index)
290
+
291
+ print("user:",len(user2index))
292
+ print("item:",len(item2index))
293
+
294
+ write_json_file(item2feature, os.path.join(args.output_path, args.dataset, f'{args.dataset}.item.json'))
295
+ # write_json_file(reviews, os.path.join(args.output_path, args.dataset, f'{args.dataset}.review.json'))
296
+
297
+
298
+ write_remap_index(user2index, os.path.join(args.output_path, args.dataset, f'{args.dataset}.user2id'))
299
+ write_remap_index(item2index, os.path.join(args.output_path, args.dataset, f'{args.dataset}.item2id'))
data_process/amazon18_recbole_data_process.py ADDED
@@ -0,0 +1,226 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import argparse
2
+ import collections
3
+ import gzip
4
+ import html
5
+ import json
6
+ import os
7
+ import random
8
+ import re
9
+ import torch
10
+ from tqdm import tqdm
11
+ import numpy as np
12
+ from utils import check_path, clean_text, amazon18_dataset2fullname,write_json_file,write_remap_index
13
+
14
+ def load_ratings(file):
15
+ users, items, inters = set(), set(), set()
16
+ with open(file, 'r') as fp:
17
+ for line in tqdm(fp, desc='Load ratings'):
18
+ try:
19
+ item, user, rating, time = line.strip().split(',')
20
+ users.add(user)
21
+ items.add(item)
22
+ inters.add((user, item, float(rating), int(time)))
23
+ except ValueError:
24
+ print(line)
25
+ return users, items, inters
26
+
27
+
28
+ def load_meta_items(file):
29
+ items = {}
30
+ # re_tag = re.compile('</?\w+[^>]*>')
31
+ with gzip.open(file, "r") as fp:
32
+ for line in tqdm(fp, desc="Load metas"):
33
+ data = json.loads(line)
34
+ item = data["asin"]
35
+ title = clean_text(data["title"])
36
+
37
+ descriptions = data["description"]
38
+ descriptions = clean_text(descriptions)
39
+ # new_descriptions = []
40
+ # for description in descriptions:
41
+ # description = re.sub(re_tag, '', description)
42
+ # new_descriptions.append(description.strip())
43
+ # descriptions = " ".join(new_descriptions).strip()
44
+
45
+ brand = data["brand"].replace("by\n", "").strip()
46
+
47
+ categories = data["category"]
48
+ new_categories = []
49
+ for category in categories:
50
+ if "</span>" in category:
51
+ break
52
+ new_categories.append(category.strip())
53
+ categories = ",".join(new_categories[1:]).strip()
54
+
55
+ items[item] = {"title": title, "description": descriptions, "brand": brand, "categories": categories}
56
+ # print(items[item])
57
+ return items
58
+
59
+
60
+ def get_user2count(inters):
61
+ user2count = collections.defaultdict(int)
62
+ for unit in inters:
63
+ user2count[unit[0]] += 1
64
+ return user2count
65
+
66
+
67
+ def get_item2count(inters):
68
+ item2count = collections.defaultdict(int)
69
+ for unit in inters:
70
+ item2count[unit[1]] += 1
71
+ return item2count
72
+
73
+
74
+ def generate_candidates(unit2count, threshold):
75
+ cans = set()
76
+ for unit, count in unit2count.items():
77
+ if count >= threshold:
78
+ cans.add(unit)
79
+ return cans, len(unit2count) - len(cans)
80
+
81
+
82
+ def filter_inters(inters, can_items=None,
83
+ user_k_core_threshold=0, item_k_core_threshold=0):
84
+ new_inters = []
85
+
86
+ # filter by meta items
87
+ if can_items:
88
+ print('\nFiltering by meta items: ')
89
+ for unit in inters:
90
+ if unit[1] in can_items.keys():
91
+ new_inters.append(unit)
92
+ inters, new_inters = new_inters, []
93
+ print(' The number of inters: ', len(inters))
94
+
95
+ # filter by k-core
96
+ if user_k_core_threshold or item_k_core_threshold:
97
+ print('\nFiltering by k-core:')
98
+ idx = 0
99
+ user2count = get_user2count(inters)
100
+ item2count = get_item2count(inters)
101
+
102
+ while True:
103
+ new_user2count = collections.defaultdict(int)
104
+ new_item2count = collections.defaultdict(int)
105
+ users, n_filtered_users = generate_candidates( # users is set
106
+ user2count, user_k_core_threshold)
107
+ items, n_filtered_items = generate_candidates(
108
+ item2count, item_k_core_threshold)
109
+ if n_filtered_users == 0 and n_filtered_items == 0:
110
+ break
111
+ for unit in inters:
112
+ if unit[0] in users and unit[1] in items:
113
+ new_inters.append(unit)
114
+ new_user2count[unit[0]] += 1
115
+ new_item2count[unit[1]] += 1
116
+ idx += 1
117
+ inters, new_inters = new_inters, []
118
+ user2count, item2count = new_user2count, new_item2count
119
+ print(' Epoch %d The number of inters: %d, users: %d, items: %d'
120
+ % (idx, len(inters), len(user2count), len(item2count)))
121
+ return inters
122
+
123
+
124
+ def make_inters_in_order(inters):
125
+ user2inters, new_inters = collections.defaultdict(list), list()
126
+ for inter in inters:
127
+ user, item, rating, timestamp = inter
128
+ user2inters[user].append((user, item, rating, timestamp))
129
+ for user in user2inters:
130
+ user_inters = user2inters[user]
131
+ user_inters.sort(key=lambda d: d[3])
132
+ interacted_item = set()
133
+ for inter in user_inters:
134
+ if inter[1] in interacted_item: # 过滤重复交互
135
+ continue
136
+ interacted_item.add(inter[1])
137
+ new_inters.append(inter)
138
+ return new_inters
139
+
140
+
141
+ def preprocess_rating(args):
142
+ dataset_full_name = amazon18_dataset2fullname[args.dataset]
143
+
144
+ print('Process rating data: ')
145
+ print(' Dataset: ', args.dataset)
146
+
147
+ # load ratings
148
+ rating_file_path = os.path.join(args.input_path, 'Ratings', dataset_full_name + '.csv')
149
+ rating_users, rating_items, rating_inters = load_ratings(rating_file_path)
150
+
151
+ # load item IDs with meta data
152
+ meta_file_path = os.path.join(args.input_path, 'Metadata', f'meta_{dataset_full_name}.json.gz')
153
+ meta_items = load_meta_items(meta_file_path)
154
+
155
+ # 1. Filter items w/o meta data;
156
+ # 2. K-core filtering;
157
+ print('The number of raw inters: ', len(rating_inters))
158
+
159
+ rating_inters = make_inters_in_order(rating_inters)
160
+
161
+ rating_inters = filter_inters(rating_inters, can_items=meta_items,
162
+ user_k_core_threshold=args.user_k,
163
+ item_k_core_threshold=args.item_k)
164
+
165
+ # sort interactions chronologically for each user
166
+ rating_inters = make_inters_in_order(rating_inters)
167
+ print('\n')
168
+
169
+ # return: list of (user_ID, item_ID, rating, timestamp)
170
+ return rating_inters, meta_items
171
+
172
+ def save_inter(args, inters):
173
+ print('Convert dataset: ')
174
+ print(' Dataset: ', args.dataset)
175
+
176
+ with open(os.path.join(args.output_path, args.dataset, f'{args.dataset}.inter'), 'w') as file:
177
+ file.write('user_id:token\titem_id:token\trating:float\ttimestamp:float\n')
178
+ for inter in inters:
179
+ user, item, rating, timestamp = inter
180
+ file.write(f'{user}\t{item}\t{rating}\t{timestamp}\n')
181
+
182
+
183
+ def save_feat(args, feat, all_items):
184
+ iid_list = list(feat.keys())
185
+ num_item = 0
186
+ with open(os.path.join(args.output_path, args.dataset, f'{args.dataset}.item'), 'w') as file:
187
+ # "title": title, "description": descriptions, "brand": brand, "categories": categories
188
+ file.write('item_id:token\ttitle:token_seq\tbrand:token\tcategories:token_seq\n')
189
+ for iid in iid_list:
190
+ if iid in all_items:
191
+ num_item += 1
192
+ title, brand, categories = feat[iid]["title"], feat[iid]["brand"], feat[iid]["categories"]
193
+ file.write(f'{iid}\t{title}\t{brand}\t{categories}\n')
194
+ print("num_item: ", num_item)
195
+
196
+
197
+ def parse_args():
198
+ parser = argparse.ArgumentParser()
199
+ parser.add_argument('--dataset', type=str, default='Arts', help='Instruments / Arts / Games')
200
+ parser.add_argument('--user_k', type=int, default=5, help='user k-core filtering')
201
+ parser.add_argument('--item_k', type=int, default=5, help='item k-core filtering')
202
+ parser.add_argument('--input_path', type=str, default='')
203
+ parser.add_argument('--output_path', type=str, default='')
204
+ return parser.parse_args()
205
+
206
+
207
+ if __name__ == '__main__':
208
+ args = parse_args()
209
+
210
+ # load interactions from raw rating file
211
+ rating_inters, meta_items = preprocess_rating(args)
212
+
213
+ check_path(os.path.join(args.output_path, args.dataset))
214
+
215
+
216
+ all_items = set()
217
+ for inter in rating_inters:
218
+ user, item, rating, timestamp = inter
219
+ all_items.add(item)
220
+
221
+ print("total item: ", len(list(all_items)))
222
+
223
+ save_inter(args,rating_inters)
224
+ save_feat(args,meta_items, all_items)
225
+
226
+
data_process/amazon_text_emb.py ADDED
@@ -0,0 +1,139 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import argparse
2
+ import collections
3
+ import gzip
4
+ import html
5
+ import json
6
+ import os
7
+ import random
8
+ import re
9
+ import torch
10
+ from tqdm import tqdm
11
+ import numpy as np
12
+ from utils import *
13
+ from transformers import LlamaForCausalLM, LlamaTokenizer, LlamaConfig, AutoTokenizer, AutoModel
14
+
15
+
16
+ def load_data(args):
17
+
18
+ item2feature_path = os.path.join(args.root, f'{args.dataset}.item.json')
19
+ item2feature = load_json(item2feature_path)
20
+
21
+ return item2feature
22
+
23
+ def generate_text(item2feature, features):
24
+ item_text_list = []
25
+
26
+ for item in item2feature:
27
+ data = item2feature[item]
28
+ text = []
29
+ for meta_key in features:
30
+ if meta_key in data:
31
+ meta_value = clean_text(data[meta_key])
32
+ text.append(meta_value.strip())
33
+
34
+ item_text_list.append([int(item), text])
35
+
36
+ return item_text_list
37
+
38
+ def preprocess_text(args):
39
+ print('Process text data: ')
40
+ print(' Dataset: ', args.dataset)
41
+
42
+ item2feature = load_data(args)
43
+ # load item text and clean
44
+ item_text_list = generate_text(item2feature, ['title', 'description'])
45
+ # item_text_list = generate_text(item2feature, ['title'])
46
+ # return: list of (item_ID, cleaned_item_text)
47
+ return item_text_list
48
+
49
+ def generate_item_embedding(args, item_text_list, tokenizer, model, word_drop_ratio=-1):
50
+ print(f'Generate Text Embedding: ')
51
+ print(' Dataset: ', args.dataset)
52
+
53
+ items, texts = zip(*item_text_list)
54
+ order_texts = [[0]] * len(items)
55
+ for item, text in zip(items, texts):
56
+ order_texts[item] = text
57
+ for text in order_texts:
58
+ assert text != [0]
59
+
60
+ embeddings = []
61
+ start, batch_size = 0, 1
62
+ with torch.no_grad():
63
+ while start < len(order_texts):
64
+ if (start+1)%100==0:
65
+ print("==>",start+1)
66
+ field_texts = order_texts[start: start + batch_size]
67
+ # print(field_texts)
68
+ field_texts = zip(*field_texts)
69
+
70
+ field_embeddings = []
71
+ for sentences in field_texts:
72
+ sentences = list(sentences)
73
+ # print(sentences)
74
+ if word_drop_ratio > 0:
75
+ print(f'Word drop with p={word_drop_ratio}')
76
+ new_sentences = []
77
+ for sent in sentences:
78
+ new_sent = []
79
+ sent = sent.split(' ')
80
+ for wd in sent:
81
+ rd = random.random()
82
+ if rd > word_drop_ratio:
83
+ new_sent.append(wd)
84
+ new_sent = ' '.join(new_sent)
85
+ new_sentences.append(new_sent)
86
+ sentences = new_sentences
87
+ encoded_sentences = tokenizer(sentences, max_length=args.max_sent_len,
88
+ truncation=True, return_tensors='pt',padding="longest").to(args.device)
89
+ outputs = model(input_ids=encoded_sentences.input_ids,
90
+ attention_mask=encoded_sentences.attention_mask)
91
+
92
+ masked_output = outputs.last_hidden_state * encoded_sentences['attention_mask'].unsqueeze(-1)
93
+ mean_output = masked_output.sum(dim=1) / encoded_sentences['attention_mask'].sum(dim=-1, keepdim=True)
94
+ mean_output = mean_output.detach().cpu()
95
+ field_embeddings.append(mean_output)
96
+
97
+ field_mean_embedding = torch.stack(field_embeddings, dim=0).mean(dim=0)
98
+ embeddings.append(field_mean_embedding)
99
+ start += batch_size
100
+
101
+ embeddings = torch.cat(embeddings, dim=0).numpy()
102
+ print('Embeddings shape: ', embeddings.shape)
103
+
104
+ file = os.path.join(args.root, args.dataset + '.emb-' + args.plm_name + "-td" + ".npy")
105
+ np.save(file, embeddings)
106
+
107
+
108
+ def parse_args():
109
+ parser = argparse.ArgumentParser()
110
+ parser.add_argument('--dataset', type=str, default='Arts', help='Instruments / Arts / Games')
111
+ parser.add_argument('--root', type=str, default="")
112
+ parser.add_argument('--gpu_id', type=int, default=2, help='ID of running GPU')
113
+ parser.add_argument('--plm_name', type=str, default='llama')
114
+ parser.add_argument('--plm_checkpoint', type=str,
115
+ default='')
116
+ parser.add_argument('--max_sent_len', type=int, default=2048)
117
+ parser.add_argument('--word_drop_ratio', type=float, default=-1, help='word drop ratio, do not drop by default')
118
+ return parser.parse_args()
119
+
120
+
121
+ if __name__ == '__main__':
122
+ args = parse_args()
123
+
124
+ args.root = os.path.join(args.root, args.dataset)
125
+
126
+ device = set_device(args.gpu_id)
127
+ args.device = device
128
+
129
+ item_text_list = preprocess_text(args)
130
+
131
+ plm_tokenizer, plm_model = load_plm(args.plm_checkpoint)
132
+ if plm_tokenizer.pad_token_id is None:
133
+ plm_tokenizer.pad_token_id = 0
134
+ plm_model = plm_model.to(device)
135
+
136
+ generate_item_embedding(args, item_text_list,plm_tokenizer,
137
+ plm_model, word_drop_ratio=args.word_drop_ratio)
138
+
139
+
data_process/get_llm_output.py ADDED
@@ -0,0 +1,374 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+
3
+ import argparse
4
+ import os
5
+ import os.path as osp
6
+ import random
7
+ import time
8
+ from logging import getLogger
9
+ import openai
10
+ from utils import get_res_batch, load_json, intention_prompt, preference_prompt_1, preference_prompt_2, amazon18_dataset2fullname, write_json_file
11
+ import json
12
+
13
+
14
+
15
+ def get_intention_train(args, inters, item2feature, reviews, api_info):
16
+
17
+ intention_train_output_file = os.path.join(args.root,"intention_train.json")
18
+
19
+
20
+ # Suggest modifying the prompt based on different datasets
21
+ prompt = intention_prompt
22
+ dataset_full_name = amazon18_dataset2fullname[args.dataset]
23
+ dataset_full_name = dataset_full_name.replace("_", " ").lower()
24
+ print(dataset_full_name)
25
+
26
+ prompt_list = []
27
+
28
+ inter_data = []
29
+
30
+ for (user,item_list) in inters.items():
31
+ user = int(user)
32
+ item = int(item_list[-3])
33
+ history = item_list[:-3]
34
+
35
+ inter_data.append((user,item,history))
36
+
37
+ review = reviews[str((user, item))]["review"]
38
+ item_title = item2feature[str(item)]["title"]
39
+ input_prompt = prompt.format(item_title=item_title,dataset_full_name=dataset_full_name,review=review)
40
+ prompt_list.append(input_prompt)
41
+
42
+ st = 0
43
+ with open(intention_train_output_file, mode='a') as f:
44
+
45
+ while st < len(prompt_list):
46
+ # while st < 3:
47
+ print(st)
48
+ # if st < 25631:
49
+ # st += args.batchsize
50
+ # continue
51
+
52
+
53
+ res = get_res_batch(args.model_name, prompt_list[st:st+args.batchsize], args.max_tokens, api_info)
54
+
55
+ for i, answer in enumerate(res):
56
+ user, item, history = inter_data[st+i]
57
+ # print(answer)
58
+ # print("=============")
59
+
60
+ if answer == '':
61
+ print("answer null error")
62
+ answer = "I enjoy high-quality item."
63
+
64
+ if answer.strip().count('\n') != 1:
65
+ if 'haracteristics:' in answer:
66
+ answer = answer.strip().split("The item's characteristics:")
67
+ else:
68
+ answer = answer.strip().split("The item's characteristic:")
69
+ else:
70
+ answer = answer.strip().split('\n')
71
+
72
+ if '' in answer:
73
+ answer.remove('')
74
+
75
+ if len(answer) == 1:
76
+ print(answer)
77
+ user_preference = item_character = answer[0]
78
+ elif len(answer) >= 3:
79
+ print(answer)
80
+ answer = answer[-1]
81
+ user_preference = item_character = answer
82
+ else:
83
+ user_preference, item_character = answer
84
+
85
+ if ':' in user_preference:
86
+ idx = user_preference.index(':')
87
+ user_preference = user_preference[idx+1:]
88
+ user_preference = user_preference.strip().replace('}','')
89
+ user_preference = user_preference.replace('\n','')
90
+
91
+ if ':' in item_character:
92
+ idx = item_character.index(':')
93
+ item_character = item_character[idx+1:]
94
+ item_character = item_character.strip().replace('}','')
95
+ item_character = item_character.replace('\n','')
96
+
97
+
98
+ dict = {"user":user, "item":item, "inters": history,
99
+ "user_related_intention":user_preference, "item_related_intention": item_character}
100
+
101
+ json.dump(dict, f)
102
+ f.write("\n")
103
+
104
+ st += args.batchsize
105
+
106
+ return intention_train_output_file
107
+
108
+
109
+ def get_intention_test(args, inters, item2feature, reviews, api_info):
110
+
111
+ intention_test_output_file = os.path.join(args.root,"intention_test.json")
112
+
113
+ # Suggest modifying the prompt based on different datasets
114
+ prompt = intention_prompt
115
+ dataset_full_name = amazon18_dataset2fullname[args.dataset]
116
+ dataset_full_name = dataset_full_name.replace("_", " ").lower()
117
+ print(dataset_full_name)
118
+
119
+ prompt_list = []
120
+
121
+ inter_data = []
122
+
123
+ for (user,item_list) in inters.items():
124
+ user = int(user)
125
+ item = int(item_list[-1])
126
+ history = item_list[:-1]
127
+
128
+ inter_data.append((user,item,history))
129
+
130
+ review = reviews[str((user, item))]["review"]
131
+ item_title = item2feature[str(item)]["title"]
132
+ input_prompt = prompt.format(item_title=item_title,dataset_full_name=dataset_full_name,review=review)
133
+ prompt_list.append(input_prompt)
134
+
135
+ st = 0
136
+ with open(intention_test_output_file, mode='a') as f:
137
+
138
+ while st < len(prompt_list):
139
+ # while st < 3:
140
+ print(st)
141
+ # if st < 4623:
142
+ # st += args.batchsize
143
+ # continue
144
+
145
+ res = get_res_batch(args.model_name, prompt_list[st:st+args.batchsize], args.max_tokens, api_info)
146
+
147
+ for i, answer in enumerate(res):
148
+ user, item, history = inter_data[st+i]
149
+
150
+ if answer == '':
151
+ print("answer null error")
152
+ answer = "I enjoy high-quality item."
153
+
154
+ if answer.strip().count('\n') != 1:
155
+ if 'haracteristics:' in answer:
156
+ answer = answer.strip().split("The item's characteristics:")
157
+ else:
158
+ answer = answer.strip().split("The item's characteristic:")
159
+ else:
160
+ answer = answer.strip().split('\n')
161
+
162
+ if '' in answer:
163
+ answer.remove('')
164
+
165
+ if len(answer) == 1:
166
+ print(answer)
167
+ user_preference = item_character = answer[0]
168
+ elif len(answer) >= 3:
169
+ print(answer)
170
+ answer = answer[-1]
171
+ user_preference = item_character = answer
172
+ else:
173
+ user_preference, item_character = answer
174
+
175
+ if ':' in user_preference:
176
+ idx = user_preference.index(':')
177
+ user_preference = user_preference[idx+1:]
178
+ user_preference = user_preference.strip().replace('}','')
179
+ user_preference = user_preference.replace('\n','')
180
+
181
+ if ':' in item_character:
182
+ idx = item_character.index(':')
183
+ item_character = item_character[idx+1:]
184
+ item_character = item_character.strip().replace('}','')
185
+ item_character = item_character.replace('\n','')
186
+
187
+
188
+ dict = {"user":user, "item":item, "inters": history,
189
+ "user_related_intention":user_preference, "item_related_intention": item_character}
190
+
191
+ json.dump(dict, f)
192
+ f.write("\n")
193
+
194
+ st += args.batchsize
195
+
196
+ return intention_test_output_file
197
+
198
+
199
+
200
+
201
+ def get_user_preference(args, inters, item2feature, reviews, api_info):
202
+
203
+ preference_output_file = os.path.join(args.root,"user_preference.json")
204
+
205
+
206
+ # Suggest modifying the prompt based on different datasets
207
+ prompt_1 = preference_prompt_1
208
+ prompt_2 = preference_prompt_2
209
+
210
+
211
+ dataset_full_name = amazon18_dataset2fullname[args.dataset]
212
+ dataset_full_name = dataset_full_name.replace("_", " ").lower()
213
+ print(dataset_full_name)
214
+
215
+ prompt_list_1 = []
216
+ prompt_list_2 = []
217
+
218
+ users = []
219
+
220
+ for (user,item_list) in inters.items():
221
+ users.append(user)
222
+ history = item_list[:-3]
223
+ item_titles = []
224
+ for j, item in enumerate(history):
225
+ item_titles.append(str(j+1) + '.' + item2feature[str(item)]["title"])
226
+ if len(item_titles) > args.max_his_len:
227
+ item_titles = item_titles[-args.max_his_len:]
228
+ item_titles = ", ".join(item_titles)
229
+
230
+ input_prompt_1 = prompt_1.format(dataset_full_name=dataset_full_name, item_titles=item_titles)
231
+ input_prompt_2 = prompt_2.format(dataset_full_name=dataset_full_name, item_titles=item_titles)
232
+
233
+ prompt_list_1.append(input_prompt_1)
234
+ prompt_list_2.append(input_prompt_2)
235
+
236
+
237
+ st = 0
238
+ with open(preference_output_file, mode='a') as f:
239
+
240
+ while st < len(prompt_list_1):
241
+ # while st < 3:
242
+ print(st)
243
+ # if st < 22895:
244
+ # st += args.batchsize
245
+ # continue
246
+
247
+ res_1 = get_res_batch(args.model_name, prompt_list_1[st:st + args.batchsize], args.max_tokens, api_info)
248
+ res_2 = get_res_batch(args.model_name, prompt_list_2[st:st + args.batchsize], args.max_tokens, api_info)
249
+ for i, answers in enumerate(zip(res_1, res_2)):
250
+
251
+ user = users[st + i]
252
+
253
+ answer_1, answer_2 = answers
254
+ # print(answers)
255
+ # print("=============")
256
+
257
+ if answer_1 == '':
258
+ print("answer null error")
259
+ answer_1 = "I enjoy high-quality item."
260
+
261
+ if answer_2 == '':
262
+ print("answer null error")
263
+ answer_2 = "I enjoy high-quality item."
264
+
265
+ if answer_2.strip().count('\n') != 1:
266
+ if 'references:' in answer_2:
267
+ answer_2 = answer_2.strip().split("Short-term preferences:")
268
+ else:
269
+ answer_2 = answer_2.strip().split("Short-term preference:")
270
+ else:
271
+ answer_2 = answer_2.strip().split('\n')
272
+
273
+ if '' in answer_2:
274
+ answer_2.remove('')
275
+
276
+ if len(answer_2) == 1:
277
+ print(answer_2)
278
+ long_preference = short_preference = answer_2[0]
279
+ elif len(answer_2) >= 3:
280
+ print(answer_2)
281
+ answer_2 = answer_2[-1]
282
+ long_preference = short_preference = answer_2
283
+ else:
284
+ long_preference, short_preference = answer_2
285
+
286
+ if ':' in long_preference:
287
+ idx = long_preference.index(':')
288
+ long_preference = long_preference[idx+1:]
289
+ long_preference = long_preference.strip().replace('}','')
290
+ long_preference = long_preference.replace('\n','')
291
+
292
+ if ':' in short_preference:
293
+ idx = short_preference.index(':')
294
+ short_preference = short_preference[idx+1:]
295
+ short_preference = short_preference.strip().replace('}','')
296
+ short_preference = short_preference.replace('\n','')
297
+
298
+ dict = {"user":user,"user_preference":[answer_1, long_preference, short_preference]}
299
+ # print(dict)
300
+ json.dump(dict, f)
301
+ f.write("\n")
302
+
303
+ st += args.batchsize
304
+
305
+ return preference_output_file
306
+
307
+ def parse_args():
308
+ parser = argparse.ArgumentParser()
309
+ parser.add_argument('--dataset', type=str, default='Instruments', help='Instruments / Arts / Games')
310
+ parser.add_argument('--root', type=str, default='')
311
+ parser.add_argument('--api_info', type=str, default='./api_info.json')
312
+ parser.add_argument('--model_name', type=str, default='text-davinci-003')
313
+ parser.add_argument('--max_tokens', type=int, default=512)
314
+ parser.add_argument('--batchsize', type=int, default=16)
315
+ parser.add_argument('--max_his_len', type=int, default=20)
316
+ return parser.parse_args()
317
+
318
+ if __name__ == "__main__":
319
+ args = parse_args()
320
+
321
+ args.root = os.path.join(args.root, args.dataset)
322
+
323
+ api_info = load_json(args.api_info)
324
+ openai.api_key = api_info["api_key_list"].pop()
325
+
326
+
327
+ inter_path = os.path.join(args.root, f'{args.dataset}.inter.json')
328
+ inters = load_json(inter_path)
329
+
330
+
331
+ item2feature_path = os.path.join(args.root, f'{args.dataset}.item.json')
332
+ item2feature = load_json(item2feature_path)
333
+
334
+ reviews_path = os.path.join(args.root, f'{args.dataset}.review.json')
335
+ reviews = load_json(reviews_path)
336
+
337
+ intention_train_output_file = get_intention_train(args, inters, item2feature, reviews, api_info)
338
+ intention_test_output_file = get_intention_test(args, inters, item2feature, reviews ,api_info)
339
+ preference_output_file = get_user_preference(args, inters, item2feature, reviews, api_info)
340
+
341
+ intention_train = {}
342
+ intention_test = {}
343
+ user_preference = {}
344
+
345
+ with open(intention_train_output_file, "r") as f:
346
+ for line in f:
347
+ # print(line)
348
+ content = json.loads(line)
349
+ if content["user"] not in intention_train:
350
+ intention_train[content["user"]] = {"item":content["item"],
351
+ "inters":content["inters"],
352
+ "querys":[ content["user_related_intention"], content["item_related_intention"] ]}
353
+
354
+
355
+ with open(intention_test_output_file, "r") as f:
356
+ for line in f:
357
+ content = json.loads(line)
358
+ if content["user"] not in intention_train:
359
+ intention_test[content["user"]] = {"item":content["item"],
360
+ "inters":content["inters"],
361
+ "querys":[ content["user_related_intention"], content["item_related_intention"] ]}
362
+
363
+
364
+ with open(preference_output_file, "r") as f:
365
+ for line in f:
366
+ content = json.loads(line)
367
+ user_preference[content["user"]] = content["user_preference"]
368
+
369
+ user_dict = {
370
+ "user_explicit_preference": user_preference,
371
+ "user_vague_intention": {"train": intention_train, "test": intention_test},
372
+ }
373
+
374
+ write_json_file(user_dict, os.path.join(args.root, f'{args.dataset}.user.json'))