Hugging Face's logo

lamthuy
/
MorganGen

Safetensors
bart
Model card Files
xet
Community
lamthuy commited on
Commit
f0a96d3
·
verified ·
1 Parent(s): da0f61f

Upload folder using huggingface_hub

Browse files
Files changed (18) hide show
  1. README.md +58 -0
  2. __pycache__/utils.cpython-312.pyc +0 -0
  3. added_tokens.json +2050 -0
  4. config.json +74 -0
  5. examples.py +25 -0
  6. generation_config.json +13 -0
  7. latest +1 -0
  8. merges.txt +0 -0
  9. model.safetensors +3 -0
  10. rng_state_0.pth +3 -0
  11. rng_state_1.pth +3 -0
  12. special_tokens_map.json +51 -0
  13. tokenizer_config.json +0 -0
  14. trainer_state.json +0 -0
  15. training_args.bin +3 -0
  16. utils.py +82 -0
  17. vocab.json +0 -0
  18. zero_to_fp32.py +760 -0
README.md ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ ---
4
+ ## MorganGen
5
+ To use it you can clone the HF gitrepo before running the following examples:
6
+
7
+ ```commandline
8
+ git lfs install # Only once if not done already
9
+ git clone https://huggingface.co/lamthuy/MorganGen
10
+ ```
11
+
12
+ A generative model trained on 120 million SMILES strings from the ZINC database. The model takes as input a sequence of indices representing the active bits in a 2048-bit Morgan fingerprint. Each index corresponds to a bit set to 1, while all other bits are 0.
13
+ ```
14
+ s = [12][184][1200]
15
+ ```
16
+ represents a fingerprint where only bits 12, 184, and 1200 are set to 1, and the remaining bits are 0.
17
+ # Running example
18
+ The following code snippet in the notebook demonstrates how to load the model from a checkpoint and generate a new SMILES string, conditioned on a given input SMILES.
19
+
20
+ ```python
21
+ from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
22
+ from utils import MorganFingerprint, morgan_fingerprint_to_text
23
+
24
+
25
+ # Load the checkpoint and the tokenizer
26
+ checkpoint_path = "lamthuy/MorganGen"
27
+ model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint_path)
28
+ tokenizer = AutoTokenizer.from_pretrained(checkpoint_path)
29
+
30
+ # Given a SMILES, get its fingerpint
31
+ smiles = "CC(=O)OC1=CC=CC=C1C(=O)O"
32
+ m = MorganFingerprint()
33
+ mf = m.smiles_to_morgan(smiles)
34
+
35
+ # convert it to the indices text format
36
+ s = morgan_fingerprint_to_text(mf)
37
+
38
+ # encode
39
+ input_ids = tokenizer.encode(s, return_tensors="pt")
40
+ # Generate output sequence
41
+ output_ids = model.generate(input_ids, max_length=64, num_beams=5)
42
+
43
+ # Decode the generated output
44
+ output_smiles = tokenizer.decode(output_ids[0], skip_special_tokens=True)
45
+ print(output_smiles)
46
+
47
+ ```
48
+
49
+ # Reference
50
+ ```
51
+ @inproceedings{hoang2024morgangen,
52
+ title={MorganGen: Generative Modeling of SMILES Using Morgan Fingerprint Features},
53
+ author={Hoang, Lam Thanh and D{\'\i}az, Ra{\'u}l Fern{\'a}ndez and Lopez, Vanessa},
54
+ booktitle={American Chemical Society (ACS) Fall Meeting},
55
+ year={2024}
56
+ }
57
+
58
+ ```
__pycache__/utils.cpython-312.pyc ADDED
Binary file (4.25 kB). View file
 
added_tokens.json ADDED
@@ -0,0 +1,2050 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "[0]": 50265,
3
+ "[1000]": 51265,
4
+ "[1001]": 51266,
5
+ "[1002]": 51267,
6
+ "[1003]": 51268,
7
+ "[1004]": 51269,
8
+ "[1005]": 51270,
9
+ "[1006]": 51271,
10
+ "[1007]": 51272,
11
+ "[1008]": 51273,
12
+ "[1009]": 51274,
13
+ "[100]": 50365,
14
+ "[1010]": 51275,
15
+ "[1011]": 51276,
16
+ "[1012]": 51277,
17
+ "[1013]": 51278,
18
+ "[1014]": 51279,
19
+ "[1015]": 51280,
20
+ "[1016]": 51281,
21
+ "[1017]": 51282,
22
+ "[1018]": 51283,
23
+ "[1019]": 51284,
24
+ "[101]": 50366,
25
+ "[1020]": 51285,
26
+ "[1021]": 51286,
27
+ "[1022]": 51287,
28
+ "[1023]": 51288,
29
+ "[1024]": 51289,
30
+ "[1025]": 51290,
31
+ "[1026]": 51291,
32
+ "[1027]": 51292,
33
+ "[1028]": 51293,
34
+ "[1029]": 51294,
35
+ "[102]": 50367,
36
+ "[1030]": 51295,
37
+ "[1031]": 51296,
38
+ "[1032]": 51297,
39
+ "[1033]": 51298,
40
+ "[1034]": 51299,
41
+ "[1035]": 51300,
42
+ "[1036]": 51301,
43
+ "[1037]": 51302,
44
+ "[1038]": 51303,
45
+ "[1039]": 51304,
46
+ "[103]": 50368,
47
+ "[1040]": 51305,
48
+ "[1041]": 51306,
49
+ "[1042]": 51307,
50
+ "[1043]": 51308,
51
+ "[1044]": 51309,
52
+ "[1045]": 51310,
53
+ "[1046]": 51311,
54
+ "[1047]": 51312,
55
+ "[1048]": 51313,
56
+ "[1049]": 51314,
57
+ "[104]": 50369,
58
+ "[1050]": 51315,
59
+ "[1051]": 51316,
60
+ "[1052]": 51317,
61
+ "[1053]": 51318,
62
+ "[1054]": 51319,
63
+ "[1055]": 51320,
64
+ "[1056]": 51321,
65
+ "[1057]": 51322,
66
+ "[1058]": 51323,
67
+ "[1059]": 51324,
68
+ "[105]": 50370,
69
+ "[1060]": 51325,
70
+ "[1061]": 51326,
71
+ "[1062]": 51327,
72
+ "[1063]": 51328,
73
+ "[1064]": 51329,
74
+ "[1065]": 51330,
75
+ "[1066]": 51331,
76
+ "[1067]": 51332,
77
+ "[1068]": 51333,
78
+ "[1069]": 51334,
79
+ "[106]": 50371,
80
+ "[1070]": 51335,
81
+ "[1071]": 51336,
82
+ "[1072]": 51337,
83
+ "[1073]": 51338,
84
+ "[1074]": 51339,
85
+ "[1075]": 51340,
86
+ "[1076]": 51341,
87
+ "[1077]": 51342,
88
+ "[1078]": 51343,
89
+ "[1079]": 51344,
90
+ "[107]": 50372,
91
+ "[1080]": 51345,
92
+ "[1081]": 51346,
93
+ "[1082]": 51347,
94
+ "[1083]": 51348,
95
+ "[1084]": 51349,
96
+ "[1085]": 51350,
97
+ "[1086]": 51351,
98
+ "[1087]": 51352,
99
+ "[1088]": 51353,
100
+ "[1089]": 51354,
101
+ "[108]": 50373,
102
+ "[1090]": 51355,
103
+ "[1091]": 51356,
104
+ "[1092]": 51357,
105
+ "[1093]": 51358,
106
+ "[1094]": 51359,
107
+ "[1095]": 51360,
108
+ "[1096]": 51361,
109
+ "[1097]": 51362,
110
+ "[1098]": 51363,
111
+ "[1099]": 51364,
112
+ "[109]": 50374,
113
+ "[10]": 50275,
114
+ "[1100]": 51365,
115
+ "[1101]": 51366,
116
+ "[1102]": 51367,
117
+ "[1103]": 51368,
118
+ "[1104]": 51369,
119
+ "[1105]": 51370,
120
+ "[1106]": 51371,
121
+ "[1107]": 51372,
122
+ "[1108]": 51373,
123
+ "[1109]": 51374,
124
+ "[110]": 50375,
125
+ "[1110]": 51375,
126
+ "[1111]": 51376,
127
+ "[1112]": 51377,
128
+ "[1113]": 51378,
129
+ "[1114]": 51379,
130
+ "[1115]": 51380,
131
+ "[1116]": 51381,
132
+ "[1117]": 51382,
133
+ "[1118]": 51383,
134
+ "[1119]": 51384,
135
+ "[111]": 50376,
136
+ "[1120]": 51385,
137
+ "[1121]": 51386,
138
+ "[1122]": 51387,
139
+ "[1123]": 51388,
140
+ "[1124]": 51389,
141
+ "[1125]": 51390,
142
+ "[1126]": 51391,
143
+ "[1127]": 51392,
144
+ "[1128]": 51393,
145
+ "[1129]": 51394,
146
+ "[112]": 50377,
147
+ "[1130]": 51395,
148
+ "[1131]": 51396,
149
+ "[1132]": 51397,
150
+ "[1133]": 51398,
151
+ "[1134]": 51399,
152
+ "[1135]": 51400,
153
+ "[1136]": 51401,
154
+ "[1137]": 51402,
155
+ "[1138]": 51403,
156
+ "[1139]": 51404,
157
+ "[113]": 50378,
158
+ "[1140]": 51405,
159
+ "[1141]": 51406,
160
+ "[1142]": 51407,
161
+ "[1143]": 51408,
162
+ "[1144]": 51409,
163
+ "[1145]": 51410,
164
+ "[1146]": 51411,
165
+ "[1147]": 51412,
166
+ "[1148]": 51413,
167
+ "[1149]": 51414,
168
+ "[114]": 50379,
169
+ "[1150]": 51415,
170
+ "[1151]": 51416,
171
+ "[1152]": 51417,
172
+ "[1153]": 51418,
173
+ "[1154]": 51419,
174
+ "[1155]": 51420,
175
+ "[1156]": 51421,
176
+ "[1157]": 51422,
177
+ "[1158]": 51423,
178
+ "[1159]": 51424,
179
+ "[115]": 50380,
180
+ "[1160]": 51425,
181
+ "[1161]": 51426,
182
+ "[1162]": 51427,
183
+ "[1163]": 51428,
184
+ "[1164]": 51429,
185
+ "[1165]": 51430,
186
+ "[1166]": 51431,
187
+ "[1167]": 51432,
188
+ "[1168]": 51433,
189
+ "[1169]": 51434,
190
+ "[116]": 50381,
191
+ "[1170]": 51435,
192
+ "[1171]": 51436,
193
+ "[1172]": 51437,
194
+ "[1173]": 51438,
195
+ "[1174]": 51439,
196
+ "[1175]": 51440,
197
+ "[1176]": 51441,
198
+ "[1177]": 51442,
199
+ "[1178]": 51443,
200
+ "[1179]": 51444,
201
+ "[117]": 50382,
202
+ "[1180]": 51445,
203
+ "[1181]": 51446,
204
+ "[1182]": 51447,
205
+ "[1183]": 51448,
206
+ "[1184]": 51449,
207
+ "[1185]": 51450,
208
+ "[1186]": 51451,
209
+ "[1187]": 51452,
210
+ "[1188]": 51453,
211
+ "[1189]": 51454,
212
+ "[118]": 50383,
213
+ "[1190]": 51455,
214
+ "[1191]": 51456,
215
+ "[1192]": 51457,
216
+ "[1193]": 51458,
217
+ "[1194]": 51459,
218
+ "[1195]": 51460,
219
+ "[1196]": 51461,
220
+ "[1197]": 51462,
221
+ "[1198]": 51463,
222
+ "[1199]": 51464,
223
+ "[119]": 50384,
224
+ "[11]": 50276,
225
+ "[1200]": 51465,
226
+ "[1201]": 51466,
227
+ "[1202]": 51467,
228
+ "[1203]": 51468,
229
+ "[1204]": 51469,
230
+ "[1205]": 51470,
231
+ "[1206]": 51471,
232
+ "[1207]": 51472,
233
+ "[1208]": 51473,
234
+ "[1209]": 51474,
235
+ "[120]": 50385,
236
+ "[1210]": 51475,
237
+ "[1211]": 51476,
238
+ "[1212]": 51477,
239
+ "[1213]": 51478,
240
+ "[1214]": 51479,
241
+ "[1215]": 51480,
242
+ "[1216]": 51481,
243
+ "[1217]": 51482,
244
+ "[1218]": 51483,
245
+ "[1219]": 51484,
246
+ "[121]": 50386,
247
+ "[1220]": 51485,
248
+ "[1221]": 51486,
249
+ "[1222]": 51487,
250
+ "[1223]": 51488,
251
+ "[1224]": 51489,
252
+ "[1225]": 51490,
253
+ "[1226]": 51491,
254
+ "[1227]": 51492,
255
+ "[1228]": 51493,
256
+ "[1229]": 51494,
257
+ "[122]": 50387,
258
+ "[1230]": 51495,
259
+ "[1231]": 51496,
260
+ "[1232]": 51497,
261
+ "[1233]": 51498,
262
+ "[1234]": 51499,
263
+ "[1235]": 51500,
264
+ "[1236]": 51501,
265
+ "[1237]": 51502,
266
+ "[1238]": 51503,
267
+ "[1239]": 51504,
268
+ "[123]": 50388,
269
+ "[1240]": 51505,
270
+ "[1241]": 51506,
271
+ "[1242]": 51507,
272
+ "[1243]": 51508,
273
+ "[1244]": 51509,
274
+ "[1245]": 51510,
275
+ "[1246]": 51511,
276
+ "[1247]": 51512,
277
+ "[1248]": 51513,
278
+ "[1249]": 51514,
279
+ "[124]": 50389,
280
+ "[1250]": 51515,
281
+ "[1251]": 51516,
282
+ "[1252]": 51517,
283
+ "[1253]": 51518,
284
+ "[1254]": 51519,
285
+ "[1255]": 51520,
286
+ "[1256]": 51521,
287
+ "[1257]": 51522,
288
+ "[1258]": 51523,
289
+ "[1259]": 51524,
290
+ "[125]": 50390,
291
+ "[1260]": 51525,
292
+ "[1261]": 51526,
293
+ "[1262]": 51527,
294
+ "[1263]": 51528,
295
+ "[1264]": 51529,
296
+ "[1265]": 51530,
297
+ "[1266]": 51531,
298
+ "[1267]": 51532,
299
+ "[1268]": 51533,
300
+ "[1269]": 51534,
301
+ "[126]": 50391,
302
+ "[1270]": 51535,
303
+ "[1271]": 51536,
304
+ "[1272]": 51537,
305
+ "[1273]": 51538,
306
+ "[1274]": 51539,
307
+ "[1275]": 51540,
308
+ "[1276]": 51541,
309
+ "[1277]": 51542,
310
+ "[1278]": 51543,
311
+ "[1279]": 51544,
312
+ "[127]": 50392,
313
+ "[1280]": 51545,
314
+ "[1281]": 51546,
315
+ "[1282]": 51547,
316
+ "[1283]": 51548,
317
+ "[1284]": 51549,
318
+ "[1285]": 51550,
319
+ "[1286]": 51551,
320
+ "[1287]": 51552,
321
+ "[1288]": 51553,
322
+ "[1289]": 51554,
323
+ "[128]": 50393,
324
+ "[1290]": 51555,
325
+ "[1291]": 51556,
326
+ "[1292]": 51557,
327
+ "[1293]": 51558,
328
+ "[1294]": 51559,
329
+ "[1295]": 51560,
330
+ "[1296]": 51561,
331
+ "[1297]": 51562,
332
+ "[1298]": 51563,
333
+ "[1299]": 51564,
334
+ "[129]": 50394,
335
+ "[12]": 50277,
336
+ "[1300]": 51565,
337
+ "[1301]": 51566,
338
+ "[1302]": 51567,
339
+ "[1303]": 51568,
340
+ "[1304]": 51569,
341
+ "[1305]": 51570,
342
+ "[1306]": 51571,
343
+ "[1307]": 51572,
344
+ "[1308]": 51573,
345
+ "[1309]": 51574,
346
+ "[130]": 50395,
347
+ "[1310]": 51575,
348
+ "[1311]": 51576,
349
+ "[1312]": 51577,
350
+ "[1313]": 51578,
351
+ "[1314]": 51579,
352
+ "[1315]": 51580,
353
+ "[1316]": 51581,
354
+ "[1317]": 51582,
355
+ "[1318]": 51583,
356
+ "[1319]": 51584,
357
+ "[131]": 50396,
358
+ "[1320]": 51585,
359
+ "[1321]": 51586,
360
+ "[1322]": 51587,
361
+ "[1323]": 51588,
362
+ "[1324]": 51589,
363
+ "[1325]": 51590,
364
+ "[1326]": 51591,
365
+ "[1327]": 51592,
366
+ "[1328]": 51593,
367
+ "[1329]": 51594,
368
+ "[132]": 50397,
369
+ "[1330]": 51595,
370
+ "[1331]": 51596,
371
+ "[1332]": 51597,
372
+ "[1333]": 51598,
373
+ "[1334]": 51599,
374
+ "[1335]": 51600,
375
+ "[1336]": 51601,
376
+ "[1337]": 51602,
377
+ "[1338]": 51603,
378
+ "[1339]": 51604,
379
+ "[133]": 50398,
380
+ "[1340]": 51605,
381
+ "[1341]": 51606,
382
+ "[1342]": 51607,
383
+ "[1343]": 51608,
384
+ "[1344]": 51609,
385
+ "[1345]": 51610,
386
+ "[1346]": 51611,
387
+ "[1347]": 51612,
388
+ "[1348]": 51613,
389
+ "[1349]": 51614,
390
+ "[134]": 50399,
391
+ "[1350]": 51615,
392
+ "[1351]": 51616,
393
+ "[1352]": 51617,
394
+ "[1353]": 51618,
395
+ "[1354]": 51619,
396
+ "[1355]": 51620,
397
+ "[1356]": 51621,
398
+ "[1357]": 51622,
399
+ "[1358]": 51623,
400
+ "[1359]": 51624,
401
+ "[135]": 50400,
402
+ "[1360]": 51625,
403
+ "[1361]": 51626,
404
+ "[1362]": 51627,
405
+ "[1363]": 51628,
406
+ "[1364]": 51629,
407
+ "[1365]": 51630,
408
+ "[1366]": 51631,
409
+ "[1367]": 51632,
410
+ "[1368]": 51633,
411
+ "[1369]": 51634,
412
+ "[136]": 50401,
413
+ "[1370]": 51635,
414
+ "[1371]": 51636,
415
+ "[1372]": 51637,
416
+ "[1373]": 51638,
417
+ "[1374]": 51639,
418
+ "[1375]": 51640,
419
+ "[1376]": 51641,
420
+ "[1377]": 51642,
421
+ "[1378]": 51643,
422
+ "[1379]": 51644,
423
+ "[137]": 50402,
424
+ "[1380]": 51645,
425
+ "[1381]": 51646,
426
+ "[1382]": 51647,
427
+ "[1383]": 51648,
428
+ "[1384]": 51649,
429
+ "[1385]": 51650,
430
+ "[1386]": 51651,
431
+ "[1387]": 51652,
432
+ "[1388]": 51653,
433
+ "[1389]": 51654,
434
+ "[138]": 50403,
435
+ "[1390]": 51655,
436
+ "[1391]": 51656,
437
+ "[1392]": 51657,
438
+ "[1393]": 51658,
439
+ "[1394]": 51659,
440
+ "[1395]": 51660,
441
+ "[1396]": 51661,
442
+ "[1397]": 51662,
443
+ "[1398]": 51663,
444
+ "[1399]": 51664,
445
+ "[139]": 50404,
446
+ "[13]": 50278,
447
+ "[1400]": 51665,
448
+ "[1401]": 51666,
449
+ "[1402]": 51667,
450
+ "[1403]": 51668,
451
+ "[1404]": 51669,
452
+ "[1405]": 51670,
453
+ "[1406]": 51671,
454
+ "[1407]": 51672,
455
+ "[1408]": 51673,
456
+ "[1409]": 51674,
457
+ "[140]": 50405,
458
+ "[1410]": 51675,
459
+ "[1411]": 51676,
460
+ "[1412]": 51677,
461
+ "[1413]": 51678,
462
+ "[1414]": 51679,
463
+ "[1415]": 51680,
464
+ "[1416]": 51681,
465
+ "[1417]": 51682,
466
+ "[1418]": 51683,
467
+ "[1419]": 51684,
468
+ "[141]": 50406,
469
+ "[1420]": 51685,
470
+ "[1421]": 51686,
471
+ "[1422]": 51687,
472
+ "[1423]": 51688,
473
+ "[1424]": 51689,
474
+ "[1425]": 51690,
475
+ "[1426]": 51691,
476
+ "[1427]": 51692,
477
+ "[1428]": 51693,
478
+ "[1429]": 51694,
479
+ "[142]": 50407,
480
+ "[1430]": 51695,
481
+ "[1431]": 51696,
482
+ "[1432]": 51697,
483
+ "[1433]": 51698,
484
+ "[1434]": 51699,
485
+ "[1435]": 51700,
486
+ "[1436]": 51701,
487
+ "[1437]": 51702,
488
+ "[1438]": 51703,
489
+ "[1439]": 51704,
490
+ "[143]": 50408,
491
+ "[1440]": 51705,
492
+ "[1441]": 51706,
493
+ "[1442]": 51707,
494
+ "[1443]": 51708,
495
+ "[1444]": 51709,
496
+ "[1445]": 51710,
497
+ "[1446]": 51711,
498
+ "[1447]": 51712,
499
+ "[1448]": 51713,
500
+ "[1449]": 51714,
501
+ "[144]": 50409,
502
+ "[1450]": 51715,
503
+ "[1451]": 51716,
504
+ "[1452]": 51717,
505
+ "[1453]": 51718,
506
+ "[1454]": 51719,
507
+ "[1455]": 51720,
508
+ "[1456]": 51721,
509
+ "[1457]": 51722,
510
+ "[1458]": 51723,
511
+ "[1459]": 51724,
512
+ "[145]": 50410,
513
+ "[1460]": 51725,
514
+ "[1461]": 51726,
515
+ "[1462]": 51727,
516
+ "[1463]": 51728,
517
+ "[1464]": 51729,
518
+ "[1465]": 51730,
519
+ "[1466]": 51731,
520
+ "[1467]": 51732,
521
+ "[1468]": 51733,
522
+ "[1469]": 51734,
523
+ "[146]": 50411,
524
+ "[1470]": 51735,
525
+ "[1471]": 51736,
526
+ "[1472]": 51737,
527
+ "[1473]": 51738,
528
+ "[1474]": 51739,
529
+ "[1475]": 51740,
530
+ "[1476]": 51741,
531
+ "[1477]": 51742,
532
+ "[1478]": 51743,
533
+ "[1479]": 51744,
534
+ "[147]": 50412,
535
+ "[1480]": 51745,
536
+ "[1481]": 51746,
537
+ "[1482]": 51747,
538
+ "[1483]": 51748,
539
+ "[1484]": 51749,
540
+ "[1485]": 51750,
541
+ "[1486]": 51751,
542
+ "[1487]": 51752,
543
+ "[1488]": 51753,
544
+ "[1489]": 51754,
545
+ "[148]": 50413,
546
+ "[1490]": 51755,
547
+ "[1491]": 51756,
548
+ "[1492]": 51757,
549
+ "[1493]": 51758,
550
+ "[1494]": 51759,
551
+ "[1495]": 51760,
552
+ "[1496]": 51761,
553
+ "[1497]": 51762,
554
+ "[1498]": 51763,
555
+ "[1499]": 51764,
556
+ "[149]": 50414,
557
+ "[14]": 50279,
558
+ "[1500]": 51765,
559
+ "[1501]": 51766,
560
+ "[1502]": 51767,
561
+ "[1503]": 51768,
562
+ "[1504]": 51769,
563
+ "[1505]": 51770,
564
+ "[1506]": 51771,
565
+ "[1507]": 51772,
566
+ "[1508]": 51773,
567
+ "[1509]": 51774,
568
+ "[150]": 50415,
569
+ "[1510]": 51775,
570
+ "[1511]": 51776,
571
+ "[1512]": 51777,
572
+ "[1513]": 51778,
573
+ "[1514]": 51779,
574
+ "[1515]": 51780,
575
+ "[1516]": 51781,
576
+ "[1517]": 51782,
577
+ "[1518]": 51783,
578
+ "[1519]": 51784,
579
+ "[151]": 50416,
580
+ "[1520]": 51785,
581
+ "[1521]": 51786,
582
+ "[1522]": 51787,
583
+ "[1523]": 51788,
584
+ "[1524]": 51789,
585
+ "[1525]": 51790,
586
+ "[1526]": 51791,
587
+ "[1527]": 51792,
588
+ "[1528]": 51793,
589
+ "[1529]": 51794,
590
+ "[152]": 50417,
591
+ "[1530]": 51795,
592
+ "[1531]": 51796,
593
+ "[1532]": 51797,
594
+ "[1533]": 51798,
595
+ "[1534]": 51799,
596
+ "[1535]": 51800,
597
+ "[1536]": 51801,
598
+ "[1537]": 51802,
599
+ "[1538]": 51803,
600
+ "[1539]": 51804,
601
+ "[153]": 50418,
602
+ "[1540]": 51805,
603
+ "[1541]": 51806,
604
+ "[1542]": 51807,
605
+ "[1543]": 51808,
606
+ "[1544]": 51809,
607
+ "[1545]": 51810,
608
+ "[1546]": 51811,
609
+ "[1547]": 51812,
610
+ "[1548]": 51813,
611
+ "[1549]": 51814,
612
+ "[154]": 50419,
613
+ "[1550]": 51815,
614
+ "[1551]": 51816,
615
+ "[1552]": 51817,
616
+ "[1553]": 51818,
617
+ "[1554]": 51819,
618
+ "[1555]": 51820,
619
+ "[1556]": 51821,
620
+ "[1557]": 51822,
621
+ "[1558]": 51823,
622
+ "[1559]": 51824,
623
+ "[155]": 50420,
624
+ "[1560]": 51825,
625
+ "[1561]": 51826,
626
+ "[1562]": 51827,
627
+ "[1563]": 51828,
628
+ "[1564]": 51829,
629
+ "[1565]": 51830,
630
+ "[1566]": 51831,
631
+ "[1567]": 51832,
632
+ "[1568]": 51833,
633
+ "[1569]": 51834,
634
+ "[156]": 50421,
635
+ "[1570]": 51835,
636
+ "[1571]": 51836,
637
+ "[1572]": 51837,
638
+ "[1573]": 51838,
639
+ "[1574]": 51839,
640
+ "[1575]": 51840,
641
+ "[1576]": 51841,
642
+ "[1577]": 51842,
643
+ "[1578]": 51843,
644
+ "[1579]": 51844,
645
+ "[157]": 50422,
646
+ "[1580]": 51845,
647
+ "[1581]": 51846,
648
+ "[1582]": 51847,
649
+ "[1583]": 51848,
650
+ "[1584]": 51849,
651
+ "[1585]": 51850,
652
+ "[1586]": 51851,
653
+ "[1587]": 51852,
654
+ "[1588]": 51853,
655
+ "[1589]": 51854,
656
+ "[158]": 50423,
657
+ "[1590]": 51855,
658
+ "[1591]": 51856,
659
+ "[1592]": 51857,
660
+ "[1593]": 51858,
661
+ "[1594]": 51859,
662
+ "[1595]": 51860,
663
+ "[1596]": 51861,
664
+ "[1597]": 51862,
665
+ "[1598]": 51863,
666
+ "[1599]": 51864,
667
+ "[159]": 50424,
668
+ "[15]": 50280,
669
+ "[1600]": 51865,
670
+ "[1601]": 51866,
671
+ "[1602]": 51867,
672
+ "[1603]": 51868,
673
+ "[1604]": 51869,
674
+ "[1605]": 51870,
675
+ "[1606]": 51871,
676
+ "[1607]": 51872,
677
+ "[1608]": 51873,
678
+ "[1609]": 51874,
679
+ "[160]": 50425,
680
+ "[1610]": 51875,
681
+ "[1611]": 51876,
682
+ "[1612]": 51877,
683
+ "[1613]": 51878,
684
+ "[1614]": 51879,
685
+ "[1615]": 51880,
686
+ "[1616]": 51881,
687
+ "[1617]": 51882,
688
+ "[1618]": 51883,
689
+ "[1619]": 51884,
690
+ "[161]": 50426,
691
+ "[1620]": 51885,
692
+ "[1621]": 51886,
693
+ "[1622]": 51887,
694
+ "[1623]": 51888,
695
+ "[1624]": 51889,
696
+ "[1625]": 51890,
697
+ "[1626]": 51891,
698
+ "[1627]": 51892,
699
+ "[1628]": 51893,
700
+ "[1629]": 51894,
701
+ "[162]": 50427,
702
+ "[1630]": 51895,
703
+ "[1631]": 51896,
704
+ "[1632]": 51897,
705
+ "[1633]": 51898,
706
+ "[1634]": 51899,
707
+ "[1635]": 51900,
708
+ "[1636]": 51901,
709
+ "[1637]": 51902,
710
+ "[1638]": 51903,
711
+ "[1639]": 51904,
712
+ "[163]": 50428,
713
+ "[1640]": 51905,
714
+ "[1641]": 51906,
715
+ "[1642]": 51907,
716
+ "[1643]": 51908,
717
+ "[1644]": 51909,
718
+ "[1645]": 51910,
719
+ "[1646]": 51911,
720
+ "[1647]": 51912,
721
+ "[1648]": 51913,
722
+ "[1649]": 51914,
723
+ "[164]": 50429,
724
+ "[1650]": 51915,
725
+ "[1651]": 51916,
726
+ "[1652]": 51917,
727
+ "[1653]": 51918,
728
+ "[1654]": 51919,
729
+ "[1655]": 51920,
730
+ "[1656]": 51921,
731
+ "[1657]": 51922,
732
+ "[1658]": 51923,
733
+ "[1659]": 51924,
734
+ "[165]": 50430,
735
+ "[1660]": 51925,
736
+ "[1661]": 51926,
737
+ "[1662]": 51927,
738
+ "[1663]": 51928,
739
+ "[1664]": 51929,
740
+ "[1665]": 51930,
741
+ "[1666]": 51931,
742
+ "[1667]": 51932,
743
+ "[1668]": 51933,
744
+ "[1669]": 51934,
745
+ "[166]": 50431,
746
+ "[1670]": 51935,
747
+ "[1671]": 51936,
748
+ "[1672]": 51937,
749
+ "[1673]": 51938,
750
+ "[1674]": 51939,
751
+ "[1675]": 51940,
752
+ "[1676]": 51941,
753
+ "[1677]": 51942,
754
+ "[1678]": 51943,
755
+ "[1679]": 51944,
756
+ "[167]": 50432,
757
+ "[1680]": 51945,
758
+ "[1681]": 51946,
759
+ "[1682]": 51947,
760
+ "[1683]": 51948,
761
+ "[1684]": 51949,
762
+ "[1685]": 51950,
763
+ "[1686]": 51951,
764
+ "[1687]": 51952,
765
+ "[1688]": 51953,
766
+ "[1689]": 51954,
767
+ "[168]": 50433,
768
+ "[1690]": 51955,
769
+ "[1691]": 51956,
770
+ "[1692]": 51957,
771
+ "[1693]": 51958,
772
+ "[1694]": 51959,
773
+ "[1695]": 51960,
774
+ "[1696]": 51961,
775
+ "[1697]": 51962,
776
+ "[1698]": 51963,
777
+ "[1699]": 51964,
778
+ "[169]": 50434,
779
+ "[16]": 50281,
780
+ "[1700]": 51965,
781
+ "[1701]": 51966,
782
+ "[1702]": 51967,
783
+ "[1703]": 51968,
784
+ "[1704]": 51969,
785
+ "[1705]": 51970,
786
+ "[1706]": 51971,
787
+ "[1707]": 51972,
788
+ "[1708]": 51973,
789
+ "[1709]": 51974,
790
+ "[170]": 50435,
791
+ "[1710]": 51975,
792
+ "[1711]": 51976,
793
+ "[1712]": 51977,
794
+ "[1713]": 51978,
795
+ "[1714]": 51979,
796
+ "[1715]": 51980,
797
+ "[1716]": 51981,
798
+ "[1717]": 51982,
799
+ "[1718]": 51983,
800
+ "[1719]": 51984,
801
+ "[171]": 50436,
802
+ "[1720]": 51985,
803
+ "[1721]": 51986,
804
+ "[1722]": 51987,
805
+ "[1723]": 51988,
806
+ "[1724]": 51989,
807
+ "[1725]": 51990,
808
+ "[1726]": 51991,
809
+ "[1727]": 51992,
810
+ "[1728]": 51993,
811
+ "[1729]": 51994,
812
+ "[172]": 50437,
813
+ "[1730]": 51995,
814
+ "[1731]": 51996,
815
+ "[1732]": 51997,
816
+ "[1733]": 51998,
817
+ "[1734]": 51999,
818
+ "[1735]": 52000,
819
+ "[1736]": 52001,
820
+ "[1737]": 52002,
821
+ "[1738]": 52003,
822
+ "[1739]": 52004,
823
+ "[173]": 50438,
824
+ "[1740]": 52005,
825
+ "[1741]": 52006,
826
+ "[1742]": 52007,
827
+ "[1743]": 52008,
828
+ "[1744]": 52009,
829
+ "[1745]": 52010,
830
+ "[1746]": 52011,
831
+ "[1747]": 52012,
832
+ "[1748]": 52013,
833
+ "[1749]": 52014,
834
+ "[174]": 50439,
835
+ "[1750]": 52015,
836
+ "[1751]": 52016,
837
+ "[1752]": 52017,
838
+ "[1753]": 52018,
839
+ "[1754]": 52019,
840
+ "[1755]": 52020,
841
+ "[1756]": 52021,
842
+ "[1757]": 52022,
843
+ "[1758]": 52023,
844
+ "[1759]": 52024,
845
+ "[175]": 50440,
846
+ "[1760]": 52025,
847
+ "[1761]": 52026,
848
+ "[1762]": 52027,
849
+ "[1763]": 52028,
850
+ "[1764]": 52029,
851
+ "[1765]": 52030,
852
+ "[1766]": 52031,
853
+ "[1767]": 52032,
854
+ "[1768]": 52033,
855
+ "[1769]": 52034,
856
+ "[176]": 50441,
857
+ "[1770]": 52035,
858
+ "[1771]": 52036,
859
+ "[1772]": 52037,
860
+ "[1773]": 52038,
861
+ "[1774]": 52039,
862
+ "[1775]": 52040,
863
+ "[1776]": 52041,
864
+ "[1777]": 52042,
865
+ "[1778]": 52043,
866
+ "[1779]": 52044,
867
+ "[177]": 50442,
868
+ "[1780]": 52045,
869
+ "[1781]": 52046,
870
+ "[1782]": 52047,
871
+ "[1783]": 52048,
872
+ "[1784]": 52049,
873
+ "[1785]": 52050,
874
+ "[1786]": 52051,
875
+ "[1787]": 52052,
876
+ "[1788]": 52053,
877
+ "[1789]": 52054,
878
+ "[178]": 50443,
879
+ "[1790]": 52055,
880
+ "[1791]": 52056,
881
+ "[1792]": 52057,
882
+ "[1793]": 52058,
883
+ "[1794]": 52059,
884
+ "[1795]": 52060,
885
+ "[1796]": 52061,
886
+ "[1797]": 52062,
887
+ "[1798]": 52063,
888
+ "[1799]": 52064,
889
+ "[179]": 50444,
890
+ "[17]": 50282,
891
+ "[1800]": 52065,
892
+ "[1801]": 52066,
893
+ "[1802]": 52067,
894
+ "[1803]": 52068,
895
+ "[1804]": 52069,
896
+ "[1805]": 52070,
897
+ "[1806]": 52071,
898
+ "[1807]": 52072,
899
+ "[1808]": 52073,
900
+ "[1809]": 52074,
901
+ "[180]": 50445,
902
+ "[1810]": 52075,
903
+ "[1811]": 52076,
904
+ "[1812]": 52077,
905
+ "[1813]": 52078,
906
+ "[1814]": 52079,
907
+ "[1815]": 52080,
908
+ "[1816]": 52081,
909
+ "[1817]": 52082,
910
+ "[1818]": 52083,
911
+ "[1819]": 52084,
912
+ "[181]": 50446,
913
+ "[1820]": 52085,
914
+ "[1821]": 52086,
915
+ "[1822]": 52087,
916
+ "[1823]": 52088,
917
+ "[1824]": 52089,
918
+ "[1825]": 52090,
919
+ "[1826]": 52091,
920
+ "[1827]": 52092,
921
+ "[1828]": 52093,
922
+ "[1829]": 52094,
923
+ "[182]": 50447,
924
+ "[1830]": 52095,
925
+ "[1831]": 52096,
926
+ "[1832]": 52097,
927
+ "[1833]": 52098,
928
+ "[1834]": 52099,
929
+ "[1835]": 52100,
930
+ "[1836]": 52101,
931
+ "[1837]": 52102,
932
+ "[1838]": 52103,
933
+ "[1839]": 52104,
934
+ "[183]": 50448,
935
+ "[1840]": 52105,
936
+ "[1841]": 52106,
937
+ "[1842]": 52107,
938
+ "[1843]": 52108,
939
+ "[1844]": 52109,
940
+ "[1845]": 52110,
941
+ "[1846]": 52111,
942
+ "[1847]": 52112,
943
+ "[1848]": 52113,
944
+ "[1849]": 52114,
945
+ "[184]": 50449,
946
+ "[1850]": 52115,
947
+ "[1851]": 52116,
948
+ "[1852]": 52117,
949
+ "[1853]": 52118,
950
+ "[1854]": 52119,
951
+ "[1855]": 52120,
952
+ "[1856]": 52121,
953
+ "[1857]": 52122,
954
+ "[1858]": 52123,
955
+ "[1859]": 52124,
956
+ "[185]": 50450,
957
+ "[1860]": 52125,
958
+ "[1861]": 52126,
959
+ "[1862]": 52127,
960
+ "[1863]": 52128,
961
+ "[1864]": 52129,
962
+ "[1865]": 52130,
963
+ "[1866]": 52131,
964
+ "[1867]": 52132,
965
+ "[1868]": 52133,
966
+ "[1869]": 52134,
967
+ "[186]": 50451,
968
+ "[1870]": 52135,
969
+ "[1871]": 52136,
970
+ "[1872]": 52137,
971
+ "[1873]": 52138,
972
+ "[1874]": 52139,
973
+ "[1875]": 52140,
974
+ "[1876]": 52141,
975
+ "[1877]": 52142,
976
+ "[1878]": 52143,
977
+ "[1879]": 52144,
978
+ "[187]": 50452,
979
+ "[1880]": 52145,
980
+ "[1881]": 52146,
981
+ "[1882]": 52147,
982
+ "[1883]": 52148,
983
+ "[1884]": 52149,
984
+ "[1885]": 52150,
985
+ "[1886]": 52151,
986
+ "[1887]": 52152,
987
+ "[1888]": 52153,
988
+ "[1889]": 52154,
989
+ "[188]": 50453,
990
+ "[1890]": 52155,
991
+ "[1891]": 52156,
992
+ "[1892]": 52157,
993
+ "[1893]": 52158,
994
+ "[1894]": 52159,
995
+ "[1895]": 52160,
996
+ "[1896]": 52161,
997
+ "[1897]": 52162,
998
+ "[1898]": 52163,
999
+ "[1899]": 52164,
1000
+ "[189]": 50454,
1001
+ "[18]": 50283,
1002
+ "[1900]": 52165,
1003
+ "[1901]": 52166,
1004
+ "[1902]": 52167,
1005
+ "[1903]": 52168,
1006
+ "[1904]": 52169,
1007
+ "[1905]": 52170,
1008
+ "[1906]": 52171,
1009
+ "[1907]": 52172,
1010
+ "[1908]": 52173,
1011
+ "[1909]": 52174,
1012
+ "[190]": 50455,
1013
+ "[1910]": 52175,
1014
+ "[1911]": 52176,
1015
+ "[1912]": 52177,
1016
+ "[1913]": 52178,
1017
+ "[1914]": 52179,
1018
+ "[1915]": 52180,
1019
+ "[1916]": 52181,
1020
+ "[1917]": 52182,
1021
+ "[1918]": 52183,
1022
+ "[1919]": 52184,
1023
+ "[191]": 50456,
1024
+ "[1920]": 52185,
1025
+ "[1921]": 52186,
1026
+ "[1922]": 52187,
1027
+ "[1923]": 52188,
1028
+ "[1924]": 52189,
1029
+ "[1925]": 52190,
1030
+ "[1926]": 52191,
1031
+ "[1927]": 52192,
1032
+ "[1928]": 52193,
1033
+ "[1929]": 52194,
1034
+ "[192]": 50457,
1035
+ "[1930]": 52195,
1036
+ "[1931]": 52196,
1037
+ "[1932]": 52197,
1038
+ "[1933]": 52198,
1039
+ "[1934]": 52199,
1040
+ "[1935]": 52200,
1041
+ "[1936]": 52201,
1042
+ "[1937]": 52202,
1043
+ "[1938]": 52203,
1044
+ "[1939]": 52204,
1045
+ "[193]": 50458,
1046
+ "[1940]": 52205,
1047
+ "[1941]": 52206,
1048
+ "[1942]": 52207,
1049
+ "[1943]": 52208,
1050
+ "[1944]": 52209,
1051
+ "[1945]": 52210,
1052
+ "[1946]": 52211,
1053
+ "[1947]": 52212,
1054
+ "[1948]": 52213,
1055
+ "[1949]": 52214,
1056
+ "[194]": 50459,
1057
+ "[1950]": 52215,
1058
+ "[1951]": 52216,
1059
+ "[1952]": 52217,
1060
+ "[1953]": 52218,
1061
+ "[1954]": 52219,
1062
+ "[1955]": 52220,
1063
+ "[1956]": 52221,
1064
+ "[1957]": 52222,
1065
+ "[1958]": 52223,
1066
+ "[1959]": 52224,
1067
+ "[195]": 50460,
1068
+ "[1960]": 52225,
1069
+ "[1961]": 52226,
1070
+ "[1962]": 52227,
1071
+ "[1963]": 52228,
1072
+ "[1964]": 52229,
1073
+ "[1965]": 52230,
1074
+ "[1966]": 52231,
1075
+ "[1967]": 52232,
1076
+ "[1968]": 52233,
1077
+ "[1969]": 52234,
1078
+ "[196]": 50461,
1079
+ "[1970]": 52235,
1080
+ "[1971]": 52236,
1081
+ "[1972]": 52237,
1082
+ "[1973]": 52238,
1083
+ "[1974]": 52239,
1084
+ "[1975]": 52240,
1085
+ "[1976]": 52241,
1086
+ "[1977]": 52242,
1087
+ "[1978]": 52243,
1088
+ "[1979]": 52244,
1089
+ "[197]": 50462,
1090
+ "[1980]": 52245,
1091
+ "[1981]": 52246,
1092
+ "[1982]": 52247,
1093
+ "[1983]": 52248,
1094
+ "[1984]": 52249,
1095
+ "[1985]": 52250,
1096
+ "[1986]": 52251,
1097
+ "[1987]": 52252,
1098
+ "[1988]": 52253,
1099
+ "[1989]": 52254,
1100
+ "[198]": 50463,
1101
+ "[1990]": 52255,
1102
+ "[1991]": 52256,
1103
+ "[1992]": 52257,
1104
+ "[1993]": 52258,
1105
+ "[1994]": 52259,
1106
+ "[1995]": 52260,
1107
+ "[1996]": 52261,
1108
+ "[1997]": 52262,
1109
+ "[1998]": 52263,
1110
+ "[1999]": 52264,
1111
+ "[199]": 50464,
1112
+ "[19]": 50284,
1113
+ "[1]": 50266,
1114
+ "[2000]": 52265,
1115
+ "[2001]": 52266,
1116
+ "[2002]": 52267,
1117
+ "[2003]": 52268,
1118
+ "[2004]": 52269,
1119
+ "[2005]": 52270,
1120
+ "[2006]": 52271,
1121
+ "[2007]": 52272,
1122
+ "[2008]": 52273,
1123
+ "[2009]": 52274,
1124
+ "[200]": 50465,
1125
+ "[2010]": 52275,
1126
+ "[2011]": 52276,
1127
+ "[2012]": 52277,
1128
+ "[2013]": 52278,
1129
+ "[2014]": 52279,
1130
+ "[2015]": 52280,
1131
+ "[2016]": 52281,
1132
+ "[2017]": 52282,
1133
+ "[2018]": 52283,
1134
+ "[2019]": 52284,
1135
+ "[201]": 50466,
1136
+ "[2020]": 52285,
1137
+ "[2021]": 52286,
1138
+ "[2022]": 52287,
1139
+ "[2023]": 52288,
1140
+ "[2024]": 52289,
1141
+ "[2025]": 52290,
1142
+ "[2026]": 52291,
1143
+ "[2027]": 52292,
1144
+ "[2028]": 52293,
1145
+ "[2029]": 52294,
1146
+ "[202]": 50467,
1147
+ "[2030]": 52295,
1148
+ "[2031]": 52296,
1149
+ "[2032]": 52297,
1150
+ "[2033]": 52298,
1151
+ "[2034]": 52299,
1152
+ "[2035]": 52300,
1153
+ "[2036]": 52301,
1154
+ "[2037]": 52302,
1155
+ "[2038]": 52303,
1156
+ "[2039]": 52304,
1157
+ "[203]": 50468,
1158
+ "[2040]": 52305,
1159
+ "[2041]": 52306,
1160
+ "[2042]": 52307,
1161
+ "[2043]": 52308,
1162
+ "[2044]": 52309,
1163
+ "[2045]": 52310,
1164
+ "[2046]": 52311,
1165
+ "[2047]": 52312,
1166
+ "[204]": 50469,
1167
+ "[205]": 50470,
1168
+ "[206]": 50471,
1169
+ "[207]": 50472,
1170
+ "[208]": 50473,
1171
+ "[209]": 50474,
1172
+ "[20]": 50285,
1173
+ "[210]": 50475,
1174
+ "[211]": 50476,
1175
+ "[212]": 50477,
1176
+ "[213]": 50478,
1177
+ "[214]": 50479,
1178
+ "[215]": 50480,
1179
+ "[216]": 50481,
1180
+ "[217]": 50482,
1181
+ "[218]": 50483,
1182
+ "[219]": 50484,
1183
+ "[21]": 50286,
1184
+ "[220]": 50485,
1185
+ "[221]": 50486,
1186
+ "[222]": 50487,
1187
+ "[223]": 50488,
1188
+ "[224]": 50489,
1189
+ "[225]": 50490,
1190
+ "[226]": 50491,
1191
+ "[227]": 50492,
1192
+ "[228]": 50493,
1193
+ "[229]": 50494,
1194
+ "[22]": 50287,
1195
+ "[230]": 50495,
1196
+ "[231]": 50496,
1197
+ "[232]": 50497,
1198
+ "[233]": 50498,
1199
+ "[234]": 50499,
1200
+ "[235]": 50500,
1201
+ "[236]": 50501,
1202
+ "[237]": 50502,
1203
+ "[238]": 50503,
1204
+ "[239]": 50504,
1205
+ "[23]": 50288,
1206
+ "[240]": 50505,
1207
+ "[241]": 50506,
1208
+ "[242]": 50507,
1209
+ "[243]": 50508,
1210
+ "[244]": 50509,
1211
+ "[245]": 50510,
1212
+ "[246]": 50511,
1213
+ "[247]": 50512,
1214
+ "[248]": 50513,
1215
+ "[249]": 50514,
1216
+ "[24]": 50289,
1217
+ "[250]": 50515,
1218
+ "[251]": 50516,
1219
+ "[252]": 50517,
1220
+ "[253]": 50518,
1221
+ "[254]": 50519,
1222
+ "[255]": 50520,
1223
+ "[256]": 50521,
1224
+ "[257]": 50522,
1225
+ "[258]": 50523,
1226
+ "[259]": 50524,
1227
+ "[25]": 50290,
1228
+ "[260]": 50525,
1229
+ "[261]": 50526,
1230
+ "[262]": 50527,
1231
+ "[263]": 50528,
1232
+ "[264]": 50529,
1233
+ "[265]": 50530,
1234
+ "[266]": 50531,
1235
+ "[267]": 50532,
1236
+ "[268]": 50533,
1237
+ "[269]": 50534,
1238
+ "[26]": 50291,
1239
+ "[270]": 50535,
1240
+ "[271]": 50536,
1241
+ "[272]": 50537,
1242
+ "[273]": 50538,
1243
+ "[274]": 50539,
1244
+ "[275]": 50540,
1245
+ "[276]": 50541,
1246
+ "[277]": 50542,
1247
+ "[278]": 50543,
1248
+ "[279]": 50544,
1249
+ "[27]": 50292,
1250
+ "[280]": 50545,
1251
+ "[281]": 50546,
1252
+ "[282]": 50547,
1253
+ "[283]": 50548,
1254
+ "[284]": 50549,
1255
+ "[285]": 50550,
1256
+ "[286]": 50551,
1257
+ "[287]": 50552,
1258
+ "[288]": 50553,
1259
+ "[289]": 50554,
1260
+ "[28]": 50293,
1261
+ "[290]": 50555,
1262
+ "[291]": 50556,
1263
+ "[292]": 50557,
1264
+ "[293]": 50558,
1265
+ "[294]": 50559,
1266
+ "[295]": 50560,
1267
+ "[296]": 50561,
1268
+ "[297]": 50562,
1269
+ "[298]": 50563,
1270
+ "[299]": 50564,
1271
+ "[29]": 50294,
1272
+ "[2]": 50267,
1273
+ "[300]": 50565,
1274
+ "[301]": 50566,
1275
+ "[302]": 50567,
1276
+ "[303]": 50568,
1277
+ "[304]": 50569,
1278
+ "[305]": 50570,
1279
+ "[306]": 50571,
1280
+ "[307]": 50572,
1281
+ "[308]": 50573,
1282
+ "[309]": 50574,
1283
+ "[30]": 50295,
1284
+ "[310]": 50575,
1285
+ "[311]": 50576,
1286
+ "[312]": 50577,
1287
+ "[313]": 50578,
1288
+ "[314]": 50579,
1289
+ "[315]": 50580,
1290
+ "[316]": 50581,
1291
+ "[317]": 50582,
1292
+ "[318]": 50583,
1293
+ "[319]": 50584,
1294
+ "[31]": 50296,
1295
+ "[320]": 50585,
1296
+ "[321]": 50586,
1297
+ "[322]": 50587,
1298
+ "[323]": 50588,
1299
+ "[324]": 50589,
1300
+ "[325]": 50590,
1301
+ "[326]": 50591,
1302
+ "[327]": 50592,
1303
+ "[328]": 50593,
1304
+ "[329]": 50594,
1305
+ "[32]": 50297,
1306
+ "[330]": 50595,
1307
+ "[331]": 50596,
1308
+ "[332]": 50597,
1309
+ "[333]": 50598,
1310
+ "[334]": 50599,
1311
+ "[335]": 50600,
1312
+ "[336]": 50601,
1313
+ "[337]": 50602,
1314
+ "[338]": 50603,
1315
+ "[339]": 50604,
1316
+ "[33]": 50298,
1317
+ "[340]": 50605,
1318
+ "[341]": 50606,
1319
+ "[342]": 50607,
1320
+ "[343]": 50608,
1321
+ "[344]": 50609,
1322
+ "[345]": 50610,
1323
+ "[346]": 50611,
1324
+ "[347]": 50612,
1325
+ "[348]": 50613,
1326
+ "[349]": 50614,
1327
+ "[34]": 50299,
1328
+ "[350]": 50615,
1329
+ "[351]": 50616,
1330
+ "[352]": 50617,
1331
+ "[353]": 50618,
1332
+ "[354]": 50619,
1333
+ "[355]": 50620,
1334
+ "[356]": 50621,
1335
+ "[357]": 50622,
1336
+ "[358]": 50623,
1337
+ "[359]": 50624,
1338
+ "[35]": 50300,
1339
+ "[360]": 50625,
1340
+ "[361]": 50626,
1341
+ "[362]": 50627,
1342
+ "[363]": 50628,
1343
+ "[364]": 50629,
1344
+ "[365]": 50630,
1345
+ "[366]": 50631,
1346
+ "[367]": 50632,
1347
+ "[368]": 50633,
1348
+ "[369]": 50634,
1349
+ "[36]": 50301,
1350
+ "[370]": 50635,
1351
+ "[371]": 50636,
1352
+ "[372]": 50637,
1353
+ "[373]": 50638,
1354
+ "[374]": 50639,
1355
+ "[375]": 50640,
1356
+ "[376]": 50641,
1357
+ "[377]": 50642,
1358
+ "[378]": 50643,
1359
+ "[379]": 50644,
1360
+ "[37]": 50302,
1361
+ "[380]": 50645,
1362
+ "[381]": 50646,
1363
+ "[382]": 50647,
1364
+ "[383]": 50648,
1365
+ "[384]": 50649,
1366
+ "[385]": 50650,
1367
+ "[386]": 50651,
1368
+ "[387]": 50652,
1369
+ "[388]": 50653,
1370
+ "[389]": 50654,
1371
+ "[38]": 50303,
1372
+ "[390]": 50655,
1373
+ "[391]": 50656,
1374
+ "[392]": 50657,
1375
+ "[393]": 50658,
1376
+ "[394]": 50659,
1377
+ "[395]": 50660,
1378
+ "[396]": 50661,
1379
+ "[397]": 50662,
1380
+ "[398]": 50663,
1381
+ "[399]": 50664,
1382
+ "[39]": 50304,
1383
+ "[3]": 50268,
1384
+ "[400]": 50665,
1385
+ "[401]": 50666,
1386
+ "[402]": 50667,
1387
+ "[403]": 50668,
1388
+ "[404]": 50669,
1389
+ "[405]": 50670,
1390
+ "[406]": 50671,
1391
+ "[407]": 50672,
1392
+ "[408]": 50673,
1393
+ "[409]": 50674,
1394
+ "[40]": 50305,
1395
+ "[410]": 50675,
1396
+ "[411]": 50676,
1397
+ "[412]": 50677,
1398
+ "[413]": 50678,
1399
+ "[414]": 50679,
1400
+ "[415]": 50680,
1401
+ "[416]": 50681,
1402
+ "[417]": 50682,
1403
+ "[418]": 50683,
1404
+ "[419]": 50684,
1405
+ "[41]": 50306,
1406
+ "[420]": 50685,
1407
+ "[421]": 50686,
1408
+ "[422]": 50687,
1409
+ "[423]": 50688,
1410
+ "[424]": 50689,
1411
+ "[425]": 50690,
1412
+ "[426]": 50691,
1413
+ "[427]": 50692,
1414
+ "[428]": 50693,
1415
+ "[429]": 50694,
1416
+ "[42]": 50307,
1417
+ "[430]": 50695,
1418
+ "[431]": 50696,
1419
+ "[432]": 50697,
1420
+ "[433]": 50698,
1421
+ "[434]": 50699,
1422
+ "[435]": 50700,
1423
+ "[436]": 50701,
1424
+ "[437]": 50702,
1425
+ "[438]": 50703,
1426
+ "[439]": 50704,
1427
+ "[43]": 50308,
1428
+ "[440]": 50705,
1429
+ "[441]": 50706,
1430
+ "[442]": 50707,
1431
+ "[443]": 50708,
1432
+ "[444]": 50709,
1433
+ "[445]": 50710,
1434
+ "[446]": 50711,
1435
+ "[447]": 50712,
1436
+ "[448]": 50713,
1437
+ "[449]": 50714,
1438
+ "[44]": 50309,
1439
+ "[450]": 50715,
1440
+ "[451]": 50716,
1441
+ "[452]": 50717,
1442
+ "[453]": 50718,
1443
+ "[454]": 50719,
1444
+ "[455]": 50720,
1445
+ "[456]": 50721,
1446
+ "[457]": 50722,
1447
+ "[458]": 50723,
1448
+ "[459]": 50724,
1449
+ "[45]": 50310,
1450
+ "[460]": 50725,
1451
+ "[461]": 50726,
1452
+ "[462]": 50727,
1453
+ "[463]": 50728,
1454
+ "[464]": 50729,
1455
+ "[465]": 50730,
1456
+ "[466]": 50731,
1457
+ "[467]": 50732,
1458
+ "[468]": 50733,
1459
+ "[469]": 50734,
1460
+ "[46]": 50311,
1461
+ "[470]": 50735,
1462
+ "[471]": 50736,
1463
+ "[472]": 50737,
1464
+ "[473]": 50738,
1465
+ "[474]": 50739,
1466
+ "[475]": 50740,
1467
+ "[476]": 50741,
1468
+ "[477]": 50742,
1469
+ "[478]": 50743,
1470
+ "[479]": 50744,
1471
+ "[47]": 50312,
1472
+ "[480]": 50745,
1473
+ "[481]": 50746,
1474
+ "[482]": 50747,
1475
+ "[483]": 50748,
1476
+ "[484]": 50749,
1477
+ "[485]": 50750,
1478
+ "[486]": 50751,
1479
+ "[487]": 50752,
1480
+ "[488]": 50753,
1481
+ "[489]": 50754,
1482
+ "[48]": 50313,
1483
+ "[490]": 50755,
1484
+ "[491]": 50756,
1485
+ "[492]": 50757,
1486
+ "[493]": 50758,
1487
+ "[494]": 50759,
1488
+ "[495]": 50760,
1489
+ "[496]": 50761,
1490
+ "[497]": 50762,
1491
+ "[498]": 50763,
1492
+ "[499]": 50764,
1493
+ "[49]": 50314,
1494
+ "[4]": 50269,
1495
+ "[500]": 50765,
1496
+ "[501]": 50766,
1497
+ "[502]": 50767,
1498
+ "[503]": 50768,
1499
+ "[504]": 50769,
1500
+ "[505]": 50770,
1501
+ "[506]": 50771,
1502
+ "[507]": 50772,
1503
+ "[508]": 50773,
1504
+ "[509]": 50774,
1505
+ "[50]": 50315,
1506
+ "[510]": 50775,
1507
+ "[511]": 50776,
1508
+ "[512]": 50777,
1509
+ "[513]": 50778,
1510
+ "[514]": 50779,
1511
+ "[515]": 50780,
1512
+ "[516]": 50781,
1513
+ "[517]": 50782,
1514
+ "[518]": 50783,
1515
+ "[519]": 50784,
1516
+ "[51]": 50316,
1517
+ "[520]": 50785,
1518
+ "[521]": 50786,
1519
+ "[522]": 50787,
1520
+ "[523]": 50788,
1521
+ "[524]": 50789,
1522
+ "[525]": 50790,
1523
+ "[526]": 50791,
1524
+ "[527]": 50792,
1525
+ "[528]": 50793,
1526
+ "[529]": 50794,
1527
+ "[52]": 50317,
1528
+ "[530]": 50795,
1529
+ "[531]": 50796,
1530
+ "[532]": 50797,
1531
+ "[533]": 50798,
1532
+ "[534]": 50799,
1533
+ "[535]": 50800,
1534
+ "[536]": 50801,
1535
+ "[537]": 50802,
1536
+ "[538]": 50803,
1537
+ "[539]": 50804,
1538
+ "[53]": 50318,
1539
+ "[540]": 50805,
1540
+ "[541]": 50806,
1541
+ "[542]": 50807,
1542
+ "[543]": 50808,
1543
+ "[544]": 50809,
1544
+ "[545]": 50810,
1545
+ "[546]": 50811,
1546
+ "[547]": 50812,
1547
+ "[548]": 50813,
1548
+ "[549]": 50814,
1549
+ "[54]": 50319,
1550
+ "[550]": 50815,
1551
+ "[551]": 50816,
1552
+ "[552]": 50817,
1553
+ "[553]": 50818,
1554
+ "[554]": 50819,
1555
+ "[555]": 50820,
1556
+ "[556]": 50821,
1557
+ "[557]": 50822,
1558
+ "[558]": 50823,
1559
+ "[559]": 50824,
1560
+ "[55]": 50320,
1561
+ "[560]": 50825,
1562
+ "[561]": 50826,
1563
+ "[562]": 50827,
1564
+ "[563]": 50828,
1565
+ "[564]": 50829,
1566
+ "[565]": 50830,
1567
+ "[566]": 50831,
1568
+ "[567]": 50832,
1569
+ "[568]": 50833,
1570
+ "[569]": 50834,
1571
+ "[56]": 50321,
1572
+ "[570]": 50835,
1573
+ "[571]": 50836,
1574
+ "[572]": 50837,
1575
+ "[573]": 50838,
1576
+ "[574]": 50839,
1577
+ "[575]": 50840,
1578
+ "[576]": 50841,
1579
+ "[577]": 50842,
1580
+ "[578]": 50843,
1581
+ "[579]": 50844,
1582
+ "[57]": 50322,
1583
+ "[580]": 50845,
1584
+ "[581]": 50846,
1585
+ "[582]": 50847,
1586
+ "[583]": 50848,
1587
+ "[584]": 50849,
1588
+ "[585]": 50850,
1589
+ "[586]": 50851,
1590
+ "[587]": 50852,
1591
+ "[588]": 50853,
1592
+ "[589]": 50854,
1593
+ "[58]": 50323,
1594
+ "[590]": 50855,
1595
+ "[591]": 50856,
1596
+ "[592]": 50857,
1597
+ "[593]": 50858,
1598
+ "[594]": 50859,
1599
+ "[595]": 50860,
1600
+ "[596]": 50861,
1601
+ "[597]": 50862,
1602
+ "[598]": 50863,
1603
+ "[599]": 50864,
1604
+ "[59]": 50324,
1605
+ "[5]": 50270,
1606
+ "[600]": 50865,
1607
+ "[601]": 50866,
1608
+ "[602]": 50867,
1609
+ "[603]": 50868,
1610
+ "[604]": 50869,
1611
+ "[605]": 50870,
1612
+ "[606]": 50871,
1613
+ "[607]": 50872,
1614
+ "[608]": 50873,
1615
+ "[609]": 50874,
1616
+ "[60]": 50325,
1617
+ "[610]": 50875,
1618
+ "[611]": 50876,
1619
+ "[612]": 50877,
1620
+ "[613]": 50878,
1621
+ "[614]": 50879,
1622
+ "[615]": 50880,
1623
+ "[616]": 50881,
1624
+ "[617]": 50882,
1625
+ "[618]": 50883,
1626
+ "[619]": 50884,
1627
+ "[61]": 50326,
1628
+ "[620]": 50885,
1629
+ "[621]": 50886,
1630
+ "[622]": 50887,
1631
+ "[623]": 50888,
1632
+ "[624]": 50889,
1633
+ "[625]": 50890,
1634
+ "[626]": 50891,
1635
+ "[627]": 50892,
1636
+ "[628]": 50893,
1637
+ "[629]": 50894,
1638
+ "[62]": 50327,
1639
+ "[630]": 50895,
1640
+ "[631]": 50896,
1641
+ "[632]": 50897,
1642
+ "[633]": 50898,
1643
+ "[634]": 50899,
1644
+ "[635]": 50900,
1645
+ "[636]": 50901,
1646
+ "[637]": 50902,
1647
+ "[638]": 50903,
1648
+ "[639]": 50904,
1649
+ "[63]": 50328,
1650
+ "[640]": 50905,
1651
+ "[641]": 50906,
1652
+ "[642]": 50907,
1653
+ "[643]": 50908,
1654
+ "[644]": 50909,
1655
+ "[645]": 50910,
1656
+ "[646]": 50911,
1657
+ "[647]": 50912,
1658
+ "[648]": 50913,
1659
+ "[649]": 50914,
1660
+ "[64]": 50329,
1661
+ "[650]": 50915,
1662
+ "[651]": 50916,
1663
+ "[652]": 50917,
1664
+ "[653]": 50918,
1665
+ "[654]": 50919,
1666
+ "[655]": 50920,
1667
+ "[656]": 50921,
1668
+ "[657]": 50922,
1669
+ "[658]": 50923,
1670
+ "[659]": 50924,
1671
+ "[65]": 50330,
1672
+ "[660]": 50925,
1673
+ "[661]": 50926,
1674
+ "[662]": 50927,
1675
+ "[663]": 50928,
1676
+ "[664]": 50929,
1677
+ "[665]": 50930,
1678
+ "[666]": 50931,
1679
+ "[667]": 50932,
1680
+ "[668]": 50933,
1681
+ "[669]": 50934,
1682
+ "[66]": 50331,
1683
+ "[670]": 50935,
1684
+ "[671]": 50936,
1685
+ "[672]": 50937,
1686
+ "[673]": 50938,
1687
+ "[674]": 50939,
1688
+ "[675]": 50940,
1689
+ "[676]": 50941,
1690
+ "[677]": 50942,
1691
+ "[678]": 50943,
1692
+ "[679]": 50944,
1693
+ "[67]": 50332,
1694
+ "[680]": 50945,
1695
+ "[681]": 50946,
1696
+ "[682]": 50947,
1697
+ "[683]": 50948,
1698
+ "[684]": 50949,
1699
+ "[685]": 50950,
1700
+ "[686]": 50951,
1701
+ "[687]": 50952,
1702
+ "[688]": 50953,
1703
+ "[689]": 50954,
1704
+ "[68]": 50333,
1705
+ "[690]": 50955,
1706
+ "[691]": 50956,
1707
+ "[692]": 50957,
1708
+ "[693]": 50958,
1709
+ "[694]": 50959,
1710
+ "[695]": 50960,
1711
+ "[696]": 50961,
1712
+ "[697]": 50962,
1713
+ "[698]": 50963,
1714
+ "[699]": 50964,
1715
+ "[69]": 50334,
1716
+ "[6]": 50271,
1717
+ "[700]": 50965,
1718
+ "[701]": 50966,
1719
+ "[702]": 50967,
1720
+ "[703]": 50968,
1721
+ "[704]": 50969,
1722
+ "[705]": 50970,
1723
+ "[706]": 50971,
1724
+ "[707]": 50972,
1725
+ "[708]": 50973,
1726
+ "[709]": 50974,
1727
+ "[70]": 50335,
1728
+ "[710]": 50975,
1729
+ "[711]": 50976,
1730
+ "[712]": 50977,
1731
+ "[713]": 50978,
1732
+ "[714]": 50979,
1733
+ "[715]": 50980,
1734
+ "[716]": 50981,
1735
+ "[717]": 50982,
1736
+ "[718]": 50983,
1737
+ "[719]": 50984,
1738
+ "[71]": 50336,
1739
+ "[720]": 50985,
1740
+ "[721]": 50986,
1741
+ "[722]": 50987,
1742
+ "[723]": 50988,
1743
+ "[724]": 50989,
1744
+ "[725]": 50990,
1745
+ "[726]": 50991,
1746
+ "[727]": 50992,
1747
+ "[728]": 50993,
1748
+ "[729]": 50994,
1749
+ "[72]": 50337,
1750
+ "[730]": 50995,
1751
+ "[731]": 50996,
1752
+ "[732]": 50997,
1753
+ "[733]": 50998,
1754
+ "[734]": 50999,
1755
+ "[735]": 51000,
1756
+ "[736]": 51001,
1757
+ "[737]": 51002,
1758
+ "[738]": 51003,
1759
+ "[739]": 51004,
1760
+ "[73]": 50338,
1761
+ "[740]": 51005,
1762
+ "[741]": 51006,
1763
+ "[742]": 51007,
1764
+ "[743]": 51008,
1765
+ "[744]": 51009,
1766
+ "[745]": 51010,
1767
+ "[746]": 51011,
1768
+ "[747]": 51012,
1769
+ "[748]": 51013,
1770
+ "[749]": 51014,
1771
+ "[74]": 50339,
1772
+ "[750]": 51015,
1773
+ "[751]": 51016,
1774
+ "[752]": 51017,
1775
+ "[753]": 51018,
1776
+ "[754]": 51019,
1777
+ "[755]": 51020,
1778
+ "[756]": 51021,
1779
+ "[757]": 51022,
1780
+ "[758]": 51023,
1781
+ "[759]": 51024,
1782
+ "[75]": 50340,
1783
+ "[760]": 51025,
1784
+ "[761]": 51026,
1785
+ "[762]": 51027,
1786
+ "[763]": 51028,
1787
+ "[764]": 51029,
1788
+ "[765]": 51030,
1789
+ "[766]": 51031,
1790
+ "[767]": 51032,
1791
+ "[768]": 51033,
1792
+ "[769]": 51034,
1793
+ "[76]": 50341,
1794
+ "[770]": 51035,
1795
+ "[771]": 51036,
1796
+ "[772]": 51037,
1797
+ "[773]": 51038,
1798
+ "[774]": 51039,
1799
+ "[775]": 51040,
1800
+ "[776]": 51041,
1801
+ "[777]": 51042,
1802
+ "[778]": 51043,
1803
+ "[779]": 51044,
1804
+ "[77]": 50342,
1805
+ "[780]": 51045,
1806
+ "[781]": 51046,
1807
+ "[782]": 51047,
1808
+ "[783]": 51048,
1809
+ "[784]": 51049,
1810
+ "[785]": 51050,
1811
+ "[786]": 51051,
1812
+ "[787]": 51052,
1813
+ "[788]": 51053,
1814
+ "[789]": 51054,
1815
+ "[78]": 50343,
1816
+ "[790]": 51055,
1817
+ "[791]": 51056,
1818
+ "[792]": 51057,
1819
+ "[793]": 51058,
1820
+ "[794]": 51059,
1821
+ "[795]": 51060,
1822
+ "[796]": 51061,
1823
+ "[797]": 51062,
1824
+ "[798]": 51063,
1825
+ "[799]": 51064,
1826
+ "[79]": 50344,
1827
+ "[7]": 50272,
1828
+ "[800]": 51065,
1829
+ "[801]": 51066,
1830
+ "[802]": 51067,
1831
+ "[803]": 51068,
1832
+ "[804]": 51069,
1833
+ "[805]": 51070,
1834
+ "[806]": 51071,
1835
+ "[807]": 51072,
1836
+ "[808]": 51073,
1837
+ "[809]": 51074,
1838
+ "[80]": 50345,
1839
+ "[810]": 51075,
1840
+ "[811]": 51076,
1841
+ "[812]": 51077,
1842
+ "[813]": 51078,
1843
+ "[814]": 51079,
1844
+ "[815]": 51080,
1845
+ "[816]": 51081,
1846
+ "[817]": 51082,
1847
+ "[818]": 51083,
1848
+ "[819]": 51084,
1849
+ "[81]": 50346,
1850
+ "[820]": 51085,
1851
+ "[821]": 51086,
1852
+ "[822]": 51087,
1853
+ "[823]": 51088,
1854
+ "[824]": 51089,
1855
+ "[825]": 51090,
1856
+ "[826]": 51091,
1857
+ "[827]": 51092,
1858
+ "[828]": 51093,
1859
+ "[829]": 51094,
1860
+ "[82]": 50347,
1861
+ "[830]": 51095,
1862
+ "[831]": 51096,
1863
+ "[832]": 51097,
1864
+ "[833]": 51098,
1865
+ "[834]": 51099,
1866
+ "[835]": 51100,
1867
+ "[836]": 51101,
1868
+ "[837]": 51102,
1869
+ "[838]": 51103,
1870
+ "[839]": 51104,
1871
+ "[83]": 50348,
1872
+ "[840]": 51105,
1873
+ "[841]": 51106,
1874
+ "[842]": 51107,
1875
+ "[843]": 51108,
1876
+ "[844]": 51109,
1877
+ "[845]": 51110,
1878
+ "[846]": 51111,
1879
+ "[847]": 51112,
1880
+ "[848]": 51113,
1881
+ "[849]": 51114,
1882
+ "[84]": 50349,
1883
+ "[850]": 51115,
1884
+ "[851]": 51116,
1885
+ "[852]": 51117,
1886
+ "[853]": 51118,
1887
+ "[854]": 51119,
1888
+ "[855]": 51120,
1889
+ "[856]": 51121,
1890
+ "[857]": 51122,
1891
+ "[858]": 51123,
1892
+ "[859]": 51124,
1893
+ "[85]": 50350,
1894
+ "[860]": 51125,
1895
+ "[861]": 51126,
1896
+ "[862]": 51127,
1897
+ "[863]": 51128,
1898
+ "[864]": 51129,
1899
+ "[865]": 51130,
1900
+ "[866]": 51131,
1901
+ "[867]": 51132,
1902
+ "[868]": 51133,
1903
+ "[869]": 51134,
1904
+ "[86]": 50351,
1905
+ "[870]": 51135,
1906
+ "[871]": 51136,
1907
+ "[872]": 51137,
1908
+ "[873]": 51138,
1909
+ "[874]": 51139,
1910
+ "[875]": 51140,
1911
+ "[876]": 51141,
1912
+ "[877]": 51142,
1913
+ "[878]": 51143,
1914
+ "[879]": 51144,
1915
+ "[87]": 50352,
1916
+ "[880]": 51145,
1917
+ "[881]": 51146,
1918
+ "[882]": 51147,
1919
+ "[883]": 51148,
1920
+ "[884]": 51149,
1921
+ "[885]": 51150,
1922
+ "[886]": 51151,
1923
+ "[887]": 51152,
1924
+ "[888]": 51153,
1925
+ "[889]": 51154,
1926
+ "[88]": 50353,
1927
+ "[890]": 51155,
1928
+ "[891]": 51156,
1929
+ "[892]": 51157,
1930
+ "[893]": 51158,
1931
+ "[894]": 51159,
1932
+ "[895]": 51160,
1933
+ "[896]": 51161,
1934
+ "[897]": 51162,
1935
+ "[898]": 51163,
1936
+ "[899]": 51164,
1937
+ "[89]": 50354,
1938
+ "[8]": 50273,
1939
+ "[900]": 51165,
1940
+ "[901]": 51166,
1941
+ "[902]": 51167,
1942
+ "[903]": 51168,
1943
+ "[904]": 51169,
1944
+ "[905]": 51170,
1945
+ "[906]": 51171,
1946
+ "[907]": 51172,
1947
+ "[908]": 51173,
1948
+ "[909]": 51174,
1949
+ "[90]": 50355,
1950
+ "[910]": 51175,
1951
+ "[911]": 51176,
1952
+ "[912]": 51177,
1953
+ "[913]": 51178,
1954
+ "[914]": 51179,
1955
+ "[915]": 51180,
1956
+ "[916]": 51181,
1957
+ "[917]": 51182,
1958
+ "[918]": 51183,
1959
+ "[919]": 51184,
1960
+ "[91]": 50356,
1961
+ "[920]": 51185,
1962
+ "[921]": 51186,
1963
+ "[922]": 51187,
1964
+ "[923]": 51188,
1965
+ "[924]": 51189,
1966
+ "[925]": 51190,
1967
+ "[926]": 51191,
1968
+ "[927]": 51192,
1969
+ "[928]": 51193,
1970
+ "[929]": 51194,
1971
+ "[92]": 50357,
1972
+ "[930]": 51195,
1973
+ "[931]": 51196,
1974
+ "[932]": 51197,
1975
+ "[933]": 51198,
1976
+ "[934]": 51199,
1977
+ "[935]": 51200,
1978
+ "[936]": 51201,
1979
+ "[937]": 51202,
1980
+ "[938]": 51203,
1981
+ "[939]": 51204,
1982
+ "[93]": 50358,
1983
+ "[940]": 51205,
1984
+ "[941]": 51206,
1985
+ "[942]": 51207,
1986
+ "[943]": 51208,
1987
+ "[944]": 51209,
1988
+ "[945]": 51210,
1989
+ "[946]": 51211,
1990
+ "[947]": 51212,
1991
+ "[948]": 51213,
1992
+ "[949]": 51214,
1993
+ "[94]": 50359,
1994
+ "[950]": 51215,
1995
+ "[951]": 51216,
1996
+ "[952]": 51217,
1997
+ "[953]": 51218,
1998
+ "[954]": 51219,
1999
+ "[955]": 51220,
2000
+ "[956]": 51221,
2001
+ "[957]": 51222,
2002
+ "[958]": 51223,
2003
+ "[959]": 51224,
2004
+ "[95]": 50360,
2005
+ "[960]": 51225,
2006
+ "[961]": 51226,
2007
+ "[962]": 51227,
2008
+ "[963]": 51228,
2009
+ "[964]": 51229,
2010
+ "[965]": 51230,
2011
+ "[966]": 51231,
2012
+ "[967]": 51232,
2013
+ "[968]": 51233,
2014
+ "[969]": 51234,
2015
+ "[96]": 50361,
2016
+ "[970]": 51235,
2017
+ "[971]": 51236,
2018
+ "[972]": 51237,
2019
+ "[973]": 51238,
2020
+ "[974]": 51239,
2021
+ "[975]": 51240,
2022
+ "[976]": 51241,
2023
+ "[977]": 51242,
2024
+ "[978]": 51243,
2025
+ "[979]": 51244,
2026
+ "[97]": 50362,
2027
+ "[980]": 51245,
2028
+ "[981]": 51246,
2029
+ "[982]": 51247,
2030
+ "[983]": 51248,
2031
+ "[984]": 51249,
2032
+ "[985]": 51250,
2033
+ "[986]": 51251,
2034
+ "[987]": 51252,
2035
+ "[988]": 51253,
2036
+ "[989]": 51254,
2037
+ "[98]": 50363,
2038
+ "[990]": 51255,
2039
+ "[991]": 51256,
2040
+ "[992]": 51257,
2041
+ "[993]": 51258,
2042
+ "[994]": 51259,
2043
+ "[995]": 51260,
2044
+ "[996]": 51261,
2045
+ "[997]": 51262,
2046
+ "[998]": 51263,
2047
+ "[999]": 51264,
2048
+ "[99]": 50364,
2049
+ "[9]": 50274
2050
+ }
config.json ADDED
@@ -0,0 +1,74 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "facebook/bart-base",
3
+ "activation_dropout": 0.1,
4
+ "activation_function": "gelu",
5
+ "add_bias_logits": false,
6
+ "add_final_layer_norm": false,
7
+ "architectures": [
8
+ "BartForConditionalGeneration"
9
+ ],
10
+ "attention_dropout": 0.1,
11
+ "bos_token_id": 0,
12
+ "classif_dropout": 0.1,
13
+ "classifier_dropout": 0.0,
14
+ "d_model": 768,
15
+ "decoder_attention_heads": 12,
16
+ "decoder_ffn_dim": 3072,
17
+ "decoder_layerdrop": 0.0,
18
+ "decoder_layers": 6,
19
+ "decoder_start_token_id": 2,
20
+ "dropout": 0.1,
21
+ "early_stopping": null,
22
+ "encoder_attention_heads": 12,
23
+ "encoder_ffn_dim": 3072,
24
+ "encoder_layerdrop": 0.0,
25
+ "encoder_layers": 6,
26
+ "eos_token_id": 2,
27
+ "forced_eos_token_id": 2,
28
+ "gradient_checkpointing": false,
29
+ "id2label": {
30
+ "0": "LABEL_0",
31
+ "1": "LABEL_1",
32
+ "2": "LABEL_2"
33
+ },
34
+ "init_std": 0.02,
35
+ "is_encoder_decoder": true,
36
+ "label2id": {
37
+ "LABEL_0": 0,
38
+ "LABEL_1": 1,
39
+ "LABEL_2": 2
40
+ },
41
+ "max_position_embeddings": 1024,
42
+ "model_type": "bart",
43
+ "no_repeat_ngram_size": null,
44
+ "normalize_before": false,
45
+ "normalize_embedding": true,
46
+ "num_beams": null,
47
+ "num_hidden_layers": 6,
48
+ "pad_token_id": 1,
49
+ "scale_embedding": false,
50
+ "task_specific_params": {
51
+ "summarization": {
52
+ "length_penalty": 1.0,
53
+ "max_length": 128,
54
+ "min_length": 12,
55
+ "num_beams": 4
56
+ },
57
+ "summarization_cnn": {
58
+ "length_penalty": 2.0,
59
+ "max_length": 142,
60
+ "min_length": 56,
61
+ "num_beams": 4
62
+ },
63
+ "summarization_xsum": {
64
+ "length_penalty": 1.0,
65
+ "max_length": 62,
66
+ "min_length": 11,
67
+ "num_beams": 6
68
+ }
69
+ },
70
+ "torch_dtype": "float16",
71
+ "transformers_version": "4.48.0",
72
+ "use_cache": true,
73
+ "vocab_size": 52313
74
+ }
examples.py ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
2
+ from utils import MorganFingerprint, morgan_fingerprint_to_text
3
+
4
+
5
+ # Load the checkpoint and the tokenizer
6
+ checkpoint_path = "lamthuy/MorganGen"
7
+ model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint_path)
8
+ tokenizer = AutoTokenizer.from_pretrained(checkpoint_path)
9
+
10
+ # Given a SMILES, get its fingerpint
11
+ smiles = "CC(=O)OC1=CC=CC=C1C(=O)O"
12
+ m = MorganFingerprint()
13
+ mf = m.smiles_to_morgan(smiles)
14
+
15
+ # convert it to the indices text format
16
+ s = morgan_fingerprint_to_text(mf)
17
+
18
+ # encode
19
+ input_ids = tokenizer.encode(s, return_tensors="pt")
20
+ # Generate output sequence
21
+ output_ids = model.generate(input_ids, max_length=64, num_beams=5)
22
+
23
+ # Decode the generated output
24
+ output_smiles = tokenizer.decode(output_ids[0], skip_special_tokens=True)
25
+ print(output_smiles)
generation_config.json ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 0,
4
+ "decoder_start_token_id": 2,
5
+ "early_stopping": true,
6
+ "eos_token_id": 2,
7
+ "forced_bos_token_id": 0,
8
+ "forced_eos_token_id": 2,
9
+ "no_repeat_ngram_size": 3,
10
+ "num_beams": 4,
11
+ "pad_token_id": 1,
12
+ "transformers_version": "4.48.0"
13
+ }
latest ADDED
@@ -0,0 +1 @@
 
 
1
+ global_step100000
merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4e2fcaaf9ed92c8748417094a12a84bf1505586822fd6a0529b22b8ff767e953
3
+ size 523179842
rng_state_0.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1a9409e7157d624dc820323a53a7f9470cf8e4d86c7b48b61509d3b80792e4cf
3
+ size 14512
rng_state_1.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d7c602e1061eed59a486be1d7ed63095a33439ef472899b1af141c7c9e77abc5
3
+ size 14512
special_tokens_map.json ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<s>",
4
+ "lstrip": false,
5
+ "normalized": true,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "cls_token": {
10
+ "content": "<s>",
11
+ "lstrip": false,
12
+ "normalized": true,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "eos_token": {
17
+ "content": "</s>",
18
+ "lstrip": false,
19
+ "normalized": true,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
+ "mask_token": {
24
+ "content": "<mask>",
25
+ "lstrip": true,
26
+ "normalized": true,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ },
30
+ "pad_token": {
31
+ "content": "<pad>",
32
+ "lstrip": false,
33
+ "normalized": true,
34
+ "rstrip": false,
35
+ "single_word": false
36
+ },
37
+ "sep_token": {
38
+ "content": "</s>",
39
+ "lstrip": false,
40
+ "normalized": true,
41
+ "rstrip": false,
42
+ "single_word": false
43
+ },
44
+ "unk_token": {
45
+ "content": "<unk>",
46
+ "lstrip": false,
47
+ "normalized": true,
48
+ "rstrip": false,
49
+ "single_word": false
50
+ }
51
+ }
tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff
 
trainer_state.json ADDED
The diff for this file is too large to render. See raw diff
 
training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:77cc9b8a8a4763f4985c6569708caf3454b802a38c401456e5c8ccd6b13ad161
3
+ size 6968
utils.py ADDED
@@ -0,0 +1,82 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from typing import Optional
2
+ import numpy as np
3
+ import py3Dmol
4
+ from rdkit import Chem, DataStructs
5
+ from rdkit.Chem import AllChem
6
+ import torch
7
+
8
+
9
+ class MorganFingerprint:
10
+ def __init__(self, shape: Optional[int] = 2048, radius: Optional[int] = 2):
11
+ self.shape = shape
12
+ self.radius = radius
13
+
14
+ @staticmethod
15
+ def canonicalize(smiles):
16
+ mol = Chem.MolFromSmiles(smiles)
17
+ if mol is not None:
18
+ return Chem.MolToSmiles(mol, isomericSmiles=True)
19
+ else:
20
+ return smiles
21
+
22
+ def smiles_to_morgan(self, smile: str) -> torch.Tensor:
23
+ try:
24
+ smile = self.canonicalize(smile)
25
+ mol = Chem.MolFromSmiles(smile)
26
+ features_vec = AllChem.GetMorganFingerprintAsBitVect(
27
+ mol, self.radius, nBits=self.shape
28
+ )
29
+ features = np.zeros((1,))
30
+ DataStructs.ConvertToNumpyArray(features_vec, features)
31
+ except Exception as e:
32
+ features = np.zeros((self.shape,))
33
+ return torch.tensor(features, dtype=torch.float32)
34
+
35
+
36
+ def get_morgan(input_sequences):
37
+ m = MorganFingerprint()
38
+ morgans = []
39
+ for s in input_sequences:
40
+ r = m.smiles_to_morgan(s)
41
+ indices_of_ones = torch.nonzero(r == 1.0, as_tuple=False)
42
+ indices_of_ones = indices_of_ones.squeeze(-1)
43
+ indices_of_ones = indices_of_ones.tolist()
44
+ s = ""
45
+ for i in indices_of_ones:
46
+ s += "[" + str(i) + "]"
47
+ morgans.append(s)
48
+ return morgans
49
+
50
+
51
+ def morgan_fingerprint_to_text(morgan_fn):
52
+ indices_of_ones = torch.nonzero(morgan_fn == 1.0, as_tuple=False)
53
+ indices_of_ones = indices_of_ones.squeeze(-1)
54
+ indices_of_ones = indices_of_ones.tolist()
55
+ s = ""
56
+ for i in indices_of_ones:
57
+ s += "[" + str(i) + "]"
58
+ return s
59
+
60
+
61
+ def smiles_to_3d(smiles_list, width=400, height=300):
62
+ # Visualize the 3D structure using py3Dmol
63
+ view = py3Dmol.view(width=width, height=height)
64
+ for smiles in smiles_list:
65
+ # Generate the RDKit molecule object
66
+ mol = Chem.MolFromSmiles(smiles)
67
+ if mol is None:
68
+ raise ValueError("Invalid SMILES string")
69
+
70
+ # Add hydrogens to the molecule
71
+ mol = Chem.AddHs(mol)
72
+
73
+ # Generate 3D coordinates
74
+ AllChem.EmbedMolecule(mol, randomSeed=42)
75
+ AllChem.UFFOptimizeMolecule(mol)
76
+
77
+ # Generate the 3D structure in the form of a pdb string
78
+ pdb = Chem.MolToPDBBlock(mol)
79
+ view.addModel(pdb, 'pdb')
80
+ view.setStyle({'stick': {}})
81
+ view.zoomTo()
82
+ return view
vocab.json ADDED
The diff for this file is too large to render. See raw diff
 
zero_to_fp32.py ADDED
@@ -0,0 +1,760 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python
2
+
3
+ # Copyright (c) Microsoft Corporation.
4
+ # SPDX-License-Identifier: Apache-2.0
5
+
6
+ # DeepSpeed Team
7
+
8
+ # This script extracts fp32 consolidated weights from a zero 1, 2 and 3 DeepSpeed checkpoints. It gets
9
+ # copied into the top level checkpoint dir, so the user can easily do the conversion at any point in
10
+ # the future. Once extracted, the weights don't require DeepSpeed and can be used in any
11
+ # application.
12
+ #
13
+ # example:
14
+ # python zero_to_fp32.py . output_dir/
15
+ # or
16
+ # python zero_to_fp32.py . output_dir/ --safe_serialization
17
+
18
+ import argparse
19
+ import torch
20
+ import glob
21
+ import math
22
+ import os
23
+ import re
24
+ import gc
25
+ import json
26
+ import numpy as np
27
+ from tqdm import tqdm
28
+ from collections import OrderedDict
29
+ from dataclasses import dataclass
30
+
31
+ # while this script doesn't use deepspeed to recover data, since the checkpoints are pickled with
32
+ # DeepSpeed data structures it has to be available in the current python environment.
33
+ from deepspeed.utils import logger
34
+ from deepspeed.checkpoint.constants import (DS_VERSION, OPTIMIZER_STATE_DICT, SINGLE_PARTITION_OF_FP32_GROUPS,
35
+ FP32_FLAT_GROUPS, ZERO_STAGE, PARTITION_COUNT, PARAM_SHAPES, BUFFER_NAMES,
36
+ FROZEN_PARAM_SHAPES, FROZEN_PARAM_FRAGMENTS)
37
+
38
+
39
+ @dataclass
40
+ class zero_model_state:
41
+ buffers: dict()
42
+ param_shapes: dict()
43
+ shared_params: list
44
+ ds_version: int
45
+ frozen_param_shapes: dict()
46
+ frozen_param_fragments: dict()
47
+
48
+
49
+ debug = 0
50
+
51
+ # load to cpu
52
+ device = torch.device('cpu')
53
+
54
+
55
+ def atoi(text):
56
+ return int(text) if text.isdigit() else text
57
+
58
+
59
+ def natural_keys(text):
60
+ '''
61
+ alist.sort(key=natural_keys) sorts in human order
62
+ http://nedbatchelder.com/blog/200712/human_sorting.html
63
+ (See Toothy's implementation in the comments)
64
+ '''
65
+ return [atoi(c) for c in re.split(r'(\d+)', text)]
66
+
67
+
68
+ def get_model_state_file(checkpoint_dir, zero_stage):
69
+ if not os.path.isdir(checkpoint_dir):
70
+ raise FileNotFoundError(f"Directory '{checkpoint_dir}' doesn't exist")
71
+
72
+ # there should be only one file
73
+ if zero_stage <= 2:
74
+ file = os.path.join(checkpoint_dir, "mp_rank_00_model_states.pt")
75
+ elif zero_stage == 3:
76
+ file = os.path.join(checkpoint_dir, "zero_pp_rank_0_mp_rank_00_model_states.pt")
77
+
78
+ if not os.path.exists(file):
79
+ raise FileNotFoundError(f"can't find model states file at '{file}'")
80
+
81
+ return file
82
+
83
+
84
+ def get_checkpoint_files(checkpoint_dir, glob_pattern):
85
+ # XXX: need to test that this simple glob rule works for multi-node setup too
86
+ ckpt_files = sorted(glob.glob(os.path.join(checkpoint_dir, glob_pattern)), key=natural_keys)
87
+
88
+ if len(ckpt_files) == 0:
89
+ raise FileNotFoundError(f"can't find {glob_pattern} files in directory '{checkpoint_dir}'")
90
+
91
+ return ckpt_files
92
+
93
+
94
+ def get_optim_files(checkpoint_dir):
95
+ return get_checkpoint_files(checkpoint_dir, "*_optim_states.pt")
96
+
97
+
98
+ def get_model_state_files(checkpoint_dir):
99
+ return get_checkpoint_files(checkpoint_dir, "*_model_states.pt")
100
+
101
+
102
+ def parse_model_states(files):
103
+ zero_model_states = []
104
+ for file in files:
105
+ state_dict = torch.load(file, map_location=device, weights_only=False)
106
+
107
+ if BUFFER_NAMES not in state_dict:
108
+ raise ValueError(f"{file} is not a model state checkpoint")
109
+ buffer_names = state_dict[BUFFER_NAMES]
110
+ if debug:
111
+ print("Found buffers:", buffer_names)
112
+
113
+ # recover just the buffers while restoring them to fp32 if they were saved in fp16
114
+ buffers = {k: v.float() for k, v in state_dict["module"].items() if k in buffer_names}
115
+ param_shapes = state_dict[PARAM_SHAPES]
116
+
117
+ # collect parameters that are included in param_shapes
118
+ param_names = []
119
+ for s in param_shapes:
120
+ for name in s.keys():
121
+ param_names.append(name)
122
+
123
+ # update with frozen parameters
124
+ frozen_param_shapes = state_dict.get(FROZEN_PARAM_SHAPES, None)
125
+ if frozen_param_shapes is not None:
126
+ if debug:
127
+ print(f"Found frozen_param_shapes: {frozen_param_shapes}")
128
+ param_names += list(frozen_param_shapes.keys())
129
+
130
+ # handle shared params
131
+ shared_params = [[k, v] for k, v in state_dict["shared_params"].items()]
132
+
133
+ ds_version = state_dict.get(DS_VERSION, None)
134
+
135
+ frozen_param_fragments = state_dict.get(FROZEN_PARAM_FRAGMENTS, None)
136
+
137
+ z_model_state = zero_model_state(buffers=buffers,
138
+ param_shapes=param_shapes,
139
+ shared_params=shared_params,
140
+ ds_version=ds_version,
141
+ frozen_param_shapes=frozen_param_shapes,
142
+ frozen_param_fragments=frozen_param_fragments)
143
+ zero_model_states.append(z_model_state)
144
+
145
+ return zero_model_states
146
+
147
+
148
+ def parse_optim_states(files, ds_checkpoint_dir):
149
+ total_files = len(files)
150
+ state_dicts = []
151
+ for f in tqdm(files, desc='Loading checkpoint shards'):
152
+ state_dict = torch.load(f, map_location=device, mmap=True, weights_only=False)
153
+ # immediately discard the potentially huge 2 optimizer states as we only care for fp32 master weights
154
+ # and also handle the case where it was already removed by another helper script
155
+ state_dict["optimizer_state_dict"].pop("optimizer_state_dict", None)
156
+ state_dicts.append(state_dict)
157
+
158
+ if not ZERO_STAGE in state_dicts[0][OPTIMIZER_STATE_DICT]:
159
+ raise ValueError(f"{files[0]} is not a zero checkpoint")
160
+ zero_stage = state_dicts[0][OPTIMIZER_STATE_DICT][ZERO_STAGE]
161
+ world_size = state_dicts[0][OPTIMIZER_STATE_DICT][PARTITION_COUNT]
162
+
163
+ # For ZeRO-2 each param group can have different partition_count as data parallelism for expert
164
+ # parameters can be different from data parallelism for non-expert parameters. So we can just
165
+ # use the max of the partition_count to get the dp world_size.
166
+
167
+ if type(world_size) is list:
168
+ world_size = max(world_size)
169
+
170
+ if world_size != total_files:
171
+ raise ValueError(
172
+ f"Expected {world_size} of '*_optim_states.pt' under '{ds_checkpoint_dir}' but found {total_files} files. "
173
+ "Possibly due to an overwrite of an old checkpoint, or a checkpoint didn't get saved by one or more processes."
174
+ )
175
+
176
+ # the groups are named differently in each stage
177
+ if zero_stage <= 2:
178
+ fp32_groups_key = SINGLE_PARTITION_OF_FP32_GROUPS
179
+ elif zero_stage == 3:
180
+ fp32_groups_key = FP32_FLAT_GROUPS
181
+ else:
182
+ raise ValueError(f"unknown zero stage {zero_stage}")
183
+
184
+ fp32_flat_groups = [state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key] for i in range(len(state_dicts))]
185
+ return zero_stage, world_size, fp32_flat_groups
186
+
187
+
188
+ def _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir, exclude_frozen_parameters):
189
+ """
190
+ Returns fp32 state_dict reconstructed from ds checkpoint
191
+
192
+ Args:
193
+ - ``ds_checkpoint_dir``: path to the deepspeed checkpoint folder (where the optimizer files are)
194
+
195
+ """
196
+ print(f"Processing zero checkpoint '{ds_checkpoint_dir}'")
197
+
198
+ optim_files = get_optim_files(ds_checkpoint_dir)
199
+ zero_stage, world_size, fp32_flat_groups = parse_optim_states(optim_files, ds_checkpoint_dir)
200
+ print(f"Detected checkpoint of type zero stage {zero_stage}, world_size: {world_size}")
201
+
202
+ model_files = get_model_state_files(ds_checkpoint_dir)
203
+
204
+ zero_model_states = parse_model_states(model_files)
205
+ print(f'Parsing checkpoint created by deepspeed=={zero_model_states[0].ds_version}')
206
+
207
+ if zero_stage <= 2:
208
+ return _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states,
209
+ exclude_frozen_parameters)
210
+ elif zero_stage == 3:
211
+ return _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states,
212
+ exclude_frozen_parameters)
213
+
214
+
215
+ def _zero2_merge_frozen_params(state_dict, zero_model_states):
216
+ if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
217
+ return
218
+
219
+ frozen_param_shapes = zero_model_states[0].frozen_param_shapes
220
+ frozen_param_fragments = zero_model_states[0].frozen_param_fragments
221
+
222
+ if debug:
223
+ num_elem = sum(s.numel() for s in frozen_param_shapes.values())
224
+ print(f'rank 0: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
225
+
226
+ wanted_params = len(frozen_param_shapes)
227
+ wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
228
+ avail_numel = sum([p.numel() for p in frozen_param_fragments.values()])
229
+ print(f'Frozen params: Have {avail_numel} numels to process.')
230
+ print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
231
+
232
+ total_params = 0
233
+ total_numel = 0
234
+ for name, shape in frozen_param_shapes.items():
235
+ total_params += 1
236
+ unpartitioned_numel = shape.numel()
237
+ total_numel += unpartitioned_numel
238
+
239
+ state_dict[name] = frozen_param_fragments[name]
240
+
241
+ if debug:
242
+ print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
243
+
244
+ print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
245
+
246
+
247
+ def _has_callable(obj, fn):
248
+ attr = getattr(obj, fn, None)
249
+ return callable(attr)
250
+
251
+
252
+ def _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
253
+ param_shapes = zero_model_states[0].param_shapes
254
+
255
+ # Reconstruction protocol:
256
+ #
257
+ # XXX: document this
258
+
259
+ if debug:
260
+ for i in range(world_size):
261
+ for j in range(len(fp32_flat_groups[0])):
262
+ print(f"{FP32_FLAT_GROUPS}[{i}][{j}].shape={fp32_flat_groups[i][j].shape}")
263
+
264
+ # XXX: memory usage doubles here (zero2)
265
+ num_param_groups = len(fp32_flat_groups[0])
266
+ merged_single_partition_of_fp32_groups = []
267
+ for i in range(num_param_groups):
268
+ merged_partitions = [sd[i] for sd in fp32_flat_groups]
269
+ full_single_fp32_vector = torch.cat(merged_partitions, 0)
270
+ merged_single_partition_of_fp32_groups.append(full_single_fp32_vector)
271
+ avail_numel = sum(
272
+ [full_single_fp32_vector.numel() for full_single_fp32_vector in merged_single_partition_of_fp32_groups])
273
+
274
+ if debug:
275
+ wanted_params = sum([len(shapes) for shapes in param_shapes])
276
+ wanted_numel = sum([sum(shape.numel() for shape in shapes.values()) for shapes in param_shapes])
277
+ # not asserting if there is a mismatch due to possible padding
278
+ print(f"Have {avail_numel} numels to process.")
279
+ print(f"Need {wanted_numel} numels in {wanted_params} params.")
280
+
281
+ # params
282
+ # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
283
+ # out-of-core computing solution
284
+ total_numel = 0
285
+ total_params = 0
286
+ for shapes, full_single_fp32_vector in zip(param_shapes, merged_single_partition_of_fp32_groups):
287
+ offset = 0
288
+ avail_numel = full_single_fp32_vector.numel()
289
+ for name, shape in shapes.items():
290
+
291
+ unpartitioned_numel = shape.numel() if _has_callable(shape, 'numel') else math.prod(shape)
292
+ total_numel += unpartitioned_numel
293
+ total_params += 1
294
+
295
+ if debug:
296
+ print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
297
+ state_dict[name] = full_single_fp32_vector.narrow(0, offset, unpartitioned_numel).view(shape)
298
+ offset += unpartitioned_numel
299
+
300
+ # Z2 started to align to 2*world_size to improve nccl performance. Therefore both offset and
301
+ # avail_numel can differ by anywhere between 0..2*world_size. Due to two unrelated complex
302
+ # paddings performed in the code it's almost impossible to predict the exact numbers w/o the
303
+ # live optimizer object, so we are checking that the numbers are within the right range
304
+ align_to = 2 * world_size
305
+
306
+ def zero2_align(x):
307
+ return align_to * math.ceil(x / align_to)
308
+
309
+ if debug:
310
+ print(f"original offset={offset}, avail_numel={avail_numel}")
311
+
312
+ offset = zero2_align(offset)
313
+ avail_numel = zero2_align(avail_numel)
314
+
315
+ if debug:
316
+ print(f"aligned offset={offset}, avail_numel={avail_numel}")
317
+
318
+ # Sanity check
319
+ if offset != avail_numel:
320
+ raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
321
+
322
+ print(f"Reconstructed fp32 state dict with {total_params} params {total_numel} elements")
323
+
324
+
325
+ def _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states,
326
+ exclude_frozen_parameters):
327
+ state_dict = OrderedDict()
328
+
329
+ # buffers
330
+ buffers = zero_model_states[0].buffers
331
+ state_dict.update(buffers)
332
+ if debug:
333
+ print(f"added {len(buffers)} buffers")
334
+
335
+ if not exclude_frozen_parameters:
336
+ _zero2_merge_frozen_params(state_dict, zero_model_states)
337
+
338
+ _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
339
+
340
+ # recover shared parameters
341
+ for pair in zero_model_states[0].shared_params:
342
+ if pair[1] in state_dict:
343
+ state_dict[pair[0]] = state_dict[pair[1]]
344
+
345
+ return state_dict
346
+
347
+
348
+ def zero3_partitioned_param_info(unpartitioned_numel, world_size):
349
+ remainder = unpartitioned_numel % world_size
350
+ padding_numel = (world_size - remainder) if remainder else 0
351
+ partitioned_numel = math.ceil(unpartitioned_numel / world_size)
352
+ return partitioned_numel, padding_numel
353
+
354
+
355
+ def _zero3_merge_frozen_params(state_dict, world_size, zero_model_states):
356
+ if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
357
+ return
358
+
359
+ if debug:
360
+ for i in range(world_size):
361
+ num_elem = sum(s.numel() for s in zero_model_states[i].frozen_param_fragments.values())
362
+ print(f'rank {i}: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
363
+
364
+ frozen_param_shapes = zero_model_states[0].frozen_param_shapes
365
+ wanted_params = len(frozen_param_shapes)
366
+ wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
367
+ avail_numel = sum([p.numel() for p in zero_model_states[0].frozen_param_fragments.values()]) * world_size
368
+ print(f'Frozen params: Have {avail_numel} numels to process.')
369
+ print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
370
+
371
+ total_params = 0
372
+ total_numel = 0
373
+ for name, shape in zero_model_states[0].frozen_param_shapes.items():
374
+ total_params += 1
375
+ unpartitioned_numel = shape.numel()
376
+ total_numel += unpartitioned_numel
377
+
378
+ param_frags = tuple(model_state.frozen_param_fragments[name] for model_state in zero_model_states)
379
+ state_dict[name] = torch.cat(param_frags, 0).narrow(0, 0, unpartitioned_numel).view(shape)
380
+
381
+ partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
382
+
383
+ if debug:
384
+ print(
385
+ f"Frozen params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
386
+ )
387
+
388
+ print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
389
+
390
+
391
+ class GatheredTensor:
392
+ """
393
+ A pseudo tensor that collects partitioned weights.
394
+ It is more memory efficient when there are multiple groups.
395
+ """
396
+
397
+ def __init__(self, flat_groups, flat_groups_offset, offset, partitioned_numel, shape):
398
+ self.flat_groups = flat_groups
399
+ self.flat_groups_offset = flat_groups_offset
400
+ self.offset = offset
401
+ self.partitioned_numel = partitioned_numel
402
+ self.shape = shape
403
+ self.dtype = self.flat_groups[0][0].dtype
404
+
405
+ def contiguous(self):
406
+ """
407
+ Merge partitioned weights from flat_groups into a single tensor.
408
+ """
409
+ end_idx = self.offset + self.partitioned_numel
410
+ world_size = len(self.flat_groups)
411
+ pad_flat_param_chunks = []
412
+
413
+ for rank_i in range(world_size):
414
+ # for each rank, we need to collect weights from related group/groups
415
+ flat_groups_at_rank_i = self.flat_groups[rank_i]
416
+ start_group_id = None
417
+ end_group_id = None
418
+ for group_id in range(len(self.flat_groups_offset)):
419
+ if self.flat_groups_offset[group_id] <= self.offset < self.flat_groups_offset[group_id + 1]:
420
+ start_group_id = group_id
421
+ if self.flat_groups_offset[group_id] < end_idx <= self.flat_groups_offset[group_id + 1]:
422
+ end_group_id = group_id
423
+ break
424
+ # collect weights from related group/groups
425
+ for group_id in range(start_group_id, end_group_id + 1):
426
+ flat_tensor = flat_groups_at_rank_i[group_id]
427
+ start_offset = self.offset - self.flat_groups_offset[group_id]
428
+ end_offset = min(end_idx, self.flat_groups_offset[group_id + 1]) - self.flat_groups_offset[group_id]
429
+ pad_flat_param_chunks.append(flat_tensor[start_offset:end_offset])
430
+
431
+ # collect weights from all ranks
432
+ pad_flat_param = torch.cat(pad_flat_param_chunks, dim=0)
433
+ param = pad_flat_param[:self.shape.numel()].view(self.shape).contiguous()
434
+ return param
435
+
436
+
437
+ def _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
438
+ param_shapes = zero_model_states[0].param_shapes
439
+ avail_numel = sum([flat_group.numel() for flat_group in fp32_flat_groups[0]]) * world_size
440
+
441
+ # Reconstruction protocol: For zero3 we need to zip the partitions together at boundary of each
442
+ # param, re-consolidating each param, while dealing with padding if any
443
+
444
+ # merge list of dicts, preserving order
445
+ param_shapes = {k: v for d in param_shapes for k, v in d.items()}
446
+
447
+ if debug:
448
+ for i in range(world_size):
449
+ print(f"{FP32_FLAT_GROUPS}[{i}].shape={fp32_flat_groups[i].shape}")
450
+
451
+ wanted_params = len(param_shapes)
452
+ wanted_numel = sum(shape.numel() for shape in param_shapes.values())
453
+ # not asserting if there is a mismatch due to possible padding
454
+ avail_numel = fp32_flat_groups[0].numel() * world_size
455
+ print(f"Trainable params: Have {avail_numel} numels to process.")
456
+ print(f"Trainable params: Need {wanted_numel} numels in {wanted_params} params.")
457
+
458
+ # params
459
+ # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
460
+ # out-of-core computing solution
461
+ offset = 0
462
+ total_numel = 0
463
+ total_params = 0
464
+ flat_groups_offset = [0] + list(np.cumsum([flat_tensor.numel() for flat_tensor in fp32_flat_groups[0]]))
465
+ for name, shape in tqdm(param_shapes.items(), desc='Gathering sharded weights'):
466
+ unpartitioned_numel = shape.numel()
467
+ total_numel += unpartitioned_numel
468
+ total_params += 1
469
+ partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
470
+
471
+ if debug:
472
+ print(
473
+ f"Trainable params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
474
+ )
475
+
476
+ # memory efficient tensor
477
+ tensor = GatheredTensor(fp32_flat_groups, flat_groups_offset, offset, partitioned_numel, shape)
478
+ state_dict[name] = tensor
479
+ offset += partitioned_numel
480
+
481
+ offset *= world_size
482
+
483
+ # Sanity check
484
+ if offset != avail_numel:
485
+ raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
486
+
487
+ print(f"Reconstructed Trainable fp32 state dict with {total_params} params {total_numel} elements")
488
+
489
+
490
+ def _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states,
491
+ exclude_frozen_parameters):
492
+ state_dict = OrderedDict()
493
+
494
+ # buffers
495
+ buffers = zero_model_states[0].buffers
496
+ state_dict.update(buffers)
497
+ if debug:
498
+ print(f"added {len(buffers)} buffers")
499
+
500
+ if not exclude_frozen_parameters:
501
+ _zero3_merge_frozen_params(state_dict, world_size, zero_model_states)
502
+
503
+ _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
504
+
505
+ # recover shared parameters
506
+ for pair in zero_model_states[0].shared_params:
507
+ if pair[1] in state_dict:
508
+ state_dict[pair[0]] = state_dict[pair[1]]
509
+
510
+ return state_dict
511
+
512
+
513
+ def to_torch_tensor(state_dict, return_empty_tensor=False):
514
+ """
515
+ Convert state_dict of GatheredTensor to torch tensor
516
+ """
517
+ torch_state_dict = {}
518
+ converted_tensors = {}
519
+ for name, tensor in state_dict.items():
520
+ tensor_id = id(tensor)
521
+ if tensor_id in converted_tensors: # shared tensors
522
+ shared_tensor = torch_state_dict[converted_tensors[tensor_id]]
523
+ torch_state_dict[name] = shared_tensor
524
+ else:
525
+ converted_tensors[tensor_id] = name
526
+ if return_empty_tensor:
527
+ torch_state_dict[name] = torch.empty(tensor.shape, dtype=tensor.dtype)
528
+ else:
529
+ torch_state_dict[name] = tensor.contiguous()
530
+ return torch_state_dict
531
+
532
+
533
+ def get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir,
534
+ tag=None,
535
+ exclude_frozen_parameters=False,
536
+ lazy_mode=False):
537
+ """
538
+ Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated state_dict that can be loaded with
539
+ ``load_state_dict()`` and used for training without DeepSpeed or shared with others, for example
540
+ via a model hub.
541
+
542
+ Args:
543
+ - ``checkpoint_dir``: path to the desired checkpoint folder
544
+ - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in 'latest' file. e.g., ``global_step14``
545
+ - ``exclude_frozen_parameters``: exclude frozen parameters
546
+ - ``lazy_mode``: get state_dict in lazy mode. It returns a dict of pesduo tensor instead of torch tensor, which is more memory efficient.
547
+ Convert the pesduo tensor to torch tensor by ``.contiguous()``
548
+
549
+ Returns:
550
+ - pytorch ``state_dict``
551
+
552
+ A typical usage might be ::
553
+
554
+ from deepspeed.utils.zero_to_fp32 import get_fp32_state_dict_from_zero_checkpoint
555
+ # do the training and checkpoint saving
556
+ state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir) # already on cpu
557
+ model = model.cpu() # move to cpu
558
+ model.load_state_dict(state_dict)
559
+ # submit to model hub or save the model to share with others
560
+
561
+ In this example the ``model`` will no longer be usable in the deepspeed context of the same
562
+ application. i.e. you will need to re-initialize the deepspeed engine, since
563
+ ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
564
+
565
+ If you want it all done for you, use ``load_state_dict_from_zero_checkpoint`` instead.
566
+
567
+ Note: the above usage may not work if your application doesn't have sufficient free CPU memory.
568
+ You may need to use the offline approach using the ``zero_to_fp32.py`` script that is saved with
569
+ the checkpoint. Or you can load state_dict in lazy mode ::
570
+
571
+ from deepspeed.utils.zero_to_fp32 import get_fp32_state_dict_from_zero_checkpoint
572
+ state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, lazy_mode=True) # not on cpu
573
+ for name, lazy_tensor in state_dict.item():
574
+ tensor = lazy_tensor.contiguous() # to cpu
575
+ print(name, tensor)
576
+ # del tensor to release memory if it no longer in use
577
+ """
578
+ if tag is None:
579
+ latest_path = os.path.join(checkpoint_dir, 'latest')
580
+ if os.path.isfile(latest_path):
581
+ with open(latest_path, 'r') as fd:
582
+ tag = fd.read().strip()
583
+ else:
584
+ raise ValueError(f"Unable to find 'latest' file at {latest_path}")
585
+
586
+ ds_checkpoint_dir = os.path.join(checkpoint_dir, tag)
587
+
588
+ if not os.path.isdir(ds_checkpoint_dir):
589
+ raise FileNotFoundError(f"Directory '{ds_checkpoint_dir}' doesn't exist")
590
+
591
+ state_dict = _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir, exclude_frozen_parameters)
592
+ if lazy_mode:
593
+ return state_dict
594
+ else:
595
+ return to_torch_tensor(state_dict)
596
+
597
+
598
+ def convert_zero_checkpoint_to_fp32_state_dict(checkpoint_dir,
599
+ output_dir,
600
+ max_shard_size="5GB",
601
+ safe_serialization=False,
602
+ tag=None,
603
+ exclude_frozen_parameters=False):
604
+ """
605
+ Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict`` file that can be
606
+ loaded with ``torch.load(file)`` + ``load_state_dict()`` and used for training without DeepSpeed.
607
+
608
+ Args:
609
+ - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
610
+ - ``output_dir``: directory to the pytorch fp32 state_dict output files
611
+ - ``max_shard_size``: the maximum size for a checkpoint before being sharded, default value is 5GB
612
+ - ``safe_serialization``: whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).
613
+ - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
614
+ - ``exclude_frozen_parameters``: exclude frozen parameters
615
+ """
616
+
617
+ # Dependency pre-check
618
+ if safe_serialization:
619
+ try:
620
+ from safetensors.torch import save_file
621
+ except ImportError:
622
+ print('If you want to use `safe_serialization`, please `pip install safetensors`')
623
+ raise
624
+ if max_shard_size is not None:
625
+ try:
626
+ from huggingface_hub import split_torch_state_dict_into_shards
627
+ except ImportError:
628
+ print('If you want to use `max_shard_size`, please `pip install huggingface_hub`')
629
+ raise
630
+
631
+ # Convert zero checkpoint to state_dict
632
+ state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir,
633
+ tag,
634
+ exclude_frozen_parameters,
635
+ lazy_mode=True)
636
+
637
+ # Shard the model if it is too big.
638
+ weights_name = "model.safetensors" if safe_serialization else "pytorch_model.bin"
639
+ if max_shard_size is not None:
640
+ filename_pattern = weights_name.replace(".bin", "{suffix}.bin").replace(".safetensors", "{suffix}.safetensors")
641
+ # an memory-efficient approach for sharding
642
+ empty_state_dict = to_torch_tensor(state_dict, return_empty_tensor=True)
643
+ state_dict_split = split_torch_state_dict_into_shards(empty_state_dict,
644
+ filename_pattern=filename_pattern,
645
+ max_shard_size=max_shard_size)
646
+ else:
647
+ from collections import namedtuple
648
+ StateDictSplit = namedtuple("StateDictSplit", ["is_sharded", "filename_to_tensors"])
649
+ state_dict_split = StateDictSplit(is_sharded=False,
650
+ filename_to_tensors={weights_name: list(state_dict.keys())})
651
+
652
+ # Save the model by shard
653
+ os.makedirs(output_dir, exist_ok=True)
654
+ filename_to_tensors = state_dict_split.filename_to_tensors.items()
655
+ for shard_file, tensors in tqdm(filename_to_tensors, desc="Saving checkpoint shards"):
656
+ shard_state_dict = {tensor_name: state_dict[tensor_name] for tensor_name in tensors}
657
+ shard_state_dict = to_torch_tensor(shard_state_dict)
658
+ output_path = os.path.join(output_dir, shard_file)
659
+ if safe_serialization:
660
+ save_file(shard_state_dict, output_path, metadata={"format": "pt"})
661
+ else:
662
+ torch.save(shard_state_dict, output_path)
663
+ # release the memory of current shard
664
+ for tensor_name in list(shard_state_dict.keys()):
665
+ del state_dict[tensor_name]
666
+ del shard_state_dict[tensor_name]
667
+ del shard_state_dict
668
+ gc.collect()
669
+
670
+ # Save index if sharded
671
+ if state_dict_split.is_sharded:
672
+ index = {
673
+ "metadata": state_dict_split.metadata,
674
+ "weight_map": state_dict_split.tensor_to_filename,
675
+ }
676
+ save_index_file = "model.safetensors.index.json" if safe_serialization else "pytorch_model.bin.index.json"
677
+ save_index_file = os.path.join(output_dir, save_index_file)
678
+ with open(save_index_file, "w", encoding="utf-8") as f:
679
+ content = json.dumps(index, indent=2, sort_keys=True) + "\n"
680
+ f.write(content)
681
+
682
+
683
+ def load_state_dict_from_zero_checkpoint(model, checkpoint_dir, tag=None):
684
+ """
685
+ 1. Put the provided model to cpu
686
+ 2. Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict``
687
+ 3. Load it into the provided model
688
+
689
+ Args:
690
+ - ``model``: the model object to update
691
+ - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
692
+ - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
693
+
694
+ Returns:
695
+ - ``model`: modified model
696
+
697
+ Make sure you have plenty of CPU memory available before you call this function. If you don't
698
+ have enough use the ``zero_to_fp32.py`` utility to do the conversion. You will find it
699
+ conveniently placed for you in the checkpoint folder.
700
+
701
+ A typical usage might be ::
702
+
703
+ from deepspeed.utils.zero_to_fp32 import load_state_dict_from_zero_checkpoint
704
+ model = load_state_dict_from_zero_checkpoint(trainer.model, checkpoint_dir)
705
+ # submit to model hub or save the model to share with others
706
+
707
+ Note, that once this was run, the ``model`` will no longer be usable in the deepspeed context
708
+ of the same application. i.e. you will need to re-initialize the deepspeed engine, since
709
+ ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
710
+
711
+ """
712
+ logger.info(f"Extracting fp32 weights")
713
+ state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag)
714
+
715
+ logger.info(f"Overwriting model with fp32 weights")
716
+ model = model.cpu()
717
+ model.load_state_dict(state_dict, strict=False)
718
+
719
+ return model
720
+
721
+
722
+ if __name__ == "__main__":
723
+ parser = argparse.ArgumentParser()
724
+ parser.add_argument("checkpoint_dir",
725
+ type=str,
726
+ help="path to the desired checkpoint folder, e.g., path/checkpoint-12")
727
+ parser.add_argument("output_dir",
728
+ type=str,
729
+ help="directory to the pytorch fp32 state_dict output files"
730
+ "(e.g. path/checkpoint-12-output/)")
731
+ parser.add_argument(
732
+ "--max_shard_size",
733
+ type=str,
734
+ default="5GB",
735
+ help="The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size"
736
+ "lower than this size. If expressed as a string, needs to be digits followed by a unit (like `5MB`"
737
+ "We default it to 5GB in order for models to be able to run easily on free-tier google colab instances"
738
+ "without CPU OOM issues.")
739
+ parser.add_argument(
740
+ "--safe_serialization",
741
+ default=False,
742
+ action='store_true',
743
+ help="Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).")
744
+ parser.add_argument("-t",
745
+ "--tag",
746
+ type=str,
747
+ default=None,
748
+ help="checkpoint tag used as a unique identifier for checkpoint. e.g., global_step1")
749
+ parser.add_argument("--exclude_frozen_parameters", action='store_true', help="exclude frozen parameters")
750
+ parser.add_argument("-d", "--debug", action='store_true', help="enable debug")
751
+ args = parser.parse_args()
752
+
753
+ debug = args.debug
754
+
755
+ convert_zero_checkpoint_to_fp32_state_dict(args.checkpoint_dir,
756
+ args.output_dir,
757
+ max_shard_size=args.max_shard_size,
758
+ safe_serialization=args.safe_serialization,
759
+ tag=args.tag,
760
+ exclude_frozen_parameters=args.exclude_frozen_parameters)