Hugging Face's logo

trappy
/
Twewy_08

Model card Files
xet
 Community
trappy commited on
Commit
234650c
·
1 Parent(s): 430f802

feat: upload project lora model

Browse files
Files changed (33) hide show
  1. .gitattributes +2 -0
  2. content/.config/.last_opt_in_prompt.yaml +1 -0
  3. content/.config/.last_survey_prompt.yaml +1 -0
  4. content/.config/.last_update_check.json +1 -0
  5. content/.config/active_config +1 -0
  6. content/.config/config_sentinel +0 -0
  7. content/.config/configurations/config_default +6 -0
  8. content/.config/default_configs.db +0 -0
  9. content/.config/gce +1 -0
  10. content/.config/logs/2023.10.05/19.31.42.136492.log +596 -0
  11. content/.config/logs/2023.10.05/19.32.16.008139.log +5 -0
  12. content/.config/logs/2023.10.05/19.32.29.466527.log +169 -0
  13. content/.config/logs/2023.10.05/19.32.44.892388.log +5 -0
  14. content/.config/logs/2023.10.05/19.32.58.628417.log +8 -0
  15. content/.config/logs/2023.10.05/19.32.59.577697.log +8 -0
  16. content/Twewy_08.safetensors +3 -0
  17. content/sample_data/README.md +19 -0
  18. content/sample_data/anscombe.json +49 -0
  19. content/sample_data/california_housing_test.csv +0 -0
  20. content/sample_data/california_housing_train.csv +0 -0
  21. content/sample_data/mnist_test.csv +3 -0
  22. content/sample_data/mnist_train_small.csv +3 -0
  23. content/sd-notebook-collection/.github/FUNDING.yml +13 -0
  24. content/sd-notebook-collection/AnimateDiff.ipynb +0 -0
  25. content/sd-notebook-collection/Anything-V3-Mirroring.ipynb +296 -0
  26. content/sd-notebook-collection/Infinite_Image_Browsing_Demo.ipynb +220 -0
  27. content/sd-notebook-collection/MANUAL.md +202 -0
  28. content/sd-notebook-collection/README.md +271 -0
  29. content/sd-notebook-collection/booru-i2i-scraper.ipynb +429 -0
  30. content/sd-notebook-collection/cagliostro-colab-ui.ipynb +1250 -0
  31. content/sd-notebook-collection/config/download_list.txt +18 -0
  32. content/sd-notebook-collection/script/vae.py +1166 -0
  33. content/sd-notebook-collection/sd-colab-toolkit.ipynb +920 -0
.gitattributes CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ content/sample_data/mnist_test.csv filter=lfs diff=lfs merge=lfs -text
37
+ content/sample_data/mnist_train_small.csv filter=lfs diff=lfs merge=lfs -text
content/.config/.last_opt_in_prompt.yaml ADDED
@@ -0,0 +1 @@
 
 
1
+ {}
content/.config/.last_survey_prompt.yaml ADDED
@@ -0,0 +1 @@
 
 
1
+ last_prompt_time: 1696534348.5676613
content/.config/.last_update_check.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"last_update_check_time": 1696534364.1681724, "last_update_check_revision": 20231002150006, "notifications": [], "last_nag_times": {}}
content/.config/active_config ADDED
@@ -0,0 +1 @@
 
 
1
+ default
content/.config/config_sentinel ADDED
File without changes
content/.config/configurations/config_default ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ [component_manager]
2
+ disable_update_check = true
3
+
4
+ [compute]
5
+ gce_metadata_read_timeout_sec = 0
6
+
content/.config/default_configs.db ADDED
Binary file (12.3 kB). View file
 
content/.config/gce ADDED
@@ -0,0 +1 @@
 
 
1
+ False
content/.config/logs/2023.10.05/19.31.42.136492.log ADDED
@@ -0,0 +1,596 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2023-10-05 19:31:42,142 DEBUG root Loaded Command Group: ['gcloud', 'components']
2
+ 2023-10-05 19:31:42,146 DEBUG root Loaded Command Group: ['gcloud', 'components', 'update']
3
+ 2023-10-05 19:31:42,149 DEBUG root Running [gcloud.components.update] with arguments: [--allow-no-backup: "True", --compile-python: "True", --quiet: "True", COMPONENT-IDS:7: "['core', 'gcloud-deps', 'bq', 'gcloud', 'gcloud-crc32c', 'gsutil', 'anthoscli']"]
4
+ 2023-10-05 19:31:42,150 INFO ___FILE_ONLY___ Beginning update. This process may take several minutes.
5
+
6
+ 2023-10-05 19:31:54,202 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
7
+ 2023-10-05 19:31:54,504 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components-2.json HTTP/1.1" 200 209942
8
+ 2023-10-05 19:31:54,526 INFO ___FILE_ONLY___
9
+
10
+ 2023-10-05 19:31:54,526 INFO ___FILE_ONLY___
11
+ Your current Google Cloud CLI version is: 449.0.0
12
+
13
+ 2023-10-05 19:31:54,527 INFO ___FILE_ONLY___ Installing components from version: 449.0.0
14
+
15
+ 2023-10-05 19:31:54,527 INFO ___FILE_ONLY___
16
+
17
+ 2023-10-05 19:31:54,528 DEBUG root Chosen display Format:table[box,title="These components will be removed."](details.display_name:label=Name:align=left,version.version_string:label=Version:align=right,data.size.size(zero="",min=1048576):label=Size:align=right)
18
+ 2023-10-05 19:31:54,528 DEBUG root Chosen display Format:table[box,title="These components will be updated."](details.display_name:label=Name:align=left,version.version_string:label=Version:align=right,data.size.size(zero="",min=1048576):label=Size:align=right)
19
+ 2023-10-05 19:31:54,529 DEBUG root Chosen display Format:table[box,title="These components will be installed."](details.display_name:label=Name:align=left,version.version_string:label=Version:align=right,data.size.size(zero="",min=1048576):label=Size:align=right)
20
+ 2023-10-05 19:31:54,538 INFO ___FILE_ONLY___ ┌─────────────────────────────────────────────────────────────────────────────┐
21
+ 2023-10-05 19:31:54,538 INFO ___FILE_ONLY___
22
+
23
+ 2023-10-05 19:31:54,538 INFO ___FILE_ONLY___ │ These components will be installed. │
24
+ 2023-10-05 19:31:54,538 INFO ___FILE_ONLY___
25
+
26
+ 2023-10-05 19:31:54,538 INFO ___FILE_ONLY___ ├─────────────────────────────────────────────────────┬────────────┬──────────┤
27
+ 2023-10-05 19:31:54,538 INFO ___FILE_ONLY___
28
+
29
+ 2023-10-05 19:31:54,538 INFO ___FILE_ONLY___ │ Name │ Version │ Size │
30
+ 2023-10-05 19:31:54,539 INFO ___FILE_ONLY___
31
+
32
+ 2023-10-05 19:31:54,539 INFO ___FILE_ONLY___ ├─────────────────────────────────────────────────────┼────────────┼──────────┤
33
+ 2023-10-05 19:31:54,539 INFO ___FILE_ONLY___
34
+
35
+ 2023-10-05 19:31:54,539 INFO ___FILE_ONLY___ │
36
+ 2023-10-05 19:31:54,539 INFO ___FILE_ONLY___ BigQuery Command Line Tool
37
+ 2023-10-05 19:31:54,539 INFO ___FILE_ONLY___
38
+ 2023-10-05 19:31:54,539 INFO ___FILE_ONLY___ │
39
+ 2023-10-05 19:31:54,539 INFO ___FILE_ONLY___ 2.0.98
40
+ 2023-10-05 19:31:54,539 INFO ___FILE_ONLY___
41
+ 2023-10-05 19:31:54,540 INFO ___FILE_ONLY___ │
42
+ 2023-10-05 19:31:54,540 INFO ___FILE_ONLY___ 1.6 MiB
43
+ 2023-10-05 19:31:54,540 INFO ___FILE_ONLY___
44
+ 2023-10-05 19:31:54,540 INFO ___FILE_ONLY___ │
45
+ 2023-10-05 19:31:54,540 INFO ___FILE_ONLY___
46
+
47
+ 2023-10-05 19:31:54,540 INFO ___FILE_ONLY___ │
48
+ 2023-10-05 19:31:54,540 INFO ___FILE_ONLY___ BigQuery Command Line Tool (Platform Specific)
49
+ 2023-10-05 19:31:54,540 INFO ___FILE_ONLY___
50
+ 2023-10-05 19:31:54,540 INFO ___FILE_ONLY___ │
51
+ 2023-10-05 19:31:54,541 INFO ___FILE_ONLY___ 2.0.98
52
+ 2023-10-05 19:31:54,541 INFO ___FILE_ONLY___
53
+ 2023-10-05 19:31:54,541 INFO ___FILE_ONLY___ │
54
+ 2023-10-05 19:31:54,541 INFO ___FILE_ONLY___ < 1 MiB
55
+ 2023-10-05 19:31:54,541 INFO ___FILE_ONLY___
56
+ 2023-10-05 19:31:54,541 INFO ___FILE_ONLY___ │
57
+ 2023-10-05 19:31:54,541 INFO ___FILE_ONLY___
58
+
59
+ 2023-10-05 19:31:54,541 INFO ___FILE_ONLY___ │
60
+ 2023-10-05 19:31:54,541 INFO ___FILE_ONLY___ Bundled Python 3.9
61
+ 2023-10-05 19:31:54,541 INFO ___FILE_ONLY___
62
+ 2023-10-05 19:31:54,542 INFO ___FILE_ONLY___ │
63
+ 2023-10-05 19:31:54,542 INFO ___FILE_ONLY___ 3.9.16
64
+ 2023-10-05 19:31:54,542 INFO ___FILE_ONLY___
65
+ 2023-10-05 19:31:54,542 INFO ___FILE_ONLY___ │
66
+ 2023-10-05 19:31:54,542 INFO ___FILE_ONLY___ 63.6 MiB
67
+ 2023-10-05 19:31:54,542 INFO ___FILE_ONLY___
68
+ 2023-10-05 19:31:54,542 INFO ___FILE_ONLY___ │
69
+ 2023-10-05 19:31:54,542 INFO ___FILE_ONLY___
70
+
71
+ 2023-10-05 19:31:54,542 INFO ___FILE_ONLY___ │
72
+ 2023-10-05 19:31:54,543 INFO ___FILE_ONLY___ Cloud Storage Command Line Tool
73
+ 2023-10-05 19:31:54,543 INFO ___FILE_ONLY___
74
+ 2023-10-05 19:31:54,543 INFO ___FILE_ONLY___ │
75
+ 2023-10-05 19:31:54,543 INFO ___FILE_ONLY___ 5.26
76
+ 2023-10-05 19:31:54,543 INFO ___FILE_ONLY___
77
+ 2023-10-05 19:31:54,543 INFO ___FILE_ONLY___ │
78
+ 2023-10-05 19:31:54,543 INFO ___FILE_ONLY___ 11.3 MiB
79
+ 2023-10-05 19:31:54,544 INFO ___FILE_ONLY___
80
+ 2023-10-05 19:31:54,544 INFO ___FILE_ONLY___ │
81
+ 2023-10-05 19:31:54,544 INFO ___FILE_ONLY___
82
+
83
+ 2023-10-05 19:31:54,544 INFO ___FILE_ONLY___ │
84
+ 2023-10-05 19:31:54,544 INFO ___FILE_ONLY___ Cloud Storage Command Line Tool (Platform Specific)
85
+ 2023-10-05 19:31:54,544 INFO ___FILE_ONLY___
86
+ 2023-10-05 19:31:54,544 INFO ___FILE_ONLY___ │
87
+ 2023-10-05 19:31:54,544 INFO ___FILE_ONLY___ 5.25
88
+ 2023-10-05 19:31:54,545 INFO ___FILE_ONLY___
89
+ 2023-10-05 19:31:54,545 INFO ___FILE_ONLY___ │
90
+ 2023-10-05 19:31:54,545 INFO ___FILE_ONLY___ < 1 MiB
91
+ 2023-10-05 19:31:54,545 INFO ___FILE_ONLY___
92
+ 2023-10-05 19:31:54,545 INFO ___FILE_ONLY___ │
93
+ 2023-10-05 19:31:54,546 INFO ___FILE_ONLY___
94
+
95
+ 2023-10-05 19:31:54,546 INFO ___FILE_ONLY___ │
96
+ 2023-10-05 19:31:54,546 INFO ___FILE_ONLY___ Google Cloud CLI Core Libraries (Platform Specific)
97
+ 2023-10-05 19:31:54,546 INFO ___FILE_ONLY___
98
+ 2023-10-05 19:31:54,546 INFO ___FILE_ONLY___ │
99
+ 2023-10-05 19:31:54,546 INFO ___FILE_ONLY___ 2023.09.15
100
+ 2023-10-05 19:31:54,546 INFO ___FILE_ONLY___
101
+ 2023-10-05 19:31:54,546 INFO ___FILE_ONLY___ │
102
+ 2023-10-05 19:31:54,547 INFO ___FILE_ONLY___ < 1 MiB
103
+ 2023-10-05 19:31:54,547 INFO ___FILE_ONLY___
104
+ 2023-10-05 19:31:54,547 INFO ___FILE_ONLY___ │
105
+ 2023-10-05 19:31:54,547 INFO ___FILE_ONLY___
106
+
107
+ 2023-10-05 19:31:54,547 INFO ___FILE_ONLY___ │
108
+ 2023-10-05 19:31:54,547 INFO ___FILE_ONLY___ Google Cloud CRC32C Hash Tool
109
+ 2023-10-05 19:31:54,547 INFO ___FILE_ONLY___
110
+ 2023-10-05 19:31:54,547 INFO ___FILE_ONLY___ │
111
+ 2023-10-05 19:31:54,547 INFO ___FILE_ONLY___ 1.0.0
112
+ 2023-10-05 19:31:54,548 INFO ___FILE_ONLY___
113
+ 2023-10-05 19:31:54,548 INFO ___FILE_ONLY___ │
114
+ 2023-10-05 19:31:54,548 INFO ___FILE_ONLY___ 1.2 MiB
115
+ 2023-10-05 19:31:54,548 INFO ___FILE_ONLY___
116
+ 2023-10-05 19:31:54,548 INFO ___FILE_ONLY___ │
117
+ 2023-10-05 19:31:54,548 INFO ___FILE_ONLY___
118
+
119
+ 2023-10-05 19:31:54,548 INFO ___FILE_ONLY___ │
120
+ 2023-10-05 19:31:54,548 INFO ___FILE_ONLY___ anthoscli
121
+ 2023-10-05 19:31:54,548 INFO ___FILE_ONLY___
122
+ 2023-10-05 19:31:54,549 INFO ___FILE_ONLY___ │
123
+ 2023-10-05 19:31:54,549 INFO ___FILE_ONLY___ 0.2.40
124
+ 2023-10-05 19:31:54,549 INFO ___FILE_ONLY___
125
+ 2023-10-05 19:31:54,549 INFO ___FILE_ONLY___ │
126
+ 2023-10-05 19:31:54,549 INFO ___FILE_ONLY___ 68.7 MiB
127
+ 2023-10-05 19:31:54,549 INFO ___FILE_ONLY___
128
+ 2023-10-05 19:31:54,549 INFO ___FILE_ONLY___ │
129
+ 2023-10-05 19:31:54,549 INFO ___FILE_ONLY___
130
+
131
+ 2023-10-05 19:31:54,549 INFO ___FILE_ONLY___ │
132
+ 2023-10-05 19:31:54,550 INFO ___FILE_ONLY___ gcloud cli dependencies
133
+ 2023-10-05 19:31:54,550 INFO ___FILE_ONLY___
134
+ 2023-10-05 19:31:54,550 INFO ___FILE_ONLY___ │
135
+ 2023-10-05 19:31:54,550 INFO ___FILE_ONLY___ 2021.04.16
136
+ 2023-10-05 19:31:54,550 INFO ___FILE_ONLY___
137
+ 2023-10-05 19:31:54,550 INFO ___FILE_ONLY___ │
138
+ 2023-10-05 19:31:54,550 INFO ___FILE_ONLY___ < 1 MiB
139
+ 2023-10-05 19:31:54,550 INFO ___FILE_ONLY___
140
+ 2023-10-05 19:31:54,550 INFO ___FILE_ONLY___ │
141
+ 2023-10-05 19:31:54,550 INFO ___FILE_ONLY___
142
+
143
+ 2023-10-05 19:31:54,551 INFO ___FILE_ONLY___ └─────────────────────────────────────────────────────┴────────────┴──────────┘
144
+ 2023-10-05 19:31:54,551 INFO ___FILE_ONLY___
145
+
146
+ 2023-10-05 19:31:54,551 INFO ___FILE_ONLY___
147
+
148
+ 2023-10-05 19:31:54,557 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
149
+ 2023-10-05 19:31:54,871 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/RELEASE_NOTES HTTP/1.1" 200 1085080
150
+ 2023-10-05 19:31:55,092 INFO ___FILE_ONLY___ For the latest full release notes, please visit:
151
+ https://cloud.google.com/sdk/release_notes
152
+
153
+
154
+ 2023-10-05 19:31:55,095 INFO ___FILE_ONLY___ ╔═════════════════════════════════════════���══════════════════╗
155
+
156
+ 2023-10-05 19:31:55,096 INFO ___FILE_ONLY___ ╠═ Creating update staging area ═╣
157
+
158
+ 2023-10-05 19:31:55,096 INFO ___FILE_ONLY___ ╚
159
+ 2023-10-05 19:31:55,096 INFO ___FILE_ONLY___ ══════
160
+ 2023-10-05 19:31:55,096 INFO ___FILE_ONLY___ ══════
161
+ 2023-10-05 19:31:55,096 INFO ___FILE_ONLY___ ══════
162
+ 2023-10-05 19:31:55,538 INFO ___FILE_ONLY___ ═
163
+ 2023-10-05 19:31:55,654 INFO ___FILE_ONLY___ ═
164
+ 2023-10-05 19:31:55,773 INFO ___FILE_ONLY___ ═
165
+ 2023-10-05 19:31:55,877 INFO ___FILE_ONLY___ ═
166
+ 2023-10-05 19:31:55,984 INFO ___FILE_ONLY___ ═
167
+ 2023-10-05 19:31:56,100 INFO ___FILE_ONLY___ ═
168
+ 2023-10-05 19:31:56,204 INFO ___FILE_ONLY___ ═
169
+ 2023-10-05 19:31:56,293 INFO ___FILE_ONLY___ ═
170
+ 2023-10-05 19:31:56,400 INFO ___FILE_ONLY___ ═
171
+ 2023-10-05 19:31:56,494 INFO ___FILE_ONLY___ ═
172
+ 2023-10-05 19:31:56,606 INFO ___FILE_ONLY___ ═
173
+ 2023-10-05 19:31:56,716 INFO ___FILE_ONLY___ ═
174
+ 2023-10-05 19:31:56,816 INFO ___FILE_ONLY___ ═
175
+ 2023-10-05 19:31:56,926 INFO ___FILE_ONLY___ ═
176
+ 2023-10-05 19:31:57,023 INFO ___FILE_ONLY___ ═
177
+ 2023-10-05 19:31:57,144 INFO ___FILE_ONLY___ ═
178
+ 2023-10-05 19:31:57,453 INFO ___FILE_ONLY___ ═
179
+ 2023-10-05 19:31:57,537 INFO ___FILE_ONLY___ ═
180
+ 2023-10-05 19:31:57,627 INFO ___FILE_ONLY___ ═
181
+ 2023-10-05 19:31:57,714 INFO ___FILE_ONLY___ ═
182
+ 2023-10-05 19:31:57,811 INFO ___FILE_ONLY___ ═
183
+ 2023-10-05 19:31:57,896 INFO ___FILE_ONLY___ ═
184
+ 2023-10-05 19:31:57,977 INFO ___FILE_ONLY___ ═
185
+ 2023-10-05 19:31:58,069 INFO ___FILE_ONLY___ ═
186
+ 2023-10-05 19:31:58,164 INFO ___FILE_ONLY___ ═
187
+ 2023-10-05 19:31:58,273 INFO ___FILE_ONLY___ ═
188
+ 2023-10-05 19:31:58,353 INFO ___FILE_ONLY___ ═
189
+ 2023-10-05 19:31:58,433 INFO ___FILE_ONLY___ ═
190
+ 2023-10-05 19:31:58,536 INFO ___FILE_ONLY___ ═
191
+ 2023-10-05 19:31:58,651 INFO ___FILE_ONLY___ ═
192
+ 2023-10-05 19:31:58,752 INFO ___FILE_ONLY___ ═
193
+ 2023-10-05 19:31:58,836 INFO ___FILE_ONLY___ ═
194
+ 2023-10-05 19:31:58,953 INFO ___FILE_ONLY___ ═
195
+ 2023-10-05 19:31:59,333 INFO ___FILE_ONLY___ ═
196
+ 2023-10-05 19:31:59,387 INFO ___FILE_ONLY___ ═
197
+ 2023-10-05 19:31:59,437 INFO ___FILE_ONLY___ ═
198
+ 2023-10-05 19:31:59,505 INFO ___FILE_ONLY___ ═
199
+ 2023-10-05 19:31:59,569 INFO ___FILE_ONLY___ ═
200
+ 2023-10-05 19:31:59,628 INFO ___FILE_ONLY___ ═
201
+ 2023-10-05 19:31:59,713 INFO ___FILE_ONLY___ ═
202
+ 2023-10-05 19:31:59,932 INFO ___FILE_ONLY___ ═
203
+ 2023-10-05 19:32:00,024 INFO ___FILE_ONLY___ ═
204
+ 2023-10-05 19:32:00,025 INFO ___FILE_ONLY___ ╝
205
+
206
+ 2023-10-05 19:32:00,270 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
207
+
208
+ 2023-10-05 19:32:00,270 INFO ___FILE_ONLY___ ╠═ Installing: BigQuery Command Line Tool ═╣
209
+
210
+ 2023-10-05 19:32:00,270 INFO ___FILE_ONLY___ ╚
211
+ 2023-10-05 19:32:00,275 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
212
+ 2023-10-05 19:32:00,410 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components/google-cloud-sdk-bq-20230913232318.tar.gz HTTP/1.1" 200 1659622
213
+ 2023-10-05 19:32:00,619 INFO ___FILE_ONLY___ ═
214
+ 2023-10-05 19:32:00,619 INFO ___FILE_ONLY___ ═
215
+ 2023-10-05 19:32:00,620 INFO ___FILE_ONLY___ ═
216
+ 2023-10-05 19:32:00,620 INFO ___FILE_ONLY___ ═
217
+ 2023-10-05 19:32:00,620 INFO ___FILE_ONLY___ ═
218
+ 2023-10-05 19:32:00,620 INFO ___FILE_ONLY___ ═
219
+ 2023-10-05 19:32:00,621 INFO ___FILE_ONLY___ ═
220
+ 2023-10-05 19:32:00,621 INFO ___FILE_ONLY___ ═
221
+ 2023-10-05 19:32:00,621 INFO ___FILE_ONLY___ ═
222
+ 2023-10-05 19:32:00,621 INFO ___FILE_ONLY___ ═
223
+ 2023-10-05 19:32:00,622 INFO ___FILE_ONLY___ ═
224
+ 2023-10-05 19:32:00,622 INFO ___FILE_ONLY___ ═
225
+ 2023-10-05 19:32:00,622 INFO ___FILE_ONLY___ ═
226
+ 2023-10-05 19:32:00,622 INFO ___FILE_ONLY___ ═
227
+ 2023-10-05 19:32:00,623 INFO ___FILE_ONLY___ ═
228
+ 2023-10-05 19:32:00,623 INFO ___FILE_ONLY___ ═
229
+ 2023-10-05 19:32:00,623 INFO ___FILE_ONLY___ ═
230
+ 2023-10-05 19:32:00,623 INFO ___FILE_ONLY___ ═
231
+ 2023-10-05 19:32:00,624 INFO ___FILE_ONLY___ ═
232
+ 2023-10-05 19:32:00,624 INFO ___FILE_ONLY___ ═
233
+ 2023-10-05 19:32:00,624 INFO ___FILE_ONLY___ ═
234
+ 2023-10-05 19:32:00,624 INFO ___FILE_ONLY___ ═
235
+ 2023-10-05 19:32:00,625 INFO ___FILE_ONLY___ ═
236
+ 2023-10-05 19:32:00,625 INFO ___FILE_ONLY___ ═
237
+ 2023-10-05 19:32:00,625 INFO ___FILE_ONLY___ ═
238
+ 2023-10-05 19:32:00,625 INFO ___FILE_ONLY___ ═
239
+ 2023-10-05 19:32:00,626 INFO ___FILE_ONLY___ ═
240
+ 2023-10-05 19:32:00,626 INFO ___FILE_ONLY___ ═
241
+ 2023-10-05 19:32:00,626 INFO ___FILE_ONLY___ ═
242
+ 2023-10-05 19:32:00,626 INFO ___FILE_ONLY___ ═
243
+ 2023-10-05 19:32:00,761 INFO ___FILE_ONLY___ ═
244
+ 2023-10-05 19:32:00,768 INFO ___FILE_ONLY___ ═
245
+ 2023-10-05 19:32:00,774 INFO ___FILE_ONLY___ ═
246
+ 2023-10-05 19:32:00,781 INFO ___FILE_ONLY___ ═
247
+ 2023-10-05 19:32:00,787 INFO ___FILE_ONLY___ ═
248
+ 2023-10-05 19:32:00,793 INFO ___FILE_ONLY___ ═
249
+ 2023-10-05 19:32:00,801 INFO ___FILE_ONLY___ ═
250
+ 2023-10-05 19:32:00,808 INFO ___FILE_ONLY___ ═
251
+ 2023-10-05 19:32:00,815 INFO ___FILE_ONLY___ ═
252
+ 2023-10-05 19:32:00,820 INFO ___FILE_ONLY___ ═
253
+ 2023-10-05 19:32:00,826 INFO ___FILE_ONLY___ ═
254
+ 2023-10-05 19:32:00,832 INFO ___FILE_ONLY___ ═
255
+ 2023-10-05 19:32:00,841 INFO ___FILE_ONLY___ ═
256
+ 2023-10-05 19:32:00,847 INFO ___FILE_ONLY___ ═
257
+ 2023-10-05 19:32:00,853 INFO ___FILE_ONLY___ ═
258
+ 2023-10-05 19:32:00,860 INFO ___FILE_ONLY___ ═
259
+ 2023-10-05 19:32:00,866 INFO ___FILE_ONLY___ ═
260
+ 2023-10-05 19:32:00,872 INFO ___FILE_ONLY___ ═
261
+ 2023-10-05 19:32:00,881 INFO ___FILE_ONLY___ ═
262
+ 2023-10-05 19:32:00,888 INFO ___FILE_ONLY___ ═
263
+ 2023-10-05 19:32:00,894 INFO ___FILE_ONLY___ ═
264
+ 2023-10-05 19:32:00,904 INFO ___FILE_ONLY___ ═
265
+ 2023-10-05 19:32:00,913 INFO ___FILE_ONLY___ ═
266
+ 2023-10-05 19:32:00,920 INFO ___FILE_ONLY___ ═
267
+ 2023-10-05 19:32:00,925 INFO ___FILE_ONLY___ ═
268
+ 2023-10-05 19:32:00,931 INFO ___FILE_ONLY___ ═
269
+ 2023-10-05 19:32:00,937 INFO ___FILE_ONLY___ ═
270
+ 2023-10-05 19:32:00,944 INFO ___FILE_ONLY___ ═
271
+ 2023-10-05 19:32:00,950 INFO ___FILE_ONLY___ ═
272
+ 2023-10-05 19:32:00,955 INFO ___FILE_ONLY___ ═
273
+ 2023-10-05 19:32:00,955 INFO ___FILE_ONLY___ ╝
274
+
275
+ 2023-10-05 19:32:00,974 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
276
+
277
+ 2023-10-05 19:32:00,974 INFO ___FILE_ONLY___ ╠═ Installing: BigQuery Command Line Tool (Platform Spec... ═╣
278
+
279
+ 2023-10-05 19:32:00,974 INFO ___FILE_ONLY___ ╚
280
+ 2023-10-05 19:32:00,979 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
281
+ 2023-10-05 19:32:01,106 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components/google-cloud-sdk-bq-nix-20230915145114.tar.gz HTTP/1.1" 200 1943
282
+ 2023-10-05 19:32:01,107 INFO ___FILE_ONLY___ ══════════════════════════════
283
+ 2023-10-05 19:32:01,108 INFO ___FILE_ONLY___ ══════════════════════════════
284
+ 2023-10-05 19:32:01,108 INFO ___FILE_ONLY___ ╝
285
+
286
+ 2023-10-05 19:32:01,119 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
287
+
288
+ 2023-10-05 19:32:01,120 INFO ___FILE_ONLY___ ╠═ Installing: Bundled Python 3.9 ═╣
289
+
290
+ 2023-10-05 19:32:01,120 INFO ___FILE_ONLY___ ╚
291
+ 2023-10-05 19:32:01,125 INFO ___FILE_ONLY___ ════════════════════════════════════════════════════════════
292
+ 2023-10-05 19:32:01,126 INFO ___FILE_ONLY___ ╝
293
+
294
+ 2023-10-05 19:32:01,129 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
295
+
296
+ 2023-10-05 19:32:01,129 INFO ___FILE_ONLY___ ╠═ Installing: Bundled Python 3.9 ═╣
297
+
298
+ 2023-10-05 19:32:01,129 INFO ___FILE_ONLY___ ╚
299
+ 2023-10-05 19:32:01,134 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
300
+ 2023-10-05 19:32:01,453 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components/google-cloud-sdk-bundled-python3-unix-linux-x86_64-20230707144938.tar.gz HTTP/1.1" 200 66719069
301
+ 2023-10-05 19:32:02,422 INFO ___FILE_ONLY___ ═
302
+ 2023-10-05 19:32:02,428 INFO ___FILE_ONLY___ ═
303
+ 2023-10-05 19:32:02,434 INFO ___FILE_ONLY___ ═
304
+ 2023-10-05 19:32:02,440 INFO ___FILE_ONLY___ ═
305
+ 2023-10-05 19:32:02,445 INFO ___FILE_ONLY___ ═
306
+ 2023-10-05 19:32:02,451 INFO ___FILE_ONLY___ ═
307
+ 2023-10-05 19:32:02,457 INFO ___FILE_ONLY___ ═
308
+ 2023-10-05 19:32:02,462 INFO ___FILE_ONLY___ ═
309
+ 2023-10-05 19:32:02,468 INFO ___FILE_ONLY___ ═
310
+ 2023-10-05 19:32:02,474 INFO ___FILE_ONLY___ ═
311
+ 2023-10-05 19:32:02,479 INFO ___FILE_ONLY___ ═
312
+ 2023-10-05 19:32:02,485 INFO ___FILE_ONLY___ ═
313
+ 2023-10-05 19:32:02,490 INFO ___FILE_ONLY___ ═
314
+ 2023-10-05 19:32:02,496 INFO ___FILE_ONLY___ ═
315
+ 2023-10-05 19:32:02,502 INFO ___FILE_ONLY___ ═
316
+ 2023-10-05 19:32:02,507 INFO ___FILE_ONLY___ ═
317
+ 2023-10-05 19:32:02,513 INFO ___FILE_ONLY___ ═
318
+ 2023-10-05 19:32:02,519 INFO ___FILE_ONLY___ ═
319
+ 2023-10-05 19:32:02,524 INFO ___FILE_ONLY___ ═
320
+ 2023-10-05 19:32:02,530 INFO ___FILE_ONLY___ ═
321
+ 2023-10-05 19:32:02,536 INFO ___FILE_ONLY___ ═
322
+ 2023-10-05 19:32:02,541 INFO ___FILE_ONLY___ ═
323
+ 2023-10-05 19:32:02,547 INFO ___FILE_ONLY___ ═
324
+ 2023-10-05 19:32:02,552 INFO ___FILE_ONLY___ ═
325
+ 2023-10-05 19:32:02,558 INFO ___FILE_ONLY___ ═
326
+ 2023-10-05 19:32:02,564 INFO ___FILE_ONLY___ ═
327
+ 2023-10-05 19:32:02,569 INFO ___FILE_ONLY___ ═
328
+ 2023-10-05 19:32:02,575 INFO ___FILE_ONLY___ ═
329
+ 2023-10-05 19:32:02,581 INFO ___FILE_ONLY___ ═
330
+ 2023-10-05 19:32:02,586 INFO ___FILE_ONLY___ ═
331
+ 2023-10-05 19:32:04,609 INFO ___FILE_ONLY___ ═
332
+ 2023-10-05 19:32:04,630 INFO ___FILE_ONLY___ ═
333
+ 2023-10-05 19:32:04,654 INFO ___FILE_ONLY___ ═
334
+ 2023-10-05 19:32:04,676 INFO ___FILE_ONLY___ ═
335
+ 2023-10-05 19:32:04,697 INFO ___FILE_ONLY___ ═
336
+ 2023-10-05 19:32:04,725 INFO ___FILE_ONLY___ ═
337
+ 2023-10-05 19:32:04,752 INFO ___FILE_ONLY___ ═
338
+ 2023-10-05 19:32:04,782 INFO ___FILE_ONLY___ ═
339
+ 2023-10-05 19:32:04,809 INFO ___FILE_ONLY___ ═
340
+ 2023-10-05 19:32:04,832 INFO ___FILE_ONLY___ ═
341
+ 2023-10-05 19:32:04,861 INFO ___FILE_ONLY___ ═
342
+ 2023-10-05 19:32:04,890 INFO ___FILE_ONLY___ ═
343
+ 2023-10-05 19:32:04,914 INFO ___FILE_ONLY___ ═
344
+ 2023-10-05 19:32:05,063 INFO ___FILE_ONLY___ ═
345
+ 2023-10-05 19:32:05,087 INFO ___FILE_ONLY___ ═
346
+ 2023-10-05 19:32:05,209 INFO ___FILE_ONLY___ ═
347
+ 2023-10-05 19:32:05,242 INFO ___FILE_ONLY___ ═
348
+ 2023-10-05 19:32:05,272 INFO ___FILE_ONLY___ ═
349
+ 2023-10-05 19:32:05,300 INFO ___FILE_ONLY___ ═
350
+ 2023-10-05 19:32:05,354 INFO ___FILE_ONLY___ ═
351
+ 2023-10-05 19:32:05,377 INFO ___FILE_ONLY___ ═
352
+ 2023-10-05 19:32:05,400 INFO ___FILE_ONLY___ ═
353
+ 2023-10-05 19:32:05,436 INFO ___FILE_ONLY___ ═
354
+ 2023-10-05 19:32:05,463 INFO ___FILE_ONLY___ ═
355
+ 2023-10-05 19:32:05,492 INFO ___FILE_ONLY___ ═
356
+ 2023-10-05 19:32:05,519 INFO ___FILE_ONLY___ ═
357
+ 2023-10-05 19:32:06,278 INFO ___FILE_ONLY___ ═
358
+ 2023-10-05 19:32:06,637 INFO ___FILE_ONLY___ ═
359
+ 2023-10-05 19:32:06,657 INFO ___FILE_ONLY___ ═
360
+ 2023-10-05 19:32:06,677 INFO ___FILE_ONLY___ ═
361
+ 2023-10-05 19:32:06,677 INFO ___FILE_ONLY___ ╝
362
+
363
+ 2023-10-05 19:32:06,753 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
364
+
365
+ 2023-10-05 19:32:06,753 INFO ___FILE_ONLY___ ╠═ Installing: Cloud Storage Command Line Tool ═╣
366
+
367
+ 2023-10-05 19:32:06,754 INFO ___FILE_ONLY___ ╚
368
+ 2023-10-05 19:32:06,758 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
369
+ 2023-10-05 19:32:07,070 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components/google-cloud-sdk-gsutil-20231002150006.tar.gz HTTP/1.1" 200 11838390
370
+ 2023-10-05 19:32:07,401 INFO ___FILE_ONLY___ ═
371
+ 2023-10-05 19:32:07,402 INFO ___FILE_ONLY___ ═
372
+ 2023-10-05 19:32:07,403 INFO ___FILE_ONLY___ ═
373
+ 2023-10-05 19:32:07,404 INFO ___FILE_ONLY___ ═
374
+ 2023-10-05 19:32:07,405 INFO ___FILE_ONLY___ ═
375
+ 2023-10-05 19:32:07,406 INFO ___FILE_ONLY___ ═
376
+ 2023-10-05 19:32:07,407 INFO ___FILE_ONLY___ ═
377
+ 2023-10-05 19:32:07,408 INFO ___FILE_ONLY___ ═
378
+ 2023-10-05 19:32:07,409 INFO ___FILE_ONLY___ ═
379
+ 2023-10-05 19:32:07,411 INFO ___FILE_ONLY___ ═
380
+ 2023-10-05 19:32:07,412 INFO ___FILE_ONLY___ ═
381
+ 2023-10-05 19:32:07,413 INFO ___FILE_ONLY___ ═
382
+ 2023-10-05 19:32:07,414 INFO ___FILE_ONLY___ ═
383
+ 2023-10-05 19:32:07,415 INFO ___FILE_ONLY___ ═
384
+ 2023-10-05 19:32:07,416 INFO ___FILE_ONLY___ ═
385
+ 2023-10-05 19:32:07,417 INFO ___FILE_ONLY___ ═
386
+ 2023-10-05 19:32:07,418 INFO ___FILE_ONLY___ ═
387
+ 2023-10-05 19:32:07,419 INFO ___FILE_ONLY___ ═
388
+ 2023-10-05 19:32:07,420 INFO ___FILE_ONLY___ ═
389
+ 2023-10-05 19:32:07,421 INFO ___FILE_ONLY___ ═
390
+ 2023-10-05 19:32:07,422 INFO ___FILE_ONLY___ ═
391
+ 2023-10-05 19:32:07,424 INFO ___FILE_ONLY___ ═
392
+ 2023-10-05 19:32:07,425 INFO ___FILE_ONLY___ ═
393
+ 2023-10-05 19:32:07,426 INFO ___FILE_ONLY___ ═
394
+ 2023-10-05 19:32:07,427 INFO ___FILE_ONLY___ ═
395
+ 2023-10-05 19:32:07,428 INFO ___FILE_ONLY___ ═
396
+ 2023-10-05 19:32:07,429 INFO ___FILE_ONLY___ ═
397
+ 2023-10-05 19:32:07,430 INFO ___FILE_ONLY___ ═
398
+ 2023-10-05 19:32:07,431 INFO ___FILE_ONLY___ ═
399
+ 2023-10-05 19:32:07,432 INFO ___FILE_ONLY___ ═
400
+ 2023-10-05 19:32:08,235 INFO ___FILE_ONLY___ ═
401
+ 2023-10-05 19:32:08,282 INFO ___FILE_ONLY___ ═
402
+ 2023-10-05 19:32:08,324 INFO ___FILE_ONLY___ ═
403
+ 2023-10-05 19:32:08,369 INFO ___FILE_ONLY___ ═
404
+ 2023-10-05 19:32:08,402 INFO ___FILE_ONLY___ ═
405
+ 2023-10-05 19:32:08,438 INFO ___FILE_ONLY___ ═
406
+ 2023-10-05 19:32:08,475 INFO ___FILE_ONLY___ ═
407
+ 2023-10-05 19:32:08,514 INFO ___FILE_ONLY___ ═
408
+ 2023-10-05 19:32:08,554 INFO ___FILE_ONLY___ ═
409
+ 2023-10-05 19:32:08,611 INFO ___FILE_ONLY___ ═
410
+ 2023-10-05 19:32:08,658 INFO ___FILE_ONLY___ ═
411
+ 2023-10-05 19:32:08,699 INFO ___FILE_ONLY___ ═
412
+ 2023-10-05 19:32:08,754 INFO ___FILE_ONLY___ ═
413
+ 2023-10-05 19:32:08,803 INFO ___FILE_ONLY___ ═
414
+ 2023-10-05 19:32:08,850 INFO ___FILE_ONLY___ ═
415
+ 2023-10-05 19:32:08,888 INFO ___FILE_ONLY___ ═
416
+ 2023-10-05 19:32:08,922 INFO ___FILE_ONLY___ ═
417
+ 2023-10-05 19:32:08,962 INFO ___FILE_ONLY___ ═
418
+ 2023-10-05 19:32:09,000 INFO ___FILE_ONLY___ ═
419
+ 2023-10-05 19:32:09,043 INFO ___FILE_ONLY___ ═
420
+ 2023-10-05 19:32:09,090 INFO ___FILE_ONLY___ ═
421
+ 2023-10-05 19:32:09,131 INFO ___FILE_ONLY___ ═
422
+ 2023-10-05 19:32:09,171 INFO ___FILE_ONLY___ ═
423
+ 2023-10-05 19:32:09,245 INFO ___FILE_ONLY___ ═
424
+ 2023-10-05 19:32:09,288 INFO ___FILE_ONLY___ ═
425
+ 2023-10-05 19:32:09,331 INFO ___FILE_ONLY___ ═
426
+ 2023-10-05 19:32:09,370 INFO ___FILE_ONLY___ ═
427
+ 2023-10-05 19:32:09,406 INFO ___FILE_ONLY___ ═
428
+ 2023-10-05 19:32:09,446 INFO ___FILE_ONLY___ ═
429
+ 2023-10-05 19:32:09,483 INFO ___FILE_ONLY___ ═
430
+ 2023-10-05 19:32:09,484 INFO ___FILE_ONLY___ ╝
431
+
432
+ 2023-10-05 19:32:09,563 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
433
+
434
+ 2023-10-05 19:32:09,563 INFO ___FILE_ONLY___ ╠═ Installing: Cloud Storage Command Line Tool (Platform... ═╣
435
+
436
+ 2023-10-05 19:32:09,563 INFO ___FILE_ONLY___ ╚
437
+ 2023-10-05 19:32:09,568 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
438
+ 2023-10-05 19:32:09,877 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components/google-cloud-sdk-gsutil-nix-20230915145114.tar.gz HTTP/1.1" 200 1955
439
+ 2023-10-05 19:32:09,878 INFO ___FILE_ONLY___ ══════════════════════════════
440
+ 2023-10-05 19:32:09,879 INFO ___FILE_ONLY___ ══════════════════════════════
441
+ 2023-10-05 19:32:09,879 INFO ___FILE_ONLY___ ╝
442
+
443
+ 2023-10-05 19:32:09,925 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
444
+
445
+ 2023-10-05 19:32:09,925 INFO ___FILE_ONLY___ ╠═ Installing: Default set of gcloud commands ═╣
446
+
447
+ 2023-10-05 19:32:09,925 INFO ___FILE_ONLY___ ╚
448
+ 2023-10-05 19:32:09,930 INFO ___FILE_ONLY___ ════════════════════════════════════════════════════════════
449
+ 2023-10-05 19:32:09,931 INFO ___FILE_ONLY___ ╝
450
+
451
+ 2023-10-05 19:32:09,933 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
452
+
453
+ 2023-10-05 19:32:09,933 INFO ___FILE_ONLY___ ╠═ Installing: Google Cloud CLI Core Libraries (Platform... ═╣
454
+
455
+ 2023-10-05 19:32:09,934 INFO ___FILE_ONLY___ ╚
456
+ 2023-10-05 19:32:09,938 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
457
+ 2023-10-05 19:32:10,243 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components/google-cloud-sdk-core-nix-20230915145114.tar.gz HTTP/1.1" 200 2319
458
+ 2023-10-05 19:32:10,244 INFO ___FILE_ONLY___ ══════════════════════════════
459
+ 2023-10-05 19:32:10,246 INFO ___FILE_ONLY___ ═══════════════
460
+ 2023-10-05 19:32:10,246 INFO ___FILE_ONLY___ ═══════════════
461
+ 2023-10-05 19:32:10,247 INFO ___FILE_ONLY___ ╝
462
+
463
+ 2023-10-05 19:32:10,257 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
464
+
465
+ 2023-10-05 19:32:10,257 INFO ___FILE_ONLY___ ╠═ Installing: Google Cloud CRC32C Hash Tool ═╣
466
+
467
+ 2023-10-05 19:32:10,257 INFO ___FILE_ONLY___ ╚
468
+ 2023-10-05 19:32:10,262 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
469
+ 2023-10-05 19:32:10,590 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components/google-cloud-sdk-gcloud-crc32c-linux-x86_64-20230922151743.tar.gz HTTP/1.1" 200 1288666
470
+ 2023-10-05 19:32:10,788 INFO ___FILE_ONLY___ ═
471
+ 2023-10-05 19:32:10,789 INFO ___FILE_ONLY___ ═
472
+ 2023-10-05 19:32:10,789 INFO ___FILE_ONLY___ ═
473
+ 2023-10-05 19:32:10,789 INFO ___FILE_ONLY___ ═
474
+ 2023-10-05 19:32:10,789 INFO ___FILE_ONLY___ ═
475
+ 2023-10-05 19:32:10,790 INFO ___FILE_ONLY___ ═
476
+ 2023-10-05 19:32:10,790 INFO ___FILE_ONLY___ ═
477
+ 2023-10-05 19:32:10,790 INFO ___FILE_ONLY___ ═
478
+ 2023-10-05 19:32:10,790 INFO ___FILE_ONLY___ ═
479
+ 2023-10-05 19:32:10,791 INFO ___FILE_ONLY___ ═
480
+ 2023-10-05 19:32:10,791 INFO ___FILE_ONLY___ ═
481
+ 2023-10-05 19:32:10,791 INFO ___FILE_ONLY___ ═
482
+ 2023-10-05 19:32:10,791 INFO ___FILE_ONLY___ ═
483
+ 2023-10-05 19:32:10,792 INFO ___FILE_ONLY___ ═
484
+ 2023-10-05 19:32:10,792 INFO ___FILE_ONLY___ ═
485
+ 2023-10-05 19:32:10,792 INFO ___FILE_ONLY___ ═
486
+ 2023-10-05 19:32:10,792 INFO ___FILE_ONLY___ ═
487
+ 2023-10-05 19:32:10,792 INFO ___FILE_ONLY___ ═
488
+ 2023-10-05 19:32:10,793 INFO ___FILE_ONLY___ ═
489
+ 2023-10-05 19:32:10,793 INFO ___FILE_ONLY___ ═
490
+ 2023-10-05 19:32:10,793 INFO ___FILE_ONLY___ ═
491
+ 2023-10-05 19:32:10,793 INFO ___FILE_ONLY___ ═
492
+ 2023-10-05 19:32:10,794 INFO ___FILE_ONLY___ ═
493
+ 2023-10-05 19:32:10,794 INFO ___FILE_ONLY___ ═
494
+ 2023-10-05 19:32:10,794 INFO ___FILE_ONLY___ ═
495
+ 2023-10-05 19:32:10,795 INFO ___FILE_ONLY___ ═
496
+ 2023-10-05 19:32:10,795 INFO ___FILE_ONLY___ ═
497
+ 2023-10-05 19:32:10,795 INFO ___FILE_ONLY___ ═
498
+ 2023-10-05 19:32:10,796 INFO ___FILE_ONLY___ ═
499
+ 2023-10-05 19:32:10,796 INFO ___FILE_ONLY___ ═
500
+ 2023-10-05 19:32:10,834 INFO ___FILE_ONLY___ ═══════════════
501
+ 2023-10-05 19:32:10,835 INFO ___FILE_ONLY___ ═══════════════
502
+ 2023-10-05 19:32:10,835 INFO ___FILE_ONLY___ ╝
503
+
504
+ 2023-10-05 19:32:10,846 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
505
+
506
+ 2023-10-05 19:32:10,846 INFO ___FILE_ONLY___ ╠═ Installing: Google Cloud CRC32C Hash Tool ═╣
507
+
508
+ 2023-10-05 19:32:10,847 INFO ___FILE_ONLY___ ╚
509
+ 2023-10-05 19:32:10,852 INFO ___FILE_ONLY___ ════════════════════════════════════════════════════════════
510
+ 2023-10-05 19:32:10,852 INFO ___FILE_ONLY___ ╝
511
+
512
+ 2023-10-05 19:32:10,856 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
513
+
514
+ 2023-10-05 19:32:10,856 INFO ___FILE_ONLY___ ╠═ Installing: anthoscli ═╣
515
+
516
+ 2023-10-05 19:32:10,856 INFO ___FILE_ONLY___ ╚
517
+ 2023-10-05 19:32:10,862 INFO ___FILE_ONLY___ ════════════════════════════════════════════════════════════
518
+ 2023-10-05 19:32:10,862 INFO ___FILE_ONLY___ ╝
519
+
520
+ 2023-10-05 19:32:10,865 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
521
+
522
+ 2023-10-05 19:32:10,865 INFO ___FILE_ONLY___ ╠═ Installing: anthoscli ═╣
523
+
524
+ 2023-10-05 19:32:10,865 INFO ___FILE_ONLY___ ╚
525
+ 2023-10-05 19:32:10,870 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
526
+ 2023-10-05 19:32:11,180 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components/google-cloud-sdk-anthoscli-linux-x86_64-20230915145114.tar.gz HTTP/1.1" 200 72006203
527
+ 2023-10-05 19:32:12,142 INFO ___FILE_ONLY___ ═
528
+ 2023-10-05 19:32:12,149 INFO ___FILE_ONLY___ ═
529
+ 2023-10-05 19:32:12,155 INFO ___FILE_ONLY___ ═
530
+ 2023-10-05 19:32:12,161 INFO ___FILE_ONLY___ ═
531
+ 2023-10-05 19:32:12,167 INFO ___FILE_ONLY___ ═
532
+ 2023-10-05 19:32:12,173 INFO ___FILE_ONLY___ ═
533
+ 2023-10-05 19:32:12,179 INFO ___FILE_ONLY___ ═
534
+ 2023-10-05 19:32:12,185 INFO ___FILE_ONLY___ ═
535
+ 2023-10-05 19:32:12,191 INFO ___FILE_ONLY___ ═
536
+ 2023-10-05 19:32:12,197 INFO ___FILE_ONLY___ ═
537
+ 2023-10-05 19:32:12,204 INFO ___FILE_ONLY___ ═
538
+ 2023-10-05 19:32:12,210 INFO ___FILE_ONLY___ ═
539
+ 2023-10-05 19:32:12,216 INFO ___FILE_ONLY___ ═
540
+ 2023-10-05 19:32:12,222 INFO ___FILE_ONLY___ ═
541
+ 2023-10-05 19:32:12,228 INFO ___FILE_ONLY___ ═
542
+ 2023-10-05 19:32:12,235 INFO ___FILE_ONLY___ ═
543
+ 2023-10-05 19:32:12,241 INFO ___FILE_ONLY___ ═
544
+ 2023-10-05 19:32:12,247 INFO ___FILE_ONLY___ ═
545
+ 2023-10-05 19:32:12,254 INFO ___FILE_ONLY___ ═
546
+ 2023-10-05 19:32:12,260 INFO ___FILE_ONLY___ ═
547
+ 2023-10-05 19:32:12,266 INFO ___FILE_ONLY___ ═
548
+ 2023-10-05 19:32:12,272 INFO ___FILE_ONLY___ ═
549
+ 2023-10-05 19:32:12,279 INFO ___FILE_ONLY___ ═
550
+ 2023-10-05 19:32:12,285 INFO ___FILE_ONLY___ ═
551
+ 2023-10-05 19:32:12,291 INFO ___FILE_ONLY___ ═
552
+ 2023-10-05 19:32:12,297 INFO ___FILE_ONLY___ ═
553
+ 2023-10-05 19:32:12,303 INFO ___FILE_ONLY___ ═
554
+ 2023-10-05 19:32:12,310 INFO ___FILE_ONLY___ ═
555
+ 2023-10-05 19:32:12,316 INFO ___FILE_ONLY___ ═
556
+ 2023-10-05 19:32:12,322 INFO ___FILE_ONLY___ ═
557
+ 2023-10-05 19:32:14,860 INFO ___FILE_ONLY___ ══════════
558
+ 2023-10-05 19:32:14,867 INFO ___FILE_ONLY___ ═════════
559
+ 2023-10-05 19:32:14,902 INFO ___FILE_ONLY___ ═══════════
560
+ 2023-10-05 19:32:14,902 INFO ___FILE_ONLY___ ╝
561
+
562
+ 2023-10-05 19:32:14,943 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
563
+
564
+ 2023-10-05 19:32:14,944 INFO ___FILE_ONLY___ ╠═ Installing: gcloud cli dependencies ═╣
565
+
566
+ 2023-10-05 19:32:14,944 INFO ___FILE_ONLY___ ╚
567
+ 2023-10-05 19:32:14,948 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
568
+ 2023-10-05 19:32:15,269 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components/google-cloud-sdk-gcloud-deps-linux-x86_64-20210416153011.tar.gz HTTP/1.1" 200 104
569
+ 2023-10-05 19:32:15,270 INFO ___FILE_ONLY___ ══════════════════════════════
570
+ 2023-10-05 19:32:15,270 INFO ___FILE_ONLY___ ══════════════════════════════
571
+ 2023-10-05 19:32:15,270 INFO ___FILE_ONLY___ ╝
572
+
573
+ 2023-10-05 19:32:15,281 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
574
+
575
+ 2023-10-05 19:32:15,281 INFO ___FILE_ONLY___ ╠═ Creating backup and activating new installation ═╣
576
+
577
+ 2023-10-05 19:32:15,281 INFO ___FILE_ONLY___ ╚
578
+ 2023-10-05 19:32:15,281 DEBUG root Attempting to move directory [/tools/google-cloud-sdk] to [/tools/google-cloud-sdk.staging/.install/.backup]
579
+ 2023-10-05 19:32:15,281 INFO ___FILE_ONLY___ ══════════════════════════════
580
+ 2023-10-05 19:32:15,282 DEBUG root Attempting to move directory [/tools/google-cloud-sdk.staging] to [/tools/google-cloud-sdk]
581
+ 2023-10-05 19:32:15,282 INFO ___FILE_ONLY___ ══════════════════════════════
582
+ 2023-10-05 19:32:15,282 INFO ___FILE_ONLY___ ╝
583
+
584
+ 2023-10-05 19:32:15,287 DEBUG root Updating notification cache...
585
+ 2023-10-05 19:32:15,287 INFO ___FILE_ONLY___
586
+
587
+ 2023-10-05 19:32:15,290 INFO ___FILE_ONLY___ Performing post processing steps...
588
+ 2023-10-05 19:32:15,290 DEBUG root Executing command: ['python3', '-S', '/tools/google-cloud-sdk/lib/gcloud.py', 'components', 'post-process']
589
+ 2023-10-05 19:32:28,383 DEBUG ___FILE_ONLY___
590
+ 2023-10-05 19:32:28,384 DEBUG ___FILE_ONLY___
591
+ 2023-10-05 19:32:28,560 INFO ___FILE_ONLY___
592
+ Update done!
593
+
594
+
595
+ 2023-10-05 19:32:28,566 DEBUG root Chosen display Format:none
596
+ 2023-10-05 19:32:28,566 INFO root Display format: "none"
content/.config/logs/2023.10.05/19.32.16.008139.log ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ 2023-10-05 19:32:16,009 DEBUG root Loaded Command Group: ['gcloud', 'components']
2
+ 2023-10-05 19:32:16,012 DEBUG root Loaded Command Group: ['gcloud', 'components', 'post_process']
3
+ 2023-10-05 19:32:16,015 DEBUG root Running [gcloud.components.post-process] with arguments: []
4
+ 2023-10-05 19:32:28,222 DEBUG root Chosen display Format:none
5
+ 2023-10-05 19:32:28,223 INFO root Display format: "none"
content/.config/logs/2023.10.05/19.32.29.466527.log ADDED
@@ -0,0 +1,169 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2023-10-05 19:32:29,467 DEBUG root Loaded Command Group: ['gcloud', 'components']
2
+ 2023-10-05 19:32:29,471 DEBUG root Loaded Command Group: ['gcloud', 'components', 'update']
3
+ 2023-10-05 19:32:29,474 DEBUG root Running [gcloud.components.update] with arguments: [--quiet: "True", COMPONENT-IDS:8: "['gcloud', 'core', 'bq', 'gsutil', 'compute', 'preview', 'alpha', 'beta']"]
4
+ 2023-10-05 19:32:29,475 INFO ___FILE_ONLY___ Beginning update. This process may take several minutes.
5
+
6
+ 2023-10-05 19:32:29,482 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
7
+ 2023-10-05 19:32:29,781 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components-2.json HTTP/1.1" 200 209942
8
+ 2023-10-05 19:32:29,882 WARNING root Component [preview] no longer exists.
9
+ 2023-10-05 19:32:29,882 WARNING root Component [compute] no longer exists.
10
+ 2023-10-05 19:32:29,883 INFO ___FILE_ONLY___
11
+
12
+ 2023-10-05 19:32:29,884 INFO ___FILE_ONLY___
13
+ Your current Google Cloud CLI version is: 449.0.0
14
+
15
+ 2023-10-05 19:32:29,884 INFO ___FILE_ONLY___ Installing components from version: 449.0.0
16
+
17
+ 2023-10-05 19:32:29,884 INFO ___FILE_ONLY___
18
+
19
+ 2023-10-05 19:32:29,885 DEBUG root Chosen display Format:table[box,title="These components will be removed."](details.display_name:label=Name:align=left,version.version_string:label=Version:align=right,data.size.size(zero="",min=1048576):label=Size:align=right)
20
+ 2023-10-05 19:32:29,885 DEBUG root Chosen display Format:table[box,title="These components will be updated."](details.display_name:label=Name:align=left,version.version_string:label=Version:align=right,data.size.size(zero="",min=1048576):label=Size:align=right)
21
+ 2023-10-05 19:32:29,886 DEBUG root Chosen display Format:table[box,title="These components will be installed."](details.display_name:label=Name:align=left,version.version_string:label=Version:align=right,data.size.size(zero="",min=1048576):label=Size:align=right)
22
+ 2023-10-05 19:32:29,888 INFO ___FILE_ONLY___ ┌──────────────────────────────────────────────┐
23
+ 2023-10-05 19:32:29,888 INFO ___FILE_ONLY___
24
+
25
+ 2023-10-05 19:32:29,888 INFO ___FILE_ONLY___ │ These components will be installed. │
26
+ 2023-10-05 19:32:29,888 INFO ___FILE_ONLY___
27
+
28
+ 2023-10-05 19:32:29,888 INFO ___FILE_ONLY___ ├───────────────────────┬────────────┬─────────┤
29
+ 2023-10-05 19:32:29,889 INFO ___FILE_ONLY___
30
+
31
+ 2023-10-05 19:32:29,889 INFO ___FILE_ONLY___ │ Name │ Version │ Size │
32
+ 2023-10-05 19:32:29,889 INFO ___FILE_ONLY___
33
+
34
+ 2023-10-05 19:32:29,889 INFO ___FILE_ONLY___ ├───────────────────────┼────────────┼─────────┤
35
+ 2023-10-05 19:32:29,889 INFO ___FILE_ONLY___
36
+
37
+ 2023-10-05 19:32:29,889 INFO ___FILE_ONLY___ │
38
+ 2023-10-05 19:32:29,889 INFO ___FILE_ONLY___ gcloud Alpha Commands
39
+ 2023-10-05 19:32:29,889 INFO ___FILE_ONLY___
40
+ 2023-10-05 19:32:29,889 INFO ___FILE_ONLY___ │
41
+ 2023-10-05 19:32:29,889 INFO ___FILE_ONLY___ 2023.10.02
42
+ 2023-10-05 19:32:29,890 INFO ___FILE_ONLY___
43
+ 2023-10-05 19:32:29,890 INFO ___FILE_ONLY___ │
44
+ 2023-10-05 19:32:29,890 INFO ___FILE_ONLY___ < 1 MiB
45
+ 2023-10-05 19:32:29,890 INFO ___FILE_ONLY___
46
+ 2023-10-05 19:32:29,890 INFO ___FILE_ONLY___ │
47
+ 2023-10-05 19:32:29,890 INFO ___FILE_ONLY___
48
+
49
+ 2023-10-05 19:32:29,890 INFO ___FILE_ONLY___ │
50
+ 2023-10-05 19:32:29,891 INFO ___FILE_ONLY___ gcloud Beta Commands
51
+ 2023-10-05 19:32:29,891 INFO ___FILE_ONLY___
52
+ 2023-10-05 19:32:29,891 INFO ___FILE_ONLY___ │
53
+ 2023-10-05 19:32:29,891 INFO ___FILE_ONLY___ 2023.10.02
54
+ 2023-10-05 19:32:29,891 INFO ___FILE_ONLY___
55
+ 2023-10-05 19:32:29,891 INFO ___FILE_ONLY___ │
56
+ 2023-10-05 19:32:29,891 INFO ___FILE_ONLY___ < 1 MiB
57
+ 2023-10-05 19:32:29,891 INFO ___FILE_ONLY___
58
+ 2023-10-05 19:32:29,892 INFO ___FILE_ONLY___ │
59
+ 2023-10-05 19:32:29,892 INFO ___FILE_ONLY___
60
+
61
+ 2023-10-05 19:32:29,892 INFO ___FILE_ONLY___ └───────────────────────┴────────────┴─────────┘
62
+ 2023-10-05 19:32:29,892 INFO ___FILE_ONLY___
63
+
64
+ 2023-10-05 19:32:29,892 INFO ___FILE_ONLY___
65
+
66
+ 2023-10-05 19:32:29,897 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
67
+ 2023-10-05 19:32:30,031 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/RELEASE_NOTES HTTP/1.1" 200 1085080
68
+ 2023-10-05 19:32:30,255 INFO ___FILE_ONLY___ For the latest full release notes, please visit:
69
+ https://cloud.google.com/sdk/release_notes
70
+
71
+
72
+ 2023-10-05 19:32:30,258 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
73
+
74
+ 2023-10-05 19:32:30,258 INFO ___FILE_ONLY___ ╠═ Creating update staging area ═╣
75
+
76
+ 2023-10-05 19:32:30,258 INFO ___FILE_ONLY___ ╚
77
+ 2023-10-05 19:32:30,258 INFO ___FILE_ONLY___ ══════
78
+ 2023-10-05 19:32:31,713 INFO ___FILE_ONLY___ ══════
79
+ 2023-10-05 19:32:31,713 INFO ___FILE_ONLY___ ══════
80
+ 2023-10-05 19:32:32,239 INFO ___FILE_ONLY___ ═
81
+ 2023-10-05 19:32:32,485 INFO ___FILE_ONLY___ ═
82
+ 2023-10-05 19:32:32,687 INFO ___FILE_ONLY___ ═
83
+ 2023-10-05 19:32:33,272 INFO ___FILE_ONLY___ ═
84
+ 2023-10-05 19:32:33,428 INFO ___FILE_ONLY___ ═
85
+ 2023-10-05 19:32:33,609 INFO ___FILE_ONLY___ ═
86
+ 2023-10-05 19:32:33,762 INFO ___FILE_ONLY___ ═
87
+ 2023-10-05 19:32:33,971 INFO ___FILE_ONLY___ ═
88
+ 2023-10-05 19:32:34,144 INFO ___FILE_ONLY___ ═
89
+ 2023-10-05 19:32:34,388 INFO ___FILE_ONLY___ ═
90
+ 2023-10-05 19:32:34,516 INFO ___FILE_ONLY___ ═
91
+ 2023-10-05 19:32:34,710 INFO ___FILE_ONLY___ ═
92
+ 2023-10-05 19:32:35,037 INFO ___FILE_ONLY___ ═
93
+ 2023-10-05 19:32:35,194 INFO ___FILE_ONLY___ ═
94
+ 2023-10-05 19:32:35,323 INFO ___FILE_ONLY___ ═
95
+ 2023-10-05 19:32:35,480 INFO ___FILE_ONLY___ ═
96
+ 2023-10-05 19:32:35,581 INFO ___FILE_ONLY___ ═
97
+ 2023-10-05 19:32:35,688 INFO ___FILE_ONLY___ ═
98
+ 2023-10-05 19:32:35,811 INFO ___FILE_ONLY___ ═
99
+ 2023-10-05 19:32:35,932 INFO ___FILE_ONLY___ ═
100
+ 2023-10-05 19:32:36,071 INFO ___FILE_ONLY___ ═
101
+ 2023-10-05 19:32:36,205 INFO ___FILE_ONLY___ ═
102
+ 2023-10-05 19:32:36,312 INFO ___FILE_ONLY___ ═
103
+ 2023-10-05 19:32:36,403 INFO ___FILE_ONLY___ ═
104
+ 2023-10-05 19:32:36,499 INFO ___FILE_ONLY___ ═
105
+ 2023-10-05 19:32:36,618 INFO ___FILE_ONLY___ ═
106
+ 2023-10-05 19:32:36,718 INFO ___FILE_ONLY___ ═
107
+ 2023-10-05 19:32:36,829 INFO ___FILE_ONLY___ ═
108
+ 2023-10-05 19:32:36,959 INFO ___FILE_ONLY___ ═
109
+ 2023-10-05 19:32:37,060 INFO ___FILE_ONLY___ ═
110
+ 2023-10-05 19:32:37,184 INFO ___FILE_ONLY___ ═
111
+ 2023-10-05 19:32:37,312 INFO ___FILE_ONLY___ ═
112
+ 2023-10-05 19:32:37,406 INFO ___FILE_ONLY___ ═
113
+ 2023-10-05 19:32:37,526 INFO ___FILE_ONLY___ ═
114
+ 2023-10-05 19:32:37,649 INFO ___FILE_ONLY___ ═
115
+ 2023-10-05 19:32:37,753 INFO ___FILE_ONLY___ ═
116
+ 2023-10-05 19:32:37,859 INFO ___FILE_ONLY___ ═
117
+ 2023-10-05 19:32:38,013 INFO ___FILE_ONLY___ ═
118
+ 2023-10-05 19:32:38,349 INFO ___FILE_ONLY___ ═
119
+ 2023-10-05 19:32:38,454 INFO ___FILE_ONLY___ ═
120
+ 2023-10-05 19:32:38,567 INFO ___FILE_ONLY___ ═
121
+ 2023-10-05 19:32:38,683 INFO ___FILE_ONLY___ ═
122
+ 2023-10-05 19:32:38,683 INFO ___FILE_ONLY___ ╝
123
+
124
+ 2023-10-05 19:32:43,513 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
125
+
126
+ 2023-10-05 19:32:43,514 INFO ___FILE_ONLY___ ╠═ Installing: gcloud Alpha Commands ═╣
127
+
128
+ 2023-10-05 19:32:43,514 INFO ___FILE_ONLY___ ╚
129
+ 2023-10-05 19:32:43,518 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
130
+ 2023-10-05 19:32:43,849 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components/google-cloud-sdk-alpha-20231002150006.tar.gz HTTP/1.1" 200 800
131
+ 2023-10-05 19:32:43,849 INFO ___FILE_ONLY___ ══════════════════════════════
132
+ 2023-10-05 19:32:43,851 INFO ___FILE_ONLY___ ══════════════════════════════
133
+ 2023-10-05 19:32:43,851 INFO ___FILE_ONLY___ ╝
134
+
135
+ 2023-10-05 19:32:43,861 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
136
+
137
+ 2023-10-05 19:32:43,861 INFO ___FILE_ONLY___ ╠═ Installing: gcloud Beta Commands ═╣
138
+
139
+ 2023-10-05 19:32:43,861 INFO ___FILE_ONLY___ ╚
140
+ 2023-10-05 19:32:43,865 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
141
+ 2023-10-05 19:32:44,149 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components/google-cloud-sdk-beta-20231002150006.tar.gz HTTP/1.1" 200 797
142
+ 2023-10-05 19:32:44,150 INFO ___FILE_ONLY___ ══════════════════════════════
143
+ 2023-10-05 19:32:44,151 INFO ___FILE_ONLY___ ══════════════════════════════
144
+ 2023-10-05 19:32:44,151 INFO ___FILE_ONLY___ ╝
145
+
146
+ 2023-10-05 19:32:44,161 INFO ___FILE_ONLY___ ��════════════════════════════════════════════════════════════╗
147
+
148
+ 2023-10-05 19:32:44,161 INFO ___FILE_ONLY___ ╠═ Creating backup and activating new installation ═╣
149
+
150
+ 2023-10-05 19:32:44,161 INFO ___FILE_ONLY___ ╚
151
+ 2023-10-05 19:32:44,161 DEBUG root Attempting to move directory [/tools/google-cloud-sdk] to [/tools/google-cloud-sdk.staging/.install/.backup]
152
+ 2023-10-05 19:32:44,162 INFO ___FILE_ONLY___ ══════════════════════════════
153
+ 2023-10-05 19:32:44,162 DEBUG root Attempting to move directory [/tools/google-cloud-sdk.staging] to [/tools/google-cloud-sdk]
154
+ 2023-10-05 19:32:44,162 INFO ___FILE_ONLY___ ══════════════════════════════
155
+ 2023-10-05 19:32:44,162 INFO ___FILE_ONLY___ ╝
156
+
157
+ 2023-10-05 19:32:44,168 DEBUG root Updating notification cache...
158
+ 2023-10-05 19:32:44,168 INFO ___FILE_ONLY___
159
+
160
+ 2023-10-05 19:32:44,171 INFO ___FILE_ONLY___ Performing post processing steps...
161
+ 2023-10-05 19:32:44,172 DEBUG root Executing command: ['python3', '-S', '/tools/google-cloud-sdk/lib/gcloud.py', 'components', 'post-process']
162
+ 2023-10-05 19:32:57,570 DEBUG ___FILE_ONLY___
163
+ 2023-10-05 19:32:57,570 DEBUG ___FILE_ONLY___
164
+ 2023-10-05 19:32:57,692 INFO ___FILE_ONLY___
165
+ Update done!
166
+
167
+
168
+ 2023-10-05 19:32:57,699 DEBUG root Chosen display Format:none
169
+ 2023-10-05 19:32:57,700 INFO root Display format: "none"
content/.config/logs/2023.10.05/19.32.44.892388.log ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ 2023-10-05 19:32:44,893 DEBUG root Loaded Command Group: ['gcloud', 'components']
2
+ 2023-10-05 19:32:44,895 DEBUG root Loaded Command Group: ['gcloud', 'components', 'post_process']
3
+ 2023-10-05 19:32:44,898 DEBUG root Running [gcloud.components.post-process] with arguments: []
4
+ 2023-10-05 19:32:57,429 DEBUG root Chosen display Format:none
5
+ 2023-10-05 19:32:57,430 INFO root Display format: "none"
content/.config/logs/2023.10.05/19.32.58.628417.log ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ 2023-10-05 19:32:58,630 DEBUG root Loaded Command Group: ['gcloud', 'config']
2
+ 2023-10-05 19:32:58,668 DEBUG root Loaded Command Group: ['gcloud', 'config', 'set']
3
+ 2023-10-05 19:32:58,671 DEBUG root Running [gcloud.config.set] with arguments: [SECTION/PROPERTY: "component_manager/disable_update_check", VALUE: "true"]
4
+ 2023-10-05 19:32:58,672 INFO ___FILE_ONLY___ Updated property [component_manager/disable_update_check].
5
+
6
+ 2023-10-05 19:32:58,673 DEBUG root Chosen display Format:default
7
+ 2023-10-05 19:32:58,674 INFO root Display format: "default"
8
+ 2023-10-05 19:32:58,675 DEBUG root SDK update checks are disabled.
content/.config/logs/2023.10.05/19.32.59.577697.log ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ 2023-10-05 19:32:59,580 DEBUG root Loaded Command Group: ['gcloud', 'config']
2
+ 2023-10-05 19:32:59,615 DEBUG root Loaded Command Group: ['gcloud', 'config', 'set']
3
+ 2023-10-05 19:32:59,618 DEBUG root Running [gcloud.config.set] with arguments: [SECTION/PROPERTY: "compute/gce_metadata_read_timeout_sec", VALUE: "0"]
4
+ 2023-10-05 19:32:59,620 INFO ___FILE_ONLY___ Updated property [compute/gce_metadata_read_timeout_sec].
5
+
6
+ 2023-10-05 19:32:59,621 DEBUG root Chosen display Format:default
7
+ 2023-10-05 19:32:59,621 INFO root Display format: "default"
8
+ 2023-10-05 19:32:59,622 DEBUG root SDK update checks are disabled.
content/Twewy_08.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:08328661afb12c7eb45acd1483843db2d78a269767ad87ee32f8b6a3b2fb4285
3
+ size 151114244
content/sample_data/README.md ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ This directory includes a few sample datasets to get you started.
2
+
3
+ * `california_housing_data*.csv` is California housing data from the 1990 US
4
+ Census; more information is available at:
5
+ https://developers.google.com/machine-learning/crash-course/california-housing-data-description
6
+
7
+ * `mnist_*.csv` is a small sample of the
8
+ [MNIST database](https://en.wikipedia.org/wiki/MNIST_database), which is
9
+ described at: http://yann.lecun.com/exdb/mnist/
10
+
11
+ * `anscombe.json` contains a copy of
12
+ [Anscombe's quartet](https://en.wikipedia.org/wiki/Anscombe%27s_quartet); it
13
+ was originally described in
14
+
15
+ Anscombe, F. J. (1973). 'Graphs in Statistical Analysis'. American
16
+ Statistician. 27 (1): 17-21. JSTOR 2682899.
17
+
18
+ and our copy was prepared by the
19
+ [vega_datasets library](https://github.com/altair-viz/vega_datasets/blob/4f67bdaad10f45e3549984e17e1b3088c731503d/vega_datasets/_data/anscombe.json).
content/sample_data/anscombe.json ADDED
@@ -0,0 +1,49 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {"Series":"I", "X":10.0, "Y":8.04},
3
+ {"Series":"I", "X":8.0, "Y":6.95},
4
+ {"Series":"I", "X":13.0, "Y":7.58},
5
+ {"Series":"I", "X":9.0, "Y":8.81},
6
+ {"Series":"I", "X":11.0, "Y":8.33},
7
+ {"Series":"I", "X":14.0, "Y":9.96},
8
+ {"Series":"I", "X":6.0, "Y":7.24},
9
+ {"Series":"I", "X":4.0, "Y":4.26},
10
+ {"Series":"I", "X":12.0, "Y":10.84},
11
+ {"Series":"I", "X":7.0, "Y":4.81},
12
+ {"Series":"I", "X":5.0, "Y":5.68},
13
+
14
+ {"Series":"II", "X":10.0, "Y":9.14},
15
+ {"Series":"II", "X":8.0, "Y":8.14},
16
+ {"Series":"II", "X":13.0, "Y":8.74},
17
+ {"Series":"II", "X":9.0, "Y":8.77},
18
+ {"Series":"II", "X":11.0, "Y":9.26},
19
+ {"Series":"II", "X":14.0, "Y":8.10},
20
+ {"Series":"II", "X":6.0, "Y":6.13},
21
+ {"Series":"II", "X":4.0, "Y":3.10},
22
+ {"Series":"II", "X":12.0, "Y":9.13},
23
+ {"Series":"II", "X":7.0, "Y":7.26},
24
+ {"Series":"II", "X":5.0, "Y":4.74},
25
+
26
+ {"Series":"III", "X":10.0, "Y":7.46},
27
+ {"Series":"III", "X":8.0, "Y":6.77},
28
+ {"Series":"III", "X":13.0, "Y":12.74},
29
+ {"Series":"III", "X":9.0, "Y":7.11},
30
+ {"Series":"III", "X":11.0, "Y":7.81},
31
+ {"Series":"III", "X":14.0, "Y":8.84},
32
+ {"Series":"III", "X":6.0, "Y":6.08},
33
+ {"Series":"III", "X":4.0, "Y":5.39},
34
+ {"Series":"III", "X":12.0, "Y":8.15},
35
+ {"Series":"III", "X":7.0, "Y":6.42},
36
+ {"Series":"III", "X":5.0, "Y":5.73},
37
+
38
+ {"Series":"IV", "X":8.0, "Y":6.58},
39
+ {"Series":"IV", "X":8.0, "Y":5.76},
40
+ {"Series":"IV", "X":8.0, "Y":7.71},
41
+ {"Series":"IV", "X":8.0, "Y":8.84},
42
+ {"Series":"IV", "X":8.0, "Y":8.47},
43
+ {"Series":"IV", "X":8.0, "Y":7.04},
44
+ {"Series":"IV", "X":8.0, "Y":5.25},
45
+ {"Series":"IV", "X":19.0, "Y":12.50},
46
+ {"Series":"IV", "X":8.0, "Y":5.56},
47
+ {"Series":"IV", "X":8.0, "Y":7.91},
48
+ {"Series":"IV", "X":8.0, "Y":6.89}
49
+ ]
content/sample_data/california_housing_test.csv ADDED
The diff for this file is too large to render. See raw diff
 
content/sample_data/california_housing_train.csv ADDED
The diff for this file is too large to render. See raw diff
 
content/sample_data/mnist_test.csv ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:51c292478d94ec3a01461bdfa82eb0885d262eb09e615679b2d69dedb6ad09e7
3
+ size 18289443
content/sample_data/mnist_train_small.csv ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1ef64781aa03180f4f5ce504314f058f5d0227277df86060473d973cf43b033e
3
+ size 36523880
content/sd-notebook-collection/.github/FUNDING.yml ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # These are supported funding model platforms
2
+
3
+ github: # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]
4
+ patreon: # Replace with a single Patreon username
5
+ open_collective: # Replace with a single Open Collective username
6
+ ko_fi: linaqruf
7
+ tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
8
+ community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
9
+ liberapay: # Replace with a single Liberapay username
10
+ issuehunt: # Replace with a single IssueHunt username
11
+ otechie: # Replace with a single Otechie username
12
+ lfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry
13
+ custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']
content/sd-notebook-collection/AnimateDiff.ipynb ADDED
The diff for this file is too large to render. See raw diff
 
content/sd-notebook-collection/Anything-V3-Mirroring.ipynb ADDED
@@ -0,0 +1,296 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "nbformat": 4,
3
+ "nbformat_minor": 0,
4
+ "metadata": {
5
+ "colab": {
6
+ "provenance": [],
7
+ "include_colab_link": true
8
+ },
9
+ "kernelspec": {
10
+ "name": "python3",
11
+ "display_name": "Python 3"
12
+ },
13
+ "language_info": {
14
+ "name": "python"
15
+ }
16
+ },
17
+ "cells": [
18
+ {
19
+ "cell_type": "markdown",
20
+ "metadata": {
21
+ "id": "view-in-github",
22
+ "colab_type": "text"
23
+ },
24
+ "source": [
25
+ "<a href=\"https://colab.research.google.com/github/Linaqruf/sd-notebook-collection/blob/main/Anything-V3-Mirroring.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
26
+ ]
27
+ },
28
+ {
29
+ "cell_type": "markdown",
30
+ "source": [
31
+ "# A Notebook to Mirror Anything V3 to Your HuggingFace Repository\n",
32
+ "### In case I make the repo private, you can use this notebook to mirror the model `(Full, fp32, fp16)` and upload/backup it to your Huggingface Repository. As long as I'm not resetting my `download` token. VAE in all model already replaced with the better one."
33
+ ],
34
+ "metadata": {
35
+ "id": "yehBslknsPvU"
36
+ }
37
+ },
38
+ {
39
+ "cell_type": "code",
40
+ "execution_count": null,
41
+ "metadata": {
42
+ "cellView": "form",
43
+ "id": "hg27LwQuvvXK"
44
+ },
45
+ "outputs": [],
46
+ "source": [
47
+ "#@title ## Install Dependencies\n",
48
+ "import os\n",
49
+ "\n",
50
+ "%cd /content/\n",
51
+ "\n",
52
+ "# Check if the directory already exists\n",
53
+ "if os.path.isdir('/content/sd-notebook-collection'):\n",
54
+ " %cd /content/sd-notebook-collection\n",
55
+ " print(\"This folder already exists, will do a !git pull instead\\n\")\n",
56
+ " !git pull\n",
57
+ "else:\n",
58
+ " !git clone https://github.com/Linaqruf/sd-notebook-collection\n",
59
+ "\n",
60
+ "%cd /content/sd-notebook-collection\n",
61
+ "\n",
62
+ "#@markdown This will install required Python packages\n",
63
+ "!pip -qqqq install --upgrade -r requirements.txt\n",
64
+ "!apt -qqqq install liblz4-tool aria2\n",
65
+ "!pip -qqqq install huggingface"
66
+ ]
67
+ },
68
+ {
69
+ "cell_type": "code",
70
+ "source": [
71
+ "#@title Mirror Model\n",
72
+ "%cd /content/\n",
73
+ "from IPython.utils import capture\n",
74
+ "\n",
75
+ "models_path = \"/content/models\"\n",
76
+ "\n",
77
+ "if not os.path.exists(models_path):\n",
78
+ " print(f\"Creating directory at {models_path}\")\n",
79
+ " os.makedirs(models_path)\n",
80
+ "\n",
81
+ "#@markdown Which model to mirror?\n",
82
+ "Anything_V_3_0_Full = True #@param {type:\"boolean\"}\n",
83
+ "Anything_V_3_0_Pruned = True #@param {type:\"boolean\"}\n",
84
+ "Anything_V_3_0_Pruned_fp16 = True #@param {type:\"boolean\"}\n",
85
+ "\n",
86
+ "#@markdown Do you want to upload diffusers format?\n",
87
+ "diffusers_format = True #@param {type:\"boolean\"}\n",
88
+ "\n",
89
+ "full_model_url = \"https://huggingface.co/Linaqruf/personal-backup/resolve/main/vae-swapped-model/anything-v3-full.safetensors\"\n",
90
+ "pruned_model_url = \"https://huggingface.co/Linaqruf/personal-backup/resolve/main/vae-swapped-model/anything-v3-fp32-pruned.safetensors\"\n",
91
+ "fp16_model_url = \"https://huggingface.co/Linaqruf/personal-backup/resolve/main/vae-swapped-model/anything-v3-fp16-pruned.safetensors\"\n",
92
+ "\n",
93
+ "full_model_path = os.path.join(models_path, os.path.basename(full_model_url))\n",
94
+ "pruned_model_path = os.path.join(models_path, os.path.basename(pruned_model_url))\n",
95
+ "fp16_model_path = os.path.join(models_path, os.path.basename(fp16_model_url))\n",
96
+ "\n",
97
+ "base_name = os.path.splitext(os.path.basename(pruned_model_path))[0]\n",
98
+ "diffusers_format = os.path.join(models_path, base_name)\n",
99
+ "\n",
100
+ "def download(url):\n",
101
+ " base_name= os.path.basename(url)\n",
102
+ " hf_token = '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' \n",
103
+ " user_header = f\"\\\"Authorization: Bearer {hf_token}\\\"\"\n",
104
+ " !aria2c --console-log-level=error --summary-interval=10 --header={user_header} -c -x 16 -k 1M -s 16 -d models -o {base_name} \"{url}\"\n",
105
+ "\n",
106
+ "def download_checkpoint():\n",
107
+ " if Anything_V_3_0_Full:\n",
108
+ " print(f\"Downloading full model from {full_model_url}\")\n",
109
+ " with capture.capture_output() as cap:\n",
110
+ " download(full_model_url)\n",
111
+ " print(\"Download completed!\")\n",
112
+ " if Anything_V_3_0_Pruned:\n",
113
+ " print(f\"Downloading pruned model from {pruned_model_url}\")\n",
114
+ " with capture.capture_output() as cap:\n",
115
+ " download(pruned_model_url)\n",
116
+ " print(\"Download completed!\")\n",
117
+ " if Anything_V_3_0_Pruned_fp16:\n",
118
+ " print(f\"Downloading pruned fp16 model from {fp16_model_url}\")\n",
119
+ " with capture.capture_output() as cap:\n",
120
+ " download(fp16_model_url)\n",
121
+ " print(\"Download completed!\")\n",
122
+ "\n",
123
+ "download_checkpoint()\n",
124
+ "\n",
125
+ "if diffusers_format:\n",
126
+ " %cd /content/sd-notebook-collection/script/\n",
127
+ " print(f\"Converting diffusers format for {diffusers_format}\")\n",
128
+ "\n",
129
+ " reference_model = \"runwayml/stable-diffusion-v1-5\"\n",
130
+ "\n",
131
+ " #@markdown Additional file for diffusers\n",
132
+ " feature_extractor = True #@param {'type': 'boolean'}\n",
133
+ " safety_checker = True #@param {'type': 'boolean'}\n",
134
+ "\n",
135
+ " print(f'feature_extractor: {feature_extractor} safety_checker: {safety_checker}')\n",
136
+ " print(f'Please wait...')\n",
137
+ " with capture.capture_output() as cap: \n",
138
+ " !python convert_diffusers20_original_sd.py \\\n",
139
+ " \"{pruned_model_path}\" \\\n",
140
+ " \"{diffusers_format}\" \\\n",
141
+ " --v1 \\\n",
142
+ " --reference_model {reference_model} \n",
143
+ "\n",
144
+ " url1 = \"https://huggingface.co/CompVis/stable-diffusion-safety-checker/resolve/main/preprocessor_config.json\"\n",
145
+ " url2 = \"https://huggingface.co/CompVis/stable-diffusion-safety-checker/resolve/main/config.json\"\n",
146
+ " url3 = \"https://huggingface.co/CompVis/stable-diffusion-safety-checker/resolve/main/pytorch_model.bin\"\n",
147
+ "\n",
148
+ " if feature_extractor == True:\n",
149
+ " if not os.path.exists(str(diffusers_format)+'/feature_extractor'):\n",
150
+ " os.makedirs(str(diffusers_format)+'/feature_extractor')\n",
151
+ " \n",
152
+ " with capture.capture_output() as cap: \n",
153
+ " !aria2c --console-log-level=error --summary-interval=10 -c -x 16 -k 1M -s 16 -d '{diffusers_format}/feature_extractor' -Z {url1}\n",
154
+ "\n",
155
+ " if safety_checker == True:\n",
156
+ " if not os.path.exists(str(diffusers_format)+'/safety_checker'):\n",
157
+ " os.makedirs(str(diffusers_format)+'/safety_checker')\n",
158
+ " \n",
159
+ " with capture.capture_output() as cap: \n",
160
+ " !aria2c --console-log-level=error --summary-interval=10 -c -x 16 -k 1M -s 16 -d '{diffusers_format}/safety_checker' -Z {url2}\n",
161
+ " !aria2c --console-log-level=error --summary-interval=10 -c -x 16 -k 1M -s 16 -d '{diffusers_format}/safety_checker' -Z {url3}\n",
162
+ " \n",
163
+ " print(\"Conversion successful\")"
164
+ ],
165
+ "metadata": {
166
+ "cellView": "form",
167
+ "id": "n0R9Hb04DRud"
168
+ },
169
+ "execution_count": null,
170
+ "outputs": []
171
+ },
172
+ {
173
+ "cell_type": "code",
174
+ "source": [
175
+ "#@title Upload Model\n",
176
+ "#@markdown ### Login to Huggingface hub\n",
177
+ "from huggingface_hub import login\n",
178
+ "from huggingface_hub import HfApi\n",
179
+ "from huggingface_hub.utils import validate_repo_id, HfHubHTTPError\n",
180
+ "from pathlib import Path\n",
181
+ "import shutil\n",
182
+ "\n",
183
+ "%cd /content/\n",
184
+ "api = HfApi()\n",
185
+ "\n",
186
+ "#@markdown You can get your huggingface token from [this link](https://huggingface.co/settings/tokens), then `create new token` or copy available token with the `Write` role.\n",
187
+ "\n",
188
+ "write_token = \"your-write-token\" #@param {type:\"string\"}\n",
189
+ "login(write_token, add_to_git_credential=True)\n",
190
+ "\n",
191
+ "user = api.whoami(write_token)\n",
192
+ "\n",
193
+ "#@markdown ### Define your Huggingface Repo\n",
194
+ "#@markdown If your model repo didn't exist, it will automatically create your repo.\n",
195
+ "model_name = \"anything-v3.0-mirror\" #@param{type:\"string\"}\n",
196
+ "make_this_model_private = True #@param{type:\"boolean\"}\n",
197
+ "\n",
198
+ "model_repo = user['name']+\"/\"+model_name.strip()\n",
199
+ "validate_repo_id(model_repo)\n",
200
+ "\n",
201
+ "if make_this_model_private:\n",
202
+ " private_repo = True\n",
203
+ "else:\n",
204
+ " private_repo = False\n",
205
+ "\n",
206
+ "try:\n",
207
+ " api.create_repo(repo_id=model_repo, \n",
208
+ " private=private_repo)\n",
209
+ " print(\"Model Repo didn't exists, creating repo\")\n",
210
+ " print(\"Model Repo: \",model_repo,\"created!\\n\")\n",
211
+ "\n",
212
+ "except HfHubHTTPError as e:\n",
213
+ " print(f\"Model Repo: {model_repo} exists, skipping create repo\\n\")\n",
214
+ "\n",
215
+ "def upload_model(model_path, is_diffusers: bool):\n",
216
+ " path_obj = Path(model_path)\n",
217
+ " trained_model = path_obj.parts[-1]\n",
218
+ " commit_message = f\"feat: upload {trained_model}\" \n",
219
+ " \n",
220
+ " if is_diffusers == True:\n",
221
+ " print(f\"Uploading diffusers model to https://huggingface.co/\"+model_repo)\n",
222
+ " print(f\"Please wait...\")\n",
223
+ "\n",
224
+ " api.upload_folder(\n",
225
+ " folder_path=model_path,\n",
226
+ " repo_id=model_repo,\n",
227
+ " commit_message=commit_message,\n",
228
+ " ignore_patterns=\".ipynb_checkpoints\"\n",
229
+ " )\n",
230
+ " print(f\"Upload success, located at https://huggingface.co/\"+model_repo+\"/tree/main\\n\")\n",
231
+ " else:\n",
232
+ " print(f\"Uploading {trained_model} to https://huggingface.co/\"+model_repo)\n",
233
+ " print(f\"Please wait...\")\n",
234
+ " api.upload_file(\n",
235
+ " path_or_fileobj=model_path,\n",
236
+ " path_in_repo=trained_model,\n",
237
+ " repo_id=model_repo,\n",
238
+ " commit_message=commit_message,\n",
239
+ " ) \n",
240
+ " print(f\"Upload success, located at https://huggingface.co/\"+model_repo+\"/blob/main/\"+trained_model+\"\\n\")\n",
241
+ "\n",
242
+ "def upload():\n",
243
+ " if Anything_V_3_0_Pruned:\n",
244
+ " upload_model(pruned_model_path, False)\n",
245
+ " if Anything_V_3_0_Pruned_fp16:\n",
246
+ " upload_model(fp16_model_path, False)\n",
247
+ " if diffusers_format:\n",
248
+ " upload_model(diffusers_format, True)\n",
249
+ " if Anything_V_3_0_Full:\n",
250
+ " upload_model(full_model_path, False)\n",
251
+ "\n",
252
+ "upload()\n"
253
+ ],
254
+ "metadata": {
255
+ "cellView": "form",
256
+ "id": "jzOIuaFHF3V9"
257
+ },
258
+ "execution_count": null,
259
+ "outputs": []
260
+ },
261
+ {
262
+ "cell_type": "code",
263
+ "source": [
264
+ "#@title Commit Model\n",
265
+ "\n",
266
+ "import shutil\n",
267
+ "from pathlib import Path\n",
268
+ "%cd /content/\n",
269
+ "\n",
270
+ "!git lfs install --skip-smudge\n",
271
+ "!export GIT_LFS_SKIP_SMUDGE=1\n",
272
+ "!git clone https://huggingface.co/{model_repo} /content/{model_name}\n",
273
+ "\n",
274
+ "path_obj = Path(full_model_path)\n",
275
+ "full_basename = path_obj.parts[-1]\n",
276
+ "\n",
277
+ "#@markdown Unfortunately you can't upload 7G model to huggingface with `hf_hub` using Colab Free, because it will crash the colab and automatically restart the runtime. So we will use git commit as alternative.\n",
278
+ "repo = f\"/content/{model_name}\"\n",
279
+ "# shutil.move(full_model_path, repo)\n",
280
+ "#@markdown Set **git commit identity**\n",
281
+ "email = \"your-email-here\" #@param {'type': 'string'}\n",
282
+ "name = \"your-username-here\" #@param {'type': 'string'}\n",
283
+ "#@markdown Set **commit message**\n",
284
+ "commit_m = f\"feat: upload {full_basename}\"\n",
285
+ "\n",
286
+ "\n"
287
+ ],
288
+ "metadata": {
289
+ "cellView": "form",
290
+ "id": "1fxyjIkGULyU"
291
+ },
292
+ "execution_count": null,
293
+ "outputs": []
294
+ }
295
+ ]
296
+ }
content/sd-notebook-collection/Infinite_Image_Browsing_Demo.ipynb ADDED
@@ -0,0 +1,220 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "nbformat": 4,
3
+ "nbformat_minor": 0,
4
+ "metadata": {
5
+ "colab": {
6
+ "provenance": [],
7
+ "include_colab_link": true
8
+ },
9
+ "kernelspec": {
10
+ "name": "python3",
11
+ "display_name": "Python 3"
12
+ },
13
+ "language_info": {
14
+ "name": "python"
15
+ }
16
+ },
17
+ "cells": [
18
+ {
19
+ "cell_type": "markdown",
20
+ "metadata": {
21
+ "id": "view-in-github",
22
+ "colab_type": "text"
23
+ },
24
+ "source": [
25
+ "<a href=\"https://colab.research.google.com/github/Linaqruf/sd-notebook-collection/blob/main/Infinite_Image_Browsing_Demo.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
26
+ ]
27
+ },
28
+ {
29
+ "cell_type": "markdown",
30
+ "source": [
31
+ "# **Infinite Image Browsing Demo**\n",
32
+ "\n",
33
+ "Project Page : [Here!](https://github.com/zanllp/sd-webui-infinite-image-browsing)"
34
+ ],
35
+ "metadata": {
36
+ "id": "x9xvjuwV9eCz"
37
+ }
38
+ },
39
+ {
40
+ "cell_type": "code",
41
+ "source": [
42
+ "#@title ##**Install**\n",
43
+ "import os\n",
44
+ "from IPython.display import clear_output\n",
45
+ "from google.colab import drive\n",
46
+ "\n",
47
+ "root_dir = \"/content\"\n",
48
+ "repo_url = \"https://github.com/zanllp/sd-webui-infinite-image-browsing.git\"\n",
49
+ "repo_dir = os.path.join(root_dir, \"sd-webui-infinite-image-browsing\")\n",
50
+ "\n",
51
+ "mount_drive = False #@param {type:\"boolean\"}\n",
52
+ "# @markdown Specify where your `images_dir` located, it also automatically create new directory if couldn't find it.\n",
53
+ "images_dir = \"/content/images\" #@param {type:\"string\"}\n",
54
+ "\n",
55
+ "if not images_dir:\n",
56
+ " images_dir = os.path.join(root_dir, \"images\")\n",
57
+ "\n",
58
+ "def clone(url, dir):\n",
59
+ " !git clone {url} {dir}\n",
60
+ "\n",
61
+ "def install_deps():\n",
62
+ " os.chdir(repo_dir)\n",
63
+ " !pip install -r requirements.txt\n",
64
+ " !pip install -U gallery-dl\n",
65
+ " clear_output(wait=True)\n",
66
+ "\n",
67
+ "def main():\n",
68
+ " os.chdir(root_dir)\n",
69
+ "\n",
70
+ " if mount_drive:\n",
71
+ " if not os.path.exists(\"/content/drive\"):\n",
72
+ " drive.mount(\"/content/drive\")\n",
73
+ "\n",
74
+ " !apt update -yqq\n",
75
+ " !apt install aria2 -yqq\n",
76
+ " \n",
77
+ " clone(repo_url, repo_dir)\n",
78
+ " \n",
79
+ " os.chdir(repo_dir)\n",
80
+ " install_deps()\n",
81
+ "\n",
82
+ " os.makedirs(images_dir, exist_ok=True)\n",
83
+ "\n",
84
+ " clear_output(wait=True)\n",
85
+ " print(\"All is done! go to the next step.\")\n",
86
+ "\n",
87
+ "main()\n"
88
+ ],
89
+ "metadata": {
90
+ "cellView": "form",
91
+ "id": "ZO5QI5BU1Po2"
92
+ },
93
+ "execution_count": null,
94
+ "outputs": []
95
+ },
96
+ {
97
+ "cell_type": "code",
98
+ "source": [
99
+ "#@title ## **Image Scraper**\n",
100
+ "import os\n",
101
+ "import html\n",
102
+ "from IPython.utils import capture\n",
103
+ "%store -r\n",
104
+ "\n",
105
+ "os.chdir(root_dir)\n",
106
+ "#@markdown Use `gallery-dl` to scrape images from an imageboard site. Specify the `prompt(s)` by separating them with commas, e.g., `hito_komoru, touhou`.\n",
107
+ "booru = \"Danbooru\" #@param [\"Danbooru\", \"Gelbooru\", \"Safebooru\"]\n",
108
+ "prompt = \"hito_komoru\" #@param {type: \"string\"}\n",
109
+ "#@markdown You can also specify a `custom_url` instead of using a predefined site.\n",
110
+ "custom_url = \"\" #@param {type: \"string\"}\n",
111
+ "user_agent = \"gdl/1.24.5\"\n",
112
+ "#@markdown You can limit the number of images to download by using the `--range` option followed by the desired range. For example `1-200`.\n",
113
+ "range = \"1-200\" #@param {type: \"string\"}\n",
114
+ "additional_arguments = \"--filename /O --no-part\"\n",
115
+ "\n",
116
+ "tags = prompt.split(',')\n",
117
+ "tags = '+'.join(tags)\n",
118
+ "\n",
119
+ "replacement_dict = {\" \": \"\", \"(\": \"%28\", \")\": \"%29\", \":\": \"%3a\"}\n",
120
+ "tags = ''.join(replacement_dict.get(c, c) for c in tags)\n",
121
+ "\n",
122
+ "if booru == \"Danbooru\":\n",
123
+ " url = \"https://danbooru.donmai.us/posts?tags={}\".format(tags)\n",
124
+ "elif booru == \"Gelbooru\":\n",
125
+ " url = \"https://gelbooru.com/index.php?page=post&s=list&tags={}\".format(tags)\n",
126
+ "else:\n",
127
+ " url = \"https://safebooru.org/index.php?page=post&s=list&tags={}\".format(tags)\n",
128
+ "\n",
129
+ "valid_url = custom_url if custom_url else url\n",
130
+ "\n",
131
+ "def scrape(config):\n",
132
+ " args = \"\"\n",
133
+ " for k, v in config.items():\n",
134
+ " if k.startswith(\"_\"):\n",
135
+ " args += f'\"{v}\" '\n",
136
+ " elif isinstance(v, str):\n",
137
+ " args += f'--{k}=\"{v}\" '\n",
138
+ " elif isinstance(v, bool) and v:\n",
139
+ " args += f\"--{k} \"\n",
140
+ " elif isinstance(v, float) and not isinstance(v, bool):\n",
141
+ " args += f\"--{k}={v} \"\n",
142
+ " elif isinstance(v, int) and not isinstance(v, bool):\n",
143
+ " args += f\"--{k}={v} \"\n",
144
+ "\n",
145
+ " return args\n",
146
+ "\n",
147
+ "get_url_config = {\n",
148
+ " \"_valid_url\" : valid_url,\n",
149
+ " \"get-urls\" : True,\n",
150
+ " \"range\" : range if range else None,\n",
151
+ " \"user-agent\" : user_agent\n",
152
+ "}\n",
153
+ "\n",
154
+ "get_url_args = scrape(get_url_config)\n",
155
+ "scraper_text = os.path.join(root_dir, \"scrape_this.txt\")\n",
156
+ "\n",
157
+ "with capture.capture_output() as cap:\n",
158
+ " !gallery-dl {get_url_args} {additional_arguments}\n",
159
+ "with open(scraper_text, \"w\") as f:\n",
160
+ " f.write(cap.stdout)\n",
161
+ "\n",
162
+ "os.chdir(images_dir)\n",
163
+ "!aria2c --console-log-level=error --summary-interval=10 -c -x 16 -k 1M -s 16 -i {scraper_text}\n",
164
+ "\n",
165
+ "clear_output(wait=True)\n",
166
+ "print(\"All is done! go to the next step.\")"
167
+ ],
168
+ "metadata": {
169
+ "cellView": "form",
170
+ "id": "7ZQy_RW52s2R"
171
+ },
172
+ "execution_count": null,
173
+ "outputs": []
174
+ },
175
+ {
176
+ "cell_type": "code",
177
+ "source": [
178
+ "#@title **Launch!**\n",
179
+ "import json\n",
180
+ "import portpicker\n",
181
+ "from threading import Thread\n",
182
+ "from google.colab.output import serve_kernel_port_as_iframe\n",
183
+ "\n",
184
+ "config = {\n",
185
+ " \"outdir_txt2img_samples\": images_dir,\n",
186
+ "}\n",
187
+ "\n",
188
+ "main_app = os.path.join(repo_dir, \"app.py\")\n",
189
+ "config_file = os.path.join(root_dir, \"config.json\")\n",
190
+ "port = portpicker.pick_unused_port()\n",
191
+ "window_height = 550 #@param {type:\"slider\", min:0, max:1000, step:1}\n",
192
+ "\n",
193
+ "def write_file(filename, config):\n",
194
+ " with open(filename, 'w',) as f:\n",
195
+ " json.dump(config, f, indent=4)\n",
196
+ "\n",
197
+ "def run_app():\n",
198
+ " !python {main_app} --port={port} --sd_webui_config={config_file}\n",
199
+ "\n",
200
+ "def main():\n",
201
+ " os.chdir(images_dir)\n",
202
+ "\n",
203
+ " write_file(config_file, config)\n",
204
+ "\n",
205
+ " thread = Thread(target=run_app)\n",
206
+ " thread.start()\n",
207
+ "\n",
208
+ " serve_kernel_port_as_iframe(port, width='100%', height=window_height, cache_in_notebook=False)\n",
209
+ "\n",
210
+ "main()"
211
+ ],
212
+ "metadata": {
213
+ "cellView": "form",
214
+ "id": "3KgFHcoRsyvU"
215
+ },
216
+ "execution_count": null,
217
+ "outputs": []
218
+ }
219
+ ]
220
+ }
content/sd-notebook-collection/MANUAL.md ADDED
@@ -0,0 +1,202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Cagliostro Colab UI User Manual
2
+
3
+ Welcome to the [Cagliostro Colab UI](https://colab.research.google.com/github/Linaqruf/sd-notebook-collection/blob/main/cagliostro-colab-ui.ipynb) User Manual! This guide will walk you through the basics of using Cagliostro Colab UI, an innovative and powerful notebook designed to launch [Automatic1111's Stable Diffusion Web UI](https://github.com/AUTOMATIC1111/stable-diffusion-webui/) in Google Colab. With its advanced features, customizability, and flexibility, Cagliostro Colab offers a seamless and efficient way to utilize Stable Diffusion Web UI for your projects.
4
+
5
+ To get started with Cagliostro Colab UI, you'll need to have a Google account. Once you're logged in, you can open up Google Colab by visiting the website and signing in with your Google credentials.
6
+
7
+ ## Table of Contents
8
+ - [Cagliostro Colab UI User Manual](#cagliostro-colab-ui-user-manual)
9
+ - [Table of Contents](#table-of-contents)
10
+ - [Features](#features)
11
+ - [Main Cell Explained](#main-cell-explained)
12
+ - [Install Stable Diffusion Web UI](#install-stable-diffusion-web-ui)
13
+ - [Download Model and VAE](#download-model-and-vae)
14
+ - [ControlNet V1.1](#controlnet-v11)
15
+ - [Custom Download Corner](#custom-download-corner)
16
+ - [Start Stable Diffusion Web UI](#start-stable-diffusion-web-ui)
17
+ - [Download Generated Images](#download-generated-images)
18
+ - [Extra Cell Explained](#extra-cell-explained)
19
+ - [Download Generated Images V2](#download-generated-images-v2)
20
+ - [Resources](#resources)
21
+ - [Default Extensions](#default-extensions)
22
+ - [Default Negative Embeddings](#default-negative-embeddings)
23
+ - [Default Custom Upscalers](#default-custom-upscalers)
24
+ - [Theme Selector](#theme-selector)
25
+
26
+ ## Features:
27
+ | Feature | Description |
28
+ |-----------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
29
+ | **Faster installation** | Unpacks a pre-installed repository instead of installing from scratch, saving time and reducing the risk of installation errors.Uses pre-installed Python dependencies and unpacks them to `/usr/local/lib/python3.9/dist-packages/` instead of downloading them during installation, further reducing installation time. |
30
+ | **Faster downloader** | Uses the `aria2c` downloader instead of `wget`.Downloads packages with up to 16 parallel connections, 16 threads per connection, and a 1MB chunk size for faster downloads.Used to download pre-installed repo, dependencies, and models. |
31
+ | **Better UI/UX** | Uses [Anapnoe's](https://github.com/anapnoe/stable-diffusion-webui-ux) forked version of [Automatic1111's Stable Diffusion Web UI](https://github.com/AUTOMATIC1111/stable-diffusion-webui/) that focuses on developing better UI/UX for Stable Diffusion Web UI. But users can still use [Automatic1111](https://github.com/AUTOMATIC1111/stable-diffusion-webui/) by disabling the `use_anapnoe_ui` option.Note that Anapnoe's still uses an old commit, so if users want to experience Gradio 3.23.0, it's better to disable the `use_anapnoe_ui` option. |
32
+ | **Up-to-date** | Enabled with `git pull` for both the repository and all extensions by default to increase user experience.Some extensions might be skipped from updating due to upgrading its Gradio to 3.23. |
33
+ | **Integrated to Google Drive** | Provides a way to enhance the user experience for `Google Drive` users.Includes options to mount Google Drive, save output to Google Drive, load the model directory from Google Drive by using `unionfs-fuse` to merge folder A and folder B to folder C, and load the model file from Google Drive by copying the model to [Automatic1111's Stable Diffusion Web UI](https://github.com/AUTOMATIC1111/stable-diffusion-webui/). |
34
+ | **Colab Optimization** | Loads the model in VRAM, merges in VRAM, and uses VRAM instead of CPU for most things to create a better environment for Colab free tier users. |
35
+ | **Better Default Model** | Provides the top trending default model and the best choice in quality.Chooses Huggingface server instead of CivitAI because it's more stable and faster. |
36
+ | **All-in-One ControlNet model** | Provides not only the ControlNet model from SD1.5 but also SD2.1.Users can set the max model, config, and adapter config before starting the Web UI. |
37
+ | **Advanced Custom Download** | Provides seven fields to download models, including custom models, VAE, embedding, LoRA, hypernetwork, control, and extension.Users can prune their model by typing `fp16:url` to prune with FP16 precision and `fp32:url` for FP32 precision.Users can also easily load models from Google Drive by merging folders with the `fuse:path` command. |
38
+ | **More Tunnels!** | Uses `sd-webui-tunnels` extension (forked by [Camenduru](https://github.com/camenduru/sd-webui-tunnels)) and set `--multiple` by default. No more boring `gradio.live`!We're also support `ngrok`.
39
+
40
+ ## Main Cell Explained
41
+ ### Install Stable Diffusion Web UI
42
+ This cell installs stable-diffusion-webui repository in Colab. It also includes several configuration options, such as mounting Google Drive, updating extensions, and choosing between different versions of the web UI.
43
+
44
+ Options | Default Value | Description
45
+ ---------------------------------------|-----------------------|---------------------------------------------------------------
46
+ Drive Config | |
47
+ `mount_drive` | `False` | Mount your Google Drive to `/content/drive/MyDrive/` to load models.
48
+ `output_to_drive` | `False` | Save generation outputs to your Google Drive instead of `/content/stable-diffusion-webui/outputs`.
49
+ `output_drive_folder` | `False` | Set your default drive folder name to save your generation outputs, default: `cagliostro-colab-ui/outputs`
50
+ Web UI Config | |
51
+ `use_anapnoe_ui` | `True` | Use Anapnoe's forked repository instead of Automatic1111's.
52
+ `update_webui` | `True` | Update the web UI to the latest version. Because we're using pre-installed repo, this option is crucial if you want to try latest update.
53
+ `update_extensions` | `True` | Update all extensions to the latest version.
54
+ `commit_hash` | `''` | Go back to a specific commit hash to prevent errors when the web UI is updated.
55
+ `colab_optimizations` | `False` | Load SDv2 the model into VRAM, optimizes demo.queue() for Colab, and loads the model onto GPU memory if available.
56
+
57
+
58
+ Note:
59
+ - The options `colab_optimizations` are not relevant anymore as the new argument `--lowram` serves the same purpose, allowing the model to be loaded onto the GPU memory instead of VRAM if available. It also not recommended to set this `True` if you have Colab Pro subscription.
60
+
61
+ ### Download Model and VAE
62
+ This code block is responsible for downloading pre-trained models and VAEs (Variational Autoencoders) from Hugging Face's model hub. The available models and VAEs are listed as boolean checkboxes, and the user can select the ones they want to download.
63
+
64
+ | Model Name | Description | Model Page |
65
+ | --- | --- | --- |
66
+ | Stable Diffusion V1.x Model | | |
67
+ | `anything_v3_0` | State-of-the-art ~~merged/overfitted/vae broken~~ model when it was first launched in November, it has influenced most anime models since then. | [Link](https://huggingface.co/AdamOswald1/Anything-Preservation/blob/4121e81acc47bb87e46480ba1344b5ab57134b88/Anything-V3.0-pruned.safetensors) |
68
+ | `anime_pastel_dream` | The best alternative model to `pastelmix`, `anime_pastel_dream` was developed to provide a better user experience. | [Link](https://huggingface.co/Lykon/AnimePastelDream/blob/main/AnimePastelDream_Soft_noVae_fp16.safetensors) |
69
+ | `anylora` (default) | The best alternative model to `anything_v4_5`, `anylora` was developed to provide a better user experience. | [Link](https://huggingface.co/Lykon/AnyLoRA/blob/main/AnyLoRA_noVae_fp16.safetensors) |
70
+ | `chilloutmix_ni` | The most used non-anime model for now, it generates real person pictures but is still influenced by anime models. | [Link](https://huggingface.co/naonovn/chilloutmix_NiPrunedFp32Fix/blob/main/chilloutmix_NiPrunedFp32Fix.safetensors) |
71
+ | Stable Diffusion V2.x Model | | |
72
+ | `replicant_v2` | An anime model based off Waifu Diffusion V1.5 Beta 2, which performs better than `waifu_diffusion_v1_5_e2_aesthetic`. | [Link](https://huggingface.co/gsdf/Replicant-V2.0/blob/main/Replicant-V2.0_fp16.safetensors) |
73
+ | `waifu_diffusion_v1_5_e2_aesthetic` | An overtrained version of Waifu Diffusion V1.5 Beta 2 trained off stable diffusion v2.1 768. It works great to generate anime art. | [Link](https://huggingface.co/waifu-diffusion/wd-1-5-beta2/blob/main/checkpoints/wd-1-5-beta2-aesthetic-fp16.safetensors) |
74
+ | VAE models | | |
75
+ | `anime` (default) | A popular anime VAE used by many model, such as AbyssOrangeMixs and Anything V3. | 🔐 |
76
+ | `waifu_diffusion` | Anime VAE developed by the Waifu Diffusion developer based on the stable diffusion VAE. It provides contrast color compared to the `anime` VAE. | [Link](https://huggingface.co/hakurei/waifu-diffusion-v1-4/blob/main/vae/kl-f8-anime.ckpt) |
77
+ | `stable_diffusion` | The original Stable Diffusion VAE trained by Stability AI. It is intended to be used with the original CompVis Stable Diffusion. | [Link](https://huggingface.co/stabilityai/sd-vae-ft-mse-original/blob/main/vae-ft-mse-840000-ema-pruned.ckpt) |
78
+
79
+ ### ControlNet V1.1
80
+ [ControlNet](https://github.com/lllyasviel/ControlNet) is a neural network structure to control diffusion models by adding extra conditions. ControlNet was introduced in [Adding Conditional Control to Text-to-Image Diffusion Models](https://arxiv.org/abs/2302.05543) by Lvmin Zhang and Maneesh Agrawala. It introduces a framework that allows for supporting various spatial contexts that can serve as additional conditionings to Diffusion models such as Stable Diffusion. It basically allow user to control Stable Diffusion generation. ControlNet 1.1 has been released nightly at this [github repository](https://github.com/lllyasviel/ControlNet-v1-1-nightly) and Cagliostro Colab UI is now support it!
81
+
82
+ | Option | Default | Description |
83
+ |--------|---------|-------------|
84
+ | `pre_download_annotator` | `True` | Specifies whether to download ControlNet pre-processor/annotator before starting the Web UI. The pre-processor/annotator includes 13 items in total, and will be downloaded to the specified path. |
85
+ | `control_v11_sd15_model` | `True` | Specifies whether to download all 14 extracted [ControlNet v1.1](https://github.com/lllyasviel/ControlNet) model developed by [Lvmin Zhang](https://twitter.com/lvminzhang) |
86
+ | `t2i_adapter_sd15_model` | `False` | Specifies whether to download all 12 [Text to Image Adapter](https://github.com/TencentARC/T2I-Adapter) model developed by [TencentArcLab](https://github.com/TencentARC). |
87
+ | `control_v10_sd21_model` | `False` | Specifies whether to download all 5 extracted [SDv21 ControlNet v1.0](https://huggingface.co/thibaud/controlnet-sd21) model developed by [thibaud](https://twitter.com/thibaudz). |
88
+ | `control_v10_wd15_model` | `False` | Specifies whether to download all 3 extracted [Waifu Diffusion 1.5 Controlnet v1.0](https://huggingface.co/furusu/ControlNet) model developed by [furusu](https://twitter.com/gcem156). |
89
+ | `control_net_max_models_num` | `2` | Specifies the maximum number of ControlNet tabs for multi-controlnet generation. |
90
+ | `control_net_model_adapter_config` | `sketch_adapter_v14.yaml` | Specifies the configuration file to use for the ControlNet model adapter. Different models require different configurations. The default configuration is `sketch_adapter_v14.yaml`. |
91
+
92
+ [T2I Adapter](https://github.com/TencentARC/T2I-Adapter) is a simple and small **(~70M parameters, ~300M storage space)** network that can provide extra guidance to pre-trained text-to-image models while freezing the original large text-to-image models. Controlnet was introduced in [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://huggingface.co/TencentARC/T2I-Adapter). It basically a cheapest ControlNet that allow user to control Text to Image generation.
93
+
94
+ | `t2i_adapter_model` | `control_net_model_adapter_config` |
95
+ |:-------------------------:|:-------------------------:|
96
+ | t2iadapter_canny_sd14v1.pth | sketch_adapter_v14.yaml |
97
+ | t2iadapter_sketch_sd14v1.pth | sketch_adapter_v14.yaml |
98
+ | t2iadapter_seg_sd14v1.pth | image_adapter_v14.yaml |
99
+ | t2iadapter_keypose_sd14v1.pth | image_adapter_v14.yaml |
100
+ | t2iadapter_openpose_sd14v1.pth | image_adapter_v14.yaml |
101
+ | t2iadapter_color_sd14v1.pth | t2iadapter_color_sd14v1.yaml |
102
+ | t2iadapter_style_sd14v1.pth | t2iadapter_style_sd14v1.yaml |
103
+
104
+ ### Custom Download Corner
105
+ This is a cell that allows you to download custom `models`, `VAEs`, `embeddings`, `LoRA`, `hypernetworks`, `upscalee` and install `extensions` by providing URLs to the files you want to download. This cell downloads custom files from various sources, including Google Drive, Huggingface, CivitAI, and other direct download links.
106
+
107
+ | Feature | Description | How to Use | Example |
108
+ | --- | --- | --- | --- |
109
+ | **Multiple Downloads** | Download multiple files at once. | Fill in the URL fields with the links to the files you want to download. Separate multiple URLs with a comma. | `url1, url2, url3` |
110
+ | **Auto-prune** | Prune models after downloading | Add `fp16:` or `fp32:` before URLs. | `fp16:url1` |
111
+ | **Copy from Google Drive** | Copy models from Google Drive and load them in the session. | Make sure you have already mounted Google Drive. Type the path to your model/lora/embedding from Google Drive. | `/content/drive/MyDrive/path/to/folder` |
112
+ | **Fusing Folder** | Fuse models/embeddings/LoRA folder to `/content/fused/{category}`. | Make sure you have already mounted Google Drive. Add `fuse:` before the path to the folder. | `fuse:/path/to/gdrive/folder` |
113
+ | **Auto-extract** | Extract files after downloading. | Add `links/to/file` ending with `.zip` or `.tar.lz4`. Extract files to specified destination directory. | `https//link.com/to/file` |
114
+ | **Install Extensions** | Install extensions for Stable Diffusion Web UI. | Add the link to the Github repository to `custom_extension_url`. | `https://github.com/user/repo` |
115
+
116
+ Once the download is complete, you can proceed to the next step.
117
+
118
+ ### Start Stable Diffusion Web UI
119
+ This is a cell for launching Stable Diffusion Web UI after a long configuration process above. However, it still has its own configuration requirements, such as defining arguments, using alternative tunnels, selecting a theme, and so on.
120
+
121
+ Option | Default | Description
122
+ ----------------|----------|-------------
123
+ | Alternative Tunnels | | Note: Recommended Tunnels: `ngrok` > `cloudflared` > `remotemoe` > `localhostrun` > `googleusercontent` > `gradio` |
124
+ `tunnel` | `multiple` | Allow users to use alternative tunnels for shared links such as `cloudflared`, `remotemoe`, `localhostrun`, and `googleusercontent`.
125
+ `ngrok_token` | `' '` | Enable ngrok tunnel for shared links. Users can get their ngrok token from [here](https://dashboard.ngrok.com/get-started/your-authtoken). If `ngrok_token` is enabled, it will automatically disable `alt_tunnels`.
126
+ `ngrok_region` | `ap` | Specify the desired region for ngrok tunnel. Users can choose between `["us", "eu", "au", "ap", "sa", "jp", "in"]`.
127
+ | Launch Arguments | | |
128
+ | `theme_selector` | `ogxBGreen` | This section used to change preferred theme for [Anapnoe's Stable Diffusion Web UI/UX](https://github.com/anapnoe/stable-diffusion-webui-ux) |
129
+ | `use_gradio_auth` | `False` | If enabled, every time a user opens shared links, it will ask for authentication. The username is set to `cagliostro` and the password will be randomly generated with 6 ASCII characters and numbers. Authentication information will be printed when the user launches the Web UI. |
130
+ | `accelerator` | `xformers` | To accelerate your generation and training, we offer 3 arguments. People commonly use `xformers`, but `opt-sdp-attention` would be a nice choice to have after Colab updates PyTorch to 2.0. It's claimed to be better than `xformers` but only for generating at lower resolution. So we choose to use `xformers` by default. |
131
+ | `quiet_mode` | `True` | This is a quality-of-life option. If enabled, it will use `--no-hashing` to skip model hashing, which is useful for not double-checking the model if it doesn't have the same hash and for merging the model. If enabled, it will also use `--disable-console-progressbars` to only have 1 line progress bars instead of updating new lines every generation. |
132
+ | `auto_select_model` | `False` | This argument allows searching for a model and randomly selecting the model in the model directory. |
133
+ | `auto_select_VAE` | `True` | This argument allows searching for a VAE and randomly selecting the VAE in the VAE directory. |
134
+ | `no_half_VAE` | `True` | This argument prevents black generation caused by certain VAEs. This usually happens because certain VAEs like `anime` have FP32. |
135
+ | `additional_args` | `--no-download-sd-model --gradio-queue` | This argument allows the user to add custom args from [here](https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Command-Line-Arguments-and-Settings). |
136
+
137
+ ### Download Generated Images
138
+ This cell useful to download outputs generated from Stable Diffusion Web UI as a compressed `.zip`
139
+
140
+ | Option | Default | Description |
141
+ | ----------- | ------- | -------------------------------------------------------------------------------------------------------- |
142
+ | `use_drive` | `True` | If enabled, this option will ask to mount your Google Drive account and save the compressed file to a preferred folder name specified by `folder_name`. It will also provide a download link after the process is completed. |
143
+ | `folder_name` | `"AI-generated Art"` | This option is used to specify the preferred folder name in Google Drive if `use_drive` is enabled. |
144
+ | `filename` | `"waifu.zip"` | This option allows the user to specify the filename of the zipfile. If `use_drive` is enabled and a duplicate filename is found in Google Drive, it will be renamed to `filename(n+1).zip`. |
145
+
146
+ ## Extra Cell Explained
147
+ ### Download Generated Images V2
148
+ Same as [Download Generated Images](#download-generated-images), the difference is it send the zipfile to Huggingface instead of Google Drive. Personally, it's better than save it to Google Drive.
149
+
150
+ | Option | Default Value | Description |
151
+ |-----------------|---------------|-------------------------------------------------------------------------------------------------------------------|
152
+ | `write_token` | `"" ` | User needs to specify a `WRITE` token from [Hugging Face settings](https://huggingface.co/settings/tokens). |
153
+ | `repo_name` | `"ai-art-dump"` | Specifies where the repository is located. If no repository is available, one will be created automatically. |
154
+ | `private_repo` | `False` | Specifies whether the repository is private or public to everyone. |
155
+ | `project_name` | `"waifu"` | Same as [Download Generated Images](#download-generated-images), this option allows the user to specify the name of the zipfile. |
156
+
157
+ ## Resources
158
+ ### Default Extensions
159
+ | Extensions | Github Repository |
160
+ | ----------------------------------------------- | --------------------------------------------------- |
161
+ | `ashen-sensored/sd_webui_stealth_pnginfo` | [Link](https://github.com/ashen-sensored/sd_webui_stealth_pnginfo) |
162
+ | `hnmr293/sd-webui-cutoff` | [Link](https://github.com/hnmr293/sd-webui-cutoff) |
163
+ | `KohakuBlueleaf/a1111-sd-webui-locon` | [Link](https://github.com/KohakuBlueleaf/a1111-sd-webui-locon) |
164
+ | `DominikDoom/a1111-sd-webui-tagcomplete` | [Link](https://github.com/DominikDoom/a1111-sd-webui-tagcomplete.git) |
165
+ | `etherealxx/batchlinks-webui` | [Link](https://github.com/etherealxx/batchlinks-webui) |
166
+ | `mcmonkeyprojects/sd-dynamic-thresholding` | [Link](https://github.com/mcmonkeyprojects/sd-dynamic-thresholding) |
167
+ | `kohya-ss/sd-webui-additional-networks` | [Link](https://github.com/kohya-ss/sd-webui-additional-networks.git) |
168
+ | `thomasasfk/sd-webui-aspect-ratio-helper` | [Link](https://github.com/thomasasfk/sd-webui-aspect-ratio-helper.git) |
169
+ | `Mikubill/sd-webui-controlnet` | [Link](https://github.com/Mikubill/sd-webui-controlnet) |
170
+ | `camenduru/sd-webui-tunnels` | [Link](https://github.com/camenduru/sd-webui-tunnels) |
171
+ | `bbc-mc/sdweb-merge-block-weighted-gui` | [Link](https://github.com/bbc-mc/sdweb-merge-block-weighted-gui.git) |
172
+ | `bbc-mc/sdweb-xyplus` | [Link](https://github.com/bbc-mc/sdweb-xyplus) |
173
+ | `opparco/stable-diffusion-webui-composable-lora` | [Link](https://github.com/opparco/stable-diffusion-webui-composable-lora.git) |
174
+ | `AlUlkesh/stable-diffusion-webui-images-browser` | [Link](https://github.com/AlUlkesh/stable-diffusion-webui-images-browser.git) |
175
+ | `arenatemp/stable-diffusion-webui-model-toolkit` | [Link](https://github.com/arenatemp/stable-diffusion-webui-model-toolkit) |
176
+ | `opparco/stable-diffusion-webui-two-shot` | [Link](https://github.com/opparco/stable-diffusion-webui-two-shot) |
177
+ | `Coyote-A/ultimate-upscale-for-automatic1111` | [Link](https://github.com/Coyote-A/ultimate-upscale-for-automatic1111.git) |
178
+
179
+ ### Default Custom Upscalers
180
+ | ESRGAN Upscaler | Mirror Link |
181
+ |------|------|
182
+ | `lollypop.pth` | [Link](https://huggingface.co/Linaqruf/stolen/blob/main/upscaler/lollypop.pth) |
183
+ | `4x-AnimeSharp.pth` | [Link](https://huggingface.co/Linaqruf/stolen/blob/main/upscaler/4x-AnimeSharp.pth) |
184
+ | `4x_foolhardy_Remacri.pth` | [Link](https://huggingface.co/Linaqruf/stolen/blob/main/upscaler/4x_foolhardy_Remacri.pth) |
185
+ | `4x-UltraSharp.pth` | [Link](https://huggingface.co/Linaqruf/stolen/blob/main/upscaler/4x-UltraSharp.pth) |
186
+
187
+ ### Default Negative Embeddings
188
+ | Negative Embeddings | Link |
189
+ |-------------------------------------------|-------------------------------------------------------------------------------------------|
190
+ | SDv1.x Negative Embeddings | |
191
+ | `EasyNegative.safetensors ` | [Link](https://huggingface.co/datasets/gsdf/EasyNegative/blob/main/EasyNegative.safetensors) |
192
+ | `bad-artist-anime.pt` | [Link](https://huggingface.co/nick-x-hacker/bad-artist/blob/main/bad-artist-anime.pt) |
193
+ | `bad-hands-5.pt` | [Link](https://huggingface.co/embed/negative/blob/main/bad-hands-5.pt) |
194
+ | `bad-artist.pt` | [Link](https://huggingface.co/nick-x-hacker/bad-artist/blob/main/bad-artist.pt) |
195
+ | `bad_prompt_version2.pt` | [Link](https://huggingface.co/datasets/Nerfgun3/bad_prompt/blob/main/bad_prompt_version2.pt) |
196
+ | `bad_prompt.pt` | [Link](https://huggingface.co/datasets/Nerfgun3/bad_prompt/blob/main/bad_prompt.pt) |
197
+ | `bad-image-v2-39000.pt` | [Link](https://huggingface.co/Xynon/models/blob/main/experimentals/TI/bad-image-v2-39000.pt) |
198
+ | `ng_deepnegative_v1_75t.pt` | [Link](https://huggingface.co/embed/negative/blob/main/ng_deepnegative_v1_75t.pt) |
199
+ | SDv2.x Negative Embeddings | |
200
+ | `rev2-badprompt.safetensors` | [Link](https://huggingface.co/gsdf/Replicant-V2.0/blob/main/rev2-badprompt.safetensors) |
201
+ | `re-badprompt.safetensors` | [Link](https://huggingface.co/gsdf/Replicant-V1.0/blob/main/re-badprompt.safetensors) |
202
+ | `wdbadprompt.pt` | [Link](https://huggingface.co/waifu-diffusion/wd-1-5-beta/blob/main/embeddings/wdbadprompt.pt) |
content/sd-notebook-collection/README.md ADDED
@@ -0,0 +1,271 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # **Cagliostro Colab UI**
2
+ All-in-One, Customizable and Flexible AUTOMATIC1111's Stable Diffusion Web UI for Google Colab. <br>
3
+
4
+ ## What's New?
5
+
6
+ ### v.3.0.0 (10/06/23)
7
+ - Rewrote `cagliostro-colab-ui` codebase from scratch.
8
+ - Used in-house module [Colablib](https://github.com/Linaqruf/colablib) as the primary library for go-to functions, such as colored print, git function, download syntax, etc.
9
+ - Added numerous trivial but important pieces of information, such as the Python version, torch version, current commit hash.
10
+ - Improved console logs, using `print_line()` and `cprint()` from `Colablib`.
11
+ - Merged `output_to_drive` with `mount_drive`.
12
+ - Now, the output path is automatically set to drive if Google Drive is mounted.
13
+ - Built-in wildcard support.
14
+ - Renamed `dpm_v2_patch` to `dpmpp_2m_v2_patch`.
15
+ - Added new section: `Optimization config`.
16
+ - Introduced `mobile_optimizations` to keep Colab tab alive for mobile users.
17
+ - Removed `Keep Tab Alive for Mobile` due to similar functionality with the new optimization.
18
+ - New extensions!
19
+ - [ilian6806/stable-diffusion-webui-state](https://github.com/ilian6806/stable-diffusion-webui-state) : Preserves Web UI parameters (inputs, sliders, checkboxes, etc.) after page reload.
20
+ - [pkuliyi2015/multidiffusion-upscaler-for-automatic1111](https://github.com/pkuliyi2015/multidiffusion-upscaler-for-automatic1111) : Offers tiled Diffusion and VAE optimization.
21
+ - [Zuellni/Stable-Diffusion-WebUI-Image-Filters](https://github.com/Zuellni/Stable-Diffusion-WebUI-Image-Filters) : A simple image postprocessing extension using the Pillow library.
22
+ - [Linaqruf/Umi-AI-debloat](https://github.com/Linaqruf/Umi-AI-debloat) : Wildcard manager, a fork of [Tsukreya/Umi-AI-debloat](https://github.com/Tsukreya/Umi-AI-debloat) that has already fixed the lowercase problem.
23
+ - [AlUlkesh/sd_delete_button](https://github.com/AlUlkesh/sd_delete_button) : Adds a delete button for Automatic1111 txt2img and img2img.
24
+ - New default models!
25
+ - Added [AnyLoRA_Anime_Mix](https://civitai.com/models/84586/) as a new AnyLoRA variant for 2D Anime generation, similar to NovelAI.
26
+ - Renamed the old `AnyLoRA` to [AnyLoRA_Default](https://civitai.com/models/23900/).
27
+ - Added [Ghost_Note_Delta](https://huggingface.co/corechan/GhostNotes#ghostnotedelta_m0528), [SDHK_V3](https://civitai.com/models/82813), for another Anime Model.
28
+ - Added [Majic_Mix_V5]() to replace `Chillout Mix`.
29
+ - Replaced `replicant_v2` with the latest and improved version, [Replicant_V3](https://huggingface.co/gsdf/Replicant-V3.0).
30
+ - Reintroduced `Illuminati Diffusion V1.1`.
31
+ - Removed Waifu Diffusion 1.5 models.
32
+ - New default VAEs!
33
+ - Added [Blessed VAE](https://huggingface.co/NoCrypt/blessed_vae).
34
+ - All default VAEs are now in `.safetensors`! Thanks to NoCrypt.
35
+ - ControlNet
36
+ - Added new Annotator, up to `Lama cleaner`.
37
+ - Included a list of new SDv2.x ControlNet Models from [thibaud/controlnet-sd21](https://huggingface.co/thibaud/controlnet-sd21).
38
+ - Added `Custom ControlNet Model` section to download custom controlnet models such as [Illumination](https://huggingface.co/ioclab/control_v1u_sd15_illumination_webui), [Brightness](https://huggingface.co/ioclab/control_v1p_sd15_brightness), the upcoming [QR Code](https://www.reddit.com/r/StableDiffusion/comments/141hg9x/controlnet_for_qr_code/) model, and any other unofficial ControlNet Model.
39
+ - Please ensure your custom ControlNet model has `sd15`/`sd21` in the filename.
40
+ - Reintroduced `t2i_adapter_model`.
41
+ - Custom Download Corner
42
+ - Added instructions for using the `fuse:` prefix, handling multiple URLs, and loading models from Google Drive.
43
+ - The `prune:`, `fp16:`, or `fp32:` prefixes currently are not available.
44
+ - Introduced a new section: `Download From Textfile`.
45
+ - This uses a similar approach to the [etherealxx/batchlinks-webui](https://github.com/etherealxx/batchlinks-webui) extensions.
46
+ - Provide a custom download URL for a `.txt` file instead of using the URL field. Edit the file: `/content/download_list.txt`.
47
+ - Available hashtags: `#model`, `#vae`, `#embedding`, `#lora`, `#hypernetwork`, `#extensions`, `#upscaler`. Aliases are not currently supported.
48
+ - Alternatively, you can input your `.txt` file into `custom_download_list_url`. This works for `pastebin`.
49
+
50
+ Example:
51
+ ```python
52
+ #model
53
+ url1
54
+ url2
55
+
56
+ #lora
57
+ url1 | filename1
58
+
59
+ #embedding
60
+ fuse:path
61
+ ```
62
+ - Launch
63
+ - Added `--opt-sdp-mem-attention` option for `accelerator`.
64
+ - Included error handling if the user forgets to download the model and vae. It will automatically download these two files before launch:
65
+ - Model: [AnyLoRA_Anime_Mix](https://civitai.com/models/84586/)
66
+ - VAE: [Animevae](https://huggingface.co/NoCrypt/resources/blob/main/VAE/any.vae.safetensors)
67
+ - Added `Token Merging Ratio` or `ToME SD` and `Negative Guidance Scale` to quicksettings
68
+ - Refactored `Download Generated Images` and other `Extras` cells by wrapping code inside functions.
69
+
70
+ - Bugfixes:
71
+ - Fixed `dpmpp_2m_v2_patch`, now it's working properly.
72
+ - Temporary fixed LoRA not applied when using Hires Fix
73
+ - Fixed file extensions undefined in `Custom Download Corner`
74
+
75
+ ### v.2.6.2 (17/05/23)
76
+ - Bugfixes:
77
+ - Attempting to fix generate button stuck problem by adding `latest_gradio` option to force update `gradio` to `3.31`
78
+
79
+ ### v.2.6.1 (17/05/23)
80
+ - Added `sd-civitai-browser` extension back, but this time using [SignalFlagZ/sd-civitai-browser](https://github.com/SignalFlagZ/sd-civitai-browser) version.
81
+ - Changed `stable-diffusion-webui-latent-two-shot` extension with [ashen-sensored](ashen-sensored/stable-diffusion-webui-two-shot)'s fork.
82
+ - Updated alternative tunnels recommendation: `ngrok` > `gradio` > `cloudflared` > `remotemoe` > `localhostrun` > `googleusercontent`. The ranking is based on the following reasons:
83
+ + Gradio: Faster and slightly more stable than the others, but may have queue problems.
84
+ - Cloudflared: Slow to start the application, loads extra networks and images.
85
+ - Remotemoe: Currently not usable.
86
+ - Localhostrun: Often gets disconnected, requiring a forced restart of cells to make it work.
87
+ - Googleusercontent: -
88
+ - Bugfixes:
89
+ - Fixed `stable-diffusion-webui-composable-lora` not being installed in `repo_type` values: `["AUTOMATIC1111", "AUTOMATIC1111-dev"]`.
90
+ - Set `commit_hash` to empty and use latest commit because the bugs already fixed.
91
+
92
+
93
+ ### v.2.6.0 (16/05/23)
94
+ - Updated to the latest commit.
95
+ - Available `repo_type` values: `["AUTOMATIC1111", "AUTOMATIC1111-dev", "Anapnoe"]`.
96
+ - Updated the new tile model with `control_v11f1e_sd15_tile_fp16.safetensors`.
97
+ - Added 2 optimization options:
98
+ - `ram_alloc_patch`, also known as the Camenduru Patch, to decrease RAM usage.
99
+ - `colab_optimizations`, also known as TheLastBen Patch, to load the model in VRAM and fix the gradio queue problem.
100
+ - Added the `use_presets` option to turn on or off the default prompt.
101
+ - Simplified the `Arguments` category options.
102
+ - Set a default value for `gradio_auth`.
103
+ - Changed [yfszzx/stable-diffusion-webui-images-browser](https://github.com/yfszzx/stable-diffusion-webui-images-browser) to [zanllp/sd-webui-infinite-image-browsing](https://github.com/zanllp/sd-webui-infinite-image-browsing).
104
+ - Added the extension [butaixianran/Stable-Diffusion-Webui-Civitai-Helper](https://github.com/butaixianran/Stable-Diffusion-Webui-Civitai-Helper).
105
+ - Added the extension [Bing-su/adetailer](https://github.com/Bing-su/adetailer).
106
+ - Added the extension [canvas-zoom](https://github.com/richrobber2/canvas-zoom).
107
+ - Added negative embeddings [EasyNegativeV2](https://huggingface.co/gsdf/Counterfeit-V3.0/blob/main/embedding/EasyNegativeV2.safetensors).
108
+ - Added a new theme for `Anapnoe-webui`: `["minimal", "minimal-orange"]`.
109
+ - Bug Fixes:
110
+ - Fixed Inpaint not showing images [#28](https://github.com/Linaqruf/sd-notebook-collection/issues/28). It looks like [Bing-su/adetailer](https://github.com/Bing-su/adetailer) and [canvas-zoom](https://github.com/richrobber2/canvas-zoom) are unusable in `Anapnoe-webui`.
111
+ - Fixed the `quicksettings` problem after the master repo was migrated from strings to a list. [#27](https://github.com/Linaqruf/sd-notebook-collection/issues/27) [#29](https://github.com/Linaqruf/sd-notebook-collection/issues/29).
112
+ - An annoying bug has recently occurred in `Anapnoe-webui`. Extra networks are not displaying Models, Embeddings, LoRA, etc., even if they exist in the respective folder. Temporary fixed by reset commit hash to [f2b9c2c](https://github.com/anapnoe/stable-diffusion-webui-ux/commit/f2b9c2cb4fa5f0e866c1b6f84e44d12ff6653af3). Track discussion [here](https://github.com/anapnoe/stable-diffusion-webui-ux/issues/141)
113
+
114
+ ### v.2.5.3 (21/04/23)
115
+ - `AUTOMATIC1111's stable-diffusion-webui` has been removed because Google Colab has prohibited the usage of any string named `stable-diffusion-webui` due to its massive usage of webui.
116
+ - `Anapnoe UI` has been set as default forcibly, also added warning messages.
117
+ - An `experimental` section has been added to install the `Anapnoe UI` integrated with `gradio 3.23.0`.
118
+ - `sd-webui-tunnels` forked by `camenduru` has been removed since the new ToS of Colab prohibits SSH shell or any similar tools.
119
+ - Now, there are two versions of the notebook available:
120
+ - [Cagliostro Colab UI](https://colab.research.google.com/github/Linaqruf/sd-notebook-collection/blob/main/cagliostro-colab-ui.ipynb): A compact and lightweight version that removes `AUTOMATIC1111's stable-diffusion-webui`.
121
+ - [Cagliostro Colab UI Pro](https://colab.research.google.com/github/Linaqruf/sd-notebook-collection/blob/main/cagliostro-colab-ui-pro.ipynb): It has all the features of the Cagliostro Colab UI update in 19/04. However, since I do not have a Colab Pro subscription, I am unlikely to maintain it.
122
+
123
+ -
124
+ ### v.2.5.2 (19/04/23)
125
+ - Reformat and simplified most cells
126
+ - Rewording all available variable to make it easier to read, example: `git_pull` to `update_webui`
127
+ - Added `output_drive_folder` to customize gdrive outputs folder name
128
+ - Removed `clean_install` option
129
+ - Merged `load_v2_in_vram` and `merge_in_vram` to `colab_optimization` and set to `False`
130
+ - Deleted `stable_diffusion_v_1_5` and `replicant_v1` from available models
131
+ - Added `custom_upscaler_url` back
132
+
133
+ ### v.2.5.1 (18/04/23)
134
+ - Added `cldm_config.yaml` for every ControlNet model (refer to [link](https://github.com/Mikubill/sd-webui-controlnet#download-models))
135
+ - Removed `custom_control_url` from `Custom Download Corner` cell
136
+ - Removed `illuminati_diffusion_v1_1` from `Available SDv2.x Model` list as the model has become exclusive to a certain image generation website
137
+ - Reworded certain arguments to make them more readable
138
+ - Removed `medvram` option from the `Arguments` list.
139
+
140
+ ### v.2.5.0 (16/04/23)
141
+ - Update Web UI to the latest version
142
+ - Moved `What's new?` to [GitHub repository](https://github.com/Linaqruf/sd-notebook-collection/blob/main/README.md)
143
+ - Added link to [Cagliostro Colab UI User Manual](https://github.com/Linaqruf/sd-notebook-collection/blob/main/MANUAL.md)
144
+ - Added `cheatsheet` in every header and subheader based on [Cagliostro Colab UI User Manual](https://github.com/Linaqruf/sd-notebook-collection/blob/main/MANUAL.md) explanation
145
+ - Reformatted Notebook to be more readable.
146
+ - If `use_anapnoe_ui` set to `True`, skip updating `stable-diffusion-webui-images-browser` when `update_extensions` set to `True`
147
+ - Added 2 more `os.environ` changes
148
+ - os.environ["PYTHONDONTWRITEBYTECODE"]='1'
149
+ - os.environ['PYTHONWARNINGS'] = 'ignore'
150
+ - Added [Replicant V2.0](https://huggingface.co/gsdf/Replicant-V2.0) as new SDv2.x model
151
+ - [**ControlNet V1.1**](https://github.com/lllyasviel/ControlNet-v1-1-nightly) Update!
152
+ - Removed old Annotator and ControlNet V1.0 model, and added new ones. Total: 13 new Annotator and 14 new ControlNet V1.1 models.
153
+ - Renamed `sd21_control_model` to `sd21_control_v10_sd21_model`
154
+ - Renamed `wd15_control_model` to `wd15_control_v10_sd21_model`
155
+ - Revamped how `Custom Download Corner` works.
156
+ - Removed `custom_upscaler_url`, `custom_control_url`, and `custom_components_url`
157
+ - Added a feature to prune model, fuse folder, copy from Google Drive. Users can find how to use it in this [link](https://github.com/Linaqruf/sd-notebook-collection/blob/main/MANUAL.md#custom-download-corner) or this table.
158
+
159
+ | Feature | Description | How to Use | Example |
160
+ | --- | --- | --- | --- |
161
+ | **Multiple Downloads** | Download multiple files at once. | Fill in the URL fields with the links to the files you want to download. Separate multiple URLs with a comma. | `url1, url2, url3` |
162
+ | **Auto-prune** | Prune models after downloading | Add `fp16:` or `fp32:` before URLs. | `fp16:url1` |
163
+ | **Copy from Google Drive** | Copy models from Google Drive and load them in the session. | Make sure you have already mounted Google Drive. Type the path to your model/lora/embedding from Google Drive. | `/content/drive/MyDrive/path/to/folder` |
164
+ | **Fusing Folder** | Fuse models/embeddings/LoRA folder to `/content/fused/{category}`. | Make sure you have already mounted Google Drive. Add `fuse:` before the path to the folder. | `fuse:/path/to/gdrive/folder` |
165
+ | **Auto-extract** | Extract files after downloading. | Add `links/to/file` ending with `.zip` or `.tar.lz4`. Extract files to specified destination directory. | `https//link.com/to/file` |
166
+ | **Install Extensions** | Install extensions for Stable Diffusion Web UI. | Add the link to the GitHub repository to `custom_extension_url`. | `https://github.com/user/repo` |
167
+
168
+ - Changed the Gradio authentication logic to use a `boolean` instead. The username is set to `cagliostro`, and the password is a randomly generated 6-character combination of ASCII letters and numbers.
169
+ - Added an option to enable `--opt-sdp-attention` instead of `xformers` to accelerate PyTorch 2.0.
170
+
171
+ ### v.2.2.0 (03/04/23)
172
+ - Update Web UI to the latest version
173
+ - Update xformers to `0.0.18`
174
+ - `git checkout` for extension using `Gradio 3.23` or above
175
+ - If using Anapnoe UI/UX, users can't `update_extensions` at the moment to prevent code conflict because of different gradio version
176
+
177
+ ### v.2.1.0 (31/03/23)
178
+ - Added Reinstall logic
179
+ - Fix bugs when upload images to ControlNet canvas, the images stretched
180
+ - Fix bugs when Web UI can't starting if `use_anapnoe_ui` set to **True**
181
+ - Changed `use_anapnoe_ui` logic by downloading and unpacking separate repo instead of `git remote origin set-url` and `git reset --hard`
182
+ - Added **Theme Selector** section to changed default (anapnoe-webui) theme before starting Web UI
183
+
184
+ ### v.2.0.1 (27/03/23)
185
+ - Clean install and update Web UI to latest version, commit hash : `955df7751eef11bb7697e2d77f6b8a6226b21e13`
186
+ - Manually edit `config.json` and `ui-config.json`
187
+ - Change `auto_model` and `auto_vae` logic, change `config.json` instead of using `--vae_path` and `--ckpt`
188
+
189
+ ### v.2.0.0 (27/03/23)
190
+ - Reformat notebook with [black python formatter](https://github.com/psf/black)
191
+ - Update Web UI and all extensions version (not latest)
192
+ - Downgrade `xformers` to `0.0.16`
193
+ - Downgrade `triton` to `2.0.0`
194
+ - Set both `lora_dir` and `additional_networks_extra_lora_path` path to `os.path.join(repo_dir, "models/Lora")`
195
+ - Set `lora_dir` path to `os.path.join(repo_dir, "models/ControlNet")`
196
+ - Reset Anapnoe UI commit hash to `802fb8a16ba4bdbfba0dca55f9cdb265f4bd86f2`
197
+ - Force reset commit hash to `3b47b000199ea8baf724080936ef985f53b3d081` when `use_anapnoe_ui` is enabled
198
+ - Added `mount_drive` function
199
+ - Added more error handling
200
+ - Set some environment variables:
201
+ - os.environ['colab_url'] = eval_js("google.colab.kernel.proxyPort(7860, {'cache': false})")
202
+ - os.environ["LD_PRELOAD"] = "libtcmalloc.so"
203
+ - os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"
204
+ - os.environ["SAFETENSORS_FAST_GPU"]= '1'
205
+ - Added 3 new default model
206
+ - [Anything V3.0](https://huggingface.co/Linaqruf/anything-v3.0)
207
+ - [AnyLoRA](https://huggingface.co/Lykon/AnyLoRA)
208
+ - [Anime Pastel Dream](https://huggingface.co/Lykon/AnimePastelDream)
209
+ - Deleted 3 default model
210
+ - Anything V3.2
211
+ - Anything V3.3
212
+ - HoloKuki V2
213
+ - Changed `Chillout-Mix` to `Chillout-Mix-Ni`
214
+ - Changed `Illuminati V1.1` download link from CivitAI to Huggingface
215
+ - Put all ControlNet things to new separated cell
216
+ - Added `pre_download_annotator` back
217
+ - Added `sd21_control_model`
218
+ - Added `wd15_control_model`
219
+ - Added option to change ControlNet model and Adapter config outside Web UI
220
+ - Added support for load custom url from Google Drive, tricky, by copying from Google Drive to destination path.
221
+ - Set `--multiple` as default alternative tunnels, please don't use Gradio URL at the moment
222
+ - Recommended Tunnels:
223
+ - `ngrok` > `cloudflared` > `remotemoe` > `localhostrun` > `googleusercontent` > `gradio`
224
+
225
+ ### v.1.4.1 (11/03/23)
226
+ - Trying to establish my own style, deleted all possible NoCrypt and TheLastBen's code legacy
227
+ - Temporary removed `pre_download_annotator`
228
+ - Deleted `PastelMix` from available models because the model license sold to [fantasy.ai](https://fantasy.ai/)
229
+ - Deleted `RefSlave-V2` from available models because the author removed his model from huggingface and civitai
230
+ - Added HoloKuki V2 as to available models
231
+ - Update t2i adapter model, currently has 8 model
232
+
233
+ ### v.1.4 (09/03/23)
234
+ - Update xformers pre-compiled wheels to `xFormers 0.0.17.dev466`
235
+ - Update pre-installed dependencies for `Python 3.9.16`
236
+ - Added new Stable Diffusion Web UI Extensions: `batchlinks-webui` and `sd-webui-llul`
237
+
238
+ ### v.1.3 (01/03/23)
239
+ - Added an option to save outputs to drive.
240
+ - Moved support button to separated and hidden section, because it looks ugly.
241
+ - Fixed some bugs where VAE can't be changed in webui.
242
+ - Updated dependencies, webui, and extensions to the latest version.
243
+ - Deleted `hitokomoru_v1_5`.
244
+ - Added [WD 1.5 Beta 2 - Aesthetic](https://huggingface.co/waifu-diffusion/wd-1-5-beta2). Releases note: [here](https://cafeai.notion.site/WD-1-5-Beta-2-Release-Notes-2852db5a9cdd456ba52fc5730b91acfd)
245
+ - Added `illuminati_diffusion_v1_1`. Required embeddings coming soon.
246
+ - Added `ref_slave_v2` and set it to default
247
+ - Used `git reset` to move the head to `3cd625854f9dc71235a432703ba82abfc5d1a3fc` when `try_new_ui_ux` set to True, as it's a stable commit history for now.
248
+ - Added `t2iadapter_sketch-fp16.safetensors` to t2iadapter model list.
249
+
250
+ ### v.1.2.2 (23/02/23)
251
+ - Added **Extra** section for optional cell, the first cell added to the section is **Download Generated Images V2**, to store your output to huggingface and download it.
252
+ - Changed how the installation works. If the folder exists, then skip unpacking.
253
+ - Added `Replicant V1.0` as default model.
254
+ - Added new UI/UX theme from [Anapnoe](https://github.com/anapnoe/stable-diffusion-webui). `[Experimental]`
255
+ - Added support for `multi-controlnet` by including a slider in certain cells (default value: `2`).
256
+ - Backed up all ControlNet annotator and model data to a personal repository.
257
+
258
+ ### v.1.2.1 (21/02/23)
259
+ - Added `tqdm` to track installation, unpacking, and download progress since original logs are disabled.
260
+ - Pulled the latest version of the repository and built-in extensions.
261
+ - Pre-downloaded ControlNet annotator/preprocessor.
262
+ - Added T2I Adapter model from TencentArc.
263
+ - Added several good custom upscalers such as `4x-Animesharp`, `4x-UltraSharp`, `Lollypop`, and others.
264
+ - Added Video Loopback extension for creating videos in Img2img, now with ControlNet support.
265
+ - Deleted CivitAI browser extension because another extension called `sd-filer` serves the same purpose. You can download models, lora, and embeddings from the web UI.
266
+ - Added Katanuki extension to convert results to transparent images.
267
+ - Deleted Haku Img Extensions because they were rarely used.
268
+ - Added 'Use Old Karras Scheduler' option to quick settings (header).
269
+ - Fixed an invalid URL for Chillout Mix Pruned and SD 1.5 Pruned.
270
+ - Fixed the 'Download Generated Images' cell when creating a duplicate zip from 'output.zip(1)' to 'output(n+1).zip'.
271
+ - Added checkboxes for randomly selecting VAE and Model.
content/sd-notebook-collection/booru-i2i-scraper.ipynb ADDED
@@ -0,0 +1,429 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "nbformat": 4,
3
+ "nbformat_minor": 0,
4
+ "metadata": {
5
+ "colab": {
6
+ "provenance": [],
7
+ "include_colab_link": true
8
+ },
9
+ "kernelspec": {
10
+ "name": "python3",
11
+ "display_name": "Python 3"
12
+ },
13
+ "language_info": {
14
+ "name": "python"
15
+ },
16
+ "gpuClass": "standard",
17
+ "accelerator": "GPU"
18
+ },
19
+ "cells": [
20
+ {
21
+ "cell_type": "markdown",
22
+ "metadata": {
23
+ "id": "view-in-github",
24
+ "colab_type": "text"
25
+ },
26
+ "source": [
27
+ "<a href=\"https://colab.research.google.com/github/Linaqruf/sd-notebook-collection/blob/main/booru-i2i-scraper.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
28
+ ]
29
+ },
30
+ {
31
+ "cell_type": "markdown",
32
+ "source": [
33
+ "\n",
34
+ "![](https://visitor-badge.glitch.me/badge?page_id=linaqruf.i2i-scraper) [![GitHub](https://badgen.net/badge/icon/github?icon=github&label)](https://github.com/Linaqruf/sd-notebook-collection/blob/main/booru-i2i-scraper.ipynb) \n",
35
+ "\n",
36
+ "# **Booru I2I Scraper**\n",
37
+ "Not perfect but usable<br>\n",
38
+ "\n",
39
+ "<details>\n",
40
+ " <summary><big>Support Us!</big></summary>\n",
41
+ " <ul>\n",
42
+ " <li>\n",
43
+ " <a href=\"https://ko-fi.com/linaqruf\">\n",
44
+ " <img src=\"https://img.shields.io/badge/Support%20me%20on%20Ko--fi-F16061?logo=ko-fi&logoColor=white&style=flat\" alt=\"Ko-fi badge\">\n",
45
+ " </a>\n",
46
+ " </li>\n",
47
+ " <li>\n",
48
+ " <a href=\"https://saweria.co/linaqruf\">\n",
49
+ " <img src=\"https://img.shields.io/badge/Saweria-7B3F00?style=flat&logo=ko-fi&logoColor=white\" alt=\"Saweria badge\">\n",
50
+ " </a>\n",
51
+ " </li>\n",
52
+ " </ul>\n",
53
+ "</details>"
54
+ ],
55
+ "metadata": {
56
+ "id": "3aKyBtukgWx3"
57
+ }
58
+ },
59
+ {
60
+ "cell_type": "code",
61
+ "source": [
62
+ "#@title Install Dependencies\n",
63
+ "import os\n",
64
+ "from IPython.utils import capture\n",
65
+ "\n",
66
+ "print(\"\u001b[1;32mInstalling...\")\n",
67
+ "print(\"\u001b[1;32mPlease wait...\\n\")\n",
68
+ "with capture.capture_output() as cap:\n",
69
+ " !pip -q install opencv-python tensorflow faiss-gpu huggingface_hub \"gradio==3.16.2\" gallery-dl\n",
70
+ " !pip -q uninstall -y Pillow\n",
71
+ " !pip -q install \"pillow>=9.1.0\"\n",
72
+ " del cap\n",
73
+ "print(\"\u001b[1;32mDone! Restarting...\")\n",
74
+ "\n",
75
+ "os.kill(os.getpid(), 9)"
76
+ ],
77
+ "metadata": {
78
+ "id": "7OaAW3Bt9m6S",
79
+ "cellView": "form"
80
+ },
81
+ "execution_count": null,
82
+ "outputs": []
83
+ },
84
+ {
85
+ "cell_type": "code",
86
+ "source": [
87
+ "#@title Launch\n",
88
+ "import argparse\n",
89
+ "import functools\n",
90
+ "import json\n",
91
+ "import os \n",
92
+ "import zipfile\n",
93
+ "import shutil\n",
94
+ "from IPython.utils import capture\n",
95
+ "from tqdm import tqdm\n",
96
+ "from pathlib import Path\n",
97
+ "from tensorflow.keras.models import load_model\n",
98
+ "import faiss\n",
99
+ "import PIL.Image\n",
100
+ "import gradio as gr\n",
101
+ "import numpy as np\n",
102
+ "\n",
103
+ "import requests\n",
104
+ "import tensorflow as tf\n",
105
+ "from huggingface_hub import hf_hub_download\n",
106
+ "\n",
107
+ "root_dir = \"/content\"\n",
108
+ "dir = os.path.join(root_dir, \"danbooru2022_image_similarity\")\n",
109
+ "repo_url = \"https://huggingface.co/spaces/SmilingWolf/danbooru2022_image_similarity\"\n",
110
+ "utils_dir = os.path.join(dir, \"Utils\")\n",
111
+ "index_dir = os.path.join(dir, \"index\")\n",
112
+ "deps_dir = os.path.join(root_dir, \"deps\")\n",
113
+ "image_dir = os.path.join(root_dir, \"images\")\n",
114
+ "app_py = os.path.join(repo_url, \"resolve/main/app.py\")\n",
115
+ "index_dir_files = [os.path.join(repo_url, \"resolve/main/index/cosine_ids.npy\"),\n",
116
+ " os.path.join(repo_url, \"resolve/main/index/cosine_infos.json\"),\n",
117
+ " os.path.join(repo_url, \"resolve/main/index/cosine_knn.index\")]\n",
118
+ "utils = os.path.join(repo_url, \"resolve/main/Utils/dbimutils.py\")\n",
119
+ "\n",
120
+ "TITLE = \"## Danbooru Explorer\"\n",
121
+ "DESCRIPTION = \"\"\"\n",
122
+ "Image similarity-based retrieval tool using:\n",
123
+ "- [SmilingWolf/wd-v1-4-convnext-tagger-v2](https://huggingface.co/SmilingWolf/wd-v1-4-convnext-tagger-v2) as feature extractor\n",
124
+ "- [Faiss](https://github.com/facebookresearch/faiss) and [autofaiss](https://github.com/criteo/autofaiss) for indexing\n",
125
+ "\"\"\"\n",
126
+ "CONV_FEXT_LAYER = \"predictions_norm\"\n",
127
+ "\n",
128
+ "files = [\"keras_metadata.pb\", \"saved_model.pb\", \"selected_tags.csv\"]\n",
129
+ "sub_dir = \"variables\"\n",
130
+ "sub_dir_files = [\"variables.data-00000-of-00001\", \"variables.index\"]\n",
131
+ "csv_file = files[-1]\n",
132
+ "model_repo = \"SmilingWolf/wd-v1-4-convnext-tagger-v2\"\n",
133
+ "model_dir = \"/content/wd14_tagger\"\n",
134
+ "\n",
135
+ "for directory in [dir, utils_dir, index_dir, deps_dir, image_dir]:\n",
136
+ " os.makedirs(directory, exist_ok=True)\n",
137
+ "\n",
138
+ "def ubuntu_deps(url, name, dst):\n",
139
+ " os.makedirs(dst, exist_ok=True)\n",
140
+ " !wget -q --show-progress {url}\n",
141
+ " with zipfile.ZipFile(name, 'r') as deps:\n",
142
+ " deps.extractall(dst)\n",
143
+ " !dpkg -i {dst}/*\n",
144
+ " os.remove(name)\n",
145
+ " shutil.rmtree(dst)\n",
146
+ "\n",
147
+ "print(\"\u001b[1;32mInstalling...\")\n",
148
+ "with capture.capture_output() as cap:\n",
149
+ " !apt -y update -qq\n",
150
+ " ubuntu_deps(\"https://huggingface.co/Linaqruf/fast-repo/resolve/main/ram_patch.zip\", \"ram_patch.zip\", deps_dir)\n",
151
+ " %env LD_PRELOAD=libtcmalloc.so\n",
152
+ " ubuntu_deps(\"https://huggingface.co/Linaqruf/fast-repo/resolve/main/deb-libs.zip\", \"deb-libs.zip\", deps_dir)\n",
153
+ " del cap\n",
154
+ "\n",
155
+ "print(\"\u001b[1;32mDownloading Danbooru Explorer...\")\n",
156
+ "with capture.capture_output() as cap:\n",
157
+ " for file in index_dir_files:\n",
158
+ " index_basename = os.path.basename(file)\n",
159
+ " !aria2c --console-log-level=error --summary-interval=10 -c -x 16 -k 1M -s 16 -d {index_dir} -o {index_basename} {file}\n",
160
+ " !aria2c --console-log-level=error --summary-interval=10 -c -x 16 -k 1M -s 16 -d {dir} -o \"app.py\" {app_py}\n",
161
+ " !aria2c --console-log-level=error --summary-interval=10 -c -x 16 -k 1M -s 16 -d {utils_dir} -o \"dbimutils.py\" {utils}\n",
162
+ " del cap\n",
163
+ "\n",
164
+ "init = \"\"\"\"\"\"\n",
165
+ "with open(os.path.join(utils_dir, \"__init__.py\"), 'w') as f:\n",
166
+ " f.write(init)\n",
167
+ "\n",
168
+ "setup = \"\"\"from setuptools import setup, find_packages\n",
169
+ "setup(name = \"Utils\", packages = find_packages())\"\"\"\n",
170
+ "with open(os.path.join(dir, \"setup.py\"), 'w') as f:\n",
171
+ " f.write(setup)\n",
172
+ "\n",
173
+ "os.chdir(dir)\n",
174
+ "print(\"\u001b[1;32mInstall setup.py...\")\n",
175
+ "with capture.capture_output() as cap:\n",
176
+ " !pip -q install .\n",
177
+ " del cap\n",
178
+ "\n",
179
+ "import Utils.dbimutils as dbimutils\n",
180
+ "\n",
181
+ "if not os.path.exists(model_dir):\n",
182
+ " print(f\"\u001b[1;32mDownloading WD 14 Tagger model from {model_repo}...\")\n",
183
+ " with capture.capture_output() as cap:\n",
184
+ " for file in files:\n",
185
+ " hf_hub_download(model_repo, file, cache_dir=model_dir, force_download=True, force_filename=file)\n",
186
+ " for file in sub_dir_files:\n",
187
+ " hf_hub_download(model_repo, file, subfolder=sub_dir, cache_dir=os.path.join(\n",
188
+ " model_dir, sub_dir), force_download=True, force_filename=file)\n",
189
+ " del cap\n",
190
+ " \n",
191
+ "def load_model(model_path, feature_extraction_layer):\n",
192
+ " full_model = tf.keras.models.load_model(model_path)\n",
193
+ " model = tf.keras.models.Model(\n",
194
+ " full_model.inputs, full_model.get_layer(feature_extraction_layer).output\n",
195
+ " )\n",
196
+ " return model\n",
197
+ "\n",
198
+ "def danbooru_id_to_url(image_id, selected_ratings, api_username=\"\", api_key=\"\"):\n",
199
+ " headers = {\"User-Agent\": \"image_similarity_tool\"}\n",
200
+ " ratings_to_letters = {\n",
201
+ " \"General\": \"g\",\n",
202
+ " \"Sensitive\": \"s\",\n",
203
+ " \"Questionable\": \"q\",\n",
204
+ " \"Explicit\": \"e\",\n",
205
+ " }\n",
206
+ "\n",
207
+ " acceptable_ratings = [ratings_to_letters[x] for x in selected_ratings]\n",
208
+ "\n",
209
+ " image_url = f\"https://danbooru.donmai.us/posts/{image_id}.json\"\n",
210
+ " if api_username != \"\" and api_key != \"\":\n",
211
+ " image_url = f\"{image_url}?api_key={api_key}&login={api_username}\"\n",
212
+ "\n",
213
+ " r = requests.get(image_url, headers=headers)\n",
214
+ " if r.status_code != 200:\n",
215
+ " return None\n",
216
+ "\n",
217
+ " content = json.loads(r.text)\n",
218
+ " image_url = content[\"large_file_url\"] if \"large_file_url\" in content else None\n",
219
+ " image_url = image_url if content[\"rating\"] in acceptable_ratings else None\n",
220
+ " return image_url\n",
221
+ "\n",
222
+ "class SimilaritySearcher:\n",
223
+ " def __init__(self, model, images_ids):\n",
224
+ " self.knn_index = None\n",
225
+ " self.knn_metric = None\n",
226
+ "\n",
227
+ " self.model = model\n",
228
+ " self.images_ids = images_ids\n",
229
+ "\n",
230
+ " def change_index(self, knn_metric):\n",
231
+ " if knn_metric == self.knn_metric:\n",
232
+ " return\n",
233
+ "\n",
234
+ " if knn_metric == \"ip\":\n",
235
+ " self.knn_index = faiss.read_index(\"index/ip_knn.index\")\n",
236
+ " config = json.loads(open(\"index/ip_infos.json\").read())[\"index_param\"]\n",
237
+ " elif knn_metric == \"cosine\":\n",
238
+ " self.knn_index = faiss.read_index(\"index/cosine_knn.index\")\n",
239
+ " config = json.loads(open(\"index/cosine_infos.json\").read())[\"index_param\"]\n",
240
+ "\n",
241
+ " faiss.ParameterSpace().set_index_parameters(self.knn_index, config)\n",
242
+ " self.knn_metric = knn_metric\n",
243
+ "\n",
244
+ "\n",
245
+ " def predict(\n",
246
+ " self, image, selected_ratings, knn_metric, api_username, api_key, n_neighbours\n",
247
+ " ):\n",
248
+ " _, height, width, _ = self.model.inputs[0].shape\n",
249
+ "\n",
250
+ " self.change_index(knn_metric)\n",
251
+ "\n",
252
+ " # Alpha to white\n",
253
+ " image = image.convert(\"RGBA\")\n",
254
+ " new_image = PIL.Image.new(\"RGBA\", image.size, \"WHITE\")\n",
255
+ " new_image.paste(image, mask=image)\n",
256
+ " image = new_image.convert(\"RGB\")\n",
257
+ " image = np.asarray(image)\n",
258
+ "\n",
259
+ " # PIL RGB to OpenCV BGR\n",
260
+ " image = image[:, :, ::-1]\n",
261
+ "\n",
262
+ " image = dbimutils.make_square(image, height)\n",
263
+ " image = dbimutils.smart_resize(image, height)\n",
264
+ " image = image.astype(np.float32)\n",
265
+ " image = np.expand_dims(image, 0)\n",
266
+ " target = self.model(image).numpy()\n",
267
+ "\n",
268
+ " if self.knn_metric == \"cosine\":\n",
269
+ " faiss.normalize_L2(target)\n",
270
+ "\n",
271
+ " dists, indexes = self.knn_index.search(target, k=n_neighbours)\n",
272
+ " neighbours_ids = self.images_ids[indexes][0]\n",
273
+ " neighbours_ids = [int(x) for x in neighbours_ids]\n",
274
+ "\n",
275
+ " captions = []\n",
276
+ " for image_id, dist in zip(neighbours_ids, dists[0]):\n",
277
+ " captions.append(f\"{image_id}/{dist:.2f}\")\n",
278
+ "\n",
279
+ " \n",
280
+ " image_urls = []\n",
281
+ " scraper_text = os.path.join(root_dir, \"scrape_this.txt\")\n",
282
+ " \n",
283
+ " for image_id in neighbours_ids:\n",
284
+ " current_url = danbooru_id_to_url(\n",
285
+ " image_id, selected_ratings, api_username, api_key\n",
286
+ " )\n",
287
+ " if current_url is not None: \n",
288
+ " image_urls.append(current_url)\n",
289
+ "\n",
290
+ " with open(scraper_text, 'w') as f:\n",
291
+ " f.write('\\n'.join(image_urls))\n",
292
+ "\n",
293
+ " !aria2c --console-log-level=error --summary-interval=10 -c -x 16 -k 1M -s 16 -d {image_dir} -i {scraper_text}\n",
294
+ "\n",
295
+ " return list(zip(image_urls, captions))\n",
296
+ "\n",
297
+ "\n",
298
+ "def main():\n",
299
+ " gpus = tf.config.list_physical_devices('GPU')\n",
300
+ " if len(gpus) > 0:\n",
301
+ " tf.config.set_visible_devices(gpus[0], 'GPU')\n",
302
+ " tf.config.experimental.set_memory_growth(gpus[0], True)\n",
303
+ " else:\n",
304
+ " print(\"No GPU found, using CPU instead.\")\n",
305
+ " model = load_model(model_dir, CONV_FEXT_LAYER)\n",
306
+ " images_ids = np.load(\"index/cosine_ids.npy\")\n",
307
+ "\n",
308
+ " searcher = SimilaritySearcher(model=model, images_ids=images_ids)\n",
309
+ "\n",
310
+ " with gr.Blocks() as demo:\n",
311
+ " gr.Markdown(TITLE)\n",
312
+ " gr.Markdown(DESCRIPTION)\n",
313
+ "\n",
314
+ " with gr.Row():\n",
315
+ " input = gr.Image(type=\"pil\", label=\"Input\")\n",
316
+ " with gr.Column():\n",
317
+ " with gr.Row():\n",
318
+ " api_username = gr.Textbox(label=\"Danbooru API Username\")\n",
319
+ " api_key = gr.Textbox(label=\"Danbooru API Key\")\n",
320
+ " selected_ratings = gr.CheckboxGroup(\n",
321
+ " choices=[\"General\", \"Sensitive\", \"Questionable\", \"Explicit\"],\n",
322
+ " value=[\"General\", \"Sensitive\"],\n",
323
+ " label=\"Ratings\",\n",
324
+ " )\n",
325
+ " with gr.Row():\n",
326
+ " selected_metric = gr.Radio(\n",
327
+ " choices=[\"cosine\"],\n",
328
+ " value=\"cosine\",\n",
329
+ " label=\"Metric selection\",\n",
330
+ " visible=False,\n",
331
+ " )\n",
332
+ " n_neighbours = gr.Slider(\n",
333
+ " minimum=1, maximum=500, value=5, step=1, label=\"# of images\"\n",
334
+ " )\n",
335
+ " find_btn = gr.Button(\"Find similar images\")\n",
336
+ " similar_images = gr.Gallery(label=\"Similar images\")\n",
337
+ "\n",
338
+ " similar_images.style(grid=5)\n",
339
+ " find_btn.click(\n",
340
+ " fn=searcher.predict,\n",
341
+ " inputs=[\n",
342
+ " input,\n",
343
+ " selected_ratings,\n",
344
+ " selected_metric,\n",
345
+ " api_username,\n",
346
+ " api_key,\n",
347
+ " n_neighbours,\n",
348
+ " ],\n",
349
+ " outputs=[similar_images],\n",
350
+ " )\n",
351
+ "\n",
352
+ " demo.queue()\n",
353
+ " demo.launch(share=True, debug=True)\n",
354
+ "\n",
355
+ "if __name__ == \"__main__\":\n",
356
+ " main()"
357
+ ],
358
+ "metadata": {
359
+ "id": "F4nNbab7-9tL",
360
+ "cellView": "form"
361
+ },
362
+ "execution_count": null,
363
+ "outputs": []
364
+ },
365
+ {
366
+ "cell_type": "code",
367
+ "source": [
368
+ "#@title ## 📝 Download Images\n",
369
+ "#@markdown Download file manually from files tab or save to Google Drive\n",
370
+ "%cd /content/\n",
371
+ "\n",
372
+ "!zip -r /content/images.zip images\n",
373
+ "\n",
374
+ "from pydrive.auth import GoogleAuth\n",
375
+ "from google.colab import drive\n",
376
+ "from pydrive.drive import GoogleDrive\n",
377
+ "from google.colab import auth\n",
378
+ "from oauth2client.client import GoogleCredentials\n",
379
+ "def create_folder(folder_name):\n",
380
+ " # Check if folder exists\n",
381
+ " file_list = drive.ListFile({'q': \"title='{}' and mimeType='application/vnd.google-apps.folder' and trashed=false\".format(folder_name)}).GetList()\n",
382
+ " if len(file_list) > 0:\n",
383
+ " # Folder exists\n",
384
+ " print('Debug: Folder exists')\n",
385
+ " folder_id = file_list[0]['id']\n",
386
+ " else:\n",
387
+ " print('Debug: Creating folder')\n",
388
+ " file = drive.CreateFile({'title': folder_name, 'mimeType': 'application/vnd.google-apps.folder'})\n",
389
+ " file.Upload()\n",
390
+ " folder_id = file.attr['metadata']['id']\n",
391
+ " # return folder id\n",
392
+ " return folder_id\n",
393
+ "# Upload file to Google Drive\n",
394
+ "def upload_file(file_name, folder_id, save_as):\n",
395
+ " # Check if file exists\n",
396
+ " file_list = drive.ListFile({'q': \"title='{}' and trashed=false\".format(save_as)}).GetList()\n",
397
+ " if len(file_list) > 0:\n",
398
+ " print('Debug: File already exists')\n",
399
+ " # Change file name to avoid overwriting\n",
400
+ " save_as = save_as + ' (1)'\n",
401
+ " file = drive.CreateFile({'title': save_as, 'parents': [{'id': folder_id}]})\n",
402
+ " file.SetContentFile(file_name)\n",
403
+ " # Upload and set permission to public\n",
404
+ " file.Upload()\n",
405
+ " file.InsertPermission({'type': 'anyone', 'value': 'anyone', 'role': 'reader'})\n",
406
+ " # return file id\n",
407
+ " return file.attr['metadata']['id']\n",
408
+ "\n",
409
+ "use_drive = True #@param {type:\"boolean\"}\n",
410
+ "folder_name = \"scraper\" #@param {type: \"string\"}\n",
411
+ "save_as = \"nakiriayame.zip\" #@param {type: \"string\"}\n",
412
+ "\n",
413
+ "if use_drive:\n",
414
+ " auth.authenticate_user()\n",
415
+ " gauth = GoogleAuth()\n",
416
+ " gauth.credentials = GoogleCredentials.get_application_default()\n",
417
+ " drive = GoogleDrive(gauth)\n",
418
+ " file_id = upload_file('/content/images.zip', create_folder(folder_name), save_as)\n",
419
+ " print(\"Your sharing link: https://drive.google.com/file/d/\" + file_id + \"/view?usp=sharing\") "
420
+ ],
421
+ "metadata": {
422
+ "cellView": "form",
423
+ "id": "1gH_zBqtTyEb"
424
+ },
425
+ "execution_count": null,
426
+ "outputs": []
427
+ }
428
+ ]
429
+ }
content/sd-notebook-collection/cagliostro-colab-ui.ipynb ADDED
@@ -0,0 +1,1250 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cells": [
3
+ {
4
+ "cell_type": "markdown",
5
+ "metadata": {
6
+ "id": "view-in-github",
7
+ "colab_type": "text"
8
+ },
9
+ "source": [
10
+ "<a href=\"https://colab.research.google.com/github/Linaqruf/sd-notebook-collection/blob/main/cagliostro-colab-ui.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
11
+ ]
12
+ },
13
+ {
14
+ "cell_type": "markdown",
15
+ "source": [
16
+ "[![visitor][visitor-badge]][visitor-stats]\n",
17
+ "[![ko-fi][ko-fi-badge]][ko-fi-link]\n",
18
+ "\n",
19
+ "# **Cagliostro Colab UI**\n",
20
+ "All-in-One, Customizable and Flexible Stable Diffusion for Google Colab.\n",
21
+ "\n",
22
+ "**Version 3.0.0** | [Github][link-to-github] | [What's New?][README]\n",
23
+ "\n",
24
+ "<!-- [visitor-badge]: https://visitor-badge.glitch.me/badge?page_id=linaqruf.cag-webui -->\n",
25
+ "[visitor-badge]: https://api.visitorbadge.io/api/visitors?path=Cagliostro%20Colab%20UI&label=Visitors&labelColor=%2334495E&countColor=%231ABC9C&style=flat&labelStyle=none\n",
26
+ "[visitor-stats]: https://visitorbadge.io/status?path=Cagliostro%20Colab%20UI\n",
27
+ "[ko-fi-badge]: https://img.shields.io/badge/Support%20me%20on%20Ko--fi-F16061?logo=ko-fi&logoColor=white&style=flat\n",
28
+ "[ko-fi-link]: https://ko-fi.com/linaqruf\n",
29
+ "[link-to-github]: https://github.com/Linaqruf/sd-notebook-collection/blob/main/cagliostro-colab-ui.ipynb\n",
30
+ "[README]: https://github.com/Linaqruf/sd-notebook-collection/blob/main/README.md#whats-new\n",
31
+ "[MANUAL]: https://github.com/Linaqruf/sd-notebook-collection/blob/main/MANUAL.md#cagliostro-colab-ui-user-manual\n",
32
+ "\n"
33
+ ],
34
+ "metadata": {
35
+ "id": "WgQr3s96015a"
36
+ }
37
+ },
38
+ {
39
+ "cell_type": "code",
40
+ "source": [
41
+ "# @title ## **Install Cagliostro Colab UI**\n",
42
+ "import os\n",
43
+ "import time\n",
44
+ "import json\n",
45
+ "import base64\n",
46
+ "import shutil\n",
47
+ "import subprocess\n",
48
+ "import threading\n",
49
+ "import sys\n",
50
+ "from IPython.display import display, HTML\n",
51
+ "from google.colab.output import eval_js\n",
52
+ "from IPython.utils import capture\n",
53
+ "from tqdm import tqdm\n",
54
+ "\n",
55
+ "python_version = \".\".join(sys.version.split(\".\")[:2])\n",
56
+ "colablib_path = f\"/usr/local/lib/python{python_version}/dist-packages/colablib\"\n",
57
+ "if not os.path.exists(colablib_path):\n",
58
+ " subprocess.run(['pip', 'install', 'git+https://github.com/Linaqruf/colablib'])\n",
59
+ "\n",
60
+ "from colablib.colored_print import cprint, print_line\n",
61
+ "from colablib.utils import py_utils, config_utils, package_utils\n",
62
+ "from colablib.utils.config_utils import pastebin_reader as read\n",
63
+ "from colablib.utils.ubuntu_utils import ubuntu_deps\n",
64
+ "from colablib.sd_models.downloader import aria2_download\n",
65
+ "from colablib.utils.git_utils import update_repo, batch_update, validate_repo, reset_repo, patch_repo\n",
66
+ "\n",
67
+ "%store -r\n",
68
+ "\n",
69
+ "################################\n",
70
+ "# COLAB ARGUMENTS GOES HERE\n",
71
+ "################################\n",
72
+ "\n",
73
+ "# @markdown ### **Drive Config**\n",
74
+ "mount_drive = False # @param {type:'boolean'}\n",
75
+ "output_drive_folder = \"cagliostro-colab-ui/outputs\" #@param {type:'string'}\n",
76
+ "# @markdown ### **Repo Config**\n",
77
+ "repo_type = \"Anapnoe\" #@param [\"AUTOMATIC1111\", \"AUTOMATIC1111-Dev\", \"Anapnoe\"]\n",
78
+ "update_webui = False # @param {type:'boolean'}\n",
79
+ "update_extensions = False # @param {type:'boolean'}\n",
80
+ "commit_hash = \"\" # @param {type:'string'}\n",
81
+ "dpmpp_2m_v2_patch = True # @param {type:'boolean'}\n",
82
+ "# @markdown ### **Optimization Config**\n",
83
+ "# @markdown > It's not recommended to set params below to `True` if you have **Colab Pro** subscription.\n",
84
+ "colab_optimizations = True # @param {type:'boolean'}\n",
85
+ "# @markdown > Specify `mobile_optimizations` to keep colab tab alive for mobile users\n",
86
+ "mobile_optimizations = False # @param {type:'boolean'}\n",
87
+ "\n",
88
+ "################################\n",
89
+ "# DIRECTORY CONFIG\n",
90
+ "################################\n",
91
+ "\n",
92
+ "# VAR\n",
93
+ "voldemort, voldy = read(\"kq6ZmHFU\")[:2]\n",
94
+ "\n",
95
+ "# ROOT DIR\n",
96
+ "root_dir = \"/content\"\n",
97
+ "drive_dir = os.path.join(root_dir, \"drive\", \"MyDrive\")\n",
98
+ "repo_dir = os.path.join(root_dir, \"cagliostro-colab-ui\")\n",
99
+ "tmp_dir = os.path.join(root_dir, \"tmp\")\n",
100
+ "patches_dir = os.path.join(root_dir, \"patches\")\n",
101
+ "deps_dir = os.path.join(root_dir, \"deps\")\n",
102
+ "fused_dir = os.path.join(root_dir, \"fused\")\n",
103
+ "\n",
104
+ "# REPO DIR\n",
105
+ "models_dir = os.path.join(repo_dir, \"models\", \"Stable-diffusion\")\n",
106
+ "vaes_dir = os.path.join(repo_dir, \"models\", \"VAE\")\n",
107
+ "hypernetworks_dir = os.path.join(repo_dir, \"models\", \"hypernetworks\")\n",
108
+ "lora_dir = os.path.join(repo_dir, \"models\", \"Lora\")\n",
109
+ "control_dir = os.path.join(repo_dir, \"models\", \"ControlNet\")\n",
110
+ "esrgan_dir = os.path.join(repo_dir, \"models\", \"ESRGAN\")\n",
111
+ "embeddings_dir = os.path.join(repo_dir, \"embeddings\")\n",
112
+ "extensions_dir = os.path.join(repo_dir, \"extensions\")\n",
113
+ "annotator_dir = os.path.join(extensions_dir, f\"{voldy}-controlnet\", \"annotator\")\n",
114
+ "output_subdir = [\"txt2img-images\", \"img2img-images\", \"extras-images\", \"txt2img-grids\", \"img2img-grids\"]\n",
115
+ "\n",
116
+ "# CONFIG\n",
117
+ "config_file = os.path.join(repo_dir, \"config.json\")\n",
118
+ "ui_config_file = os.path.join(repo_dir, \"ui-config.json\")\n",
119
+ "style_path = os.path.join(repo_dir, \"style.css\")\n",
120
+ "download_list = os.path.join(root_dir, \"download_list.txt\")\n",
121
+ "\n",
122
+ "\n",
123
+ "################################\n",
124
+ "# REPO TYPE CONFIG\n",
125
+ "################################\n",
126
+ "\n",
127
+ "repo_type_lower = repo_type.lower()\n",
128
+ "\n",
129
+ "package_url = [\n",
130
+ " f\"https://huggingface.co/Linaqruf/fast-repo/resolve/main/{repo_type_lower}-webui.tar.lz4\",\n",
131
+ " f\"https://huggingface.co/Linaqruf/fast-repo/resolve/main/{repo_type_lower}-webui-deps.tar.lz4\",\n",
132
+ " f\"https://huggingface.co/Linaqruf/fast-repo/resolve/main/{repo_type_lower}-webui-cache.tar.lz4\",\n",
133
+ "]\n",
134
+ "\n",
135
+ "repo_type_to_repo_name = {\n",
136
+ " \"anapnoe\" : f\"anapnoe/{voldemort}-ux\",\n",
137
+ " \"automatic1111\" : f\"AUTOMATIC1111/{voldemort}\",\n",
138
+ " \"automatic1111-dev\" : f\"AUTOMATIC1111/{voldemort}\",\n",
139
+ "}\n",
140
+ "\n",
141
+ "branch_type_to_branch = {\n",
142
+ " \"automatic1111\" : \"master\",\n",
143
+ " \"automatic1111-dev\" : \"dev\"\n",
144
+ "}\n",
145
+ "\n",
146
+ "with capture.capture_output() as cap:\n",
147
+ " for dir in [\"root_dir\", \"fused_dir\", \"repo_dir\", \"tmp_dir\", \"models_dir\", \"vaes_dir\", \"hypernetworks_dir\", \"embeddings_dir\", \"extensions_dir\", \"lora_dir\", \"control_dir\", \"esrgan_dir\"]:\n",
148
+ " %store {dir}\n",
149
+ " for file in [\"config_file\", \"ui_config_file\", \"style_path\", \"download_list\"]:\n",
150
+ " %store {file}\n",
151
+ " for var in [\"voldemort\", \"voldy\"]:\n",
152
+ " %store {var}\n",
153
+ " del cap\n",
154
+ "\n",
155
+ "def mount_func(directory):\n",
156
+ " output_dir = os.path.join(repo_dir, \"outputs\")\n",
157
+ "\n",
158
+ " if mount_drive:\n",
159
+ " print_line(80, color=\"green\")\n",
160
+ " if not os.path.exists(directory):\n",
161
+ " from google.colab import drive\n",
162
+ " cprint(\"Mounting google drive...\", color=\"green\", reset=False)\n",
163
+ " drive.mount(os.path.dirname(directory))\n",
164
+ " output_dir = os.path.join(directory, output_drive_folder)\n",
165
+ " cprint(\"Set default output path to:\", output_dir, color=\"green\")\n",
166
+ "\n",
167
+ " return output_dir\n",
168
+ "\n",
169
+ "def setup_directories():\n",
170
+ " for dir in [fused_dir, models_dir, vaes_dir,\n",
171
+ " hypernetworks_dir, embeddings_dir, extensions_dir,\n",
172
+ " lora_dir, control_dir, esrgan_dir]:\n",
173
+ " os.makedirs(dir, exist_ok=True)\n",
174
+ "\n",
175
+ "def pre_download(dir, urls, desc, overwrite=False):\n",
176
+ " gpu_info = py_utils.get_gpu_info()\n",
177
+ " version = py_utils.get_python_version().split()[0]\n",
178
+ " major_minor = \".\".join(version.split(\".\")[:2])\n",
179
+ " xformers_version = \"0.0.20\"\n",
180
+ " python_path = f\"/usr/local/lib/python{major_minor}/dist-packages/\"\n",
181
+ " ffmpy_path = os.path.join(python_path, \"ffmpy-0.3.0.dist-info\")\n",
182
+ "\n",
183
+ " for url in tqdm(urls, desc=desc):\n",
184
+ " filename = py_utils.get_filename(url)\n",
185
+ " aria2_download(dir, filename, url, quiet=True)\n",
186
+ " if filename == f\"{repo_type.lower()}-webui-deps.tar.lz4\":\n",
187
+ " package_utils.extract_package(filename, python_path, overwrite=True)\n",
188
+ " else:\n",
189
+ " package_utils.extract_package(filename, \"/\", overwrite=overwrite)\n",
190
+ " os.remove(filename)\n",
191
+ "\n",
192
+ " if os.path.exists(ffmpy_path):\n",
193
+ " shutil.rmtree(ffmpy_path)\n",
194
+ "\n",
195
+ " if not 'T4' in gpu_info:\n",
196
+ " subprocess.run(['pip', 'uninstall', '-y', 'xformers'], check=True)\n",
197
+ " subprocess.run(['pip', 'install', '-q', f'xformers=={xformers_version}'], check=True)\n",
198
+ "\n",
199
+ "def install_dependencies():\n",
200
+ " ubuntu_deps = [\"aria2\", \"lz4\", \"unionfs-fuse\"]\n",
201
+ " cprint(\"Installing ubuntu dependencies\", color=\"green\")\n",
202
+ " subprocess.run([\"apt\", \"install\"] + ubuntu_deps)\n",
203
+ "\n",
204
+ "def install_webui(repo_dir, desc):\n",
205
+ " try:\n",
206
+ " if not os.path.exists(repo_dir):\n",
207
+ " pre_download(root_dir, package_url, desc, overwrite=False)\n",
208
+ " return\n",
209
+ "\n",
210
+ " repo_name, _, current_branch = validate_repo(repo_dir)\n",
211
+ " repo_type_lower = repo_type.lower()\n",
212
+ " expected_repo_name = repo_type_to_repo_name.get(repo_type_lower)\n",
213
+ "\n",
214
+ " if expected_repo_name == repo_name:\n",
215
+ " expected_branch = branch_type_to_branch.get(repo_type_lower)\n",
216
+ " if expected_branch is None or expected_branch == current_branch:\n",
217
+ " cprint(f\"'{repo_name}' {current_branch if expected_branch else ''} already installed, skipping...\", color=\"green\")\n",
218
+ " return\n",
219
+ "\n",
220
+ " cprint(f\"Another repository exist. Uninstall '{repo_name}'...\", color=\"green\")\n",
221
+ " shutil.rmtree(repo_dir)\n",
222
+ " pre_download(root_dir, package_url, desc)\n",
223
+ " except Exception as e:\n",
224
+ " cprint(f\"An error occurred: {e}\", color=\"green\")\n",
225
+ "\n",
226
+ "def configure_output_path(config_path, output_dir, output_subdir):\n",
227
+ " config = config_utils.read_config(config_path)\n",
228
+ " config_updates = {\n",
229
+ " \"outdir_txt2img_samples\" : os.path.join(output_dir, output_subdir[0]),\n",
230
+ " \"outdir_img2img_samples\" : os.path.join(output_dir, output_subdir[1]),\n",
231
+ " \"outdir_extras_samples\" : os.path.join(output_dir, output_subdir[2]),\n",
232
+ " \"outdir_txt2img_grids\" : os.path.join(output_dir, output_subdir[3]),\n",
233
+ " \"outdir_img2img_grids\" : os.path.join(output_dir, output_subdir[4])\n",
234
+ " }\n",
235
+ "\n",
236
+ " config.update(config_updates)\n",
237
+ " config_utils.write_config(config_path, config)\n",
238
+ "\n",
239
+ " for dir in output_subdir:\n",
240
+ " os.makedirs(os.path.join(output_dir, dir), exist_ok=True)\n",
241
+ "\n",
242
+ "def prepare_environment():\n",
243
+ " cprint(f\"Preparing environment...\", color=\"green\")\n",
244
+ "\n",
245
+ " os.environ[\"colab_url\"] = eval_js(\"google.colab.kernel.proxyPort(7860, {'cache': false})\")\n",
246
+ " os.environ[\"TF_CPP_MIN_LOG_LEVEL\"] = \"3\"\n",
247
+ " os.environ[\"SAFETENSORS_FAST_GPU\"] = \"1\"\n",
248
+ " os.environ['PYTORCH_CUDA_ALLOC_CONF'] = \"garbage_collection_threshold:0.9,max_split_size_mb:512\"\n",
249
+ " os.environ[\"PYTHONWARNINGS\"] = \"ignore\"\n",
250
+ "\n",
251
+ "def play_audio(url):\n",
252
+ " display(HTML(f'<audio src=\"{url}\" controls autoplay style=\"display:none\"></audio>'))\n",
253
+ "\n",
254
+ "def main():\n",
255
+ " global output_dir\n",
256
+ "\n",
257
+ " os.chdir(root_dir)\n",
258
+ " start_time = time.time()\n",
259
+ "\n",
260
+ " output_dir = mount_func(drive_dir)\n",
261
+ "\n",
262
+ " gpu_info = py_utils.get_gpu_info(get_gpu_name=True)\n",
263
+ " python_info = py_utils.get_python_version()\n",
264
+ " torch_info = py_utils.get_torch_version()\n",
265
+ "\n",
266
+ " print_line(80, color=\"green\")\n",
267
+ " cprint(f\" [-] Current GPU:\", gpu_info, color=\"flat_yellow\")\n",
268
+ " cprint(f\" [-] Python\", python_info, color=\"flat_yellow\")\n",
269
+ " cprint(f\" [-] Torch\", torch_info, color=\"flat_yellow\")\n",
270
+ " print_line(80, color=\"green\")\n",
271
+ "\n",
272
+ " install_dependencies()\n",
273
+ "\n",
274
+ " print_line(80, color=\"green\")\n",
275
+ " install_webui(repo_dir, cprint(f\"Unpacking {repo_type} Webui\", color=\"green\", tqdm_desc=True))\n",
276
+ " prepare_environment()\n",
277
+ "\n",
278
+ " configure_output_path(config_file, output_dir, output_subdir)\n",
279
+ "\n",
280
+ " print_line(80, color=\"green\")\n",
281
+ " if update_webui and not commit_hash:\n",
282
+ " update_repo(cwd=repo_dir, args=\"-X theirs --rebase --autostash\")\n",
283
+ "\n",
284
+ " setup_directories ()\n",
285
+ "\n",
286
+ " if commit_hash:\n",
287
+ " reset_repo(repo_dir, commit_hash)\n",
288
+ "\n",
289
+ " repo_name, current_commit_hash, current_branch = validate_repo(repo_dir)\n",
290
+ " cprint(f\"Using '{repo_name}' repository...\", color=\"green\")\n",
291
+ " cprint(f\"Branch: {current_branch}, Commit hash: {current_commit_hash}\", color=\"green\")\n",
292
+ "\n",
293
+ " print_line(80, color=\"green\")\n",
294
+ " cprint(\"Hotfixes and Optimization:\", color=\"green\")\n",
295
+ "\n",
296
+ " if dpmpp_2m_v2_patch:\n",
297
+ " dpmpp_2m_v2_url = \"https://gist.githubusercontent.com/Linaqruf/514d40676e97a70ffc3a2451bbf51555/raw/3fa447ebfac6b98a25485374b70447f848267589/01-add-DPMPP-2M-V2.patch\"\n",
298
+ " patch_repo(url=dpmpp_2m_v2_url, dir=patches_dir, cwd=repo_dir, whitespace_fix=True, quiet=True)\n",
299
+ " shutil.rmtree(patches_dir)\n",
300
+ " cprint(\" [-] DPM++ 2m V2 and DPM++ 2m Karras V2 patch done!\", color=\"green\")\n",
301
+ "\n",
302
+ " if colab_optimizations:\n",
303
+ " lowram_patch_url = \"https://raw.githubusercontent.com/ddPn08/automatic1111-colab/main/patches/stablediffusion-lowram.patch\"\n",
304
+ " stable_diffusion_repo_dir = os.path.join(repo_dir, \"repositories/stable-diffusion-stability-ai\")\n",
305
+ " patch_repo(url=lowram_patch_url, dir=patches_dir, cwd=stable_diffusion_repo_dir, quiet=True)\n",
306
+ " shutil.rmtree(patches_dir)\n",
307
+ " cprint(\" [-] Stable Diffusion V2.x lowram patch done!\", color=\"green\")\n",
308
+ "\n",
309
+ " subprocess.run([\"sed\", \"-i\", f\"s@os.path.splitext(checkpoint_file)@os.path.splitext(checkpoint_file); map_location='cuda'@\", os.path.join(repo_dir, \"modules\", \"sd_models.py\")])\n",
310
+ " subprocess.run([\"sed\", \"-i\", f\"s@ui.create_ui().*@ui.create_ui();shared.demo.queue(concurrency_count=999999,status_update_rate=0.1)@\", os.path.join(repo_dir, \"webui.py\")])\n",
311
+ " subprocess.run([\"sed\", \"-i\", f\"s@map_location='cpu'@map_location='cuda'@\", os.path.join(repo_dir, \"modules\", \"extras.py\")])\n",
312
+ " cprint(\" [-] TheLastben's colab optimization done!\", color=\"green\")\n",
313
+ "\n",
314
+ " if mobile_optimizations:\n",
315
+ " audio_url = \"https://raw.githubusercontent.com/KoboldAI/KoboldAI-Client/main/colab/silence.m4a\"\n",
316
+ " audio_thread = threading.Thread(target=play_audio, args=(audio_url,))\n",
317
+ " audio_thread.start()\n",
318
+ " cprint(\" [-] Mobile Optimization done!\", color=\"green\")\n",
319
+ "\n",
320
+ " if \"anapnoe\" in repo_name and \"9931e861dfb128735c4a928a7beb5b5c0af30593\" in current_commit_hash:\n",
321
+ " hires_prompt_fix = \"https://gist.githubusercontent.com/Linaqruf/8fef456d53604f8c3bcd16722ea7d2f6/raw/a3382087c6e32f9a171f4b5e8aeb572a61682801/0001-Add-New-Label-for-Hires-Prompt.patch\"\n",
322
+ " patch_repo(url=hires_prompt_fix, dir=patches_dir, cwd=repo_dir, whitespace_fix=True, quiet=True)\n",
323
+ " shutil.rmtree(patches_dir)\n",
324
+ " cprint(\" [-] Hires Prompt patch done!\", color=\"green\")\n",
325
+ "\n",
326
+ " print_line(80, color=\"green\")\n",
327
+ "\n",
328
+ " if update_extensions:\n",
329
+ " batch_update(fetch=True, directory=extensions_dir, desc=cprint(f\"Updating extensions\", color=\"green\", tqdm_desc=True))\n",
330
+ "\n",
331
+ " if not os.path.exists(download_list):\n",
332
+ " download_list_url = \"https://raw.githubusercontent.com/Linaqruf/sd-notebook-collection/main/config/download_list.txt\"\n",
333
+ " aria2_download(os.path.dirname(download_list), os.path.basename(download_list), download_list_url, quiet=True)\n",
334
+ "\n",
335
+ " elapsed_time = py_utils.calculate_elapsed_time(start_time)\n",
336
+ " print_line(80, color=\"green\")\n",
337
+ " cprint(f\"Finished installation. Took {elapsed_time}.\", color=\"flat_yellow\")\n",
338
+ " cprint(\"All is done! Go to the next step.\", color=\"flat_yellow\")\n",
339
+ " print_line(80, color=\"green\")\n",
340
+ "\n",
341
+ "main()"
342
+ ],
343
+ "metadata": {
344
+ "id": "A6c7-qjDdb0X",
345
+ "cellView": "form"
346
+ },
347
+ "execution_count": null,
348
+ "outputs": []
349
+ },
350
+ {
351
+ "cell_type": "code",
352
+ "source": [
353
+ "# @title ## **Download Model and VAE**\n",
354
+ "import os\n",
355
+ "import time\n",
356
+ "from colablib.utils import py_utils\n",
357
+ "from colablib.colored_print import cprint, print_line\n",
358
+ "from colablib.sd_models.downloader import aria2_download, get_modelname\n",
359
+ "\n",
360
+ "%store -r\n",
361
+ "\n",
362
+ "################################\n",
363
+ "# COLAB ARGUMENTS GOES HERE\n",
364
+ "################################\n",
365
+ "\n",
366
+ "# @markdown ### **Stable Diffusion v1.x Model**\n",
367
+ "Anything_V3_0 = False # @param {type: 'boolean'}\n",
368
+ "AnyLoRA_Default = False # @param {type: 'boolean'}\n",
369
+ "AnyLoRA_Anime_Mix = True # @param {type: 'boolean'}\n",
370
+ "Ghost_Note_Delta = False # @param {type: 'boolean'}\n",
371
+ "SDHK_V3 = False # @param {type: 'boolean'}\n",
372
+ "Majic_Mix_V5 = False # @param {type: 'boolean'}\n",
373
+ "# @markdown ### **Stable Diffusion v2.x Model**\n",
374
+ "Replicant_V3 = False # @param {type: 'boolean'}\n",
375
+ "Illuminati_Diffusion = False # @param {type: 'boolean'}\n",
376
+ "# @markdown ### **VAE Model**\n",
377
+ "Anime = True # @param {type: 'boolean'}\n",
378
+ "Blessed = False # @param {type: 'boolean'}\n",
379
+ "Waifu_Diffusion = False # @param {type: 'boolean'}\n",
380
+ "Stable_Diffusion = False # @param {type: 'boolean'}\n",
381
+ "\n",
382
+ "# VAR\n",
383
+ "read_token = \"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\"\n",
384
+ "user_header = f\"Authorization: Bearer {read_token}\"\n",
385
+ "\n",
386
+ "################################\n",
387
+ "# URL DICT GOES HERE\n",
388
+ "################################\n",
389
+ "\n",
390
+ "model_dict = {\n",
391
+ " \"Anything_V3_0\" : \"https://huggingface.co/Linaqruf/stolen/resolve/main/pruned-models/anything-v3-0-pruned.ckpt\",\n",
392
+ " \"AnyLoRA_Default\" : \"https://huggingface.co/Linaqruf/stolen/resolve/main/pruned-models/AnyLoRA_noVae_fp16-pruned.safetensors\",\n",
393
+ " \"AnyLoRA_Anime_Mix\" : \"https://huggingface.co/Linaqruf/stolen/resolve/main/fp16/aamAnyloraAnimeMixAnime_v10-fp16-pruned.safetensors\",\n",
394
+ " \"Ghost_Note_Delta\" : \"https://huggingface.co/Linaqruf/stolen/resolve/main/fp16/GhostNoteDelta_m0528_fp16.safetensors\",\n",
395
+ " \"SDHK_V3\" : \"https://huggingface.co/Linaqruf/stolen/resolve/main/fp16/sdhk_v30.safetensors\",\n",
396
+ " \"Majic_Mix_V5\" : \"https://huggingface.co/Linaqruf/stolen/resolve/main/fp16/majicmixRealistic_v5.safetensors\",\n",
397
+ " \"Replicant_V3\" : \"https://huggingface.co/gsdf/Replicant-V3.0/resolve/main/Replicant-V3.0_fp16.safetensors\",\n",
398
+ " \"Illuminati_Diffusion\" : \"https://huggingface.co/Linaqruf/stolen/resolve/main/pruned-models/illuminatiDiffusionV1_v11.safetensors\",\n",
399
+ "}\n",
400
+ "\n",
401
+ "vae_dict = {\n",
402
+ " \"Anime\" : \"https://huggingface.co/NoCrypt/resources/resolve/main/VAE/any.vae.safetensors\",\n",
403
+ " \"Blessed\" : \"https://huggingface.co/NoCrypt/resources/resolve/main/VAE/blessed2.vae.safetensors\",\n",
404
+ " \"Waifu_Diffusion\" : \"https://huggingface.co/NoCrypt/resources/resolve/main/VAE/wd.vae.safetensors\",\n",
405
+ " \"Stable_Diffusion\" : \"https://huggingface.co/stabilityai/sd-vae-ft-mse-original/resolve/main/vae-ft-mse-840000-ema-pruned.safetensors\",\n",
406
+ "}\n",
407
+ "\n",
408
+ "def filter_dict_items(dict_items):\n",
409
+ " result_list = []\n",
410
+ " for key, url in dict_items.items():\n",
411
+ " if globals().get(key):\n",
412
+ " result_list.append((key, url))\n",
413
+ " return result_list\n",
414
+ "\n",
415
+ "def main():\n",
416
+ " start_time = time.time()\n",
417
+ "\n",
418
+ " download_list = [\n",
419
+ " (filter_dict_items(model_dict), models_dir),\n",
420
+ " (filter_dict_items(vae_dict), vaes_dir)\n",
421
+ " ]\n",
422
+ "\n",
423
+ " print_line(80, color=\"green\")\n",
424
+ " cprint(\" [-] Downloading Stable Diffusion Models and VAEs...\", color=\"flat_yellow\")\n",
425
+ " for lst, dst in download_list:\n",
426
+ " for key, url in lst:\n",
427
+ " print_line(80, color=\"green\")\n",
428
+ " extensions = os.path.splitext(get_modelname(url))[1]\n",
429
+ " if dst == vaes_dir:\n",
430
+ " extensions = \".vae\" + extensions\n",
431
+ " aria2_download(url=url, download_dir=dst, filename=key + extensions, user_header=user_header)\n",
432
+ "\n",
433
+ " elapsed_time = py_utils.calculate_elapsed_time(start_time)\n",
434
+ " print_line(80, color=\"green\")\n",
435
+ " cprint(f\"Download finished. Took {elapsed_time}.\", color=\"flat_yellow\")\n",
436
+ " cprint(\"All is done! Go to the next step.\", color=\"flat_yellow\")\n",
437
+ " print_line(80, color=\"green\")\n",
438
+ "\n",
439
+ "main()"
440
+ ],
441
+ "metadata": {
442
+ "id": "7nUM-wRFhBa3",
443
+ "cellView": "form"
444
+ },
445
+ "execution_count": null,
446
+ "outputs": []
447
+ },
448
+ {
449
+ "cell_type": "code",
450
+ "source": [
451
+ "# @title ## **ControlNet v1.1**\n",
452
+ "import time\n",
453
+ "import shutil\n",
454
+ "import os\n",
455
+ "import subprocess\n",
456
+ "from tqdm import tqdm\n",
457
+ "from colablib.sd_models.downloader import aria2_download, download\n",
458
+ "from colablib.utils import config_utils, py_utils, git_utils\n",
459
+ "from colablib.colored_print import cprint, print_line\n",
460
+ "from colablib.utils.py_utils import get_filename\n",
461
+ "\n",
462
+ "%store -r\n",
463
+ "\n",
464
+ "# @markdown ### **ControlNet Annotator**\n",
465
+ "pre_download_annotator = True # @param {type: 'boolean'}\n",
466
+ "# @markdown ### **SDv1.x ControlNet Model**\n",
467
+ "control_v11_sd15_model = True # @param {type: 'boolean'}\n",
468
+ "t2i_adapter_model = False # @param {type: 'boolean'}\n",
469
+ "# @markdown ### **SDv2.x ControlNet Model**\n",
470
+ "control_v11_sd21_model = False # @param {type: 'boolean'}\n",
471
+ "# @markdown ### **Custom ControlNet Model**\n",
472
+ "# @markdown - Make sure your custom controlnet model has `sd15`/`sd21` in the filename.\n",
473
+ "# @markdown - Use comma separation for multiple URLs, e.g. `url1, url2, url3`.\n",
474
+ "custom_controlnet_url = \"\" #@param [\"\", \"https://huggingface.co/ioclab/ioc-controlnet/resolve/main/models/control_v1p_sd15_illumination.safetensors\", \"https://huggingface.co/ioclab/ioc-controlnet/resolve/main/models/control_v1p_sd15_brightness.safetensors\"] {allow-input: true}\n",
475
+ "# @markdown ### **ControlNet Config**\n",
476
+ "control_net_max_models_num = 2 # @param {type:\"slider\", min:1, max:10, step:1}\n",
477
+ "\n",
478
+ "custom_controlnet_dict = {\n",
479
+ " \"control_v1p_sd15_illumination\" : \"https://huggingface.co/ioclab/ioc-controlnet/resolve/main/models/control_v1p_sd15_illumination.safetensors\",\n",
480
+ " \"control_v1p_sd15_brightness\" : \"https://huggingface.co/ioclab/ioc-controlnet/resolve/main/models/control_v1p_sd15_brightness.safetensors\"\n",
481
+ "}\n",
482
+ "\n",
483
+ "annotator_dict = {\n",
484
+ " \"midas\" : \"https://huggingface.co/lllyasviel/ControlNet/resolve/main/annotator/ckpts/dpt_hybrid-midas-501f0c75.pt\",\n",
485
+ " \"leres\" : [\n",
486
+ " \"https://huggingface.co/lllyasviel/Annotators/resolve/main/res101.pth\",\n",
487
+ " \"https://huggingface.co/lllyasviel/Annotators/resolve/main/latest_net_G.pth\"\n",
488
+ " ],\n",
489
+ " \"hed\" : \"https://huggingface.co/lllyasviel/Annotators/resolve/main/ControlNetHED.pth\",\n",
490
+ " \"mlsd\" : \"https://huggingface.co/lllyasviel/ControlNet/resolve/main/annotator/ckpts/mlsd_large_512_fp32.pth\",\n",
491
+ " \"openpose\" : [\n",
492
+ " \"https://huggingface.co/lllyasviel/Annotators/resolve/main/body_pose_model.pth\",\n",
493
+ " \"https://huggingface.co/lllyasviel/Annotators/resolve/main/hand_pose_model.pth\",\n",
494
+ " \"https://huggingface.co/lllyasviel/Annotators/resolve/main/facenet.pth\"\n",
495
+ " ],\n",
496
+ " \"clip_vision\" : \"https://huggingface.co/openai/clip-vit-large-patch14/resolve/main/pytorch_model.bin\",\n",
497
+ " \"pidinet\" : \"https://huggingface.co/lllyasviel/Annotators/resolve/main/table5_pidinet.pth\",\n",
498
+ " \"uniformer\" : \"https://huggingface.co/lllyasviel/ControlNet/resolve/main/annotator/ckpts/upernet_global_small.pth\",\n",
499
+ " \"zoedepth\" : \"https://huggingface.co/lllyasviel/Annotators/resolve/main/ZoeD_M12_N.pt\",\n",
500
+ " \"normal_bae\" : \"https://huggingface.co/lllyasviel/Annotators/resolve/main/scannet.pt\",\n",
501
+ " \"oneformer\" : [\n",
502
+ " \"https://huggingface.co/lllyasviel/Annotators/resolve/main/150_16_swin_l_oneformer_coco_100ep.pth\",\n",
503
+ " \"https://huggingface.co/lllyasviel/Annotators/resolve/main/250_16_swin_l_oneformer_ade20k_160k.pth\"\n",
504
+ " ],\n",
505
+ " \"lineart\" : [\n",
506
+ " \"https://huggingface.co/lllyasviel/Annotators/resolve/main/sk_model.pth\",\n",
507
+ " \"https://huggingface.co/lllyasviel/Annotators/resolve/main/sk_model2.pth\"\n",
508
+ " ],\n",
509
+ " \"lineart_anime\" : \"https://huggingface.co/lllyasviel/Annotators/resolve/main/netG.pth\",\n",
510
+ " \"manga_line\" : \"https://huggingface.co/lllyasviel/Annotators/resolve/main/erika.pth\"\n",
511
+ "}\n",
512
+ "\n",
513
+ "control_v11_sd15_url = [\n",
514
+ " \"https://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11e_sd15_ip2p_fp16.safetensors\",\n",
515
+ " \"https://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11e_sd15_shuffle_fp16.safetensors\",\n",
516
+ " \"https://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_canny_fp16.safetensors\",\n",
517
+ " \"https://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11f1p_sd15_depth_fp16.safetensors\",\n",
518
+ " \"https://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_inpaint_fp16.safetensors\",\n",
519
+ " \"https://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_lineart_fp16.safetensors\",\n",
520
+ " \"https://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_mlsd_fp16.safetensors\",\n",
521
+ " \"https://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_normalbae_fp16.safetensors\",\n",
522
+ " \"https://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_openpose_fp16.safetensors\",\n",
523
+ " \"https://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_scribble_fp16.safetensors\",\n",
524
+ " \"https://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_seg_fp16.safetensors\",\n",
525
+ " \"https://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_softedge_fp16.safetensors\",\n",
526
+ " \"https://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15s2_lineart_anime_fp16.safetensors\",\n",
527
+ " \"https://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11f1e_sd15_tile_fp16.safetensors\",\n",
528
+ "]\n",
529
+ "\n",
530
+ "control_v11_sd21_url = [\n",
531
+ " \"https://huggingface.co/thibaud/controlnet-sd21/resolve/main/control_v11p_sd21_ade20k.safetensors\",\n",
532
+ " \"https://huggingface.co/thibaud/controlnet-sd21/resolve/main/control_v11p_sd21_canny.safetensors\",\n",
533
+ " \"https://huggingface.co/thibaud/controlnet-sd21/resolve/main/control_v11p_sd21_color.safetensors\",\n",
534
+ " \"https://huggingface.co/thibaud/controlnet-sd21/resolve/main/control_v11p_sd21_depth.safetensors\",\n",
535
+ " \"https://huggingface.co/thibaud/controlnet-sd21/resolve/main/control_v11p_sd21_hed.safetensors\",\n",
536
+ " \"https://huggingface.co/thibaud/controlnet-sd21/resolve/main/control_v11p_sd21_lineart.safetensors\",\n",
537
+ " \"https://huggingface.co/thibaud/controlnet-sd21/resolve/main/control_v11p_sd21_normalbae.safetensors\",\n",
538
+ " \"https://huggingface.co/thibaud/controlnet-sd21/resolve/main/control_v11p_sd21_openpose.safetensors\",\n",
539
+ " \"https://huggingface.co/thibaud/controlnet-sd21/resolve/main/control_v11p_sd21_openposev2.safetensors\",\n",
540
+ " \"https://huggingface.co/thibaud/controlnet-sd21/resolve/main/control_v11p_sd21_scribble.safetensors\",\n",
541
+ " \"https://huggingface.co/thibaud/controlnet-sd21/resolve/main/control_v11p_sd21_zoedepth.safetensors\"\n",
542
+ "]\n",
543
+ "\n",
544
+ "t2i_adapter_url = [\n",
545
+ " \"https://huggingface.co/TencentARC/T2I-Adapter/resolve/main/models/t2iadapter_canny_sd14v1.pth\",\n",
546
+ " \"https://huggingface.co/TencentARC/T2I-Adapter/resolve/main/models/t2iadapter_canny_sd15v2.pth\",\n",
547
+ " \"https://huggingface.co/TencentARC/T2I-Adapter/resolve/main/models/t2iadapter_color_sd14v1.pth\",\n",
548
+ " \"https://huggingface.co/TencentARC/T2I-Adapter/resolve/main/models/t2iadapter_depth_sd14v1.pth\",\n",
549
+ " \"https://huggingface.co/TencentARC/T2I-Adapter/resolve/main/models/t2iadapter_depth_sd15v2.pth\",\n",
550
+ " \"https://huggingface.co/TencentARC/T2I-Adapter/resolve/main/models/t2iadapter_keypose_sd14v1.pth\",\n",
551
+ " \"https://huggingface.co/TencentARC/T2I-Adapter/resolve/main/models/t2iadapter_openpose_sd14v1.pth\",\n",
552
+ " \"https://huggingface.co/TencentARC/T2I-Adapter/resolve/main/models/t2iadapter_seg_sd14v1.pth\",\n",
553
+ " \"https://huggingface.co/TencentARC/T2I-Adapter/resolve/main/models/t2iadapter_sketch_sd14v1.pth\",\n",
554
+ " \"https://huggingface.co/TencentARC/T2I-Adapter/resolve/main/models/t2iadapter_sketch_sd15v2.pth\",\n",
555
+ " \"https://huggingface.co/TencentARC/T2I-Adapter/resolve/main/models/t2iadapter_style_sd14v1.pth\",\n",
556
+ " \"https://huggingface.co/TencentARC/T2I-Adapter/resolve/main/models/t2iadapter_zoedepth_sd15v1.pth\"\n",
557
+ "]\n",
558
+ "\n",
559
+ "def cldm_config_path(destination_path):\n",
560
+ " if \"control\" in destination_path:\n",
561
+ " if \"sd15\" in destination_path:\n",
562
+ " return \"control_v11e_sd15_shuffle.yaml\" if \"_shuffle_\" in destination_path else \"cldm_v15.yaml\"\n",
563
+ " if \"sd21\" in destination_path:\n",
564
+ " return \"cldm_v21.yaml\"\n",
565
+ " elif \"t2i\" in destination_path:\n",
566
+ " adapter_name = os.path.splitext(os.path.basename(destination_path))[0]\n",
567
+ " return adapter_name + \".yaml\"\n",
568
+ " return None\n",
569
+ "\n",
570
+ "def cldm_config(destination_path):\n",
571
+ " repo_name, _, _ = git_utils.validate_repo(repo_dir)\n",
572
+ "\n",
573
+ " control_net_model_config = cldm_config_path(destination_path)\n",
574
+ " if control_net_model_config is not None:\n",
575
+ " cldm_config_src = os.path.join(extensions_dir, os.path.join(f\"{voldy}-controlnet\", \"models\", control_net_model_config))\n",
576
+ " cldm_config_dst = os.path.splitext(destination_path)[0] + \".yaml\"\n",
577
+ " if not os.path.exists(cldm_config_dst):\n",
578
+ " shutil.copy(cldm_config_src, cldm_config_dst)\n",
579
+ "\n",
580
+ "def batch_download(urls, dst, desc=None, quiet=False, cldm_model=False):\n",
581
+ " for url in tqdm(urls, disable=quiet, desc=cprint(desc, color=\"green\", tqdm_desc=True)):\n",
582
+ " filename = get_filename(url)\n",
583
+ " aria2_download(url=url, download_dir=dst, filename=filename, quiet=True)\n",
584
+ " if cldm_model:\n",
585
+ " cldm_config(os.path.join(dst, filename))\n",
586
+ "\n",
587
+ "\n",
588
+ "def custom_controlnet_download(urls, dst):\n",
589
+ " for url in urls.split(\",\"):\n",
590
+ " url = url.strip()\n",
591
+ " if url != \"\":\n",
592
+ " print_line(80, color=\"green\")\n",
593
+ " filename = get_filename(url)\n",
594
+ " download(url=url, filename=filename, dst=control_dir)\n",
595
+ " cldm_config(os.path.join(dst, filename))\n",
596
+ "\n",
597
+ "def download_annotator(directory, desc):\n",
598
+ " for category, urls in tqdm(annotator_dict.items(), desc=cprint(desc, color=\"green\", tqdm_desc=True)):\n",
599
+ " if category == \"clip_vision\":\n",
600
+ " dst = os.path.join(directory, \"clip_vision\")\n",
601
+ " else:\n",
602
+ " dst = os.path.join(directory, \"downloads\", category)\n",
603
+ " os.makedirs(dst, exist_ok=True)\n",
604
+ " urls = [urls] if isinstance(urls, str) else urls\n",
605
+ " batch_download(urls, dst, quiet=True)\n",
606
+ "\n",
607
+ "def main():\n",
608
+ " start_time = time.time()\n",
609
+ "\n",
610
+ " repo_name, _, _ = git_utils.validate_repo(repo_dir)\n",
611
+ "\n",
612
+ " config = config_utils.read_config(config_file)\n",
613
+ " config[\"control_net_max_models_num\"] = control_net_max_models_num\n",
614
+ " config[\"control_net_models_path\"] = control_dir\n",
615
+ " config[\"control_net_allow_script_control\"] = True\n",
616
+ " config_utils.write_config(config_file, config)\n",
617
+ "\n",
618
+ " print_line(80, color=\"green\")\n",
619
+ " cprint(\" [-] Downloading ControlNet Models...\", color=\"flat_yellow\")\n",
620
+ " print_line(80, color=\"green\")\n",
621
+ "\n",
622
+ " if pre_download_annotator:\n",
623
+ " download_annotator(annotator_dir, \"Downloading ControlNet Annotator/Preprocessor\")\n",
624
+ " if control_v11_sd15_model:\n",
625
+ " batch_download(control_v11_sd15_url, control_dir, \"Downloading SDv1.x ControlNet Model\", cldm_model=True)\n",
626
+ " if control_v11_sd21_model:\n",
627
+ " batch_download(control_v11_sd21_url, control_dir, \"Downloading SDv2.x ControlNet Model\", cldm_model=True)\n",
628
+ " if t2i_adapter_model:\n",
629
+ " batch_download(t2i_adapter_url, control_dir, \"Downloading SDv1.x Text2Image Adapter Model\", cldm_model=True)\n",
630
+ " if custom_controlnet_url:\n",
631
+ " print_line(80, color=\"green\")\n",
632
+ " cprint(\" [-] Downloading Custom ControlNet Models...\", color=\"flat_yellow\")\n",
633
+ " custom_controlnet_download(custom_controlnet_url, control_dir)\n",
634
+ "\n",
635
+ " print_line(80, color=\"green\")\n",
636
+ " elapsed_time = py_utils.calculate_elapsed_time(start_time)\n",
637
+ " cprint(f\"Download finished. Took {elapsed_time}.\", color=\"flat_yellow\")\n",
638
+ " cprint(\"All is done! Go to the next step.\", color=\"flat_yellow\")\n",
639
+ " print_line(80, color=\"green\")\n",
640
+ "\n",
641
+ "main()"
642
+ ],
643
+ "metadata": {
644
+ "id": "K28QYTFhf7Cu",
645
+ "cellView": "form"
646
+ },
647
+ "execution_count": null,
648
+ "outputs": []
649
+ },
650
+ {
651
+ "cell_type": "code",
652
+ "source": [
653
+ "# @title ## **Custom Download Corner**\n",
654
+ "import os\n",
655
+ "import time\n",
656
+ "from pydantic import BaseModel\n",
657
+ "from colablib.utils.py_utils import get_filename\n",
658
+ "from colablib.sd_models.downloader import aria2_download, download\n",
659
+ "from colablib.utils.ubuntu_utils import unionfuse\n",
660
+ "from colablib.utils.git_utils import clone_repo\n",
661
+ "from colablib.colored_print import cprint, print_line\n",
662
+ "from colablib.utils.config_utils import read_config\n",
663
+ "\n",
664
+ "%store -r\n",
665
+ "\n",
666
+ "# @markdown\n",
667
+ "# @markdown ### **Download from Custom URLs**\n",
668
+ "# @markdown - Use comma separation for multiple URLs, e.g. `url1, url2, url3`.\n",
669
+ "# @markdown - To load Google Drive, use `fuse:` followed by path, e.g. `fuse:/content/MyDrive/LoRA`.\n",
670
+ "# @markdown - Copy your model path from Google Drive to URL fields to copy your model to the web UI models directory.\n",
671
+ "custom_model_url = \"\" # @param {'type': 'string'}\n",
672
+ "custom_vae_url = \"\" # @param {'type': 'string'}\n",
673
+ "custom_embedding_url = \"\" # @param {'type': 'string'}\n",
674
+ "custom_LoRA_url = \"\" # @param {'type': 'string'}\n",
675
+ "custom_hypernetwork_url = \"\" # @param {'type': 'string'}\n",
676
+ "custom_extensions_url = \"\" # @param {'type': 'string'}\n",
677
+ "custom_upscaler_url = \"\" # @param {'type': 'string'}\n",
678
+ "# @markdown ### <br>`NEW` **Download from Textfile**\n",
679
+ "# @markdown - Provide a custom download URL for a `.txt` file instead of using the URL field. Edit the file: `/content/download_list.txt`.\n",
680
+ "# @markdown - Available hashtags: `#model`, `#vae`, `#embedding`, `#lora`, `#hypernetwork`, `#extensions`, `#upscaler`.\n",
681
+ "# @markdown - Or you can input your `.txt` file in `custom_download_list_url` below. Works for `pastebin`.\n",
682
+ "custom_download_list_url = \"\" # @param {'type': 'string'}\n",
683
+ "\n",
684
+ "class CustomDirs(BaseModel):\n",
685
+ " url: str\n",
686
+ " dst: str\n",
687
+ "\n",
688
+ "custom_dirs = {\n",
689
+ " \"model\" : CustomDirs(url=custom_model_url, dst=models_dir),\n",
690
+ " \"vae\" : CustomDirs(url=custom_vae_url, dst=vaes_dir),\n",
691
+ " \"embedding\" : CustomDirs(url=custom_embedding_url, dst=embeddings_dir),\n",
692
+ " \"lora\" : CustomDirs(url=custom_LoRA_url, dst=lora_dir),\n",
693
+ " \"hypernetwork\": CustomDirs(url=custom_hypernetwork_url, dst=hypernetworks_dir),\n",
694
+ " \"extensions\" : CustomDirs(url=custom_extensions_url, dst=extensions_dir),\n",
695
+ " \"upscaler\" : CustomDirs(url=custom_upscaler_url, dst=esrgan_dir)\n",
696
+ "}\n",
697
+ "\n",
698
+ "def fuse(url, key, dst):\n",
699
+ " if \"extensions\" in key:\n",
700
+ " cprint(f\"Folder can't be fused, skipping...\")\n",
701
+ " return\n",
702
+ "\n",
703
+ " path = url.split(\"fuse:\")[1].strip()\n",
704
+ " category_dir = os.path.join(fused_dir, key)\n",
705
+ " if os.path.exists(category_dir):\n",
706
+ " cprint(f\"Folder '{category_dir}' is already fused, skipping...\", color=\"yellow\")\n",
707
+ " return\n",
708
+ "\n",
709
+ " cprint(f\"Fusing process started for PATH: '{path}'\", color=\"green\")\n",
710
+ " unionfuse(category_dir, path, dst)\n",
711
+ " cprint(f\"Fusing process completed. Valid '{key}' folder located at: '{category_dir}' \", color=\"green\")\n",
712
+ "\n",
713
+ "def parse_urls(filename):\n",
714
+ " content = read_config(filename)\n",
715
+ " lines = content.strip().split('\\n')\n",
716
+ " result = {}\n",
717
+ " key = ''\n",
718
+ " for line in lines:\n",
719
+ " if not line.strip():\n",
720
+ " continue\n",
721
+ " if line.startswith('//'):\n",
722
+ " continue\n",
723
+ " if line.startswith('#'):\n",
724
+ " key = line[1:].lower()\n",
725
+ " result[key] = []\n",
726
+ " else:\n",
727
+ " urls = [url.strip() for url in line.split(',') if url.strip() != '']\n",
728
+ " result[key].extend(urls)\n",
729
+ " return result\n",
730
+ "\n",
731
+ "def custom_download(custom_dirs):\n",
732
+ " for key, value in custom_dirs.items():\n",
733
+ " urls = value.url.split(\",\") # Split the comma-separated URLs\n",
734
+ " dst = value.dst\n",
735
+ "\n",
736
+ " if value.url:\n",
737
+ " print_line(80, color=\"green\")\n",
738
+ " cprint(f\" [-] Downloading Custom {key}...\", color=\"flat_yellow\")\n",
739
+ "\n",
740
+ " for url in urls:\n",
741
+ " url = url.strip() # Remove leading/trailing whitespaces from each URL\n",
742
+ " if url != \"\":\n",
743
+ " print_line(80, color=\"green\")\n",
744
+ " if \"|\" in url:\n",
745
+ " url, filename = map(str.strip, url.split(\"|\"))\n",
746
+ " if not filename.endswith((\".safetensors\", \".ckpt\", \".pt\", \"pth\")):\n",
747
+ " filename = filename + os.path.splitext(get_filename(url))[1]\n",
748
+ " else:\n",
749
+ " if not url.startswith(\"fuse:\"):\n",
750
+ " filename = get_filename(url)\n",
751
+ "\n",
752
+ " if url.startswith(\"fuse:\"):\n",
753
+ " fuse(url, key, dst)\n",
754
+ " elif key == \"extensions\":\n",
755
+ " clone_repo(url, cwd=dst)\n",
756
+ " else:\n",
757
+ " download(url=url, filename=filename, dst=dst, quiet=False)\n",
758
+ "\n",
759
+ "def download_from_textfile(filename):\n",
760
+ " for key, urls in parse_urls(filename).items():\n",
761
+ " key_lower = key.lower()\n",
762
+ " if key_lower in custom_dirs:\n",
763
+ " if custom_dirs[key_lower].url:\n",
764
+ " custom_dirs[key_lower].url += ',' + ','.join(urls)\n",
765
+ " else:\n",
766
+ " custom_dirs[key_lower].url = ','.join(urls)\n",
767
+ " else:\n",
768
+ " cprint(f\"Warning: Category '{key}' from the file is not found in custom_dirs.\", color=\"yellow\")\n",
769
+ "\n",
770
+ "def custom_download_list(url):\n",
771
+ " filename = \"custom_download_list.txt\"\n",
772
+ " filepath = os.path.join(root_dir, filename)\n",
773
+ " if os.path.exists(filepath):\n",
774
+ " os.remove(filepath)\n",
775
+ " if 'pastebin.com' in url:\n",
776
+ " if 'raw' not in url:\n",
777
+ " url = url.replace('pastebin.com', 'pastebin.com/raw')\n",
778
+ " download(url=url, filename=filename, dst=root_dir, quiet=True)\n",
779
+ " return filepath\n",
780
+ "\n",
781
+ "def main():\n",
782
+ " start_time = time.time()\n",
783
+ " textfile_path = download_list\n",
784
+ " if custom_download_list_url:\n",
785
+ " textfile_path = custom_download_list(custom_download_list_url)\n",
786
+ " download_from_textfile(textfile_path)\n",
787
+ " custom_download(custom_dirs)\n",
788
+ "\n",
789
+ " elapsed_time = py_utils.calculate_elapsed_time(start_time)\n",
790
+ " print_line(80, color=\"green\")\n",
791
+ " cprint(f\"Download finished. Took {elapsed_time}.\", color=\"flat_yellow\")\n",
792
+ " cprint(\"All is done! Go to the next step.\", color=\"flat_yellow\")\n",
793
+ " print_line(80, color=\"green\")\n",
794
+ "\n",
795
+ "main()"
796
+ ],
797
+ "metadata": {
798
+ "cellView": "form",
799
+ "id": "SoZcrfQk6Y2z"
800
+ },
801
+ "execution_count": null,
802
+ "outputs": []
803
+ },
804
+ {
805
+ "cell_type": "code",
806
+ "source": [
807
+ "#@title ## **Start Cagliostro Colab UI**\n",
808
+ "import random\n",
809
+ "import string\n",
810
+ "from pydantic import BaseModel\n",
811
+ "from typing import List, Optional\n",
812
+ "from colablib.utils import config_utils\n",
813
+ "from colablib.colored_print import cprint, print_line\n",
814
+ "from colablib.utils.git_utils import validate_repo\n",
815
+ "\n",
816
+ "%store -r\n",
817
+ "\n",
818
+ "################################\n",
819
+ "# COLAB ARGUMENTS GOES HERE\n",
820
+ "################################\n",
821
+ "\n",
822
+ "# @markdown ### **Alternative Tunnel**\n",
823
+ "# @markdown > Recommended Tunnels: `ngrok` > `gradio` > `cloudflared` > `remotemoe` > `localhostrun` > `googleusercontent`\n",
824
+ "select_tunnel = \"multiple\" # @param ['gradio', 'multiple','cloudflared', 'localhostrun', 'remotemoe', \"googleusercontent\"]\n",
825
+ "# @markdown > Get your `ngrok_token` [here](https://dashboard.ngrok.com/get-started/your-authtoken)\n",
826
+ "ngrok_token = \"\" # @param {type: 'string'}\n",
827
+ "ngrok_region = \"ap\" # @param [\"us\", \"eu\", \"au\", \"ap\", \"sa\", \"jp\", \"in\"]\n",
828
+ "# @markdown ### **UI/UX Config**\n",
829
+ "select_theme = \"minimal_orange\" # @param ['moonlight', 'ogxRed', 'fun', 'ogxCyan', 'ogxCyanInvert', 'ogxBGreen', 'default_orange', 'tron2', 'd-230-52-94', 'minimal', 'ogxRedYellow', 'retrog', 'ogxRedPurple', 'ogxGreen', 'tron', 'default_cyan', 'default', 'backup', 'minimal_orange', 'Golde']\n",
830
+ "# @markdown Set `use_preset` for using default prompt, resolution, sampler, and other settings\n",
831
+ "use_presets = True # @param {type: 'boolean'}\n",
832
+ "# @markdown ### **Arguments**\n",
833
+ "use_gradio_auth = False # @param {type: 'boolean'}\n",
834
+ "accelerator = \"xformers\" # @param ['xformers', 'opt-sdp-attention', 'opt-sdp-no-mem-attention', 'opt-split-attention']\n",
835
+ "auto_select_model = False # @param {type: 'boolean'}\n",
836
+ "auto_select_vae = True # @param {type: 'boolean'}\n",
837
+ "additional_arguments = \"--lowram --theme dark --no-half-vae\" #@param {type: 'string'}\n",
838
+ "\n",
839
+ "# GRADIO AUTH\n",
840
+ "user = \"cagliostro\"\n",
841
+ "password = \"\".join(random.choices(string.ascii_letters + string.digits, k=6))\n",
842
+ "\n",
843
+ "def change_theme(filename):\n",
844
+ " themes_folder = os.path.join(repo_dir, \"extensions-builtin\", \"sd_theme_editor\", \"themes\")\n",
845
+ " themes_file = os.path.join(themes_folder, f\"{filename}.css\")\n",
846
+ "\n",
847
+ " style_config = config_utils.read_config(style_path)\n",
848
+ " style_contents = style_config.split(\"/*BREAKPOINT_CSS_CONTENT*/\")[1]\n",
849
+ "\n",
850
+ " theme_config = config_utils.read_config(themes_file)\n",
851
+ " style_data = \":host{\" + theme_config + \"}\" + \"/*BREAKPOINT_CSS_CONTENT*/\" + style_contents\n",
852
+ " config_utils.write_config(style_path, style_data)\n",
853
+ "\n",
854
+ "def is_valid(valid_dir, file_types):\n",
855
+ " return [f for f in os.listdir(valid_dir) if f.endswith(file_types)]\n",
856
+ "\n",
857
+ "def auto_select_file(valid_dir, config_key, file_types):\n",
858
+ " valid_files = is_valid(valid_dir, file_types)\n",
859
+ " if valid_files:\n",
860
+ " file_path = random.choice(valid_files)\n",
861
+ " if os.path.exists(os.path.join(valid_dir, file_path)):\n",
862
+ " config = config_utils.read_config(config_file)\n",
863
+ " config[config_key] = file_path\n",
864
+ " config_utils.write_config(config_file, config)\n",
865
+ " return file_path\n",
866
+ " else:\n",
867
+ " return None\n",
868
+ "\n",
869
+ "def ui_preset_config():\n",
870
+ " global default_upscaler, default_sampler_v2\n",
871
+ "\n",
872
+ " default_prompt = \"masterpiece, best quality,\"\n",
873
+ " default_neg_prompt = \"(worst quality, low quality:1.4)\"\n",
874
+ " default_sampler = \"DPM++ 2M Karras\"\n",
875
+ " default_steps = 20\n",
876
+ " default_width = 512\n",
877
+ " default_height = 768\n",
878
+ " default_strength = 0.55\n",
879
+ " default_cfg_scale = 7\n",
880
+ " default_upscaler = \"Latent (nearest-exact)\"\n",
881
+ "\n",
882
+ " config = {\n",
883
+ " \"Prompt/value\" : default_prompt,\n",
884
+ " \"Negative prompt/value\" : default_neg_prompt,\n",
885
+ " \"Sampling method/value\" : default_sampler,\n",
886
+ " \"Sampling steps/value\" : default_steps,\n",
887
+ " \"Width/value\" : default_width,\n",
888
+ " \"Height/value\" : default_height,\n",
889
+ " \"Denoising strength/value\" : default_strength,\n",
890
+ " \"CFG Scale/value\" : default_cfg_scale\n",
891
+ " }\n",
892
+ "\n",
893
+ " return config\n",
894
+ "\n",
895
+ "def configure_main_settings(config_file: str, lora_dir: str, use_presets: bool, ui_config_file: str):\n",
896
+ " config = config_utils.read_config(config_file)\n",
897
+ "\n",
898
+ " config[\"additional_networks_extra_lora_path\"] = lora_dir\n",
899
+ " config[\"CLIP_stop_at_last_layers\"] = 2\n",
900
+ " config[\"eta_noise_seed_delta\"] = 0\n",
901
+ " config[\"show_progress_every_n_steps\"] = 10\n",
902
+ " config[\"show_progressbar\"] = True\n",
903
+ " config[\"samples_filename_pattern\"] = \"[model_name]_[seed]\"\n",
904
+ " config[\"show_progress_type\"] = \"Approx NN\" # Full, Approx NN, TAESD, Approx cheap\n",
905
+ " config[\"live_preview_content\"] = \"Prompt\" # Combined, Prompt, Negative Prompt\n",
906
+ " config[\"hires_fix_show_sampler\"] = True\n",
907
+ " config[\"hires_fix_show_prompts\"] = True\n",
908
+ " config[\"state\"] = [\"tabs\"]\n",
909
+ " config[\"state_txt2img\"] = [\"prompt\", \"negative_prompt\", \"styles\", \"sampling\", \"sampling_steps\", \"width\", \"height\", \"batch_count\", \"batch_size\", \"hires_resize_y\", \"hires_resize_x\", \"hires_scale\", \"hires_steps\", \"hires_upscaler\", \"hires_fix\", \"tiling\", \"restore_faces\", \"cfg_scale\", \"hires_denoising_strength\"]\n",
910
+ " config[\"state_img2img\"] = [\"prompt\", \"negative_prompt\", \"styles\", \"sampling\", \"resize_mode\", \"sampling_steps\", \"tiling\", \"restore_faces\", \"width\", \"height\", \"batch_count\", \"batch_size\", \"cfg_scale\", \"denoising_strength\"]\n",
911
+ " config[\"state_extensions\"] = [\"control-net\"]\n",
912
+ "\n",
913
+ " quicksettings_values = [\"sd_model_checkpoint\", \"sd_vae\", \"CLIP_stop_at_last_layers\",\n",
914
+ " \"use_old_karras_scheduler_sigmas\", \"always_discard_next_to_last_sigma\",\n",
915
+ " \"token_merging_ratio\", \"s_min_uncond\"]\n",
916
+ "\n",
917
+ " if \"quicksettings\" in config:\n",
918
+ " config[\"quicksettings\"] = \", \".join(quicksettings_values)\n",
919
+ " elif \"quicksettings_list\" in config:\n",
920
+ " config[\"quicksettings_list\"] = quicksettings_values\n",
921
+ "\n",
922
+ " config_utils.write_config(config_file, config)\n",
923
+ "\n",
924
+ " if use_presets:\n",
925
+ " configure_ui_settings(ui_config_file)\n",
926
+ "\n",
927
+ "def configure_ui_settings(ui_config_file: str):\n",
928
+ " config = config_utils.read_config(ui_config_file)\n",
929
+ " preset_config = ui_preset_config()\n",
930
+ " for key in [\"txt2img\", \"img2img\"]:\n",
931
+ " for subkey, value in preset_config.items():\n",
932
+ " config[f\"{key}/{subkey}\"] = value\n",
933
+ "\n",
934
+ " config[\"txt2img/Upscaler/value\"] = default_upscaler\n",
935
+ " config_utils.write_config(ui_config_file, config)\n",
936
+ "\n",
937
+ "def is_dir_exist(cloned_dir, original_dir):\n",
938
+ " if os.path.exists(cloned_dir):\n",
939
+ " return cloned_dir\n",
940
+ " else:\n",
941
+ " return original_dir\n",
942
+ "\n",
943
+ "def parse_args(config):\n",
944
+ " args = \"\"\n",
945
+ " for k, v in config.items():\n",
946
+ " if k.startswith(\"_\"):\n",
947
+ " args += f'\"{v}\" '\n",
948
+ " elif isinstance(v, str):\n",
949
+ " args += f'--{k}=\"{v}\" '\n",
950
+ " elif isinstance(v, bool) and v:\n",
951
+ " args += f\"--{k} \"\n",
952
+ " elif isinstance(v, float) and not isinstance(v, bool):\n",
953
+ " args += f\"--{k}={v} \"\n",
954
+ " elif isinstance(v, int) and not isinstance(v, bool):\n",
955
+ " args += f\"--{k}={v} \"\n",
956
+ "\n",
957
+ " return args\n",
958
+ "\n",
959
+ "def main():\n",
960
+ " global auto_select_model, auto_select_vae\n",
961
+ "\n",
962
+ " repo_name, _, _ = validate_repo(repo_dir)\n",
963
+ " if \"anapnoe\" in repo_name:\n",
964
+ " change_theme(select_theme)\n",
965
+ "\n",
966
+ " valid_ckpt_dir = is_dir_exist(os.path.join(fused_dir, \"model\"), models_dir)\n",
967
+ " valid_vae_dir = is_dir_exist(os.path.join(fused_dir, \"vae\"), vaes_dir)\n",
968
+ " valid_embedding_dir = is_dir_exist(os.path.join(fused_dir, \"embedding\"), embeddings_dir)\n",
969
+ " valid_lora_dir = is_dir_exist(os.path.join(fused_dir, \"lora\"), lora_dir)\n",
970
+ " valid_hypernetwork_dir = is_dir_exist(os.path.join(fused_dir, \"hypernetwork\"), hypernetworks_dir)\n",
971
+ "\n",
972
+ " print_line(80, color=\"green\")\n",
973
+ " cprint(f\"Launching '{repo_name}'\", color=\"flat_yellow\")\n",
974
+ " print_line(80, color=\"green\")\n",
975
+ "\n",
976
+ " if not is_valid(valid_ckpt_dir, ('.ckpt', '.safetensors')):\n",
977
+ " cprint(f\"No checkpoints were found in the directory '{valid_ckpt_dir}'.\", color=\"yellow\")\n",
978
+ " url = \"https://huggingface.co/Linaqruf/stolen/resolve/main/fp16/aamAnyloraAnimeMixAnime_v10-fp16-pruned.safetensors\"\n",
979
+ " filename = \"AnyLoRA_Anime_mix.safetensors\"\n",
980
+ " aria2_download(url=url, download_dir=valid_ckpt_dir, filename=filename)\n",
981
+ " print_line(80, color=\"green\")\n",
982
+ " auto_select_model = True\n",
983
+ "\n",
984
+ " if not is_valid(valid_vae_dir, ('.vae.pt', '.vae.safetensors', '.pt', '.ckpt')):\n",
985
+ " cprint(f\"No VAEs were found in the directory '{valid_vae_dir}'.\", color=\"yellow\")\n",
986
+ " url = \"https://huggingface.co/NoCrypt/resources/resolve/main/any.vae.safetensors\"\n",
987
+ " filename = \"Anime.vae.safetensors\"\n",
988
+ " aria2_download(url=url, download_dir=valid_vae_dir, filename=filename)\n",
989
+ " print_line(80, color=\"green\")\n",
990
+ " auto_select_vae = True\n",
991
+ "\n",
992
+ " if auto_select_model:\n",
993
+ " selected_model = auto_select_file(valid_ckpt_dir, \"sd_model_checkpoint\", ('.ckpt', '.safetensors'))\n",
994
+ " cprint(f\"Selected Model: {selected_model}\", color=\"green\")\n",
995
+ "\n",
996
+ " if auto_select_vae:\n",
997
+ " selected_vae = auto_select_file(valid_vae_dir, \"sd_vae\", ('.vae.pt', '.vae.safetensors', '.pt', '.ckpt'))\n",
998
+ " cprint(f\"Selected VAE: {selected_vae}\", color=\"green\")\n",
999
+ "\n",
1000
+ " print_line(80, color=\"green\")\n",
1001
+ "\n",
1002
+ " configure_main_settings(config_file, valid_lora_dir, use_presets, ui_config_file)\n",
1003
+ "\n",
1004
+ " if use_gradio_auth:\n",
1005
+ " cprint(\"Gradio Auth (use this account to login):\", color=\"green\")\n",
1006
+ " cprint(\"[-] Username: cagliostro\", color=\"green\")\n",
1007
+ " cprint(\"[-] Password:\", password, color=\"green\")\n",
1008
+ " print_line(80, color=\"green\")\n",
1009
+ "\n",
1010
+ " config = {\n",
1011
+ " \"enable-insecure-extension-access\": True,\n",
1012
+ " \"disable-safe-unpickle\" : True,\n",
1013
+ " f\"{accelerator}\" : True,\n",
1014
+ " f\"{select_tunnel}\" : True if not select_tunnel == \"gradio\" and not ngrok_token else False,\n",
1015
+ " \"share\" : True if not ngrok_token else False,\n",
1016
+ " \"gradio-auth\" : f\"{user}:{password}\" if use_gradio_auth else None,\n",
1017
+ " \"no-hashing\" : True,\n",
1018
+ " \"disable-console-progressbars\" : True,\n",
1019
+ " \"ngrok\" : ngrok_token if ngrok_token else None,\n",
1020
+ " \"ngrok-region\" : ngrok_region if ngrok_token else None,\n",
1021
+ " \"opt-sub-quad-attention\" : True,\n",
1022
+ " \"opt-channelslast\" : True,\n",
1023
+ " \"no-download-sd-model\" : True,\n",
1024
+ " \"gradio-queue\" : True,\n",
1025
+ " \"listen\" : True,\n",
1026
+ " \"ckpt-dir\" : valid_ckpt_dir,\n",
1027
+ " \"vae-dir\" : valid_vae_dir,\n",
1028
+ " \"hypernetwork-dir\" : valid_hypernetwork_dir,\n",
1029
+ " \"embeddings-dir\" : valid_embedding_dir,\n",
1030
+ " \"lora-dir\" : valid_lora_dir,\n",
1031
+ " \"lyco-dir\" : valid_lora_dir,\n",
1032
+ " }\n",
1033
+ "\n",
1034
+ " args = parse_args(config)\n",
1035
+ " final_args = f\"python launch.py {args} {additional_arguments}\"\n",
1036
+ "\n",
1037
+ " cprint()\n",
1038
+ " os.chdir(repo_dir)\n",
1039
+ " os.environ[\"TCMALLOC_AGGRESSIVE_DECOMMIT\"] = \"t\"\n",
1040
+ " !{final_args}\n",
1041
+ "\n",
1042
+ "main()"
1043
+ ],
1044
+ "metadata": {
1045
+ "cellView": "form",
1046
+ "id": "Oyrwg8cMyDXj"
1047
+ },
1048
+ "execution_count": null,
1049
+ "outputs": []
1050
+ },
1051
+ {
1052
+ "cell_type": "code",
1053
+ "source": [
1054
+ "# @title ## **Download Generated Images**\n",
1055
+ "# @markdown Download file manually from files tab or save to Google Drive\n",
1056
+ "import os\n",
1057
+ "\n",
1058
+ "from pydrive.auth import GoogleAuth\n",
1059
+ "from pydrive.drive import GoogleDrive\n",
1060
+ "from google.colab import auth, drive\n",
1061
+ "from oauth2client.client import GoogleCredentials\n",
1062
+ "from colablib.colored_print import cprint, print_line\n",
1063
+ "\n",
1064
+ "%store -r\n",
1065
+ "\n",
1066
+ "os.chdir(output_dir)\n",
1067
+ "\n",
1068
+ "use_drive = False # @param {type:\"boolean\"}\n",
1069
+ "folder_name = \"cagliostro-colab-ui\" # @param {type: \"string\"}\n",
1070
+ "filename = \"waifu.zip\" # @param {type: \"string\"}\n",
1071
+ "save_as = filename\n",
1072
+ "\n",
1073
+ "if os.path.exists(filename):\n",
1074
+ " i = 1\n",
1075
+ " while os.path.exists(f\"waifu({i}).zip\"):\n",
1076
+ " i += 1\n",
1077
+ " filename = f\"waifu({i}).zip\"\n",
1078
+ "\n",
1079
+ "os.system('zip -r /content/outputs.zip .')\n",
1080
+ "\n",
1081
+ "if use_drive:\n",
1082
+ " auth.authenticate_user()\n",
1083
+ " gauth = GoogleAuth()\n",
1084
+ " gauth.credentials = GoogleCredentials.get_application_default()\n",
1085
+ " drive = GoogleDrive(gauth)\n",
1086
+ "\n",
1087
+ " def create_folder(folder_name):\n",
1088
+ " file_list = drive.ListFile({\n",
1089
+ " \"q\": f\"title='{folder_name}' and mimeType='application/vnd.google-apps.folder' and trashed=false\"\n",
1090
+ " }).GetList()\n",
1091
+ " if file_list:\n",
1092
+ " cprint(\"Debug: Folder exists\", color=\"green\")\n",
1093
+ " folder_id = file_list[0][\"id\"]\n",
1094
+ " else:\n",
1095
+ " cprint(\"Debug: Creating folder\", color=\"green\")\n",
1096
+ " file = drive.CreateFile({\n",
1097
+ " \"title\": folder_name,\n",
1098
+ " \"mimeType\": \"application/vnd.google-apps.folder\"\n",
1099
+ " })\n",
1100
+ " file.Upload()\n",
1101
+ " folder_id = file.attr[\"metadata\"][\"id\"]\n",
1102
+ " return folder_id\n",
1103
+ "\n",
1104
+ " def upload_file(file_name, folder_id, save_as):\n",
1105
+ " file_list = drive.ListFile({\"q\": f\"title='{save_as}' and trashed=false\"}).GetList()\n",
1106
+ " if file_list:\n",
1107
+ " cprint(\"Debug: File already exists\", color=\"green\")\n",
1108
+ " i = 1\n",
1109
+ " while True:\n",
1110
+ " new_name = f\"{os.path.splitext(save_as)[0]}({i}){os.path.splitext(save_as)[1]}\"\n",
1111
+ " file_list = drive.ListFile({\"q\": f\"title='{new_name}' and trashed=false\"}).GetList()\n",
1112
+ " if not file_list:\n",
1113
+ " save_as = new_name\n",
1114
+ " break\n",
1115
+ " i += 1\n",
1116
+ " file = drive.CreateFile({\"title\": save_as, \"parents\": [{\"id\": folder_id}]})\n",
1117
+ " file.SetContentFile(file_name)\n",
1118
+ " file.Upload()\n",
1119
+ " file.InsertPermission({\"type\": \"anyone\", \"value\": \"anyone\", \"role\": \"reader\"})\n",
1120
+ " return file.attr[\"metadata\"][\"id\"]\n",
1121
+ "\n",
1122
+ " file_id = upload_file(\"/content/outputs.zip\", create_folder(folder_name), save_as)\n",
1123
+ " cprint(f\"Your sharing link: https://drive.google.com/file/d/{file_id}/view?usp=sharing\", color=\"green\")\n"
1124
+ ],
1125
+ "metadata": {
1126
+ "cellView": "form",
1127
+ "id": "Ojb4nAieATxI"
1128
+ },
1129
+ "execution_count": null,
1130
+ "outputs": []
1131
+ },
1132
+ {
1133
+ "cell_type": "markdown",
1134
+ "source": [
1135
+ "# Extras"
1136
+ ],
1137
+ "metadata": {
1138
+ "id": "4SUHPtGLz2m4"
1139
+ }
1140
+ },
1141
+ {
1142
+ "cell_type": "code",
1143
+ "source": [
1144
+ "# @title ## **Download Generated Images V2**\n",
1145
+ "import shutil\n",
1146
+ "import os\n",
1147
+ "from IPython.utils import capture\n",
1148
+ "from huggingface_hub import login, HfApi\n",
1149
+ "from huggingface_hub.utils import validate_repo_id, HfHubHTTPError\n",
1150
+ "from colablib.colored_print import cprint, print_line\n",
1151
+ "\n",
1152
+ "# @markdown Download your output by upload it to **Huggingface** instead of Google Drive.\n",
1153
+ "# @markdown > Get **your** huggingface `WRITE` token [here](https://huggingface.co/settings/tokens)\n",
1154
+ "write_token = \"\" # @param {type:\"string\"}\n",
1155
+ "# @markdown Specify where is your repo located, it will automatically create your repo if you didn't have one.\n",
1156
+ "repo_name = \"cagliostro-colab-ui\" # @param{type:\"string\"}\n",
1157
+ "private_repo = False # @param{type:\"boolean\"}\n",
1158
+ "# @markdown This will be compressed to zip and uploaded to datasets repo\n",
1159
+ "project_name = \"waifu\" # @param {type :\"string\"}\n",
1160
+ "\n",
1161
+ "repo_name = repo_name.replace(\" \", \"-\")\n",
1162
+ "project_name = project_name.replace(\" \", \"_\")\n",
1163
+ "\n",
1164
+ "if not project_name:\n",
1165
+ " project_name = \"waifu\"\n",
1166
+ "\n",
1167
+ "dataset_zip = f\"{project_name}.zip\"\n",
1168
+ "output_path = os.path.join(root_dir, dataset_zip)\n",
1169
+ "commit_message = f\"Feat: Upload {dataset_zip} with Cagliostro Colab UI\"\n",
1170
+ "\n",
1171
+ "def create_or_validate_repo(api, datasets_repo):\n",
1172
+ " try:\n",
1173
+ " validate_repo_id(datasets_repo)\n",
1174
+ " api.create_repo(\n",
1175
+ " repo_id=datasets_repo, repo_type=\"dataset\", private=private_repo\n",
1176
+ " )\n",
1177
+ " cprint(f\"Repo created, located at \"\n",
1178
+ " f\"https://huggingface.co/datasets/{datasets_repo}\", color=\"green\")\n",
1179
+ "\n",
1180
+ " except HfHubHTTPError:\n",
1181
+ " cprint(f\"Repo exists, skipping...\", color=\"green\")\n",
1182
+ "\n",
1183
+ "def compress_to_zip(output_path):\n",
1184
+ " os.chdir(output_dir)\n",
1185
+ " cprint(f\"Compressing to ZIP...\", color=\"green\")\n",
1186
+ " with capture.capture_output() as cap:\n",
1187
+ " !zip -rv {output_path} .\n",
1188
+ "\n",
1189
+ "def upload_and_cleanup(api, output_path, datasets_repo):\n",
1190
+ " cprint(f\"Uploading generated images... Please wait...\", color=\"green\")\n",
1191
+ "\n",
1192
+ " api.upload_file(\n",
1193
+ " path_or_fileobj=output_path,\n",
1194
+ " path_in_repo=dataset_zip,\n",
1195
+ " repo_id=datasets_repo,\n",
1196
+ " repo_type=\"dataset\",\n",
1197
+ " commit_message=commit_message,\n",
1198
+ " )\n",
1199
+ "\n",
1200
+ " cprint(f\"Upload success, download directly at \"\n",
1201
+ " f\"https://huggingface.co/datasets/{datasets_repo}/resolve/main/{dataset_zip}\", color=\"green\")\n",
1202
+ "\n",
1203
+ " os.remove(output_path)\n",
1204
+ "\n",
1205
+ "def main():\n",
1206
+ " with capture.capture_output() as cap:\n",
1207
+ " login(write_token, add_to_git_credential=True)\n",
1208
+ " output = cap.stdout.strip()\n",
1209
+ "\n",
1210
+ " if \"Token is valid.\" in output:\n",
1211
+ " cprint(f\"Login Successful.\", color=\"green\")\n",
1212
+ "\n",
1213
+ " api = HfApi()\n",
1214
+ " user = api.whoami(write_token)\n",
1215
+ " datasets_repo = f\"{user['name']}/{repo_name.strip()}\"\n",
1216
+ "\n",
1217
+ " if repo_name:\n",
1218
+ " create_or_validate_repo(api, datasets_repo)\n",
1219
+ " compress_to_zip(output_path)\n",
1220
+ " upload_and_cleanup(api, output_path, datasets_repo)\n",
1221
+ "\n",
1222
+ "main()\n"
1223
+ ],
1224
+ "metadata": {
1225
+ "cellView": "form",
1226
+ "id": "EGXqJLXwnJQB"
1227
+ },
1228
+ "execution_count": null,
1229
+ "outputs": []
1230
+ }
1231
+ ],
1232
+ "metadata": {
1233
+ "accelerator": "GPU",
1234
+ "colab": {
1235
+ "private_outputs": true,
1236
+ "provenance": [],
1237
+ "include_colab_link": true
1238
+ },
1239
+ "gpuClass": "standard",
1240
+ "kernelspec": {
1241
+ "display_name": "Python 3",
1242
+ "name": "python3"
1243
+ },
1244
+ "language_info": {
1245
+ "name": "python"
1246
+ }
1247
+ },
1248
+ "nbformat": 4,
1249
+ "nbformat_minor": 0
1250
+ }
content/sd-notebook-collection/config/download_list.txt ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ // Instructions:
2
+ // 1. Start with a key name preceded by a '#'. Key names should match those in the `custom_dirs` dictionary in your script.
3
+ // 2. Key names are not case-sensitive.
4
+ // 3. List your URLs for download under the key. Each URL should be on a new line.
5
+ // 4. To download multiple files under the same key, just continue listing the URLs.
6
+ // 5. To specify a filename for the download, format it as "URL | filename". Example: https://civitai.com/api/download/models/84623 | filename1.pth
7
+ // 6. Lines starting with "//" are considered comments and will be ignored by the script.
8
+ //
9
+ // Available hashtag:
10
+ // `#model`, `#vae`, `#embedding`, `#lora`, `#hypernetwork`, `#extensions`, `#upscaler`
11
+ //
12
+ // Example:
13
+ //
14
+ // #model
15
+ // https://civitai.com/api/download/models/82446
16
+ // https://civitai.com/api/download/models/76907 | model.safetensors
17
+ //
18
+ // Start Here...
content/sd-notebook-collection/script/vae.py ADDED
@@ -0,0 +1,1166 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #from pathlib import Path
2
+
3
+ import os
4
+
5
+ import numpy as np
6
+ import math
7
+ import random
8
+ import tqdm
9
+ import gc
10
+ import argparse
11
+ import time
12
+
13
+ #import matplotlib.pyplot as plt
14
+
15
+ import torch
16
+ #import torch.nn.functional as F
17
+ from torchvision import transforms
18
+ from torch.utils.data import Dataset
19
+ from torchvision import transforms
20
+ from safetensors.torch import save_file
21
+
22
+ from diffusers.optimization import get_constant_schedule
23
+
24
+ #from diffusers import AutoencoderKL
25
+ import lpips
26
+
27
+ from transformers.optimization import Adafactor, AdafactorSchedule
28
+ #from sd_vae import AutoencoderKL
29
+
30
+ from PIL import Image
31
+ import cv2
32
+ #vaeロード面倒だからsd-scriptからそのまま借りる
33
+ from library.model_util import load_vae
34
+ import yaml
35
+
36
+ try:
37
+ import torch.utils.tensorboard.writer as tensorboardX
38
+ except:
39
+ pass
40
+ import datetime
41
+
42
+ def load_vae_from_sd_checkpoint(model_path, dtype=torch.float16, use_xformers: bool = True):
43
+ print(f"load dtype: {dtype}")
44
+ vae = load_vae(model_path, dtype=dtype)
45
+
46
+ #xformersのフラグをオンにする
47
+ for i in range(len(vae.encoder.mid_block.attentions)):
48
+ vae.encoder.mid_block.attentions[i].set_use_memory_efficient_attention_xformers(use_xformers)
49
+ for i in range(len(vae.decoder.mid_block.attentions)):
50
+ vae.decoder.mid_block.attentions[i].set_use_memory_efficient_attention_xformers(use_xformers)
51
+
52
+ print(f"xformers: {vae.decoder.mid_block.attentions[0]._use_memory_efficient_attention_xformers}")
53
+ return vae
54
+
55
+ #from src.utils import ImageFolder, VAEHandler, denormalize, preprocess_images
56
+ class IMAGE_DIC():
57
+ def __init__(self, file_path: str) -> None:
58
+ self.size : tuple[int, int] = None
59
+ self.org_size : tuple[int, int] = None
60
+ self.path : str = file_path
61
+ self.data = None
62
+ self.latent = None
63
+ self.ratio : float = None
64
+ self.area_size : int = None
65
+ self.ratio_error : float = None
66
+ self.scale : float = None
67
+
68
+ class VAE_TRAIN_DATASET(Dataset):
69
+ def __init__(self, data_path, batch_size=1, gradient_accumulation_steps=1, shuffle=True,
70
+ resolution=(256,256), min_resolution=(128,128), max_size=512, min_size=128, divisible=64, bucket_serch_step=1, make_clipping=0., make_clip_num=1) -> None:
71
+ '''
72
+ data_path = ディレクトリパス　か　画像のパスリスト
73
+ '''
74
+ super().__init__()
75
+ #ファイルパスとか
76
+ self.dataset_dir_path = data_path
77
+ self.file_paths = None
78
+ self.data_list = {}
79
+ self.dots = ["png", "jpg"]
80
+ #学習に関するデータセット設定
81
+ self.batch_size = batch_size
82
+ self.gradient_accumulation_steps = gradient_accumulation_steps
83
+ self._data_len : int = 0
84
+ self._data_len_add : int = 0
85
+ self.shuffle : bool = shuffle
86
+ self.make_clipping = make_clipping
87
+ if self.make_clipping >= 1.: self.make_clipping=0.
88
+ self.make_clip_num = make_clip_num
89
+ #画像サイズに関する変数
90
+ self.resolution = resolution
91
+ self.min_resolution = min_resolution
92
+ self.max_area_size = (self.resolution[0]//divisible) * (self.resolution[1]//divisible)
93
+ self.min_area_size = (self.min_resolution[0]//divisible) * (self.min_resolution[1]//divisible)
94
+ self.max_size = max_size
95
+ self.min_size = min_size
96
+ self.divisible = divisible
97
+ self.bucket_serch_step = bucket_serch_step
98
+ #bucket関連の変数（画像読み込み時）
99
+ self.buckets_lists = []
100
+ self.area_size_list = []
101
+ self.bucket_area_size_resos_list = []
102
+ self.bucket_area_size_ratio_list = []
103
+ self.add_index = []
104
+ # bucket関連の変数（学習時に使うやつ）
105
+ #index -> vsize key
106
+ self.index_to_enable_bucket_list : list[tuple[int,int]] = []
107
+ # vsize key -> dataset key
108
+ self.enable_bucket_vsize_to_resos_lens : dict[tuple[int,int], int] = {}
109
+ self.enable_bucket_vsize_to_keys_list : dict[tuple[int,int], list] = {}
110
+ # vsize key -> keys_list index (-> dataset)
111
+ self.enable_bucket_vsize_to_keys_indexs : dict[tuple[int,int], list] = {}
112
+ #画像データセット
113
+ self.data_list : dict[str, IMAGE_DIC] = {}
114
+ #
115
+ self.image_transforms = transforms.Compose([transforms.ToTensor(), transforms.Normalize([0.5], [0.5])])
116
+
117
+ #ファイルのパスをとりあえず取得する
118
+ self.get_files_path()
119
+ #先にデータリストの入れ物を作る
120
+ self.make_datalist()
121
+ #バケットを作成する
122
+ self.make_buckets()
123
+ #画像を読み込む
124
+ self.load_images()
125
+ #有効なバケット情報をまとめる
126
+ self.create_enable_buckets()
127
+
128
+ #テスト出力
129
+ #for k, v in self.data_list.items():
130
+ # print(f"{k}: {v.org_size} -> {v.size} er({v.ratio_error})")
131
+ #
132
+ #読み込むデータリスト作成���連
133
+ def get_files_path(self):
134
+ file_paths = []
135
+ if type(self.dataset_dir_path)==str:
136
+ for root, dirs, files in os.walk(self.dataset_dir_path, followlinks=True):
137
+ # ファイルを格納
138
+ for file in files:
139
+ for dot in self.dots:
140
+ if dot in os.path.splitext(file)[-1]:
141
+ file_paths.append(os.path.join(root, file))
142
+ self.file_paths = file_paths
143
+ else:
144
+ self.file_paths = self.dataset_dir_path
145
+ self.dataset_dir_path = None
146
+ def make_datalist(self):
147
+ for file_path in self.file_paths:
148
+ key = os.path.splitext(file_path)[0]
149
+ if self.dataset_dir_path==None:
150
+ key = os.path.basename(key)
151
+ else:
152
+ key = key[len(self.dataset_dir_path)+1:]
153
+ img_data = IMAGE_DIC(file_path)
154
+ self.data_list[key] = img_data
155
+ # バケット作成
156
+ def make_buckets(self):
157
+ _max_area = self.max_area_size
158
+ while _max_area >= self.min_area_size:
159
+ resos = set()
160
+ size = int(math.sqrt(_max_area)) * self.divisible
161
+ resos.add((size, size))
162
+ size = self.min_size
163
+ while size <= self.max_size:
164
+ width = size
165
+ height = min(self.max_size, (_max_area // (size // self.divisible))*self.divisible)
166
+ if height >= self.min_size:
167
+ resos.add((width, height))
168
+ resos.add((height, width))
169
+ size += self.divisible
170
+ resos = list(resos)
171
+ resos.sort()
172
+
173
+ self.area_size_list.append(_max_area)
174
+ self.bucket_area_size_resos_list.append(resos)
175
+ ratio = [w/h for w, h in resos]
176
+ self.bucket_area_size_ratio_list.append(np.array(ratio))
177
+ _max_area -= 1
178
+
179
+ self.area_size_list = np.array(self.area_size_list)
180
+ # 画像読み込み処理
181
+ def load_image(self, img_path):
182
+ image = Image.open(img_path)
183
+ if not image.mode == "RGB":
184
+ image=image.convert("RGB")
185
+ return np.array(image, np.uint8)
186
+ def load_images(self):
187
+ print("画像読み込み中...")
188
+ append_list = {}
189
+ for key, img_data in tqdm.tqdm(self.data_list.items()):
190
+ image_org = self.load_image(img_data.path)
191
+ if not type(image_org)==np.ndarray:continue
192
+ #画像サイズを決める
193
+ img_data.org_size = [image_org.shape[1], image_org.shape[0]]
194
+ img_data.size, img_data.ratio, img_data.ratio_error = self.sel_bucket_size(img_data.org_size[0], img_data.org_size[1])
195
+ #画像加工処理
196
+ image, img_data.scale = self.resize_image(image_org, img_data.size, img_data.ratio)
197
+ image = self.image_transforms(image)
198
+ img_data.data = image
199
+ ############################################
200
+ #クリッピング画像追加処理
201
+ if self.make_clipping>0.:
202
+ if img_data.scale<=self.make_clipping:
203
+ #脳筋処理方法(処理工数に無駄はあるけど確実　極端なクリッピングを防ぐためスケーリング処理が必要だった)
204
+ new_img = image_org
205
+ #クリッピングサイズが極端すぎないか確認する
206
+ if img_data.scale < 0.33:
207
+ new_scale = img_data.scale * 2 #おおよそ半解像度位をクリッピングするのが良さそうな気がする
208
+ resize = []
209
+ for i in range(2):
210
+ resize.append(int(new_img.shape[1-i] * new_scale + .5))
211
+ new_img = cv2.resize(new_img, resize, interpolation=cv2.INTER_AREA)
212
+ #print(f"resize: {img_data.scale} -> {new_scale}")
213
+ #追加リスト作成処理
214
+ for _ in range(self.make_clip_num):
215
+ i = 0
216
+ while True:
217
+ new_key = f"{key}+{i}"
218
+ if (not new_key in self.data_list) and (not new_key in append_list): break
219
+ i+=1
220
+ append_list[new_key] = IMAGE_DIC(new_key)
221
+ append_list[new_key].data = new_img
222
+ pos = []
223
+ for i in range(2):
224
+ pos.append(random.randint(0, append_list[new_key].data.shape[i]-img_data.size[1-i]-1))
225
+ append_list[new_key].data = append_list[new_key].data[pos[0]:pos[0]+img_data.size[1],pos[1]:pos[1]+img_data.size[0]]
226
+ #ここから下は情報を再取得していく
227
+ append_list[new_key].org_size = [append_list[new_key].data.shape[1], append_list[new_key].data.shape[0]]
228
+ append_list[new_key].size, append_list[new_key].ratio, append_list[new_key].ratio_error = self.sel_bucket_size(append_list[new_key].org_size[0], append_list[new_key].org_size[1])
229
+ append_list[new_key].data, append_list[new_key].scale = self.resize_image(append_list[new_key].data, append_list[new_key].size, append_list[new_key].ratio)
230
+ append_list[new_key].data = self.image_transforms(append_list[new_key].data)
231
+ for k, v in append_list.items():
232
+ self.data_list[k] = v
233
+
234
+ def resize_image(self, image, _resized_size, ratio):
235
+ img_size = image.shape[0:2]
236
+ resized_size = [_resized_size[1], _resized_size[0]] #img_sizeからそのまま呼び出すと並びが　h,w なので処理をすっきりするために入れ替えておく
237
+ #5/1=2.5 10/3=0.3 1/5=0.2 3/10=0.3
238
+ re_retio = _resized_size[0] / _resized_size[1]
239
+ if re_retio >= ratio:
240
+ base_size = 1
241
+ else:
242
+ base_size = 0
243
+ #cv2の方がPILより高品質な拡大縮小ができる
244
+ resize_scale = resized_size[base_size] / img_size[base_size]
245
+ resize = []
246
+ for i in range(2):
247
+ resize.append(int(img_size[1-i] * resize_scale + .5))
248
+ if img_size[base_size] > resized_size[base_size]:
249
+ image = cv2.resize(image, resize, interpolation=cv2.INTER_AREA) # INTER_AREAでやりたいのでcv2でリサイズ
250
+ elif img_size[base_size] < resized_size[base_size]:
251
+ image = cv2.resize(image, resize, interpolation=cv2.INTER_CUBIC)#遅い代わりに高品質らしい
252
+ #
253
+ img_size = image.shape[0:2]
254
+ p = [0, 0]
255
+ for i in range(len(img_size)):
256
+ if img_size[i] > resized_size[i]:
257
+ trim_size = img_size[i] - resized_size[i]
258
+ p[i] = trim_size // 2
259
+ image = image[p[0]:p[0] + resized_size[0], p[1]:p[1] + resized_size[1]]
260
+ assert image.shape[0] == resized_size[0] and image.shape[1] == resized_size[1], f"resized error {image.shape} to {resized_size}"
261
+ return image, resize_scale
262
+
263
+ def sel_bucket_size(self, img_width, img_height):
264
+ area_size = (img_width//self.divisible) * (img_height//self.divisible)
265
+ img_ratio = img_width / img_height
266
+ area_size_er = self.area_size_list - area_size
267
+ area_size_id = np.abs(area_size_er).argmin()
268
+ area_size_id_list = [area_size_id]
269
+ #探査範囲のsize id listを作成する
270
+ for i in range(self.bucket_serch_step):
271
+ if area_size_id -i <= 0:
272
+ area_size_id_list.append(area_size_id+i+1)
273
+ elif area_size_id + i + 1 >= len(self.bucket_area_size_resos_list):
274
+ area_size_id_list.append(area_size_id-i-1)
275
+ else:
276
+ area_size_id_list.append(area_size_id-i-1)
277
+ area_size_id_list.append(area_size_id+i+1)
278
+ min_error = 10000
279
+ min_area_size_id = area_size_id
280
+ for area_size_id in area_size_id_list:
281
+ area_ratio = self.bucket_area_size_ratio_list[area_size_id]
282
+ ratio_errors = area_ratio - img_ratio
283
+ ratio_error = np.abs(ratio_errors).min()
284
+ if min_error > ratio_error:
285
+ min_error = ratio_error
286
+ min_area_size_id = area_size_id
287
+ if min_error==0.:
288
+ break
289
+ area_size_id = min_area_size_id
290
+ #ここから普通のバケットサイズ取得
291
+ area_resos = self.bucket_area_size_resos_list[area_size_id]
292
+ area_ratio = self.bucket_area_size_ratio_list[area_size_id]
293
+ ratio_errors = area_ratio - img_ratio
294
+ bucket_id = np.abs(ratio_errors).argmin()
295
+ bucket_size = area_resos[bucket_id]
296
+
297
+ return bucket_size, img_ratio, np.abs(ratio_errors).min()
298
+
299
+ def make_latent(self, vae):
300
+ print("latent作成中...")
301
+ for img_data in tqdm.tqdm(self.data_list.values()):
302
+ image = img_data.data
303
+ image_tensor = image.unsqueeze(0).to(device=vae.device, dtype=vae.dtype)
304
+ try:
305
+ with torch.no_grad():
306
+ img_data.latent = vae.encode(image_tensor).latent_dist.mode().squeeze(0).to("cpu")
307
+ except:
308
+ print(f"error: {img_data.path} {image_tensor.size()}")
309
+ #学習時に使うリストに関する関数
310
+ def create_enable_buckets(self):
311
+ for k, v in self.data_list.items():
312
+ if not v.size in self.enable_bucket_vsize_to_keys_list:
313
+ self.enable_bucket_vsize_to_keys_list[v.size] = [k]
314
+ else:
315
+ self.enable_bucket_vsize_to_keys_list[v.size].append(k)
316
+ for k, v in self.enable_bucket_vsize_to_keys_list.items():
317
+ count = len(v)
318
+ self.enable_bucket_vsize_to_resos_lens[k] = count
319
+ self.reset_indexs_list(k) #enable bucketsを作成する時に初期化しておく
320
+ self._data_len += (count//self.batch_size) + (count%self.batch_size>0)
321
+ for _ in range((count//self.batch_size) + (count%self.batch_size>0)):
322
+ self.index_to_enable_bucket_list.append(k)
323
+ #gradient accumulation stepsのための計算
324
+ self._data_len_add = self.gradient_accumulation_steps - (self._data_len % self.gradient_accumulation_steps)
325
+
326
+ def reset_indexs_list(self, vsize):
327
+ now_list = [i for i in range(self.enable_bucket_vsize_to_resos_lens[vsize])]
328
+ self.enable_bucket_vsize_to_keys_indexs[vsize]= now_list
329
+
330
+ self.shuffle_indexs_list(vsize) #リセット時についでにシャッフルしたほうが楽
331
+ def shuffle_indexs_list(self, vsize):
332
+ if self.shuffle:
333
+ now_list = self.enable_bucket_vsize_to_keys_indexs[vsize]
334
+ random.shuffle(now_list)
335
+ self.enable_bucket_vsize_to_keys_indexs[vsize] = now_list
336
+ else: pass
337
+ def reset_add_indexs_list(self):
338
+ self.add_index = random.sample(range(self._data_len), self._data_len)
339
+ #各バケット内の要素を取り出すためのkeyをindexとして扱うための関数
340
+ def get_key(self, vsize):
341
+ keys = self.enable_bucket_vsize_to_keys_list[vsize]
342
+ key_index = self.enable_bucket_vsize_to_keys_indexs[vsize].pop(0) #popで取り出すので取り出した要素は消える
343
+ #listの中を使い切ったら初期化して補充
344
+ if len(self.enable_bucket_vsize_to_keys_indexs[vsize])==0:
345
+ self.reset_indexs_list(vsize)
346
+ return keys[key_index]
347
+
348
+ def get_index_to_bucket_key(self, index):
349
+ return self.index_to_enable_bucket_list[index]
350
+
351
+ def __len__(self):
352
+ return self._data_len + self._data_len_add
353
+ def __getitem__(self, index):
354
+ #まずはindexから使用するバケットサイズを決定する
355
+ if index >= self._data_len:
356
+ if len(self.add_index) == 0:
357
+ self.reset_add_indexs_list()
358
+ index = self.add_index.pop(0)
359
+ vsize = self.get_index_to_bucket_key(index)
360
+
361
+ teacher = []
362
+ latents = []
363
+ for i in range(self.batch_size):
364
+ #取り出したvsize_keyから画像データにアクセスするためのkeyを取り出す
365
+ key = self.get_key(vsize)
366
+ img_data = self.data_list[key]
367
+ latents.append(img_data.latent)
368
+ teacher.append(img_data.data)
369
+ latents = torch.stack(latents)
370
+ teacher = torch.stack(teacher)
371
+
372
+ data = {"latents": latents, "teacher": teacher}
373
+
374
+ #動作確認用出力
375
+ #print("=====================")
376
+ #print(f"[{index}]: {key}")
377
+ #print(self.data_list[key].data.size())
378
+ #print(f"min: {self.data_list[key].data.min()} max: {self.data_list[key].data.max()}")
379
+
380
+ return data
381
+
382
+ def save_vae(theta, output_file, save_type="safetensors"):
383
+ vae_conversion_map = [
384
+ ("conv_shortcut", "nin_shortcut"),
385
+ ("conv_norm_out", "norm_out"),
386
+ ("mid_block.attentions.0.", "mid.attn_1."),
387
+ ]
388
+
389
+ for i in range(4):
390
+ # down_blocks have two resnets
391
+ for j in range(2):
392
+ hf_down_prefix = f"encoder.down_blocks.{i}.resnets.{j}."
393
+ sd_down_prefix = f"encoder.down.{i}.block.{j}."
394
+ vae_conversion_map.append((hf_down_prefix, sd_down_prefix))
395
+
396
+ if i < 3:
397
+ hf_downsample_prefix = f"down_blocks.{i}.downsamplers.0."
398
+ sd_downsample_prefix = f"down.{i}.downsample."
399
+ vae_conversion_map.append((hf_downsample_prefix, sd_downsample_prefix))
400
+
401
+ hf_upsample_prefix = f"up_blocks.{i}.upsamplers.0."
402
+ sd_upsample_prefix= f"up.{3-i}.upsample."
403
+ vae_conversion_map.append((hf_upsample_prefix, sd_upsample_prefix))
404
+
405
+ # up_blocks have three resnets
406
+ # also, up blocks in hf are numbered in reverse from sd
407
+ for j in range(3):
408
+ hf_up_prefix = f"decoder.up_blocks.{i}.resnets.{j}."
409
+ sd_up_prefix = f"decoder.up.{3-i}.block.{j}."
410
+ vae_conversion_map.append((hf_up_prefix, sd_up_prefix))
411
+
412
+ # this part accounts for mid blocks in both the encoder and the decoder
413
+ for i in range(2):
414
+ hf_mid_res_prefix = f"mid_block.resnets.{i}."
415
+ sd_mid_res_prefix = f"mid.block_{i+1}."
416
+ vae_conversion_map.append((hf_mid_res_prefix, sd_mid_res_prefix))
417
+
418
+ vae_conversion_map_attn = [
419
+ # (HF Diffusers, stable-diffusion)
420
+ ("group_norm.", "norm."),
421
+ ("query.", "q."),
422
+ ("key.", "k."),
423
+ ("value.", "v."),
424
+ ("proj_attn.", "proj_out."),
425
+ ]
426
+
427
+ mapping = {k: k for k in theta.keys()}
428
+ for k, v in mapping.items():
429
+ for hf_part, sd_part in vae_conversion_map:
430
+ v = v.replace(hf_part, sd_part)
431
+ mapping[k] = v
432
+ for k, v in mapping.items():
433
+ if "mid_block.attentions" in k:
434
+ for hf_part, sd_part in vae_conversion_map_attn:
435
+ v = v.replace(hf_part, sd_part)
436
+ mapping[k] = v
437
+ new_state_dict = {v: theta[k] for k, v in mapping.items()}
438
+
439
+ weights_to_convert = ["q", "k", "v", "proj_out"]
440
+ for k, v in new_state_dict.items():
441
+ for weight_name in weights_to_convert:
442
+ if f"mid.attn_1.{weight_name}.weight" in k:
443
+ new_state_dict[k] = new_state_dict[k].unsqueeze(dim=2).unsqueeze(dim=2)
444
+
445
+ if save_type=="pt":
446
+ torch.save({
447
+ "state_dict": new_state_dict
448
+ }, output_file)
449
+ else:
450
+ save_file(new_state_dict, output_file)
451
+
452
+ print(f"saved ... {output_file}")
453
+ return
454
+
455
+ def collate_fn(examples):
456
+ return examples[0]
457
+
458
+ #compVisが採用してるDiscriminatorそのまま
459
+ import functools
460
+ class NLayerDiscriminator(torch.nn.Module):
461
+ """Defines a PatchGAN discriminator as in Pix2Pix
462
+ --> see https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/models/networks.py
463
+ """
464
+ def __init__(self, ndf=64, n_layers=3):
465
+ super(NLayerDiscriminator, self).__init__()
466
+ input_nc = 3
467
+ norm_layer = torch.nn.BatchNorm2d
468
+ use_bias = False
469
+ kw = 4
470
+ padw = 1
471
+ sequence = [torch.nn.Conv2d(input_nc, ndf, kernel_size=kw, stride=2, padding=padw), torch.nn.LeakyReLU(0.2, True)]
472
+ nf_mult = 1
473
+ nf_mult_prev = 1
474
+ for n in range(1, n_layers): # gradually increase the number of filters
475
+ nf_mult_prev = nf_mult
476
+ nf_mult = min(2 ** n, 8)
477
+ sequence += [
478
+ torch.nn.Conv2d(ndf * nf_mult_prev, ndf * nf_mult, kernel_size=kw, stride=2, padding=padw, bias=use_bias),
479
+ norm_layer(ndf * nf_mult),
480
+ torch.nn.LeakyReLU(0.2, True)
481
+ ]
482
+
483
+ nf_mult_prev = nf_mult
484
+ nf_mult = min(2 ** n_layers, 8)
485
+ sequence += [
486
+ torch.nn.Conv2d(ndf * nf_mult_prev, ndf * nf_mult, kernel_size=kw, stride=1, padding=padw, bias=use_bias),
487
+ norm_layer(ndf * nf_mult),
488
+ torch.nn.LeakyReLU(0.2, True)
489
+ ]
490
+
491
+ sequence += [
492
+ torch.nn.Conv2d(ndf * nf_mult, 1, kernel_size=kw, stride=1, padding=padw)] # output 1 channel prediction map
493
+ self.main = torch.nn.Sequential(*sequence)
494
+ def forward(self, input):
495
+ """Standard forward."""
496
+ return self.main(input)
497
+ def loss_func_discriminator(self, inputs, teacher):
498
+ dis_fake = self(inputs)
499
+ dis_true = self(teacher)
500
+ loss_fake = torch.mean(torch.nn.functional.relu(1. + dis_fake)) * 0.5
501
+ loss_real = torch.mean(torch.nn.functional.relu(1. - dis_true)) * 0.5
502
+ return loss_fake, loss_real
503
+ def loss_func(self, inputs, teacher):
504
+ dis_fake = self(inputs)
505
+ loss_real = -torch.mean(dis_fake)
506
+ return loss_real
507
+
508
+ class Discriminator_Block(torch.nn.Module):
509
+ def __init__(self, in_dims, hid_dims=16, down_scale=2) -> None:
510
+ super(Discriminator_Block, self).__init__()
511
+ self.conv_in = torch.nn.Conv2d(in_dims, hid_dims, 3, 1, 1)
512
+ self.down1 = torch.nn.Conv2d(hid_dims, hid_dims, 3, down_scale)
513
+ self.down2 = torch.nn.Conv2d(in_dims, hid_dims, 3, down_scale)
514
+ self.activate1 = lambda x: x * torch.sigmoid(x)
515
+ self.activate2 = lambda x: x * torch.sigmoid(x)
516
+ def forward(self, inputs):
517
+ h = self.conv_in(inputs)
518
+ h = self.activate1(h)
519
+ h = self.down1(h)
520
+ h = self.activate2(h)
521
+ res = self.down2(inputs)
522
+ return (h + res) / math.sqrt(2)
523
+
524
+ class Discriminator(torch.nn.Module):
525
+ def __init__(self, hid_dims=16, down_scale=2) -> None:
526
+ super(Discriminator, self).__init__()
527
+ self.conv1 = Discriminator_Block(3, hid_dims*2, down_scale)
528
+ self.conv2 = Discriminator_Block(hid_dims*2, hid_dims, 1)
529
+ self.conv_out = torch.nn.Conv2d(hid_dims, hid_dims, 1)
530
+ def forward(self, inputs):
531
+ h = self.conv1(inputs)
532
+ h = self.conv2(h)
533
+ h = self.conv_out(h)
534
+ return h
535
+ def loss_func_discriminator(self, inputs, teacher):
536
+ dis_fake = self(inputs)
537
+ dis_real = self(teacher)
538
+ #loss_fake = F.softplus(dis_fake).mean()
539
+ #loss_real = F.softplus(-dis_real).mean()
540
+ loss_fake = torch.nn.functional.mse_loss(dis_fake, torch.zeros_like(dis_fake))
541
+ loss_real = torch.nn.functional.mse_loss(dis_real, torch.ones_like(dis_real))
542
+ return loss_fake, loss_real
543
+ def loss_func(self, inputs, teacher):
544
+ dis_fake = self(inputs)
545
+ loss_real = torch.nn.functional.mse_loss(dis_fake, torch.ones_like(dis_fake))
546
+ return loss_real
547
+
548
+ ### cahiner時代にシンプルさを追及して作ったDiscriminator 精度は知らん
549
+ ### conv2を積層にしたら広い範囲を見れる構造 ただその場合padding計算してskip connect必要
550
+ class SimpleDiscriminator(torch.nn.Module):
551
+ def __init__(self, hid_dims=16, alpha=1.75) -> None:
552
+ super(SimpleDiscriminator, self).__init__()
553
+ self.alpha = alpha
554
+ self.conv1 = torch.nn.Conv2d(3, hid_dims, 3, bias=False)
555
+ self.activation1 = lambda x: x * torch.sigmoid(x)
556
+ self.conv2 = torch.nn.Conv2d(hid_dims, hid_dims*2, 3, 1, 0, 2, bias=False)
557
+ self.activation2 = lambda x: x * torch.sigmoid(x)
558
+ self.conv3 = torch.nn.Conv2d(hid_dims*2, hid_dims, 3, bias=False)
559
+ def forward(self, inputs):
560
+ h = self.conv1(inputs)
561
+ h = self.activation1(h)
562
+ h = self.conv2(h)
563
+ h = self.activation2(h)
564
+ h = self.conv3(h)
565
+ return h
566
+ def loss_func_discriminator(self, inputs, teacher):
567
+ dis_fake = self(inputs*self.alpha-teacher)
568
+ dis_real = self(teacher*self.alpha-teacher)
569
+ loss_fake = torch.nn.functional.softplus(dis_fake).mean()
570
+ loss_real = torch.nn.functional.softplus(-dis_real).mean()
571
+ #loss_real = F.softplus(-dis_real).mean()
572
+ return loss_fake, loss_real
573
+ def loss_func(self, inputs, teacher):
574
+ dis_fake = self(inputs*self.alpha-teacher)
575
+ loss_fake = torch.nn.functional.softplus(-dis_fake).mean()
576
+ #loss_real = F.softplus(-dis_real).mean()
577
+ return loss_fake
578
+
579
+ def torch_fix_seed(seed=42):
580
+ # Python random
581
+ random.seed(seed)
582
+ # Numpy
583
+ np.random.seed(seed)
584
+ # Pytorch
585
+ torch.manual_seed(seed)
586
+ torch.cuda.manual_seed(seed)
587
+ torch.backends.cudnn.deterministic = True
588
+ torch.use_deterministic_algorithms = True
589
+ class print_command():
590
+ DEL = "\033[2K\033[G"
591
+ def args_str_to_list(args_str):
592
+ tmp = tuple([int(r) for r in args_str.split(',')])
593
+ if len(tmp) == 1:
594
+ tmp = (tmp[0], tmp[0])
595
+ return tmp
596
+ def gen_img(vae, latents, output_name):
597
+ with torch.no_grad():
598
+ test_img = vae.decode(latents).sample
599
+
600
+ test_img = (test_img / 2 + 0.5).clamp(0, 1)
601
+ test_img = test_img.cpu().permute(0, 2, 3, 1).float().numpy()
602
+ test_img = (test_img * 255).round().astype("uint8")
603
+ test_img = [Image.fromarray(im) for im in test_img]
604
+ test_img[0].save(f"{output_name}.png")
605
+
606
+ def train(args):
607
+ #必要な場合最初にseed値は固定してしまう
608
+ if args.seed is not None:
609
+ torch_fix_seed(args.seed)
610
+ data_dir = args.dataset
611
+ model_file = args.model
612
+ batch_size = args.batch
613
+ shuffle_flag = args.not_shuffle
614
+ resolution = args_str_to_list(args.resolution)
615
+ min_resolution = args_str_to_list(args.min_resolution)
616
+ max_size = args.max_size
617
+ min_size = args.min_size
618
+ divisible = args.divisible
619
+ bucket_serch_step = args.bucket_serch_step
620
+ val_rate = args.val_rate
621
+ make_clipping = args.make_clipping
622
+ make_clip_num = args.make_clip_num
623
+
624
+ max_grad_norm = args.grad_clip
625
+ max_grad_norm_flag = (max_grad_norm>0.)
626
+ max_data_loader_n_workers = args.max_data_loader_n_workers
627
+ persistent_data_loader_workers = True
628
+ n_workers = min(max_data_loader_n_workers, os.cpu_count() - 1) # cpu_count-1 ただし最大で指定された数まで
629
+
630
+ output_file = f"{args.output_file}.{args.save_type}"
631
+ output_dir = os.path.split(output_file)[0]
632
+ if not output_dir=="":
633
+ if not os.path.isdir(output_dir):
634
+ print(f"{output_dir} ディレクトリを作成しました")
635
+ os.makedirs(output_dir)
636
+ save_every_n_epoch = args.save_every_n_epoch
637
+ pre_epoch = args.pre_epoch
638
+ epoch = args.epoch
639
+ gradient_accumulation_steps = args.gradient_accumulation_steps
640
+ optimizer_type = args.optimizer
641
+ optimizer_arg = {"lr": args.lr}
642
+ warmup_init = args.warmup_init
643
+ decoder_param_split = args.decoder_param_split
644
+
645
+ latent_dropout = args.latent_dropout
646
+ max_latent_dropout_epoch = args.max_latent_dropout_epoch
647
+ if max_latent_dropout_epoch<=0:
648
+ max_latent_dropout_epoch = epoch
649
+
650
+ latent_noise_rate = args.latent_noise_rate
651
+
652
+ rec_alpha_def = args.decode_alpha
653
+ rec_alpha = args.decode_alpha
654
+ rec_alpha_max = args.decode_alpha_max_epoch
655
+ rec_alpha_linear = args.decode_alpha_linear
656
+ rec_alpha_blight = 1.0 * (args.decode_alpha_blight)
657
+ rec_alpha_step = 0.
658
+
659
+ rec_rgb_alpha_def = []
660
+ rec_rgb_alpha_step = []
661
+ rec_rgb_alpha_max_epoch = args.decode_rgb_alpha_max_epoch
662
+ rec_rgb_alpha_linear = args.decode_rgb_alpha_linear
663
+ rec_rgb_alpha_blight = 1.0 * (args.decode_rgb_alpha_blight)
664
+ rec_rgb_name = ["r", "g", "b"]
665
+
666
+ l2_cost = args.l2_cost
667
+ lpips_cost = args.lpips_cost
668
+ disc_cost = args.discriminator_cost
669
+
670
+ discriminator_type = args.discriminator_type
671
+ discriminator_dim = args.discriminator_dim
672
+
673
+ lpips_name = args.lpips_name
674
+
675
+ enable_tensorboard = args.enable_tensorboard
676
+ _t = datetime.datetime.today().strftime('%Y%m%d_%H%M')
677
+ tensorboard_output = f"runs/vae_{_t}"
678
+ del _t
679
+
680
+ dtype = torch.float
681
+ device = torch.device("cuda" if torch.cuda.is_available() else "cpu") # "mps"を考量してない
682
+ use_xformers_flag = args.not_use_xformers
683
+
684
+ #######################################################################################
685
+ if val_rate>0:
686
+ pass
687
+ #######################################################################################
688
+ dataset = VAE_TRAIN_DATASET(data_dir, batch_size, gradient_accumulation_steps, shuffle_flag, resolution, min_resolution, max_size, min_size, divisible, bucket_serch_step, make_clipping, make_clip_num)
689
+
690
+ if rec_alpha_max>0:
691
+ rec_alpha_step = (rec_alpha-1.) / (rec_alpha_max * (len(dataset)/gradient_accumulation_steps))
692
+ if rec_alpha_linear and rec_alpha_step==0:
693
+ print(f"decode_alpha もしくは decode_alpha_max_epoch の指定に誤りがあったため decode_alpha_linear をオフにします")
694
+ rec_alpha_linear = False
695
+ if rec_rgb_alpha_linear and rec_rgb_alpha_max_epoch==0:
696
+ print(f"decode_rgb_alpha もしくは decode_rgb_alpha_max_epoch の指定に誤りがあったため decode_rgb_alpha_linear をオフにします")
697
+ rec_rgb_alpha_linear = False
698
+ if rec_rgb_alpha_max_epoch>0:
699
+ if args.decode_rgb_alpha is not None or args.decode_rgb_alpha!="":
700
+ rec_rgb_alpha_def = [1.,1.,1.]
701
+ rec_rgb_alpha_step = [0.,0.,0.]
702
+ tmp = args.decode_rgb_alpha.replace(" ","").split(",")
703
+ for s in tmp:
704
+ color, alpha = s.split("=")
705
+ if color == "r" or color == "R":
706
+ target_id = 0
707
+ elif color=="g" or color == "G":
708
+ target_id = 1
709
+ elif color=="b" or color == "B":
710
+ target_id = 2
711
+ rec_rgb_alpha_def[target_id] = float(alpha)
712
+ rec_rgb_alpha_step[target_id] = (float(alpha)-1.)/(rec_rgb_alpha_max_epoch*(len(dataset)/gradient_accumulation_steps))
713
+ rec_rgb_alpha = rec_rgb_alpha_def[:]
714
+ vae = load_vae_from_sd_checkpoint(model_file, dtype, use_xformers_flag)
715
+ vae.to(device)
716
+ #vae.requires_grad_(False)
717
+ vae.eval()
718
+ dataset.make_latent(vae)
719
+ #########################################################################
720
+ #いったん情報をまとめて表示
721
+ print("\n[学習設定]\n==================================")
722
+ print(f"学習セット： {data_dir}")
723
+ print(f"ベースモデルファイル: {model_file}")
724
+ print(f"epoch: {epoch} / pre epoch: {pre_epoch}")
725
+ print(f"batch size: {batch_size}")
726
+ print(f"gradient_accumulation_steps: {gradient_accumulation_steps}")
727
+ print(f"1epochあたりのstep数: {len(dataset)}")
728
+ if gradient_accumulation_steps>1:
729
+ print(f"実質的な1epochあたりのstep数(重みの更新回数): {int(len(dataset)//gradient_accumulation_steps)}")
730
+ print(f"全体のstep数(重みの更新回数): {int(len(dataset)//gradient_accumulation_steps) * epoch}")
731
+ print(f"max_data_loader_n_workers :{n_workers}")
732
+ if args.seed is not None:
733
+ print(f"seed: {args.seed}")
734
+ print("---------------------------------")
735
+ print(f"画像サイズ reso: {resolution} min_reso: {min_resolution} 画像サイズの単位({divisible})")
736
+ print(f"max_size: {max_size} min_size: {min_size}")
737
+ print(f"optimizer: {optimizer_type} lr: {args.lr} warmup_init: {warmup_init}")
738
+ print(f"decoderをmidとupで分割するかどうか: {decoder_param_split}")
739
+ print(f"xformersの利用: {use_xformers_flag}")
740
+ print(f"LPIPSのモデル名: {lpips_name}")
741
+ print(f"discriminator_type: {discriminator_type}")
742
+ print(f"discriminatorのネットワークサイズ: {discriminator_dim}")
743
+ print("---------------------------------")
744
+ print(f"latentのdropout率: {latent_dropout}")
745
+ print(f"latentのdropoutを何epochまで続けるか: {max_latent_dropout_epoch}")
746
+ print(f"latentに加えるノイズ強度: {latent_noise_rate}")
747
+ print(f"decoderの出力補正: {rec_alpha}")
748
+ if rec_alpha_step>0:
749
+ print(f"decoderの出力補正を続けるepoch数: {rec_alpha_max} linear : {rec_alpha_linear}")
750
+ print("----------------------------------")
751
+ print(f"l2_lossの適用率: {l2_cost}")
752
+ print(f"lpips_lossの適用率: {lpips_cost}")
753
+ print(f"discriminatorの適用率: {disc_cost}")
754
+ if max_grad_norm_flag:
755
+ print(f"max_grad_norm: {max_grad_norm}")
756
+ print("----------------------------------")
757
+ print(f"出力ファイル名: {output_file}")
758
+ if save_every_n_epoch > 0:
759
+ print(f"epoch毎の保存: {save_every_n_epoch}")
760
+ if enable_tensorboard:
761
+ print(f"tensor board log dirctory: {tensorboard_output}")
762
+ log_board = tensorboardX.SummaryWriter(tensorboard_output)
763
+ print("==================================\n")
764
+ #########################################################################
765
+ if args.debug:
766
+ #テスト出力
767
+ data = dataset.__getitem__(0)
768
+ test_latents = data["latents"].to(device=vae.device, dtype=vae.dtype)
769
+ teacher = data["teacher"]
770
+ test_img = (teacher / 2 + 0.5).clamp(0, 1)
771
+ test_img = test_img.cpu().permute(0, 2, 3, 1).float().numpy()
772
+ test_img = (test_img * 255).round().astype("uint8")
773
+ test_img = [Image.fromarray(im) for im in test_img]
774
+ test_img[0].save("test_0.png")
775
+
776
+ gen_img(vae, test_latents, "test_1")
777
+ vae.to("cpu")
778
+ torch.cuda.empty_cache()#VRAM確保のために余計な情報は消去
779
+
780
+ '''
781
+ with torch.no_grad():
782
+ test_img = vae.decode(test_latents).sample
783
+
784
+ test_img = (test_img / 2 + 0.5).clamp(0, 1)
785
+ test_img = test_img.cpu().permute(0, 2, 3, 1).float().numpy()
786
+ test_img = (test_img * 255).round().astype("uint8")
787
+ test_img = [Image.fromarray(im) for im in test_img]
788
+ test_img[0].save("test_1.png")
789
+ '''
790
+ #########################################################################
791
+ #latentを作成したのでとりあえずいったんVRAMを掃除
792
+ vae.to("cpu")
793
+ torch.cuda.empty_cache()#VRAM確保のために余計な情報は消去
794
+ gc.collect()
795
+ ######################## 学習に必要なものを作っていく
796
+ #lpips
797
+ print("///// ここのwarnigは無視していい //////")
798
+ if lpips_name=="alex":
799
+ loss_fn_lpips_loss = lpips.LPIPS(net='alex') # best forward scores
800
+ else:
801
+ loss_fn_lpips_loss = lpips.LPIPS(net='vgg') # closer to "traditional" perceptual loss, when used for optimization
802
+ loss_fn_lpips_loss.to(device=device)
803
+ print("/////////////////////////////////////")
804
+
805
+ #discriminator
806
+ if discriminator_type=="simple":
807
+ disc = SimpleDiscriminator(discriminator_dim, 1.75)
808
+ elif discriminator_type=="compvis":
809
+ disc = NLayerDiscriminator(discriminator_dim, 3)
810
+ else:
811
+ disc = Discriminator(discriminator_dim, 1)
812
+ disc_opt = Adafactor(disc.parameters())
813
+ disc_opt_sche = AdafactorSchedule(disc_opt)
814
+ disc.to(device=device)
815
+ disc.train()
816
+
817
+ if optimizer_type == "8bitAdam":
818
+ try:
819
+ import bitsandbytes as bnb
820
+ optimizer_class = bnb.optim.AdamW8bit
821
+ except:
822
+ print("8bit-Adamが取得できませんでした optimizer に Adafactor を設定します")
823
+ optimizer_type = "Adafactor"
824
+
825
+ if optimizer_type == "Adafactor":
826
+ optimizer_class = Adafactor
827
+ optimizer_arg["lr"] = None
828
+ if warmup_init:
829
+ optimizer_arg["warmup_init"] = True
830
+ else:
831
+ optimizer_class = torch.optim.AdamW
832
+
833
+ parameters = []
834
+
835
+ vae.post_quant_conv.to(device=device)
836
+ #vae.post_quant_conv.requires_grad_(True)
837
+ vae.post_quant_conv.train()
838
+ parameters.append({"params": vae.post_quant_conv.parameters()})
839
+
840
+ vae.decoder.to(device=device)
841
+ #vae.decoder.requires_grad_(True)
842
+ vae.decoder.train()
843
+ if decoder_param_split:
844
+ lr_names = ["post_quant_conv", "mid_block", "up_blocks"]
845
+ parameters.append({"params": [p for n, p in vae.decoder.named_parameters() if (n.startswith("mid_block") or n.startswith("conv_in")) and p.requires_grad]})
846
+ parameters.append({"params": [p for n, p in vae.decoder.named_parameters() if (not (n.startswith("mid_block") or n.startswith("conv_in"))) and p.requires_grad]})
847
+ else:
848
+ lr_names = ["post_quant_conv", "decoder", "decoder"]
849
+ parameters.append({"params": vae.decoder.parameters()})
850
+
851
+ optimizer = optimizer_class(parameters, **optimizer_arg)
852
+ if optimizer_type == "Adafactor":
853
+ initial_lr = args.lr if not args.lr==None else 0
854
+ optimizer_scheduler = AdafactorSchedule(optimizer, initial_lr=initial_lr)
855
+ else:
856
+ optimizer_scheduler = get_constant_schedule(optimizer)
857
+
858
+ train_dataloader = torch.utils.data.DataLoader(dataset, batch_size=1, shuffle=True, collate_fn=collate_fn, num_workers=n_workers, persistent_workers=persistent_data_loader_workers)
859
+
860
+ datalen = len(dataset)
861
+ pre_total_step = pre_epoch * datalen
862
+ total_step = epoch * datalen
863
+ #先にdiscriminatorをある程度学習させておく
864
+ print("\n======================\n[ Pre Train / Discriminatorだけ先に少し学習をしておく ]\n======================\n")
865
+ with tqdm.tqdm(range(pre_total_step), desc="steps") as pbar:
866
+ global_step = 0
867
+ for i in range(pre_epoch):
868
+ fake_losse_sum = 0.
869
+ real_losse_sum = 0.
870
+ pbar.set_description("[Epoch %d] " % i)
871
+ for step, data in enumerate(train_dataloader):
872
+ latents = data["latents"].to(device=device, dtype=dtype)
873
+ teacher = data["teacher"].to(device=device, dtype=dtype)
874
+ #####################################################
875
+ #discriminator
876
+ with torch.no_grad():
877
+ reconstructions = vae.decode(latents).sample
878
+ loss_fake, loss_real = disc.loss_func_discriminator(reconstructions, teacher)
879
+ #dis_fake = disc(reconstructions)
880
+ #dis_real = disc(teacher)
881
+ #loss_fake = F.softplus(dis_fake).mean()
882
+ #loss_real = F.softplus(-dis_real).mean()
883
+ #loss_fake = torch.nn.functional.mse_loss(dis_fake, torch.zeros_like(dis_fake))
884
+ #loss_real = torch.nn.functional.mse_loss(dis_real, torch.ones_like(dis_real))
885
+ loss_disc = loss_fake + loss_real
886
+ disc_opt.zero_grad()
887
+ loss_disc.backward()
888
+ now_loss = loss_fake.detach().clone().item()
889
+ fake_losse_sum += now_loss
890
+ if enable_tensorboard:
891
+ log_board.add_scalar("discriminator/fake_loss", now_loss, global_step)
892
+ now_loss = loss_real.detach().clone().item()
893
+ real_losse_sum += now_loss
894
+ if enable_tensorboard:
895
+ log_board.add_scalar("discriminator/real_loss", now_loss, global_step)
896
+ pbar.set_postfix({"loss_fake": f"{fake_losse_sum/(step+1):.4f}", "loss_real": f"{real_losse_sum/(step+1):.4f}"})
897
+ del loss_disc, loss_fake, loss_real
898
+ torch.cuda.empty_cache()
899
+ disc_opt.step()
900
+ disc_opt_sche.step()
901
+ #del dis_fake, dis_real
902
+ torch.cuda.empty_cache()
903
+ #####################################################
904
+ global_step += 1
905
+ pbar.update(1)
906
+ print(f"{print_command.DEL}[Epoch {i}] loss_fake: {fake_losse_sum/(step+1):.4f} loss_real: {real_losse_sum/(step+1):.4f}")
907
+ print("\n======================\n[ Train / 本番の学習　Decoderの学習 ]\n======================\n")
908
+ with tqdm.tqdm(range(total_step), desc="steps") as pbar:
909
+ global_step = 0
910
+ train_step = 0
911
+ if latent_dropout<=0.:
912
+ dropout_func = lambda x: x
913
+ else:
914
+ print("\nset dropout func")
915
+ dropout_func = torch.nn.Dropout(latent_dropout).to(device=vae.device, dtype=vae.dtype)
916
+ for i in range(epoch):
917
+ epst = time.time()
918
+ loss_sum = 0.
919
+ fake_losse_sum = 0.
920
+ real_losse_sum = 0.
921
+ loss_vae_sum = 0.
922
+ loss_fake_sum = 0.
923
+ loss_lpips_sum = 0.
924
+ logs = {}
925
+ if max_latent_dropout_epoch <= i:
926
+ dropout_func = lambda x: x
927
+ pbar.set_description("[Epoch %d] " % i)
928
+ for step, data in enumerate(train_dataloader):
929
+ if rec_alpha_linear:
930
+ rec_alpha = rec_alpha_def - (train_step * rec_alpha_step)
931
+ if rec_alpha_max-i <= 0: rec_alpha = 1.
932
+ ####################################################
933
+ if len(rec_rgb_alpha) == 3:
934
+ for _i in range(3):
935
+ if rec_rgb_alpha_linear:
936
+ rec_rgb_alpha[_i] = rec_rgb_alpha_def[_i] - (train_step * rec_rgb_alpha_step[_i])
937
+ if rec_rgb_alpha_max_epoch-i <= 0:
938
+ rec_rgb_alpha[_i] = 1.
939
+ ####################################################
940
+ latents = data["latents"].to(device=device, dtype=dtype)
941
+ if latent_noise_rate>0:
942
+ latent_noise = torch.normal(mean=0, std=1, size=latents.size()) * latent_noise_rate
943
+ latents = latents + latent_noise.to(device=device, dtype=dtype)
944
+ latents = dropout_func(latents)
945
+ teacher = data["teacher"].to(device=device, dtype=dtype)
946
+ #####################################################
947
+ #discriminator
948
+ with torch.no_grad():
949
+ reconstructions = vae.decode(latents).sample
950
+ disc.requires_grad_(True)
951
+ loss_fake, loss_real = disc.loss_func_discriminator(reconstructions, teacher)
952
+ #dis_fake = disc(reconstructions)
953
+ #dis_real = disc(teacher)
954
+ #loss_fake = F.softplus(dis_fake).mean()
955
+ #loss_real = F.softplus(-dis_real).mean()
956
+ #loss_fake = torch.nn.functional.mse_loss(dis_fake, torch.zeros_like(dis_fake))
957
+ #loss_real = torch.nn.functional.mse_loss(dis_real, torch.ones_like(dis_real))
958
+ loss_disc = loss_fake + loss_real
959
+ disc_opt.zero_grad()
960
+ loss_disc.backward()
961
+ now_loss = loss_fake.detach().clone().item()
962
+ fake_losse_sum += now_loss
963
+ now_loss = loss_real.detach().clone().item()
964
+ real_losse_sum += now_loss
965
+ del loss_disc, loss_fake, loss_real
966
+ torch.cuda.empty_cache()
967
+ disc_opt.step()
968
+ disc_opt_sche.step()
969
+ #del dis_fake, dis_real
970
+ torch.cuda.empty_cache()
971
+ #####################################################
972
+ #learning
973
+ z = vae.post_quant_conv(latents)
974
+ reconstructions = ((vae.decoder(z) + rec_alpha_blight) * rec_alpha) - rec_alpha_blight
975
+
976
+ if len(rec_rgb_alpha) == 3:
977
+ for _i in range(3):
978
+ reconstructions[:, _i] = ((reconstructions[:, _i] + rec_rgb_alpha_blight) * rec_rgb_alpha[_i]) - rec_rgb_alpha_blight
979
+
980
+ loss_vae = torch.nn.functional.mse_loss(reconstructions, teacher, reduction="none").mean()
981
+ #loss = torch.mean(torch.square(reconstructions - teacher))
982
+
983
+ #discriminator loss
984
+ disc.requires_grad_(False)
985
+ loss_fake = disc.loss_func(reconstructions, teacher)
986
+ #dis_fake = disc(reconstructions)
987
+ #loss_fake = F.softplus(-dis_fake).mean()
988
+ #loss_fake = torch.nn.functional.mse_loss(dis_fake, torch.ones_like(dis_fake)).mean()
989
+
990
+ #lpips loss
991
+ loss_fn_lpips_loss.requires_grad_(False)
992
+ loss_lpips = loss_fn_lpips_loss(teacher, reconstructions).mean()
993
+
994
+ loss = (loss_vae * l2_cost) + (loss_fake * disc_cost) + (loss_lpips * lpips_cost)
995
+
996
+ now_loss = loss.detach().clone().item()
997
+ loss_sum += now_loss
998
+
999
+ loss /= gradient_accumulation_steps
1000
+
1001
+ loss.backward()
1002
+
1003
+ now_loss = loss_vae.detach().clone().item()
1004
+ loss_vae_sum += now_loss
1005
+ if enable_tensorboard:
1006
+ log_board.add_scalar("decoder/MSE_loss", now_loss, global_step)
1007
+ now_loss = loss_fake.detach().clone().item()
1008
+ loss_fake_sum += now_loss
1009
+ if enable_tensorboard:
1010
+ log_board.add_scalar("decoder/Discriminator_loss", now_loss, global_step)
1011
+ now_loss = loss_lpips.detach().clone().item()
1012
+ loss_lpips_sum += now_loss
1013
+ if enable_tensorboard:
1014
+ log_board.add_scalar("decoder/LPIPS_loss", now_loss, global_step)
1015
+
1016
+ logs["loss"]= f"{loss_sum / (step+1):.4f}"
1017
+ pbar.set_postfix(logs)
1018
+
1019
+ del loss, loss_fake, loss_vae, loss_lpips#, dis_fake
1020
+ torch.cuda.empty_cache()
1021
+ if (step+1)%gradient_accumulation_steps==0:
1022
+ if max_grad_norm_flag:
1023
+ torch.nn.utils.clip_grad_norm_(vae.decoder.parameters(), max_grad_norm)
1024
+ torch.nn.utils.clip_grad_norm_(vae.post_quant_conv.parameters(), max_grad_norm)
1025
+ optimizer.step()
1026
+ optimizer_scheduler.step()
1027
+ optimizer.zero_grad()
1028
+ torch.cuda.empty_cache()
1029
+ if enable_tensorboard:
1030
+ log_board.add_scalar("decoder_alpha/decode_alpha", rec_alpha, global_step)
1031
+ if len(rec_rgb_alpha) == 3:
1032
+ for _i in range(3):
1033
+ log_board.add_scalar(f"decoder_alpha/decode_{rec_rgb_name[_i]}_alpha", rec_rgb_alpha[_i], global_step)
1034
+ for lr_num, now_lr in enumerate(optimizer_scheduler.get_lr()):
1035
+ log_board.add_scalar(f"decoder_lr/lr_{lr_names[lr_num]}", now_lr, global_step)
1036
+ train_step += 1
1037
+ #####################################################
1038
+ global_step += 1
1039
+ pbar.update(1)
1040
+ epoch_time = time.time() - epst
1041
+ now_epoch_time = epoch_time
1042
+ epoch_time_log = ""
1043
+ for t in range(2):
1044
+ et = now_epoch_time // (60**(2-t))
1045
+ epoch_time_log = f"{epoch_time_log}:{int(et):02d}"
1046
+ now_epoch_time = now_epoch_time % (60**(2-t))
1047
+ epoch_time_log = f"{epoch_time_log}:{now_epoch_time:02.02f} ({datalen/epoch_time:02.03f} step/time)"
1048
+ print(f"{print_command.DEL}[Epoch {i}] estime: {epoch_time_log} (loss: {loss_sum/(step+1):.6f})")
1049
+ print(f"discriminator: loss_fake: {fake_losse_sum/(step+1):.4f} loss_real: {real_losse_sum/(step+1):.4f}")
1050
+ print(f"VAE decoder: MSEloss: {loss_vae_sum/(step+1):.4f} loss_dis: {loss_fake_sum/(step+1):.4f} loss_lpips: {loss_lpips_sum/(step+1):.4f}")
1051
+ if save_every_n_epoch>0 and (i+1)<epoch:
1052
+ if (i+1)%save_every_n_epoch==0:
1053
+ save_vae(vae.state_dict(), f"{args.output_file}_epoch{i:03d}.{args.save_type}", args.save_type)
1054
+
1055
+ if args.debug:
1056
+ gen_img(vae, test_latents, "test_2")
1057
+ '''
1058
+ with torch.no_grad():
1059
+ test_img = vae.decode(test_latents).sample
1060
+
1061
+ test_img = (test_img / 2 + 0.5).clamp(0, 1)
1062
+ test_img = test_img.cpu().permute(0, 2, 3, 1).float().numpy()
1063
+ vae.to("cpu")
1064
+ torch.cuda.empty_cache()#VRAM確保のために余計な情報は消去
1065
+ test_img = (test_img * 255).round().astype("uint8")
1066
+ test_img = [Image.fromarray(im) for im in test_img]
1067
+ test_img[0].save("test_2.png")
1068
+ '''
1069
+ vae.to("cpu")
1070
+ torch.cuda.empty_cache()#VRAM確保のために余計な情報は消去
1071
+ save_vae(vae.state_dict(), output_file, args.save_type)
1072
+
1073
+
1074
+
1075
+ if __name__ == '__main__':
1076
+ parser = argparse.ArgumentParser()
1077
+ parser.add_argument("--config", type=str, default=None, help="yamlフ��イルから設定を読み込む場合に使う 拡張子不要 学習終了後に設定項目は <output_file>_日付_時分.yamlに保存される")
1078
+ parser.add_argument("--dataset", type=str, default=None, help="学習に使うデータセットのディレクトリ　階層は下の方まで探索する　指定しなければ起動時にダイアログで設定できる")
1079
+ parser.add_argument("--model", type=str, default=None, help="学習に使うベースとなるvaeファイル 指定しなければ起動時いダイアログで設定できる")
1080
+
1081
+ parser.add_argument("--epoch", type=int, default=10, help="epoch数")
1082
+ parser.add_argument("--pre_epoch", type=int, default=1, help="Decoderの学習前にあらかじめDiscriminatorの学習を回しておく epoch数")
1083
+ parser.add_argument("--batch", type=int, default=1, help="batchサイズ")
1084
+ parser.add_argument("--gradient_accumulation_steps", type=int, default=1, help="バッチを大きくする代わりの仕組み VRAM節約のための仕組み 実質的なバッチサイズ = batch * これ")
1085
+ parser.add_argument("--not_shuffle", action="store_false", help="データセットの順番を維持したまま学習したい時用")
1086
+ parser.add_argument("--val_rate", type=float, default=0., help="[WIP]データセットを評価用と学習用に分割する時に使う・大規模学習じゃないならいらない気もする")
1087
+ parser.add_argument("--seed", type=int, default=None, help="学習の乱数固定")
1088
+
1089
+ parser.add_argument("--resolution", type=str, default="256,256", help="画像の最大サイズ")
1090
+ parser.add_argument("--min_resolution", type=str, default="128,128", help="画像の最小サイズ")
1091
+ parser.add_argument("--max_size", type=int, default=512, help="画像の最大幅 長辺の最大サイズ")
1092
+ parser.add_argument("--min_size", type=int, default=128, help="画像の最小幅 短辺の最小サイズ")
1093
+ parser.add_argument("--divisible", type=int, default=64, help="画像サイズの単位 8の倍数 VAEのみ学習するので8でも問題ないけど実際のところどうなのかはわからない")
1094
+ parser.add_argument("--bucket_serch_step", type=int, default=1, help="画像のアス比に近くなるサイズを探索する範囲　値を大きくするほどアス比は最適化されるが拡大縮小幅も大きくなる")
1095
+ parser.add_argument("--make_clipping", type=float, default=0., help="指定した割合よりも縮小する場合ランダムにクリッピングした教師データも追加する 0もしくは1以上で無効化")
1096
+ parser.add_argument("--make_clip_num", type=int, default=1, help="make_clippingで水増しするデータ数")
1097
+
1098
+ parser.add_argument("--optimizer", type=str, default="Adafactor", help="[AdamW|8bitAdam|Adafactor] のどれか選べる")
1099
+ parser.add_argument("--lr", type=float, default=None, help="Adafactorの場合 initial_lrとして設定される")
1100
+ parser.add_argument("--warmup_init", action="store_true", help="Adafactorのwarmup_initを有効にするかどうか")
1101
+ parser.add_argument("--grad_clip", type=float, default=1., help="学習しすぎるのを防ぐための値　小さいほうが防ぐけどつまり学習の進みを遅くする仕組みでもある 0以下指定で無効化")
1102
+ parser.add_argument("--decoder_param_split", action="store_true", help="デコーダーのパラメータをmidとupで分割してLr調整をするか Adafactor限定")
1103
+
1104
+ parser.add_argument("--decode_alpha", type=float, default=1., help="decoderの出力に掛ける補正値")
1105
+ parser.add_argument("--decode_alpha_max_epoch", type=int, default=0, help="decoder alphaを適用するepoch数 0だと無効化 ")
1106
+ parser.add_argument("--decode_alpha_linear", action="store_true", help="decode_alpha_max_epochの指定を使った時 直線的にdecode_alphaの値を1.0に変化させていく")
1107
+ parser.add_argument("--decode_alpha_blight", action="store_true", help="alpha値の計算を0を起点としたものにする 通常の計算式の場合正規化された値なので強さというよりコントラストに近い")
1108
+ parser.add_argument("--decode_rgb_alpha", type=str, default=None, help="RGB毎にdecoderの出力にalpha値を乗算する r＝0.5,b＝1.0 等のように指定する（＝は半角　説明文が全角なのはhelpが半角＝だとエラーになるから）")
1109
+ parser.add_argument("--decode_rgb_alpha_max_epoch", type=int, default=0, help="RGB毎にdecoderno出力にalphaを適用する最大epoch数 0だと無効化 RGB毎に設定はできないので注意")
1110
+ parser.add_argument("--decode_rgb_alpha_linear", action="store_true", help="RGB毎に適用するalpha値をmax_epochで指定したepochまで直線的にalphaの値を1.0にしていく")
1111
+ parser.add_argument("--decode_rgb_alpha_blight", action="store_true", help="RGB値に対するalpha値の計算を0を起点としたものにする 通常の計算式の場合正規化された値なので強さというよりコントラストに近い")
1112
+ parser.add_argument("--l2_cost", type=float, default=1., help="l2 lossの補正値 教師画像と見比べた時に実際の違いに対しての注目度みたいなもの")
1113
+ parser.add_argument("--lpips_cost", type=float, default=0.5, help="lpips loss の補正値 LPIPSに教師画像と生成画像を見比べさせた時の評価値に対する注目度みたいなもの")
1114
+ parser.add_argument("--discriminator_cost", type=float, default=0.5, help="discriminator loss の補正値 Discriminatorに生成画像を見せた時に本物と騙されてる程度に対する注目度みたいなもの")
1115
+
1116
+ parser.add_argument("--discriminator_type", type=str, default="def", help="discriminatorのタイプ defで通常 simpleで非常にシンプルなdiscriminatorを使う[def|simple|compvis]")
1117
+ parser.add_argument("--discriminator_dim", type=int, default=64, help="discriminatorのネットワークサイズ 大きければいいってわけでもない気がするけど適切なサイズいまいちわからない")
1118
+ parser.add_argument("--not_use_xformers", action="store_false", help="xformersを使いたくない時用")
1119
+ parser.add_argument("--latent_dropout", type=float, default=0., help="decoder入力前のlatentにdropoutを適用する SDのVAEのようなモデルで使う仕組みじゃない気がするけど")
1120
+ parser.add_argument("--max_latent_dropout_epoch", type=int, default=0, help="latent dropout を適用するepoch数 0なら最後まで適用")
1121
+ parser.add_argument("--latent_noise_rate", type=float, default=0., help="latentにノイズを加える混ぜ具合 1.0なら100パーセント 0なら0パーセント(無効化)")
1122
+
1123
+ parser.add_argument("--lpips_name", type=str, default="alex", help="画像評価モデルのLPIPSに使うモデル名 [alex|vgg]")
1124
+ parser.add_argument("--max_data_loader_n_workers", type=int, default=1, help="DataLoaderの数 VAE学習自体は軽いから適当に増やしていいかもしれないCPUに合わせて上限を超えて設定しても勝手に調整する")
1125
+
1126
+ parser.add_argument("--output_file", type=str, default="vae_output\\testvae", help="出力ファイル名　ディレクトリから指定できる ディレクトリがなかったら作る　拡張子は自動で補完するからいらない")
1127
+ parser.add_argument("--save_type", type=str, default="safetensors", help="保存形式[pt|safetensors]")
1128
+ parser.add_argument("--save_every_n_epoch", type=int, default=0, help="指定したepoch毎に途中経過を保存する 0なら無効")
1129
+
1130
+ parser.add_argument("--debug", action="store_true", help="現状では単純に縮小した元画像　学習前のDecoder出力 学習後のDecoder出力 出すだけ")
1131
+ parser.add_argument("--enable_tensorboard", action="store_true", help="学習の経過情報に関する記録をtensorboardを使って出力する")
1132
+ args = parser.parse_args()
1133
+
1134
+ #yamlをコンフィグとして読み込む場合 読み込み前でargs.configを使ってるから読み込み時のyamlファイルは何の影響もない
1135
+ if (args.config is not None) and (not args.config==""):
1136
+ if os.path.splitext(args.config)[-1] == ".yaml":
1137
+ args.config = os.path.splitext(args.config)[0]
1138
+ config_name = args.config
1139
+ config_path = f"{args.config}.yaml"
1140
+ _output_file = ""
1141
+ if os.path.exists(config_path):
1142
+ if not args.output_file == "vae_output\\testvae":
1143
+ _output_file = args.output_file
1144
+ print(f"{config_path} から設定を読み込み中...")
1145
+ margs, rest = parser.parse_known_args()
1146
+ with open(config_path, mode="r") as f:
1147
+ configs = yaml.unsafe_load(f)
1148
+ args_names = argparse.Namespace(**configs)
1149
+ args = parser.parse_args(args=rest, namespace=args_names)
1150
+ args.config = config_name
1151
+ if not _output_file == "":
1152
+ args.output_file = _output_file
1153
+ if args.lr is not None:
1154
+ args.lr = float(args.lr)
1155
+ else:
1156
+ print(f"{config_path} が見つかりませんでした")
1157
+
1158
+ train(args)
1159
+
1160
+ #学習が終わったら現在のargsを保存する
1161
+ _t = datetime.datetime.today().strftime('%Y%m%d_%H%M')
1162
+ config_name = f"vae_config_{os.path.basename(args.output_file)}_{_t}.yaml"
1163
+ print(f"{config_name} に設定を書き出し中...")
1164
+ with open(config_name, mode="w") as f:
1165
+ yaml.dump(args.__dict__, f, indent=4)
1166
+ print("done!")
content/sd-notebook-collection/sd-colab-toolkit.ipynb ADDED
@@ -0,0 +1,920 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "nbformat": 4,
3
+ "nbformat_minor": 0,
4
+ "metadata": {
5
+ "colab": {
6
+ "provenance": [],
7
+ "include_colab_link": true
8
+ },
9
+ "kernelspec": {
10
+ "name": "python3",
11
+ "display_name": "Python 3"
12
+ },
13
+ "language_info": {
14
+ "name": "python"
15
+ },
16
+ "accelerator": "GPU",
17
+ "gpuClass": "standard"
18
+ },
19
+ "cells": [
20
+ {
21
+ "cell_type": "markdown",
22
+ "metadata": {
23
+ "id": "view-in-github",
24
+ "colab_type": "text"
25
+ },
26
+ "source": [
27
+ "<a href=\"https://colab.research.google.com/github/Linaqruf/sd-notebook-collection/blob/main/sd-colab-toolkit.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
28
+ ]
29
+ },
30
+ {
31
+ "cell_type": "markdown",
32
+ "source": [
33
+ "# I. Installation"
34
+ ],
35
+ "metadata": {
36
+ "id": "akFjukqz5PbB"
37
+ }
38
+ },
39
+ {
40
+ "cell_type": "code",
41
+ "source": [
42
+ "#@title ## 1.1 Install Colab Toolkit\n",
43
+ "#@markdown This will install required Python packages\n",
44
+ "import os\n",
45
+ "import zipfile\n",
46
+ "import shutil\n",
47
+ "import subprocess\n",
48
+ "import requests\n",
49
+ "import re\n",
50
+ "import time\n",
51
+ "from tqdm import tqdm\n",
52
+ "from urllib.parse import urlparse, unquote\n",
53
+ "\n",
54
+ "root_dir = \"/content\"\n",
55
+ "repo_dir = os.path.join(root_dir, \"sd-scripts\")\n",
56
+ "models_dir = os.path.join(root_dir, \"models\")\n",
57
+ "vaes_dir = os.path.join(root_dir, \"vae\")\n",
58
+ "lora_dir = os.path.join(root_dir, \"network_weight\")\n",
59
+ "deps_dir = os.path.join(root_dir, \"deps\")\n",
60
+ "drive_dir = os.path.join(root_dir, \"drive/MyDrive\")\n",
61
+ "tools_dir = os.path.join(repo_dir, \"tools\")\n",
62
+ "\n",
63
+ "repo_url = \"https://github.com/kohya-ss/sd-scripts\"\n",
64
+ "\n",
65
+ "def cprint(*args, color=\"default\", reset=True, tqdm_desc=False):\n",
66
+ " color_codes = {\n",
67
+ " \"default\" : \"\\033[0m\",\n",
68
+ " \"green\" : \"\\033[0;32m\",\n",
69
+ " \"red\" : \"\\033[0;31m\",\n",
70
+ " \"bold_green\" : \"\\033[1;32m\",\n",
71
+ " \"bold_red\" : \"\\033[1;31m\",\n",
72
+ " }\n",
73
+ " \n",
74
+ " if color in color_codes:\n",
75
+ " color_start = color_codes[color]\n",
76
+ " color_end = \"\"\n",
77
+ " if reset:\n",
78
+ " color_end = color_codes[\"default\"]\n",
79
+ " formatted_text = \" \".join(str(arg) for arg in args)\n",
80
+ " if tqdm_desc:\n",
81
+ " color_end = \"\"\n",
82
+ " return color_start + formatted_text + color_end\n",
83
+ " else:\n",
84
+ " print(color_start + formatted_text + color_end)\n",
85
+ " else:\n",
86
+ " if tqdm_desc:\n",
87
+ " return \" \".join(str(arg) for arg in args)\n",
88
+ " else:\n",
89
+ " print(*args)\n",
90
+ "\n",
91
+ "def clone_repo(url, dir):\n",
92
+ " if not os.path.exists(dir):\n",
93
+ " subprocess.run([\"git\", \"clone\", url, dir], check=True)\n",
94
+ "\n",
95
+ "def ubuntu_deps(url, dst, desc):\n",
96
+ " os.makedirs(dst, exist_ok=True)\n",
97
+ " filename = get_filename(url)\n",
98
+ " subprocess.run([\"wget\", url], stdout=subprocess.DEVNULL)\n",
99
+ "\n",
100
+ " with zipfile.ZipFile(filename, \"r\") as deps:\n",
101
+ " deps.extractall(dst)\n",
102
+ "\n",
103
+ " for file in tqdm(os.listdir(dst), desc=desc):\n",
104
+ " if file.endswith(\".deb\"):\n",
105
+ " subprocess.run([\"dpkg\", \"-i\", os.path.join(dst, file)], stdout=subprocess.DEVNULL)\n",
106
+ " \n",
107
+ " os.remove(filename)\n",
108
+ " shutil.rmtree(dst)\n",
109
+ "\n",
110
+ "def get_filename(url):\n",
111
+ " response = requests.get(url, stream=True)\n",
112
+ " response.raise_for_status()\n",
113
+ "\n",
114
+ " if 'content-disposition' in response.headers:\n",
115
+ " content_disposition = response.headers['content-disposition']\n",
116
+ " filename = re.findall('filename=\"?([^\"]+)\"?', content_disposition)[0]\n",
117
+ " else:\n",
118
+ " url_path = urlparse(url).path\n",
119
+ " filename = unquote(os.path.basename(url_path))\n",
120
+ " \n",
121
+ " return filename\n",
122
+ "\n",
123
+ "def update_requirements(repo_dir, desired_module, desired_version, filepath):\n",
124
+ " with open(filepath, \"r\") as f:\n",
125
+ " lines = f.readlines()\n",
126
+ "\n",
127
+ " updated_lines = []\n",
128
+ " for line in lines:\n",
129
+ " if desired_module in line:\n",
130
+ " line = f\"{desired_module}=={desired_version}\\n\"\n",
131
+ " updated_lines.append(line)\n",
132
+ "\n",
133
+ " with open(filepath, \"w\") as f:\n",
134
+ " f.writelines(updated_lines)\n",
135
+ "\n",
136
+ "def install_dependencies():\n",
137
+ " requirements_file = os.path.join(repo_dir, \"requirements.txt\")\n",
138
+ "\n",
139
+ " ram_patch_url = \"https://huggingface.co/Linaqruf/fast-repo/resolve/main/ram_patch.zip\"\n",
140
+ "\n",
141
+ " cprint(f\"Installing ubuntu dependencies...\", color=\"green\")\n",
142
+ " subprocess.run([\"apt\", \"install\", \"aria2\", \"lz4\", \"libunwind8-dev\", \"-y\"], check=True)\n",
143
+ "\n",
144
+ " ubuntu_deps(ram_patch_url, deps_dir, cprint(\"Installing RAM allocation patch\", color=\"green\", tqdm_desc=True))\n",
145
+ " \n",
146
+ " cprint(f\"Installing requirements...\", color=\"green\")\n",
147
+ " update_requirements(repo_dir, \"requests\", \"2.27.1\", requirements_file)\n",
148
+ " subprocess.run(['pip', 'install', '--upgrade', '--no-cache-dir', 'gdown'], check=True)\n",
149
+ " subprocess.run(['pip', 'install', '--upgrade', '-r', requirements_file], cwd=repo_dir, check=True)\n",
150
+ "\n",
151
+ "def calculate_elapsed_time(start_time):\n",
152
+ " end_time = time.time()\n",
153
+ " elapsed_time = int(end_time - start_time)\n",
154
+ "\n",
155
+ " if elapsed_time < 60:\n",
156
+ " return f\"{elapsed_time} sec\"\n",
157
+ " else:\n",
158
+ " mins, secs = divmod(elapsed_time, 60)\n",
159
+ " return f\"{mins} mins {secs} sec\"\n",
160
+ "\n",
161
+ "def main():\n",
162
+ " os.chdir(root_dir)\n",
163
+ " start_time = time.time()\n",
164
+ "\n",
165
+ " for dir in [models_dir, vaes_dir]:\n",
166
+ " os.makedirs(dir, exist_ok=True)\n",
167
+ "\n",
168
+ " cprint(f\"Installing 'kohya-ss/sd-scripts'...\", color=\"green\")\n",
169
+ " clone_repo(repo_url, repo_dir)\n",
170
+ " install_dependencies()\n",
171
+ "\n",
172
+ " elapsed_time = calculate_elapsed_time(start_time)\n",
173
+ " \n",
174
+ " cprint(f\"\\nFinished installation. Took {elapsed_time}.\", color=\"green\")\n",
175
+ " cprint(f\"All is done! Go to the next step.\", color=\"green\")\n",
176
+ "\n",
177
+ "main()\n"
178
+ ],
179
+ "metadata": {
180
+ "id": "0BicRIFqIjg0",
181
+ "cellView": "form"
182
+ },
183
+ "execution_count": null,
184
+ "outputs": []
185
+ },
186
+ {
187
+ "cell_type": "code",
188
+ "source": [
189
+ "# @title ## **1.2. Download Model**\n",
190
+ "import os\n",
191
+ "import re\n",
192
+ "import json\n",
193
+ "import glob\n",
194
+ "import gdown\n",
195
+ "import time\n",
196
+ "import requests\n",
197
+ "import subprocess\n",
198
+ "from IPython.utils import capture\n",
199
+ "from urllib.parse import urlparse, unquote\n",
200
+ "from pathlib import Path\n",
201
+ "\n",
202
+ "os.chdir(root_dir)\n",
203
+ "\n",
204
+ "# @markdown ### **Download Stable Diffusion Model**\n",
205
+ "\n",
206
+ "model_url = \"https://civitai.com/api/download/models/82446\" #@param [\"Anime Model\", \"Anything V3.1\", \"AnyLoRA\", \"ChilloutMix Ni\", \"Stable Diffusion V1.5\", \"Replicant V3\", \"Illuminati Diffusion V1.1\", \"Waifu Diffusion V1.5 Beta 3\", \"Stable Diffusion V2.1\"] {allow-input: true}\n",
207
+ "# @markdown ### **Download VAE Model**\n",
208
+ "vae_url = \"\" #@param [\"\", \"Anime / Anything VAE\", \"Blessed VAE\", \"Waifu Diffusion VAE\", \"Stable Diffusion VAE\"] {allow-input: true}\n",
209
+ "# @markdown ### **Download LoRA Model**\n",
210
+ "lora_url = \"\" #@param {type: \"string\"}\n",
211
+ "\n",
212
+ "available_models = {\n",
213
+ " # SDv1.x Pretrained Model\n",
214
+ " \"Anime Model\" : \"https://huggingface.co/Linaqruf/personal-backup/resolve/main/models/animefull-final-pruned.ckpt\",\n",
215
+ " \"Anything V3.1\" : \"https://huggingface.co/Linaqruf/anything-v3.0/resolve/main/anything-v3-fp16-pruned.safetensors\",\n",
216
+ " \"AnyLoRA\" : \"https://huggingface.co/Lykon/AnyLoRA/resolve/main/AnyLoRA_noVae_fp16-pruned.ckpt\",\n",
217
+ " \"ChilloutMix Ni\" : \"https://huggingface.co/naonovn/chilloutmix_NiPrunedFp32Fix/resolve/main/chilloutmix_NiPrunedFp32Fix.safetensors\",\n",
218
+ " \"Stable Diffusion V1.5\" : \"https://huggingface.co/Linaqruf/stolen/resolve/main/pruned-models/stable_diffusion_1_5-pruned.safetensors\",\n",
219
+ " # SDv2.x Pretrained Model\n",
220
+ " \"Replicant V3\" : \"https://huggingface.co/gsdf/Replicant-V3.0/resolve/main/Replicant-V3.0_fp16.safetensors\",\n",
221
+ " \"Illuminati Diffusion V1.1\" : \"https://huggingface.co/Linaqruf/stolen/resolve/main/pruned-models/illuminatiDiffusionV1_v11.safetensors\",\n",
222
+ " \"Waifu Diffusion V1.5 Beta 3\" : \"https://huggingface.co/waifu-diffusion/wd-1-5-beta3/resolve/main/wd-beta3-base-fp16.safetensors\",\n",
223
+ " \"Stable Diffusion V2.1\" : \"https://huggingface.co/stabilityai/stable-diffusion-2-1/resolve/main/v2-1_768-ema-pruned.safetensors\",\n",
224
+ "}\n",
225
+ "\n",
226
+ "available_vaes = {\n",
227
+ " \"Anime / Anything VAE\" : \"https://huggingface.co/Linaqruf/personal-backup/resolve/main/vae/animevae.pt\",\n",
228
+ " \"Blessed VAE\" : \"https://huggingface.co/NoCrypt/blessed_vae/resolve/main/blessed2.vae.pt\",\n",
229
+ " \"Waifu Diffusion VAE\" : \"https://huggingface.co/hakurei/waifu-diffusion-v1-4/resolve/main/vae/kl-f8-anime.ckpt\",\n",
230
+ " \"Stable Diffusion VAE\" : \"https://huggingface.co/stabilityai/sd-vae-ft-mse-original/resolve/main/vae-ft-mse-840000-ema-pruned.ckpt\",\n",
231
+ "}\n",
232
+ "\n",
233
+ "if model_url is not None:\n",
234
+ " valid_model_url = model_url\n",
235
+ " if model_url in available_models:\n",
236
+ " valid_model_url = available_models[model_url]\n",
237
+ "\n",
238
+ "if vae_url is not None:\n",
239
+ " valid_vae_url = vae_url\n",
240
+ " if vae_url in available_vaes:\n",
241
+ " valid_vae_url = available_vaes[vae_url]\n",
242
+ "\n",
243
+ "def get_supported_extensions():\n",
244
+ " return tuple([\".ckpt\", \".safetensors\", \".pt\", \".pth\"])\n",
245
+ "\n",
246
+ "def get_filename(url, quiet=True):\n",
247
+ " extensions = get_supported_extensions()\n",
248
+ "\n",
249
+ " if url.startswith(\"/content/drive/MyDrive/\") or url.endswith(tuple(extensions)):\n",
250
+ " filename = os.path.basename(url)\n",
251
+ " else:\n",
252
+ " response = requests.get(url, stream=True)\n",
253
+ " response.raise_for_status()\n",
254
+ "\n",
255
+ " if 'content-disposition' in response.headers:\n",
256
+ " content_disposition = response.headers['content-disposition']\n",
257
+ " filename = re.findall('filename=\"?([^\"]+)\"?', content_disposition)[0]\n",
258
+ " else:\n",
259
+ " url_path = urlparse(url).path\n",
260
+ " filename = unquote(os.path.basename(url_path))\n",
261
+ "\n",
262
+ " if filename is not None and filename.endswith(tuple(extensions)):\n",
263
+ " return filename\n",
264
+ " else:\n",
265
+ " return None\n",
266
+ "\n",
267
+ "def parse_args(config):\n",
268
+ " args = []\n",
269
+ "\n",
270
+ " for k, v in config.items():\n",
271
+ " if k.startswith(\"_\"):\n",
272
+ " args.append(f\"{v}\")\n",
273
+ " elif isinstance(v, str) and v is not None:\n",
274
+ " args.append(f'--{k}={v}')\n",
275
+ " elif isinstance(v, bool) and v:\n",
276
+ " args.append(f\"--{k}\")\n",
277
+ " elif isinstance(v, float) and not isinstance(v, bool):\n",
278
+ " args.append(f\"--{k}={v}\")\n",
279
+ " elif isinstance(v, int) and not isinstance(v, bool):\n",
280
+ " args.append(f\"--{k}={v}\")\n",
281
+ "\n",
282
+ " return args\n",
283
+ "\n",
284
+ "def aria2_download(dir, filename, url):\n",
285
+ " hf_token = \"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\"\n",
286
+ " user_header = f\"Authorization: Bearer {hf_token}\"\n",
287
+ "\n",
288
+ " aria2_config = {\n",
289
+ " \"console-log-level\" : \"error\",\n",
290
+ " \"summary-interval\" : 10,\n",
291
+ " \"header\" : user_header if \"huggingface.co\" in url else None,\n",
292
+ " \"continue\" : True,\n",
293
+ " \"max-connection-per-server\" : 16,\n",
294
+ " \"min-split-size\" : \"1M\",\n",
295
+ " \"split\" : 16,\n",
296
+ " \"dir\" : dir,\n",
297
+ " \"out\" : filename,\n",
298
+ " \"_url\" : url,\n",
299
+ " }\n",
300
+ " aria2_args = parse_args(aria2_config)\n",
301
+ " subprocess.run([\"aria2c\", *aria2_args])\n",
302
+ "\n",
303
+ "def gdown_download(url, dst):\n",
304
+ " if \"/uc?id/\" in url:\n",
305
+ " return gdown.download(url, dst + \"/\", quiet=False)\n",
306
+ " elif \"/file/d/\" in url:\n",
307
+ " return gdown.download(url, dst + \"/\", quiet=False, fuzzy=True)\n",
308
+ " elif \"/drive/folders/\" in url:\n",
309
+ " os.chdir(dst)\n",
310
+ " return gdown.download_folder(url, quiet=True, use_cookies=False)\n",
311
+ "\n",
312
+ "def download(urls, dst, target):\n",
313
+ " for url in tqdm(urls, desc=cprint(f\"Downloading {target} from url\", color=\"green\", tqdm_desc=True)):\n",
314
+ " with capture.capture_output() as cap:\n",
315
+ " url = url.replace(\" \", \"\")\n",
316
+ " try:\n",
317
+ " filename = get_filename(url, quiet=False)\n",
318
+ " except Exception:\n",
319
+ " filename = None\n",
320
+ " continue\n",
321
+ "\n",
322
+ " if \"drive.google.com\" in url:\n",
323
+ " try:\n",
324
+ " gdown = gdown_download(url, dst)\n",
325
+ " except Exception as e:\n",
326
+ " print(f\"Error occurred: {str(e)}\")\n",
327
+ " elif url.startswith(\"/content/drive/MyDrive/\"):\n",
328
+ " filepath = os.path.join(dst, filename)\n",
329
+ " Path(filepath).write_bytes(Path(url).read_bytes())\n",
330
+ " else:\n",
331
+ " if \"huggingface.co\" in url:\n",
332
+ " if \"/blob/\" in url:\n",
333
+ " url = url.replace(\"/blob/\", \"/resolve/\")\n",
334
+ " aria2_download(dst, filename, url)\n",
335
+ " \n",
336
+ "def main():\n",
337
+ " global model_path, vae_path, lora_path\n",
338
+ "\n",
339
+ " start_time = time.time()\n",
340
+ " model_path = vae_path = lora_path = None\n",
341
+ "\n",
342
+ " download_targets = {\n",
343
+ " \"model\": (valid_model_url.split(','), models_dir),\n",
344
+ " \"vae\": (valid_vae_url.split(','), vaes_dir),\n",
345
+ " \"lora\": (lora_url.split(','), lora_dir),\n",
346
+ " }\n",
347
+ "\n",
348
+ " for target, (urls, dst) in download_targets.items():\n",
349
+ " if urls and urls != \"PASTE {} URL OR GDRIVE PATH HERE\".format(target.upper()):\n",
350
+ " initial_files = glob.glob(os.path.join(dst, \"*\"))\n",
351
+ " download(urls, dst, target)\n",
352
+ "\n",
353
+ " downloaded_files = []\n",
354
+ " for filename in initial_files:\n",
355
+ " filepath = os.path.join(dst, filename)\n",
356
+ " if os.path.exists(filepath):\n",
357
+ " downloaded_files.append(filepath)\n",
358
+ "\n",
359
+ " if len(downloaded_files) == 0:\n",
360
+ " downloaded_files = sorted(\n",
361
+ " glob.glob(os.path.join(dst, \"*\")), key=os.path.getmtime, reverse=True\n",
362
+ " )\n",
363
+ "\n",
364
+ " cprint(f\"Downloaded files for {target}:\", color=\"green\")\n",
365
+ " for file in downloaded_files:\n",
366
+ " cprint(\" - \", os.path.basename(file), color=\"green\")\n",
367
+ "\n",
368
+ " elapsed_time = calculate_elapsed_time(start_time)\n",
369
+ " \n",
370
+ " cprint(f\"\\nFinished installation. Took {elapsed_time}.\", color=\"green\")\n",
371
+ " cprint(f\"All is done! Go to the next step.\", color=\"green\")\n",
372
+ "\n",
373
+ "main()\n"
374
+ ],
375
+ "metadata": {
376
+ "id": "A2NC0tm8vU-K",
377
+ "cellView": "form"
378
+ },
379
+ "execution_count": null,
380
+ "outputs": []
381
+ },
382
+ {
383
+ "cell_type": "markdown",
384
+ "source": [
385
+ "# 2. Model Conversion"
386
+ ],
387
+ "metadata": {
388
+ "id": "WDPfF4uc5pd1"
389
+ }
390
+ },
391
+ {
392
+ "cell_type": "code",
393
+ "source": [
394
+ "import os\n",
395
+ "%store -r\n",
396
+ "#@title ## 7.2. Model Pruner\n",
397
+ "\n",
398
+ "os.chdir(tools_dir)\n",
399
+ "\n",
400
+ "if not os.path.exists('prune.py'):\n",
401
+ " !wget https://raw.githubusercontent.com/lopho/stable-diffusion-prune/main/prune.py\n",
402
+ "\n",
403
+ "#@markdown Convert to Float16\n",
404
+ "fp16 = True #@param {'type':'boolean'}\n",
405
+ "#@markdown Use EMA for weights\n",
406
+ "ema = False #@param {'type':'boolean'}\n",
407
+ "#@markdown Strip CLIP weights\n",
408
+ "no_clip = False #@param {'type':'boolean'}\n",
409
+ "#@markdown Strip VAE weights\n",
410
+ "no_vae = False #@param {'type':'boolean'}\n",
411
+ "#@markdown Strip depth model weights\n",
412
+ "no_depth = False #@param {'type':'boolean'}\n",
413
+ "#@markdown Strip UNet weights\n",
414
+ "no_unet = False #@param {'type':'boolean'}\n",
415
+ "\n",
416
+ "model_path = \"/content/models/Counterfeit-V3.0_fp16.safetensors\" #@param {'type' : 'string'}\n",
417
+ "\n",
418
+ "config = {\n",
419
+ " \"fp16\": fp16,\n",
420
+ " \"ema\": ema,\n",
421
+ " \"no_clip\": no_clip,\n",
422
+ " \"no_vae\": no_vae,\n",
423
+ " \"no_depth\": no_depth,\n",
424
+ " \"no_unet\": no_unet,\n",
425
+ "}\n",
426
+ "\n",
427
+ "suffixes = {\n",
428
+ " \"fp16\": \"-fp16\",\n",
429
+ " \"ema\": \"-ema\",\n",
430
+ " \"no_clip\": \"-no-clip\",\n",
431
+ " \"no_vae\": \"-no-vae\",\n",
432
+ " \"no_depth\": \"-no-depth\",\n",
433
+ " \"no_unet\": \"-no-unet\",\n",
434
+ "}\n",
435
+ "\n",
436
+ "print(f\"Loading model from {model_path}\")\n",
437
+ "\n",
438
+ "dir_name = os.path.dirname(model_path)\n",
439
+ "base_name = os.path.basename(model_path)\n",
440
+ "output_name = base_name.split('.')[0]\n",
441
+ "\n",
442
+ "for option, suffix in suffixes.items():\n",
443
+ " if config[option]:\n",
444
+ " print(f\"Applying option {option}\")\n",
445
+ " output_name += suffix\n",
446
+ " \n",
447
+ "output_name += '-pruned'\n",
448
+ "output_path = os.path.join(dir_name, output_name + ('.ckpt' if model_path.endswith(\".ckpt\") else \".safetensors\"))\n",
449
+ "\n",
450
+ "args = \"\"\n",
451
+ "for k, v in config.items():\n",
452
+ " if k.startswith(\"_\"):\n",
453
+ " args += f'\"{v}\" '\n",
454
+ " elif isinstance(v, str):\n",
455
+ " args += f'--{k}=\"{v}\" '\n",
456
+ " elif isinstance(v, bool) and v:\n",
457
+ " args += f\"--{k} \"\n",
458
+ " elif isinstance(v, float) and not isinstance(v, bool):\n",
459
+ " args += f\"--{k}={v} \"\n",
460
+ " elif isinstance(v, int) and not isinstance(v, bool):\n",
461
+ " args += f\"--{k}={v} \"\n",
462
+ "\n",
463
+ "final_args = f\"python3 prune.py {model_path} {output_path} {args}\"\n",
464
+ "!{final_args}\n",
465
+ "\n",
466
+ "print(f\"Saving pruned model to {output_path}\")"
467
+ ],
468
+ "metadata": {
469
+ "cellView": "form",
470
+ "id": "fb3rxuCaSYta"
471
+ },
472
+ "execution_count": null,
473
+ "outputs": []
474
+ },
475
+ {
476
+ "cell_type": "code",
477
+ "source": [
478
+ "#@title ## 7.1. Convert Diffusers to Checkpoint\n",
479
+ "import os\n",
480
+ "%store -r\n",
481
+ "\n",
482
+ "os.chdir(tools_dir)\n",
483
+ "\n",
484
+ "#@markdown ### Conversion Config\n",
485
+ "model_to_load = \"\" #@param {'type': 'string'}\n",
486
+ "model_to_save = os.path.splitext(model_to_load)[0]\n",
487
+ "convert = \"checkpoint_to_diffusers\" #@param [\"diffusers_to_checkpoint\", \"checkpoint_to_diffusers\"] {'allow-input': false}\n",
488
+ "v2 = True #@param {type:'boolean'}\n",
489
+ "global_step = 0 #@param {'type': 'number'}\n",
490
+ "epoch = 0 #@param {'type': 'number'}\n",
491
+ "use_safetensors = True #@param {'type': 'boolean'}\n",
492
+ "save_precision_as = \"--float\" #@param [\"--fp16\",\"--bf16\",\"--float\"] {'allow-input': false}\n",
493
+ "\n",
494
+ "#@markdown Additional option for diffusers\n",
495
+ "feature_extractor = True #@param {'type': 'boolean'}\n",
496
+ "safety_checker = True #@param {'type': 'boolean'}\n",
497
+ "\n",
498
+ "reference_model = \"stabilityai/stable-diffusion-2-1\" if v2 else \"runwayml/stable-diffusion-v1-5\" \n",
499
+ "model_output = f\"{model_to_save}.safetensors\" if use_safetensors else f\"{model_to_save}.ckpt\"\n",
500
+ "\n",
501
+ "urls = [\n",
502
+ " (\"preprocessor_config.json\", \"https://huggingface.co/CompVis/stable-diffusion-safety-checker/resolve/main/preprocessor_config.json\"),\n",
503
+ " (\"config.json\", \"https://huggingface.co/CompVis/stable-diffusion-safety-checker/resolve/main/config.json\"),\n",
504
+ " (\"pytorch_model.bin\", \"https://huggingface.co/CompVis/stable-diffusion-safety-checker/resolve/main/pytorch_model.bin\"),\n",
505
+ "]\n",
506
+ "\n",
507
+ "diffusers_to_sd_dict = {\n",
508
+ " \"_model_to_load\": model_to_load,\n",
509
+ " \"_model_to_save\": model_output,\n",
510
+ " \"global_step\": global_step,\n",
511
+ " \"epoch\": epoch,\n",
512
+ " \"save_precision_as\": save_precision_as,\n",
513
+ "}\n",
514
+ "\n",
515
+ "sd_to_diffusers_dict = {\n",
516
+ " \"_model_to_load\": model_to_load,\n",
517
+ " \"_model_to_save\": model_to_save,\n",
518
+ " \"v2\": True if v2 else False,\n",
519
+ " \"v1\": True if not v2 else False,\n",
520
+ " \"global_step\": global_step,\n",
521
+ " \"epoch\": epoch,\n",
522
+ " \"fp16\": True if save_precision_as == \"fp16\" else False,\n",
523
+ " \"use_safetensors\": use_safetensors,\n",
524
+ " \"reference_model\": reference_model\n",
525
+ "}\n",
526
+ "\n",
527
+ "def convert_dict(config):\n",
528
+ " args = \"\"\n",
529
+ " for k, v in config.items():\n",
530
+ " if k.startswith(\"_\"):\n",
531
+ " args += f'\"{v}\" '\n",
532
+ " elif isinstance(v, str):\n",
533
+ " args += f'--{k}=\"{v}\" '\n",
534
+ " elif isinstance(v, bool) and v:\n",
535
+ " args += f\"--{k} \"\n",
536
+ " elif isinstance(v, float) and not isinstance(v, bool):\n",
537
+ " args += f\"--{k}={v} \"\n",
538
+ " elif isinstance(v, int) and not isinstance(v, bool):\n",
539
+ " args += f\"--{k}={v} \"\n",
540
+ "\n",
541
+ " return args\n",
542
+ "\n",
543
+ "def run_script(script_name, script_args):\n",
544
+ " !python {script_name} {script_args}\n",
545
+ "\n",
546
+ "def download(output, url, save_dir):\n",
547
+ " !aria2c --console-log-level=error --summary-interval=10 -c -x 16 -k 1M -s 16 -d '{save_dir}' -o '{output}' {url}\n",
548
+ "\n",
549
+ "diffusers_to_sd_args = convert_dict(diffusers_to_sd_dict)\n",
550
+ "sd_to_diffusers_args = convert_dict(sd_to_diffusers_dict)\n",
551
+ "\n",
552
+ "if convert == \"diffusers_to_checkpoint\":\n",
553
+ " if model_to_load.endswith((\"ckpt\",\"safetensors\")):\n",
554
+ " print(f\"{os.path.basename(model_to_load)} is not in diffusers format\")\n",
555
+ " else:\n",
556
+ " run_script(\"convert_diffusers20_original_sd.py\", diffusers_to_sd_args)\n",
557
+ "else:\n",
558
+ " if not model_to_load.endswith((\"ckpt\",\"safetensors\")):\n",
559
+ " print(f\"{os.path.basename(model_to_load)} is not in ckpt/safetensors format\")\n",
560
+ " else: \n",
561
+ " run_script(\"convert_diffusers20_original_sd.py\", sd_to_diffusers_args)\n",
562
+ "\n",
563
+ " if feature_extractor:\n",
564
+ " save_dir = os.path.join(model_to_save, \"feature_extractor\")\n",
565
+ " os.makedirs(save_dir, exist_ok=True)\n",
566
+ " output, url = urls[0]\n",
567
+ " download(output, url, save_dir)\n",
568
+ " \n",
569
+ " if safety_checker:\n",
570
+ " save_dir = os.path.join(model_to_save, \"safety_checker\")\n",
571
+ " os.makedirs(save_dir, exist_ok=True)\n",
572
+ " for output, url in urls[1:]:\n",
573
+ " download(output, url, save_dir)"
574
+ ],
575
+ "metadata": {
576
+ "cellView": "form",
577
+ "id": "TdCb8_dSSzzr"
578
+ },
579
+ "execution_count": null,
580
+ "outputs": []
581
+ },
582
+ {
583
+ "cell_type": "code",
584
+ "source": [
585
+ "#@title 2.3. Replace VAE of Existing Model \n",
586
+ "\n",
587
+ "os.chdir(tools_dir)\n",
588
+ "if not os.path.exists('merge_vae.py'):\n",
589
+ " !wget https://raw.githubusercontent.com/Linaqruf/kohya-trainer/main/tools/merge_vae.py\n",
590
+ "\n",
591
+ "#@markdown You need to input model ends with `.ckpt`, because `.safetensors` model won't work.\n",
592
+ "\n",
593
+ "target_model = \"\" #@param {'type': 'string'}\n",
594
+ "target_vae = \"/content/vae/anime.vae.pt\" #@param {'type': 'string'}\n",
595
+ "use_safetensors = False #@param {type:'boolean'}\n",
596
+ "# get the base file name and directory\n",
597
+ "base_name = os.path.basename(target_model)\n",
598
+ "base_dir = os.path.dirname(target_model)\n",
599
+ "\n",
600
+ "# get the file name without extension\n",
601
+ "file_name = os.path.splitext(base_name)[0]\n",
602
+ "\n",
603
+ "# create the new file name\n",
604
+ "new_file_name = file_name + \"-vae-swapped\"\n",
605
+ "\n",
606
+ "# get the file extension\n",
607
+ "file_ext = os.path.splitext(base_name)[1]\n",
608
+ "\n",
609
+ "# create the output file path\n",
610
+ "output_model = os.path.join(base_dir, new_file_name + file_ext)\n",
611
+ "\n",
612
+ "!python merge_vae.py \\\n",
613
+ " {target_model} \\\n",
614
+ " {target_vae} \\\n",
615
+ " {output_model}\n",
616
+ "\n"
617
+ ],
618
+ "metadata": {
619
+ "cellView": "form",
620
+ "id": "g2QfhhlfbGsf"
621
+ },
622
+ "execution_count": null,
623
+ "outputs": []
624
+ },
625
+ {
626
+ "cell_type": "code",
627
+ "source": [
628
+ "#@title 2.4. Convert CKPT-2-Safetensors\n",
629
+ "\n",
630
+ "import os\n",
631
+ "import torch\n",
632
+ "from safetensors.torch import load_file, save_file\n",
633
+ "from torch import load, save\n",
634
+ "\n",
635
+ "model_path = \"\" #@param {type: 'string'}\n",
636
+ "\n",
637
+ "def is_safetensors(path):\n",
638
+ " return os.path.splitext(path)[1].lower() == '.safetensors'\n",
639
+ "\n",
640
+ "def convert(model_path):\n",
641
+ " print(\"Loading model:\", os.path.basename(model_path))\n",
642
+ " \n",
643
+ " try:\n",
644
+ " with torch.no_grad():\n",
645
+ " print(\"Conversion in progress, please wait...\")\n",
646
+ " if is_safetensors(model_path):\n",
647
+ " model = load_file(model_path, device=\"cpu\")\n",
648
+ " else:\n",
649
+ " model = load(model_path, map_location=\"cpu\")\n",
650
+ " \n",
651
+ " if 'state_dict' in model:\n",
652
+ " sd = model['state_dict']\n",
653
+ " else:\n",
654
+ " sd = model\n",
655
+ "\n",
656
+ " save_to = \".ckpt\" if is_safetensors(model_path) else \".safetensors\"\n",
657
+ " output = os.path.splitext(model_path)[0] + save_to\n",
658
+ "\n",
659
+ " if is_safetensors(model_path):\n",
660
+ " save(sd, output)\n",
661
+ " else:\n",
662
+ " save_file(sd, output)\n",
663
+ "\n",
664
+ " print(f'Successfully converted {os.path.basename(model_path)} to {os.path.basename(output)}')\n",
665
+ " print(f'located in this path : {output}')\n",
666
+ " except Exception as ex:\n",
667
+ " print(f'ERROR converting {os.path.basename(model_path)}: {ex}')\n",
668
+ "\n",
669
+ " print('Done!')\n",
670
+ "\n",
671
+ "def main():\n",
672
+ " convert(model_path)\n",
673
+ "main()\n"
674
+ ],
675
+ "metadata": {
676
+ "cellView": "form",
677
+ "id": "L8WI17pmnlVM"
678
+ },
679
+ "execution_count": null,
680
+ "outputs": []
681
+ },
682
+ {
683
+ "cell_type": "markdown",
684
+ "source": [
685
+ "# VIII. Deployment"
686
+ ],
687
+ "metadata": {
688
+ "id": "nyIl9BhNXKUq"
689
+ }
690
+ },
691
+ {
692
+ "cell_type": "code",
693
+ "source": [
694
+ "# @title ## 7.1. Upload Config\n",
695
+ "from huggingface_hub import login\n",
696
+ "from huggingface_hub import HfApi\n",
697
+ "from huggingface_hub.utils import validate_repo_id, HfHubHTTPError\n",
698
+ "\n",
699
+ "# @markdown Login to Huggingface Hub\n",
700
+ "# @markdown > Get **your** huggingface `WRITE` token [here](https://huggingface.co/settings/tokens)\n",
701
+ "write_token = \"\" # @param {type:\"string\"}\n",
702
+ "# @markdown Fill this if you want to upload to your organization, or just leave it empty.\n",
703
+ "orgs_name = \"\" # @param{type:\"string\"}\n",
704
+ "# @markdown If your model/dataset repo does not exist, it will automatically create it.\n",
705
+ "repo_name = \"stolen\" # @param{type:\"string\"}\n",
706
+ "make_private = False # @param{type:\"boolean\"}\n",
707
+ "\n",
708
+ "def authenticate(write_token):\n",
709
+ " login(write_token, add_to_git_credential=True)\n",
710
+ " api = HfApi()\n",
711
+ " return api.whoami(write_token), api\n",
712
+ "\n",
713
+ "def create_repo(api, user, orgs_name, repo_name, make_private=False):\n",
714
+ " if orgs_name == \"\":\n",
715
+ " repo_id = user[\"name\"] + \"/\" + repo_name.strip()\n",
716
+ " else:\n",
717
+ " repo_id = orgs_name + \"/\" + repo_name.strip()\n",
718
+ "\n",
719
+ " try:\n",
720
+ " validate_repo_id(repo_id)\n",
721
+ " api.create_repo(repo_id=repo_id, private=make_private)\n",
722
+ " print(f\"Model repo '{repo_id}' didn't exist, creating repo\")\n",
723
+ " except HfHubHTTPError as e:\n",
724
+ " print(f\"Model repo '{repo_id}' exists, skipping create repo\")\n",
725
+ " \n",
726
+ " print(f\"Model repo '{repo_id}' link: https://huggingface.co/{repo_id}\\n\")\n",
727
+ " return repo_id\n",
728
+ "\n",
729
+ "user, api = authenticate(write_token)\n",
730
+ "\n",
731
+ "model_repo = create_repo(api, user, orgs_name, repo_name, make_private)\n"
732
+ ],
733
+ "metadata": {
734
+ "cellView": "form",
735
+ "id": "QTXsM170GUpk"
736
+ },
737
+ "execution_count": null,
738
+ "outputs": []
739
+ },
740
+ {
741
+ "cell_type": "markdown",
742
+ "source": [
743
+ "## 8.2. Upload with Huggingface Hub"
744
+ ],
745
+ "metadata": {
746
+ "id": "Fuxghk8MnG6j"
747
+ }
748
+ },
749
+ {
750
+ "cell_type": "code",
751
+ "source": [
752
+ "#@title ### 8.2.1. Upload Model\n",
753
+ "from huggingface_hub import HfApi\n",
754
+ "from huggingface_hub.utils import validate_repo_id, HfHubHTTPError\n",
755
+ "from pathlib import Path\n",
756
+ "\n",
757
+ "api = HfApi()\n",
758
+ "\n",
759
+ "#@markdown This will be uploaded to model repo\n",
760
+ "\n",
761
+ "model_path = \"/content/models\" #@param {type :\"string\"}\n",
762
+ "path_in_repo = \"fp16\" #@param {type :\"string\"}\n",
763
+ "revision = \"\" #@param {type :\"string\"}\n",
764
+ "if revision:\n",
765
+ " api.create_branch(repo_id=model_repo, \n",
766
+ " branch=revision, \n",
767
+ " exist_ok=True)\n",
768
+ "else:\n",
769
+ " revision = \"main\"\n",
770
+ "project_name = os.path.basename(model_path)\n",
771
+ "if project_name in [\".safetensors\", \"ckpt\", \"pt\"]:\n",
772
+ " project_name = os.path.split(model_path)[0]\n",
773
+ "# @markdown Other Information\n",
774
+ "commit_message = \"\" # @param {type :\"string\"}\n",
775
+ "\n",
776
+ "if not commit_message:\n",
777
+ " commit_message = \"feat: upload \" + project_name + \" checkpoint\"\n",
778
+ "\n",
779
+ "if os.path.exists(model_path):\n",
780
+ " vae_exists = os.path.exists(os.path.join(model_path, 'vae'))\n",
781
+ " unet_exists = os.path.exists(os.path.join(model_path, 'unet'))\n",
782
+ " text_encoder_exists = os.path.exists(os.path.join(model_path, 'text_encoder'))\n",
783
+ " \n",
784
+ "def upload_model(model_paths, is_folder :bool, commit_message):\n",
785
+ " path_obj = Path(model_paths)\n",
786
+ " trained_model = path_obj.parts[-1]\n",
787
+ " \n",
788
+ " if path_in_repo:\n",
789
+ " trained_model = path_in_repo\n",
790
+ " \n",
791
+ " if is_folder == True:\n",
792
+ " print(f\"Uploading {trained_model} to https://huggingface.co/\"+model_repo)\n",
793
+ " print(f\"Please wait...\")\n",
794
+ "\n",
795
+ " if vae_exists and unet_exists and text_encoder_exists:\n",
796
+ " if not commit_message:\n",
797
+ " commit_message = f\"feat: upload diffusers version of {trained_model}\"\n",
798
+ "\n",
799
+ " api.upload_folder(\n",
800
+ " folder_path=model_paths,\n",
801
+ " repo_id=model_repo,\n",
802
+ " revision=revision,\n",
803
+ " commit_message=commit_message,\n",
804
+ " ignore_patterns=\".ipynb_checkpoints\"\n",
805
+ " )\n",
806
+ " \n",
807
+ " else:\n",
808
+ " if not commit_message:\n",
809
+ " commit_message = f\"feat: upload {trained_model} checkpoint folder\"\n",
810
+ "\n",
811
+ " api.upload_folder(\n",
812
+ " folder_path=model_paths,\n",
813
+ " path_in_repo=trained_model,\n",
814
+ " repo_id=model_repo,\n",
815
+ " revision=revision,\n",
816
+ " commit_message=commit_message,\n",
817
+ " ignore_patterns=\".ipynb_checkpoints\"\n",
818
+ " )\n",
819
+ " print(f\"Upload success, located at https://huggingface.co/\"+model_repo+\"/tree/main\\n\")\n",
820
+ " else: \n",
821
+ " print(f\"Uploading {trained_model} to https://huggingface.co/\"+model_repo)\n",
822
+ " print(f\"Please wait...\")\n",
823
+ " if not commit_message:\n",
824
+ " if model_paths.endswith(\".safetensors\"):\n",
825
+ " commit_message = f\"feat: upload safetensors version of {trained_model} \"\n",
826
+ " else:\n",
827
+ " commit_message = f\"feat: upload {trained_model} checkpoint\"\n",
828
+ " \n",
829
+ " api.upload_file(\n",
830
+ " path_or_fileobj=model_paths,\n",
831
+ " path_in_repo=trained_model,\n",
832
+ " repo_id=model_repo,\n",
833
+ " revision=revision,\n",
834
+ " commit_message=commit_message,\n",
835
+ " )\n",
836
+ " \n",
837
+ " print(f\"Upload success, located at https://huggingface.co/\"+model_repo+\"/blob/main/\"+trained_model+\"\\n\")\n",
838
+ " \n",
839
+ "def upload():\n",
840
+ " if model_path.endswith((\".ckpt\", \".safetensors\", \".pt\")):\n",
841
+ " upload_model(model_path, False, commit_message)\n",
842
+ " else:\n",
843
+ " upload_model(model_path, True, commit_message)\n",
844
+ "\n",
845
+ "upload()"
846
+ ],
847
+ "metadata": {
848
+ "cellView": "form",
849
+ "id": "CIeoJA-eO-8t"
850
+ },
851
+ "execution_count": null,
852
+ "outputs": []
853
+ },
854
+ {
855
+ "cell_type": "markdown",
856
+ "source": [
857
+ "## 8.3. Upload with GIT (Alternative)"
858
+ ],
859
+ "metadata": {
860
+ "id": "CKZpg4keWS5c"
861
+ }
862
+ },
863
+ {
864
+ "cell_type": "code",
865
+ "source": [
866
+ "#@title ### 8.3.1. Clone Repository\n",
867
+ "%cd /content/\n",
868
+ "clone_model = True #@param {'type': 'boolean'}\n",
869
+ "\n",
870
+ "!git lfs install --skip-smudge\n",
871
+ "!export GIT_LFS_SKIP_SMUDGE=1\n",
872
+ "\n",
873
+ "if clone_model:\n",
874
+ " !git clone https://huggingface.co/{model_repo} /content/{repo_name}"
875
+ ],
876
+ "metadata": {
877
+ "cellView": "form",
878
+ "id": "6nBlrOrytO9F"
879
+ },
880
+ "execution_count": null,
881
+ "outputs": []
882
+ },
883
+ {
884
+ "cell_type": "code",
885
+ "source": [
886
+ "#@title ### 8.3.2. Commit using Git \n",
887
+ "import os\n",
888
+ "\n",
889
+ "os.chdir(root_dir)\n",
890
+ "\n",
891
+ "#@markdown Choose which repo you want to commit\n",
892
+ "commit_model = True #@param {'type': 'boolean'}\n",
893
+ "#@markdown #### Other Information\n",
894
+ "commit_message = \"\" #@param {type :\"string\"}\n",
895
+ "\n",
896
+ "if not commit_message:\n",
897
+ " commit_message = f\"feat: upload {repo_name}\"\n",
898
+ "\n",
899
+ "!git config --global user.email \"example@mail.com\"\n",
900
+ "!git config --global user.name \"example\"\n",
901
+ "\n",
902
+ "def commit(repo_folder, commit_message):\n",
903
+ " os.chdir(os.path.join(root_dir, repo_folder))\n",
904
+ " !git lfs install\n",
905
+ " !huggingface-cli lfs-enable-largefiles .\n",
906
+ " !git add .\n",
907
+ " !git commit -m \"{commit_message}\"\n",
908
+ " !git push\n",
909
+ "\n",
910
+ "commit(repo_name, commit_message)\n"
911
+ ],
912
+ "metadata": {
913
+ "cellView": "form",
914
+ "id": "7bJev4PzOFFB"
915
+ },
916
+ "execution_count": null,
917
+ "outputs": []
918
+ }
919
+ ]
920
+ }