2023-10-31 16:13:27,664 INFO StreamThr :1081067 [internal.py:wandb_internal():86] W&B internal server running at pid: 1081067, started at: 2023-10-31 16:13:27.663059 2023-10-31 16:13:27,667 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status 2023-10-31 16:13:27,669 INFO WriterThread:1081067 [datastore.py:open_for_write():85] open: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/run-8441xowv.wandb 2023-10-31 16:13:27,671 DEBUG SenderThread:1081067 [sender.py:send():380] send: header 2023-10-31 16:13:27,676 DEBUG SenderThread:1081067 [sender.py:send():380] send: run 2023-10-31 16:13:28,262 INFO SenderThread:1081067 [dir_watcher.py:__init__():211] watching files in: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files 2023-10-31 16:13:28,263 INFO SenderThread:1081067 [sender.py:_start_run_threads():1122] run started: 8441xowv with start time 1698736407.663163 2023-10-31 16:13:28,280 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: check_version 2023-10-31 16:13:28,281 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: check_version 2023-10-31 16:13:28,438 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: run_start 2023-10-31 16:13:28,482 DEBUG HandlerThread:1081067 [system_info.py:__init__():32] System info init 2023-10-31 16:13:28,482 DEBUG HandlerThread:1081067 [system_info.py:__init__():47] System info init done 2023-10-31 16:13:28,482 INFO HandlerThread:1081067 [system_monitor.py:start():194] Starting system monitor 2023-10-31 16:13:28,483 INFO SystemMonitor:1081067 [system_monitor.py:_start():158] Starting system asset monitoring threads 2023-10-31 16:13:28,483 INFO HandlerThread:1081067 [system_monitor.py:probe():214] Collecting system info 2023-10-31 16:13:28,484 INFO SystemMonitor:1081067 [interfaces.py:start():190] Started cpu monitoring 2023-10-31 16:13:28,485 INFO SystemMonitor:1081067 [interfaces.py:start():190] Started disk monitoring 2023-10-31 16:13:28,486 INFO SystemMonitor:1081067 [interfaces.py:start():190] Started gpu monitoring 2023-10-31 16:13:28,487 INFO SystemMonitor:1081067 [interfaces.py:start():190] Started memory monitoring 2023-10-31 16:13:28,488 INFO SystemMonitor:1081067 [interfaces.py:start():190] Started network monitoring 2023-10-31 16:13:28,545 DEBUG HandlerThread:1081067 [system_info.py:probe():196] Probing system 2023-10-31 16:13:28,548 DEBUG HandlerThread:1081067 [system_info.py:_probe_git():181] Probing git 2023-10-31 16:13:28,558 DEBUG HandlerThread:1081067 [system_info.py:_probe_git():189] Probing git done 2023-10-31 16:13:28,558 DEBUG HandlerThread:1081067 [system_info.py:probe():244] Probing system done 2023-10-31 16:13:28,558 DEBUG HandlerThread:1081067 [system_monitor.py:probe():223] {'os': 'Linux-5.4.0-109-generic-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2023-10-31T07:13:28.546062', 'startedAt': '2023-10-31T07:13:27.653386', 'docker': None, 'cuda': None, 'args': ('--pretrained_model_name_or_path=stabilityai/stable-diffusion-2', '--instance_data_dir=dog', '--output_dir=/home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/', '--instance_prompt=a photo of sks dog', '--resolution=512', '--train_batch_size=1', '--gradient_accumulation_steps=1', '--checkpointing_steps=100', '--learning_rate=1e-4', '--report_to=wandb', '--lr_scheduler=constant', '--lr_warmup_steps=0', '--max_train_steps=500', '--validation_prompt=A photo of sks dog in a bucket', '--validation_epochs=50', '--seed=0', '--push_to_hub'), 'state': 'running', 'program': '/home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/train_dreambooth_lora.py', 'codePathLocal': 'train_dreambooth_lora.py', 'codePath': 'examples/dreambooth/train_dreambooth_lora.py', 'git': {'remote': 'https://github.com/huggingface/diffusers.git', 'commit': '32fea1cc9bd36e91e42558f978ea224f971f0b51'}, 'email': None, 'root': '/home/ubuntu/Desktop/jaeho/diffusers', 'host': 'GPUSVR12', 'username': 'ubuntu', 'executable': '/home/ubuntu/anaconda3/envs/dreambooth/bin/python', 'cpu_count': 52, 'cpu_count_logical': 104, 'cpu_freq': {'current': 2382.8386057692323, 'min': 1000.0, 'max': 4000.0}, 'cpu_freq_per_core': [{'current': 1006.326, 'min': 1000.0, 'max': 4000.0}, {'current': 1509.222, 'min': 1000.0, 'max': 4000.0}, {'current': 1507.0, 'min': 1000.0, 'max': 4000.0}, {'current': 999.992, 'min': 1000.0, 'max': 4000.0}, {'current': 1401.495, 'min': 1000.0, 'max': 4000.0}, {'current': 1007.377, 'min': 1000.0, 'max': 4000.0}, {'current': 1519.841, 'min': 1000.0, 'max': 4000.0}, {'current': 1201.375, 'min': 1000.0, 'max': 4000.0}, {'current': 1000.786, 'min': 1000.0, 'max': 4000.0}, {'current': 1000.384, 'min': 1000.0, 'max': 4000.0}, {'current': 1289.83, 'min': 1000.0, 'max': 4000.0}, {'current': 1054.753, 'min': 1000.0, 'max': 4000.0}, {'current': 999.984, 'min': 1000.0, 'max': 4000.0}, {'current': 1563.155, 'min': 1000.0, 'max': 4000.0}, {'current': 1323.271, 'min': 1000.0, 'max': 4000.0}, {'current': 1006.915, 'min': 1000.0, 'max': 4000.0}, {'current': 1139.581, 'min': 1000.0, 'max': 4000.0}, {'current': 999.989, 'min': 1000.0, 'max': 4000.0}, {'current': 999.985, 'min': 1000.0, 'max': 4000.0}, {'current': 3627.15, 'min': 1000.0, 'max': 4000.0}, {'current': 1006.957, 'min': 1000.0, 'max': 4000.0}, {'current': 1336.875, 'min': 1000.0, 'max': 4000.0}, {'current': 1106.274, 'min': 1000.0, 'max': 4000.0}, {'current': 1006.373, 'min': 1000.0, 'max': 4000.0}, {'current': 1004.658, 'min': 1000.0, 'max': 4000.0}, {'current': 1000.0, 'min': 1000.0, 'max': 4000.0}, {'current': 1101.629, 'min': 1000.0, 'max': 4000.0}, {'current': 1000.001, 'min': 1000.0, 'max': 4000.0}, {'current': 1004.104, 'min': 1000.0, 'max': 4000.0}, {'current': 1101.349, 'min': 1000.0, 'max': 4000.0}, {'current': 1001.263, 'min': 1000.0, 'max': 4000.0}, {'current': 1000.536, 'min': 1000.0, 'max': 4000.0}, {'current': 1106.625, 'min': 1000.0, 'max': 4000.0}, {'current': 1101.509, 'min': 1000.0, 'max': 4000.0}, {'current': 1000.002, 'min': 1000.0, 'max': 4000.0}, {'current': 1002.199, 'min': 1000.0, 'max': 4000.0}, {'current': 1000.227, 'min': 1000.0, 'max': 4000.0}, {'current': 1101.74, 'min': 1000.0, 'max': 4000.0}, {'current': 1106.07, 'min': 1000.0, 'max': 4000.0}, {'current': 1188.662, 'min': 1000.0, 'max': 4000.0}, {'current': 999.997, 'min': 1000.0, 'max': 4000.0}, {'current': 3346.359, 'min': 1000.0, 'max': 4000.0}, {'current': 1004.15, 'min': 1000.0, 'max': 4000.0}, {'current': 1106.981, 'min': 1000.0, 'max': 4000.0}, {'current': 999.992, 'min': 1000.0, 'max': 4000.0}, {'current': 1000.681, 'min': 1000.0, 'max': 4000.0}, {'current': 1002.734, 'min': 1000.0, 'max': 4000.0}, {'current': 1000.035, 'min': 1000.0, 'max': 4000.0}, {'current': 999.999, 'min': 1000.0, 'max': 4000.0}, {'current': 1105.129, 'min': 1000.0, 'max': 4000.0}, {'current': 1101.069, 'min': 1000.0, 'max': 4000.0}, {'current': 1002.323, 'min': 1000.0, 'max': 4000.0}, {'current': 1000.004, 'min': 1000.0, 'max': 4000.0}, {'current': 1506.458, 'min': 1000.0, 'max': 4000.0}, {'current': 1505.831, 'min': 1000.0, 'max': 4000.0}, {'current': 999.99, 'min': 1000.0, 'max': 4000.0}, {'current': 1404.417, 'min': 1000.0, 'max': 4000.0}, {'current': 1006.417, 'min': 1000.0, 'max': 4000.0}, {'current': 1525.018, 'min': 1000.0, 'max': 4000.0}, {'current': 1200.874, 'min': 1000.0, 'max': 4000.0}, {'current': 1000.556, 'min': 1000.0, 'max': 4000.0}, {'current': 1000.001, 'min': 1000.0, 'max': 4000.0}, {'current': 1003.1, 'min': 1000.0, 'max': 4000.0}, {'current': 1006.194, 'min': 1000.0, 'max': 4000.0}, {'current': 999.997, 'min': 1000.0, 'max': 4000.0}, {'current': 1504.437, 'min': 1000.0, 'max': 4000.0}, {'current': 1308.935, 'min': 1000.0, 'max': 4000.0}, {'current': 1005.385, 'min': 1000.0, 'max': 4000.0}, {'current': 1003.726, 'min': 1000.0, 'max': 4000.0}, {'current': 1000.018, 'min': 1000.0, 'max': 4000.0}, {'current': 999.999, 'min': 1000.0, 'max': 4000.0}, {'current': 3018.89, 'min': 1000.0, 'max': 4000.0}, {'current': 1005.26, 'min': 1000.0, 'max': 4000.0}, {'current': 1319.682, 'min': 1000.0, 'max': 4000.0}, {'current': 1104.723, 'min': 1000.0, 'max': 4000.0}, {'current': 1005.172, 'min': 1000.0, 'max': 4000.0}, {'current': 1003.527, 'min': 1000.0, 'max': 4000.0}, {'current': 999.986, 'min': 1000.0, 'max': 4000.0}, {'current': 1357.038, 'min': 1000.0, 'max': 4000.0}, {'current': 1000.003, 'min': 1000.0, 'max': 4000.0}, {'current': 1164.476, 'min': 1000.0, 'max': 4000.0}, {'current': 1327.551, 'min': 1000.0, 'max': 4000.0}, {'current': 1177.744, 'min': 1000.0, 'max': 4000.0}, {'current': 1000.561, 'min': 1000.0, 'max': 4000.0}, {'current': 1315.663, 'min': 1000.0, 'max': 4000.0}, {'current': 1355.862, 'min': 1000.0, 'max': 4000.0}, {'current': 1000.078, 'min': 1000.0, 'max': 4000.0}, {'current': 1008.032, 'min': 1000.0, 'max': 4000.0}, {'current': 1000.637, 'min': 1000.0, 'max': 4000.0}, {'current': 1344.143, 'min': 1000.0, 'max': 4000.0}, {'current': 1314.664, 'min': 1000.0, 'max': 4000.0}, {'current': 1000.662, 'min': 1000.0, 'max': 4000.0}, {'current': 1000.471, 'min': 1000.0, 'max': 4000.0}, {'current': 3161.899, 'min': 1000.0, 'max': 4000.0}, {'current': 1160.106, 'min': 1000.0, 'max': 4000.0}, {'current': 1316.706, 'min': 1000.0, 'max': 4000.0}, {'current': 999.873, 'min': 1000.0, 'max': 4000.0}, {'current': 1001.067, 'min': 1000.0, 'max': 4000.0}, {'current': 1000.342, 'min': 1000.0, 'max': 4000.0}, {'current': 1000.222, 'min': 1000.0, 'max': 4000.0}, {'current': 997.878, 'min': 1000.0, 'max': 4000.0}, {'current': 1074.419, 'min': 1000.0, 'max': 4000.0}, {'current': 1067.65, 'min': 1000.0, 'max': 4000.0}, {'current': 1000.079, 'min': 1000.0, 'max': 4000.0}], 'disk': {'/': {'total': 878.2507591247559, 'used': 771.6865081787109}}, 'gpu': 'NVIDIA A100 80GB PCIe', 'gpu_count': 8, 'gpu_devices': [{'name': 'NVIDIA A100 80GB PCIe', 'memory_total': 85899345920}, {'name': 'NVIDIA A100 80GB PCIe', 'memory_total': 85899345920}, {'name': 'NVIDIA A100 80GB PCIe', 'memory_total': 85899345920}, {'name': 'NVIDIA A100 80GB PCIe', 'memory_total': 85899345920}, {'name': 'NVIDIA A100 80GB PCIe', 'memory_total': 85899345920}, {'name': 'NVIDIA A100 80GB PCIe', 'memory_total': 85899345920}, {'name': 'NVIDIA A100 80GB PCIe', 'memory_total': 85899345920}, {'name': 'NVIDIA A100 80GB PCIe', 'memory_total': 85899345920}], 'memory': {'total': 1007.5172653198242}} 2023-10-31 16:13:28,558 INFO HandlerThread:1081067 [system_monitor.py:probe():224] Finished collecting system info 2023-10-31 16:13:28,558 INFO HandlerThread:1081067 [system_monitor.py:probe():227] Publishing system info 2023-10-31 16:13:28,558 DEBUG HandlerThread:1081067 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment 2023-10-31 16:13:28,559 DEBUG HandlerThread:1081067 [system_info.py:_save_pip():68] Saving pip packages done 2023-10-31 16:13:28,559 DEBUG HandlerThread:1081067 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment 2023-10-31 16:13:29,265 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/requirements.txt 2023-10-31 16:13:29,265 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/conda-environment.yaml 2023-10-31 16:13:31,507 DEBUG HandlerThread:1081067 [system_info.py:_save_conda():87] Saving conda packages done 2023-10-31 16:13:31,507 DEBUG HandlerThread:1081067 [system_info.py:_save_code():90] Saving code 2023-10-31 16:13:31,512 DEBUG HandlerThread:1081067 [system_info.py:_save_code():111] Saving code done 2023-10-31 16:13:31,512 DEBUG HandlerThread:1081067 [system_info.py:_save_patches():128] Saving git patches 2023-10-31 16:13:31,542 DEBUG HandlerThread:1081067 [system_info.py:_save_patches():170] Saving git patches done 2023-10-31 16:13:31,544 INFO HandlerThread:1081067 [system_monitor.py:probe():229] Finished publishing system info 2023-10-31 16:13:31,549 DEBUG SenderThread:1081067 [sender.py:send():380] send: files 2023-10-31 16:13:31,549 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-metadata.json with policy now 2023-10-31 16:13:31,549 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file code/examples/dreambooth/train_dreambooth_lora.py with policy now 2023-10-31 16:13:31,550 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file diff.patch with policy now 2023-10-31 16:13:31,563 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: stop_status 2023-10-31 16:13:31,564 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: internal_messages 2023-10-31 16:13:31,564 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: stop_status 2023-10-31 16:13:31,933 DEBUG SenderThread:1081067 [sender.py:send():380] send: telemetry 2023-10-31 16:13:31,934 DEBUG SenderThread:1081067 [sender.py:send():380] send: config 2023-10-31 16:13:32,265 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/conda-environment.yaml 2023-10-31 16:13:32,265 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/diff.patch 2023-10-31 16:13:32,266 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:13:32,266 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/code/examples/dreambooth/train_dreambooth_lora.py 2023-10-31 16:13:32,266 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-metadata.json 2023-10-31 16:13:32,267 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/code/examples/dreambooth 2023-10-31 16:13:32,267 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/code 2023-10-31 16:13:32,267 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/code/examples 2023-10-31 16:13:32,536 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:13:32,617 INFO wandb-upload_0:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/n2tlykkb-wandb-metadata.json 2023-10-31 16:13:32,723 INFO wandb-upload_2:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/g8wqkyct-diff.patch 2023-10-31 16:13:32,781 INFO wandb-upload_1:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/nca7sh7b-code/examples/dreambooth/train_dreambooth_lora.py 2023-10-31 16:13:32,900 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:13:32,900 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:13:32,901 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:13:32,901 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:13:32,904 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:13:33,241 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:13:33,242 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:13:33,243 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:13:33,243 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:13:33,266 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:13:33,573 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:13:33,574 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:13:33,574 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:13:33,574 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:13:33,757 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:13:33,758 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:13:33,758 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:13:33,758 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:13:34,266 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:13:34,267 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:13:36,268 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:13:38,064 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:13:42,271 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:13:43,808 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:13:46,560 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: stop_status 2023-10-31 16:13:46,561 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: stop_status 2023-10-31 16:13:46,561 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: internal_messages 2023-10-31 16:13:48,861 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:13:53,862 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:13:58,829 DEBUG SenderThread:1081067 [sender.py:send():380] send: files 2023-10-31 16:13:58,829 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file media/images/validation_5_41e961890c35f4bf255b.png with policy now 2023-10-31 16:13:58,870 DEBUG SenderThread:1081067 [sender.py:send():380] send: files 2023-10-31 16:13:58,870 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file media/images/validation_5_5704b95a281bf598e7e6.png with policy now 2023-10-31 16:13:58,876 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:13:58,927 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:13:59,149 DEBUG SenderThread:1081067 [sender.py:send():380] send: files 2023-10-31 16:13:59,149 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file media/images/validation_5_ee3cb3e213e1f623716d.png with policy now 2023-10-31 16:13:59,149 DEBUG SenderThread:1081067 [sender.py:send():380] send: files 2023-10-31 16:13:59,149 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file media/images/validation_5_14744afbcade8fa19a30.png with policy now 2023-10-31 16:13:59,150 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:13:59,150 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:13:59,152 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:13:59,283 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:13:59,284 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/config.yaml 2023-10-31 16:13:59,284 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/media/images/validation_5_ee3cb3e213e1f623716d.png 2023-10-31 16:13:59,284 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/media/images/validation_5_41e961890c35f4bf255b.png 2023-10-31 16:13:59,284 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/media/images/validation_5_5704b95a281bf598e7e6.png 2023-10-31 16:13:59,284 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/media/images/validation_5_14744afbcade8fa19a30.png 2023-10-31 16:13:59,285 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/media/images 2023-10-31 16:13:59,285 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/media 2023-10-31 16:13:59,423 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:13:59,772 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:13:59,773 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:13:59,773 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:13:59,773 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:00,046 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:00,048 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:00,048 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:00,049 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:00,285 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:00,285 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:00,364 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:00,365 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:00,365 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:00,366 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:00,540 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:00,542 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:00,542 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:00,544 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:00,956 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:00,957 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:00,957 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:00,957 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:01,286 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:01,294 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:01,295 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:01,296 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:01,300 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:01,560 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: stop_status 2023-10-31 16:14:01,561 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: internal_messages 2023-10-31 16:14:01,561 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: stop_status 2023-10-31 16:14:01,640 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:01,796 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:01,797 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:01,797 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:01,967 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:01,968 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:01,968 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:01,969 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:02,161 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:02,163 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:02,163 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:02,164 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:02,286 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:02,287 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:02,637 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:02,638 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:02,638 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:02,638 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:02,907 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:02,908 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:02,908 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:02,909 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:03,238 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:03,238 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:03,239 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:03,239 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:03,287 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:03,581 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:03,582 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:03,582 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:03,584 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:03,770 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:03,771 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:03,771 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:03,772 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:04,172 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:14:04,180 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:04,181 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:04,181 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:04,182 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:04,288 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:04,288 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:04,499 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:04,500 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:04,500 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:04,500 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:04,834 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:04,835 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:04,836 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:04,837 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:05,153 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:05,155 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:05,155 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:05,156 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:05,289 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:05,336 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:05,336 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:05,336 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:05,337 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:05,819 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:05,820 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:05,820 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:05,821 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:06,147 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:06,149 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:06,149 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:06,150 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:06,289 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:06,290 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:06,469 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:06,470 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:06,470 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:06,471 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:06,731 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:06,732 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:06,733 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:06,733 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:06,932 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:06,933 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:06,933 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:06,933 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:07,290 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:07,398 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:07,398 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:07,398 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:07,399 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:07,724 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:07,726 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:07,726 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:07,727 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:08,048 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:08,049 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:08,049 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:08,049 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:08,291 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:08,292 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:08,308 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:08,310 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:08,310 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:08,310 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:08,489 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:08,490 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:08,490 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:08,490 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:08,902 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:08,904 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:08,905 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:08,905 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:09,204 INFO wandb-upload_0:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/1cte6zo4-media/images/validation_5_41e961890c35f4bf255b.png 2023-10-31 16:14:09,222 INFO wandb-upload_1:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/n360dx1d-media/images/validation_5_5704b95a281bf598e7e6.png 2023-10-31 16:14:09,236 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:09,237 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:14:09,238 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:09,239 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:09,239 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:09,292 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:09,554 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:09,554 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:09,555 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:09,555 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:09,885 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:09,886 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:09,887 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:09,887 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:10,067 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:10,068 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:10,069 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:10,069 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:10,196 INFO wandb-upload_2:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/wn8talg7-media/images/validation_5_ee3cb3e213e1f623716d.png 2023-10-31 16:14:10,292 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:10,293 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:10,561 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:10,562 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:10,562 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:10,563 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:10,894 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:10,895 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:10,895 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:10,895 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:11,198 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:11,198 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:11,199 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:11,199 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:11,293 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:11,459 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:11,461 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:11,461 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:11,462 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:11,646 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:11,647 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:11,648 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:11,648 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:12,066 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:12,067 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:12,068 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:12,068 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:12,294 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:12,294 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:12,392 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:12,393 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:12,393 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:12,393 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:12,734 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:12,735 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:12,735 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:12,736 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:13,057 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:13,057 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:13,058 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:13,058 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:13,245 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:13,246 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:13,247 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:13,247 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:13,294 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:13,649 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:13,650 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:13,650 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:13,650 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:13,981 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:13,982 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:13,982 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:13,983 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:14,290 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:14:14,295 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:14,295 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:14,296 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:14,297 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:14,297 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:14,297 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:14,622 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:14,623 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:14,623 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:14,624 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:14,808 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:14,809 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:14,809 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:14,810 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:15,212 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:15,213 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:15,213 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:15,214 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:15,296 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:15,525 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:15,526 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:15,526 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:15,527 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:15,883 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:15,883 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:15,884 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:15,884 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:16,219 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:16,219 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:16,220 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:16,220 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:16,297 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:16,297 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:16,390 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:16,391 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:16,391 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:16,392 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:16,560 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: stop_status 2023-10-31 16:14:16,561 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: internal_messages 2023-10-31 16:14:16,561 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: stop_status 2023-10-31 16:14:16,862 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:16,862 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:16,863 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:16,863 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:17,121 INFO wandb-upload_0:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/pimbb122-media/images/validation_5_41e961890c35f4bf255b.png 2023-10-31 16:14:17,173 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:17,174 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:17,174 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:17,175 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:17,297 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:17,506 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:17,507 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:17,507 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:17,507 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:17,767 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:17,769 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:17,769 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:17,770 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:17,956 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:17,957 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:17,957 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:17,958 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:18,298 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:18,299 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:18,373 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:18,374 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:18,374 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:18,374 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:18,709 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:18,710 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:18,710 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:18,710 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:18,814 INFO wandb-upload_3:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/za918r3k-media/images/validation_5_14744afbcade8fa19a30.png 2023-10-31 16:14:19,031 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:19,032 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:19,032 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:19,033 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:19,299 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:19,366 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:19,368 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:14:19,368 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:19,369 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:19,370 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:19,544 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:19,546 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:19,546 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:19,546 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:20,021 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:20,022 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:20,022 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:20,022 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:20,300 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:20,300 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:20,342 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:20,343 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:20,343 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:20,344 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:20,660 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:20,662 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:20,662 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:20,662 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:20,932 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:20,933 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:20,933 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:20,934 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:21,109 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:21,111 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:21,111 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:21,112 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:21,300 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:21,566 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:21,567 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:21,568 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:21,568 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:21,816 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:21,818 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:21,818 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:21,819 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:22,151 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:22,152 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:22,153 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:22,153 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:22,301 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:22,301 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:22,470 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:22,472 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:22,472 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:22,472 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:22,666 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:22,667 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:22,667 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:22,667 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:23,137 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:23,138 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:23,138 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:23,139 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:23,302 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:23,465 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:23,466 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:23,466 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:23,467 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:23,718 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:23,719 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:23,719 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:23,720 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:23,878 INFO wandb-upload_1:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/peimdn30-media/images/validation_5_5704b95a281bf598e7e6.png 2023-10-31 16:14:24,042 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:24,043 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:24,043 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:24,047 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:24,227 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:24,228 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:24,228 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:24,229 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:24,302 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:24,303 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:24,626 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:24,628 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:14:24,628 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:24,628 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:24,629 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:24,936 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:24,937 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:24,937 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:24,938 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:25,277 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:25,278 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:25,279 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:25,279 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:25,303 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:25,604 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:25,606 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:25,606 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:25,607 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:25,793 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:25,795 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:25,795 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:25,796 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:26,247 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:26,248 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:26,249 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:26,249 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:26,304 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:26,304 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:26,518 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:26,518 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:26,519 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:26,519 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:26,848 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:26,849 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:26,849 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:26,850 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:27,164 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:27,165 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:27,165 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:27,166 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:27,304 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:27,354 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:27,355 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:27,355 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:27,355 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:27,799 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:27,800 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:27,801 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:27,802 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:28,073 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:28,075 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:28,075 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:28,076 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:28,305 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:28,306 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:28,412 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:28,414 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:28,414 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:28,415 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:28,488 DEBUG SystemMonitor:1081067 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2023-10-31 16:14:28,494 DEBUG SenderThread:1081067 [sender.py:send():380] send: stats 2023-10-31 16:14:28,723 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:28,724 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:28,724 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:28,725 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:28,979 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:28,980 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:28,980 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:28,980 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:29,306 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:29,376 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:29,378 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:29,378 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:29,378 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:29,704 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:29,705 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:14:29,705 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:29,706 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:29,706 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:30,039 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:30,041 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:30,041 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:30,042 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:30,307 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:30,308 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:30,366 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:30,368 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:30,368 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:30,369 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:30,543 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:30,545 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:30,545 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:30,546 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:30,622 INFO wandb-upload_2:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/3i7tn3qn-media/images/validation_5_ee3cb3e213e1f623716d.png 2023-10-31 16:14:31,020 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:31,022 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:31,022 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:31,023 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:31,308 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:31,347 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:31,348 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:31,349 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:31,349 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:31,560 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: stop_status 2023-10-31 16:14:31,561 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: stop_status 2023-10-31 16:14:31,563 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: internal_messages 2023-10-31 16:14:31,653 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:31,787 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:31,787 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:31,788 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:31,915 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:31,916 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:31,916 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:31,916 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:32,094 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:32,095 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:32,096 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:32,096 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:32,309 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:32,310 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:32,549 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:32,550 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:32,550 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:32,550 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:32,895 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:32,896 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:32,897 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:32,897 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:33,227 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:33,228 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:33,228 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:33,229 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:33,311 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:33,485 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:33,486 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:33,486 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:33,487 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:33,666 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:33,667 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:33,668 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:33,668 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:34,149 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:34,150 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:34,150 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:34,151 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:34,312 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:34,313 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:34,470 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:34,472 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:34,472 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:34,472 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:34,560 INFO wandb-upload_3:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/aumzzvfm-media/images/validation_5_14744afbcade8fa19a30.png 2023-10-31 16:14:34,815 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:34,816 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:14:34,816 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:34,817 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:34,817 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:35,060 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:35,061 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:35,061 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:35,061 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:35,240 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:35,241 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:35,241 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:35,241 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:35,313 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:35,664 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:35,665 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:35,665 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:35,665 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:35,959 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:35,961 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:35,961 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:35,962 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:36,279 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:36,280 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:36,280 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:36,281 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:36,314 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:36,314 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:36,594 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:36,595 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:36,595 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:36,596 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:36,794 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:36,795 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:36,795 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:36,795 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:37,218 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:37,219 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:37,220 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:37,220 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:37,315 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:37,522 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:37,523 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:37,523 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:37,523 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:37,837 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:37,838 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:37,838 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:37,839 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:38,171 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:38,172 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:38,173 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:38,173 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:38,316 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:38,317 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:38,345 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:38,347 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:38,347 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:38,348 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:38,810 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:38,811 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:38,812 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:38,812 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:39,139 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:39,140 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:39,140 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:39,141 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:39,317 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:39,388 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:39,389 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:39,389 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:39,390 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:39,710 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:39,711 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:39,712 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:39,712 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:39,884 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:14:39,884 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:39,885 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:39,886 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:39,886 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:40,318 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:40,318 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:40,352 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:40,354 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:40,354 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:40,354 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:40,612 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:40,613 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:40,613 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:40,614 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:40,913 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:40,914 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:40,915 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:40,916 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:41,319 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:41,384 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:41,386 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:41,386 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:41,387 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:41,567 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:41,568 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:41,568 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:41,569 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:42,056 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:42,057 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:42,058 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:42,058 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:42,313 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:42,315 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:42,315 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:42,316 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:42,325 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:42,325 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:42,636 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:42,637 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:42,637 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:42,638 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:42,954 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:42,955 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:42,955 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:42,955 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:43,130 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:43,131 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:43,131 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:43,132 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:43,326 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:43,511 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:43,512 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:43,512 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:43,513 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:43,852 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:43,854 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:43,854 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:43,855 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:44,183 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:44,185 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:44,185 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:44,186 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:44,326 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:44,327 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:44,503 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:44,505 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:44,505 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:44,505 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:44,699 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:44,701 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:44,701 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:44,702 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:45,174 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:14:45,175 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:45,176 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:45,176 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:45,177 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:45,327 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:45,490 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:45,491 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:45,491 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:45,492 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:45,755 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:45,757 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:45,757 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:45,758 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:46,071 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:46,073 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:46,073 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:46,076 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:46,245 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:46,246 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:46,247 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:46,247 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:46,328 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:46,329 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:46,561 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: stop_status 2023-10-31 16:14:46,561 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: internal_messages 2023-10-31 16:14:46,562 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: stop_status 2023-10-31 16:14:46,660 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:46,849 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:46,849 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:46,850 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:46,994 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:46,995 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:46,996 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:46,996 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:47,322 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:47,323 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:47,324 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:47,325 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:47,329 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:47,640 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:47,642 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:47,642 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:47,643 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:47,822 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:47,823 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:47,823 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:47,824 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:48,261 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:48,262 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:48,263 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:48,263 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:48,330 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:48,331 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:48,606 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:48,607 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:48,608 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:48,608 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:48,877 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:48,878 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:48,879 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:48,879 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:49,197 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:49,198 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:49,199 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:49,200 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:49,331 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:49,377 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:49,378 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:49,378 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:49,379 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:49,857 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:49,859 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:49,859 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:49,860 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:50,107 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:50,108 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:50,109 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:50,110 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:50,332 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:50,333 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:50,415 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:14:50,416 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:50,417 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:50,417 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:50,418 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:50,758 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:50,759 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:50,759 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:50,760 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:50,947 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:50,948 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:50,949 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:50,949 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:51,333 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:51,399 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:51,401 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:51,401 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:51,402 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:51,743 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:51,744 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:51,744 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:51,745 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:52,006 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:52,007 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:52,008 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:52,008 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:52,334 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:52,334 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:52,339 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:52,340 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:52,341 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:52,341 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:52,526 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:52,527 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:52,528 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:52,528 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:53,000 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:53,001 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:53,001 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:53,002 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:53,270 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:53,271 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:53,272 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:53,272 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:53,335 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:53,623 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:53,624 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:53,624 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:53,625 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:53,942 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:53,944 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:53,944 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:53,945 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:54,142 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:54,144 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:54,144 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:54,145 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:54,335 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:54,336 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:54,542 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:54,543 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:54,543 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:54,544 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:54,868 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:54,869 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:54,869 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:54,870 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:55,227 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:55,228 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:55,228 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:55,229 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:55,336 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:55,573 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:14:55,574 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:55,575 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:55,575 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:55,576 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:55,762 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:55,764 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:55,764 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:55,765 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:56,177 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:56,178 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:56,179 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:56,179 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:56,337 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:56,338 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:56,521 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:56,521 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:56,522 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:56,522 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:56,828 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:56,829 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:56,830 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:56,831 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:57,141 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:57,142 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:57,142 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:57,143 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:57,324 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:57,325 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:57,326 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:57,326 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:57,338 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:57,731 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:57,732 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:57,732 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:57,732 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:58,051 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:58,052 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:58,052 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:58,053 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:58,339 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:58,339 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:14:58,391 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:58,392 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:58,393 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:58,393 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:58,497 DEBUG SenderThread:1081067 [sender.py:send():380] send: stats 2023-10-31 16:14:58,706 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:58,708 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:58,708 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:58,709 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:58,899 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:58,900 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:58,900 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:58,901 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:59,339 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:14:59,396 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:59,397 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:59,398 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:59,399 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:59,665 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:59,667 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:59,667 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:59,668 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:14:59,990 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:14:59,992 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:14:59,992 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:14:59,993 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:00,302 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:00,303 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:00,304 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:00,305 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:00,340 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:00,341 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:15:00,573 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:00,575 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:15:00,575 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:00,575 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:00,576 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:01,030 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:01,031 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:01,032 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:01,032 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:01,279 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:01,280 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:01,281 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:01,281 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:01,341 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:01,561 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: stop_status 2023-10-31 16:15:01,562 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: stop_status 2023-10-31 16:15:01,562 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: internal_messages 2023-10-31 16:15:01,645 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:01,772 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:01,773 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:01,773 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:01,944 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:01,945 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:01,945 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:01,946 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:02,128 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:02,130 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:02,130 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:02,131 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:02,342 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:02,342 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:15:02,623 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:02,624 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:02,624 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:02,625 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:02,936 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:02,937 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:02,938 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:02,938 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:03,200 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:03,202 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:03,202 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:03,203 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:03,342 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:03,512 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:03,513 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:03,513 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:03,514 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:03,698 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:03,699 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:03,700 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:03,700 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:04,165 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:04,166 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:04,167 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:04,167 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:04,343 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:04,344 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:15:04,426 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:04,427 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:04,427 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:04,428 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:04,730 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:04,731 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:04,732 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:04,732 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:05,072 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:05,073 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:05,073 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:05,073 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:05,251 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:05,252 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:05,252 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:05,253 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:05,344 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:05,639 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:15:05,640 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:05,642 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:05,642 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:05,643 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:05,974 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:05,975 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:05,976 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:05,976 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:06,295 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:06,296 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:06,297 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:06,297 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:06,345 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:06,346 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:15:06,613 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:06,614 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:06,615 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:06,615 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:06,805 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:06,807 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:06,807 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:06,808 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:07,204 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:07,206 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:07,206 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:07,207 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:07,346 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:07,523 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:07,524 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:07,525 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:07,525 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:07,869 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:07,870 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:07,871 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:07,871 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:08,179 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:08,180 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:08,181 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:08,181 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:08,347 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:08,347 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:15:08,368 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:08,369 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:08,369 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:08,370 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:08,797 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:08,798 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:08,799 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:08,799 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:09,107 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:09,108 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:09,109 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:09,109 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:09,347 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:09,442 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:09,443 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:09,444 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:09,444 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:09,761 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:09,762 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:09,762 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:09,763 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:09,948 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:09,950 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:09,950 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:09,950 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:10,349 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:10,350 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:15:10,411 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:10,412 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:10,413 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:10,413 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:10,681 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:15:10,681 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:10,683 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:10,683 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:10,683 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:11,002 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:11,003 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:11,004 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:11,004 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:11,337 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:11,339 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:11,339 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:11,340 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:11,350 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:11,519 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:11,520 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:11,521 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:11,522 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:11,993 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:11,994 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:11,994 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:11,994 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:12,331 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:12,333 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:12,333 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:12,334 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:12,350 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:12,351 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:15:12,601 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:12,602 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:12,603 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:12,603 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:12,926 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:12,927 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:12,928 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:12,928 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:13,108 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:13,109 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:13,109 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:13,109 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:13,351 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:13,499 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:13,500 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:13,500 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:13,500 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:13,850 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:13,851 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:13,852 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:13,852 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:14,178 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:14,179 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:14,179 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:14,180 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:14,352 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:14,353 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:15:14,510 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:14,511 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:14,511 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:14,512 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:14,690 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:14,691 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:14,692 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:14,692 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:15,100 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:15,101 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:15,102 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:15,103 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:15,353 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:15,396 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:15,397 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:15,398 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:15,398 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:15,716 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:15:15,717 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:15,718 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:15,718 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:15,719 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:16,056 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:16,057 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:16,058 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:16,058 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:16,229 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:16,230 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:16,230 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:16,231 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:16,354 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:16,354 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:15:16,561 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: stop_status 2023-10-31 16:15:16,562 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: stop_status 2023-10-31 16:15:16,562 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: internal_messages 2023-10-31 16:15:16,636 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:16,814 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:16,815 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:16,815 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:16,967 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:16,968 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:16,968 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:16,969 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:17,294 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:17,295 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:17,296 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:17,296 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:17,355 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:17,598 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:17,599 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:17,599 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:17,600 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:17,781 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:17,783 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:17,784 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:17,784 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:18,355 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:18,355 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:15:20,356 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:15:21,590 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:15:26,591 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:15:28,501 DEBUG SenderThread:1081067 [sender.py:send():380] send: stats 2023-10-31 16:15:31,561 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: stop_status 2023-10-31 16:15:31,562 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: stop_status 2023-10-31 16:15:31,562 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: internal_messages 2023-10-31 16:15:31,834 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:15:36,202 DEBUG SenderThread:1081067 [sender.py:send():380] send: files 2023-10-31 16:15:36,202 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file media/images/validation_255_0a1fec2bd7993ca80414.png with policy now 2023-10-31 16:15:36,204 DEBUG SenderThread:1081067 [sender.py:send():380] send: files 2023-10-31 16:15:36,204 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file media/images/validation_255_18721f418025f2635d64.png with policy now 2023-10-31 16:15:36,211 DEBUG SenderThread:1081067 [sender.py:send():380] send: files 2023-10-31 16:15:36,211 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file media/images/validation_255_c163a2b2c79634b7e2cf.png with policy now 2023-10-31 16:15:36,211 DEBUG SenderThread:1081067 [sender.py:send():380] send: files 2023-10-31 16:15:36,211 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file media/images/validation_255_8ae29b8050025fdae4a1.png with policy now 2023-10-31 16:15:36,294 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:36,296 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:36,296 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:36,297 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:36,367 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:36,368 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/media/images/validation_255_18721f418025f2635d64.png 2023-10-31 16:15:36,368 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/media/images/validation_255_0a1fec2bd7993ca80414.png 2023-10-31 16:15:36,368 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/media/images/validation_255_8ae29b8050025fdae4a1.png 2023-10-31 16:15:36,369 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/media/images/validation_255_c163a2b2c79634b7e2cf.png 2023-10-31 16:15:36,369 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/media/images 2023-10-31 16:15:36,778 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:37,116 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:37,117 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:15:37,117 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:37,117 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:37,118 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:37,368 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:37,444 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:37,445 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:37,445 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:37,445 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:37,759 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:37,759 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:37,760 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:37,760 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:37,951 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:37,952 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:37,952 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:37,953 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:38,368 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:38,369 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:15:38,451 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:38,452 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:38,453 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:38,453 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:38,704 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:38,705 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:38,705 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:38,708 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:38,725 INFO wandb-upload_2:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/0br636ye-media/images/validation_255_18721f418025f2635d64.png 2023-10-31 16:15:39,042 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:39,042 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:39,043 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:39,043 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:39,365 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:39,367 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:39,367 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:39,368 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:39,370 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:39,565 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:39,566 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:39,566 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:39,567 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:39,732 INFO wandb-upload_0:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/25duhgxm-media/images/validation_255_c163a2b2c79634b7e2cf.png 2023-10-31 16:15:40,037 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:40,038 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:40,038 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:40,039 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:40,297 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:40,298 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:40,298 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:40,299 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:40,371 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:40,371 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:15:40,636 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:40,637 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:40,637 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:40,637 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:40,976 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:40,977 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:40,977 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:40,978 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:41,172 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:41,172 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:41,173 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:41,173 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:41,371 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:41,640 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:41,641 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:41,641 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:41,642 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:41,993 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:41,994 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:41,994 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:41,994 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:42,310 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:15:42,311 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:42,311 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:42,312 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:42,312 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:42,372 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:42,372 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:15:42,571 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:42,572 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:42,573 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:42,573 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:42,756 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:42,757 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:42,758 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:42,758 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:43,233 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:43,234 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:43,234 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:43,235 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:43,373 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:43,579 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:43,580 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:43,580 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:43,580 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:43,841 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:43,842 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:43,842 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:43,843 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:43,866 INFO wandb-upload_2:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/fbqm60yi-media/images/validation_255_18721f418025f2635d64.png 2023-10-31 16:15:44,156 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:44,156 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:44,157 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:44,157 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:44,340 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:44,341 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:44,341 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:44,342 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:44,374 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:44,374 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:15:44,589 INFO wandb-upload_1:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/8tmpwoa1-media/images/validation_255_0a1fec2bd7993ca80414.png 2023-10-31 16:15:44,819 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:44,821 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:44,821 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:44,821 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:45,073 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:45,074 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:45,075 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:45,076 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:45,157 INFO wandb-upload_3:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/noh2dhtz-media/images/validation_255_8ae29b8050025fdae4a1.png 2023-10-31 16:15:45,376 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:45,423 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:45,424 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:45,424 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:45,425 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:45,737 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:45,737 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:45,738 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:45,738 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:45,931 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:45,932 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:45,932 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:45,932 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:46,377 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:46,377 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:15:46,417 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:46,418 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:46,418 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:46,419 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:46,561 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: stop_status 2023-10-31 16:15:46,562 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: stop_status 2023-10-31 16:15:46,562 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: internal_messages 2023-10-31 16:15:46,769 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:46,856 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:46,856 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:46,857 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:47,063 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:47,064 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:47,064 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:47,065 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:47,320 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:15:47,320 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:47,321 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:47,322 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:47,322 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:47,378 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:47,512 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:47,513 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:47,513 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:47,514 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:47,919 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:47,920 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:47,920 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:47,921 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:48,242 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:48,243 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:48,243 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:48,244 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:48,378 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:48,379 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:15:48,578 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:48,580 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:48,580 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:48,581 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:48,893 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:48,894 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:48,895 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:48,898 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:49,077 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:49,078 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:49,078 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:49,079 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:49,379 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:49,539 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:49,540 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:49,540 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:49,540 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:49,863 INFO wandb-upload_0:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/h58nyib8-media/images/validation_255_c163a2b2c79634b7e2cf.png 2023-10-31 16:15:49,870 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:49,870 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:49,871 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:49,871 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:50,133 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:50,134 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:50,134 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:50,135 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:50,380 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:50,380 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:15:50,454 INFO wandb-upload_1:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/jzx6h16t-media/images/validation_255_0a1fec2bd7993ca80414.png 2023-10-31 16:15:50,477 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:50,478 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:50,478 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:50,479 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:50,752 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:50,752 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:50,753 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:50,753 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:51,178 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:51,179 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:51,179 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:51,180 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:51,381 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:51,493 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:51,495 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:51,495 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:51,496 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:51,809 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:51,810 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:51,810 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:51,810 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:52,146 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:52,147 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:52,147 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:52,148 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:52,322 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:52,324 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:15:52,324 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:52,324 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:52,325 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:52,383 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:52,383 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:15:52,801 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:52,801 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:52,802 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:52,802 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:53,143 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:53,143 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:53,144 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:53,144 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:53,383 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:53,396 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:53,398 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:53,398 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:53,398 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:53,713 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:53,714 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:53,714 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:53,715 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:53,909 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:53,910 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:53,910 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:53,910 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:54,384 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:54,384 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:15:54,402 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:54,403 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:54,403 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:54,403 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:54,663 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:54,664 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:54,665 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:54,665 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:54,989 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:54,990 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:54,991 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:54,992 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:55,312 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:55,313 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:55,313 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:55,314 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:55,385 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:55,494 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:55,495 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:55,495 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:55,496 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:55,572 INFO wandb-upload_3:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/yap2e0gm-media/images/validation_255_8ae29b8050025fdae4a1.png 2023-10-31 16:15:55,992 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:55,993 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:55,993 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:55,994 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:56,253 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:56,254 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:56,254 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:56,255 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:56,385 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:56,386 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:15:56,579 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:56,580 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:56,580 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:56,581 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:56,898 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:56,900 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:56,900 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:56,900 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:57,077 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:57,078 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:57,078 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:57,079 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:57,386 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:57,560 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:15:57,560 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:57,561 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:57,562 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:57,562 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:57,884 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:57,885 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:57,885 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:57,886 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:58,204 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:58,206 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:58,206 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:58,206 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:58,387 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:58,387 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:15:58,471 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:58,472 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:58,472 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:58,472 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:58,504 DEBUG SenderThread:1081067 [sender.py:send():380] send: stats 2023-10-31 16:15:58,664 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:58,664 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:58,664 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:58,665 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:59,133 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:59,134 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:59,134 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:59,138 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:59,387 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:15:59,439 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:59,441 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:59,441 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:59,442 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:59,689 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:15:59,690 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:15:59,691 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:15:59,691 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:15:59,999 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:00,000 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:00,001 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:00,001 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:00,170 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:00,171 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:00,172 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:00,172 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:00,388 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:00,388 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:00,630 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:00,631 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:00,631 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:00,632 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:00,939 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:00,940 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:00,940 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:00,941 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:01,189 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:01,190 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:01,191 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:01,192 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:01,389 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:01,482 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:01,483 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:01,484 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:01,485 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:01,561 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: stop_status 2023-10-31 16:16:01,562 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: internal_messages 2023-10-31 16:16:01,562 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: stop_status 2023-10-31 16:16:01,658 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:01,868 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:01,869 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:01,870 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:02,102 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:02,104 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:02,104 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:02,105 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:02,345 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:02,347 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:02,347 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:02,348 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:02,390 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:02,390 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:02,665 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:16:02,666 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:02,667 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:02,668 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:02,668 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:02,968 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:02,969 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:02,970 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:02,970 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:03,143 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:03,145 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:03,145 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:03,146 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:03,390 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:03,529 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:03,531 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:03,531 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:03,532 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:03,833 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:03,834 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:03,835 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:03,835 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:04,151 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:04,153 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:04,153 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:04,154 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:04,391 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:04,392 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:04,464 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:04,465 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:04,466 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:04,467 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:04,653 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:04,654 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:04,655 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:04,655 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:05,051 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:05,052 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:05,053 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:05,053 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:05,366 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:05,367 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:05,367 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:05,368 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:05,392 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:05,679 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:05,681 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:05,681 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:05,682 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:06,006 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:06,007 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:06,008 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:06,008 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:06,182 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:06,183 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:06,184 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:06,184 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:06,393 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:06,393 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:06,651 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:06,652 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:06,652 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:06,652 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:06,961 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:06,962 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:06,962 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:06,963 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:07,293 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:07,294 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:07,295 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:07,295 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:07,394 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:07,613 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:07,613 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:07,614 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:07,614 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:07,786 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:16:07,788 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:07,789 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:07,790 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:07,790 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:08,303 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:08,304 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:08,304 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:08,305 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:08,394 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:08,395 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:08,562 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:08,563 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:08,563 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:08,564 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:08,905 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:08,906 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:08,906 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:08,907 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:09,227 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:09,227 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:09,228 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:09,229 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:09,395 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:09,402 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:09,402 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:09,403 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:09,403 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:09,811 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:09,811 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:09,812 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:09,812 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:10,125 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:10,126 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:10,126 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:10,127 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:10,396 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:10,396 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:10,472 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:10,473 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:10,473 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:10,473 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:10,790 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:10,791 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:10,791 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:10,791 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:10,977 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:10,978 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:10,978 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:10,979 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:11,397 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:11,516 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:11,518 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:11,518 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:11,519 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:11,868 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:11,869 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:11,869 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:11,870 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:12,166 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:12,167 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:12,167 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:12,168 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:12,398 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:12,399 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:12,515 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:12,516 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:12,516 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:12,516 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:12,700 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:12,701 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:12,701 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:12,702 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:13,154 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:16:13,154 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:13,155 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:13,155 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:13,156 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:13,400 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:13,546 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:13,547 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:13,547 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:13,548 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:13,900 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:13,901 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:13,902 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:13,902 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:14,273 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:14,275 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:14,275 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:14,276 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:14,400 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:14,401 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:14,481 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:14,482 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:14,482 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:14,483 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:15,002 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:15,003 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:15,003 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:15,004 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:15,354 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:15,355 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:15,355 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:15,356 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:15,401 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:15,710 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:15,711 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:15,711 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:15,711 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:15,962 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:15,962 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:15,963 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:15,963 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:16,182 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:16,184 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:16,184 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:16,185 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:16,402 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:16,402 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:16,562 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: stop_status 2023-10-31 16:16:16,562 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: stop_status 2023-10-31 16:16:16,564 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: internal_messages 2023-10-31 16:16:16,730 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:16,822 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:16,822 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:16,823 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:17,007 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:17,008 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:17,008 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:17,009 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:17,392 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:17,393 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:17,393 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:17,393 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:17,402 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:17,783 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:17,784 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:17,784 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:17,784 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:17,976 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:17,978 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:17,978 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:17,979 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:18,403 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:18,403 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:18,524 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:18,525 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:16:18,526 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:18,526 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:18,526 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:18,920 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:18,920 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:18,921 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:18,922 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:19,303 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:19,304 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:19,304 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:19,306 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:19,404 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:19,582 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:19,583 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:19,584 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:19,584 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:19,779 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:19,781 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:19,781 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:19,781 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:20,332 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:20,333 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:20,333 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:20,334 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:20,405 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:20,405 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:20,604 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:20,605 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:20,605 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:20,606 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:20,985 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:20,986 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:20,986 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:20,987 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:21,368 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:21,369 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:21,370 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:21,371 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:21,406 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:21,570 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:21,571 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:21,572 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:21,573 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:22,134 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:22,135 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:22,136 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:22,136 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:22,406 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:22,407 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:22,416 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:22,417 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:22,417 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:22,417 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:22,754 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:22,755 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:22,755 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:22,756 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:23,102 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:23,103 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:23,103 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:23,104 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:23,359 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:23,360 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:23,361 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:23,361 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:23,407 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:23,777 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:16:23,777 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:23,778 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:23,778 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:23,779 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:24,134 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:24,135 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:24,136 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:24,136 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:24,408 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:24,409 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:24,467 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:24,468 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:24,468 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:24,468 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:24,823 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:24,824 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:24,824 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:24,825 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:25,015 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:25,015 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:25,016 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:25,016 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:25,410 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:25,448 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:25,449 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:25,450 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:25,450 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:25,805 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:25,806 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:25,806 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:25,807 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:26,127 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:26,128 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:26,128 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:26,128 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:26,410 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:26,411 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:26,477 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:26,478 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:26,478 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:26,478 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:26,659 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:26,660 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:26,660 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:26,661 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:27,132 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:27,133 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:27,134 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:27,134 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:27,411 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:27,478 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:27,479 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:27,479 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:27,479 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:27,804 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:27,805 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:27,805 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:27,806 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:28,073 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:28,074 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:28,075 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:28,075 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:28,260 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:28,261 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:28,261 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:28,262 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:28,412 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:28,412 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:28,507 DEBUG SenderThread:1081067 [sender.py:send():380] send: stats 2023-10-31 16:16:28,688 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:28,689 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:28,690 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:28,690 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:29,025 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:16:29,025 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:29,026 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:29,026 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:29,027 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:29,354 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:29,355 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:29,355 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:29,359 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:29,413 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:29,687 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:29,688 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:29,688 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:29,689 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:29,867 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:29,868 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:29,868 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:29,869 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:30,375 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:30,376 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:30,376 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:30,377 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:30,413 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:30,414 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:30,704 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:30,704 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:30,705 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:30,705 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:31,048 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:31,049 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:31,049 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:31,050 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:31,314 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:31,314 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:31,314 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:31,315 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:31,414 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:31,509 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:31,510 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:31,510 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:31,510 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:31,561 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: stop_status 2023-10-31 16:16:31,562 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: stop_status 2023-10-31 16:16:31,562 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: internal_messages 2023-10-31 16:16:31,949 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:31,950 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:31,951 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:31,952 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:32,257 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:32,257 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:32,258 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:32,258 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:32,415 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:32,415 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:32,584 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:32,585 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:32,585 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:32,586 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:32,921 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:32,921 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:32,922 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:32,922 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:33,119 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:33,120 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:33,120 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:33,120 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:33,416 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:33,524 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:33,526 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:33,526 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:33,527 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:33,850 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:33,850 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:33,851 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:33,851 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:34,206 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:16:34,206 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:34,207 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:34,207 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:34,207 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:34,417 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:34,417 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:34,539 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:34,540 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:34,541 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:34,541 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:34,713 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:34,714 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:34,715 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:34,715 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:35,145 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:35,146 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:35,146 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:35,146 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:35,417 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:35,492 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:35,493 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:35,493 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:35,494 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:35,809 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:35,810 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:35,810 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:35,810 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:36,131 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:36,132 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:36,132 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:36,132 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:36,317 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:36,318 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:36,318 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:36,319 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:36,418 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:36,419 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:36,784 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:36,785 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:36,785 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:36,786 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:37,117 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:37,119 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:37,119 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:37,119 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:37,419 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:37,453 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:37,454 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:37,454 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:37,455 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:37,718 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:37,719 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:37,719 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:37,720 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:37,904 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:37,906 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:37,906 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:37,906 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:38,416 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:38,416 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:38,416 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:38,417 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:38,420 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:38,420 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:38,689 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:38,689 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:38,690 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:38,690 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:39,020 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:39,022 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:39,022 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:39,023 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:39,365 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:16:39,366 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:39,367 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:39,367 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:39,368 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:39,420 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:39,550 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:39,552 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:39,552 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:39,553 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:40,043 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:40,044 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:40,044 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:40,044 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:40,376 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:40,377 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:40,377 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:40,377 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:40,421 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:40,421 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:40,719 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:40,719 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:40,720 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:40,720 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:40,983 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:40,984 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:40,985 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:40,985 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:41,164 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:41,165 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:41,165 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:41,165 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:41,422 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:41,679 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:41,680 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:41,680 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:41,680 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:41,937 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:41,938 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:41,939 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:41,939 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:42,261 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:42,263 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:42,263 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:42,264 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:42,423 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:42,424 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:42,580 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:42,582 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:42,582 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:42,583 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:42,774 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:42,774 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:42,774 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:42,775 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:43,277 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:43,278 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:43,278 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:43,279 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:43,424 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:43,539 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:43,540 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:43,540 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:43,540 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:43,872 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:43,873 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:43,873 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:43,874 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:44,199 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:44,201 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:44,201 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:44,202 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:44,382 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:44,384 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:16:44,385 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:44,386 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:44,386 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:44,425 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:44,425 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:44,863 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:44,865 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:44,865 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:44,866 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:45,212 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:45,213 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:45,213 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:45,213 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:45,426 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:45,464 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:45,465 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:45,465 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:45,466 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:45,792 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:45,793 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:45,793 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:45,794 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:45,984 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:45,984 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:45,984 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:45,985 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:46,426 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:46,427 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:46,462 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:46,462 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:46,462 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:46,463 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:46,561 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: stop_status 2023-10-31 16:16:46,562 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: stop_status 2023-10-31 16:16:46,562 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: internal_messages 2023-10-31 16:16:46,726 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:46,806 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:46,806 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:46,807 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:47,069 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:47,070 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:47,070 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:47,071 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:47,381 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:47,381 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:47,382 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:47,382 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:47,427 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:47,565 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:47,566 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:47,566 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:47,567 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:48,031 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:48,032 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:48,033 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:48,033 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:48,371 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:48,372 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:48,372 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:48,373 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:48,427 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:48,428 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:48,720 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:48,721 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:48,722 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:48,722 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:48,976 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:48,977 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:48,977 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:48,977 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:49,147 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:49,149 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:49,149 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:49,150 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:49,428 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:49,553 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:49,554 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:16:49,555 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:49,555 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:49,558 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:49,891 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:49,892 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:49,892 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:49,892 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:50,223 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:50,224 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:50,224 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:50,225 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:50,430 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:50,430 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:50,558 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:50,559 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:50,559 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:50,560 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:50,752 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:50,753 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:50,753 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:50,754 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:51,248 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:51,249 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:51,249 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:51,250 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:51,431 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:51,563 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:51,563 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:51,564 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:51,564 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:51,823 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:51,824 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:51,824 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:51,825 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:52,133 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:52,135 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:52,135 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:52,136 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:52,314 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:52,315 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:52,315 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:52,316 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:52,432 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:52,432 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:52,829 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:52,830 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:52,830 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:52,830 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:53,075 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:53,076 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:53,076 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:53,077 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:53,405 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:53,406 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:53,407 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:53,407 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:53,433 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:53,712 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:53,714 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:53,714 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:53,714 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:53,901 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:53,902 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:53,902 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:53,903 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:54,395 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:54,396 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:54,396 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:54,397 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:54,434 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:54,434 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:54,656 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:16:54,658 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:54,659 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:54,660 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:54,660 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:54,995 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:54,997 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:54,998 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:54,998 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:55,318 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:55,319 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:55,320 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:55,320 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:55,435 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:55,575 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:16:55,575 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:16:55,575 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:16:55,576 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:16:56,436 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:16:56,436 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:58,437 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:16:58,510 DEBUG SenderThread:1081067 [sender.py:send():380] send: stats 2023-10-31 16:16:59,726 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:17:00,438 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:17:01,562 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: stop_status 2023-10-31 16:17:01,562 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: internal_messages 2023-10-31 16:17:01,563 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: stop_status 2023-10-31 16:17:02,440 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:17:04,441 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:17:04,832 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:17:06,443 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:17:08,444 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:17:09,945 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:17:10,446 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:17:12,448 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:17:14,450 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:17:15,074 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:17:16,452 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:17:16,562 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: stop_status 2023-10-31 16:17:16,563 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: stop_status 2023-10-31 16:17:16,566 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: internal_messages 2023-10-31 16:17:18,455 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:17:20,017 DEBUG SenderThread:1081067 [sender.py:send():380] send: files 2023-10-31 16:17:20,017 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file media/images/test_500_98dec093c81016dddf63.png with policy now 2023-10-31 16:17:20,019 DEBUG SenderThread:1081067 [sender.py:send():380] send: files 2023-10-31 16:17:20,019 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file media/images/test_500_df3d23f28a4d191e1c20.png with policy now 2023-10-31 16:17:20,024 DEBUG SenderThread:1081067 [sender.py:send():380] send: files 2023-10-31 16:17:20,025 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file media/images/test_500_926cf1c3ebc9fd521a6c.png with policy now 2023-10-31 16:17:20,025 DEBUG SenderThread:1081067 [sender.py:send():380] send: files 2023-10-31 16:17:20,025 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file media/images/test_500_05f539e30de56f4440b6.png with policy now 2023-10-31 16:17:20,117 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: partial_history 2023-10-31 16:17:20,118 DEBUG SenderThread:1081067 [sender.py:send():380] send: history 2023-10-31 16:17:20,118 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: summary_record 2023-10-31 16:17:20,119 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:17:20,119 INFO SenderThread:1081067 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-10-31 16:17:20,457 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/wandb-summary.json 2023-10-31 16:17:20,457 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:17:20,458 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/media/images/test_500_926cf1c3ebc9fd521a6c.png 2023-10-31 16:17:20,458 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/media/images/test_500_05f539e30de56f4440b6.png 2023-10-31 16:17:20,459 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/media/images/test_500_df3d23f28a4d191e1c20.png 2023-10-31 16:17:20,459 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/media/images/test_500_98dec093c81016dddf63.png 2023-10-31 16:17:20,460 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/media/images 2023-10-31 16:17:23,432 INFO wandb-upload_3:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/ikx1bdks-media/images/test_500_05f539e30de56f4440b6.png 2023-10-31 16:17:24,830 INFO wandb-upload_2:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/9twurp93-media/images/test_500_926cf1c3ebc9fd521a6c.png 2023-10-31 16:17:25,025 INFO wandb-upload_0:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/yi78from-media/images/test_500_98dec093c81016dddf63.png 2023-10-31 16:17:25,120 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:17:25,354 INFO wandb-upload_1:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/x6r3u6u6-media/images/test_500_df3d23f28a4d191e1c20.png 2023-10-31 16:17:28,511 DEBUG SenderThread:1081067 [sender.py:send():380] send: stats 2023-10-31 16:17:29,993 INFO wandb-upload_3:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/15rynzqb-media/images/test_500_05f539e30de56f4440b6.png 2023-10-31 16:17:30,513 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:17:31,562 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: stop_status 2023-10-31 16:17:31,563 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: internal_messages 2023-10-31 16:17:31,563 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: stop_status 2023-10-31 16:17:33,220 INFO wandb-upload_2:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/c19fvo2k-media/images/test_500_98dec093c81016dddf63.png 2023-10-31 16:17:35,826 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:17:36,276 INFO wandb-upload_1:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/pgymjdum-media/images/test_500_926cf1c3ebc9fd521a6c.png 2023-10-31 16:17:38,472 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:17:40,051 INFO wandb-upload_3:1081067 [upload_job.py:push():131] Uploaded file /tmp/tmpnqvc7crfwandb/nwz7gi7a-media/images/test_500_df3d23f28a4d191e1c20.png 2023-10-31 16:17:41,374 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:17:42,477 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:17:44,479 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:17:46,410 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:17:46,481 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:17:46,563 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: stop_status 2023-10-31 16:17:46,563 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: internal_messages 2023-10-31 16:17:46,564 DEBUG SenderThread:1081067 [sender.py:send_request():407] send_request: stop_status 2023-10-31 16:17:48,483 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:17:50,485 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:17:51,420 DEBUG HandlerThread:1081067 [handler.py:handle_request():146] handle_request: status_report 2023-10-31 16:17:52,487 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log 2023-10-31 16:17:54,489 INFO Thread-12 :1081067 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/Desktop/jaeho/diffusers/examples/dreambooth/wandb/run-20231031_161327-8441xowv/files/output.log