| 2024-03-28 18:18:36,241 INFO StreamThr :603355 [internal.py:wandb_internal():86] W&B internal server running at pid: 603355, started at: 2024-03-28 18:18:36.240864 |
| 2024-03-28 18:18:36,242 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status |
| 2024-03-28 18:18:36,245 INFO WriterThread:603355 [datastore.py:open_for_write():87] open: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/run-ljeg9p3r.wandb |
| 2024-03-28 18:18:36,245 DEBUG SenderThread:603355 [sender.py:send():382] send: header |
| 2024-03-28 18:18:36,369 DEBUG SenderThread:603355 [sender.py:send():382] send: run |
| 2024-03-28 18:18:36,577 INFO SenderThread:603355 [dir_watcher.py:__init__():211] watching files in: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files |
| 2024-03-28 18:18:36,577 INFO SenderThread:603355 [sender.py:_start_run_threads():1136] run started: ljeg9p3r with start time 1711664316.241124 |
| 2024-03-28 18:18:36,581 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: check_version |
| 2024-03-28 18:18:36,582 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: check_version |
| 2024-03-28 18:18:36,633 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: run_start |
| 2024-03-28 18:18:36,640 DEBUG HandlerThread:603355 [system_info.py:__init__():27] System info init |
| 2024-03-28 18:18:36,640 DEBUG HandlerThread:603355 [system_info.py:__init__():42] System info init done |
| 2024-03-28 18:18:36,641 INFO HandlerThread:603355 [system_monitor.py:start():194] Starting system monitor |
| 2024-03-28 18:18:36,641 INFO SystemMonitor:603355 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2024-03-28 18:18:36,641 INFO HandlerThread:603355 [system_monitor.py:probe():214] Collecting system info |
| 2024-03-28 18:18:36,641 INFO SystemMonitor:603355 [interfaces.py:start():190] Started cpu monitoring |
| 2024-03-28 18:18:36,642 INFO SystemMonitor:603355 [interfaces.py:start():190] Started disk monitoring |
| 2024-03-28 18:18:36,642 INFO SystemMonitor:603355 [interfaces.py:start():190] Started gpu monitoring |
| 2024-03-28 18:18:36,643 INFO SystemMonitor:603355 [interfaces.py:start():190] Started memory monitoring |
| 2024-03-28 18:18:36,645 INFO SystemMonitor:603355 [interfaces.py:start():190] Started network monitoring |
| 2024-03-28 18:18:36,671 DEBUG HandlerThread:603355 [system_info.py:probe():151] Probing system |
| 2024-03-28 18:18:36,675 DEBUG HandlerThread:603355 [system_info.py:_probe_git():136] Probing git |
| 2024-03-28 18:18:36,686 DEBUG HandlerThread:603355 [system_info.py:_probe_git():144] Probing git done |
| 2024-03-28 18:18:36,686 DEBUG HandlerThread:603355 [system_info.py:probe():199] Probing system done |
| 2024-03-28 18:18:36,686 DEBUG HandlerThread:603355 [system_monitor.py:probe():223] {'os': 'Linux-5.14.0-284.40.1.el9_2.x86_64-x86_64-with-glibc2.17', 'python': '3.8.18', 'heartbeatAt': '2024-03-28T22:18:36.671449', 'startedAt': '2024-03-28T22:18:36.234595', 'docker': None, 'cuda': None, 'args': ('--model_name_or_path=openai-community/gpt2', '--report_to=wandb', '--learning_rate=1.41e-5', '--per_device_train_batch_size=32', '--per_device_eval_batch_size=4', '--gradient_accumulation_steps=16', '--output_dir=/scratch/km3888/gcode_peft/sweep/4_16', '--logging_steps=1', '--num_train_epochs=6', '--max_seq_length=512', '--max_steps=20000', '--push_to_hub', '--evaluation_strategy=steps', '--eval_steps=5', '--eval_accumulation_steps=16', '--dataset_name=gcode', '--gradient_checkpointing', '--use_peft', '--lora_r=64', '--lora_alpha=16', '--new_tokenizer=new'), 'state': 'running', 'program': 'examples/scripts/sft.py', 'codePathLocal': 'examples/scripts/sft.py', 'codePath': 'examples/scripts/sft.py', 'git': {'remote': 'https://github.com/huggingface/trl.git', 'commit': 'a46cd84a6405312837f0d0e56fd1cf4d45585770'}, 'email': 'km3888@nyu.edu', 'root': '/home/km3888/trl', 'host': 'ga009.hpc.nyu.edu', 'username': 'km3888', 'executable': '/ext3/miniconda3/bin/python', 'cpu_count': 64, 'cpu_count_logical': 128, 'cpu_freq': {'current': 3.3672968749999956, 'min': 800.0, 'max': 3400.0}, 'cpu_freq_per_core': [{'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.299, 'min': 800.0, 'max': 3400.0}, {'current': 3.299, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.299, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.299, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.299, 'min': 800.0, 'max': 3400.0}, {'current': 3.299, 'min': 800.0, 'max': 3400.0}, {'current': 3.299, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.299, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.299, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.299, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.299, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.3, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}, {'current': 3.4, 'min': 800.0, 'max': 3400.0}], 'disk': {'/': {'total': 52.354671478271484, 'used': 11.097156524658203}}, 'gpu': 'NVIDIA A100-SXM4-80GB', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}], 'memory': {'total': 503.01668548583984}} |
| 2024-03-28 18:18:36,687 INFO HandlerThread:603355 [system_monitor.py:probe():224] Finished collecting system info |
| 2024-03-28 18:18:36,687 INFO HandlerThread:603355 [system_monitor.py:probe():227] Publishing system info |
| 2024-03-28 18:18:36,687 DEBUG HandlerThread:603355 [system_info.py:_save_conda():208] Saving list of conda packages installed into the current environment |
| 2024-03-28 18:18:37,578 INFO Thread-12 :603355 [dir_watcher.py:_on_file_created():271] file/dir created: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/conda-environment.yaml |
| 2024-03-28 18:18:46,205 DEBUG HandlerThread:603355 [system_info.py:_save_conda():220] Saving conda packages done |
| 2024-03-28 18:18:46,206 INFO HandlerThread:603355 [system_monitor.py:probe():229] Finished publishing system info |
| 2024-03-28 18:18:46,211 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:18:46,211 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: keepalive |
| 2024-03-28 18:18:46,212 DEBUG SenderThread:603355 [sender.py:send():382] send: files |
| 2024-03-28 18:18:46,212 INFO SenderThread:603355 [sender.py:_save_file():1403] saving file wandb-metadata.json with policy now |
| 2024-03-28 18:18:46,216 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: python_packages |
| 2024-03-28 18:18:46,216 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: python_packages |
| 2024-03-28 18:18:46,217 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: stop_status |
| 2024-03-28 18:18:46,217 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: stop_status |
| 2024-03-28 18:18:46,219 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: internal_messages |
| 2024-03-28 18:18:46,262 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: partial_history |
| 2024-03-28 18:18:46,310 DEBUG SenderThread:603355 [sender.py:send():382] send: telemetry |
| 2024-03-28 18:18:46,310 DEBUG SenderThread:603355 [sender.py:send():382] send: config |
| 2024-03-28 18:18:46,310 DEBUG SenderThread:603355 [sender.py:send():382] send: metric |
| 2024-03-28 18:18:46,311 DEBUG SenderThread:603355 [sender.py:send():382] send: telemetry |
| 2024-03-28 18:18:46,311 DEBUG SenderThread:603355 [sender.py:send():382] send: metric |
| 2024-03-28 18:18:46,311 WARNING SenderThread:603355 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen) |
| 2024-03-28 18:18:46,311 DEBUG SenderThread:603355 [sender.py:send():382] send: telemetry |
| 2024-03-28 18:18:46,311 DEBUG SenderThread:603355 [sender.py:send():382] send: metric |
| 2024-03-28 18:18:46,311 DEBUG SenderThread:603355 [sender.py:send():382] send: metric |
| 2024-03-28 18:18:46,311 DEBUG SenderThread:603355 [sender.py:send():382] send: metric |
| 2024-03-28 18:18:46,311 DEBUG SenderThread:603355 [sender.py:send():382] send: metric |
| 2024-03-28 18:18:46,311 DEBUG SenderThread:603355 [sender.py:send():382] send: metric |
| 2024-03-28 18:18:46,311 DEBUG SenderThread:603355 [sender.py:send():382] send: history |
| 2024-03-28 18:18:46,311 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: summary_record |
| 2024-03-28 18:18:46,314 INFO SenderThread:603355 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end |
| 2024-03-28 18:18:46,528 INFO wandb-upload_0:603355 [upload_job.py:push():131] Uploaded file /state/partition1/job-44751296/tmpylawwan9wandb/ki7nntdh-wandb-metadata.json |
| 2024-03-28 18:18:46,579 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/conda-environment.yaml |
| 2024-03-28 18:18:46,579 INFO Thread-12 :603355 [dir_watcher.py:_on_file_created():271] file/dir created: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/wandb-summary.json |
| 2024-03-28 18:18:46,579 INFO Thread-12 :603355 [dir_watcher.py:_on_file_created():271] file/dir created: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:18:46,579 INFO Thread-12 :603355 [dir_watcher.py:_on_file_created():271] file/dir created: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/wandb-metadata.json |
| 2024-03-28 18:18:46,579 INFO Thread-12 :603355 [dir_watcher.py:_on_file_created():271] file/dir created: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/requirements.txt |
| 2024-03-28 18:18:47,505 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:18:48,579 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:18:52,702 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:18:57,703 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:19:01,217 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: stop_status |
| 2024-03-28 18:19:01,217 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: stop_status |
| 2024-03-28 18:19:01,243 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: internal_messages |
| 2024-03-28 18:19:03,011 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:19:03,051 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: partial_history |
| 2024-03-28 18:19:03,052 DEBUG SenderThread:603355 [sender.py:send():382] send: metric |
| 2024-03-28 18:19:03,053 DEBUG SenderThread:603355 [sender.py:send():382] send: metric |
| 2024-03-28 18:19:03,053 DEBUG SenderThread:603355 [sender.py:send():382] send: metric |
| 2024-03-28 18:19:03,053 DEBUG SenderThread:603355 [sender.py:send():382] send: history |
| 2024-03-28 18:19:03,053 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: summary_record |
| 2024-03-28 18:19:03,054 INFO SenderThread:603355 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end |
| 2024-03-28 18:19:03,584 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/wandb-summary.json |
| 2024-03-28 18:19:04,585 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:19:08,059 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:19:08,586 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/config.yaml |
| 2024-03-28 18:19:13,168 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:19:16,241 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: internal_messages |
| 2024-03-28 18:19:16,241 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: stop_status |
| 2024-03-28 18:19:16,241 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: stop_status |
| 2024-03-28 18:19:17,488 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: partial_history |
| 2024-03-28 18:19:17,489 DEBUG SenderThread:603355 [sender.py:send():382] send: history |
| 2024-03-28 18:19:17,489 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: summary_record |
| 2024-03-28 18:19:17,491 INFO SenderThread:603355 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end |
| 2024-03-28 18:19:17,589 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/wandb-summary.json |
| 2024-03-28 18:19:18,492 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:19:18,589 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:19:23,492 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:19:28,493 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:19:31,341 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: stop_status |
| 2024-03-28 18:19:31,341 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: internal_messages |
| 2024-03-28 18:19:31,341 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: stop_status |
| 2024-03-28 18:19:31,908 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: partial_history |
| 2024-03-28 18:19:31,909 DEBUG SenderThread:603355 [sender.py:send():382] send: history |
| 2024-03-28 18:19:31,909 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: summary_record |
| 2024-03-28 18:19:31,911 INFO SenderThread:603355 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end |
| 2024-03-28 18:19:32,592 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/wandb-summary.json |
| 2024-03-28 18:19:32,592 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:19:33,912 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:19:34,593 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:19:36,645 DEBUG SystemMonitor:603355 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2024-03-28 18:19:36,647 DEBUG SenderThread:603355 [sender.py:send():382] send: stats |
| 2024-03-28 18:19:39,648 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:19:44,648 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:19:46,279 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: partial_history |
| 2024-03-28 18:19:46,280 DEBUG SenderThread:603355 [sender.py:send():382] send: history |
| 2024-03-28 18:19:46,280 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: summary_record |
| 2024-03-28 18:19:46,282 INFO SenderThread:603355 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end |
| 2024-03-28 18:19:46,341 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: stop_status |
| 2024-03-28 18:19:46,342 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: internal_messages |
| 2024-03-28 18:19:46,342 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: stop_status |
| 2024-03-28 18:19:46,595 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/wandb-summary.json |
| 2024-03-28 18:19:46,595 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:19:48,596 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:19:50,412 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:19:55,413 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:20:00,413 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:20:00,731 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: partial_history |
| 2024-03-28 18:20:00,732 DEBUG SenderThread:603355 [sender.py:send():382] send: history |
| 2024-03-28 18:20:00,732 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: summary_record |
| 2024-03-28 18:20:00,733 INFO SenderThread:603355 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end |
| 2024-03-28 18:20:01,341 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: internal_messages |
| 2024-03-28 18:20:01,342 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: stop_status |
| 2024-03-28 18:20:01,342 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: stop_status |
| 2024-03-28 18:20:01,598 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/wandb-summary.json |
| 2024-03-28 18:20:02,599 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:20:05,516 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:20:06,599 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:20:06,648 DEBUG SenderThread:603355 [sender.py:send():382] send: stats |
| 2024-03-28 18:20:10,649 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:20:15,602 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:20:16,287 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:20:16,342 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: internal_messages |
| 2024-03-28 18:20:16,342 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: stop_status |
| 2024-03-28 18:20:16,342 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: stop_status |
| 2024-03-28 18:20:21,323 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:20:23,604 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:20:26,992 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:20:31,342 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: stop_status |
| 2024-03-28 18:20:31,342 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: internal_messages |
| 2024-03-28 18:20:31,342 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: stop_status |
| 2024-03-28 18:20:32,030 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:20:33,605 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:20:35,606 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:20:36,650 DEBUG SenderThread:603355 [sender.py:send():382] send: stats |
| 2024-03-28 18:20:37,650 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:20:42,651 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:20:45,608 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:20:46,342 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: stop_status |
| 2024-03-28 18:20:46,342 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: internal_messages |
| 2024-03-28 18:20:46,342 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: stop_status |
| 2024-03-28 18:20:48,135 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:20:53,136 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:20:55,093 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: partial_history |
| 2024-03-28 18:20:55,094 DEBUG SenderThread:603355 [sender.py:send():382] send: history |
| 2024-03-28 18:20:55,094 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: summary_record |
| 2024-03-28 18:20:55,096 INFO SenderThread:603355 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end |
| 2024-03-28 18:20:55,610 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/wandb-summary.json |
| 2024-03-28 18:20:56,610 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:20:57,611 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:20:58,176 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:21:01,348 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: internal_messages |
| 2024-03-28 18:21:01,348 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: stop_status |
| 2024-03-28 18:21:01,349 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: stop_status |
| 2024-03-28 18:21:03,491 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:21:06,653 DEBUG SenderThread:603355 [sender.py:send():382] send: stats |
| 2024-03-28 18:21:08,654 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:21:09,629 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: partial_history |
| 2024-03-28 18:21:09,630 DEBUG SenderThread:603355 [sender.py:send():382] send: history |
| 2024-03-28 18:21:09,630 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: summary_record |
| 2024-03-28 18:21:09,631 INFO SenderThread:603355 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end |
| 2024-03-28 18:21:10,614 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/wandb-summary.json |
| 2024-03-28 18:21:10,614 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:21:11,614 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:21:14,631 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:21:16,354 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: internal_messages |
| 2024-03-28 18:21:16,354 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: stop_status |
| 2024-03-28 18:21:16,355 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: stop_status |
| 2024-03-28 18:21:20,511 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:21:24,026 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: partial_history |
| 2024-03-28 18:21:24,027 DEBUG SenderThread:603355 [sender.py:send():382] send: history |
| 2024-03-28 18:21:24,027 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: summary_record |
| 2024-03-28 18:21:24,028 INFO SenderThread:603355 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end |
| 2024-03-28 18:21:24,617 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/wandb-summary.json |
| 2024-03-28 18:21:25,617 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:21:26,028 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:21:26,617 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:21:31,029 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:21:31,355 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: stop_status |
| 2024-03-28 18:21:31,355 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: internal_messages |
| 2024-03-28 18:21:31,355 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: stop_status |
| 2024-03-28 18:21:36,463 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:21:36,652 DEBUG SenderThread:603355 [sender.py:send():382] send: stats |
| 2024-03-28 18:21:38,448 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: partial_history |
| 2024-03-28 18:21:38,449 DEBUG SenderThread:603355 [sender.py:send():382] send: history |
| 2024-03-28 18:21:38,449 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: summary_record |
| 2024-03-28 18:21:38,450 INFO SenderThread:603355 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end |
| 2024-03-28 18:21:38,620 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/wandb-summary.json |
| 2024-03-28 18:21:40,620 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:21:41,620 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:21:42,451 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:21:46,355 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: internal_messages |
| 2024-03-28 18:21:46,355 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: stop_status |
| 2024-03-28 18:21:46,356 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: stop_status |
| 2024-03-28 18:21:48,423 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:21:52,862 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: partial_history |
| 2024-03-28 18:21:52,863 DEBUG SenderThread:603355 [sender.py:send():382] send: history |
| 2024-03-28 18:21:52,863 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: summary_record |
| 2024-03-28 18:21:52,864 INFO SenderThread:603355 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end |
| 2024-03-28 18:21:53,623 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/wandb-summary.json |
| 2024-03-28 18:21:53,864 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:21:54,623 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:21:55,623 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:21:58,865 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:22:01,357 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: internal_messages |
| 2024-03-28 18:22:01,357 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: stop_status |
| 2024-03-28 18:22:01,357 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: stop_status |
| 2024-03-28 18:22:04,479 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:22:06,659 DEBUG SenderThread:603355 [sender.py:send():382] send: stats |
| 2024-03-28 18:22:07,282 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: partial_history |
| 2024-03-28 18:22:07,283 DEBUG SenderThread:603355 [sender.py:send():382] send: history |
| 2024-03-28 18:22:07,283 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: summary_record |
| 2024-03-28 18:22:07,284 INFO SenderThread:603355 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end |
| 2024-03-28 18:22:07,626 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/wandb-summary.json |
| 2024-03-28 18:22:08,626 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:22:09,627 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:22:10,288 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:22:11,627 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:22:13,628 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:22:15,354 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:22:15,628 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:22:16,356 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: stop_status |
| 2024-03-28 18:22:16,357 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: internal_messages |
| 2024-03-28 18:22:16,357 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: stop_status |
| 2024-03-28 18:22:20,930 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:22:21,630 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:22:25,930 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:22:29,631 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:22:31,357 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: internal_messages |
| 2024-03-28 18:22:31,357 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:22:31,357 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: stop_status |
| 2024-03-28 18:22:31,357 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: stop_status |
| 2024-03-28 18:22:31,632 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:22:36,509 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:22:36,655 DEBUG SenderThread:603355 [sender.py:send():382] send: stats |
| 2024-03-28 18:22:39,634 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:22:41,634 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:22:41,808 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:22:42,012 DEBUG SenderThread:603355 [sender.py:send():382] send: exit |
| 2024-03-28 18:22:42,012 INFO SenderThread:603355 [sender.py:send_exit():589] handling exit code: 255 |
| 2024-03-28 18:22:42,012 INFO SenderThread:603355 [sender.py:send_exit():591] handling runtime: 245 |
| 2024-03-28 18:22:42,012 INFO SenderThread:603355 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end |
| 2024-03-28 18:22:42,012 INFO SenderThread:603355 [sender.py:send_exit():597] send defer |
| 2024-03-28 18:22:42,013 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: defer |
| 2024-03-28 18:22:42,013 INFO HandlerThread:603355 [handler.py:handle_request_defer():172] handle defer: 0 |
| 2024-03-28 18:22:42,013 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: defer |
| 2024-03-28 18:22:42,013 INFO SenderThread:603355 [sender.py:send_request_defer():613] handle sender defer: 0 |
| 2024-03-28 18:22:42,013 INFO SenderThread:603355 [sender.py:transition_state():617] send defer: 1 |
| 2024-03-28 18:22:42,013 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: defer |
| 2024-03-28 18:22:42,013 INFO HandlerThread:603355 [handler.py:handle_request_defer():172] handle defer: 1 |
| 2024-03-28 18:22:42,013 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: defer |
| 2024-03-28 18:22:42,013 INFO SenderThread:603355 [sender.py:send_request_defer():613] handle sender defer: 1 |
| 2024-03-28 18:22:42,013 INFO SenderThread:603355 [sender.py:transition_state():617] send defer: 2 |
| 2024-03-28 18:22:42,013 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: defer |
| 2024-03-28 18:22:42,013 INFO HandlerThread:603355 [handler.py:handle_request_defer():172] handle defer: 2 |
| 2024-03-28 18:22:42,013 INFO HandlerThread:603355 [system_monitor.py:finish():203] Stopping system monitor |
| 2024-03-28 18:22:42,013 DEBUG SystemMonitor:603355 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2024-03-28 18:22:42,014 INFO HandlerThread:603355 [interfaces.py:finish():202] Joined cpu monitor |
| 2024-03-28 18:22:42,014 DEBUG SystemMonitor:603355 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2024-03-28 18:22:42,014 INFO HandlerThread:603355 [interfaces.py:finish():202] Joined disk monitor |
| 2024-03-28 18:22:42,119 INFO HandlerThread:603355 [interfaces.py:finish():202] Joined gpu monitor |
| 2024-03-28 18:22:42,119 INFO HandlerThread:603355 [interfaces.py:finish():202] Joined memory monitor |
| 2024-03-28 18:22:42,119 INFO HandlerThread:603355 [interfaces.py:finish():202] Joined network monitor |
| 2024-03-28 18:22:42,119 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: defer |
| 2024-03-28 18:22:42,119 INFO SenderThread:603355 [sender.py:send_request_defer():613] handle sender defer: 2 |
| 2024-03-28 18:22:42,120 INFO SenderThread:603355 [sender.py:transition_state():617] send defer: 3 |
| 2024-03-28 18:22:42,120 DEBUG SenderThread:603355 [sender.py:send():382] send: stats |
| 2024-03-28 18:22:42,120 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: defer |
| 2024-03-28 18:22:42,120 INFO HandlerThread:603355 [handler.py:handle_request_defer():172] handle defer: 3 |
| 2024-03-28 18:22:42,120 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: defer |
| 2024-03-28 18:22:42,120 INFO SenderThread:603355 [sender.py:send_request_defer():613] handle sender defer: 3 |
| 2024-03-28 18:22:42,120 INFO SenderThread:603355 [sender.py:transition_state():617] send defer: 4 |
| 2024-03-28 18:22:42,120 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: defer |
| 2024-03-28 18:22:42,121 INFO HandlerThread:603355 [handler.py:handle_request_defer():172] handle defer: 4 |
| 2024-03-28 18:22:42,121 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: defer |
| 2024-03-28 18:22:42,121 INFO SenderThread:603355 [sender.py:send_request_defer():613] handle sender defer: 4 |
| 2024-03-28 18:22:42,121 INFO SenderThread:603355 [sender.py:transition_state():617] send defer: 5 |
| 2024-03-28 18:22:42,121 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: defer |
| 2024-03-28 18:22:42,121 INFO HandlerThread:603355 [handler.py:handle_request_defer():172] handle defer: 5 |
| 2024-03-28 18:22:42,121 DEBUG SenderThread:603355 [sender.py:send():382] send: summary |
| 2024-03-28 18:22:42,121 INFO SenderThread:603355 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end |
| 2024-03-28 18:22:42,121 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: defer |
| 2024-03-28 18:22:42,122 INFO SenderThread:603355 [sender.py:send_request_defer():613] handle sender defer: 5 |
| 2024-03-28 18:22:42,122 INFO SenderThread:603355 [sender.py:transition_state():617] send defer: 6 |
| 2024-03-28 18:22:42,122 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: defer |
| 2024-03-28 18:22:42,122 INFO HandlerThread:603355 [handler.py:handle_request_defer():172] handle defer: 6 |
| 2024-03-28 18:22:42,122 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: defer |
| 2024-03-28 18:22:42,122 INFO SenderThread:603355 [sender.py:send_request_defer():613] handle sender defer: 6 |
| 2024-03-28 18:22:42,122 INFO SenderThread:603355 [sender.py:transition_state():617] send defer: 7 |
| 2024-03-28 18:22:42,122 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: status_report |
| 2024-03-28 18:22:42,122 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: defer |
| 2024-03-28 18:22:42,122 INFO HandlerThread:603355 [handler.py:handle_request_defer():172] handle defer: 7 |
| 2024-03-28 18:22:42,122 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: defer |
| 2024-03-28 18:22:42,122 INFO SenderThread:603355 [sender.py:send_request_defer():613] handle sender defer: 7 |
| 2024-03-28 18:22:42,634 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/wandb-summary.json |
| 2024-03-28 18:22:42,783 INFO SenderThread:603355 [sender.py:transition_state():617] send defer: 8 |
| 2024-03-28 18:22:42,783 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: defer |
| 2024-03-28 18:22:42,783 INFO HandlerThread:603355 [handler.py:handle_request_defer():172] handle defer: 8 |
| 2024-03-28 18:22:42,783 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: defer |
| 2024-03-28 18:22:42,783 INFO SenderThread:603355 [sender.py:send_request_defer():613] handle sender defer: 8 |
| 2024-03-28 18:22:42,783 INFO SenderThread:603355 [job_builder.py:build():296] Attempting to build job artifact |
| 2024-03-28 18:22:42,784 INFO SenderThread:603355 [job_builder.py:_get_source_type():426] is repo sourced job |
| 2024-03-28 18:22:42,817 INFO SenderThread:603355 [job_builder.py:build():402] adding wandb-job metadata file |
| 2024-03-28 18:22:42,821 INFO SenderThread:603355 [sender.py:transition_state():617] send defer: 9 |
| 2024-03-28 18:22:42,821 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: defer |
| 2024-03-28 18:22:42,821 DEBUG SenderThread:603355 [sender.py:send():382] send: artifact |
| 2024-03-28 18:22:42,821 INFO HandlerThread:603355 [handler.py:handle_request_defer():172] handle defer: 9 |
| 2024-03-28 18:22:43,012 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: poll_exit |
| 2024-03-28 18:22:43,635 INFO Thread-12 :603355 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:22:46,200 INFO SenderThread:603355 [sender.py:send_artifact():1494] sent artifact job-https___github.com_huggingface_trl.git_examples_scripts_sft.py - {'id': 'QXJ0aWZhY3Q6NzczMzg5MDg0', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE0MzYxMzYzNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NzczMzg5MDg0', 'versionIndex': 15}}} |
| 2024-03-28 18:22:46,200 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: defer |
| 2024-03-28 18:22:46,200 INFO SenderThread:603355 [sender.py:send_request_defer():613] handle sender defer: 9 |
| 2024-03-28 18:22:46,200 INFO SenderThread:603355 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2024-03-28 18:22:46,636 INFO SenderThread:603355 [dir_watcher.py:finish():388] scan: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files |
| 2024-03-28 18:22:46,636 INFO SenderThread:603355 [dir_watcher.py:finish():402] scan save: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/conda-environment.yaml conda-environment.yaml |
| 2024-03-28 18:22:46,636 INFO SenderThread:603355 [dir_watcher.py:finish():402] scan save: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/wandb-metadata.json wandb-metadata.json |
| 2024-03-28 18:22:46,636 INFO SenderThread:603355 [dir_watcher.py:finish():402] scan save: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/config.yaml config.yaml |
| 2024-03-28 18:22:46,638 INFO SenderThread:603355 [dir_watcher.py:finish():402] scan save: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/requirements.txt requirements.txt |
| 2024-03-28 18:22:46,641 INFO SenderThread:603355 [dir_watcher.py:finish():402] scan save: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/wandb-summary.json wandb-summary.json |
| 2024-03-28 18:22:46,641 INFO SenderThread:603355 [dir_watcher.py:finish():402] scan save: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log output.log |
| 2024-03-28 18:22:46,645 INFO SenderThread:603355 [sender.py:transition_state():617] send defer: 10 |
| 2024-03-28 18:22:46,645 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: poll_exit |
| 2024-03-28 18:22:46,645 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: defer |
| 2024-03-28 18:22:46,647 INFO HandlerThread:603355 [handler.py:handle_request_defer():172] handle defer: 10 |
| 2024-03-28 18:22:46,648 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: defer |
| 2024-03-28 18:22:46,648 INFO SenderThread:603355 [sender.py:send_request_defer():613] handle sender defer: 10 |
| 2024-03-28 18:22:46,648 INFO SenderThread:603355 [file_pusher.py:finish():175] shutting down file pusher |
| 2024-03-28 18:22:46,922 INFO wandb-upload_0:603355 [upload_job.py:push():131] Uploaded file /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/conda-environment.yaml |
| 2024-03-28 18:22:46,933 INFO wandb-upload_1:603355 [upload_job.py:push():131] Uploaded file /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/config.yaml |
| 2024-03-28 18:22:47,005 INFO wandb-upload_3:603355 [upload_job.py:push():131] Uploaded file /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/wandb-summary.json |
| 2024-03-28 18:22:47,008 INFO wandb-upload_4:603355 [upload_job.py:push():131] Uploaded file /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/output.log |
| 2024-03-28 18:22:47,009 INFO wandb-upload_2:603355 [upload_job.py:push():131] Uploaded file /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/files/requirements.txt |
| 2024-03-28 18:22:47,013 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: poll_exit |
| 2024-03-28 18:22:47,013 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: poll_exit |
| 2024-03-28 18:22:47,209 INFO Thread-11 :603355 [sender.py:transition_state():617] send defer: 11 |
| 2024-03-28 18:22:47,210 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: defer |
| 2024-03-28 18:22:47,210 INFO HandlerThread:603355 [handler.py:handle_request_defer():172] handle defer: 11 |
| 2024-03-28 18:22:47,210 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: defer |
| 2024-03-28 18:22:47,210 INFO SenderThread:603355 [sender.py:send_request_defer():613] handle sender defer: 11 |
| 2024-03-28 18:22:47,210 INFO SenderThread:603355 [file_pusher.py:join():181] waiting for file pusher |
| 2024-03-28 18:22:47,210 INFO SenderThread:603355 [sender.py:transition_state():617] send defer: 12 |
| 2024-03-28 18:22:47,210 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: defer |
| 2024-03-28 18:22:47,210 INFO HandlerThread:603355 [handler.py:handle_request_defer():172] handle defer: 12 |
| 2024-03-28 18:22:47,210 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: defer |
| 2024-03-28 18:22:47,210 INFO SenderThread:603355 [sender.py:send_request_defer():613] handle sender defer: 12 |
| 2024-03-28 18:22:47,210 INFO SenderThread:603355 [file_stream.py:finish():595] file stream finish called |
| 2024-03-28 18:22:47,385 INFO SenderThread:603355 [file_stream.py:finish():599] file stream finish is done |
| 2024-03-28 18:22:47,385 INFO SenderThread:603355 [sender.py:transition_state():617] send defer: 13 |
| 2024-03-28 18:22:47,385 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: defer |
| 2024-03-28 18:22:47,385 INFO HandlerThread:603355 [handler.py:handle_request_defer():172] handle defer: 13 |
| 2024-03-28 18:22:47,385 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: defer |
| 2024-03-28 18:22:47,385 INFO SenderThread:603355 [sender.py:send_request_defer():613] handle sender defer: 13 |
| 2024-03-28 18:22:47,386 INFO SenderThread:603355 [sender.py:transition_state():617] send defer: 14 |
| 2024-03-28 18:22:47,386 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: defer |
| 2024-03-28 18:22:47,386 DEBUG SenderThread:603355 [sender.py:send():382] send: final |
| 2024-03-28 18:22:47,386 INFO HandlerThread:603355 [handler.py:handle_request_defer():172] handle defer: 14 |
| 2024-03-28 18:22:47,386 DEBUG SenderThread:603355 [sender.py:send():382] send: footer |
| 2024-03-28 18:22:47,386 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: defer |
| 2024-03-28 18:22:47,386 INFO SenderThread:603355 [sender.py:send_request_defer():613] handle sender defer: 14 |
| 2024-03-28 18:22:47,386 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: poll_exit |
| 2024-03-28 18:22:47,387 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: poll_exit |
| 2024-03-28 18:22:47,387 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: poll_exit |
| 2024-03-28 18:22:47,387 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: server_info |
| 2024-03-28 18:22:47,387 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: poll_exit |
| 2024-03-28 18:22:47,387 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: get_summary |
| 2024-03-28 18:22:47,387 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: server_info |
| 2024-03-28 18:22:47,388 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: sampled_history |
| 2024-03-28 18:22:47,390 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: internal_messages |
| 2024-03-28 18:22:47,390 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: job_info |
| 2024-03-28 18:22:47,441 DEBUG SenderThread:603355 [sender.py:send_request():409] send_request: job_info |
| 2024-03-28 18:22:47,441 INFO MainThread:603355 [wandb_run.py:_footer_history_summary_info():3857] rendering history |
| 2024-03-28 18:22:47,442 INFO MainThread:603355 [wandb_run.py:_footer_history_summary_info():3889] rendering summary |
| 2024-03-28 18:22:47,442 INFO MainThread:603355 [wandb_run.py:_footer_sync_info():3816] logging synced files |
| 2024-03-28 18:22:47,442 DEBUG HandlerThread:603355 [handler.py:handle_request():146] handle_request: shutdown |
| 2024-03-28 18:22:47,442 INFO HandlerThread:603355 [handler.py:finish():869] shutting down handler |
| 2024-03-28 18:22:48,391 INFO WriterThread:603355 [datastore.py:close():296] close: /home/km3888/trl/wandb/run-20240328_181836-ljeg9p3r/run-ljeg9p3r.wandb |
| 2024-03-28 18:22:48,441 INFO SenderThread:603355 [sender.py:finish():1572] shutting down sender |
| 2024-03-28 18:22:48,441 INFO SenderThread:603355 [file_pusher.py:finish():175] shutting down file pusher |
| 2024-03-28 18:22:48,441 INFO SenderThread:603355 [file_pusher.py:join():181] waiting for file pusher |
|
|
