Blame for run/train.out

[22:17:31] device=cuda:0 bf16=True dev=NVIDIA GeForce RTX 5090 world=8
[22:17:31] dane: train=1,889,519,627B block=4096 batch=256 tok/step=1,048,576
[22:17:42] model GPT-ref: 1073.9M param (L20 d2048 h16)
[22:17:43] optimizer=muon muon_lr=0.02 (mult=33.3x)
[22:17:43] r11 pamiec: act_ckpt=True loss_chunk=1024
[22:17:43] r13: mixer_layout=AAAAAAAAAAAAAAAAAAAA gdn_head_dim=64 attn_bias=0 rope_style=interleaved
[22:17:46] torch.compile enabled
[22:17:52] RESUME @ 2250
[22:17:52] START gpt-ref: steps=5720 (od 2250) lr=0.0006->6e-05
[22:17:52] sampler epoch: 461,307 okien/epoke (offset0 2245), start od kroku 2250 = epoka 1.249
[22:17:52] data-dir-2: /root/pilot1b/data_p01 1,880,520,445 tok, 459,110 okien/epoke = 1793 krokow, od kroku strumienia 1801
[22:17:52] r24 train3: 1,883,249,429 tok, 459,777 okien/epoke = 1796 krokow, od kroku strumienia 3594
[22:17:52] r24 train4: 1,890,850,536 tok, 461,632 okien/epoke = 1803 krokow, od kroku strumienia 5389
[rank2]:W1023 22:17:52.342000 8204 site-packages/torch/_logging/_internal.py:1154] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
[rank3]:W1023 22:17:52.353000 8205 site-packages/torch/_logging/_internal.py:1154] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
[rank5]:W1023 22:17:52.355000 8207 site-packages/torch/_logging/_internal.py:1154] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
[rank6]:W1023 22:17:52.355000 8208 site-packages/torch/_logging/_internal.py:1154] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
[rank1]:W1023 22:17:52.356000 8203 site-packages/torch/_logging/_internal.py:1154] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
[rank4]:W1023 22:17:52.357000 8206 site-packages/torch/_logging/_internal.py:1154] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
[rank7]:W1023 22:17:52.358000 8209 site-packages/torch/_logging/_internal.py:1154] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
[rank0]:W1023 22:17:52.361000 8202 site-packages/torch/_logging/_internal.py:1154] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
[22:21:05] step 2260/5720 loss 2.6379 lr 6.00e-04 gnorm 0.10 54,164 tok/s peakVRAM 27.0GB
[22:22:38] step 2270/5720 loss 2.6461 lr 6.00e-04 gnorm 0.08 113,489 tok/s peakVRAM 27.0GB
[22:24:10] step 2280/5720 loss 2.6532 lr 6.00e-04 gnorm 0.08 113,231 tok/s peakVRAM 27.0GB
[22:25:43] step 2290/5720 loss 2.6472 lr 6.00e-04 gnorm 0.08 113,191 tok/s peakVRAM 27.0GB
[22:27:16] step 2300/5720 loss 2.6436 lr 6.00e-04 gnorm 0.08 113,099 tok/s peakVRAM 27.0GB
[22:28:48] step 2310/5720 loss 2.6365 lr 6.00e-04 gnorm 0.10 113,222 tok/s peakVRAM 27.0GB
[22:30:21] step 2320/5720 loss 2.6297 lr 6.00e-04 gnorm 0.07 113,237 tok/s peakVRAM 27.0GB
[22:31:54] step 2330/5720 loss 2.6421 lr 6.00e-04 gnorm 0.09 113,292 tok/s peakVRAM 27.0GB
[22:33:26] step 2340/5720 loss 2.6351 lr 6.00e-04 gnorm 0.09 113,238 tok/s peakVRAM 27.0GB
[22:34:59] step 2350/5720 loss 2.6249 lr 6.00e-04 gnorm 0.07 113,263 tok/s peakVRAM 27.0GB
[22:36:31] step 2360/5720 loss 2.6175 lr 6.00e-04 gnorm 0.08 113,301 tok/s peakVRAM 27.0GB
[22:38:04] step 2370/5720 loss 2.6197 lr 6.00e-04 gnorm 0.07 113,345 tok/s peakVRAM 27.0GB
[22:39:36] step 2380/5720 loss 2.6259 lr 6.00e-04 gnorm 0.08 113,443 tok/s peakVRAM 27.0GB
[22:41:09] step 2390/5720 loss 2.6177 lr 6.00e-04 gnorm 0.07 113,353 tok/s peakVRAM 27.0GB
[22:42:41] step 2400/5720 loss 2.6315 lr 6.00e-04 gnorm 0.08 113,380 tok/s peakVRAM 27.0GB
[22:42:50] ckpt @ 2400
[22:44:22] step 2410/5720 loss 2.6364 lr 6.00e-04 gnorm 0.09 103,710 tok/s peakVRAM 27.0GB
[22:45:55] step 2420/5720 loss 2.6255 lr 6.00e-04 gnorm 0.09 113,315 tok/s peakVRAM 27.0GB
[22:47:27] step 2430/5720 loss 2.6181 lr 6.00e-04 gnorm 0.08 113,253 tok/s peakVRAM 27.0GB
[22:49:00] step 2440/5720 loss 2.6159 lr 6.00e-04 gnorm 0.08 113,251 tok/s peakVRAM 27.0GB
[22:50:33] step 2450/5720 loss 2.6271 lr 6.00e-04 gnorm 0.08 113,260 tok/s peakVRAM 27.0GB
[22:52:05] step 2460/5720 loss 2.6269 lr 6.00e-04 gnorm 0.09 113,266 tok/s peakVRAM 27.0GB
[22:53:38] step 2470/5720 loss 2.6160 lr 6.00e-04 gnorm 0.08 113,419 tok/s peakVRAM 27.0GB
[22:55:10] step 2480/5720 loss 2.6256 lr 6.00e-04 gnorm 0.08 113,388 tok/s peakVRAM 27.0GB
[22:56:43] step 2490/5720 loss 2.6332 lr 6.00e-04 gnorm 0.07 113,374 tok/s peakVRAM 27.0GB
[22:58:15] step 2500/5720 loss 2.6236 lr 6.00e-04 gnorm 0.07 113,430 tok/s peakVRAM 27.0GB
[22:59:48] step 2510/5720 loss 2.6180 lr 6.00e-04 gnorm 0.08 113,347 tok/s peakVRAM 27.0GB
[23:01:20] step 2520/5720 loss 2.6165 lr 6.00e-04 gnorm 0.07 113,325 tok/s peakVRAM 27.0GB
[23:02:53] step 2530/5720 loss 2.6012 lr 6.00e-04 gnorm 0.07 113,268 tok/s peakVRAM 27.0GB
[23:04:25] step 2540/5720 loss 2.6064 lr 6.00e-04 gnorm 0.09 113,220 tok/s peakVRAM 27.0GB
[23:05:58] step 2550/5720 loss 2.6060 lr 6.00e-04 gnorm 0.08 113,163 tok/s peakVRAM 27.0GB
[23:06:06] ckpt @ 2550
[23:07:38] step 2560/5720 loss 2.5938 lr 6.00e-04 gnorm 0.07 104,295 tok/s peakVRAM 27.0GB
[23:09:11] step 2570/5720 loss 2.6062 lr 6.00e-04 gnorm 0.08 113,333 tok/s peakVRAM 27.0GB
[23:10:44] step 2580/5720 loss 2.6028 lr 6.00e-04 gnorm 0.08 113,215 tok/s peakVRAM 27.0GB
[23:12:16] step 2590/5720 loss 2.6091 lr 6.00e-04 gnorm 0.08 113,178 tok/s peakVRAM 27.0GB
[23:13:49] step 2600/5720 loss 2.6079 lr 6.00e-04 gnorm 0.08 113,194 tok/s peakVRAM 27.0GB
[23:15:22] step 2610/5720 loss 2.6177 lr 6.00e-04 gnorm 0.08 113,199 tok/s peakVRAM 27.0GB
[23:16:54] step 2620/5720 loss 2.5998 lr 6.00e-04 gnorm 0.08 113,183 tok/s peakVRAM 27.0GB
[23:18:27] step 2630/5720 loss 2.5994 lr 6.00e-04 gnorm 0.08 112,992 tok/s peakVRAM 27.0GB
[23:20:00] step 2640/5720 loss 2.6001 lr 6.00e-04 gnorm 0.08 113,121 tok/s peakVRAM 27.0GB
[23:21:32] step 2650/5720 loss 2.6212 lr 6.00e-04 gnorm 0.08 113,194 tok/s peakVRAM 27.0GB
[23:23:05] step 2660/5720 loss 2.5814 lr 6.00e-04 gnorm 0.08 113,228 tok/s peakVRAM 27.0GB
[23:24:37] step 2670/5720 loss 2.6000 lr 6.00e-04 gnorm 0.09 113,302 tok/s peakVRAM 27.0GB
[23:26:10] step 2680/5720 loss 2.5943 lr 6.00e-04 gnorm 0.07 113,406 tok/s peakVRAM 27.0GB
[23:27:42] step 2690/5720 loss 2.5914 lr 6.00e-04 gnorm 0.09 113,344 tok/s peakVRAM 27.0GB
[23:29:15] step 2700/5720 loss 2.5959 lr 6.00e-04 gnorm 0.09 113,295 tok/s peakVRAM 27.0GB
[23:29:23] ckpt @ 2700
[23:30:56] step 2710/5720 loss 2.6002 lr 6.00e-04 gnorm 0.07 104,284 tok/s peakVRAM 27.0GB
[23:32:28] step 2720/5720 loss 2.5896 lr 6.00e-04 gnorm 0.08 113,130 tok/s peakVRAM 27.0GB
[23:34:01] step 2730/5720 loss 2.6118 lr 6.00e-04 gnorm 0.07 113,009 tok/s peakVRAM 27.0GB
[23:35:34] step 2740/5720 loss 2.5942 lr 6.00e-04 gnorm 0.07 113,004 tok/s peakVRAM 27.0GB
[23:37:07] step 2750/5720 loss 2.5820 lr 6.00e-04 gnorm 0.07 113,043 tok/s peakVRAM 27.0GB
[23:38:39] step 2760/5720 loss 2.5676 lr 6.00e-04 gnorm 0.08 113,155 tok/s peakVRAM 27.0GB
[23:40:12] step 2770/5720 loss 2.5999 lr 6.00e-04 gnorm 0.07 113,241 tok/s peakVRAM 27.0GB
[23:41:45] step 2780/5720 loss 2.5972 lr 6.00e-04 gnorm 0.09 113,110 tok/s peakVRAM 27.0GB
[23:43:17] step 2790/5720 loss 2.5851 lr 6.00e-04 gnorm 0.08 113,233 tok/s peakVRAM 27.0GB
[23:44:50] step 2800/5720 loss 2.5806 lr 6.00e-04 gnorm 0.08 113,205 tok/s peakVRAM 27.0GB
[23:46:22] step 2810/5720 loss 2.5821 lr 6.00e-04 gnorm 0.07 113,123 tok/s peakVRAM 27.0GB
[23:47:55] step 2820/5720 loss 2.5823 lr 6.00e-04 gnorm 0.08 113,254 tok/s peakVRAM 27.0GB
[23:49:28] step 2830/5720 loss 2.5675 lr 6.00e-04 gnorm 0.07 113,404 tok/s peakVRAM 27.0GB
[23:51:00] step 2840/5720 loss 2.5864 lr 6.00e-04 gnorm 0.07 113,377 tok/s peakVRAM 27.0GB
[23:52:32] step 2850/5720 loss 2.5572 lr 6.00e-04 gnorm 0.07 113,451 tok/s peakVRAM 27.0GB
[23:52:41] ckpt @ 2850
[23:54:13] step 2860/5720 loss 2.5817 lr 6.00e-04 gnorm 0.08 103,743 tok/s peakVRAM 27.0GB
[23:55:46] step 2870/5720 loss 2.5747 lr 6.00e-04 gnorm 0.07 113,433 tok/s peakVRAM 27.0GB
[23:57:19] step 2880/5720 loss 2.5715 lr 6.00e-04 gnorm 0.08 113,227 tok/s peakVRAM 27.0GB
[23:58:51] step 2890/5720 loss 2.5776 lr 6.00e-04 gnorm 0.07 113,315 tok/s peakVRAM 27.0GB
[00:00:24] step 2900/5720 loss 2.5869 lr 6.00e-04 gnorm 0.09 113,436 tok/s peakVRAM 27.0GB
[00:01:56] step 2910/5720 loss 2.5714 lr 6.00e-04 gnorm 0.07 113,418 tok/s peakVRAM 27.0GB
[00:03:28] step 2920/5720 loss 2.5774 lr 6.00e-04 gnorm 0.07 113,436 tok/s peakVRAM 27.0GB
[00:05:01] step 2930/5720 loss 2.5815 lr 6.00e-04 gnorm 0.08 113,427 tok/s peakVRAM 27.0GB
[00:06:33] step 2940/5720 loss 2.5607 lr 6.00e-04 gnorm 0.07 113,484 tok/s peakVRAM 27.0GB
[00:08:06] step 2950/5720 loss 2.5818 lr 6.00e-04 gnorm 0.09 113,380 tok/s peakVRAM 27.0GB
[00:09:38] step 2960/5720 loss 2.5656 lr 6.00e-04 gnorm 0.07 113,392 tok/s peakVRAM 27.0GB
[00:11:11] step 2970/5720 loss 2.5829 lr 6.00e-04 gnorm 0.08 113,329 tok/s peakVRAM 27.0GB
[00:12:43] step 2980/5720 loss 2.5874 lr 6.00e-04 gnorm 0.08 113,352 tok/s peakVRAM 27.0GB
[00:14:16] step 2990/5720 loss 2.5585 lr 6.00e-04 gnorm 0.06 113,331 tok/s peakVRAM 27.0GB
[00:15:48] step 3000/5720 loss 2.5749 lr 6.00e-04 gnorm 0.07 113,327 tok/s peakVRAM 27.0GB
[00:15:57] ckpt @ 3000
[00:17:29] step 3010/5720 loss 2.5600 lr 6.00e-04 gnorm 0.08 104,129 tok/s peakVRAM 27.0GB
[00:19:02] step 3020/5720 loss 2.5549 lr 6.00e-04 gnorm 0.07 113,342 tok/s peakVRAM 27.0GB
[00:20:34] step 3030/5720 loss 2.5535 lr 6.00e-04 gnorm 0.08 113,236 tok/s peakVRAM 27.0GB
[00:22:07] step 3040/5720 loss 2.5648 lr 6.00e-04 gnorm 0.07 113,214 tok/s peakVRAM 27.0GB
[00:23:39] step 3050/5720 loss 2.5730 lr 6.00e-04 gnorm 0.08 113,294 tok/s peakVRAM 27.0GB
[00:25:12] step 3060/5720 loss 2.5516 lr 6.00e-04 gnorm 0.06 113,304 tok/s peakVRAM 27.0GB
[00:26:44] step 3070/5720 loss 2.5735 lr 6.00e-04 gnorm 0.08 113,346 tok/s peakVRAM 27.0GB
[00:28:17] step 3080/5720 loss 2.5782 lr 6.00e-04 gnorm 0.08 113,366 tok/s peakVRAM 27.0GB
[00:29:49] step 3090/5720 loss 2.5622 lr 6.00e-04 gnorm 0.07 113,345 tok/s peakVRAM 27.0GB
[00:31:22] step 3100/5720 loss 2.5571 lr 6.00e-04 gnorm 0.07 113,372 tok/s peakVRAM 27.0GB
[00:32:54] step 3110/5720 loss 2.5389 lr 6.00e-04 gnorm 0.07 113,364 tok/s peakVRAM 27.0GB
[00:34:27] step 3120/5720 loss 2.5615 lr 6.00e-04 gnorm 0.07 113,394 tok/s peakVRAM 27.0GB
[00:35:59] step 3130/5720 loss 2.5639 lr 6.00e-04 gnorm 0.08 113,415 tok/s peakVRAM 27.0GB
[00:37:32] step 3140/5720 loss 2.5503 lr 6.00e-04 gnorm 0.07 113,358 tok/s peakVRAM 27.0GB
[00:39:04] step 3150/5720 loss 2.5463 lr 6.00e-04 gnorm 0.07 113,405 tok/s peakVRAM 27.0GB
[00:39:13] ckpt @ 3150
[00:40:45] step 3160/5720 loss 2.5686 lr 6.00e-04 gnorm 0.07 104,167 tok/s peakVRAM 27.0GB
[00:42:17] step 3170/5720 loss 2.5492 lr 6.00e-04 gnorm 0.08 113,506 tok/s peakVRAM 27.0GB
[00:43:50] step 3180/5720 loss 2.5450 lr 6.00e-04 gnorm 0.08 113,475 tok/s peakVRAM 27.0GB
[00:45:22] step 3190/5720 loss 2.5459 lr 6.00e-04 gnorm 0.07 113,522 tok/s peakVRAM 27.0GB
[00:46:55] step 3200/5720 loss 2.5286 lr 6.00e-04 gnorm 0.07 113,402 tok/s peakVRAM 27.0GB
[00:48:27] step 3210/5720 loss 2.5580 lr 6.00e-04 gnorm 0.07 113,391 tok/s peakVRAM 27.0GB
[00:49:59] step 3220/5720 loss 2.5494 lr 6.00e-04 gnorm 0.08 113,361 tok/s peakVRAM 27.0GB
[00:51:32] step 3230/5720 loss 2.5754 lr 6.00e-04 gnorm 0.07 113,407 tok/s peakVRAM 27.0GB
[00:53:04] step 3240/5720 loss 2.5362 lr 6.00e-04 gnorm 0.07 113,385 tok/s peakVRAM 27.0GB
[00:54:37] step 3250/5720 loss 2.5611 lr 6.00e-04 gnorm 0.06 113,386 tok/s peakVRAM 27.0GB
[00:56:09] step 3260/5720 loss 2.5556 lr 6.00e-04 gnorm 0.07 113,390 tok/s peakVRAM 27.0GB
[00:57:42] step 3270/5720 loss 2.5300 lr 6.00e-04 gnorm 0.07 113,442 tok/s peakVRAM 27.0GB
[00:59:14] step 3280/5720 loss 2.5741 lr 6.00e-04 gnorm 0.07 113,384 tok/s peakVRAM 27.0GB
[01:00:47] step 3290/5720 loss 2.5494 lr 6.00e-04 gnorm 0.06 113,408 tok/s peakVRAM 27.0GB
[01:02:19] step 3300/5720 loss 2.5484 lr 6.00e-04 gnorm 0.06 113,464 tok/s peakVRAM 27.0GB
[01:02:28] ckpt @ 3300
[01:04:00] step 3310/5720 loss 2.5522 lr 6.00e-04 gnorm 0.08 103,995 tok/s peakVRAM 27.0GB
W1024 01:04:03.500000 8184 site-packages/torch/distributed/elastic/agent/server/api.py:723] Received 15 death signal, shutting down workers
W1024 01:04:03.502000 8184 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 8202 closing signal SIGTERM
W1024 01:04:03.503000 8184 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 8203 closing signal SIGTERM
W1024 01:04:03.504000 8184 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 8204 closing signal SIGTERM
W1024 01:04:03.506000 8184 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 8205 closing signal SIGTERM
W1024 01:04:03.510000 8184 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 8206 closing signal SIGTERM
W1024 01:04:03.532000 8184 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 8207 closing signal SIGTERM
W1024 01:04:03.533000 8184 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 8208 closing signal SIGTERM
W1024 01:04:03.549000 8184 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 8209 closing signal SIGTERM
Traceback (most recent call last):
  File "/opt/conda/bin/torchrun", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 357, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/distributed/run.py", line 901, in main
    run(args)
  File "/opt/conda/lib/python3.11/site-packages/torch/distributed/run.py", line 892, in run
    elastic_launch(
  File "/opt/conda/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 143, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent
    result = agent.run()
             ^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
    result = f(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/distributed/elastic/agent/server/api.py", line 715, in run
    result = self._invoke_run(role)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/distributed/elastic/agent/server/api.py", line 879, in _invoke_run
    time.sleep(monitor_interval)
  File "/opt/conda/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
    raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
torch.distributed.elastic.multiprocessing.api.SignalException: Process 8184 got signal: 15
[01:06:08] device=cuda:0 bf16=True dev=NVIDIA GeForce RTX 5090 world=8
[01:06:08] dane: train=1,889,519,627B block=4096 batch=256 tok/step=1,048,576
[01:06:16] model GPT-ref: 1073.9M param (L20 d2048 h16)
[01:06:17] optimizer=muon muon_lr=0.02 (mult=33.3x)
[01:06:17] r11 pamiec: act_ckpt=True loss_chunk=1024
[01:06:17] r13: mixer_layout=AAAAAAAAAAAAAAAAAAAA gdn_head_dim=64 attn_bias=0 rope_style=interleaved
[01:06:28] torch.compile enabled
[rank1]:W1024 01:06:33.636000 44893 site-packages/torch/_logging/_internal.py:1154] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
[01:06:34] RESUME @ 3300
[01:06:34] START gpt-ref: steps=5720 (od 3300) lr=0.0006->6e-05
[01:06:34] sampler epoch: 461,307 okien/epoke (offset0 2245), start od kroku 3300 = epoka 1.831
[01:06:34] data-dir-2: /root/pilot1b/data_p01 1,880,520,445 tok, 459,110 okien/epoke = 1793 krokow, od kroku strumienia 1801
[01:06:34] r24 train3: 1,883,249,429 tok, 459,777 okien/epoke = 1796 krokow, od kroku strumienia 3594
[01:06:34] r24 train4: 1,890,850,536 tok, 461,632 okien/epoke = 1803 krokow, od kroku strumienia 5389
[rank2]:W1024 01:06:34.105000 44894 site-packages/torch/_logging/_internal.py:1154] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
[rank0]:W1024 01:06:34.117000 44892 site-packages/torch/_logging/_internal.py:1154] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
[rank7]:W1024 01:06:34.186000 44899 site-packages/torch/_logging/_internal.py:1154] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
[rank4]:W1024 01:06:34.217000 44896 site-packages/torch/_logging/_internal.py:1154] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
[rank6]:W1024 01:06:34.217000 44898 site-packages/torch/_logging/_internal.py:1154] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
[rank5]:W1024 01:06:34.517000 44897 site-packages/torch/_logging/_internal.py:1154] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
[rank3]:W1024 01:06:34.569000 44895 site-packages/torch/_logging/_internal.py:1154] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
[01:09:05] step 3310/5720 loss 2.5522 lr 6.00e-04 gnorm 0.08 69,367 tok/s peakVRAM 27.0GB
[01:10:37] step 3320/5720 loss 2.5476 lr 6.00e-04 gnorm 0.07 113,742 tok/s peakVRAM 27.0GB
[01:12:09] step 3330/5720 loss 2.5545 lr 6.00e-04 gnorm 0.06 113,417 tok/s peakVRAM 27.0GB
[01:13:42] step 3340/5720 loss 2.5607 lr 6.00e-04 gnorm 0.06 113,416 tok/s peakVRAM 27.0GB
[01:15:14] step 3350/5720 loss 2.5461 lr 6.00e-04 gnorm 0.07 113,434 tok/s peakVRAM 27.0GB
[01:16:47] step 3360/5720 loss 2.5295 lr 6.00e-04 gnorm 0.07 113,333 tok/s peakVRAM 27.0GB
[01:18:19] step 3370/5720 loss 2.5509 lr 6.00e-04 gnorm 0.07 113,383 tok/s peakVRAM 27.0GB
[01:19:52] step 3380/5720 loss 2.5389 lr 6.00e-04 gnorm 0.07 113,391 tok/s peakVRAM 27.0GB
[01:21:24] step 3390/5720 loss 2.5575 lr 6.00e-04 gnorm 0.06 113,447 tok/s peakVRAM 27.0GB
[01:22:57] step 3400/5720 loss 2.5319 lr 6.00e-04 gnorm 0.07 113,367 tok/s peakVRAM 27.0GB
[01:24:29] step 3410/5720 loss 2.5480 lr 6.00e-04 gnorm 0.07 113,478 tok/s peakVRAM 27.0GB
[01:26:02] step 3420/5720 loss 2.5433 lr 6.00e-04 gnorm 0.06 113,481 tok/s peakVRAM 27.0GB
[01:27:34] step 3430/5720 loss 2.5304 lr 6.00e-04 gnorm 0.06 113,440 tok/s peakVRAM 27.0GB
[01:29:06] step 3440/5720 loss 2.5441 lr 6.00e-04 gnorm 0.07 113,379 tok/s peakVRAM 27.0GB
[01:30:39] step 3450/5720 loss 2.5423 lr 6.00e-04 gnorm 0.07 113,376 tok/s peakVRAM 27.0GB
[01:30:49] ckpt @ 3450
[01:32:21] step 3460/5720 loss 2.5297 lr 6.00e-04 gnorm 0.07 103,035 tok/s peakVRAM 27.0GB
[01:33:53] step 3470/5720 loss 2.5408 lr 6.00e-04 gnorm 0.07 113,496 tok/s peakVRAM 27.0GB
[01:35:26] step 3480/5720 loss 2.5270 lr 6.00e-04 gnorm 0.11 113,438 tok/s peakVRAM 27.0GB
[01:36:58] step 3490/5720 loss 2.5178 lr 6.00e-04 gnorm 0.07 113,416 tok/s peakVRAM 27.0GB
[01:38:30] step 3500/5720 loss 2.5450 lr 6.00e-04 gnorm 0.06 113,433 tok/s peakVRAM 27.0GB
[01:40:03] step 3510/5720 loss 2.5315 lr 6.00e-04 gnorm 0.07 113,433 tok/s peakVRAM 27.0GB
[01:41:35] step 3520/5720 loss 2.5568 lr 6.00e-04 gnorm 0.06 113,368 tok/s peakVRAM 27.0GB
[01:43:08] step 3530/5720 loss 2.5248 lr 6.00e-04 gnorm 0.07 113,401 tok/s peakVRAM 27.0GB
[01:44:40] step 3540/5720 loss 2.5320 lr 6.00e-04 gnorm 0.06 113,415 tok/s peakVRAM 27.0GB
[01:46:13] step 3550/5720 loss 2.5185 lr 6.00e-04 gnorm 0.07 113,454 tok/s peakVRAM 27.0GB
[01:47:45] step 3560/5720 loss 2.5318 lr 6.00e-04 gnorm 0.08 113,512 tok/s peakVRAM 27.0GB
[01:49:17] step 3570/5720 loss 2.5362 lr 6.00e-04 gnorm 0.06 113,549 tok/s peakVRAM 27.0GB
[01:50:50] step 3580/5720 loss 2.5203 lr 6.00e-04 gnorm 0.08 113,541 tok/s peakVRAM 27.0GB
[01:52:22] step 3590/5720 loss 2.5193 lr 6.00e-04 gnorm 0.07 113,590 tok/s peakVRAM 27.0GB
[01:53:54] step 3600/5720 loss 2.5297 lr 6.00e-04 gnorm 0.07 113,468 tok/s peakVRAM 27.0GB
[01:54:03] ckpt @ 3600
[01:55:35] step 3610/5720 loss 2.5293 lr 6.00e-04 gnorm 0.07 103,847 tok/s peakVRAM 27.0GB
[01:57:08] step 3620/5720 loss 2.5099 lr 6.00e-04 gnorm 0.06 113,307 tok/s peakVRAM 27.0GB
[01:58:41] step 3630/5720 loss 2.5283 lr 6.00e-04 gnorm 0.07 113,299 tok/s peakVRAM 27.0GB
[02:00:13] step 3640/5720 loss 2.5160 lr 6.00e-04 gnorm 0.07 113,425 tok/s peakVRAM 27.0GB
[02:01:45] step 3650/5720 loss 2.5353 lr 6.00e-04 gnorm 0.07 113,360 tok/s peakVRAM 27.0GB
[02:03:18] step 3660/5720 loss 2.5368 lr 6.00e-04 gnorm 0.06 113,458 tok/s peakVRAM 27.0GB
[02:04:50] step 3670/5720 loss 2.5289 lr 6.00e-04 gnorm 0.07 113,415 tok/s peakVRAM 27.0GB
[02:06:23] step 3680/5720 loss 2.5235 lr 6.00e-04 gnorm 0.07 113,398 tok/s peakVRAM 27.0GB
[02:07:55] step 3690/5720 loss 2.5084 lr 6.00e-04 gnorm 0.07 113,492 tok/s peakVRAM 27.0GB
[02:09:28] step 3700/5720 loss 2.5221 lr 6.00e-04 gnorm 0.07 113,470 tok/s peakVRAM 27.0GB
[02:11:00] step 3710/5720 loss 2.5199 lr 6.00e-04 gnorm 0.06 113,455 tok/s peakVRAM 27.0GB
[02:12:32] step 3720/5720 loss 2.5170 lr 6.00e-04 gnorm 0.07 113,444 tok/s peakVRAM 27.0GB
[02:14:05] step 3730/5720 loss 2.5182 lr 6.00e-04 gnorm 0.07 113,426 tok/s peakVRAM 27.0GB
[02:15:37] step 3740/5720 loss 2.5190 lr 6.00e-04 gnorm 0.06 113,474 tok/s peakVRAM 27.0GB
[02:17:10] step 3750/5720 loss 2.5225 lr 6.00e-04 gnorm 0.07 113,414 tok/s peakVRAM 27.0GB
[02:17:19] ckpt @ 3750
[02:18:51] step 3760/5720 loss 2.5228 lr 6.00e-04 gnorm 0.07 103,658 tok/s peakVRAM 27.0GB
[02:20:23] step 3770/5720 loss 2.5047 lr 6.00e-04 gnorm 0.06 113,352 tok/s peakVRAM 27.0GB
[02:21:56] step 3780/5720 loss 2.5295 lr 6.00e-04 gnorm 0.06 113,185 tok/s peakVRAM 27.0GB
[02:23:29] step 3790/5720 loss 2.5281 lr 6.00e-04 gnorm 0.06 113,212 tok/s peakVRAM 27.0GB
[02:25:01] step 3800/5720 loss 2.5217 lr 6.00e-04 gnorm 0.07 113,285 tok/s peakVRAM 27.0GB
[02:26:34] step 3810/5720 loss 2.5048 lr 6.00e-04 gnorm 0.06 113,373 tok/s peakVRAM 27.0GB
[02:28:06] step 3820/5720 loss 2.4998 lr 6.00e-04 gnorm 0.06 113,377 tok/s peakVRAM 27.0GB
[02:29:39] step 3830/5720 loss 2.5178 lr 6.00e-04 gnorm 0.07 113,413 tok/s peakVRAM 27.0GB
[02:31:11] step 3840/5720 loss 2.5171 lr 6.00e-04 gnorm 0.07 113,319 tok/s peakVRAM 27.0GB
[02:32:44] step 3850/5720 loss 2.5172 lr 6.00e-04 gnorm 0.06 113,274 tok/s peakVRAM 27.0GB
[02:34:16] step 3860/5720 loss 2.5162 lr 6.00e-04 gnorm 0.07 113,267 tok/s peakVRAM 27.0GB
[02:35:49] step 3870/5720 loss 2.5158 lr 6.00e-04 gnorm 0.07 113,344 tok/s peakVRAM 27.0GB
[02:37:21] step 3880/5720 loss 2.4965 lr 6.00e-04 gnorm 0.06 113,398 tok/s peakVRAM 27.0GB
[02:38:54] step 3890/5720 loss 2.4945 lr 6.00e-04 gnorm 0.06 113,395 tok/s peakVRAM 27.0GB
[02:40:26] step 3900/5720 loss 2.5360 lr 6.00e-04 gnorm 0.07 113,331 tok/s peakVRAM 27.0GB
[02:40:35] ckpt @ 3900
[02:42:07] step 3910/5720 loss 2.5101 lr 6.00e-04 gnorm 0.07 103,925 tok/s peakVRAM 27.0GB
[02:43:40] step 3920/5720 loss 2.5018 lr 6.00e-04 gnorm 0.08 113,529 tok/s peakVRAM 27.0GB
[02:45:12] step 3930/5720 loss 2.5049 lr 6.00e-04 gnorm 0.06 113,536 tok/s peakVRAM 27.0GB
[02:46:44] step 3940/5720 loss 2.5074 lr 6.00e-04 gnorm 0.06 113,478 tok/s peakVRAM 27.0GB
[02:48:17] step 3950/5720 loss 2.5109 lr 6.00e-04 gnorm 0.06 113,382 tok/s peakVRAM 27.0GB
[02:49:49] step 3960/5720 loss 2.5092 lr 6.00e-04 gnorm 0.07 113,400 tok/s peakVRAM 27.0GB
[02:51:22] step 3970/5720 loss 2.5176 lr 6.00e-04 gnorm 0.07 113,381 tok/s peakVRAM 27.0GB
[02:52:54] step 3980/5720 loss 2.4976 lr 6.00e-04 gnorm 0.06 113,400 tok/s peakVRAM 27.0GB
[02:54:27] step 3990/5720 loss 2.5135 lr 6.00e-04 gnorm 0.06 113,426 tok/s peakVRAM 27.0GB
[02:55:59] step 4000/5720 loss 2.5116 lr 6.00e-04 gnorm 0.07 113,425 tok/s peakVRAM 27.0GB
[02:57:32] step 4010/5720 loss 2.5003 lr 6.00e-04 gnorm 0.06 113,398 tok/s peakVRAM 27.0GB
[02:59:04] step 4020/5720 loss 2.4992 lr 6.00e-04 gnorm 0.07 113,343 tok/s peakVRAM 27.0GB
[03:00:37] step 4030/5720 loss 2.5163 lr 6.00e-04 gnorm 0.06 113,397 tok/s peakVRAM 27.0GB
[03:02:09] step 4040/5720 loss 2.5014 lr 6.00e-04 gnorm 0.06 113,433 tok/s peakVRAM 27.0GB
[03:03:42] step 4050/5720 loss 2.4976 lr 6.00e-04 gnorm 0.07 113,376 tok/s peakVRAM 27.0GB
[03:03:51] ckpt @ 4050
[03:05:23] step 4060/5720 loss 2.4932 lr 6.00e-04 gnorm 0.06 103,368 tok/s peakVRAM 27.0GB
[03:06:56] step 4070/5720 loss 2.5038 lr 6.00e-04 gnorm 0.06 113,305 tok/s peakVRAM 27.0GB
[03:08:28] step 4080/5720 loss 2.4877 lr 6.00e-04 gnorm 0.06 113,205 tok/s peakVRAM 27.0GB
[03:10:01] step 4090/5720 loss 2.5058 lr 6.00e-04 gnorm 0.08 113,272 tok/s peakVRAM 27.0GB
[03:11:33] step 4100/5720 loss 2.5023 lr 6.00e-04 gnorm 0.06 113,365 tok/s peakVRAM 27.0GB
[03:13:06] step 4110/5720 loss 2.4858 lr 6.00e-04 gnorm 0.06 113,409 tok/s peakVRAM 27.0GB
[03:14:38] step 4120/5720 loss 2.5062 lr 6.00e-04 gnorm 0.06 113,426 tok/s peakVRAM 27.0GB
[03:16:11] step 4130/5720 loss 2.4939 lr 6.00e-04 gnorm 0.07 113,465 tok/s peakVRAM 27.0GB
[03:17:43] step 4140/5720 loss 2.4891 lr 6.00e-04 gnorm 0.07 113,440 tok/s peakVRAM 27.0GB
[03:19:15] step 4150/5720 loss 2.5099 lr 6.00e-04 gnorm 0.07 113,423 tok/s peakVRAM 27.0GB
[03:20:48] step 4160/5720 loss 2.5032 lr 6.00e-04 gnorm 0.06 113,441 tok/s peakVRAM 27.0GB
[03:22:20] step 4170/5720 loss 2.5015 lr 6.00e-04 gnorm 0.06 113,453 tok/s peakVRAM 27.0GB
[03:23:53] step 4180/5720 loss 2.4885 lr 6.00e-04 gnorm 0.06 113,358 tok/s peakVRAM 27.0GB
[03:25:25] step 4190/5720 loss 2.4744 lr 6.00e-04 gnorm 0.06 113,463 tok/s peakVRAM 27.0GB
[03:26:58] step 4200/5720 loss 2.4930 lr 6.00e-04 gnorm 0.06 113,399 tok/s peakVRAM 27.0GB
[03:27:07] ckpt @ 4200
[03:28:39] step 4210/5720 loss 2.4865 lr 6.00e-04 gnorm 0.06 103,676 tok/s peakVRAM 27.0GB
[03:30:11] step 4220/5720 loss 2.5061 lr 6.00e-04 gnorm 0.06 113,469 tok/s peakVRAM 27.0GB
[03:31:44] step 4230/5720 loss 2.4962 lr 6.00e-04 gnorm 0.07 113,397 tok/s peakVRAM 27.0GB
[03:33:16] step 4240/5720 loss 2.4810 lr 6.00e-04 gnorm 0.06 113,409 tok/s peakVRAM 27.0GB
[03:34:49] step 4250/5720 loss 2.4803 lr 6.00e-04 gnorm 0.06 113,468 tok/s peakVRAM 27.0GB
[03:36:21] step 4260/5720 loss 2.4771 lr 6.00e-04 gnorm 0.06 113,387 tok/s peakVRAM 27.0GB
[03:37:54] step 4270/5720 loss 2.4969 lr 6.00e-04 gnorm 0.05 113,421 tok/s peakVRAM 27.0GB
[03:39:26] step 4280/5720 loss 2.4803 lr 6.00e-04 gnorm 0.07 113,524 tok/s peakVRAM 27.0GB
[03:40:58] step 4290/5720 loss 2.4952 lr 6.00e-04 gnorm 0.06 113,518 tok/s peakVRAM 27.0GB
[03:42:31] step 4300/5720 loss 2.4723 lr 6.00e-04 gnorm 0.08 113,462 tok/s peakVRAM 27.0GB
[03:44:03] step 4310/5720 loss 2.4940 lr 6.00e-04 gnorm 0.06 113,483 tok/s peakVRAM 27.0GB
[03:45:35] step 4320/5720 loss 2.4725 lr 6.00e-04 gnorm 0.06 113,486 tok/s peakVRAM 27.0GB
[03:47:08] step 4330/5720 loss 2.4984 lr 6.00e-04 gnorm 0.05 113,468 tok/s peakVRAM 27.0GB
[03:48:40] step 4340/5720 loss 2.4840 lr 6.00e-04 gnorm 0.06 113,541 tok/s peakVRAM 27.0GB
[03:50:13] step 4350/5720 loss 2.4792 lr 6.00e-04 gnorm 0.06 113,480 tok/s peakVRAM 27.0GB
[03:50:22] ckpt @ 4350
[03:51:54] step 4360/5720 loss 2.4763 lr 6.00e-04 gnorm 0.06 103,524 tok/s peakVRAM 27.0GB
[03:53:26] step 4370/5720 loss 2.4752 lr 6.00e-04 gnorm 0.06 113,575 tok/s peakVRAM 27.0GB
[03:54:59] step 4380/5720 loss 2.4888 lr 6.00e-04 gnorm 0.06 113,538 tok/s peakVRAM 27.0GB
[03:56:31] step 4390/5720 loss 2.4892 lr 6.00e-04 gnorm 0.06 113,517 tok/s peakVRAM 27.0GB
[03:58:03] step 4400/5720 loss 2.4695 lr 6.00e-04 gnorm 0.07 113,477 tok/s peakVRAM 27.0GB
[03:59:36] step 4410/5720 loss 2.5139 lr 6.00e-04 gnorm 0.06 113,638 tok/s peakVRAM 27.0GB
[04:01:08] step 4420/5720 loss 2.4856 lr 6.00e-04 gnorm 0.06 113,707 tok/s peakVRAM 27.0GB
[04:02:40] step 4430/5720 loss 2.4762 lr 6.00e-04 gnorm 0.06 113,674 tok/s peakVRAM 27.0GB
[04:04:12] step 4440/5720 loss 2.4879 lr 6.00e-04 gnorm 0.06 113,639 tok/s peakVRAM 27.0GB
[04:05:45] step 4450/5720 loss 2.4720 lr 6.00e-04 gnorm 0.06 113,623 tok/s peakVRAM 27.0GB
[04:07:17] step 4460/5720 loss 2.4603 lr 6.00e-04 gnorm 0.07 113,674 tok/s peakVRAM 27.0GB
[04:08:49] step 4470/5720 loss 2.4891 lr 6.00e-04 gnorm 0.06 113,640 tok/s peakVRAM 27.0GB
[04:10:22] step 4480/5720 loss 2.4717 lr 6.00e-04 gnorm 0.06 113,602 tok/s peakVRAM 27.0GB
[04:11:54] step 4490/5720 loss 2.4672 lr 6.00e-04 gnorm 0.06 113,598 tok/s peakVRAM 27.0GB
[04:13:26] step 4500/5720 loss 2.4725 lr 6.00e-04 gnorm 0.06 113,554 tok/s peakVRAM 27.0GB
[04:13:35] ckpt @ 4500
[04:15:07] step 4510/5720 loss 2.4826 lr 6.00e-04 gnorm 0.07 103,577 tok/s peakVRAM 27.0GB
[04:16:40] step 4520/5720 loss 2.4781 lr 6.00e-04 gnorm 0.05 113,484 tok/s peakVRAM 27.0GB
[04:18:12] step 4530/5720 loss 2.4872 lr 6.00e-04 gnorm 0.05 113,480 tok/s peakVRAM 27.0GB
[04:19:45] step 4540/5720 loss 2.5000 lr 6.00e-04 gnorm 0.06 113,492 tok/s peakVRAM 27.0GB
[04:21:17] step 4550/5720 loss 2.4716 lr 6.00e-04 gnorm 0.06 113,541 tok/s peakVRAM 27.0GB
[04:22:49] step 4560/5720 loss 2.4682 lr 6.00e-04 gnorm 0.06 113,577 tok/s peakVRAM 27.0GB
[04:24:22] step 4570/5720 loss 2.4745 lr 6.00e-04 gnorm 0.06 113,596 tok/s peakVRAM 27.0GB
[04:25:26] ckpt @ 4576
[04:26:03] step 4580/5720 loss 2.4728 lr 6.00e-04 gnorm 0.06 103,453 tok/s peakVRAM 27.0GB
[04:27:35] step 4590/5720 loss 2.4994 lr 6.00e-04 gnorm 0.06 113,740 tok/s peakVRAM 27.0GB
[04:29:07] step 4600/5720 loss 2.4828 lr 6.00e-04 gnorm 0.06 113,582 tok/s peakVRAM 27.0GB
[04:30:40] step 4610/5720 loss 2.4645 lr 6.00e-04 gnorm 0.05 113,497 tok/s peakVRAM 27.0GB
[04:32:12] step 4620/5720 loss 2.4547 lr 6.00e-04 gnorm 0.07 113,496 tok/s peakVRAM 27.0GB
[04:33:45] step 4630/5720 loss 2.4641 lr 6.00e-04 gnorm 0.06 113,575 tok/s peakVRAM 27.0GB
[04:35:17] step 4640/5720 loss 2.4668 lr 6.00e-04 gnorm 0.06 113,600 tok/s peakVRAM 27.0GB
[04:36:49] step 4650/5720 loss 2.4624 lr 6.00e-04 gnorm 0.05 113,640 tok/s peakVRAM 27.0GB
[04:36:58] ckpt @ 4650
[04:38:30] step 4660/5720 loss 2.4818 lr 6.00e-04 gnorm 0.06 103,917 tok/s peakVRAM 27.0GB
[04:40:02] step 4670/5720 loss 2.4701 lr 6.00e-04 gnorm 0.07 113,585 tok/s peakVRAM 27.0GB
[04:41:35] step 4680/5720 loss 2.4618 lr 6.00e-04 gnorm 0.06 113,590 tok/s peakVRAM 27.0GB
[04:43:07] step 4690/5720 loss 2.4736 lr 6.00e-04 gnorm 0.06 113,529 tok/s peakVRAM 27.0GB
[04:44:39] step 4700/5720 loss 2.4750 lr 6.00e-04 gnorm 0.06 113,609 tok/s peakVRAM 27.0GB
[04:46:12] step 4710/5720 loss 2.4840 lr 6.00e-04 gnorm 0.05 113,561 tok/s peakVRAM 27.0GB
[04:47:44] step 4720/5720 loss 2.4700 lr 6.00e-04 gnorm 0.05 113,551 tok/s peakVRAM 27.0GB
[04:49:16] step 4730/5720 loss 2.4700 lr 6.00e-04 gnorm 0.05 113,481 tok/s peakVRAM 27.0GB
[04:50:49] step 4740/5720 loss 2.4530 lr 6.00e-04 gnorm 0.06 113,569 tok/s peakVRAM 27.0GB
[04:52:21] step 4750/5720 loss 2.4710 lr 6.00e-04 gnorm 0.06 113,573 tok/s peakVRAM 27.0GB
[04:53:53] step 4760/5720 loss 2.4664 lr 6.00e-04 gnorm 0.06 113,604 tok/s peakVRAM 27.0GB
[04:55:26] step 4770/5720 loss 2.4679 lr 6.00e-04 gnorm 0.05 113,602 tok/s peakVRAM 27.0GB
[04:56:58] step 4780/5720 loss 2.4541 lr 6.00e-04 gnorm 0.06 113,679 tok/s peakVRAM 27.0GB
[04:58:30] step 4790/5720 loss 2.4816 lr 6.00e-04 gnorm 0.06 113,651 tok/s peakVRAM 27.0GB
[05:00:02] step 4800/5720 loss 2.4758 lr 6.00e-04 gnorm 0.05 113,560 tok/s peakVRAM 27.0GB
[05:00:12] ckpt @ 4800
[05:01:44] step 4810/5720 loss 2.4680 lr 6.00e-04 gnorm 0.06 103,726 tok/s peakVRAM 27.0GB
[05:03:16] step 4820/5720 loss 2.4712 lr 6.00e-04 gnorm 0.06 113,625 tok/s peakVRAM 27.0GB
[05:04:48] step 4830/5720 loss 2.4593 lr 6.00e-04 gnorm 0.06 113,556 tok/s peakVRAM 27.0GB
[05:06:21] step 4840/5720 loss 2.4654 lr 6.00e-04 gnorm 0.06 113,478 tok/s peakVRAM 27.0GB
[05:07:53] step 4850/5720 loss 2.4658 lr 6.00e-04 gnorm 0.06 113,490 tok/s peakVRAM 27.0GB
[05:09:25] step 4860/5720 loss 2.4545 lr 6.00e-04 gnorm 0.06 113,481 tok/s peakVRAM 27.0GB
[05:10:58] step 4870/5720 loss 2.4667 lr 6.00e-04 gnorm 0.06 113,478 tok/s peakVRAM 27.0GB
[05:12:30] step 4880/5720 loss 2.4510 lr 6.00e-04 gnorm 0.06 113,546 tok/s peakVRAM 27.0GB
[05:14:02] step 4890/5720 loss 2.4516 lr 6.00e-04 gnorm 0.06 113,631 tok/s peakVRAM 27.0GB
[05:15:35] step 4900/5720 loss 2.4708 lr 6.00e-04 gnorm 0.06 113,538 tok/s peakVRAM 27.0GB
[05:17:07] step 4910/5720 loss 2.4488 lr 6.00e-04 gnorm 0.06 113,455 tok/s peakVRAM 27.0GB
[05:18:40] step 4920/5720 loss 2.4655 lr 6.00e-04 gnorm 0.06 113,427 tok/s peakVRAM 27.0GB
[05:20:12] step 4930/5720 loss 2.4648 lr 6.00e-04 gnorm 0.06 113,461 tok/s peakVRAM 27.0GB
[05:21:45] step 4940/5720 loss 2.4561 lr 6.00e-04 gnorm 0.06 113,460 tok/s peakVRAM 27.0GB
[05:23:17] step 4950/5720 loss 2.4695 lr 6.00e-04 gnorm 0.06 113,573 tok/s peakVRAM 27.0GB
[05:23:26] ckpt @ 4950
[05:24:58] step 4960/5720 loss 2.4649 lr 6.00e-04 gnorm 0.06 103,876 tok/s peakVRAM 27.0GB
[05:26:30] step 4970/5720 loss 2.4609 lr 6.00e-04 gnorm 0.06 113,532 tok/s peakVRAM 27.0GB
[05:28:03] step 4980/5720 loss 2.4566 lr 6.00e-04 gnorm 0.06 113,478 tok/s peakVRAM 27.0GB
[05:29:35] step 4990/5720 loss 2.4424 lr 6.00e-04 gnorm 0.05 113,539 tok/s peakVRAM 27.0GB
[05:31:07] step 5000/5720 loss 2.4350 lr 6.00e-04 gnorm 0.05 113,491 tok/s peakVRAM 27.0GB
[05:32:40] step 5010/5720 loss 2.4711 lr 6.00e-04 gnorm 0.06 113,661 tok/s peakVRAM 27.0GB
[05:34:12] step 5020/5720 loss 2.4644 lr 6.00e-04 gnorm 0.05 113,671 tok/s peakVRAM 27.0GB
[05:35:44] step 5030/5720 loss 2.4588 lr 6.00e-04 gnorm 0.06 113,622 tok/s peakVRAM 27.0GB
[05:37:16] step 5040/5720 loss 2.4524 lr 6.00e-04 gnorm 0.05 113,573 tok/s peakVRAM 27.0GB
[05:38:49] step 5050/5720 loss 2.4439 lr 6.00e-04 gnorm 0.05 113,549 tok/s peakVRAM 27.0GB
[05:40:21] step 5060/5720 loss 2.4672 lr 6.00e-04 gnorm 0.05 113,565 tok/s peakVRAM 27.0GB
[05:41:53] step 5070/5720 loss 2.4733 lr 6.00e-04 gnorm 0.06 113,577 tok/s peakVRAM 27.0GB
[05:43:26] step 5080/5720 loss 2.4475 lr 6.00e-04 gnorm 0.06 113,594 tok/s peakVRAM 27.0GB
[05:44:58] step 5090/5720 loss 2.4591 lr 6.00e-04 gnorm 0.05 113,671 tok/s peakVRAM 27.0GB
[05:46:30] step 5100/5720 loss 2.4651 lr 6.00e-04 gnorm 0.06 113,745 tok/s peakVRAM 27.0GB
[05:46:39] ckpt @ 5100
[05:48:11] step 5110/5720 loss 2.4532 lr 6.00e-04 gnorm 0.06 103,674 tok/s peakVRAM 27.0GB
[05:49:44] step 5120/5720 loss 2.4670 lr 6.00e-04 gnorm 0.05 113,645 tok/s peakVRAM 27.0GB
[05:51:16] step 5130/5720 loss 2.4614 lr 6.00e-04 gnorm 0.06 113,652 tok/s peakVRAM 27.0GB
[05:52:48] step 5140/5720 loss 2.4420 lr 6.00e-04 gnorm 0.06 113,699 tok/s peakVRAM 27.0GB
[05:54:20] step 5150/5720 loss 2.4372 lr 6.00e-04 gnorm 0.05 113,722 tok/s peakVRAM 27.0GB
[05:55:52] step 5160/5720 loss 2.4453 lr 6.00e-04 gnorm 0.06 113,718 tok/s peakVRAM 27.0GB
[05:57:25] step 5170/5720 loss 2.4357 lr 6.00e-04 gnorm 0.05 113,762 tok/s peakVRAM 27.0GB
[05:58:57] step 5180/5720 loss 2.4591 lr 6.00e-04 gnorm 0.06 113,793 tok/s peakVRAM 27.0GB
[06:00:29] step 5190/5720 loss 2.4595 lr 6.00e-04 gnorm 0.07 113,707 tok/s peakVRAM 27.0GB
[06:02:01] step 5200/5720 loss 2.4505 lr 6.00e-04 gnorm 0.05 113,753 tok/s peakVRAM 27.0GB
[06:03:33] step 5210/5720 loss 2.4398 lr 6.00e-04 gnorm 0.05 113,728 tok/s peakVRAM 27.0GB
[06:05:06] step 5220/5720 loss 2.4542 lr 6.00e-04 gnorm 0.05 113,741 tok/s peakVRAM 27.0GB
[06:06:38] step 5230/5720 loss 2.4414 lr 6.00e-04 gnorm 0.06 113,674 tok/s peakVRAM 27.0GB
[06:08:10] step 5240/5720 loss 2.4512 lr 6.00e-04 gnorm 0.06 113,699 tok/s peakVRAM 27.0GB
[06:09:42] step 5250/5720 loss 2.4385 lr 6.00e-04 gnorm 0.05 113,676 tok/s peakVRAM 27.0GB
[06:09:51] ckpt @ 5250
[06:11:23] step 5260/5720 loss 2.4331 lr 6.00e-04 gnorm 0.05 103,747 tok/s peakVRAM 27.0GB
[06:12:56] step 5270/5720 loss 2.4446 lr 6.00e-04 gnorm 0.05 113,602 tok/s peakVRAM 27.0GB
[06:14:28] step 5280/5720 loss 2.4504 lr 6.00e-04 gnorm 0.06 113,475 tok/s peakVRAM 27.0GB
[06:16:00] step 5290/5720 loss 2.4449 lr 6.00e-04 gnorm 0.05 113,529 tok/s peakVRAM 27.0GB
[06:17:33] step 5300/5720 loss 2.4384 lr 6.00e-04 gnorm 0.06 113,653 tok/s peakVRAM 27.0GB
[06:19:05] step 5310/5720 loss 2.4586 lr 6.00e-04 gnorm 0.05 113,652 tok/s peakVRAM 27.0GB
[06:20:37] step 5320/5720 loss 2.4554 lr 6.00e-04 gnorm 0.06 113,625 tok/s peakVRAM 27.0GB
[06:22:10] step 5330/5720 loss 2.4484 lr 6.00e-04 gnorm 0.05 113,641 tok/s peakVRAM 27.0GB
[06:23:42] step 5340/5720 loss 2.4275 lr 6.00e-04 gnorm 0.06 113,642 tok/s peakVRAM 27.0GB
[06:25:14] step 5350/5720 loss 2.4413 lr 6.00e-04 gnorm 0.06 113,608 tok/s peakVRAM 27.0GB
[06:26:46] step 5360/5720 loss 2.4409 lr 6.00e-04 gnorm 0.06 113,614 tok/s peakVRAM 27.0GB
[06:28:19] step 5370/5720 loss 2.4339 lr 6.00e-04 gnorm 0.06 113,686 tok/s peakVRAM 27.0GB
[06:29:51] step 5380/5720 loss 2.4382 lr 6.00e-04 gnorm 0.06 113,681 tok/s peakVRAM 27.0GB
[06:31:23] step 5390/5720 loss 2.4466 lr 6.00e-04 gnorm 0.05 113,681 tok/s peakVRAM 27.0GB
[06:32:55] step 5400/5720 loss 2.4364 lr 6.00e-04 gnorm 0.05 113,704 tok/s peakVRAM 27.0GB
[06:33:04] ckpt @ 5400
[06:34:36] step 5410/5720 loss 2.4301 lr 6.00e-04 gnorm 0.05 103,637 tok/s peakVRAM 27.0GB
[06:36:09] step 5420/5720 loss 2.4480 lr 6.00e-04 gnorm 0.05 113,724 tok/s peakVRAM 27.0GB
[06:37:41] step 5430/5720 loss 2.4492 lr 6.00e-04 gnorm 0.05 113,732 tok/s peakVRAM 27.0GB
[06:39:13] step 5440/5720 loss 2.4397 lr 6.00e-04 gnorm 0.05 113,647 tok/s peakVRAM 27.0GB
[06:40:45] step 5450/5720 loss 2.4379 lr 6.00e-04 gnorm 0.06 113,588 tok/s peakVRAM 27.0GB
[06:42:18] step 5460/5720 loss 2.4482 lr 6.00e-04 gnorm 0.06 113,606 tok/s peakVRAM 27.0GB
[06:43:50] step 5470/5720 loss 2.4460 lr 6.00e-04 gnorm 0.06 113,590 tok/s peakVRAM 27.0GB
[06:45:22] step 5480/5720 loss 2.4472 lr 6.00e-04 gnorm 0.05 113,613 tok/s peakVRAM 27.0GB
[06:46:55] step 5490/5720 loss 2.4374 lr 6.00e-04 gnorm 0.05 113,740 tok/s peakVRAM 27.0GB
[06:48:27] step 5500/5720 loss 2.4384 lr 6.00e-04 gnorm 0.05 113,749 tok/s peakVRAM 27.0GB
[06:49:59] step 5510/5720 loss 2.4428 lr 6.00e-04 gnorm 0.06 113,730 tok/s peakVRAM 27.0GB
[06:51:31] step 5520/5720 loss 2.4301 lr 6.00e-04 gnorm 0.05 113,717 tok/s peakVRAM 27.0GB
[06:53:03] step 5530/5720 loss 2.4413 lr 6.00e-04 gnorm 0.06 113,766 tok/s peakVRAM 27.0GB
[06:54:36] step 5540/5720 loss 2.4417 lr 6.00e-04 gnorm 0.05 113,676 tok/s peakVRAM 27.0GB
[06:56:08] step 5550/5720 loss 2.4430 lr 6.00e-04 gnorm 0.05 113,650 tok/s peakVRAM 27.0GB
[06:56:17] ckpt @ 5550
[06:57:49] step 5560/5720 loss 2.4515 lr 6.00e-04 gnorm 0.05 103,950 tok/s peakVRAM 27.0GB
[06:59:21] step 5570/5720 loss 2.4355 lr 6.00e-04 gnorm 0.05 113,729 tok/s peakVRAM 27.0GB
[07:00:53] step 5580/5720 loss 2.4500 lr 6.00e-04 gnorm 0.05 113,739 tok/s peakVRAM 27.0GB
[07:02:25] step 5590/5720 loss 2.4494 lr 6.00e-04 gnorm 0.06 113,699 tok/s peakVRAM 27.0GB
[07:03:58] step 5600/5720 loss 2.4437 lr 6.00e-04 gnorm 0.06 113,682 tok/s peakVRAM 27.0GB
[07:05:30] step 5610/5720 loss 2.4474 lr 6.00e-04 gnorm 0.05 113,640 tok/s peakVRAM 27.0GB
[07:07:02] step 5620/5720 loss 2.4527 lr 6.00e-04 gnorm 0.05 113,734 tok/s peakVRAM 27.0GB
[07:08:34] step 5630/5720 loss 2.4373 lr 6.00e-04 gnorm 0.06 113,689 tok/s peakVRAM 27.0GB
[07:10:06] step 5640/5720 loss 2.4450 lr 6.00e-04 gnorm 0.05 113,696 tok/s peakVRAM 27.0GB
[07:11:39] step 5650/5720 loss 2.4493 lr 6.00e-04 gnorm 0.06 113,643 tok/s peakVRAM 27.0GB
[07:13:11] step 5660/5720 loss 2.4615 lr 6.00e-04 gnorm 0.05 113,659 tok/s peakVRAM 27.0GB
[07:14:43] step 5670/5720 loss 2.4395 lr 6.00e-04 gnorm 0.05 113,718 tok/s peakVRAM 27.0GB
[07:16:15] step 5680/5720 loss 2.4378 lr 6.00e-04 gnorm 0.05 113,697 tok/s peakVRAM 27.0GB
[07:17:48] step 5690/5720 loss 2.4196 lr 6.00e-04 gnorm 0.05 113,737 tok/s peakVRAM 27.0GB
[07:19:20] step 5700/5720 loss 2.4312 lr 6.00e-04 gnorm 0.06 113,698 tok/s peakVRAM 27.0GB
[07:19:29] ckpt @ 5700
[07:21:01] step 5710/5720 loss 2.4392 lr 6.00e-04 gnorm 0.06 103,762 tok/s peakVRAM 27.0GB
[07:22:33] step 5720/5720 loss 2.4187 lr 6.00e-04 gnorm 0.06 113,655 tok/s peakVRAM 27.0GB
[07:22:42] ckpt @ 5720
[07:22:42] DONE