new data

logs/main_log.txt

@@ -12205,3 +12205,86 @@ Number of parameters without embeddings: 1.20860672 billion
  iteration    18200/  296023 | consumed samples:      4238784 | consumed tokens:   8681029632 | elapsed time per iteration (ms): 4638.8 | learning rate: 1.986E-04 | global batch size:   512 | lm loss: 1.881505E+00 | loss scale: 32768.0 | grad norm: 4246.005 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
 [Rank 0] (after 18200 iterations) memory (MB) | allocated: 1631.6650390625 | max allocated: 3929.2744140625 | reserved: 6816.0 | max reserved: 6816.0
  iteration    18400/  296023 | consumed samples:      4341184 | consumed tokens:   8890744832 | elapsed time per iteration (ms): 4630.3 | learning rate: 1.985E-04 | global batch size:   512 | lm loss: 1.853725E+00 | loss scale: 16384.0 | grad norm: 2445.581 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+ iteration    18600/  296023 | consumed samples:      4443584 | consumed tokens:   9100460032 | elapsed time per iteration (ms): 4642.4 | learning rate: 1.984E-04 | global batch size:   512 | lm loss: 1.849770E+00 | loss scale: 16384.0 | grad norm: 2880.275 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+Killing subprocess 73114
+Killing subprocess 73115
+Killing subprocess 73116
+Killing subprocess 90168
+Killing subprocess 78775
+Killing subprocess 73117
+Killing subprocess 90017
+Killing subprocess 90169
+Killing subprocess 78776
+Main process received SIGTERM, exiting
+Killing subprocess 85084
+Killing subprocess 86427
+Killing subprocess 90170
+Killing subprocess 67764
+Killing subprocess 90018
+Killing subprocess 75423
+Killing subprocess 76711
+Killing subprocess 481575
+Killing subprocess 78777
+Killing subprocess 85085
+Killing subprocess 78778
+Killing subprocess 69992
+Killing subprocess 90171
+Killing subprocess 86428
+Killing subprocess 73589
+Killing subprocess 90019
+Killing subprocess 76712
+Killing subprocess 481576
+Killing subprocess 75424
+Killing subprocess 90020
+Killing subprocess 85086
+Killing subprocess 67765
+Main process received SIGTERM, exiting
+Killing subprocess 86429
+Killing subprocess 69993
+Killing subprocess 85087
+Main process received SIGTERM, exiting
+Killing subprocess 481612
+Killing subprocess 73590
+Killing subprocess 76713
+Killing subprocess 75425
+Killing subprocess 69467
+Killing subprocess 68888
+Killing subprocess 481577
+Killing subprocess 86430
+Main process received SIGTERM, exiting
+Killing subprocess 67766
+slurmstepd: error: *** STEP 1840970.0 ON r10i3n3 CANCELLED AT 2021-11-05T12:10:00 ***
+Killing subprocess 76714
+Killing subprocess 73591
+Killing subprocess 69994
+Killing subprocess 67767
+Killing subprocess 75426
+Main process received SIGTERM, exiting
+Main process received SIGTERM, exiting
+Killing subprocess 481613
+Killing subprocess 73592
+Killing subprocess 69468
+Main process received SIGTERM, exiting
+Killing subprocess 68889
+Killing subprocess 481578
+Killing subprocess 69469
+Main process received SIGTERM, exiting
+Killing subprocess 69995
+Main process received SIGTERM, exiting
+Main process received SIGTERM, exiting
+Killing subprocess 481614
+Killing subprocess 69470
+Killing subprocess 68890
+Killing subprocess 76149
+Main process received SIGTERM, exiting
+Killing subprocess 68891
+Main process received SIGTERM, exiting
+Main process received SIGTERM, exiting
+Killing subprocess 481616
+Killing subprocess 76150
+Main process received SIGTERM, exiting
+Main process received SIGTERM, exiting
+Killing subprocess 76151
+Killing subprocess 76152
+Main process received SIGTERM, exiting
+srun: Job step aborted: Waiting up to 62 seconds for job step to finish.