new data

logs/main_log.txt

@@ -57155,3 +57155,179 @@ time (ms)
 time (ms)
  iteration       48/  292968 | consumed samples:        98304 | consumed tokens:      6291456 | elapsed time per iteration (ms): 95739.1 | learning rate: 2.727E-05 | global batch size:  2048 | lm loss: 8.545946E+00 | loss scale: 4096.0 | grad norm: 47054.317 | num zeros: 0.0 | curriculum seqlen:    64 | number of skipped iterations:   0 | number of nan iterations:   0 |
 time (ms)
+ iteration       49/  292968 | consumed samples:       100352 | consumed tokens:      6422528 | elapsed time per iteration (ms): 94691.8 | learning rate: 2.783E-05 | global batch size:  2048 | lm loss: 8.737078E+00 | loss scale: 4096.0 | grad norm: 147984.860 | num zeros: 0.0 | curriculum seqlen:    64 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration       50/  292968 | consumed samples:       102400 | consumed tokens:      6553600 | elapsed time per iteration (ms): 96272.3 | learning rate: 2.840E-05 | global batch size:  2048 | lm loss: 8.645372E+00 | loss scale: 4096.0 | grad norm: 100115.276 | num zeros: 0.0 | curriculum seqlen:    64 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration       51/  292968 | consumed samples:       104448 | consumed tokens:      6684672 | elapsed time per iteration (ms): 96225.8 | learning rate: 2.897E-05 | global batch size:  2048 | lm loss: 8.786609E+00 | loss scale: 4096.0 | grad norm: 138446.949 | num zeros: 0.0 | curriculum seqlen:    64 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration       52/  292968 | consumed samples:       106496 | consumed tokens:      6815744 | elapsed time per iteration (ms): 93767.5 | learning rate: 2.954E-05 | global batch size:  2048 | lm loss: 8.520951E+00 | loss scale: 4096.0 | grad norm: 72259.747 | num zeros: 0.0 | curriculum seqlen:    64 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration       53/  292968 | consumed samples:       108544 | consumed tokens:      6946816 | elapsed time per iteration (ms): 95896.3 | learning rate: 3.011E-05 | global batch size:  2048 | lm loss: 8.274112E+00 | loss scale: 4096.0 | grad norm: 30192.728 | num zeros: 0.0 | curriculum seqlen:    64 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration       54/  292968 | consumed samples:       110592 | consumed tokens:      7077888 | elapsed time per iteration (ms): 94348.2 | learning rate: 3.067E-05 | global batch size:  2048 | lm loss: 8.363799E+00 | loss scale: 4096.0 | grad norm: 70109.113 | num zeros: 0.0 | curriculum seqlen:    64 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration       55/  292968 | consumed samples:       112640 | consumed tokens:      7208960 | elapsed time per iteration (ms): 96086.5 | learning rate: 3.124E-05 | global batch size:  2048 | lm loss: 8.283342E+00 | loss scale: 4096.0 | grad norm: 32869.639 | num zeros: 0.0 | curriculum seqlen:    64 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+Killing subprocess 4027110
+slurmstepd: error: *** STEP 1655850.0 ON r6i4n5 CANCELLED AT 2021-10-22T19:05:02 ***
+Killing subprocess 4027111
+Killing subprocess 4027112
+Killing subprocess 1360613
+Killing subprocess 4027114
+Main process received SIGTERM, exiting
+Killing subprocess 512071
+Killing subprocess 1360614
+Killing subprocess 392892
+Killing subprocess 1360615
+Killing subprocess 2123183
+Killing subprocess 1360617
+Killing subprocess 392893
+Killing subprocess 512072
+Killing subprocess 512073
+Killing subprocess 2123184
+Killing subprocess 512074
+Killing subprocess 392894
+Main process received SIGTERM, exiting
+Killing subprocess 1339268
+Main process received SIGTERM, exiting
+Killing subprocess 4161749
+Killing subprocess 392895
+Killing subprocess 4117263
+Main process received SIGTERM, exiting
+Killing subprocess 1339269
+Killing subprocess 642447
+Killing subprocess 4161750
+Killing subprocess 4161751
+Killing subprocess 1311333
+Killing subprocess 1795309
+Killing subprocess 1084264
+Killing subprocess 2354914
+Killing subprocess 4161753
+Killing subprocess 1311334
+Killing subprocess 2123185
+Killing subprocess 1036398
+Killing subprocess 1795310
+Killing subprocess 105206
+Killing subprocess 2123186
+Killing subprocess 1084265
+Main process received SIGTERM, exiting
+Killing subprocess 1795311
+Killing subprocess 2354347
+Killing subprocess 2354915
+Killing subprocess 642448
+Killing subprocess 533215
+Killing subprocess 1084266
+Killing subprocess 1339270
+Killing subprocess 105207
+Killing subprocess 4117264
+Killing subprocess 23037
+Killing subprocess 19863
+Killing subprocess 1311335
+Killing subprocess 3570489
+Killing subprocess 837305
+Killing subprocess 743334
+Killing subprocess 1339271
+Killing subprocess 4117265
+Killing subprocess 2354348
+Killing subprocess 1036399
+Killing subprocess 3570490
+Killing subprocess 1484262
+Killing subprocess 533216
+Killing subprocess 743335
+Killing subprocess 1084268
+Main process received SIGTERM, exiting
+Killing subprocess 4117266
+Killing subprocess 23038
+Killing subprocess 19864
+Killing subprocess 2354349
+Killing subprocess 1036400
+Killing subprocess 2354916
+Killing subprocess 1795313
+Killing subprocess 837306
+Killing subprocess 105208
+Main process received SIGTERM, exiting
+Killing subprocess 23039
+Killing subprocess 19865
+Killing subprocess 2354350
+Killing subprocess 2354918
+Killing subprocess 3570491
+Killing subprocess 2376378
+Main process received SIGTERM, exiting
+Killing subprocess 1484263
+Killing subprocess 837307
+Killing subprocess 1380399
+Killing subprocess 642449
+Killing subprocess 533217
+Killing subprocess 743336
+Killing subprocess 105209
+Killing subprocess 19867
+Killing subprocess 1311336
+Killing subprocess 572722
+Killing subprocess 700598
+Main process received SIGTERM, exiting
+Killing subprocess 2376379
+Killing subprocess 2581538
+Killing subprocess 837309
+Killing subprocess 1380400
+Killing subprocess 642450
+Killing subprocess 533218
+Killing subprocess 743337
+Main process received SIGTERM, exiting
+Killing subprocess 23040
+Main process received SIGTERM, exiting
+Killing subprocess 1036401
+Main process received SIGTERM, exiting
+Killing subprocess 572723
+Killing subprocess 1729138
+Killing subprocess 2376380
+Killing subprocess 1484264
+Main process received SIGTERM, exiting
+Main process received SIGTERM, exiting
+Main process received SIGTERM, exiting
+Main process received SIGTERM, exiting
+Main process received SIGTERM, exiting
+Main process received SIGTERM, exiting
+Killing subprocess 572724
+Killing subprocess 700599
+Killing subprocess 1959733
+Killing subprocess 1618027
+Killing subprocess 1654112
+Killing subprocess 2376381
+Killing subprocess 1484266
+Killing subprocess 2581539
+Killing subprocess 1380401
+Main process received SIGTERM, exiting
+Killing subprocess 700600
+Killing subprocess 3570492
+Main process received SIGTERM, exiting
+Killing subprocess 1380403
+Main process received SIGTERM, exiting
+Killing subprocess 1959734
+Killing subprocess 1618028
+Main process received SIGTERM, exiting
+Main process received SIGTERM, exiting
+Killing subprocess 1729139
+Killing subprocess 1654113
+Killing subprocess 2581540
+Killing subprocess 572725
+Killing subprocess 700601
+Killing subprocess 1618029
+Killing subprocess 2581542
+Main process received SIGTERM, exiting
+Main process received SIGTERM, exiting
+Killing subprocess 1959735
+Main process received SIGTERM, exiting
+Killing subprocess 1618030
+Killing subprocess 1654114
+Killing subprocess 1729140
+Killing subprocess 1729141
+Killing subprocess 1959737
+Main process received SIGTERM, exiting
+Killing subprocess 1654116
+Main process received SIGTERM, exiting
+Main process received SIGTERM, exiting
+Main process received SIGTERM, exiting
+Main process received SIGTERM, exiting
+Main process received SIGTERM, exiting
+srun: Job step aborted: Waiting up to 62 seconds for job step to finish.