ViV1T / viv1t_003 /output.log
bryanlimy's picture
rename folders
e148497
Use bfloat16 for core module.
Use parallel attention and MLP in ViViT.
Epoch 001/400
Train loss: 112704208.00 correlation: 0.0126
Validation loss: 199545376.00 correlation: 0.0293
Elapse: 540.35s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 002/400
Train loss: 97414480.00 correlation: 0.0393
Validation loss: 198820640.00 correlation: 0.0410
Elapse: 549.83s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 003/400
Train loss: 95851936.00 correlation: 0.0545
Validation loss: 197542912.00 correlation: 0.0493
Elapse: 552.73s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 004/400
Train loss: 94762680.00 correlation: 0.0658
Validation loss: 195978896.00 correlation: 0.0592
Elapse: 553.22s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 005/400
Train loss: 93366904.00 correlation: 0.0807
Validation loss: 193138784.00 correlation: 0.0786
Elapse: 551.37s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 006/400
Train loss: 91530000.00 correlation: 0.0996
Validation loss: 190414640.00 correlation: 0.0957
Elapse: 548.87s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 007/400
Train loss: 90052112.00 correlation: 0.1151
Validation loss: 187958800.00 correlation: 0.1117
Elapse: 545.45s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 008/400
Train loss: 88550296.00 correlation: 0.1303
Validation loss: 185819104.00 correlation: 0.1231
Elapse: 542.13s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 009/400
Train loss: 87211088.00 correlation: 0.1444
Validation loss: 183781008.00 correlation: 0.1379
Elapse: 540.39s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 010/400
Train loss: 85912744.00 correlation: 0.1575
Validation loss: 182160864.00 correlation: 0.1483
Elapse: 539.93s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 011/400
Train loss: 84825776.00 correlation: 0.1681
Validation loss: 180796960.00 correlation: 0.1574
Elapse: 539.05s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 012/400
Train loss: 83932096.00 correlation: 0.1768
Validation loss: 179665872.00 correlation: 0.1651
Elapse: 539.83s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 013/400
Train loss: 83185960.00 correlation: 0.1841
Validation loss: 178867920.00 correlation: 0.1706
Elapse: 540.49s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 014/400
Train loss: 82619600.00 correlation: 0.1896
Validation loss: 178183616.00 correlation: 0.1749
Elapse: 540.25s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 015/400
Train loss: 82049104.00 correlation: 0.1953
Validation loss: 177195872.00 correlation: 0.1811
Elapse: 539.98s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 016/400
Train loss: 81438744.00 correlation: 0.2013
Validation loss: 176732480.00 correlation: 0.1848
Elapse: 539.92s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 017/400
Train loss: 80981184.00 correlation: 0.2058
Validation loss: 175869328.00 correlation: 0.1895
Elapse: 540.34s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 018/400
Train loss: 80463984.00 correlation: 0.2108
Validation loss: 175137504.00 correlation: 0.1941
Elapse: 540.52s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 019/400
Train loss: 80076456.00 correlation: 0.2145
Validation loss: 174743216.00 correlation: 0.1976
Elapse: 540.79s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 020/400
Train loss: 79770240.00 correlation: 0.2180
Validation loss: 174354656.00 correlation: 0.1999
Elapse: 541.02s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 021/400
Train loss: 79389680.00 correlation: 0.2210
Validation loss: 173737072.00 correlation: 0.2049
Elapse: 540.85s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 022/400
Train loss: 79099264.00 correlation: 0.2238
Validation loss: 173426080.00 correlation: 0.2057
Elapse: 542.08s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 023/400
Train loss: 78893584.00 correlation: 0.2259
Validation loss: 172965088.00 correlation: 0.2094
Elapse: 542.73s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 024/400
Train loss: 78610448.00 correlation: 0.2286
Validation loss: 172757408.00 correlation: 0.2102
Elapse: 543.17s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 025/400
Train loss: 78350984.00 correlation: 0.2311
Validation loss: 172301040.00 correlation: 0.2136
Elapse: 543.01s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 026/400
Train loss: 78082256.00 correlation: 0.2338
Validation loss: 172086432.00 correlation: 0.2145
Elapse: 543.35s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 027/400
Train loss: 77827456.00 correlation: 0.2363
Validation loss: 171786464.00 correlation: 0.2169
Elapse: 543.66s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 028/400
Train loss: 77638136.00 correlation: 0.2379
Validation loss: 171793264.00 correlation: 0.2171
Elapse: 543.62s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 029/400
Train loss: 77553520.00 correlation: 0.2390
Validation loss: 171183824.00 correlation: 0.2206
Elapse: 543.14s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 030/400
Train loss: 77293184.00 correlation: 0.2414
Validation loss: 171063472.00 correlation: 0.2218
Elapse: 542.93s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 031/400
Train loss: 77137136.00 correlation: 0.2425
Validation loss: 170927232.00 correlation: 0.2215
Elapse: 542.90s
Epoch 032/400
Train loss: 77036008.00 correlation: 0.2439
Validation loss: 170678112.00 correlation: 0.2241
Elapse: 544.22s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 033/400
Train loss: 76850512.00 correlation: 0.2458
Validation loss: 170585168.00 correlation: 0.2249
Elapse: 543.38s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 034/400
Train loss: 76745528.00 correlation: 0.2467
Validation loss: 170361008.00 correlation: 0.2261
Elapse: 543.46s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 035/400
Train loss: 76631640.00 correlation: 0.2479
Validation loss: 170159952.00 correlation: 0.2272
Elapse: 544.31s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 036/400
Train loss: 76572216.00 correlation: 0.2483
Validation loss: 170068624.00 correlation: 0.2280
Elapse: 544.03s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 037/400
Train loss: 76306000.00 correlation: 0.2511
Validation loss: 170078272.00 correlation: 0.2281
Elapse: 543.66s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 038/400
Train loss: 76521968.00 correlation: 0.2488
Validation loss: 170034304.00 correlation: 0.2284
Elapse: 543.78s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 039/400
Train loss: 76277552.00 correlation: 0.2512
Validation loss: 169955232.00 correlation: 0.2283
Elapse: 544.11s
Epoch 040/400
Train loss: 76083856.00 correlation: 0.2532
Validation loss: 169547568.00 correlation: 0.2316
Elapse: 544.27s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 041/400
Train loss: 76034800.00 correlation: 0.2536
Validation loss: 169778400.00 correlation: 0.2301
Elapse: 544.31s
Epoch 042/400
Train loss: 75948104.00 correlation: 0.2542
Validation loss: 169703472.00 correlation: 0.2304
Elapse: 543.89s
Epoch 043/400
Train loss: 75900336.00 correlation: 0.2549
Validation loss: 169460816.00 correlation: 0.2314
Elapse: 543.95s
Epoch 044/400
Train loss: 75856872.00 correlation: 0.2554
Validation loss: 169446544.00 correlation: 0.2314
Elapse: 543.85s
Epoch 045/400
Train loss: 75771024.00 correlation: 0.2563
Validation loss: 169373568.00 correlation: 0.2320
Elapse: 544.58s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 046/400
Train loss: 75747888.00 correlation: 0.2564
Validation loss: 169484960.00 correlation: 0.2316
Elapse: 544.29s
Epoch 047/400
Train loss: 75614488.00 correlation: 0.2578
Validation loss: 169374048.00 correlation: 0.2332
Elapse: 544.59s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 048/400
Train loss: 75621768.00 correlation: 0.2577
Validation loss: 169829152.00 correlation: 0.2292
Elapse: 544.60s
Epoch 049/400
Train loss: 75540576.00 correlation: 0.2584
Validation loss: 169151584.00 correlation: 0.2322
Elapse: 545.07s
Epoch 050/400
Train loss: 75552112.00 correlation: 0.2587
Validation loss: 169218528.00 correlation: 0.2327
Elapse: 544.47s
Epoch 051/400
Train loss: 75440728.00 correlation: 0.2597
Validation loss: 169177248.00 correlation: 0.2329
Elapse: 544.75s
Epoch 052/400
Train loss: 75499728.00 correlation: 0.2591
Validation loss: 169380720.00 correlation: 0.2327
Elapse: 544.77s
Loaded checkpoint from epoch 47 (correlation: 0.2332).
Reduce learning rate of core to 1.4400e-03 (num. reduce: 1).
Reduce learning rate of readouts to 1.0800e-03 (num. reduce: 1).
Reduce learning rate of shifters to 1.0800e-03 (num. reduce: 1).
Epoch 053/400
Train loss: 73633064.00 correlation: 0.2750
Validation loss: 167526240.00 correlation: 0.2446
Elapse: 545.49s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 054/400
Train loss: 72992880.00 correlation: 0.2809
Validation loss: 167505792.00 correlation: 0.2447
Elapse: 545.66s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 055/400
Train loss: 72791024.00 correlation: 0.2827
Validation loss: 167337248.00 correlation: 0.2451
Elapse: 545.82s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 056/400
Train loss: 72699672.00 correlation: 0.2837
Validation loss: 167396864.00 correlation: 0.2453
Elapse: 545.65s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 057/400
Train loss: 72666128.00 correlation: 0.2841
Validation loss: 167320224.00 correlation: 0.2457
Elapse: 546.33s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 058/400
Train loss: 72662680.00 correlation: 0.2840
Validation loss: 167358624.00 correlation: 0.2457
Elapse: 545.38s
Epoch 059/400
Train loss: 72528456.00 correlation: 0.2854
Validation loss: 167347344.00 correlation: 0.2456
Elapse: 546.44s
Epoch 060/400
Train loss: 72553424.00 correlation: 0.2854
Validation loss: 167294624.00 correlation: 0.2462
Elapse: 546.17s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 061/400
Train loss: 72496144.00 correlation: 0.2860
Validation loss: 167428896.00 correlation: 0.2444
Elapse: 545.96s
Epoch 062/400
Train loss: 72479344.00 correlation: 0.2859
Validation loss: 167235104.00 correlation: 0.2454
Elapse: 546.11s
Epoch 063/400
Train loss: 72437912.00 correlation: 0.2863
Validation loss: 167384528.00 correlation: 0.2447
Elapse: 546.05s
Epoch 064/400
Train loss: 72443952.00 correlation: 0.2864
Validation loss: 167276464.00 correlation: 0.2452
Elapse: 546.23s
Epoch 065/400
Train loss: 72421648.00 correlation: 0.2869
Validation loss: 167359664.00 correlation: 0.2448
Elapse: 546.55s
Loaded checkpoint from epoch 60 (correlation: 0.2462).
Reduce learning rate of core to 4.3200e-04 (num. reduce: 1).
Reduce learning rate of readouts to 3.2400e-04 (num. reduce: 1).
Reduce learning rate of shifters to 3.2400e-04 (num. reduce: 1).
Epoch 066/400
Train loss: 71631360.00 correlation: 0.2929
Validation loss: 166762592.00 correlation: 0.2488
Elapse: 546.04s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 067/400
Train loss: 71301752.00 correlation: 0.2963
Validation loss: 166766816.00 correlation: 0.2491
Elapse: 546.30s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 068/400
Train loss: 71239384.00 correlation: 0.2970
Validation loss: 166698608.00 correlation: 0.2493
Elapse: 546.54s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 069/400
Train loss: 71140904.00 correlation: 0.2977
Validation loss: 166688480.00 correlation: 0.2493
Elapse: 546.57s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 070/400
Train loss: 71157128.00 correlation: 0.2979
Validation loss: 166701680.00 correlation: 0.2493
Elapse: 546.93s
Epoch 071/400
Train loss: 71109936.00 correlation: 0.2983
Validation loss: 166666112.00 correlation: 0.2493
Elapse: 546.33s
Epoch 072/400
Train loss: 71014384.00 correlation: 0.2990
Validation loss: 166637680.00 correlation: 0.2493
Elapse: 546.25s
Epoch 073/400
Train loss: 71012144.00 correlation: 0.2989
Validation loss: 166633760.00 correlation: 0.2495
Elapse: 546.14s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 074/400
Train loss: 70962312.00 correlation: 0.2996
Validation loss: 166724848.00 correlation: 0.2491
Elapse: 546.66s
Epoch 075/400
Train loss: 70909096.00 correlation: 0.3002
Validation loss: 166680192.00 correlation: 0.2492
Elapse: 546.39s
Epoch 076/400
Train loss: 70859696.00 correlation: 0.3007
Validation loss: 166666160.00 correlation: 0.2493
Elapse: 546.76s
Epoch 077/400
Train loss: 70862616.00 correlation: 0.3006
Validation loss: 166644512.00 correlation: 0.2493
Elapse: 546.35s
Epoch 078/400
Train loss: 70866920.00 correlation: 0.3008
Validation loss: 166689088.00 correlation: 0.2489
Elapse: 546.43s
Loaded checkpoint from epoch 73 (correlation: 0.2495).
Reduce learning rate of core to 1.2960e-04 (num. reduce: 1).
Reduce learning rate of readouts to 9.7200e-05 (num. reduce: 1).
Reduce learning rate of shifters to 9.7200e-05 (num. reduce: 1).
Epoch 079/400
Train loss: 70659088.00 correlation: 0.3023
Validation loss: 166495968.00 correlation: 0.2504
Elapse: 546.32s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 080/400
Train loss: 70583640.00 correlation: 0.3029
Validation loss: 166499952.00 correlation: 0.2503
Elapse: 546.55s
Epoch 081/400
Train loss: 70541232.00 correlation: 0.3031
Validation loss: 166486608.00 correlation: 0.2504
Elapse: 546.44s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 082/400
Train loss: 70561208.00 correlation: 0.3030
Validation loss: 166498080.00 correlation: 0.2505
Elapse: 546.48s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 083/400
Train loss: 70504848.00 correlation: 0.3037
Validation loss: 166479744.00 correlation: 0.2504
Elapse: 545.91s
Epoch 084/400
Train loss: 70483280.00 correlation: 0.3037
Validation loss: 166527136.00 correlation: 0.2504
Elapse: 546.39s
Epoch 085/400
Train loss: 70486104.00 correlation: 0.3039
Validation loss: 166507024.00 correlation: 0.2504
Elapse: 546.66s
Epoch 086/400
Train loss: 70448464.00 correlation: 0.3042
Validation loss: 166480160.00 correlation: 0.2505
Elapse: 546.13s
Epoch 087/400
Train loss: 70433528.00 correlation: 0.3043
Validation loss: 166521664.00 correlation: 0.2504
Elapse: 546.51s
Loaded checkpoint from epoch 82 (correlation: 0.2505).
Reduce learning rate of core to 3.8880e-05 (num. reduce: 1).
Reduce learning rate of readouts to 2.9160e-05 (num. reduce: 1).
Reduce learning rate of shifters to 2.9160e-05 (num. reduce: 1).
Epoch 088/400
Train loss: 70419248.00 correlation: 0.3041
Validation loss: 166470848.00 correlation: 0.2507
Elapse: 546.72s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 089/400
Train loss: 70456936.00 correlation: 0.3039
Validation loss: 166478544.00 correlation: 0.2507
Elapse: 546.31s
Epoch 090/400
Train loss: 70377392.00 correlation: 0.3047
Validation loss: 166470880.00 correlation: 0.2508
Elapse: 546.64s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 091/400
Train loss: 70370720.00 correlation: 0.3048
Validation loss: 166471712.00 correlation: 0.2507
Elapse: 546.55s
Epoch 092/400
Train loss: 70332624.00 correlation: 0.3052
Validation loss: 166468176.00 correlation: 0.2506
Elapse: 546.51s
Epoch 093/400
Train loss: 70375248.00 correlation: 0.3046
Validation loss: 166466304.00 correlation: 0.2507
Elapse: 547.09s
Epoch 094/400
Train loss: 70312208.00 correlation: 0.3053
Validation loss: 166426080.00 correlation: 0.2509
Elapse: 546.36s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 095/400
Train loss: 70306024.00 correlation: 0.3054
Validation loss: 166448992.00 correlation: 0.2509
Elapse: 546.42s
Epoch 096/400
Train loss: 70301760.00 correlation: 0.3053
Validation loss: 166460400.00 correlation: 0.2508
Elapse: 545.99s
Epoch 097/400
Train loss: 70339392.00 correlation: 0.3050
Validation loss: 166448912.00 correlation: 0.2507
Elapse: 546.74s
Epoch 098/400
Train loss: 70365064.00 correlation: 0.3048
Validation loss: 166476400.00 correlation: 0.2506
Elapse: 546.17s
Epoch 099/400
Train loss: 70304792.00 correlation: 0.3053
Validation loss: 166451168.00 correlation: 0.2507
Elapse: 545.82s
Loaded checkpoint from epoch 94 (correlation: 0.2509).
Reduce learning rate of core to 1.1664e-05 (num. reduce: 1).
Reduce learning rate of readouts to 8.7480e-06 (num. reduce: 1).
Reduce learning rate of shifters to 8.7480e-06 (num. reduce: 1).
Epoch 100/400
Train loss: 70282520.00 correlation: 0.3055
Validation loss: 166444240.00 correlation: 0.2509
Elapse: 546.60s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 101/400
Train loss: 70321056.00 correlation: 0.3050
Validation loss: 166451424.00 correlation: 0.2509
Elapse: 546.61s
Epoch 102/400
Train loss: 70312536.00 correlation: 0.3052
Validation loss: 166450992.00 correlation: 0.2509
Elapse: 546.64s
Epoch 103/400
Train loss: 70294928.00 correlation: 0.3057
Validation loss: 166437664.00 correlation: 0.2509
Elapse: 546.47s
Epoch 104/400
Train loss: 70265648.00 correlation: 0.3058
Validation loss: 166443488.00 correlation: 0.2509
Elapse: 546.86s
Epoch 105/400
Train loss: 70228080.00 correlation: 0.3058
Validation loss: 166447312.00 correlation: 0.2508
Elapse: 546.77s
Loaded checkpoint from epoch 100 (correlation: 0.2509).
Reduce learning rate of core to 3.4992e-06 (num. reduce: 1).
Reduce learning rate of readouts to 2.6244e-06 (num. reduce: 1).
Reduce learning rate of shifters to 2.6244e-06 (num. reduce: 1).
Epoch 106/400
Train loss: 70303288.00 correlation: 0.3053
Validation loss: 166441600.00 correlation: 0.2509
Elapse: 546.61s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 107/400
Train loss: 70329360.00 correlation: 0.3052
Validation loss: 166438976.00 correlation: 0.2509
Elapse: 546.25s
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt.
Epoch 108/400
Train loss: 70249752.00 correlation: 0.3063
Validation loss: 166440032.00 correlation: 0.2509
Elapse: 546.40s
Epoch 109/400
Train loss: 70245592.00 correlation: 0.3059
Validation loss: 166436832.00 correlation: 0.2509
Elapse: 546.34s
Epoch 110/400
Train loss: 70323520.00 correlation: 0.3050
Validation loss: 166439552.00 correlation: 0.2509
Elapse: 546.33s
Epoch 111/400
Train loss: 70291648.00 correlation: 0.3053
Validation loss: 166438448.00 correlation: 0.2509
Elapse: 546.45s
Epoch 112/400
Train loss: 70337264.00 correlation: 0.3050
Validation loss: 166435040.00 correlation: 0.2509
Elapse: 546.37s
Loaded checkpoint from epoch 107 (correlation: 0.2509).
Reduce learning rate of core to 1.0498e-06 (num. reduce: 1).
Reduce learning rate of readouts to 7.8732e-07 (num. reduce: 1).
Reduce learning rate of shifters to 7.8732e-07 (num. reduce: 1).
Epoch 113/400
Train loss: 70326192.00 correlation: 0.3050
Validation loss: 166438976.00 correlation: 0.2509
Elapse: 546.86s
Epoch 114/400
Train loss: 70297808.00 correlation: 0.3056
Validation loss: 166439680.00 correlation: 0.2509
Elapse: 546.99s
Epoch 115/400
Train loss: 70375312.00 correlation: 0.3045
Validation loss: 166439312.00 correlation: 0.2509
Elapse: 547.26s
Epoch 116/400
Train loss: 70331832.00 correlation: 0.3050
Validation loss: 166439392.00 correlation: 0.2509
Elapse: 546.75s
Epoch 117/400
Train loss: 70241608.00 correlation: 0.3060
Validation loss: 166441104.00 correlation: 0.2509
Elapse: 546.43s
Loaded checkpoint from epoch 107 (correlation: 0.2509).
Reduce learning rate of core to 3.1493e-07 (num. reduce: 2).
Reduce learning rate of readouts to 2.3620e-07 (num. reduce: 2).
Reduce learning rate of shifters to 2.3620e-07 (num. reduce: 2).
Epoch 118/400
Train loss: 70344976.00 correlation: 0.3049
Validation loss: 166439680.00 correlation: 0.2509
Elapse: 546.75s
Epoch 119/400
Train loss: 70278184.00 correlation: 0.3054
Validation loss: 166439840.00 correlation: 0.2509
Elapse: 547.33s
Epoch 120/400
Train loss: 70224856.00 correlation: 0.3062
Validation loss: 166439440.00 correlation: 0.2509
Elapse: 546.94s
Epoch 121/400
Train loss: 70317744.00 correlation: 0.3049
Validation loss: 166439280.00 correlation: 0.2509
Elapse: 547.22s
Epoch 122/400
Train loss: 70322624.00 correlation: 0.3052
Validation loss: 166438784.00 correlation: 0.2509
Elapse: 546.83s
Model has not improved after 2 LR reductions.
Loaded checkpoint from epoch 107 (correlation: 0.2509).
ValidationA: 0.2475 B: 0.2751 C: 0.2730 D: 0.2364 E: 0.2355 F: 0.2343 G: 0.2543 H: 0.2344 I: 0.2602 J: 0.2588 average: 0.2509
Results saved to /home/storage/runs/vivit_ensemble/012.