| Use bfloat16 for core module. | |
| Use parallel attention and MLP in ViViT. | |
| Epoch 001/400 | |
| Train loss: 112704208.00 correlation: 0.0126 | |
| Validation loss: 199545376.00 correlation: 0.0293 | |
| Elapse: 540.35s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 002/400 | |
| Train loss: 97414480.00 correlation: 0.0393 | |
| Validation loss: 198820640.00 correlation: 0.0410 | |
| Elapse: 549.83s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 003/400 | |
| Train loss: 95851936.00 correlation: 0.0545 | |
| Validation loss: 197542912.00 correlation: 0.0493 | |
| Elapse: 552.73s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 004/400 | |
| Train loss: 94762680.00 correlation: 0.0658 | |
| Validation loss: 195978896.00 correlation: 0.0592 | |
| Elapse: 553.22s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 005/400 | |
| Train loss: 93366904.00 correlation: 0.0807 | |
| Validation loss: 193138784.00 correlation: 0.0786 | |
| Elapse: 551.37s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 006/400 | |
| Train loss: 91530000.00 correlation: 0.0996 | |
| Validation loss: 190414640.00 correlation: 0.0957 | |
| Elapse: 548.87s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 007/400 | |
| Train loss: 90052112.00 correlation: 0.1151 | |
| Validation loss: 187958800.00 correlation: 0.1117 | |
| Elapse: 545.45s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 008/400 | |
| Train loss: 88550296.00 correlation: 0.1303 | |
| Validation loss: 185819104.00 correlation: 0.1231 | |
| Elapse: 542.13s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 009/400 | |
| Train loss: 87211088.00 correlation: 0.1444 | |
| Validation loss: 183781008.00 correlation: 0.1379 | |
| Elapse: 540.39s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 010/400 | |
| Train loss: 85912744.00 correlation: 0.1575 | |
| Validation loss: 182160864.00 correlation: 0.1483 | |
| Elapse: 539.93s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 011/400 | |
| Train loss: 84825776.00 correlation: 0.1681 | |
| Validation loss: 180796960.00 correlation: 0.1574 | |
| Elapse: 539.05s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 012/400 | |
| Train loss: 83932096.00 correlation: 0.1768 | |
| Validation loss: 179665872.00 correlation: 0.1651 | |
| Elapse: 539.83s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 013/400 | |
| Train loss: 83185960.00 correlation: 0.1841 | |
| Validation loss: 178867920.00 correlation: 0.1706 | |
| Elapse: 540.49s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 014/400 | |
| Train loss: 82619600.00 correlation: 0.1896 | |
| Validation loss: 178183616.00 correlation: 0.1749 | |
| Elapse: 540.25s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 015/400 | |
| Train loss: 82049104.00 correlation: 0.1953 | |
| Validation loss: 177195872.00 correlation: 0.1811 | |
| Elapse: 539.98s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 016/400 | |
| Train loss: 81438744.00 correlation: 0.2013 | |
| Validation loss: 176732480.00 correlation: 0.1848 | |
| Elapse: 539.92s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 017/400 | |
| Train loss: 80981184.00 correlation: 0.2058 | |
| Validation loss: 175869328.00 correlation: 0.1895 | |
| Elapse: 540.34s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 018/400 | |
| Train loss: 80463984.00 correlation: 0.2108 | |
| Validation loss: 175137504.00 correlation: 0.1941 | |
| Elapse: 540.52s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 019/400 | |
| Train loss: 80076456.00 correlation: 0.2145 | |
| Validation loss: 174743216.00 correlation: 0.1976 | |
| Elapse: 540.79s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 020/400 | |
| Train loss: 79770240.00 correlation: 0.2180 | |
| Validation loss: 174354656.00 correlation: 0.1999 | |
| Elapse: 541.02s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 021/400 | |
| Train loss: 79389680.00 correlation: 0.2210 | |
| Validation loss: 173737072.00 correlation: 0.2049 | |
| Elapse: 540.85s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 022/400 | |
| Train loss: 79099264.00 correlation: 0.2238 | |
| Validation loss: 173426080.00 correlation: 0.2057 | |
| Elapse: 542.08s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 023/400 | |
| Train loss: 78893584.00 correlation: 0.2259 | |
| Validation loss: 172965088.00 correlation: 0.2094 | |
| Elapse: 542.73s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 024/400 | |
| Train loss: 78610448.00 correlation: 0.2286 | |
| Validation loss: 172757408.00 correlation: 0.2102 | |
| Elapse: 543.17s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 025/400 | |
| Train loss: 78350984.00 correlation: 0.2311 | |
| Validation loss: 172301040.00 correlation: 0.2136 | |
| Elapse: 543.01s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 026/400 | |
| Train loss: 78082256.00 correlation: 0.2338 | |
| Validation loss: 172086432.00 correlation: 0.2145 | |
| Elapse: 543.35s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 027/400 | |
| Train loss: 77827456.00 correlation: 0.2363 | |
| Validation loss: 171786464.00 correlation: 0.2169 | |
| Elapse: 543.66s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 028/400 | |
| Train loss: 77638136.00 correlation: 0.2379 | |
| Validation loss: 171793264.00 correlation: 0.2171 | |
| Elapse: 543.62s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 029/400 | |
| Train loss: 77553520.00 correlation: 0.2390 | |
| Validation loss: 171183824.00 correlation: 0.2206 | |
| Elapse: 543.14s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 030/400 | |
| Train loss: 77293184.00 correlation: 0.2414 | |
| Validation loss: 171063472.00 correlation: 0.2218 | |
| Elapse: 542.93s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 031/400 | |
| Train loss: 77137136.00 correlation: 0.2425 | |
| Validation loss: 170927232.00 correlation: 0.2215 | |
| Elapse: 542.90s | |
| Epoch 032/400 | |
| Train loss: 77036008.00 correlation: 0.2439 | |
| Validation loss: 170678112.00 correlation: 0.2241 | |
| Elapse: 544.22s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 033/400 | |
| Train loss: 76850512.00 correlation: 0.2458 | |
| Validation loss: 170585168.00 correlation: 0.2249 | |
| Elapse: 543.38s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 034/400 | |
| Train loss: 76745528.00 correlation: 0.2467 | |
| Validation loss: 170361008.00 correlation: 0.2261 | |
| Elapse: 543.46s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 035/400 | |
| Train loss: 76631640.00 correlation: 0.2479 | |
| Validation loss: 170159952.00 correlation: 0.2272 | |
| Elapse: 544.31s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 036/400 | |
| Train loss: 76572216.00 correlation: 0.2483 | |
| Validation loss: 170068624.00 correlation: 0.2280 | |
| Elapse: 544.03s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 037/400 | |
| Train loss: 76306000.00 correlation: 0.2511 | |
| Validation loss: 170078272.00 correlation: 0.2281 | |
| Elapse: 543.66s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 038/400 | |
| Train loss: 76521968.00 correlation: 0.2488 | |
| Validation loss: 170034304.00 correlation: 0.2284 | |
| Elapse: 543.78s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 039/400 | |
| Train loss: 76277552.00 correlation: 0.2512 | |
| Validation loss: 169955232.00 correlation: 0.2283 | |
| Elapse: 544.11s | |
| Epoch 040/400 | |
| Train loss: 76083856.00 correlation: 0.2532 | |
| Validation loss: 169547568.00 correlation: 0.2316 | |
| Elapse: 544.27s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 041/400 | |
| Train loss: 76034800.00 correlation: 0.2536 | |
| Validation loss: 169778400.00 correlation: 0.2301 | |
| Elapse: 544.31s | |
| Epoch 042/400 | |
| Train loss: 75948104.00 correlation: 0.2542 | |
| Validation loss: 169703472.00 correlation: 0.2304 | |
| Elapse: 543.89s | |
| Epoch 043/400 | |
| Train loss: 75900336.00 correlation: 0.2549 | |
| Validation loss: 169460816.00 correlation: 0.2314 | |
| Elapse: 543.95s | |
| Epoch 044/400 | |
| Train loss: 75856872.00 correlation: 0.2554 | |
| Validation loss: 169446544.00 correlation: 0.2314 | |
| Elapse: 543.85s | |
| Epoch 045/400 | |
| Train loss: 75771024.00 correlation: 0.2563 | |
| Validation loss: 169373568.00 correlation: 0.2320 | |
| Elapse: 544.58s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 046/400 | |
| Train loss: 75747888.00 correlation: 0.2564 | |
| Validation loss: 169484960.00 correlation: 0.2316 | |
| Elapse: 544.29s | |
| Epoch 047/400 | |
| Train loss: 75614488.00 correlation: 0.2578 | |
| Validation loss: 169374048.00 correlation: 0.2332 | |
| Elapse: 544.59s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 048/400 | |
| Train loss: 75621768.00 correlation: 0.2577 | |
| Validation loss: 169829152.00 correlation: 0.2292 | |
| Elapse: 544.60s | |
| Epoch 049/400 | |
| Train loss: 75540576.00 correlation: 0.2584 | |
| Validation loss: 169151584.00 correlation: 0.2322 | |
| Elapse: 545.07s | |
| Epoch 050/400 | |
| Train loss: 75552112.00 correlation: 0.2587 | |
| Validation loss: 169218528.00 correlation: 0.2327 | |
| Elapse: 544.47s | |
| Epoch 051/400 | |
| Train loss: 75440728.00 correlation: 0.2597 | |
| Validation loss: 169177248.00 correlation: 0.2329 | |
| Elapse: 544.75s | |
| Epoch 052/400 | |
| Train loss: 75499728.00 correlation: 0.2591 | |
| Validation loss: 169380720.00 correlation: 0.2327 | |
| Elapse: 544.77s | |
| Loaded checkpoint from epoch 47 (correlation: 0.2332). | |
| Reduce learning rate of core to 1.4400e-03 (num. reduce: 1). | |
| Reduce learning rate of readouts to 1.0800e-03 (num. reduce: 1). | |
| Reduce learning rate of shifters to 1.0800e-03 (num. reduce: 1). | |
| Epoch 053/400 | |
| Train loss: 73633064.00 correlation: 0.2750 | |
| Validation loss: 167526240.00 correlation: 0.2446 | |
| Elapse: 545.49s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 054/400 | |
| Train loss: 72992880.00 correlation: 0.2809 | |
| Validation loss: 167505792.00 correlation: 0.2447 | |
| Elapse: 545.66s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 055/400 | |
| Train loss: 72791024.00 correlation: 0.2827 | |
| Validation loss: 167337248.00 correlation: 0.2451 | |
| Elapse: 545.82s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 056/400 | |
| Train loss: 72699672.00 correlation: 0.2837 | |
| Validation loss: 167396864.00 correlation: 0.2453 | |
| Elapse: 545.65s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 057/400 | |
| Train loss: 72666128.00 correlation: 0.2841 | |
| Validation loss: 167320224.00 correlation: 0.2457 | |
| Elapse: 546.33s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 058/400 | |
| Train loss: 72662680.00 correlation: 0.2840 | |
| Validation loss: 167358624.00 correlation: 0.2457 | |
| Elapse: 545.38s | |
| Epoch 059/400 | |
| Train loss: 72528456.00 correlation: 0.2854 | |
| Validation loss: 167347344.00 correlation: 0.2456 | |
| Elapse: 546.44s | |
| Epoch 060/400 | |
| Train loss: 72553424.00 correlation: 0.2854 | |
| Validation loss: 167294624.00 correlation: 0.2462 | |
| Elapse: 546.17s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 061/400 | |
| Train loss: 72496144.00 correlation: 0.2860 | |
| Validation loss: 167428896.00 correlation: 0.2444 | |
| Elapse: 545.96s | |
| Epoch 062/400 | |
| Train loss: 72479344.00 correlation: 0.2859 | |
| Validation loss: 167235104.00 correlation: 0.2454 | |
| Elapse: 546.11s | |
| Epoch 063/400 | |
| Train loss: 72437912.00 correlation: 0.2863 | |
| Validation loss: 167384528.00 correlation: 0.2447 | |
| Elapse: 546.05s | |
| Epoch 064/400 | |
| Train loss: 72443952.00 correlation: 0.2864 | |
| Validation loss: 167276464.00 correlation: 0.2452 | |
| Elapse: 546.23s | |
| Epoch 065/400 | |
| Train loss: 72421648.00 correlation: 0.2869 | |
| Validation loss: 167359664.00 correlation: 0.2448 | |
| Elapse: 546.55s | |
| Loaded checkpoint from epoch 60 (correlation: 0.2462). | |
| Reduce learning rate of core to 4.3200e-04 (num. reduce: 1). | |
| Reduce learning rate of readouts to 3.2400e-04 (num. reduce: 1). | |
| Reduce learning rate of shifters to 3.2400e-04 (num. reduce: 1). | |
| Epoch 066/400 | |
| Train loss: 71631360.00 correlation: 0.2929 | |
| Validation loss: 166762592.00 correlation: 0.2488 | |
| Elapse: 546.04s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 067/400 | |
| Train loss: 71301752.00 correlation: 0.2963 | |
| Validation loss: 166766816.00 correlation: 0.2491 | |
| Elapse: 546.30s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 068/400 | |
| Train loss: 71239384.00 correlation: 0.2970 | |
| Validation loss: 166698608.00 correlation: 0.2493 | |
| Elapse: 546.54s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 069/400 | |
| Train loss: 71140904.00 correlation: 0.2977 | |
| Validation loss: 166688480.00 correlation: 0.2493 | |
| Elapse: 546.57s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 070/400 | |
| Train loss: 71157128.00 correlation: 0.2979 | |
| Validation loss: 166701680.00 correlation: 0.2493 | |
| Elapse: 546.93s | |
| Epoch 071/400 | |
| Train loss: 71109936.00 correlation: 0.2983 | |
| Validation loss: 166666112.00 correlation: 0.2493 | |
| Elapse: 546.33s | |
| Epoch 072/400 | |
| Train loss: 71014384.00 correlation: 0.2990 | |
| Validation loss: 166637680.00 correlation: 0.2493 | |
| Elapse: 546.25s | |
| Epoch 073/400 | |
| Train loss: 71012144.00 correlation: 0.2989 | |
| Validation loss: 166633760.00 correlation: 0.2495 | |
| Elapse: 546.14s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 074/400 | |
| Train loss: 70962312.00 correlation: 0.2996 | |
| Validation loss: 166724848.00 correlation: 0.2491 | |
| Elapse: 546.66s | |
| Epoch 075/400 | |
| Train loss: 70909096.00 correlation: 0.3002 | |
| Validation loss: 166680192.00 correlation: 0.2492 | |
| Elapse: 546.39s | |
| Epoch 076/400 | |
| Train loss: 70859696.00 correlation: 0.3007 | |
| Validation loss: 166666160.00 correlation: 0.2493 | |
| Elapse: 546.76s | |
| Epoch 077/400 | |
| Train loss: 70862616.00 correlation: 0.3006 | |
| Validation loss: 166644512.00 correlation: 0.2493 | |
| Elapse: 546.35s | |
| Epoch 078/400 | |
| Train loss: 70866920.00 correlation: 0.3008 | |
| Validation loss: 166689088.00 correlation: 0.2489 | |
| Elapse: 546.43s | |
| Loaded checkpoint from epoch 73 (correlation: 0.2495). | |
| Reduce learning rate of core to 1.2960e-04 (num. reduce: 1). | |
| Reduce learning rate of readouts to 9.7200e-05 (num. reduce: 1). | |
| Reduce learning rate of shifters to 9.7200e-05 (num. reduce: 1). | |
| Epoch 079/400 | |
| Train loss: 70659088.00 correlation: 0.3023 | |
| Validation loss: 166495968.00 correlation: 0.2504 | |
| Elapse: 546.32s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 080/400 | |
| Train loss: 70583640.00 correlation: 0.3029 | |
| Validation loss: 166499952.00 correlation: 0.2503 | |
| Elapse: 546.55s | |
| Epoch 081/400 | |
| Train loss: 70541232.00 correlation: 0.3031 | |
| Validation loss: 166486608.00 correlation: 0.2504 | |
| Elapse: 546.44s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 082/400 | |
| Train loss: 70561208.00 correlation: 0.3030 | |
| Validation loss: 166498080.00 correlation: 0.2505 | |
| Elapse: 546.48s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 083/400 | |
| Train loss: 70504848.00 correlation: 0.3037 | |
| Validation loss: 166479744.00 correlation: 0.2504 | |
| Elapse: 545.91s | |
| Epoch 084/400 | |
| Train loss: 70483280.00 correlation: 0.3037 | |
| Validation loss: 166527136.00 correlation: 0.2504 | |
| Elapse: 546.39s | |
| Epoch 085/400 | |
| Train loss: 70486104.00 correlation: 0.3039 | |
| Validation loss: 166507024.00 correlation: 0.2504 | |
| Elapse: 546.66s | |
| Epoch 086/400 | |
| Train loss: 70448464.00 correlation: 0.3042 | |
| Validation loss: 166480160.00 correlation: 0.2505 | |
| Elapse: 546.13s | |
| Epoch 087/400 | |
| Train loss: 70433528.00 correlation: 0.3043 | |
| Validation loss: 166521664.00 correlation: 0.2504 | |
| Elapse: 546.51s | |
| Loaded checkpoint from epoch 82 (correlation: 0.2505). | |
| Reduce learning rate of core to 3.8880e-05 (num. reduce: 1). | |
| Reduce learning rate of readouts to 2.9160e-05 (num. reduce: 1). | |
| Reduce learning rate of shifters to 2.9160e-05 (num. reduce: 1). | |
| Epoch 088/400 | |
| Train loss: 70419248.00 correlation: 0.3041 | |
| Validation loss: 166470848.00 correlation: 0.2507 | |
| Elapse: 546.72s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 089/400 | |
| Train loss: 70456936.00 correlation: 0.3039 | |
| Validation loss: 166478544.00 correlation: 0.2507 | |
| Elapse: 546.31s | |
| Epoch 090/400 | |
| Train loss: 70377392.00 correlation: 0.3047 | |
| Validation loss: 166470880.00 correlation: 0.2508 | |
| Elapse: 546.64s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 091/400 | |
| Train loss: 70370720.00 correlation: 0.3048 | |
| Validation loss: 166471712.00 correlation: 0.2507 | |
| Elapse: 546.55s | |
| Epoch 092/400 | |
| Train loss: 70332624.00 correlation: 0.3052 | |
| Validation loss: 166468176.00 correlation: 0.2506 | |
| Elapse: 546.51s | |
| Epoch 093/400 | |
| Train loss: 70375248.00 correlation: 0.3046 | |
| Validation loss: 166466304.00 correlation: 0.2507 | |
| Elapse: 547.09s | |
| Epoch 094/400 | |
| Train loss: 70312208.00 correlation: 0.3053 | |
| Validation loss: 166426080.00 correlation: 0.2509 | |
| Elapse: 546.36s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 095/400 | |
| Train loss: 70306024.00 correlation: 0.3054 | |
| Validation loss: 166448992.00 correlation: 0.2509 | |
| Elapse: 546.42s | |
| Epoch 096/400 | |
| Train loss: 70301760.00 correlation: 0.3053 | |
| Validation loss: 166460400.00 correlation: 0.2508 | |
| Elapse: 545.99s | |
| Epoch 097/400 | |
| Train loss: 70339392.00 correlation: 0.3050 | |
| Validation loss: 166448912.00 correlation: 0.2507 | |
| Elapse: 546.74s | |
| Epoch 098/400 | |
| Train loss: 70365064.00 correlation: 0.3048 | |
| Validation loss: 166476400.00 correlation: 0.2506 | |
| Elapse: 546.17s | |
| Epoch 099/400 | |
| Train loss: 70304792.00 correlation: 0.3053 | |
| Validation loss: 166451168.00 correlation: 0.2507 | |
| Elapse: 545.82s | |
| Loaded checkpoint from epoch 94 (correlation: 0.2509). | |
| Reduce learning rate of core to 1.1664e-05 (num. reduce: 1). | |
| Reduce learning rate of readouts to 8.7480e-06 (num. reduce: 1). | |
| Reduce learning rate of shifters to 8.7480e-06 (num. reduce: 1). | |
| Epoch 100/400 | |
| Train loss: 70282520.00 correlation: 0.3055 | |
| Validation loss: 166444240.00 correlation: 0.2509 | |
| Elapse: 546.60s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 101/400 | |
| Train loss: 70321056.00 correlation: 0.3050 | |
| Validation loss: 166451424.00 correlation: 0.2509 | |
| Elapse: 546.61s | |
| Epoch 102/400 | |
| Train loss: 70312536.00 correlation: 0.3052 | |
| Validation loss: 166450992.00 correlation: 0.2509 | |
| Elapse: 546.64s | |
| Epoch 103/400 | |
| Train loss: 70294928.00 correlation: 0.3057 | |
| Validation loss: 166437664.00 correlation: 0.2509 | |
| Elapse: 546.47s | |
| Epoch 104/400 | |
| Train loss: 70265648.00 correlation: 0.3058 | |
| Validation loss: 166443488.00 correlation: 0.2509 | |
| Elapse: 546.86s | |
| Epoch 105/400 | |
| Train loss: 70228080.00 correlation: 0.3058 | |
| Validation loss: 166447312.00 correlation: 0.2508 | |
| Elapse: 546.77s | |
| Loaded checkpoint from epoch 100 (correlation: 0.2509). | |
| Reduce learning rate of core to 3.4992e-06 (num. reduce: 1). | |
| Reduce learning rate of readouts to 2.6244e-06 (num. reduce: 1). | |
| Reduce learning rate of shifters to 2.6244e-06 (num. reduce: 1). | |
| Epoch 106/400 | |
| Train loss: 70303288.00 correlation: 0.3053 | |
| Validation loss: 166441600.00 correlation: 0.2509 | |
| Elapse: 546.61s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 107/400 | |
| Train loss: 70329360.00 correlation: 0.3052 | |
| Validation loss: 166438976.00 correlation: 0.2509 | |
| Elapse: 546.25s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. | |
| Epoch 108/400 | |
| Train loss: 70249752.00 correlation: 0.3063 | |
| Validation loss: 166440032.00 correlation: 0.2509 | |
| Elapse: 546.40s | |
| Epoch 109/400 | |
| Train loss: 70245592.00 correlation: 0.3059 | |
| Validation loss: 166436832.00 correlation: 0.2509 | |
| Elapse: 546.34s | |
| Epoch 110/400 | |
| Train loss: 70323520.00 correlation: 0.3050 | |
| Validation loss: 166439552.00 correlation: 0.2509 | |
| Elapse: 546.33s | |
| Epoch 111/400 | |
| Train loss: 70291648.00 correlation: 0.3053 | |
| Validation loss: 166438448.00 correlation: 0.2509 | |
| Elapse: 546.45s | |
| Epoch 112/400 | |
| Train loss: 70337264.00 correlation: 0.3050 | |
| Validation loss: 166435040.00 correlation: 0.2509 | |
| Elapse: 546.37s | |
| Loaded checkpoint from epoch 107 (correlation: 0.2509). | |
| Reduce learning rate of core to 1.0498e-06 (num. reduce: 1). | |
| Reduce learning rate of readouts to 7.8732e-07 (num. reduce: 1). | |
| Reduce learning rate of shifters to 7.8732e-07 (num. reduce: 1). | |
| Epoch 113/400 | |
| Train loss: 70326192.00 correlation: 0.3050 | |
| Validation loss: 166438976.00 correlation: 0.2509 | |
| Elapse: 546.86s | |
| Epoch 114/400 | |
| Train loss: 70297808.00 correlation: 0.3056 | |
| Validation loss: 166439680.00 correlation: 0.2509 | |
| Elapse: 546.99s | |
| Epoch 115/400 | |
| Train loss: 70375312.00 correlation: 0.3045 | |
| Validation loss: 166439312.00 correlation: 0.2509 | |
| Elapse: 547.26s | |
| Epoch 116/400 | |
| Train loss: 70331832.00 correlation: 0.3050 | |
| Validation loss: 166439392.00 correlation: 0.2509 | |
| Elapse: 546.75s | |
| Epoch 117/400 | |
| Train loss: 70241608.00 correlation: 0.3060 | |
| Validation loss: 166441104.00 correlation: 0.2509 | |
| Elapse: 546.43s | |
| Loaded checkpoint from epoch 107 (correlation: 0.2509). | |
| Reduce learning rate of core to 3.1493e-07 (num. reduce: 2). | |
| Reduce learning rate of readouts to 2.3620e-07 (num. reduce: 2). | |
| Reduce learning rate of shifters to 2.3620e-07 (num. reduce: 2). | |
| Epoch 118/400 | |
| Train loss: 70344976.00 correlation: 0.3049 | |
| Validation loss: 166439680.00 correlation: 0.2509 | |
| Elapse: 546.75s | |
| Epoch 119/400 | |
| Train loss: 70278184.00 correlation: 0.3054 | |
| Validation loss: 166439840.00 correlation: 0.2509 | |
| Elapse: 547.33s | |
| Epoch 120/400 | |
| Train loss: 70224856.00 correlation: 0.3062 | |
| Validation loss: 166439440.00 correlation: 0.2509 | |
| Elapse: 546.94s | |
| Epoch 121/400 | |
| Train loss: 70317744.00 correlation: 0.3049 | |
| Validation loss: 166439280.00 correlation: 0.2509 | |
| Elapse: 547.22s | |
| Epoch 122/400 | |
| Train loss: 70322624.00 correlation: 0.3052 | |
| Validation loss: 166438784.00 correlation: 0.2509 | |
| Elapse: 546.83s | |
| Model has not improved after 2 LR reductions. | |
| Loaded checkpoint from epoch 107 (correlation: 0.2509). | |
| ValidationA: 0.2475 B: 0.2751 C: 0.2730 D: 0.2364 E: 0.2355 F: 0.2343 G: 0.2543 H: 0.2344 I: 0.2602 J: 0.2588 average: 0.2509 | |
| Results saved to /home/storage/runs/vivit_ensemble/012. | |