|
Use bfloat16 for core module. |
|
Use parallel attention and MLP in ViViT. |
|
|
|
Epoch 001/400 |
|
Train loss: 112704208.00 correlation: 0.0126 |
|
Validation loss: 199545376.00 correlation: 0.0293 |
|
Elapse: 540.35s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 002/400 |
|
Train loss: 97414480.00 correlation: 0.0393 |
|
Validation loss: 198820640.00 correlation: 0.0410 |
|
Elapse: 549.83s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 003/400 |
|
Train loss: 95851936.00 correlation: 0.0545 |
|
Validation loss: 197542912.00 correlation: 0.0493 |
|
Elapse: 552.73s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 004/400 |
|
Train loss: 94762680.00 correlation: 0.0658 |
|
Validation loss: 195978896.00 correlation: 0.0592 |
|
Elapse: 553.22s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 005/400 |
|
Train loss: 93366904.00 correlation: 0.0807 |
|
Validation loss: 193138784.00 correlation: 0.0786 |
|
Elapse: 551.37s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 006/400 |
|
Train loss: 91530000.00 correlation: 0.0996 |
|
Validation loss: 190414640.00 correlation: 0.0957 |
|
Elapse: 548.87s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 007/400 |
|
Train loss: 90052112.00 correlation: 0.1151 |
|
Validation loss: 187958800.00 correlation: 0.1117 |
|
Elapse: 545.45s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 008/400 |
|
Train loss: 88550296.00 correlation: 0.1303 |
|
Validation loss: 185819104.00 correlation: 0.1231 |
|
Elapse: 542.13s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 009/400 |
|
Train loss: 87211088.00 correlation: 0.1444 |
|
Validation loss: 183781008.00 correlation: 0.1379 |
|
Elapse: 540.39s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 010/400 |
|
Train loss: 85912744.00 correlation: 0.1575 |
|
Validation loss: 182160864.00 correlation: 0.1483 |
|
Elapse: 539.93s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 011/400 |
|
Train loss: 84825776.00 correlation: 0.1681 |
|
Validation loss: 180796960.00 correlation: 0.1574 |
|
Elapse: 539.05s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 012/400 |
|
Train loss: 83932096.00 correlation: 0.1768 |
|
Validation loss: 179665872.00 correlation: 0.1651 |
|
Elapse: 539.83s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 013/400 |
|
Train loss: 83185960.00 correlation: 0.1841 |
|
Validation loss: 178867920.00 correlation: 0.1706 |
|
Elapse: 540.49s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 014/400 |
|
Train loss: 82619600.00 correlation: 0.1896 |
|
Validation loss: 178183616.00 correlation: 0.1749 |
|
Elapse: 540.25s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 015/400 |
|
Train loss: 82049104.00 correlation: 0.1953 |
|
Validation loss: 177195872.00 correlation: 0.1811 |
|
Elapse: 539.98s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 016/400 |
|
Train loss: 81438744.00 correlation: 0.2013 |
|
Validation loss: 176732480.00 correlation: 0.1848 |
|
Elapse: 539.92s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 017/400 |
|
Train loss: 80981184.00 correlation: 0.2058 |
|
Validation loss: 175869328.00 correlation: 0.1895 |
|
Elapse: 540.34s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 018/400 |
|
Train loss: 80463984.00 correlation: 0.2108 |
|
Validation loss: 175137504.00 correlation: 0.1941 |
|
Elapse: 540.52s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 019/400 |
|
Train loss: 80076456.00 correlation: 0.2145 |
|
Validation loss: 174743216.00 correlation: 0.1976 |
|
Elapse: 540.79s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 020/400 |
|
Train loss: 79770240.00 correlation: 0.2180 |
|
Validation loss: 174354656.00 correlation: 0.1999 |
|
Elapse: 541.02s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 021/400 |
|
Train loss: 79389680.00 correlation: 0.2210 |
|
Validation loss: 173737072.00 correlation: 0.2049 |
|
Elapse: 540.85s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 022/400 |
|
Train loss: 79099264.00 correlation: 0.2238 |
|
Validation loss: 173426080.00 correlation: 0.2057 |
|
Elapse: 542.08s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 023/400 |
|
Train loss: 78893584.00 correlation: 0.2259 |
|
Validation loss: 172965088.00 correlation: 0.2094 |
|
Elapse: 542.73s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 024/400 |
|
Train loss: 78610448.00 correlation: 0.2286 |
|
Validation loss: 172757408.00 correlation: 0.2102 |
|
Elapse: 543.17s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 025/400 |
|
Train loss: 78350984.00 correlation: 0.2311 |
|
Validation loss: 172301040.00 correlation: 0.2136 |
|
Elapse: 543.01s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 026/400 |
|
Train loss: 78082256.00 correlation: 0.2338 |
|
Validation loss: 172086432.00 correlation: 0.2145 |
|
Elapse: 543.35s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 027/400 |
|
Train loss: 77827456.00 correlation: 0.2363 |
|
Validation loss: 171786464.00 correlation: 0.2169 |
|
Elapse: 543.66s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 028/400 |
|
Train loss: 77638136.00 correlation: 0.2379 |
|
Validation loss: 171793264.00 correlation: 0.2171 |
|
Elapse: 543.62s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 029/400 |
|
Train loss: 77553520.00 correlation: 0.2390 |
|
Validation loss: 171183824.00 correlation: 0.2206 |
|
Elapse: 543.14s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 030/400 |
|
Train loss: 77293184.00 correlation: 0.2414 |
|
Validation loss: 171063472.00 correlation: 0.2218 |
|
Elapse: 542.93s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 031/400 |
|
Train loss: 77137136.00 correlation: 0.2425 |
|
Validation loss: 170927232.00 correlation: 0.2215 |
|
Elapse: 542.90s |
|
|
|
Epoch 032/400 |
|
Train loss: 77036008.00 correlation: 0.2439 |
|
Validation loss: 170678112.00 correlation: 0.2241 |
|
Elapse: 544.22s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 033/400 |
|
Train loss: 76850512.00 correlation: 0.2458 |
|
Validation loss: 170585168.00 correlation: 0.2249 |
|
Elapse: 543.38s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 034/400 |
|
Train loss: 76745528.00 correlation: 0.2467 |
|
Validation loss: 170361008.00 correlation: 0.2261 |
|
Elapse: 543.46s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 035/400 |
|
Train loss: 76631640.00 correlation: 0.2479 |
|
Validation loss: 170159952.00 correlation: 0.2272 |
|
Elapse: 544.31s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 036/400 |
|
Train loss: 76572216.00 correlation: 0.2483 |
|
Validation loss: 170068624.00 correlation: 0.2280 |
|
Elapse: 544.03s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 037/400 |
|
Train loss: 76306000.00 correlation: 0.2511 |
|
Validation loss: 170078272.00 correlation: 0.2281 |
|
Elapse: 543.66s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 038/400 |
|
Train loss: 76521968.00 correlation: 0.2488 |
|
Validation loss: 170034304.00 correlation: 0.2284 |
|
Elapse: 543.78s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 039/400 |
|
Train loss: 76277552.00 correlation: 0.2512 |
|
Validation loss: 169955232.00 correlation: 0.2283 |
|
Elapse: 544.11s |
|
|
|
Epoch 040/400 |
|
Train loss: 76083856.00 correlation: 0.2532 |
|
Validation loss: 169547568.00 correlation: 0.2316 |
|
Elapse: 544.27s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 041/400 |
|
Train loss: 76034800.00 correlation: 0.2536 |
|
Validation loss: 169778400.00 correlation: 0.2301 |
|
Elapse: 544.31s |
|
|
|
Epoch 042/400 |
|
Train loss: 75948104.00 correlation: 0.2542 |
|
Validation loss: 169703472.00 correlation: 0.2304 |
|
Elapse: 543.89s |
|
|
|
Epoch 043/400 |
|
Train loss: 75900336.00 correlation: 0.2549 |
|
Validation loss: 169460816.00 correlation: 0.2314 |
|
Elapse: 543.95s |
|
|
|
Epoch 044/400 |
|
Train loss: 75856872.00 correlation: 0.2554 |
|
Validation loss: 169446544.00 correlation: 0.2314 |
|
Elapse: 543.85s |
|
|
|
Epoch 045/400 |
|
Train loss: 75771024.00 correlation: 0.2563 |
|
Validation loss: 169373568.00 correlation: 0.2320 |
|
Elapse: 544.58s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 046/400 |
|
Train loss: 75747888.00 correlation: 0.2564 |
|
Validation loss: 169484960.00 correlation: 0.2316 |
|
Elapse: 544.29s |
|
|
|
Epoch 047/400 |
|
Train loss: 75614488.00 correlation: 0.2578 |
|
Validation loss: 169374048.00 correlation: 0.2332 |
|
Elapse: 544.59s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 048/400 |
|
Train loss: 75621768.00 correlation: 0.2577 |
|
Validation loss: 169829152.00 correlation: 0.2292 |
|
Elapse: 544.60s |
|
|
|
Epoch 049/400 |
|
Train loss: 75540576.00 correlation: 0.2584 |
|
Validation loss: 169151584.00 correlation: 0.2322 |
|
Elapse: 545.07s |
|
|
|
Epoch 050/400 |
|
Train loss: 75552112.00 correlation: 0.2587 |
|
Validation loss: 169218528.00 correlation: 0.2327 |
|
Elapse: 544.47s |
|
|
|
Epoch 051/400 |
|
Train loss: 75440728.00 correlation: 0.2597 |
|
Validation loss: 169177248.00 correlation: 0.2329 |
|
Elapse: 544.75s |
|
|
|
Epoch 052/400 |
|
Train loss: 75499728.00 correlation: 0.2591 |
|
Validation loss: 169380720.00 correlation: 0.2327 |
|
Elapse: 544.77s |
|
|
|
Loaded checkpoint from epoch 47 (correlation: 0.2332). |
|
|
|
Reduce learning rate of core to 1.4400e-03 (num. reduce: 1). |
|
Reduce learning rate of readouts to 1.0800e-03 (num. reduce: 1). |
|
Reduce learning rate of shifters to 1.0800e-03 (num. reduce: 1). |
|
|
|
Epoch 053/400 |
|
Train loss: 73633064.00 correlation: 0.2750 |
|
Validation loss: 167526240.00 correlation: 0.2446 |
|
Elapse: 545.49s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 054/400 |
|
Train loss: 72992880.00 correlation: 0.2809 |
|
Validation loss: 167505792.00 correlation: 0.2447 |
|
Elapse: 545.66s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 055/400 |
|
Train loss: 72791024.00 correlation: 0.2827 |
|
Validation loss: 167337248.00 correlation: 0.2451 |
|
Elapse: 545.82s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 056/400 |
|
Train loss: 72699672.00 correlation: 0.2837 |
|
Validation loss: 167396864.00 correlation: 0.2453 |
|
Elapse: 545.65s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 057/400 |
|
Train loss: 72666128.00 correlation: 0.2841 |
|
Validation loss: 167320224.00 correlation: 0.2457 |
|
Elapse: 546.33s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 058/400 |
|
Train loss: 72662680.00 correlation: 0.2840 |
|
Validation loss: 167358624.00 correlation: 0.2457 |
|
Elapse: 545.38s |
|
|
|
Epoch 059/400 |
|
Train loss: 72528456.00 correlation: 0.2854 |
|
Validation loss: 167347344.00 correlation: 0.2456 |
|
Elapse: 546.44s |
|
|
|
Epoch 060/400 |
|
Train loss: 72553424.00 correlation: 0.2854 |
|
Validation loss: 167294624.00 correlation: 0.2462 |
|
Elapse: 546.17s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 061/400 |
|
Train loss: 72496144.00 correlation: 0.2860 |
|
Validation loss: 167428896.00 correlation: 0.2444 |
|
Elapse: 545.96s |
|
|
|
Epoch 062/400 |
|
Train loss: 72479344.00 correlation: 0.2859 |
|
Validation loss: 167235104.00 correlation: 0.2454 |
|
Elapse: 546.11s |
|
|
|
Epoch 063/400 |
|
Train loss: 72437912.00 correlation: 0.2863 |
|
Validation loss: 167384528.00 correlation: 0.2447 |
|
Elapse: 546.05s |
|
|
|
Epoch 064/400 |
|
Train loss: 72443952.00 correlation: 0.2864 |
|
Validation loss: 167276464.00 correlation: 0.2452 |
|
Elapse: 546.23s |
|
|
|
Epoch 065/400 |
|
Train loss: 72421648.00 correlation: 0.2869 |
|
Validation loss: 167359664.00 correlation: 0.2448 |
|
Elapse: 546.55s |
|
|
|
Loaded checkpoint from epoch 60 (correlation: 0.2462). |
|
|
|
Reduce learning rate of core to 4.3200e-04 (num. reduce: 1). |
|
Reduce learning rate of readouts to 3.2400e-04 (num. reduce: 1). |
|
Reduce learning rate of shifters to 3.2400e-04 (num. reduce: 1). |
|
|
|
Epoch 066/400 |
|
Train loss: 71631360.00 correlation: 0.2929 |
|
Validation loss: 166762592.00 correlation: 0.2488 |
|
Elapse: 546.04s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 067/400 |
|
Train loss: 71301752.00 correlation: 0.2963 |
|
Validation loss: 166766816.00 correlation: 0.2491 |
|
Elapse: 546.30s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 068/400 |
|
Train loss: 71239384.00 correlation: 0.2970 |
|
Validation loss: 166698608.00 correlation: 0.2493 |
|
Elapse: 546.54s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 069/400 |
|
Train loss: 71140904.00 correlation: 0.2977 |
|
Validation loss: 166688480.00 correlation: 0.2493 |
|
Elapse: 546.57s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 070/400 |
|
Train loss: 71157128.00 correlation: 0.2979 |
|
Validation loss: 166701680.00 correlation: 0.2493 |
|
Elapse: 546.93s |
|
|
|
Epoch 071/400 |
|
Train loss: 71109936.00 correlation: 0.2983 |
|
Validation loss: 166666112.00 correlation: 0.2493 |
|
Elapse: 546.33s |
|
|
|
Epoch 072/400 |
|
Train loss: 71014384.00 correlation: 0.2990 |
|
Validation loss: 166637680.00 correlation: 0.2493 |
|
Elapse: 546.25s |
|
|
|
Epoch 073/400 |
|
Train loss: 71012144.00 correlation: 0.2989 |
|
Validation loss: 166633760.00 correlation: 0.2495 |
|
Elapse: 546.14s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 074/400 |
|
Train loss: 70962312.00 correlation: 0.2996 |
|
Validation loss: 166724848.00 correlation: 0.2491 |
|
Elapse: 546.66s |
|
|
|
Epoch 075/400 |
|
Train loss: 70909096.00 correlation: 0.3002 |
|
Validation loss: 166680192.00 correlation: 0.2492 |
|
Elapse: 546.39s |
|
|
|
Epoch 076/400 |
|
Train loss: 70859696.00 correlation: 0.3007 |
|
Validation loss: 166666160.00 correlation: 0.2493 |
|
Elapse: 546.76s |
|
|
|
Epoch 077/400 |
|
Train loss: 70862616.00 correlation: 0.3006 |
|
Validation loss: 166644512.00 correlation: 0.2493 |
|
Elapse: 546.35s |
|
|
|
Epoch 078/400 |
|
Train loss: 70866920.00 correlation: 0.3008 |
|
Validation loss: 166689088.00 correlation: 0.2489 |
|
Elapse: 546.43s |
|
|
|
Loaded checkpoint from epoch 73 (correlation: 0.2495). |
|
|
|
Reduce learning rate of core to 1.2960e-04 (num. reduce: 1). |
|
Reduce learning rate of readouts to 9.7200e-05 (num. reduce: 1). |
|
Reduce learning rate of shifters to 9.7200e-05 (num. reduce: 1). |
|
|
|
Epoch 079/400 |
|
Train loss: 70659088.00 correlation: 0.3023 |
|
Validation loss: 166495968.00 correlation: 0.2504 |
|
Elapse: 546.32s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 080/400 |
|
Train loss: 70583640.00 correlation: 0.3029 |
|
Validation loss: 166499952.00 correlation: 0.2503 |
|
Elapse: 546.55s |
|
|
|
Epoch 081/400 |
|
Train loss: 70541232.00 correlation: 0.3031 |
|
Validation loss: 166486608.00 correlation: 0.2504 |
|
Elapse: 546.44s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 082/400 |
|
Train loss: 70561208.00 correlation: 0.3030 |
|
Validation loss: 166498080.00 correlation: 0.2505 |
|
Elapse: 546.48s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 083/400 |
|
Train loss: 70504848.00 correlation: 0.3037 |
|
Validation loss: 166479744.00 correlation: 0.2504 |
|
Elapse: 545.91s |
|
|
|
Epoch 084/400 |
|
Train loss: 70483280.00 correlation: 0.3037 |
|
Validation loss: 166527136.00 correlation: 0.2504 |
|
Elapse: 546.39s |
|
|
|
Epoch 085/400 |
|
Train loss: 70486104.00 correlation: 0.3039 |
|
Validation loss: 166507024.00 correlation: 0.2504 |
|
Elapse: 546.66s |
|
|
|
Epoch 086/400 |
|
Train loss: 70448464.00 correlation: 0.3042 |
|
Validation loss: 166480160.00 correlation: 0.2505 |
|
Elapse: 546.13s |
|
|
|
Epoch 087/400 |
|
Train loss: 70433528.00 correlation: 0.3043 |
|
Validation loss: 166521664.00 correlation: 0.2504 |
|
Elapse: 546.51s |
|
|
|
Loaded checkpoint from epoch 82 (correlation: 0.2505). |
|
|
|
Reduce learning rate of core to 3.8880e-05 (num. reduce: 1). |
|
Reduce learning rate of readouts to 2.9160e-05 (num. reduce: 1). |
|
Reduce learning rate of shifters to 2.9160e-05 (num. reduce: 1). |
|
|
|
Epoch 088/400 |
|
Train loss: 70419248.00 correlation: 0.3041 |
|
Validation loss: 166470848.00 correlation: 0.2507 |
|
Elapse: 546.72s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 089/400 |
|
Train loss: 70456936.00 correlation: 0.3039 |
|
Validation loss: 166478544.00 correlation: 0.2507 |
|
Elapse: 546.31s |
|
|
|
Epoch 090/400 |
|
Train loss: 70377392.00 correlation: 0.3047 |
|
Validation loss: 166470880.00 correlation: 0.2508 |
|
Elapse: 546.64s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 091/400 |
|
Train loss: 70370720.00 correlation: 0.3048 |
|
Validation loss: 166471712.00 correlation: 0.2507 |
|
Elapse: 546.55s |
|
|
|
Epoch 092/400 |
|
Train loss: 70332624.00 correlation: 0.3052 |
|
Validation loss: 166468176.00 correlation: 0.2506 |
|
Elapse: 546.51s |
|
|
|
Epoch 093/400 |
|
Train loss: 70375248.00 correlation: 0.3046 |
|
Validation loss: 166466304.00 correlation: 0.2507 |
|
Elapse: 547.09s |
|
|
|
Epoch 094/400 |
|
Train loss: 70312208.00 correlation: 0.3053 |
|
Validation loss: 166426080.00 correlation: 0.2509 |
|
Elapse: 546.36s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 095/400 |
|
Train loss: 70306024.00 correlation: 0.3054 |
|
Validation loss: 166448992.00 correlation: 0.2509 |
|
Elapse: 546.42s |
|
|
|
Epoch 096/400 |
|
Train loss: 70301760.00 correlation: 0.3053 |
|
Validation loss: 166460400.00 correlation: 0.2508 |
|
Elapse: 545.99s |
|
|
|
Epoch 097/400 |
|
Train loss: 70339392.00 correlation: 0.3050 |
|
Validation loss: 166448912.00 correlation: 0.2507 |
|
Elapse: 546.74s |
|
|
|
Epoch 098/400 |
|
Train loss: 70365064.00 correlation: 0.3048 |
|
Validation loss: 166476400.00 correlation: 0.2506 |
|
Elapse: 546.17s |
|
|
|
Epoch 099/400 |
|
Train loss: 70304792.00 correlation: 0.3053 |
|
Validation loss: 166451168.00 correlation: 0.2507 |
|
Elapse: 545.82s |
|
|
|
Loaded checkpoint from epoch 94 (correlation: 0.2509). |
|
|
|
Reduce learning rate of core to 1.1664e-05 (num. reduce: 1). |
|
Reduce learning rate of readouts to 8.7480e-06 (num. reduce: 1). |
|
Reduce learning rate of shifters to 8.7480e-06 (num. reduce: 1). |
|
|
|
Epoch 100/400 |
|
Train loss: 70282520.00 correlation: 0.3055 |
|
Validation loss: 166444240.00 correlation: 0.2509 |
|
Elapse: 546.60s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 101/400 |
|
Train loss: 70321056.00 correlation: 0.3050 |
|
Validation loss: 166451424.00 correlation: 0.2509 |
|
Elapse: 546.61s |
|
|
|
Epoch 102/400 |
|
Train loss: 70312536.00 correlation: 0.3052 |
|
Validation loss: 166450992.00 correlation: 0.2509 |
|
Elapse: 546.64s |
|
|
|
Epoch 103/400 |
|
Train loss: 70294928.00 correlation: 0.3057 |
|
Validation loss: 166437664.00 correlation: 0.2509 |
|
Elapse: 546.47s |
|
|
|
Epoch 104/400 |
|
Train loss: 70265648.00 correlation: 0.3058 |
|
Validation loss: 166443488.00 correlation: 0.2509 |
|
Elapse: 546.86s |
|
|
|
Epoch 105/400 |
|
Train loss: 70228080.00 correlation: 0.3058 |
|
Validation loss: 166447312.00 correlation: 0.2508 |
|
Elapse: 546.77s |
|
|
|
Loaded checkpoint from epoch 100 (correlation: 0.2509). |
|
|
|
Reduce learning rate of core to 3.4992e-06 (num. reduce: 1). |
|
Reduce learning rate of readouts to 2.6244e-06 (num. reduce: 1). |
|
Reduce learning rate of shifters to 2.6244e-06 (num. reduce: 1). |
|
|
|
Epoch 106/400 |
|
Train loss: 70303288.00 correlation: 0.3053 |
|
Validation loss: 166441600.00 correlation: 0.2509 |
|
Elapse: 546.61s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 107/400 |
|
Train loss: 70329360.00 correlation: 0.3052 |
|
Validation loss: 166438976.00 correlation: 0.2509 |
|
Elapse: 546.25s |
|
|
|
Checkpoint saved to /home/storage/runs/vivit_ensemble/012/ckpt/model_state.pt. |
|
|
|
Epoch 108/400 |
|
Train loss: 70249752.00 correlation: 0.3063 |
|
Validation loss: 166440032.00 correlation: 0.2509 |
|
Elapse: 546.40s |
|
|
|
Epoch 109/400 |
|
Train loss: 70245592.00 correlation: 0.3059 |
|
Validation loss: 166436832.00 correlation: 0.2509 |
|
Elapse: 546.34s |
|
|
|
Epoch 110/400 |
|
Train loss: 70323520.00 correlation: 0.3050 |
|
Validation loss: 166439552.00 correlation: 0.2509 |
|
Elapse: 546.33s |
|
|
|
Epoch 111/400 |
|
Train loss: 70291648.00 correlation: 0.3053 |
|
Validation loss: 166438448.00 correlation: 0.2509 |
|
Elapse: 546.45s |
|
|
|
Epoch 112/400 |
|
Train loss: 70337264.00 correlation: 0.3050 |
|
Validation loss: 166435040.00 correlation: 0.2509 |
|
Elapse: 546.37s |
|
|
|
Loaded checkpoint from epoch 107 (correlation: 0.2509). |
|
|
|
Reduce learning rate of core to 1.0498e-06 (num. reduce: 1). |
|
Reduce learning rate of readouts to 7.8732e-07 (num. reduce: 1). |
|
Reduce learning rate of shifters to 7.8732e-07 (num. reduce: 1). |
|
|
|
Epoch 113/400 |
|
Train loss: 70326192.00 correlation: 0.3050 |
|
Validation loss: 166438976.00 correlation: 0.2509 |
|
Elapse: 546.86s |
|
|
|
Epoch 114/400 |
|
Train loss: 70297808.00 correlation: 0.3056 |
|
Validation loss: 166439680.00 correlation: 0.2509 |
|
Elapse: 546.99s |
|
|
|
Epoch 115/400 |
|
Train loss: 70375312.00 correlation: 0.3045 |
|
Validation loss: 166439312.00 correlation: 0.2509 |
|
Elapse: 547.26s |
|
|
|
Epoch 116/400 |
|
Train loss: 70331832.00 correlation: 0.3050 |
|
Validation loss: 166439392.00 correlation: 0.2509 |
|
Elapse: 546.75s |
|
|
|
Epoch 117/400 |
|
Train loss: 70241608.00 correlation: 0.3060 |
|
Validation loss: 166441104.00 correlation: 0.2509 |
|
Elapse: 546.43s |
|
|
|
Loaded checkpoint from epoch 107 (correlation: 0.2509). |
|
|
|
Reduce learning rate of core to 3.1493e-07 (num. reduce: 2). |
|
Reduce learning rate of readouts to 2.3620e-07 (num. reduce: 2). |
|
Reduce learning rate of shifters to 2.3620e-07 (num. reduce: 2). |
|
|
|
Epoch 118/400 |
|
Train loss: 70344976.00 correlation: 0.3049 |
|
Validation loss: 166439680.00 correlation: 0.2509 |
|
Elapse: 546.75s |
|
|
|
Epoch 119/400 |
|
Train loss: 70278184.00 correlation: 0.3054 |
|
Validation loss: 166439840.00 correlation: 0.2509 |
|
Elapse: 547.33s |
|
|
|
Epoch 120/400 |
|
Train loss: 70224856.00 correlation: 0.3062 |
|
Validation loss: 166439440.00 correlation: 0.2509 |
|
Elapse: 546.94s |
|
|
|
Epoch 121/400 |
|
Train loss: 70317744.00 correlation: 0.3049 |
|
Validation loss: 166439280.00 correlation: 0.2509 |
|
Elapse: 547.22s |
|
|
|
Epoch 122/400 |
|
Train loss: 70322624.00 correlation: 0.3052 |
|
Validation loss: 166438784.00 correlation: 0.2509 |
|
Elapse: 546.83s |
|
|
|
Model has not improved after 2 LR reductions. |
|
|
|
Loaded checkpoint from epoch 107 (correlation: 0.2509). |
|
|
|
|
|
ValidationA: 0.2475 B: 0.2751 C: 0.2730 D: 0.2364 E: 0.2355 F: 0.2343 G: 0.2543 H: 0.2344 I: 0.2602 J: 0.2588 average: 0.2509 |
|
|
|
Results saved to /home/storage/runs/vivit_ensemble/012. |
|
|