| [2024-09-04 13:11:59,143][__main__][INFO] - Using GPU for training. Main device : cuda:0 | |
| [2024-09-04 13:11:59,144][__main__][INFO] - Using multiple GPUs : [0, 1, 2, 3] | |
| [2024-09-04 13:12:08,055][__main__][INFO] - PSF shape : torch.Size([1, 380, 507, 3]) | |
| [2024-09-04 13:12:08,081][__main__][INFO] - PSF min : 0.0 | |
| [2024-09-04 13:12:08,091][__main__][INFO] - PSF max : 0.04469018802046776 | |
| [2024-09-04 13:12:08,091][__main__][INFO] - PSF dtype : torch.float32 | |
| [2024-09-04 13:12:08,115][__main__][INFO] - PSF norm : 1.000011682510376 | |
| [2024-09-04 13:12:08,115][__main__][INFO] - Reconstruction a few images with ADMM... | |
| [2024-09-04 13:12:08,715][__main__][INFO] - Cropped shape : (123, 123, 3) | |
| [2024-09-04 13:12:08,931][__main__][INFO] - Cropped shape : (123, 123, 3) | |
| [2024-09-04 13:12:11,081][__main__][INFO] - Train test size : 21250 | |
| [2024-09-04 13:12:11,082][__main__][INFO] - Test test size : 3750 | |
| [2024-09-04 13:12:11,189][__main__][INFO] - Training model with 8716570 parameters | |
| [2024-09-04 13:12:11,190][__main__][INFO] - -- Post-process model with 8010848 parameters | |
| [2024-09-04 13:12:11,190][__main__][INFO] - -- Background subtraction model with 127740 parameters | |
| [2024-09-04 13:12:11,191][__main__][INFO] - Setup time : 0.10877776145935059 s | |
| [2024-09-04 13:12:11,191][__main__][INFO] - PSF shape : torch.Size([1, 380, 507, 3]) | |
| [2024-09-04 13:12:11,191][__main__][INFO] - Results saved in /root/FORKS/LenslessPiCamStefan/outputs/2024-09-04/13-11-53 | |
| [2024-09-04 13:18:43,747][__main__][INFO] - Epoch 0 with learning rate [0.0, 0.0] | |
| [2024-09-04 14:28:10,981][__main__][INFO] - loss : 0.9142370790876994 | |
| [2024-09-04 14:34:45,871][__main__][INFO] - Epoch 1 with learning rate [0.0001, 0.0001] | |
| [2024-09-04 15:43:56,820][__main__][INFO] - loss : 0.6776803907602283 | |
| [2024-09-04 15:50:35,414][__main__][INFO] - Epoch 2 with learning rate [9.957224306869053e-05, 9.957224306869053e-05] | |
| [2024-09-04 16:59:56,057][__main__][INFO] - loss : 0.6371981697905533 | |
| [2024-09-04 17:06:30,410][__main__][INFO] - Epoch 3 with learning rate [9.829629131445342e-05, 9.829629131445342e-05] | |
| [2024-09-04 18:15:30,296][__main__][INFO] - loss : 0.577169242897838 | |
| [2024-09-04 18:22:02,173][__main__][INFO] - Epoch 4 with learning rate [9.619397662556435e-05, 9.619397662556435e-05] | |
| [2024-09-04 19:30:54,714][__main__][INFO] - loss : 0.5359458207702671 | |
| [2024-09-04 19:37:26,551][__main__][INFO] - Epoch 5 with learning rate [9.330127018922194e-05, 9.330127018922194e-05] | |
| [2024-09-04 20:47:59,199][__main__][INFO] - loss : 0.5197215331727079 | |
| [2024-09-04 20:54:27,830][__main__][INFO] - Epoch 6 with learning rate [8.966766701456177e-05, 8.966766701456177e-05] | |
| [2024-09-04 22:03:27,225][__main__][INFO] - loss : 0.5092100189350501 | |
| [2024-09-04 22:10:00,395][__main__][INFO] - Epoch 7 with learning rate [8.535533905932738e-05, 8.535533905932738e-05] | |
| [2024-09-04 23:19:02,952][__main__][INFO] - loss : 0.500233816821182 | |
| [2024-09-04 23:25:34,406][__main__][INFO] - Epoch 8 with learning rate [8.043807145043604e-05, 8.043807145043604e-05] | |
| [2024-09-05 00:34:26,623][__main__][INFO] - loss : 0.4928267903018764 | |
| [2024-09-05 00:41:02,065][__main__][INFO] - Epoch 9 with learning rate [7.500000000000001e-05, 7.500000000000001e-05] | |
| [2024-09-05 01:50:02,148][__main__][INFO] - loss : 0.4848265762486603 | |
| [2024-09-05 01:56:34,875][__main__][INFO] - Epoch 10 with learning rate [6.91341716182545e-05, 6.91341716182545e-05] | |
| [2024-09-05 03:05:19,683][__main__][INFO] - loss : 0.47705518910491235 | |
| [2024-09-05 03:11:51,373][__main__][INFO] - Epoch 11 with learning rate [6.294095225512603e-05, 6.294095225512603e-05] | |
| [2024-09-05 04:20:44,779][__main__][INFO] - loss : 0.4692753138637301 | |
| [2024-09-05 04:27:17,105][__main__][INFO] - Epoch 12 with learning rate [5.6526309611002594e-05, 5.6526309611002594e-05] | |
| [2024-09-05 05:36:26,552][__main__][INFO] - loss : 0.4619376632692944 | |
| [2024-09-05 05:43:00,033][__main__][INFO] - Epoch 13 with learning rate [5e-05, 5e-05] | |
| [2024-09-05 06:51:54,266][__main__][INFO] - loss : 0.45511861081830635 | |
| [2024-09-05 06:58:29,971][__main__][INFO] - Epoch 14 with learning rate [4.347369038899744e-05, 4.347369038899744e-05] | |
| [2024-09-05 08:07:31,556][__main__][INFO] - loss : 0.4489527845223383 | |
| [2024-09-05 08:14:05,068][__main__][INFO] - Epoch 15 with learning rate [3.705904774487396e-05, 3.705904774487396e-05] | |
| [2024-09-05 09:23:02,780][__main__][INFO] - loss : 0.44313734614443023 | |
| [2024-09-05 09:29:33,977][__main__][INFO] - Epoch 16 with learning rate [3.086582838174551e-05, 3.086582838174551e-05] | |
| [2024-09-05 10:38:28,650][__main__][INFO] - loss : 0.4379278795263225 | |
| [2024-09-05 10:45:06,512][__main__][INFO] - Epoch 17 with learning rate [2.500000000000001e-05, 2.500000000000001e-05] | |
| [2024-09-05 11:54:14,096][__main__][INFO] - loss : 0.4334318825423613 | |
| [2024-09-05 12:00:49,002][__main__][INFO] - Epoch 18 with learning rate [1.9561928549563968e-05, 1.9561928549563968e-05] | |
| [2024-09-05 13:09:47,602][__main__][INFO] - loss : 0.4294679476764718 | |
| [2024-09-05 13:16:28,250][__main__][INFO] - Epoch 19 with learning rate [1.4644660940672627e-05, 1.4644660940672627e-05] | |
| [2024-09-05 14:25:33,194][__main__][INFO] - loss : 0.4254841720616998 | |
| [2024-09-05 14:32:08,839][__main__][INFO] - Epoch 20 with learning rate [1.0332332985438248e-05, 1.0332332985438248e-05] | |
| [2024-09-05 15:41:02,212][__main__][INFO] - loss : 0.4224701670184093 | |
| [2024-09-05 15:47:35,726][__main__][INFO] - Epoch 21 with learning rate [6.698729810778065e-06, 6.698729810778065e-06] | |
| [2024-09-05 16:56:37,445][__main__][INFO] - loss : 0.42015853027222205 | |
| [2024-09-05 17:03:10,351][__main__][INFO] - Epoch 22 with learning rate [3.8060233744356633e-06, 3.8060233744356633e-06] | |
| [2024-09-05 18:12:11,833][__main__][INFO] - loss : 0.418189093310706 | |
| [2024-09-05 18:18:44,048][__main__][INFO] - Epoch 23 with learning rate [1.70370868554659e-06, 1.70370868554659e-06] | |
| [2024-09-05 19:28:00,720][__main__][INFO] - loss : 0.41705528846259093 | |
| [2024-09-05 19:34:32,818][__main__][INFO] - Epoch 24 with learning rate [4.277569313094809e-07, 4.277569313094809e-07] | |
| [2024-09-05 20:43:39,013][__main__][INFO] - loss : 0.41623980265742483 | |
| [2024-09-05 20:50:10,074][__main__][INFO] - Train time [hour] : 31.6324996623728 h | |
| [2024-09-05 20:50:10,075][__main__][INFO] - Results saved in /root/FORKS/LenslessPiCamStefan/outputs/2024-09-04/13-11-53 | |