| [2024-09-08 08:20:32,507][__main__][INFO] - Using GPU for training. Main device : cuda:2 | |
| [2024-09-08 08:20:32,508][__main__][INFO] - Using multiple GPUs : [2, 3, 1, 0] | |
| [2024-09-08 08:20:39,617][__main__][INFO] - PSF shape : torch.Size([1, 380, 507, 3]) | |
| [2024-09-08 08:20:39,640][__main__][INFO] - PSF min : 0.0 | |
| [2024-09-08 08:20:39,650][__main__][INFO] - PSF max : 0.04469018802046776 | |
| [2024-09-08 08:20:39,650][__main__][INFO] - PSF dtype : torch.float32 | |
| [2024-09-08 08:20:39,675][__main__][INFO] - PSF norm : 1.000011682510376 | |
| [2024-09-08 08:20:39,675][__main__][INFO] - Reconstruction a few images with ADMM... | |
| [2024-09-08 08:20:40,269][__main__][INFO] - Cropped shape : (123, 123, 3) | |
| [2024-09-08 08:20:40,491][__main__][INFO] - Cropped shape : (123, 123, 3) | |
| [2024-09-08 08:20:42,611][__main__][INFO] - Train test size : 21250 | |
| [2024-09-08 08:20:42,611][__main__][INFO] - Test test size : 3750 | |
| [2024-09-08 08:20:42,722][__main__][INFO] - Training model with 8680318 parameters | |
| [2024-09-08 08:20:42,722][__main__][INFO] - -- Pre-process model with 4051168 parameters | |
| [2024-09-08 08:20:42,723][__main__][INFO] - -- Post-process model with 4051168 parameters | |
| [2024-09-08 08:20:42,723][__main__][INFO] - Setup time : 0.11139488220214844 s | |
| [2024-09-08 08:20:42,723][__main__][INFO] - PSF shape : torch.Size([1, 380, 507, 3]) | |
| [2024-09-08 08:20:42,723][__main__][INFO] - Results saved in /root/FORKS/LenslessPiCamStefan/outputs/2024-09-08/08-20-25 | |
| [2024-09-08 08:27:48,959][__main__][INFO] - Epoch 0 with learning rate [0.0, 0.0] | |
| [2024-09-08 09:53:11,989][__main__][INFO] - loss : 0.9253862211310697 | |
| [2024-09-08 10:00:12,323][__main__][INFO] - Epoch 1 with learning rate [0.0001, 0.0001] | |
| [2024-09-08 11:25:26,971][__main__][INFO] - loss : 0.6835226396618627 | |
| [2024-09-08 11:32:29,290][__main__][INFO] - Epoch 2 with learning rate [9.957224306869053e-05, 9.957224306869053e-05] | |
| [2024-09-08 12:57:53,168][__main__][INFO] - loss : 0.5906448548748384 | |
| [2024-09-08 13:04:55,052][__main__][INFO] - Epoch 3 with learning rate [9.829629131445342e-05, 9.829629131445342e-05] | |
| [2024-09-08 14:30:11,814][__main__][INFO] - loss : 0.5477840927009314 | |
| [2024-09-08 14:37:17,384][__main__][INFO] - Epoch 4 with learning rate [9.619397662556435e-05, 9.619397662556435e-05] | |
| [2024-09-08 16:02:34,856][__main__][INFO] - loss : 0.5260317196633758 | |
| [2024-09-08 16:09:42,195][__main__][INFO] - Epoch 5 with learning rate [9.330127018922194e-05, 9.330127018922194e-05] | |
| [2024-09-08 17:34:50,636][__main__][INFO] - loss : 0.5041433336504574 | |
| [2024-09-08 17:41:53,581][__main__][INFO] - Epoch 6 with learning rate [8.966766701456177e-05, 8.966766701456177e-05] | |
| [2024-09-08 19:06:49,208][__main__][INFO] - loss : 0.48811491332147394 | |
| [2024-09-08 19:13:56,813][__main__][INFO] - Epoch 7 with learning rate [8.535533905932738e-05, 8.535533905932738e-05] | |
| [2024-09-08 20:38:49,366][__main__][INFO] - loss : 0.4768871894758697 | |
| [2024-09-08 20:45:50,506][__main__][INFO] - Epoch 8 with learning rate [8.043807145043604e-05, 8.043807145043604e-05] | |
| [2024-09-08 22:10:50,243][__main__][INFO] - loss : 0.4666666754295471 | |
| [2024-09-08 22:17:53,607][__main__][INFO] - Epoch 9 with learning rate [7.500000000000001e-05, 7.500000000000001e-05] | |
| [2024-09-08 23:42:32,061][__main__][INFO] - loss : 0.45133162885988526 | |
| [2024-09-08 23:49:34,632][__main__][INFO] - Epoch 10 with learning rate [6.91341716182545e-05, 6.91341716182545e-05] | |
| [2024-09-09 01:14:50,369][__main__][INFO] - loss : 0.4366503630107613 | |
| [2024-09-09 01:21:54,027][__main__][INFO] - Epoch 11 with learning rate [6.294095225512603e-05, 6.294095225512603e-05] | |
| [2024-09-09 02:47:08,753][__main__][INFO] - loss : 0.42409106613933245 | |
| [2024-09-09 02:54:14,500][__main__][INFO] - Epoch 12 with learning rate [5.6526309611002594e-05, 5.6526309611002594e-05] | |
| [2024-09-09 04:19:17,402][__main__][INFO] - loss : 0.4141611980445035 | |
| [2024-09-09 04:26:19,644][__main__][INFO] - Epoch 13 with learning rate [5e-05, 5e-05] | |
| [2024-09-09 05:51:46,463][__main__][INFO] - loss : 0.40566003849316457 | |
| [2024-09-09 05:58:52,693][__main__][INFO] - Epoch 14 with learning rate [4.347369038899744e-05, 4.347369038899744e-05] | |
| [2024-09-09 07:24:16,928][__main__][INFO] - loss : 0.3984212905488267 | |
| [2024-09-09 07:31:20,836][__main__][INFO] - Epoch 15 with learning rate [3.705904774487396e-05, 3.705904774487396e-05] | |
| [2024-09-09 08:56:31,213][__main__][INFO] - loss : 0.39157586989150484 | |
| [2024-09-09 09:03:38,823][__main__][INFO] - Epoch 16 with learning rate [3.086582838174551e-05, 3.086582838174551e-05] | |
| [2024-09-09 10:28:41,628][__main__][INFO] - loss : 0.3850977401202303 | |
| [2024-09-09 10:35:47,706][__main__][INFO] - Epoch 17 with learning rate [2.500000000000001e-05, 2.500000000000001e-05] | |
| [2024-09-09 12:01:04,108][__main__][INFO] - loss : 0.3790526455592563 | |
| [2024-09-09 12:08:08,846][__main__][INFO] - Epoch 18 with learning rate [1.9561928549563968e-05, 1.9561928549563968e-05] | |
| [2024-09-09 13:33:15,071][__main__][INFO] - loss : 0.3737031583387299 | |
| [2024-09-09 13:40:16,914][__main__][INFO] - Epoch 19 with learning rate [1.4644660940672627e-05, 1.4644660940672627e-05] | |
| [2024-09-09 15:05:29,132][__main__][INFO] - loss : 0.3689893393889676 | |
| [2024-09-09 15:12:33,943][__main__][INFO] - Epoch 20 with learning rate [1.0332332985438248e-05, 1.0332332985438248e-05] | |
| [2024-09-09 16:37:45,482][__main__][INFO] - loss : 0.3648544587580553 | |
| [2024-09-09 16:44:50,581][__main__][INFO] - Epoch 21 with learning rate [6.698729810778065e-06, 6.698729810778065e-06] | |
| [2024-09-09 18:10:20,957][__main__][INFO] - loss : 0.36169400309161137 | |
| [2024-09-09 18:17:30,492][__main__][INFO] - Epoch 22 with learning rate [3.8060233744356633e-06, 3.8060233744356633e-06] | |
| [2024-09-09 19:42:56,851][__main__][INFO] - loss : 0.35911366006658596 | |
| [2024-09-09 19:50:01,181][__main__][INFO] - Epoch 23 with learning rate [1.70370868554659e-06, 1.70370868554659e-06] | |
| [2024-09-09 21:14:57,272][__main__][INFO] - loss : 0.3574252754320835 | |
| [2024-09-09 21:21:59,072][__main__][INFO] - Epoch 24 with learning rate [4.277569313094809e-07, 4.277569313094809e-07] | |
| [2024-09-09 22:47:10,675][__main__][INFO] - loss : 0.3564971802756542 | |
| [2024-09-09 22:54:14,895][__main__][INFO] - Train time [hour] : 38.55840078751246 h | |
| [2024-09-09 22:54:14,895][__main__][INFO] - Results saved in /root/FORKS/LenslessPiCamStefan/outputs/2024-09-08/08-20-25 | |