Spaces:
Runtime error
Runtime error
| 2023-10-06 03:49:36,638 MJV2 INFO {'data': {'filter_length': 2048, 'hop_length': 400, 'max_wav_value': 32768.0, 'mel_fmax': None, 'mel_fmin': 0.0, 'n_mel_channels': 125, 'sampling_rate': 40000, 'win_length': 2048, 'training_files': './logs/MJV2/filelist.txt'}, 'model': {'filter_channels': 768, 'gin_channels': 256, 'hidden_channels': 192, 'inter_channels': 192, 'kernel_size': 3, 'n_heads': 2, 'n_layers': 6, 'p_dropout': 0, 'resblock': '1', 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'resblock_kernel_sizes': [3, 7, 11], 'spk_embed_dim': 109, 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [16, 16, 4, 4], 'upsample_rates': [10, 10, 2, 2], 'use_spectral_norm': False}, 'train': {'batch_size': 7, 'betas': [0.8, 0.99], 'c_kl': 1.0, 'c_mel': 45, 'epochs': 20000, 'eps': 1e-09, 'fp16_run': True, 'init_lr_ratio': 1, 'learning_rate': 0.0001, 'log_interval': 200, 'lr_decay': 0.999875, 'seed': 1234, 'segment_size': 12800, 'warmup_epochs': 0}, 'model_dir': './logs/MJV2', 'experiment_dir': './logs/MJV2', 'save_every_epoch': 20, 'name': 'MJV2', 'total_epoch': 300, 'pretrainG': 'assets/pretrained_v2/f0G40k.pth', 'pretrainD': 'assets/pretrained_v2/f0D40k.pth', 'version': 'v2', 'gpus': '0', 'sample_rate': '40k', 'if_f0': 1, 'if_latest': 1, 'save_every_weights': '1', 'if_cache_data_in_gpu': 1} | |
| 2023-10-06 03:49:41,021 MJV2 INFO loaded pretrained assets/pretrained_v2/f0G40k.pth | |
| 2023-10-06 03:49:41,611 MJV2 INFO <All keys matched successfully> | |
| 2023-10-06 03:49:41,612 MJV2 INFO loaded pretrained assets/pretrained_v2/f0D40k.pth | |
| 2023-10-06 03:49:42,543 MJV2 INFO <All keys matched successfully> | |
| 2023-10-06 03:50:16,825 MJV2 INFO Train Epoch: 1 [0%] | |
| 2023-10-06 03:50:16,826 MJV2 INFO [0, 0.0001] | |
| 2023-10-06 03:50:16,826 MJV2 INFO loss_disc=4.437, loss_gen=2.698, loss_fm=0.584,loss_mel=27.227, loss_kl=9.000 | |
| 2023-10-06 03:50:17,898 MJV2 INFO ====> Epoch: 1 [2023-10-06 03:50:17] | (0:00:13.210963) | |
| 2023-10-06 03:50:18,450 MJV2 INFO ====> Epoch: 2 [2023-10-06 03:50:18] | (0:00:00.539312) | |
| 2023-10-06 03:50:18,940 MJV2 INFO ====> Epoch: 3 [2023-10-06 03:50:18] | (0:00:00.485277) | |
| 2023-10-06 03:50:19,435 MJV2 INFO ====> Epoch: 4 [2023-10-06 03:50:19] | (0:00:00.489769) | |
| 2023-10-06 03:50:19,952 MJV2 INFO ====> Epoch: 5 [2023-10-06 03:50:19] | (0:00:00.505072) | |
| 2023-10-06 03:50:20,535 MJV2 INFO ====> Epoch: 6 [2023-10-06 03:50:20] | (0:00:00.577565) | |
| 2023-10-06 03:50:21,114 MJV2 INFO ====> Epoch: 7 [2023-10-06 03:50:21] | (0:00:00.571026) | |
| 2023-10-06 03:50:21,723 MJV2 INFO ====> Epoch: 8 [2023-10-06 03:50:21] | (0:00:00.601737) | |
| 2023-10-06 03:50:22,325 MJV2 INFO ====> Epoch: 9 [2023-10-06 03:50:22] | (0:00:00.594052) | |
| 2023-10-06 03:50:22,978 MJV2 INFO ====> Epoch: 10 [2023-10-06 03:50:22] | (0:00:00.646191) | |
| 2023-10-06 03:50:23,624 MJV2 INFO ====> Epoch: 11 [2023-10-06 03:50:23] | (0:00:00.637738) | |
| 2023-10-06 03:50:24,261 MJV2 INFO ====> Epoch: 12 [2023-10-06 03:50:24] | (0:00:00.631217) | |
| 2023-10-06 03:50:24,886 MJV2 INFO ====> Epoch: 13 [2023-10-06 03:50:24] | (0:00:00.617479) | |
| 2023-10-06 03:50:25,533 MJV2 INFO ====> Epoch: 14 [2023-10-06 03:50:25] | (0:00:00.639175) | |
| 2023-10-06 03:50:26,161 MJV2 INFO ====> Epoch: 15 [2023-10-06 03:50:26] | (0:00:00.619377) | |
| 2023-10-06 03:50:26,819 MJV2 INFO ====> Epoch: 16 [2023-10-06 03:50:26] | (0:00:00.650853) | |
| 2023-10-06 03:50:27,484 MJV2 INFO ====> Epoch: 17 [2023-10-06 03:50:27] | (0:00:00.656293) | |
| 2023-10-06 03:50:28,089 MJV2 INFO ====> Epoch: 18 [2023-10-06 03:50:28] | (0:00:00.598655) | |
| 2023-10-06 03:50:28,626 MJV2 INFO ====> Epoch: 19 [2023-10-06 03:50:28] | (0:00:00.532480) | |
| 2023-10-06 03:50:29,157 MJV2 INFO Saving model and optimizer state at epoch 20 to ./logs/MJV2/G_2333333.pth | |
| 2023-10-06 03:50:34,855 MJV2 INFO Saving model and optimizer state at epoch 20 to ./logs/MJV2/D_2333333.pth | |
| 2023-10-06 03:50:38,736 MJV2 INFO saving ckpt MJV2_e20:Success. | |
| 2023-10-06 03:50:38,737 MJV2 INFO ====> Epoch: 20 [2023-10-06 03:50:38] | (0:00:10.105891) | |
| 2023-10-06 03:50:39,447 MJV2 INFO ====> Epoch: 21 [2023-10-06 03:50:39] | (0:00:00.701162) | |
| 2023-10-06 03:50:40,071 MJV2 INFO ====> Epoch: 22 [2023-10-06 03:50:40] | (0:00:00.616781) | |
| 2023-10-06 03:50:40,643 MJV2 INFO ====> Epoch: 23 [2023-10-06 03:50:40] | (0:00:00.560207) | |
| 2023-10-06 03:50:41,264 MJV2 INFO ====> Epoch: 24 [2023-10-06 03:50:41] | (0:00:00.614343) | |
| 2023-10-06 03:50:41,890 MJV2 INFO ====> Epoch: 25 [2023-10-06 03:50:41] | (0:00:00.618039) | |
| 2023-10-06 03:50:42,537 MJV2 INFO ====> Epoch: 26 [2023-10-06 03:50:42] | (0:00:00.638990) | |
| 2023-10-06 03:50:43,167 MJV2 INFO ====> Epoch: 27 [2023-10-06 03:50:43] | (0:00:00.621153) | |
| 2023-10-06 03:50:43,819 MJV2 INFO ====> Epoch: 28 [2023-10-06 03:50:43] | (0:00:00.644734) | |
| 2023-10-06 03:50:44,563 MJV2 INFO ====> Epoch: 29 [2023-10-06 03:50:44] | (0:00:00.735037) | |
| 2023-10-06 03:50:45,813 MJV2 INFO ====> Epoch: 30 [2023-10-06 03:50:45] | (0:00:01.214035) | |
| 2023-10-06 03:50:46,632 MJV2 INFO ====> Epoch: 31 [2023-10-06 03:50:46] | (0:00:00.808276) | |
| 2023-10-06 03:50:47,428 MJV2 INFO ====> Epoch: 32 [2023-10-06 03:50:47] | (0:00:00.786112) | |
| 2023-10-06 03:50:48,155 MJV2 INFO ====> Epoch: 33 [2023-10-06 03:50:48] | (0:00:00.718137) | |
| 2023-10-06 03:50:48,950 MJV2 INFO ====> Epoch: 34 [2023-10-06 03:50:48] | (0:00:00.787028) | |
| 2023-10-06 03:50:49,795 MJV2 INFO ====> Epoch: 35 [2023-10-06 03:50:49] | (0:00:00.835644) | |
| 2023-10-06 03:50:50,688 MJV2 INFO ====> Epoch: 36 [2023-10-06 03:50:50] | (0:00:00.883101) | |
| 2023-10-06 03:50:51,400 MJV2 INFO ====> Epoch: 37 [2023-10-06 03:50:51] | (0:00:00.700183) | |
| 2023-10-06 03:50:52,093 MJV2 INFO ====> Epoch: 38 [2023-10-06 03:50:52] | (0:00:00.684977) | |
| 2023-10-06 03:50:52,702 MJV2 INFO ====> Epoch: 39 [2023-10-06 03:50:52] | (0:00:00.604089) | |
| 2023-10-06 03:50:53,367 MJV2 INFO Saving model and optimizer state at epoch 40 to ./logs/MJV2/G_2333333.pth | |
| 2023-10-06 03:51:00,311 MJV2 INFO Saving model and optimizer state at epoch 40 to ./logs/MJV2/D_2333333.pth | |
| 2023-10-06 03:51:04,292 MJV2 INFO saving ckpt MJV2_e40:Success. | |
| 2023-10-06 03:51:04,292 MJV2 INFO ====> Epoch: 40 [2023-10-06 03:51:04] | (0:00:11.582100) | |
| 2023-10-06 03:51:04,910 MJV2 INFO ====> Epoch: 41 [2023-10-06 03:51:04] | (0:00:00.611901) | |
| 2023-10-06 03:51:05,430 MJV2 INFO ====> Epoch: 42 [2023-10-06 03:51:05] | (0:00:00.515362) | |
| 2023-10-06 03:51:05,960 MJV2 INFO ====> Epoch: 43 [2023-10-06 03:51:05] | (0:00:00.524792) | |
| 2023-10-06 03:51:06,496 MJV2 INFO ====> Epoch: 44 [2023-10-06 03:51:06] | (0:00:00.531301) | |
| 2023-10-06 03:51:07,028 MJV2 INFO ====> Epoch: 45 [2023-10-06 03:51:07] | (0:00:00.526814) | |
| 2023-10-06 03:51:07,565 MJV2 INFO ====> Epoch: 46 [2023-10-06 03:51:07] | (0:00:00.532899) | |
| 2023-10-06 03:51:08,100 MJV2 INFO ====> Epoch: 47 [2023-10-06 03:51:08] | (0:00:00.530152) | |
| 2023-10-06 03:51:08,649 MJV2 INFO ====> Epoch: 48 [2023-10-06 03:51:08] | (0:00:00.543636) | |
| 2023-10-06 03:51:09,182 MJV2 INFO ====> Epoch: 49 [2023-10-06 03:51:09] | (0:00:00.527989) | |
| 2023-10-06 03:51:09,714 MJV2 INFO ====> Epoch: 50 [2023-10-06 03:51:09] | (0:00:00.527200) | |
| 2023-10-06 03:51:10,281 MJV2 INFO ====> Epoch: 51 [2023-10-06 03:51:10] | (0:00:00.562587) | |
| 2023-10-06 03:51:11,008 MJV2 INFO ====> Epoch: 52 [2023-10-06 03:51:11] | (0:00:00.722259) | |
| 2023-10-06 03:51:11,806 MJV2 INFO ====> Epoch: 53 [2023-10-06 03:51:11] | (0:00:00.787369) | |
| 2023-10-06 03:51:12,535 MJV2 INFO ====> Epoch: 54 [2023-10-06 03:51:12] | (0:00:00.719133) | |
| 2023-10-06 03:51:13,213 MJV2 INFO ====> Epoch: 55 [2023-10-06 03:51:13] | (0:00:00.668560) | |
| 2023-10-06 03:51:13,814 MJV2 INFO ====> Epoch: 56 [2023-10-06 03:51:13] | (0:00:00.594812) | |
| 2023-10-06 03:51:14,550 MJV2 INFO ====> Epoch: 57 [2023-10-06 03:51:14] | (0:00:00.728585) | |
| 2023-10-06 03:51:15,360 MJV2 INFO ====> Epoch: 58 [2023-10-06 03:51:15] | (0:00:00.793917) | |
| 2023-10-06 03:51:15,973 MJV2 INFO ====> Epoch: 59 [2023-10-06 03:51:15] | (0:00:00.607627) | |
| 2023-10-06 03:51:17,197 MJV2 INFO Saving model and optimizer state at epoch 60 to ./logs/MJV2/G_2333333.pth | |
| 2023-10-06 03:51:21,659 MJV2 INFO Saving model and optimizer state at epoch 60 to ./logs/MJV2/D_2333333.pth | |
| 2023-10-06 03:51:25,817 MJV2 INFO saving ckpt MJV2_e60:Success. | |
| 2023-10-06 03:51:25,818 MJV2 INFO ====> Epoch: 60 [2023-10-06 03:51:25] | (0:00:09.836865) | |
| 2023-10-06 03:51:26,451 MJV2 INFO ====> Epoch: 61 [2023-10-06 03:51:26] | (0:00:00.625652) | |
| 2023-10-06 03:51:26,984 MJV2 INFO ====> Epoch: 62 [2023-10-06 03:51:26] | (0:00:00.528752) | |
| 2023-10-06 03:51:27,519 MJV2 INFO ====> Epoch: 63 [2023-10-06 03:51:27] | (0:00:00.529525) | |
| 2023-10-06 03:51:28,203 MJV2 INFO ====> Epoch: 64 [2023-10-06 03:51:28] | (0:00:00.679488) | |
| 2023-10-06 03:51:28,817 MJV2 INFO ====> Epoch: 65 [2023-10-06 03:51:28] | (0:00:00.605069) | |
| 2023-10-06 03:51:29,421 MJV2 INFO ====> Epoch: 66 [2023-10-06 03:51:29] | (0:00:00.597881) | |
| 2023-10-06 03:51:30,070 MJV2 INFO ====> Epoch: 67 [2023-10-06 03:51:30] | (0:00:00.642525) | |
| 2023-10-06 03:51:30,708 MJV2 INFO ====> Epoch: 68 [2023-10-06 03:51:30] | (0:00:00.632335) | |
| 2023-10-06 03:51:31,303 MJV2 INFO ====> Epoch: 69 [2023-10-06 03:51:31] | (0:00:00.587060) | |
| 2023-10-06 03:51:32,011 MJV2 INFO ====> Epoch: 70 [2023-10-06 03:51:32] | (0:00:00.701323) | |
| 2023-10-06 03:51:32,803 MJV2 INFO ====> Epoch: 71 [2023-10-06 03:51:32] | (0:00:00.779498) | |
| 2023-10-06 03:51:33,535 MJV2 INFO ====> Epoch: 72 [2023-10-06 03:51:33] | (0:00:00.722884) | |
| 2023-10-06 03:51:34,283 MJV2 INFO ====> Epoch: 73 [2023-10-06 03:51:34] | (0:00:00.739816) | |
| 2023-10-06 03:51:35,099 MJV2 INFO ====> Epoch: 74 [2023-10-06 03:51:35] | (0:00:00.799174) | |
| 2023-10-06 03:51:36,151 MJV2 INFO ====> Epoch: 75 [2023-10-06 03:51:36] | (0:00:01.039519) | |
| 2023-10-06 03:51:37,172 MJV2 INFO ====> Epoch: 76 [2023-10-06 03:51:37] | (0:00:01.005433) | |
| 2023-10-06 03:51:38,225 MJV2 INFO ====> Epoch: 77 [2023-10-06 03:51:38] | (0:00:01.040676) | |
| 2023-10-06 03:51:39,322 MJV2 INFO ====> Epoch: 78 [2023-10-06 03:51:39] | (0:00:01.082663) | |
| 2023-10-06 03:51:40,388 MJV2 INFO ====> Epoch: 79 [2023-10-06 03:51:40] | (0:00:01.044772) | |
| 2023-10-06 03:51:41,309 MJV2 INFO Saving model and optimizer state at epoch 80 to ./logs/MJV2/G_2333333.pth | |
| 2023-10-06 03:51:46,761 MJV2 INFO Saving model and optimizer state at epoch 80 to ./logs/MJV2/D_2333333.pth | |
| 2023-10-06 03:51:54,511 MJV2 INFO saving ckpt MJV2_e80:Success. | |
| 2023-10-06 03:51:54,512 MJV2 INFO ====> Epoch: 80 [2023-10-06 03:51:54] | (0:00:14.117232) | |
| 2023-10-06 03:51:55,186 MJV2 INFO ====> Epoch: 81 [2023-10-06 03:51:55] | (0:00:00.665237) | |
| 2023-10-06 03:51:55,804 MJV2 INFO ====> Epoch: 82 [2023-10-06 03:51:55] | (0:00:00.608887) | |
| 2023-10-06 03:51:56,422 MJV2 INFO ====> Epoch: 83 [2023-10-06 03:51:56] | (0:00:00.611605) | |
| 2023-10-06 03:51:57,034 MJV2 INFO ====> Epoch: 84 [2023-10-06 03:51:57] | (0:00:00.603133) | |
| 2023-10-06 03:51:57,735 MJV2 INFO ====> Epoch: 85 [2023-10-06 03:51:57] | (0:00:00.690840) | |
| 2023-10-06 03:51:59,189 MJV2 INFO ====> Epoch: 86 [2023-10-06 03:51:59] | (0:00:01.441537) | |
| 2023-10-06 03:52:00,082 MJV2 INFO ====> Epoch: 87 [2023-10-06 03:52:00] | (0:00:00.871573) | |
| 2023-10-06 03:52:01,017 MJV2 INFO ====> Epoch: 88 [2023-10-06 03:52:01] | (0:00:00.923933) | |
| 2023-10-06 03:52:01,642 MJV2 INFO ====> Epoch: 89 [2023-10-06 03:52:01] | (0:00:00.612630) | |
| 2023-10-06 03:52:02,263 MJV2 INFO ====> Epoch: 90 [2023-10-06 03:52:02] | (0:00:00.614190) | |
| 2023-10-06 03:52:02,906 MJV2 INFO ====> Epoch: 91 [2023-10-06 03:52:02] | (0:00:00.635062) | |
| 2023-10-06 03:52:03,471 MJV2 INFO ====> Epoch: 92 [2023-10-06 03:52:03] | (0:00:00.557012) | |
| 2023-10-06 03:52:04,042 MJV2 INFO ====> Epoch: 93 [2023-10-06 03:52:04] | (0:00:00.566189) | |
| 2023-10-06 03:52:04,637 MJV2 INFO ====> Epoch: 94 [2023-10-06 03:52:04] | (0:00:00.583105) | |
| 2023-10-06 03:52:05,203 MJV2 INFO ====> Epoch: 95 [2023-10-06 03:52:05] | (0:00:00.561535) | |
| 2023-10-06 03:52:05,890 MJV2 INFO ====> Epoch: 96 [2023-10-06 03:52:05] | (0:00:00.679752) | |
| 2023-10-06 03:52:06,455 MJV2 INFO ====> Epoch: 97 [2023-10-06 03:52:06] | (0:00:00.559880) | |
| 2023-10-06 03:52:07,007 MJV2 INFO ====> Epoch: 98 [2023-10-06 03:52:07] | (0:00:00.546814) | |
| 2023-10-06 03:52:07,558 MJV2 INFO ====> Epoch: 99 [2023-10-06 03:52:07] | (0:00:00.546054) | |
| 2023-10-06 03:52:08,143 MJV2 INFO Saving model and optimizer state at epoch 100 to ./logs/MJV2/G_2333333.pth | |
| 2023-10-06 03:52:14,116 MJV2 INFO Saving model and optimizer state at epoch 100 to ./logs/MJV2/D_2333333.pth | |
| 2023-10-06 03:52:24,697 MJV2 INFO saving ckpt MJV2_e100:Success. | |
| 2023-10-06 03:52:24,698 MJV2 INFO ====> Epoch: 100 [2023-10-06 03:52:24] | (0:00:17.136264) | |
| 2023-10-06 03:52:25,391 MJV2 INFO ====> Epoch: 101 [2023-10-06 03:52:25] | (0:00:00.679721) | |
| 2023-10-06 03:52:26,012 MJV2 INFO ====> Epoch: 102 [2023-10-06 03:52:26] | (0:00:00.613850) | |
| 2023-10-06 03:52:26,717 MJV2 INFO ====> Epoch: 103 [2023-10-06 03:52:26] | (0:00:00.699744) | |
| 2023-10-06 03:52:27,548 MJV2 INFO ====> Epoch: 104 [2023-10-06 03:52:27] | (0:00:00.822955) | |
| 2023-10-06 03:52:28,489 MJV2 INFO ====> Epoch: 105 [2023-10-06 03:52:28] | (0:00:00.925860) | |
| 2023-10-06 03:52:29,338 MJV2 INFO ====> Epoch: 106 [2023-10-06 03:52:29] | (0:00:00.836484) | |
| 2023-10-06 03:52:30,043 MJV2 INFO ====> Epoch: 107 [2023-10-06 03:52:30] | (0:00:00.691541) | |
| 2023-10-06 03:52:30,899 MJV2 INFO ====> Epoch: 108 [2023-10-06 03:52:30] | (0:00:00.843734) | |
| 2023-10-06 03:52:31,756 MJV2 INFO ====> Epoch: 109 [2023-10-06 03:52:31] | (0:00:00.846437) | |
| 2023-10-06 03:52:32,794 MJV2 INFO ====> Epoch: 110 [2023-10-06 03:52:32] | (0:00:01.028016) | |
| 2023-10-06 03:52:33,735 MJV2 INFO ====> Epoch: 111 [2023-10-06 03:52:33] | (0:00:00.932825) | |
| 2023-10-06 03:52:34,410 MJV2 INFO ====> Epoch: 112 [2023-10-06 03:52:34] | (0:00:00.668154) | |
| 2023-10-06 03:52:35,036 MJV2 INFO ====> Epoch: 113 [2023-10-06 03:52:35] | (0:00:00.618588) | |
| 2023-10-06 03:52:35,619 MJV2 INFO ====> Epoch: 114 [2023-10-06 03:52:35] | (0:00:00.576706) | |
| 2023-10-06 03:52:36,216 MJV2 INFO ====> Epoch: 115 [2023-10-06 03:52:36] | (0:00:00.590554) | |
| 2023-10-06 03:52:36,829 MJV2 INFO ====> Epoch: 116 [2023-10-06 03:52:36] | (0:00:00.605132) | |
| 2023-10-06 03:52:37,443 MJV2 INFO ====> Epoch: 117 [2023-10-06 03:52:37] | (0:00:00.607876) | |
| 2023-10-06 03:52:38,038 MJV2 INFO ====> Epoch: 118 [2023-10-06 03:52:38] | (0:00:00.586923) | |
| 2023-10-06 03:52:38,649 MJV2 INFO ====> Epoch: 119 [2023-10-06 03:52:38] | (0:00:00.605610) | |
| 2023-10-06 03:52:39,262 MJV2 INFO Saving model and optimizer state at epoch 120 to ./logs/MJV2/G_2333333.pth | |
| 2023-10-06 03:52:44,533 MJV2 INFO Saving model and optimizer state at epoch 120 to ./logs/MJV2/D_2333333.pth | |
| 2023-10-06 03:52:58,405 MJV2 INFO saving ckpt MJV2_e120:Success. | |
| 2023-10-06 03:52:58,408 MJV2 INFO ====> Epoch: 120 [2023-10-06 03:52:58] | (0:00:19.752815) | |
| 2023-10-06 03:52:59,169 MJV2 INFO ====> Epoch: 121 [2023-10-06 03:52:59] | (0:00:00.753933) | |
| 2023-10-06 03:52:59,879 MJV2 INFO ====> Epoch: 122 [2023-10-06 03:52:59] | (0:00:00.702880) | |
| 2023-10-06 03:53:00,663 MJV2 INFO ====> Epoch: 123 [2023-10-06 03:53:00] | (0:00:00.773274) | |
| 2023-10-06 03:53:01,556 MJV2 INFO ====> Epoch: 124 [2023-10-06 03:53:01] | (0:00:00.884808) | |
| 2023-10-06 03:53:02,377 MJV2 INFO ====> Epoch: 125 [2023-10-06 03:53:02] | (0:00:00.812450) | |
| 2023-10-06 03:53:03,142 MJV2 INFO ====> Epoch: 126 [2023-10-06 03:53:03] | (0:00:00.753508) | |
| 2023-10-06 03:53:03,844 MJV2 INFO ====> Epoch: 127 [2023-10-06 03:53:03] | (0:00:00.691548) | |
| 2023-10-06 03:53:04,487 MJV2 INFO ====> Epoch: 128 [2023-10-06 03:53:04] | (0:00:00.636253) | |
| 2023-10-06 03:53:05,506 MJV2 INFO ====> Epoch: 129 [2023-10-06 03:53:05] | (0:00:01.011236) | |
| 2023-10-06 03:53:06,566 MJV2 INFO ====> Epoch: 130 [2023-10-06 03:53:06] | (0:00:01.048733) | |
| 2023-10-06 03:53:07,309 MJV2 INFO ====> Epoch: 131 [2023-10-06 03:53:07] | (0:00:00.726694) | |
| 2023-10-06 03:53:07,885 MJV2 INFO ====> Epoch: 132 [2023-10-06 03:53:07] | (0:00:00.569697) | |
| 2023-10-06 03:53:08,504 MJV2 INFO ====> Epoch: 133 [2023-10-06 03:53:08] | (0:00:00.614753) | |
| 2023-10-06 03:53:09,109 MJV2 INFO ====> Epoch: 134 [2023-10-06 03:53:09] | (0:00:00.597936) | |
| 2023-10-06 03:53:09,678 MJV2 INFO ====> Epoch: 135 [2023-10-06 03:53:09] | (0:00:00.564048) | |
| 2023-10-06 03:53:10,280 MJV2 INFO ====> Epoch: 136 [2023-10-06 03:53:10] | (0:00:00.596084) | |
| 2023-10-06 03:53:10,928 MJV2 INFO ====> Epoch: 137 [2023-10-06 03:53:10] | (0:00:00.642712) | |
| 2023-10-06 03:53:11,547 MJV2 INFO ====> Epoch: 138 [2023-10-06 03:53:11] | (0:00:00.611499) | |
| 2023-10-06 03:53:12,150 MJV2 INFO ====> Epoch: 139 [2023-10-06 03:53:12] | (0:00:00.596271) | |
| 2023-10-06 03:53:12,750 MJV2 INFO Saving model and optimizer state at epoch 140 to ./logs/MJV2/G_2333333.pth | |
| 2023-10-06 03:53:19,037 MJV2 INFO Saving model and optimizer state at epoch 140 to ./logs/MJV2/D_2333333.pth | |
| 2023-10-06 03:53:34,706 MJV2 INFO saving ckpt MJV2_e140:Success. | |
| 2023-10-06 03:53:34,707 MJV2 INFO ====> Epoch: 140 [2023-10-06 03:53:34] | (0:00:22.552880) | |
| 2023-10-06 03:53:35,611 MJV2 INFO ====> Epoch: 141 [2023-10-06 03:53:35] | (0:00:00.894458) | |
| 2023-10-06 03:53:36,466 MJV2 INFO ====> Epoch: 142 [2023-10-06 03:53:36] | (0:00:00.834391) | |
| 2023-10-06 03:53:37,364 MJV2 INFO ====> Epoch: 143 [2023-10-06 03:53:37] | (0:00:00.887085) | |
| 2023-10-06 03:53:38,335 MJV2 INFO ====> Epoch: 144 [2023-10-06 03:53:38] | (0:00:00.961163) | |
| 2023-10-06 03:53:38,919 MJV2 INFO ====> Epoch: 145 [2023-10-06 03:53:38] | (0:00:00.576101) | |
| 2023-10-06 03:53:39,467 MJV2 INFO ====> Epoch: 146 [2023-10-06 03:53:39] | (0:00:00.543167) | |
| 2023-10-06 03:53:40,068 MJV2 INFO ====> Epoch: 147 [2023-10-06 03:53:40] | (0:00:00.595421) | |
| 2023-10-06 03:53:40,675 MJV2 INFO ====> Epoch: 148 [2023-10-06 03:53:40] | (0:00:00.599768) | |
| 2023-10-06 03:53:41,285 MJV2 INFO ====> Epoch: 149 [2023-10-06 03:53:41] | (0:00:00.603598) | |
| 2023-10-06 03:53:41,873 MJV2 INFO ====> Epoch: 150 [2023-10-06 03:53:41] | (0:00:00.582782) | |
| 2023-10-06 03:53:42,471 MJV2 INFO ====> Epoch: 151 [2023-10-06 03:53:42] | (0:00:00.591710) | |
| 2023-10-06 03:53:43,044 MJV2 INFO ====> Epoch: 152 [2023-10-06 03:53:43] | (0:00:00.566762) | |
| 2023-10-06 03:53:43,649 MJV2 INFO ====> Epoch: 153 [2023-10-06 03:53:43] | (0:00:00.600362) | |
| 2023-10-06 03:53:44,297 MJV2 INFO ====> Epoch: 154 [2023-10-06 03:53:44] | (0:00:00.640482) | |
| 2023-10-06 03:53:44,950 MJV2 INFO ====> Epoch: 155 [2023-10-06 03:53:44] | (0:00:00.645727) | |
| 2023-10-06 03:53:45,559 MJV2 INFO ====> Epoch: 156 [2023-10-06 03:53:45] | (0:00:00.602849) | |
| 2023-10-06 03:53:46,163 MJV2 INFO ====> Epoch: 157 [2023-10-06 03:53:46] | (0:00:00.592601) | |
| 2023-10-06 03:53:46,758 MJV2 INFO ====> Epoch: 158 [2023-10-06 03:53:46] | (0:00:00.588324) | |
| 2023-10-06 03:53:47,355 MJV2 INFO ====> Epoch: 159 [2023-10-06 03:53:47] | (0:00:00.591694) | |
| 2023-10-06 03:53:48,009 MJV2 INFO Saving model and optimizer state at epoch 160 to ./logs/MJV2/G_2333333.pth | |
| 2023-10-06 03:53:51,031 MJV2 INFO Saving model and optimizer state at epoch 160 to ./logs/MJV2/D_2333333.pth | |
| 2023-10-06 03:54:04,590 MJV2 INFO saving ckpt MJV2_e160:Success. | |
| 2023-10-06 03:54:04,590 MJV2 INFO ====> Epoch: 160 [2023-10-06 03:54:04] | (0:00:17.223768) | |
| 2023-10-06 03:54:05,412 MJV2 INFO ====> Epoch: 161 [2023-10-06 03:54:05] | (0:00:00.815681) | |
| 2023-10-06 03:54:06,371 MJV2 INFO ====> Epoch: 162 [2023-10-06 03:54:06] | (0:00:00.940540) | |
| 2023-10-06 03:54:07,297 MJV2 INFO ====> Epoch: 163 [2023-10-06 03:54:07] | (0:00:00.910478) | |
| 2023-10-06 03:54:07,997 MJV2 INFO ====> Epoch: 164 [2023-10-06 03:54:07] | (0:00:00.691057) | |
| 2023-10-06 03:54:08,684 MJV2 INFO ====> Epoch: 165 [2023-10-06 03:54:08] | (0:00:00.679572) | |
| 2023-10-06 03:54:09,316 MJV2 INFO ====> Epoch: 166 [2023-10-06 03:54:09] | (0:00:00.625722) | |
| 2023-10-06 03:54:09,945 MJV2 INFO ====> Epoch: 167 [2023-10-06 03:54:09] | (0:00:00.621918) | |
| 2023-10-06 03:54:10,621 MJV2 INFO ====> Epoch: 168 [2023-10-06 03:54:10] | (0:00:00.668251) | |
| 2023-10-06 03:54:11,274 MJV2 INFO ====> Epoch: 169 [2023-10-06 03:54:11] | (0:00:00.644905) | |
| 2023-10-06 03:54:11,883 MJV2 INFO ====> Epoch: 170 [2023-10-06 03:54:11] | (0:00:00.601653) | |
| 2023-10-06 03:54:12,429 MJV2 INFO ====> Epoch: 171 [2023-10-06 03:54:12] | (0:00:00.540736) | |
| 2023-10-06 03:54:12,985 MJV2 INFO ====> Epoch: 172 [2023-10-06 03:54:12] | (0:00:00.550989) | |
| 2023-10-06 03:54:13,534 MJV2 INFO ====> Epoch: 173 [2023-10-06 03:54:13] | (0:00:00.544413) | |
| 2023-10-06 03:54:14,111 MJV2 INFO ====> Epoch: 174 [2023-10-06 03:54:14] | (0:00:00.572376) | |
| 2023-10-06 03:54:14,781 MJV2 INFO ====> Epoch: 175 [2023-10-06 03:54:14] | (0:00:00.662149) | |
| 2023-10-06 03:54:15,370 MJV2 INFO ====> Epoch: 176 [2023-10-06 03:54:15] | (0:00:00.582636) | |
| 2023-10-06 03:54:15,959 MJV2 INFO ====> Epoch: 177 [2023-10-06 03:54:15] | (0:00:00.580516) | |
| 2023-10-06 03:54:16,542 MJV2 INFO ====> Epoch: 178 [2023-10-06 03:54:16] | (0:00:00.577851) | |
| 2023-10-06 03:54:17,187 MJV2 INFO ====> Epoch: 179 [2023-10-06 03:54:17] | (0:00:00.641118) | |
| 2023-10-06 03:54:17,913 MJV2 INFO Saving model and optimizer state at epoch 180 to ./logs/MJV2/G_2333333.pth | |
| 2023-10-06 03:54:20,057 MJV2 INFO Saving model and optimizer state at epoch 180 to ./logs/MJV2/D_2333333.pth | |
| 2023-10-06 03:54:34,201 MJV2 INFO saving ckpt MJV2_e180:Success. | |
| 2023-10-06 03:54:34,202 MJV2 INFO ====> Epoch: 180 [2023-10-06 03:54:34] | (0:00:17.006622) | |
| 2023-10-06 03:54:34,931 MJV2 INFO ====> Epoch: 181 [2023-10-06 03:54:34] | (0:00:00.719226) | |
| 2023-10-06 03:54:35,594 MJV2 INFO ====> Epoch: 182 [2023-10-06 03:54:35] | (0:00:00.654828) | |
| 2023-10-06 03:54:36,276 MJV2 INFO ====> Epoch: 183 [2023-10-06 03:54:36] | (0:00:00.673454) | |
| 2023-10-06 03:54:36,921 MJV2 INFO ====> Epoch: 184 [2023-10-06 03:54:36] | (0:00:00.637752) | |
| 2023-10-06 03:54:37,661 MJV2 INFO ====> Epoch: 185 [2023-10-06 03:54:37] | (0:00:00.731861) | |
| 2023-10-06 03:54:38,289 MJV2 INFO ====> Epoch: 186 [2023-10-06 03:54:38] | (0:00:00.619867) | |
| 2023-10-06 03:54:38,914 MJV2 INFO ====> Epoch: 187 [2023-10-06 03:54:38] | (0:00:00.618333) | |
| 2023-10-06 03:54:39,653 MJV2 INFO ====> Epoch: 188 [2023-10-06 03:54:39] | (0:00:00.730853) | |
| 2023-10-06 03:54:40,686 MJV2 INFO ====> Epoch: 189 [2023-10-06 03:54:40] | (0:00:01.022155) | |
| 2023-10-06 03:54:41,542 MJV2 INFO ====> Epoch: 190 [2023-10-06 03:54:41] | (0:00:00.840247) | |
| 2023-10-06 03:54:42,472 MJV2 INFO ====> Epoch: 191 [2023-10-06 03:54:42] | (0:00:00.919018) | |
| 2023-10-06 03:54:43,376 MJV2 INFO ====> Epoch: 192 [2023-10-06 03:54:43] | (0:00:00.892642) | |
| 2023-10-06 03:54:44,428 MJV2 INFO ====> Epoch: 193 [2023-10-06 03:54:44] | (0:00:01.044749) | |
| 2023-10-06 03:54:45,094 MJV2 INFO ====> Epoch: 194 [2023-10-06 03:54:45] | (0:00:00.658799) | |
| 2023-10-06 03:54:45,740 MJV2 INFO ====> Epoch: 195 [2023-10-06 03:54:45] | (0:00:00.637691) | |
| 2023-10-06 03:54:46,384 MJV2 INFO ====> Epoch: 196 [2023-10-06 03:54:46] | (0:00:00.639791) | |
| 2023-10-06 03:54:46,994 MJV2 INFO ====> Epoch: 197 [2023-10-06 03:54:46] | (0:00:00.604131) | |
| 2023-10-06 03:54:47,595 MJV2 INFO ====> Epoch: 198 [2023-10-06 03:54:47] | (0:00:00.593502) | |
| 2023-10-06 03:54:48,205 MJV2 INFO ====> Epoch: 199 [2023-10-06 03:54:48] | (0:00:00.602847) | |
| 2023-10-06 03:54:48,818 MJV2 INFO Saving model and optimizer state at epoch 200 to ./logs/MJV2/G_2333333.pth | |
| 2023-10-06 03:54:54,252 MJV2 INFO Saving model and optimizer state at epoch 200 to ./logs/MJV2/D_2333333.pth | |
| 2023-10-06 03:55:15,566 MJV2 INFO saving ckpt MJV2_e200:Success. | |
| 2023-10-06 03:55:15,570 MJV2 INFO ====> Epoch: 200 [2023-10-06 03:55:15] | (0:00:27.357656) | |
| 2023-10-06 03:55:16,646 MJV2 INFO Train Epoch: 201 [0%] | |
| 2023-10-06 03:55:16,647 MJV2 INFO [200, 9.753083879807726e-05] | |
| 2023-10-06 03:55:16,647 MJV2 INFO loss_disc=5.307, loss_gen=2.022, loss_fm=1.850,loss_mel=2.464, loss_kl=-0.004 | |
| 2023-10-06 03:55:18,285 MJV2 INFO ====> Epoch: 201 [2023-10-06 03:55:18] | (0:00:02.702535) | |
| 2023-10-06 03:55:18,945 MJV2 INFO ====> Epoch: 202 [2023-10-06 03:55:18] | (0:00:00.643940) | |
| 2023-10-06 03:55:19,496 MJV2 INFO ====> Epoch: 203 [2023-10-06 03:55:19] | (0:00:00.546200) | |
| 2023-10-06 03:55:20,044 MJV2 INFO ====> Epoch: 204 [2023-10-06 03:55:20] | (0:00:00.543616) | |
| 2023-10-06 03:55:20,594 MJV2 INFO ====> Epoch: 205 [2023-10-06 03:55:20] | (0:00:00.544458) | |
| 2023-10-06 03:55:21,144 MJV2 INFO ====> Epoch: 206 [2023-10-06 03:55:21] | (0:00:00.545627) | |
| 2023-10-06 03:55:21,690 MJV2 INFO ====> Epoch: 207 [2023-10-06 03:55:21] | (0:00:00.540758) | |
| 2023-10-06 03:55:22,230 MJV2 INFO ====> Epoch: 208 [2023-10-06 03:55:22] | (0:00:00.536133) | |
| 2023-10-06 03:55:22,776 MJV2 INFO ====> Epoch: 209 [2023-10-06 03:55:22] | (0:00:00.540699) | |
| 2023-10-06 03:55:23,316 MJV2 INFO ====> Epoch: 210 [2023-10-06 03:55:23] | (0:00:00.535896) | |
| 2023-10-06 03:55:23,873 MJV2 INFO ====> Epoch: 211 [2023-10-06 03:55:23] | (0:00:00.551941) | |
| 2023-10-06 03:55:24,489 MJV2 INFO ====> Epoch: 212 [2023-10-06 03:55:24] | (0:00:00.611887) | |
| 2023-10-06 03:55:25,132 MJV2 INFO ====> Epoch: 213 [2023-10-06 03:55:25] | (0:00:00.635606) | |
| 2023-10-06 03:55:25,789 MJV2 INFO ====> Epoch: 214 [2023-10-06 03:55:25] | (0:00:00.649869) | |
| 2023-10-06 03:55:26,462 MJV2 INFO ====> Epoch: 215 [2023-10-06 03:55:26] | (0:00:00.666291) | |
| 2023-10-06 03:55:27,084 MJV2 INFO ====> Epoch: 216 [2023-10-06 03:55:27] | (0:00:00.614961) | |
| 2023-10-06 03:55:27,680 MJV2 INFO ====> Epoch: 217 [2023-10-06 03:55:27] | (0:00:00.590715) | |
| 2023-10-06 03:55:28,297 MJV2 INFO ====> Epoch: 218 [2023-10-06 03:55:28] | (0:00:00.609961) | |
| 2023-10-06 03:55:29,076 MJV2 INFO ====> Epoch: 219 [2023-10-06 03:55:29] | (0:00:00.774164) | |
| 2023-10-06 03:55:29,904 MJV2 INFO Saving model and optimizer state at epoch 220 to ./logs/MJV2/G_2333333.pth | |
| 2023-10-06 03:55:40,216 MJV2 INFO Saving model and optimizer state at epoch 220 to ./logs/MJV2/D_2333333.pth | |
| 2023-10-06 03:55:50,822 MJV2 INFO saving ckpt MJV2_e220:Success. | |
| 2023-10-06 03:55:50,825 MJV2 INFO ====> Epoch: 220 [2023-10-06 03:55:50] | (0:00:21.740222) | |
| 2023-10-06 03:55:51,594 MJV2 INFO ====> Epoch: 221 [2023-10-06 03:55:51] | (0:00:00.755274) | |
| 2023-10-06 03:55:52,252 MJV2 INFO ====> Epoch: 222 [2023-10-06 03:55:52] | (0:00:00.650103) | |
| 2023-10-06 03:55:52,873 MJV2 INFO ====> Epoch: 223 [2023-10-06 03:55:52] | (0:00:00.613572) | |
| 2023-10-06 03:55:53,508 MJV2 INFO ====> Epoch: 224 [2023-10-06 03:55:53] | (0:00:00.627875) | |
| 2023-10-06 03:55:54,112 MJV2 INFO ====> Epoch: 225 [2023-10-06 03:55:54] | (0:00:00.599724) | |
| 2023-10-06 03:55:54,714 MJV2 INFO ====> Epoch: 226 [2023-10-06 03:55:54] | (0:00:00.593971) | |
| 2023-10-06 03:55:55,264 MJV2 INFO ====> Epoch: 227 [2023-10-06 03:55:55] | (0:00:00.545647) | |
| 2023-10-06 03:55:55,807 MJV2 INFO ====> Epoch: 228 [2023-10-06 03:55:55] | (0:00:00.538006) | |
| 2023-10-06 03:55:56,435 MJV2 INFO ====> Epoch: 229 [2023-10-06 03:55:56] | (0:00:00.622771) | |
| 2023-10-06 03:55:57,066 MJV2 INFO ====> Epoch: 230 [2023-10-06 03:55:57] | (0:00:00.624770) | |
| 2023-10-06 03:55:57,756 MJV2 INFO ====> Epoch: 231 [2023-10-06 03:55:57] | (0:00:00.683317) | |
| 2023-10-06 03:55:58,446 MJV2 INFO ====> Epoch: 232 [2023-10-06 03:55:58] | (0:00:00.683250) | |
| 2023-10-06 03:55:59,151 MJV2 INFO ====> Epoch: 233 [2023-10-06 03:55:59] | (0:00:00.695991) | |
| 2023-10-06 03:56:00,160 MJV2 INFO ====> Epoch: 234 [2023-10-06 03:56:00] | (0:00:00.996514) | |
| 2023-10-06 03:56:00,985 MJV2 INFO ====> Epoch: 235 [2023-10-06 03:56:00] | (0:00:00.816494) | |
| 2023-10-06 03:56:02,025 MJV2 INFO ====> Epoch: 236 [2023-10-06 03:56:02] | (0:00:01.028410) | |
| 2023-10-06 03:56:03,145 MJV2 INFO ====> Epoch: 237 [2023-10-06 03:56:03] | (0:00:01.106992) | |
| 2023-10-06 03:56:04,001 MJV2 INFO ====> Epoch: 238 [2023-10-06 03:56:04] | (0:00:00.835461) | |
| 2023-10-06 03:56:04,674 MJV2 INFO ====> Epoch: 239 [2023-10-06 03:56:04] | (0:00:00.657481) | |
| 2023-10-06 03:56:05,306 MJV2 INFO Saving model and optimizer state at epoch 240 to ./logs/MJV2/G_2333333.pth | |
| 2023-10-06 03:56:15,733 MJV2 INFO Saving model and optimizer state at epoch 240 to ./logs/MJV2/D_2333333.pth | |
| 2023-10-06 03:56:25,590 MJV2 INFO saving ckpt MJV2_e240:Success. | |
| 2023-10-06 03:56:25,591 MJV2 INFO ====> Epoch: 240 [2023-10-06 03:56:25] | (0:00:20.909496) | |
| 2023-10-06 03:56:26,332 MJV2 INFO ====> Epoch: 241 [2023-10-06 03:56:26] | (0:00:00.729054) | |
| 2023-10-06 03:56:26,965 MJV2 INFO ====> Epoch: 242 [2023-10-06 03:56:26] | (0:00:00.626688) | |
| 2023-10-06 03:56:27,585 MJV2 INFO ====> Epoch: 243 [2023-10-06 03:56:27] | (0:00:00.613657) | |
| 2023-10-06 03:56:28,228 MJV2 INFO ====> Epoch: 244 [2023-10-06 03:56:28] | (0:00:00.638880) | |
| 2023-10-06 03:56:28,857 MJV2 INFO ====> Epoch: 245 [2023-10-06 03:56:28] | (0:00:00.621633) | |
| 2023-10-06 03:56:29,480 MJV2 INFO ====> Epoch: 246 [2023-10-06 03:56:29] | (0:00:00.616075) | |
| 2023-10-06 03:56:30,111 MJV2 INFO ====> Epoch: 247 [2023-10-06 03:56:30] | (0:00:00.625484) | |
| 2023-10-06 03:56:30,692 MJV2 INFO ====> Epoch: 248 [2023-10-06 03:56:30] | (0:00:00.573028) | |
| 2023-10-06 03:56:31,241 MJV2 INFO ====> Epoch: 249 [2023-10-06 03:56:31] | (0:00:00.544812) | |
| 2023-10-06 03:56:31,841 MJV2 INFO ====> Epoch: 250 [2023-10-06 03:56:31] | (0:00:00.595134) | |
| 2023-10-06 03:56:32,551 MJV2 INFO ====> Epoch: 251 [2023-10-06 03:56:32] | (0:00:00.700838) | |
| 2023-10-06 03:56:33,302 MJV2 INFO ====> Epoch: 252 [2023-10-06 03:56:33] | (0:00:00.744068) | |
| 2023-10-06 03:56:34,225 MJV2 INFO ====> Epoch: 253 [2023-10-06 03:56:34] | (0:00:00.914344) | |
| 2023-10-06 03:56:35,074 MJV2 INFO ====> Epoch: 254 [2023-10-06 03:56:35] | (0:00:00.835004) | |
| 2023-10-06 03:56:35,976 MJV2 INFO ====> Epoch: 255 [2023-10-06 03:56:35] | (0:00:00.886215) | |
| 2023-10-06 03:56:36,874 MJV2 INFO ====> Epoch: 256 [2023-10-06 03:56:36] | (0:00:00.887418) | |
| 2023-10-06 03:56:37,825 MJV2 INFO ====> Epoch: 257 [2023-10-06 03:56:37] | (0:00:00.939730) | |
| 2023-10-06 03:56:38,786 MJV2 INFO ====> Epoch: 258 [2023-10-06 03:56:38] | (0:00:00.942782) | |
| 2023-10-06 03:56:39,669 MJV2 INFO ====> Epoch: 259 [2023-10-06 03:56:39] | (0:00:00.875538) | |
| 2023-10-06 03:56:40,611 MJV2 INFO Saving model and optimizer state at epoch 260 to ./logs/MJV2/G_2333333.pth | |
| 2023-10-06 03:56:42,480 MJV2 INFO Saving model and optimizer state at epoch 260 to ./logs/MJV2/D_2333333.pth | |
| 2023-10-06 03:56:57,918 MJV2 INFO saving ckpt MJV2_e260:Success. | |
| 2023-10-06 03:56:57,919 MJV2 INFO ====> Epoch: 260 [2023-10-06 03:56:57] | (0:00:18.239052) | |
| 2023-10-06 03:56:58,627 MJV2 INFO ====> Epoch: 261 [2023-10-06 03:56:58] | (0:00:00.691500) | |
| 2023-10-06 03:56:59,222 MJV2 INFO ====> Epoch: 262 [2023-10-06 03:56:59] | (0:00:00.587209) | |
| 2023-10-06 03:56:59,819 MJV2 INFO ====> Epoch: 263 [2023-10-06 03:56:59] | (0:00:00.591587) | |
| 2023-10-06 03:57:00,400 MJV2 INFO ====> Epoch: 264 [2023-10-06 03:57:00] | (0:00:00.573412) | |
| 2023-10-06 03:57:00,949 MJV2 INFO ====> Epoch: 265 [2023-10-06 03:57:00] | (0:00:00.543658) | |
| 2023-10-06 03:57:01,498 MJV2 INFO ====> Epoch: 266 [2023-10-06 03:57:01] | (0:00:00.544362) | |
| 2023-10-06 03:57:02,057 MJV2 INFO ====> Epoch: 267 [2023-10-06 03:57:02] | (0:00:00.554105) | |
| 2023-10-06 03:57:02,607 MJV2 INFO ====> Epoch: 268 [2023-10-06 03:57:02] | (0:00:00.545201) | |
| 2023-10-06 03:57:03,160 MJV2 INFO ====> Epoch: 269 [2023-10-06 03:57:03] | (0:00:00.548308) | |
| 2023-10-06 03:57:03,712 MJV2 INFO ====> Epoch: 270 [2023-10-06 03:57:03] | (0:00:00.547160) | |
| 2023-10-06 03:57:04,264 MJV2 INFO ====> Epoch: 271 [2023-10-06 03:57:04] | (0:00:00.546357) | |
| 2023-10-06 03:57:04,811 MJV2 INFO ====> Epoch: 272 [2023-10-06 03:57:04] | (0:00:00.542045) | |
| 2023-10-06 03:57:05,356 MJV2 INFO ====> Epoch: 273 [2023-10-06 03:57:05] | (0:00:00.540888) | |
| 2023-10-06 03:57:05,905 MJV2 INFO ====> Epoch: 274 [2023-10-06 03:57:05] | (0:00:00.544092) | |
| 2023-10-06 03:57:06,500 MJV2 INFO ====> Epoch: 275 [2023-10-06 03:57:06] | (0:00:00.590528) | |
| 2023-10-06 03:57:07,145 MJV2 INFO ====> Epoch: 276 [2023-10-06 03:57:07] | (0:00:00.637404) | |
| 2023-10-06 03:57:08,002 MJV2 INFO ====> Epoch: 277 [2023-10-06 03:57:08] | (0:00:00.848961) | |
| 2023-10-06 03:57:08,943 MJV2 INFO ====> Epoch: 278 [2023-10-06 03:57:08] | (0:00:00.921772) | |
| 2023-10-06 03:57:09,899 MJV2 INFO ====> Epoch: 279 [2023-10-06 03:57:09] | (0:00:00.942280) | |
| 2023-10-06 03:57:10,780 MJV2 INFO Saving model and optimizer state at epoch 280 to ./logs/MJV2/G_2333333.pth | |
| 2023-10-06 03:57:17,977 MJV2 INFO Saving model and optimizer state at epoch 280 to ./logs/MJV2/D_2333333.pth | |
| 2023-10-06 03:57:33,219 MJV2 INFO saving ckpt MJV2_e280:Success. | |
| 2023-10-06 03:57:33,221 MJV2 INFO ====> Epoch: 280 [2023-10-06 03:57:33] | (0:00:23.314079) | |
| 2023-10-06 03:57:33,921 MJV2 INFO ====> Epoch: 281 [2023-10-06 03:57:33] | (0:00:00.688499) | |
| 2023-10-06 03:57:34,534 MJV2 INFO ====> Epoch: 282 [2023-10-06 03:57:34] | (0:00:00.602221) | |
| 2023-10-06 03:57:35,168 MJV2 INFO ====> Epoch: 283 [2023-10-06 03:57:35] | (0:00:00.628034) | |
| 2023-10-06 03:57:35,796 MJV2 INFO ====> Epoch: 284 [2023-10-06 03:57:35] | (0:00:00.620763) | |
| 2023-10-06 03:57:36,394 MJV2 INFO ====> Epoch: 285 [2023-10-06 03:57:36] | (0:00:00.591844) | |
| 2023-10-06 03:57:36,943 MJV2 INFO ====> Epoch: 286 [2023-10-06 03:57:36] | (0:00:00.544362) | |
| 2023-10-06 03:57:37,489 MJV2 INFO ====> Epoch: 287 [2023-10-06 03:57:37] | (0:00:00.541447) | |
| 2023-10-06 03:57:38,049 MJV2 INFO ====> Epoch: 288 [2023-10-06 03:57:38] | (0:00:00.555393) | |
| 2023-10-06 03:57:38,654 MJV2 INFO ====> Epoch: 289 [2023-10-06 03:57:38] | (0:00:00.599946) | |
| 2023-10-06 03:57:39,299 MJV2 INFO ====> Epoch: 290 [2023-10-06 03:57:39] | (0:00:00.636895) | |
| 2023-10-06 03:57:39,926 MJV2 INFO ====> Epoch: 291 [2023-10-06 03:57:39] | (0:00:00.619722) | |
| 2023-10-06 03:57:40,549 MJV2 INFO ====> Epoch: 292 [2023-10-06 03:57:40] | (0:00:00.615366) | |
| 2023-10-06 03:57:41,203 MJV2 INFO ====> Epoch: 293 [2023-10-06 03:57:41] | (0:00:00.647124) | |
| 2023-10-06 03:57:41,829 MJV2 INFO ====> Epoch: 294 [2023-10-06 03:57:41] | (0:00:00.617758) | |
| 2023-10-06 03:57:42,548 MJV2 INFO ====> Epoch: 295 [2023-10-06 03:57:42] | (0:00:00.711541) | |
| 2023-10-06 03:57:43,594 MJV2 INFO ====> Epoch: 296 [2023-10-06 03:57:43] | (0:00:01.035691) | |
| 2023-10-06 03:57:44,518 MJV2 INFO ====> Epoch: 297 [2023-10-06 03:57:44] | (0:00:00.916965) | |
| 2023-10-06 03:57:45,387 MJV2 INFO ====> Epoch: 298 [2023-10-06 03:57:45] | (0:00:00.860600) | |
| 2023-10-06 03:57:46,124 MJV2 INFO ====> Epoch: 299 [2023-10-06 03:57:46] | (0:00:00.728063) | |
| 2023-10-06 03:57:46,744 MJV2 INFO Saving model and optimizer state at epoch 300 to ./logs/MJV2/G_2333333.pth | |
| 2023-10-06 03:57:50,985 MJV2 INFO Saving model and optimizer state at epoch 300 to ./logs/MJV2/D_2333333.pth | |
| 2023-10-06 03:58:03,341 MJV2 INFO saving ckpt MJV2_e300:Success. | |
| 2023-10-06 03:58:03,343 MJV2 INFO ====> Epoch: 300 [2023-10-06 03:58:03] | (0:00:17.211895) | |
| 2023-10-06 03:58:03,344 MJV2 INFO Training is done. The program is closed. | |
| 2023-10-06 03:58:07,679 MJV2 INFO saving final ckpt:Success. | |