Training in progress, step 4000

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c532ed3d8b8f60fb357e05510cdf806743712b41a62c2e9cdbbe550d425b38c3
+oid sha256:46bfc78745340fef27b401f3b8a5de1f05a7ca7fc2b06227789acc71b8f732ce
 size 1262817457

wandb/run-20220129_215451-1vipdbow/files/output.log

@@ -4153,3 +4153,595 @@ Configuration saved in ./checkpoint-3500/config.json
 {'eval_loss': 0.333638072013855, 'eval_wer': 0.42922545537489004, 'eval_runtime': 297.0402, 'eval_samples_per_second': 19.499, 'eval_steps_per_second': 0.306, 'epoch': 0.51}
 Model weights saved in ./checkpoint-3500/pytorch_model.bin
 Configuration saved in ./checkpoint-3500/preprocessor_config.json
+Configuration saved in ./preprocessor_config.json
+Deleting older checkpoint [checkpoint-2500] due to args.save_total_limit
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 26%|████████████████████████████████████████▋                                                                                                                   | 3600/13822 [3:53:42<4:45:51,  1.68s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 27%|█████████████████████████████████████████▋                                                                                                                  | 3699/13822 [3:58:53<4:48:54,  1.71s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 27%|██████████████████████████████████████████▉                                                                                                                 | 3799/13822 [4:04:04<4:48:05,  1.72s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 28%|████████████████████████████████████████████                                                                                                                | 3899/13822 [4:09:16<4:48:23,  1.74s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 29%|█████████████████████████████████████████████▏                                                                                                              | 4000/13822 [4:14:32<4:22:56,  1.61s/it]The following columns in the evaluation set  don't have a corresponding argument in `Wav2Vec2ForCTC.forward` and have been ignored: input_length.
+***** Running Evaluation *****
+  Num examples = 5792
+  Batch size = 64
+{'loss': 1.0968, 'learning_rate': 7.975166369095926e-05, 'epoch': 0.58}
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Configuration saved in ./checkpoint-4000/config.json
+{'eval_loss': 0.31949570775032043, 'eval_wer': 0.4180162273127179, 'eval_runtime': 296.0172, 'eval_samples_per_second': 19.566, 'eval_steps_per_second': 0.307, 'epoch': 0.58}
+Model weights saved in ./checkpoint-4000/pytorch_model.bin
+Configuration saved in ./checkpoint-4000/preprocessor_config.json

wandb/run-20220129_215451-1vipdbow/run-1vipdbow.wandb

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:09d895df04f8a49a047748b42a92396c0ec2b9399014e75370eaa65fe6e8fe9e
-size 24317915
+oid sha256:ca2883907183a25ab58f3c8968be35166274236f2b82afc34594b95c94230448
+size 27912985