Training in progress, step 5000

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3f3b35216dc11b2d8eb1144ad6b8b6c2bbdb57c5f263916fe79dc60e3bf481c4
+oid sha256:c4546cfbb989fdb0398aab4dbb04fd96901a18da23c866e881a7828ba50c551a
 size 1262817457

wandb/run-20220129_215451-1vipdbow/files/output.log

@@ -5337,3 +5337,596 @@ Configuration saved in ./checkpoint-4500/config.json
 {'eval_loss': 0.30754634737968445, 'eval_wer': 0.41408973899442797, 'eval_runtime': 296.2378, 'eval_samples_per_second': 19.552, 'eval_steps_per_second': 0.307, 'epoch': 0.65}
 Model weights saved in ./checkpoint-4500/pytorch_model.bin
 Configuration saved in ./checkpoint-4500/preprocessor_config.json
+Configuration saved in ./preprocessor_config.json
+Deleting older checkpoint [checkpoint-3500] due to args.save_total_limit
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 33%|███████████████████████████████████████████████████▉                                                                                                        | 4600/13822 [4:58:39<4:09:13,  1.62s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 34%|█████████████████████████████████████████████████████                                                                                                       | 4699/13822 [5:03:50<4:20:56,  1.72s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 35%|██████████████████████████████████████████████████████▏                                                                                                     | 4799/13822 [5:09:04<4:15:11,  1.70s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 35%|███████████████████████████████████████████████████████▎                                                                                                    | 4900/13822 [5:14:21<4:07:48,  1.67s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 36%|████████████████████████████████████████████████████████▍                                                                                                   | 4998/13822 [5:19:31<4:25:34,  1.81s/it]
+ 36%|████████████████████████████████████████████████████████▍                                                                                                   | 5000/13822 [5:19:34<4:00:23,  1.63s/it]The following columns in the evaluation set  don't have a corresponding argument in `Wav2Vec2ForCTC.forward` and have been ignored: input_length.
+***** Running Evaluation *****
+  Num examples = 5792
+  Batch size = 64
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 91/91 [04:07<00:00,  2.35s/it]
+
+Configuration saved in ./checkpoint-5000/config.json
+Model weights saved in ./checkpoint-5000/pytorch_model.bin
+Configuration saved in ./checkpoint-5000/preprocessor_config.json

wandb/run-20220129_215451-1vipdbow/run-1vipdbow.wandb

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3f46b515385f03d285516ebc3155783c7908614a53b79847fa8608c1e96b83bc
-size 31621187
+oid sha256:56b0ae02ddd23e1dc8dce84b5897394035ec24e854f73a9e502d7010aafc6b4d
+size 35389946