Training in progress, step 5500

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7e90080bf523412654da109eb8a0e3ae8edacc7c025143a1ccd52be304e4b1d5
+oid sha256:87f309cbf1a814ca57bc11006cd2a393307279e9dd48b83967fc1fd5bb40d204
 size 1262817457

wandb/run-20220129_131141-h6nhqm30/files/output.log

@@ -5293,3 +5293,534 @@ Configuration saved in ./checkpoint-5000/config.json
 {'eval_loss': 0.3355213701725006, 'eval_wer': 0.4330439988027537, 'eval_runtime': 63.7849, 'eval_samples_per_second': 28.831, 'eval_steps_per_second': 0.455, 'epoch': 1.16}
 Model weights saved in ./checkpoint-5000/pytorch_model.bin
 Configuration saved in ./checkpoint-5000/preprocessor_config.json
+Configuration saved in ./preprocessor_config.json
+Deleting older checkpoint [checkpoint-3500] due to args.save_total_limit
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 24%|████████████████████████████████████▉                                                                                                                       | 5100/21520 [4:12:48<8:56:31,  1.96s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 24%|█████████████████████████████████████▋                                                                                                                      | 5200/21520 [4:17:12<9:28:46,  2.09s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 25%|██████████████████████████████████████▍                                                                                                                     | 5299/21520 [4:21:32<8:51:36,  1.97s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 25%|███████████████████████████████████████▏                                                                                                                    | 5399/21520 [4:25:55<9:07:16,  2.04s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 26%|███████████████████████████████████████▊                                                                                                                    | 5500/21520 [4:30:20<8:58:27,  2.02s/it]The following columns in the evaluation set  don't have a corresponding argument in `Wav2Vec2ForCTC.forward` and have been ignored: input_length.
+***** Running Evaluation *****
+  Num examples = 1839
+  Batch size = 64
+{'loss': 1.1568, 'learning_rate': 5.6041958041958036e-05, 'epoch': 1.28}
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Configuration saved in ./checkpoint-5500/config.json
+{'eval_loss': 0.327897846698761, 'eval_wer': 0.42490272373540855, 'eval_runtime': 64.3828, 'eval_samples_per_second': 28.564, 'eval_steps_per_second': 0.45, 'epoch': 1.28}
+Model weights saved in ./checkpoint-5500/pytorch_model.bin
+Configuration saved in ./checkpoint-5500/preprocessor_config.json

wandb/run-20220129_131141-h6nhqm30/run-h6nhqm30.wandb

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:53a09fb2a08f71ee4b89bfa0dfaf906dc6e5a01b9a4e4ab7901f827b7817f498
-size 35019521
+oid sha256:1b912162042956cbe2e98aeca3bbb927a8b0bf94d2a3c9c5a23f2080c45117fe
+size 38830210