itaihay commited on
Commit
dc3156c
·
1 Parent(s): 28430a0

Training in progress, step 10000

Browse files
nohup.out CHANGED
@@ -1,2 +1,55 @@
1
- /home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/bin/python3: can't open file 'fine_tune.py': [Errno 2] No such file or directory
2
- /home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/bin/python3: can't open file 'wav2vec2_fine_tune.py': [Errno 2] No such file or directory
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ /home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated
2
+ and will be removed in future. Use torchrun.
3
+ Note that --use_env is set by default in torchrun.
4
+ If your script expects `--local_rank` argument to be set, please
5
+ change it to read from `os.environ['LOCAL_RANK']` instead. See
6
+ https://pytorch.org/docs/stable/distributed.html#launch-utility for
7
+ further instructions
8
+
9
+ warnings.warn(
10
+ WARNING:torch.distributed.run:
11
+ *****************************************
12
+ Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
13
+ *****************************************
14
+ /home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/bin/python: can't open file 'wav2vec2_fine_tune.py': [Errno 2] No such file or directory
15
+ /home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/bin/python: can't open file 'wav2vec2_fine_tune.py': [Errno 2] No such file or directory
16
+ ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 2) local_rank: 0 (pid: 7680) of binary: /home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/bin/python
17
+ Traceback (most recent call last):
18
+ File "/home/itai/.pyenv/versions/3.8.10/lib/python3.8/runpy.py", line 194, in _run_module_as_main
19
+ return _run_code(code, main_globals, None,
20
+ File "/home/itai/.pyenv/versions/3.8.10/lib/python3.8/runpy.py", line 87, in _run_code
21
+ exec(code, run_globals)
22
+ File "/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in <module>
23
+ main()
24
+ File "/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main
25
+ launch(args)
26
+ File "/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch
27
+ run(args)
28
+ File "/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/run.py", line 715, in run
29
+ elastic_launch(
30
+ File "/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
31
+ return launch_agent(self._config, self._entrypoint, list(args))
32
+ File "/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent
33
+ raise ChildFailedError(
34
+ torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
35
+ ============================================================
36
+ wav2vec2_fine_tune.py FAILED
37
+ ------------------------------------------------------------
38
+ Failures:
39
+ [1]:
40
+ time : 2022-04-13_09:16:59
41
+ host : itai-gpu-dev1.internal.cloudapp.net
42
+ rank : 1 (local_rank: 1)
43
+ exitcode : 2 (pid: 7681)
44
+ error_file: <N/A>
45
+ traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
46
+ ------------------------------------------------------------
47
+ Root Cause (first observed failure):
48
+ [0]:
49
+ time : 2022-04-13_09:16:59
50
+ host : itai-gpu-dev1.internal.cloudapp.net
51
+ rank : 0 (local_rank: 0)
52
+ exitcode : 2 (pid: 7680)
53
+ error_file: <N/A>
54
+ traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
55
+ ============================================================
preprocessor_config.json CHANGED
@@ -4,7 +4,6 @@
4
  "feature_size": 1,
5
  "padding_side": "right",
6
  "padding_value": 0.0,
7
- "processor_class": "Wav2Vec2Processor",
8
  "return_attention_mask": true,
9
  "sampling_rate": 16000
10
  }
 
4
  "feature_size": 1,
5
  "padding_side": "right",
6
  "padding_value": 0.0,
 
7
  "return_attention_mask": true,
8
  "sampling_rate": 16000
9
  }
pytorch_model.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:b1b24b486a4f5bef6aa904512277fcd48021483717c9339e56aaf8dab50468c8
3
  size 1262057521
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:770c1d2f84b1ffa78e34ae25197bc401a66c2fb8f078496b048083282424a370
3
  size 1262057521
runs/Apr13_09-02-30_itai-gpu-dev1/1649840569.2990751/events.out.tfevents.1649840569.itai-gpu-dev1.5010.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2726e98002245a6bcb52f1567640ac4c223518ad17e8d84cd569dff0177d9aa1
3
+ size 4910
runs/Apr13_09-02-30_itai-gpu-dev1/events.out.tfevents.1649840569.itai-gpu-dev1.5010.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8046756f7025881355f21df6318c99f5a67b1c5f84bdd98e0a0aed0e57339322
3
+ size 4136
runs/Apr13_09-17-31_itai-gpu-dev1/1649841458.8038676/events.out.tfevents.1649841458.itai-gpu-dev1.8429.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3abd3804a8ea4a1920c346f6f9843f6df9073b4fb1599d60306a5a14d41de5a5
3
+ size 4913
runs/Apr13_09-17-31_itai-gpu-dev1/events.out.tfevents.1649841458.itai-gpu-dev1.8429.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1ea60e89a0699861c39e331eb75f57668a2d06f414b2d0dd1917c42341485d18
3
+ size 4136
runs/Apr13_09-17-58_itai-gpu-dev1/1649841485.831911/events.out.tfevents.1649841485.itai-gpu-dev1.9232.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:820143094cbde875b7976022a75c6b34644788842c5486098155e968f991413d
3
+ size 4913
runs/Apr13_09-17-58_itai-gpu-dev1/events.out.tfevents.1649841485.itai-gpu-dev1.9232.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d51e0ef7553c42d446dec800eeea05f01c6cdbc860b72591b37333ab70d4f51f
3
+ size 4136
runs/Apr13_09-19-19_itai-gpu-dev1/1649841565.8537362/events.out.tfevents.1649841565.itai-gpu-dev1.9858.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:afb3a1bcc49bd137ec43f6cce7001653cced1623588f19d4209813bcf8fe07c1
3
+ size 4913
runs/Apr13_09-19-19_itai-gpu-dev1/events.out.tfevents.1649841565.itai-gpu-dev1.9858.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bdaf0f653117020d3b20e158e327f5af33b83a6d49e032027920b8b91390e247
3
+ size 4915
runs/Apr13_09-27-30_itai-gpu-dev1/1649842080.0518692/events.out.tfevents.1649842080.itai-gpu-dev1.2639.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:21f54b12b53445c552be6e7b3f38faeb38f9c80c2e7c9e2d7e969d9c1ac8b895
3
+ size 4913
runs/Apr13_09-27-30_itai-gpu-dev1/events.out.tfevents.1649842080.itai-gpu-dev1.2639.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:246e0dfcfa014169a5cd6baadf9c6c0792e55faee78f0b73a6f5e1b35525e0d5
3
+ size 6803
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:926103311e0e00908b8d9742cdd1df68221d08ed4c7ea38fa5aa21654f3eae41
3
  size 3183
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:aede962b491f33bb9333cf1284f02ebd7e6a84e0e15922f80fbd6611c106c42c
3
  size 3183