Training in progress, step 10000
Browse files- nohup.out +55 -2
- preprocessor_config.json +0 -1
- pytorch_model.bin +1 -1
- runs/Apr13_09-02-30_itai-gpu-dev1/1649840569.2990751/events.out.tfevents.1649840569.itai-gpu-dev1.5010.1 +3 -0
- runs/Apr13_09-02-30_itai-gpu-dev1/events.out.tfevents.1649840569.itai-gpu-dev1.5010.0 +3 -0
- runs/Apr13_09-17-31_itai-gpu-dev1/1649841458.8038676/events.out.tfevents.1649841458.itai-gpu-dev1.8429.1 +3 -0
- runs/Apr13_09-17-31_itai-gpu-dev1/events.out.tfevents.1649841458.itai-gpu-dev1.8429.0 +3 -0
- runs/Apr13_09-17-58_itai-gpu-dev1/1649841485.831911/events.out.tfevents.1649841485.itai-gpu-dev1.9232.1 +3 -0
- runs/Apr13_09-17-58_itai-gpu-dev1/events.out.tfevents.1649841485.itai-gpu-dev1.9232.0 +3 -0
- runs/Apr13_09-19-19_itai-gpu-dev1/1649841565.8537362/events.out.tfevents.1649841565.itai-gpu-dev1.9858.1 +3 -0
- runs/Apr13_09-19-19_itai-gpu-dev1/events.out.tfevents.1649841565.itai-gpu-dev1.9858.0 +3 -0
- runs/Apr13_09-27-30_itai-gpu-dev1/1649842080.0518692/events.out.tfevents.1649842080.itai-gpu-dev1.2639.1 +3 -0
- runs/Apr13_09-27-30_itai-gpu-dev1/events.out.tfevents.1649842080.itai-gpu-dev1.2639.0 +3 -0
- training_args.bin +1 -1
nohup.out
CHANGED
@@ -1,2 +1,55 @@
|
|
1 |
-
/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/
|
2 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated
|
2 |
+
and will be removed in future. Use torchrun.
|
3 |
+
Note that --use_env is set by default in torchrun.
|
4 |
+
If your script expects `--local_rank` argument to be set, please
|
5 |
+
change it to read from `os.environ['LOCAL_RANK']` instead. See
|
6 |
+
https://pytorch.org/docs/stable/distributed.html#launch-utility for
|
7 |
+
further instructions
|
8 |
+
|
9 |
+
warnings.warn(
|
10 |
+
WARNING:torch.distributed.run:
|
11 |
+
*****************************************
|
12 |
+
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
|
13 |
+
*****************************************
|
14 |
+
/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/bin/python: can't open file 'wav2vec2_fine_tune.py': [Errno 2] No such file or directory
|
15 |
+
/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/bin/python: can't open file 'wav2vec2_fine_tune.py': [Errno 2] No such file or directory
|
16 |
+
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 2) local_rank: 0 (pid: 7680) of binary: /home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/bin/python
|
17 |
+
Traceback (most recent call last):
|
18 |
+
File "/home/itai/.pyenv/versions/3.8.10/lib/python3.8/runpy.py", line 194, in _run_module_as_main
|
19 |
+
return _run_code(code, main_globals, None,
|
20 |
+
File "/home/itai/.pyenv/versions/3.8.10/lib/python3.8/runpy.py", line 87, in _run_code
|
21 |
+
exec(code, run_globals)
|
22 |
+
File "/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in <module>
|
23 |
+
main()
|
24 |
+
File "/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main
|
25 |
+
launch(args)
|
26 |
+
File "/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch
|
27 |
+
run(args)
|
28 |
+
File "/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/run.py", line 715, in run
|
29 |
+
elastic_launch(
|
30 |
+
File "/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
|
31 |
+
return launch_agent(self._config, self._entrypoint, list(args))
|
32 |
+
File "/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent
|
33 |
+
raise ChildFailedError(
|
34 |
+
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
|
35 |
+
============================================================
|
36 |
+
wav2vec2_fine_tune.py FAILED
|
37 |
+
------------------------------------------------------------
|
38 |
+
Failures:
|
39 |
+
[1]:
|
40 |
+
time : 2022-04-13_09:16:59
|
41 |
+
host : itai-gpu-dev1.internal.cloudapp.net
|
42 |
+
rank : 1 (local_rank: 1)
|
43 |
+
exitcode : 2 (pid: 7681)
|
44 |
+
error_file: <N/A>
|
45 |
+
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
|
46 |
+
------------------------------------------------------------
|
47 |
+
Root Cause (first observed failure):
|
48 |
+
[0]:
|
49 |
+
time : 2022-04-13_09:16:59
|
50 |
+
host : itai-gpu-dev1.internal.cloudapp.net
|
51 |
+
rank : 0 (local_rank: 0)
|
52 |
+
exitcode : 2 (pid: 7680)
|
53 |
+
error_file: <N/A>
|
54 |
+
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
|
55 |
+
============================================================
|
preprocessor_config.json
CHANGED
@@ -4,7 +4,6 @@
|
|
4 |
"feature_size": 1,
|
5 |
"padding_side": "right",
|
6 |
"padding_value": 0.0,
|
7 |
-
"processor_class": "Wav2Vec2Processor",
|
8 |
"return_attention_mask": true,
|
9 |
"sampling_rate": 16000
|
10 |
}
|
|
|
4 |
"feature_size": 1,
|
5 |
"padding_side": "right",
|
6 |
"padding_value": 0.0,
|
|
|
7 |
"return_attention_mask": true,
|
8 |
"sampling_rate": 16000
|
9 |
}
|
pytorch_model.bin
CHANGED
@@ -1,3 +1,3 @@
|
|
1 |
version https://git-lfs.github.com/spec/v1
|
2 |
-
oid sha256:
|
3 |
size 1262057521
|
|
|
1 |
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:770c1d2f84b1ffa78e34ae25197bc401a66c2fb8f078496b048083282424a370
|
3 |
size 1262057521
|
runs/Apr13_09-02-30_itai-gpu-dev1/1649840569.2990751/events.out.tfevents.1649840569.itai-gpu-dev1.5010.1
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:2726e98002245a6bcb52f1567640ac4c223518ad17e8d84cd569dff0177d9aa1
|
3 |
+
size 4910
|
runs/Apr13_09-02-30_itai-gpu-dev1/events.out.tfevents.1649840569.itai-gpu-dev1.5010.0
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:8046756f7025881355f21df6318c99f5a67b1c5f84bdd98e0a0aed0e57339322
|
3 |
+
size 4136
|
runs/Apr13_09-17-31_itai-gpu-dev1/1649841458.8038676/events.out.tfevents.1649841458.itai-gpu-dev1.8429.1
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:3abd3804a8ea4a1920c346f6f9843f6df9073b4fb1599d60306a5a14d41de5a5
|
3 |
+
size 4913
|
runs/Apr13_09-17-31_itai-gpu-dev1/events.out.tfevents.1649841458.itai-gpu-dev1.8429.0
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:1ea60e89a0699861c39e331eb75f57668a2d06f414b2d0dd1917c42341485d18
|
3 |
+
size 4136
|
runs/Apr13_09-17-58_itai-gpu-dev1/1649841485.831911/events.out.tfevents.1649841485.itai-gpu-dev1.9232.1
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:820143094cbde875b7976022a75c6b34644788842c5486098155e968f991413d
|
3 |
+
size 4913
|
runs/Apr13_09-17-58_itai-gpu-dev1/events.out.tfevents.1649841485.itai-gpu-dev1.9232.0
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:d51e0ef7553c42d446dec800eeea05f01c6cdbc860b72591b37333ab70d4f51f
|
3 |
+
size 4136
|
runs/Apr13_09-19-19_itai-gpu-dev1/1649841565.8537362/events.out.tfevents.1649841565.itai-gpu-dev1.9858.1
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:afb3a1bcc49bd137ec43f6cce7001653cced1623588f19d4209813bcf8fe07c1
|
3 |
+
size 4913
|
runs/Apr13_09-19-19_itai-gpu-dev1/events.out.tfevents.1649841565.itai-gpu-dev1.9858.0
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:bdaf0f653117020d3b20e158e327f5af33b83a6d49e032027920b8b91390e247
|
3 |
+
size 4915
|
runs/Apr13_09-27-30_itai-gpu-dev1/1649842080.0518692/events.out.tfevents.1649842080.itai-gpu-dev1.2639.1
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:21f54b12b53445c552be6e7b3f38faeb38f9c80c2e7c9e2d7e969d9c1ac8b895
|
3 |
+
size 4913
|
runs/Apr13_09-27-30_itai-gpu-dev1/events.out.tfevents.1649842080.itai-gpu-dev1.2639.0
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:246e0dfcfa014169a5cd6baadf9c6c0792e55faee78f0b73a6f5e1b35525e0d5
|
3 |
+
size 6803
|
training_args.bin
CHANGED
@@ -1,3 +1,3 @@
|
|
1 |
version https://git-lfs.github.com/spec/v1
|
2 |
-
oid sha256:
|
3 |
size 3183
|
|
|
1 |
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:aede962b491f33bb9333cf1284f02ebd7e6a84e0e15922f80fbd6611c106c42c
|
3 |
size 3183
|