/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated and will be removed in future. Use torchrun. Note that --use_env is set by default in torchrun. If your script expects `--local_rank` argument to be set, please change it to read from `os.environ['LOCAL_RANK']` instead. See https://pytorch.org/docs/stable/distributed.html#launch-utility for further instructions warnings.warn( WARNING:torch.distributed.run: ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. ***************************************** /home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/bin/python: can't open file 'wav2vec2_fine_tune.py': [Errno 2] No such file or directory /home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/bin/python: can't open file 'wav2vec2_fine_tune.py': [Errno 2] No such file or directory ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 2) local_rank: 0 (pid: 7680) of binary: /home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/bin/python Traceback (most recent call last): File "/home/itai/.pyenv/versions/3.8.10/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/itai/.pyenv/versions/3.8.10/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in main() File "/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main launch(args) File "/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch run(args) File "/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/run.py", line 715, in run elastic_launch( File "/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ wav2vec2_fine_tune.py FAILED ------------------------------------------------------------ Failures: [1]: time : 2022-04-13_09:16:59 host : itai-gpu-dev1.internal.cloudapp.net rank : 1 (local_rank: 1) exitcode : 2 (pid: 7681) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2022-04-13_09:16:59 host : itai-gpu-dev1.internal.cloudapp.net rank : 0 (local_rank: 0) exitcode : 2 (pid: 7680) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================