[2023-12-11 05:30:52,558] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-12-11 05:30:54,351] [WARNING] [runner.py:203:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only. Detected CUDA_VISIBLE_DEVICES=1,2,3: setting --include=localhost:1,2,3 Traceback (most recent call last): File "/home/t-sokumar/miniconda3/envs/ft/bin/deepspeed", line 6, in main() File "/home/t-sokumar/miniconda3/envs/ft/lib/python3.11/site-packages/deepspeed/launcher/runner.py", line 430, in main active_resources = parse_inclusion_exclusion(resource_pool, args.include, args.exclude) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/t-sokumar/miniconda3/envs/ft/lib/python3.11/site-packages/deepspeed/launcher/runner.py", line 351, in parse_inclusion_exclusion return parse_resource_filter(active_resources, include_str=inclusion, exclude_str=exclusion) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/t-sokumar/miniconda3/envs/ft/lib/python3.11/site-packages/deepspeed/launcher/runner.py", line 303, in parse_resource_filter raise ValueError(f"No slot '{slot}' specified on host '{hostname}'") ValueError: No slot '3' specified on host 'localhost'