| nohup: ignoring input |
| W1030 19:02:35.769000 8870 site-packages/torch/distributed/run.py:793] |
| W1030 19:02:35.769000 8870 site-packages/torch/distributed/run.py:793] ***************************************** |
| W1030 19:02:35.769000 8870 site-packages/torch/distributed/run.py:793] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. |
| W1030 19:02:35.769000 8870 site-packages/torch/distributed/run.py:793] ***************************************** |
| /root/miniconda3/envs/meditron/bin/python: can't open file '/mnt/dg_tunning/lora4_meditron_7b/tune_combined.py': [Errno 2] No such file or directory |
| /root/miniconda3/envs/meditron/bin/python: can't open file '/mnt/dg_tunning/lora4_meditron_7b/tune_combined.py': [Errno 2] No such file or directory |
| /root/miniconda3/envs/meditron/bin/python: can't open file '/mnt/dg_tunning/lora4_meditron_7b/tune_combined.py': [Errno 2] No such file or directory |
| /root/miniconda3/envs/meditron/bin/python: can't open file '/mnt/dg_tunning/lora4_meditron_7b/tune_combined.py': [Errno 2] No such file or directory |
| /root/miniconda3/envs/meditron/bin/python: can't open file '/mnt/dg_tunning/lora4_meditron_7b/tune_combined.py': [Errno 2] No such file or directory |
| /root/miniconda3/envs/meditron/bin/python: can't open file '/mnt/dg_tunning/lora4_meditron_7b/tune_combined.py': [Errno 2] No such file or directory |
| /root/miniconda3/envs/meditron/bin/python: can't open file '/mnt/dg_tunning/lora4_meditron_7b/tune_combined.py': [Errno 2] No such file or directory |
| E1030 19:02:35.887000 8870 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 2) local_rank: 0 (pid: 8935) of binary: /root/miniconda3/envs/meditron/bin/python |
| Traceback (most recent call last): |
| File "/root/miniconda3/envs/meditron/bin/torchrun", line 8, in <module> |
| sys.exit(main()) |
| File "/root/miniconda3/envs/meditron/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper |
| return f(*args, **kwargs) |
| File "/root/miniconda3/envs/meditron/lib/python3.10/site-packages/torch/distributed/run.py", line 919, in main |
| run(args) |
| File "/root/miniconda3/envs/meditron/lib/python3.10/site-packages/torch/distributed/run.py", line 910, in run |
| elastic_launch( |
| File "/root/miniconda3/envs/meditron/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 138, in __call__ |
| return launch_agent(self._config, self._entrypoint, list(args)) |
| File "/root/miniconda3/envs/meditron/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent |
| raise ChildFailedError( |
| torch.distributed.elastic.multiprocessing.errors.ChildFailedError: |
| ============================================================ |
| tune_combined.py FAILED |
| ------------------------------------------------------------ |
| Failures: |
| [1]: |
| time : 2024-10-30_19:02:35 |
| host : 3f2e085cdcba |
| rank : 1 (local_rank: 1) |
| exitcode : 2 (pid: 8936) |
| error_file: <N/A> |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html |
| [2]: |
| time : 2024-10-30_19:02:35 |
| host : 3f2e085cdcba |
| rank : 2 (local_rank: 2) |
| exitcode : 2 (pid: 8937) |
| error_file: <N/A> |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html |
| [3]: |
| time : 2024-10-30_19:02:35 |
| host : 3f2e085cdcba |
| rank : 3 (local_rank: 3) |
| exitcode : 2 (pid: 8938) |
| error_file: <N/A> |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html |
| [4]: |
| time : 2024-10-30_19:02:35 |
| host : 3f2e085cdcba |
| rank : 4 (local_rank: 4) |
| exitcode : 2 (pid: 8939) |
| error_file: <N/A> |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html |
| [5]: |
| time : 2024-10-30_19:02:35 |
| host : 3f2e085cdcba |
| rank : 5 (local_rank: 5) |
| exitcode : 2 (pid: 8940) |
| error_file: <N/A> |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html |
| [6]: |
| time : 2024-10-30_19:02:35 |
| host : 3f2e085cdcba |
| rank : 6 (local_rank: 6) |
| exitcode : 2 (pid: 8941) |
| error_file: <N/A> |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html |
| ------------------------------------------------------------ |
| Root Cause (first observed failure): |
| [0]: |
| time : 2024-10-30_19:02:35 |
| host : 3f2e085cdcba |
| rank : 0 (local_rank: 0) |
| exitcode : 2 (pid: 8935) |
| error_file: <N/A> |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html |
| ============================================================ |
| |
