script_1/logs/22088281_2.err

Setting Data Parallel size to 8
Tensor parallelism has not been tested for a while, use at your own risk
WARNING 26-04-21 06:43:05.956792 - 0:00:00 - Signal handler installed.
WARNING 26-04-21 06:43:05.957136 - 0:00:00 - WARNING: Setting MKL_SERVICE_FORCE_INTEL to GNU
WARNING 26-04-21 06:43:05.957216 - 0:00:00 - WARNING: Setting MKL_NUM_THREADS to 1
WARNING 26-04-21 06:43:05.957273 - 0:00:00 - WARNING: Setting ENABLE_INTRA_NODE_COMM to 1
WARNING 26-04-21 06:43:05.957319 - 0:00:00 - WARNING: Setting TORCH_NCCL_AVOID_RECORD_STREAMS to 1
WARNING 26-04-21 06:43:05.957360 - 0:00:00 - WARNING: Setting NCCL_IB_TIMEOUT to 22
WARNING 26-04-21 06:43:05.957400 - 0:00:00 - WARNING: Setting NCCL_DEBUG to INFO
WARNING 26-04-21 06:43:05.957438 - 0:00:00 - WARNING: Setting TORCH_NCCL_ASYNC_ERROR_HANDLING to 1
WARNING 26-04-21 06:43:05.957473 - 0:00:00 - WARNING: Setting TRITON_CACHE_DIR to /scratch/tmpeghijfw5
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/fsx/craffel/lingua_logs/script_1/code/apps/main/train.py", line 687, in <module>
    main()
  File "/fsx/craffel/lingua_logs/script_1/code/apps/main/train.py", line 683, in main
    train(cfg)
  File "/fsx/craffel/lingua_logs/script_1/code/apps/main/train.py", line 242, in train
    setup_torch_distributed(args.distributed)
  File "/fsx/craffel/lingua_logs/script_1/code/lingua/distributed.py", line 243, in setup_torch_distributed
    mp.set_start_method(dist_args.spawn_method)
  File "/fsx/craffel/miniconda3/envs/lingua_250401/lib/python3.11/multiprocessing/context.py", line 247, in set_start_method
    raise RuntimeError('context has already been set')
RuntimeError: context has already been set