script_1/logs/22088281_7.err

Setting Data Parallel size to 8
Tensor parallelism has not been tested for a while, use at your own risk
WARNING 26-04-21 06:43:05.964694 - 0:00:00 - Signal handler installed.
WARNING 26-04-21 06:43:05.965020 - 0:00:00 - WARNING: Setting MKL_SERVICE_FORCE_INTEL to GNU
WARNING 26-04-21 06:43:05.965089 - 0:00:00 - WARNING: Setting MKL_NUM_THREADS to 1
WARNING 26-04-21 06:43:05.965155 - 0:00:00 - WARNING: Setting ENABLE_INTRA_NODE_COMM to 1
WARNING 26-04-21 06:43:05.965202 - 0:00:00 - WARNING: Setting TORCH_NCCL_AVOID_RECORD_STREAMS to 1
WARNING 26-04-21 06:43:05.965239 - 0:00:00 - WARNING: Setting NCCL_IB_TIMEOUT to 22
WARNING 26-04-21 06:43:05.965276 - 0:00:00 - WARNING: Setting NCCL_DEBUG to INFO
WARNING 26-04-21 06:43:05.965311 - 0:00:00 - WARNING: Setting TORCH_NCCL_ASYNC_ERROR_HANDLING to 1
WARNING 26-04-21 06:43:05.965344 - 0:00:00 - WARNING: Setting TRITON_CACHE_DIR to /scratch/tmpgslflncp
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/fsx/craffel/lingua_logs/script_1/code/apps/main/train.py", line 687, in <module>
    main()
  File "/fsx/craffel/lingua_logs/script_1/code/apps/main/train.py", line 683, in main
    train(cfg)
  File "/fsx/craffel/lingua_logs/script_1/code/apps/main/train.py", line 242, in train
    setup_torch_distributed(args.distributed)
  File "/fsx/craffel/lingua_logs/script_1/code/lingua/distributed.py", line 243, in setup_torch_distributed
    mp.set_start_method(dist_args.spawn_method)
  File "/fsx/craffel/miniconda3/envs/lingua_250401/lib/python3.11/multiprocessing/context.py", line 247, in set_start_method
    raise RuntimeError('context has already been set')
RuntimeError: context has already been set