script_1/logs/22088281_0.err

Setting Data Parallel size to 8
Tensor parallelism has not been tested for a while, use at your own risk
WARNING 26-04-21 06:43:06.013601 - 0:00:00 - Signal handler installed.
WARNING 26-04-21 06:43:06.013987 - 0:00:00 - WARNING: Setting MKL_SERVICE_FORCE_INTEL to GNU
WARNING 26-04-21 06:43:06.014081 - 0:00:00 - WARNING: Setting MKL_NUM_THREADS to 1
WARNING 26-04-21 06:43:06.014149 - 0:00:00 - WARNING: Setting ENABLE_INTRA_NODE_COMM to 1
WARNING 26-04-21 06:43:06.014201 - 0:00:00 - WARNING: Setting TORCH_NCCL_AVOID_RECORD_STREAMS to 1
WARNING 26-04-21 06:43:06.014251 - 0:00:00 - WARNING: Setting NCCL_IB_TIMEOUT to 22
WARNING 26-04-21 06:43:06.014302 - 0:00:00 - WARNING: Setting NCCL_DEBUG to INFO
WARNING 26-04-21 06:43:06.014352 - 0:00:00 - WARNING: Setting TORCH_NCCL_ASYNC_ERROR_HANDLING to 1
WARNING 26-04-21 06:43:06.014397 - 0:00:00 - WARNING: Setting TRITON_CACHE_DIR to /scratch/tmpzpz3qait
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/fsx/craffel/lingua_logs/script_1/code/apps/main/train.py", line 687, in <module>
    main()
  File "/fsx/craffel/lingua_logs/script_1/code/apps/main/train.py", line 683, in main
    train(cfg)
  File "/fsx/craffel/lingua_logs/script_1/code/apps/main/train.py", line 242, in train
    setup_torch_distributed(args.distributed)
  File "/fsx/craffel/lingua_logs/script_1/code/lingua/distributed.py", line 243, in setup_torch_distributed
    mp.set_start_method(dist_args.spawn_method)
  File "/fsx/craffel/miniconda3/envs/lingua_250401/lib/python3.11/multiprocessing/context.py", line 247, in set_start_method
    raise RuntimeError('context has already been set')
RuntimeError: context has already been set