script_1/logs/22089041_0.err

Setting Data Parallel size to 8
Tensor parallelism has not been tested for a while, use at your own risk
WARNING 26-04-21 17:22:55.417186 - 0:00:00 - Signal handler installed.
WARNING 26-04-21 17:22:55.419626 - 0:00:00 - WARNING: Setting MKL_SERVICE_FORCE_INTEL to GNU
WARNING 26-04-21 17:22:55.419758 - 0:00:00 - WARNING: Setting MKL_NUM_THREADS to 1
WARNING 26-04-21 17:22:55.419850 - 0:00:00 - WARNING: Setting ENABLE_INTRA_NODE_COMM to 1
WARNING 26-04-21 17:22:55.419922 - 0:00:00 - WARNING: Setting TORCH_NCCL_AVOID_RECORD_STREAMS to 1
WARNING 26-04-21 17:22:55.419988 - 0:00:00 - WARNING: Setting NCCL_IB_TIMEOUT to 22
WARNING 26-04-21 17:22:55.420053 - 0:00:00 - WARNING: Setting NCCL_DEBUG to INFO
WARNING 26-04-21 17:22:55.420121 - 0:00:00 - WARNING: Setting TORCH_NCCL_ASYNC_ERROR_HANDLING to 1
WARNING 26-04-21 17:22:55.420191 - 0:00:00 - WARNING: Setting TRITON_CACHE_DIR to /scratch/tmp9riq9mr6
/fsx/craffel/miniconda3/envs/lingua_250401/lib/python3.11/site-packages/torch/autograd/graph.py:825: UserWarning: cuDNN SDPA backward got grad_output.strides() != output.strides(), attempting to materialize a grad_output with matching strides... (Triggered internally at ../aten/src/ATen/native/cudnn/MHA.cpp:674.)
  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[rank0]:[W421 17:25:36.949081863 CPUAllocator.cpp:249] Memory block of unknown size was allocated before the profiling started, profiler results will not include the deallocation event
[rank0]:[W426 04:29:30.940741292 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())