W0618 14:50:54.254000 20975 site-packages/torch/distributed/run.py:862] 
W0618 14:50:54.254000 20975 site-packages/torch/distributed/run.py:862] *****************************************
W0618 14:50:54.254000 20975 site-packages/torch/distributed/run.py:862] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0618 14:50:54.254000 20975 site-packages/torch/distributed/run.py:862] *****************************************
W0618 14:51:30.170000 21047 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.278000 21047 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.319000 21043 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.362000 21043 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.697000 21042 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.701000 21045 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.731000 21044 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.741000 21042 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.743000 21045 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.769000 21049 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.772000 21044 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.780000 21046 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.784000 21048 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.804000 21049 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.815000 21046 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.818000 21048 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yamlcfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml

cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml
cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml
cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yamlcfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml

cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml
cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO cudaDriverVersion 13000
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO cudaDriverVersion 13000
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO cudaDriverVersion 13000
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO cudaDriverVersion 13000
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Bootstrap: Using eth0:10.178.128.103<0>
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Bootstrap: Using eth0:10.178.128.103<0>
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO NCCL version 2.29.7+cuda13.2
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO NCCL version 2.29.7+cuda13.2
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO NCCL git version stable b81d6a5a3
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO NCCL git version stable b81d6a5a3
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO cudaDriverVersion 13000
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Bootstrap: Using eth0:10.178.128.103<0>
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO NCCL version 2.29.7+cuda13.2
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO NCCL git version stable b81d6a5a3
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Bootstrap: Using eth0:10.178.128.103<0>
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO NCCL version 2.29.7+cuda13.2
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO NCCL git version stable b81d6a5a3
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Comm config Blocking set to 1
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO cudaDriverVersion 13000
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Comm config Blocking set to 1
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Comm config Blocking set to 1
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Comm config Blocking set to 1
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Bootstrap: Using eth0:10.178.128.103<0>
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO NCCL version 2.29.7+cuda13.2
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO NCCL git version stable b81d6a5a3
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Comm config Blocking set to 1
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Bootstrap: Using eth0:10.178.128.103<0>
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO NCCL version 2.29.7+cuda13.2
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO NCCL git version stable b81d6a5a3
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Comm config Blocking set to 1
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO cudaDriverVersion 13000
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO cudaDriverVersion 13000
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Bootstrap: Using eth0:10.178.128.103<0>
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO NCCL version 2.29.7+cuda13.2
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO NCCL git version stable b81d6a5a3
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Bootstrap: Using eth0:10.178.128.103<0>
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO NCCL version 2.29.7+cuda13.2
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO NCCL git version stable b81d6a5a3
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Comm config Blocking set to 1
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Comm config Blocking set to 1
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO transport/net_ib/init.cc:396 -> 3
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO transport/net_ib/init.cc:396 -> 3
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO transport/net_ib/init.cc:396 -> 3
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO transport/net_ib/init.cc:396 -> 3
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Failed to initialize NET plugin IB
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Failed to initialize NET plugin IB
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO transport/net_ib/init.cc:396 -> 3
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO transport/net_ib/init.cc:396 -> 3
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Failed to initialize NET plugin IB
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Failed to initialize NET plugin IB
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Failed to initialize NET plugin IB
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Failed to initialize NET plugin IB
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO NET/Socket : Using [0]eth0:10.178.128.103<0>
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Initialized NET plugin Socket
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Assigned NET plugin Socket to comm
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO NET/Socket : Using [0]eth0:10.178.128.103<0>
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO NET/Socket : Using [0]eth0:10.178.128.103<0>
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Initialized NET plugin Socket
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Initialized NET plugin Socket
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO transport/net_ib/init.cc:396 -> 3
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Failed to initialize NET plugin IB
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Assigned NET plugin Socket to comm
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Assigned NET plugin Socket to comm
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO NET/Socket : Using [0]eth0:10.178.128.103<0>
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO NET/Socket : Using [0]eth0:10.178.128.103<0>
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO NET/Socket : Using [0]eth0:10.178.128.103<0>
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Initialized NET plugin Socket
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Initialized NET plugin Socket
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Initialized NET plugin Socket
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Assigned NET plugin Socket to comm
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Assigned NET plugin Socket to comm
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Assigned NET plugin Socket to comm
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO NET/Socket : Using [0]eth0:10.178.128.103<0>
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Initialized NET plugin Socket
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Assigned NET plugin Socket to comm
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Failed to initialize any GIN plugin
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Using network Socket
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Failed to initialize any GIN plugin
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Using network Socket
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Failed to initialize any GIN plugin
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Failed to initialize any GIN plugin
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Using network Socket
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Using network Socket
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Failed to initialize any GIN plugin
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Failed to initialize any GIN plugin
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Using network Socket
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Failed to initialize any GIN plugin
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Using network Socket
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Using network Socket
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO transport/net_ib/init.cc:396 -> 3
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Failed to initialize NET plugin IB
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO NET/Socket : Using [0]eth0:10.178.128.103<0>
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Initialized NET plugin Socket
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Assigned NET plugin Socket to comm
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Failed to initialize any GIN plugin
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Using network Socket
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO [Rank 11] ncclCommInitRankConfig comm 0x5651a9e17960 rank 11 nranks 32 cudaDev 3 nvmlDev 3 busId 4e000 commId 0x7a425254b479deb7 - Init START
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO [Rank 14] ncclCommInitRankConfig comm 0x5604d34c5fa0 rank 14 nranks 32 cudaDev 6 nvmlDev 6 busId db000 commId 0x7a425254b479deb7 - Init START
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO [Rank 13] ncclCommInitRankConfig comm 0x5591c0a3fa30 rank 13 nranks 32 cudaDev 5 nvmlDev 5 busId a1000 commId 0x7a425254b479deb7 - Init START
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO [Rank 9] ncclCommInitRankConfig comm 0x56515db472e0 rank 9 nranks 32 cudaDev 1 nvmlDev 1 busId 10000 commId 0x7a425254b479deb7 - Init START
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO [Rank 15] ncclCommInitRankConfig comm 0x56183b6f0ae0 rank 15 nranks 32 cudaDev 7 nvmlDev 7 busId e0000 commId 0x7a425254b479deb7 - Init START
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO [Rank 10] ncclCommInitRankConfig comm 0x56414f024bb0 rank 10 nranks 32 cudaDev 2 nvmlDev 2 busId 48000 commId 0x7a425254b479deb7 - Init START
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO [Rank 8] ncclCommInitRankConfig comm 0x55de0c9cabb0 rank 8 nranks 32 cudaDev 0 nvmlDev 0 busId b000 commId 0x7a425254b479deb7 - Init START
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO [Rank 12] ncclCommInitRankConfig comm 0x5568eccd24d0 rank 12 nranks 32 cudaDev 4 nvmlDev 4 busId 9b000 commId 0x7a425254b479deb7 - Init START
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Bootstrap timings total 0.366470 (create 0.000107, send 0.000655, recv 0.032611, ring 0.331141, delay 0.000003)
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Bootstrap timings total 0.334157 (create 0.000082, send 0.000627, recv 0.001341, ring 0.331225, delay 0.000003)
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Bootstrap timings total 0.334732 (create 0.000078, send 0.000501, recv 0.001245, ring 0.332147, delay 0.000003)
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Bootstrap timings total 0.436544 (create 0.000089, send 0.000604, recv 0.104207, ring 0.330540, delay 0.000003)
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Bootstrap timings total 0.334165 (create 0.000132, send 0.000621, recv 0.001859, ring 0.330700, delay 0.000003)
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Bootstrap timings total 0.398533 (create 0.000105, send 0.000639, recv 0.000765, ring 0.330577, delay 0.000003)
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Bootstrap timings total 0.419006 (create 0.000081, send 0.000532, recv 0.071832, ring 0.345517, delay 0.000003)
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Bootstrap timings total 0.348157 (create 0.000091, send 0.000585, recv 0.335943, ring 0.010122, delay 0.000003)
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 5 is 96-111,224-239. (GPU affinity = 96-111,224-239 ; CPU affinity = 0-119,128-247).
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO NVLS multicast support is not available on dev 5 (NVLS_NCHANNELS 0)
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 4 is 96-111,224-239. (GPU affinity = 96-111,224-239 ; CPU affinity = 0-119,128-247).
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO NVLS multicast support is not available on dev 4 (NVLS_NCHANNELS 0)
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 2 is 16-31,144-159. (GPU affinity = 16-31,144-159 ; CPU affinity = 0-119,128-247).
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO NVLS multicast support is not available on dev 2 (NVLS_NCHANNELS 0)
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 6 is 64-79,192-207. (GPU affinity = 64-79,192-207 ; CPU affinity = 0-119,128-247).
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO NVLS multicast support is not available on dev 6 (NVLS_NCHANNELS 0)
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 0 is 48-63,176-191. (GPU affinity = 48-63,176-191 ; CPU affinity = 0-119,128-247).
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 1 is 48-63,176-191. (GPU affinity = 48-63,176-191 ; CPU affinity = 0-119,128-247).
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO NVLS multicast support is not available on dev 0 (NVLS_NCHANNELS 0)
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO NVLS multicast support is not available on dev 1 (NVLS_NCHANNELS 0)
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 7 is 64-79,192-207. (GPU affinity = 64-79,192-207 ; CPU affinity = 0-119,128-247).
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 3 is 16-31,144-159. (GPU affinity = 16-31,144-159 ; CPU affinity = 0-119,128-247).
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO NVLS multicast support is not available on dev 7 (NVLS_NCHANNELS 0)
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO NVLS multicast support is not available on dev 3 (NVLS_NCHANNELS 0)
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO comm 0x5591c0a3fa30 rank 13 nRanks 32 nNodes 4 localRanks 8 localRank 5 MNNVL 0
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO comm 0x5604d34c5fa0 rank 14 nRanks 32 nNodes 4 localRanks 8 localRank 6 MNNVL 0
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO comm 0x56183b6f0ae0 rank 15 nRanks 32 nNodes 4 localRanks 8 localRank 7 MNNVL 0
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Trees [0] 14/-1/-1->13->12 [1] 14/-1/-1->13->12
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO P2P Chunksize set to 131072
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO comm 0x5568eccd24d0 rank 12 nRanks 32 nNodes 4 localRanks 8 localRank 4 MNNVL 0
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Trees [0] -1/-1/-1->15->14 [1] -1/-1/-1->15->14
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Trees [0] 15/-1/-1->14->13 [1] 15/-1/-1->14->13
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO P2P Chunksize set to 131072
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO P2P Chunksize set to 131072
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Trees [0] 13/-1/-1->12->11 [1] 13/-1/-1->12->11
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO P2P Chunksize set to 131072
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO comm 0x5651a9e17960 rank 11 nRanks 32 nNodes 4 localRanks 8 localRank 3 MNNVL 0
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO comm 0x56515db472e0 rank 9 nRanks 32 nNodes 4 localRanks 8 localRank 1 MNNVL 0
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO comm 0x55de0c9cabb0 rank 8 nRanks 32 nNodes 4 localRanks 8 localRank 0 MNNVL 0
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO comm 0x56414f024bb0 rank 10 nRanks 32 nNodes 4 localRanks 8 localRank 2 MNNVL 0
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Trees [0] 9/-1/-1->8->17 [1] 9/0/-1->8->24
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Trees [0] 10/-1/-1->9->8 [1] 10/16/-1->9->8
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO P2P Chunksize set to 131072
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO P2P Chunksize set to 131072
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Trees [0] 12/-1/-1->11->10 [1] 12/-1/-1->11->10
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO P2P Chunksize set to 131072
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Trees [0] 11/-1/-1->10->9 [1] 11/-1/-1->10->9
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO P2P Chunksize set to 131072
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
h-liuchonghan-translate-a8nodes-main-w-0:21046:21114 [0] NCCL INFO [Proxy Service] Device 4 CPU core 230
h-liuchonghan-translate-a8nodes-main-w-0:21046:21115 [0] NCCL INFO [Proxy Service UDS] Device 4 CPU core 111
h-liuchonghan-translate-a8nodes-main-w-0:21047:21118 [0] NCCL INFO [Proxy Service] Device 5 CPU core 236
h-liuchonghan-translate-a8nodes-main-w-0:21049:21116 [0] NCCL INFO [Proxy Service] Device 7 CPU core 73
h-liuchonghan-translate-a8nodes-main-w-0:21042:21119 [0] NCCL INFO [Proxy Service] Device 0 CPU core 180
h-liuchonghan-translate-a8nodes-main-w-0:21048:21117 [0] NCCL INFO [Proxy Service] Device 6 CPU core 205
h-liuchonghan-translate-a8nodes-main-w-0:21042:21123 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 184
h-liuchonghan-translate-a8nodes-main-w-0:21049:21120 [0] NCCL INFO [Proxy Service UDS] Device 7 CPU core 78
h-liuchonghan-translate-a8nodes-main-w-0:21048:21121 [0] NCCL INFO [Proxy Service UDS] Device 6 CPU core 206
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
h-liuchonghan-translate-a8nodes-main-w-0:21047:21122 [0] NCCL INFO [Proxy Service UDS] Device 5 CPU core 228
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
h-liuchonghan-translate-a8nodes-main-w-0:21045:21129 [0] NCCL INFO [Proxy Service UDS] Device 3 CPU core 28
h-liuchonghan-translate-a8nodes-main-w-0:21043:21124 [0] NCCL INFO [Proxy Service] Device 1 CPU core 189
h-liuchonghan-translate-a8nodes-main-w-0:21045:21127 [0] NCCL INFO [Proxy Service] Device 3 CPU core 21
h-liuchonghan-translate-a8nodes-main-w-0:21044:21128 [0] NCCL INFO [Proxy Service UDS] Device 2 CPU core 155
h-liuchonghan-translate-a8nodes-main-w-0:21043:21125 [0] NCCL INFO [Proxy Service UDS] Device 1 CPU core 185
h-liuchonghan-translate-a8nodes-main-w-0:21044:21126 [0] NCCL INFO [Proxy Service] Device 2 CPU core 21
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Symmetric VA size=80GB
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Symmetric VA size=80GB
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Symmetric VA size=80GB
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Symmetric VA size=80GB
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Symmetric VA size=80GB
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Symmetric VA size=80GB
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Symmetric VA size=80GB
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Symmetric VA size=80GB
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO ncclCommInitRankConfig comm 0x56515db472e0 rank 9 nranks 32 cudaDev 1 nvmlDev 1 busId 10000 commId 0x7a425254b479deb7 - Init COMPLETE
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO ncclCommInitRankConfig comm 0x5591c0a3fa30 rank 13 nranks 32 cudaDev 5 nvmlDev 5 busId a1000 commId 0x7a425254b479deb7 - Init COMPLETE
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO ncclCommInitRankConfig comm 0x5604d34c5fa0 rank 14 nranks 32 cudaDev 6 nvmlDev 6 busId db000 commId 0x7a425254b479deb7 - Init COMPLETE
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO ncclCommInitRankConfig comm 0x56183b6f0ae0 rank 15 nranks 32 cudaDev 7 nvmlDev 7 busId e0000 commId 0x7a425254b479deb7 - Init COMPLETE
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Init timings - ncclCommInitRankConfig: rank 13 nranks 32 total 1.27 (kernels 0.39, alloc 0.16, bootstrap 0.40, allgathers 0.03, topo 0.27, graphs 0.00, connections 0.01, rest 0.01)
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO ncclCommInitRankConfig comm 0x56414f024bb0 rank 10 nranks 32 cudaDev 2 nvmlDev 2 busId 48000 commId 0x7a425254b479deb7 - Init COMPLETE
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 9 nranks 32 total 1.27 (kernels 0.39, alloc 0.19, bootstrap 0.37, allgathers 0.03, topo 0.28, graphs 0.00, connections 0.01, rest 0.01)
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Init timings - ncclCommInitRankConfig: rank 15 nranks 32 total 1.26 (kernels 0.38, alloc 0.21, bootstrap 0.35, allgathers 0.03, topo 0.28, graphs 0.00, connections 0.01, rest 0.01)
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Init timings - ncclCommInitRankConfig: rank 14 nranks 32 total 1.27 (kernels 0.39, alloc 0.14, bootstrap 0.42, allgathers 0.03, topo 0.27, graphs 0.00, connections 0.01, rest 0.01)
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Init timings - ncclCommInitRankConfig: rank 10 nranks 32 total 1.26 (kernels 0.38, alloc 0.22, bootstrap 0.33, allgathers 0.03, topo 0.27, graphs 0.00, connections 0.01, rest 0.00)
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO ncclCommInitRankConfig comm 0x55de0c9cabb0 rank 8 nranks 32 cudaDev 0 nvmlDev 0 busId b000 commId 0x7a425254b479deb7 - Init COMPLETE
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO ncclCommInitRankConfig comm 0x5651a9e17960 rank 11 nranks 32 cudaDev 3 nvmlDev 3 busId 4e000 commId 0x7a425254b479deb7 - Init COMPLETE
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO ncclCommInitRankConfig comm 0x5568eccd24d0 rank 12 nranks 32 cudaDev 4 nvmlDev 4 busId 9b000 commId 0x7a425254b479deb7 - Init COMPLETE
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 8 nranks 32 total 1.27 (kernels 0.39, alloc 0.22, bootstrap 0.33, allgathers 0.03, topo 0.27, graphs 0.00, connections 0.01, rest 0.01)
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Init timings - ncclCommInitRankConfig: rank 11 nranks 32 total 1.27 (kernels 0.39, alloc 0.12, bootstrap 0.44, allgathers 0.02, topo 0.28, graphs 0.00, connections 0.01, rest 0.01)
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Init timings - ncclCommInitRankConfig: rank 12 nranks 32 total 1.27 (kernels 0.41, alloc 0.21, bootstrap 0.33, allgathers 0.02, topo 0.27, graphs 0.01, connections 0.01, rest 0.01)
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Channel 00/0 : 9[1] -> 10[2] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Channel 01/0 : 9[1] -> 10[2] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Channel 00/0 : 10[2] -> 11[3] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Channel 00/0 : 11[3] -> 12[4] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Channel 01/0 : 10[2] -> 11[3] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Channel 00/0 : 12[4] -> 13[5] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Channel 00/0 : 14[6] -> 15[7] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Channel 01/0 : 11[3] -> 12[4] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 00/0 : 8[0] -> 9[1] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Channel 01/0 : 12[4] -> 13[5] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Channel 01/0 : 14[6] -> 15[7] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Channel 00/0 : 13[5] -> 14[6] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 01/0 : 8[0] -> 9[1] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Channel 01/0 : 13[5] -> 14[6] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Channel 00/0 : 15[7] -> 14[6] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Channel 01/0 : 15[7] -> 14[6] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Channel 00/0 : 10[2] -> 9[1] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Channel 00/0 : 11[3] -> 10[2] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Channel 01/0 : 10[2] -> 9[1] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Channel 01/0 : 11[3] -> 10[2] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Channel 00/0 : 14[6] -> 13[5] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Channel 01/0 : 14[6] -> 13[5] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Channel 00/0 : 13[5] -> 12[4] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Channel 00/0 : 12[4] -> 11[3] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Channel 01/0 : 12[4] -> 11[3] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Channel 01/0 : 13[5] -> 12[4] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21043:21131 [0] NCCL INFO [Proxy Progress] Device 1 CPU core 181
h-liuchonghan-translate-a8nodes-main-w-0:21042:21130 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 181
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Channel 01/0 : 9[1] -> 16[0] [send] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 01/0 : 0[0] -> 8[0] [receive] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 00/0 : 8[0] -> 17[1] [send] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Channel 01/0 : 16[0] -> 9[1] [receive] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 01/0 : 24[0] -> 8[0] [receive] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 01/0 : 8[0] -> 24[0] [send] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 00/0 : 17[1] -> 8[0] [receive] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 01/0 : 8[0] -> 0[0] [send] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Channel 00/0 : 9[1] -> 8[0] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Channel 01/0 : 9[1] -> 8[0] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Connected all trees
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Connected all trees
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Connected all trees
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Connected all trees
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Connected all trees
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Connected all trees
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Connected all trees
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Connected all trees

Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights:   1%|          | 2/399 [00:00<00:23, 16.61it/s]
Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights:   1%|          | 4/399 [00:00<00:11, 35.69it/s]
Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights:   1%|          | 2/399 [00:00<00:25, 15.35it/s]
Loading weights:   1%|          | 3/399 [00:00<00:13, 29.53it/s]
Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights:   1%|▏         | 5/399 [00:00<00:11, 33.27it/s]
Loading weights:   2%|▏         | 7/399 [00:00<00:17, 22.37it/s]
Loading weights:   5%|▍         | 18/399 [00:00<00:02, 128.79it/s]
Loading weights:   2%|▏         | 6/399 [00:00<00:13, 28.11it/s]
Loading weights:   5%|▍         | 19/399 [00:00<00:02, 188.13it/s]
Loading weights:   2%|▏         | 8/399 [00:00<00:18, 21.38it/s]
Loading weights:   2%|▏         | 6/399 [00:00<00:18, 21.82it/s]
Loading weights:   2%|▏         | 9/399 [00:00<00:05, 68.27it/s]
Loading weights:   4%|▍         | 17/399 [00:00<00:07, 49.81it/s]
Loading weights:   3%|▎         | 11/399 [00:00<00:18, 21.39it/s]
Loading weights:   3%|▎         | 10/399 [00:00<00:14, 27.66it/s]
Loading weights:   3%|▎         | 11/399 [00:00<00:18, 21.35it/s]
Loading weights:   3%|▎         | 10/399 [00:00<00:16, 23.77it/s]
Loading weights:   3%|▎         | 13/399 [00:00<00:13, 28.22it/s]
Loading weights:   4%|▍         | 16/399 [00:00<00:09, 41.32it/s]
Loading weights:   4%|▎         | 14/399 [00:00<00:14, 25.97it/s]
Loading weights:   4%|▎         | 14/399 [00:00<00:20, 18.64it/s]
Loading weights:   6%|▌         | 23/399 [00:00<00:10, 36.49it/s]
Loading weights:   4%|▍         | 17/399 [00:00<00:14, 25.80it/s]
Loading weights:   4%|▎         | 14/399 [00:00<00:23, 16.05it/s]
Loading weights:   4%|▍         | 16/399 [00:00<00:17, 21.57it/s]
Loading weights:   5%|▍         | 18/399 [00:00<00:17, 21.95it/s]
Loading weights:   5%|▌         | 21/399 [00:00<00:13, 27.70it/s]
Loading weights:   5%|▌         | 21/399 [00:00<00:18, 20.55it/s]
Loading weights:   7%|▋         | 28/399 [00:00<00:13, 27.74it/s]
Loading weights:   8%|▊         | 31/399 [00:00<00:11, 31.10it/s] 
Loading weights:   4%|▍         | 16/399 [00:01<00:29, 12.89it/s]
Loading weights:   5%|▍         | 19/399 [00:00<00:20, 18.23it/s]
Loading weights:   5%|▌         | 20/399 [00:00<00:20, 18.75it/s]
Loading weights:   6%|▋         | 25/399 [00:00<00:14, 25.41it/s]
Loading weights:  10%|▉         | 38/399 [00:00<00:09, 37.99it/s] 
Loading weights:   8%|▊         | 32/399 [00:01<00:12, 29.62it/s]
Loading weights:   6%|▌         | 24/399 [00:01<00:18, 19.84it/s]
Loading weights:   6%|▌         | 24/399 [00:01<00:18, 20.67it/s]
Loading weights:   5%|▌         | 21/399 [00:01<00:21, 17.29it/s]
Loading weights:   6%|▌         | 22/399 [00:01<00:21, 17.31it/s]
Loading weights:  10%|▉         | 38/399 [00:01<00:11, 30.74it/s]
Loading weights:   9%|▉         | 36/399 [00:01<00:13, 27.86it/s]
Loading weights:   6%|▋         | 25/399 [00:01<00:17, 21.48it/s]
Loading weights:   7%|▋         | 28/399 [00:01<00:16, 22.01it/s]
Loading weights:   6%|▋         | 25/399 [00:01<00:19, 19.10it/s]
Loading weights:   7%|▋         | 27/399 [00:01<00:21, 17.48it/s]
Loading weights:  10%|█         | 40/399 [00:01<00:12, 29.01it/s]
Loading weights:   8%|▊         | 31/399 [00:01<00:16, 22.42it/s]
Loading weights:   8%|▊         | 31/399 [00:01<00:16, 21.97it/s]
Loading weights:   8%|▊         | 33/399 [00:01<00:13, 27.11it/s]
Loading weights:  11%|█         | 43/399 [00:01<00:13, 26.73it/s]
Loading weights:   7%|▋         | 27/399 [00:01<00:25, 14.87it/s]
Loading weights:  11%|█         | 44/399 [00:01<00:12, 27.97it/s]
Loading weights:   9%|▉         | 37/399 [00:01<00:13, 26.66it/s]
Loading weights:   7%|▋         | 28/399 [00:01<00:23, 15.89it/s]
Loading weights:   7%|▋         | 29/399 [00:01<00:24, 15.37it/s]
Loading weights:  11%|█         | 42/399 [00:01<00:10, 35.52it/s]
Loading weights:  12%|█▏        | 47/399 [00:01<00:13, 26.29it/s]
Loading weights:   9%|▉         | 37/399 [00:01<00:14, 25.35it/s]
Loading weights:  12%|█▏        | 48/399 [00:01<00:12, 27.34it/s]
Loading weights:  12%|█▏        | 48/399 [00:01<00:12, 27.59it/s]
Loading weights:  10%|█         | 40/399 [00:01<00:14, 25.46it/s]
Loading weights:   8%|▊         | 31/399 [00:01<00:21, 16.90it/s]
Loading weights:   8%|▊         | 31/399 [00:01<00:24, 15.29it/s]
Loading weights:  12%|█▏        | 46/399 [00:01<00:11, 30.37it/s]
Loading weights:  12%|█▏        | 46/399 [00:01<00:11, 30.80it/s]
Loading weights:  10%|█         | 40/399 [00:01<00:15, 22.50it/s]
Loading weights:  14%|█▎        | 54/399 [00:01<00:12, 27.75it/s]
Loading weights:   9%|▊         | 34/399 [00:01<00:21, 16.93it/s]
Loading weights:  13%|█▎        | 51/399 [00:01<00:15, 22.23it/s]
Loading weights:   9%|▉         | 35/399 [00:01<00:20, 18.05it/s]
Loading weights:  13%|█▎        | 51/399 [00:01<00:15, 22.55it/s]
Loading weights:  13%|█▎        | 51/399 [00:01<00:10, 33.40it/s]
Loading weights:  14%|█▎        | 54/399 [00:01<00:08, 40.76it/s]
Loading weights:  14%|█▎        | 54/399 [00:01<00:15, 22.51it/s]
Loading weights:   9%|▉         | 36/399 [00:01<00:23, 15.75it/s]
Loading weights:  14%|█▎        | 54/399 [00:01<00:15, 21.91it/s]
Loading weights:  10%|▉         | 38/399 [00:01<00:20, 17.71it/s]
Loading weights:  11%|█         | 43/399 [00:02<00:19, 18.58it/s]
Loading weights:  14%|█▍        | 55/399 [00:02<00:11, 30.01it/s]
Loading weights:  15%|█▍        | 59/399 [00:01<00:13, 25.68it/s]
Loading weights:  10%|█         | 40/399 [00:02<00:19, 18.13it/s]
Loading weights:  14%|█▍        | 57/399 [00:02<00:15, 21.91it/s]
Loading weights:  15%|█▍        | 59/399 [00:01<00:11, 30.19it/s]
Loading weights:  11%|█▏        | 45/399 [00:02<00:20, 17.41it/s]
Loading weights:  14%|█▍        | 57/399 [00:02<00:18, 18.27it/s]
Loading weights:  10%|▉         | 39/399 [00:02<00:25, 14.40it/s]
Loading weights:  11%|█         | 42/399 [00:02<00:19, 17.97it/s]
Loading weights:  15%|█▌        | 61/399 [00:02<00:13, 24.44it/s]
Loading weights:  16%|█▌        | 63/399 [00:02<00:13, 24.27it/s]
Loading weights:  15%|█▍        | 59/399 [00:02<00:13, 24.83it/s]
Loading weights:  11%|█         | 43/399 [00:02<00:18, 19.12it/s]
Loading weights:  12%|█▏        | 48/399 [00:02<00:19, 18.14it/s]
Loading weights:  15%|█▌        | 60/399 [00:02<00:18, 18.48it/s]
Loading weights:  16%|█▌        | 63/399 [00:02<00:13, 25.61it/s]
Loading weights:  16%|█▌        | 64/399 [00:02<00:13, 24.71it/s]
Loading weights:  16%|█▌        | 63/399 [00:02<00:12, 27.61it/s]
Loading weights:  11%|█         | 44/399 [00:02<00:23, 14.85it/s]
Loading weights:  13%|█▎        | 52/399 [00:02<00:16, 21.44it/s]
Loading weights:  16%|█▌        | 62/399 [00:02<00:18, 17.96it/s]
Loading weights:  12%|█▏        | 46/399 [00:02<00:19, 18.37it/s]
Loading weights:  17%|█▋        | 67/399 [00:02<00:15, 21.91it/s]
Loading weights:  17%|█▋        | 67/399 [00:02<00:12, 27.36it/s]
Loading weights:  18%|█▊        | 70/399 [00:02<00:12, 26.49it/s]
Loading weights:  16%|█▌        | 64/399 [00:02<00:18, 17.70it/s]
Loading weights:  12%|█▏        | 49/399 [00:02<00:18, 19.18it/s]
Loading weights:  18%|█▊        | 71/399 [00:02<00:13, 24.10it/s]
Loading weights:  17%|█▋        | 67/399 [00:02<00:15, 21.73it/s]
Loading weights:  12%|█▏        | 49/399 [00:02<00:19, 18.37it/s]
Loading weights:  14%|█▍        | 56/399 [00:02<00:17, 20.05it/s]
Loading weights:  18%|█▊        | 71/399 [00:02<00:12, 26.70it/s]
Loading weights:  19%|█▉        | 77/399 [00:02<00:09, 34.29it/s]
Loading weights:  13%|█▎        | 53/399 [00:02<00:16, 20.35it/s]
Loading weights:  18%|█▊        | 70/399 [00:02<00:15, 21.34it/s]
Loading weights:  19%|█▉        | 75/399 [00:02<00:13, 23.90it/s]
Loading weights:  13%|█▎        | 52/399 [00:02<00:18, 18.85it/s]
Loading weights:  17%|█▋        | 66/399 [00:02<00:22, 14.60it/s]
Loading weights:  19%|█▊        | 74/399 [00:02<00:12, 25.98it/s]
Loading weights:  21%|██        | 82/399 [00:02<00:09, 32.45it/s]
Loading weights:  15%|█▍        | 59/399 [00:03<00:18, 18.57it/s]
Loading weights:  19%|█▉        | 77/399 [00:03<00:11, 26.86it/s]
Loading weights:  14%|█▍        | 55/399 [00:02<00:17, 19.80it/s]
Loading weights:  18%|█▊        | 73/399 [00:02<00:15, 20.40it/s]
Loading weights:  21%|██        | 83/399 [00:02<00:10, 29.31it/s]
Loading weights:  18%|█▊        | 73/399 [00:03<00:14, 22.59it/s]
Loading weights:  14%|█▍        | 56/399 [00:02<00:18, 18.24it/s]
Loading weights:  15%|█▌        | 61/399 [00:03<00:18, 17.90it/s]
Loading weights:  20%|██        | 81/399 [00:03<00:11, 27.97it/s]
Loading weights:  22%|██▏       | 86/399 [00:02<00:10, 28.92it/s]
Loading weights:  19%|█▉        | 76/399 [00:02<00:14, 21.84it/s]
Loading weights:  20%|██        | 80/399 [00:03<00:10, 31.72it/s]
Loading weights:  16%|█▋        | 65/399 [00:03<00:15, 21.64it/s]
Loading weights:  15%|█▍        | 58/399 [00:03<00:18, 18.63it/s]
Loading weights:  15%|█▍        | 58/399 [00:03<00:19, 17.82it/s]
Loading weights:  35%|███▌      | 140/399 [00:03<00:01, 165.30it/s]
Loading weights:  36%|███▌      | 143/399 [00:03<00:01, 139.93it/s]
Loading weights: 100%|██████████| 399/399 [00:03<00:00, 121.18it/s]

Loading weights: 100%|██████████| 399/399 [00:03<00:00, 122.56it/s]

Loading weights:  43%|████▎     | 171/399 [00:03<00:01, 214.25it/s]
Loading weights: 100%|██████████| 399/399 [00:03<00:00, 127.46it/s]

Loading weights: 100%|██████████| 399/399 [00:03<00:00, 127.92it/s]

Loading weights: 100%|██████████| 399/399 [00:03<00:00, 131.71it/s]

Loading weights:  22%|██▏       | 87/399 [00:03<00:11, 26.15it/s]
Loading weights:  55%|█████▍    | 219/399 [00:03<00:00, 333.24it/s]
Loading weights: 100%|██████████| 399/399 [00:03<00:00, 124.81it/s]

Loading weights: 100%|██████████| 399/399 [00:03<00:00, 123.47it/s]

Loading weights: 100%|██████████| 399/399 [00:03<00:00, 128.97it/s]
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Channel 00/0 : 9[1] -> 16[0] [send] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 00/0 : 1[1] -> 8[0] [receive] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 01/0 : 1[1] -> 8[0] [receive] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 00/0 : 8[0] -> 15[7] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 01/0 : 8[0] -> 15[7] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 00/1 : 8[0] -> 0[0] [send] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Channel 00/1 : 9[1] -> 0[0] [send] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-0:21049:22249 [0] NCCL INFO [Proxy Progress] Device 7 CPU core 73
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Channel 00/1 : 15[7] -> 0[0] [send] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-0:21044:22250 [0] NCCL INFO [Proxy Progress] Device 2 CPU core 157
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Channel 00/1 : 10[2] -> 0[0] [send] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-0:21045:22251 [0] NCCL INFO [Proxy Progress] Device 3 CPU core 16
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Channel 00/1 : 11[3] -> 0[0] [send] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-0:21047:22253 [0] NCCL INFO [Proxy Progress] Device 5 CPU core 97
h-liuchonghan-translate-a8nodes-main-w-0:21046:22252 [0] NCCL INFO [Proxy Progress] Device 4 CPU core 97
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Channel 00/1 : 12[4] -> 0[0] [send] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Channel 00/1 : 13[5] -> 0[0] [send] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-0:21048:22254 [0] NCCL INFO [Proxy Progress] Device 6 CPU core 67
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Channel 00/1 : 14[6] -> 0[0] [send] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 01/1 : 0[0] -> 8[0] [receive] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Channel 01/1 : 0[0] -> 11[3] [receive] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Channel 01/1 : 0[0] -> 10[2] [receive] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Channel 01/1 : 0[0] -> 12[4] [receive] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Channel 01/1 : 0[0] -> 14[6] [receive] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Channel 01/1 : 0[0] -> 13[5] [receive] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Channel 01/1 : 0[0] -> 15[7] [receive] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Channel 01/1 : 0[0] -> 9[1] [receive] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO comm 0x5568eccd24d0 rank 12 nranks 32 cudaDev 4 busId 9b000 - Destroy COMPLETE
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO comm 0x56515db472e0 rank 9 nranks 32 cudaDev 1 busId 10000 - Destroy COMPLETE
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO comm 0x56183b6f0ae0 rank 15 nranks 32 cudaDev 7 busId e0000 - Destroy COMPLETE
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO comm 0x5604d34c5fa0 rank 14 nranks 32 cudaDev 6 busId db000 - Destroy COMPLETE
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO comm 0x5591c0a3fa30 rank 13 nranks 32 cudaDev 5 busId a1000 - Destroy COMPLETE
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO comm 0x56414f024bb0 rank 10 nranks 32 cudaDev 2 busId 48000 - Destroy COMPLETE
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO comm 0x55de0c9cabb0 rank 8 nranks 32 cudaDev 0 busId b000 - Destroy COMPLETE
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO comm 0x5651a9e17960 rank 11 nranks 32 cudaDev 3 busId 4e000 - Destroy COMPLETE
h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault