| W0618 14:50:54.254000 20975 site-packages/torch/distributed/run.py:862] |
| W0618 14:50:54.254000 20975 site-packages/torch/distributed/run.py:862] ***************************************** |
| W0618 14:50:54.254000 20975 site-packages/torch/distributed/run.py:862] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. |
| W0618 14:50:54.254000 20975 site-packages/torch/distributed/run.py:862] ***************************************** |
| W0618 14:51:30.170000 21047 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release. |
| W0618 14:51:30.278000 21047 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release. |
| W0618 14:51:30.319000 21043 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release. |
| W0618 14:51:30.362000 21043 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release. |
| W0618 14:51:30.697000 21042 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release. |
| W0618 14:51:30.701000 21045 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release. |
| W0618 14:51:30.731000 21044 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release. |
| W0618 14:51:30.741000 21042 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release. |
| W0618 14:51:30.743000 21045 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release. |
| W0618 14:51:30.769000 21049 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release. |
| W0618 14:51:30.772000 21044 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release. |
| W0618 14:51:30.780000 21046 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release. |
| W0618 14:51:30.784000 21048 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release. |
| W0618 14:51:30.804000 21049 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release. |
| W0618 14:51:30.815000 21046 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release. |
| W0618 14:51:30.818000 21048 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release. |
| cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yamlcfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml |
|
|
| cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml |
| cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml |
| cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yamlcfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml |
|
|
| cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml |
| cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO cudaDriverVersion 13000 |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO cudaDriverVersion 13000 |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO cudaDriverVersion 13000 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO cudaDriverVersion 13000 |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Bootstrap: Using eth0:10.178.128.103<0> |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Bootstrap: Using eth0:10.178.128.103<0> |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO NCCL version 2.29.7+cuda13.2 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO NCCL version 2.29.7+cuda13.2 |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO NCCL git version stable b81d6a5a3 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO NCCL git version stable b81d6a5a3 |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO cudaDriverVersion 13000 |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Bootstrap: Using eth0:10.178.128.103<0> |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO NCCL version 2.29.7+cuda13.2 |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO NCCL git version stable b81d6a5a3 |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Bootstrap: Using eth0:10.178.128.103<0> |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO NCCL version 2.29.7+cuda13.2 |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO NCCL git version stable b81d6a5a3 |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Comm config Blocking set to 1 |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO cudaDriverVersion 13000 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Comm config Blocking set to 1 |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Comm config Blocking set to 1 |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Comm config Blocking set to 1 |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Bootstrap: Using eth0:10.178.128.103<0> |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO NCCL version 2.29.7+cuda13.2 |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO NCCL git version stable b81d6a5a3 |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Comm config Blocking set to 1 |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Bootstrap: Using eth0:10.178.128.103<0> |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO NCCL version 2.29.7+cuda13.2 |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO NCCL git version stable b81d6a5a3 |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Comm config Blocking set to 1 |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO cudaDriverVersion 13000 |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO cudaDriverVersion 13000 |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Bootstrap: Using eth0:10.178.128.103<0> |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO NCCL version 2.29.7+cuda13.2 |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO NCCL git version stable b81d6a5a3 |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Bootstrap: Using eth0:10.178.128.103<0> |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO NCCL version 2.29.7+cuda13.2 |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO NCCL git version stable b81d6a5a3 |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Comm config Blocking set to 1 |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Comm config Blocking set to 1 |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO NET/Plugin: Could not find: libnccl-net.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO NET/Plugin: Could not find: libnccl-net.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO NET/Plugin: Could not find: libnccl-net.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO NET/Plugin: Could not find: libnccl-net.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO NCCL_IB_DISABLE set by environment to 1. |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO NCCL_IB_DISABLE set by environment to 1. |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO NCCL_IB_DISABLE set by environment to 1. |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO NCCL_IB_DISABLE set by environment to 1. |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO NET/Plugin: Could not find: libnccl-net.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO transport/net_ib/init.cc:396 -> 3 |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO transport/net_ib/init.cc:396 -> 3 |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO transport/net_ib/init.cc:396 -> 3 |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO transport/net_ib/init.cc:396 -> 3 |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Failed to initialize NET plugin IB |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Failed to initialize NET plugin IB |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1. |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO NCCL_IB_DISABLE set by environment to 1. |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO transport/net_ib/init.cc:396 -> 3 |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO transport/net_ib/init.cc:396 -> 3 |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Failed to initialize NET plugin IB |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Failed to initialize NET plugin IB |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Failed to initialize NET plugin IB |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Failed to initialize NET plugin IB |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO NET/Socket : Using [0]eth0:10.178.128.103<0> |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Initialized NET plugin Socket |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Assigned NET plugin Socket to comm |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO NET/Socket : Using [0]eth0:10.178.128.103<0> |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO NET/Socket : Using [0]eth0:10.178.128.103<0> |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO NET/Plugin: Could not find: libnccl-net.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Initialized NET plugin Socket |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Initialized NET plugin Socket |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 1. |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO transport/net_ib/init.cc:396 -> 3 |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Failed to initialize NET plugin IB |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Assigned NET plugin Socket to comm |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Assigned NET plugin Socket to comm |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO NET/Socket : Using [0]eth0:10.178.128.103<0> |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO NET/Socket : Using [0]eth0:10.178.128.103<0> |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO NET/Socket : Using [0]eth0:10.178.128.103<0> |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Initialized NET plugin Socket |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Initialized NET plugin Socket |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Initialized NET plugin Socket |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Assigned NET plugin Socket to comm |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Assigned NET plugin Socket to comm |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Assigned NET plugin Socket to comm |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO NET/Socket : Using [0]eth0:10.178.128.103<0> |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Initialized NET plugin Socket |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Assigned NET plugin Socket to comm |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Failed to initialize any GIN plugin |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Using network Socket |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Failed to initialize any GIN plugin |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Using network Socket |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Failed to initialize any GIN plugin |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Failed to initialize any GIN plugin |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Using network Socket |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Using network Socket |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Failed to initialize any GIN plugin |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Failed to initialize any GIN plugin |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Using network Socket |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Failed to initialize any GIN plugin |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Using network Socket |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Using network Socket |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO NET/Plugin: Could not find: libnccl-net.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO NCCL_IB_DISABLE set by environment to 1. |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO transport/net_ib/init.cc:396 -> 3 |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Failed to initialize NET plugin IB |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO NET/Socket : Using [0]eth0:10.178.128.103<0> |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Initialized NET plugin Socket |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Assigned NET plugin Socket to comm |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Failed to initialize any GIN plugin |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Using network Socket |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO [Rank 11] ncclCommInitRankConfig comm 0x5651a9e17960 rank 11 nranks 32 cudaDev 3 nvmlDev 3 busId 4e000 commId 0x7a425254b479deb7 - Init START |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO [Rank 14] ncclCommInitRankConfig comm 0x5604d34c5fa0 rank 14 nranks 32 cudaDev 6 nvmlDev 6 busId db000 commId 0x7a425254b479deb7 - Init START |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO [Rank 13] ncclCommInitRankConfig comm 0x5591c0a3fa30 rank 13 nranks 32 cudaDev 5 nvmlDev 5 busId a1000 commId 0x7a425254b479deb7 - Init START |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO [Rank 9] ncclCommInitRankConfig comm 0x56515db472e0 rank 9 nranks 32 cudaDev 1 nvmlDev 1 busId 10000 commId 0x7a425254b479deb7 - Init START |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO [Rank 15] ncclCommInitRankConfig comm 0x56183b6f0ae0 rank 15 nranks 32 cudaDev 7 nvmlDev 7 busId e0000 commId 0x7a425254b479deb7 - Init START |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO RAS client listening socket at 127.0.0.1<28028> |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO [Rank 10] ncclCommInitRankConfig comm 0x56414f024bb0 rank 10 nranks 32 cudaDev 2 nvmlDev 2 busId 48000 commId 0x7a425254b479deb7 - Init START |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO [Rank 8] ncclCommInitRankConfig comm 0x55de0c9cabb0 rank 8 nranks 32 cudaDev 0 nvmlDev 0 busId b000 commId 0x7a425254b479deb7 - Init START |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO [Rank 12] ncclCommInitRankConfig comm 0x5568eccd24d0 rank 12 nranks 32 cudaDev 4 nvmlDev 4 busId 9b000 commId 0x7a425254b479deb7 - Init START |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO RAS client listening socket at 127.0.0.1<28028> |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO RAS client listening socket at 127.0.0.1<28028> |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028> |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO RAS client listening socket at 127.0.0.1<28028> |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO RAS client listening socket at 127.0.0.1<28028> |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO RAS client listening socket at 127.0.0.1<28028> |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO RAS client listening socket at 127.0.0.1<28028> |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Bootstrap timings total 0.366470 (create 0.000107, send 0.000655, recv 0.032611, ring 0.331141, delay 0.000003) |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Bootstrap timings total 0.334157 (create 0.000082, send 0.000627, recv 0.001341, ring 0.331225, delay 0.000003) |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Bootstrap timings total 0.334732 (create 0.000078, send 0.000501, recv 0.001245, ring 0.332147, delay 0.000003) |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Bootstrap timings total 0.436544 (create 0.000089, send 0.000604, recv 0.104207, ring 0.330540, delay 0.000003) |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Bootstrap timings total 0.334165 (create 0.000132, send 0.000621, recv 0.001859, ring 0.330700, delay 0.000003) |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Bootstrap timings total 0.398533 (create 0.000105, send 0.000639, recv 0.000765, ring 0.330577, delay 0.000003) |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Bootstrap timings total 0.419006 (create 0.000081, send 0.000532, recv 0.071832, ring 0.345517, delay 0.000003) |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Bootstrap timings total 0.348157 (create 0.000091, send 0.000585, recv 0.335943, ring 0.010122, delay 0.000003) |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 5 is 96-111,224-239. (GPU affinity = 96-111,224-239 ; CPU affinity = 0-119,128-247). |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO NVLS multicast support is not available on dev 5 (NVLS_NCHANNELS 0) |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 4 is 96-111,224-239. (GPU affinity = 96-111,224-239 ; CPU affinity = 0-119,128-247). |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO NVLS multicast support is not available on dev 4 (NVLS_NCHANNELS 0) |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 2 is 16-31,144-159. (GPU affinity = 16-31,144-159 ; CPU affinity = 0-119,128-247). |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO NVLS multicast support is not available on dev 2 (NVLS_NCHANNELS 0) |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 6 is 64-79,192-207. (GPU affinity = 64-79,192-207 ; CPU affinity = 0-119,128-247). |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO NVLS multicast support is not available on dev 6 (NVLS_NCHANNELS 0) |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 0 is 48-63,176-191. (GPU affinity = 48-63,176-191 ; CPU affinity = 0-119,128-247). |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 1 is 48-63,176-191. (GPU affinity = 48-63,176-191 ; CPU affinity = 0-119,128-247). |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO NVLS multicast support is not available on dev 0 (NVLS_NCHANNELS 0) |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO NVLS multicast support is not available on dev 1 (NVLS_NCHANNELS 0) |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 7 is 64-79,192-207. (GPU affinity = 64-79,192-207 ; CPU affinity = 0-119,128-247). |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 3 is 16-31,144-159. (GPU affinity = 16-31,144-159 ; CPU affinity = 0-119,128-247). |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO NVLS multicast support is not available on dev 7 (NVLS_NCHANNELS 0) |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO NVLS multicast support is not available on dev 3 (NVLS_NCHANNELS 0) |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO comm 0x5591c0a3fa30 rank 13 nRanks 32 nNodes 4 localRanks 8 localRank 5 MNNVL 0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO comm 0x5604d34c5fa0 rank 14 nRanks 32 nNodes 4 localRanks 8 localRank 6 MNNVL 0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO comm 0x56183b6f0ae0 rank 15 nRanks 32 nNodes 4 localRanks 8 localRank 7 MNNVL 0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Trees [0] 14/-1/-1->13->12 [1] 14/-1/-1->13->12 |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO P2P Chunksize set to 131072 |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO comm 0x5568eccd24d0 rank 12 nRanks 32 nNodes 4 localRanks 8 localRank 4 MNNVL 0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Trees [0] -1/-1/-1->15->14 [1] -1/-1/-1->15->14 |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Trees [0] 15/-1/-1->14->13 [1] 15/-1/-1->14->13 |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO P2P Chunksize set to 131072 |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO P2P Chunksize set to 131072 |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Trees [0] 13/-1/-1->12->11 [1] 13/-1/-1->12->11 |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO P2P Chunksize set to 131072 |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO comm 0x5651a9e17960 rank 11 nRanks 32 nNodes 4 localRanks 8 localRank 3 MNNVL 0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO comm 0x56515db472e0 rank 9 nRanks 32 nNodes 4 localRanks 8 localRank 1 MNNVL 0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO comm 0x55de0c9cabb0 rank 8 nRanks 32 nNodes 4 localRanks 8 localRank 0 MNNVL 0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO comm 0x56414f024bb0 rank 10 nRanks 32 nNodes 4 localRanks 8 localRank 2 MNNVL 0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Trees [0] 9/-1/-1->8->17 [1] 9/0/-1->8->24 |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Trees [0] 10/-1/-1->9->8 [1] 10/16/-1->9->8 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO P2P Chunksize set to 131072 |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO P2P Chunksize set to 131072 |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Trees [0] 12/-1/-1->11->10 [1] 12/-1/-1->11->10 |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO P2P Chunksize set to 131072 |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Trees [0] 11/-1/-1->10->9 [1] 11/-1/-1->10->9 |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO P2P Chunksize set to 131072 |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21114 [0] NCCL INFO [Proxy Service] Device 4 CPU core 230 |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21115 [0] NCCL INFO [Proxy Service UDS] Device 4 CPU core 111 |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21118 [0] NCCL INFO [Proxy Service] Device 5 CPU core 236 |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21116 [0] NCCL INFO [Proxy Service] Device 7 CPU core 73 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21119 [0] NCCL INFO [Proxy Service] Device 0 CPU core 180 |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21117 [0] NCCL INFO [Proxy Service] Device 6 CPU core 205 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21123 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 184 |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21120 [0] NCCL INFO [Proxy Service UDS] Device 7 CPU core 78 |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21121 [0] NCCL INFO [Proxy Service UDS] Device 6 CPU core 206 |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21122 [0] NCCL INFO [Proxy Service UDS] Device 5 CPU core 228 |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21129 [0] NCCL INFO [Proxy Service UDS] Device 3 CPU core 28 |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21124 [0] NCCL INFO [Proxy Service] Device 1 CPU core 189 |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21127 [0] NCCL INFO [Proxy Service] Device 3 CPU core 21 |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21128 [0] NCCL INFO [Proxy Service UDS] Device 2 CPU core 155 |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21125 [0] NCCL INFO [Proxy Service UDS] Device 1 CPU core 185 |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21126 [0] NCCL INFO [Proxy Service] Device 2 CPU core 21 |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Symmetric VA size=80GB |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1 |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Symmetric VA size=80GB |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Symmetric VA size=80GB |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1 |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Symmetric VA size=80GB |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1 |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Symmetric VA size=80GB |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1 |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Symmetric VA size=80GB |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1 |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Symmetric VA size=80GB |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1 |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Symmetric VA size=80GB |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1 |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO ncclCommInitRankConfig comm 0x56515db472e0 rank 9 nranks 32 cudaDev 1 nvmlDev 1 busId 10000 commId 0x7a425254b479deb7 - Init COMPLETE |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO ncclCommInitRankConfig comm 0x5591c0a3fa30 rank 13 nranks 32 cudaDev 5 nvmlDev 5 busId a1000 commId 0x7a425254b479deb7 - Init COMPLETE |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO ncclCommInitRankConfig comm 0x5604d34c5fa0 rank 14 nranks 32 cudaDev 6 nvmlDev 6 busId db000 commId 0x7a425254b479deb7 - Init COMPLETE |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO ncclCommInitRankConfig comm 0x56183b6f0ae0 rank 15 nranks 32 cudaDev 7 nvmlDev 7 busId e0000 commId 0x7a425254b479deb7 - Init COMPLETE |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Init timings - ncclCommInitRankConfig: rank 13 nranks 32 total 1.27 (kernels 0.39, alloc 0.16, bootstrap 0.40, allgathers 0.03, topo 0.27, graphs 0.00, connections 0.01, rest 0.01) |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO ncclCommInitRankConfig comm 0x56414f024bb0 rank 10 nranks 32 cudaDev 2 nvmlDev 2 busId 48000 commId 0x7a425254b479deb7 - Init COMPLETE |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 9 nranks 32 total 1.27 (kernels 0.39, alloc 0.19, bootstrap 0.37, allgathers 0.03, topo 0.28, graphs 0.00, connections 0.01, rest 0.01) |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Init timings - ncclCommInitRankConfig: rank 15 nranks 32 total 1.26 (kernels 0.38, alloc 0.21, bootstrap 0.35, allgathers 0.03, topo 0.28, graphs 0.00, connections 0.01, rest 0.01) |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Init timings - ncclCommInitRankConfig: rank 14 nranks 32 total 1.27 (kernels 0.39, alloc 0.14, bootstrap 0.42, allgathers 0.03, topo 0.27, graphs 0.00, connections 0.01, rest 0.01) |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Init timings - ncclCommInitRankConfig: rank 10 nranks 32 total 1.26 (kernels 0.38, alloc 0.22, bootstrap 0.33, allgathers 0.03, topo 0.27, graphs 0.00, connections 0.01, rest 0.00) |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO ncclCommInitRankConfig comm 0x55de0c9cabb0 rank 8 nranks 32 cudaDev 0 nvmlDev 0 busId b000 commId 0x7a425254b479deb7 - Init COMPLETE |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO ncclCommInitRankConfig comm 0x5651a9e17960 rank 11 nranks 32 cudaDev 3 nvmlDev 3 busId 4e000 commId 0x7a425254b479deb7 - Init COMPLETE |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO ncclCommInitRankConfig comm 0x5568eccd24d0 rank 12 nranks 32 cudaDev 4 nvmlDev 4 busId 9b000 commId 0x7a425254b479deb7 - Init COMPLETE |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 8 nranks 32 total 1.27 (kernels 0.39, alloc 0.22, bootstrap 0.33, allgathers 0.03, topo 0.27, graphs 0.00, connections 0.01, rest 0.01) |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Init timings - ncclCommInitRankConfig: rank 11 nranks 32 total 1.27 (kernels 0.39, alloc 0.12, bootstrap 0.44, allgathers 0.02, topo 0.28, graphs 0.00, connections 0.01, rest 0.01) |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Init timings - ncclCommInitRankConfig: rank 12 nranks 32 total 1.27 (kernels 0.41, alloc 0.21, bootstrap 0.33, allgathers 0.02, topo 0.27, graphs 0.01, connections 0.01, rest 0.01) |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Channel 00/0 : 9[1] -> 10[2] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Channel 01/0 : 9[1] -> 10[2] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Channel 00/0 : 10[2] -> 11[3] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Channel 00/0 : 11[3] -> 12[4] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Channel 01/0 : 10[2] -> 11[3] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Channel 00/0 : 12[4] -> 13[5] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Channel 00/0 : 14[6] -> 15[7] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Channel 01/0 : 11[3] -> 12[4] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 00/0 : 8[0] -> 9[1] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Channel 01/0 : 12[4] -> 13[5] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Channel 01/0 : 14[6] -> 15[7] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Channel 00/0 : 13[5] -> 14[6] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 01/0 : 8[0] -> 9[1] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Channel 01/0 : 13[5] -> 14[6] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Channel 00/0 : 15[7] -> 14[6] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Channel 01/0 : 15[7] -> 14[6] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Channel 00/0 : 10[2] -> 9[1] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Channel 00/0 : 11[3] -> 10[2] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Channel 01/0 : 10[2] -> 9[1] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Channel 01/0 : 11[3] -> 10[2] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Channel 00/0 : 14[6] -> 13[5] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Channel 01/0 : 14[6] -> 13[5] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Channel 00/0 : 13[5] -> 12[4] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Channel 00/0 : 12[4] -> 11[3] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Channel 01/0 : 12[4] -> 11[3] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Channel 01/0 : 13[5] -> 12[4] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21131 [0] NCCL INFO [Proxy Progress] Device 1 CPU core 181 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21130 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 181 |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Channel 01/0 : 9[1] -> 16[0] [send] via NET/Socket/0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 01/0 : 0[0] -> 8[0] [receive] via NET/Socket/0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 00/0 : 8[0] -> 17[1] [send] via NET/Socket/0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Channel 01/0 : 16[0] -> 9[1] [receive] via NET/Socket/0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 01/0 : 24[0] -> 8[0] [receive] via NET/Socket/0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 01/0 : 8[0] -> 24[0] [send] via NET/Socket/0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 00/0 : 17[1] -> 8[0] [receive] via NET/Socket/0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 01/0 : 8[0] -> 0[0] [send] via NET/Socket/0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Channel 00/0 : 9[1] -> 8[0] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Channel 01/0 : 9[1] -> 8[0] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Connected all trees |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Connected all trees |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Connected all trees |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Connected all trees |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Connected all trees |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Connected all trees |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Connected all trees |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Connected all trees |
|
Loading weights: 0%| | 0/399 [00:00<?, ?it/s]
Loading weights: 0%| | 0/399 [00:00<?, ?it/s]
Loading weights: 0%| | 0/399 [00:00<?, ?it/s]
Loading weights: 1%| | 2/399 [00:00<00:23, 16.61it/s]
Loading weights: 0%| | 0/399 [00:00<?, ?it/s]
Loading weights: 1%| | 4/399 [00:00<00:11, 35.69it/s]
Loading weights: 0%| | 0/399 [00:00<?, ?it/s]
Loading weights: 0%| | 0/399 [00:00<?, ?it/s]
Loading weights: 1%| | 2/399 [00:00<00:25, 15.35it/s]
Loading weights: 1%| | 3/399 [00:00<00:13, 29.53it/s]
Loading weights: 0%| | 0/399 [00:00<?, ?it/s]
Loading weights: 0%| | 0/399 [00:00<?, ?it/s]
Loading weights: 1%|▏ | 5/399 [00:00<00:11, 33.27it/s]
Loading weights: 2%|▏ | 7/399 [00:00<00:17, 22.37it/s]
Loading weights: 5%|▍ | 18/399 [00:00<00:02, 128.79it/s]
Loading weights: 2%|▏ | 6/399 [00:00<00:13, 28.11it/s]
Loading weights: 5%|▍ | 19/399 [00:00<00:02, 188.13it/s]
Loading weights: 2%|▏ | 8/399 [00:00<00:18, 21.38it/s]
Loading weights: 2%|▏ | 6/399 [00:00<00:18, 21.82it/s]
Loading weights: 2%|▏ | 9/399 [00:00<00:05, 68.27it/s]
Loading weights: 4%|▍ | 17/399 [00:00<00:07, 49.81it/s]
Loading weights: 3%|▎ | 11/399 [00:00<00:18, 21.39it/s]
Loading weights: 3%|▎ | 10/399 [00:00<00:14, 27.66it/s]
Loading weights: 3%|▎ | 11/399 [00:00<00:18, 21.35it/s]
Loading weights: 3%|▎ | 10/399 [00:00<00:16, 23.77it/s]
Loading weights: 3%|▎ | 13/399 [00:00<00:13, 28.22it/s]
Loading weights: 4%|▍ | 16/399 [00:00<00:09, 41.32it/s]
Loading weights: 4%|▎ | 14/399 [00:00<00:14, 25.97it/s]
Loading weights: 4%|▎ | 14/399 [00:00<00:20, 18.64it/s]
Loading weights: 6%|▌ | 23/399 [00:00<00:10, 36.49it/s]
Loading weights: 4%|▍ | 17/399 [00:00<00:14, 25.80it/s]
Loading weights: 4%|▎ | 14/399 [00:00<00:23, 16.05it/s]
Loading weights: 4%|▍ | 16/399 [00:00<00:17, 21.57it/s]
Loading weights: 5%|▍ | 18/399 [00:00<00:17, 21.95it/s]
Loading weights: 5%|▌ | 21/399 [00:00<00:13, 27.70it/s]
Loading weights: 5%|▌ | 21/399 [00:00<00:18, 20.55it/s]
Loading weights: 7%|▋ | 28/399 [00:00<00:13, 27.74it/s]
Loading weights: 8%|▊ | 31/399 [00:00<00:11, 31.10it/s]
Loading weights: 4%|▍ | 16/399 [00:01<00:29, 12.89it/s]
Loading weights: 5%|▍ | 19/399 [00:00<00:20, 18.23it/s]
Loading weights: 5%|▌ | 20/399 [00:00<00:20, 18.75it/s]
Loading weights: 6%|▋ | 25/399 [00:00<00:14, 25.41it/s]
Loading weights: 10%|▉ | 38/399 [00:00<00:09, 37.99it/s]
Loading weights: 8%|▊ | 32/399 [00:01<00:12, 29.62it/s]
Loading weights: 6%|▌ | 24/399 [00:01<00:18, 19.84it/s]
Loading weights: 6%|▌ | 24/399 [00:01<00:18, 20.67it/s]
Loading weights: 5%|▌ | 21/399 [00:01<00:21, 17.29it/s]
Loading weights: 6%|▌ | 22/399 [00:01<00:21, 17.31it/s]
Loading weights: 10%|▉ | 38/399 [00:01<00:11, 30.74it/s]
Loading weights: 9%|▉ | 36/399 [00:01<00:13, 27.86it/s]
Loading weights: 6%|▋ | 25/399 [00:01<00:17, 21.48it/s]
Loading weights: 7%|▋ | 28/399 [00:01<00:16, 22.01it/s]
Loading weights: 6%|▋ | 25/399 [00:01<00:19, 19.10it/s]
Loading weights: 7%|▋ | 27/399 [00:01<00:21, 17.48it/s]
Loading weights: 10%|█ | 40/399 [00:01<00:12, 29.01it/s]
Loading weights: 8%|▊ | 31/399 [00:01<00:16, 22.42it/s]
Loading weights: 8%|▊ | 31/399 [00:01<00:16, 21.97it/s]
Loading weights: 8%|▊ | 33/399 [00:01<00:13, 27.11it/s]
Loading weights: 11%|█ | 43/399 [00:01<00:13, 26.73it/s]
Loading weights: 7%|▋ | 27/399 [00:01<00:25, 14.87it/s]
Loading weights: 11%|█ | 44/399 [00:01<00:12, 27.97it/s]
Loading weights: 9%|▉ | 37/399 [00:01<00:13, 26.66it/s]
Loading weights: 7%|▋ | 28/399 [00:01<00:23, 15.89it/s]
Loading weights: 7%|▋ | 29/399 [00:01<00:24, 15.37it/s]
Loading weights: 11%|█ | 42/399 [00:01<00:10, 35.52it/s]
Loading weights: 12%|█▏ | 47/399 [00:01<00:13, 26.29it/s]
Loading weights: 9%|▉ | 37/399 [00:01<00:14, 25.35it/s]
Loading weights: 12%|█▏ | 48/399 [00:01<00:12, 27.34it/s]
Loading weights: 12%|█▏ | 48/399 [00:01<00:12, 27.59it/s]
Loading weights: 10%|█ | 40/399 [00:01<00:14, 25.46it/s]
Loading weights: 8%|▊ | 31/399 [00:01<00:21, 16.90it/s]
Loading weights: 8%|▊ | 31/399 [00:01<00:24, 15.29it/s]
Loading weights: 12%|█▏ | 46/399 [00:01<00:11, 30.37it/s]
Loading weights: 12%|█▏ | 46/399 [00:01<00:11, 30.80it/s]
Loading weights: 10%|█ | 40/399 [00:01<00:15, 22.50it/s]
Loading weights: 14%|█▎ | 54/399 [00:01<00:12, 27.75it/s]
Loading weights: 9%|▊ | 34/399 [00:01<00:21, 16.93it/s]
Loading weights: 13%|█▎ | 51/399 [00:01<00:15, 22.23it/s]
Loading weights: 9%|▉ | 35/399 [00:01<00:20, 18.05it/s]
Loading weights: 13%|█▎ | 51/399 [00:01<00:15, 22.55it/s]
Loading weights: 13%|█▎ | 51/399 [00:01<00:10, 33.40it/s]
Loading weights: 14%|█▎ | 54/399 [00:01<00:08, 40.76it/s]
Loading weights: 14%|█▎ | 54/399 [00:01<00:15, 22.51it/s]
Loading weights: 9%|▉ | 36/399 [00:01<00:23, 15.75it/s]
Loading weights: 14%|█▎ | 54/399 [00:01<00:15, 21.91it/s]
Loading weights: 10%|▉ | 38/399 [00:01<00:20, 17.71it/s]
Loading weights: 11%|█ | 43/399 [00:02<00:19, 18.58it/s]
Loading weights: 14%|█▍ | 55/399 [00:02<00:11, 30.01it/s]
Loading weights: 15%|█▍ | 59/399 [00:01<00:13, 25.68it/s]
Loading weights: 10%|█ | 40/399 [00:02<00:19, 18.13it/s]
Loading weights: 14%|█▍ | 57/399 [00:02<00:15, 21.91it/s]
Loading weights: 15%|█▍ | 59/399 [00:01<00:11, 30.19it/s]
Loading weights: 11%|█▏ | 45/399 [00:02<00:20, 17.41it/s]
Loading weights: 14%|█▍ | 57/399 [00:02<00:18, 18.27it/s]
Loading weights: 10%|▉ | 39/399 [00:02<00:25, 14.40it/s]
Loading weights: 11%|█ | 42/399 [00:02<00:19, 17.97it/s]
Loading weights: 15%|█▌ | 61/399 [00:02<00:13, 24.44it/s]
Loading weights: 16%|█▌ | 63/399 [00:02<00:13, 24.27it/s]
Loading weights: 15%|█▍ | 59/399 [00:02<00:13, 24.83it/s]
Loading weights: 11%|█ | 43/399 [00:02<00:18, 19.12it/s]
Loading weights: 12%|█▏ | 48/399 [00:02<00:19, 18.14it/s]
Loading weights: 15%|█▌ | 60/399 [00:02<00:18, 18.48it/s]
Loading weights: 16%|█▌ | 63/399 [00:02<00:13, 25.61it/s]
Loading weights: 16%|█▌ | 64/399 [00:02<00:13, 24.71it/s]
Loading weights: 16%|█▌ | 63/399 [00:02<00:12, 27.61it/s]
Loading weights: 11%|█ | 44/399 [00:02<00:23, 14.85it/s]
Loading weights: 13%|█▎ | 52/399 [00:02<00:16, 21.44it/s]
Loading weights: 16%|█▌ | 62/399 [00:02<00:18, 17.96it/s]
Loading weights: 12%|█▏ | 46/399 [00:02<00:19, 18.37it/s]
Loading weights: 17%|█▋ | 67/399 [00:02<00:15, 21.91it/s]
Loading weights: 17%|█▋ | 67/399 [00:02<00:12, 27.36it/s]
Loading weights: 18%|█▊ | 70/399 [00:02<00:12, 26.49it/s]
Loading weights: 16%|█▌ | 64/399 [00:02<00:18, 17.70it/s]
Loading weights: 12%|█▏ | 49/399 [00:02<00:18, 19.18it/s]
Loading weights: 18%|█▊ | 71/399 [00:02<00:13, 24.10it/s]
Loading weights: 17%|█▋ | 67/399 [00:02<00:15, 21.73it/s]
Loading weights: 12%|█▏ | 49/399 [00:02<00:19, 18.37it/s]
Loading weights: 14%|█▍ | 56/399 [00:02<00:17, 20.05it/s]
Loading weights: 18%|█▊ | 71/399 [00:02<00:12, 26.70it/s]
Loading weights: 19%|█▉ | 77/399 [00:02<00:09, 34.29it/s]
Loading weights: 13%|█▎ | 53/399 [00:02<00:16, 20.35it/s]
Loading weights: 18%|█▊ | 70/399 [00:02<00:15, 21.34it/s]
Loading weights: 19%|█▉ | 75/399 [00:02<00:13, 23.90it/s]
Loading weights: 13%|█▎ | 52/399 [00:02<00:18, 18.85it/s]
Loading weights: 17%|█▋ | 66/399 [00:02<00:22, 14.60it/s]
Loading weights: 19%|█▊ | 74/399 [00:02<00:12, 25.98it/s]
Loading weights: 21%|██ | 82/399 [00:02<00:09, 32.45it/s]
Loading weights: 15%|█▍ | 59/399 [00:03<00:18, 18.57it/s]
Loading weights: 19%|█▉ | 77/399 [00:03<00:11, 26.86it/s]
Loading weights: 14%|█▍ | 55/399 [00:02<00:17, 19.80it/s]
Loading weights: 18%|█▊ | 73/399 [00:02<00:15, 20.40it/s]
Loading weights: 21%|██ | 83/399 [00:02<00:10, 29.31it/s]
Loading weights: 18%|█▊ | 73/399 [00:03<00:14, 22.59it/s]
Loading weights: 14%|█▍ | 56/399 [00:02<00:18, 18.24it/s]
Loading weights: 15%|█▌ | 61/399 [00:03<00:18, 17.90it/s]
Loading weights: 20%|██ | 81/399 [00:03<00:11, 27.97it/s]
Loading weights: 22%|██▏ | 86/399 [00:02<00:10, 28.92it/s]
Loading weights: 19%|█▉ | 76/399 [00:02<00:14, 21.84it/s]
Loading weights: 20%|██ | 80/399 [00:03<00:10, 31.72it/s]
Loading weights: 16%|█▋ | 65/399 [00:03<00:15, 21.64it/s]
Loading weights: 15%|█▍ | 58/399 [00:03<00:18, 18.63it/s]
Loading weights: 15%|█▍ | 58/399 [00:03<00:19, 17.82it/s]
Loading weights: 35%|███▌ | 140/399 [00:03<00:01, 165.30it/s]
Loading weights: 36%|███▌ | 143/399 [00:03<00:01, 139.93it/s]
Loading weights: 100%|██████████| 399/399 [00:03<00:00, 121.18it/s] |
|
Loading weights: 100%|██████████| 399/399 [00:03<00:00, 122.56it/s] |
|
Loading weights: 43%|████▎ | 171/399 [00:03<00:01, 214.25it/s]
Loading weights: 100%|██████████| 399/399 [00:03<00:00, 127.46it/s] |
|
Loading weights: 100%|██████████| 399/399 [00:03<00:00, 127.92it/s] |
|
Loading weights: 100%|██████████| 399/399 [00:03<00:00, 131.71it/s] |
|
Loading weights: 22%|██▏ | 87/399 [00:03<00:11, 26.15it/s]
Loading weights: 55%|█████▍ | 219/399 [00:03<00:00, 333.24it/s]
Loading weights: 100%|██████████| 399/399 [00:03<00:00, 124.81it/s] |
|
Loading weights: 100%|██████████| 399/399 [00:03<00:00, 123.47it/s] |
|
Loading weights: 100%|██████████| 399/399 [00:03<00:00, 128.97it/s] |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Channel 00/0 : 9[1] -> 16[0] [send] via NET/Socket/0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 00/0 : 1[1] -> 8[0] [receive] via NET/Socket/0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 01/0 : 1[1] -> 8[0] [receive] via NET/Socket/0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 00/0 : 8[0] -> 15[7] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 01/0 : 8[0] -> 15[7] via P2P/CUMEM/read |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Connected all rings, use ring PXN 0 GDR 0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Connected all rings, use ring PXN 0 GDR 0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Connected all rings, use ring PXN 0 GDR 0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Connected all rings, use ring PXN 0 GDR 0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Connected all rings, use ring PXN 0 GDR 0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Connected all rings, use ring PXN 0 GDR 0 |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 00/1 : 8[0] -> 0[0] [send] via NET/Socket/0/Shared |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Channel 00/1 : 9[1] -> 0[0] [send] via NET/Socket/0/Shared |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:22249 [0] NCCL INFO [Proxy Progress] Device 7 CPU core 73 |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Channel 00/1 : 15[7] -> 0[0] [send] via NET/Socket/0/Shared |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:22250 [0] NCCL INFO [Proxy Progress] Device 2 CPU core 157 |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Channel 00/1 : 10[2] -> 0[0] [send] via NET/Socket/0/Shared |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:22251 [0] NCCL INFO [Proxy Progress] Device 3 CPU core 16 |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Channel 00/1 : 11[3] -> 0[0] [send] via NET/Socket/0/Shared |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:22253 [0] NCCL INFO [Proxy Progress] Device 5 CPU core 97 |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:22252 [0] NCCL INFO [Proxy Progress] Device 4 CPU core 97 |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Channel 00/1 : 12[4] -> 0[0] [send] via NET/Socket/0/Shared |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Channel 00/1 : 13[5] -> 0[0] [send] via NET/Socket/0/Shared |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:22254 [0] NCCL INFO [Proxy Progress] Device 6 CPU core 67 |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Channel 00/1 : 14[6] -> 0[0] [send] via NET/Socket/0/Shared |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO Channel 01/1 : 0[0] -> 8[0] [receive] via NET/Socket/0/Shared |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO Channel 01/1 : 0[0] -> 11[3] [receive] via NET/Socket/0/Shared |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO Channel 01/1 : 0[0] -> 10[2] [receive] via NET/Socket/0/Shared |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO Channel 01/1 : 0[0] -> 12[4] [receive] via NET/Socket/0/Shared |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO Channel 01/1 : 0[0] -> 14[6] [receive] via NET/Socket/0/Shared |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO Channel 01/1 : 0[0] -> 13[5] [receive] via NET/Socket/0/Shared |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO Channel 01/1 : 0[0] -> 15[7] [receive] via NET/Socket/0/Shared |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO Channel 01/1 : 0[0] -> 9[1] [receive] via NET/Socket/0/Shared |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO comm 0x5568eccd24d0 rank 12 nranks 32 cudaDev 4 busId 9b000 - Destroy COMPLETE |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO comm 0x56515db472e0 rank 9 nranks 32 cudaDev 1 busId 10000 - Destroy COMPLETE |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO comm 0x56183b6f0ae0 rank 15 nranks 32 cudaDev 7 busId e0000 - Destroy COMPLETE |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO comm 0x5604d34c5fa0 rank 14 nranks 32 cudaDev 6 busId db000 - Destroy COMPLETE |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO comm 0x5591c0a3fa30 rank 13 nranks 32 cudaDev 5 busId a1000 - Destroy COMPLETE |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO comm 0x56414f024bb0 rank 10 nranks 32 cudaDev 2 busId 48000 - Destroy COMPLETE |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO comm 0x55de0c9cabb0 rank 8 nranks 32 cudaDev 0 busId b000 - Destroy COMPLETE |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO comm 0x5651a9e17960 rank 11 nranks 32 cudaDev 3 busId 4e000 - Destroy COMPLETE |
| h-liuchonghan-translate-a8nodes-main-w-0:21047:21047 [5] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault |
| h-liuchonghan-translate-a8nodes-main-w-0:21049:21049 [7] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault |
| h-liuchonghan-translate-a8nodes-main-w-0:21042:21042 [0] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault |
| h-liuchonghan-translate-a8nodes-main-w-0:21043:21043 [1] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault |
| h-liuchonghan-translate-a8nodes-main-w-0:21048:21048 [6] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault |
| h-liuchonghan-translate-a8nodes-main-w-0:21044:21044 [2] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault |
| h-liuchonghan-translate-a8nodes-main-w-0:21045:21045 [3] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault |
| h-liuchonghan-translate-a8nodes-main-w-0:21046:21046 [4] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault |
| |
