File size: 11,564 Bytes
4d12519 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 | dsw-251511-c5cfcb8-lwcpt:167971:167971 [1] NCCL INFO cudaDriverVersion 12010
dsw-251511-c5cfcb8-lwcpt:167971:167971 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
dsw-251511-c5cfcb8-lwcpt:167971:167971 [1] NCCL INFO Bootstrap : Using eth0:22.8.12.250<0>
dsw-251511-c5cfcb8-lwcpt:167971:167971 [1] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
dsw-251511-c5cfcb8-lwcpt:167974:167974 [4] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
dsw-251511-c5cfcb8-lwcpt:167974:167974 [4] NCCL INFO NET/Plugin: Using internal network plugin.
dsw-251511-c5cfcb8-lwcpt:167975:167975 [5] NCCL INFO Comm config Blocking set to 1
dsw-251511-c5cfcb8-lwcpt:167975:169648 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
dsw-251511-c5cfcb8-lwcpt:167975:169648 [5] NCCL INFO NCCL_IB_HCA set to mlx5
dsw-251511-c5cfcb8-lwcpt:167975:169648 [5] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:22.8.12.250<0>
dsw-251511-c5cfcb8-lwcpt:167975:169648 [5] NCCL INFO Using non-device net plugin version 0
dsw-251511-c5cfcb8-lwcpt:167975:169648 [5] NCCL INFO Using network IB
dsw-251511-c5cfcb8-lwcpt:167972:169645 [2] NCCL INFO ncclCommInitRank comm 0x562dcdsw-251511-c5cfcb8-lwcpt:167505:169641 [0] NCCL INFO ncclCommInitRank comm 0x555f1c018290 radsw-251511-c5cfcb8-lwcpt:167975:169648 [5] NCCL INFO Setting affinity for GPU 5 todsw-251511-c5cfcb8-dsw-251511-c5cfcb8-lwcpt:167975:169648 [5] NCCL INFO NVLS multicast support is notdsw-251511-c5cfcb8-ldsw-251511-c5cfcb8-lwcpt:167973:169644 [3] NCCL INFO comm 0x56420157b060 rank 3 nRanks 8 nNodes 1 localRanks 8 localRank 3 MNNVL 0
dsw-251511-c5cfcb8-lwcpt:167976:169643 [6] NCCL INFO NCCL_MIN_NCHANNELS set by environment to 4.
dsw-251511-c5cfcb8-lwcpt:167976:169643 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 7/-1/-1->6->5 [11] 7/-1/-1->6->5 [12] 7/-1/-1->6->5 [13] 7/-1/-1->6->5 [14] 7/-1/-1->6->5 [15] 7/-1/-1->6->5
dsw-251511-c5cfcb8-lwcpt:167976:169643 [6] NCCL INFO P2P Chunksize set to 524288
dsw-251511-c5cfcb8-lwcpt:167976:169643 [6] NCCL INFO Channel 00/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167976:169643 [6] NCCL INFO Channel 01/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167976:169643 [6] NCCL INFO Channel 02/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167976:169643 [6] NCCL INFO Channel 03/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167976:169643 [6] NCCL INFO Channel 04/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167976:169643 [6] NCCL INFO Channel 05/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167976:169643 [6] NCCL INFO Channel 06/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167976:169643 [6] NCCL INFO Channel 07/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167976:169643 [6] NCCL INFO Channel 08/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167976:169643 [6] NCCL INFO Channel 09/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167976:169643 [6] NCCL INFO Channel 10/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167976:169643 [6] NCCL INFO Channel 11/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167976:169643 [6] NCCL INFO Channel 12/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167976:169643 [6] NCCL INFO Channel 13/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167976:169643 [6] NCCL INFO Channel 14/0 : 6[6] -> 7[7] via P2P/CUMEM/read
dsw-251511-c5cfcdsw-251511-c5cfcb8-lwcpt:167977:169642 [7] NCCL INFO Channel 15/0 : 7[7] -> 0[0] viadsw-251511-c5cfcb8-lwcpt:167976:169643 [6] NCCL INFO Connected all rings
dsw-251511-c5cfcb8-lwcpt:167974:169647 [4] NCCL INFO Channel 00/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167974:169647 [4] NCCL INFO Channel 01/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167974:169647 [4] NCCL INFO Channel 02/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167974:169647 [4] NCCL INFO Channel 03/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167974:169647 [4] NCCL INFO Channel 04/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167974:169647 [4] NCCL INFO Channel 05/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167974:169647 [4] NCCL INFO Channel 06/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167974:169647 [4] NCCL INFO Channel 07/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167974:169647 [4] NCCL INFO Channel 08/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167974:169647 [4] NCCL INFO Channel 09/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167974:169647 [4] NCCL INFO Channel 10/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167974:169647 [4] NCCL INFO Channel 11/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167974:169647 [4] NCCL INFO Channel 12/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167974:169647 [4] NCCL INFO Channel 13/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167974:169647 [4] NCCL INFO Channel 14/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167974:169647 [4] NCCL INFO Channel 15/0 : 4[4] -> 3[3] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167975:169648 [5] NCCL INFO Connected all trees
dsw-251511-c5cfcb8-lwcpt:167976:169643 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
dsw-251511-c5cfcb8-lwcpt:167976:169643 [6] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 16 p2p channels per peer
dsw-251511-c5cfcb8-lwcpt:167975:169648 [5] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared objdsw-251511-c5cfcb8-lwcpt:167505:169641 [0] NCCL INFO TUNER/Plugin: Pdsw-251511-c5cfcdsw-251511-c5cfcb8-lwcpt:167977:169642 [7] NCCL INFO TUNER/Plugin: Using internadsw-251511-c5cfcdsw-251511-c5cfcb8dsw-251511-c5cfcb8-lwcpt:167505:169641 [0] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
dsw-251511-c5cfcb8-lwcpt:167505:169641 [0] NCCLdsw-251511-c5cfcb8-lwcpt:167972:167972 [2] NCCL INFO Comm config Blocking set to 1
dsw-251511-c5cfcb8-lwcpt:167972:170590 [2] NCCL INFO Using non-device net plugin version 0
dsw-251511-c5cfcdsw-251511-c5cfcb8-lwcpt:167977:170588 [7] NCCL INFO Udsw-251511-c5cfcb8-lwcpt:167972:170590 [2] NCCL INFO ncclCommInitRank comm 0x562df3678100 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId 30 commId 0x3f91aa992886cd66 - Init START
dsw-251511-c5cfcb8-lwcpt:167975:170586 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffffff
dsw-251511-c5cfcb8-lwcpt:167975:170586 [5] NCCL INFO NVLS multicast support is not available on dev 5
dsw-251511-c5cfcdsw-251511-c5cfcb8-lwcpt:167977:170588 [7] NCCL INFO comm 0x55bff7f83c40 rank 7 nRanks 8 nNodes 1 localRanks 8 locadsw-251511-c5cfcb8-lwcpt:167974:170585 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3dsw-251511-c5cfcb8-lwcpt:167505:170584 [0] NCCL INFO Channel 00/16 : 0 1 2 3 4 5 6 7
dsw-251511-c5cfcb8-lwcpt:167505:170584 [0] NCCL INFO Channel 01/16 : 0 1 2 3 4 5 6 7
dsw-251511-c5cfcb8-dsw-251511-c5cfcb8-lwcpt:167972:dsw-251511-c5cfcb8-lwcpt:167977:170588 [7] NCCL Idsw-251511-c5cfcb8-lwcpt:167972:dsw-251511-c5cfcb8-lwcpt:167977:170588 [7] NCCL INFO Channel 00/0 : dsw-251511-c5cfcb8-lwcpt:167972:dsw-251511-c5cfcb8-lwcpt:167977:170588 [7] NCCL INFO Channel 01/0 : dsw-251511-c5cfcb8-lwcpt:167972:dsw-251511-c5cfcb8-lwcpt:167977:170588 [7] NCCL INFO Channel 02/0 : dsw-251511-c5cfcb8-lwcpt:167972:dsw-251511-c5cfcb8-lwcpt:167977:170588 [7] NCCL INFO Channel 03/0 : dsw-251511-c5cfcb8-lwcpt:167972:dsw-251511-c5cfcb8-lwcpt:167977:170588 [7] NCCL INFO Channel 04/0 : dsw-251511-c5cfcb8-lwcpt:167972:dsw-251511-c5cfcb8-lwcpt:167977:170588 [7] NCCL INFO Channel 05/0 : dsw-251511-c5cfcb8-lwcpt:167972:dsw-251511-c5cfcb8-lwcpt:167977:170588 [7] NCCL INFO Channel 06/0 : dsw-251511-c5cfcb8-lwcpt:167972:dsw-251511-c5cfcb8-lwcpt:167977:170588 [7] NCCL INFO Channel 07/0 : dsw-251511-c5cfcb8-lwcpt:167972:dsw-251511-c5cfcb8-lwcpt:167977:170588 [7] NCCL INFO Channel 08/0 : dsw-251511-c5cfcb8-lwcpt:167972:dsw-251511-c5cfcb8-lwcpt:167977:170588 [7] NCCL INFO Channel 09/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167977:170588 [7] NCCL INFO Channel 10/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167977:170588 [7] NCCL INFO Channel 11/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167977:170588 [7] NCCL INFO Channel 12/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167977:170588 [7] NCCL INFO Channel 13/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167977:170588 [7] NCCL INFO Channel 14/0 : 7[7] -> 0[0] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167977:170588 [7] NCCL INFO Channel 15/0 : dsw-251511-c5cfcb8-lwcpt:167975:170586 [5] NCCL INFO Connected all rings
dsw-251511-c5cfcb8-lwcpt:167975:170586 [5] NCCL INFO Channel 00/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167975:170586 [5] NCCL INFO Channel 01/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167975:170586 [5] NCCL INFO Channel 02/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167975:170586 [5] NCCL INFO Channel 03/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167975:170586 [5] NCCL INFO Channel 04/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167975:170586 [5] NCCL INFO Channel 05/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167975:170586 [5] NCCL INFO Channel 06/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167975:170586 [5] NCCL INFO Channel 07/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167975:170586 [5] NCCL INFO Channel 08/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167975:170586 [5] NCCL INFO Channel 09/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167975:170586 [5] NCCL INFO Channel 10/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167975:170586 [5] NCCL INFO Channel 11/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167975:170586 [5] NCCL INFO Channel 12/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167975:170586 [5] NCCL INFO Channel 13/0 : 5[5] -> 4[4] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167976:170589 [6] NCCL INFO Channel 14/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167976:170589 [6] NCCL INFO Channel 15/0 : 6[6] -> 5[5] via P2P/CUMEM/read
dsw-251511-c5cfcb8-lwcpt:167974:170585 [4] NCCL INFO Connected all trees
dsw-251511-c5cfcb8-lwcpt:167974:170585 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
dsw-251511-c5cfcb8-lwcpt:167974:170585 [4] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 16 p2p channels per peer
dsw-251511-c5cfcb8-lwcpt:167973:170587 [3] NCCL INFO ncclCommInitRank comm 0x5642080b6540 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId 40 commId 0x3f91aa992886cd66 - Init COMPLETE
dsw-251511-c5cfcb8-lwcpt:167973:169699 [3] NCCL INFO [Service thread] Connection closed by localRank 3
dsw-251511-c5cfcb8-lwcpt:167973:258299 [3] NCCL INFO comm 0x56420157b060 rank 3 nranks 8 cudaDev 3 busId 40 - Abort COMPLETE
Dev 7 busId 80 - Abort COMPLETE
|