train_node3.log

W0618 14:50:53.210000 23025 site-packages/torch/distributed/run.py:862] 
W0618 14:50:53.210000 23025 site-packages/torch/distributed/run.py:862] *****************************************
W0618 14:50:53.210000 23025 site-packages/torch/distributed/run.py:862] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0618 14:50:53.210000 23025 site-packages/torch/distributed/run.py:862] *****************************************
[W618 14:50:53.098749912 socket.cpp:764] [c10d] The client socket cannot be initialized to connect to [h-liuchonghan-translate-a8nodes-main-m-0.h-liuchonghan-translate-a8nodes-main.hbox-aigc.svc.hbox2-zzzc2-prd.local]:29501 (errno: 97 - Address family not supported by protocol).
W0618 14:51:29.232000 23095 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:29.232000 23096 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:29.232000 23098 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:29.232000 23097 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:29.232000 23092 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:29.232000 23099 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:29.232000 23093 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:29.232000 23094 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:29.347000 23095 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:29.347000 23094 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:29.347000 23093 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:29.347000 23098 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:29.347000 23097 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:29.347000 23092 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:29.347000 23096 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:29.347000 23099 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml
cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yamlcfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml

cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml
cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yamlcfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml

cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml
cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml
[W618 14:51:39.837580499 socket.cpp:764] [c10d] The client socket cannot be initialized to connect to [h-liuchonghan-translate-a8nodes-main-m-0.h-liuchonghan-translate-a8nodes-main.hbox-aigc.svc.hbox2-zzzc2-prd.local]:29501 (errno: 97 - Address family not supported by protocol).
[W618 14:51:39.840459157 socket.cpp:764] [c10d] The client socket cannot be initialized to connect to [h-liuchonghan-translate-a8nodes-main-m-0.h-liuchonghan-translate-a8nodes-main.hbox-aigc.svc.hbox2-zzzc2-prd.local]:29501 (errno: 97 - Address family not supported by protocol).
[W618 14:51:39.841161115 socket.cpp:764] [c10d] The client socket cannot be initialized to connect to [h-liuchonghan-translate-a8nodes-main-m-0.h-liuchonghan-translate-a8nodes-main.hbox-aigc.svc.hbox2-zzzc2-prd.local]:29501 (errno: 97 - Address family not supported by protocol).
[W618 14:51:39.844603505 socket.cpp:764] [c10d] The client socket cannot be initialized to connect to [h-liuchonghan-translate-a8nodes-main-m-0.h-liuchonghan-translate-a8nodes-main.hbox-aigc.svc.hbox2-zzzc2-prd.local]:29501 (errno: 97 - Address family not supported by protocol).
[W618 14:51:39.853779183 socket.cpp:764] [c10d] The client socket cannot be initialized to connect to [h-liuchonghan-translate-a8nodes-main-m-0.h-liuchonghan-translate-a8nodes-main.hbox-aigc.svc.hbox2-zzzc2-prd.local]:29501 (errno: 97 - Address family not supported by protocol).
[W618 14:51:39.854733079 socket.cpp:764] [c10d] The client socket cannot be initialized to connect to [h-liuchonghan-translate-a8nodes-main-m-0.h-liuchonghan-translate-a8nodes-main.hbox-aigc.svc.hbox2-zzzc2-prd.local]:29501 (errno: 97 - Address family not supported by protocol).
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO cudaDriverVersion 13000
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO cudaDriverVersion 13000
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO Bootstrap: Using eth0:10.178.159.168<0>
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO NCCL version 2.29.7+cuda13.2
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO NCCL git version stable b81d6a5a3
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO cudaDriverVersion 13000
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO Bootstrap: Using eth0:10.178.159.168<0>
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO NCCL version 2.29.7+cuda13.2
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO NCCL git version stable b81d6a5a3
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO Bootstrap: Using eth0:10.178.159.168<0>
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO NCCL version 2.29.7+cuda13.2
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO NCCL git version stable b81d6a5a3
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO Comm config Blocking set to 1
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO Comm config Blocking set to 1
[W618 14:51:39.861322966 socket.cpp:764] [c10d] The client socket cannot be initialized to connect to [h-liuchonghan-translate-a8nodes-main-m-0.h-liuchonghan-translate-a8nodes-main.hbox-aigc.svc.hbox2-zzzc2-prd.local]:29501 (errno: 97 - Address family not supported by protocol).
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO Comm config Blocking set to 1
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO cudaDriverVersion 13000
[W618 14:51:39.864920834 socket.cpp:764] [c10d] The client socket cannot be initialized to connect to [h-liuchonghan-translate-a8nodes-main-m-0.h-liuchonghan-translate-a8nodes-main.hbox-aigc.svc.hbox2-zzzc2-prd.local]:29501 (errno: 97 - Address family not supported by protocol).
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO Bootstrap: Using eth0:10.178.159.168<0>
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO NCCL version 2.29.7+cuda13.2
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO NCCL git version stable b81d6a5a3
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO Comm config Blocking set to 1
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO cudaDriverVersion 13000
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO cudaDriverVersion 13000
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Bootstrap: Using eth0:10.178.159.168<0>
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO NCCL version 2.29.7+cuda13.2
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO NCCL git version stable b81d6a5a3
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO Bootstrap: Using eth0:10.178.159.168<0>
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO NCCL version 2.29.7+cuda13.2
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO NCCL git version stable b81d6a5a3
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Comm config Blocking set to 1
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO cudaDriverVersion 13000
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO Comm config Blocking set to 1
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO Bootstrap: Using eth0:10.178.159.168<0>
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO NCCL version 2.29.7+cuda13.2
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO NCCL git version stable b81d6a5a3
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO Comm config Blocking set to 1
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO cudaDriverVersion 13000
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO Bootstrap: Using eth0:10.178.159.168<0>
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO NCCL version 2.29.7+cuda13.2
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO NCCL git version stable b81d6a5a3
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO Comm config Blocking set to 1
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO transport/net_ib/init.cc:396 -> 3
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO Failed to initialize NET plugin IB
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO NET/Socket : Using [0]eth0:10.178.159.168<0>
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO Initialized NET plugin Socket
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO Assigned NET plugin Socket to comm
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO Failed to initialize any GIN plugin
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO Using network Socket
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO transport/net_ib/init.cc:396 -> 3
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO Failed to initialize NET plugin IB
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO NET/Socket : Using [0]eth0:10.178.159.168<0>
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO Initialized NET plugin Socket
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO Assigned NET plugin Socket to comm
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO Failed to initialize any GIN plugin
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO Using network Socket
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO transport/net_ib/init.cc:396 -> 3
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO Failed to initialize NET plugin IB
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO NET/Socket : Using [0]eth0:10.178.159.168<0>
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO Initialized NET plugin Socket
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO Assigned NET plugin Socket to comm
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO Failed to initialize any GIN plugin
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO Using network Socket
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO [Rank 28] ncclCommInitRankConfig comm 0x55c3ac99f730 rank 28 nranks 32 cudaDev 4 nvmlDev 4 busId 91000 commId 0x7a425254b479deb7 - Init START
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO transport/net_ib/init.cc:396 -> 3
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO Failed to initialize NET plugin IB
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO NET/Socket : Using [0]eth0:10.178.159.168<0>
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO Initialized NET plugin Socket
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO Assigned NET plugin Socket to comm
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO Failed to initialize any GIN plugin
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO Using network Socket
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO [Rank 30] ncclCommInitRankConfig comm 0x55f9da72a480 rank 30 nranks 32 cudaDev 6 nvmlDev 6 busId cd000 commId 0x7a425254b479deb7 - Init START
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO transport/net_ib/init.cc:396 -> 3
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Failed to initialize NET plugin IB
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO NET/Socket : Using [0]eth0:10.178.159.168<0>
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Initialized NET plugin Socket
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Assigned NET plugin Socket to comm
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Failed to initialize any GIN plugin
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Using network Socket
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO transport/net_ib/init.cc:396 -> 3
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO Failed to initialize NET plugin IB
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO NET/Socket : Using [0]eth0:10.178.159.168<0>
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO Initialized NET plugin Socket
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO Assigned NET plugin Socket to comm
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO Failed to initialize any GIN plugin
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO Using network Socket
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO transport/net_ib/init.cc:396 -> 3
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO Failed to initialize NET plugin IB
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO NET/Socket : Using [0]eth0:10.178.159.168<0>
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO Initialized NET plugin Socket
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO Assigned NET plugin Socket to comm
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO Failed to initialize any GIN plugin
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO Using network Socket
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO [Rank 27] ncclCommInitRankConfig comm 0x55e83475cf70 rank 27 nranks 32 cudaDev 3 nvmlDev 3 busId 4f000 commId 0x7a425254b479deb7 - Init START
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO [Rank 31] ncclCommInitRankConfig comm 0x55d87ca342e0 rank 31 nranks 32 cudaDev 7 nvmlDev 7 busId d2000 commId 0x7a425254b479deb7 - Init START
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO [Rank 29] ncclCommInitRankConfig comm 0x558ee2eefda0 rank 29 nranks 32 cudaDev 5 nvmlDev 5 busId 97000 commId 0x7a425254b479deb7 - Init START
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO [Rank 25] ncclCommInitRankConfig comm 0x55b547d2a650 rank 25 nranks 32 cudaDev 1 nvmlDev 1 busId 13000 commId 0x7a425254b479deb7 - Init START
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO [Rank 24] ncclCommInitRankConfig comm 0x555be7db0ec0 rank 24 nranks 32 cudaDev 0 nvmlDev 0 busId e000 commId 0x7a425254b479deb7 - Init START
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO transport/net_ib/init.cc:396 -> 3
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO Failed to initialize NET plugin IB
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO NET/Socket : Using [0]eth0:10.178.159.168<0>
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO Initialized NET plugin Socket
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO Assigned NET plugin Socket to comm
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO Failed to initialize any GIN plugin
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO Using network Socket
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO [Rank 26] ncclCommInitRankConfig comm 0x560985a2e2c0 rank 26 nranks 32 cudaDev 2 nvmlDev 2 busId 49000 commId 0x7a425254b479deb7 - Init START
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO Bootstrap timings total 1.661113 (create 0.000084, send 0.034499, recv 0.015181, ring 1.608392, delay 0.000004)
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO Bootstrap timings total 1.637535 (create 0.000093, send 0.010804, recv 0.015769, ring 1.608165, delay 0.000003)
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO Bootstrap timings total 1.656155 (create 0.000090, send 0.000524, recv 0.016438, ring 1.608109, delay 0.000003)
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO Bootstrap timings total 1.650098 (create 0.000080, send 0.000694, recv 0.037369, ring 1.607020, delay 0.000003)
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Bootstrap timings total 1.635468 (create 0.000084, send 0.000396, recv 0.000815, ring 0.004503, delay 0.000003)
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO Bootstrap timings total 1.637684 (create 0.000078, send 0.000473, recv 0.026578, ring 1.607852, delay 0.000003)
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO Bootstrap timings total 1.640929 (create 0.000087, send 0.000471, recv 0.001635, ring 1.624032, delay 0.000002)
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO Bootstrap timings total 1.627117 (create 0.000066, send 0.000562, recv 0.016361, ring 1.607486, delay 0.000004)
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 1 is 48-63,176-191. (GPU affinity = 48-63,176-191 ; CPU affinity = 0-254).
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO NVLS multicast support is not available on dev 1 (NVLS_NCHANNELS 0)
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 3 is 16-31,144-159. (GPU affinity = 16-31,144-159 ; CPU affinity = 0-254).
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO NVLS multicast support is not available on dev 3 (NVLS_NCHANNELS 0)
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 2 is 16-31,144-159. (GPU affinity = 16-31,144-159 ; CPU affinity = 0-254).
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO NVLS multicast support is not available on dev 2 (NVLS_NCHANNELS 0)
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 0 is 48-63,176-191. (GPU affinity = 48-63,176-191 ; CPU affinity = 0-254).
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 6 is 80-95,208-223. (GPU affinity = 80-95,208-223 ; CPU affinity = 0-254).
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 4 is 112-127,240-254. (GPU affinity = 112-127,240-254 ; CPU affinity = 0-254).
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO NVLS multicast support is not available on dev 6 (NVLS_NCHANNELS 0)
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO NVLS multicast support is not available on dev 4 (NVLS_NCHANNELS 0)
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 7 is 80-95,208-223. (GPU affinity = 80-95,208-223 ; CPU affinity = 0-254).
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO NVLS multicast support is not available on dev 7 (NVLS_NCHANNELS 0)
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 5 is 112-127,240-254. (GPU affinity = 112-127,240-254 ; CPU affinity = 0-254).
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO NVLS multicast support is not available on dev 0 (NVLS_NCHANNELS 0)
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO NVLS multicast support is not available on dev 5 (NVLS_NCHANNELS 0)
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO comm 0x55e83475cf70 rank 27 nRanks 32 nNodes 4 localRanks 8 localRank 3 MNNVL 0
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO comm 0x555be7db0ec0 rank 24 nRanks 32 nNodes 4 localRanks 8 localRank 0 MNNVL 0
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO comm 0x55c3ac99f730 rank 28 nRanks 32 nNodes 4 localRanks 8 localRank 4 MNNVL 0
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO comm 0x560985a2e2c0 rank 26 nRanks 32 nNodes 4 localRanks 8 localRank 2 MNNVL 0
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO comm 0x558ee2eefda0 rank 29 nRanks 32 nNodes 4 localRanks 8 localRank 5 MNNVL 0
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO Trees [0] 28/-1/-1->27->26 [1] 28/-1/-1->27->26
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO comm 0x55b547d2a650 rank 25 nRanks 32 nNodes 4 localRanks 8 localRank 1 MNNVL 0
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO P2P Chunksize set to 131072
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Trees [0] 25/-1/-1->24->16 [1] 25/8/-1->24->-1
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO Trees [0] 27/-1/-1->26->25 [1] 27/-1/-1->26->25
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO Trees [0] 29/-1/-1->28->27 [1] 29/-1/-1->28->27
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO Trees [0] 30/-1/-1->29->28 [1] 30/-1/-1->29->28
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO Trees [0] 26/-1/-1->25->24 [1] 26/-1/-1->25->24
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO P2P Chunksize set to 131072
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO P2P Chunksize set to 131072
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO P2P Chunksize set to 131072
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO P2P Chunksize set to 131072
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO P2P Chunksize set to 131072
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO comm 0x55f9da72a480 rank 30 nRanks 32 nNodes 4 localRanks 8 localRank 6 MNNVL 0
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO comm 0x55d87ca342e0 rank 31 nRanks 32 nNodes 4 localRanks 8 localRank 7 MNNVL 0
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO Trees [0] -1/-1/-1->31->30 [1] -1/-1/-1->31->30
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO P2P Chunksize set to 131072
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO Trees [0] 31/-1/-1->30->29 [1] 31/-1/-1->30->29
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO P2P Chunksize set to 131072
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
h-liuchonghan-translate-a8nodes-main-w-2:23093:23164 [0] NCCL INFO [Proxy Service] Device 1 CPU core 178
h-liuchonghan-translate-a8nodes-main-w-2:23099:23165 [0] NCCL INFO [Proxy Service] Device 7 CPU core 218
h-liuchonghan-translate-a8nodes-main-w-2:23096:23166 [0] NCCL INFO [Proxy Service] Device 4 CPU core 242
h-liuchonghan-translate-a8nodes-main-w-2:23098:23167 [0] NCCL INFO [Proxy Service] Device 6 CPU core 209
h-liuchonghan-translate-a8nodes-main-w-2:23099:23168 [0] NCCL INFO [Proxy Service UDS] Device 7 CPU core 83
h-liuchonghan-translate-a8nodes-main-w-2:23092:23171 [0] NCCL INFO [Proxy Service] Device 0 CPU core 188
h-liuchonghan-translate-a8nodes-main-w-2:23094:23169 [0] NCCL INFO [Proxy Service] Device 2 CPU core 150
h-liuchonghan-translate-a8nodes-main-w-2:23093:23173 [0] NCCL INFO [Proxy Service UDS] Device 1 CPU core 184
h-liuchonghan-translate-a8nodes-main-w-2:23092:23178 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 189
h-liuchonghan-translate-a8nodes-main-w-2:23095:23177 [0] NCCL INFO [Proxy Service] Device 3 CPU core 153
h-liuchonghan-translate-a8nodes-main-w-2:23096:23172 [0] NCCL INFO [Proxy Service UDS] Device 4 CPU core 122
h-liuchonghan-translate-a8nodes-main-w-2:23098:23174 [0] NCCL INFO [Proxy Service UDS] Device 6 CPU core 92
h-liuchonghan-translate-a8nodes-main-w-2:23094:23176 [0] NCCL INFO [Proxy Service UDS] Device 2 CPU core 154
h-liuchonghan-translate-a8nodes-main-w-2:23095:23179 [0] NCCL INFO [Proxy Service UDS] Device 3 CPU core 155
h-liuchonghan-translate-a8nodes-main-w-2:23097:23170 [0] NCCL INFO [Proxy Service] Device 5 CPU core 248
h-liuchonghan-translate-a8nodes-main-w-2:23097:23175 [0] NCCL INFO [Proxy Service UDS] Device 5 CPU core 126
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO Symmetric VA size=80GB
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO Symmetric VA size=80GB
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO Symmetric VA size=80GB
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Symmetric VA size=80GB
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO Symmetric VA size=80GB
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO Symmetric VA size=80GB
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO Symmetric VA size=80GB
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO Symmetric VA size=80GB
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO ncclCommInitRankConfig comm 0x560985a2e2c0 rank 26 nranks 32 cudaDev 2 nvmlDev 2 busId 49000 commId 0x7a425254b479deb7 - Init COMPLETE
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO ncclCommInitRankConfig comm 0x55f9da72a480 rank 30 nranks 32 cudaDev 6 nvmlDev 6 busId cd000 commId 0x7a425254b479deb7 - Init COMPLETE
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO Init timings - ncclCommInitRankConfig: rank 26 nranks 32 total 2.13 (kernels 0.18, alloc 0.01, bootstrap 1.63, allgathers 0.14, topo 0.16, graphs 0.00, connections 0.01, rest 0.01)
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO Init timings - ncclCommInitRankConfig: rank 30 nranks 32 total 2.17 (kernels 0.18, alloc 0.01, bootstrap 1.66, allgathers 0.14, topo 0.16, graphs 0.00, connections 0.01, rest 0.01)
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO ncclCommInitRankConfig comm 0x555be7db0ec0 rank 24 nranks 32 cudaDev 0 nvmlDev 0 busId e000 commId 0x7a425254b479deb7 - Init COMPLETE
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO ncclCommInitRankConfig comm 0x55c3ac99f730 rank 28 nranks 32 cudaDev 4 nvmlDev 4 busId 91000 commId 0x7a425254b479deb7 - Init COMPLETE
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 24 nranks 32 total 2.15 (kernels 0.17, alloc 0.02, bootstrap 1.64, allgathers 0.14, topo 0.16, graphs 0.00, connections 0.01, rest 0.01)
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO Init timings - ncclCommInitRankConfig: rank 28 nranks 32 total 2.17 (kernels 0.17, alloc 0.01, bootstrap 1.66, allgathers 0.14, topo 0.16, graphs 0.00, connections 0.01, rest 0.01)
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO ncclCommInitRankConfig comm 0x55b547d2a650 rank 25 nranks 32 cudaDev 1 nvmlDev 1 busId 13000 commId 0x7a425254b479deb7 - Init COMPLETE
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO ncclCommInitRankConfig comm 0x558ee2eefda0 rank 29 nranks 32 cudaDev 5 nvmlDev 5 busId 97000 commId 0x7a425254b479deb7 - Init COMPLETE
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO Init timings - ncclCommInitRankConfig: rank 29 nranks 32 total 2.15 (kernels 0.18, alloc 0.01, bootstrap 1.64, allgathers 0.14, topo 0.16, graphs 0.00, connections 0.01, rest 0.01)
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 25 nranks 32 total 2.15 (kernels 0.18, alloc 0.02, bootstrap 1.64, allgathers 0.15, topo 0.16, graphs 0.00, connections 0.01, rest 0.01)
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO ncclCommInitRankConfig comm 0x55d87ca342e0 rank 31 nranks 32 cudaDev 7 nvmlDev 7 busId d2000 commId 0x7a425254b479deb7 - Init COMPLETE
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO Init timings - ncclCommInitRankConfig: rank 31 nranks 32 total 2.16 (kernels 0.18, alloc 0.02, bootstrap 1.64, allgathers 0.14, topo 0.16, graphs 0.00, connections 0.02, rest 0.01)
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO ncclCommInitRankConfig comm 0x55e83475cf70 rank 27 nranks 32 cudaDev 3 nvmlDev 3 busId 4f000 commId 0x7a425254b479deb7 - Init COMPLETE
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO Init timings - ncclCommInitRankConfig: rank 27 nranks 32 total 2.17 (kernels 0.18, alloc 0.02, bootstrap 1.65, allgathers 0.15, topo 0.16, graphs 0.00, connections 0.02, rest 0.00)
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Channel 00/0 : 24[0] -> 25[1] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Channel 01/0 : 24[0] -> 25[1] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO Channel 00/0 : 30[6] -> 31[7] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO Channel 00/0 : 29[5] -> 30[6] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO Channel 01/0 : 30[6] -> 31[7] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO Channel 01/0 : 29[5] -> 30[6] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO Channel 00/0 : 31[7] -> 30[6] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO Channel 00/0 : 26[2] -> 27[3] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO Channel 01/0 : 31[7] -> 30[6] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO Channel 01/0 : 26[2] -> 27[3] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO Channel 00/0 : 30[6] -> 29[5] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO Channel 01/0 : 30[6] -> 29[5] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO Channel 00/0 : 27[3] -> 28[4] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO Channel 00/0 : 28[4] -> 29[5] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO Channel 01/0 : 27[3] -> 28[4] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO Channel 01/0 : 28[4] -> 29[5] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO Channel 00/0 : 29[5] -> 28[4] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO Channel 00/0 : 27[3] -> 26[2] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO Channel 00/0 : 28[4] -> 27[3] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO Channel 00/0 : 25[1] -> 26[2] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO Channel 01/0 : 29[5] -> 28[4] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO Channel 01/0 : 28[4] -> 27[3] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO Channel 01/0 : 25[1] -> 26[2] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO Channel 01/0 : 27[3] -> 26[2] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO Channel 00/0 : 26[2] -> 25[1] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO Channel 00/0 : 25[1] -> 24[0] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO Channel 01/0 : 26[2] -> 25[1] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO Channel 01/0 : 25[1] -> 24[0] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23092:23180 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 51
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Channel 00/0 : 16[0] -> 24[0] [receive] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Channel 01/0 : 8[0] -> 24[0] [receive] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Channel 01/0 : 24[0] -> 8[0] [send] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Channel 00/0 : 24[0] -> 16[0] [send] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO Connected all trees
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO Connected all trees
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO Connected all trees
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO Connected all trees
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO Connected all trees
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO Connected all trees
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Connected all trees
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO Connected all trees

Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights:   2%|▏         | 6/399 [00:00<00:07, 51.85it/s]
Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights:   0%|          | 1/399 [00:00<00:41,  9.59it/s]
Loading weights:   0%|          | 1/399 [00:00<00:42,  9.47it/s]
Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights:   0%|          | 1/399 [00:00<00:41,  9.59it/s]
Loading weights:   1%|          | 3/399 [00:00<00:16, 23.65it/s]
Loading weights:   3%|▎         | 12/399 [00:00<00:08, 45.11it/s]
Loading weights:   1%|▏         | 5/399 [00:00<00:14, 26.43it/s]
Loading weights:   1%|          | 4/399 [00:00<00:18, 21.16it/s]
Loading weights:   1%|▏         | 5/399 [00:00<00:09, 40.81it/s]
Loading weights:   2%|▏         | 7/399 [00:00<00:13, 29.11it/s]
Loading weights:   2%|▏         | 6/399 [00:00<00:15, 25.06it/s]
Loading weights:   2%|▏         | 6/399 [00:00<00:12, 31.86it/s]
Loading weights:   2%|▏         | 6/399 [00:00<00:10, 36.80it/s]
Loading weights:   3%|▎         | 10/399 [00:00<00:10, 36.37it/s]
Loading weights:   4%|▍         | 17/399 [00:00<00:09, 40.08it/s]
Loading weights:   4%|▍         | 17/399 [00:00<00:06, 55.29it/s]
Loading weights:   3%|▎         | 10/399 [00:00<00:09, 41.80it/s]
Loading weights:   3%|▎         | 10/399 [00:00<00:14, 27.77it/s]
Loading weights:   2%|▏         | 8/399 [00:00<00:17, 22.08it/s]
Loading weights:   3%|▎         | 10/399 [00:00<00:17, 22.12it/s]
Loading weights:   3%|▎         | 13/399 [00:00<00:15, 24.78it/s]
Loading weights:   3%|▎         | 10/399 [00:00<00:17, 22.52it/s]
Loading weights:   4%|▎         | 14/399 [00:00<00:15, 25.46it/s]
Loading weights:   6%|▌         | 23/399 [00:00<00:08, 42.87it/s]
Loading weights:   4%|▍         | 15/399 [00:00<00:12, 29.58it/s]
Loading weights:   3%|▎         | 12/399 [00:00<00:18, 20.66it/s]
Loading weights:   3%|▎         | 13/399 [00:00<00:16, 22.90it/s]
Loading weights:   4%|▍         | 15/399 [00:00<00:13, 28.56it/s]
Loading weights:   4%|▍         | 16/399 [00:00<00:16, 23.14it/s]
Loading weights:   5%|▍         | 18/399 [00:00<00:14, 26.38it/s]
Loading weights:   7%|▋         | 28/399 [00:00<00:09, 39.83it/s]
Loading weights:   6%|▌         | 22/399 [00:00<00:15, 24.26it/s]
Loading weights:   5%|▌         | 21/399 [00:00<00:11, 34.26it/s]
Loading weights:   5%|▌         | 20/399 [00:00<00:13, 27.62it/s]
Loading weights:   4%|▍         | 16/399 [00:00<00:20, 18.88it/s]
Loading weights:   4%|▍         | 15/399 [00:00<00:22, 17.20it/s]
Loading weights:   5%|▍         | 19/399 [00:00<00:20, 18.86it/s]
Loading weights:   7%|▋         | 26/399 [00:00<00:16, 22.73it/s]
Loading weights:   6%|▌         | 23/399 [00:00<00:13, 27.48it/s]
Loading weights:   5%|▌         | 21/399 [00:00<00:18, 20.59it/s]
Loading weights:   5%|▍         | 18/399 [00:00<00:20, 18.92it/s]
Loading weights:   8%|▊         | 33/399 [00:00<00:12, 29.90it/s]
Loading weights:   4%|▍         | 17/399 [00:01<00:24, 15.28it/s]
Loading weights:   6%|▌         | 22/399 [00:01<00:20, 18.79it/s]
Loading weights:   6%|▌         | 24/399 [00:01<00:18, 20.53it/s]
Loading weights:   7%|▋         | 26/399 [00:00<00:15, 23.89it/s]
Loading weights:   5%|▌         | 20/399 [00:00<00:20, 18.17it/s]
Loading weights:   7%|▋         | 29/399 [00:01<00:19, 19.03it/s]
Loading weights:   6%|▋         | 25/399 [00:01<00:17, 21.14it/s]
Loading weights:   5%|▍         | 19/399 [00:01<00:24, 15.64it/s]
Loading weights:   6%|▋         | 25/399 [00:01<00:17, 21.27it/s]
Loading weights:   7%|▋         | 27/399 [00:01<00:16, 22.60it/s]
Loading weights:   9%|▉         | 37/399 [00:01<00:14, 25.51it/s]
Loading weights:   7%|▋         | 29/399 [00:01<00:15, 24.62it/s]
Loading weights:   8%|▊         | 32/399 [00:01<00:17, 20.77it/s]
Loading weights:   6%|▌         | 24/399 [00:01<00:18, 20.09it/s]
Loading weights:   6%|▌         | 22/399 [00:01<00:21, 17.72it/s]
Loading weights:   7%|▋         | 28/399 [00:01<00:17, 21.09it/s]
Loading weights:  10%|█         | 40/399 [00:01<00:15, 22.88it/s]
Loading weights:   6%|▌         | 24/399 [00:01<00:20, 18.06it/s]
Loading weights:   7%|▋         | 28/399 [00:01<00:20, 17.74it/s]
Loading weights:   8%|▊         | 33/399 [00:01<00:13, 26.19it/s]
Loading weights:   7%|▋         | 27/399 [00:01<00:18, 19.93it/s]
Loading weights:   8%|▊         | 32/399 [00:01<00:18, 19.98it/s]
Loading weights:   8%|▊         | 30/399 [00:01<00:21, 16.85it/s]
Loading weights:   9%|▉         | 35/399 [00:01<00:19, 18.57it/s]
Loading weights:  12%|█▏        | 46/399 [00:01<00:11, 29.93it/s]
Loading weights:  10%|▉         | 38/399 [00:01<00:13, 26.36it/s]
Loading weights:   8%|▊         | 33/399 [00:01<00:18, 19.29it/s]
Loading weights:  10%|█         | 40/399 [00:01<00:15, 23.75it/s]
Loading weights:   8%|▊         | 30/399 [00:01<00:24, 14.94it/s]
Loading weights:   9%|▉         | 37/399 [00:01<00:15, 24.12it/s]
Loading weights:   7%|▋         | 29/399 [00:01<00:22, 16.34it/s]
Loading weights:   7%|▋         | 26/399 [00:01<00:25, 14.65it/s]
Loading weights:  13%|█▎        | 50/399 [00:01<00:12, 28.21it/s]
Loading weights:  10%|█         | 41/399 [00:01<00:13, 26.76it/s]
Loading weights:  10%|█         | 40/399 [00:01<00:14, 24.88it/s]
Loading weights:   7%|▋         | 29/399 [00:01<00:20, 17.65it/s]
Loading weights:   9%|▉         | 35/399 [00:01<00:17, 21.07it/s]
Loading weights:  10%|▉         | 39/399 [00:01<00:15, 23.27it/s]
Loading weights:  11%|█         | 43/399 [00:01<00:17, 19.85it/s]
Loading weights:  11%|█         | 44/399 [00:01<00:14, 24.83it/s]
Loading weights:   8%|▊         | 32/399 [00:01<00:23, 15.58it/s]
Loading weights:   8%|▊         | 33/399 [00:01<00:16, 22.62it/s]
Loading weights:  11%|█         | 43/399 [00:01<00:14, 24.07it/s]
Loading weights:  14%|█▎        | 54/399 [00:01<00:14, 24.08it/s]
Loading weights:  12%|█▏        | 47/399 [00:01<00:11, 31.91it/s]
Loading weights:  13%|█▎        | 52/399 [00:01<00:09, 36.93it/s]
Loading weights:  10%|▉         | 39/399 [00:01<00:17, 20.86it/s]
Loading weights:   9%|▉         | 36/399 [00:01<00:17, 20.18it/s]
Loading weights:   9%|▉         | 37/399 [00:01<00:14, 25.76it/s]
Loading weights:  12%|█▏        | 48/399 [00:01<00:14, 23.50it/s]
Loading weights:  15%|█▍        | 58/399 [00:01<00:13, 26.03it/s]
Loading weights:  13%|█▎        | 51/399 [00:01<00:10, 32.79it/s]
Loading weights:  12%|█▏        | 48/399 [00:01<00:13, 25.58it/s]
Loading weights:  10%|▉         | 39/399 [00:01<00:17, 20.94it/s]
Loading weights:  15%|█▌        | 61/399 [00:02<00:12, 26.69it/s]
Loading weights:  11%|█         | 43/399 [00:02<00:16, 21.52it/s]
Loading weights:  10%|█         | 40/399 [00:02<00:15, 23.43it/s]
Loading weights:  13%|█▎        | 51/399 [00:02<00:15, 22.66it/s]
Loading weights:  14%|█▍        | 57/399 [00:01<00:10, 32.18it/s]
Loading weights:  14%|█▍        | 55/399 [00:02<00:10, 34.18it/s]
Loading weights:  11%|█         | 42/399 [00:02<00:15, 22.74it/s]
Loading weights:  16%|█▌        | 64/399 [00:02<00:12, 27.20it/s]
Loading weights:  12%|█▏        | 46/399 [00:02<00:15, 22.96it/s]
Loading weights:  11%|█         | 43/399 [00:02<00:15, 23.16it/s]
Loading weights:  14%|█▍        | 55/399 [00:02<00:14, 23.13it/s]
Loading weights:  13%|█▎        | 51/399 [00:02<00:17, 20.36it/s]
Loading weights:  15%|█▍        | 59/399 [00:02<00:11, 30.73it/s]
Loading weights:  15%|█▌        | 61/399 [00:02<00:12, 27.49it/s]
Loading weights:  12%|█▏        | 46/399 [00:02<00:15, 23.18it/s]
Loading weights:  13%|█▎        | 50/399 [00:02<00:14, 23.49it/s]
Loading weights:  17%|█▋        | 67/399 [00:02<00:13, 24.01it/s]
Loading weights:  14%|█▎        | 54/399 [00:02<00:16, 21.42it/s]
Loading weights:  11%|█▏        | 45/399 [00:02<00:20, 17.04it/s]
Loading weights:  15%|█▍        | 58/399 [00:02<00:17, 20.01it/s]
Loading weights:  18%|█▊        | 70/399 [00:02<00:13, 23.69it/s]
Loading weights:  14%|█▍        | 56/399 [00:02<00:11, 29.32it/s]
Loading weights:  14%|█▎        | 54/399 [00:02<00:10, 34.05it/s]
Loading weights:  14%|█▍        | 57/399 [00:02<00:15, 22.34it/s]
Loading weights:  16%|█▌        | 63/399 [00:02<00:14, 23.95it/s]
Loading weights:  15%|█▌        | 61/399 [00:02<00:16, 20.95it/s]
Loading weights:  18%|█▊        | 73/399 [00:02<00:13, 24.74it/s]
Loading weights:  12%|█▏        | 49/399 [00:02<00:18, 18.85it/s]
Loading weights:  16%|█▋        | 65/399 [00:02<00:15, 21.47it/s]
Loading weights:  15%|█▌        | 60/399 [00:02<00:15, 22.57it/s]
Loading weights:  19%|█▉        | 76/399 [00:02<00:12, 25.73it/s]
Loading weights:  14%|█▎        | 54/399 [00:02<00:14, 24.12it/s]
Loading weights:  17%|█▋        | 69/399 [00:02<00:12, 26.35it/s]
Loading weights:  15%|█▌        | 60/399 [00:02<00:14, 24.15it/s]
Loading weights:  17%|█▋        | 68/399 [00:02<00:15, 21.94it/s]
Loading weights:  16%|█▌        | 64/399 [00:02<00:18, 18.29it/s]
Loading weights:  15%|█▍        | 58/399 [00:02<00:15, 21.91it/s]
Loading weights:  16%|█▌        | 63/399 [00:02<00:13, 24.36it/s]
Loading weights:  16%|█▌        | 63/399 [00:02<00:18, 18.57it/s]
Loading weights:  14%|█▍        | 57/399 [00:02<00:14, 23.03it/s]
Loading weights:  20%|█▉        | 79/399 [00:02<00:15, 20.55it/s]
Loading weights:  18%|█▊        | 73/399 [00:02<00:13, 24.22it/s]
Loading weights:  18%|█▊        | 71/399 [00:02<00:16, 20.04it/s]
Loading weights:  17%|█▋        | 66/399 [00:02<00:18, 17.61it/s]
Loading weights:  16%|█▌        | 64/399 [00:02<00:12, 27.22it/s]
Loading weights:  17%|█▋        | 66/399 [00:02<00:13, 24.17it/s]
Loading weights:  17%|█▋        | 68/399 [00:02<00:14, 22.18it/s]
Loading weights:  19%|█▉        | 76/399 [00:03<00:12, 25.26it/s]
Loading weights:  21%|██        | 84/399 [00:03<00:11, 26.39it/s]
Loading weights:  18%|█▊        | 70/399 [00:03<00:15, 21.06it/s]
Loading weights:  15%|█▌        | 60/399 [00:02<00:16, 20.03it/s]
Loading weights:  17%|█▋        | 69/399 [00:03<00:13, 23.80it/s]
Loading weights:  19%|█▊        | 74/399 [00:02<00:17, 19.04it/s]
Loading weights:  19%|█▉        | 75/399 [00:03<00:10, 30.15it/s]
Loading weights:  20%|█▉        | 79/399 [00:03<00:13, 23.98it/s]
Loading weights:  18%|█▊        | 72/399 [00:03<00:13, 25.14it/s]
Loading weights:  17%|█▋        | 68/399 [00:03<00:14, 22.86it/s]
Loading weights:  22%|██▏       | 88/399 [00:03<00:13, 23.32it/s]
Loading weights:  19%|█▉        | 77/399 [00:03<00:16, 19.61it/s]
Loading weights:  20%|█▉        | 79/399 [00:03<00:10, 31.69it/s]
Loading weights:  68%|██████▊   | 272/399 [00:03<00:00, 432.93it/s]
Loading weights: 100%|██████████| 399/399 [00:03<00:00, 122.03it/s]

Loading weights: 100%|██████████| 399/399 [00:03<00:00, 126.34it/s]

Loading weights: 100%|██████████| 399/399 [00:03<00:00, 126.06it/s]

Loading weights: 100%|██████████| 399/399 [00:03<00:00, 120.89it/s]

Loading weights:  16%|█▌        | 63/399 [00:03<00:19, 17.46it/s]
Loading weights:  18%|█▊        | 73/399 [00:03<00:18, 17.71it/s]
Loading weights:  19%|█▉        | 75/399 [00:03<00:12, 26.26it/s]
Loading weights: 100%|██████████| 399/399 [00:03<00:00, 123.25it/s]

Loading weights: 100%|██████████| 399/399 [00:03<00:00, 118.16it/s]

Loading weights:  34%|███▍      | 135/399 [00:03<00:01, 136.20it/s]
Loading weights: 100%|██████████| 399/399 [00:03<00:00, 120.55it/s]

Loading weights: 100%|██████████| 399/399 [00:03<00:00, 120.81it/s]
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Channel 00/0 : 17[1] -> 24[0] [receive] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Channel 01/0 : 17[1] -> 24[0] [receive] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Channel 00/0 : 24[0] -> 31[7] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Channel 01/0 : 24[0] -> 31[7] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-2:23093:23526 [0] NCCL INFO [Proxy Progress] Device 1 CPU core 181
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO Channel 00/0 : 25[1] -> 0[0] [send] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO Channel 01/0 : 25[1] -> 0[0] [send] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
/llm-align/liuchonghan/env/envs/automodel/lib/python3.12/site-packages/torch/_inductor/select_algorithm.py:4628: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  current_out_size = out_base.storage().size()
/llm-align/liuchonghan/env/envs/automodel/lib/python3.12/site-packages/torch/_inductor/select_algorithm.py:4628: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  current_out_size = out_base.storage().size()
/llm-align/liuchonghan/env/envs/automodel/lib/python3.12/site-packages/torch/_inductor/select_algorithm.py:4628: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  current_out_size = out_base.storage().size()
/llm-align/liuchonghan/env/envs/automodel/lib/python3.12/site-packages/torch/_inductor/select_algorithm.py:4628: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  current_out_size = out_base.storage().size()
/llm-align/liuchonghan/env/envs/automodel/lib/python3.12/site-packages/torch/_inductor/select_algorithm.py:4628: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  current_out_size = out_base.storage().size()
/llm-align/liuchonghan/env/envs/automodel/lib/python3.12/site-packages/torch/_inductor/select_algorithm.py:4628: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  current_out_size = out_base.storage().size()
/llm-align/liuchonghan/env/envs/automodel/lib/python3.12/site-packages/torch/_inductor/select_algorithm.py:4628: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  current_out_size = out_base.storage().size()
/llm-align/liuchonghan/env/envs/automodel/lib/python3.12/site-packages/torch/_inductor/select_algorithm.py:4628: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  current_out_size = out_base.storage().size()
Autotune Choices Stats:
{"num_choices": 7, "num_triton_choices": 7, "best_kernel": "triton_flex_attention_6", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8", "best_time": 2.0670080184936523, "best_triton_pos": 0}
AUTOTUNE flex_attention(1x32x3747x128, 1x32x2048x128, 1x32x2048x128, 1x32x3747, 1x32x3747, 1x1x30, 1x1x30x16, 1x1x30, 1x1x30x16, 3747)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*s37, s37, 1], [32*s37, s37, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_6 2.0670 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8
  triton_flex_attention_2 2.1216 ms 97.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=8
  triton_flex_attention_4 2.6052 ms 79.3% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_5 3.2198 ms 64.2% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_3 69.2817 ms 3.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=8
  triton_flex_attention_0 79.7318 ms 2.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_1 87.4941 ms 2.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 2.0869 seconds and 0.1110 seconds precompiling for 7 choices
Autotune Choices Stats:
{"num_choices": 7, "num_triton_choices": 7, "best_kernel": "triton_flex_attention_6", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8", "best_time": 2.520128011703491, "best_triton_pos": 0}
AUTOTUNE flex_attention(1x32x4396x128, 1x32x2048x128, 1x32x2048x128, 1x32x4396, 1x32x4396, 1x1x35, 1x1x35x16, 1x1x35, 1x1x35x16, 4396)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*s37, s37, 1], [32*s37, s37, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_6 2.5201 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8
  triton_flex_attention_2 2.5562 ms 98.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=8
  triton_flex_attention_4 3.0449 ms 82.8% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_5 3.7991 ms 66.3% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_3 81.0687 ms 3.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=8
  triton_flex_attention_0 93.6508 ms 2.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_1 102.6583 ms 2.5% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 2.1807 seconds and 0.1182 seconds precompiling for 7 choices
Autotune Choices Stats:
{"num_choices": 7, "num_triton_choices": 7, "best_kernel": "triton_flex_attention_6", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8", "best_time": 2.781440019607544, "best_triton_pos": 0}
AUTOTUNE flex_attention(1x32x5089x128, 1x32x2048x128, 1x32x2048x128, 1x32x5089, 1x32x5089, 1x1x40, 1x1x40x16, 1x1x40, 1x1x40x16, 5089)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*s37, s37, 1], [32*s37, s37, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_6 2.7814 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8
  triton_flex_attention_2 2.8198 ms 98.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=8
  triton_flex_attention_4 3.4846 ms 79.8% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_5 4.3413 ms 64.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_3 92.2290 ms 3.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=8
  triton_flex_attention_0 106.0014 ms 2.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_1 116.6964 ms 2.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 2.4511 seconds and 0.0821 seconds precompiling for 7 choices
Autotune Choices Stats:
{"num_choices": 7, "num_triton_choices": 7, "best_kernel": "triton_flex_attention_6", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8", "best_time": 2.959359884262085, "best_triton_pos": 0}
AUTOTUNE flex_attention(1x32x5218x128, 1x32x2048x128, 1x32x2048x128, 1x32x5218, 1x32x5218, 1x1x41, 1x1x41x16, 1x1x41, 1x1x41x16, 5218)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*s37, s37, 1], [32*s37, s37, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_6 2.9594 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8
  triton_flex_attention_2 3.0111 ms 98.3% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=8
  triton_flex_attention_4 3.6014 ms 82.2% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_5 4.4945 ms 65.8% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_3 94.3351 ms 3.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=8
  triton_flex_attention_0 108.2686 ms 2.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_1 120.8303 ms 2.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 2.5401 seconds and 0.0962 seconds precompiling for 7 choices
Autotune Choices Stats:
{"num_choices": 7, "num_triton_choices": 7, "best_kernel": "triton_flex_attention_6", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8", "best_time": 3.922528028488159, "best_triton_pos": 0}
AUTOTUNE flex_attention(1x32x6976x128, 1x32x2048x128, 1x32x2048x128, 1x32x6976, 1x32x6976, 1x1x55, 1x1x55x16, 1x1x55, 1x1x55x16, 6976)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*s37, s37, 1], [32*s37, s37, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_6 3.9225 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8
  triton_flex_attention_2 3.9877 ms 98.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=8
  triton_flex_attention_4 4.7732 ms 82.2% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_5 5.9617 ms 65.8% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_3 125.4410 ms 3.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=8
  triton_flex_attention_0 144.7373 ms 2.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_1 159.4861 ms 2.5% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 3.3062 seconds and 0.1222 seconds precompiling for 7 choices
Autotune Choices Stats:
{"num_choices": 7, "num_triton_choices": 7, "best_kernel": "triton_flex_attention_6", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8", "best_time": 4.138944149017334, "best_triton_pos": 0}
AUTOTUNE flex_attention(1x32x7398x128, 1x32x2048x128, 1x32x2048x128, 1x32x7398, 1x32x7398, 1x1x58, 1x1x58x16, 1x1x58, 1x1x58x16, 7398)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*s37, s37, 1], [32*s37, s37, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_6 4.1389 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8
  triton_flex_attention_2 4.1805 ms 99.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=8
  triton_flex_attention_4 5.0643 ms 81.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_5 6.3068 ms 65.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_3 132.9866 ms 3.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=8
  triton_flex_attention_0 153.4245 ms 2.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_1 168.7180 ms 2.5% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 3.4720 seconds and 0.0796 seconds precompiling for 7 choices
Autotune Choices Stats:
{"num_choices": 7, "num_triton_choices": 7, "best_kernel": "triton_flex_attention_6", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8", "best_time": 4.596672058105469, "best_triton_pos": 0}
AUTOTUNE flex_attention(1x32x8211x128, 1x32x2048x128, 1x32x2048x128, 1x32x8211, 1x32x8211, 1x1x65, 1x1x65x16, 1x1x65, 1x1x65x16, 8211)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*s37, s37, 1], [32*s37, s37, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_6 4.5967 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8
  triton_flex_attention_2 4.6261 ms 99.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=8
  triton_flex_attention_4 5.6120 ms 81.9% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_5 6.9436 ms 66.2% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_3 149.3384 ms 3.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=8
  triton_flex_attention_0 171.6647 ms 2.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_1 188.0713 ms 2.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 3.8756 seconds and 0.1095 seconds precompiling for 7 choices
Autotune Choices Stats:
{"num_choices": 7, "num_triton_choices": 7, "best_kernel": "triton_flex_attention_6", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8", "best_time": 4.606688022613525, "best_triton_pos": 0}
AUTOTUNE flex_attention(1x32x8328x128, 1x32x2048x128, 1x32x2048x128, 1x32x8328, 1x32x8328, 1x1x66, 1x1x66x16, 1x1x66, 1x1x66x16, 8328)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*s37, s37, 1], [32*s37, s37, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_6 4.6067 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8
  triton_flex_attention_2 4.6397 ms 99.3% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=8
  triton_flex_attention_4 5.6684 ms 81.3% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_5 7.0370 ms 65.5% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_3 151.0815 ms 3.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=8
  triton_flex_attention_0 173.9183 ms 2.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_1 191.6995 ms 2.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 3.9164 seconds and 0.1302 seconds precompiling for 7 choices
Autotune Choices Stats:
{"num_choices": 13, "num_triton_choices": 13, "best_kernel": "triton_flex_attention_backward_19", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4", "best_time": 9.601344108581543, "best_triton_pos": 0}
AUTOTUNE flex_attention_backward(1x32x4396x128, 1x32x2048x128, 1x32x2048x128, 1x32x4396, 1x32x4396, 1x32x4396x128, 1x32x4396x128, 1x32x2048x128, 1x1x35, 1x1x35x16, 1x1x16, 1x1x16x35, 1x1x35, 1x1x35x16, 1x1x16, 1x1x16x35, 4396)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*Max(1, s37), Max(1, s37), 1], [32*s37, s37, 1], [4096*Max(1, s37), 128*Max(1, s37), 128, 1], [4096*s37, 128, 4096, 1], [4096*s43, 128*s43, 128, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s56, s56, 1], [s22*s84, s22*s84, s22, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [s99, s99, 1], [s22*s5, s22*s5, s22, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_backward_19 9.6013 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4
  triton_flex_attention_backward_12 9.8102 ms 97.9% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_13 9.8673 ms 97.3% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_14 10.0766 ms 95.3% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_11 10.7134 ms 89.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_15 11.7937 ms 81.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_8 11.8433 ms 81.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_10 11.8523 ms 81.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_9 11.8964 ms 80.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_16 13.7207 ms 70.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 2.3278 seconds and 0.1840 seconds precompiling for 13 choices
Autotune Choices Stats:
{"num_choices": 13, "num_triton_choices": 13, "best_kernel": "triton_flex_attention_backward_19", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4", "best_time": 8.3090238571167, "best_triton_pos": 0}
Autotune Choices Stats:
{"num_choices": 13, "num_triton_choices": 13, "best_kernel": "triton_flex_attention_backward_19", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4", "best_time": 15.383999824523926, "best_triton_pos": 0}
AUTOTUNE flex_attention_backward(1x32x3747x128, 1x32x2048x128, 1x32x2048x128, 1x32x3747, 1x32x3747, 1x32x3747x128, 1x32x3747x128, 1x32x2048x128, 1x1x30, 1x1x30x16, 1x1x16, 1x1x16x30, 1x1x30, 1x1x30x16, 1x1x16, 1x1x16x30, 3747)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*Max(1, s37), Max(1, s37), 1], [32*s37, s37, 1], [4096*Max(1, s37), 128*Max(1, s37), 128, 1], [4096*s37, 128, 4096, 1], [4096*s43, 128*s43, 128, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s56, s56, 1], [s22*s84, s22*s84, s22, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [s99, s99, 1], [s22*s5, s22*s5, s22, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_backward_19 8.3090 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4
  triton_flex_attention_backward_13 8.4509 ms 98.3% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_12 8.4548 ms 98.3% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_14 8.6562 ms 96.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_11 9.1361 ms 90.9% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_15 10.1126 ms 82.2% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_9 10.1545 ms 81.8% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_10 10.2014 ms 81.5% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_8 10.2567 ms 81.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_16 11.7667 ms 70.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 2.3581 seconds and 0.1913 seconds precompiling for 13 choices
AUTOTUNE flex_attention_backward(1x32x6976x128, 1x32x2048x128, 1x32x2048x128, 1x32x6976, 1x32x6976, 1x32x6976x128, 1x32x6976x128, 1x32x2048x128, 1x1x55, 1x1x55x16, 1x1x16, 1x1x16x55, 1x1x55, 1x1x55x16, 1x1x16, 1x1x16x55, 6976)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*Max(1, s37), Max(1, s37), 1], [32*s37, s37, 1], [4096*Max(1, s37), 128*Max(1, s37), 128, 1], [4096*s37, 128, 4096, 1], [4096*s43, 128*s43, 128, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s56, s56, 1], [s22*s84, s22*s84, s22, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [s99, s99, 1], [s22*s5, s22*s5, s22, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_backward_19 15.3840 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4
  triton_flex_attention_backward_12 15.5822 ms 98.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_13 15.6728 ms 98.2% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_14 15.8828 ms 96.9% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_11 17.0598 ms 90.2% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_8 18.8572 ms 81.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_9 18.8664 ms 81.5% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_10 18.8788 ms 81.5% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_15 18.9304 ms 81.3% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_7 20.4811 ms 75.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 2.8335 seconds and 0.2328 seconds precompiling for 13 choices
Autotune Choices Stats:
{"num_choices": 13, "num_triton_choices": 13, "best_kernel": "triton_flex_attention_backward_19", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4", "best_time": 16.32636833190918, "best_triton_pos": 0}
AUTOTUNE flex_attention_backward(1x32x7398x128, 1x32x2048x128, 1x32x2048x128, 1x32x7398, 1x32x7398, 1x32x7398x128, 1x32x7398x128, 1x32x2048x128, 1x1x58, 1x1x58x16, 1x1x16, 1x1x16x58, 1x1x58, 1x1x58x16, 1x1x16, 1x1x16x58, 7398)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*Max(1, s37), Max(1, s37), 1], [32*s37, s37, 1], [4096*Max(1, s37), 128*Max(1, s37), 128, 1], [4096*s37, 128, 4096, 1], [4096*s43, 128*s43, 128, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s56, s56, 1], [s22*s84, s22*s84, s22, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [s99, s99, 1], [s22*s5, s22*s5, s22, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_backward_19 16.3264 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4
  triton_flex_attention_backward_12 16.5317 ms 98.8% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_13 16.6498 ms 98.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_14 17.0993 ms 95.5% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_11 18.0677 ms 90.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_9 19.7461 ms 82.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_10 19.9825 ms 81.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_15 20.0093 ms 81.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_8 20.0158 ms 81.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_7 21.7237 ms 75.2% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 2.9586 seconds and 0.2426 seconds precompiling for 13 choices
Autotune Choices Stats:
{"num_choices": 13, "num_triton_choices": 13, "best_kernel": "triton_flex_attention_backward_19", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4", "best_time": 11.443807601928711, "best_triton_pos": 0}
AUTOTUNE flex_attention_backward(1x32x5218x128, 1x32x2048x128, 1x32x2048x128, 1x32x5218, 1x32x5218, 1x32x5218x128, 1x32x5218x128, 1x32x2048x128, 1x1x41, 1x1x41x16, 1x1x16, 1x1x16x41, 1x1x41, 1x1x41x16, 1x1x16, 1x1x16x41, 5218)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*Max(1, s37), Max(1, s37), 1], [32*s37, s37, 1], [4096*Max(1, s37), 128*Max(1, s37), 128, 1], [4096*s37, 128, 4096, 1], [4096*s43, 128*s43, 128, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s56, s56, 1], [s22*s84, s22*s84, s22, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [s99, s99, 1], [s22*s5, s22*s5, s22, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_backward_19 11.4438 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4
  triton_flex_attention_backward_12 11.5820 ms 98.8% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_13 11.6444 ms 98.3% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_14 11.8923 ms 96.2% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_11 12.6873 ms 90.2% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_9 14.0003 ms 81.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_15 14.0029 ms 81.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_8 14.0669 ms 81.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_10 14.0884 ms 81.2% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_16 16.3234 ms 70.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 2.4232 seconds and 0.1552 seconds precompiling for 13 choices
Autotune Choices Stats:
{"num_choices": 13, "num_triton_choices": 13, "best_kernel": "triton_flex_attention_backward_19", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4", "best_time": 11.261119842529297, "best_triton_pos": 0}
AUTOTUNE flex_attention_backward(1x32x5089x128, 1x32x2048x128, 1x32x2048x128, 1x32x5089, 1x32x5089, 1x32x5089x128, 1x32x5089x128, 1x32x2048x128, 1x1x40, 1x1x40x16, 1x1x16, 1x1x16x40, 1x1x40, 1x1x40x16, 1x1x16, 1x1x16x40, 5089)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*Max(1, s37), Max(1, s37), 1], [32*s37, s37, 1], [4096*Max(1, s37), 128*Max(1, s37), 128, 1], [4096*s37, 128, 4096, 1], [4096*s43, 128*s43, 128, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s56, s56, 1], [s22*s84, s22*s84, s22, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [s99, s99, 1], [s22*s5, s22*s5, s22, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_backward_19 11.2611 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4
  triton_flex_attention_backward_12 11.3733 ms 99.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_13 11.3824 ms 98.9% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_14 11.6578 ms 96.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_11 12.4052 ms 90.8% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_10 13.7058 ms 82.2% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_9 13.7391 ms 82.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_8 13.7461 ms 81.9% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_15 13.8195 ms 81.5% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_16 15.9383 ms 70.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 2.6362 seconds and 0.1962 seconds precompiling for 13 choices
Autotune Choices Stats:
{"num_choices": 13, "num_triton_choices": 13, "best_kernel": "triton_flex_attention_backward_19", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4", "best_time": 18.197471618652344, "best_triton_pos": 0}
AUTOTUNE flex_attention_backward(1x32x8211x128, 1x32x2048x128, 1x32x2048x128, 1x32x8211, 1x32x8211, 1x32x8211x128, 1x32x8211x128, 1x32x2048x128, 1x1x65, 1x1x65x16, 1x1x16, 1x1x16x65, 1x1x65, 1x1x65x16, 1x1x16, 1x1x16x65, 8211)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*Max(1, s37), Max(1, s37), 1], [32*s37, s37, 1], [4096*Max(1, s37), 128*Max(1, s37), 128, 1], [4096*s37, 128, 4096, 1], [4096*s43, 128*s43, 128, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s56, s56, 1], [s22*s84, s22*s84, s22, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [s99, s99, 1], [s22*s5, s22*s5, s22, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_backward_19 18.1975 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4
  triton_flex_attention_backward_13 18.4260 ms 98.8% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_12 18.4368 ms 98.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_14 18.9088 ms 96.2% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_11 20.1213 ms 90.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_9 22.1255 ms 82.2% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_8 22.2547 ms 81.8% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_10 22.2740 ms 81.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_15 22.3125 ms 81.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_7 24.3591 ms 74.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 3.2006 seconds and 0.1525 seconds precompiling for 13 choices
Autotune Choices Stats:
{"num_choices": 13, "num_triton_choices": 13, "best_kernel": "triton_flex_attention_backward_19", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4", "best_time": 18.499168395996094, "best_triton_pos": 0}
AUTOTUNE flex_attention_backward(1x32x8328x128, 1x32x2048x128, 1x32x2048x128, 1x32x8328, 1x32x8328, 1x32x8328x128, 1x32x8328x128, 1x32x2048x128, 1x1x66, 1x1x66x16, 1x1x16, 1x1x16x66, 1x1x66, 1x1x66x16, 1x1x16, 1x1x16x66, 8328)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*Max(1, s37), Max(1, s37), 1], [32*s37, s37, 1], [4096*Max(1, s37), 128*Max(1, s37), 128, 1], [4096*s37, 128, 4096, 1], [4096*s43, 128*s43, 128, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s56, s56, 1], [s22*s84, s22*s84, s22, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [s99, s99, 1], [s22*s5, s22*s5, s22, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_backward_19 18.4992 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4
  triton_flex_attention_backward_12 18.6988 ms 98.9% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_13 18.7300 ms 98.8% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_14 19.1956 ms 96.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_11 20.4240 ms 90.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_9 22.5433 ms 82.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_8 22.5777 ms 81.9% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_10 22.5824 ms 81.9% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_15 22.6349 ms 81.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_7 24.8504 ms 74.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 3.2417 seconds and 0.2461 seconds precompiling for 13 choices
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO Channel 01/1 : 25[1] -> 0[0] [send] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Channel 01/1 : 24[0] -> 0[0] [send] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-2:23098:24931 [0] NCCL INFO [Proxy Progress] Device 6 CPU core 223
h-liuchonghan-translate-a8nodes-main-w-2:23099:24932 [0] NCCL INFO [Proxy Progress] Device 7 CPU core 85
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO Channel 01/1 : 30[6] -> 0[0] [send] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO Channel 01/1 : 31[7] -> 0[0] [send] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-2:23095:24933 [0] NCCL INFO [Proxy Progress] Device 3 CPU core 156
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO Channel 01/1 : 27[3] -> 0[0] [send] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-2:23094:24934 [0] NCCL INFO [Proxy Progress] Device 2 CPU core 23
h-liuchonghan-translate-a8nodes-main-w-2:23097:24935 [0] NCCL INFO [Proxy Progress] Device 5 CPU core 250
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO Channel 01/1 : 26[2] -> 0[0] [send] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO Channel 01/1 : 29[5] -> 0[0] [send] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-2:23096:24936 [0] NCCL INFO [Proxy Progress] Device 4 CPU core 252
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO Channel 01/1 : 28[4] -> 0[0] [send] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO Channel 00/1 : 0[0] -> 24[0] [receive] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO Channel 00/1 : 0[0] -> 29[5] [receive] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO Channel 00/1 : 0[0] -> 31[7] [receive] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO Channel 00/1 : 0[0] -> 28[4] [receive] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO Channel 00/1 : 0[0] -> 30[6] [receive] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO Channel 00/1 : 0[0] -> 26[2] [receive] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO Channel 00/1 : 0[0] -> 27[3] [receive] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO Channel 00/1 : 0[0] -> 25[1] [receive] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO comm 0x55f9da72a480 rank 30 nranks 32 cudaDev 6 busId cd000 - Destroy COMPLETE
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO comm 0x55e83475cf70 rank 27 nranks 32 cudaDev 3 busId 4f000 - Destroy COMPLETE
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO comm 0x560985a2e2c0 rank 26 nranks 32 cudaDev 2 busId 49000 - Destroy COMPLETE
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO comm 0x558ee2eefda0 rank 29 nranks 32 cudaDev 5 busId 97000 - Destroy COMPLETE
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO comm 0x55b547d2a650 rank 25 nranks 32 cudaDev 1 busId 13000 - Destroy COMPLETE
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO comm 0x55c3ac99f730 rank 28 nranks 32 cudaDev 4 busId 91000 - Destroy COMPLETE
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO comm 0x555be7db0ec0 rank 24 nranks 32 cudaDev 0 busId e000 - Destroy COMPLETE
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO comm 0x55d87ca342e0 rank 31 nranks 32 cudaDev 7 busId d2000 - Destroy COMPLETE
h-liuchonghan-translate-a8nodes-main-w-2:23098:23098 [6] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
h-liuchonghan-translate-a8nodes-main-w-2:23095:23095 [3] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
h-liuchonghan-translate-a8nodes-main-w-2:23094:23094 [2] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
h-liuchonghan-translate-a8nodes-main-w-2:23097:23097 [5] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
h-liuchonghan-translate-a8nodes-main-w-2:23093:23093 [1] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
h-liuchonghan-translate-a8nodes-main-w-2:23096:23096 [4] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
h-liuchonghan-translate-a8nodes-main-w-2:23092:23092 [0] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
h-liuchonghan-translate-a8nodes-main-w-2:23099:23099 [7] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault