train_node2.log

W0618 14:50:52.303000 23952 site-packages/torch/distributed/run.py:862] 
W0618 14:50:52.303000 23952 site-packages/torch/distributed/run.py:862] *****************************************
W0618 14:50:52.303000 23952 site-packages/torch/distributed/run.py:862] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0618 14:50:52.303000 23952 site-packages/torch/distributed/run.py:862] *****************************************
[W618 14:50:53.972259801 socket.cpp:764] [c10d] The client socket cannot be initialized to connect to [h-liuchonghan-translate-a8nodes-main-m-0.h-liuchonghan-translate-a8nodes-main.hbox-aigc.svc.hbox2-zzzc2-prd.local]:29501 (errno: 97 - Address family not supported by protocol).
W0618 14:51:30.579000 24019 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.579000 24022 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.579000 24021 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.579000 24026 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.579000 24023 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.579000 24020 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.579000 24024 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.579000 24025 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.793000 24022 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.793000 24020 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.793000 24026 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.793000 24021 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.793000 24023 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.793000 24024 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.793000 24019 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0618 14:51:30.793000 24025 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml
cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml
cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml
cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml
cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml
cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml
cfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yamlcfg-path: examples/speculative/p-eagle/p_eagle_kv_reuse.yaml

[W618 14:51:40.773822829 socket.cpp:764] [c10d] The client socket cannot be initialized to connect to [h-liuchonghan-translate-a8nodes-main-m-0.h-liuchonghan-translate-a8nodes-main.hbox-aigc.svc.hbox2-zzzc2-prd.local]:29501 (errno: 97 - Address family not supported by protocol).
[W618 14:51:40.776196292 socket.cpp:764] [c10d] The client socket cannot be initialized to connect to [h-liuchonghan-translate-a8nodes-main-m-0.h-liuchonghan-translate-a8nodes-main.hbox-aigc.svc.hbox2-zzzc2-prd.local]:29501 (errno: 97 - Address family not supported by protocol).
[W618 14:51:40.787639289 socket.cpp:764] [c10d] The client socket cannot be initialized to connect to [h-liuchonghan-translate-a8nodes-main-m-0.h-liuchonghan-translate-a8nodes-main.hbox-aigc.svc.hbox2-zzzc2-prd.local]:29501 (errno: 97 - Address family not supported by protocol).
[W618 14:51:40.791606446 socket.cpp:764] [c10d] The client socket cannot be initialized to connect to [h-liuchonghan-translate-a8nodes-main-m-0.h-liuchonghan-translate-a8nodes-main.hbox-aigc.svc.hbox2-zzzc2-prd.local]:29501 (errno: 97 - Address family not supported by protocol).
[W618 14:51:40.796026140 socket.cpp:764] [c10d] The client socket cannot be initialized to connect to [h-liuchonghan-translate-a8nodes-main-m-0.h-liuchonghan-translate-a8nodes-main.hbox-aigc.svc.hbox2-zzzc2-prd.local]:29501 (errno: 97 - Address family not supported by protocol).
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO cudaDriverVersion 13000
[W618 14:51:40.798293086 socket.cpp:764] [c10d] The client socket cannot be initialized to connect to [h-liuchonghan-translate-a8nodes-main-m-0.h-liuchonghan-translate-a8nodes-main.hbox-aigc.svc.hbox2-zzzc2-prd.local]:29501 (errno: 97 - Address family not supported by protocol).
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Bootstrap: Using eth0:10.178.160.221<0>
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO NCCL version 2.29.7+cuda13.2
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO NCCL git version stable b81d6a5a3
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Comm config Blocking set to 1
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO cudaDriverVersion 13000
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO Bootstrap: Using eth0:10.178.160.221<0>
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO NCCL version 2.29.7+cuda13.2
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO NCCL git version stable b81d6a5a3
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO Comm config Blocking set to 1
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO cudaDriverVersion 13000
[W618 14:51:40.814452556 socket.cpp:764] [c10d] The client socket cannot be initialized to connect to [h-liuchonghan-translate-a8nodes-main-m-0.h-liuchonghan-translate-a8nodes-main.hbox-aigc.svc.hbox2-zzzc2-prd.local]:29501 (errno: 97 - Address family not supported by protocol).
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO Bootstrap: Using eth0:10.178.160.221<0>
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO NCCL version 2.29.7+cuda13.2
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO NCCL git version stable b81d6a5a3
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO cudaDriverVersion 13000
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO cudaDriverVersion 13000
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO Bootstrap: Using eth0:10.178.160.221<0>
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO NCCL version 2.29.7+cuda13.2
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO NCCL git version stable b81d6a5a3
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO Bootstrap: Using eth0:10.178.160.221<0>
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO NCCL version 2.29.7+cuda13.2
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO NCCL git version stable b81d6a5a3
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO Comm config Blocking set to 1
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO Comm config Blocking set to 1
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO Comm config Blocking set to 1
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO cudaDriverVersion 13000
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Bootstrap: Using eth0:10.178.160.221<0>
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO NCCL version 2.29.7+cuda13.2
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO NCCL git version stable b81d6a5a3
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Comm config Blocking set to 1
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO cudaDriverVersion 13000
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO Bootstrap: Using eth0:10.178.160.221<0>
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO NCCL version 2.29.7+cuda13.2
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO NCCL git version stable b81d6a5a3
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO Comm config Blocking set to 1
[W618 14:51:40.839747982 socket.cpp:764] [c10d] The client socket cannot be initialized to connect to [h-liuchonghan-translate-a8nodes-main-m-0.h-liuchonghan-translate-a8nodes-main.hbox-aigc.svc.hbox2-zzzc2-prd.local]:29501 (errno: 97 - Address family not supported by protocol).
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO cudaDriverVersion 13000
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO Bootstrap: Using eth0:10.178.160.221<0>
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO NCCL version 2.29.7+cuda13.2
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO NCCL git version stable b81d6a5a3
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO Comm config Blocking set to 1
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO transport/net_ib/init.cc:396 -> 3
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO Failed to initialize NET plugin IB
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO transport/net_ib/init.cc:396 -> 3
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Failed to initialize NET plugin IB
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO transport/net_ib/init.cc:396 -> 3
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO Failed to initialize NET plugin IB
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO transport/net_ib/init.cc:396 -> 3
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO Failed to initialize NET plugin IB
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO transport/net_ib/init.cc:396 -> 3
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO Failed to initialize NET plugin IB
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO NET/Socket : Using [0]eth0:10.178.160.221<0>
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO Initialized NET plugin Socket
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO transport/net_ib/init.cc:396 -> 3
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO transport/net_ib/init.cc:396 -> 3
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO Failed to initialize NET plugin IB
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Failed to initialize NET plugin IB
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO Assigned NET plugin Socket to comm
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO NET/Socket : Using [0]eth0:10.178.160.221<0>
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Initialized NET plugin Socket
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO NET/Socket : Using [0]eth0:10.178.160.221<0>
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO Initialized NET plugin Socket
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO NET/Socket : Using [0]eth0:10.178.160.221<0>
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO Initialized NET plugin Socket
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO NET/Socket : Using [0]eth0:10.178.160.221<0>
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO Assigned NET plugin Socket to comm
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO Initialized NET plugin Socket
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Assigned NET plugin Socket to comm
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO Assigned NET plugin Socket to comm
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO Assigned NET plugin Socket to comm
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO transport/net_ib/init.cc:396 -> 3
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO Failed to initialize NET plugin IB
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO NET/Socket : Using [0]eth0:10.178.160.221<0>
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO NET/Socket : Using [0]eth0:10.178.160.221<0>
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Initialized NET plugin Socket
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO Initialized NET plugin Socket
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO Assigned NET plugin Socket to comm
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Assigned NET plugin Socket to comm
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO NET/Socket : Using [0]eth0:10.178.160.221<0>
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO Initialized NET plugin Socket
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO Assigned NET plugin Socket to comm
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO Failed to initialize any GIN plugin
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO Using network Socket
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO Failed to initialize any GIN plugin
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO Using network Socket
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO Failed to initialize any GIN plugin
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO Using network Socket
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Failed to initialize any GIN plugin
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Using network Socket
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO Failed to initialize any GIN plugin
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO Using network Socket
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Failed to initialize any GIN plugin
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO Failed to initialize any GIN plugin
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Using network Socket
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO Using network Socket
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO Failed to initialize any GIN plugin
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO Using network Socket
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO [Rank 18] ncclCommInitRankConfig comm 0x55de3b5c9510 rank 18 nranks 32 cudaDev 2 nvmlDev 2 busId 49000 commId 0x7a425254b479deb7 - Init START
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO [Rank 16] ncclCommInitRankConfig comm 0x5563bd0ab130 rank 16 nranks 32 cudaDev 0 nvmlDev 0 busId e000 commId 0x7a425254b479deb7 - Init START
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO [Rank 17] ncclCommInitRankConfig comm 0x55cc6d5b47b0 rank 17 nranks 32 cudaDev 1 nvmlDev 1 busId 13000 commId 0x7a425254b479deb7 - Init START
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO [Rank 20] ncclCommInitRankConfig comm 0x562c6a8fbb60 rank 20 nranks 32 cudaDev 4 nvmlDev 4 busId 91000 commId 0x7a425254b479deb7 - Init START
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO [Rank 22] ncclCommInitRankConfig comm 0x55fcd06c2a60 rank 22 nranks 32 cudaDev 6 nvmlDev 6 busId cd000 commId 0x7a425254b479deb7 - Init START
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO [Rank 23] ncclCommInitRankConfig comm 0x5628e9cd4dd0 rank 23 nranks 32 cudaDev 7 nvmlDev 7 busId d2000 commId 0x7a425254b479deb7 - Init START
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO [Rank 19] ncclCommInitRankConfig comm 0x55d9f63f4ec0 rank 19 nranks 32 cudaDev 3 nvmlDev 3 busId 4f000 commId 0x7a425254b479deb7 - Init START
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO [Rank 21] ncclCommInitRankConfig comm 0x55f5bce25f20 rank 21 nranks 32 cudaDev 5 nvmlDev 5 busId 97000 commId 0x7a425254b479deb7 - Init START
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO Bootstrap timings total 0.008144 (create 0.000100, send 0.000540, recv 0.001250, ring 0.002949, delay 0.000002)
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO Bootstrap timings total 0.012142 (create 0.000077, send 0.000500, recv 0.005856, ring 0.002904, delay 0.000003)
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO Bootstrap timings total 0.007954 (create 0.000077, send 0.000317, recv 0.002187, ring 0.002870, delay 0.000002)
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO Bootstrap timings total 0.020599 (create 0.000084, send 0.000650, recv 0.013112, ring 0.003288, delay 0.000003)
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Bootstrap timings total 0.012442 (create 0.000079, send 0.000529, recv 0.001696, ring 0.006790, delay 0.000004)
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Bootstrap timings total 0.013131 (create 0.000095, send 0.000642, recv 0.001783, ring 0.007871, delay 0.000003)
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO Bootstrap timings total 0.009725 (create 0.000123, send 0.000510, recv 0.001501, ring 0.002585, delay 0.000004)
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO Bootstrap timings total 0.009764 (create 0.000086, send 0.000455, recv 0.001025, ring 0.004274, delay 0.000003)
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 0 is 48-63,176-191. (GPU affinity = 48-63,176-191 ; CPU affinity = 0-254).
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO NVLS multicast support is not available on dev 0 (NVLS_NCHANNELS 0)
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 6 is 80-95,208-223. (GPU affinity = 80-95,208-223 ; CPU affinity = 0-254).
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO NVLS multicast support is not available on dev 6 (NVLS_NCHANNELS 0)
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 3 is 16-31,144-159. (GPU affinity = 16-31,144-159 ; CPU affinity = 0-254).
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 4 is 112-127,240-254. (GPU affinity = 112-127,240-254 ; CPU affinity = 0-254).
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO NVLS multicast support is not available on dev 4 (NVLS_NCHANNELS 0)
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO NVLS multicast support is not available on dev 3 (NVLS_NCHANNELS 0)
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 5 is 112-127,240-254. (GPU affinity = 112-127,240-254 ; CPU affinity = 0-254).
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 2 is 16-31,144-159. (GPU affinity = 16-31,144-159 ; CPU affinity = 0-254).
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 7 is 80-95,208-223. (GPU affinity = 80-95,208-223 ; CPU affinity = 0-254).
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO NVLS multicast support is not available on dev 5 (NVLS_NCHANNELS 0)
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO NVLS multicast support is not available on dev 2 (NVLS_NCHANNELS 0)
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO NVLS multicast support is not available on dev 7 (NVLS_NCHANNELS 0)
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 1 is 48-63,176-191. (GPU affinity = 48-63,176-191 ; CPU affinity = 0-254).
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO NVLS multicast support is not available on dev 1 (NVLS_NCHANNELS 0)
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO comm 0x5628e9cd4dd0 rank 23 nRanks 32 nNodes 4 localRanks 8 localRank 7 MNNVL 0
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO Trees [0] -1/-1/-1->23->22 [1] -1/-1/-1->23->22
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO P2P Chunksize set to 131072
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO comm 0x55f5bce25f20 rank 21 nRanks 32 nNodes 4 localRanks 8 localRank 5 MNNVL 0
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO comm 0x55fcd06c2a60 rank 22 nRanks 32 nNodes 4 localRanks 8 localRank 6 MNNVL 0
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO Trees [0] 22/-1/-1->21->20 [1] 22/-1/-1->21->20
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO P2P Chunksize set to 131072
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO Trees [0] 23/-1/-1->22->21 [1] 23/-1/-1->22->21
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO P2P Chunksize set to 131072
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO comm 0x562c6a8fbb60 rank 20 nRanks 32 nNodes 4 localRanks 8 localRank 4 MNNVL 0
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO comm 0x55d9f63f4ec0 rank 19 nRanks 32 nNodes 4 localRanks 8 localRank 3 MNNVL 0
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO Trees [0] 21/-1/-1->20->19 [1] 21/-1/-1->20->19
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO Trees [0] 20/-1/-1->19->18 [1] 20/-1/-1->19->18
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO P2P Chunksize set to 131072
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO P2P Chunksize set to 131072
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO comm 0x55de3b5c9510 rank 18 nRanks 32 nNodes 4 localRanks 8 localRank 2 MNNVL 0
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO comm 0x55cc6d5b47b0 rank 17 nRanks 32 nNodes 4 localRanks 8 localRank 1 MNNVL 0
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO Trees [0] 19/-1/-1->18->17 [1] 19/-1/-1->18->17
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO P2P Chunksize set to 131072
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Trees [0] 18/8/-1->17->16 [1] 18/-1/-1->17->16
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO comm 0x5563bd0ab130 rank 16 nRanks 32 nNodes 4 localRanks 8 localRank 0 MNNVL 0
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO P2P Chunksize set to 131072
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Trees [0] 17/24/-1->16->0 [1] 17/-1/-1->16->9
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO P2P Chunksize set to 131072
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
h-liuchonghan-translate-a8nodes-main-w-1:24020:24091 [0] NCCL INFO [Proxy Service] Device 1 CPU core 50
h-liuchonghan-translate-a8nodes-main-w-1:24023:24094 [0] NCCL INFO [Proxy Service] Device 4 CPU core 241
h-liuchonghan-translate-a8nodes-main-w-1:24021:24092 [0] NCCL INFO [Proxy Service] Device 2 CPU core 153
h-liuchonghan-translate-a8nodes-main-w-1:24023:24098 [0] NCCL INFO [Proxy Service UDS] Device 4 CPU core 242
h-liuchonghan-translate-a8nodes-main-w-1:24020:24100 [0] NCCL INFO [Proxy Service UDS] Device 1 CPU core 180
h-liuchonghan-translate-a8nodes-main-w-1:24019:24093 [0] NCCL INFO [Proxy Service] Device 0 CPU core 185
h-liuchonghan-translate-a8nodes-main-w-1:24024:24095 [0] NCCL INFO [Proxy Service] Device 5 CPU core 123
h-liuchonghan-translate-a8nodes-main-w-1:24021:24101 [0] NCCL INFO [Proxy Service UDS] Device 2 CPU core 146
h-liuchonghan-translate-a8nodes-main-w-1:24026:24103 [0] NCCL INFO [Proxy Service] Device 7 CPU core 92
h-liuchonghan-translate-a8nodes-main-w-1:24022:24099 [0] NCCL INFO [Proxy Service] Device 3 CPU core 17
h-liuchonghan-translate-a8nodes-main-w-1:24024:24102 [0] NCCL INFO [Proxy Service UDS] Device 5 CPU core 124
h-liuchonghan-translate-a8nodes-main-w-1:24019:24096 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 187
h-liuchonghan-translate-a8nodes-main-w-1:24026:24106 [0] NCCL INFO [Proxy Service UDS] Device 7 CPU core 93
h-liuchonghan-translate-a8nodes-main-w-1:24022:24105 [0] NCCL INFO [Proxy Service UDS] Device 3 CPU core 147
h-liuchonghan-translate-a8nodes-main-w-1:24025:24097 [0] NCCL INFO [Proxy Service] Device 6 CPU core 222
h-liuchonghan-translate-a8nodes-main-w-1:24025:24104 [0] NCCL INFO [Proxy Service UDS] Device 6 CPU core 219
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO Symmetric VA size=80GB
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO Symmetric VA size=80GB
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO Symmetric VA size=80GB
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO Symmetric VA size=80GB
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO Symmetric VA size=80GB
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Symmetric VA size=80GB
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO Symmetric VA size=80GB
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 1 p2p channels per peer
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Symmetric VA size=80GB
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Symmetric memory is not supported. cuMemEnable 1, globalGinSupport 0, globalNicFused 0 cuMemGdrSupport 1
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO ncclCommInitRankConfig comm 0x5628e9cd4dd0 rank 23 nranks 32 cudaDev 7 nvmlDev 7 busId d2000 commId 0x7a425254b479deb7 - Init COMPLETE
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO ncclCommInitRankConfig comm 0x55d9f63f4ec0 rank 19 nranks 32 cudaDev 3 nvmlDev 3 busId 4f000 commId 0x7a425254b479deb7 - Init COMPLETE
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO Init timings - ncclCommInitRankConfig: rank 23 nranks 32 total 0.95 (kernels 0.18, alloc 0.44, bootstrap 0.01, allgathers 0.16, topo 0.14, graphs 0.00, connections 0.01, rest 0.01)
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO ncclCommInitRankConfig comm 0x55f5bce25f20 rank 21 nranks 32 cudaDev 5 nvmlDev 5 busId 97000 commId 0x7a425254b479deb7 - Init COMPLETE
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO Init timings - ncclCommInitRankConfig: rank 19 nranks 32 total 0.97 (kernels 0.19, alloc 0.45, bootstrap 0.01, allgathers 0.16, topo 0.14, graphs 0.00, connections 0.01, rest 0.01)
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO Init timings - ncclCommInitRankConfig: rank 21 nranks 32 total 0.97 (kernels 0.18, alloc 0.46, bootstrap 0.01, allgathers 0.16, topo 0.14, graphs 0.00, connections 0.01, rest 0.01)
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO ncclCommInitRankConfig comm 0x55fcd06c2a60 rank 22 nranks 32 cudaDev 6 nvmlDev 6 busId cd000 commId 0x7a425254b479deb7 - Init COMPLETE
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO ncclCommInitRankConfig comm 0x55cc6d5b47b0 rank 17 nranks 32 cudaDev 1 nvmlDev 1 busId 13000 commId 0x7a425254b479deb7 - Init COMPLETE
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO ncclCommInitRankConfig comm 0x55de3b5c9510 rank 18 nranks 32 cudaDev 2 nvmlDev 2 busId 49000 commId 0x7a425254b479deb7 - Init COMPLETE
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO Init timings - ncclCommInitRankConfig: rank 22 nranks 32 total 0.93 (kernels 0.18, alloc 0.42, bootstrap 0.01, allgathers 0.16, topo 0.14, graphs 0.00, connections 0.01, rest 0.01)
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 17 nranks 32 total 0.96 (kernels 0.18, alloc 0.44, bootstrap 0.01, allgathers 0.16, topo 0.15, graphs 0.00, connections 0.01, rest 0.01)
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO Init timings - ncclCommInitRankConfig: rank 18 nranks 32 total 0.97 (kernels 0.18, alloc 0.45, bootstrap 0.02, allgathers 0.16, topo 0.14, graphs 0.00, connections 0.01, rest 0.01)
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO ncclCommInitRankConfig comm 0x562c6a8fbb60 rank 20 nranks 32 cudaDev 4 nvmlDev 4 busId 91000 commId 0x7a425254b479deb7 - Init COMPLETE
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO Init timings - ncclCommInitRankConfig: rank 20 nranks 32 total 0.98 (kernels 0.17, alloc 0.47, bootstrap 0.01, allgathers 0.16, topo 0.14, graphs 0.00, connections 0.01, rest 0.00)
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO ncclCommInitRankConfig comm 0x5563bd0ab130 rank 16 nranks 32 cudaDev 0 nvmlDev 0 busId e000 commId 0x7a425254b479deb7 - Init COMPLETE
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 16 nranks 32 total 0.99 (kernels 0.19, alloc 0.46, bootstrap 0.01, allgathers 0.16, topo 0.14, graphs 0.00, connections 0.01, rest 0.00)
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO Channel 00/0 : 21[5] -> 22[6] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO Channel 00/0 : 22[6] -> 23[7] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO Channel 01/0 : 21[5] -> 22[6] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO Channel 01/0 : 22[6] -> 23[7] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Channel 00/0 : 16[0] -> 17[1] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO Channel 00/0 : 23[7] -> 22[6] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO Channel 00/0 : 19[3] -> 20[4] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Channel 01/0 : 16[0] -> 17[1] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO Channel 01/0 : 23[7] -> 22[6] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Channel 00/0 : 17[1] -> 18[2] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO Channel 01/0 : 19[3] -> 20[4] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Channel 01/0 : 17[1] -> 18[2] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO Channel 00/0 : 22[6] -> 21[5] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO Channel 00/0 : 18[2] -> 19[3] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO Channel 01/0 : 22[6] -> 21[5] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO Channel 01/0 : 18[2] -> 19[3] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO Channel 00/0 : 20[4] -> 21[5] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO Channel 00/0 : 18[2] -> 17[1] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO Channel 01/0 : 18[2] -> 17[1] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO Channel 01/0 : 20[4] -> 21[5] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO Channel 00/0 : 19[3] -> 18[2] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO Channel 00/0 : 20[4] -> 19[3] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO Channel 00/0 : 21[5] -> 20[4] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO Channel 01/0 : 19[3] -> 18[2] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO Channel 01/0 : 20[4] -> 19[3] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO Channel 01/0 : 21[5] -> 20[4] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24020:24107 [0] NCCL INFO [Proxy Progress] Device 1 CPU core 53
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Channel 00/0 : 8[0] -> 17[1] [receive] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-1:24019:24108 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 54
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Channel 01/0 : 9[1] -> 16[0] [receive] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Channel 00/0 : 16[0] -> 24[0] [send] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Channel 00/0 : 17[1] -> 8[0] [send] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Channel 00/0 : 0[0] -> 16[0] [receive] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Channel 00/0 : 16[0] -> 0[0] [send] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Channel 00/0 : 24[0] -> 16[0] [receive] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Channel 01/0 : 16[0] -> 9[1] [send] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Channel 00/0 : 17[1] -> 16[0] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Channel 01/0 : 17[1] -> 16[0] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO Connected all trees
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO Connected all trees
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO Connected all trees
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO Connected all trees
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO Connected all trees
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO Connected all trees
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Connected all trees
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Connected all trees

Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights:   1%|          | 2/399 [00:00<00:26, 15.24it/s]
Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights:   1%|          | 3/399 [00:00<00:14, 27.05it/s]
Loading weights:   1%|          | 4/399 [00:00<00:19, 20.46it/s]
Loading weights:   0%|          | 1/399 [00:00<00:43,  9.13it/s]
Loading weights:   2%|▏         | 6/399 [00:00<00:16, 23.25it/s]
Loading weights:   1%|          | 4/399 [00:00<00:18, 20.96it/s]
Loading weights:   1%|          | 4/399 [00:00<00:14, 28.09it/s]
Loading weights:   1%|▏         | 5/399 [00:00<00:14, 27.43it/s]
Loading weights:   1%|▏         | 5/399 [00:00<00:19, 20.11it/s]
Loading weights:   3%|▎         | 10/399 [00:00<00:13, 28.04it/s]
Loading weights:   1%|          | 3/399 [00:00<00:27, 14.35it/s]
Loading weights:   3%|▎         | 10/399 [00:00<00:12, 30.99it/s]
Loading weights:   2%|▏         | 6/399 [00:00<00:25, 15.70it/s]
Loading weights:   2%|▏         | 8/399 [00:00<00:16, 23.73it/s]
Loading weights:   3%|▎         | 12/399 [00:00<00:14, 27.34it/s]
Loading weights:   3%|▎         | 10/399 [00:00<00:10, 36.33it/s]
Loading weights:   2%|▏         | 7/399 [00:00<00:18, 20.76it/s]
Loading weights:   4%|▍         | 16/399 [00:00<00:10, 35.51it/s]
Loading weights:   4%|▍         | 15/399 [00:00<00:13, 27.64it/s]
Loading weights:   3%|▎         | 11/399 [00:00<00:15, 24.92it/s]
Loading weights:   2%|▏         | 8/399 [00:00<00:22, 17.47it/s]
Loading weights:   2%|▏         | 9/399 [00:00<00:22, 17.18it/s]
Loading weights:   5%|▌         | 20/399 [00:00<00:11, 32.03it/s]
Loading weights:   3%|▎         | 10/399 [00:00<00:19, 19.79it/s]
Loading weights:   4%|▎         | 14/399 [00:00<00:16, 23.15it/s]
Loading weights:   3%|▎         | 10/399 [00:00<00:21, 17.81it/s]
Loading weights:   5%|▍         | 18/399 [00:00<00:13, 27.34it/s]
Loading weights:   3%|▎         | 13/399 [00:00<00:18, 21.06it/s]
Loading weights:   6%|▌         | 24/399 [00:00<00:12, 30.84it/s]
Loading weights:   4%|▍         | 15/399 [00:00<00:16, 23.53it/s]
Loading weights:   4%|▎         | 14/399 [00:00<00:20, 18.57it/s]
Loading weights:   4%|▎         | 14/399 [00:00<00:17, 22.28it/s]
Loading weights:   5%|▌         | 21/399 [00:00<00:14, 25.89it/s]
Loading weights:   4%|▎         | 14/399 [00:00<00:19, 19.92it/s]
Loading weights:   4%|▍         | 17/399 [00:00<00:18, 20.92it/s]
Loading weights:   7%|▋         | 28/399 [00:00<00:12, 30.86it/s]
Loading weights:   4%|▍         | 17/399 [00:00<00:20, 18.40it/s]
Loading weights:   4%|▍         | 17/399 [00:00<00:18, 20.92it/s]
Loading weights:   4%|▍         | 17/399 [00:00<00:17, 21.60it/s]
Loading weights:   6%|▌         | 24/399 [00:01<00:17, 21.52it/s]
Loading weights:   5%|▍         | 19/399 [00:00<00:17, 21.18it/s]
Loading weights:   4%|▍         | 16/399 [00:00<00:22, 16.92it/s]
Loading weights:   5%|▌         | 21/399 [00:00<00:18, 20.43it/s]
Loading weights:   5%|▌         | 20/399 [00:01<00:19, 19.19it/s]
Loading weights:   8%|▊         | 32/399 [00:01<00:13, 27.36it/s]
Loading weights:   7%|▋         | 27/399 [00:01<00:16, 23.17it/s]
Loading weights:   5%|▌         | 20/399 [00:00<00:18, 20.55it/s]
Loading weights:   5%|▍         | 18/399 [00:00<00:22, 16.75it/s]
Loading weights:   6%|▌         | 22/399 [00:01<00:19, 19.40it/s]
Loading weights:   9%|▉         | 35/399 [00:01<00:13, 27.42it/s]
Loading weights:   6%|▌         | 23/399 [00:01<00:20, 18.67it/s]
Loading weights:   8%|▊         | 30/399 [00:01<00:17, 21.12it/s]
Loading weights:   6%|▌         | 24/399 [00:01<00:18, 20.65it/s]
Loading weights:   6%|▋         | 25/399 [00:01<00:17, 20.84it/s]
Loading weights:   6%|▌         | 24/399 [00:01<00:21, 17.22it/s]
Loading weights:  10%|▉         | 39/399 [00:01<00:12, 27.99it/s]
Loading weights:   6%|▌         | 23/399 [00:01<00:19, 18.97it/s]
Loading weights:   5%|▌         | 20/399 [00:01<00:29, 13.00it/s]
Loading weights:   7%|▋         | 26/399 [00:01<00:19, 19.45it/s]
Loading weights:   7%|▋         | 28/399 [00:01<00:16, 22.42it/s]
Loading weights:   7%|▋         | 28/399 [00:01<00:17, 21.04it/s]
Loading weights:   7%|▋         | 26/399 [00:01<00:18, 20.23it/s]
Loading weights:   8%|▊         | 33/399 [00:01<00:20, 18.08it/s]
Loading weights:   6%|▌         | 24/399 [00:01<00:23, 16.12it/s]
Loading weights:   7%|▋         | 29/399 [00:01<00:18, 20.09it/s]
Loading weights:   8%|▊         | 33/399 [00:01<00:13, 27.13it/s]
Loading weights:   7%|▋         | 27/399 [00:01<00:21, 17.16it/s]
Loading weights:   8%|▊         | 31/399 [00:01<00:17, 20.55it/s]
Loading weights:  11%|█         | 42/399 [00:01<00:17, 20.60it/s]
Loading weights:   8%|▊         | 30/399 [00:01<00:16, 22.44it/s]
Loading weights:   9%|▉         | 37/399 [00:01<00:17, 21.22it/s]
Loading weights:   7%|▋         | 29/399 [00:01<00:17, 21.06it/s]
Loading weights:   8%|▊         | 30/399 [00:01<00:18, 19.60it/s]
Loading weights:   9%|▉         | 36/399 [00:01<00:14, 25.44it/s]
Loading weights:   9%|▉         | 37/399 [00:01<00:13, 27.82it/s]
Loading weights:  12%|█▏        | 48/399 [00:01<00:13, 26.88it/s]
Loading weights:   8%|▊         | 33/399 [00:01<00:16, 22.83it/s]
Loading weights:  10%|█         | 40/399 [00:01<00:16, 22.04it/s]
Loading weights:   8%|▊         | 32/399 [00:01<00:20, 17.69it/s]
Loading weights:   8%|▊         | 32/399 [00:01<00:17, 21.06it/s]
Loading weights:  10%|█         | 41/399 [00:01<00:11, 30.66it/s]
Loading weights:   9%|▊         | 34/399 [00:01<00:20, 18.06it/s]
Loading weights:  13%|█▎        | 52/399 [00:01<00:12, 27.29it/s]
Loading weights:  11%|█         | 43/399 [00:01<00:16, 22.04it/s]
Loading weights:   8%|▊         | 33/399 [00:01<00:22, 16.57it/s]
Loading weights:  10%|█         | 41/399 [00:01<00:14, 24.82it/s]
Loading weights:  11%|█▏        | 45/399 [00:01<00:12, 28.44it/s]
Loading weights:  10%|█         | 41/399 [00:01<00:12, 28.64it/s]
Loading weights:   9%|▉         | 35/399 [00:01<00:18, 19.20it/s]
Loading weights:   9%|▉         | 35/399 [00:01<00:21, 17.22it/s]
Loading weights:   9%|▉         | 36/399 [00:01<00:22, 15.93it/s]
Loading weights:  12%|█▏        | 47/399 [00:02<00:14, 23.56it/s]
Loading weights:  11%|█         | 44/399 [00:02<00:16, 21.85it/s]
Loading weights:  14%|█▍        | 55/399 [00:02<00:15, 21.75it/s]
Loading weights:  10%|▉         | 38/399 [00:02<00:17, 20.09it/s]
Loading weights:  10%|▉         | 38/399 [00:01<00:19, 18.65it/s]
Loading weights:  13%|█▎        | 50/399 [00:02<00:10, 33.29it/s]
Loading weights:  12%|█▏        | 47/399 [00:02<00:16, 21.99it/s]
Loading weights:  12%|█▏        | 49/399 [00:02<00:15, 22.03it/s]
Loading weights:  10%|█         | 41/399 [00:02<00:17, 20.87it/s]
Loading weights:  10%|▉         | 38/399 [00:02<00:26, 13.55it/s]
Loading weights:  13%|█▎        | 52/399 [00:02<00:14, 23.28it/s]
Loading weights:  10%|█         | 41/399 [00:02<00:18, 19.13it/s]
Loading weights:  14%|█▍        | 55/399 [00:02<00:09, 34.71it/s]
Loading weights:  11%|█         | 44/399 [00:02<00:17, 20.47it/s]
Loading weights:  13%|█▎        | 52/399 [00:02<00:16, 20.75it/s]
Loading weights:  15%|█▍        | 59/399 [00:02<00:09, 34.02it/s]
Loading weights:  11%|█         | 44/399 [00:02<00:18, 19.22it/s]
Loading weights:  15%|█▍        | 58/399 [00:02<00:21, 15.91it/s]
Loading weights:  11%|█         | 44/399 [00:02<00:19, 18.26it/s]
Loading weights:  14%|█▍        | 56/399 [00:02<00:15, 22.41it/s]
Loading weights:  13%|█▎        | 50/399 [00:02<00:18, 18.78it/s]
Loading weights:  16%|█▌        | 62/399 [00:02<00:17, 19.61it/s]
Loading weights:  12%|█▏        | 46/399 [00:02<00:18, 18.83it/s]
Loading weights:  12%|█▏        | 47/399 [00:02<00:17, 19.66it/s]
Loading weights:  15%|█▌        | 61/399 [00:02<00:13, 24.80it/s]
Loading weights:  14%|█▍        | 55/399 [00:02<00:18, 18.69it/s]
Loading weights:  14%|█▍        | 57/399 [00:02<00:13, 25.44it/s]
Loading weights:  12%|█▏        | 47/399 [00:02<00:20, 17.22it/s]
Loading weights:  16%|█▌        | 63/399 [00:02<00:12, 27.69it/s]
Loading weights:  12%|█▏        | 49/399 [00:02<00:17, 20.00it/s]
Loading weights:  13%|█▎        | 50/399 [00:02<00:16, 20.70it/s]
Loading weights:  15%|█▍        | 58/399 [00:02<00:16, 20.70it/s]
Loading weights:  15%|█▌        | 60/399 [00:02<00:13, 25.87it/s]
Loading weights:  16%|█▋        | 65/399 [00:02<00:17, 18.68it/s]
Loading weights:  16%|█▌        | 64/399 [00:02<00:13, 24.52it/s]
Loading weights:  12%|█▏        | 49/399 [00:02<00:20, 17.44it/s]
Loading weights:  17%|█▋        | 66/399 [00:02<00:13, 25.11it/s]
Loading weights:  17%|█▋        | 68/399 [00:02<00:15, 20.78it/s]
Loading weights:  13%|█▎        | 52/399 [00:02<00:17, 20.04it/s]
Loading weights:  15%|█▌        | 61/399 [00:02<00:15, 22.06it/s]
Loading weights:  13%|█▎        | 53/399 [00:02<00:17, 19.84it/s]
Loading weights:  13%|█▎        | 51/399 [00:02<00:20, 17.34it/s]
Loading weights:  17%|█▋        | 67/399 [00:02<00:15, 21.84it/s]
Loading weights:  15%|█▍        | 58/399 [00:02<00:11, 28.55it/s]
Loading weights:  16%|█▋        | 65/399 [00:02<00:13, 25.40it/s]
Loading weights:  18%|█▊        | 71/399 [00:02<00:11, 28.11it/s]
Loading weights:  18%|█▊        | 73/399 [00:02<00:13, 24.91it/s]
Loading weights:  16%|█▌        | 63/399 [00:02<00:16, 20.31it/s]
Loading weights:  14%|█▍        | 56/399 [00:02<00:16, 20.95it/s]
Loading weights:  14%|█▎        | 54/399 [00:02<00:18, 18.17it/s]
Loading weights:  18%|█▊        | 70/399 [00:03<00:14, 22.46it/s]
Loading weights:  16%|█▌        | 62/399 [00:02<00:11, 29.47it/s]
Loading weights:  19%|█▉        | 76/399 [00:03<00:12, 25.34it/s]
Loading weights:  17%|█▋        | 68/399 [00:02<00:13, 24.51it/s]
Loading weights:  17%|█▋        | 67/399 [00:03<00:15, 22.11it/s]
Loading weights:  16%|█▌        | 62/399 [00:03<00:12, 26.94it/s]
Loading weights:  16%|█▌        | 62/399 [00:03<00:11, 29.95it/s]
Loading weights:  19%|█▊        | 74/399 [00:03<00:15, 21.23it/s]
Loading weights:  20%|█▉        | 79/399 [00:03<00:13, 23.67it/s]
Loading weights:  18%|█▊        | 71/399 [00:03<00:14, 22.93it/s]
Loading weights:  17%|█▋        | 69/399 [00:03<00:09, 35.11it/s]
Loading weights:  17%|█▋        | 66/399 [00:03<00:10, 31.50it/s]
Loading weights:  19%|█▊        | 74/399 [00:03<00:11, 27.30it/s]
Loading weights:  18%|█▊        | 73/399 [00:03<00:18, 17.86it/s]
Loading weights: 100%|██████████| 399/399 [00:03<00:00, 123.53it/s]

Loading weights: 100%|██████████| 399/399 [00:03<00:00, 122.85it/s]

Loading weights:  21%|██        | 82/399 [00:03<00:12, 25.03it/s]
Loading weights:  19%|█▊        | 74/399 [00:03<00:13, 24.50it/s]
Loading weights: 100%|██████████| 399/399 [00:03<00:00, 119.47it/s]

Loading weights: 100%|██████████| 399/399 [00:03<00:00, 118.99it/s]

Loading weights:  19%|█▉        | 77/399 [00:03<00:15, 21.25it/s]
Loading weights: 100%|██████████| 399/399 [00:03<00:00, 124.69it/s]

Loading weights:  17%|█▋        | 66/399 [00:03<00:15, 21.04it/s]
Loading weights: 100%|██████████| 399/399 [00:03<00:00, 123.85it/s]

Loading weights:  18%|█▊        | 70/399 [00:03<00:10, 32.52it/s]
Loading weights: 100%|██████████| 399/399 [00:03<00:00, 120.68it/s]

Loading weights: 100%|██████████| 399/399 [00:03<00:00, 122.40it/s]
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Channel 00/0 : 9[1] -> 16[0] [receive] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Channel 00/0 : 16[0] -> 23[7] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Channel 00/0 : 17[1] -> 24[0] [send] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Channel 01/0 : 17[1] -> 24[0] [send] via NET/Socket/0
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Channel 01/0 : 16[0] -> 23[7] via P2P/CUMEM/read
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 0
/llm-align/liuchonghan/env/envs/automodel/lib/python3.12/site-packages/torch/_inductor/select_algorithm.py:4628: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  current_out_size = out_base.storage().size()
/llm-align/liuchonghan/env/envs/automodel/lib/python3.12/site-packages/torch/_inductor/select_algorithm.py:4628: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  current_out_size = out_base.storage().size()
/llm-align/liuchonghan/env/envs/automodel/lib/python3.12/site-packages/torch/_inductor/select_algorithm.py:4628: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  current_out_size = out_base.storage().size()
/llm-align/liuchonghan/env/envs/automodel/lib/python3.12/site-packages/torch/_inductor/select_algorithm.py:4628: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  current_out_size = out_base.storage().size()
/llm-align/liuchonghan/env/envs/automodel/lib/python3.12/site-packages/torch/_inductor/select_algorithm.py:4628: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  current_out_size = out_base.storage().size()
/llm-align/liuchonghan/env/envs/automodel/lib/python3.12/site-packages/torch/_inductor/select_algorithm.py:4628: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  current_out_size = out_base.storage().size()
/llm-align/liuchonghan/env/envs/automodel/lib/python3.12/site-packages/torch/_inductor/select_algorithm.py:4628: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  current_out_size = out_base.storage().size()
/llm-align/liuchonghan/env/envs/automodel/lib/python3.12/site-packages/torch/_inductor/select_algorithm.py:4628: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  current_out_size = out_base.storage().size()
Autotune Choices Stats:
{"num_choices": 7, "num_triton_choices": 7, "best_kernel": "triton_flex_attention_6", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8", "best_time": 3.2159039974212646, "best_triton_pos": 0}
AUTOTUNE flex_attention(1x32x5531x128, 1x32x2048x128, 1x32x2048x128, 1x32x5531, 1x32x5531, 1x1x44, 1x1x44x16, 1x1x44, 1x1x44x16, 5531)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*s37, s37, 1], [32*s37, s37, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_6 3.2159 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8
  triton_flex_attention_2 3.2401 ms 99.3% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=8
  triton_flex_attention_4 3.7833 ms 85.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_5 4.6468 ms 69.2% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_3 101.3400 ms 3.2% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=8
  triton_flex_attention_0 116.7961 ms 2.8% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_1 129.0050 ms 2.5% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 2.7037 seconds and 0.0910 seconds precompiling for 7 choices
Autotune Choices Stats:
{"num_choices": 7, "num_triton_choices": 7, "best_kernel": "triton_flex_attention_6", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8", "best_time": 3.910207986831665, "best_triton_pos": 0}
AUTOTUNE flex_attention(1x32x7023x128, 1x32x2048x128, 1x32x2048x128, 1x32x7023, 1x32x7023, 1x1x55, 1x1x55x16, 1x1x55, 1x1x55x16, 7023)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*s37, s37, 1], [32*s37, s37, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_6 3.9102 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8
  triton_flex_attention_2 3.9763 ms 98.3% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=8
  triton_flex_attention_4 4.7724 ms 81.9% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_5 5.9565 ms 65.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_3 126.1691 ms 3.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=8
  triton_flex_attention_0 146.1305 ms 2.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_1 159.1604 ms 2.5% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 3.3306 seconds and 0.1279 seconds precompiling for 7 choices
Autotune Choices Stats:
{"num_choices": 7, "num_triton_choices": 7, "best_kernel": "triton_flex_attention_6", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8", "best_time": 4.152544021606445, "best_triton_pos": 0}
AUTOTUNE flex_attention(1x32x7633x128, 1x32x2048x128, 1x32x2048x128, 1x32x7633, 1x32x7633, 1x1x60, 1x1x60x16, 1x1x60, 1x1x60x16, 7633)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*s37, s37, 1], [32*s37, s37, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_6 4.1525 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8
  triton_flex_attention_2 4.2154 ms 98.5% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=8
  triton_flex_attention_4 5.2201 ms 79.5% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_5 6.5163 ms 63.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_3 138.4308 ms 3.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=8
  triton_flex_attention_0 158.1575 ms 2.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_1 173.9761 ms 2.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 3.5911 seconds and 0.1097 seconds precompiling for 7 choices
Autotune Choices Stats:
{"num_choices": 7, "num_triton_choices": 7, "best_kernel": "triton_flex_attention_6", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8", "best_time": 4.4054718017578125, "best_triton_pos": 0}
AUTOTUNE flex_attention(1x32x8025x128, 1x32x2048x128, 1x32x2048x128, 1x32x8025, 1x32x8025, 1x1x63, 1x1x63x16, 1x1x63, 1x1x63x16, 8025)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*s37, s37, 1], [32*s37, s37, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_6 4.4055 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8
  triton_flex_attention_2 4.4361 ms 99.3% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=8
  triton_flex_attention_4 5.4957 ms 80.2% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_5 6.7724 ms 65.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_3 144.2378 ms 3.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=8
  triton_flex_attention_0 166.2305 ms 2.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_1 182.5952 ms 2.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 3.7615 seconds and 0.1067 seconds precompiling for 7 choices
Autotune Choices Stats:
{"num_choices": 7, "num_triton_choices": 7, "best_kernel": "triton_flex_attention_6", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8", "best_time": 4.598944187164307, "best_triton_pos": 0}
AUTOTUNE flex_attention(1x32x8307x128, 1x32x2048x128, 1x32x2048x128, 1x32x8307, 1x32x8307, 1x1x65, 1x1x65x16, 1x1x65, 1x1x65x16, 8307)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*s37, s37, 1], [32*s37, s37, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_6 4.5989 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8
  triton_flex_attention_2 4.6375 ms 99.2% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=8
  triton_flex_attention_4 5.6414 ms 81.5% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_5 7.0231 ms 65.5% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_3 148.9349 ms 3.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=8
  triton_flex_attention_0 171.4363 ms 2.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_1 187.8983 ms 2.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 3.8732 seconds and 0.1165 seconds precompiling for 7 choices
Autotune Choices Stats:
{"num_choices": 7, "num_triton_choices": 7, "best_kernel": "triton_flex_attention_6", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8", "best_time": 4.604063987731934, "best_triton_pos": 0}
AUTOTUNE flex_attention(1x32x8386x128, 1x32x2048x128, 1x32x2048x128, 1x32x8386, 1x32x8386, 1x1x66, 1x1x66x16, 1x1x66, 1x1x66x16, 8386)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*s37, s37, 1], [32*s37, s37, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_6 4.6041 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8
  triton_flex_attention_2 4.6350 ms 99.3% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=8
  triton_flex_attention_4 5.7515 ms 80.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_5 7.1936 ms 64.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_3 151.0328 ms 3.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=8
  triton_flex_attention_0 172.9462 ms 2.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_1 191.4085 ms 2.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 3.9295 seconds and 0.1110 seconds precompiling for 7 choices
Autotune Choices Stats:
{"num_choices": 7, "num_triton_choices": 7, "best_kernel": "triton_flex_attention_6", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8", "best_time": 4.6029438972473145, "best_triton_pos": 0}
AUTOTUNE flex_attention(1x32x8436x128, 1x32x2048x128, 1x32x2048x128, 1x32x8436, 1x32x8436, 1x1x66, 1x1x66x16, 1x1x66, 1x1x66x16, 8436)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*s37, s37, 1], [32*s37, s37, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_6 4.6029 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8
  triton_flex_attention_2 4.6357 ms 99.3% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=8
  triton_flex_attention_4 5.7491 ms 80.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_5 7.1902 ms 64.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_3 151.0985 ms 3.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=8
  triton_flex_attention_0 173.0056 ms 2.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_1 191.2873 ms 2.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 3.9289 seconds and 0.1115 seconds precompiling for 7 choices
Autotune Choices Stats:
{"num_choices": 7, "num_triton_choices": 7, "best_kernel": "triton_flex_attention_6", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8", "best_time": 4.606207847595215, "best_triton_pos": 0}
AUTOTUNE flex_attention(1x32x8433x128, 1x32x2048x128, 1x32x2048x128, 1x32x8433, 1x32x8433, 1x1x66, 1x1x66x16, 1x1x66, 1x1x66x16, 8433)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*s37, s37, 1], [32*s37, s37, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_6 4.6062 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=8
  triton_flex_attention_2 4.6672 ms 98.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=8
  triton_flex_attention_4 5.7809 ms 79.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_5 7.1962 ms 64.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=64, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_3 150.7012 ms 3.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=8
  triton_flex_attention_0 173.5201 ms 2.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_1 191.5892 ms 2.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=128, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 3.9374 seconds and 0.1514 seconds precompiling for 7 choices
Autotune Choices Stats:
{"num_choices": 13, "num_triton_choices": 13, "best_kernel": "triton_flex_attention_backward_12", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4", "best_time": 16.89833641052246, "best_triton_pos": 0}
AUTOTUNE flex_attention_backward(1x32x7633x128, 1x32x2048x128, 1x32x2048x128, 1x32x7633, 1x32x7633, 1x32x7633x128, 1x32x7633x128, 1x32x2048x128, 1x1x60, 1x1x60x16, 1x1x16, 1x1x16x60, 1x1x60, 1x1x60x16, 1x1x16, 1x1x16x60, 7633)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*Max(1, s37), Max(1, s37), 1], [32*s37, s37, 1], [4096*Max(1, s37), 128*Max(1, s37), 128, 1], [4096*s37, 128, 4096, 1], [4096*s43, 128*s43, 128, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s56, s56, 1], [s22*s84, s22*s84, s22, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [s99, s99, 1], [s22*s5, s22*s5, s22, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_backward_12 16.8983 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_19 16.9292 ms 99.8% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4
  triton_flex_attention_backward_13 17.1818 ms 98.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_14 17.5042 ms 96.5% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_11 18.6420 ms 90.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_9 20.4663 ms 82.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_10 20.5108 ms 82.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_8 20.5287 ms 82.3% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_15 20.7022 ms 81.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_7 22.4195 ms 75.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 2.8127 seconds and 0.2152 seconds precompiling for 13 choices
Autotune Choices Stats:
{"num_choices": 13, "num_triton_choices": 13, "best_kernel": "triton_flex_attention_backward_19", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4", "best_time": 12.132736206054688, "best_triton_pos": 0}
AUTOTUNE flex_attention_backward(1x32x5531x128, 1x32x2048x128, 1x32x2048x128, 1x32x5531, 1x32x5531, 1x32x5531x128, 1x32x5531x128, 1x32x2048x128, 1x1x44, 1x1x44x16, 1x1x16, 1x1x16x44, 1x1x44, 1x1x44x16, 1x1x16, 1x1x16x44, 5531)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*Max(1, s37), Max(1, s37), 1], [32*s37, s37, 1], [4096*Max(1, s37), 128*Max(1, s37), 128, 1], [4096*s37, 128, 4096, 1], [4096*s43, 128*s43, 128, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s56, s56, 1], [s22*s84, s22*s84, s22, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [s99, s99, 1], [s22*s5, s22*s5, s22, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_backward_19 12.1327 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4
  triton_flex_attention_backward_12 12.3736 ms 98.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_13 12.4137 ms 97.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_14 12.6724 ms 95.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_11 13.4284 ms 90.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_10 14.8106 ms 81.9% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_9 14.8963 ms 81.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_15 14.8977 ms 81.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_8 14.9450 ms 81.2% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_16 17.3011 ms 70.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 2.4451 seconds and 0.1559 seconds precompiling for 13 choices
Autotune Choices Stats:
{"num_choices": 13, "num_triton_choices": 13, "best_kernel": "triton_flex_attention_backward_19", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4", "best_time": 17.831104278564453, "best_triton_pos": 0}
AUTOTUNE flex_attention_backward(1x32x8025x128, 1x32x2048x128, 1x32x2048x128, 1x32x8025, 1x32x8025, 1x32x8025x128, 1x32x8025x128, 1x32x2048x128, 1x1x63, 1x1x63x16, 1x1x16, 1x1x16x63, 1x1x63, 1x1x63x16, 1x1x16, 1x1x16x63, 8025)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*Max(1, s37), Max(1, s37), 1], [32*s37, s37, 1], [4096*Max(1, s37), 128*Max(1, s37), 128, 1], [4096*s37, 128, 4096, 1], [4096*s43, 128*s43, 128, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s56, s56, 1], [s22*s84, s22*s84, s22, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [s99, s99, 1], [s22*s5, s22*s5, s22, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_backward_19 17.8311 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4
  triton_flex_attention_backward_13 18.0515 ms 98.8% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_12 18.0954 ms 98.5% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_14 18.5161 ms 96.3% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_11 19.5753 ms 91.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_9 21.5507 ms 82.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_8 21.6270 ms 82.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_10 21.7430 ms 82.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_15 21.7490 ms 82.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_7 23.5572 ms 75.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 3.2410 seconds and 0.1913 seconds precompiling for 13 choices
Autotune Choices Stats:
{"num_choices": 13, "num_triton_choices": 13, "best_kernel": "triton_flex_attention_backward_19", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4", "best_time": 18.667360305786133, "best_triton_pos": 0}
Autotune Choices Stats:
{"num_choices": 13, "num_triton_choices": 13, "best_kernel": "triton_flex_attention_backward_19", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4", "best_time": 18.70262336730957, "best_triton_pos": 0}
AUTOTUNE flex_attention_backward(1x32x8386x128, 1x32x2048x128, 1x32x2048x128, 1x32x8386, 1x32x8386, 1x32x8386x128, 1x32x8386x128, 1x32x2048x128, 1x1x66, 1x1x66x16, 1x1x16, 1x1x16x66, 1x1x66, 1x1x66x16, 1x1x16, 1x1x16x66, 8386)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*Max(1, s37), Max(1, s37), 1], [32*s37, s37, 1], [4096*Max(1, s37), 128*Max(1, s37), 128, 1], [4096*s37, 128, 4096, 1], [4096*s43, 128*s43, 128, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s56, s56, 1], [s22*s84, s22*s84, s22, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [s99, s99, 1], [s22*s5, s22*s5, s22, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_backward_19 18.6674 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4
  triton_flex_attention_backward_12 18.8078 ms 99.3% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_13 18.9399 ms 98.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_14 19.3102 ms 96.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_11 20.5880 ms 90.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_10 22.7014 ms 82.2% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_9 22.7124 ms 82.2% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_8 22.7410 ms 82.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_15 22.7648 ms 82.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_7 25.0880 ms 74.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 3.0870 seconds and 0.2283 seconds precompiling for 13 choices
Autotune Choices Stats:
{"num_choices": 13, "num_triton_choices": 13, "best_kernel": "triton_flex_attention_backward_19", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4", "best_time": 18.652223587036133, "best_triton_pos": 0}
AUTOTUNE flex_attention_backward(1x32x8433x128, 1x32x2048x128, 1x32x2048x128, 1x32x8433, 1x32x8433, 1x32x8433x128, 1x32x8433x128, 1x32x2048x128, 1x1x66, 1x1x66x16, 1x1x16, 1x1x16x66, 1x1x66, 1x1x66x16, 1x1x16, 1x1x16x66, 8433)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*Max(1, s37), Max(1, s37), 1], [32*s37, s37, 1], [4096*Max(1, s37), 128*Max(1, s37), 128, 1], [4096*s37, 128, 4096, 1], [4096*s43, 128*s43, 128, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s56, s56, 1], [s22*s84, s22*s84, s22, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [s99, s99, 1], [s22*s5, s22*s5, s22, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_backward_19 18.7026 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4
  triton_flex_attention_backward_12 18.8859 ms 99.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_13 18.9763 ms 98.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_14 19.3380 ms 96.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_11 20.6652 ms 90.5% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_9 22.7428 ms 82.2% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_15 22.8070 ms 82.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_10 22.8169 ms 82.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_8 22.9125 ms 81.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_7 25.3636 ms 73.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 3.1158 seconds and 0.2219 seconds precompiling for 13 choices
AUTOTUNE flex_attention_backward(1x32x8436x128, 1x32x2048x128, 1x32x2048x128, 1x32x8436, 1x32x8436, 1x32x8436x128, 1x32x8436x128, 1x32x2048x128, 1x1x66, 1x1x66x16, 1x1x16, 1x1x16x66, 1x1x66, 1x1x66x16, 1x1x16, 1x1x16x66, 8436)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*Max(1, s37), Max(1, s37), 1], [32*s37, s37, 1], [4096*Max(1, s37), 128*Max(1, s37), 128, 1], [4096*s37, 128, 4096, 1], [4096*s43, 128*s43, 128, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s56, s56, 1], [s22*s84, s22*s84, s22, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [s99, s99, 1], [s22*s5, s22*s5, s22, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_backward_19 18.6522 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4
  triton_flex_attention_backward_12 18.8603 ms 98.9% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_13 18.9502 ms 98.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_14 19.3232 ms 96.5% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_11 20.6350 ms 90.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_9 22.7743 ms 81.9% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_10 22.7935 ms 81.8% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_15 22.8346 ms 81.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_8 22.9176 ms 81.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_7 25.2455 ms 73.9% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 3.1078 seconds and 0.2172 seconds precompiling for 13 choices
Autotune Choices Stats:
{"num_choices": 13, "num_triton_choices": 13, "best_kernel": "triton_flex_attention_backward_19", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4", "best_time": 18.391231536865234, "best_triton_pos": 0}
AUTOTUNE flex_attention_backward(1x32x8307x128, 1x32x2048x128, 1x32x2048x128, 1x32x8307, 1x32x8307, 1x32x8307x128, 1x32x8307x128, 1x32x2048x128, 1x1x65, 1x1x65x16, 1x1x16, 1x1x16x65, 1x1x65, 1x1x65x16, 1x1x16, 1x1x16x65, 8307)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*Max(1, s37), Max(1, s37), 1], [32*s37, s37, 1], [4096*Max(1, s37), 128*Max(1, s37), 128, 1], [4096*s37, 128, 4096, 1], [4096*s43, 128*s43, 128, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s56, s56, 1], [s22*s84, s22*s84, s22, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [s99, s99, 1], [s22*s5, s22*s5, s22, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_backward_19 18.3912 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4
  triton_flex_attention_backward_12 18.5940 ms 98.9% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_13 18.6374 ms 98.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_14 19.1026 ms 96.3% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_11 20.3500 ms 90.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_9 22.4148 ms 82.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_15 22.4269 ms 82.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_10 22.4855 ms 81.8% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_8 22.5205 ms 81.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_7 24.6826 ms 74.5% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 3.2393 seconds and 0.2707 seconds precompiling for 13 choices
Autotune Choices Stats:
{"num_choices": 13, "num_triton_choices": 13, "best_kernel": "triton_flex_attention_backward_19", "best_kernel_desc": "BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4", "best_time": 15.482303619384766, "best_triton_pos": 0}
AUTOTUNE flex_attention_backward(1x32x7023x128, 1x32x2048x128, 1x32x2048x128, 1x32x7023, 1x32x7023, 1x32x7023x128, 1x32x7023x128, 1x32x2048x128, 1x1x55, 1x1x55x16, 1x1x16, 1x1x16x55, 1x1x55, 1x1x55x16, 1x1x16, 1x1x16x55, 7023)
strides: [4096*s37, 128, 4096, 1], [4096*s0, 128*s0, 128, 1], [4096*s43, 128*s43, 128, 1], [32*Max(1, s37), Max(1, s37), 1], [32*s37, s37, 1], [4096*Max(1, s37), 128*Max(1, s37), 128, 1], [4096*s37, 128, 4096, 1], [4096*s43, 128*s43, 128, 1], [s22, s22, 1], [s22*s72, s22*s72, s72, 1], [s56, s56, 1], [s22*s84, s22*s84, s22, 1], [s22, s22, 1], [s22*s4, s22*s4, s4, 1], [s99, s99, 1], [s22*s5, s22*s5, s22, 1], [1]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64
  triton_flex_attention_backward_19 15.4823 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=2, num_warps=4
  triton_flex_attention_backward_12 15.6948 ms 98.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_13 15.7264 ms 98.4% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_14 16.1180 ms 96.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_11 17.2020 ms 90.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_8 18.9128 ms 81.9% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
  triton_flex_attention_backward_9 18.9159 ms 81.8% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=4, num_warps=4
  triton_flex_attention_backward_10 18.9885 ms 81.5% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=5, num_warps=4
  triton_flex_attention_backward_15 18.9983 ms 81.5% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=1, num_warps=4
  triton_flex_attention_backward_16 21.9098 ms 70.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION="'ieee'", GQA_SHARED_HEADS=1, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=False, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=128, QK_HEAD_DIM_ROUNDED=128, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=True, SM_SCALE=0.08838834764831845, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=128, V_HEAD_DIM_ROUNDED=128, WRITE_DQ=True, num_stages=3, num_warps=4
SingleProcess AUTOTUNE benchmarking takes 2.9795 seconds and 0.1614 seconds precompiling for 13 choices
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Channel 01/1 : 17[1] -> 0[0] [send] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Channel 01/1 : 16[0] -> 0[0] [send] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-1:24022:25874 [0] NCCL INFO [Proxy Progress] Device 3 CPU core 154
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO Channel 01/1 : 19[3] -> 0[0] [send] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-1:24021:25875 [0] NCCL INFO [Proxy Progress] Device 2 CPU core 28
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO Channel 01/1 : 18[2] -> 0[0] [send] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-1:24026:25876 [0] NCCL INFO [Proxy Progress] Device 7 CPU core 92
h-liuchonghan-translate-a8nodes-main-w-1:24024:25877 [0] NCCL INFO [Proxy Progress] Device 5 CPU core 122
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO Channel 01/1 : 23[7] -> 0[0] [send] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO Channel 01/1 : 21[5] -> 0[0] [send] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-1:24025:25878 [0] NCCL INFO [Proxy Progress] Device 6 CPU core 223
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO Channel 01/1 : 22[6] -> 0[0] [send] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-1:24023:25879 [0] NCCL INFO [Proxy Progress] Device 4 CPU core 253
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO Channel 01/1 : 20[4] -> 0[0] [send] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO Channel 01/1 : 0[0] -> 16[0] [receive] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO Channel 01/1 : 0[0] -> 17[1] [receive] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO Channel 01/1 : 0[0] -> 23[7] [receive] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO Channel 01/1 : 0[0] -> 22[6] [receive] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO Channel 01/1 : 0[0] -> 18[2] [receive] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO Channel 01/1 : 0[0] -> 19[3] [receive] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO Channel 01/1 : 0[0] -> 21[5] [receive] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO Channel 01/1 : 0[0] -> 20[4] [receive] via NET/Socket/0/Shared
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO comm 0x55fcd06c2a60 rank 22 nranks 32 cudaDev 6 busId cd000 - Destroy COMPLETE
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO comm 0x562c6a8fbb60 rank 20 nranks 32 cudaDev 4 busId 91000 - Destroy COMPLETE
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO comm 0x5628e9cd4dd0 rank 23 nranks 32 cudaDev 7 busId d2000 - Destroy COMPLETE
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO comm 0x55f5bce25f20 rank 21 nranks 32 cudaDev 5 busId 97000 - Destroy COMPLETE
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO comm 0x5563bd0ab130 rank 16 nranks 32 cudaDev 0 busId e000 - Destroy COMPLETE
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO comm 0x55d9f63f4ec0 rank 19 nranks 32 cudaDev 3 busId 4f000 - Destroy COMPLETE
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO comm 0x55de3b5c9510 rank 18 nranks 32 cudaDev 2 busId 49000 - Destroy COMPLETE
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO comm 0x55cc6d5b47b0 rank 17 nranks 32 cudaDev 1 busId 13000 - Destroy COMPLETE
h-liuchonghan-translate-a8nodes-main-w-1:24026:24026 [7] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
h-liuchonghan-translate-a8nodes-main-w-1:24025:24025 [6] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
h-liuchonghan-translate-a8nodes-main-w-1:24023:24023 [4] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
h-liuchonghan-translate-a8nodes-main-w-1:24024:24024 [5] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
h-liuchonghan-translate-a8nodes-main-w-1:24019:24019 [0] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
h-liuchonghan-translate-a8nodes-main-w-1:24022:24022 [3] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
h-liuchonghan-translate-a8nodes-main-w-1:24021:24021 [2] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault
h-liuchonghan-translate-a8nodes-main-w-1:24020:24020 [1] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault