{
"version": "1",
"metadata": {
"marimo_version": "0.25.2",
"script_metadata_hash": "1cb25ad342be9ccf432fa880e3c72e41"
},
"cells": [
{
"id": "Hbol",
"code_hash": "1d0db38904205bec4d6f6f6a1f6cec3e",
"outputs": [],
"console": []
},
{
"id": "MJUe",
"code_hash": "8c3add7f504aee86f92a5fd2a080e930",
"outputs": [],
"console": []
},
{
"id": "vblA",
"code_hash": "905d09ca55d5b894841a2106100436a4",
"outputs": [
{
"type": "error",
"ename": "interruption",
"evalue": "This cell was interrupted and needs to be re-run",
"traceback": []
}
],
"console": [
{
"type": "stream",
"name": "stderr",
"text": "Traceback (most recent call last):\n File "/tmp/marimo_794/__marimo__cell_vblA_.py", line 3, in <module>\n result = subprocess.run(\n ['python', 'wikitext.py', '--sources', 'cosmopedia', '--cosmopedia-mb', '1000'],\n capture_output=True, text=True\n )\n File "/usr/local/lib/python3.13/subprocess.py", line 556, in run\n stdout, stderr = process.communicate(input, timeout=timeout)\n ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.13/subprocess.py", line 1222, in communicate\n stdout, stderr = self._communicate(input, endtime, timeout)\n ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.13/subprocess.py", line 2154, in _communicate\n ready = selector.select(timeout)\n File "/usr/local/lib/python3.13/selectors.py", line 398, in select\n fd_event_list = self._selector.poll(timeout)\n File "/usr/local/lib/python3.13/site-packages/marimo/_utils/signals.py", line 33, in __call__\n self._handler(signum, frame)\n ~~~~~~~~~~~~~^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.13/site-packages/marimo/_runtime/handlers.py", line 74, in interrupt_handler\n raise MarimoInterrupt\nKeyboardInterrupt\n
\ud83d\udd0d GPU Diagnosis Summary \u2014 Molab RTX 6000 Pro Blackwell
\nBased on the diagnostic cells in this notebook, here is what I found:\nRoot Cause: The GPU is not being exposed to this Molab container/runtime
\n\n\n
\n\n \n\n\nCheck \nResult \n\n \n \nnvidia-smi binary\u274c Not found anywhere on the system \n\n \n \ntorch.cuda.is_available()\u274c Returns \nFalse\n \n \n/dev/nvidia* devices\u274c Not present \n\n \n \n/proc/driver/nvidia\u274c Not present (no kernel module loaded) \n\n \n \nfuture.py benchmark output\u274c Ran on \ndevice=cpu with peak_mem=0.00 MB\n \n\n \nnvidia-smi -pm 1 command\u274c \nFileNotFoundError: No such file or directory: 'nvidia-smi'Key Observations
\n\n
\nnvidia-smi, no /dev/nvidia*, no /proc/driver/nvidia \u2014 this means the NVIDIA driver is not accessible inside the container at all. The GPU hardware may exist on the host, but the container cannot reach it.jax_cuda13_plugin v0.11.0 is incompatible with jaxlib v0.11.2.What Needs to Happen
\nThe problem is not with your Python scripts or PyTorch installation \u2014 it's that the Molab container does not have GPU access configured. See the fix steps below."
}
}
],
"console": []
},
{
"id": "PKri",
"code_hash": "12bf8c42b2c408cde25119da5ff58681",
"outputs": [],
"console": []
},
{
"id": "Xref",
"code_hash": "805b618d7b70e0a56483e6d9d7a2ff32",
"outputs": [],
"console": []
},
{
"id": "SFPL",
"code_hash": "d4e2adcce77bf7bd657b4f45bba3331f",
"outputs": [],
"console": []
},
{
"id": "BYtC",
"code_hash": "46e0411ba9267491b44043205bb69992",
"outputs": [],
"console": []
},
{
"id": "RGSE",
"code_hash": "6470f3209ebb8c510b3f5b2f231bfc62",
"outputs": [
{
"type": "data",
"data": {
"text/html": "
0" } } ], "console": [ { "type": "stream", "name": "stdout", "text": "[kernel] import: cuda_avail=True cuda_ver=13.0 file=/marimo/kernel.py\n", "mimetype": "text/plain" }, { "type": "stream", "name": "stderr", "text": "/tmp/uv-venv/lib/python3.13/site-packages/torchao/quantization/quant_api.py:1558: SyntaxWarning: invalid escape sequence '\\.'\n * regex for parameter names, must start with `re:`, e.g. `re:language\\.layers\\..+\\.q_proj.weight`.\n/marimo/model.py:391: UserWarning: config module has no attribute 'eos_id' -- falling back to default None. If this is unexpected, check for a casing mismatch or rename in your config.py.\n warnings.warn(\n/marimo/model.py:391: UserWarning: config module has no attribute 'moe_fixed_capacity' -- falling back to default None. If this is unexpected, check for a casing mismatch or rename in your config.py.\n warnings.warn(\n", "mimetype": "text/plain" }, { "type": "stream", "name": "stdout", "text": "[Tok] Found 3 file(s) in ./training_data\n[Tok] Fingerprint: 3894af946a164b07 (document_boundaries=True)\n[Tok] Tokenizing with 3 worker(s) ...\n[Tok] 1/3 files done | 184,072,225 tokens | 106.1s\n[Tok] 2/3 files done | 504,955,780 tokens | 123.7s\n[Tok] 3/3 files done | 879,185,385 tokens | 147.3s\n[Tok] Assembling 879,185,385 tokens into ./training_data/.token_cache/tokens_3894af946a164b07.tmp.npy ...\n[Tok] Ranges sidecar -> ./training_data/.token_cache/tokens_3894af946a164b07.ranges.json (3 entries)\n[Tok] Done in 173.2s | 879,185,385 tokens | 3.52 GB\n[Tok] Saved -> ./training_data/.token_cache/tokens_3894af946a164b07.npy\n./training_data/.token_cache/tokens_3894af946a164b07.npy\n", "mimetype": "text/plain" } ] }, { "id": "Kclp", "code_hash": "b923242b1bd575e4dcf31971197d3570", "outputs": [], "console": [] }, { "id": "ZHIE", "code_hash": null, "outputs": [], "console": [] }, { "id": "emfo", "code_hash": null, "outputs": [], "console": [] }, { "id": "Hstk", "code_hash": "16f7001224229d5030ea9ad7077bab8a", "outputs": [ { "type": "error", "ename": "interruption", "evalue": "This cell was interrupted and needs to be re-run", "traceback": [] } ], "console": [ { "type": "stream", "name": "stdout", "text": "[kernel] import: cuda_avail=True cuda_ver=13.0 file=/marimo/kernel.py\n", "mimetype": "text/plain" }, { "type": "stream", "name": "stderr", "text": "/marimo/model.py:391: UserWarning: config module has no attribute 'eos_id' -- falling back to default None. If this is unexpected, check for a casing mismatch or rename in your config.py.\n warnings.warn(\n/marimo/model.py:391: UserWarning: config module has no attribute 'moe_fixed_capacity' -- falling back to default None. If this is unexpected, check for a casing mismatch or rename in your config.py.\n warnings.warn(\n[RANK 0] Detected kernel version 4.19.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.\n", "mimetype": "text/plain" }, { "type": "stream", "name": "stdout", "text": "[Setup] GPU compute capability 12.x (Ampere+) -- TF32 enabled for fp32 matmuls/convs.\n[Setup] Parallelism: plain data-parallel (DDP) only -- no FSDP/tensor-parallel/pipeline-parallel. Appropriate for small device counts (e.g. dual T4); set use_fsdp=True in config.py when sharding is actually needed.\n[Setup] Using device: cuda\n[Setup] Num processes (GPUs): 1\n[Tokenizer] Loaded, vocab_size=12288\n[Dataset] Found 3 files in ./training_data (curriculum OFF)\n[Dataset] Cache hit -> ./training_data/.token_cache/tokens_3894af946a164b07.npy (879,185,417 tokens)\n[Dataset] train=870,393,532 tokens, val=8,791,853 tokens\n[Val] fixed validation set: 33 batches x 8 seqs across 3 file(s) (cosmopedia_1000mb.txt:7, meyank_hindi_1500mb.txt:12, tinystories_1500mb.txt:14)\n", "mimetype": "text/plain" }, { "type": "stream", "name": "stderr", "text": "/usr/local/lib/python3.13/site-packages/torch/_inductor/select_algorithm.py:3686: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()\n current_out_size = out_base.storage().size()\nAutotune Choices Stats:\n{\"num_choices\": 6, \"num_triton_choices\": 6, \"best_kernel\": \"triton_flex_attention_24\", \"best_kernel_desc\": \"BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\\\"'tf32'\\\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\", \"best_time\": 6.675551891326904, \"best_triton_pos\": 0}\nAUTOTUNE flex_attention(320x16x1024x20, 320x8x1024x20, 320x8x1024x20, 320x16x1024, 320x16x1024, 320x1x8, 320x1x8x8, 320x1x8, 320x1x8x8, 320x1024)\nstrides: [327680, 20, 320, 1], [163840, 20, 160, 1], [655360, 20, 640, 1], [16384, 1024, 1], [16384, 1024, 1], [8, 8, 1], [64, 64, 8, 1], [8, 8, 1], [64, 64, 8, 1], [1024, 1]\ndtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64\n triton_flex_attention_24 6.6756 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'tf32'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_25 6.6775 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'tf32'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_23 6.6796 ms 99.9% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'tf32'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_20 6.9734 ms 95.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'tf32'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_22 7.0103 ms 95.2% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'tf32'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_21 7.0175 ms 95.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'tf32'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\nSingleProcess AUTOTUNE benchmarking takes 0.4729 seconds and 1.2006 seconds precompiling for 6 choices\nW1101 10:56:17.149000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf749: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.153000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf838: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.157000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf927: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.161000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1016: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.164000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1094: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.168000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf738: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.171000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf827: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.174000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf916: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.178000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1005: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.181000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1083: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.184000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf727: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.187000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf816: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.190000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf905: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.193000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf994: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.197000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1072: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.200000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf716: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.203000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf805: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.206000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf894: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.209000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf983: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.213000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1061: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.217000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf705: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.220000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf794: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.223000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf883: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.226000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf972: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.230000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1050: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.233000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf694: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.236000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf783: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.239000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf872: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.242000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf961: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.245000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1039: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.248000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf683: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.252000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf772: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.255000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf861: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.258000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf950: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.262000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1028: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.265000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf672: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.269000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf761: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.272000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf850: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.275000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf939: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.278000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf661: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.720000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf927: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.724000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf916: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.728000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf939: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.731000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf905: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.735000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf950: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.738000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf894: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.741000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf961: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.745000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf883: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.748000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf972: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.751000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf872: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.755000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf983: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.758000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf861: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.761000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf994: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.765000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf850: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.768000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1005: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.771000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf838: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.774000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1016: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.778000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf827: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.781000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1028: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.785000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf816: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.788000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1039: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.791000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf805: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.794000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1050: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.797000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf794: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.801000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1061: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.804000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf783: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.807000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1072: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.810000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf772: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.814000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1083: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.817000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf761: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.820000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1094: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.823000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf749: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.827000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf738: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.830000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf727: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.833000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf716: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.836000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf705: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.839000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf694: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.843000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf683: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.846000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf672: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:17.849000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf661: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.240000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf927: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.243000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf916: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.246000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf939: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.250000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf905: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.253000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf950: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.257000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf894: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.260000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf961: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.263000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf883: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.266000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf972: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.269000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf872: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.272000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf983: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.276000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf861: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.279000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf994: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.282000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf850: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.286000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1005: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.289000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf838: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.292000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1016: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.295000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf827: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.298000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1028: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.301000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf816: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.304000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1039: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.308000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf805: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.311000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1050: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.314000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf794: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.317000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1061: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.320000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf783: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.324000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1072: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.327000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf772: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.330000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1083: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.333000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf761: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.336000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1094: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.339000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf749: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.342000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf738: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.346000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf727: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.349000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf716: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.352000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf705: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.355000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf694: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.359000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf683: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.362000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf672: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:18.365000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf661: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.293000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf927: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.296000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf916: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.300000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf939: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.304000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf905: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.307000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf950: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.311000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf894: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.314000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf961: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.317000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf883: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.321000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf972: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.324000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf872: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.328000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf983: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.331000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf861: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.334000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf994: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.337000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf850: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.341000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1005: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.344000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf838: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.347000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1016: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.351000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf827: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.354000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1028: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.357000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf816: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.361000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1039: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.364000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf805: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.367000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1050: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.370000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf794: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.374000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1061: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.377000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf783: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.381000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1072: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.384000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf772: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.387000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1083: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.391000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf761: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.394000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1094: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.397000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf749: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.400000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf738: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.404000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf727: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.407000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf716: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.410000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf705: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.414000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf694: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.417000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf683: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.421000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf672: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nW1101 10:56:19.424000 40715 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf661: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[8192, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 8192], stride=[1, 12288])\nE1101 10:56:20.384000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:20.384000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:20.384000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:56:20.537000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:20.537000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:20.537000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:56:20.789000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:20.789000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:20.789000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:56:20.795000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:20.795000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:20.795000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"triton_mm_16\", \"best_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\", \"best_time\": 0.5324479937553406, \"best_triton_pos\": 0}\nAUTOTUNE mm(327680x320, 320x640)\nstrides: [320, 1], [1, 320]\ndtypes: torch.bfloat16, torch.bfloat16\n triton_mm_16 0.5324 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_17 0.6113 ms 87.1% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_15 0.6308 ms 84.4% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=8\n triton_mm_13 0.6512 ms 81.8% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_9 0.6574 ms 81.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_11 0.6635 ms 80.2% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_10 0.6676 ms 79.8% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_14 0.6850 ms 77.7% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n mm 0.7300 ms 72.9% \n triton_mm_7 0.8642 ms 61.6% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\nSingleProcess AUTOTUNE benchmarking takes 1.0179 seconds and 0.0002 seconds precompiling for 21 choices\nAutotune Choices Stats:\n{\"num_choices\": 12, \"num_triton_choices\": 11, \"best_kernel\": \"triton_mm_27\", \"best_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=16, BLOCK_M=32, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=2\", \"best_time\": 0.01740800030529499, \"best_triton_pos\": 0}\nAUTOTUNE mm(327680x16, 16x16)\nstrides: [16, 1], [1, 16]\ndtypes: torch.bfloat16, torch.bfloat16\n triton_mm_27 0.0174 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=16, BLOCK_M=32, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=2\n triton_mm_35 0.0179 ms 97.5% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=16, BLOCK_M=128, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=8\n triton_mm_33 0.0182 ms 95.4% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=16, BLOCK_M=128, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_28 0.0183 ms 95.3% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=16, BLOCK_M=32, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=2\n triton_mm_29 0.0183 ms 95.3% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=16, BLOCK_M=64, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4\n triton_mm_30 0.0183 ms 95.1% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=16, BLOCK_M=64, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4\n triton_mm_34 0.0183 ms 95.1% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=16, BLOCK_M=128, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_26 0.0183 ms 94.9% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=16, BLOCK_M=32, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=1, num_warps=2\n triton_mm_31 0.0184 ms 94.8% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=16, BLOCK_M=64, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_32 0.0184 ms 94.8% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=16, BLOCK_M=64, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=4\nSingleProcess AUTOTUNE benchmarking takes 0.3442 seconds and 0.0001 seconds precompiling for 12 choices\nE1101 10:56:21.589000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:21.589000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:21.589000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:56:21.742000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:21.742000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:21.742000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:56:21.998000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:21.998000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:21.998000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:56:22.004000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:22.004000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:22.004000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"triton_mm_53\", \"best_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\", \"best_time\": 0.35017600655555725, \"best_triton_pos\": 0}\nAUTOTUNE mm(327680x320, 320x320)\nstrides: [320, 1], [1, 320]\ndtypes: torch.bfloat16, torch.bfloat16\n triton_mm_53 0.3502 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n mm 0.3634 ms 96.4% \n triton_mm_54 0.3665 ms 95.5% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_50 0.3666 ms 95.5% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_51 0.3767 ms 93.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_52 0.3952 ms 88.6% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=8\n triton_mm_48 0.4444 ms 78.8% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_44 0.4647 ms 75.3% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\n triton_mm_46 0.4686 ms 74.7% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_47 0.4710 ms 74.3% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\nSingleProcess AUTOTUNE benchmarking takes 0.8634 seconds and 0.0002 seconds precompiling for 21 choices\nE1101 10:56:22.538000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:22.538000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:22.538000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:56:22.699000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:22.699000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:22.699000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:56:22.962000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:22.962000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:22.962000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:56:22.968000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:22.968000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:22.968000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"triton_mm_73\", \"best_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\", \"best_time\": 0.98198401927948, \"best_triton_pos\": 0}\nAUTOTUNE mm(327680x320, 320x1280)\nstrides: [320, 1], [1, 320]\ndtypes: torch.bfloat16, torch.bfloat16\n triton_mm_73 0.9820 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_74 1.1262 ms 87.2% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n mm 1.1950 ms 82.2% \n triton_mm_72 1.2337 ms 79.6% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=8\n triton_mm_70 1.2820 ms 76.6% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_66 1.2871 ms 76.3% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_67 1.3015 ms 75.5% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_68 1.3035 ms 75.3% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_71 1.3117 ms 74.9% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_63 1.7049 ms 57.6% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4\nSingleProcess AUTOTUNE benchmarking takes 0.9636 seconds and 0.0002 seconds precompiling for 21 choices\nE1101 10:56:23.436000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:23.436000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:23.436000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:56:23.589000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:23.589000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:23.589000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:56:23.840000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:23.840000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:23.840000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:56:23.847000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:23.847000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:23.847000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"mm\", \"best_time\": 0.5550079941749573, \"best_triton_pos\": 1, \"best_triton_time\": 0.5722560286521912, \"best_triton_kernel\": \"triton_mm_93\", \"best_triton_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\"}\nAUTOTUNE mm(327680x640, 640x320)\nstrides: [640, 1], [1, 640]\ndtypes: torch.bfloat16, torch.bfloat16\n mm 0.5550 ms 100.0% \n triton_mm_93 0.5723 ms 97.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_94 0.6062 ms 91.6% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_90 0.6573 ms 84.4% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_91 0.6676 ms 83.1% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_86 0.7905 ms 70.2% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_88 0.7936 ms 69.9% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_87 0.7966 ms 69.7% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_84 0.8542 ms 65.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\n triton_mm_83 0.8765 ms 63.3% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4\nSingleProcess AUTOTUNE benchmarking takes 0.8781 seconds and 0.0002 seconds precompiling for 21 choices\nE1101 10:56:24.296000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:24.296000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:24.296000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:56:24.453000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:24.453000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:24.453000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:56:24.713000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:24.713000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:24.713000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:56:24.719000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:24.719000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:24.719000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"triton_mm_113\", \"best_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\", \"best_time\": 0.3152959942817688, \"best_triton_pos\": 0}\nAUTOTUNE mm(327680x256, 256x320)\nstrides: [256, 1], [1, 256]\ndtypes: torch.bfloat16, torch.bfloat16\n triton_mm_113 0.3153 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n mm 0.3183 ms 99.1% \n triton_mm_110 0.3192 ms 98.8% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_114 0.3235 ms 97.5% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_111 0.3256 ms 96.8% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_108 0.3396 ms 92.8% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_112 0.3451 ms 91.4% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=8\n triton_mm_106 0.3540 ms 89.1% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_107 0.3623 ms 87.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_104 0.3654 ms 86.3% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\nSingleProcess AUTOTUNE benchmarking takes 0.8705 seconds and 0.0002 seconds precompiling for 21 choices\nE1101 10:56:25.238000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:25.238000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:25.238000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:56:25.396000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:25.396000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:25.396000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:56:25.657000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:25.657000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:25.657000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:56:25.663000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:25.663000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:25.663000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"mm\", \"best_time\": 0.25596800446510315, \"best_triton_pos\": 1, \"best_triton_time\": 0.25619199872016907, \"best_triton_kernel\": \"triton_mm_3508\", \"best_triton_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\"}\nAUTOTUNE mm(8192x320, 320x12288)\nstrides: [320, 1], [1, 320]\ndtypes: torch.bfloat16, torch.bfloat16\n mm 0.2560 ms 100.0% \n triton_mm_3508 0.2562 ms 99.9% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3509 0.2908 ms 88.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3507 0.3074 ms 83.3% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=8\n triton_mm_3505 0.3204 ms 79.9% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3501 0.3276 ms 78.1% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3506 0.3307 ms 77.4% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_3503 0.3316 ms 77.2% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3502 0.3358 ms 76.2% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_3499 0.4218 ms 60.7% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\nSingleProcess AUTOTUNE benchmarking takes 0.8730 seconds and 0.0001 seconds precompiling for 21 choices\nE1101 10:56:26.213000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:26.213000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:26.213000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:56:26.361000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:26.361000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:26.361000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:56:26.605000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:26.605000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:26.605000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:56:26.612000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:26.612000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:26.612000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"mm\", \"best_time\": 0.2949120104312897, \"best_triton_pos\": 1, \"best_triton_time\": 0.3553279936313629, \"best_triton_kernel\": \"triton_mm_1666\", \"best_triton_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\"}\nAUTOTUNE mm(8192x12288, 12288x320)\nstrides: [12288, 1], [320, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n mm 0.2949 ms 100.0% \n triton_mm_1666 0.3553 ms 83.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_1665 0.3666 ms 80.4% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_1668 0.4075 ms 72.4% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_1661 0.4198 ms 70.2% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_1663 0.4618 ms 63.9% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_1658 0.4731 ms 62.3% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4\n triton_mm_1662 0.4813 ms 61.3% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_1667 0.4813 ms 61.3% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=8\n triton_mm_1669 0.4813 ms 61.3% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\nSingleProcess AUTOTUNE benchmarking takes 0.8356 seconds and 0.0001 seconds precompiling for 21 choices\nE1101 10:56:27.052000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:27.052000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:27.052000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:56:27.203000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:27.203000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:27.203000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:56:27.450000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:27.450000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:27.450000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:56:27.456000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:56:27.456000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:56:27.456000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"mm\", \"best_time\": 0.27136000990867615, \"best_triton_pos\": 1, \"best_triton_time\": 0.3102720081806183, \"best_triton_kernel\": \"triton_mm_1688\", \"best_triton_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\"}\nAUTOTUNE mm(12288x8192, 8192x320)\nstrides: [1, 12288], [320, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n mm 0.2714 ms 100.0% \n triton_mm_1688 0.3103 ms 87.5% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_1689 0.3336 ms 81.3% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_1685 0.3502 ms 77.5% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_1687 0.3583 ms 75.7% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=8\n triton_mm_1681 0.4096 ms 66.3% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_1679 0.4351 ms 62.4% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\n triton_mm_1683 0.4402 ms 61.7% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_1678 0.4444 ms 61.1% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4\n triton_mm_1682 0.4465 ms 60.8% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\nSingleProcess AUTOTUNE benchmarking takes 0.8425 seconds and 0.0002 seconds precompiling for 21 choices\nAutotune Choices Stats:\n{\"num_choices\": 13, \"num_triton_choices\": 13, \"best_kernel\": \"triton_flex_attention_backward_4115\", \"best_kernel_desc\": \"BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_M1=32, BLOCK_M2=128, BLOCK_N=64, BLOCK_N1=128, BLOCK_N2=32, FLOAT32_PRECISION=\\\"'tf32'\\\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\", \"best_time\": 18.97577667236328, \"best_triton_pos\": 0}\nAUTOTUNE flex_attention_backward(320x16x1024x20, 320x8x1024x20, 320x8x1024x20, 320x16x1024, 320x16x1024, 320x16x1024x20, 320x16x1024x20, 320x8x1024x20, 320x1x8, 320x1x8x8, 320x1x8, 320x1x8x8, 320x1x8, 320x1x8x8, 320x1x8, 320x1x8x8, 320x1024)\nstrides: [327680, 20, 320, 1], [163840, 20, 160, 1], [655360, 20, 640, 1], [16384, 1024, 1], [16384, 1024, 1], [327680, 20480, 20, 1], [327680, 20, 320, 1], [163840, 20, 160, 1], [8, 8, 1], [64, 64, 8, 1], [8, 8, 1], [64, 64, 8, 1], [8, 8, 1], [64, 64, 8, 1], [8, 8, 1], [64, 64, 8, 1], [1024, 1]\ndtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64\n triton_flex_attention_backward_4115 18.9758 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_M1=32, BLOCK_M2=128, BLOCK_N=64, BLOCK_N1=128, BLOCK_N2=32, FLOAT32_PRECISION=\"'tf32'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_backward_4113 23.2663 ms 81.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION=\"'tf32'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_backward_4111 23.2684 ms 81.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION=\"'tf32'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_backward_4112 23.2684 ms 81.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION=\"'tf32'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_backward_4114 23.2735 ms 81.5% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION=\"'tf32'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_backward_4108 24.8586 ms 76.3% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION=\"'tf32'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_backward_4109 24.8587 ms 76.3% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION=\"'tf32'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_backward_4110 24.8638 ms 76.3% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION=\"'tf32'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_backward_4107 24.8648 ms 76.3% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION=\"'tf32'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_backward_4105 48.7229 ms 38.9% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N=64, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION=\"'tf32'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\nSingleProcess AUTOTUNE benchmarking takes 3.2074 seconds and 1.5122 seconds precompiling for 13 choices\nAutotune Choices Stats:\n{\"num_choices\": 11, \"num_triton_choices\": 0, \"best_kernel\": \"decompose_k_mm_10_split_11\", \"best_kernel_desc\": \"k_split=10\", \"best_time\": 0.9830399751663208}\nAUTOTUNE mm(1280x327680, 327680x320)\nstrides: [1, 1280], [320, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n decompose_k_mm_10_split_11 0.9830 ms 100.0% k_split=10\n mm 1.0271 ms 95.7% \n decompose_k_mm_5_split_10 1.0905 ms 90.1% k_split=5\n decompose_k_mm_40_split_13 1.0925 ms 90.0% k_split=40\n decompose_k_mm_16_split_19 1.1448 ms 85.9% k_split=16\n decompose_k_mm_8_split_18 1.1930 ms 82.4% k_split=8\n decompose_k_mm_160_split_15 1.2032 ms 81.7% k_split=160\n decompose_k_mm_4_split_17 1.2073 ms 81.4% k_split=4\n decompose_k_mm_80_split_14 1.2114 ms 81.2% k_split=80\n decompose_k_mm_20_split_12 1.2369 ms 79.5% k_split=20\nSingleProcess AUTOTUNE benchmarking takes 9.6060 seconds and 0.0001 seconds precompiling for 11 choices\nAutotune Choices Stats:\n{\"num_choices\": 11, \"num_triton_choices\": 0, \"best_kernel\": \"decompose_k_mm_20_split_2\", \"best_kernel_desc\": \"k_split=20\", \"best_time\": 0.5590400099754333}\nAUTOTUNE mm(320x327680, 327680x640)\nstrides: [1, 320], [640, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n decompose_k_mm_20_split_2 0.5590 ms 100.0% k_split=20\n mm 0.5652 ms 98.9% \n decompose_k_mm_40_split_3 0.5846 ms 95.6% k_split=40\n decompose_k_mm_10_split_1 0.5949 ms 94.0% k_split=10\n decompose_k_mm_8_split_9 0.6501 ms 86.0% k_split=8\n decompose_k_mm_80_split_4 0.6604 ms 84.7% k_split=80\n decompose_k_mm_160_split_5 0.7055 ms 79.2% k_split=160\n decompose_k_mm_5_split_0 0.8663 ms 64.5% k_split=5\n decompose_k_mm_320_split_6 0.8765 ms 63.8% k_split=320\n decompose_k_mm_4_split_8 1.0568 ms 52.9% k_split=4\nSingleProcess AUTOTUNE benchmarking takes 10.1940 seconds and 0.0001 seconds precompiling for 11 choices\nAutotune Choices Stats:\n{\"num_choices\": 11, \"num_triton_choices\": 0, \"best_kernel\": \"decompose_k_mm_20_split_42\", \"best_kernel_desc\": \"k_split=20\", \"best_time\": 0.5578879714012146}\nAUTOTUNE mm(640x327680, 327680x320)\nstrides: [1, 640], [320, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n decompose_k_mm_20_split_42 0.5579 ms 100.0% k_split=20\n mm 0.5632 ms 99.1% \n decompose_k_mm_40_split_43 0.5869 ms 95.1% k_split=40\n decompose_k_mm_10_split_41 0.6001 ms 93.0% k_split=10\n decompose_k_mm_8_split_49 0.6361 ms 87.7% k_split=8\n decompose_k_mm_80_split_44 0.6572 ms 84.9% k_split=80\n decompose_k_mm_160_split_45 0.7024 ms 79.4% k_split=160\n decompose_k_mm_320_split_46 0.8630 ms 64.6% k_split=320\n decompose_k_mm_5_split_40 0.9636 ms 57.9% k_split=5\n decompose_k_mm_4_split_48 1.0568 ms 52.8% k_split=4\nSingleProcess AUTOTUNE benchmarking takes 4.1517 seconds and 0.0001 seconds precompiling for 11 choices\nAutotune Choices Stats:\n{\"num_choices\": 11, \"num_triton_choices\": 0, \"best_kernel\": \"mm\", \"best_time\": 0.3932479918003082}\nAUTOTUNE mm(320x327680, 327680x320)\nstrides: [1, 320], [320, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n mm 0.3932 ms 100.0% \n decompose_k_mm_20_split_22 0.3963 ms 99.2% k_split=20\n decompose_k_mm_40_split_23 0.4012 ms 98.0% k_split=40\n decompose_k_mm_80_split_24 0.4116 ms 95.5% k_split=80\n decompose_k_mm_10_split_21 0.4280 ms 91.9% k_split=10\n decompose_k_mm_160_split_25 0.4494 ms 87.5% k_split=160\n decompose_k_mm_320_split_26 0.5498 ms 71.5% k_split=320\n decompose_k_mm_5_split_20 0.6441 ms 61.1% k_split=5\n decompose_k_mm_640_split_27 0.6973 ms 56.4% k_split=640\n decompose_k_mm_4_split_29 0.7035 ms 55.9% k_split=4\nSingleProcess AUTOTUNE benchmarking takes 11.3703 seconds and 0.0001 seconds precompiling for 11 choices\nAutotune Choices Stats:\n{\"num_choices\": 11, \"num_triton_choices\": 0, \"best_kernel\": \"decompose_k_mm_20_split_402\", \"best_kernel_desc\": \"k_split=20\", \"best_time\": 0.3604480028152466}\nAUTOTUNE mm(320x327680, 327680x256)\nstrides: [1, 320], [256, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n decompose_k_mm_20_split_402 0.3604 ms 100.0% k_split=20\n mm 0.3625 ms 99.4% \n decompose_k_mm_40_split_403 0.3664 ms 98.4% k_split=40\n decompose_k_mm_80_split_404 0.3748 ms 96.2% k_split=80\n decompose_k_mm_10_split_401 0.3748 ms 96.2% k_split=10\n decompose_k_mm_160_split_405 0.3911 ms 92.2% k_split=160\n decompose_k_mm_5_split_400 0.4334 ms 83.2% k_split=5\n decompose_k_mm_4_split_409 0.4403 ms 81.9% k_split=4\n decompose_k_mm_320_split_406 0.4731 ms 76.2% k_split=320\n decompose_k_mm_640_split_407 0.5753 ms 62.7% k_split=640\nSingleProcess AUTOTUNE benchmarking takes 10.4363 seconds and 0.0001 seconds precompiling for 11 choices\nAutotune Choices Stats:\n{\"num_choices\": 11, \"num_triton_choices\": 0, \"best_kernel\": \"decompose_k_mm_320_split_36\", \"best_kernel_desc\": \"k_split=320\", \"best_time\": 0.03471999987959862}\nAUTOTUNE mm(16x327680, 327680x16)\nstrides: [1, 16], [16, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n decompose_k_mm_320_split_36 0.0347 ms 100.0% k_split=320\n decompose_k_mm_2560_split_39 0.0348 ms 99.7% k_split=2560\n decompose_k_mm_1280_split_38 0.0349 ms 99.4% k_split=1280\n decompose_k_mm_640_split_37 0.0356 ms 97.4% k_split=640\n decompose_k_mm_160_split_35 0.0398 ms 87.3% k_split=160\n mm 0.0409 ms 85.0% \n decompose_k_mm_80_split_34 0.0512 ms 67.8% k_split=80\n decompose_k_mm_40_split_33 0.0851 ms 40.8% k_split=40\n decompose_k_mm_20_split_32 0.1565 ms 22.2% k_split=20\n decompose_k_mm_10_split_31 0.2978 ms 11.7% k_split=10\nSingleProcess AUTOTUNE benchmarking takes 12.6107 seconds and 0.0001 seconds precompiling for 11 choices\nE1101 11:00:42.348000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 11:00:42.348000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 11:00:42.348000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 11:00:42.499000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 11:00:42.499000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 11:00:42.499000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 11:00:42.747000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 11:00:42.747000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 11:00:42.747000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 11:00:42.754000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 11:00:42.754000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 11:00:42.754000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"triton_mm_4048\", \"best_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\", \"best_time\": 0.5293759703636169, \"best_triton_pos\": 0}\nAUTOTUNE mm(327680x320, 320x640)\nstrides: [320, 1], [640, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n triton_mm_4048 0.5294 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4049 0.6011 ms 88.1% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4047 0.6236 ms 84.9% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=8\n triton_mm_4045 0.6543 ms 80.9% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4041 0.6614 ms 80.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4043 0.6635 ms 79.8% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4042 0.6748 ms 78.5% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_4046 0.6902 ms 76.7% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n mm 0.7321 ms 72.3% \n triton_mm_4039 0.8591 ms 61.6% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\nSingleProcess AUTOTUNE benchmarking takes 0.8651 seconds and 0.0002 seconds precompiling for 21 choices\nE1101 11:00:43.269000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 11:00:43.269000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 11:00:43.269000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 11:00:43.427000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 11:00:43.427000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 11:00:43.427000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 11:00:43.683000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 11:00:43.683000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 11:00:43.683000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 11:00:43.689000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 11:00:43.689000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 11:00:43.689000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"mm\", \"best_time\": 0.9625279903411865, \"best_triton_pos\": 1, \"best_triton_time\": 1.0751999616622925, \"best_triton_kernel\": \"triton_mm_4068\", \"best_triton_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\"}\nAUTOTUNE mm(327680x1280, 1280x320)\nstrides: [1280, 1], [320, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n mm 0.9625 ms 100.0% \n triton_mm_4068 1.0752 ms 89.5% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4069 1.1100 ms 86.7% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4067 1.2493 ms 77.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=8\n triton_mm_4065 1.2984 ms 74.1% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4066 1.3015 ms 74.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_4063 1.5527 ms 62.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4061 1.5647 ms 61.5% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4062 1.5647 ms 61.5% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_4059 1.6978 ms 56.7% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\nSingleProcess AUTOTUNE benchmarking takes 0.9271 seconds and 0.0001 seconds precompiling for 21 choices\nE1101 11:00:44.136000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 11:00:44.136000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 11:00:44.136000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 11:00:44.289000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 11:00:44.289000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 11:00:44.289000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 11:00:44.543000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 11:00:44.543000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 11:00:44.543000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 11:00:44.549000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 11:00:44.549000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 11:00:44.549000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"triton_mm_4088\", \"best_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\", \"best_time\": 0.343968003988266, \"best_triton_pos\": 0}\nAUTOTUNE mm(327680x320, 320x320)\nstrides: [320, 1], [320, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n triton_mm_4088 0.3440 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n mm 0.3563 ms 96.5% \n triton_mm_4089 0.3635 ms 94.6% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4085 0.3644 ms 94.4% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4086 0.3787 ms 90.8% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_4087 0.3870 ms 88.9% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=8\n triton_mm_4083 0.4484 ms 76.7% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4081 0.4502 ms 76.4% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4082 0.4555 ms 75.5% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_4079 0.4638 ms 74.2% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\nSingleProcess AUTOTUNE benchmarking takes 0.8581 seconds and 0.0002 seconds precompiling for 21 choices\nAutotune Choices Stats:\n{\"num_choices\": 12, \"num_triton_choices\": 11, \"best_kernel\": \"triton_mm_4092\", \"best_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=16, BLOCK_M=32, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=1, num_warps=2\", \"best_time\": 0.018079999834299088, \"best_triton_pos\": 0}\nAUTOTUNE mm(327680x16, 16x16)\nstrides: [16, 1], [16, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n triton_mm_4092 0.0181 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=16, BLOCK_M=32, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=1, num_warps=2\n triton_mm_4102 0.0181 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=16, BLOCK_M=128, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8\n triton_mm_4095 0.0182 ms 99.3% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=16, BLOCK_M=64, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4\n triton_mm_4097 0.0182 ms 99.3% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=16, BLOCK_M=64, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4093 0.0182 ms 99.1% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=16, BLOCK_M=32, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=2\n triton_mm_4099 0.0183 ms 98.9% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=16, BLOCK_M=128, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4098 0.0183 ms 98.6% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=16, BLOCK_M=64, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=4\n triton_mm_4100 0.0183 ms 98.6% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=16, BLOCK_M=128, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_4094 0.0184 ms 98.3% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=16, BLOCK_M=32, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=2\n triton_mm_4096 0.0184 ms 98.3% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=16, BLOCK_M=64, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4\nSingleProcess AUTOTUNE benchmarking takes 0.3395 seconds and 0.0001 seconds precompiling for 12 choices\nE1101 11:00:45.362000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 11:00:45.362000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 11:00:45.362000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 11:00:45.513000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 11:00:45.513000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 11:00:45.513000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 11:00:45.759000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 11:00:45.759000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 11:00:45.759000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 11:00:45.765000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 11:00:45.765000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 11:00:45.765000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"mm\", \"best_time\": 0.5589439868927002, \"best_triton_pos\": 1, \"best_triton_time\": 0.5613440275192261, \"best_triton_kernel\": \"triton_mm_4132\", \"best_triton_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\"}\nAUTOTUNE mm(327680x640, 640x320)\nstrides: [640, 1], [320, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n mm 0.5589 ms 100.0% \n triton_mm_4132 0.5613 ms 99.6% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4133 0.5919 ms 94.4% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4129 0.6655 ms 84.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4130 0.6665 ms 83.9% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_4131 0.6738 ms 83.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=8\n triton_mm_4127 0.7895 ms 70.8% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4125 0.7987 ms 70.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4126 0.7997 ms 69.9% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_4123 0.8550 ms 65.4% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\nSingleProcess AUTOTUNE benchmarking takes 0.8577 seconds and 0.0001 seconds precompiling for 21 choices\nE1101 11:00:46.463000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 11:00:46.463000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 11:00:46.463000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 11:00:46.619000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 11:00:46.619000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 11:00:46.619000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 11:00:46.874000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 11:00:46.874000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 11:00:46.874000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 11:00:46.880000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 11:00:46.880000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 11:00:46.880000 40715 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"triton_mm_4875\", \"best_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\", \"best_time\": 0.2986240088939667, \"best_triton_pos\": 0}\nAUTOTUNE mm(327680x320, 320x256)\nstrides: [320, 1], [256, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n triton_mm_4875 0.2986 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n mm 0.2988 ms 99.9% \n triton_mm_4880 0.2989 ms 99.9% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4873 0.2990 ms 99.9% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4881 0.3012 ms 99.1% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4874 0.3091 ms 96.6% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_4877 0.3092 ms 96.6% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_4879 0.3175 ms 94.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=8\n triton_mm_4878 0.3184 ms 93.8% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_4871 0.3686 ms 81.0% ACC_TYPE='tl.float32', ALLOW_TF32=True, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\nSingleProcess AUTOTUNE benchmarking takes 0.8573 seconds and 0.0001 seconds precompiling for 21 choices\n", "mimetype": "text/plain" }, { "type": "stream", "name": "stderr", "text": "
Traceback (most recent call last):\n File "/tmp/marimo_794/__marimo__cell_Hstk_.py", line 1, in <module>\n subprocess.call(["python", "train.py"] )\n ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.13/subprocess.py", line 397, in call\n return p.wait(timeout=timeout)\n ~~~~~~^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.13/subprocess.py", line 1280, in wait\n return self._wait(timeout=timeout)\n ~~~~~~~~~~^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.13/subprocess.py", line 2085, in _wait\n (pid, sts) = self._try_wait(0)\n ~~~~~~~~~~~~~~^^^\n File "/usr/local/lib/python3.13/subprocess.py", line 2043, in _try_wait\n (pid, sts) = os.waitpid(self.pid, wait_flags)\n ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.13/site-packages/marimo/_utils/signals.py", line 33, in __call__\n self._handler(signum, frame)\n ~~~~~~~~~~~~~^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.13/site-packages/marimo/_runtime/handlers.py", line 74, in interrupt_handler\n raise MarimoInterrupt\nKeyboardInterrupt\n0" } } ], "console": [ { "type": "stream", "name": "stdout", "text": "[kernel] import: cuda_avail=True cuda_ver=13.0 file=/marimo/kernel.py\n", "mimetype": "text/plain" }, { "type": "stream", "name": "stderr", "text": "/marimo/model.py:391: UserWarning: config module has no attribute 'eos_id' -- falling back to default None. If this is unexpected, check for a casing mismatch or rename in your config.py.\n warnings.warn(\n/marimo/model.py:391: UserWarning: config module has no attribute 'moe_fixed_capacity' -- falling back to default None. If this is unexpected, check for a casing mismatch or rename in your config.py.\n warnings.warn(\n/usr/local/lib/python3.13/site-packages/torch/profiler/profiler.py:224: UserWarning: Warning: Profiler clears events at the end of each cycle.Only events from the current cycle will be reported.To keep events across cycles, set acc_events=True.\n _warn_once(\n/usr/local/lib/python3.13/site-packages/torch/_inductor/select_algorithm.py:3686: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()\n current_out_size = out_base.storage().size()\nAutotune Choices Stats:\n{\"num_choices\": 6, \"num_triton_choices\": 6, \"best_kernel\": \"triton_flex_attention_23\", \"best_kernel_desc\": \"BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\\\"'ieee'\\\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\", \"best_time\": 0.20691199600696564, \"best_triton_pos\": 0}\nAUTOTUNE flex_attention(8x16x1024x20, 8x8x1024x20, 8x8x1024x20, 8x16x1024, 8x16x1024, 8x1x8, 8x1x8x8, 8x1x8, 8x1x8x8, 8x1024)\nstrides: [327680, 20, 320, 1], [163840, 20, 160, 1], [655360, 20, 640, 1], [16384, 1024, 1], [16384, 1024, 1], [8, 8, 1], [64, 64, 8, 1], [8, 8, 1], [64, 64, 8, 1], [1024, 1]\ndtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64\n triton_flex_attention_23 0.2069 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_20 0.2088 ms 99.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_21 0.2088 ms 99.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_22 0.2089 ms 99.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_24 0.2089 ms 99.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_25 0.2090 ms 99.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\nSingleProcess AUTOTUNE benchmarking takes 0.2819 seconds and 1.2034 seconds precompiling for 6 choices\nW1101 10:46:13.914000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf765: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.917000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf870: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.920000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf975: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.923000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1067: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.926000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf752: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.929000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf857: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.932000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf962: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.934000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1054: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.937000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf739: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.940000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf844: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.943000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf949: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.946000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1041: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.948000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf726: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.952000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf831: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.954000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf936: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.957000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1028: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.960000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf713: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.962000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf818: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.965000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf923: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.968000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1015: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.971000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf700: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.974000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf805: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.976000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf910: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.979000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1002: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.982000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf687: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.985000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf792: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.987000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf897: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.990000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf989: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.993000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf674: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.995000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf779: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:13.998000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf884: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.001000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf661: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.387000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf923: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.390000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf910: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.394000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf936: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.397000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf897: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.400000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf949: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.403000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf884: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.405000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf962: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.408000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf870: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.410000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf975: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.413000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf857: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.416000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf989: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.419000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf844: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.422000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1002: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.424000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf831: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.427000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1015: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.430000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf818: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.433000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1028: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.435000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf805: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.438000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1041: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.441000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf792: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.443000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1054: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.446000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf779: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.448000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1067: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.452000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf765: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.454000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf752: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.457000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf739: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.460000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf726: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.462000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf713: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.465000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf700: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.467000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf687: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.470000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf674: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.473000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf661: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.804000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf923: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.807000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf910: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.810000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf936: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.812000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf897: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.815000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf949: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.818000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf884: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.821000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf962: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.824000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf870: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.827000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf975: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.829000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf857: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.832000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf989: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.835000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf844: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.837000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1002: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.840000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf831: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.843000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1015: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.846000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf818: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.848000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1028: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.851000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf805: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.854000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1041: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.856000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf792: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.859000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1054: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.862000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf779: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.865000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1067: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.868000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf765: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.870000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf752: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.873000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf739: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.875000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf726: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.878000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf713: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.881000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf700: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.884000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf687: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.887000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf674: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:14.889000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf661: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.677000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf923: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.680000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf910: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.683000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf936: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.686000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf897: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.690000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf949: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.693000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf884: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.696000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf962: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.700000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf870: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.703000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf975: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.706000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf857: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.708000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf989: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.712000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf844: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.715000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1002: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.717000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf831: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.719000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1015: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.722000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf818: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.725000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1028: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.728000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf805: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.732000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1041: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.735000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf792: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.738000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1054: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.740000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf779: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.743000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf1067: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.746000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf765: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.749000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf752: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.752000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf739: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.755000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf726: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.757000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf713: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.760000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf700: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.763000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf687: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.766000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf674: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nW1101 10:46:15.768000 36509 site-packages/torch/_inductor/scheduler.py:3820] [2/0] Layout conflict detected for buf661: template expects FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[256, 1]) but layout is frozen to FixedLayout('cuda:0', torch.bfloat16, size=[12288, 256], stride=[1, 12288])\nE1101 10:46:16.529000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:16.529000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:16.529000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:46:16.624000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:16.624000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:16.624000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:46:16.780000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:16.780000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:16.780000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:46:16.786000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:16.786000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:16.786000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"mm\", \"best_time\": 0.02864000014960766, \"best_triton_pos\": 1, \"best_triton_time\": 0.030559999868273735, \"best_triton_kernel\": \"triton_mm_11\", \"best_triton_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\"}\nAUTOTUNE mm(8192x320, 320x640)\nstrides: [320, 1], [1, 320]\ndtypes: torch.bfloat16, torch.bfloat16\n mm 0.0286 ms 100.0% \n triton_mm_11 0.0306 ms 93.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_17 0.0307 ms 93.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_9 0.0319 ms 89.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_16 0.0326 ms 87.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_13 0.0327 ms 87.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_14 0.0347 ms 82.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_10 0.0347 ms 82.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_15 0.0348 ms 82.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=8\n triton_mm_7 0.0348 ms 82.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\nSingleProcess AUTOTUNE benchmarking takes 0.6742 seconds and 0.0002 seconds precompiling for 21 choices\nAutotune Choices Stats:\n{\"num_choices\": 12, \"num_triton_choices\": 11, \"best_kernel\": \"triton_mm_31\", \"best_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\", \"best_time\": 0.005183999892324209, \"best_triton_pos\": 0}\nAUTOTUNE mm(8192x16, 16x16)\nstrides: [16, 1], [1, 16]\ndtypes: torch.bfloat16, torch.bfloat16\n triton_mm_31 0.0052 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_26 0.0060 ms 87.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=32, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=1, num_warps=2\n triton_mm_32 0.0060 ms 87.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=4\n mm 0.0060 ms 86.2% \n triton_mm_30 0.0060 ms 86.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4\n triton_mm_34 0.0061 ms 85.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=128, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_35 0.0061 ms 85.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=128, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=8\n triton_mm_27 0.0061 ms 84.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=32, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=2\n triton_mm_29 0.0061 ms 84.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4\n triton_mm_33 0.0061 ms 84.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=128, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\nSingleProcess AUTOTUNE benchmarking takes 0.3330 seconds and 0.0001 seconds precompiling for 12 choices\nE1101 10:46:17.386000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:17.386000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:17.386000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:46:17.481000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:17.481000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:17.481000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:46:17.637000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:17.637000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:17.637000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:46:17.643000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:17.643000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:17.643000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"mm\", \"best_time\": 0.020287999883294106, \"best_triton_pos\": 1, \"best_triton_time\": 0.021663999184966087, \"best_triton_kernel\": \"triton_mm_44\", \"best_triton_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\"}\nAUTOTUNE mm(8192x320, 320x320)\nstrides: [320, 1], [1, 320]\ndtypes: torch.bfloat16, torch.bfloat16\n mm 0.0203 ms 100.0% \n triton_mm_44 0.0217 ms 93.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\n triton_mm_48 0.0244 ms 83.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_51 0.0245 ms 82.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_54 0.0266 ms 76.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_39 0.0278 ms 73.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=32, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8\n triton_mm_50 0.0285 ms 71.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_40 0.0285 ms 71.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8\n triton_mm_47 0.0286 ms 71.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_46 0.0297 ms 68.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\nSingleProcess AUTOTUNE benchmarking takes 0.5220 seconds and 0.0002 seconds precompiling for 21 choices\nE1101 10:46:17.967000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:17.967000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:17.967000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:46:18.534000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:18.534000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:18.534000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:46:18.699000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:18.699000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:18.699000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:46:18.705000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:18.705000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:18.705000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"triton_mm_74\", \"best_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\", \"best_time\": 0.04700800031423569, \"best_triton_pos\": 0}\nAUTOTUNE mm(8192x320, 320x1280)\nstrides: [320, 1], [1, 320]\ndtypes: torch.bfloat16, torch.bfloat16\n triton_mm_74 0.0470 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_66 0.0471 ms 99.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_70 0.0471 ms 99.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_73 0.0471 ms 99.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_68 0.0484 ms 97.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n mm 0.0499 ms 94.2% \n triton_mm_71 0.0511 ms 91.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_67 0.0524 ms 89.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_72 0.0532 ms 88.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=8\n triton_mm_63 0.0572 ms 82.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4\nSingleProcess AUTOTUNE benchmarking takes 1.0601 seconds and 0.0002 seconds precompiling for 21 choices\nE1101 10:46:18.991000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:18.991000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:18.991000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:46:19.091000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:19.091000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:19.091000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:46:19.254000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:19.254000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:19.254000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:46:19.260000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:19.260000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:19.260000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"mm\", \"best_time\": 0.030559999868273735, \"best_triton_pos\": 1, \"best_triton_time\": 0.03686400130391121, \"best_triton_kernel\": \"triton_mm_84\", \"best_triton_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\"}\nAUTOTUNE mm(8192x640, 640x320)\nstrides: [640, 1], [1, 640]\ndtypes: torch.bfloat16, torch.bfloat16\n mm 0.0306 ms 100.0% \n triton_mm_84 0.0369 ms 82.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\n triton_mm_88 0.0401 ms 76.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_91 0.0430 ms 71.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_90 0.0462 ms 66.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_79 0.0470 ms 65.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=32, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8\n triton_mm_80 0.0470 ms 65.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8\n triton_mm_87 0.0471 ms 64.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_94 0.0471 ms 64.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_93 0.0483 ms 63.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\nSingleProcess AUTOTUNE benchmarking takes 0.5544 seconds and 0.0002 seconds precompiling for 21 choices\nE1101 10:46:19.530000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:19.530000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:19.530000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:46:19.624000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:19.624000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:19.624000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:46:19.776000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:19.776000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:19.776000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:46:19.782000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:19.782000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:19.782000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"mm\", \"best_time\": 0.018271999433636665, \"best_triton_pos\": 1, \"best_triton_time\": 0.01958400011062622, \"best_triton_kernel\": \"triton_mm_104\", \"best_triton_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\"}\nAUTOTUNE mm(8192x256, 256x320)\nstrides: [256, 1], [1, 256]\ndtypes: torch.bfloat16, torch.bfloat16\n mm 0.0183 ms 100.0% \n triton_mm_104 0.0196 ms 93.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\n triton_mm_108 0.0204 ms 89.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_111 0.0217 ms 84.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_114 0.0225 ms 81.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_100 0.0236 ms 77.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8\n triton_mm_106 0.0245 ms 74.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_107 0.0256 ms 71.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_99 0.0257 ms 71.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=32, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8\n triton_mm_110 0.0264 ms 69.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\nSingleProcess AUTOTUNE benchmarking takes 0.5198 seconds and 0.0002 seconds precompiling for 21 choices\nE1101 10:46:20.156000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:20.156000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:20.156000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:46:20.252000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:20.252000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:20.252000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:46:20.410000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:20.410000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:20.410000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:46:20.416000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:20.416000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:20.416000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"triton_mm_3263\", \"best_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\", \"best_time\": 0.02454400062561035, \"best_triton_pos\": 0}\nAUTOTUNE mm(256x320, 320x12288)\nstrides: [320, 1], [1, 320]\ndtypes: torch.bfloat16, torch.bfloat16\n triton_mm_3263 0.0245 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3259 0.0246 ms 99.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\n mm 0.0263 ms 93.2% \n triton_mm_3269 0.0266 ms 92.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3268 0.0317 ms 77.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3266 0.0319 ms 76.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_3265 0.0327 ms 75.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3255 0.0347 ms 70.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8\n triton_mm_3261 0.0348 ms 70.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3262 0.0360 ms 68.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\nSingleProcess AUTOTUNE benchmarking takes 0.5343 seconds and 0.0001 seconds precompiling for 21 choices\nE1101 10:46:27.235000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:27.235000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:27.235000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:46:27.368000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:27.368000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:27.368000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:46:27.606000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:27.606000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:27.606000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:46:27.612000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:27.612000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:27.612000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 30, \"num_triton_choices\": 20, \"best_kernel\": \"decompose_k_mm_16_split_7\", \"best_kernel_desc\": \"k_split=16\", \"best_time\": 0.027871999889612198, \"best_triton_pos\": 10, \"best_triton_time\": 0.05526399984955788, \"best_triton_kernel\": \"triton_mm_1656\", \"best_triton_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4\"}\nAUTOTUNE mm(256x12288, 12288x320)\nstrides: [12288, 1], [320, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n decompose_k_mm_16_split_7 0.0279 ms 100.0% k_split=16\n decompose_k_mm_12_split_2 0.0286 ms 97.3% k_split=12\n decompose_k_mm_8_split_6 0.0287 ms 97.2% k_split=8\n mm 0.0287 ms 97.1% \n decompose_k_mm_24_split_3 0.0307 ms 90.8% k_split=24\n decompose_k_mm_6_split_1 0.0309 ms 90.1% k_split=6\n decompose_k_mm_32_split_8 0.0327 ms 85.1% k_split=32\n decompose_k_mm_4_split_5 0.0399 ms 69.8% k_split=4\n decompose_k_mm_3_split_0 0.0419 ms 66.5% k_split=3\n decompose_k_mm_2_split_4 0.0462 ms 60.4% k_split=2\nSingleProcess AUTOTUNE benchmarking takes 7.1899 seconds and 0.0001 seconds precompiling for 30 choices\nE1101 10:46:27.892000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:27.892000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:27.892000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:46:27.985000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:27.985000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:27.985000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:46:28.138000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:28.138000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:28.138000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:46:28.144000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:46:28.144000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:46:28.144000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"mm\", \"best_time\": 0.021888000890612602, \"best_triton_pos\": 1, \"best_triton_time\": 0.023744000121951103, \"best_triton_kernel\": \"triton_mm_1679\", \"best_triton_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\"}\nAUTOTUNE mm(12288x256, 256x320)\nstrides: [1, 12288], [320, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n mm 0.0219 ms 100.0% \n triton_mm_1679 0.0237 ms 92.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\n triton_mm_1683 0.0244 ms 89.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_1688 0.0266 ms 82.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_1689 0.0266 ms 82.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_1681 0.0278 ms 78.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_1685 0.0285 ms 76.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_1682 0.0299 ms 73.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_1686 0.0306 ms 71.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_1674 0.0307 ms 71.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=32, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8\nSingleProcess AUTOTUNE benchmarking takes 0.5223 seconds and 0.0001 seconds precompiling for 21 choices\nAutotune Choices Stats:\n{\"num_choices\": 13, \"num_triton_choices\": 13, \"best_kernel\": \"triton_flex_attention_backward_3695\", \"best_kernel_desc\": \"BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_M1=32, BLOCK_M2=128, BLOCK_N=64, BLOCK_N1=128, BLOCK_N2=32, FLOAT32_PRECISION=\\\"'ieee'\\\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\", \"best_time\": 0.6520959734916687, \"best_triton_pos\": 0}\nAUTOTUNE flex_attention_backward(8x16x1024x20, 8x8x1024x20, 8x8x1024x20, 8x16x1024, 8x16x1024, 8x16x1024x20, 8x16x1024x20, 8x8x1024x20, 8x1x8, 8x1x8x8, 8x1x8, 8x1x8x8, 8x1x8, 8x1x8x8, 8x1x8, 8x1x8x8, 8x1024)\nstrides: [327680, 20, 320, 1], [163840, 20, 160, 1], [655360, 20, 640, 1], [16384, 1024, 1], [16384, 1024, 1], [327680, 20480, 20, 1], [327680, 20, 320, 1], [163840, 20, 160, 1], [8, 8, 1], [64, 64, 8, 1], [8, 8, 1], [64, 64, 8, 1], [8, 8, 1], [64, 64, 8, 1], [8, 8, 1], [64, 64, 8, 1], [1024, 1]\ndtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64\n triton_flex_attention_backward_3695 0.6521 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_M1=32, BLOCK_M2=128, BLOCK_N=64, BLOCK_N1=128, BLOCK_N2=32, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_backward_3691 0.6879 ms 94.8% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_backward_3694 0.6902 ms 94.5% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_backward_3693 0.6934 ms 94.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_backward_3692 0.6943 ms 93.9% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_M1=64, BLOCK_M2=64, BLOCK_N=64, BLOCK_N1=64, BLOCK_N2=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_backward_3687 0.7405 ms 88.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_backward_3690 0.7424 ms 87.8% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_backward_3688 0.7434 ms 87.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_backward_3689 0.7434 ms 87.7% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_M1=32, BLOCK_M2=64, BLOCK_N=64, BLOCK_N1=64, BLOCK_N2=32, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_backward_3683 1.3548 ms 48.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_M1=32, BLOCK_M2=32, BLOCK_N=64, BLOCK_N1=32, BLOCK_N2=32, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=True, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\nSingleProcess AUTOTUNE benchmarking takes 0.7854 seconds and 1.9881 seconds precompiling for 13 choices\nE1101 10:48:32.913000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:32.913000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:32.913000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:33.028000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:33.028000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:33.028000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:33.241000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:33.241000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:33.241000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:33.247000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:33.247000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:33.247000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"mm\", \"best_time\": 0.057312000542879105, \"best_triton_pos\": 1, \"best_triton_time\": 0.08591999858617783, \"best_triton_kernel\": \"triton_mm_3599\", \"best_triton_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\"}\nAUTOTUNE mm(1280x8192, 8192x320)\nstrides: [1, 1280], [320, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n mm 0.0573 ms 100.0% \n triton_mm_3599 0.0859 ms 66.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\n triton_mm_3602 0.0954 ms 60.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_3594 0.0982 ms 58.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=32, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8\n triton_mm_3603 0.0983 ms 58.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3595 0.1004 ms 57.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8\n triton_mm_3606 0.1044 ms 54.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_3596 0.1045 ms 54.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4\n triton_mm_3593 0.1085 ms 52.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=32, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4\n triton_mm_3605 0.1301 ms 44.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\nSingleProcess AUTOTUNE benchmarking takes 0.6935 seconds and 0.0002 seconds precompiling for 21 choices\nE1101 10:48:34.087000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:34.087000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:34.087000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:34.203000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:34.203000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:34.203000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:34.415000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:34.415000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:34.415000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:34.422000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:34.422000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:34.422000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"mm\", \"best_time\": 0.03478400036692619, \"best_triton_pos\": 1, \"best_triton_time\": 0.056384000927209854, \"best_triton_kernel\": \"triton_mm_3556\", \"best_triton_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4\"}\nAUTOTUNE mm(320x8192, 8192x640)\nstrides: [1, 320], [640, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n mm 0.0348 ms 100.0% \n triton_mm_3556 0.0564 ms 61.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4\n triton_mm_3554 0.0688 ms 50.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=32, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8\n triton_mm_3555 0.0696 ms 50.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8\n triton_mm_3559 0.0811 ms 42.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\n triton_mm_3553 0.0872 ms 39.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=32, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4\n triton_mm_3562 0.0933 ms 37.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_3563 0.0942 ms 36.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3566 0.0962 ms 36.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_3561 0.1300 ms 26.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\nSingleProcess AUTOTUNE benchmarking takes 0.6804 seconds and 0.0001 seconds precompiling for 21 choices\nE1101 10:48:35.150000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:35.150000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:35.150000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:35.264000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:35.264000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:35.264000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:35.475000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:35.475000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:35.475000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:35.481000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:35.481000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:35.481000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"mm\", \"best_time\": 0.03478400036692619, \"best_triton_pos\": 1, \"best_triton_time\": 0.053119998425245285, \"best_triton_kernel\": \"triton_mm_3700\", \"best_triton_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4\"}\nAUTOTUNE mm(640x8192, 8192x320)\nstrides: [1, 640], [320, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n mm 0.0348 ms 100.0% \n triton_mm_3700 0.0531 ms 65.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4\n triton_mm_3699 0.0694 ms 50.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8\n triton_mm_3698 0.0696 ms 50.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=32, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8\n triton_mm_3703 0.0809 ms 43.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\n triton_mm_3697 0.0842 ms 41.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=32, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4\n triton_mm_3706 0.0921 ms 37.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_3707 0.0954 ms 36.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3710 0.0962 ms 36.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_3709 0.1290 ms 27.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\nSingleProcess AUTOTUNE benchmarking takes 0.6714 seconds and 0.0002 seconds precompiling for 21 choices\nE1101 10:48:36.359000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:36.359000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:36.359000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:36.472000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:36.472000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:36.472000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:36.681000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:36.681000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:36.681000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:36.686000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:36.686000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:36.686000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"mm\", \"best_time\": 0.02457600086927414, \"best_triton_pos\": 1, \"best_triton_time\": 0.04291199892759323, \"best_triton_kernel\": \"triton_mm_3636\", \"best_triton_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4\"}\nAUTOTUNE mm(320x8192, 8192x320)\nstrides: [1, 320], [320, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n mm 0.0246 ms 100.0% \n triton_mm_3636 0.0429 ms 57.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4\n triton_mm_3634 0.0634 ms 38.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=32, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8\n triton_mm_3635 0.0646 ms 38.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8\n triton_mm_3639 0.0787 ms 31.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\n triton_mm_3633 0.0819 ms 30.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=32, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4\n triton_mm_3642 0.0880 ms 27.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_3643 0.0913 ms 26.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3646 0.0941 ms 26.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_3641 0.1247 ms 19.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\nSingleProcess AUTOTUNE benchmarking takes 0.6591 seconds and 0.0001 seconds precompiling for 21 choices\nE1101 10:48:37.505000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:37.505000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:37.505000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:37.621000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:37.621000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:37.621000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:37.832000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:37.832000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:37.832000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:37.837000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:37.837000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:37.837000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"mm\", \"best_time\": 0.024351999163627625, \"best_triton_pos\": 1, \"best_triton_time\": 0.04297599941492081, \"best_triton_kernel\": \"triton_mm_5028\", \"best_triton_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4\"}\nAUTOTUNE mm(320x8192, 8192x256)\nstrides: [1, 320], [256, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n mm 0.0244 ms 100.0% \n triton_mm_5028 0.0430 ms 56.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4\n triton_mm_5027 0.0653 ms 37.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8\n triton_mm_5026 0.0655 ms 37.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=32, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8\n triton_mm_5031 0.0757 ms 32.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\n triton_mm_5025 0.0778 ms 31.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=32, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4\n triton_mm_5034 0.0860 ms 28.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_5035 0.0901 ms 27.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_5038 0.0932 ms 26.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_5033 0.1241 ms 19.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\nSingleProcess AUTOTUNE benchmarking takes 0.6647 seconds and 0.0001 seconds precompiling for 21 choices\nAutotune Choices Stats:\n{\"num_choices\": 7, \"num_triton_choices\": 0, \"best_kernel\": \"decompose_k_mm_64_split_293\", \"best_kernel_desc\": \"k_split=64\", \"best_time\": 0.008383999578654766}\nAUTOTUNE mm(16x8192, 8192x16)\nstrides: [1, 16], [16, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n decompose_k_mm_64_split_293 0.0084 ms 100.0% k_split=64\n decompose_k_mm_32_split_292 0.0103 ms 81.6% k_split=32\n decompose_k_mm_16_split_291 0.0122 ms 68.6% k_split=16\n decompose_k_mm_8_split_290 0.0164 ms 51.3% k_split=8\n mm 0.0197 ms 42.5% \n decompose_k_mm_4_split_289 0.0237 ms 35.4% k_split=4\n decompose_k_mm_2_split_288 0.0390 ms 21.5% k_split=2\nSingleProcess AUTOTUNE benchmarking takes 3.7511 seconds and 0.0001 seconds precompiling for 7 choices\nE1101 10:48:44.535000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:44.535000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:44.535000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:44.627000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:44.627000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:44.627000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:44.776000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:44.776000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:44.776000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:44.781000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:44.781000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:44.781000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"mm\", \"best_time\": 0.02844800055027008, \"best_triton_pos\": 1, \"best_triton_time\": 0.028672000393271446, \"best_triton_kernel\": \"triton_mm_3581\", \"best_triton_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\"}\nAUTOTUNE mm(8192x320, 320x640)\nstrides: [320, 1], [640, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n mm 0.0284 ms 100.0% \n triton_mm_3581 0.0287 ms 99.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3588 0.0298 ms 95.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3583 0.0299 ms 95.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3585 0.0307 ms 92.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3589 0.0317 ms 89.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3582 0.0326 ms 87.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_3579 0.0327 ms 86.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\n triton_mm_3587 0.0337 ms 84.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=8\n triton_mm_3586 0.0360 ms 79.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\nSingleProcess AUTOTUNE benchmarking takes 0.5241 seconds and 0.0002 seconds precompiling for 21 choices\nE1101 10:48:45.093000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:45.093000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:45.093000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:45.192000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:45.192000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:45.192000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:45.355000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:45.355000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:45.355000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:45.360000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:45.360000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:45.360000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"mm\", \"best_time\": 0.050303999334573746, \"best_triton_pos\": 1, \"best_triton_time\": 0.05929600074887276, \"best_triton_kernel\": \"triton_mm_3626\", \"best_triton_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\"}\nAUTOTUNE mm(8192x1280, 1280x320)\nstrides: [1280, 1], [320, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n mm 0.0503 ms 100.0% \n triton_mm_3626 0.0593 ms 84.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_3625 0.0606 ms 83.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3623 0.0612 ms 82.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3619 0.0614 ms 82.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\n triton_mm_3628 0.0614 ms 81.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3621 0.0626 ms 80.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3622 0.0676 ms 74.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_3618 0.0696 ms 72.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4\n triton_mm_3629 0.0717 ms 70.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\nSingleProcess AUTOTUNE benchmarking takes 0.5767 seconds and 0.0002 seconds precompiling for 21 choices\nE1101 10:48:45.624000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:45.624000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:45.624000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:45.714000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:45.714000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:45.714000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:45.860000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:45.860000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:45.860000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:45.865000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:45.865000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:45.865000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"mm\", \"best_time\": 0.018400000408291817, \"best_triton_pos\": 1, \"best_triton_time\": 0.020479999482631683, \"best_triton_kernel\": \"triton_mm_3663\", \"best_triton_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\"}\nAUTOTUNE mm(8192x320, 320x320)\nstrides: [320, 1], [320, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n mm 0.0184 ms 100.0% \n triton_mm_3663 0.0205 ms 89.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3666 0.0224 ms 82.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_3665 0.0236 ms 77.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3659 0.0238 ms 77.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\n triton_mm_3662 0.0245 ms 75.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_3669 0.0245 ms 75.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3661 0.0265 ms 69.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3668 0.0266 ms 69.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3654 0.0278 ms 66.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=32, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8\nSingleProcess AUTOTUNE benchmarking takes 0.5028 seconds and 0.0001 seconds precompiling for 21 choices\nAutotune Choices Stats:\n{\"num_choices\": 12, \"num_triton_choices\": 11, \"best_kernel\": \"triton_mm_3672\", \"best_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=32, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=1, num_warps=2\", \"best_time\": 0.006047999951988459, \"best_triton_pos\": 0}\nAUTOTUNE mm(8192x16, 16x16)\nstrides: [16, 1], [16, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n triton_mm_3672 0.0060 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=32, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=1, num_warps=2\n triton_mm_3678 0.0061 ms 99.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=4\n triton_mm_3681 0.0061 ms 99.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=128, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=8\n triton_mm_3673 0.0061 ms 99.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=32, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=2\n triton_mm_3674 0.0061 ms 99.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=32, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=2\n triton_mm_3675 0.0061 ms 99.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4\n triton_mm_3676 0.0061 ms 99.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4\n triton_mm_3677 0.0061 ms 99.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3680 0.0061 ms 99.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=128, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_3682 0.0061 ms 99.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=128, BLOCK_N=16, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8\nSingleProcess AUTOTUNE benchmarking takes 0.3310 seconds and 0.0001 seconds precompiling for 12 choices\nE1101 10:48:46.487000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:46.487000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:46.487000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:46.580000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:46.580000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:46.580000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:46.733000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:46.733000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:46.733000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:46.739000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:46.739000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:46.739000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"mm\", \"best_time\": 0.028351999819278717, \"best_triton_pos\": 1, \"best_triton_time\": 0.03590400144457817, \"best_triton_kernel\": \"triton_mm_3723\", \"best_triton_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\"}\nAUTOTUNE mm(8192x640, 640x320)\nstrides: [640, 1], [320, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n mm 0.0284 ms 100.0% \n triton_mm_3723 0.0359 ms 79.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\n triton_mm_3727 0.0360 ms 78.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3733 0.0408 ms 69.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3730 0.0409 ms 69.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_3725 0.0422 ms 67.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3718 0.0430 ms 66.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=32, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8\n triton_mm_3732 0.0430 ms 66.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_3726 0.0430 ms 65.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_3729 0.0430 ms 65.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\nSingleProcess AUTOTUNE benchmarking takes 0.5305 seconds and 0.0002 seconds precompiling for 21 choices\nE1101 10:48:47.134000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:47.134000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:47.134000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:47.223000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:47.223000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:47.223000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:47.371000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:47.371000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:47.371000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nE1101 10:48:47.377000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Runtime error during autotuning: \nE1101 10:48:47.377000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:48:47.377000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/0] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"triton_mm_5055\", \"best_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\", \"best_time\": 0.01759999990463257, \"best_triton_pos\": 0}\nAUTOTUNE mm(8192x320, 320x256)\nstrides: [320, 1], [256, 1]\ndtypes: torch.bfloat16, torch.bfloat16\n triton_mm_5055 0.0176 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_5061 0.0183 ms 96.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_5051 0.0196 ms 90.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\n mm 0.0203 ms 86.8% \n triton_mm_5054 0.0217 ms 81.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_5058 0.0224 ms 78.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_5046 0.0236 ms 74.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=32, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8\n triton_mm_5057 0.0244 ms 72.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_5060 0.0245 ms 71.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_5053 0.0246 ms 71.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\nSingleProcess AUTOTUNE benchmarking takes 0.5021 seconds and 0.0001 seconds precompiling for 21 choices\nAutotune Choices Stats:\n{\"num_choices\": 6, \"num_triton_choices\": 6, \"best_kernel\": \"triton_flex_attention_6678\", \"best_kernel_desc\": \"BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\\\"'ieee'\\\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=False, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\", \"best_time\": 0.19990399479866028, \"best_triton_pos\": 0}\nAUTOTUNE flex_attention(8x16x1024x20, 8x8x1024x20, 8x8x1024x20, 8x16x1024, 8x16x1024, 8x1x8, 8x1x8x8, 8x1x8, 8x1x8x8, 8x1024)\nstrides: [327680, 20480, 20, 1], [163840, 20480, 20, 1], [655360, 20, 640, 1], [16384, 1024, 1], [16384, 1024, 1], [8, 8, 1], [64, 64, 8, 1], [8, 8, 1], [64, 64, 8, 1], [1024, 1]\ndtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, torch.int32, torch.int32, torch.int32, torch.int32, torch.int64\n triton_flex_attention_6678 0.1999 ms 100.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=False, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_6680 0.2007 ms 99.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=False, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_6676 0.2017 ms 99.1% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=False, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_6677 0.2019 ms 99.0% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=False, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_6681 0.2022 ms 98.9% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=False, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\n triton_flex_attention_6679 0.2028 ms 98.6% BLOCKS_ARE_CONTIGUOUS=False, BLOCK_M=128, BLOCK_N=64, FLOAT32_PRECISION=\"'ieee'\", GQA_SHARED_HEADS=2, HAS_FULL_BLOCKS=True, IS_DIVISIBLE=True, OUTPUT_LOGSUMEXP=False, OUTPUT_MAX=False, PRESCALE_QK=False, QK_HEAD_DIM=20, QK_HEAD_DIM_ROUNDED=32, ROWS_GUARANTEED_SAFE=False, SAFE_HEAD_DIM=False, SM_SCALE=0.22360679774997896, SPARSE_KV_BLOCK_SIZE=128, SPARSE_Q_BLOCK_SIZE=128, USE_TMA=False, V_HEAD_DIM=20, V_HEAD_DIM_ROUNDED=32, WRITE_DQ=True, num_stages=2, num_warps=4\nSingleProcess AUTOTUNE benchmarking takes 0.2741 seconds and 0.7981 seconds precompiling for 6 choices\nE1101 10:51:35.378000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/2] Runtime error during autotuning: \nE1101 10:51:35.378000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/2] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:51:35.378000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/2] Ignoring this choice.\nE1101 10:51:35.515000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/2] Runtime error during autotuning: \nE1101 10:51:35.515000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/2] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 147456 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:51:35.515000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/2] Ignoring this choice.\nE1101 10:51:35.734000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/2] Runtime error during autotuning: \nE1101 10:51:35.734000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/2] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 131072 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:51:35.734000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/2] Ignoring this choice.\nE1101 10:51:35.740000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/2] Runtime error during autotuning: \nE1101 10:51:35.740000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/2] No valid triton configs. OutOfMemoryError: out of resource: triton_mm Required: 196608 Hardware limit:101376 Reducing block sizes or `num_stages` may help.. \nE1101 10:51:35.740000 36509 site-packages/torch/_inductor/select_algorithm.py:3924] [2/2] Ignoring this choice.\nAutotune Choices Stats:\n{\"num_choices\": 21, \"num_triton_choices\": 20, \"best_kernel\": \"mm\", \"best_time\": 0.14313599467277527, \"best_triton_pos\": 1, \"best_triton_time\": 0.14467200636863708, \"best_triton_kernel\": \"triton_mm_8304\", \"best_triton_kernel_desc\": \"ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\"}\nAUTOTUNE mm(4096x320, 320x12288)\nstrides: [320, 1], [1, 320]\ndtypes: torch.bfloat16, torch.bfloat16\n mm 0.1431 ms 100.0% \n triton_mm_8304 0.1447 ms 98.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_8305 0.1589 ms 90.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_8303 0.1714 ms 83.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=8\n triton_mm_8301 0.1716 ms 83.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_8297 0.1758 ms 81.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_8299 0.1764 ms 81.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4\n triton_mm_8302 0.1815 ms 78.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_8298 0.1820 ms 78.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8\n triton_mm_8295 0.2191 ms 65.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8\nSingleProcess AUTOTUNE benchmarking takes 0.8009 seconds and 0.0001 seconds precompiling for 21 choices\n", "mimetype": "text/plain" }, { "type": "stream", "name": "stdout", "text": "[CLI] Overriding config.py: {'use_liger': False}\n[Setup] device=cuda (cuda)\n\n=== Speed diagnosis: attention implementation ===\n device: cuda\n config pattern: 'dense'\n attention backend: lazy torch.compile(torch.nn.attention.flex_attention) (pending first call)\n FP8 training: off (config.use_fp8=False)\n DenseMHA calls kernel.fused_attention; no Splash, xFormers, or flash_attn dispatch is implemented in kernel.py.\n [OK] architecture gate: this pattern uses DenseMHA and the FlexAttention path.\n[Tokenizer] Loaded, vocab_size=12288\n[Dataset] Found 3 files in ./training_data (curriculum OFF)\n[Dataset] Cache hit -> ./training_data/.token_cache/tokens_3894af946a164b07.npy (879,185,417 tokens)\n[Engram] layers=[1, 8] max_ngram=3 compressed_vocab=10304 (raw 12288)\n[DocMask] ON: BOS=12286, EOS=12287, windows=[None]\n[Model] 21.16M params, pattern=dense\n[Engram] 2,466,304 engram params\n[Precision] autocast dtype=torch.bfloat16 (device_kind=cuda)\n[Compile] Note: hardest_file_check uses batch=1 (different shape from learnability_check/gpu_profile_check's batch=8) -- expect one additional recompilation when that section runs, not a bug.\n[Compile] mode=max-autotune\n\n=== Learnability check: SKIPPED (--skip-learnability) ===\n Model is at random initialization -- hardest_file_check is also skipped (its loss ranking would be meaningless on untrained weights). Only gpu_profile_check's speed/roofline numbers run, since those reflect architecture shape only, not learned behavior.\n\n=== Hardest-file check: SKIPPED (--skip-learnability) ===\n Model is at random initialization -- the loss ranking would be meaningless.\n\n=== GPU profiling (CUDA) ===\n device: NVIDIA RTX PRO 6000 Blackwell Server Edition\n SMs: 188 | total VRAM: 102.0 GB\n\n --- torch.profiler: 3 warmup + 8 recorded steps ---\n\n Top operators by self_cuda_time_total (8-step window):\n------------------------------------------------------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ \n Name Self CPU % Self CPU CPU total % CPU total CPU time avg Self CUDA Self CUDA % CUDA total CUDA time avg CPU Mem Self CPU Mem CUDA Mem Self CUDA Mem # of Calls Total MFLOPs \n------------------------------------------------------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ \n## Call CompiledFxGraph frr3dragej4ihoykk2tsooswynnp... 0.00% 0.000us 0.00% 0.000us 0.000us 68.057ms 56.86% 68.057ms 8.507ms 0 B 0 B 0 B 0 B 8 -- \n CompiledFunctionBackward 1.41% 2.146ms 10.71% 16.317ms 1.020ms 67.437ms 56.34% 67.454ms 4.216ms 0 B 0 B -40.00 KB 0 B 16 -- \n## Call CompiledFxGraph fj262purbsknvkmw7tvr5ifalioi... 0.00% 0.000us 0.00% 0.000us 0.000us 53.641ms 44.82% 53.641ms 6.705ms 0 B 0 B 0 B 0 B 8 -- \n CompiledFunction 2.46% 3.748ms 12.82% 19.530ms 1.221ms 51.862ms 43.33% 51.946ms 3.247ms 0 B 0 B -1.50 KB 0 B 16 -- \ntriton_tem_fused__unsafe_view_flex_attention_backwar... 0.00% 0.000us 0.00% 0.000us 0.000us 18.361ms 15.34% 18.361ms 143.447us 0 B 0 B 0 B 0 B 128 -- \nvoid cutlass::Kernel2
0" } } ], "console": [ { "type": "stream", "name": "stderr", "text": "WARNING: The directory '/home/marimo/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.\n", "mimetype": "text/plain" }, { "type": "stream", "name": "stdout", "text": "Requirement already satisfied: liger-kernel in /tmp/uv-venv/lib/python3.13/site-packages (0.8.5)\nRequirement already satisfied: rustbpe in /tmp/uv-venv/lib/python3.13/site-packages (0.1.0)\nRequirement already satisfied: torchao in /tmp/uv-venv/lib/python3.13/site-packages (0.18.0)\nRequirement already satisfied: triton in /tmp/uv-venv/lib/python3.13/site-packages (3.9.0)\nCollecting accelerate\n Downloading accelerate-1.15.0-py3-none-any.whl.metadata (19 kB)\nRequirement already satisfied: safetensors in /tmp/uv-venv/lib/python3.13/site-packages (0.8.0)\nCollecting helion\n Downloading helion-1.5.2-py3-none-any.whl.metadata (23 kB)\nRequirement already satisfied: torch>=2.1.2 in /usr/local/lib/python3.13/site-packages (from liger-kernel) (2.11.0)\nRequirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.13/site-packages (from accelerate) (2.4.6)\nRequirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.13/site-packages (from accelerate) (26.2)\nRequirement already satisfied: psutil in /usr/local/lib/python3.13/site-packages (from accelerate) (7.2.2)\nRequirement already satisfied: pyyaml in /usr/local/lib/python3.13/site-packages (from accelerate) (6.0.3)\nRequirement already satisfied: huggingface_hub>=0.21.0 in /tmp/uv-venv/lib/python3.13/site-packages (from accelerate) (1.33.0)\nCollecting filecheck (from helion)\n Downloading filecheck-1.0.6-py3-none-any.whl.metadata (5.2 kB)\nRequirement already satisfied: rich in /tmp/uv-venv/lib/python3.13/site-packages (from helion) (15.0.0)\nRequirement already satisfied: scikit-learn>=1.3.0 in /usr/local/lib/python3.13/site-packages (from helion) (1.9.0)\nRequirement already satisfied: typing-extensions>=4.0.0 in /usr/local/lib/python3.13/site-packages (from helion) (4.16.0)\nRequirement already satisfied: click<9.0.0,>=8.4.2 in /usr/local/lib/python3.13/site-packages (from huggingface_hub>=0.21.0->accelerate) (8.4.2)\nRequirement already satisfied: filelock>=3.10.0 in /tmp/uv-venv/lib/python3.13/site-packages (from huggingface_hub>=0.21.0->accelerate) (4.2.6)\nRequirement already satisfied: fsspec>=2023.5.0 in /tmp/uv-venv/lib/python3.13/site-packages (from huggingface_hub>=0.21.0->accelerate) (2026.9.0)\nRequirement already satisfied: hf-xet<2.0.0,>=1.6.0 in /tmp/uv-venv/lib/python3.13/site-packages (from huggingface_hub>=0.21.0->accelerate) (1.8.0)\nRequirement already satisfied: httpx<1,>=0.23.0 in /tmp/uv-venv/lib/python3.13/site-packages (from huggingface_hub>=0.21.0->accelerate) (0.28.1)\nRequirement already satisfied: tqdm>=4.42.1 in /tmp/uv-venv/lib/python3.13/site-packages (from huggingface_hub>=0.21.0->accelerate) (4.70.1)\nRequirement already satisfied: anyio in /tmp/uv-venv/lib/python3.13/site-packages (from httpx<1,>=0.23.0->huggingface_hub>=0.21.0->accelerate) (4.15.1)\nRequirement already satisfied: certifi in /tmp/uv-venv/lib/python3.13/site-packages (from httpx<1,>=0.23.0->huggingface_hub>=0.21.0->accelerate) (2026.10.23)\nRequirement already satisfied: httpcore==1.* in /tmp/uv-venv/lib/python3.13/site-packages (from httpx<1,>=0.23.0->huggingface_hub>=0.21.0->accelerate) (1.0.9)\nRequirement already satisfied: idna in /tmp/uv-venv/lib/python3.13/site-packages (from httpx<1,>=0.23.0->huggingface_hub>=0.21.0->accelerate) (3.20)\nRequirement already satisfied: h11>=0.16 in /tmp/uv-venv/lib/python3.13/site-packages (from httpcore==1.*->httpx<1,>=0.23.0->huggingface_hub>=0.21.0->accelerate) (0.16.0)\nRequirement already satisfied: scipy>=1.10.0 in /usr/local/lib/python3.13/site-packages (from scikit-learn>=1.3.0->helion) (1.18.0)\nRequirement already satisfied: joblib>=1.4.0 in /usr/local/lib/python3.13/site-packages (from scikit-learn>=1.3.0->helion) (1.5.3)\nRequirement already satisfied: narwhals>=2.0.1 in /usr/local/lib/python3.13/site-packages (from scikit-learn>=1.3.0->helion) (2.24.0)\nRequirement already satisfied: threadpoolctl>=3.5.0 in /usr/local/lib/python3.13/site-packages (from scikit-learn>=1.3.0->helion) (3.6.0)\nCollecting setuptools<82 (from torch>=2.1.2->liger-kernel)\n Downloading setuptools-81.0.0-py3-none-any.whl.metadata (6.6 kB)\nRequirement already satisfied: sympy>=1.13.3 in /usr/local/lib/python3.13/site-packages (from torch>=2.1.2->liger-kernel) (1.14.0)\nRequirement already satisfied: networkx>=2.5.1 in /usr/local/lib/python3.13/site-packages (from torch>=2.1.2->liger-kernel) (3.6.1)\nRequirement already satisfied: jinja2 in /usr/local/lib/python3.13/site-packages (from torch>=2.1.2->liger-kernel) (3.1.6)\nRequirement already satisfied: cuda-toolkit==13.0.2 in /usr/local/lib/python3.13/site-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.2; platform_system == \"Linux\"->torch>=2.1.2->liger-kernel) (13.0.2)\nRequirement already satisfied: cuda-bindings<14,>=13.0.3 in /usr/local/lib/python3.13/site-packages (from torch>=2.1.2->liger-kernel) (13.3.1)\nRequirement already satisfied: nvidia-cudnn-cu13==9.19.0.56 in /usr/local/lib/python3.13/site-packages (from torch>=2.1.2->liger-kernel) (9.19.0.56)\nRequirement already satisfied: nvidia-cusparselt-cu13==0.8.0 in /usr/local/lib/python3.13/site-packages (from torch>=2.1.2->liger-kernel) (0.8.0)\nRequirement already satisfied: nvidia-nccl-cu13==2.28.9 in /usr/local/lib/python3.13/site-packages (from torch>=2.1.2->liger-kernel) (2.28.9)\nRequirement already satisfied: nvidia-nvshmem-cu13==3.4.5 in /usr/local/lib/python3.13/site-packages (from torch>=2.1.2->liger-kernel) (3.4.5)\nCollecting triton\n Downloading triton-3.6.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (1.7 kB)\nRequirement already satisfied: nvidia-cublas==13.1.0.3.* in /usr/local/lib/python3.13/site-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.2; platform_system == \"Linux\"->torch>=2.1.2->liger-kernel) (13.1.0.3)\nRequirement already satisfied: nvidia-cuda-runtime==13.0.96.* in /usr/local/lib/python3.13/site-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.2; platform_system == \"Linux\"->torch>=2.1.2->liger-kernel) (13.0.96)\nRequirement already satisfied: nvidia-cufft==12.0.0.61.* in /usr/local/lib/python3.13/site-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.2; platform_system == \"Linux\"->torch>=2.1.2->liger-kernel) (12.0.0.61)\nRequirement already satisfied: nvidia-cufile==1.15.1.6.* in /usr/local/lib/python3.13/site-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.2; platform_system == \"Linux\"->torch>=2.1.2->liger-kernel) (1.15.1.6)\nRequirement already satisfied: nvidia-cuda-cupti==13.0.85.* in /usr/local/lib/python3.13/site-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.2; platform_system == \"Linux\"->torch>=2.1.2->liger-kernel) (13.0.85)\nRequirement already satisfied: nvidia-curand==10.4.0.35.* in /usr/local/lib/python3.13/site-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.2; platform_system == \"Linux\"->torch>=2.1.2->liger-kernel) (10.4.0.35)\nRequirement already satisfied: nvidia-cusolver==12.0.4.66.* in /usr/local/lib/python3.13/site-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.2; platform_system == \"Linux\"->torch>=2.1.2->liger-kernel) (12.0.4.66)\nRequirement already satisfied: nvidia-cusparse==12.6.3.3.* in /usr/local/lib/python3.13/site-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.2; platform_system == \"Linux\"->torch>=2.1.2->liger-kernel) (12.6.3.3)\nRequirement already satisfied: nvidia-nvjitlink==13.0.88.* in /usr/local/lib/python3.13/site-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.2; platform_system == \"Linux\"->torch>=2.1.2->liger-kernel) (13.0.88)\nRequirement already satisfied: nvidia-cuda-nvrtc==13.0.88.* in /usr/local/lib/python3.13/site-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.2; platform_system == \"Linux\"->torch>=2.1.2->liger-kernel) (13.0.88)\nRequirement already satisfied: nvidia-nvtx==13.0.85.* in /usr/local/lib/python3.13/site-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.2; platform_system == \"Linux\"->torch>=2.1.2->liger-kernel) (13.0.85)\nRequirement already satisfied: cuda-pathfinder>=1.4.2 in /usr/local/lib/python3.13/site-packages (from cuda-bindings<14,>=13.0.3->torch>=2.1.2->liger-kernel) (1.5.6)\nRequirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.13/site-packages (from sympy>=1.13.3->torch>=2.1.2->liger-kernel) (1.3.0)\nRequirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.13/site-packages (from jinja2->torch>=2.1.2->liger-kernel) (3.0.3)\nRequirement already satisfied: markdown-it-py>=2.2.0 in /tmp/uv-venv/lib/python3.13/site-packages (from rich->helion) (4.2.0)\nRequirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.13/site-packages (from rich->helion) (2.20.0)\nRequirement already satisfied: mdurl~=0.1 in /tmp/uv-venv/lib/python3.13/site-packages (from markdown-it-py>=2.2.0->rich->helion) (0.1.2)\nDownloading accelerate-1.15.0-py3-none-any.whl (394 kB)\nDownloading helion-1.5.2-py3-none-any.whl (5.9 MB)\n \u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501 5.9/5.9 MB 32.7 MB/s 0:00:00\nDownloading triton-3.6.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (188.3 MB)\n \u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501 188.3/188.3 MB 32.9 MB/s 0:00:05\nDownloading setuptools-81.0.0-py3-none-any.whl (1.1 MB)\n \u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501 1.1/1.1 MB 233.2 MB/s 0:00:00\nDownloading filecheck-1.0.6-py3-none-any.whl (24 kB)\nInstalling collected packages: triton, setuptools, filecheck, helion, accelerate\n Attempting uninstall: triton\n Found existing installation: triton 3.9.0\n Uninstalling triton-3.9.0:\n Successfully uninstalled triton-3.9.0\n Attempting uninstall: setuptools\n Found existing installation: setuptools 84.2.1\n Uninstalling setuptools-84.2.1:\n Successfully uninstalled setuptools-84.2.1\n\nSuccessfully installed accelerate-1.15.0 filecheck-1.0.6 helion-1.5.2 setuptools-81.0.0 triton-3.6.0\n", "mimetype": "text/plain" }, { "type": "stream", "name": "stderr", "text": "WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.\n\n[notice] A new release of pip is available: 25.3 -> 26.2.2\n[notice] To update, run: pip install --upgrade pip\n", "mimetype": "text/plain" } ] }, { "id": "aLJB", "code_hash": "b23d6205e0dff43a7a9b1cab8bd8e97a", "outputs": [], "console": [] }, { "id": "nHfw", "code_hash": "5b6dacd0df25215f730c002feaf2a8d1", "outputs": [], "console": [] }, { "id": "xXTn", "code_hash": null, "outputs": [], "console": [] }, { "id": "AjVT", "code_hash": "18d0318a90bb4be1e137141badd32161", "outputs": [], "console": [] }, { "id": "pHFh", "code_hash": "950061ed21b5034668672f8f1537fc20", "outputs": [ { "type": "data", "data": { "text/markdown": "
arya_latest.pt and demonstrates how to generate text using a given prompt."
}
}
],
"console": []
},
{
"id": "NCOB",
"code_hash": "4866b2239ec08cc06925a93d8a5443d0",
"outputs": [],
"console": []
},
{
"id": "aqbW",
"code_hash": "5900b60296183a7a06eaf25ddc611723",
"outputs": [
{
"type": "data",
"data": {
"text/html": "0" } } ], "console": [ { "type": "stream", "name": "stdout", "text": "[kernel] import: cuda_avail=True cuda_ver=13.0 file=/marimo/kernel.py\n", "mimetype": "text/plain" }, { "type": "stream", "name": "stderr", "text": "/marimo/model.py:391: UserWarning: config module has no attribute 'eos_id' -- falling back to default None. If this is unexpected, check for a casing mismatch or rename in your config.py.\n warnings.warn(\n/marimo/model.py:391: UserWarning: config module has no attribute 'moe_fixed_capacity' -- falling back to default None. If this is unexpected, check for a casing mismatch or rename in your config.py.\n warnings.warn(\n[Inference] Loaded checkpoint (step 4000, 17.23M params) onto cuda\n", "mimetype": "text/plain" }, { "type": "stream", "name": "stdout", "text": "\u090f\u0915 \u0938\u092e\u092f \u0915\u0940 \u092c\u093e\u0924 \u0939\u0948 \u091c\u092c \u0932\u094b\u0917\u094b\u0902 \u0915\u094b \u092f\u0939 \u0926\u0947\u0916\u0928\u0947 \u092e\u0947\u0902 \u092e\u0926\u0926 \u0915\u0930 \u0930\u0939\u093e \u0925\u093e\u0964 \u0935\u093f\u0921\u0902\u092c\u0928\u093e \u0928\u093f\u0930\u093e\u0936\u093e\u091c\u0928\u0915 \u092e\u0939\u0938\u0942\u0938 \u0939\u0941\u0908\u0964 \u0909\u0928\u094d\u0939\u0947\u0902 \u092a\u0924\u094d\u0930 \u0928\u0939\u0940\u0902 \u092c\u091a\u093e\u0928\u093e, \u092c\u0932\u094d\u0915\u093f \u0909\u0928\u0915\u0940 \u0938\u094d\u092e\u0943\u0924\u093f\u092f\u094b\u0902 \u0915\u094b\u0964 \u0909\u0938\u0928\u0947 \u090f\u0915 \u0928\u0908 \u091a\u093f\u0902\u0917\u093e\u0930\u0940 \u092c\u0928\u093e\u0908, \u091c\u0948\u0938\u0947-\u091c\u0948\u0938\u0947, \u0914\u0930 \u0905\u092a\u0928\u093e\u092a\u0928 \u0915\u0930\u0928\u0947\u0964 \u0927\u0940\u0930\u0947-\u0927\u0940\u0930\u0947, \u0939\u0930 \u0916\u0941\u0936\u092c\u0942 \u092a\u094d\u0930\u0924\u093f\u0915\u094d\u0930\u093f\u092f\u093e\u0913\u0902 \u092a\u0930 \u092b\u0948\u0932 \u0917\u0908\u0964 \u0932\u0921\u093c\u0915\u0940, \u0905\u092c \u0909\u0938\u0915\u0947 \u092c\u0917\u0932 \u092e\u0947\u0902, \u0921\u0930, \u0917\u0939\u0930\u093e\u0908 \u0924\u0915 \u091a\u0932\u0924\u0940\u0964 \u0935\u0939 \u0938\u092d\u094d\u092f\u0924\u093e \u0907\u0915\u091f\u094d\u0920\u0941-\u0935\u0947\u0935 \u0917\u0941\u0930\u0941\u0924\u094d\u0935\u093e\u0915\u0930\u094d\u0937\u0923 \u092c\u0928\u0928\u093e\u090f \u092f\u0924\u093f \u0930\u0916\u0924\u0940\u0964 \u0930\u0902\u0917\u0940\u0928 \u092c\u0948\u0928\u0930 \u0939\u0935\u0947\u0932\u0939\u094b \u0917\u090f \u0925\u0947! \u0907\u0938 \u0924\u0930\u0939 \u0938\u092c\u0942\u0924 \u0935\u093e\u092a\u0938 \u0906 \u0917\u092f\u093e! \u092e\u0939\u093f\u0932\u093e\u0964\" \u0936\u093e\u092f\u0926 \u0939\u092e \u0907\u0928 \u0915\u0939\u093e\u0928\u093f\u092f\u093e\u0901 \u092a\u0922\u093c\u0928\u093e\u0924\u0947? \"\u0932\u0921\u093c\u0915\u093e, \u0906\u0936\u094d\u091a\u0930\u094d\u092f\u091a\u0915\u093f\u0924\u0964 \u0905\u0917\u0932\u0940! \u091b\u094b\u091f\u0940, \u0905\u0927\u093f\u0915 \u091f\u0942\u091f\u094d\u091f\u094d\u0921\u0940!\"\n", "mimetype": "text/plain" } ] }, { "id": "TRpd", "code_hash": "e181efcf155eef21719f37511de1f32f", "outputs": [ { "type": "data", "data": { "text/markdown": "" } } ], "console": [] }, { "id": "TXez", "code_hash": "9573a4817336aebce40126886c37e8aa", "outputs": [ { "type": "data", "data": { "text/markdown": "Now, let's define a function to generate text and then test it with an example prompt." } } ], "console": [] }, { "id": "dNNg", "code_hash": "ba640786c6f9c69d151de28a936ea8cd", "outputs": [ { "type": "data", "data": { "text/plain": "" } } ], "console": [ { "type": "stream", "name": "stdout", "text": "\u26a0 ftfy not installed \u2014 mojibake repair skipped. Run `pip install ftfy` for full encoding cleanup.\n\u2713 Authenticated with Hugging Face token (CLI login or HF_TOKEN)\nBulk Meyank/Tiny_Stories_Hindi/default: 6 shard(s) available, target 1500.0 MB, 8 workers -> ./training_data/meyank_hindi_1500mb.txt\n", "mimetype": "text/plain" }, { "type": "stream", "name": "stderr", "text": "Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.\n", "mimetype": "text/plain" }, { "type": "stream", "name": "stdout", "text": " shard 1/6: default/train/0000.parquet (227 MB on disk)\n shard 1 done | 476.1 MB written so far\n shard 2/6: default/train/0001.parquet (227 MB on disk)\n shard 2 done | 952.2 MB written so far\n shard 3/6: default/train/0002.parquet (227 MB on disk)\n shard 3 done | 1428.2 MB written so far\n shard 4/6: default/train/0003.parquet (227 MB on disk)\n shard 4 done | 1500.0 MB written so far\n\u2713 ./training_data/meyank_hindi_1500mb.txt | 1500.0/1500.0 MB | 577,827 lines | 288,916 filtered | 0 dupes dropped\n \u2713 verified: 200,000 lines sampled, 0 artifacts\n\nfinished rc=0 in 0.7 min\n", "mimetype": "text/plain" } ] }, { "id": "yCnT", "code_hash": "b421797a6ede1124767e74a6671c2da9", "outputs": [], "console": [] }, { "id": "wlCL", "code_hash": "08809d03fe76b2df1122d25d14fe74d5", "outputs": [], "console": [] }, { "id": "kqZH", "code_hash": "13122d375dfe64b0638eed09336e688c", "outputs": [], "console": [] }, { "id": "wAgl", "code_hash": "c8fc96583ecaa55247aa6caf3e855d22", "outputs": [], "console": [] }, { "id": "rEll", "code_hash": "d6e07408db81621dabaca73882dcacb1", "outputs": [], "console": [] }, { "id": "dGlV", "code_hash": "9ea3e14a9bb177fa422b4038a8daaca4", "outputs": [ { "type": "data", "data": { "text/html": "
0" } } ], "console": [ { "type": "stream", "name": "stdout", "text": "[token_train] 3 files, 4.05 GB corpus, vocab_size=12288\n[token_train] Trained. mergeable ranks = 12286\n[token_train] tiktoken.Encoding built: n_vocab=12288, <|bos|> id=12286, <|eos|> id=12287\n[token_train] Saved tokenizer/tokenizer.pkl\n", "mimetype": "text/plain" } ] }, { "id": "SdmI", "code_hash": "c72b1f6f545da1a69cbaa110db3ed3da", "outputs": [], "console": [] }, { "id": "lgWD", "code_hash": "b5cb5e47bc909cfe489c2c8e8038ad4f", "outputs": [], "console": [] }, { "id": "yOPj", "code_hash": "d10c338ecd83ffd0ca2127d594ec829b", "outputs": [], "console": [] }, { "id": "fwwy", "code_hash": "e8ef51bde9d8394667597f0800e234e7", "outputs": [ { "type": "data", "data": { "text/plain": "" } } ], "console": [] }, { "id": "LJZf", "code_hash": "2d219b399b6cb81ceb159de5efe0331b", "outputs": [], "console": [ { "type": "stream", "name": "stdout", "text": "[auth] ArushBuilds (type=user)\n[repo] ready: https://huggingface.co/RuVI-AI/pragya-mini-exp\n[preflight] 19 local file(s) match ALLOW:\n __marimo__/session/notebook.py.json\n __pycache__/tokenizer.cpython-313.pyc\n checkpoints/model_step4000.safetensors\n checkpoints/o.safetensors\n checkpoints/x.pt\n lock.txt\n profile_trace.json\n tokenizer.model\n tokenizer.py\n tokenizer.vocab\n tokenizer/tokenizer.pkl\n tokenizer/tokenizer_config.json\n tokenizer_1.model\n tokenizer_1.py\n tokenizer_1.vocab\n training_data/.token_cache/tokens_3894af946a164b07.ranges.json\n training_data/cosmopedia_1000mb.txt\n training_data/meyank_hindi_1500mb.txt\n training_data/tinystories_1500mb.txt\n[upload] starting (whitelist: ['*.safetensors', '*.json', '*.model', '*.txt', '*.md', 'tokenizer*', 'tokenizer_files/**', '*.pkl', '*.pt', 'checkpoints/**'])\n", "mimetype": "text/plain" } ] }, { "id": "urSm", "code_hash": "35717b83aa01ab740b9698a3184aeb99", "outputs": [], "console": [] }, { "id": "jxvo", "code_hash": "8ba9c69f5c18b12e4f905eb4e412f6e8", "outputs": [], "console": [] } ] }