2026-07-18:15:17:55 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
2026-07-18:15:17:58 INFO     [_cli.run:376] Selected Tasks: ['babilong_longctx']
2026-07-18:15:18:00 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-07-18:15:18:00 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
2026-07-18:15:18:03 INFO     [models.huggingface:161] Using device 'cuda:0'
2026-07-18:15:18:03 INFO     [models.huggingface:548] Model type cannot be determined. Using default model type 'causal'
2026-07-18:15:18:03 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
2026-07-18:15:18:05 WARNING  [api.task:856] babilong_qa5: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
2026-07-18:15:18:05 WARNING  [api.task:856] babilong_qa4: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
2026-07-18:15:18:05 WARNING  [api.task:856] babilong_qa3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
2026-07-18:15:18:06 WARNING  [api.task:856] babilong_qa2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
2026-07-18:15:18:06 WARNING  [api.task:856] babilong_qa1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
2026-07-18:15:18:06 INFO     [tasks:700] Selected tasks:
2026-07-18:15:18:06 INFO     [tasks:703] Group: babilong_longctx
2026-07-18:15:18:06 INFO     [tasks:726] ConfigurableGroup(group=babilong_longctx,group_alias=None): {'babilong_qa1': ConfigurableTask(task_name=babilong_qa1,output_type=generate_until,num_fewshot=2,num_samples=1000), 'babilong_qa2': ConfigurableTask(task_name=babilong_qa2,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa3': ConfigurableTask(task_name=babilong_qa3,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa4': ConfigurableTask(task_name=babilong_qa4,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa5': ConfigurableTask(task_name=babilong_qa5,output_type=generate_until,num_fewshot=2,num_samples=999)}
2026-07-18:15:18:06 INFO     [evaluator:314] babilong_qa1: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
2026-07-18:15:18:06 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa1 from 2 to 2
2026-07-18:15:18:06 INFO     [evaluator:314] babilong_qa2: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
2026-07-18:15:18:06 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa2 from 2 to 2
2026-07-18:15:18:06 INFO     [evaluator:314] babilong_qa3: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
2026-07-18:15:18:06 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa3 from 2 to 2
2026-07-18:15:18:06 INFO     [evaluator:314] babilong_qa4: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
2026-07-18:15:18:06 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa4 from 2 to 2
2026-07-18:15:18:06 INFO     [evaluator:314] babilong_qa5: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
2026-07-18:15:18:06 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa5 from 2 to 2
2026-07-18:15:18:06 INFO     [api.task:311] Building contexts for babilong_qa1 on rank 0...


  0%|          | 0/50 [00:00<?, ?it/s]
100%|██████████| 50/50 [00:00<00:00, 375.50it/s]
2026-07-18:15:18:06 INFO     [api.task:311] Building contexts for babilong_qa2 on rank 0...


  0%|          | 0/50 [00:00<?, ?it/s]
100%|██████████| 50/50 [00:00<00:00, 370.78it/s]
2026-07-18:15:18:06 INFO     [api.task:311] Building contexts for babilong_qa3 on rank 0...


  0%|          | 0/50 [00:00<?, ?it/s]
100%|██████████| 50/50 [00:00<00:00, 369.97it/s]
2026-07-18:15:18:07 INFO     [api.task:311] Building contexts for babilong_qa4 on rank 0...


  0%|          | 0/50 [00:00<?, ?it/s]
100%|██████████| 50/50 [00:00<00:00, 374.34it/s]
2026-07-18:15:18:07 INFO     [api.task:311] Building contexts for babilong_qa5 on rank 0...


  0%|          | 0/50 [00:00<?, ?it/s]
100%|██████████| 50/50 [00:00<00:00, 372.53it/s]
2026-07-18:15:18:07 INFO     [evaluator:584] Running generate_until requests


Running generate_until requests:   0%|          | 0/250 [00:00<?, ?it/s]

Running generate_until requests:   0%|          | 1/250 [00:06<25:38,  6.18s/it]

Running generate_until requests:   1%|          | 2/250 [00:07<13:51,  3.35s/it]

Running generate_until requests:   1%|          | 3/250 [00:08<10:05,  2.45s/it]

Running generate_until requests:   2%|▏         | 4/250 [00:10<08:17,  2.02s/it]

Running generate_until requests:   2%|▏         | 5/250 [00:11<07:16,  1.78s/it]

Running generate_until requests:   2%|▏         | 6/250 [00:13<06:39,  1.64s/it]

Running generate_until requests:   3%|▎         | 7/250 [00:14<06:15,  1.55s/it]

Running generate_until requests:   3%|▎         | 8/250 [00:15<05:59,  1.49s/it]

Running generate_until requests:   4%|▎         | 9/250 [00:17<05:47,  1.44s/it]

Running generate_until requests:   4%|▍         | 10/250 [00:18<05:39,  1.42s/it]

Running generate_until requests:   4%|▍         | 11/250 [00:19<05:33,  1.40s/it]

Running generate_until requests:   5%|▍         | 12/250 [00:21<05:28,  1.38s/it]

Running generate_until requests:   5%|▌         | 13/250 [00:22<05:24,  1.37s/it]

Running generate_until requests:   6%|▌         | 14/250 [00:23<05:21,  1.36s/it]

Running generate_until requests:   6%|▌         | 15/250 [00:25<05:19,  1.36s/it]

Running generate_until requests:   6%|▋         | 16/250 [00:26<05:17,  1.36s/it]

Running generate_until requests:   7%|▋         | 17/250 [00:27<05:15,  1.35s/it]

Running generate_until requests:   7%|▋         | 18/250 [00:29<05:12,  1.35s/it]

Running generate_until requests:   8%|▊         | 19/250 [00:30<05:11,  1.35s/it]

Running generate_until requests:   8%|▊         | 20/250 [00:31<05:09,  1.35s/it]

Running generate_until requests:   8%|▊         | 21/250 [00:33<05:08,  1.35s/it]

Running generate_until requests:   9%|▉         | 22/250 [00:34<05:06,  1.34s/it]

Running generate_until requests:   9%|▉         | 23/250 [00:35<05:05,  1.35s/it]

Running generate_until requests:  10%|▉         | 24/250 [00:37<05:04,  1.35s/it]

Running generate_until requests:  10%|█         | 25/250 [00:38<05:02,  1.35s/it]

Running generate_until requests:  10%|█         | 26/250 [00:39<05:01,  1.35s/it]

Running generate_until requests:  11%|█         | 27/250 [00:41<05:00,  1.35s/it]

Running generate_until requests:  11%|█         | 28/250 [00:42<04:58,  1.35s/it]

Running generate_until requests:  12%|█▏        | 29/250 [00:43<04:57,  1.34s/it]

Running generate_until requests:  12%|█▏        | 30/250 [00:45<04:55,  1.34s/it]

Running generate_until requests:  12%|█▏        | 31/250 [00:46<04:54,  1.34s/it]

Running generate_until requests:  13%|█▎        | 32/250 [00:48<04:53,  1.34s/it]

Running generate_until requests:  13%|█▎        | 33/250 [00:49<04:51,  1.35s/it]

Running generate_until requests:  14%|█▎        | 34/250 [00:50<04:50,  1.34s/it]

Running generate_until requests:  14%|█▍        | 35/250 [00:52<04:49,  1.34s/it]

Running generate_until requests:  14%|█▍        | 36/250 [00:53<04:47,  1.34s/it]

Running generate_until requests:  15%|█▍        | 37/250 [00:54<04:46,  1.34s/it]

Running generate_until requests:  15%|█▌        | 38/250 [00:56<04:45,  1.34s/it]

Running generate_until requests:  16%|█▌        | 39/250 [00:57<04:44,  1.35s/it]

Running generate_until requests:  16%|█▌        | 40/250 [00:58<04:42,  1.35s/it]

Running generate_until requests:  16%|█▋        | 41/250 [01:00<04:41,  1.34s/it]

Running generate_until requests:  17%|█▋        | 42/250 [01:01<04:39,  1.34s/it]

Running generate_until requests:  17%|█▋        | 43/250 [01:02<04:38,  1.34s/it]

Running generate_until requests:  18%|█▊        | 44/250 [01:04<04:36,  1.34s/it]

Running generate_until requests:  18%|█▊        | 45/250 [01:05<04:35,  1.34s/it]

Running generate_until requests:  18%|█▊        | 46/250 [01:06<04:35,  1.35s/it]

Running generate_until requests:  19%|█▉        | 47/250 [01:08<04:34,  1.35s/it]

Running generate_until requests:  19%|█▉        | 48/250 [01:09<04:33,  1.35s/it]

Running generate_until requests:  20%|█▉        | 49/250 [01:10<04:31,  1.35s/it]

Running generate_until requests:  20%|██        | 50/250 [01:12<04:29,  1.35s/it]

Running generate_until requests:  20%|██        | 51/250 [01:13<04:28,  1.35s/it]

Running generate_until requests:  21%|██        | 52/250 [01:14<04:26,  1.35s/it]

Running generate_until requests:  21%|██        | 53/250 [01:16<04:25,  1.35s/it]

Running generate_until requests:  22%|██▏       | 54/250 [01:17<04:23,  1.35s/it]

Running generate_until requests:  22%|██▏       | 55/250 [01:18<04:22,  1.35s/it]

Running generate_until requests:  22%|██▏       | 56/250 [01:20<04:20,  1.34s/it]

Running generate_until requests:  23%|██▎       | 57/250 [01:21<04:19,  1.34s/it]

Running generate_until requests:  23%|██▎       | 58/250 [01:23<04:17,  1.34s/it]

Running generate_until requests:  24%|██▎       | 59/250 [01:24<04:15,  1.34s/it]

Running generate_until requests:  24%|██▍       | 60/250 [01:25<04:13,  1.34s/it]

Running generate_until requests:  24%|██▍       | 61/250 [01:27<04:12,  1.34s/it]

Running generate_until requests:  25%|██▍       | 62/250 [01:28<04:11,  1.34s/it]

Running generate_until requests:  25%|██▌       | 63/250 [01:29<04:10,  1.34s/it]

Running generate_until requests:  26%|██▌       | 64/250 [01:31<04:09,  1.34s/it]

Running generate_until requests:  26%|██▌       | 65/250 [01:32<04:07,  1.34s/it]

Running generate_until requests:  26%|██▋       | 66/250 [01:33<04:06,  1.34s/it]

Running generate_until requests:  27%|██▋       | 67/250 [01:35<04:04,  1.34s/it]

Running generate_until requests:  27%|██▋       | 68/250 [01:36<04:03,  1.34s/it]

Running generate_until requests:  28%|██▊       | 69/250 [01:37<04:02,  1.34s/it]

Running generate_until requests:  28%|██▊       | 70/250 [01:39<04:00,  1.34s/it]

Running generate_until requests:  28%|██▊       | 71/250 [01:40<03:59,  1.34s/it]

Running generate_until requests:  29%|██▉       | 72/250 [01:41<03:58,  1.34s/it]

Running generate_until requests:  29%|██▉       | 73/250 [01:43<03:57,  1.34s/it]

Running generate_until requests:  30%|██▉       | 74/250 [01:44<03:55,  1.34s/it]

Running generate_until requests:  30%|███       | 75/250 [01:45<03:54,  1.34s/it]

Running generate_until requests:  30%|███       | 76/250 [01:47<03:53,  1.34s/it]

Running generate_until requests:  31%|███       | 77/250 [01:48<03:52,  1.34s/it]

Running generate_until requests:  31%|███       | 78/250 [01:49<03:50,  1.34s/it]

Running generate_until requests:  32%|███▏      | 79/250 [01:51<03:49,  1.34s/it]

Running generate_until requests:  32%|███▏      | 80/250 [01:52<03:47,  1.34s/it]

Running generate_until requests:  32%|███▏      | 81/250 [01:53<03:46,  1.34s/it]

Running generate_until requests:  33%|███▎      | 82/250 [01:55<03:44,  1.34s/it]

Running generate_until requests:  33%|███▎      | 83/250 [01:56<03:43,  1.34s/it]

Running generate_until requests:  34%|███▎      | 84/250 [01:57<03:42,  1.34s/it]

Running generate_until requests:  34%|███▍      | 85/250 [01:59<03:40,  1.34s/it]

Running generate_until requests:  34%|███▍      | 86/250 [02:00<03:39,  1.34s/it]

Running generate_until requests:  35%|███▍      | 87/250 [02:01<03:38,  1.34s/it]

Running generate_until requests:  35%|███▌      | 88/250 [02:03<03:36,  1.34s/it]

Running generate_until requests:  36%|███▌      | 89/250 [02:04<03:35,  1.34s/it]

Running generate_until requests:  36%|███▌      | 90/250 [02:05<03:34,  1.34s/it]

Running generate_until requests:  36%|███▋      | 91/250 [02:07<03:33,  1.34s/it]

Running generate_until requests:  37%|███▋      | 92/250 [02:08<03:31,  1.34s/it]

Running generate_until requests:  37%|███▋      | 93/250 [02:09<03:30,  1.34s/it]

Running generate_until requests:  38%|███▊      | 94/250 [02:11<03:28,  1.34s/it]

Running generate_until requests:  38%|███▊      | 95/250 [02:12<03:27,  1.34s/it]

Running generate_until requests:  38%|███▊      | 96/250 [02:13<03:25,  1.34s/it]

Running generate_until requests:  39%|███▉      | 97/250 [02:15<03:24,  1.33s/it]

Running generate_until requests:  39%|███▉      | 98/250 [02:16<03:22,  1.33s/it]

Running generate_until requests:  40%|███▉      | 99/250 [02:17<03:21,  1.33s/it]

Running generate_until requests:  40%|████      | 100/250 [02:19<03:19,  1.33s/it]

Running generate_until requests:  40%|████      | 101/250 [02:20<03:18,  1.33s/it]

Running generate_until requests:  41%|████      | 102/250 [02:21<03:16,  1.33s/it]

Running generate_until requests:  41%|████      | 103/250 [02:23<03:15,  1.33s/it]

Running generate_until requests:  42%|████▏     | 104/250 [02:24<03:14,  1.33s/it]

Running generate_until requests:  42%|████▏     | 105/250 [02:25<03:12,  1.33s/it]

Running generate_until requests:  42%|████▏     | 106/250 [02:27<03:11,  1.33s/it]

Running generate_until requests:  43%|████▎     | 107/250 [02:28<03:10,  1.33s/it]

Running generate_until requests:  43%|████▎     | 108/250 [02:29<03:08,  1.33s/it]

Running generate_until requests:  44%|████▎     | 109/250 [02:31<03:07,  1.33s/it]

Running generate_until requests:  44%|████▍     | 110/250 [02:32<03:06,  1.33s/it]

Running generate_until requests:  44%|████▍     | 111/250 [02:33<03:04,  1.33s/it]

Running generate_until requests:  45%|████▍     | 112/250 [02:35<03:03,  1.33s/it]

Running generate_until requests:  45%|████▌     | 113/250 [02:36<03:02,  1.33s/it]

Running generate_until requests:  46%|████▌     | 114/250 [02:37<03:00,  1.33s/it]

Running generate_until requests:  46%|████▌     | 115/250 [02:39<02:59,  1.33s/it]

Running generate_until requests:  46%|████▋     | 116/250 [02:40<02:58,  1.33s/it]

Running generate_until requests:  47%|████▋     | 117/250 [02:41<02:56,  1.33s/it]

Running generate_until requests:  47%|████▋     | 118/250 [02:43<02:55,  1.33s/it]

Running generate_until requests:  48%|████▊     | 119/250 [02:44<02:54,  1.33s/it]

Running generate_until requests:  48%|████▊     | 120/250 [02:45<02:53,  1.33s/it]

Running generate_until requests:  48%|████▊     | 121/250 [02:47<02:51,  1.33s/it]

Running generate_until requests:  49%|████▉     | 122/250 [02:48<02:50,  1.33s/it]

Running generate_until requests:  49%|████▉     | 123/250 [02:49<02:48,  1.33s/it]

Running generate_until requests:  50%|████▉     | 124/250 [02:51<02:47,  1.33s/it]

Running generate_until requests:  50%|█████     | 125/250 [02:52<02:46,  1.33s/it]

Running generate_until requests:  50%|█████     | 126/250 [02:53<02:44,  1.33s/it]

Running generate_until requests:  51%|█████     | 127/250 [02:55<02:43,  1.33s/it]

Running generate_until requests:  51%|█████     | 128/250 [02:56<02:42,  1.33s/it]

Running generate_until requests:  52%|█████▏    | 129/250 [02:57<02:41,  1.33s/it]

Running generate_until requests:  52%|█████▏    | 130/250 [02:59<02:39,  1.33s/it]

Running generate_until requests:  52%|█████▏    | 131/250 [03:00<02:38,  1.33s/it]

Running generate_until requests:  53%|█████▎    | 132/250 [03:01<02:36,  1.33s/it]

Running generate_until requests:  53%|█████▎    | 133/250 [03:03<02:35,  1.33s/it]

Running generate_until requests:  54%|█████▎    | 134/250 [03:04<02:34,  1.33s/it]

Running generate_until requests:  54%|█████▍    | 135/250 [03:05<02:32,  1.33s/it]

Running generate_until requests:  54%|█████▍    | 136/250 [03:07<02:30,  1.32s/it]

Running generate_until requests:  55%|█████▍    | 137/250 [03:08<02:29,  1.32s/it]

Running generate_until requests:  55%|█████▌    | 138/250 [03:09<02:28,  1.32s/it]

Running generate_until requests:  56%|█████▌    | 139/250 [03:11<02:26,  1.32s/it]

Running generate_until requests:  56%|█████▌    | 140/250 [03:12<02:25,  1.33s/it]

Running generate_until requests:  56%|█████▋    | 141/250 [03:13<02:24,  1.32s/it]

Running generate_until requests:  57%|█████▋    | 142/250 [03:15<02:22,  1.32s/it]

Running generate_until requests:  57%|█████▋    | 143/250 [03:16<02:21,  1.33s/it]

Running generate_until requests:  58%|█████▊    | 144/250 [03:17<02:20,  1.33s/it]

Running generate_until requests:  58%|█████▊    | 145/250 [03:19<02:19,  1.32s/it]

Running generate_until requests:  58%|█████▊    | 146/250 [03:20<02:17,  1.33s/it]

Running generate_until requests:  59%|█████▉    | 147/250 [03:21<02:16,  1.33s/it]

Running generate_until requests:  59%|█████▉    | 148/250 [03:22<02:15,  1.33s/it]

Running generate_until requests:  60%|█████▉    | 149/250 [03:24<02:13,  1.33s/it]

Running generate_until requests:  60%|██████    | 150/250 [03:25<02:12,  1.33s/it]

Running generate_until requests:  60%|██████    | 151/250 [03:26<02:11,  1.33s/it]

Running generate_until requests:  61%|██████    | 152/250 [03:28<02:09,  1.33s/it]

Running generate_until requests:  61%|██████    | 153/250 [03:29<02:08,  1.33s/it]

Running generate_until requests:  62%|██████▏   | 154/250 [03:30<02:07,  1.32s/it]

Running generate_until requests:  62%|██████▏   | 155/250 [03:32<02:05,  1.32s/it]

Running generate_until requests:  62%|██████▏   | 156/250 [03:33<02:04,  1.32s/it]

Running generate_until requests:  63%|██████▎   | 157/250 [03:34<02:02,  1.32s/it]

Running generate_until requests:  63%|██████▎   | 158/250 [03:36<02:01,  1.32s/it]

Running generate_until requests:  64%|██████▎   | 159/250 [03:37<02:00,  1.32s/it]

Running generate_until requests:  64%|██████▍   | 160/250 [03:38<01:59,  1.32s/it]

Running generate_until requests:  64%|██████▍   | 161/250 [03:40<01:57,  1.32s/it]

Running generate_until requests:  65%|██████▍   | 162/250 [03:41<01:56,  1.32s/it]

Running generate_until requests:  65%|██████▌   | 163/250 [03:42<01:55,  1.32s/it]

Running generate_until requests:  66%|██████▌   | 164/250 [03:44<01:53,  1.32s/it]

Running generate_until requests:  66%|██████▌   | 165/250 [03:45<01:52,  1.32s/it]

Running generate_until requests:  66%|██████▋   | 166/250 [03:46<01:51,  1.32s/it]

Running generate_until requests:  67%|██████▋   | 167/250 [03:48<01:49,  1.32s/it]

Running generate_until requests:  67%|██████▋   | 168/250 [03:49<01:48,  1.32s/it]

Running generate_until requests:  68%|██████▊   | 169/250 [03:50<01:47,  1.32s/it]

Running generate_until requests:  68%|██████▊   | 170/250 [03:52<01:45,  1.32s/it]

Running generate_until requests:  68%|██████▊   | 171/250 [03:53<01:44,  1.32s/it]

Running generate_until requests:  69%|██████▉   | 172/250 [03:54<01:43,  1.32s/it]

Running generate_until requests:  69%|██████▉   | 173/250 [03:56<01:41,  1.32s/it]

Running generate_until requests:  70%|██████▉   | 174/250 [03:57<01:40,  1.32s/it]

Running generate_until requests:  70%|███████   | 175/250 [03:58<01:39,  1.32s/it]

Running generate_until requests:  70%|███████   | 176/250 [04:00<01:37,  1.32s/it]

Running generate_until requests:  71%|███████   | 177/250 [04:01<01:36,  1.32s/it]

Running generate_until requests:  71%|███████   | 178/250 [04:02<01:35,  1.32s/it]

Running generate_until requests:  72%|███████▏  | 179/250 [04:04<01:33,  1.32s/it]

Running generate_until requests:  72%|███████▏  | 180/250 [04:05<01:32,  1.32s/it]

Running generate_until requests:  72%|███████▏  | 181/250 [04:06<01:31,  1.32s/it]

Running generate_until requests:  73%|███████▎  | 182/250 [04:07<01:29,  1.32s/it]

Running generate_until requests:  73%|███████▎  | 183/250 [04:09<01:28,  1.32s/it]

Running generate_until requests:  74%|███████▎  | 184/250 [04:10<01:27,  1.32s/it]

Running generate_until requests:  74%|███████▍  | 185/250 [04:11<01:25,  1.32s/it]

Running generate_until requests:  74%|███████▍  | 186/250 [04:13<01:24,  1.32s/it]

Running generate_until requests:  75%|███████▍  | 187/250 [04:14<01:23,  1.32s/it]

Running generate_until requests:  75%|███████▌  | 188/250 [04:15<01:21,  1.32s/it]

Running generate_until requests:  76%|███████▌  | 189/250 [04:17<01:20,  1.32s/it]

Running generate_until requests:  76%|███████▌  | 190/250 [04:18<01:19,  1.32s/it]

Running generate_until requests:  76%|███████▋  | 191/250 [04:19<01:17,  1.32s/it]

Running generate_until requests:  77%|███████▋  | 192/250 [04:21<01:16,  1.32s/it]

Running generate_until requests:  77%|███████▋  | 193/250 [04:22<01:15,  1.32s/it]

Running generate_until requests:  78%|███████▊  | 194/250 [04:23<01:13,  1.32s/it]

Running generate_until requests:  78%|███████▊  | 195/250 [04:25<01:12,  1.32s/it]

Running generate_until requests:  78%|███████▊  | 196/250 [04:26<01:11,  1.32s/it]

Running generate_until requests:  79%|███████▉  | 197/250 [04:27<01:09,  1.32s/it]

Running generate_until requests:  79%|███████▉  | 198/250 [04:29<01:08,  1.32s/it]

Running generate_until requests:  80%|███████▉  | 199/250 [04:30<01:07,  1.32s/it]

Running generate_until requests:  80%|████████  | 200/250 [04:31<01:05,  1.32s/it]

Running generate_until requests:  80%|████████  | 201/250 [04:32<01:04,  1.31s/it]

Running generate_until requests:  81%|████████  | 202/250 [04:34<01:03,  1.31s/it]

Running generate_until requests:  81%|████████  | 203/250 [04:35<01:01,  1.31s/it]

Running generate_until requests:  82%|████████▏ | 204/250 [04:36<01:00,  1.31s/it]

Running generate_until requests:  82%|████████▏ | 205/250 [04:38<00:56,  1.25s/it]

Running generate_until requests:  82%|████████▏ | 206/250 [04:39<00:53,  1.21s/it]

Running generate_until requests:  83%|████████▎ | 207/250 [04:40<00:50,  1.18s/it]

Running generate_until requests:  83%|████████▎ | 208/250 [04:41<00:48,  1.16s/it]

Running generate_until requests:  84%|████████▎ | 209/250 [04:42<00:46,  1.14s/it]

Running generate_until requests:  84%|████████▍ | 210/250 [04:43<00:45,  1.13s/it]

Running generate_until requests:  84%|████████▍ | 211/250 [04:44<00:43,  1.12s/it]

Running generate_until requests:  85%|████████▍ | 212/250 [04:45<00:42,  1.12s/it]

Running generate_until requests:  85%|████████▌ | 213/250 [04:46<00:41,  1.12s/it]

Running generate_until requests:  86%|████████▌ | 214/250 [04:48<00:40,  1.11s/it]

Running generate_until requests:  86%|████████▌ | 215/250 [04:49<00:38,  1.11s/it]

Running generate_until requests:  86%|████████▋ | 216/250 [04:50<00:37,  1.11s/it]

Running generate_until requests:  87%|████████▋ | 217/250 [04:51<00:36,  1.11s/it]

Running generate_until requests:  87%|████████▋ | 218/250 [04:52<00:35,  1.11s/it]

Running generate_until requests:  88%|████████▊ | 219/250 [04:53<00:34,  1.10s/it]

Running generate_until requests:  88%|████████▊ | 220/250 [04:54<00:33,  1.10s/it]

Running generate_until requests:  88%|████████▊ | 221/250 [04:55<00:31,  1.10s/it]

Running generate_until requests:  89%|████████▉ | 222/250 [04:56<00:30,  1.10s/it]

Running generate_until requests:  89%|████████▉ | 223/250 [04:57<00:29,  1.10s/it]

Running generate_until requests:  90%|████████▉ | 224/250 [04:59<00:28,  1.10s/it]

Running generate_until requests:  90%|█████████ | 225/250 [05:00<00:27,  1.10s/it]

Running generate_until requests:  90%|█████████ | 226/250 [05:01<00:26,  1.10s/it]

Running generate_until requests:  91%|█████████ | 227/250 [05:02<00:25,  1.10s/it]

Running generate_until requests:  91%|█████████ | 228/250 [05:03<00:24,  1.09s/it]

Running generate_until requests:  92%|█████████▏| 229/250 [05:04<00:22,  1.09s/it]

Running generate_until requests:  92%|█████████▏| 230/250 [05:05<00:21,  1.09s/it]

Running generate_until requests:  92%|█████████▏| 231/250 [05:06<00:20,  1.09s/it]

Running generate_until requests:  93%|█████████▎| 232/250 [05:07<00:19,  1.08s/it]

Running generate_until requests:  93%|█████████▎| 233/250 [05:08<00:18,  1.08s/it]

Running generate_until requests:  94%|█████████▎| 234/250 [05:09<00:17,  1.08s/it]

Running generate_until requests:  94%|█████████▍| 235/250 [05:10<00:16,  1.08s/it]

Running generate_until requests:  94%|█████████▍| 236/250 [05:12<00:15,  1.07s/it]

Running generate_until requests:  95%|█████████▍| 237/250 [05:13<00:13,  1.07s/it]

Running generate_until requests:  95%|█████████▌| 238/250 [05:14<00:12,  1.06s/it]

Running generate_until requests:  96%|█████████▌| 239/250 [05:15<00:11,  1.06s/it]

Running generate_until requests:  96%|█████████▌| 240/250 [05:16<00:10,  1.06s/it]

Running generate_until requests:  96%|█████████▋| 241/250 [05:17<00:09,  1.05s/it]

Running generate_until requests:  97%|█████████▋| 242/250 [05:18<00:08,  1.05s/it]

Running generate_until requests:  97%|█████████▋| 243/250 [05:19<00:07,  1.05s/it]

Running generate_until requests:  98%|█████████▊| 244/250 [05:20<00:06,  1.05s/it]

Running generate_until requests:  98%|█████████▊| 245/250 [05:21<00:05,  1.05s/it]

Running generate_until requests:  98%|█████████▊| 246/250 [05:22<00:04,  1.04s/it]

Running generate_until requests:  99%|█████████▉| 248/250 [05:24<00:02,  1.01s/it]

Running generate_until requests: 100%|██████████| 250/250 [05:26<00:00,  1.01it/s]
Running generate_until requests: 100%|██████████| 250/250 [05:26<00:00,  1.31s/it]
2026-07-18:15:23:38 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
2026-07-18:15:23:38 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to /workspace/outputs/eval/token_t045/babilong8k_qa1_qa5_n50/__workspace__outputs__l2a_style__stage2__checkpoint-25/*.jsonl
hf ({'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}), gen_kwargs: ({}), limit: 50.0, num_fewshot: 2, batch_size: 1
|     Tasks      |Version|Filter|n-shot|Metric|   |Value|   |Stderr|
|----------------|------:|------|-----:|------|---|----:|---|-----:|
|babilong_longctx|      0|none  |      |acc   |↑  |0.588|±  |0.0298|
| - babilong_qa1 |      0|none  |     2|acc   |↑  |0.640|±  |0.0686|
| - babilong_qa2 |      0|none  |     2|acc   |↑  |0.480|±  |0.0714|
| - babilong_qa3 |      0|none  |     2|acc   |↑  |0.340|±  |0.0677|
| - babilong_qa4 |      0|none  |     2|acc   |↑  |0.720|±  |0.0641|
| - babilong_qa5 |      0|none  |     2|acc   |↑  |0.760|±  |0.0610|

|     Groups     |Version|Filter|n-shot|Metric|   |Value|   |Stderr|
|----------------|------:|------|------|------|---|----:|---|-----:|
|babilong_longctx|      0|none  |      |acc   |↑  |0.588|±  |0.0298|