2026-07-18:14:17:11 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
2026-07-18:14:17:15 INFO     [_cli.run:376] Selected Tasks: ['niah_single_1', 'niah_single_3', 'niah_multikey_2', 'niah_multiquery', 'ruler_vt', 'ruler_fwe', 'ruler_qa_hotpot']
2026-07-18:14:17:16 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-07-18:14:17:16 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
2026-07-18:14:17:20 INFO     [models.huggingface:161] Using device 'cuda:0'
2026-07-18:14:17:20 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
2026-07-18:14:17:29 WARNING  [api.task:856] niah_single_1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
2026-07-18:14:17:29 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.


Generating synthetic samples: repeat | 8192:   0%|          | 0/500 [00:00<?, ?it/s]

Generating synthetic samples: repeat | 8192:  11%|█▏        | 57/500 [00:01<00:07, 56.50it/s]

Generating synthetic samples: repeat | 8192:  23%|██▎       | 114/500 [00:02<00:06, 56.05it/s]

Generating synthetic samples: repeat | 8192:  34%|███▍      | 171/500 [00:03<00:05, 55.36it/s]

Generating synthetic samples: repeat | 8192:  45%|████▌     | 227/500 [00:04<00:04, 55.19it/s]

Generating synthetic samples: repeat | 8192:  57%|█████▋    | 283/500 [00:05<00:03, 55.10it/s]

Generating synthetic samples: repeat | 8192:  68%|██████▊   | 339/500 [00:06<00:02, 55.08it/s]

Generating synthetic samples: repeat | 8192:  79%|███████▉  | 396/500 [00:07<00:01, 55.44it/s]

Generating synthetic samples: repeat | 8192:  90%|█████████ | 452/500 [00:08<00:00, 55.35it/s]
Generating synthetic samples: repeat | 8192: 100%|██████████| 500/500 [00:09<00:00, 55.34it/s]
2026-07-18:14:17:39 WARNING  [api.task:856] niah_single_3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.


essays.parquet:   0%|          | 0.00/1.19M [00:00<?, ?B/s]
essays.parquet: 100%|██████████| 1.19M/1.19M [00:00<00:00, 4.34MB/s]


Generating train split:   0%|          | 0/218 [00:00<?, ? examples/s]
Generating train split: 100%|██████████| 218/218 [00:00<00:00, 16342.71 examples/s]


Generating synthetic samples: essay | 8192:   0%|          | 0/500 [00:00<?, ?it/s]

Generating synthetic samples: essay | 8192:  11%|█         | 54/500 [00:01<00:08, 53.43it/s]

Generating synthetic samples: essay | 8192:  22%|██▏       | 108/500 [00:02<00:07, 52.72it/s]

Generating synthetic samples: essay | 8192:  32%|███▏      | 162/500 [00:03<00:06, 53.10it/s]

Generating synthetic samples: essay | 8192:  43%|████▎     | 216/500 [00:04<00:05, 53.06it/s]

Generating synthetic samples: essay | 8192:  54%|█████▍    | 270/500 [00:05<00:04, 53.09it/s]

Generating synthetic samples: essay | 8192:  65%|██████▌   | 325/500 [00:06<00:03, 53.45it/s]

Generating synthetic samples: essay | 8192:  76%|███████▌  | 380/500 [00:07<00:02, 53.88it/s]

Generating synthetic samples: essay | 8192:  87%|████████▋ | 435/500 [00:08<00:01, 54.08it/s]

Generating synthetic samples: essay | 8192:  98%|█████████▊| 490/500 [00:09<00:00, 53.95it/s]
Generating synthetic samples: essay | 8192: 100%|██████████| 500/500 [00:09<00:00, 53.60it/s]
2026-07-18:14:17:50 WARNING  [api.task:856] niah_multikey_2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.


Generating synthetic samples: needle | 8192:   0%|          | 0/500 [00:00<?, ?it/s]

Generating synthetic samples: needle | 8192:  13%|█▎        | 64/500 [00:01<00:06, 63.24it/s]

Generating synthetic samples: needle | 8192:  26%|██▌       | 128/500 [00:02<00:05, 63.21it/s]

Generating synthetic samples: needle | 8192:  38%|███▊      | 192/500 [00:03<00:04, 62.90it/s]

Generating synthetic samples: needle | 8192:  52%|█████▏    | 258/500 [00:04<00:03, 63.68it/s]

Generating synthetic samples: needle | 8192:  65%|██████▍   | 323/500 [00:05<00:02, 63.97it/s]

Generating synthetic samples: needle | 8192:  78%|███████▊  | 390/500 [00:06<00:01, 64.80it/s]

Generating synthetic samples: needle | 8192:  92%|█████████▏| 458/500 [00:07<00:00, 65.62it/s]
Generating synthetic samples: needle | 8192: 100%|██████████| 500/500 [00:07<00:00, 64.74it/s]
2026-07-18:14:17:58 WARNING  [api.task:856] niah_multiquery: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.


Generating synthetic samples: essay | 8192:   0%|          | 0/500 [00:00<?, ?it/s]

Generating synthetic samples: essay | 8192:  11%|█         | 55/500 [00:01<00:08, 54.29it/s]

Generating synthetic samples: essay | 8192:  22%|██▏       | 110/500 [00:02<00:07, 54.00it/s]

Generating synthetic samples: essay | 8192:  33%|███▎      | 165/500 [00:03<00:06, 52.38it/s]

Generating synthetic samples: essay | 8192:  44%|████▎     | 218/500 [00:04<00:05, 51.12it/s]

Generating synthetic samples: essay | 8192:  54%|█████▍    | 270/500 [00:05<00:04, 51.20it/s]

Generating synthetic samples: essay | 8192:  64%|██████▍   | 322/500 [00:06<00:03, 51.14it/s]

Generating synthetic samples: essay | 8192:  75%|███████▌  | 376/500 [00:07<00:02, 51.95it/s]

Generating synthetic samples: essay | 8192:  86%|████████▌ | 429/500 [00:08<00:01, 51.90it/s]

Generating synthetic samples: essay | 8192:  96%|█████████▌| 481/500 [00:09<00:00, 51.30it/s]
Generating synthetic samples: essay | 8192: 100%|██████████| 500/500 [00:09<00:00, 51.70it/s]
2026-07-18:14:18:08 WARNING  [api.task:856] ruler_vt: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
2026-07-18:14:18:08 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
Max length 500 | Current length 307 | Noises: 5
Max length 500 | Current length 410 | Noises: 10
Max length 500 | Current length 531 | Noises: 15
Num noises: 10


  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:00<00:00, 939.79it/s]
Max length 8192 | Current length 791 | Noises: 10
Max length 8192 | Current length 1025 | Noises: 20
Max length 8192 | Current length 1273 | Noises: 30
Max length 8192 | Current length 1510 | Noises: 40
Max length 8192 | Current length 1749 | Noises: 50
Max length 8192 | Current length 1996 | Noises: 60
Max length 8192 | Current length 2231 | Noises: 70
Max length 8192 | Current length 2473 | Noises: 80
Max length 8192 | Current length 2705 | Noises: 90
Max length 8192 | Current length 2953 | Noises: 100
Max length 8192 | Current length 3186 | Noises: 110
Max length 8192 | Current length 3431 | Noises: 120
Max length 8192 | Current length 3671 | Noises: 130
Max length 8192 | Current length 3916 | Noises: 140
Max length 8192 | Current length 4149 | Noises: 150
Max length 8192 | Current length 4386 | Noises: 160
Max length 8192 | Current length 4630 | Noises: 170
Max length 8192 | Current length 4871 | Noises: 180
Max length 8192 | Current length 5110 | Noises: 190
Max length 8192 | Current length 5349 | Noises: 200
Max length 8192 | Current length 5590 | Noises: 210
Max length 8192 | Current length 5832 | Noises: 220
Max length 8192 | Current length 6070 | Noises: 230
Max length 8192 | Current length 6311 | Noises: 240
Max length 8192 | Current length 6544 | Noises: 250
Max length 8192 | Current length 6791 | Noises: 260
Max length 8192 | Current length 7031 | Noises: 270
Max length 8192 | Current length 7271 | Noises: 280
Max length 8192 | Current length 7513 | Noises: 290
Max length 8192 | Current length 7751 | Noises: 300
Max length 8192 | Current length 7986 | Noises: 310
Max length 8192 | Current length 8230 | Noises: 320
Num noises: 310


  0%|          | 0/500 [00:00<?, ?it/s]

 12%|█▏        | 59/500 [00:01<00:07, 58.56it/s]

 24%|██▎       | 118/500 [00:02<00:06, 57.94it/s]

 35%|███▌      | 176/500 [00:03<00:05, 57.91it/s]

 47%|████▋     | 234/500 [00:04<00:04, 56.14it/s]

 58%|█████▊    | 291/500 [00:05<00:03, 55.52it/s]

 69%|██████▉   | 347/500 [00:06<00:02, 54.63it/s]

 80%|████████  | 402/500 [00:07<00:01, 54.48it/s]

 91%|█████████▏| 457/500 [00:08<00:00, 54.48it/s]
100%|██████████| 500/500 [00:09<00:00, 55.38it/s]
2026-07-18:14:18:18 WARNING  [api.task:856] ruler_fwe: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.


Generating FWE Samples | 8192:   0%|          | 0/500 [00:00<?, ?it/s]

Generating FWE Samples | 8192:   8%|▊         | 40/500 [00:01<00:11, 39.74it/s]

Generating FWE Samples | 8192:  16%|█▌        | 81/500 [00:02<00:10, 40.12it/s]

Generating FWE Samples | 8192:  24%|██▍       | 122/500 [00:03<00:09, 39.79it/s]

Generating FWE Samples | 8192:  32%|███▏      | 162/500 [00:04<00:08, 39.25it/s]

Generating FWE Samples | 8192:  40%|████      | 202/500 [00:05<00:07, 38.10it/s]

Generating FWE Samples | 8192:  48%|████▊     | 241/500 [00:06<00:07, 36.79it/s]

Generating FWE Samples | 8192:  56%|█████▌    | 278/500 [00:07<00:06, 36.23it/s]

Generating FWE Samples | 8192:  63%|██████▎   | 315/500 [00:08<00:05, 36.28it/s]

Generating FWE Samples | 8192:  70%|███████   | 352/500 [00:09<00:04, 36.25it/s]

Generating FWE Samples | 8192:  78%|███████▊  | 389/500 [00:10<00:03, 35.93it/s]

Generating FWE Samples | 8192:  85%|████████▌ | 426/500 [00:11<00:02, 36.05it/s]

Generating FWE Samples | 8192:  93%|█████████▎| 463/500 [00:12<00:01, 35.81it/s]

Generating FWE Samples | 8192: 100%|█████████▉| 499/500 [00:13<00:00, 35.80it/s]
Generating FWE Samples | 8192: 100%|██████████| 500/500 [00:13<00:00, 36.80it/s]
2026-07-18:14:18:32 WARNING  [api.task:856] ruler_qa_hotpot: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.


Generating QA Samples | 8192:   0%|          | 0/500 [00:00<?, ?it/s]

Generating QA Samples | 8192:   5%|▍         | 23/500 [00:01<00:21, 22.17it/s]

Generating QA Samples | 8192:  10%|▉         | 48/500 [00:02<00:19, 23.58it/s]

Generating QA Samples | 8192:  14%|█▍        | 72/500 [00:03<00:18, 23.42it/s]

Generating QA Samples | 8192:  19%|█▉        | 96/500 [00:04<00:17, 23.56it/s]

Generating QA Samples | 8192:  24%|██▍       | 121/500 [00:05<00:15, 23.92it/s]

Generating QA Samples | 8192:  29%|██▉       | 147/500 [00:06<00:14, 24.37it/s]

Generating QA Samples | 8192:  34%|███▍      | 172/500 [00:07<00:13, 24.55it/s]

Generating QA Samples | 8192:  39%|███▉      | 197/500 [00:08<00:12, 24.24it/s]

Generating QA Samples | 8192:  44%|████▍     | 222/500 [00:09<00:11, 24.10it/s]

Generating QA Samples | 8192:  49%|████▉     | 247/500 [00:10<00:10, 23.96it/s]

Generating QA Samples | 8192:  54%|█████▍    | 271/500 [00:11<00:09, 23.39it/s]

Generating QA Samples | 8192:  59%|█████▉    | 297/500 [00:12<00:08, 23.94it/s]

Generating QA Samples | 8192:  65%|██████▍   | 323/500 [00:13<00:07, 24.31it/s]

Generating QA Samples | 8192:  70%|██████▉   | 348/500 [00:14<00:06, 23.74it/s]

Generating QA Samples | 8192:  74%|███████▍  | 372/500 [00:15<00:05, 23.74it/s]

Generating QA Samples | 8192:  79%|███████▉  | 396/500 [00:16<00:04, 23.77it/s]

Generating QA Samples | 8192:  84%|████████▍ | 422/500 [00:17<00:03, 24.25it/s]

Generating QA Samples | 8192:  89%|████████▉ | 447/500 [00:18<00:02, 24.01it/s]

Generating QA Samples | 8192:  94%|█████████▍| 472/500 [00:19<00:01, 24.22it/s]

Generating QA Samples | 8192:  99%|█████████▉| 497/500 [00:20<00:00, 23.74it/s]
Generating QA Samples | 8192: 100%|██████████| 500/500 [00:20<00:00, 23.90it/s]
2026-07-18:14:18:55 INFO     [tasks:700] Selected tasks:
2026-07-18:14:18:55 INFO     [tasks:691] Task: ruler_qa_hotpot (ruler/qa_hotpot.yaml)
2026-07-18:14:18:55 INFO     [tasks:691] Task: ruler_fwe (ruler/fwe.yaml)
2026-07-18:14:18:55 INFO     [tasks:691] Task: ruler_vt (ruler/vt.yaml)
2026-07-18:14:18:55 INFO     [tasks:691] Task: niah_multiquery (ruler/niah_multiquery.yaml)
2026-07-18:14:18:55 INFO     [tasks:691] Task: niah_multikey_2 (ruler/niah_multikey_2.yaml)
2026-07-18:14:18:55 INFO     [tasks:691] Task: niah_single_3 (ruler/niah_single_3.yaml)
2026-07-18:14:18:55 INFO     [tasks:691] Task: niah_single_1 (ruler/niah_single_1.yaml)
2026-07-18:14:18:55 INFO     [evaluator:314] ruler_qa_hotpot: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 32, 'until': []}
2026-07-18:14:18:55 INFO     [evaluator:314] ruler_fwe: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 50, 'until': []}
2026-07-18:14:18:55 INFO     [evaluator:314] ruler_vt: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 30, 'until': []}
2026-07-18:14:18:55 INFO     [evaluator:314] niah_multiquery: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
2026-07-18:14:18:55 INFO     [evaluator:314] niah_multikey_2: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
2026-07-18:14:18:55 INFO     [evaluator:314] niah_single_3: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
2026-07-18:14:18:55 INFO     [evaluator:314] niah_single_1: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
2026-07-18:14:18:55 INFO     [api.task:311] Building contexts for ruler_qa_hotpot on rank 0...


  0%|          | 0/20 [00:00<?, ?it/s]
100%|██████████| 20/20 [00:00<00:00, 890.78it/s]
2026-07-18:14:18:55 INFO     [api.task:311] Building contexts for ruler_fwe on rank 0...


  0%|          | 0/20 [00:00<?, ?it/s]
100%|██████████| 20/20 [00:00<00:00, 27.23it/s]
2026-07-18:14:18:56 INFO     [api.task:311] Building contexts for ruler_vt on rank 0...


  0%|          | 0/20 [00:00<?, ?it/s]
100%|██████████| 20/20 [00:00<00:00, 1111.88it/s]
2026-07-18:14:18:56 INFO     [api.task:311] Building contexts for niah_multiquery on rank 0...


  0%|          | 0/20 [00:00<?, ?it/s]
100%|██████████| 20/20 [00:00<00:00, 1060.84it/s]
2026-07-18:14:18:56 INFO     [api.task:311] Building contexts for niah_multikey_2 on rank 0...


  0%|          | 0/20 [00:00<?, ?it/s]
100%|██████████| 20/20 [00:00<00:00, 1124.90it/s]
2026-07-18:14:18:56 INFO     [api.task:311] Building contexts for niah_single_3 on rank 0...


  0%|          | 0/20 [00:00<?, ?it/s]
100%|██████████| 20/20 [00:00<00:00, 1082.43it/s]
2026-07-18:14:18:56 INFO     [api.task:311] Building contexts for niah_single_1 on rank 0...


  0%|          | 0/20 [00:00<?, ?it/s]
100%|██████████| 20/20 [00:00<00:00, 1113.64it/s]
2026-07-18:14:18:56 INFO     [evaluator:584] Running generate_until requests


Running generate_until requests:   0%|          | 0/140 [00:00<?, ?it/s]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:   1%|          | 1/140 [00:02<05:45,  2.49s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:   1%|▏         | 2/140 [00:04<04:25,  1.93s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:   2%|▏         | 3/140 [00:05<03:59,  1.75s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:   3%|▎         | 4/140 [00:07<03:45,  1.66s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:   4%|▎         | 5/140 [00:08<03:37,  1.61s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:   4%|▍         | 6/140 [00:10<03:31,  1.58s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:   5%|▌         | 7/140 [00:11<03:27,  1.56s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:   6%|▌         | 8/140 [00:13<03:24,  1.55s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:   6%|▋         | 9/140 [00:14<03:21,  1.54s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:   7%|▋         | 10/140 [00:16<03:18,  1.53s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:   8%|▊         | 11/140 [00:17<03:16,  1.52s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:   9%|▊         | 12/140 [00:19<03:14,  1.52s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:   9%|▉         | 13/140 [00:20<03:11,  1.51s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  10%|█         | 14/140 [00:22<03:09,  1.50s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  11%|█         | 15/140 [00:23<03:06,  1.50s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  11%|█▏        | 16/140 [00:25<03:04,  1.49s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  12%|█▏        | 17/140 [00:26<03:02,  1.48s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  13%|█▎        | 18/140 [00:28<03:01,  1.49s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  14%|█▎        | 19/140 [00:29<02:59,  1.48s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  14%|█▍        | 20/140 [00:31<02:57,  1.48s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  15%|█▌        | 21/140 [00:33<03:32,  1.78s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  16%|█▌        | 22/140 [00:35<03:47,  1.92s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  16%|█▋        | 23/140 [00:38<03:57,  2.03s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  17%|█▋        | 24/140 [00:40<04:03,  2.10s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  18%|█▊        | 25/140 [00:42<04:06,  2.15s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  19%|█▊        | 26/140 [00:44<04:08,  2.18s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  19%|█▉        | 27/140 [00:47<04:09,  2.21s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  20%|██        | 28/140 [00:49<04:08,  2.22s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  21%|██        | 29/140 [00:51<04:07,  2.23s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  21%|██▏       | 30/140 [00:53<04:06,  2.24s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  22%|██▏       | 31/140 [00:56<04:04,  2.24s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  23%|██▎       | 32/140 [00:58<04:03,  2.25s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  24%|██▎       | 33/140 [01:00<04:00,  2.25s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  24%|██▍       | 34/140 [01:02<03:58,  2.25s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  25%|██▌       | 35/140 [01:05<03:56,  2.25s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  26%|██▌       | 36/140 [01:07<03:54,  2.26s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  26%|██▋       | 37/140 [01:09<03:52,  2.26s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  27%|██▋       | 38/140 [01:11<03:49,  2.25s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  28%|██▊       | 39/140 [01:14<03:47,  2.25s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  29%|██▊       | 40/140 [01:16<03:44,  2.25s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  29%|██▉       | 41/140 [01:18<03:29,  2.11s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  30%|███       | 42/140 [01:19<03:07,  1.91s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  31%|███       | 43/140 [01:21<02:52,  1.78s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  31%|███▏      | 44/140 [01:22<02:41,  1.68s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  32%|███▏      | 45/140 [01:24<02:32,  1.61s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  33%|███▎      | 46/140 [01:25<02:26,  1.56s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  34%|███▎      | 47/140 [01:26<02:22,  1.53s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  34%|███▍      | 48/140 [01:28<02:19,  1.51s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  35%|███▌      | 49/140 [01:29<02:15,  1.49s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  36%|███▌      | 50/140 [01:31<02:13,  1.48s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  36%|███▋      | 51/140 [01:32<02:11,  1.48s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  37%|███▋      | 52/140 [01:34<02:08,  1.46s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  38%|███▊      | 53/140 [01:35<02:06,  1.46s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  39%|███▊      | 54/140 [01:37<02:04,  1.45s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  39%|███▉      | 55/140 [01:38<02:03,  1.45s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  40%|████      | 56/140 [01:39<02:01,  1.45s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  41%|████      | 57/140 [01:41<01:59,  1.45s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  41%|████▏     | 58/140 [01:42<01:58,  1.44s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  42%|████▏     | 59/140 [01:44<01:56,  1.44s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  43%|████▎     | 60/140 [01:45<01:55,  1.44s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  44%|████▎     | 61/140 [01:52<04:00,  3.05s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  44%|████▍     | 62/140 [01:58<04:54,  3.78s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  45%|████▌     | 63/140 [02:03<05:30,  4.29s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  46%|████▌     | 64/140 [02:08<05:53,  4.65s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  46%|████▋     | 65/140 [02:14<06:07,  4.90s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  47%|████▋     | 66/140 [02:19<06:15,  5.08s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  48%|████▊     | 67/140 [02:25<06:19,  5.20s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  49%|████▊     | 68/140 [02:30<06:20,  5.29s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  49%|████▉     | 69/140 [02:36<06:19,  5.34s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  50%|█████     | 70/140 [02:41<06:17,  5.39s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  51%|█████     | 71/140 [02:47<06:13,  5.41s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  51%|█████▏    | 72/140 [02:52<06:09,  5.43s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  52%|█████▏    | 73/140 [02:58<06:05,  5.45s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  53%|█████▎    | 74/140 [03:03<06:00,  5.46s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  54%|█████▎    | 75/140 [03:09<05:55,  5.47s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  54%|█████▍    | 76/140 [03:14<05:49,  5.47s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  55%|█████▌    | 77/140 [03:20<05:44,  5.47s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  56%|█████▌    | 78/140 [03:25<05:39,  5.48s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  56%|█████▋    | 79/140 [03:31<05:34,  5.48s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  57%|█████▋    | 80/140 [03:36<05:28,  5.48s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  58%|█████▊    | 81/140 [03:42<05:23,  5.48s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  59%|█████▊    | 82/140 [03:47<05:17,  5.48s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  59%|█████▉    | 83/140 [03:53<05:12,  5.48s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  60%|██████    | 84/140 [03:58<05:07,  5.48s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  61%|██████    | 85/140 [04:04<05:02,  5.50s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  61%|██████▏   | 86/140 [04:09<04:57,  5.51s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  62%|██████▏   | 87/140 [04:15<04:52,  5.52s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  63%|██████▎   | 88/140 [04:20<04:47,  5.52s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  64%|██████▎   | 89/140 [04:26<04:41,  5.52s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  64%|██████▍   | 90/140 [04:31<04:36,  5.53s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  65%|██████▌   | 91/140 [04:37<04:30,  5.53s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  66%|██████▌   | 92/140 [04:42<04:25,  5.52s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  66%|██████▋   | 93/140 [04:48<04:19,  5.52s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  67%|██████▋   | 94/140 [04:53<04:13,  5.52s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  68%|██████▊   | 95/140 [04:59<04:08,  5.52s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  69%|██████▊   | 96/140 [05:04<04:02,  5.52s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  69%|██████▉   | 97/140 [05:10<03:57,  5.53s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  70%|███████   | 98/140 [05:15<03:51,  5.51s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  71%|███████   | 99/140 [05:21<03:45,  5.51s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  71%|███████▏  | 100/140 [05:26<03:40,  5.50s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  72%|███████▏  | 101/140 [05:32<03:34,  5.50s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  73%|███████▎  | 102/140 [05:37<03:28,  5.49s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  74%|███████▎  | 103/140 [05:43<03:23,  5.49s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  74%|███████▍  | 104/140 [05:48<03:17,  5.49s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  75%|███████▌  | 105/140 [05:54<03:12,  5.49s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  76%|███████▌  | 106/140 [05:59<03:06,  5.49s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  76%|███████▋  | 107/140 [06:05<03:01,  5.50s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  77%|███████▋  | 108/140 [06:10<02:55,  5.49s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  78%|███████▊  | 109/140 [06:16<02:49,  5.48s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  79%|███████▊  | 110/140 [06:21<02:44,  5.48s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  79%|███████▉  | 111/140 [06:27<02:38,  5.48s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  80%|████████  | 112/140 [06:32<02:33,  5.47s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  81%|████████  | 113/140 [06:38<02:27,  5.47s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  81%|████████▏ | 114/140 [06:43<02:22,  5.47s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  82%|████████▏ | 115/140 [06:49<02:16,  5.47s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  83%|████████▎ | 116/140 [06:54<02:11,  5.47s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  84%|████████▎ | 117/140 [07:00<02:05,  5.47s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  84%|████████▍ | 118/140 [07:05<02:00,  5.47s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  85%|████████▌ | 119/140 [07:11<01:55,  5.48s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  86%|████████▌ | 120/140 [07:16<01:49,  5.48s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  86%|████████▋ | 121/140 [07:21<01:44,  5.48s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  87%|████████▋ | 122/140 [07:27<01:38,  5.48s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  88%|████████▊ | 123/140 [07:32<01:33,  5.48s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  89%|████████▊ | 124/140 [07:38<01:27,  5.47s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  89%|████████▉ | 125/140 [07:43<01:22,  5.47s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  90%|█████████ | 126/140 [07:49<01:16,  5.47s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  91%|█████████ | 127/140 [07:54<01:11,  5.47s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  91%|█████████▏| 128/140 [07:59<01:03,  5.27s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  92%|█████████▏| 129/140 [08:03<00:54,  4.96s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  93%|█████████▎| 130/140 [08:08<00:47,  4.74s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  94%|█████████▎| 131/140 [08:12<00:41,  4.58s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  94%|█████████▍| 132/140 [08:16<00:35,  4.47s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  95%|█████████▌| 133/140 [08:20<00:30,  4.40s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  96%|█████████▌| 134/140 [08:24<00:26,  4.35s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  96%|█████████▋| 135/140 [08:29<00:21,  4.31s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  97%|█████████▋| 136/140 [08:33<00:17,  4.28s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  98%|█████████▊| 137/140 [08:37<00:12,  4.27s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  99%|█████████▊| 138/140 [08:41<00:08,  4.25s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests:  99%|█████████▉| 139/140 [08:46<00:04,  4.24s/it]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


Running generate_until requests: 100%|██████████| 140/140 [08:50<00:00,  4.24s/it]
Running generate_until requests: 100%|██████████| 140/140 [08:50<00:00,  3.79s/it]
2026-07-18:14:27:50 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
2026-07-18:14:27:50 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to /workspace/outputs/eval/vanilla/ruler8k_splits/a/lm_eval/__workspace/*.jsonl
hf ({'pretrained': '/workspace', 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}), gen_kwargs: ({}), limit: 20.0, num_fewshot: None, batch_size: 1
|     Tasks     |Version|Filter|n-shot|Metric|   | Value |   |Stderr|
|---------------|------:|------|-----:|-----:|---|------:|---|------|
|niah_multikey_2|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
|               |       |none  |     0|  8192|↑  | 1.0000|±  |   N/A|
|niah_multiquery|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
|               |       |none  |     0|  8192|↑  | 0.9875|±  |   N/A|
|niah_single_1  |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
|               |       |none  |     0|  8192|↑  | 1.0000|±  |   N/A|
|niah_single_3  |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
|               |       |none  |     0|  8192|↑  | 1.0000|±  |   N/A|
|ruler_fwe      |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
|               |       |none  |     0|  8192|↑  | 0.7667|±  |   N/A|
|ruler_qa_hotpot|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
|               |       |none  |     0|  8192|↑  | 0.4500|±  |   N/A|
|ruler_vt       |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
|               |       |none  |     0|  8192|↑  | 0.8600|±  |   N/A|