Delete lm_eval/test_eval.log

lm_eval/test_eval.log

@@ -1,94 +0,0 @@
-The following values were not passed to `accelerate launch` and had defaults used instead:
-	`--num_processes` was set to a value of `2`
-		More than one GPU was found, enabling multi-GPU training.
-		If this was unintended please pass in `--num_processes=1`.
-	`--num_machines` was set to a value of `1`
-	`--mixed_precision` was set to a value of `'no'`
-	`--dynamo_backend` was set to a value of `'no'`
-To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
-2026-03-19:14:15:25 INFO     [_cli.run:375] Including path: ./
-2026-03-19:14:15:25 INFO     [_cli.run:376] Selected Tasks: ['arc_easy_mi', 'arc_challenge_mi', 'hellaswag', 'piqa']
-2026-03-19:14:15:25 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
-2026-03-19:14:15:25 INFO     [evaluator:236] Initializing cloverlm model, with arguments: {'pretrained': 'daslab-testing/CloverLM', 'dtype': 'bfloat16', 'quartet_2_impl': 'quartet2', 'attn_backend': 'pytorch', 'trust_remote_code': True}
-2026-03-19:14:15:25 INFO     [_cli.run:375] Including path: ./
-2026-03-19:14:15:25 INFO     [_cli.run:376] Selected Tasks: ['arc_easy_mi', 'arc_challenge_mi', 'hellaswag', 'piqa']
-2026-03-19:14:15:25 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
-2026-03-19:14:15:25 INFO     [evaluator:236] Initializing cloverlm model, with arguments: {'pretrained': 'daslab-testing/CloverLM', 'dtype': 'bfloat16', 'quartet_2_impl': 'quartet2', 'attn_backend': 'pytorch', 'trust_remote_code': True}
-2026-03-19:14:15:26 INFO     [models.huggingface:178] Using `accelerate launch` or `parallelize=True`, device 'cuda:0' will be overridden when placing model.
-2026-03-19:14:15:26 INFO     [models.huggingface:178] Using `accelerate launch` or `parallelize=True`, device 'cuda:0' will be overridden when placing model.
-2026-03-19:14:15:26 INFO     [models.huggingface:548] Model type cannot be determined. Using default model type 'causal'
-2026-03-19:14:15:26 INFO     [models.huggingface:548] Model type cannot be determined. Using default model type 'causal'
-2026-03-19:14:15:28 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
-2026-03-19:14:15:28 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
-
-
-The tied weights mapping and config for this model specifies to tie transformer.emb.weight to transformer.linear.weight, but both are present in the checkpoints, so we will NOT tie them. You should update the config with `tie_word_embeddings=False` to silence this warning
-The tied weights mapping and config for this model specifies to tie transformer.emb.weight to transformer.linear.weight, but both are present in the checkpoints, so we will NOT tie them. You should update the config with `tie_word_embeddings=False` to silence this warning
-2026-03-19:14:15:45 INFO     [tasks:700] Selected tasks:
-2026-03-19:14:15:45 INFO     [tasks:691] Task: piqa (.venv/lib/python3.11/site-packages/lm_eval/tasks/piqa/piqa.yaml)
-2026-03-19:14:15:45 INFO     [tasks:691] Task: hellaswag (.venv/lib/python3.11/site-packages/lm_eval/tasks/hellaswag/hellaswag.yaml)
-2026-03-19:14:15:45 INFO     [tasks:691] Task: arc_challenge_mi (arc_challenge.yaml)
-2026-03-19:14:15:45 INFO     [tasks:691] Task: arc_easy_mi (arc_easy_mi.yaml)
-2026-03-19:14:15:45 WARNING  [evaluator:333] Overwriting default num_fewshot of piqa from None to 0
-2026-03-19:14:15:45 WARNING  [evaluator:333] Overwriting default num_fewshot of hellaswag from None to 0
-2026-03-19:14:15:45 WARNING  [evaluator:333] Overwriting default num_fewshot of arc_challenge_mi from None to 0
-2026-03-19:14:15:45 WARNING  [evaluator:333] Overwriting default num_fewshot of arc_easy_mi from None to 0
-2026-03-19:14:15:45 INFO     [api.task:311] Building contexts for piqa on rank 0...
-
  0%|          | 0/919 [00:00<?, ?it/s]
 10%|▉         | 91/919 [00:00<00:00, 900.62it/s]2026-03-19:14:15:45 INFO     [tasks:700] Selected tasks:
-2026-03-19:14:15:45 INFO     [tasks:691] Task: piqa (.venv/lib/python3.11/site-packages/lm_eval/tasks/piqa/piqa.yaml)
-2026-03-19:14:15:45 INFO     [tasks:691] Task: hellaswag (.venv/lib/python3.11/site-packages/lm_eval/tasks/hellaswag/hellaswag.yaml)
-2026-03-19:14:15:45 INFO     [tasks:691] Task: arc_challenge_mi (arc_challenge.yaml)
-2026-03-19:14:15:45 INFO     [tasks:691] Task: arc_easy_mi (arc_easy_mi.yaml)
-2026-03-19:14:15:45 WARNING  [evaluator:333] Overwriting default num_fewshot of piqa from None to 0
-2026-03-19:14:15:45 WARNING  [evaluator:333] Overwriting default num_fewshot of hellaswag from None to 0
-2026-03-19:14:15:45 WARNING  [evaluator:333] Overwriting default num_fewshot of arc_challenge_mi from None to 0
-2026-03-19:14:15:45 WARNING  [evaluator:333] Overwriting default num_fewshot of arc_easy_mi from None to 0
-2026-03-19:14:15:45 INFO     [api.task:311] Building contexts for piqa on rank 1...
-
 20%|█▉        | 182/919 [00:00<00:00, 905.61it/s]
  0%|          | 0/919 [00:00<?, ?it/s]
 30%|██▉       | 274/919 [00:00<00:00, 910.65it/s]
 10%|▉         | 91/919 [00:00<00:00, 907.36it/s]
 40%|███▉      | 366/919 [00:00<00:00, 911.36it/s]
 20%|█▉        | 183/919 [00:00<00:00, 911.26it/s]
 50%|████▉     | 458/919 [00:00<00:00, 911.45it/s]
 30%|██▉       | 275/919 [00:00<00:00, 915.08it/s]
 60%|█████▉    | 550/919 [00:00<00:00, 911.49it/s]
 40%|███▉      | 367/919 [00:00<00:00, 915.11it/s]
 70%|██████▉   | 642/919 [00:00<00:00, 912.08it/s]
 50%|████▉     | 459/919 [00:00<00:00, 916.54it/s]
 80%|███████▉  | 734/919 [00:00<00:00, 911.69it/s]
 60%|█████▉    | 551/919 [00:00<00:00, 914.63it/s]
 90%|████████▉ | 826/919 [00:00<00:00, 912.45it/s]
 70%|██████▉   | 643/919 [00:00<00:00, 914.29it/s]
-
 80%|███████▉  | 735/919 [00:00<00:00, 912.73it/s]
 90%|████████▉ | 827/919 [00:00<00:00, 913.03it/s]
-2026-03-19:14:15:48 INFO     [api.task:311] Building contexts for hellaswag on rank 0...
-2026-03-19:14:15:48 INFO     [api.task:311] Building contexts for hellaswag on rank 1...
-
  0%|          | 0/5021 [00:00<?, ?it/s]
  6%|▌         | 278/5021 [00:00<00:01, 2773.97it/s]
 11%|█         | 564/5021 [00:00<00:01, 2824.23it/s]
  0%|          | 0/5021 [00:00<?, ?it/s]
 17%|█▋        | 852/5021 [00:00<00:01, 2845.58it/s]
  4%|▎         | 178/5021 [00:00<00:02, 1779.16it/s]
 23%|██▎       | 1140/5021 [00:00<00:01, 2855.94it/s]
  7%|▋         | 362/5021 [00:00<00:02, 1813.26it/s]
 28%|██▊       | 1427/5021 [00:00<00:01, 2860.60it/s]
 11%|█         | 547/5021 [00:00<00:02, 1826.42it/s]
 34%|███▍      | 1714/5021 [00:00<00:01, 2863.10it/s]
 15%|█▍        | 730/5021 [00:00<00:02, 1826.29it/s]
 40%|███▉      | 2001/5021 [00:00<00:01, 2863.11it/s]
 18%|█▊        | 913/5021 [00:00<00:02, 1824.48it/s]
 46%|████▌     | 2289/5021 [00:00<00:00, 2868.32it/s]
 22%|██▏       | 1096/5021 [00:00<00:02, 1819.99it/s]
 51%|█████▏    | 2577/5021 [00:00<00:00, 2871.27it/s]
 25%|██▌       | 1279/5021 [00:00<00:02, 1815.56it/s]
 29%|██▉       | 1461/5021 [00:00<00:01, 1802.42it/s]
 57%|█████▋    | 2865/5021 [00:01<00:01, 1693.34it/s]
 63%|██████▎   | 3149/5021 [00:01<00:00, 1928.96it/s]
 68%|██████▊   | 3433/5021 [00:01<00:00, 2135.30it/s]
 33%|███▎      | 1642/5021 [00:01<00:03, 1001.41it/s]
 74%|███████▍  | 3718/5021 [00:01<00:00, 2308.62it/s]
 36%|███▋      | 1823/5021 [00:01<00:02, 1159.85it/s]
 80%|███████▉  | 4004/5021 [00:01<00:00, 2449.72it/s]
 40%|███▉      | 2002/5021 [00:01<00:02, 1297.94it/s]
 85%|████████▌ | 4290/5021 [00:01<00:00, 2558.71it/s]
 43%|████▎     | 2183/5021 [00:01<00:02, 1418.79it/s]
 91%|█████████ | 4577/5021 [00:01<00:00, 2643.21it/s]
 47%|████▋     | 2364/5021 [00:01<00:01, 1516.12it/s]
 97%|█████████▋| 4864/5021 [00:01<00:00, 2705.33it/s]
 51%|█████     | 2544/5021 [00:01<00:01, 1591.38it/s]
-
 54%|█████▍    | 2725/5021 [00:01<00:01, 1650.39it/s]
 58%|█████▊    | 2905/5021 [00:01<00:01, 1692.57it/s]
 61%|██████▏   | 3085/5021 [00:01<00:01, 1722.37it/s]
 65%|██████▌   | 3266/5021 [00:02<00:01, 1745.32it/s]
 69%|██████▊   | 3447/5021 [00:02<00:00, 1762.66it/s]
 72%|███████▏  | 3627/5021 [00:02<00:00, 1772.66it/s]
 76%|███████▌  | 3807/5021 [00:02<00:00, 1779.65it/s]
 79%|███████▉  | 3987/5021 [00:02<00:00, 1781.31it/s]
 83%|████████▎ | 4168/5021 [00:02<00:00, 1787.31it/s]
 87%|████████▋ | 4349/5021 [00:02<00:00, 1792.50it/s]
 90%|█████████ | 4529/5021 [00:02<00:00, 1790.62it/s]
 94%|█████████▍| 4709/5021 [00:02<00:00, 1790.61it/s]
 97%|█████████▋| 4890/5021 [00:02<00:00, 1793.65it/s]
-2026-03-19:14:15:52 INFO     [api.task:311] Building contexts for arc_challenge_mi on rank 1...
-2026-03-19:14:15:52 INFO     [api.task:311] Building contexts for arc_challenge_mi on rank 0...
-
  0%|          | 0/586 [00:00<?, ?it/s]
  0%|          | 0/586 [00:00<?, ?it/s]
 17%|█▋        | 98/586 [00:00<00:00, 971.82it/s]
 11%|█         | 62/586 [00:00<00:00, 613.37it/s]
 34%|███▎      | 197/586 [00:00<00:00, 976.00it/s]
 21%|██▏       | 125/586 [00:00<00:00, 619.71it/s]
 51%|█████     | 297/586 [00:00<00:00, 985.47it/s]
 32%|███▏      | 189/586 [00:00<00:00, 624.63it/s]
 68%|██████▊   | 396/586 [00:00<00:00, 986.66it/s]
 43%|████▎     | 253/586 [00:00<00:00, 627.18it/s]
 85%|████████▍ | 496/586 [00:00<00:00, 989.81it/s]
 54%|█████▍    | 317/586 [00:00<00:00, 628.58it/s]
-
 65%|██████▌   | 381/586 [00:00<00:00, 629.34it/s]
 76%|███████▌  | 445/586 [00:00<00:00, 630.51it/s]
 87%|████████▋ | 509/586 [00:00<00:00, 632.25it/s]
 98%|█████████▊| 573/586 [00:00<00:00, 632.82it/s]
-2026-03-19:14:15:53 INFO     [api.task:311] Building contexts for arc_easy_mi on rank 0...
-2026-03-19:14:15:53 INFO     [api.task:311] Building contexts for arc_easy_mi on rank 1...
-
  0%|          | 0/1188 [00:00<?, ?it/s]
  0%|          | 0/1188 [00:00<?, ?it/s]
  8%|▊         | 100/1188 [00:00<00:01, 993.05it/s]
  5%|▌         | 63/1188 [00:00<00:01, 626.46it/s]
 17%|█▋        | 200/1188 [00:00<00:00, 991.06it/s]
 11%|█         | 127/1188 [00:00<00:01, 629.71it/s]
 25%|██▌       | 300/1188 [00:00<00:00, 994.71it/s]
 16%|█▌        | 191/1188 [00:00<00:01, 630.93it/s]
 34%|███▎      | 400/1188 [00:00<00:00, 993.35it/s]
 21%|██▏       | 255/1188 [00:00<00:01, 633.00it/s]
 42%|████▏     | 500/1188 [00:00<00:00, 993.59it/s]
 27%|██▋       | 319/1188 [00:00<00:01, 634.81it/s]
 51%|█████     | 600/1188 [00:00<00:00, 993.86it/s]
 32%|███▏      | 383/1188 [00:00<00:01, 636.25it/s]
 59%|█████▉    | 700/1188 [00:00<00:00, 992.08it/s]
 38%|███▊      | 447/1188 [00:00<00:01, 636.11it/s]
 67%|██████▋   | 800/1188 [00:00<00:00, 988.12it/s]
 43%|████▎     | 511/1188 [00:00<00:01, 634.75it/s]
 76%|███████▌  | 899/1188 [00:00<00:00, 988.66it/s]
 48%|████▊     | 575/1188 [00:00<00:00, 634.67it/s]
 84%|████████▍ | 999/1188 [00:01<00:00, 991.15it/s]
 54%|█████▍    | 639/1188 [00:01<00:00, 633.38it/s]
 93%|█████████▎| 1099/1188 [00:01<00:00, 993.44it/s]
 59%|█████▉    | 703/1188 [00:01<00:00, 633.82it/s]
-
 65%|██████▍   | 768/1188 [00:01<00:00, 635.72it/s]
 70%|███████   | 832/1188 [00:01<00:00, 636.49it/s]
 75%|███████▌  | 896/1188 [00:01<00:00, 632.00it/s]
 81%|████████  | 960/1188 [00:01<00:00, 628.76it/s]
 86%|████████▌ | 1023/1188 [00:01<00:00, 624.90it/s]
 91%|█████████▏| 1086/1188 [00:01<00:00, 625.37it/s]
 97%|█████████▋| 1149/1188 [00:01<00:00, 625.85it/s]
-2026-03-19:14:15:55 INFO     [evaluator:584] Running loglikelihood requests
-2026-03-19:14:15:55 INFO     [evaluator:584] Running loglikelihood requests
-
-Passed argument batch_size = auto:1. Detecting largest batch size
-Determined largest batch size: 64
-Determined largest batch size: 64
-
-[rank1]:W0319 14:20:19.946000 1458598 .venv/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py:1676] [0/8]    function: 'abs_max' (/home/matin/convert_dir/CloverLM/lm_eval/.venv/lib/python3.11/site-packages/quartet2/linear.py:147)
-[rank1]:W0319 14:20:19.946000 1458598 .venv/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py:1676] [0/8]    last reason: 0/5: tensor 'x' requires_grad mismatch. expected requires_grad=1
-[rank1]:W0319 14:20:19.946000 1458598 .venv/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py:1676] [0/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
-[rank1]:W0319 14:20:19.946000 1458598 .venv/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py:1676] [0/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/compile/programming_model.recompilation.html
-
-[rank0]:W0319 14:20:22.844000 1458597 .venv/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py:1676] [0/8]    function: 'abs_max' (/home/matin/convert_dir/CloverLM/lm_eval/.venv/lib/python3.11/site-packages/quartet2/linear.py:147)
-[rank0]:W0319 14:20:22.844000 1458597 .venv/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py:1676] [0/8]    last reason: 0/5: tensor 'x' requires_grad mismatch. expected requires_grad=1
-[rank0]:W0319 14:20:22.844000 1458597 .venv/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py:1676] [0/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
-[rank0]:W0319 14:20:22.844000 1458597 .venv/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py:1676] [0/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/compile/programming_model.recompilation.html
-
-fatal: not a git repository (or any of the parent directories): .git
-2026-03-19:14:20:29 INFO     [loggers.evaluation_tracker:316] Output path not provided, skipping saving results aggregated
-cloverlm ({'pretrained': 'daslab-testing/CloverLM', 'dtype': 'bfloat16', 'quartet_2_impl': 'quartet2', 'attn_backend': 'pytorch'}), gen_kwargs: ({}), limit: None, num_fewshot: 0, batch_size: auto (64)
-|     Tasks      |Version|Filter|n-shot|    Metric     |   |Value |   |Stderr|
-|----------------|------:|------|-----:|---------------|---|-----:|---|-----:|
-|arc_challenge_mi|      1|none  |     0|acc            |↑  |0.4642|±  |0.0146|
-|                |       |none  |     0|acc_mutual_info|↑  |0.5017|±  |0.0146|
-|                |       |none  |     0|acc_norm       |↑  |0.4940|±  |0.0146|
-|arc_easy_mi     |      1|none  |     0|acc            |↑  |0.8005|±  |0.0082|
-|                |       |none  |     0|acc_mutual_info|↑  |0.7193|±  |0.0092|
-|                |       |none  |     0|acc_norm       |↑  |0.7740|±  |0.0086|
-|hellaswag       |      1|none  |     0|acc            |↑  |0.5392|±  |0.0050|
-|                |       |none  |     0|acc_norm       |↑  |0.7169|±  |0.0045|
-|piqa            |      1|none  |     0|acc            |↑  |0.7911|±  |0.0095|
-|                |       |none  |     0|acc_norm       |↑  |0.8090|±  |0.0092|
-
-[rank0]:[W319 14:20:30.213375773 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())