eval_lighteval|gpqa:diamond|0.log

INFO 10-26 08:02:52 [__init__.py:235] Automatically detected platform cuda.
[2025-10-26 08:02:53,753] [    INFO]: --- INIT SEEDS --- (pipeline.py:249)
[2025-10-26 08:02:53,754] [    INFO]: --- LOADING TASKS --- (pipeline.py:210)
[2025-10-26 08:02:53,757] [ WARNING]: Careful, the task gpqa:diamond is using evaluation data to build the few shot examples. (lighteval_task.py:269)
[2025-10-26 08:03:00,271] [    INFO]: --- LOADING MODEL --- (pipeline.py:177)
`torch_dtype` is deprecated! Use `dtype` instead!
[2025-10-26 08:03:06,633] [    INFO]: Using max model len 32768 (config.py:1604)
[2025-10-26 08:03:07,276] [    INFO]: Chunked prefill is enabled with max_num_batched_tokens=2048. (config.py:2434)
INFO 10-26 08:03:11 [__init__.py:235] Automatically detected platform cuda.
INFO 10-26 08:03:13 [core.py:572] Waiting for init message from front-end.
INFO 10-26 08:03:13 [core.py:71] Initializing a V1 LLM engine (v0.10.0) with config: model='/mnt/public/wucanhui/outputs/Qwen3-4B-math-reasoning/checkpoint-2562', speculative_config=None, tokenizer='/mnt/public/wucanhui/outputs/Qwen3-4B-math-reasoning/checkpoint-2562', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, override_neuron_config={}, tokenizer_revision=main, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=1234, served_model_name=/mnt/public/wucanhui/outputs/Qwen3-4B-math-reasoning/checkpoint-2562, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":0,"local_cache_dir":null}
INFO 10-26 08:03:17 [parallel_state.py:1102] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
WARNING 10-26 08:03:17 [topk_topp_sampler.py:59] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
INFO 10-26 08:03:17 [gpu_model_runner.py:1843] Starting to load model /mnt/public/wucanhui/outputs/Qwen3-4B-math-reasoning/checkpoint-2562...
INFO 10-26 08:03:18 [gpu_model_runner.py:1875] Loading model from scratch...
INFO 10-26 08:03:18 [cuda.py:290] Using Flash Attention backend on V1 engine.

Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]

Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:30<00:30, 30.81s/it]

Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:52<00:00, 25.24s/it]

Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:52<00:00, 26.07s/it]

INFO 10-26 08:04:11 [default_loader.py:262] Loading weights took 52.58 seconds
INFO 10-26 08:04:11 [gpu_model_runner.py:1892] Model loading took 7.5552 GiB and 53.175289 seconds
INFO 10-26 08:04:12 [gpu_worker.py:255] Available KV cache memory: 117.60 GiB
INFO 10-26 08:04:12 [kv_cache_utils.py:833] GPU KV cache size: 856,336 tokens
INFO 10-26 08:04:12 [kv_cache_utils.py:837] Maximum concurrency for 32,768 tokens per request: 26.13x
INFO 10-26 08:04:13 [core.py:193] init engine (profile, create kv cache, warmup model) took 1.35 seconds
[2025-10-26 08:04:13,641] [    INFO]: [CACHING] Initializing data cache (cache_management.py:105)
[2025-10-26 08:04:13,649] [    INFO]: --- RUNNING MODEL --- (pipeline.py:330)
[2025-10-26 08:04:13,650] [    INFO]: Running SamplingMethod.GENERATIVE requests (pipeline.py:313)
[2025-10-26 08:04:19,459] [    INFO]: Cache: Starting to process 198/198 samples (not found in cache) for tasks lighteval|gpqa:diamond|0 (cd454966f6c36e03, GENERATIVE) (cache_management.py:399)
[2025-10-26 08:04:19,461] [ WARNING]: You cannot select the number of dataset splits for a generative evaluation at the moment. Automatically inferring. (data.py:206)

Splits:   0%|          | 0/1 [00:00<?, ?it/s][2025-10-26 08:04:19,562] [ WARNING]: context_size + max_new_tokens=35595 which is greater than self.max_length=32768. Truncating context to 0 tokens. (vllm_model.py:367)


Adding requests:   0%|          | 0/198 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 198/198 [00:00<00:00, 14780.77it/s]


Processed prompts:   0%|          | 0/198 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]

Processed prompts:   1%|          | 1/198 [00:04<14:14,  4.34s/it, est. speed input: 49.11 toks/s, output: 55.79 toks/s]

Processed prompts:   1%|          | 2/198 [00:04<06:07,  1.87s/it, est. speed input: 96.51 toks/s, output: 111.44 toks/s]

Processed prompts:   2%|▏         | 3/198 [00:05<04:39,  1.43s/it, est. speed input: 105.96 toks/s, output: 151.15 toks/s]

Processed prompts:   4%|▎         | 7/198 [00:05<01:21,  2.34it/s, est. speed input: 272.77 toks/s, output: 382.30 toks/s]

Processed prompts:   5%|▍         | 9/198 [00:05<00:59,  3.18it/s, est. speed input: 332.27 toks/s, output: 487.53 toks/s]

Processed prompts:   6%|▌         | 11/198 [00:05<00:46,  4.02it/s, est. speed input: 422.78 toks/s, output: 590.19 toks/s]

Processed prompts:   7%|▋         | 13/198 [00:06<00:43,  4.27it/s, est. speed input: 457.06 toks/s, output: 669.98 toks/s]

Processed prompts:   7%|▋         | 14/198 [00:06<00:39,  4.60it/s, est. speed input: 483.34 toks/s, output: 716.18 toks/s]

Processed prompts:   8%|▊         | 15/198 [00:07<00:57,  3.21it/s, est. speed input: 462.59 toks/s, output: 709.27 toks/s]

Processed prompts:   8%|▊         | 16/198 [00:07<00:49,  3.69it/s, est. speed input: 565.37 toks/s, output: 759.02 toks/s]

Processed prompts:   9%|▊         | 17/198 [00:07<00:47,  3.79it/s, est. speed input: 578.76 toks/s, output: 796.59 toks/s]

Processed prompts:   9%|▉         | 18/198 [00:07<00:42,  4.21it/s, est. speed input: 599.94 toks/s, output: 841.74 toks/s]

Processed prompts:  10%|▉         | 19/198 [00:07<00:36,  4.93it/s, est. speed input: 617.73 toks/s, output: 891.80 toks/s]

Processed prompts:  10%|█         | 20/198 [00:08<00:53,  3.34it/s, est. speed input: 605.67 toks/s, output: 894.24 toks/s]

Processed prompts:  11%|█         | 21/198 [00:08<00:53,  3.31it/s, est. speed input: 617.19 toks/s, output: 924.15 toks/s]

Processed prompts:  11%|█         | 22/198 [00:08<00:50,  3.48it/s, est. speed input: 629.00 toks/s, output: 960.05 toks/s]

Processed prompts:  12%|█▏        | 23/198 [00:10<01:34,  1.85it/s, est. speed input: 578.91 toks/s, output: 912.54 toks/s]

Processed prompts:  13%|█▎        | 25/198 [00:10<01:05,  2.64it/s, est. speed input: 620.69 toks/s, output: 1002.76 toks/s]

Processed prompts:  14%|█▎        | 27/198 [00:10<00:50,  3.39it/s, est. speed input: 654.55 toks/s, output: 1094.40 toks/s]

Processed prompts:  15%|█▌        | 30/198 [00:10<00:31,  5.40it/s, est. speed input: 747.08 toks/s, output: 1263.64 toks/s]

Processed prompts:  16%|█▌        | 32/198 [00:11<00:24,  6.88it/s, est. speed input: 773.07 toks/s, output: 1342.85 toks/s]

Processed prompts:  17%|█▋        | 34/198 [00:11<00:20,  7.99it/s, est. speed input: 816.04 toks/s, output: 1448.44 toks/s]

Processed prompts:  18%|█▊        | 36/198 [00:11<00:18,  8.85it/s, est. speed input: 840.03 toks/s, output: 1550.12 toks/s]

Processed prompts:  19%|█▉        | 38/198 [00:11<00:19,  8.17it/s, est. speed input: 893.79 toks/s, output: 1636.07 toks/s]

Processed prompts:  20%|██        | 40/198 [00:11<00:20,  7.87it/s, est. speed input: 993.76 toks/s, output: 1724.64 toks/s]

Processed prompts:  21%|██        | 42/198 [00:12<00:22,  6.82it/s, est. speed input: 1006.86 toks/s, output: 1794.18 toks/s]

Processed prompts:  23%|██▎       | 46/198 [00:12<00:14, 10.42it/s, est. speed input: 1081.84 toks/s, output: 1993.65 toks/s]

Processed prompts:  24%|██▍       | 48/198 [00:12<00:16,  9.07it/s, est. speed input: 1113.51 toks/s, output: 2070.03 toks/s]

Processed prompts:  25%|██▌       | 50/198 [00:13<00:16,  8.81it/s, est. speed input: 1140.91 toks/s, output: 2156.40 toks/s]

Processed prompts:  26%|██▋       | 52/198 [00:13<00:19,  7.43it/s, est. speed input: 1142.02 toks/s, output: 2192.37 toks/s]

Processed prompts:  27%|██▋       | 54/198 [00:14<00:25,  5.56it/s, est. speed input: 1148.01 toks/s, output: 2222.83 toks/s]

Processed prompts:  28%|██▊       | 56/198 [00:14<00:25,  5.57it/s, est. speed input: 1163.71 toks/s, output: 2292.22 toks/s]

Processed prompts:  29%|██▉       | 57/198 [00:14<00:27,  5.17it/s, est. speed input: 1165.49 toks/s, output: 2313.27 toks/s]

Processed prompts:  30%|██▉       | 59/198 [00:14<00:22,  6.10it/s, est. speed input: 1209.92 toks/s, output: 2406.87 toks/s]

Processed prompts:  30%|███       | 60/198 [00:15<00:26,  5.14it/s, est. speed input: 1204.03 toks/s, output: 2417.93 toks/s]

Processed prompts:  31%|███▏      | 62/198 [00:15<00:22,  5.92it/s, est. speed input: 1228.17 toks/s, output: 2504.77 toks/s]

Processed prompts:  32%|███▏      | 64/198 [00:15<00:20,  6.64it/s, est. speed input: 1256.62 toks/s, output: 2593.51 toks/s]

Processed prompts:  33%|███▎      | 65/198 [00:15<00:22,  5.92it/s, est. speed input: 1254.45 toks/s, output: 2586.31 toks/s]

Processed prompts:  34%|███▍      | 67/198 [00:16<00:18,  7.12it/s, est. speed input: 1286.11 toks/s, output: 2683.52 toks/s]

Processed prompts:  34%|███▍      | 68/198 [00:16<00:25,  5.07it/s, est. speed input: 1288.49 toks/s, output: 2678.29 toks/s]

Processed prompts:  36%|███▌      | 71/198 [00:16<00:16,  7.74it/s, est. speed input: 1329.31 toks/s, output: 2822.87 toks/s]

Processed prompts:  37%|███▋      | 73/198 [00:17<00:19,  6.40it/s, est. speed input: 1324.21 toks/s, output: 2834.80 toks/s]

Processed prompts:  37%|███▋      | 74/198 [00:17<00:18,  6.57it/s, est. speed input: 1331.69 toks/s, output: 2875.72 toks/s]

Processed prompts:  38%|███▊      | 75/198 [00:17<00:17,  6.87it/s, est. speed input: 1334.10 toks/s, output: 2918.51 toks/s]

Processed prompts:  38%|███▊      | 76/198 [00:17<00:26,  4.63it/s, est. speed input: 1307.74 toks/s, output: 2860.04 toks/s]

Processed prompts:  39%|███▉      | 77/198 [00:18<00:29,  4.08it/s, est. speed input: 1298.10 toks/s, output: 2852.18 toks/s]

Processed prompts:  39%|███▉      | 78/198 [00:18<00:25,  4.71it/s, est. speed input: 1304.17 toks/s, output: 2878.15 toks/s]

Processed prompts:  40%|███▉      | 79/198 [00:18<00:31,  3.83it/s, est. speed input: 1298.55 toks/s, output: 2881.28 toks/s]

Processed prompts:  40%|████      | 80/198 [00:19<00:40,  2.94it/s, est. speed input: 1274.53 toks/s, output: 2846.22 toks/s]

Processed prompts:  41%|████      | 81/198 [00:19<00:50,  2.31it/s, est. speed input: 1243.60 toks/s, output: 2796.53 toks/s]

Processed prompts:  42%|████▏     | 83/198 [00:20<00:36,  3.18it/s, est. speed input: 1265.71 toks/s, output: 2877.40 toks/s]

Processed prompts:  42%|████▏     | 84/198 [00:20<00:30,  3.79it/s, est. speed input: 1269.51 toks/s, output: 2900.81 toks/s]

Processed prompts:  43%|████▎     | 85/198 [00:20<00:30,  3.76it/s, est. speed input: 1263.91 toks/s, output: 2926.20 toks/s]

Processed prompts:  45%|████▍     | 89/198 [00:20<00:13,  7.86it/s, est. speed input: 1293.98 toks/s, output: 3030.93 toks/s]

Processed prompts:  46%|████▌     | 91/198 [00:21<00:16,  6.51it/s, est. speed input: 1288.31 toks/s, output: 3063.37 toks/s]

Processed prompts:  47%|████▋     | 93/198 [00:21<00:18,  5.54it/s, est. speed input: 1281.48 toks/s, output: 3080.15 toks/s]

Processed prompts:  48%|████▊     | 95/198 [00:21<00:17,  5.90it/s, est. speed input: 1280.79 toks/s, output: 3089.64 toks/s]

Processed prompts:  49%|████▉     | 97/198 [00:22<00:14,  7.03it/s, est. speed input: 1287.78 toks/s, output: 3139.40 toks/s]

Processed prompts:  49%|████▉     | 98/198 [00:22<00:16,  6.22it/s, est. speed input: 1293.76 toks/s, output: 3168.83 toks/s]

Processed prompts:  50%|█████     | 99/198 [00:22<00:21,  4.54it/s, est. speed input: 1281.66 toks/s, output: 3169.18 toks/s]

Processed prompts:  52%|█████▏    | 102/198 [00:22<00:14,  6.83it/s, est. speed input: 1297.13 toks/s, output: 3283.92 toks/s]

Processed prompts:  52%|█████▏    | 103/198 [00:23<00:18,  5.26it/s, est. speed input: 1287.00 toks/s, output: 3293.20 toks/s]

Processed prompts:  53%|█████▎    | 105/198 [00:23<00:21,  4.41it/s, est. speed input: 1270.06 toks/s, output: 3294.28 toks/s]

Processed prompts:  54%|█████▎    | 106/198 [00:24<00:19,  4.81it/s, est. speed input: 1273.73 toks/s, output: 3340.82 toks/s]

Processed prompts:  54%|█████▍    | 107/198 [00:24<00:23,  3.88it/s, est. speed input: 1267.14 toks/s, output: 3346.09 toks/s]

Processed prompts:  55%|█████▌    | 109/198 [00:24<00:17,  5.19it/s, est. speed input: 1279.38 toks/s, output: 3450.72 toks/s]

Processed prompts:  56%|█████▌    | 111/198 [00:25<00:16,  5.18it/s, est. speed input: 1273.63 toks/s, output: 3487.43 toks/s]

Processed prompts:  57%|█████▋    | 112/198 [00:25<00:17,  4.88it/s, est. speed input: 1267.40 toks/s, output: 3486.67 toks/s]

Processed prompts:  58%|█████▊    | 114/198 [00:25<00:12,  6.68it/s, est. speed input: 1276.72 toks/s, output: 3558.72 toks/s]

Processed prompts:  58%|█████▊    | 115/198 [00:25<00:12,  6.51it/s, est. speed input: 1278.41 toks/s, output: 3599.78 toks/s]

Processed prompts:  59%|█████▊    | 116/198 [00:26<00:25,  3.28it/s, est. speed input: 1244.65 toks/s, output: 3518.60 toks/s]

Processed prompts:  59%|█████▉    | 117/198 [00:26<00:25,  3.21it/s, est. speed input: 1236.25 toks/s, output: 3511.89 toks/s]

Processed prompts:  60%|█████▉    | 118/198 [00:27<00:37,  2.15it/s, est. speed input: 1203.39 toks/s, output: 3453.06 toks/s]

Processed prompts:  60%|██████    | 119/198 [00:29<01:06,  1.19it/s, est. speed input: 1134.20 toks/s, output: 3274.82 toks/s]

Processed prompts:  61%|██████    | 121/198 [00:29<00:40,  1.90it/s, est. speed input: 1137.73 toks/s, output: 3348.52 toks/s]

Processed prompts:  62%|██████▏   | 122/198 [00:30<00:43,  1.75it/s, est. speed input: 1115.86 toks/s, output: 3302.50 toks/s]

Processed prompts:  62%|██████▏   | 123/198 [00:30<00:34,  2.18it/s, est. speed input: 1116.14 toks/s, output: 3324.28 toks/s]

Processed prompts:  63%|██████▎   | 124/198 [00:31<00:37,  2.00it/s, est. speed input: 1102.25 toks/s, output: 3324.86 toks/s]

Processed prompts:  63%|██████▎   | 125/198 [00:31<00:39,  1.87it/s, est. speed input: 1091.62 toks/s, output: 3325.71 toks/s]

Processed prompts:  64%|██████▍   | 127/198 [00:32<00:32,  2.19it/s, est. speed input: 1081.11 toks/s, output: 3345.98 toks/s]

Processed prompts:  65%|██████▍   | 128/198 [00:32<00:30,  2.27it/s, est. speed input: 1078.32 toks/s, output: 3372.93 toks/s]

Processed prompts:  65%|██████▌   | 129/198 [00:34<00:48,  1.42it/s, est. speed input: 1039.36 toks/s, output: 3294.16 toks/s]

Processed prompts:  66%|██████▌   | 130/198 [00:35<00:46,  1.47it/s, est. speed input: 1026.28 toks/s, output: 3277.33 toks/s]

Processed prompts:  66%|██████▌   | 131/198 [00:36<00:51,  1.31it/s, est. speed input: 1006.22 toks/s, output: 3255.03 toks/s]

Processed prompts:  67%|██████▋   | 132/198 [00:36<00:43,  1.53it/s, est. speed input: 1003.48 toks/s, output: 3287.93 toks/s]

Processed prompts:  67%|██████▋   | 133/198 [00:37<00:41,  1.55it/s, est. speed input: 993.76 toks/s, output: 3300.07 toks/s] 

Processed prompts:  68%|██████▊   | 134/198 [00:37<00:43,  1.48it/s, est. speed input: 983.46 toks/s, output: 3301.39 toks/s]

Processed prompts:  68%|██████▊   | 135/198 [00:38<00:38,  1.63it/s, est. speed input: 981.06 toks/s, output: 3327.89 toks/s]

Processed prompts:  69%|██████▉   | 137/198 [00:38<00:28,  2.12it/s, est. speed input: 975.17 toks/s, output: 3366.98 toks/s]

Processed prompts:  70%|██████▉   | 138/198 [00:39<00:24,  2.43it/s, est. speed input: 974.96 toks/s, output: 3414.96 toks/s]

Processed prompts:  70%|███████   | 139/198 [00:39<00:30,  1.94it/s, est. speed input: 962.24 toks/s, output: 3412.48 toks/s]

Processed prompts:  71%|███████   | 140/198 [00:40<00:28,  2.03it/s, est. speed input: 958.02 toks/s, output: 3443.46 toks/s]

Processed prompts:  71%|███████   | 141/198 [00:41<00:33,  1.68it/s, est. speed input: 946.21 toks/s, output: 3438.58 toks/s]

Processed prompts:  72%|███████▏  | 142/198 [00:41<00:33,  1.65it/s, est. speed input: 935.92 toks/s, output: 3436.02 toks/s]

Processed prompts:  72%|███████▏  | 143/198 [00:42<00:27,  2.02it/s, est. speed input: 934.67 toks/s, output: 3472.42 toks/s]

Processed prompts:  73%|███████▎  | 145/198 [00:42<00:15,  3.34it/s, est. speed input: 942.39 toks/s, output: 3578.35 toks/s]

Processed prompts:  74%|███████▍  | 147/198 [00:42<00:11,  4.62it/s, est. speed input: 948.25 toks/s, output: 3679.33 toks/s]

Processed prompts:  75%|███████▍  | 148/198 [00:46<00:58,  1.17s/it, est. speed input: 861.22 toks/s, output: 3389.12 toks/s]

Processed prompts:  75%|███████▌  | 149/198 [00:48<01:02,  1.27s/it, est. speed input: 839.64 toks/s, output: 3345.07 toks/s]

Processed prompts:  76%|███████▌  | 150/198 [00:49<00:52,  1.10s/it, est. speed input: 835.95 toks/s, output: 3374.36 toks/s]

Processed prompts:  76%|███████▋  | 151/198 [00:50<00:52,  1.11s/it, est. speed input: 822.89 toks/s, output: 3365.56 toks/s]

Processed prompts:  77%|███████▋  | 153/198 [00:51<00:40,  1.11it/s, est. speed input: 811.38 toks/s, output: 3400.47 toks/s]

Processed prompts:  78%|███████▊  | 155/198 [00:51<00:27,  1.58it/s, est. speed input: 815.39 toks/s, output: 3498.80 toks/s]

Processed prompts:  79%|███████▉  | 156/198 [00:52<00:23,  1.79it/s, est. speed input: 815.40 toks/s, output: 3547.59 toks/s]

Processed prompts:  79%|███████▉  | 157/198 [00:54<00:40,  1.01it/s, est. speed input: 782.71 toks/s, output: 3449.67 toks/s]

Processed prompts:  80%|███████▉  | 158/198 [00:55<00:35,  1.12it/s, est. speed input: 778.23 toks/s, output: 3481.79 toks/s]

Processed prompts:  80%|████████  | 159/198 [00:56<00:35,  1.10it/s, est. speed input: 776.61 toks/s, output: 3492.18 toks/s]

Processed prompts:  81%|████████  | 160/198 [00:57<00:42,  1.11s/it, est. speed input: 758.84 toks/s, output: 3462.22 toks/s]

Processed prompts:  81%|████████▏ | 161/198 [01:02<01:15,  2.05s/it, est. speed input: 706.52 toks/s, output: 3268.86 toks/s]

Processed prompts:  82%|████████▏ | 162/198 [01:07<01:42,  2.85s/it, est. speed input: 658.28 toks/s, output: 3102.65 toks/s]

Processed prompts:  82%|████████▏ | 163/198 [01:08<01:23,  2.37s/it, est. speed input: 648.27 toks/s, output: 3103.14 toks/s]

Processed prompts:  83%|████████▎ | 164/198 [01:12<01:37,  2.87s/it, est. speed input: 615.22 toks/s, output: 2999.45 toks/s]

Processed prompts:  83%|████████▎ | 165/198 [01:13<01:20,  2.44s/it, est. speed input: 605.61 toks/s, output: 3002.98 toks/s]

Processed prompts:  84%|████████▍ | 166/198 [01:16<01:18,  2.47s/it, est. speed input: 587.00 toks/s, output: 2960.16 toks/s]

Processed prompts:  84%|████████▍ | 167/198 [01:17<01:08,  2.20s/it, est. speed input: 577.86 toks/s, output: 2965.74 toks/s]

Processed prompts:  85%|████████▍ | 168/198 [01:19<01:00,  2.02s/it, est. speed input: 568.89 toks/s, output: 2973.02 toks/s]

Processed prompts:  85%|████████▌ | 169/198 [01:20<00:48,  1.67s/it, est. speed input: 564.84 toks/s, output: 3006.33 toks/s]

Processed prompts:  86%|████████▌ | 170/198 [01:21<00:45,  1.64s/it, est. speed input: 556.40 toks/s, output: 3012.16 toks/s]

Processed prompts:  86%|████████▋ | 171/198 [01:22<00:38,  1.41s/it, est. speed input: 554.01 toks/s, output: 3050.86 toks/s]

Processed prompts:  87%|████████▋ | 172/198 [01:22<00:26,  1.04s/it, est. speed input: 556.31 toks/s, output: 3116.39 toks/s]

Processed prompts:  88%|████████▊ | 174/198 [01:23<00:14,  1.61it/s, est. speed input: 558.90 toks/s, output: 3232.29 toks/s]

Processed prompts:  88%|████████▊ | 175/198 [01:27<00:37,  1.63s/it, est. speed input: 531.29 toks/s, output: 3122.55 toks/s]

Processed prompts:  89%|████████▉ | 176/198 [01:29<00:36,  1.65s/it, est. speed input: 523.74 toks/s, output: 3133.99 toks/s]

Processed prompts:  89%|████████▉ | 177/198 [01:32<00:42,  2.00s/it, est. speed input: 508.91 toks/s, output: 3099.40 toks/s]

Processed prompts:  90%|████████▉ | 178/198 [01:35<00:44,  2.25s/it, est. speed input: 496.60 toks/s, output: 3078.25 toks/s]

Processed prompts:  90%|█████████ | 179/198 [01:37<00:40,  2.13s/it, est. speed input: 489.62 toks/s, output: 3092.54 toks/s]

Processed prompts:  91%|█████████ | 180/198 [01:44<01:04,  3.61s/it, est. speed input: 457.18 toks/s, output: 2941.81 toks/s]

Processed prompts:  91%|█████████▏| 181/198 [01:46<00:54,  3.20s/it, est. speed input: 449.32 toks/s, output: 2953.95 toks/s]

Processed prompts:  92%|█████████▏| 182/198 [01:50<00:55,  3.47s/it, est. speed input: 434.95 toks/s, output: 2917.33 toks/s]

Processed prompts:  92%|█████████▏| 183/198 [02:01<01:25,  5.68s/it, est. speed input: 396.94 toks/s, output: 2720.90 toks/s]

Processed prompts:  93%|█████████▎| 185/198 [02:08<01:01,  4.74s/it, est. speed input: 378.03 toks/s, output: 2705.20 toks/s]

Processed prompts:  94%|█████████▍| 186/198 [02:09<00:43,  3.66s/it, est. speed input: 378.64 toks/s, output: 2766.62 toks/s]

Processed prompts:  94%|█████████▍| 187/198 [02:11<00:34,  3.17s/it, est. speed input: 375.14 toks/s, output: 2803.41 toks/s]

Processed prompts:  95%|█████████▍| 188/198 [02:15<00:34,  3.45s/it, est. speed input: 365.32 toks/s, output: 2791.42 toks/s]

Processed prompts:  95%|█████████▌| 189/198 [02:35<01:12,  8.06s/it, est. speed input: 319.61 toks/s, output: 2505.53 toks/s]

Processed prompts:  96%|█████████▌| 190/198 [07:01<10:52, 81.57s/it, est. speed input: 124.57 toks/s, output: 995.01 toks/s] 

Processed prompts:  96%|█████████▋| 191/198 [07:37<08:00, 68.65s/it, est. speed input: 115.12 toks/s, output: 985.74 toks/s]

Processed prompts:  97%|█████████▋| 192/198 [07:38<04:52, 48.74s/it, est. speed input: 115.47 toks/s, output: 1055.41 toks/s]

Processed prompts:  97%|█████████▋| 193/198 [07:38<02:52, 34.44s/it, est. speed input: 115.89 toks/s, output: 1125.79 toks/s]

Processed prompts:  98%|█████████▊| 194/198 [07:39<01:37, 24.40s/it, est. speed input: 116.13 toks/s, output: 1195.36 toks/s]

Processed prompts:  98%|█████████▊| 195/198 [07:46<00:57, 19.13s/it, est. speed input: 114.92 toks/s, output: 1248.11 toks/s]

Processed prompts:  99%|█████████▉| 197/198 [07:46<00:10, 10.51s/it, est. speed input: 115.57 toks/s, output: 1385.70 toks/s]

Processed prompts: 100%|██████████| 198/198 [07:49<00:00,  8.58s/it, est. speed input: 115.14 toks/s, output: 1447.26 toks/s]

Processed prompts: 100%|██████████| 198/198 [07:49<00:00,  8.58s/it, est. speed input: 115.14 toks/s, output: 1447.26 toks/s]
Processed prompts: 100%|██████████| 198/198 [07:49<00:00,  2.37s/it, est. speed input: 115.14 toks/s, output: 1447.26 toks/s]

Splits: 100%|██████████| 1/1 [07:49<00:00, 469.64s/it]
Splits: 100%|██████████| 1/1 [07:49<00:00, 469.64s/it]

Creating parquet from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 1/1 [00:00<00:00, 18.94ba/s]
[2025-10-26 08:12:12,128] [    INFO]: Cached 198 samples of lighteval|gpqa:diamond|0 (cd454966f6c36e03, GENERATIVE) at /mnt/public/wucanhui/outputs/Qwen3-4B-math-reasoning/checkpoint-2562/0619260e1176b049/lighteval|gpqa:diamond|0/cd454966f6c36e03/GENERATIVE.parquet. (cache_management.py:345)

Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 198 examples [00:00, 5401.52 examples/s]
[rank0]:[W1026 08:12:16.799573779 ProcessGroupNCCL.cpp:1479] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
[2025-10-26 08:12:17,647] [    INFO]: --- POST-PROCESSING MODEL RESPONSES --- (pipeline.py:344)
[2025-10-26 08:12:17,650] [    INFO]: --- COMPUTING METRICS --- (pipeline.py:371)
[2025-10-26 08:12:17,651] [ WARNING]: n undefined in the pass@k. We assume it's the same as the sample's number of predictions. (metrics_sample.py:1302)
[2025-10-26 08:12:17,807] [    INFO]: --- DISPLAYING RESULTS --- (pipeline.py:432)
[2025-10-26 08:12:17,817] [    INFO]: --- SAVING AND PUSHING RESULTS --- (pipeline.py:422)
[2025-10-26 08:12:17,818] [    INFO]: Saving experiment tracker (evaluation_tracker.py:246)
[2025-10-26 08:12:18,285] [    INFO]: Saving results to /mnt/public/wucanhui/lighteval/results/results/mnt/public/wucanhui/outputs/Qwen3-4B-math-reasoning/checkpoint-2562/results_2025-10-26T08-12-17.819223.json (evaluation_tracker.py:310)
|          Task          |Version|      Metric      |Value |   |Stderr|
|------------------------|-------|------------------|-----:|---|-----:|
|all                     |       |gpqa_pass@k_with_k|0.3889|±  |0.0347|
|lighteval:gpqa:diamond:0|       |gpqa_pass@k_with_k|0.3889|±  |0.0347|