| INFO 10-26 08:02:52 [__init__.py:235] Automatically detected platform cuda. | |
| [2025-10-26 08:02:53,782] [[32m INFO[0m]: --- INIT SEEDS --- (pipeline.py:249)[0m | |
| [2025-10-26 08:02:53,783] [[32m INFO[0m]: --- LOADING TASKS --- (pipeline.py:210)[0m | |
| [2025-10-26 08:02:53,786] [[33m WARNING[0m]: Careful, the task aime24 is using evaluation data to build the few shot examples. (lighteval_task.py:269)[0m | |
| [2025-10-26 08:02:58,782] [[32m INFO[0m]: --- LOADING MODEL --- (pipeline.py:177)[0m | |
| `torch_dtype` is deprecated! Use `dtype` instead! | |
| [2025-10-26 08:03:06,003] [[32m INFO[0m]: Using max model len 32768 (config.py:1604)[0m | |
| [2025-10-26 08:03:06,707] [[32m INFO[0m]: Chunked prefill is enabled with max_num_batched_tokens=2048. (config.py:2434)[0m | |
| INFO 10-26 08:03:11 [__init__.py:235] Automatically detected platform cuda. | |
| INFO 10-26 08:03:13 [core.py:572] Waiting for init message from front-end. | |
| INFO 10-26 08:03:13 [core.py:71] Initializing a V1 LLM engine (v0.10.0) with config: model='/mnt/public/wucanhui/outputs/Qwen3-4B-math-reasoning/checkpoint-2562', speculative_config=None, tokenizer='/mnt/public/wucanhui/outputs/Qwen3-4B-math-reasoning/checkpoint-2562', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, override_neuron_config={}, tokenizer_revision=main, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=1234, served_model_name=/mnt/public/wucanhui/outputs/Qwen3-4B-math-reasoning/checkpoint-2562, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":0,"local_cache_dir":null} | |
| INFO 10-26 08:03:17 [parallel_state.py:1102] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0 | |
| WARNING 10-26 08:03:17 [topk_topp_sampler.py:59] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer. | |
| INFO 10-26 08:03:17 [gpu_model_runner.py:1843] Starting to load model /mnt/public/wucanhui/outputs/Qwen3-4B-math-reasoning/checkpoint-2562... | |
| INFO 10-26 08:03:18 [gpu_model_runner.py:1875] Loading model from scratch... | |
| INFO 10-26 08:03:18 [cuda.py:290] Using Flash Attention backend on V1 engine. | |
| Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:31<00:31, 31.34s/it] | |
| Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:52<00:00, 25.45s/it] | |
| Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:52<00:00, 26.34s/it] | |
| INFO 10-26 08:04:11 [default_loader.py:262] Loading weights took 53.10 seconds | |
| INFO 10-26 08:04:11 [gpu_model_runner.py:1892] Model loading took 7.5552 GiB and 53.217135 seconds | |
| INFO 10-26 08:04:12 [gpu_worker.py:255] Available KV cache memory: 117.60 GiB | |
| INFO 10-26 08:04:12 [kv_cache_utils.py:833] GPU KV cache size: 856,336 tokens | |
| INFO 10-26 08:04:12 [kv_cache_utils.py:837] Maximum concurrency for 32,768 tokens per request: 26.13x | |
| INFO 10-26 08:04:13 [core.py:193] init engine (profile, create kv cache, warmup model) took 1.35 seconds | |
| [2025-10-26 08:04:13,688] [[32m INFO[0m]: [CACHING] Initializing data cache (cache_management.py:105)[0m | |
| [2025-10-26 08:04:13,695] [[32m INFO[0m]: --- RUNNING MODEL --- (pipeline.py:330)[0m | |
| [2025-10-26 08:04:13,696] [[32m INFO[0m]: Running SamplingMethod.GENERATIVE requests (pipeline.py:313)[0m | |
| [2025-10-26 08:04:14,670] [[32m INFO[0m]: Cache: Starting to process 30/30 samples (not found in cache) for tasks lighteval|aime24|0 (12234f074d9327ac, GENERATIVE) (cache_management.py:399)[0m | |
| [2025-10-26 08:04:14,671] [[33m WARNING[0m]: You cannot select the number of dataset splits for a generative evaluation at the moment. Automatically inferring. (data.py:206)[0m | |
| Splits: 0%| | 0/1 [00:00<?, ?it/s][2025-10-26 08:04:14,701] [[33m WARNING[0m]: context_size + max_new_tokens=33247 which is greater than self.max_length=32768. Truncating context to 0 tokens. (vllm_model.py:367)[0m | |
| Adding requests: 0%| | 0/30 [00:00<?, ?it/s][A Adding requests: 100%|██████████| 30/30 [00:00<00:00, 2828.89it/s] | |
| Processed prompts: 0%| | 0/480 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s][A | |
| Processed prompts: 3%|▎ | 16/480 [00:19<09:23, 1.21s/it, est. speed input: 173.05 toks/s, output: 842.39 toks/s][A | |
| Processed prompts: 7%|▋ | 32/480 [02:10<34:21, 4.60s/it, est. speed input: 52.41 toks/s, output: 303.21 toks/s] [A | |
| Processed prompts: 10%|█ | 48/480 [02:35<23:02, 3.20s/it, est. speed input: 67.72 toks/s, output: 431.83 toks/s][A | |
| Processed prompts: 13%|█▎ | 64/480 [06:48<56:43, 8.18s/it, est. speed input: 36.14 toks/s, output: 414.29 toks/s][A | |
| Processed prompts: 17%|█▋ | 80/480 [11:58<1:21:30, 12.23s/it, est. speed input: 26.84 toks/s, output: 518.27 toks/s][A | |
| Processed prompts: 20%|██ | 96/480 [19:04<1:49:32, 17.12s/it, est. speed input: 23.55 toks/s, output: 586.62 toks/s][A | |
| Processed prompts: 23%|██▎ | 112/480 [19:16<1:12:09, 11.76s/it, est. speed input: 26.59 toks/s, output: 800.47 toks/s][A | |
| Processed prompts: 27%|██▋ | 128/480 [19:16<47:04, 8.03s/it, est. speed input: 29.80 toks/s, output: 1216.72 toks/s] [A | |
| Processed prompts: 30%|███ | 144/480 [19:19<31:12, 5.57s/it, est. speed input: 32.27 toks/s, output: 1322.60 toks/s][A | |
| Processed prompts: 30%|███ | 144/480 [19:30<31:12, 5.57s/it, est. speed input: 32.27 toks/s, output: 1322.60 toks/s][A | |
| Processed prompts: 33%|███▎ | 160/480 [19:52<23:53, 4.48s/it, est. speed input: 33.90 toks/s, output: 1406.40 toks/s][A | |
| Processed prompts: 37%|███▋ | 176/480 [19:55<16:00, 3.16s/it, est. speed input: 36.47 toks/s, output: 1508.94 toks/s][A | |
| Processed prompts: 37%|███▋ | 176/480 [20:12<16:00, 3.16s/it, est. speed input: 36.47 toks/s, output: 1508.94 toks/s][A | |
| Processed prompts: 40%|████ | 192/480 [20:23<13:10, 2.74s/it, est. speed input: 38.56 toks/s, output: 1605.36 toks/s][A | |
| Processed prompts: 43%|████▎ | 208/480 [21:56<16:40, 3.68s/it, est. speed input: 37.53 toks/s, output: 1508.98 toks/s][A | |
| Processed prompts: 47%|████▋ | 224/480 [25:14<26:51, 6.30s/it, est. speed input: 34.53 toks/s, output: 1477.46 toks/s][A | |
| Processed prompts: 50%|█████ | 240/480 [25:18<17:55, 4.48s/it, est. speed input: 35.98 toks/s, output: 1513.08 toks/s][A | |
| Processed prompts: 50%|█████ | 240/480 [25:34<17:55, 4.48s/it, est. speed input: 35.98 toks/s, output: 1513.08 toks/s][A | |
| Processed prompts: 53%|█████▎ | 256/480 [27:19<20:10, 5.40s/it, est. speed input: 34.98 toks/s, output: 1447.90 toks/s][A | |
| Processed prompts: 57%|█████▋ | 272/480 [28:37<18:11, 5.25s/it, est. speed input: 35.09 toks/s, output: 1500.56 toks/s][A | |
| Processed prompts: 60%|██████ | 288/480 [28:40<11:56, 3.73s/it, est. speed input: 36.28 toks/s, output: 1511.14 toks/s][A | |
| Processed prompts: 63%|██████▎ | 304/480 [29:22<09:57, 3.39s/it, est. speed input: 36.93 toks/s, output: 1538.79 toks/s][A | |
| Processed prompts: 67%|██████▋ | 320/480 [29:29<06:39, 2.50s/it, est. speed input: 38.27 toks/s, output: 1599.71 toks/s][A | |
| Processed prompts: 70%|███████ | 336/480 [35:40<20:53, 8.71s/it, est. speed input: 32.76 toks/s, output: 1412.11 toks/s][A | |
| Processed prompts: 73%|███████▎ | 352/480 [36:02<13:52, 6.51s/it, est. speed input: 33.63 toks/s, output: 1527.18 toks/s][A | |
| Processed prompts: 77%|███████▋ | 368/480 [37:35<11:46, 6.31s/it, est. speed input: 33.59 toks/s, output: 1596.24 toks/s][A | |
| Processed prompts: 80%|████████ | 384/480 [37:39<07:11, 4.49s/it, est. speed input: 34.75 toks/s, output: 1658.85 toks/s][A | |
| Processed prompts: 83%|████████▎ | 400/480 [38:45<05:50, 4.38s/it, est. speed input: 34.84 toks/s, output: 1784.84 toks/s][A | |
| Processed prompts: 87%|████████▋ | 416/480 [39:05<03:40, 3.44s/it, est. speed input: 35.55 toks/s, output: 1890.39 toks/s][A | |
| Processed prompts: 90%|█████████ | 432/480 [40:29<03:10, 3.98s/it, est. speed input: 35.40 toks/s, output: 1928.69 toks/s][A | |
| Processed prompts: 93%|█████████▎| 448/480 [40:34<01:32, 2.89s/it, est. speed input: 36.23 toks/s, output: 1998.74 toks/s][A | |
| Processed prompts: 97%|█████████▋| 464/480 [42:23<01:04, 4.05s/it, est. speed input: 35.60 toks/s, output: 1961.46 toks/s][A | |
| Processed prompts: 100%|██████████| 480/480 [43:03<00:00, 3.60s/it, est. speed input: 35.86 toks/s, output: 1958.48 toks/s][A | |
| Processed prompts: 100%|██████████| 480/480 [43:03<00:00, 3.60s/it, est. speed input: 35.86 toks/s, output: 1958.48 toks/s][A Processed prompts: 100%|██████████| 480/480 [43:04<00:00, 5.38s/it, est. speed input: 35.86 toks/s, output: 1958.48 toks/s] | |
| Splits: 100%|██████████| 1/1 [43:04<00:00, 2584.05s/it] Splits: 100%|██████████| 1/1 [43:04<00:00, 2584.05s/it] | |
| Creating parquet from Arrow format: 0%| | 0/1 [00:00<?, ?ba/s] Creating parquet from Arrow format: 100%|██████████| 1/1 [00:00<00:00, 4.79ba/s] Creating parquet from Arrow format: 100%|██████████| 1/1 [00:00<00:00, 4.77ba/s] | |
| [2025-10-26 08:47:25,202] [[32m INFO[0m]: Cached 30 samples of lighteval|aime24|0 (12234f074d9327ac, GENERATIVE) at /mnt/public/wucanhui/outputs/Qwen3-4B-math-reasoning/checkpoint-2562/0619260e1176b049/lighteval|aime24|0/12234f074d9327ac/GENERATIVE.parquet. (cache_management.py:345)[0m | |
| Generating train split: 0 examples [00:00, ? examples/s] Generating train split: 30 examples [00:00, 190.79 examples/s] Generating train split: 30 examples [00:00, 188.72 examples/s] | |
| [rank0]:[W1026 08:47:30.553971068 ProcessGroupNCCL.cpp:1479] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) | |
| [2025-10-26 08:47:31,519] [[32m INFO[0m]: --- POST-PROCESSING MODEL RESPONSES --- (pipeline.py:344)[0m | |
| [2025-10-26 08:47:31,531] [[32m INFO[0m]: --- COMPUTING METRICS --- (pipeline.py:371)[0m | |
| [2025-10-26 08:47:31,532] [[33m WARNING[0m]: n undefined in the pass@k. We assume it's the same as the sample's number of predictions. (metrics_sample.py:1302)[0m | |
| [2025-10-26 08:47:36,019] [[32m INFO[0m]: --- DISPLAYING RESULTS --- (pipeline.py:432)[0m | |
| [2025-10-26 08:47:36,036] [[32m INFO[0m]: --- SAVING AND PUSHING RESULTS --- (pipeline.py:422)[0m | |
| [2025-10-26 08:47:36,037] [[32m INFO[0m]: Saving experiment tracker (evaluation_tracker.py:246)[0m | |
| [2025-10-26 08:47:38,929] [[32m INFO[0m]: Saving results to /mnt/public/wucanhui/lighteval/results/results/mnt/public/wucanhui/outputs/Qwen3-4B-math-reasoning/checkpoint-2562/results_2025-10-26T08-47-36.038278.json (evaluation_tracker.py:310)[0m | |
| | Task |Version| Metric |Value | |Stderr| | |
| |------------------|-------|-------------|-----:|---|-----:| | |
| |all | |pass@k_with_k|0.7333|± |0.0821| | |
| | | |avg@k_with_k |0.3604|± |0.0684| | |
| |lighteval:aime24:0| |pass@k_with_k|0.7333|± |0.0821| | |
| | | |avg@k_with_k |0.3604|± |0.0684| | |