Buckets:

Mercity
/

Desicion-Model-Custom

Files

xet

Mercity/Desicion-Model-Custom / run3_2b /logs /v2b_eval_base.log

Pranav2748

8 days ago

download

raw

18.7 kB

(MainProcess pid=152146) INFO 11-01 02:15:57 [model.py:733] Resolved architecture: LlamaForCausalLM
(MainProcess pid=152146) INFO 11-01 02:15:57 [model.py:2152] Using max model len 32768
(MainProcess pid=152146) INFO 11-01 02:15:57 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=2048.
(MainProcess pid=152146) INFO 11-01 02:15:57 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])

Fetching 10 files:   0%|          | 0/10 [00:00<?, ?it/s]
Fetching 10 files: 100%|██████████| 10/10 [00:00<00:00, 992.34it/s]
(EngineCore pid=152420) INFO 11-01 02:16:00 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='openbmb/MiniCPM5-2B', speculative_config=None, tokenizer='openbmb/MiniCPM5-2B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, tokenizer_revision=main, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=openbmb/MiniCPM5-2B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None)
(EngineCore pid=152420) INFO 11-01 02:16:01 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_9f6fbeb7f44b4f5cb90eb8ffc540bf4b backend=nccl
(EngineCore pid=152420) INFO 11-01 02:16:01 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=152420) INFO 11-01 02:16:01 [gpu_worker.py:518] Using V2 Model Runner
(EngineCore pid=152420) INFO 11-01 02:16:02 [model_runner.py:392] Loading model from scratch...
(EngineCore pid=152420) INFO 11-01 02:16:03 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=152420) INFO 11-01 02:16:03 [flash_attn.py:1117] Using FlashAttention version 2
(EngineCore pid=152420) INFO 11-01 02:16:06 [base_loader.py:82] Initializing model took 3.765978 seconds
(EngineCore pid=152420) INFO 11-01 02:16:06 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 4.69 GiB. Available RAM: 129.82 GiB.
(EngineCore pid=152420) INFO 11-01 02:16:06 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.

Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]

Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.71it/s]

Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.71it/s]

(EngineCore pid=152420) INFO 11-01 02:16:07 [default_loader.py:501] Loading weights took 0.59 seconds
(EngineCore pid=152420) INFO 11-01 02:16:08 [model_runner.py:424] Model loading took 4.75 GiB memory and 6.755927 seconds
(EngineCore pid=152420) INFO 11-01 02:16:09 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=152420) INFO 11-01 02:16:09 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override.
(EngineCore pid=152420) INFO 11-01 02:16:09 [utils.py:347] Using LBNHC KV cache layout.
(EngineCore pid=152420) INFO 11-01 02:16:09 [monitor.py:40] [backbone] Starting torch.compile
(EngineCore pid=152420) INFO 11-01 02:16:18 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/4ae45673a2/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=152420) INFO 11-01 02:16:18 [backends.py:1166] [backbone] Dynamo bytecode transform time: 8.98 s
(EngineCore pid=152420) INFO 11-01 02:16:26 [backends.py:398] [backbone] Compiling a graph for compile range (1, 2048) takes 7.47 s
(EngineCore pid=152420) INFO 11-01 02:16:32 [backends.py:927] collected artifacts: 43 entries, 3 artifacts, 4889897 bytes total
(EngineCore pid=152420) INFO 11-01 02:16:32 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/8af31eb29db492efb42d43f0b7d6f9e21786afa38ef612607b7e13dce12a0d17/rank_0_0/model
(EngineCore pid=152420) INFO 11-01 02:16:32 [monitor.py:62] [backbone] torch.compile took 22.15 s in total
(EngineCore pid=152420) INFO 11-01 02:16:32 [monitor.py:91] [backbone] Initial profiling/warmup run took 0.20 s

Capturing CUDA graphs (PIECEWISE):   0%|          | 0/51 [00:00<?, ?it/s]
Capturing CUDA graphs (PIECEWISE):   4%|▍         | 2/51 [00:00<00:03, 12.49it/s]
Capturing CUDA graphs (PIECEWISE):   8%|▊         | 4/51 [00:00<00:03, 12.78it/s]
Capturing CUDA graphs (PIECEWISE):  12%|█▏        | 6/51 [00:00<00:03, 13.49it/s]
Capturing CUDA graphs (PIECEWISE):  16%|█▌        | 8/51 [00:00<00:03, 13.71it/s]
Capturing CUDA graphs (PIECEWISE):  20%|█▉        | 10/51 [00:00<00:03, 13.63it/s]
Capturing CUDA graphs (PIECEWISE):  24%|██▎       | 12/51 [00:00<00:02, 13.95it/s]
Capturing CUDA graphs (PIECEWISE):  27%|██▋       | 14/51 [00:01<00:02, 13.90it/s]
Capturing CUDA graphs (PIECEWISE):  31%|███▏      | 16/51 [00:01<00:02, 14.18it/s]
Capturing CUDA graphs (PIECEWISE):  35%|███▌      | 18/51 [00:01<00:02, 12.96it/s]
Capturing CUDA graphs (PIECEWISE):  39%|███▉      | 20/51 [00:01<00:02, 12.88it/s]
Capturing CUDA graphs (PIECEWISE):  43%|████▎     | 22/51 [00:01<00:02, 13.13it/s]
Capturing CUDA graphs (PIECEWISE):  47%|████▋     | 24/51 [00:01<00:02, 13.10it/s]
Capturing CUDA graphs (PIECEWISE):  51%|█████     | 26/51 [00:01<00:01, 13.39it/s]
Capturing CUDA graphs (PIECEWISE):  55%|█████▍    | 28/51 [00:02<00:01, 13.53it/s]
Capturing CUDA graphs (PIECEWISE):  59%|█████▉    | 30/51 [00:02<00:01, 13.38it/s]
Capturing CUDA graphs (PIECEWISE):  63%|██████▎   | 32/51 [00:02<00:01, 13.53it/s]
Capturing CUDA graphs (PIECEWISE):  67%|██████▋   | 34/51 [00:02<00:01, 13.35it/s]
Capturing CUDA graphs (PIECEWISE):  71%|███████   | 36/51 [00:02<00:01, 13.62it/s]
Capturing CUDA graphs (PIECEWISE):  75%|███████▍  | 38/51 [00:02<00:00, 13.50it/s]
Capturing CUDA graphs (PIECEWISE):  78%|███████▊  | 40/51 [00:03<00:00, 12.62it/s]
Capturing CUDA graphs (PIECEWISE):  82%|████████▏ | 42/51 [00:03<00:00, 13.11it/s]
Capturing CUDA graphs (PIECEWISE):  86%|████████▋ | 44/51 [00:03<00:00, 13.50it/s]
Capturing CUDA graphs (PIECEWISE):  90%|█████████ | 46/51 [00:03<00:00, 13.62it/s]
Capturing CUDA graphs (PIECEWISE):  94%|█████████▍| 48/51 [00:03<00:00, 13.12it/s]
Capturing CUDA graphs (PIECEWISE):  98%|█████████▊| 50/51 [00:03<00:00, 13.30it/s]
Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:03<00:00, 13.27it/s]

Capturing CUDA graphs (FULL):   0%|          | 0/2 [00:00<?, ?it/s]
Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 23.17it/s]
(EngineCore pid=152420) INFO 11-01 02:16:39 [model_runner.py:1127] Graph capturing finished in 5 secs, took 0.40 GiB
(EngineCore pid=152420) INFO 11-01 02:16:39 [gpu_worker.py:759] Available KV cache memory: 65.26 GiB
(EngineCore pid=152420) INFO 11-01 02:16:39 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9000 is equivalent to --gpu-memory-utilization=0.8943 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9057. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=152420) INFO 11-01 02:16:39 [kv_cache_utils.py:2327] GPU KV cache size: 1,629,360 tokens, Maximum concurrency for 32,768 tokens per request: 49.72x
(EngineCore pid=152420) INFO 11-01 02:16:39 [kernel_warmup.py:176] JIT kernel warmup starting.
(EngineCore pid=152420) INFO 11-01 02:16:39 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s.

Capturing CUDA graphs (PIECEWISE):   0%|          | 0/51 [00:00<?, ?it/s]
Capturing CUDA graphs (PIECEWISE):   4%|▍         | 2/51 [00:00<00:04, 12.02it/s]
Capturing CUDA graphs (PIECEWISE):   8%|▊         | 4/51 [00:00<00:03, 13.16it/s]
Capturing CUDA graphs (PIECEWISE):  12%|█▏        | 6/51 [00:00<00:03, 13.83it/s]
Capturing CUDA graphs (PIECEWISE):  16%|█▌        | 8/51 [00:00<00:03, 14.06it/s]
Capturing CUDA graphs (PIECEWISE):  20%|█▉        | 10/51 [00:00<00:02, 14.47it/s]
Capturing CUDA graphs (PIECEWISE):  24%|██▎       | 12/51 [00:00<00:02, 14.86it/s]
Capturing CUDA graphs (PIECEWISE):  27%|██▋       | 14/51 [00:00<00:02, 15.13it/s]
Capturing CUDA graphs (PIECEWISE):  31%|███▏      | 16/51 [00:01<00:02, 15.35it/s]
Capturing CUDA graphs (PIECEWISE):  35%|███▌      | 18/51 [00:01<00:02, 14.47it/s]
Capturing CUDA graphs (PIECEWISE):  39%|███▉      | 20/51 [00:01<00:02, 14.22it/s]
Capturing CUDA graphs (PIECEWISE):  43%|████▎     | 22/51 [00:01<00:02, 14.34it/s]
Capturing CUDA graphs (PIECEWISE):  47%|████▋     | 24/51 [00:01<00:01, 14.16it/s]
Capturing CUDA graphs (PIECEWISE):  51%|█████     | 26/51 [00:01<00:01, 14.20it/s]
Capturing CUDA graphs (PIECEWISE):  55%|█████▍    | 28/51 [00:01<00:01, 14.18it/s]
Capturing CUDA graphs (PIECEWISE):  59%|█████▉    | 30/51 [00:02<00:01, 14.18it/s]
Capturing CUDA graphs (PIECEWISE):  63%|██████▎   | 32/51 [00:02<00:01, 14.19it/s]
Capturing CUDA graphs (PIECEWISE):  67%|██████▋   | 34/51 [00:02<00:01, 14.24it/s]
Capturing CUDA graphs (PIECEWISE):  71%|███████   | 36/51 [00:02<00:01, 14.24it/s]
Capturing CUDA graphs (PIECEWISE):  75%|███████▍  | 38/51 [00:02<00:00, 14.26it/s]
Capturing CUDA graphs (PIECEWISE):  78%|███████▊  | 40/51 [00:02<00:00, 14.20it/s]
Capturing CUDA graphs (PIECEWISE):  82%|████████▏ | 42/51 [00:02<00:00, 14.25it/s]
Capturing CUDA graphs (PIECEWISE):  86%|████████▋ | 44/51 [00:03<00:00, 14.33it/s]
Capturing CUDA graphs (PIECEWISE):  90%|█████████ | 46/51 [00:03<00:00, 14.37it/s]
Capturing CUDA graphs (PIECEWISE):  94%|█████████▍| 48/51 [00:03<00:00, 14.34it/s]
Capturing CUDA graphs (PIECEWISE):  98%|█████████▊| 50/51 [00:03<00:00, 14.26it/s]
Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:03<00:00, 14.29it/s]

Capturing CUDA graphs (FULL):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (FULL):   9%|▊         | 3/35 [00:00<00:01, 24.49it/s]
Capturing CUDA graphs (FULL):  17%|█▋        | 6/35 [00:00<00:01, 24.04it/s]
Capturing CUDA graphs (FULL):  26%|██▌       | 9/35 [00:00<00:01, 24.13it/s]
Capturing CUDA graphs (FULL):  34%|███▍      | 12/35 [00:00<00:00, 24.35it/s]
Capturing CUDA graphs (FULL):  43%|████▎     | 15/35 [00:00<00:00, 24.52it/s]
Capturing CUDA graphs (FULL):  51%|█████▏    | 18/35 [00:00<00:00, 24.75it/s]
Capturing CUDA graphs (FULL):  60%|██████    | 21/35 [00:00<00:00, 24.73it/s]
Capturing CUDA graphs (FULL):  69%|██████▊   | 24/35 [00:00<00:00, 24.43it/s]
Capturing CUDA graphs (FULL):  77%|███████▋  | 27/35 [00:01<00:00, 24.79it/s]
Capturing CUDA graphs (FULL):  86%|████████▌ | 30/35 [00:01<00:00, 25.19it/s]
Capturing CUDA graphs (FULL):  94%|█████████▍| 33/35 [00:01<00:00, 24.95it/s]
Capturing CUDA graphs (FULL): 100%|██████████| 35/35 [00:01<00:00, 24.74it/s]
(EngineCore pid=152420) INFO 11-01 02:17:04 [model_runner.py:1127] Graph capturing finished in 6 secs, took 0.23 GiB
(EngineCore pid=152420) INFO 11-01 02:17:04 [gpu_worker.py:971] CUDA graph pool memory: 0.23 GiB (actual), 0.46 GiB (estimated), difference: 0.23 GiB (100.9%).
(EngineCore pid=152420) INFO 11-01 02:17:04 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.9, 71.33 GiB). Actual usage is 5.1 GiB for consumed memory (weights + non-torch), 0.96 GiB for peak activation, and 0.23 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=69675340698` (64.89 GiB) to fit into requested memory, or `--kv-cache-memory=77741406720` (72.4 GiB) to fully utilize gpu memory. Current kv cache memory in use is 65.26 GiB.
(EngineCore pid=152420) INFO 11-01 02:17:04 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(EngineCore pid=152420) INFO 11-01 02:17:05 [core.py:395] init engine (profile, create kv cache, warmup model) took 56.14 s (compilation: 22.15 s)
(EngineCore pid=152420) INFO 11-01 02:17:05 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(MainProcess pid=152146) WARNING 11-01 02:17:06 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
(EngineCore pid=152420) WARNING 11-01 02:17:06 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
readout base2b: 5957 items in 47s
(MainProcess pid=152146) INFO 11-01 02:17:52 [core_client.py:770] [shutdown] MPClient: start timeout=default
(MainProcess pid=152146) INFO 11-01 02:17:52 [core_client.py:772] [shutdown] MPClient: stopping engine manager
(MainProcess pid=152146) INFO 11-01 02:17:52 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore
(EngineCore pid=152420) INFO 11-01 02:17:52 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=152420) INFO 11-01 02:17:52 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s
(EngineCore pid=152420) INFO 11-01 02:17:52 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=152420) INFO 11-01 02:17:52 [core.py:1499] [shutdown] EngineCore: exiting busy loop
(MainProcess pid=152146) INFO 11-01 02:17:57 [core_client.py:774] [shutdown] MPClient: engine manager stopped
(MainProcess pid=152146) INFO 11-01 02:17:57 [core_client.py:775] [shutdown] MPClient: cleaning up background resources
(MainProcess pid=152146) INFO 11-01 02:17:57 [core_client.py:777] [shutdown] MPClient: complete

Xet Storage Details

Size:: 18.7 kB
Xet hash:: ed079c6616c70b511c6546225b7c2f1167e5d713099eb7373e81f601d8b5b8be

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	(MainProcess pid=152146) INFO 11-01 02:15:57 [model.py:733] Resolved architecture: LlamaForCausalLM
	(MainProcess pid=152146) INFO 11-01 02:15:57 [model.py:2152] Using max model len 32768
	(MainProcess pid=152146) INFO 11-01 02:15:57 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=2048.
	(MainProcess pid=152146) INFO 11-01 02:15:57 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
	Fetching 10 files: 0%\| \| 0/10 [00:00<?, ?it/s] Fetching 10 files: 100%\|██████████\| 10/10 [00:00<00:00, 992.34it/s]
	(EngineCore pid=152420) INFO 11-01 02:16:00 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='openbmb/MiniCPM5-2B', speculative_config=None, tokenizer='openbmb/MiniCPM5-2B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, tokenizer_revision=main, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=openbmb/MiniCPM5-2B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None)
	(EngineCore pid=152420) INFO 11-01 02:16:01 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_9f6fbeb7f44b4f5cb90eb8ffc540bf4b backend=nccl
	(EngineCore pid=152420) INFO 11-01 02:16:01 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A
	(EngineCore pid=152420) INFO 11-01 02:16:01 [gpu_worker.py:518] Using V2 Model Runner
	(EngineCore pid=152420) INFO 11-01 02:16:02 [model_runner.py:392] Loading model from scratch...
	(EngineCore pid=152420) INFO 11-01 02:16:03 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
	(EngineCore pid=152420) INFO 11-01 02:16:03 [flash_attn.py:1117] Using FlashAttention version 2
	(EngineCore pid=152420) INFO 11-01 02:16:06 [base_loader.py:82] Initializing model took 3.765978 seconds
	(EngineCore pid=152420) INFO 11-01 02:16:06 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 4.69 GiB. Available RAM: 129.82 GiB.
	(EngineCore pid=152420) INFO 11-01 02:16:06 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
	Loading safetensors checkpoint shards: 0% Completed \| 0/1 [00:00<?, ?it/s]
	Loading safetensors checkpoint shards: 100% Completed \| 1/1 [00:00<00:00, 1.71it/s]
	Loading safetensors checkpoint shards: 100% Completed \| 1/1 [00:00<00:00, 1.71it/s]

	(EngineCore pid=152420) INFO 11-01 02:16:07 [default_loader.py:501] Loading weights took 0.59 seconds
	(EngineCore pid=152420) INFO 11-01 02:16:08 [model_runner.py:424] Model loading took 4.75 GiB memory and 6.755927 seconds
	(EngineCore pid=152420) INFO 11-01 02:16:09 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling.
	(EngineCore pid=152420) INFO 11-01 02:16:09 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override.
	(EngineCore pid=152420) INFO 11-01 02:16:09 [utils.py:347] Using LBNHC KV cache layout.
	(EngineCore pid=152420) INFO 11-01 02:16:09 [monitor.py:40] [backbone] Starting torch.compile
	(EngineCore pid=152420) INFO 11-01 02:16:18 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/4ae45673a2/rank_0_0/backbone for vLLM's torch.compile
	(EngineCore pid=152420) INFO 11-01 02:16:18 [backends.py:1166] [backbone] Dynamo bytecode transform time: 8.98 s
	(EngineCore pid=152420) INFO 11-01 02:16:26 [backends.py:398] [backbone] Compiling a graph for compile range (1, 2048) takes 7.47 s
	(EngineCore pid=152420) INFO 11-01 02:16:32 [backends.py:927] collected artifacts: 43 entries, 3 artifacts, 4889897 bytes total
	(EngineCore pid=152420) INFO 11-01 02:16:32 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/8af31eb29db492efb42d43f0b7d6f9e21786afa38ef612607b7e13dce12a0d17/rank_0_0/model
	(EngineCore pid=152420) INFO 11-01 02:16:32 [monitor.py:62] [backbone] torch.compile took 22.15 s in total
	(EngineCore pid=152420) INFO 11-01 02:16:32 [monitor.py:91] [backbone] Initial profiling/warmup run took 0.20 s
	Capturing CUDA graphs (PIECEWISE): 0%\| \| 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 4%\|▍ \| 2/51 [00:00<00:03, 12.49it/s] Capturing CUDA graphs (PIECEWISE): 8%\|▊ \| 4/51 [00:00<00:03, 12.78it/s] Capturing CUDA graphs (PIECEWISE): 12%\|█▏ \| 6/51 [00:00<00:03, 13.49it/s] Capturing CUDA graphs (PIECEWISE): 16%\|█▌ \| 8/51 [00:00<00:03, 13.71it/s] Capturing CUDA graphs (PIECEWISE): 20%\|█▉ \| 10/51 [00:00<00:03, 13.63it/s] Capturing CUDA graphs (PIECEWISE): 24%\|██▎ \| 12/51 [00:00<00:02, 13.95it/s] Capturing CUDA graphs (PIECEWISE): 27%\|██▋ \| 14/51 [00:01<00:02, 13.90it/s] Capturing CUDA graphs (PIECEWISE): 31%\|███▏ \| 16/51 [00:01<00:02, 14.18it/s] Capturing CUDA graphs (PIECEWISE): 35%\|███▌ \| 18/51 [00:01<00:02, 12.96it/s] Capturing CUDA graphs (PIECEWISE): 39%\|███▉ \| 20/51 [00:01<00:02, 12.88it/s] Capturing CUDA graphs (PIECEWISE): 43%\|████▎ \| 22/51 [00:01<00:02, 13.13it/s] Capturing CUDA graphs (PIECEWISE): 47%\|████▋ \| 24/51 [00:01<00:02, 13.10it/s] Capturing CUDA graphs (PIECEWISE): 51%\|█████ \| 26/51 [00:01<00:01, 13.39it/s] Capturing CUDA graphs (PIECEWISE): 55%\|█████▍ \| 28/51 [00:02<00:01, 13.53it/s] Capturing CUDA graphs (PIECEWISE): 59%\|█████▉ \| 30/51 [00:02<00:01, 13.38it/s] Capturing CUDA graphs (PIECEWISE): 63%\|██████▎ \| 32/51 [00:02<00:01, 13.53it/s] Capturing CUDA graphs (PIECEWISE): 67%\|██████▋ \| 34/51 [00:02<00:01, 13.35it/s] Capturing CUDA graphs (PIECEWISE): 71%\|███████ \| 36/51 [00:02<00:01, 13.62it/s] Capturing CUDA graphs (PIECEWISE): 75%\|███████▍ \| 38/51 [00:02<00:00, 13.50it/s] Capturing CUDA graphs (PIECEWISE): 78%\|███████▊ \| 40/51 [00:03<00:00, 12.62it/s] Capturing CUDA graphs (PIECEWISE): 82%\|████████▏ \| 42/51 [00:03<00:00, 13.11it/s] Capturing CUDA graphs (PIECEWISE): 86%\|████████▋ \| 44/51 [00:03<00:00, 13.50it/s] Capturing CUDA graphs (PIECEWISE): 90%\|█████████ \| 46/51 [00:03<00:00, 13.62it/s] Capturing CUDA graphs (PIECEWISE): 94%\|█████████▍\| 48/51 [00:03<00:00, 13.12it/s] Capturing CUDA graphs (PIECEWISE): 98%\|█████████▊\| 50/51 [00:03<00:00, 13.30it/s] Capturing CUDA graphs (PIECEWISE): 100%\|██████████\| 51/51 [00:03<00:00, 13.27it/s]
	Capturing CUDA graphs (FULL): 0%\| \| 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%\|██████████\| 2/2 [00:00<00:00, 23.17it/s]
	(EngineCore pid=152420) INFO 11-01 02:16:39 [model_runner.py:1127] Graph capturing finished in 5 secs, took 0.40 GiB
	(EngineCore pid=152420) INFO 11-01 02:16:39 [gpu_worker.py:759] Available KV cache memory: 65.26 GiB
	(EngineCore pid=152420) INFO 11-01 02:16:39 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9000 is equivalent to --gpu-memory-utilization=0.8943 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9057. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
	(EngineCore pid=152420) INFO 11-01 02:16:39 [kv_cache_utils.py:2327] GPU KV cache size: 1,629,360 tokens, Maximum concurrency for 32,768 tokens per request: 49.72x
	(EngineCore pid=152420) INFO 11-01 02:16:39 [kernel_warmup.py:176] JIT kernel warmup starting.
	(EngineCore pid=152420) INFO 11-01 02:16:39 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s.
	Capturing CUDA graphs (PIECEWISE): 0%\| \| 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 4%\|▍ \| 2/51 [00:00<00:04, 12.02it/s] Capturing CUDA graphs (PIECEWISE): 8%\|▊ \| 4/51 [00:00<00:03, 13.16it/s] Capturing CUDA graphs (PIECEWISE): 12%\|█▏ \| 6/51 [00:00<00:03, 13.83it/s] Capturing CUDA graphs (PIECEWISE): 16%\|█▌ \| 8/51 [00:00<00:03, 14.06it/s] Capturing CUDA graphs (PIECEWISE): 20%\|█▉ \| 10/51 [00:00<00:02, 14.47it/s] Capturing CUDA graphs (PIECEWISE): 24%\|██▎ \| 12/51 [00:00<00:02, 14.86it/s] Capturing CUDA graphs (PIECEWISE): 27%\|██▋ \| 14/51 [00:00<00:02, 15.13it/s] Capturing CUDA graphs (PIECEWISE): 31%\|███▏ \| 16/51 [00:01<00:02, 15.35it/s] Capturing CUDA graphs (PIECEWISE): 35%\|███▌ \| 18/51 [00:01<00:02, 14.47it/s] Capturing CUDA graphs (PIECEWISE): 39%\|███▉ \| 20/51 [00:01<00:02, 14.22it/s] Capturing CUDA graphs (PIECEWISE): 43%\|████▎ \| 22/51 [00:01<00:02, 14.34it/s] Capturing CUDA graphs (PIECEWISE): 47%\|████▋ \| 24/51 [00:01<00:01, 14.16it/s] Capturing CUDA graphs (PIECEWISE): 51%\|█████ \| 26/51 [00:01<00:01, 14.20it/s] Capturing CUDA graphs (PIECEWISE): 55%\|█████▍ \| 28/51 [00:01<00:01, 14.18it/s] Capturing CUDA graphs (PIECEWISE): 59%\|█████▉ \| 30/51 [00:02<00:01, 14.18it/s] Capturing CUDA graphs (PIECEWISE): 63%\|██████▎ \| 32/51 [00:02<00:01, 14.19it/s] Capturing CUDA graphs (PIECEWISE): 67%\|██████▋ \| 34/51 [00:02<00:01, 14.24it/s] Capturing CUDA graphs (PIECEWISE): 71%\|███████ \| 36/51 [00:02<00:01, 14.24it/s] Capturing CUDA graphs (PIECEWISE): 75%\|███████▍ \| 38/51 [00:02<00:00, 14.26it/s] Capturing CUDA graphs (PIECEWISE): 78%\|███████▊ \| 40/51 [00:02<00:00, 14.20it/s] Capturing CUDA graphs (PIECEWISE): 82%\|████████▏ \| 42/51 [00:02<00:00, 14.25it/s] Capturing CUDA graphs (PIECEWISE): 86%\|████████▋ \| 44/51 [00:03<00:00, 14.33it/s] Capturing CUDA graphs (PIECEWISE): 90%\|█████████ \| 46/51 [00:03<00:00, 14.37it/s] Capturing CUDA graphs (PIECEWISE): 94%\|█████████▍\| 48/51 [00:03<00:00, 14.34it/s] Capturing CUDA graphs (PIECEWISE): 98%\|█████████▊\| 50/51 [00:03<00:00, 14.26it/s] Capturing CUDA graphs (PIECEWISE): 100%\|██████████\| 51/51 [00:03<00:00, 14.29it/s]
	Capturing CUDA graphs (FULL): 0%\| \| 0/35 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 9%\|▊ \| 3/35 [00:00<00:01, 24.49it/s] Capturing CUDA graphs (FULL): 17%\|█▋ \| 6/35 [00:00<00:01, 24.04it/s] Capturing CUDA graphs (FULL): 26%\|██▌ \| 9/35 [00:00<00:01, 24.13it/s] Capturing CUDA graphs (FULL): 34%\|███▍ \| 12/35 [00:00<00:00, 24.35it/s] Capturing CUDA graphs (FULL): 43%\|████▎ \| 15/35 [00:00<00:00, 24.52it/s] Capturing CUDA graphs (FULL): 51%\|█████▏ \| 18/35 [00:00<00:00, 24.75it/s] Capturing CUDA graphs (FULL): 60%\|██████ \| 21/35 [00:00<00:00, 24.73it/s] Capturing CUDA graphs (FULL): 69%\|██████▊ \| 24/35 [00:00<00:00, 24.43it/s] Capturing CUDA graphs (FULL): 77%\|███████▋ \| 27/35 [00:01<00:00, 24.79it/s] Capturing CUDA graphs (FULL): 86%\|████████▌ \| 30/35 [00:01<00:00, 25.19it/s] Capturing CUDA graphs (FULL): 94%\|█████████▍\| 33/35 [00:01<00:00, 24.95it/s] Capturing CUDA graphs (FULL): 100%\|██████████\| 35/35 [00:01<00:00, 24.74it/s]
	(EngineCore pid=152420) INFO 11-01 02:17:04 [model_runner.py:1127] Graph capturing finished in 6 secs, took 0.23 GiB
	(EngineCore pid=152420) INFO 11-01 02:17:04 [gpu_worker.py:971] CUDA graph pool memory: 0.23 GiB (actual), 0.46 GiB (estimated), difference: 0.23 GiB (100.9%).
	(EngineCore pid=152420) INFO 11-01 02:17:04 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.9, 71.33 GiB). Actual usage is 5.1 GiB for consumed memory (weights + non-torch), 0.96 GiB for peak activation, and 0.23 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=69675340698` (64.89 GiB) to fit into requested memory, or `--kv-cache-memory=77741406720` (72.4 GiB) to fully utilize gpu memory. Current kv cache memory in use is 65.26 GiB.
	(EngineCore pid=152420) INFO 11-01 02:17:04 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
	(EngineCore pid=152420) INFO 11-01 02:17:05 [core.py:395] init engine (profile, create kv cache, warmup model) took 56.14 s (compilation: 22.15 s)
	(EngineCore pid=152420) INFO 11-01 02:17:05 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
	(MainProcess pid=152146) WARNING 11-01 02:17:06 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
	(EngineCore pid=152420) WARNING 11-01 02:17:06 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
	readout base2b: 5957 items in 47s
	(MainProcess pid=152146) INFO 11-01 02:17:52 [core_client.py:770] [shutdown] MPClient: start timeout=default
	(MainProcess pid=152146) INFO 11-01 02:17:52 [core_client.py:772] [shutdown] MPClient: stopping engine manager
	(MainProcess pid=152146) INFO 11-01 02:17:52 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore
	(EngineCore pid=152420) INFO 11-01 02:17:52 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM
	(EngineCore pid=152420) INFO 11-01 02:17:52 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s
	(EngineCore pid=152420) INFO 11-01 02:17:52 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown
	(EngineCore pid=152420) INFO 11-01 02:17:52 [core.py:1499] [shutdown] EngineCore: exiting busy loop
	(MainProcess pid=152146) INFO 11-01 02:17:57 [core_client.py:774] [shutdown] MPClient: engine manager stopped
	(MainProcess pid=152146) INFO 11-01 02:17:57 [core_client.py:775] [shutdown] MPClient: cleaning up background resources
	(MainProcess pid=152146) INFO 11-01 02:17:57 [core_client.py:777] [shutdown] MPClient: complete