Buckets:
| {"selected": 64, "by_source": {"clinc": 10, "snli": 12, "sgd": 4, "synthetic_constraints": 2, "synthetic_ordinal": 16, "boolq": 11, "dbpedia": 7, "goemotions": 2}} | |
| W1106 14:39:10.842000 24337 torch/_opaque_base.py:6] torch._opaque_base is deprecated, use torch._custom_class_base instead | |
| W1106 14:39:10.843000 24337 torch/_library/opaque_object.py:288] register_opaque_type is deprecated, use register_custom_class instead | |
| W1106 14:39:10.844000 24337 torch/_library/opaque_object.py:211] typ='value' is deprecated, use typ='constant' instead | |
| (MainProcess pid=24337) INFO 11-06 14:39:17 [startup.py:96] non-default args: {'dtype': 'bfloat16', 'max_model_len': 8192, 'enable_prefix_caching': True, 'gpu_memory_utilization': 0.45, 'max_num_seqs': 256, 'disable_log_stats': True, 'model': '/root/hf/qwen3.5-9b'} | |
| (MainProcess pid=24337) INFO 11-06 14:39:17 [model.py:744] Resolved architecture: Qwen3_5ForConditionalGeneration | |
| (MainProcess pid=24337) INFO 11-06 14:39:17 [model.py:2170] Using max model len 8192 | |
| (MainProcess pid=24337) INFO 11-06 14:39:20 [scheduler.py:310] Chunked prefill is enabled with max_num_batched_tokens=8192. | |
| (MainProcess pid=24337) INFO 11-06 14:39:20 [config.py:766] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration by default when prefix caching is enabled | |
| (MainProcess pid=24337) INFO 11-06 14:39:20 [kernel.py:480] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=24337) WARNING 11-06 14:39:30 [system_utils.py:154] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized | |
| W1106 14:39:35.327000 24677 torch/_opaque_base.py:6] torch._opaque_base is deprecated, use torch._custom_class_base instead | |
| W1106 14:39:35.328000 24677 torch/_library/opaque_object.py:288] register_opaque_type is deprecated, use register_custom_class instead | |
| W1106 14:39:35.329000 24677 torch/_library/opaque_object.py:211] typ='value' is deprecated, use typ='constant' instead | |
| (EngineCore pid=24677) INFO 11-06 14:39:41 [core.py:139] Initializing a V1 LLM engine (v0.33.0) with config: model='/root/hf/qwen3.5-9b', speculative_config=None, tokenizer='/root/hf/qwen3.5-9b', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=/root/hf/qwen3.5-9b, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, monokernel=False, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None) | |
| (EngineCore pid=24677) INFO 11-06 14:39:43 [parallel_state.py:1929] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_35d9ec12bdf34c37916872a3063f0b78 backend=nccl | |
| (EngineCore pid=24677) INFO 11-06 14:39:43 [parallel_state.py:2385] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A | |
| (EngineCore pid=24677) INFO 11-06 14:39:43 [gpu_worker.py:513] Using V2 Model Runner | |
| (EngineCore pid=24677) INFO 11-06 14:39:45 [model_runner.py:399] Loading model from scratch... | |
| (EngineCore pid=24677) INFO 11-06 14:39:46 [cuda.py:567] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention | |
| (EngineCore pid=24677) INFO 11-06 14:39:46 [mm_encoder_attention.py:372] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention. | |
| (EngineCore pid=24677) INFO 11-06 14:39:46 [qwen_gdn_linear_attn.py:300] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128). | |
| (EngineCore pid=24677) INFO 11-06 14:39:47 [cuda.py:508] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. | |
| (EngineCore pid=24677) INFO 11-06 14:39:47 [flash_attn.py:1122] Using FlashAttention version 2 | |
| (EngineCore pid=24677) INFO 11-06 14:39:47 [base_loader.py:82] Initializing model took 2.218908 seconds | |
| (EngineCore pid=24677) INFO 11-06 14:39:47 [weight_utils.py:1144] Filesystem type for checkpoints: OVERLAY. Checkpoint size: 17.98 GiB. Available RAM: 68.75 GiB. | |
| (EngineCore pid=24677) INFO 11-06 14:39:47 [weight_utils.py:1167] Auto-prefetch is disabled because the filesystem (OVERLAY) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. | |
| Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:01, 1.57it/s] | |
| Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:01<00:01, 1.53it/s] | |
| Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:01<00:00, 1.51it/s] | |
| (EngineCore pid=24677) WARNING 11-06 14:39:49 [utils.py:333] Attempted to load weight layers.0.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=24677) WARNING 11-06 14:39:49 [utils.py:333] Attempted to load weight layers.1.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=24677) WARNING 11-06 14:39:49 [utils.py:333] Attempted to load weight layers.10.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=24677) WARNING 11-06 14:39:50 [utils.py:333] Attempted to load weight layers.12.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=24677) WARNING 11-06 14:39:50 [utils.py:333] Attempted to load weight layers.13.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=24677) WARNING 11-06 14:39:50 [utils.py:333] Attempted to load weight layers.14.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=24677) WARNING 11-06 14:39:50 [utils.py:333] Attempted to load weight layers.16.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=24677) WARNING 11-06 14:39:50 [utils.py:333] Attempted to load weight layers.17.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=24677) WARNING 11-06 14:39:50 [utils.py:333] Attempted to load weight layers.18.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=24677) WARNING 11-06 14:39:50 [utils.py:333] Attempted to load weight layers.2.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=24677) WARNING 11-06 14:39:50 [utils.py:333] Attempted to load weight layers.20.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=24677) WARNING 11-06 14:39:50 [utils.py:333] Attempted to load weight layers.21.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=24677) WARNING 11-06 14:39:50 [utils.py:333] Attempted to load weight layers.22.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=24677) WARNING 11-06 14:39:50 [utils.py:333] Attempted to load weight layers.24.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=24677) WARNING 11-06 14:39:50 [utils.py:333] Attempted to load weight layers.25.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=24677) WARNING 11-06 14:39:50 [utils.py:333] Attempted to load weight layers.26.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=24677) WARNING 11-06 14:39:50 [utils.py:333] Attempted to load weight layers.28.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=24677) WARNING 11-06 14:39:50 [utils.py:333] Attempted to load weight layers.29.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=24677) WARNING 11-06 14:39:50 [utils.py:333] Attempted to load weight layers.30.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=24677) WARNING 11-06 14:39:50 [utils.py:333] Attempted to load weight layers.4.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=24677) WARNING 11-06 14:39:50 [utils.py:333] Attempted to load weight layers.5.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=24677) WARNING 11-06 14:39:50 [utils.py:333] Attempted to load weight layers.6.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=24677) WARNING 11-06 14:39:50 [utils.py:333] Attempted to load weight layers.8.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=24677) WARNING 11-06 14:39:50 [utils.py:333] Attempted to load weight layers.9.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00, 1.15s/it] | |
| Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00, 1.03it/s] | |
| [transformers] Qwen3VL video processing does not apply the per-frame pixel cap the reference implementation (qwen-vl-utils) applies, so some videos cost far more tokens than they would there. In v5.22 the capped behavior will become the default and `cap_pixels_per_frame` will be removed. Pass `cap_pixels_per_frame=True` to adopt the reference behavior now, or `False` to keep the current behavior and silence this warning. | |
| (EngineCore pid=24677) INFO 11-06 14:39:51 [default_loader.py:506] Loading weights took 4.01 seconds | |
| (MainProcess pid=24337) INFO 11-06 14:39:52 [base.py:283] Multi-modal warmup completed in 21.349s | |
| (EngineCore pid=24677) INFO 11-06 14:39:53 [model_runner.py:428] Model loading took 17.57 GiB memory and 7.992604 seconds | |
| (EngineCore pid=24677) INFO 11-06 14:39:53 [topk_topp_sampler.py:149] Using FlashInfer for top-p & top-k sampling. | |
| (EngineCore pid=24677) INFO 11-06 14:39:53 [torch_utils.py:281] Reducing Torch threads from 255 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override. | |
| (EngineCore pid=24677) INFO 11-06 14:39:53 [interface.py:1108] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size. | |
| (EngineCore pid=24677) INFO 11-06 14:39:53 [interface.py:1132] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal. | |
| (EngineCore pid=24677) INFO 11-06 14:39:53 [utils.py:381] Using LBNHC KV cache layout. | |
| (EngineCore pid=24677) INFO 11-06 14:40:00 [encoder_runner.py:131] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size. | |
| (EngineCore pid=24677) INFO 11-06 14:40:11 [caching.py:342] reconstructed serializable fn from standalone compile artifacts. num_artifacts=13 num_submods=33 | |
| (EngineCore pid=24677) INFO 11-06 14:40:11 [decorators.py:326] [backbone] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/570fe35d686514fcab15b2f5a7f595cad3825ba85c7952aa6d2232b00327468d/rank_0_0/model | |
| (EngineCore pid=24677) INFO 11-06 14:40:11 [monitor.py:62] [backbone] torch.compile took 0.69 s in total | |
| (EngineCore pid=24677) INFO 11-06 14:40:13 [monitor.py:91] [backbone] Initial profiling/warmup run took 1.46 s | |
| 2026-11-06 14:40:13,367 - INFO - core.py:594 - flashinfer.jit: Building JIT module sampling; this can take several minutes on first use. | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:01<01:09, 1.38s/it] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:02<00:59, 1.22s/it] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:03<00:47, 1.01it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:03<00:40, 1.17it/s] Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:04<00:36, 1.25it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:05<00:33, 1.35it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:05<00:30, 1.44it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:06<00:27, 1.54it/s] Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:06<00:25, 1.63it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:07<00:23, 1.75it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:07<00:21, 1.82it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:08<00:20, 1.90it/s] Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:08<00:20, 1.88it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:09<00:18, 1.98it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:09<00:17, 2.09it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:10<00:15, 2.24it/s] Capturing CUDA graphs (PIECEWISE): 33%|███▎ | 17/51 [00:11<00:19, 1.71it/s] Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:11<00:16, 1.99it/s] Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 19/51 [00:11<00:12, 2.47it/s] Capturing CUDA graphs (PIECEWISE): 39%|███▉ | 20/51 [00:11<00:10, 3.02it/s] Capturing CUDA graphs (PIECEWISE): 41%|████ | 21/51 [00:11<00:08, 3.62it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:11<00:07, 4.08it/s] Capturing CUDA graphs (PIECEWISE): 45%|████▌ | 23/51 [00:12<00:06, 4.54it/s] Capturing CUDA graphs (PIECEWISE): 47%|████▋ | 24/51 [00:12<00:05, 5.12it/s] Capturing CUDA graphs (PIECEWISE): 49%|████▉ | 25/51 [00:12<00:04, 5.45it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████ | 26/51 [00:12<00:04, 5.54it/s] Capturing CUDA graphs (PIECEWISE): 53%|█████▎ | 27/51 [00:12<00:04, 5.85it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:12<00:03, 6.03it/s] Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 29/51 [00:13<00:03, 6.04it/s] Capturing CUDA graphs (PIECEWISE): 59%|█████▉ | 30/51 [00:13<00:03, 6.25it/s] Capturing CUDA graphs (PIECEWISE): 61%|██████ | 31/51 [00:13<00:03, 6.41it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 32/51 [00:13<00:03, 6.24it/s] Capturing CUDA graphs (PIECEWISE): 65%|██████▍ | 33/51 [00:13<00:02, 6.47it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:13<00:02, 6.63it/s] Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 35/51 [00:13<00:02, 6.37it/s] Capturing CUDA graphs (PIECEWISE): 71%|███████ | 36/51 [00:14<00:02, 6.89it/s] Capturing CUDA graphs (PIECEWISE): 73%|███████▎ | 37/51 [00:14<00:02, 5.62it/s] Capturing CUDA graphs (PIECEWISE): 75%|███████▍ | 38/51 [00:14<00:02, 5.67it/s] Capturing CUDA graphs (PIECEWISE): 76%|███████▋ | 39/51 [00:14<00:01, 6.03it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:14<00:01, 6.37it/s] Capturing CUDA graphs (PIECEWISE): 80%|████████ | 41/51 [00:14<00:01, 6.45it/s] Capturing CUDA graphs (PIECEWISE): 82%|████████▏ | 42/51 [00:15<00:01, 6.49it/s] Capturing CUDA graphs (PIECEWISE): 84%|████████▍ | 43/51 [00:15<00:01, 6.89it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▋ | 44/51 [00:15<00:00, 7.19it/s] Capturing CUDA graphs (PIECEWISE): 88%|████████▊ | 45/51 [00:15<00:00, 6.92it/s] Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:15<00:00, 7.06it/s] Capturing CUDA graphs (PIECEWISE): 92%|█████████▏| 47/51 [00:15<00:00, 7.23it/s] Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 48/51 [00:15<00:00, 7.39it/s] Capturing CUDA graphs (PIECEWISE): 96%|█████████▌| 49/51 [00:16<00:00, 7.13it/s] Capturing CUDA graphs (PIECEWISE): 98%|█████████▊| 50/51 [00:16<00:00, 7.33it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:16<00:00, 5.61it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:16<00:00, 3.10it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 14.50it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 14.48it/s] | |
| (EngineCore pid=24677) INFO 11-06 14:41:58 [model_runner.py:1163] Graph capturing finished in 18 secs, took 0.43 GiB | |
| (EngineCore pid=24677) INFO 11-06 14:41:58 [gpu_worker.py:772] Available KV cache memory: 15.46 GiB | |
| (EngineCore pid=24677) INFO 11-06 14:41:58 [gpu_worker.py:787] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.4500 is equivalent to --gpu-memory-utilization=0.4440 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.4560. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. | |
| (EngineCore pid=24677) INFO 11-06 14:41:58 [kv_cache_utils.py:1147] GPU KV cache size: 357,096 tokens, Maximum concurrency for 8,192 tokens per request: 43.59x | |
| (EngineCore pid=24677) INFO 11-06 14:41:58 [kv_cache_utils.py:865] kv cache group sizes (528, 528, 528, 528) | |
| (EngineCore pid=24677) INFO 11-06 14:41:58 [kv_cache_utils.py:866] kv lcm block sizes 528 | |
| (EngineCore pid=24677) INFO 11-06 14:41:59 [kernel_warmup.py:179] JIT kernel warmup starting. | |
| (EngineCore pid=24677) INFO 11-06 14:41:59 [kernel_warmup.py:192] JIT kernel warmup finished in 0.00s. | |
| (EngineCore pid=24677) INFO 11-06 14:41:59 [qwen_triton_warmup.py:370] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text. | |
| (EngineCore pid=24677) INFO 11-06 14:42:01 [qwen_vl_triton_warmup.py:57] Warmed position embedding and vision rotary kernels on grids=[(1, 16, 16), (1, 16, 2), (1, 2, 16), (1, 2, 2)]. | |
| (EngineCore pid=24677) INFO 11-06 14:42:01 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel. | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:00<00:42, 1.17it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:01<00:40, 1.21it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:02<00:37, 1.27it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:03<00:34, 1.36it/s] Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:03<00:33, 1.38it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:04<00:31, 1.43it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:05<00:30, 1.46it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:05<00:27, 1.56it/s] Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:06<00:25, 1.64it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:06<00:24, 1.69it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:07<00:23, 1.68it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:07<00:21, 1.81it/s] Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:08<00:20, 1.85it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:08<00:19, 1.93it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:09<00:17, 2.04it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:09<00:16, 2.18it/s] Capturing CUDA graphs (PIECEWISE): 33%|███▎ | 17/51 [00:09<00:12, 2.75it/s] Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:09<00:09, 3.35it/s] Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 19/51 [00:10<00:08, 3.81it/s] Capturing CUDA graphs (PIECEWISE): 39%|███▉ | 20/51 [00:10<00:07, 4.39it/s] Capturing CUDA graphs (PIECEWISE): 41%|████ | 21/51 [00:10<00:06, 4.84it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:10<00:05, 5.08it/s] Capturing CUDA graphs (PIECEWISE): 45%|████▌ | 23/51 [00:10<00:05, 5.43it/s] Capturing CUDA graphs (PIECEWISE): 47%|████▋ | 24/51 [00:10<00:04, 5.74it/s] Capturing CUDA graphs (PIECEWISE): 49%|████▉ | 25/51 [00:10<00:04, 5.63it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████ | 26/51 [00:11<00:04, 5.94it/s] Capturing CUDA graphs (PIECEWISE): 53%|█████▎ | 27/51 [00:11<00:03, 6.22it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:11<00:03, 6.07it/s] Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 29/51 [00:11<00:03, 6.25it/s] Capturing CUDA graphs (PIECEWISE): 59%|█████▉ | 30/51 [00:11<00:03, 6.41it/s] Capturing CUDA graphs (PIECEWISE): 61%|██████ | 31/51 [00:11<00:03, 6.26it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 32/51 [00:12<00:02, 6.36it/s] Capturing CUDA graphs (PIECEWISE): 65%|██████▍ | 33/51 [00:12<00:02, 6.67it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:12<00:02, 6.97it/s] Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 35/51 [00:12<00:02, 6.65it/s] Capturing CUDA graphs (PIECEWISE): 71%|███████ | 36/51 [00:12<00:02, 6.89it/s] Capturing CUDA graphs (PIECEWISE): 73%|███████▎ | 37/51 [00:12<00:01, 7.02it/s] Capturing CUDA graphs (PIECEWISE): 75%|███████▍ | 38/51 [00:12<00:01, 6.73it/s] Capturing CUDA graphs (PIECEWISE): 76%|███████▋ | 39/51 [00:13<00:01, 6.92it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:13<00:01, 7.12it/s] Capturing CUDA graphs (PIECEWISE): 80%|████████ | 41/51 [00:13<00:01, 7.24it/s] Capturing CUDA graphs (PIECEWISE): 82%|████████▏ | 42/51 [00:13<00:01, 6.98it/s] Capturing CUDA graphs (PIECEWISE): 84%|████████▍ | 43/51 [00:13<00:01, 7.08it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▋ | 44/51 [00:13<00:00, 7.24it/s] Capturing CUDA graphs (PIECEWISE): 88%|████████▊ | 45/51 [00:13<00:00, 7.29it/s] Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:14<00:00, 6.91it/s] Capturing CUDA graphs (PIECEWISE): 92%|█████████▏| 47/51 [00:14<00:00, 7.08it/s] Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 48/51 [00:14<00:00, 7.23it/s] Capturing CUDA graphs (PIECEWISE): 96%|█████████▌| 49/51 [00:14<00:00, 6.92it/s] Capturing CUDA graphs (PIECEWISE): 98%|█████████▊| 50/51 [00:14<00:00, 7.04it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:14<00:00, 7.27it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:14<00:00, 3.46it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/35 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 6%|▌ | 2/35 [00:00<00:02, 13.09it/s] Capturing CUDA graphs (FULL): 11%|█▏ | 4/35 [00:00<00:02, 13.17it/s] Capturing CUDA graphs (FULL): 17%|█▋ | 6/35 [00:00<00:02, 13.37it/s] Capturing CUDA graphs (FULL): 23%|██▎ | 8/35 [00:00<00:02, 13.46it/s] Capturing CUDA graphs (FULL): 29%|██▊ | 10/35 [00:00<00:01, 13.28it/s] Capturing CUDA graphs (FULL): 34%|███▍ | 12/35 [00:00<00:01, 13.40it/s] Capturing CUDA graphs (FULL): 40%|████ | 14/35 [00:01<00:01, 13.62it/s] Capturing CUDA graphs (FULL): 46%|████▌ | 16/35 [00:01<00:01, 13.63it/s] Capturing CUDA graphs (FULL): 51%|█████▏ | 18/35 [00:01<00:01, 13.68it/s] Capturing CUDA graphs (FULL): 57%|█████▋ | 20/35 [00:01<00:01, 14.35it/s] Capturing CUDA graphs (FULL): 63%|██████▎ | 22/35 [00:01<00:00, 14.84it/s] Capturing CUDA graphs (FULL): 69%|██████▊ | 24/35 [00:01<00:00, 14.57it/s] Capturing CUDA graphs (FULL): 74%|███████▍ | 26/35 [00:01<00:00, 15.01it/s] Capturing CUDA graphs (FULL): 80%|████████ | 28/35 [00:01<00:00, 15.65it/s] Capturing CUDA graphs (FULL): 86%|████████▌ | 30/35 [00:02<00:00, 16.34it/s] Capturing CUDA graphs (FULL): 91%|█████████▏| 32/35 [00:02<00:00, 14.33it/s] Capturing CUDA graphs (FULL): 97%|█████████▋| 34/35 [00:02<00:00, 15.65it/s] Capturing CUDA graphs (FULL): 100%|██████████| 35/35 [00:02<00:00, 14.56it/s] | |
| (EngineCore pid=24677) INFO 11-06 14:42:21 [model_runner.py:1163] Graph capturing finished in 18 secs, took 0.27 GiB | |
| (EngineCore pid=24677) INFO 11-06 14:42:21 [gpu_worker.py:1003] CUDA graph pool memory: 0.27 GiB (actual), 0.48 GiB (estimated), difference: 0.21 GiB (78.8%). | |
| (EngineCore pid=24677) INFO 11-06 14:42:21 [gpu_worker.py:1066] Free memory on device (74.86/79.25 GiB) on startup. Desired GPU memory utilization is (0.45, 35.66 GiB). Actual usage is 17.94 GiB for consumed memory (weights + non-torch), 2.27 GiB for peak activation, and 0.27 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=16150956749` (15.04 GiB) to fit into requested memory, or `--kv-cache-memory=58242514432` (54.24 GiB) to fully utilize gpu memory. Current kv cache memory in use is 15.46 GiB. | |
| (EngineCore pid=24677) INFO 11-06 14:42:23 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. | |
| (EngineCore pid=24677) INFO 11-06 14:42:24 [core.py:405] init engine (profile, create kv cache, warmup model) took 150.74 s (compilation: 0.69 s) | |
| (EngineCore pid=24677) INFO 11-06 14:42:24 [kernel.py:480] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=24337) INFO 11-06 14:42:24 [hf.py:736] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this. | |
| Rendering prompts: 0%| | 0/64 [00:00<?, ?it/s] Rendering prompts: 100%|██████████| 64/64 [00:00<00:00, 699.98it/s] | |
| Processed prompts: 0%| | 0/64 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s] Processed prompts: 2%|▏ | 1/64 [00:05<05:32, 5.28s/it, est. speed input: 61.21 toks/s, output: 9.66 toks/s] Processed prompts: 5%|▍ | 3/64 [00:05<01:26, 1.42s/it, est. speed input: 188.21 toks/s, output: 29.03 toks/s] Processed prompts: 6%|▋ | 4/64 [00:05<00:58, 1.03it/s, est. speed input: 215.41 toks/s, output: 38.64 toks/s] Processed prompts: 9%|▉ | 6/64 [00:05<00:30, 1.91it/s, est. speed input: 309.03 toks/s, output: 59.02 toks/s] Processed prompts: 12%|█▎ | 8/64 [00:05<00:18, 3.01it/s, est. speed input: 385.95 toks/s, output: 79.43 toks/s] Processed prompts: 17%|█▋ | 11/64 [00:05<00:10, 5.07it/s, est. speed input: 492.52 toks/s, output: 110.59 toks/s] Processed prompts: 27%|██▋ | 17/64 [00:06<00:04, 10.37it/s, est. speed input: 678.24 toks/s, output: 175.94 toks/s] Processed prompts: 33%|███▎ | 21/64 [00:06<00:03, 13.01it/s, est. speed input: 797.25 toks/s, output: 217.52 toks/s] Processed prompts: 42%|████▏ | 27/64 [00:06<00:01, 18.82it/s, est. speed input: 980.86 toks/s, output: 283.27 toks/s] Processed prompts: 48%|████▊ | 31/64 [00:06<00:01, 20.41it/s, est. speed input: 1214.94 toks/s, output: 324.97 toks/s] Processed prompts: 53%|█████▎ | 34/64 [00:06<00:01, 19.91it/s, est. speed input: 1292.69 toks/s, output: 354.63 toks/s] Processed prompts: 66%|██████▌ | 42/64 [00:06<00:00, 29.98it/s, est. speed input: 1580.88 toks/s, output: 448.92 toks/s] Processed prompts: 73%|███████▎ | 47/64 [00:06<00:00, 32.46it/s, est. speed input: 1779.73 toks/s, output: 505.00 toks/s] Processed prompts: 80%|███████▉ | 51/64 [00:07<00:00, 32.88it/s, est. speed input: 1950.66 toks/s, output: 549.58 toks/s] Processed prompts: 86%|████████▌ | 55/64 [00:07<00:00, 29.49it/s, est. speed input: 2122.53 toks/s, output: 589.39 toks/s] Processed prompts: 92%|█████████▏| 59/64 [00:07<00:00, 13.16it/s, est. speed input: 2131.70 toks/s, output: 588.78 toks/s] Processed prompts: 97%|█████████▋| 62/64 [00:08<00:00, 14.55it/s, est. speed input: 2225.61 toks/s, output: 624.87 toks/s] Processed prompts: 100%|██████████| 64/64 [00:08<00:00, 14.55it/s, est. speed input: 2286.32 toks/s, output: 645.40 toks/s] Processed prompts: 100%|██████████| 64/64 [00:08<00:00, 7.78it/s, est. speed input: 2286.32 toks/s, output: 645.40 toks/s] | |
| {"event": "done", "out": "runs/e3/notes_pilot.jsonl", "stats": {"gold_label_named": 6, "ok": 51, "phrase": 7}} | |
| (MainProcess pid=24337) INFO 11-06 14:42:33 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore | |
| (EngineCore pid=24677) INFO 11-06 14:42:33 [core.py:1525] [shutdown] EngineCore: trigger received signal=SIGTERM | |
| (EngineCore pid=24677) INFO 11-06 14:42:33 [core.py:1712] [shutdown] EngineCore: start mode=abort timeout=0s | |
| (EngineCore pid=24677) INFO 11-06 14:42:33 [core.py:1743] [shutdown] EngineCore: request processing complete; starting resource teardown | |
| (EngineCore pid=24677) INFO 11-06 14:42:33 [core.py:1573] [shutdown] EngineCore: exiting busy loop | |
Xet Storage Details
- Size:
- 34.2 kB
- Xet hash:
- 823f9efffa3422f34458e936403ed9a47b940bf76d4b76340842572898f9a81b
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.