Buckets:
| (MainProcess pid=16152) INFO 10-31 00:28:16 [model.py:733] Resolved architecture: Qwen3_5ForConditionalGeneration | |
| (MainProcess pid=16152) INFO 10-31 00:28:16 [model.py:2152] Using max model len 32768 | |
| Fetching 12 files: 0%| | 0/12 [00:00, ?it/s] Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1097.60it/s] | |
| (MainProcess pid=16152) INFO 10-31 00:28:18 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=2048. | |
| (MainProcess pid=16152) INFO 10-31 00:28:18 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration by default when prefix caching is enabled | |
| Parse safetensors files: 0%| | 0/1 [00:00, ?it/s] Parse safetensors files: 100%|██████████| 1/1 [00:00<00:00, 1.44it/s] Parse safetensors files: 100%|██████████| 1/1 [00:00<00:00, 1.44it/s] | |
| (MainProcess pid=16152) INFO 10-31 00:28:19 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=16152) WARNING 10-31 00:28:31 [system_utils.py:153] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized | |
| (EngineCore pid=16572) INFO 10-31 00:28:41 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='Qwen/Qwen3.5-0.8B', speculative_config=None, tokenizer='Qwen/Qwen3.5-0.8B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, tokenizer_revision=main, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=Qwen/Qwen3.5-0.8B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None) | |
| (EngineCore pid=16572) INFO 10-31 00:28:44 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_7a7dcd1adae2497e8cca8f83c758f6a1 backend=nccl | |
| (EngineCore pid=16572) INFO 10-31 00:28:44 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A | |
| (EngineCore pid=16572) INFO 10-31 00:28:44 [gpu_worker.py:518] Using V2 Model Runner | |
| (EngineCore pid=16572) INFO 10-31 00:28:45 [model_runner.py:392] Loading model from scratch... | |
| Fetching 12 files: 0%| | 0/12 [00:00, ?it/s] Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 747.54it/s] | |
| (EngineCore pid=16572) INFO 10-31 00:28:46 [cuda.py:565] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention | |
| (EngineCore pid=16572) INFO 10-31 00:28:46 [mm_encoder_attention.py:372] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention. | |
| (EngineCore pid=16572) INFO 10-31 00:28:46 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128). | |
| (EngineCore pid=16572) INFO 10-31 00:28:47 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. | |
| (EngineCore pid=16572) INFO 10-31 00:28:47 [flash_attn.py:1117] Using FlashAttention version 2 | |
| (EngineCore pid=16572) INFO 10-31 00:28:48 [base_loader.py:82] Initializing model took 2.363736 seconds | |
| (EngineCore pid=16572) INFO 10-31 00:28:48 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 1.63 GiB. Available RAM: 197.66 GiB. | |
| (EngineCore pid=16572) INFO 10-31 00:28:48 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. | |
| Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00, ?it/s] | |
| (EngineCore pid=16572) WARNING 10-31 00:28:48 [utils.py:333] Attempted to load weight layers.0.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=16572) WARNING 10-31 00:28:48 [utils.py:333] Attempted to load weight layers.1.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=16572) WARNING 10-31 00:28:48 [utils.py:333] Attempted to load weight layers.10.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=16572) WARNING 10-31 00:28:48 [utils.py:333] Attempted to load weight layers.12.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=16572) WARNING 10-31 00:28:48 [utils.py:333] Attempted to load weight layers.13.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=16572) WARNING 10-31 00:28:48 [utils.py:333] Attempted to load weight layers.14.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=16572) WARNING 10-31 00:28:48 [utils.py:333] Attempted to load weight layers.16.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=16572) WARNING 10-31 00:28:48 [utils.py:333] Attempted to load weight layers.17.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=16572) WARNING 10-31 00:28:48 [utils.py:333] Attempted to load weight layers.18.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=16572) WARNING 10-31 00:28:48 [utils.py:333] Attempted to load weight layers.2.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=16572) WARNING 10-31 00:28:48 [utils.py:333] Attempted to load weight layers.20.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=16572) WARNING 10-31 00:28:48 [utils.py:333] Attempted to load weight layers.21.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=16572) WARNING 10-31 00:28:48 [utils.py:333] Attempted to load weight layers.22.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=16572) WARNING 10-31 00:28:48 [utils.py:333] Attempted to load weight layers.4.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=16572) WARNING 10-31 00:28:48 [utils.py:333] Attempted to load weight layers.5.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=16572) WARNING 10-31 00:28:48 [utils.py:333] Attempted to load weight layers.6.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=16572) WARNING 10-31 00:28:48 [utils.py:333] Attempted to load weight layers.8.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=16572) WARNING 10-31 00:28:48 [utils.py:333] Attempted to load weight layers.9.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 4.40it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 4.39it/s] | |
| (EngineCore pid=16572) INFO 10-31 00:28:48 [default_loader.py:501] Loading weights took 0.23 seconds | |
| (EngineCore pid=16572) INFO 10-31 00:28:49 [punica_selector.py:20] Using PunicaWrapperGPU. | |
| (EngineCore pid=16572) INFO 10-31 00:28:49 [model_manager.py:242] Qwen3_5ForConditionalGeneration supports adding LoRA to the tower modules. If needed, please set `enable_tower_connector_lora=True`. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.merger.linear_fc1 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.merger.linear_fc2 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.0.attn.qkv will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.0.attn.proj will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.0.mlp.linear_fc1 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.0.mlp.linear_fc2 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.1.attn.qkv will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.1.attn.proj will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.1.mlp.linear_fc1 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.1.mlp.linear_fc2 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.2.attn.qkv will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.2.attn.proj will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.2.mlp.linear_fc1 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.2.mlp.linear_fc2 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.3.attn.qkv will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.3.attn.proj will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.3.mlp.linear_fc1 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.3.mlp.linear_fc2 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.4.attn.qkv will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.4.attn.proj will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.4.mlp.linear_fc1 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.4.mlp.linear_fc2 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.5.attn.qkv will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.5.attn.proj will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.5.mlp.linear_fc1 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.5.mlp.linear_fc2 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.6.attn.qkv will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.6.attn.proj will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.6.mlp.linear_fc1 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.6.mlp.linear_fc2 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.7.attn.qkv will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.7.attn.proj will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.7.mlp.linear_fc1 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.7.mlp.linear_fc2 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.8.attn.qkv will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.8.attn.proj will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.8.mlp.linear_fc1 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.8.mlp.linear_fc2 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.9.attn.qkv will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.9.attn.proj will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.9.mlp.linear_fc1 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.9.mlp.linear_fc2 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.10.attn.qkv will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.10.attn.proj will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.10.mlp.linear_fc1 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.10.mlp.linear_fc2 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.11.attn.qkv will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.11.attn.proj will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.11.mlp.linear_fc1 will be ignored. | |
| (EngineCore pid=16572) WARNING 10-31 00:28:49 [model_manager.py:496] Regarding Qwen3_5ForConditionalGeneration, no matching PunicaWrapper is found; visual.blocks.11.mlp.linear_fc2 will be ignored. | |
| (EngineCore pid=16572) INFO 10-31 00:28:49 [model_runner.py:424] Model loading took 1.88 GiB memory and 4.687602 seconds | |
| (EngineCore pid=16572) INFO 10-31 00:28:50 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling. | |
| (EngineCore pid=16572) INFO 10-31 00:28:50 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override. | |
| (EngineCore pid=16572) INFO 10-31 00:28:50 [interface.py:1096] Setting attention block size to 544 tokens to ensure that attention page size is >= mamba page size. | |
| (EngineCore pid=16572) INFO 10-31 00:28:50 [interface.py:1120] Padding mamba page size by 2.64% to ensure that mamba page size and attention page size are exactly equal. | |
| (EngineCore pid=16572) INFO 10-31 00:28:50 [utils.py:347] Using LBNHC KV cache layout. | |
| [transformers] Qwen3VL video processing does not apply the per-frame pixel cap the reference implementation (qwen-vl-utils) applies, so some videos cost far more tokens than they would there. In v5.22 the capped behavior will become the default and `cap_pixels_per_frame` will be removed. Pass `cap_pixels_per_frame=True` to adopt the reference behavior now, or `False` to keep the current behavior and silence this warning. | |
| (MainProcess pid=16152) INFO 10-31 00:28:51 [base.py:280] Multi-modal warmup completed in 20.323s | |
| (EngineCore pid=16572) INFO 10-31 00:28:57 [encoder_runner.py:131] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size. | |
| (EngineCore pid=16572) INFO 10-31 00:28:59 [monitor.py:40] [backbone] Starting torch.compile | |
| (EngineCore pid=16572) INFO 10-31 00:29:08 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/57ae865412/rank_0_0/backbone for vLLM's torch.compile | |
| (EngineCore pid=16572) INFO 10-31 00:29:08 [backends.py:1166] [backbone] Dynamo bytecode transform time: 8.83 s | |
| (EngineCore pid=16572) INFO 10-31 00:29:30 [backends.py:398] [backbone] Compiling a graph for compile range (1, 2048) takes 21.24 s | |
| (EngineCore pid=16572) INFO 10-31 00:29:35 [backends.py:927] collected artifacts: 25 entries, 11 artifacts, 23112457 bytes total | |
| (EngineCore pid=16572) INFO 10-31 00:29:35 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/d417b3f6c043a70533088b77168c0635b58c3977707bbfdcada89da858cccda5/rank_0_0/model | |
| (EngineCore pid=16572) INFO 10-31 00:29:35 [monitor.py:62] [backbone] torch.compile took 35.94 s in total | |
| (EngineCore pid=16572) WARNING 10-31 00:29:35 [utils.py:276] Using default LoRA kernel configs | |
| (EngineCore pid=16572) INFO 10-31 00:31:03 [monitor.py:91] [backbone] Initial profiling/warmup run took 88.16 s | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/70 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 1%|▏ | 1/70 [00:00<01:01, 1.12it/s] Capturing CUDA graphs (PIECEWISE): 3%|▎ | 2/70 [00:01<00:39, 1.72it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 3/70 [00:01<00:41, 1.61it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 4/70 [00:04<01:39, 1.51s/it] Capturing CUDA graphs (PIECEWISE): 7%|▋ | 5/70 [00:04<01:07, 1.04s/it] Capturing CUDA graphs (PIECEWISE): 9%|▊ | 6/70 [00:05<00:50, 1.26it/s] Capturing CUDA graphs (PIECEWISE): 10%|█ | 7/70 [00:05<00:37, 1.66it/s] Capturing CUDA graphs (PIECEWISE): 11%|█▏ | 8/70 [00:05<00:31, 1.94it/s] Capturing CUDA graphs (PIECEWISE): 13%|█▎ | 9/70 [00:06<00:25, 2.41it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▍ | 10/70 [00:06<00:22, 2.64it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 11/70 [00:06<00:18, 3.14it/s] Capturing CUDA graphs (PIECEWISE): 17%|█▋ | 12/70 [00:06<00:17, 3.32it/s] Capturing CUDA graphs (PIECEWISE): 19%|█▊ | 13/70 [00:06<00:15, 3.80it/s] Capturing CUDA graphs (PIECEWISE): 20%|██ | 14/70 [00:07<00:15, 3.57it/s] Capturing CUDA graphs (PIECEWISE): 21%|██▏ | 15/70 [00:07<00:13, 4.08it/s] Capturing CUDA graphs (PIECEWISE): 23%|██▎ | 16/70 [00:07<00:12, 4.25it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▍ | 17/70 [00:07<00:11, 4.50it/s] Capturing CUDA graphs (PIECEWISE): 26%|██▌ | 18/70 [00:08<00:10, 4.75it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 19/70 [00:08<00:11, 4.49it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▊ | 20/70 [00:08<00:10, 4.80it/s] Capturing CUDA graphs (PIECEWISE): 30%|███ | 21/70 [00:08<00:09, 5.20it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 22/70 [00:08<00:10, 4.77it/s] Capturing CUDA graphs (PIECEWISE): 33%|███▎ | 23/70 [00:08<00:08, 5.41it/s] Capturing CUDA graphs (PIECEWISE): 34%|███▍ | 24/70 [00:09<00:09, 4.87it/s] Capturing CUDA graphs (PIECEWISE): 36%|███▌ | 25/70 [00:11<00:43, 1.03it/s] Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 26/70 [00:12<00:32, 1.34it/s] Capturing CUDA graphs (PIECEWISE): 39%|███▊ | 27/70 [00:12<00:24, 1.79it/s] Capturing CUDA graphs (PIECEWISE): 40%|████ | 28/70 [00:12<00:18, 2.28it/s] Capturing CUDA graphs (PIECEWISE): 41%|████▏ | 29/70 [00:12<00:14, 2.85it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 30/70 [00:12<00:11, 3.40it/s] Capturing CUDA graphs (PIECEWISE): 44%|████▍ | 31/70 [00:12<00:09, 4.23it/s] Capturing CUDA graphs (PIECEWISE): 46%|████▌ | 32/70 [00:13<00:09, 4.14it/s] Capturing CUDA graphs (PIECEWISE): 47%|████▋ | 33/70 [00:14<00:17, 2.09it/s] Capturing CUDA graphs (PIECEWISE): 50%|█████ | 35/70 [00:14<00:11, 3.01it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████▏ | 36/70 [00:16<00:23, 1.46it/s] Capturing CUDA graphs (PIECEWISE): 54%|█████▍ | 38/70 [00:18<00:24, 1.31it/s] Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 40/70 [00:18<00:15, 1.92it/s] Capturing CUDA graphs (PIECEWISE): 60%|██████ | 42/70 [00:18<00:10, 2.74it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 44/70 [00:18<00:06, 3.75it/s] Capturing CUDA graphs (PIECEWISE): 66%|██████▌ | 46/70 [00:18<00:05, 4.73it/s] Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 48/70 [00:18<00:03, 6.03it/s] Capturing CUDA graphs (PIECEWISE): 71%|███████▏ | 50/70 [00:18<00:02, 7.34it/s] Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 52/70 [00:19<00:02, 8.19it/s] Capturing CUDA graphs (PIECEWISE): 77%|███████▋ | 54/70 [00:19<00:01, 8.95it/s] Capturing CUDA graphs (PIECEWISE): 80%|████████ | 56/70 [00:19<00:01, 10.15it/s] Capturing CUDA graphs (PIECEWISE): 83%|████████▎ | 58/70 [00:19<00:01, 10.41it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▌ | 60/70 [00:21<00:03, 2.82it/s] Capturing CUDA graphs (PIECEWISE): 87%|████████▋ | 61/70 [00:22<00:03, 2.44it/s] Capturing CUDA graphs (PIECEWISE): 89%|████████▊ | 62/70 [00:22<00:02, 2.86it/s] Capturing CUDA graphs (PIECEWISE): 91%|█████████▏| 64/70 [00:22<00:01, 4.06it/s] Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 66/70 [00:22<00:00, 5.40it/s] Capturing CUDA graphs (PIECEWISE): 97%|█████████▋| 68/70 [00:22<00:00, 6.84it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 70/70 [00:26<00:00, 1.55it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 70/70 [00:26<00:00, 2.68it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 16.28it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 16.25it/s] | |
| (EngineCore pid=16572) INFO 10-31 00:31:33 [model_runner.py:1127] Graph capturing finished in 27 secs, took 0.45 GiB | |
| (EngineCore pid=16572) INFO 10-31 00:31:33 [gpu_worker.py:759] Available KV cache memory: 65.09 GiB | |
| (EngineCore pid=16572) INFO 10-31 00:31:33 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9000 is equivalent to --gpu-memory-utilization=0.8934 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9066. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. | |
| (EngineCore pid=16572) INFO 10-31 00:31:33 [kv_cache_utils.py:2327] GPU KV cache size: 5,112,786 tokens, Maximum concurrency for 32,768 tokens per request: 156.03x | |
| (EngineCore pid=16572) INFO 10-31 00:31:33 [kv_cache_utils.py:853] kv cache group sizes (544, 544, 544, 544) | |
| (EngineCore pid=16572) INFO 10-31 00:31:33 [kv_cache_utils.py:854] kv lcm block sizes 544 | |
| (EngineCore pid=16572) INFO 10-31 00:31:34 [kernel_warmup.py:176] JIT kernel warmup starting. | |
| (EngineCore pid=16572) INFO 10-31 00:31:34 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s. | |
| (EngineCore pid=16572) INFO 10-31 00:31:34 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text. | |
| (EngineCore pid=16572) INFO 10-31 00:31:35 [qwen_vl_triton_warmup.py:57] Warmed position embedding and vision rotary kernels on grids=[(1, 16, 16), (1, 16, 2), (1, 2, 16), (1, 2, 2)]. | |
| (EngineCore pid=16572) INFO 10-31 00:31:35 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel. | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/70 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 1%|▏ | 1/70 [00:00<00:16, 4.11it/s] Capturing CUDA graphs (PIECEWISE): 3%|▎ | 2/70 [00:00<00:18, 3.60it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 3/70 [00:00<00:17, 3.94it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 4/70 [00:01<00:16, 3.95it/s] Capturing CUDA graphs (PIECEWISE): 7%|▋ | 5/70 [00:01<00:15, 4.14it/s] Capturing CUDA graphs (PIECEWISE): 9%|▊ | 6/70 [00:01<00:15, 4.02it/s] Capturing CUDA graphs (PIECEWISE): 10%|█ | 7/70 [00:01<00:16, 3.73it/s] Capturing CUDA graphs (PIECEWISE): 11%|█▏ | 8/70 [00:07<01:56, 1.88s/it] Capturing CUDA graphs (PIECEWISE): 13%|█▎ | 9/70 [00:07<01:22, 1.35s/it] Capturing CUDA graphs (PIECEWISE): 14%|█▍ | 10/70 [00:07<01:00, 1.01s/it] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 11/70 [00:07<00:44, 1.33it/s] Capturing CUDA graphs (PIECEWISE): 17%|█▋ | 12/70 [00:07<00:34, 1.69it/s] Capturing CUDA graphs (PIECEWISE): 19%|█▊ | 13/70 [00:08<00:27, 2.10it/s] Capturing CUDA graphs (PIECEWISE): 20%|██ | 14/70 [00:08<00:21, 2.55it/s] Capturing CUDA graphs (PIECEWISE): 21%|██▏ | 15/70 [00:08<00:18, 2.90it/s] Capturing CUDA graphs (PIECEWISE): 23%|██▎ | 16/70 [00:08<00:16, 3.34it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▍ | 17/70 [00:08<00:13, 3.85it/s] Capturing CUDA graphs (PIECEWISE): 26%|██▌ | 18/70 [00:09<00:12, 4.10it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 19/70 [00:09<00:10, 4.69it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▊ | 20/70 [00:09<00:10, 4.79it/s] Capturing CUDA graphs (PIECEWISE): 30%|███ | 21/70 [00:09<00:09, 5.18it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 22/70 [00:09<00:08, 5.36it/s] Capturing CUDA graphs (PIECEWISE): 33%|███▎ | 23/70 [00:09<00:08, 5.77it/s] Capturing CUDA graphs (PIECEWISE): 34%|███▍ | 24/70 [00:10<00:08, 5.64it/s] Capturing CUDA graphs (PIECEWISE): 36%|███▌ | 25/70 [00:10<00:07, 6.21it/s] Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 26/70 [00:10<00:07, 6.07it/s] Capturing CUDA graphs (PIECEWISE): 39%|███▊ | 27/70 [00:10<00:07, 5.83it/s] Capturing CUDA graphs (PIECEWISE): 40%|████ | 28/70 [00:10<00:06, 6.02it/s] Capturing CUDA graphs (PIECEWISE): 41%|████▏ | 29/70 [00:10<00:06, 6.36it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 30/70 [00:11<00:07, 5.50it/s] Capturing CUDA graphs (PIECEWISE): 44%|████▍ | 31/70 [00:11<00:06, 6.27it/s] Capturing CUDA graphs (PIECEWISE): 46%|████▌ | 32/70 [00:11<00:05, 6.40it/s] Capturing CUDA graphs (PIECEWISE): 49%|████▊ | 34/70 [00:11<00:04, 8.23it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████▏ | 36/70 [00:11<00:03, 10.09it/s] Capturing CUDA graphs (PIECEWISE): 54%|█████▍ | 38/70 [00:11<00:02, 11.54it/s] Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 40/70 [00:12<00:02, 12.60it/s] Capturing CUDA graphs (PIECEWISE): 60%|██████ | 42/70 [00:12<00:02, 12.06it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 44/70 [00:12<00:02, 12.67it/s] Capturing CUDA graphs (PIECEWISE): 66%|██████▌ | 46/70 [00:12<00:01, 13.41it/s] Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 48/70 [00:12<00:01, 13.76it/s] Capturing CUDA graphs (PIECEWISE): 71%|███████▏ | 50/70 [00:12<00:01, 13.84it/s] Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 52/70 [00:12<00:01, 14.20it/s] Capturing CUDA graphs (PIECEWISE): 77%|███████▋ | 54/70 [00:13<00:01, 14.38it/s] Capturing CUDA graphs (PIECEWISE): 80%|████████ | 56/70 [00:13<00:01, 13.85it/s] Capturing CUDA graphs (PIECEWISE): 83%|████████▎ | 58/70 [00:13<00:00, 14.30it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▌ | 60/70 [00:13<00:00, 14.52it/s] Capturing CUDA graphs (PIECEWISE): 89%|████████▊ | 62/70 [00:13<00:00, 14.86it/s] Capturing CUDA graphs (PIECEWISE): 91%|█████████▏| 64/70 [00:13<00:00, 15.12it/s] Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 66/70 [00:13<00:00, 15.29it/s] Capturing CUDA graphs (PIECEWISE): 97%|█████████▋| 68/70 [00:13<00:00, 15.27it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 70/70 [00:14<00:00, 15.48it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 70/70 [00:14<00:00, 4.98it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/38 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 5%|▌ | 2/38 [00:00<00:02, 12.05it/s] Capturing CUDA graphs (FULL): 11%|█ | 4/38 [00:00<00:02, 14.47it/s] Capturing CUDA graphs (FULL): 16%|█▌ | 6/38 [00:00<00:02, 15.01it/s] Capturing CUDA graphs (FULL): 21%|██ | 8/38 [00:00<00:02, 14.40it/s] Capturing CUDA graphs (FULL): 26%|██▋ | 10/38 [00:00<00:01, 14.99it/s] Capturing CUDA graphs (FULL): 32%|███▏ | 12/38 [00:00<00:01, 15.12it/s] Capturing CUDA graphs (FULL): 37%|███▋ | 14/38 [00:00<00:01, 15.39it/s] Capturing CUDA graphs (FULL): 42%|████▏ | 16/38 [00:01<00:01, 14.77it/s] Capturing CUDA graphs (FULL): 47%|████▋ | 18/38 [00:01<00:01, 15.10it/s] Capturing CUDA graphs (FULL): 53%|█████▎ | 20/38 [00:01<00:01, 15.48it/s] Capturing CUDA graphs (FULL): 58%|█████▊ | 22/38 [00:01<00:01, 14.85it/s] Capturing CUDA graphs (FULL): 63%|██████▎ | 24/38 [00:01<00:01, 13.63it/s] Capturing CUDA graphs (FULL): 68%|██████▊ | 26/38 [00:01<00:00, 14.38it/s] Capturing CUDA graphs (FULL): 74%|███████▎ | 28/38 [00:01<00:00, 14.86it/s] Capturing CUDA graphs (FULL): 79%|███████▉ | 30/38 [00:02<00:00, 14.70it/s] Capturing CUDA graphs (FULL): 84%|████████▍ | 32/38 [00:02<00:00, 15.12it/s] Capturing CUDA graphs (FULL): 89%|████████▉ | 34/38 [00:02<00:00, 15.56it/s] Capturing CUDA graphs (FULL): 95%|█████████▍| 36/38 [00:02<00:00, 15.84it/s] Capturing CUDA graphs (FULL): 100%|██████████| 38/38 [00:02<00:00, 15.39it/s] Capturing CUDA graphs (FULL): 100%|██████████| 38/38 [00:02<00:00, 14.94it/s] | |
| (EngineCore pid=16572) INFO 10-31 00:32:12 [model_runner.py:1127] Graph capturing finished in 18 secs, took 0.23 GiB | |
| (EngineCore pid=16572) INFO 10-31 00:32:12 [gpu_worker.py:971] CUDA graph pool memory: 0.23 GiB (actual), 0.52 GiB (estimated), difference: 0.29 GiB (125.4%). | |
| (EngineCore pid=16572) INFO 10-31 00:32:12 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.9, 71.33 GiB). Actual usage is 4.49 GiB for consumed memory (weights + non-torch), 1.75 GiB for peak activation, and 0.23 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=69480321946` (64.71 GiB) to fit into requested memory, or `--kv-cache-memory=77546387968` (72.22 GiB) to fully utilize gpu memory. Current kv cache memory in use is 65.09 GiB. | |
| (EngineCore pid=16572) INFO 10-31 00:32:12 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. | |
| (EngineCore pid=16572) INFO 10-31 00:32:13 [core.py:395] init engine (profile, create kv cache, warmup model) took 203.42 s (compilation: 35.94 s) | |
| Parse safetensors files: 0%| | 0/1 [00:00<?, ?it/s] Parse safetensors files: 100%|██████████| 1/1 [00:00<00:00, 4.87it/s] Parse safetensors files: 100%|██████████| 1/1 [00:00<00:00, 4.86it/s] | |
| (EngineCore pid=16572) INFO 10-31 00:32:13 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=16152) WARNING 10-31 00:32:13 [input_processor.py:289] vLLM has deprecated support for supporting different tokenizers for different LoRAs. By default, vLLM uses base model's tokenizer. If you are using a LoRA with its own tokenizer, consider specifying `--tokenizer [lora_path]` to use the LoRA tokenizer. | |
| (MainProcess pid=16152) WARNING 10-31 00:32:13 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat(). | |
| (EngineCore pid=16572) WARNING 10-31 00:32:14 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _lora_expand_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (EngineCore pid=16572) WARNING 10-31 00:32:14 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| readout joint: 130 items in 4s | |
| readout gold_only: 130 items in 3s | |
| readout votes_only: 130 items in 2s | |
| (MainProcess pid=16152) INFO 10-31 00:32:22 [core_client.py:770] [shutdown] MPClient: start timeout=default | |
| (MainProcess pid=16152) INFO 10-31 00:32:22 [core_client.py:772] [shutdown] MPClient: stopping engine manager | |
| (MainProcess pid=16152) INFO 10-31 00:32:22 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore | |
| (EngineCore pid=16572) INFO 10-31 00:32:22 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM | |
| (EngineCore pid=16572) INFO 10-31 00:32:22 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s | |
| (EngineCore pid=16572) INFO 10-31 00:32:22 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown | |
| (EngineCore pid=16572) INFO 10-31 00:32:22 [core.py:1499] [shutdown] EngineCore: exiting busy loop | |
| (MainProcess pid=16152) INFO 10-31 00:32:28 [core_client.py:774] [shutdown] MPClient: engine manager stopped | |
| (MainProcess pid=16152) INFO 10-31 00:32:28 [core_client.py:775] [shutdown] MPClient: cleaning up background resources | |
| (MainProcess pid=16152) INFO 10-31 00:32:28 [core_client.py:777] [shutdown] MPClient: complete | |
Xet Storage Details
- Size:
- 41.5 kB
- Xet hash:
- a89ff62b4e9260b867a7b8311fe6c5308f06b696a49ef3e036050c553ef71f61
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.