| (APIServer pid=1746049) INFO 07-28 07:22:50 [api_utils.py:339]
|
| (APIServer pid=1746049) INFO 07-28 07:22:50 [api_utils.py:339] █ █ █▄ ▄█
|
| (APIServer pid=1746049) INFO 07-28 07:22:50 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.25.0
|
| (APIServer pid=1746049) INFO 07-28 07:22:50 [api_utils.py:339] █▄█▀ █ █ █ █ model Qwen/Qwen3.6-35B-A3B-FP8
|
| (APIServer pid=1746049) INFO 07-28 07:22:50 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
|
| (APIServer pid=1746049) INFO 07-28 07:22:50 [api_utils.py:339] |
| (APIServer pid=1746049) INFO 07-28 07:22:50 [api_utils.py:273] non-default args: {'model_tag': 'Qwen/Qwen3.6-35B-A3B-FP8', 'default_chat_template_kwargs': {'enable_thinking': False}, 'host': '127.0.0.1', 'port': 8397, 'model': 'Qwen/Qwen3.6-35B-A3B-FP8', 'max_model_len': 8192, 'enforce_eager': True, 'served_model_name': ['student'], 'pipeline_parallel_size': 3, 'gpu_memory_utilization': 0.95, 'language_model_only': True} |
| (APIServer pid=1746049) INFO 07-28 07:22:51 [model.py:619] Resolved architecture: Qwen3_5MoeForConditionalGeneration |
| (APIServer pid=1746049) INFO 07-28 07:22:51 [model.py:1776] Using max model len 8192 |
| (APIServer pid=1746049) INFO 07-28 07:22:52 [vllm.py:1042] Asynchronous scheduling is enabled. |
| (APIServer pid=1746049) WARNING 07-28 07:22:52 [vllm.py:1096] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none |
| (APIServer pid=1746049) WARNING 07-28 07:22:52 [vllm.py:1144] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. |
| (APIServer pid=1746049) INFO 07-28 07:22:52 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) |
| (APIServer pid=1746049) INFO 07-28 07:22:52 [vllm.py:1322] Cudagraph is disabled under eager mode |
| (APIServer pid=1746049) INFO 07-28 07:22:52 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant |
| (APIServer pid=1746049) INFO 07-28 07:22:55 [registry.py:134] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode. |
| (EngineCore pid=1746297) INFO 07-28 07:23:05 [core.py:114] Initializing a V1 LLM engine (v0.25.0) with config: model='Qwen/Qwen3.6-35B-A3B-FP8', speculative_config=None, tokenizer='Qwen/Qwen3.6-35B-A3B-FP8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=3, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=fp8, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=student, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+quant_fp8', 'all', '+quant_fp8'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, moe_backend='auto', linear_backend='auto') |
| (EngineCore pid=1746297) WARNING 07-28 07:23:05 [multiproc_executor.py:1067] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. |
| (EngineCore pid=1746297) INFO 07-28 07:23:05 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=192.168.0.15 (local), world_size=3, local_world_size=3 |
| INFO 07-28 07:23:15 [registry.py:134] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode. |
| (Worker pid=1746409) INFO 07-28 07:23:15 [parallel_state.py:1607] world_size=3 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:54897 backend=nccl |
| INFO 07-28 07:23:15 [registry.py:134] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode. |
| INFO 07-28 07:23:15 [registry.py:134] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode. |
| (Worker pid=1746411) INFO 07-28 07:23:15 [parallel_state.py:1607] world_size=3 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:54897 backend=nccl |
| (Worker pid=1746410) INFO 07-28 07:23:15 [parallel_state.py:1607] world_size=3 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:54897 backend=nccl |
| (Worker pid=1746409) INFO 07-28 07:23:16 [pynccl.py:113] vLLM is using nccl==2.28.9 |
| (Worker pid=1746409) INFO 07-28 07:23:17 [cuda_communicator.py:264] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'pp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'AITER_CUSTOM', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. |
| (Worker pid=1746409) INFO 07-28 07:23:17 [parallel_state.py:1942] rank 0 in world size 3 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A |
| (Worker pid=1746409) INFO 07-28 07:23:17 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. |
| (Worker_PP0 pid=1746409) INFO 07-28 07:23:17 [gpu_model_runner.py:5209] Starting to load model Qwen/Qwen3.6-35B-A3B-FP8... |
| (Worker_PP0 pid=1746409) INFO 07-28 07:23:17 [cuda.py:535] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention |
| (Worker_PP0 pid=1746409) INFO 07-28 07:23:17 [mm_encoder_attention.py:373] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention. |
| (Worker_PP1 pid=1746410) INFO 07-28 07:23:17 [utils.py:162] Hidden layers were unevenly partitioned: [13,14,13]. This can be manually overridden using the VLLM_PP_LAYER_PARTITION environment variable |
| (Worker_PP2 pid=1746411) INFO 07-28 07:23:17 [utils.py:162] Hidden layers were unevenly partitioned: [13,14,13]. This can be manually overridden using the VLLM_PP_LAYER_PARTITION environment variable |
| (Worker_PP0 pid=1746409) INFO 07-28 07:23:17 [utils.py:162] Hidden layers were unevenly partitioned: [13,14,13]. This can be manually overridden using the VLLM_PP_LAYER_PARTITION environment variable |
| (Worker_PP0 pid=1746409) INFO 07-28 07:23:17 [__init__.py:600] Selected MarlinFP8ScaledMMLinearKernel for Fp8LinearMethod |
| (Worker_PP0 pid=1746409) INFO 07-28 07:23:17 [qwen_gdn_linear_attn.py:228] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128). |
| (Worker_PP0 pid=1746409) INFO 07-28 07:23:17 [fp8.py:405] Using MARLIN Fp8 MoE backend out of potential backends: ['AITER', 'FLASHINFER_TRTLLM', 'FLASHINFER_CUTLASS', 'DEEPGEMM', 'TRITON', 'MARLIN', 'HUMMING', 'BATCHED_DEEPGEMM', 'BATCHED_TRITON', 'XPU', 'CPU', 'HPC']. |
| (Worker_PP0 pid=1746409) INFO 07-28 07:23:17 [cuda.py:476] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. |
| (Worker_PP0 pid=1746409) INFO 07-28 07:23:17 [flash_attn.py:718] Using FlashAttention version 2 |
| (Worker_PP0 pid=1746409) INFO 07-28 07:23:19 [weight_utils.py:849] Filesystem type for checkpoints: EXT4. Checkpoint size: 34.89 GiB. Available RAM: 106.59 GiB. |
| (Worker_PP0 pid=1746409) INFO 07-28 07:23:19 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 0% Completed | 0/42 [00:00<?, ?it/s] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 2% Completed | 1/42 [00:01<01:13, 1.80s/it] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 5% Completed | 2/42 [00:03<01:09, 1.73s/it] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 7% Completed | 3/42 [00:05<01:04, 1.66s/it] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 10% Completed | 4/42 [00:06<01:06, 1.74s/it] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 12% Completed | 5/42 [00:08<01:05, 1.78s/it] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 14% Completed | 6/42 [00:10<01:03, 1.77s/it] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 17% Completed | 7/42 [00:12<01:03, 1.80s/it] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 19% Completed | 8/42 [00:14<01:01, 1.81s/it] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 21% Completed | 9/42 [00:16<00:59, 1.81s/it] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 24% Completed | 10/42 [00:17<00:58, 1.84s/it] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 26% Completed | 11/42 [00:19<00:58, 1.89s/it] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 29% Completed | 12/42 [00:21<00:55, 1.86s/it] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 31% Completed | 13/42 [00:23<00:55, 1.90s/it] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 33% Completed | 14/42 [00:23<00:38, 1.36s/it] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 38% Completed | 16/42 [00:24<00:19, 1.30it/s] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 43% Completed | 18/42 [00:24<00:11, 2.02it/s] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 48% Completed | 20/42 [00:24<00:07, 2.88it/s] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 52% Completed | 22/42 [00:24<00:05, 3.87it/s] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 57% Completed | 24/42 [00:24<00:03, 4.97it/s] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 62% Completed | 26/42 [00:24<00:02, 6.10it/s] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 67% Completed | 28/42 [00:25<00:01, 7.22it/s] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 71% Completed | 30/42 [00:27<00:04, 2.41it/s] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 74% Completed | 31/42 [00:28<00:05, 1.90it/s] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 76% Completed | 32/42 [00:29<00:06, 1.57it/s] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 79% Completed | 33/42 [00:30<00:06, 1.38it/s] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 81% Completed | 34/42 [00:31<00:06, 1.26it/s] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 83% Completed | 35/42 [00:32<00:06, 1.16it/s] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 86% Completed | 36/42 [00:33<00:05, 1.11it/s] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 88% Completed | 37/42 [00:34<00:04, 1.06it/s] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 90% Completed | 38/42 [00:35<00:03, 1.04it/s] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 93% Completed | 39/42 [00:36<00:02, 1.03it/s] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 95% Completed | 40/42 [00:37<00:01, 1.08it/s] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 98% Completed | 41/42 [00:37<00:00, 1.19it/s] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 100% Completed | 42/42 [00:39<00:00, 1.22s/it] |
| (Worker_PP0 pid=1746409)
Loading safetensors checkpoint shards: 100% Completed | 42/42 [00:39<00:00, 1.05it/s] |
| (Worker_PP0 pid=1746409) |
| (Worker_PP0 pid=1746409) INFO 07-28 07:23:59 [default_loader.py:430] Loading weights took 39.94 seconds |
| (Worker_PP0 pid=1746409) WARNING 07-28 07:23:59 [marlin_utils_fp8.py:112] Your GPU does not have native support for FP8 computation but FP8 quantization is being used. Weight-only FP8 compression will be used leveraging the Marlin kernel. This may degrade performance for compute-heavy workloads. |
| (Worker_PP0 pid=1746409) INFO 07-28 07:23:59 [fp8.py:688] Using MoEPrepareAndFinalizeNoDPEPModular |
| (Worker_PP0 pid=1746409) INFO 07-28 07:24:01 [gpu_model_runner.py:5306] Model loading took 11.42 GiB memory and 42.831946 seconds |
| (Worker_PP0 pid=1746409) INFO 07-28 07:24:01 [interface.py:890] Setting attention block size to 1056 tokens to ensure that attention page size is >= mamba page size. |
| (Worker_PP0 pid=1746409) INFO 07-28 07:24:01 [interface.py:914] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal. |
| (Worker_PP1 pid=1746410) INFO 07-28 07:24:01 [interface.py:890] Setting attention block size to 1056 tokens to ensure that attention page size is >= mamba page size. |
| (Worker_PP1 pid=1746410) INFO 07-28 07:24:01 [interface.py:914] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal. |
| (Worker_PP2 pid=1746411) INFO 07-28 07:24:09 [interface.py:890] Setting attention block size to 1056 tokens to ensure that attention page size is >= mamba page size. |
| (Worker_PP2 pid=1746411) INFO 07-28 07:24:09 [interface.py:914] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal. |
| (Worker_PP0 pid=1746409) INFO 07-28 07:24:10 [marlin_utils.py:599] Marlin kernel can achieve better performance for small size_n with experimental use_atomic_add feature. You can consider set environment variable VLLM_MARLIN_USE_ATOMIC_ADD to 1 if possible. |
| (Worker_PP0 pid=1746409) INFO 07-28 07:24:10 [gpu_worker.py:538] Available KV cache memory: 10.65 GiB |
| (EngineCore pid=1746297) INFO 07-28 07:24:11 [kv_cache_utils.py:2146] GPU KV cache size: 825,157 tokens |
| (EngineCore pid=1746297) INFO 07-28 07:24:11 [kv_cache_utils.py:2147] Maximum concurrency for 8,192 tokens per request: 100.73x |
| (Worker_PP2 pid=1746411) INFO 07-28 07:24:11 [qwen_triton_warmup.py:372] Warming up Qwen Triton kernels for model_type=qwen3_5_moe_text. |
| (Worker_PP1 pid=1746410) INFO 07-28 07:24:11 [qwen_triton_warmup.py:372] Warming up Qwen Triton kernels for model_type=qwen3_5_moe_text. |
| (Worker_PP0 pid=1746409) INFO 07-28 07:24:11 [qwen_triton_warmup.py:372] Warming up Qwen Triton kernels for model_type=qwen3_5_moe_text. |
| (Worker_PP1 pid=1746410) INFO 07-28 07:24:12 [cutedsl_warmup.py:97] Skipping CuTeDSL warmup because no compile units were requested. |
| (Worker_PP0 pid=1746409) INFO 07-28 07:24:12 [cutedsl_warmup.py:97] Skipping CuTeDSL warmup because no compile units were requested. |
| (Worker_PP1 pid=1746410) INFO 07-28 07:24:12 [jit_monitor.py:73] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. |
| (Worker_PP0 pid=1746409) INFO 07-28 07:24:12 [jit_monitor.py:73] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. |
| (Worker_PP2 pid=1746411) INFO 07-28 07:24:12 [cutedsl_warmup.py:97] Skipping CuTeDSL warmup because no compile units were requested. |
| (Worker_PP2 pid=1746411) INFO 07-28 07:24:12 [jit_monitor.py:73] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. |
| (EngineCore pid=1746297) INFO 07-28 07:24:13 [core.py:344] init engine (profile, create kv cache, warmup model) took 3.91 s |
| (EngineCore pid=1746297) INFO 07-28 07:24:15 [registry.py:134] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode. |
| (EngineCore pid=1746297) INFO 07-28 07:24:15 [vllm.py:1042] Asynchronous scheduling is enabled. |
| (EngineCore pid=1746297) WARNING 07-28 07:24:15 [vllm.py:1096] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none |
| (EngineCore pid=1746297) WARNING 07-28 07:24:15 [vllm.py:1144] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. |
| (EngineCore pid=1746297) INFO 07-28 07:24:15 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) |
| (EngineCore pid=1746297) INFO 07-28 07:24:15 [vllm.py:1322] Cudagraph is disabled under eager mode |
| (EngineCore pid=1746297) INFO 07-28 07:24:15 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant |
| (APIServer pid=1746049) INFO 07-28 07:24:15 [api_server.py:612] Supported tasks: ['generate'] |
| (APIServer pid=1746049) WARNING 07-28 07:24:16 [model.py:1528] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 1.0, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. |
| (APIServer pid=1746049) INFO 07-28 07:24:19 [hf.py:548] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this. |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [api_server.py:616] Starting vLLM server on http://127.0.0.1:8397 |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:37] Available routes are: |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /docs, Methods: GET, HEAD |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /redoc, Methods: GET, HEAD |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /load, Methods: GET |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /version, Methods: GET |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /health, Methods: GET |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /metrics, Methods: GET |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /tokenize, Methods: POST |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /detokenize, Methods: POST |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /v1/models, Methods: GET |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /ping, Methods: GET |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /ping, Methods: POST |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /invocations, Methods: POST |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /v1/chat/completions, Methods: POST |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /v1/responses, Methods: POST |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /v1/completions, Methods: POST |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /v1/messages, Methods: POST |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /generative_scoring, Methods: POST |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /v1/completions/render, Methods: POST |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /v1/completions/derender, Methods: POST |
| (APIServer pid=1746049) INFO 07-28 07:24:20 [launcher.py:46] Route: /inference/v1/generate, Methods: POST |
| (APIServer pid=1746049) INFO: Started server process [1746049] |
| (APIServer pid=1746049) INFO: Waiting for application startup. |
| (APIServer pid=1746049) INFO: Application startup complete. |
| (APIServer pid=1746049) INFO: 127.0.0.1:35764 - "GET /health HTTP/1.1" 200 OK |
| (APIServer pid=1746049) INFO 07-28 07:24:27 [launcher.py:100] [shutdown] API server: shutdown triggered |
| (APIServer pid=1746049) INFO 07-28 07:24:27 [launcher.py:116] [shutdown] API server: stopping engine client mode=abort timeout=0s |
| (APIServer pid=1746049) INFO 07-28 07:24:27 [core_client.py:655] [shutdown] MPClient: start timeout=0s |
| (APIServer pid=1746049) INFO 07-28 07:24:27 [core_client.py:657] [shutdown] MPClient: stopping engine manager |
| (APIServer pid=1746049) WARNING 07-28 07:24:27 [utils.py:626] [shutdown] Process manager: force killing remaining processes count=1 |
| (EngineCore pid=1746297) INFO 07-28 07:24:27 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM |
| (EngineCore pid=1746297) INFO 07-28 07:24:27 [core.py:1333] [shutdown] EngineCore: start mode=abort timeout=0s |
| (EngineCore pid=1746297) INFO 07-28 07:24:27 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown |
| (EngineCore pid=1746297) INFO 07-28 07:24:27 [core.py:1227] [shutdown] EngineCore: exiting busy loop |
| (Worker_PP0 pid=1746409) INFO 07-28 07:24:27 [multiproc_executor.py:790] Parent process exited, terminating worker queues |
| (EngineCore pid=1746297) INFO 07-28 07:24:27 [multiproc_executor.py:426] [shutdown] Executor: waiting for worker exit count=3 |
| (APIServer pid=1746049) INFO 07-28 07:24:27 [core_client.py:659] [shutdown] MPClient: engine manager stopped |
| (APIServer pid=1746049) INFO 07-28 07:24:27 [core_client.py:660] [shutdown] MPClient: cleaning up background resources |
| (APIServer pid=1746049) INFO 07-28 07:24:27 [core_client.py:662] [shutdown] MPClient: complete |
| (APIServer pid=1746049) ERROR 07-28 07:24:27 [async_llm.py:704] AsyncLLM output_handler failed.
|
| (APIServer pid=1746049) ERROR 07-28 07:24:27 [async_llm.py:704] Traceback (most recent call last):
|
| (APIServer pid=1746049) ERROR 07-28 07:24:27 [async_llm.py:704] File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 660, in output_handler
|
| (APIServer pid=1746049) ERROR 07-28 07:24:27 [async_llm.py:704] outputs = await engine_core.get_output_async()
|
| (APIServer pid=1746049) ERROR 07-28 07:24:27 [async_llm.py:704] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| (APIServer pid=1746049) ERROR 07-28 07:24:27 [async_llm.py:704] File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 1061, in get_output_async
|
| (APIServer pid=1746049) ERROR 07-28 07:24:27 [async_llm.py:704] raise self._format_exception(outputs) from None
|
| (APIServer pid=1746049) ERROR 07-28 07:24:27 [async_llm.py:704] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. |
| (APIServer pid=1746049) INFO 07-28 07:24:27 [launcher.py:125] [shutdown] API server: engine client stopped |
| (APIServer pid=1746049) INFO 07-28 07:24:27 [launcher.py:128] [shutdown] API server: signalling HTTP server shutdown |
| (APIServer pid=1746049) INFO 07-28 07:24:28 [launcher.py:149] [shutdown] API server: shutting down FastAPI HTTP server |
| (APIServer pid=1746049) INFO: Shutting down |
| (APIServer pid=1746049) INFO: Waiting for application shutdown. |
| (APIServer pid=1746049) INFO: Application shutdown complete. |
| /home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown |
| warnings.warn('resource_tracker: There appear to be %d ' |
| /home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 4 leaked shared_memory objects to clean up at shutdown |
| warnings.warn('resource_tracker: There appear to be %d ' |
| |
