(APIServer pid=1744989) INFO 07-28 07:15:09 [api_utils.py:339] 
(APIServer pid=1744989) INFO 07-28 07:15:09 [api_utils.py:339]        █     █     █▄   ▄█
(APIServer pid=1744989) INFO 07-28 07:15:09 [api_utils.py:339]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.25.0
(APIServer pid=1744989) INFO 07-28 07:15:09 [api_utils.py:339]   █▄█▀ █     █     █     █  model   Qwen/Qwen3.6-35B-A3B
(APIServer pid=1744989) INFO 07-28 07:15:09 [api_utils.py:339]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=1744989) INFO 07-28 07:15:09 [api_utils.py:339] 
(APIServer pid=1744989) INFO 07-28 07:15:09 [api_utils.py:273] non-default args: {'model_tag': 'Qwen/Qwen3.6-35B-A3B', 'default_chat_template_kwargs': {'enable_thinking': False}, 'host': '127.0.0.1', 'port': 8397, 'model': 'Qwen/Qwen3.6-35B-A3B', 'max_model_len': 8192, 'enforce_eager': True, 'served_model_name': ['student'], 'pipeline_parallel_size': 3, 'gpu_memory_utilization': 0.95, 'language_model_only': True, 'max_num_seqs': 4}
(APIServer pid=1744989) INFO 07-28 07:15:10 [model.py:619] Resolved architecture: Qwen3_5MoeForConditionalGeneration
(APIServer pid=1744989) INFO 07-28 07:15:10 [model.py:1776] Using max model len 8192
(APIServer pid=1744989) INFO 07-28 07:15:10 [vllm.py:1042] Asynchronous scheduling is enabled.
(APIServer pid=1744989) WARNING 07-28 07:15:10 [vllm.py:1096] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
(APIServer pid=1744989) WARNING 07-28 07:15:10 [vllm.py:1144] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(APIServer pid=1744989) INFO 07-28 07:15:10 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
(APIServer pid=1744989) INFO 07-28 07:15:10 [vllm.py:1322] Cudagraph is disabled under eager mode
(APIServer pid=1744989) INFO 07-28 07:15:10 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant
(APIServer pid=1744989) INFO 07-28 07:15:14 [registry.py:134] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode.
(EngineCore pid=1745230) INFO 07-28 07:15:23 [core.py:114] Initializing a V1 LLM engine (v0.25.0) with config: model='Qwen/Qwen3.6-35B-A3B', speculative_config=None, tokenizer='Qwen/Qwen3.6-35B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=3, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=student, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=1745230) WARNING 07-28 07:15:23 [multiproc_executor.py:1067] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore pid=1745230) INFO 07-28 07:15:23 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=192.168.0.15 (local), world_size=3, local_world_size=3
INFO 07-28 07:15:33 [registry.py:134] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode.
(Worker pid=1745345) INFO 07-28 07:15:33 [parallel_state.py:1607] world_size=3 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:47793 backend=nccl
INFO 07-28 07:15:33 [registry.py:134] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode.
INFO 07-28 07:15:33 [registry.py:134] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode.
(Worker pid=1745346) INFO 07-28 07:15:34 [parallel_state.py:1607] world_size=3 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:47793 backend=nccl
(Worker pid=1745347) INFO 07-28 07:15:34 [parallel_state.py:1607] world_size=3 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:47793 backend=nccl
(Worker pid=1745345) INFO 07-28 07:15:34 [pynccl.py:113] vLLM is using nccl==2.28.9
(Worker pid=1745345) INFO 07-28 07:15:35 [cuda_communicator.py:264] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'pp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'AITER_CUSTOM', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=1745345) INFO 07-28 07:15:35 [parallel_state.py:1942] rank 0 in world size 3 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(Worker pid=1745345) INFO 07-28 07:15:35 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(Worker_PP0 pid=1745345) INFO 07-28 07:15:35 [gpu_model_runner.py:5209] Starting to load model Qwen/Qwen3.6-35B-A3B...
(Worker_PP0 pid=1745345) INFO 07-28 07:15:36 [cuda.py:535] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(Worker_PP0 pid=1745345) INFO 07-28 07:15:36 [mm_encoder_attention.py:373] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(Worker_PP2 pid=1745347) INFO 07-28 07:15:36 [utils.py:162] Hidden layers were unevenly partitioned: [13,14,13]. This can be manually overridden using the VLLM_PP_LAYER_PARTITION environment variable
(Worker_PP0 pid=1745345) INFO 07-28 07:15:36 [utils.py:162] Hidden layers were unevenly partitioned: [13,14,13]. This can be manually overridden using the VLLM_PP_LAYER_PARTITION environment variable
(Worker_PP0 pid=1745345) INFO 07-28 07:15:36 [qwen_gdn_linear_attn.py:228] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(Worker_PP1 pid=1745346) INFO 07-28 07:15:36 [utils.py:162] Hidden layers were unevenly partitioned: [13,14,13]. This can be manually overridden using the VLLM_PP_LAYER_PARTITION environment variable
(Worker_PP0 pid=1745345) INFO 07-28 07:15:36 [unquantized.py:262] Using TRITON Unquantized MoE backend out of potential backends: ['FlashInfer TRTLLM', 'FlashInfer CUTLASS', 'TRITON', 'BATCHED_TRITON'].
(Worker_PP0 pid=1745345) INFO 07-28 07:15:36 [cuda.py:476] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(Worker_PP0 pid=1745345) INFO 07-28 07:15:36 [flash_attn.py:718] Using FlashAttention version 2
(Worker_PP0 pid=1745345) INFO 07-28 07:15:37 [weight_utils.py:849] Filesystem type for checkpoints: EXT4. Checkpoint size: 66.97 GiB. Available RAM: 106.83 GiB.
(Worker_PP0 pid=1745345) INFO 07-28 07:15:37 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(Worker_PP0 pid=1745345) 
Loading safetensors checkpoint shards:   0% Completed | 0/26 [00:00<?, ?it/s]
(Worker_PP0 pid=1745345) 
Loading safetensors checkpoint shards:   4% Completed | 1/26 [00:00<00:15,  1.58it/s]
(Worker_PP0 pid=1745345) 
Loading safetensors checkpoint shards:   8% Completed | 2/26 [00:00<00:09,  2.59it/s]
(Worker_PP0 pid=1745345) 
Loading safetensors checkpoint shards:  12% Completed | 3/26 [00:01<00:11,  1.98it/s]
(Worker_PP0 pid=1745345) 
Loading safetensors checkpoint shards:  15% Completed | 4/26 [00:02<00:11,  1.84it/s]
(Worker_PP0 pid=1745345) 
Loading safetensors checkpoint shards:  19% Completed | 5/26 [00:02<00:11,  1.80it/s]
(Worker_PP0 pid=1745345) 
Loading safetensors checkpoint shards:  23% Completed | 6/26 [00:03<00:11,  1.68it/s]
(Worker_PP0 pid=1745345) 
Loading safetensors checkpoint shards:  27% Completed | 7/26 [00:03<00:09,  1.97it/s]
(Worker_PP0 pid=1745345) 
Loading safetensors checkpoint shards:  31% Completed | 8/26 [00:04<00:08,  2.10it/s]
(Worker_PP0 pid=1745345) 
Loading safetensors checkpoint shards:  35% Completed | 9/26 [00:04<00:06,  2.58it/s]
(Worker_PP0 pid=1745345) 
Loading safetensors checkpoint shards: 100% Completed | 26/26 [00:04<00:00,  6.05it/s]
(Worker_PP0 pid=1745345) 
(Worker_PP0 pid=1745345) INFO 07-28 07:15:41 [default_loader.py:430] Loading weights took 4.30 seconds
(Worker_PP0 pid=1745345) INFO 07-28 07:15:41 [unquantized.py:334] Using MoEPrepareAndFinalizeNoDPEPModular
(Worker_PP2 pid=1745347) INFO 07-28 07:15:42 [interface.py:890] Setting attention block size to 1056 tokens to ensure that attention page size is >= mamba page size.
(Worker_PP2 pid=1745347) INFO 07-28 07:15:42 [interface.py:914] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal.
(Worker_PP0 pid=1745345) INFO 07-28 07:15:42 [gpu_model_runner.py:5306] Model loading took 21.44 GiB memory and 5.475079 seconds
(Worker_PP0 pid=1745345) INFO 07-28 07:15:42 [interface.py:890] Setting attention block size to 1056 tokens to ensure that attention page size is >= mamba page size.
(Worker_PP0 pid=1745345) INFO 07-28 07:15:42 [interface.py:914] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal.
(Worker_PP1 pid=1745346) INFO 07-28 07:15:42 [interface.py:890] Setting attention block size to 1056 tokens to ensure that attention page size is >= mamba page size.
(Worker_PP1 pid=1745346) INFO 07-28 07:15:42 [interface.py:914] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal.
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004] WorkerProc hit an exception.
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004] Traceback (most recent call last):
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 996, in worker_busy_loop
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     output = func(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]              ^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return func(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 468, in determine_available_memory
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     self.model_runner.profile_run()
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6350, in profile_run
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     hidden_states, last_hidden_states = self._dummy_run(
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]                                         ^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return func(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6010, in _dummy_run
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     outputs = self.model(
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]               ^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return self._call_impl(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return forward_call(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_5.py", line 496, in forward
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     hidden_states = self.language_model.model(
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 507, in __call__
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return self.forward(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_next.py", line 612, in forward
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     hidden_states, residual = layer(
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]                               ^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return self._call_impl(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return forward_call(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_next.py", line 485, in forward
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     hidden_states, residual = self.input_layernorm(hidden_states, residual)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return self._call_impl(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return forward_call(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/model_executor/custom_op.py", line 136, in forward
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return self._forward_method(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/model_executor/layers/layernorm.py", line 164, in forward_cuda
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return self.forward_native(x, residual)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/model_executor/layers/layernorm.py", line 157, in forward_native
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return ir.ops.fused_add_rms_norm(x, residual, weight, self.variance_epsilon)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/ir/op.py", line 370, in __call__
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return self._inner_call(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/ir/op.py", line 313, in _inner_call
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return impl.func_impl_fn(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/ir/op.py", line 656, in func_impl_fn
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return self.impl_fn(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/ir/ops/layernorm.py", line 57, in fused_add_rms_norm
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     x = x.to(weight.dtype) * weight
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]         ~~~~~~~~~~~~~~~~~~~^~~~~~~~
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004] torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 MiB. GPU 1 has a total capacity of 23.56 GiB of which 13.38 MiB is free. Including non-PyTorch memory, this process has 23.52 GiB memory in use. Of the allocated memory 23.07 GiB is allocated by PyTorch, and 30.22 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004] Traceback (most recent call last):
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 996, in worker_busy_loop
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     output = func(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]              ^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return func(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 468, in determine_available_memory
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     self.model_runner.profile_run()
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6350, in profile_run
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     hidden_states, last_hidden_states = self._dummy_run(
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]                                         ^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return func(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 6010, in _dummy_run
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     outputs = self.model(
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]               ^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return self._call_impl(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return forward_call(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_5.py", line 496, in forward
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     hidden_states = self.language_model.model(
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 507, in __call__
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return self.forward(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_next.py", line 612, in forward
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     hidden_states, residual = layer(
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]                               ^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return self._call_impl(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return forward_call(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_next.py", line 485, in forward
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     hidden_states, residual = self.input_layernorm(hidden_states, residual)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return self._call_impl(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return forward_call(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/model_executor/custom_op.py", line 136, in forward
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return self._forward_method(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/model_executor/layers/layernorm.py", line 164, in forward_cuda
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return self.forward_native(x, residual)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/model_executor/layers/layernorm.py", line 157, in forward_native
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return ir.ops.fused_add_rms_norm(x, residual, weight, self.variance_epsilon)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/ir/op.py", line 370, in __call__
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return self._inner_call(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/ir/op.py", line 313, in _inner_call
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return impl.func_impl_fn(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/ir/op.py", line 656, in func_impl_fn
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     return self.impl_fn(*args, **kwargs)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/ir/ops/layernorm.py", line 57, in fused_add_rms_norm
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]     x = x.to(weight.dtype) * weight
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004]         ~~~~~~~~~~~~~~~~~~~^~~~~~~~
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004] torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 MiB. GPU 1 has a total capacity of 23.56 GiB of which 13.38 MiB is free. Including non-PyTorch memory, this process has 23.52 GiB memory in use. Of the allocated memory 23.07 GiB is allocated by PyTorch, and 30.22 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
(Worker_PP1 pid=1745346) ERROR 07-28 07:15:42 [multiproc_executor.py:1004] 
(Worker_PP0 pid=1745345) WARNING 07-28 07:15:43 [fused_moe.py:1106] Using default MoE config. Performance might be sub-optimal! Config file not found at /home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/configs/E=256,N=512,device_name=NVIDIA_GeForce_RTX_3090.json
(Worker_PP0 pid=1745345) INFO 07-28 07:15:44 [gpu_worker.py:538] Available KV cache memory: 0.63 GiB
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231] EngineCore failed to start.
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231] Traceback (most recent call last):
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1200, in run_engine_core
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]     return func(*args, **kwargs)
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 966, in __init__
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]     super().__init__(
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 133, in __init__
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]     return func(*args, **kwargs)
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 283, in _initialize_kv_caches
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]     return self.collective_rpc("determine_available_memory")
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 402, in collective_rpc
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]     return future if non_block else future.result()
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]                                     ^^^^^^^^^^^^^^^
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 91, in result
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]     return super().result()
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]            ^^^^^^^^^^^^^^^^
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]   File "/home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 449, in result
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]     return self.__get_result()
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]   File "/home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]     raise self._exception
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 95, in _wait_for_response
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]     response = self.aggregate(self.get_response())
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]                               ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 391, in get_response
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231]     raise RuntimeError(
(EngineCore pid=1745230) ERROR 07-28 07:15:44 [core.py:1231] RuntimeError: Worker failed with error 'CUDA out of memory. Tried to allocate 16.00 MiB. GPU 1 has a total capacity of 23.56 GiB of which 13.38 MiB is free. Including non-PyTorch memory, this process has 23.52 GiB memory in use. Of the allocated memory 23.07 GiB is allocated by PyTorch, and 30.22 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)', please check the stack trace above for the root cause
(EngineCore pid=1745230) ERROR 07-28 07:15:46 [multiproc_executor.py:284] Worker proc VllmWorker-1 died unexpectedly, shutting down executor.
(EngineCore pid=1745230) INFO 07-28 07:15:46 [multiproc_executor.py:426] [shutdown] Executor: waiting for worker exit count=3
(EngineCore pid=1745230) Process EngineCore:
(EngineCore pid=1745230) Traceback (most recent call last):
(EngineCore pid=1745230)   File "/home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=1745230)     self.run()
(EngineCore pid=1745230)   File "/home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=1745230)     self._target(*self._args, **self._kwargs)
(EngineCore pid=1745230)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1235, in run_engine_core
(EngineCore pid=1745230)     raise e
(EngineCore pid=1745230)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1200, in run_engine_core
(EngineCore pid=1745230)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=1745230)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1745230)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=1745230)     return func(*args, **kwargs)
(EngineCore pid=1745230)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1745230)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 966, in __init__
(EngineCore pid=1745230)     super().__init__(
(EngineCore pid=1745230)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 133, in __init__
(EngineCore pid=1745230)     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=1745230)                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1745230)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=1745230)     return func(*args, **kwargs)
(EngineCore pid=1745230)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1745230)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 283, in _initialize_kv_caches
(EngineCore pid=1745230)     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=1745230)                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1745230)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=1745230)     return self.collective_rpc("determine_available_memory")
(EngineCore pid=1745230)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1745230)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 402, in collective_rpc
(EngineCore pid=1745230)     return future if non_block else future.result()
(EngineCore pid=1745230)                                     ^^^^^^^^^^^^^^^
(EngineCore pid=1745230)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 91, in result
(EngineCore pid=1745230)     return super().result()
(EngineCore pid=1745230)            ^^^^^^^^^^^^^^^^
(EngineCore pid=1745230)   File "/home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 449, in result
(EngineCore pid=1745230)     return self.__get_result()
(EngineCore pid=1745230)            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1745230)   File "/home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(EngineCore pid=1745230)     raise self._exception
(EngineCore pid=1745230)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 95, in _wait_for_response
(EngineCore pid=1745230)     response = self.aggregate(self.get_response())
(EngineCore pid=1745230)                               ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1745230)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 391, in get_response
(EngineCore pid=1745230)     raise RuntimeError(
(EngineCore pid=1745230) RuntimeError: Worker failed with error 'CUDA out of memory. Tried to allocate 16.00 MiB. GPU 1 has a total capacity of 23.56 GiB of which 13.38 MiB is free. Including non-PyTorch memory, this process has 23.52 GiB memory in use. Of the allocated memory 23.07 GiB is allocated by PyTorch, and 30.22 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)', please check the stack trace above for the root cause
(APIServer pid=1744989) Traceback (most recent call last):
(APIServer pid=1744989)   File "<frozen runpy>", line 198, in _run_module_as_main
(APIServer pid=1744989)   File "<frozen runpy>", line 88, in _run_code
(APIServer pid=1744989)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 101, in <module>
(APIServer pid=1744989)     main()
(APIServer pid=1744989)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 95, in main
(APIServer pid=1744989)     args.dispatch_function(args)
(APIServer pid=1744989)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 148, in cmd
(APIServer pid=1744989)     uvloop.run(run_server(args))
(APIServer pid=1744989)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=1744989)     return __asyncio.run(
(APIServer pid=1744989)            ^^^^^^^^^^^^^^
(APIServer pid=1744989)   File "/home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 195, in run
(APIServer pid=1744989)     return runner.run(main)
(APIServer pid=1744989)            ^^^^^^^^^^^^^^^^
(APIServer pid=1744989)   File "/home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=1744989)     return self._loop.run_until_complete(task)
(APIServer pid=1744989)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1744989)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=1744989)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=1744989)     return await main
(APIServer pid=1744989)            ^^^^^^^^^^
(APIServer pid=1744989)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 698, in run_server
(APIServer pid=1744989)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=1744989)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 712, in run_server_worker
(APIServer pid=1744989)     async with build_async_engine_client(
(APIServer pid=1744989)                ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1744989)   File "/home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=1744989)     return await anext(self.gen)
(APIServer pid=1744989)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1744989)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client
(APIServer pid=1744989)     async with build_async_engine_client_from_engine_args(
(APIServer pid=1744989)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1744989)   File "/home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=1744989)     return await anext(self.gen)
(APIServer pid=1744989)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1744989)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args
(APIServer pid=1744989)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=1744989)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1744989)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config
(APIServer pid=1744989)     return cls(
(APIServer pid=1744989)            ^^^^
(APIServer pid=1744989)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 146, in __init__
(APIServer pid=1744989)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=1744989)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1744989)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1744989)     return func(*args, **kwargs)
(APIServer pid=1744989)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1744989)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 132, in make_async_mp_client
(APIServer pid=1744989)     return AsyncMPClient(*client_args)
(APIServer pid=1744989)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1744989)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1744989)     return func(*args, **kwargs)
(APIServer pid=1744989)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1744989)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 963, in __init__
(APIServer pid=1744989)     super().__init__(
(APIServer pid=1744989)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 573, in __init__
(APIServer pid=1744989)     with launch_core_engines(
(APIServer pid=1744989)          ^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1744989)   File "/home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=1744989)     next(self.gen)
(APIServer pid=1744989)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1213, in launch_core_engines
(APIServer pid=1744989)     wait_for_engine_startup(
(APIServer pid=1744989)   File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1272, in wait_for_engine_startup
(APIServer pid=1744989)     raise RuntimeError(
(APIServer pid=1744989) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
/home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '