| (APIServer pid=1597600) INFO 07-26 20:26:44 [api_utils.py:339]
|
| (APIServer pid=1597600) INFO 07-26 20:26:44 [api_utils.py:339] █ █ █▄ ▄█
|
| (APIServer pid=1597600) INFO 07-26 20:26:44 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.25.0
|
| (APIServer pid=1597600) INFO 07-26 20:26:44 [api_utils.py:339] █▄█▀ █ █ █ █ model /home/henry/qwen_healed/keep50/checkpoints/step0200
|
| (APIServer pid=1597600) INFO 07-26 20:26:44 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
|
| (APIServer pid=1597600) INFO 07-26 20:26:44 [api_utils.py:339] |
| (APIServer pid=1597600) INFO 07-26 20:26:44 [api_utils.py:273] non-default args: {'model_tag': '/home/henry/qwen_healed/keep50/checkpoints/step0200', 'default_chat_template_kwargs': {'enable_thinking': False}, 'host': '127.0.0.1', 'port': 8398, 'model': '/home/henry/qwen_healed/keep50/checkpoints/step0200', 'max_model_len': 8192, 'enforce_eager': True, 'served_model_name': ['student'], 'pipeline_parallel_size': 2, 'gpu_memory_utilization': 0.95} |
| (APIServer pid=1597600) INFO 07-26 20:26:44 [model.py:619] Resolved architecture: PrunedQwen3_5MoeForCausalLM |
| (APIServer pid=1597600) INFO 07-26 20:26:44 [model.py:1776] Using max model len 8192 |
| (APIServer pid=1597600) INFO 07-26 20:26:44 [vllm.py:1042] Asynchronous scheduling is enabled. |
| (APIServer pid=1597600) WARNING 07-26 20:26:44 [vllm.py:1096] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none |
| (APIServer pid=1597600) WARNING 07-26 20:26:44 [vllm.py:1144] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. |
| (APIServer pid=1597600) INFO 07-26 20:26:44 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) |
| (APIServer pid=1597600) INFO 07-26 20:26:44 [vllm.py:1322] Cudagraph is disabled under eager mode |
| (APIServer pid=1597600) INFO 07-26 20:26:44 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant |
| (EngineCore pid=1597716) INFO 07-26 20:26:56 [core.py:114] Initializing a V1 LLM engine (v0.25.0) with config: model='/home/henry/qwen_healed/keep50/checkpoints/step0200', speculative_config=None, tokenizer='/home/henry/qwen_healed/keep50/checkpoints/step0200', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=2, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=student, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, moe_backend='auto', linear_backend='auto') |
| (EngineCore pid=1597716) WARNING 07-26 20:26:56 [multiproc_executor.py:1067] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. |
| (EngineCore pid=1597716) INFO 07-26 20:26:56 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=192.168.0.15 (local), world_size=2, local_world_size=2 |
| (Worker pid=1597838) INFO 07-26 20:27:04 [parallel_state.py:1607] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:37317 backend=nccl |
| (Worker pid=1597839) INFO 07-26 20:27:04 [parallel_state.py:1607] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:37317 backend=nccl |
| (Worker pid=1597838) INFO 07-26 20:27:05 [pynccl.py:113] vLLM is using nccl==2.28.9 |
| (Worker pid=1597838) INFO 07-26 20:27:05 [cuda_communicator.py:264] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'pp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'AITER_CUSTOM', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. |
| (Worker pid=1597838) INFO 07-26 20:27:05 [parallel_state.py:1942] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A |
| (Worker pid=1597838) INFO 07-26 20:27:06 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. |
| (Worker_PP0 pid=1597838) INFO 07-26 20:27:06 [gpu_model_runner.py:5209] Starting to load model /home/henry/qwen_healed/keep50/checkpoints/step0200... |
| (Worker_PP0 pid=1597838) INFO 07-26 20:27:06 [qwen_gdn_linear_attn.py:228] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128). |
| (Worker_PP0 pid=1597838) INFO 07-26 20:27:06 [cuda.py:476] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. |
| (Worker_PP0 pid=1597838) INFO 07-26 20:27:06 [flash_attn.py:718] Using FlashAttention version 2 |
| (Worker_PP0 pid=1597838) INFO 07-26 20:27:06 [weight_utils.py:849] Filesystem type for checkpoints: EXT4. Checkpoint size: 34.55 GiB. Available RAM: 107.21 GiB. |
| (Worker_PP0 pid=1597838) INFO 07-26 20:27:06 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. |
| (Worker_PP0 pid=1597838)
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s] |
| (Worker_PP1 pid=1597839) INFO 07-26 20:27:22 [interface.py:890] Setting attention block size to 544 tokens to ensure that attention page size is >= mamba page size. |
| (Worker_PP1 pid=1597839) INFO 07-26 20:27:22 [interface.py:914] Padding mamba page size by 1.49% to ensure that mamba page size and attention page size are exactly equal. |
| (Worker_PP0 pid=1597838)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:17<00:00, 17.93s/it] |
| (Worker_PP0 pid=1597838)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:17<00:00, 17.93s/it] |
| (Worker_PP0 pid=1597838) |
| (Worker_PP0 pid=1597838) INFO 07-26 20:27:25 [default_loader.py:430] Loading weights took 18.84 seconds |
| (Worker_PP0 pid=1597838) INFO 07-26 20:27:26 [gpu_model_runner.py:5306] Model loading took 17.4 GiB memory and 19.161217 seconds |
| (Worker_PP0 pid=1597838) INFO 07-26 20:27:26 [interface.py:890] Setting attention block size to 544 tokens to ensure that attention page size is >= mamba page size. |
| (Worker_PP0 pid=1597838) INFO 07-26 20:27:26 [interface.py:914] Padding mamba page size by 1.49% to ensure that mamba page size and attention page size are exactly equal. |
| (Worker_PP0 pid=1597838) INFO 07-26 20:27:28 [gpu_worker.py:538] Available KV cache memory: 4.76 GiB |
| (EngineCore pid=1597716) INFO 07-26 20:27:28 [kv_cache_utils.py:2146] GPU KV cache size: 255,245 tokens |
| (EngineCore pid=1597716) INFO 07-26 20:27:28 [kv_cache_utils.py:2147] Maximum concurrency for 8,192 tokens per request: 31.16x |
| (Worker_PP1 pid=1597839) INFO 07-26 20:27:28 [cutedsl_warmup.py:97] Skipping CuTeDSL warmup because no compile units were requested. |
| (Worker_PP0 pid=1597838) INFO 07-26 20:27:28 [cutedsl_warmup.py:97] Skipping CuTeDSL warmup because no compile units were requested. |
| (Worker_PP0 pid=1597838) INFO 07-26 20:27:28 [jit_monitor.py:73] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. |
| (Worker_PP1 pid=1597839) INFO 07-26 20:27:28 [jit_monitor.py:73] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. |
| (EngineCore pid=1597716) INFO 07-26 20:27:28 [core.py:344] init engine (profile, create kv cache, warmup model) took 2.44 s |
| (EngineCore pid=1597716) INFO 07-26 20:27:31 [vllm.py:1042] Asynchronous scheduling is enabled. |
| (EngineCore pid=1597716) WARNING 07-26 20:27:31 [vllm.py:1096] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none |
| (EngineCore pid=1597716) WARNING 07-26 20:27:31 [vllm.py:1144] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. |
| (EngineCore pid=1597716) INFO 07-26 20:27:31 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) |
| (EngineCore pid=1597716) INFO 07-26 20:27:31 [vllm.py:1322] Cudagraph is disabled under eager mode |
| (EngineCore pid=1597716) INFO 07-26 20:27:31 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant |
| (APIServer pid=1597600) INFO 07-26 20:27:31 [api_server.py:612] Supported tasks: ['generate'] |
| (APIServer pid=1597600) INFO 07-26 20:27:32 [hf.py:548] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this. |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [api_server.py:616] Starting vLLM server on http://127.0.0.1:8398 |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:37] Available routes are: |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /docs, Methods: HEAD, GET |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /redoc, Methods: HEAD, GET |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /load, Methods: GET |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /version, Methods: GET |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /health, Methods: GET |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /metrics, Methods: GET |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /tokenize, Methods: POST |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /detokenize, Methods: POST |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /v1/models, Methods: GET |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /ping, Methods: GET |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /ping, Methods: POST |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /invocations, Methods: POST |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /v1/chat/completions, Methods: POST |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /v1/responses, Methods: POST |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /v1/completions, Methods: POST |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /v1/messages, Methods: POST |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /generative_scoring, Methods: POST |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /v1/completions/render, Methods: POST |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /v1/completions/derender, Methods: POST |
| (APIServer pid=1597600) INFO 07-26 20:27:33 [launcher.py:46] Route: /inference/v1/generate, Methods: POST |
| (APIServer pid=1597600) INFO: Started server process [1597600] |
| (APIServer pid=1597600) INFO: Waiting for application startup. |
| (APIServer pid=1597600) INFO: Application startup complete. |
| (APIServer pid=1597600) INFO: 127.0.0.1:37108 - "GET /health HTTP/1.1" 200 OK |
| (Worker_PP0 pid=1597838) WARNING 07-26 20:27:40 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _zero_kv_blocks_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (Worker_PP0 pid=1597838) WARNING 07-26 20:27:40 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (Worker_PP0 pid=1597838) WARNING 07-26 20:27:41 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _causal_conv1d_fwd_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (Worker_PP0 pid=1597838) WARNING 07-26 20:27:41 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _fused_post_conv_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (Worker_PP0 pid=1597838) WARNING 07-26 20:27:41 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _plan_tiles. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (Worker_PP0 pid=1597838) WARNING 07-26 20:27:41 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _gate_up_silu. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (Worker_PP0 pid=1597838) WARNING 07-26 20:27:41 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _down_proj. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (Worker_PP0 pid=1597838) /home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/distributed/parallel_state.py:813: UserWarning: The given buffer is not writable, and PyTorch does not support non-writable tensors. This means you can write to the underlying (supposedly non-writable) buffer using the tensor. You may want to copy the buffer to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /pytorch/torch/csrc/utils/tensor_new.cpp:1586.) |
| (Worker_PP0 pid=1597838) object_tensor = torch.frombuffer(pickle.dumps(obj), dtype=torch.uint8) |
| [rank0]:[W726 20:27:41.326140621 ProcessGroupNCCL.cpp:4094] Warning: An unbatched P2P op (send/recv) was called on this ProcessGroup with size 2. In lazy initialization mode, this will result in a new 2-rank NCCL communicator to be created. (function operator()) |
| [rank1]:[W726 20:27:41.326392459 ProcessGroupNCCL.cpp:4094] Warning: An unbatched P2P op (send/recv) was called on this ProcessGroup with size 2. In lazy initialization mode, this will result in a new 2-rank NCCL communicator to be created. (function operator()) |
| (Worker_PP0 pid=1597838) WARNING 07-26 20:27:41 [jit_monitor.py:129] Triton kernel JIT compilation during inference: fused_sigmoid_gating_delta_rule_update_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (Worker_PP0 pid=1597838) WARNING 07-26 20:27:41 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _count_experts. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (Worker_PP0 pid=1597838) WARNING 07-26 20:27:41 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _scatter_routes. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (Worker_PP0 pid=1597838) WARNING 07-26 20:27:42 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _causal_conv1d_update_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (Worker_PP0 pid=1597838) WARNING 07-26 20:27:42 [jit_monitor.py:129] Triton kernel JIT compilation during inference: fused_recurrent_gated_delta_rule_packed_decode_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (Worker_PP0 pid=1597838) WARNING 07-26 20:27:42 [jit_monitor.py:129] Triton kernel JIT compilation during inference: layer_norm_fwd_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. |
| (APIServer pid=1597600) INFO 07-26 20:27:43 [loggers.py:273] Engine 000: Avg prompt throughput: 152.1 tokens/s, Avg generation throughput: 6.1 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:40610 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:40580 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:27:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 218.3 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:40602 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:40558 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:40594 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:40584 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:28:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 152.4 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:40618 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:28:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 76.4 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:40572 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:28:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 60.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:28:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 50.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:28:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 50.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:28:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 50.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:40550 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:29:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:29:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 27.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:29:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 26.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:29:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 26.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:29:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 27.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:29:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 27.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:30:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 27.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:30:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 27.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:30:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 27.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:40542 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:30:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 14.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:30:43 [loggers.py:273] Engine 000: Avg prompt throughput: 376.5 tokens/s, Avg generation throughput: 337.1 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:30:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 393.6 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53956 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:31:03 [loggers.py:273] Engine 000: Avg prompt throughput: 20.5 tokens/s, Avg generation throughput: 394.9 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53870 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:31:13 [loggers.py:273] Engine 000: Avg prompt throughput: 76.1 tokens/s, Avg generation throughput: 372.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:31:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 396.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:31:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 396.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:31:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 393.6 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:31:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 392.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:32:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 392.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:32:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 395.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:32:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 391.9 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:32:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 392.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:32:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 391.9 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:32:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 395.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:33:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 390.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:33:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 393.6 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:33:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 393.6 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 30.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:33:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 392.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 31.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:33:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 392.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:33:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 396.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 34.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:34:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 393.6 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 35.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:34:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 387.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 37.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:34:23 [loggers.py:273] Engine 000: Avg prompt throughput: 16.7 tokens/s, Avg generation throughput: 374.1 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 36.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:34:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 382.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 37.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:34:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 388.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 38.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53968 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:34:53 [loggers.py:273] Engine 000: Avg prompt throughput: 278.4 tokens/s, Avg generation throughput: 349.1 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:35:03 [loggers.py:273] Engine 000: Avg prompt throughput: 19.3 tokens/s, Avg generation throughput: 374.1 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53956 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:35:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 379.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:35:23 [loggers.py:273] Engine 000: Avg prompt throughput: 56.9 tokens/s, Avg generation throughput: 382.1 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53870 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:35:33 [loggers.py:273] Engine 000: Avg prompt throughput: 118.5 tokens/s, Avg generation throughput: 365.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:35:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 388.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:35:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 387.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:36:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 387.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:36:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 393.6 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:36:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 395.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:36:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 385.6 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:36:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 382.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:36:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 384.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:37:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 384.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:37:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 387.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:37:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 380.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:37:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 382.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:37:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 390.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 30.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:37:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 395.1 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 31.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:38:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 396.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:38:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 396.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:38:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 398.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 35.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:38:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 398.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 36.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:38:43 [loggers.py:273] Engine 000: Avg prompt throughput: 16.7 tokens/s, Avg generation throughput: 396.5 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 36.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:38:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 398.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 37.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53968 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:39:03 [loggers.py:273] Engine 000: Avg prompt throughput: 228.2 tokens/s, Avg generation throughput: 372.3 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:39:13 [loggers.py:273] Engine 000: Avg prompt throughput: 16.7 tokens/s, Avg generation throughput: 390.1 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:39:23 [loggers.py:273] Engine 000: Avg prompt throughput: 16.3 tokens/s, Avg generation throughput: 386.9 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53956 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:39:33 [loggers.py:273] Engine 000: Avg prompt throughput: 36.5 tokens/s, Avg generation throughput: 377.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53870 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:39:43 [loggers.py:273] Engine 000: Avg prompt throughput: 78.8 tokens/s, Avg generation throughput: 383.1 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:39:53 [loggers.py:273] Engine 000: Avg prompt throughput: 69.0 tokens/s, Avg generation throughput: 389.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:40:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 393.6 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:40:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 393.6 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:40:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 387.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:40:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 385.6 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:40:43 [loggers.py:273] Engine 000: Avg prompt throughput: 27.6 tokens/s, Avg generation throughput: 383.7 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:40:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 387.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:41:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 384.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:41:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 382.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:41:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 382.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:41:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 395.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:41:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 395.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:41:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 396.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:42:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 398.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:42:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 396.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 30.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:42:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 398.3 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:42:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 396.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:42:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 395.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 35.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:42:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 396.6 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:43:03 [loggers.py:273] Engine 000: Avg prompt throughput: 16.7 tokens/s, Avg generation throughput: 391.9 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 35.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53968 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:43:13 [loggers.py:273] Engine 000: Avg prompt throughput: 131.6 tokens/s, Avg generation throughput: 383.6 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:43:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 395.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:43:33 [loggers.py:273] Engine 000: Avg prompt throughput: 69.0 tokens/s, Avg generation throughput: 391.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53956 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:43:43 [loggers.py:273] Engine 000: Avg prompt throughput: 44.3 tokens/s, Avg generation throughput: 391.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53870 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:43:53 [loggers.py:273] Engine 000: Avg prompt throughput: 33.5 tokens/s, Avg generation throughput: 391.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:44:03 [loggers.py:273] Engine 000: Avg prompt throughput: 98.1 tokens/s, Avg generation throughput: 382.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:44:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 390.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53956 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:44:23 [loggers.py:273] Engine 000: Avg prompt throughput: 69.0 tokens/s, Avg generation throughput: 386.5 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:44:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 388.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:44:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 393.6 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:44:53 [loggers.py:273] Engine 000: Avg prompt throughput: 27.6 tokens/s, Avg generation throughput: 396.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:45:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 396.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:45:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 396.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:45:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 393.6 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:45:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 395.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:45:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 395.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:45:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 395.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:46:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 393.6 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:46:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 396.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:46:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 398.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:46:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 398.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 31.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:46:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 395.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:46:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 398.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 34.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:47:03 [loggers.py:273] Engine 000: Avg prompt throughput: 16.7 tokens/s, Avg generation throughput: 393.3 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:47:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 395.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 35.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53956 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53968 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:47:23 [loggers.py:273] Engine 000: Avg prompt throughput: 131.6 tokens/s, Avg generation throughput: 385.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:47:33 [loggers.py:273] Engine 000: Avg prompt throughput: 20.5 tokens/s, Avg generation throughput: 391.6 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:47:43 [loggers.py:273] Engine 000: Avg prompt throughput: 48.5 tokens/s, Avg generation throughput: 390.1 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:47:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 390.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53870 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:48:03 [loggers.py:273] Engine 000: Avg prompt throughput: 44.3 tokens/s, Avg generation throughput: 385.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:48:13 [loggers.py:273] Engine 000: Avg prompt throughput: 69.7 tokens/s, Avg generation throughput: 384.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:48:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 398.3 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:48:33 [loggers.py:273] Engine 000: Avg prompt throughput: 61.9 tokens/s, Avg generation throughput: 393.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:48:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 398.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:48:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 396.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53870 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:49:03 [loggers.py:273] Engine 000: Avg prompt throughput: 69.0 tokens/s, Avg generation throughput: 393.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:49:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 396.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:49:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 396.7 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:49:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 395.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:49:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 395.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.9%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:49:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 393.6 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:50:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 396.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:50:13 [loggers.py:273] Engine 000: Avg prompt throughput: 27.6 tokens/s, Avg generation throughput: 394.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:50:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 395.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:50:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 398.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:50:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 398.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 30.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:50:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 398.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 31.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:51:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 398.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:51:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 400.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:51:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 398.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 35.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53956 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53892 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53900 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53914 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53968 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53996 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:51:33 [loggers.py:273] Engine 000: Avg prompt throughput: 16.7 tokens/s, Avg generation throughput: 377.3 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:51:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 275.0 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53982 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:51:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 257.4 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:52:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 252.0 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53956 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53938 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:52:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 227.4 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.6%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53924 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53874 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53888 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:52:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 157.5 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.3%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53994 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:52:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 125.0 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53944 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:52:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 76.8 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:52:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 75.6 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53870 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:53984 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:53:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 66.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:53:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 27.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:53:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 27.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:53:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 27.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:53:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 27.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:53:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 26.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:54:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 26.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:53918 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:54:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 14.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:55860 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:55884 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:55864 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:54:23 [loggers.py:273] Engine 000: Avg prompt throughput: 61.8 tokens/s, Avg generation throughput: 138.9 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:55880 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:55852 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:55848 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:54:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 101.2 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO 07-26 20:54:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 74.7 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:55886 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO: 127.0.0.1:55836 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:54:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 44.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1597600) INFO: 127.0.0.1:55826 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
| (APIServer pid=1597600) INFO 07-26 20:54:59 [launcher.py:100] [shutdown] API server: shutdown triggered |
| (APIServer pid=1597600) INFO 07-26 20:54:59 [launcher.py:116] [shutdown] API server: stopping engine client mode=abort timeout=0s |
| (APIServer pid=1597600) INFO 07-26 20:54:59 [core_client.py:655] [shutdown] MPClient: start timeout=0s |
| (APIServer pid=1597600) INFO 07-26 20:54:59 [core_client.py:657] [shutdown] MPClient: stopping engine manager |
| (APIServer pid=1597600) WARNING 07-26 20:54:59 [utils.py:626] [shutdown] Process manager: force killing remaining processes count=1 |
| (EngineCore pid=1597716) INFO 07-26 20:54:59 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM |
| (EngineCore pid=1597716) INFO 07-26 20:54:59 [core.py:1333] [shutdown] EngineCore: start mode=abort timeout=0s |
| (EngineCore pid=1597716) INFO 07-26 20:54:59 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown |
| (EngineCore pid=1597716) INFO 07-26 20:54:59 [core.py:1227] [shutdown] EngineCore: exiting busy loop |
| (Worker_PP0 pid=1597838) INFO 07-26 20:54:59 [multiproc_executor.py:790] Parent process exited, terminating worker queues |
| (EngineCore pid=1597716) INFO 07-26 20:54:59 [multiproc_executor.py:426] [shutdown] Executor: waiting for worker exit count=2 |
| (APIServer pid=1597600) INFO 07-26 20:54:59 [core_client.py:659] [shutdown] MPClient: engine manager stopped |
| (APIServer pid=1597600) INFO 07-26 20:54:59 [core_client.py:660] [shutdown] MPClient: cleaning up background resources |
| (APIServer pid=1597600) INFO 07-26 20:54:59 [core_client.py:662] [shutdown] MPClient: complete |
| (APIServer pid=1597600) INFO 07-26 20:54:59 [launcher.py:125] [shutdown] API server: engine client stopped |
| (APIServer pid=1597600) INFO 07-26 20:54:59 [launcher.py:128] [shutdown] API server: signalling HTTP server shutdown |
| (APIServer pid=1597600) INFO 07-26 20:54:59 [launcher.py:149] [shutdown] API server: shutting down FastAPI HTTP server |
| (APIServer pid=1597600) INFO: Shutting down |
| (APIServer pid=1597600) INFO: Waiting for application shutdown. |
| (APIServer pid=1597600) INFO: Application shutdown complete. |
| /home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 2 leaked semaphore objects to clean up at shutdown |
| warnings.warn('resource_tracker: There appear to be %d ' |
| /home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked shared_memory objects to clean up at shutdown |
| warnings.warn('resource_tracker: There appear to be %d ' |
|
|
