(APIServer pid=1162132) INFO 07-24 00:45:09 [api_utils.py:339] (APIServer pid=1162132) INFO 07-24 00:45:09 [api_utils.py:339] █ █ █▄ ▄█ (APIServer pid=1162132) INFO 07-24 00:45:09 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.25.0 (APIServer pid=1162132) INFO 07-24 00:45:09 [api_utils.py:339] █▄█▀ █ █ █ █ model outputs/healed_dl/keep25_step0200 (APIServer pid=1162132) INFO 07-24 00:45:09 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ (APIServer pid=1162132) INFO 07-24 00:45:09 [api_utils.py:339] (APIServer pid=1162132) INFO 07-24 00:45:09 [api_utils.py:273] non-default args: {'model_tag': 'outputs/healed_dl/keep25_step0200', 'default_chat_template_kwargs': {'enable_thinking': False}, 'host': '127.0.0.1', 'port': 8399, 'model': 'outputs/healed_dl/keep25_step0200', 'max_model_len': 8192, 'enforce_eager': True, 'served_model_name': ['student'], 'pipeline_parallel_size': 2, 'gpu_memory_utilization': 0.9} (APIServer pid=1162132) INFO 07-24 00:45:09 [model.py:619] Resolved architecture: PrunedQwen3_5MoeForCausalLM (APIServer pid=1162132) INFO 07-24 00:45:09 [model.py:1776] Using max model len 8192 (APIServer pid=1162132) INFO 07-24 00:45:09 [vllm.py:1042] Asynchronous scheduling is enabled. (APIServer pid=1162132) WARNING 07-24 00:45:09 [vllm.py:1096] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none (APIServer pid=1162132) WARNING 07-24 00:45:09 [vllm.py:1144] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. (APIServer pid=1162132) INFO 07-24 00:45:09 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) (APIServer pid=1162132) INFO 07-24 00:45:09 [vllm.py:1322] Cudagraph is disabled under eager mode (APIServer pid=1162132) INFO 07-24 00:45:09 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant (EngineCore pid=1162266) INFO 07-24 00:45:21 [core.py:114] Initializing a V1 LLM engine (v0.25.0) with config: model='outputs/healed_dl/keep25_step0200', speculative_config=None, tokenizer='outputs/healed_dl/keep25_step0200', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=2, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=student, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, moe_backend='auto', linear_backend='auto') (EngineCore pid=1162266) WARNING 07-24 00:45:21 [multiproc_executor.py:1067] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. (EngineCore pid=1162266) INFO 07-24 00:45:21 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=192.168.0.15 (local), world_size=2, local_world_size=2 (Worker pid=1162380) INFO 07-24 00:45:29 [parallel_state.py:1607] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:36663 backend=nccl (Worker pid=1162381) INFO 07-24 00:45:29 [parallel_state.py:1607] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:36663 backend=nccl (Worker pid=1162380) INFO 07-24 00:45:30 [pynccl.py:113] vLLM is using nccl==2.28.9 (Worker pid=1162380) INFO 07-24 00:45:30 [cuda_communicator.py:264] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'pp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'AITER_CUSTOM', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. (Worker pid=1162380) INFO 07-24 00:45:31 [parallel_state.py:1942] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A (Worker pid=1162380) INFO 07-24 00:45:31 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. (Worker_PP0 pid=1162380) INFO 07-24 00:45:31 [gpu_model_runner.py:5209] Starting to load model outputs/healed_dl/keep25_step0200... (Worker_PP0 pid=1162380) INFO 07-24 00:45:31 [qwen_gdn_linear_attn.py:228] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128). (Worker_PP1 pid=1162381) /home/henry/.cache/glean/megablocks-variable-93a1479bc15b/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available. (Worker_PP1 pid=1162381) warnings.warn('Grouped GEMM not available.') (Worker_PP0 pid=1162380) /home/henry/.cache/glean/megablocks-variable-93a1479bc15b/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available. (Worker_PP0 pid=1162380) warnings.warn('Grouped GEMM not available.') (Worker_PP0 pid=1162380) INFO 07-24 00:45:31 [cuda.py:476] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. (Worker_PP0 pid=1162380) INFO 07-24 00:45:31 [flash_attn.py:718] Using FlashAttention version 2 (Worker_PP0 pid=1162380) INFO 07-24 00:45:32 [weight_utils.py:849] Filesystem type for checkpoints: EXT4. Checkpoint size: 19.54 GiB. Available RAM: 106.72 GiB. (Worker_PP0 pid=1162380) INFO 07-24 00:45:32 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. (Worker_PP0 pid=1162380) Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00= mamba page size. (Worker_PP0 pid=1162380) INFO 07-24 00:45:38 [interface.py:914] Padding mamba page size by 1.49% to ensure that mamba page size and attention page size are exactly equal. (Worker_PP1 pid=1162381) INFO 07-24 00:45:38 [interface.py:890] Setting attention block size to 544 tokens to ensure that attention page size is >= mamba page size. (Worker_PP1 pid=1162381) INFO 07-24 00:45:38 [interface.py:914] Padding mamba page size by 1.49% to ensure that mamba page size and attention page size are exactly equal. (Worker_PP0 pid=1162380) INFO 07-24 00:45:40 [gpu_worker.py:538] Available KV cache memory: 11.07 GiB (EngineCore pid=1162266) INFO 07-24 00:45:40 [kv_cache_utils.py:2146] GPU KV cache size: 781,258 tokens (EngineCore pid=1162266) INFO 07-24 00:45:40 [kv_cache_utils.py:2147] Maximum concurrency for 8,192 tokens per request: 95.37x (Worker_PP1 pid=1162381) INFO 07-24 00:45:40 [cutedsl_warmup.py:97] Skipping CuTeDSL warmup because no compile units were requested. (Worker_PP0 pid=1162380) INFO 07-24 00:45:40 [cutedsl_warmup.py:97] Skipping CuTeDSL warmup because no compile units were requested. (Worker_PP0 pid=1162380) INFO 07-24 00:45:40 [jit_monitor.py:73] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. (Worker_PP1 pid=1162381) INFO 07-24 00:45:40 [jit_monitor.py:73] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. (EngineCore pid=1162266) INFO 07-24 00:45:41 [core.py:344] init engine (profile, create kv cache, warmup model) took 2.39 s (EngineCore pid=1162266) INFO 07-24 00:45:43 [vllm.py:1042] Asynchronous scheduling is enabled. (EngineCore pid=1162266) WARNING 07-24 00:45:43 [vllm.py:1096] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none (EngineCore pid=1162266) WARNING 07-24 00:45:43 [vllm.py:1144] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. (EngineCore pid=1162266) INFO 07-24 00:45:43 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) (EngineCore pid=1162266) INFO 07-24 00:45:43 [vllm.py:1322] Cudagraph is disabled under eager mode (EngineCore pid=1162266) INFO 07-24 00:45:43 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant (APIServer pid=1162132) INFO 07-24 00:45:43 [api_server.py:612] Supported tasks: ['generate'] (APIServer pid=1162132) INFO 07-24 00:45:45 [hf.py:548] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this. (APIServer pid=1162132) INFO 07-24 00:45:45 [api_server.py:616] Starting vLLM server on http://127.0.0.1:8399 (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:37] Available routes are: (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /docs, Methods: HEAD, GET (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /redoc, Methods: HEAD, GET (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /load, Methods: GET (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /version, Methods: GET (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /health, Methods: GET (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /metrics, Methods: GET (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /tokenize, Methods: POST (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /detokenize, Methods: POST (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /v1/models, Methods: GET (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /ping, Methods: GET (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /ping, Methods: POST (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /invocations, Methods: POST (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /v1/chat/completions, Methods: POST (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /v1/responses, Methods: POST (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /v1/completions, Methods: POST (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /v1/messages, Methods: POST (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /generative_scoring, Methods: POST (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /v1/completions/render, Methods: POST (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /v1/completions/derender, Methods: POST (APIServer pid=1162132) INFO 07-24 00:45:45 [launcher.py:46] Route: /inference/v1/generate, Methods: POST (APIServer pid=1162132) INFO: Started server process [1162132] (APIServer pid=1162132) INFO: Waiting for application startup. (APIServer pid=1162132) INFO: Application startup complete. (APIServer pid=1162132) INFO: 127.0.0.1:39172 - "GET /health HTTP/1.1" 200 OK (Worker_PP0 pid=1162380) WARNING 07-24 00:46:18 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _zero_kv_blocks_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (Worker_PP0 pid=1162380) WARNING 07-24 00:46:18 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (Worker_PP0 pid=1162380) WARNING 07-24 00:46:18 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _causal_conv1d_fwd_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (Worker_PP0 pid=1162380) WARNING 07-24 00:46:18 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _fused_post_conv_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (Worker_PP0 pid=1162380) WARNING 07-24 00:46:18 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _build_route_rows. This causes a latency spike; consider extending warmup to cover this shape/config. (Worker_PP0 pid=1162380) /home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/distributed/parallel_state.py:813: UserWarning: The given buffer is not writable, and PyTorch does not support non-writable tensors. This means you can write to the underlying (supposedly non-writable) buffer using the tensor. You may want to copy the buffer to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /pytorch/torch/csrc/utils/tensor_new.cpp:1586.) (Worker_PP0 pid=1162380) object_tensor = torch.frombuffer(pickle.dumps(obj), dtype=torch.uint8) [rank0]:[W724 00:46:18.064289498 ProcessGroupNCCL.cpp:4094] Warning: An unbatched P2P op (send/recv) was called on this ProcessGroup with size 2. In lazy initialization mode, this will result in a new 2-rank NCCL communicator to be created. (function operator()) [rank1]:[W724 00:46:18.064578297 ProcessGroupNCCL.cpp:4094] Warning: An unbatched P2P op (send/recv) was called on this ProcessGroup with size 2. In lazy initialization mode, this will result in a new 2-rank NCCL communicator to be created. (function operator()) (Worker_PP0 pid=1162380) WARNING 07-24 00:46:18 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _causal_conv1d_update_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (Worker_PP0 pid=1162380) WARNING 07-24 00:46:18 [jit_monitor.py:129] Triton kernel JIT compilation during inference: fused_recurrent_gated_delta_rule_packed_decode_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (Worker_PP0 pid=1162380) WARNING 07-24 00:46:18 [jit_monitor.py:129] Triton kernel JIT compilation during inference: layer_norm_fwd_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (Worker_PP0 pid=1162380) WARNING 07-24 00:46:18 [jit_monitor.py:129] Triton kernel JIT compilation during inference: fused_sigmoid_gating_delta_rule_update_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (APIServer pid=1162132) INFO 07-24 00:46:25 [loggers.py:273] Engine 000: Avg prompt throughput: 4745.1 tokens/s, Avg generation throughput: 37.8 tokens/s, Running: 32 reqs, Waiting: 32 reqs, GPU KV cache usage: 10.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:46:35 [loggers.py:273] Engine 000: Avg prompt throughput: 6136.4 tokens/s, Avg generation throughput: 228.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:46:45 [loggers.py:273] Engine 000: Avg prompt throughput: 4124.9 tokens/s, Avg generation throughput: 345.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:46:55 [loggers.py:273] Engine 000: Avg prompt throughput: 3015.1 tokens/s, Avg generation throughput: 431.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:47:05 [loggers.py:273] Engine 000: Avg prompt throughput: 3399.7 tokens/s, Avg generation throughput: 411.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:47:15 [loggers.py:273] Engine 000: Avg prompt throughput: 2657.1 tokens/s, Avg generation throughput: 464.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:47:25 [loggers.py:273] Engine 000: Avg prompt throughput: 2242.8 tokens/s, Avg generation throughput: 489.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:47:35 [loggers.py:273] Engine 000: Avg prompt throughput: 2380.3 tokens/s, Avg generation throughput: 489.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:47:45 [loggers.py:273] Engine 000: Avg prompt throughput: 2463.2 tokens/s, Avg generation throughput: 483.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:47:55 [loggers.py:273] Engine 000: Avg prompt throughput: 2574.9 tokens/s, Avg generation throughput: 450.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:48:05 [loggers.py:273] Engine 000: Avg prompt throughput: 3073.4 tokens/s, Avg generation throughput: 444.4 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:48:15 [loggers.py:273] Engine 000: Avg prompt throughput: 2207.5 tokens/s, Avg generation throughput: 509.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:48:25 [loggers.py:273] Engine 000: Avg prompt throughput: 2753.9 tokens/s, Avg generation throughput: 450.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:48:35 [loggers.py:273] Engine 000: Avg prompt throughput: 3078.1 tokens/s, Avg generation throughput: 437.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:48:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1919.4 tokens/s, Avg generation throughput: 548.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:48:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1728.5 tokens/s, Avg generation throughput: 560.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:49:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1716.3 tokens/s, Avg generation throughput: 554.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:49:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1492.9 tokens/s, Avg generation throughput: 561.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:49:25 [loggers.py:273] Engine 000: Avg prompt throughput: 2886.3 tokens/s, Avg generation throughput: 457.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:49:35 [loggers.py:273] Engine 000: Avg prompt throughput: 2338.4 tokens/s, Avg generation throughput: 495.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:49:45 [loggers.py:273] Engine 000: Avg prompt throughput: 2760.1 tokens/s, Avg generation throughput: 470.0 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:49:55 [loggers.py:273] Engine 000: Avg prompt throughput: 2878.7 tokens/s, Avg generation throughput: 463.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:50:05 [loggers.py:273] Engine 000: Avg prompt throughput: 2782.1 tokens/s, Avg generation throughput: 450.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:50:15 [loggers.py:273] Engine 000: Avg prompt throughput: 2617.0 tokens/s, Avg generation throughput: 476.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:50:25 [loggers.py:273] Engine 000: Avg prompt throughput: 2147.0 tokens/s, Avg generation throughput: 503.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:50:35 [loggers.py:273] Engine 000: Avg prompt throughput: 2076.1 tokens/s, Avg generation throughput: 528.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:50:45 [loggers.py:273] Engine 000: Avg prompt throughput: 2665.9 tokens/s, Avg generation throughput: 463.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:50:55 [loggers.py:273] Engine 000: Avg prompt throughput: 2014.1 tokens/s, Avg generation throughput: 528.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:51:05 [loggers.py:273] Engine 000: Avg prompt throughput: 2231.6 tokens/s, Avg generation throughput: 522.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:51:15 [loggers.py:273] Engine 000: Avg prompt throughput: 2547.3 tokens/s, Avg generation throughput: 463.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:51:25 [loggers.py:273] Engine 000: Avg prompt throughput: 2395.1 tokens/s, Avg generation throughput: 508.7 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:51:35 [loggers.py:273] Engine 000: Avg prompt throughput: 2679.8 tokens/s, Avg generation throughput: 476.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:51:45 [loggers.py:273] Engine 000: Avg prompt throughput: 2157.6 tokens/s, Avg generation throughput: 515.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:51:55 [loggers.py:273] Engine 000: Avg prompt throughput: 2379.4 tokens/s, Avg generation throughput: 489.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:52:05 [loggers.py:273] Engine 000: Avg prompt throughput: 2315.1 tokens/s, Avg generation throughput: 496.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:52:15 [loggers.py:273] Engine 000: Avg prompt throughput: 2180.4 tokens/s, Avg generation throughput: 509.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:52:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1748.2 tokens/s, Avg generation throughput: 567.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:52:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1611.4 tokens/s, Avg generation throughput: 554.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:52:45 [loggers.py:273] Engine 000: Avg prompt throughput: 2103.5 tokens/s, Avg generation throughput: 515.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:52:55 [loggers.py:273] Engine 000: Avg prompt throughput: 3517.0 tokens/s, Avg generation throughput: 417.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:53:05 [loggers.py:273] Engine 000: Avg prompt throughput: 2511.1 tokens/s, Avg generation throughput: 483.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:53:15 [loggers.py:273] Engine 000: Avg prompt throughput: 2678.7 tokens/s, Avg generation throughput: 457.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:53:25 [loggers.py:273] Engine 000: Avg prompt throughput: 2742.7 tokens/s, Avg generation throughput: 489.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:53:35 [loggers.py:273] Engine 000: Avg prompt throughput: 2495.4 tokens/s, Avg generation throughput: 495.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:53:45 [loggers.py:273] Engine 000: Avg prompt throughput: 3650.9 tokens/s, Avg generation throughput: 389.2 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:53:55 [loggers.py:273] Engine 000: Avg prompt throughput: 2137.0 tokens/s, Avg generation throughput: 521.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:54:05 [loggers.py:273] Engine 000: Avg prompt throughput: 2388.8 tokens/s, Avg generation throughput: 500.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:54:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1973.2 tokens/s, Avg generation throughput: 553.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:54:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1931.8 tokens/s, Avg generation throughput: 527.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:54:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1553.4 tokens/s, Avg generation throughput: 580.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:54:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1531.1 tokens/s, Avg generation throughput: 573.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:54:55 [loggers.py:273] Engine 000: Avg prompt throughput: 2290.0 tokens/s, Avg generation throughput: 521.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:55:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1277.1 tokens/s, Avg generation throughput: 599.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:55:15 [loggers.py:273] Engine 000: Avg prompt throughput: 588.6 tokens/s, Avg generation throughput: 651.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:55:25 [loggers.py:273] Engine 000: Avg prompt throughput: 970.9 tokens/s, Avg generation throughput: 619.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:55:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1322.9 tokens/s, Avg generation throughput: 592.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:55:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1722.0 tokens/s, Avg generation throughput: 547.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:55:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1562.2 tokens/s, Avg generation throughput: 566.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:56:05 [loggers.py:273] Engine 000: Avg prompt throughput: 842.3 tokens/s, Avg generation throughput: 606.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:56:15 [loggers.py:273] Engine 000: Avg prompt throughput: 2281.6 tokens/s, Avg generation throughput: 507.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:56:25 [loggers.py:273] Engine 000: Avg prompt throughput: 2465.0 tokens/s, Avg generation throughput: 513.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:56:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1272.5 tokens/s, Avg generation throughput: 580.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:56:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1772.0 tokens/s, Avg generation throughput: 560.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:56:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1614.9 tokens/s, Avg generation throughput: 566.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:57:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1466.8 tokens/s, Avg generation throughput: 585.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:57:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1624.8 tokens/s, Avg generation throughput: 566.5 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:57:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1505.7 tokens/s, Avg generation throughput: 580.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:57:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1315.1 tokens/s, Avg generation throughput: 586.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:57:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1276.7 tokens/s, Avg generation throughput: 592.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:57:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1812.0 tokens/s, Avg generation throughput: 553.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:58:05 [loggers.py:273] Engine 000: Avg prompt throughput: 623.1 tokens/s, Avg generation throughput: 651.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:58:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1405.6 tokens/s, Avg generation throughput: 580.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:58:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1307.8 tokens/s, Avg generation throughput: 586.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:58:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1753.0 tokens/s, Avg generation throughput: 572.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:58:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1924.5 tokens/s, Avg generation throughput: 533.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:58:55 [loggers.py:273] Engine 000: Avg prompt throughput: 2097.1 tokens/s, Avg generation throughput: 546.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:59:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1443.3 tokens/s, Avg generation throughput: 573.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:59:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1400.4 tokens/s, Avg generation throughput: 579.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:59:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1326.2 tokens/s, Avg generation throughput: 599.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:59:35 [loggers.py:273] Engine 000: Avg prompt throughput: 2099.0 tokens/s, Avg generation throughput: 540.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:59:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1757.8 tokens/s, Avg generation throughput: 559.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 00:59:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1517.3 tokens/s, Avg generation throughput: 586.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:00:05 [loggers.py:273] Engine 000: Avg prompt throughput: 2022.3 tokens/s, Avg generation throughput: 533.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:00:15 [loggers.py:273] Engine 000: Avg prompt throughput: 2565.4 tokens/s, Avg generation throughput: 494.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:00:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1667.2 tokens/s, Avg generation throughput: 560.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:00:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1401.3 tokens/s, Avg generation throughput: 580.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:00:45 [loggers.py:273] Engine 000: Avg prompt throughput: 2763.4 tokens/s, Avg generation throughput: 494.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:00:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1516.0 tokens/s, Avg generation throughput: 566.7 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:01:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1956.3 tokens/s, Avg generation throughput: 560.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:01:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1737.4 tokens/s, Avg generation throughput: 553.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:01:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1665.2 tokens/s, Avg generation throughput: 572.9 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:01:35 [loggers.py:273] Engine 000: Avg prompt throughput: 2825.3 tokens/s, Avg generation throughput: 468.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:01:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1900.7 tokens/s, Avg generation throughput: 553.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:01:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1901.1 tokens/s, Avg generation throughput: 553.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:02:05 [loggers.py:273] Engine 000: Avg prompt throughput: 2158.5 tokens/s, Avg generation throughput: 533.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:02:15 [loggers.py:273] Engine 000: Avg prompt throughput: 2636.1 tokens/s, Avg generation throughput: 494.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:02:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1697.2 tokens/s, Avg generation throughput: 560.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:02:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1817.2 tokens/s, Avg generation throughput: 541.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:02:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1882.0 tokens/s, Avg generation throughput: 548.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:02:55 [loggers.py:273] Engine 000: Avg prompt throughput: 920.4 tokens/s, Avg generation throughput: 600.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:03:05 [loggers.py:273] Engine 000: Avg prompt throughput: 890.9 tokens/s, Avg generation throughput: 632.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:03:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1672.6 tokens/s, Avg generation throughput: 548.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:03:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1104.7 tokens/s, Avg generation throughput: 587.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:03:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1199.5 tokens/s, Avg generation throughput: 593.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:03:45 [loggers.py:273] Engine 000: Avg prompt throughput: 332.1 tokens/s, Avg generation throughput: 658.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:03:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1347.6 tokens/s, Avg generation throughput: 586.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:04:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1202.4 tokens/s, Avg generation throughput: 600.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:04:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1057.9 tokens/s, Avg generation throughput: 594.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:04:25 [loggers.py:273] Engine 000: Avg prompt throughput: 724.8 tokens/s, Avg generation throughput: 613.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:04:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1042.7 tokens/s, Avg generation throughput: 613.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:04:45 [loggers.py:273] Engine 000: Avg prompt throughput: 688.2 tokens/s, Avg generation throughput: 632.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:04:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1777.5 tokens/s, Avg generation throughput: 541.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:05:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1222.0 tokens/s, Avg generation throughput: 587.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:05:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1556.0 tokens/s, Avg generation throughput: 561.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:05:25 [loggers.py:273] Engine 000: Avg prompt throughput: 711.2 tokens/s, Avg generation throughput: 626.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:05:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1376.3 tokens/s, Avg generation throughput: 574.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:05:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1384.7 tokens/s, Avg generation throughput: 567.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:05:55 [loggers.py:273] Engine 000: Avg prompt throughput: 870.7 tokens/s, Avg generation throughput: 613.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:06:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1035.8 tokens/s, Avg generation throughput: 600.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:06:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1353.9 tokens/s, Avg generation throughput: 580.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:06:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1060.9 tokens/s, Avg generation throughput: 600.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:06:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1238.7 tokens/s, Avg generation throughput: 587.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:06:45 [loggers.py:273] Engine 000: Avg prompt throughput: 843.0 tokens/s, Avg generation throughput: 619.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:06:55 [loggers.py:273] Engine 000: Avg prompt throughput: 870.0 tokens/s, Avg generation throughput: 613.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:07:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1308.5 tokens/s, Avg generation throughput: 573.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:07:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1372.6 tokens/s, Avg generation throughput: 580.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:07:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1201.8 tokens/s, Avg generation throughput: 593.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:07:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1715.2 tokens/s, Avg generation throughput: 541.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:07:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1167.3 tokens/s, Avg generation throughput: 593.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:07:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1361.5 tokens/s, Avg generation throughput: 574.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:08:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1204.2 tokens/s, Avg generation throughput: 594.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:08:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1021.5 tokens/s, Avg generation throughput: 606.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:08:25 [loggers.py:273] Engine 000: Avg prompt throughput: 834.6 tokens/s, Avg generation throughput: 619.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:08:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1360.3 tokens/s, Avg generation throughput: 587.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:08:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1214.0 tokens/s, Avg generation throughput: 574.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:08:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1556.4 tokens/s, Avg generation throughput: 580.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:09:05 [loggers.py:273] Engine 000: Avg prompt throughput: 842.8 tokens/s, Avg generation throughput: 619.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:09:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1367.8 tokens/s, Avg generation throughput: 574.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:09:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1028.5 tokens/s, Avg generation throughput: 599.7 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:09:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1405.3 tokens/s, Avg generation throughput: 574.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:09:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1221.0 tokens/s, Avg generation throughput: 587.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:09:55 [loggers.py:273] Engine 000: Avg prompt throughput: 2057.6 tokens/s, Avg generation throughput: 522.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:10:05 [loggers.py:273] Engine 000: Avg prompt throughput: 539.3 tokens/s, Avg generation throughput: 652.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:10:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1543.3 tokens/s, Avg generation throughput: 555.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:10:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1362.1 tokens/s, Avg generation throughput: 574.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:10:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1217.7 tokens/s, Avg generation throughput: 593.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:10:45 [loggers.py:273] Engine 000: Avg prompt throughput: 885.4 tokens/s, Avg generation throughput: 619.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:10:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1356.7 tokens/s, Avg generation throughput: 561.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:11:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1229.1 tokens/s, Avg generation throughput: 606.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:11:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1349.6 tokens/s, Avg generation throughput: 587.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:11:25 [loggers.py:273] Engine 000: Avg prompt throughput: 540.4 tokens/s, Avg generation throughput: 632.3 tokens/s, Running: 63 reqs, Waiting: 1 reqs, GPU KV cache usage: 27.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:11:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1276.1 tokens/s, Avg generation throughput: 600.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:11:45 [loggers.py:273] Engine 000: Avg prompt throughput: 522.9 tokens/s, Avg generation throughput: 639.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:11:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1039.2 tokens/s, Avg generation throughput: 619.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:12:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1044.7 tokens/s, Avg generation throughput: 593.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:12:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1354.9 tokens/s, Avg generation throughput: 600.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:12:25 [loggers.py:273] Engine 000: Avg prompt throughput: 683.7 tokens/s, Avg generation throughput: 626.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:12:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1906.6 tokens/s, Avg generation throughput: 535.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:12:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1058.1 tokens/s, Avg generation throughput: 593.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:12:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1391.1 tokens/s, Avg generation throughput: 574.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:13:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1151.4 tokens/s, Avg generation throughput: 580.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:13:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1072.3 tokens/s, Avg generation throughput: 626.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:13:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1301.1 tokens/s, Avg generation throughput: 587.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:13:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1428.2 tokens/s, Avg generation throughput: 574.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:13:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1024.9 tokens/s, Avg generation throughput: 613.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:13:55 [loggers.py:273] Engine 000: Avg prompt throughput: 898.2 tokens/s, Avg generation throughput: 613.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:14:05 [loggers.py:273] Engine 000: Avg prompt throughput: 997.3 tokens/s, Avg generation throughput: 613.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:14:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1062.4 tokens/s, Avg generation throughput: 613.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:14:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1202.3 tokens/s, Avg generation throughput: 593.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:14:35 [loggers.py:273] Engine 000: Avg prompt throughput: 998.3 tokens/s, Avg generation throughput: 613.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:14:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1209.8 tokens/s, Avg generation throughput: 593.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:14:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1885.4 tokens/s, Avg generation throughput: 529.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:15:05 [loggers.py:273] Engine 000: Avg prompt throughput: 697.2 tokens/s, Avg generation throughput: 645.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:15:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1427.4 tokens/s, Avg generation throughput: 568.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:15:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1371.5 tokens/s, Avg generation throughput: 580.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:15:35 [loggers.py:273] Engine 000: Avg prompt throughput: 987.7 tokens/s, Avg generation throughput: 600.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:15:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1158.7 tokens/s, Avg generation throughput: 587.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:15:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1358.6 tokens/s, Avg generation throughput: 587.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:16:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1288.9 tokens/s, Avg generation throughput: 581.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:16:15 [loggers.py:273] Engine 000: Avg prompt throughput: 554.2 tokens/s, Avg generation throughput: 652.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:16:25 [loggers.py:273] Engine 000: Avg prompt throughput: 863.3 tokens/s, Avg generation throughput: 626.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:16:35 [loggers.py:273] Engine 000: Avg prompt throughput: 847.6 tokens/s, Avg generation throughput: 626.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:16:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1055.4 tokens/s, Avg generation throughput: 606.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:16:55 [loggers.py:273] Engine 000: Avg prompt throughput: 675.0 tokens/s, Avg generation throughput: 645.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:17:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1843.9 tokens/s, Avg generation throughput: 541.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:17:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1416.5 tokens/s, Avg generation throughput: 580.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:17:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1205.5 tokens/s, Avg generation throughput: 600.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:17:35 [loggers.py:273] Engine 000: Avg prompt throughput: 873.5 tokens/s, Avg generation throughput: 619.7 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:17:45 [loggers.py:273] Engine 000: Avg prompt throughput: 840.0 tokens/s, Avg generation throughput: 626.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:17:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1148.7 tokens/s, Avg generation throughput: 606.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:18:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1609.7 tokens/s, Avg generation throughput: 554.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:18:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1581.5 tokens/s, Avg generation throughput: 567.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:18:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1074.7 tokens/s, Avg generation throughput: 606.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:18:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1220.0 tokens/s, Avg generation throughput: 600.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:18:45 [loggers.py:273] Engine 000: Avg prompt throughput: 848.3 tokens/s, Avg generation throughput: 625.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:18:55 [loggers.py:273] Engine 000: Avg prompt throughput: 532.1 tokens/s, Avg generation throughput: 652.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:19:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1194.0 tokens/s, Avg generation throughput: 580.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:19:15 [loggers.py:273] Engine 000: Avg prompt throughput: 670.7 tokens/s, Avg generation throughput: 652.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:19:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1750.6 tokens/s, Avg generation throughput: 548.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:19:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1094.8 tokens/s, Avg generation throughput: 587.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:19:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1242.8 tokens/s, Avg generation throughput: 587.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:19:55 [loggers.py:273] Engine 000: Avg prompt throughput: 848.9 tokens/s, Avg generation throughput: 639.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:20:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1142.4 tokens/s, Avg generation throughput: 606.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:20:15 [loggers.py:273] Engine 000: Avg prompt throughput: 856.8 tokens/s, Avg generation throughput: 606.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:20:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1213.5 tokens/s, Avg generation throughput: 613.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:20:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1487.5 tokens/s, Avg generation throughput: 561.5 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:20:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1587.6 tokens/s, Avg generation throughput: 561.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:20:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1049.6 tokens/s, Avg generation throughput: 599.8 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:21:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1847.5 tokens/s, Avg generation throughput: 561.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:21:15 [loggers.py:273] Engine 000: Avg prompt throughput: 360.4 tokens/s, Avg generation throughput: 671.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:21:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1555.0 tokens/s, Avg generation throughput: 574.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:21:35 [loggers.py:273] Engine 000: Avg prompt throughput: 835.6 tokens/s, Avg generation throughput: 613.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:21:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1041.0 tokens/s, Avg generation throughput: 632.4 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:21:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1714.9 tokens/s, Avg generation throughput: 541.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:22:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1863.4 tokens/s, Avg generation throughput: 561.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:22:15 [loggers.py:273] Engine 000: Avg prompt throughput: 985.1 tokens/s, Avg generation throughput: 619.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:22:25 [loggers.py:273] Engine 000: Avg prompt throughput: 701.1 tokens/s, Avg generation throughput: 639.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:22:35 [loggers.py:273] Engine 000: Avg prompt throughput: 880.3 tokens/s, Avg generation throughput: 626.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:22:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1768.2 tokens/s, Avg generation throughput: 554.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:22:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1010.7 tokens/s, Avg generation throughput: 613.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:23:05 [loggers.py:273] Engine 000: Avg prompt throughput: 712.0 tokens/s, Avg generation throughput: 639.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:23:15 [loggers.py:273] Engine 000: Avg prompt throughput: 688.8 tokens/s, Avg generation throughput: 626.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:23:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1224.6 tokens/s, Avg generation throughput: 606.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:23:35 [loggers.py:273] Engine 000: Avg prompt throughput: 706.4 tokens/s, Avg generation throughput: 626.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:23:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1271.0 tokens/s, Avg generation throughput: 593.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:23:55 [loggers.py:273] Engine 000: Avg prompt throughput: 854.2 tokens/s, Avg generation throughput: 632.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:24:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1238.8 tokens/s, Avg generation throughput: 593.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:24:15 [loggers.py:273] Engine 000: Avg prompt throughput: 862.9 tokens/s, Avg generation throughput: 626.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:24:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1549.0 tokens/s, Avg generation throughput: 561.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:24:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1389.7 tokens/s, Avg generation throughput: 580.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:24:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1536.2 tokens/s, Avg generation throughput: 561.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:24:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1769.3 tokens/s, Avg generation throughput: 561.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:25:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1729.1 tokens/s, Avg generation throughput: 554.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:25:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1578.4 tokens/s, Avg generation throughput: 567.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:25:25 [loggers.py:273] Engine 000: Avg prompt throughput: 365.1 tokens/s, Avg generation throughput: 665.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:25:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1080.4 tokens/s, Avg generation throughput: 613.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:25:45 [loggers.py:273] Engine 000: Avg prompt throughput: 851.1 tokens/s, Avg generation throughput: 626.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:25:55 [loggers.py:273] Engine 000: Avg prompt throughput: 522.3 tokens/s, Avg generation throughput: 652.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:26:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1255.1 tokens/s, Avg generation throughput: 593.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:26:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1253.4 tokens/s, Avg generation throughput: 587.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:26:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1512.7 tokens/s, Avg generation throughput: 574.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:26:35 [loggers.py:273] Engine 000: Avg prompt throughput: 703.5 tokens/s, Avg generation throughput: 639.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:26:45 [loggers.py:273] Engine 000: Avg prompt throughput: 691.7 tokens/s, Avg generation throughput: 639.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:26:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1059.9 tokens/s, Avg generation throughput: 619.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:27:05 [loggers.py:273] Engine 000: Avg prompt throughput: 2029.1 tokens/s, Avg generation throughput: 515.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:27:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1392.5 tokens/s, Avg generation throughput: 600.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:27:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1393.4 tokens/s, Avg generation throughput: 580.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:27:35 [loggers.py:273] Engine 000: Avg prompt throughput: 365.2 tokens/s, Avg generation throughput: 665.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:27:45 [loggers.py:273] Engine 000: Avg prompt throughput: 906.8 tokens/s, Avg generation throughput: 626.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:27:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1244.0 tokens/s, Avg generation throughput: 593.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:28:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1178.0 tokens/s, Avg generation throughput: 600.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:28:15 [loggers.py:273] Engine 000: Avg prompt throughput: 687.8 tokens/s, Avg generation throughput: 645.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:28:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1711.6 tokens/s, Avg generation throughput: 554.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:28:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1014.0 tokens/s, Avg generation throughput: 613.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:28:45 [loggers.py:273] Engine 000: Avg prompt throughput: 984.6 tokens/s, Avg generation throughput: 619.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:28:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1151.0 tokens/s, Avg generation throughput: 606.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:29:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1594.3 tokens/s, Avg generation throughput: 567.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:29:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1019.0 tokens/s, Avg generation throughput: 613.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:29:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1205.8 tokens/s, Avg generation throughput: 593.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:29:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1709.3 tokens/s, Avg generation throughput: 554.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:29:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1236.2 tokens/s, Avg generation throughput: 587.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:29:55 [loggers.py:273] Engine 000: Avg prompt throughput: 770.3 tokens/s, Avg generation throughput: 639.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:30:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1210.8 tokens/s, Avg generation throughput: 600.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:30:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1374.9 tokens/s, Avg generation throughput: 580.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:30:25 [loggers.py:273] Engine 000: Avg prompt throughput: 982.9 tokens/s, Avg generation throughput: 619.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:30:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1270.7 tokens/s, Avg generation throughput: 593.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:30:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1207.0 tokens/s, Avg generation throughput: 600.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:30:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1486.3 tokens/s, Avg generation throughput: 574.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:31:05 [loggers.py:273] Engine 000: Avg prompt throughput: 985.9 tokens/s, Avg generation throughput: 619.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:31:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1875.5 tokens/s, Avg generation throughput: 541.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:31:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1668.8 tokens/s, Avg generation throughput: 561.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:31:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1305.6 tokens/s, Avg generation throughput: 580.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:31:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1923.3 tokens/s, Avg generation throughput: 541.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:31:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1731.3 tokens/s, Avg generation throughput: 554.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:32:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1343.3 tokens/s, Avg generation throughput: 587.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:32:15 [loggers.py:273] Engine 000: Avg prompt throughput: 2111.0 tokens/s, Avg generation throughput: 522.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:32:25 [loggers.py:273] Engine 000: Avg prompt throughput: 2934.0 tokens/s, Avg generation throughput: 463.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:32:35 [loggers.py:273] Engine 000: Avg prompt throughput: 3490.1 tokens/s, Avg generation throughput: 417.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:32:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1808.0 tokens/s, Avg generation throughput: 548.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:32:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1930.4 tokens/s, Avg generation throughput: 541.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:33:05 [loggers.py:273] Engine 000: Avg prompt throughput: 2489.1 tokens/s, Avg generation throughput: 496.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:33:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1460.1 tokens/s, Avg generation throughput: 580.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:33:25 [loggers.py:273] Engine 000: Avg prompt throughput: 967.5 tokens/s, Avg generation throughput: 619.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:33:35 [loggers.py:273] Engine 000: Avg prompt throughput: 831.8 tokens/s, Avg generation throughput: 613.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:33:45 [loggers.py:273] Engine 000: Avg prompt throughput: 2018.8 tokens/s, Avg generation throughput: 547.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:33:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1461.1 tokens/s, Avg generation throughput: 580.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:34:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1167.9 tokens/s, Avg generation throughput: 606.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:34:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1143.0 tokens/s, Avg generation throughput: 606.7 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:34:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1810.3 tokens/s, Avg generation throughput: 541.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:34:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1402.1 tokens/s, Avg generation throughput: 580.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:34:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1488.1 tokens/s, Avg generation throughput: 561.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:34:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1951.8 tokens/s, Avg generation throughput: 554.4 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:35:05 [loggers.py:273] Engine 000: Avg prompt throughput: 2182.5 tokens/s, Avg generation throughput: 522.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:35:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1993.1 tokens/s, Avg generation throughput: 522.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:35:25 [loggers.py:273] Engine 000: Avg prompt throughput: 2034.5 tokens/s, Avg generation throughput: 548.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:35:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1346.8 tokens/s, Avg generation throughput: 587.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:35:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1771.4 tokens/s, Avg generation throughput: 541.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:35:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1396.4 tokens/s, Avg generation throughput: 599.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:36:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1859.8 tokens/s, Avg generation throughput: 535.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:36:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1437.9 tokens/s, Avg generation throughput: 580.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:36:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1135.2 tokens/s, Avg generation throughput: 606.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:36:35 [loggers.py:273] Engine 000: Avg prompt throughput: 2103.6 tokens/s, Avg generation throughput: 522.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:36:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1010.3 tokens/s, Avg generation throughput: 613.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:36:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1798.1 tokens/s, Avg generation throughput: 548.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:37:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1855.1 tokens/s, Avg generation throughput: 548.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:37:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1093.8 tokens/s, Avg generation throughput: 599.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:37:25 [loggers.py:273] Engine 000: Avg prompt throughput: 2269.6 tokens/s, Avg generation throughput: 514.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:37:35 [loggers.py:273] Engine 000: Avg prompt throughput: 2271.2 tokens/s, Avg generation throughput: 507.6 tokens/s, Running: 63 reqs, Waiting: 1 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:37:45 [loggers.py:273] Engine 000: Avg prompt throughput: 3152.9 tokens/s, Avg generation throughput: 417.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:37:55 [loggers.py:273] Engine 000: Avg prompt throughput: 2804.0 tokens/s, Avg generation throughput: 475.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:38:05 [loggers.py:273] Engine 000: Avg prompt throughput: 2929.1 tokens/s, Avg generation throughput: 456.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:38:15 [loggers.py:273] Engine 000: Avg prompt throughput: 3139.4 tokens/s, Avg generation throughput: 443.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:38:25 [loggers.py:273] Engine 000: Avg prompt throughput: 3437.3 tokens/s, Avg generation throughput: 416.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:38:35 [loggers.py:273] Engine 000: Avg prompt throughput: 3590.1 tokens/s, Avg generation throughput: 410.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:38:45 [loggers.py:273] Engine 000: Avg prompt throughput: 3377.8 tokens/s, Avg generation throughput: 417.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:38:55 [loggers.py:273] Engine 000: Avg prompt throughput: 3706.8 tokens/s, Avg generation throughput: 390.0 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:39:05 [loggers.py:273] Engine 000: Avg prompt throughput: 3849.2 tokens/s, Avg generation throughput: 371.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:39:15 [loggers.py:273] Engine 000: Avg prompt throughput: 3407.9 tokens/s, Avg generation throughput: 429.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:39:25 [loggers.py:273] Engine 000: Avg prompt throughput: 3652.4 tokens/s, Avg generation throughput: 403.2 tokens/s, Running: 62 reqs, Waiting: 2 reqs, GPU KV cache usage: 23.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:39:35 [loggers.py:273] Engine 000: Avg prompt throughput: 4231.9 tokens/s, Avg generation throughput: 363.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:39:45 [loggers.py:273] Engine 000: Avg prompt throughput: 3845.7 tokens/s, Avg generation throughput: 396.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:39:55 [loggers.py:273] Engine 000: Avg prompt throughput: 3350.1 tokens/s, Avg generation throughput: 423.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:40:05 [loggers.py:273] Engine 000: Avg prompt throughput: 3524.9 tokens/s, Avg generation throughput: 416.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:40:15 [loggers.py:273] Engine 000: Avg prompt throughput: 3956.9 tokens/s, Avg generation throughput: 383.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:40:25 [loggers.py:273] Engine 000: Avg prompt throughput: 3320.6 tokens/s, Avg generation throughput: 429.4 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:40:35 [loggers.py:273] Engine 000: Avg prompt throughput: 3519.8 tokens/s, Avg generation throughput: 403.2 tokens/s, Running: 62 reqs, Waiting: 1 reqs, GPU KV cache usage: 22.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:40:45 [loggers.py:273] Engine 000: Avg prompt throughput: 3772.0 tokens/s, Avg generation throughput: 403.2 tokens/s, Running: 60 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:40:55 [loggers.py:273] Engine 000: Avg prompt throughput: 3853.6 tokens/s, Avg generation throughput: 384.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:41:05 [loggers.py:273] Engine 000: Avg prompt throughput: 3347.3 tokens/s, Avg generation throughput: 435.9 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:41:15 [loggers.py:273] Engine 000: Avg prompt throughput: 3897.8 tokens/s, Avg generation throughput: 377.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:41:25 [loggers.py:273] Engine 000: Avg prompt throughput: 3504.7 tokens/s, Avg generation throughput: 409.8 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:41:35 [loggers.py:273] Engine 000: Avg prompt throughput: 3451.4 tokens/s, Avg generation throughput: 423.2 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:41:45 [loggers.py:273] Engine 000: Avg prompt throughput: 3676.3 tokens/s, Avg generation throughput: 416.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:41:55 [loggers.py:273] Engine 000: Avg prompt throughput: 3271.1 tokens/s, Avg generation throughput: 423.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:42:05 [loggers.py:273] Engine 000: Avg prompt throughput: 3490.8 tokens/s, Avg generation throughput: 422.9 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:42:15 [loggers.py:273] Engine 000: Avg prompt throughput: 2749.7 tokens/s, Avg generation throughput: 462.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:42:25 [loggers.py:273] Engine 000: Avg prompt throughput: 2847.0 tokens/s, Avg generation throughput: 475.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:42:35 [loggers.py:273] Engine 000: Avg prompt throughput: 2955.5 tokens/s, Avg generation throughput: 468.7 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:42:45 [loggers.py:273] Engine 000: Avg prompt throughput: 2768.9 tokens/s, Avg generation throughput: 469.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:42:55 [loggers.py:273] Engine 000: Avg prompt throughput: 3551.0 tokens/s, Avg generation throughput: 410.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:43:05 [loggers.py:273] Engine 000: Avg prompt throughput: 2457.2 tokens/s, Avg generation throughput: 489.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:43:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1580.8 tokens/s, Avg generation throughput: 580.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:43:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1007.7 tokens/s, Avg generation throughput: 619.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:43:35 [loggers.py:273] Engine 000: Avg prompt throughput: 556.3 tokens/s, Avg generation throughput: 645.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:43:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1132.1 tokens/s, Avg generation throughput: 599.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:43:55 [loggers.py:273] Engine 000: Avg prompt throughput: 692.9 tokens/s, Avg generation throughput: 639.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:44:05 [loggers.py:273] Engine 000: Avg prompt throughput: 551.8 tokens/s, Avg generation throughput: 645.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:44:15 [loggers.py:273] Engine 000: Avg prompt throughput: 945.1 tokens/s, Avg generation throughput: 619.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:44:25 [loggers.py:273] Engine 000: Avg prompt throughput: 847.6 tokens/s, Avg generation throughput: 606.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:44:35 [loggers.py:273] Engine 000: Avg prompt throughput: 728.2 tokens/s, Avg generation throughput: 645.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:44:45 [loggers.py:273] Engine 000: Avg prompt throughput: 139.0 tokens/s, Avg generation throughput: 684.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:44:55 [loggers.py:273] Engine 000: Avg prompt throughput: 132.7 tokens/s, Avg generation throughput: 684.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:45:05 [loggers.py:273] Engine 000: Avg prompt throughput: 758.3 tokens/s, Avg generation throughput: 626.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:45:15 [loggers.py:273] Engine 000: Avg prompt throughput: 2495.2 tokens/s, Avg generation throughput: 482.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:45:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1353.0 tokens/s, Avg generation throughput: 574.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:45:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1382.0 tokens/s, Avg generation throughput: 587.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:45:45 [loggers.py:273] Engine 000: Avg prompt throughput: 686.7 tokens/s, Avg generation throughput: 638.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:45:55 [loggers.py:273] Engine 000: Avg prompt throughput: 895.1 tokens/s, Avg generation throughput: 619.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:46:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1055.4 tokens/s, Avg generation throughput: 606.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:46:15 [loggers.py:273] Engine 000: Avg prompt throughput: 401.7 tokens/s, Avg generation throughput: 652.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:46:25 [loggers.py:273] Engine 000: Avg prompt throughput: 963.8 tokens/s, Avg generation throughput: 599.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:46:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1230.8 tokens/s, Avg generation throughput: 600.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:46:45 [loggers.py:273] Engine 000: Avg prompt throughput: 293.8 tokens/s, Avg generation throughput: 665.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:46:55 [loggers.py:273] Engine 000: Avg prompt throughput: 419.9 tokens/s, Avg generation throughput: 664.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:47:05 [loggers.py:273] Engine 000: Avg prompt throughput: 636.1 tokens/s, Avg generation throughput: 645.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:47:15 [loggers.py:273] Engine 000: Avg prompt throughput: 719.2 tokens/s, Avg generation throughput: 645.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:47:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1592.1 tokens/s, Avg generation throughput: 561.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:47:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1459.6 tokens/s, Avg generation throughput: 580.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:47:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1356.4 tokens/s, Avg generation throughput: 573.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:47:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1403.3 tokens/s, Avg generation throughput: 587.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:48:05 [loggers.py:273] Engine 000: Avg prompt throughput: 830.0 tokens/s, Avg generation throughput: 632.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:48:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1010.7 tokens/s, Avg generation throughput: 606.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:48:25 [loggers.py:273] Engine 000: Avg prompt throughput: 418.3 tokens/s, Avg generation throughput: 671.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:48:35 [loggers.py:273] Engine 000: Avg prompt throughput: 416.3 tokens/s, Avg generation throughput: 664.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:48:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1362.0 tokens/s, Avg generation throughput: 587.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:48:55 [loggers.py:273] Engine 000: Avg prompt throughput: 445.0 tokens/s, Avg generation throughput: 665.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:49:05 [loggers.py:273] Engine 000: Avg prompt throughput: 780.9 tokens/s, Avg generation throughput: 632.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:49:15 [loggers.py:273] Engine 000: Avg prompt throughput: 963.7 tokens/s, Avg generation throughput: 619.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:49:25 [loggers.py:273] Engine 000: Avg prompt throughput: 797.3 tokens/s, Avg generation throughput: 632.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:49:35 [loggers.py:273] Engine 000: Avg prompt throughput: 895.6 tokens/s, Avg generation throughput: 632.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:49:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1742.9 tokens/s, Avg generation throughput: 554.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:49:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1274.4 tokens/s, Avg generation throughput: 593.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:50:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1657.8 tokens/s, Avg generation throughput: 561.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:50:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1483.6 tokens/s, Avg generation throughput: 580.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:50:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1064.9 tokens/s, Avg generation throughput: 593.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:50:35 [loggers.py:273] Engine 000: Avg prompt throughput: 763.6 tokens/s, Avg generation throughput: 651.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:50:45 [loggers.py:273] Engine 000: Avg prompt throughput: 323.6 tokens/s, Avg generation throughput: 671.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:50:55 [loggers.py:273] Engine 000: Avg prompt throughput: 690.7 tokens/s, Avg generation throughput: 639.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:51:05 [loggers.py:273] Engine 000: Avg prompt throughput: 739.5 tokens/s, Avg generation throughput: 639.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:51:15 [loggers.py:273] Engine 000: Avg prompt throughput: 130.8 tokens/s, Avg generation throughput: 690.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:51:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1171.4 tokens/s, Avg generation throughput: 599.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:51:35 [loggers.py:273] Engine 000: Avg prompt throughput: 932.7 tokens/s, Avg generation throughput: 619.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:51:45 [loggers.py:273] Engine 000: Avg prompt throughput: 548.9 tokens/s, Avg generation throughput: 645.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:51:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1159.8 tokens/s, Avg generation throughput: 593.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:52:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1363.4 tokens/s, Avg generation throughput: 586.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:52:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1538.0 tokens/s, Avg generation throughput: 560.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:52:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1687.5 tokens/s, Avg generation throughput: 554.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:52:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1082.7 tokens/s, Avg generation throughput: 593.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:52:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1132.8 tokens/s, Avg generation throughput: 613.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:52:55 [loggers.py:273] Engine 000: Avg prompt throughput: 863.3 tokens/s, Avg generation throughput: 625.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:53:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1095.2 tokens/s, Avg generation throughput: 600.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:53:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1014.4 tokens/s, Avg generation throughput: 612.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:53:25 [loggers.py:273] Engine 000: Avg prompt throughput: 343.3 tokens/s, Avg generation throughput: 671.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:53:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1011.9 tokens/s, Avg generation throughput: 612.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:53:45 [loggers.py:273] Engine 000: Avg prompt throughput: 846.6 tokens/s, Avg generation throughput: 619.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:53:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1356.4 tokens/s, Avg generation throughput: 587.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:54:05 [loggers.py:273] Engine 000: Avg prompt throughput: 913.1 tokens/s, Avg generation throughput: 619.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:54:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1406.7 tokens/s, Avg generation throughput: 580.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:54:25 [loggers.py:273] Engine 000: Avg prompt throughput: 420.0 tokens/s, Avg generation throughput: 645.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:54:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1676.4 tokens/s, Avg generation throughput: 567.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:54:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1150.7 tokens/s, Avg generation throughput: 600.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:54:55 [loggers.py:273] Engine 000: Avg prompt throughput: 932.9 tokens/s, Avg generation throughput: 625.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:55:05 [loggers.py:273] Engine 000: Avg prompt throughput: 843.1 tokens/s, Avg generation throughput: 625.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:55:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1003.3 tokens/s, Avg generation throughput: 619.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:55:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1214.6 tokens/s, Avg generation throughput: 587.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:55:35 [loggers.py:273] Engine 000: Avg prompt throughput: 996.9 tokens/s, Avg generation throughput: 625.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:55:45 [loggers.py:273] Engine 000: Avg prompt throughput: 740.0 tokens/s, Avg generation throughput: 619.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:55:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1013.5 tokens/s, Avg generation throughput: 632.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:56:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1218.4 tokens/s, Avg generation throughput: 593.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:56:15 [loggers.py:273] Engine 000: Avg prompt throughput: 744.0 tokens/s, Avg generation throughput: 632.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:56:25 [loggers.py:273] Engine 000: Avg prompt throughput: 774.0 tokens/s, Avg generation throughput: 632.5 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:56:35 [loggers.py:273] Engine 000: Avg prompt throughput: 974.5 tokens/s, Avg generation throughput: 619.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:56:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1051.3 tokens/s, Avg generation throughput: 606.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:56:55 [loggers.py:273] Engine 000: Avg prompt throughput: 833.9 tokens/s, Avg generation throughput: 625.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:57:05 [loggers.py:273] Engine 000: Avg prompt throughput: 733.1 tokens/s, Avg generation throughput: 632.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:57:15 [loggers.py:273] Engine 000: Avg prompt throughput: 765.6 tokens/s, Avg generation throughput: 626.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:57:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1036.9 tokens/s, Avg generation throughput: 606.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:57:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1270.4 tokens/s, Avg generation throughput: 587.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:57:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1709.7 tokens/s, Avg generation throughput: 554.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:57:55 [loggers.py:273] Engine 000: Avg prompt throughput: 840.9 tokens/s, Avg generation throughput: 626.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:58:05 [loggers.py:273] Engine 000: Avg prompt throughput: 855.8 tokens/s, Avg generation throughput: 606.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:58:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1094.9 tokens/s, Avg generation throughput: 606.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:58:25 [loggers.py:273] Engine 000: Avg prompt throughput: 908.6 tokens/s, Avg generation throughput: 632.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:58:35 [loggers.py:273] Engine 000: Avg prompt throughput: 858.4 tokens/s, Avg generation throughput: 625.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:58:45 [loggers.py:273] Engine 000: Avg prompt throughput: 842.3 tokens/s, Avg generation throughput: 625.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:58:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1088.9 tokens/s, Avg generation throughput: 600.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:59:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1149.1 tokens/s, Avg generation throughput: 606.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:59:15 [loggers.py:273] Engine 000: Avg prompt throughput: 880.6 tokens/s, Avg generation throughput: 625.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:59:25 [loggers.py:273] Engine 000: Avg prompt throughput: 417.8 tokens/s, Avg generation throughput: 658.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:59:35 [loggers.py:273] Engine 000: Avg prompt throughput: 969.8 tokens/s, Avg generation throughput: 613.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:59:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1334.4 tokens/s, Avg generation throughput: 587.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 01:59:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1354.6 tokens/s, Avg generation throughput: 586.7 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:00:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1232.1 tokens/s, Avg generation throughput: 593.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:00:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1011.7 tokens/s, Avg generation throughput: 612.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:00:25 [loggers.py:273] Engine 000: Avg prompt throughput: 617.5 tokens/s, Avg generation throughput: 639.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:00:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1027.1 tokens/s, Avg generation throughput: 606.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:00:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1516.7 tokens/s, Avg generation throughput: 561.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:00:55 [loggers.py:273] Engine 000: Avg prompt throughput: 706.3 tokens/s, Avg generation throughput: 632.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:01:05 [loggers.py:273] Engine 000: Avg prompt throughput: 768.5 tokens/s, Avg generation throughput: 626.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:01:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1410.1 tokens/s, Avg generation throughput: 573.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:01:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1093.0 tokens/s, Avg generation throughput: 606.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:01:35 [loggers.py:273] Engine 000: Avg prompt throughput: 674.0 tokens/s, Avg generation throughput: 625.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:01:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1278.8 tokens/s, Avg generation throughput: 593.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:01:55 [loggers.py:273] Engine 000: Avg prompt throughput: 444.0 tokens/s, Avg generation throughput: 652.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:02:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1289.4 tokens/s, Avg generation throughput: 587.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:02:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1023.7 tokens/s, Avg generation throughput: 612.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:02:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1246.3 tokens/s, Avg generation throughput: 600.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:02:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1253.8 tokens/s, Avg generation throughput: 587.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:02:45 [loggers.py:273] Engine 000: Avg prompt throughput: 434.7 tokens/s, Avg generation throughput: 664.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:02:55 [loggers.py:273] Engine 000: Avg prompt throughput: 443.1 tokens/s, Avg generation throughput: 658.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:03:05 [loggers.py:273] Engine 000: Avg prompt throughput: 569.4 tokens/s, Avg generation throughput: 645.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:03:15 [loggers.py:273] Engine 000: Avg prompt throughput: 425.2 tokens/s, Avg generation throughput: 665.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:03:25 [loggers.py:273] Engine 000: Avg prompt throughput: 2254.1 tokens/s, Avg generation throughput: 515.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:03:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1267.5 tokens/s, Avg generation throughput: 593.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:03:45 [loggers.py:273] Engine 000: Avg prompt throughput: 578.5 tokens/s, Avg generation throughput: 651.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:03:55 [loggers.py:273] Engine 000: Avg prompt throughput: 874.7 tokens/s, Avg generation throughput: 613.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:04:05 [loggers.py:273] Engine 000: Avg prompt throughput: 696.9 tokens/s, Avg generation throughput: 645.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:04:15 [loggers.py:273] Engine 000: Avg prompt throughput: 770.7 tokens/s, Avg generation throughput: 632.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:04:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1698.5 tokens/s, Avg generation throughput: 548.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:04:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1221.6 tokens/s, Avg generation throughput: 586.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:04:45 [loggers.py:273] Engine 000: Avg prompt throughput: 1469.1 tokens/s, Avg generation throughput: 586.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:04:55 [loggers.py:273] Engine 000: Avg prompt throughput: 970.6 tokens/s, Avg generation throughput: 613.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:05:05 [loggers.py:273] Engine 000: Avg prompt throughput: 672.2 tokens/s, Avg generation throughput: 639.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:05:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1259.7 tokens/s, Avg generation throughput: 580.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:05:25 [loggers.py:273] Engine 000: Avg prompt throughput: 827.5 tokens/s, Avg generation throughput: 639.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:05:35 [loggers.py:273] Engine 000: Avg prompt throughput: 872.5 tokens/s, Avg generation throughput: 613.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:05:45 [loggers.py:273] Engine 000: Avg prompt throughput: 734.1 tokens/s, Avg generation throughput: 645.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:05:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1134.8 tokens/s, Avg generation throughput: 600.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:06:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1130.0 tokens/s, Avg generation throughput: 593.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:06:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1472.8 tokens/s, Avg generation throughput: 574.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:06:25 [loggers.py:273] Engine 000: Avg prompt throughput: 880.9 tokens/s, Avg generation throughput: 619.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:06:35 [loggers.py:273] Engine 000: Avg prompt throughput: 1008.9 tokens/s, Avg generation throughput: 612.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:06:45 [loggers.py:273] Engine 000: Avg prompt throughput: 591.7 tokens/s, Avg generation throughput: 645.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:06:55 [loggers.py:273] Engine 000: Avg prompt throughput: 1450.1 tokens/s, Avg generation throughput: 580.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:07:05 [loggers.py:273] Engine 000: Avg prompt throughput: 1833.5 tokens/s, Avg generation throughput: 547.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:07:15 [loggers.py:273] Engine 000: Avg prompt throughput: 1574.5 tokens/s, Avg generation throughput: 573.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:07:25 [loggers.py:273] Engine 000: Avg prompt throughput: 1929.0 tokens/s, Avg generation throughput: 547.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:07:35 [loggers.py:273] Engine 000: Avg prompt throughput: 2431.3 tokens/s, Avg generation throughput: 507.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:07:45 [loggers.py:273] Engine 000: Avg prompt throughput: 2553.8 tokens/s, Avg generation throughput: 494.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:07:55 [loggers.py:273] Engine 000: Avg prompt throughput: 2482.7 tokens/s, Avg generation throughput: 507.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:08:05 [loggers.py:273] Engine 000: Avg prompt throughput: 2768.5 tokens/s, Avg generation throughput: 474.7 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:08:15 [loggers.py:273] Engine 000: Avg prompt throughput: 3631.6 tokens/s, Avg generation throughput: 409.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:08:25 [loggers.py:273] Engine 000: Avg prompt throughput: 3223.8 tokens/s, Avg generation throughput: 448.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:08:35 [loggers.py:273] Engine 000: Avg prompt throughput: 3667.6 tokens/s, Avg generation throughput: 409.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:08:45 [loggers.py:273] Engine 000: Avg prompt throughput: 3785.7 tokens/s, Avg generation throughput: 397.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:08:55 [loggers.py:273] Engine 000: Avg prompt throughput: 3343.6 tokens/s, Avg generation throughput: 441.5 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:09:05 [loggers.py:273] Engine 000: Avg prompt throughput: 4690.1 tokens/s, Avg generation throughput: 336.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:09:15 [loggers.py:273] Engine 000: Avg prompt throughput: 3987.0 tokens/s, Avg generation throughput: 383.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:09:25 [loggers.py:273] Engine 000: Avg prompt throughput: 4005.1 tokens/s, Avg generation throughput: 389.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:09:35 [loggers.py:273] Engine 000: Avg prompt throughput: 4114.8 tokens/s, Avg generation throughput: 376.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:09:45 [loggers.py:273] Engine 000: Avg prompt throughput: 4305.5 tokens/s, Avg generation throughput: 383.0 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:09:55 [loggers.py:273] Engine 000: Avg prompt throughput: 4179.4 tokens/s, Avg generation throughput: 369.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:10:05 [loggers.py:273] Engine 000: Avg prompt throughput: 4458.7 tokens/s, Avg generation throughput: 363.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:10:15 [loggers.py:273] Engine 000: Avg prompt throughput: 4349.7 tokens/s, Avg generation throughput: 337.1 tokens/s, Running: 61 reqs, Waiting: 2 reqs, GPU KV cache usage: 21.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:10:25 [loggers.py:273] Engine 000: Avg prompt throughput: 4014.8 tokens/s, Avg generation throughput: 401.7 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:10:35 [loggers.py:273] Engine 000: Avg prompt throughput: 4693.0 tokens/s, Avg generation throughput: 342.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:10:45 [loggers.py:273] Engine 000: Avg prompt throughput: 3932.0 tokens/s, Avg generation throughput: 396.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:10:55 [loggers.py:273] Engine 000: Avg prompt throughput: 3931.0 tokens/s, Avg generation throughput: 402.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:11:05 [loggers.py:273] Engine 000: Avg prompt throughput: 4150.3 tokens/s, Avg generation throughput: 388.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:11:15 [loggers.py:273] Engine 000: Avg prompt throughput: 4960.0 tokens/s, Avg generation throughput: 316.7 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:11:25 [loggers.py:273] Engine 000: Avg prompt throughput: 3943.0 tokens/s, Avg generation throughput: 395.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:11:35 [loggers.py:273] Engine 000: Avg prompt throughput: 4714.2 tokens/s, Avg generation throughput: 325.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:11:45 [loggers.py:273] Engine 000: Avg prompt throughput: 5679.9 tokens/s, Avg generation throughput: 258.2 tokens/s, Running: 60 reqs, Waiting: 3 reqs, GPU KV cache usage: 23.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:11:55 [loggers.py:273] Engine 000: Avg prompt throughput: 5276.7 tokens/s, Avg generation throughput: 247.6 tokens/s, Running: 60 reqs, Waiting: 1 reqs, GPU KV cache usage: 24.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:12:05 [loggers.py:273] Engine 000: Avg prompt throughput: 6508.8 tokens/s, Avg generation throughput: 183.3 tokens/s, Running: 61 reqs, Waiting: 2 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:12:15 [loggers.py:273] Engine 000: Avg prompt throughput: 5605.3 tokens/s, Avg generation throughput: 247.6 tokens/s, Running: 61 reqs, Waiting: 1 reqs, GPU KV cache usage: 25.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:12:25 [loggers.py:273] Engine 000: Avg prompt throughput: 6170.5 tokens/s, Avg generation throughput: 217.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:12:35 [loggers.py:273] Engine 000: Avg prompt throughput: 5056.7 tokens/s, Avg generation throughput: 252.5 tokens/s, Running: 62 reqs, Waiting: 2 reqs, GPU KV cache usage: 26.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:12:45 [loggers.py:273] Engine 000: Avg prompt throughput: 5729.9 tokens/s, Avg generation throughput: 250.5 tokens/s, Running: 59 reqs, Waiting: 4 reqs, GPU KV cache usage: 24.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:12:55 [loggers.py:273] Engine 000: Avg prompt throughput: 5858.5 tokens/s, Avg generation throughput: 220.8 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:13:05 [loggers.py:273] Engine 000: Avg prompt throughput: 5824.8 tokens/s, Avg generation throughput: 230.5 tokens/s, Running: 62 reqs, Waiting: 1 reqs, GPU KV cache usage: 26.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:13:15 [loggers.py:273] Engine 000: Avg prompt throughput: 4875.2 tokens/s, Avg generation throughput: 267.6 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:13:25 [loggers.py:273] Engine 000: Avg prompt throughput: 6353.0 tokens/s, Avg generation throughput: 192.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:13:35 [loggers.py:273] Engine 000: Avg prompt throughput: 5459.6 tokens/s, Avg generation throughput: 260.7 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:13:45 [loggers.py:273] Engine 000: Avg prompt throughput: 6169.3 tokens/s, Avg generation throughput: 205.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:13:55 [loggers.py:273] Engine 000: Avg prompt throughput: 5509.8 tokens/s, Avg generation throughput: 234.0 tokens/s, Running: 62 reqs, Waiting: 1 reqs, GPU KV cache usage: 26.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:14:05 [loggers.py:273] Engine 000: Avg prompt throughput: 5813.1 tokens/s, Avg generation throughput: 247.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:14:15 [loggers.py:273] Engine 000: Avg prompt throughput: 4776.8 tokens/s, Avg generation throughput: 298.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:14:25 [loggers.py:273] Engine 000: Avg prompt throughput: 3779.5 tokens/s, Avg generation throughput: 378.7 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:14:35 [loggers.py:273] Engine 000: Avg prompt throughput: 4175.9 tokens/s, Avg generation throughput: 372.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:14:45 [loggers.py:273] Engine 000: Avg prompt throughput: 4156.1 tokens/s, Avg generation throughput: 352.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:14:55 [loggers.py:273] Engine 000: Avg prompt throughput: 4035.7 tokens/s, Avg generation throughput: 378.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:15:05 [loggers.py:273] Engine 000: Avg prompt throughput: 4040.2 tokens/s, Avg generation throughput: 371.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:15:15 [loggers.py:273] Engine 000: Avg prompt throughput: 3557.1 tokens/s, Avg generation throughput: 405.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:15:25 [loggers.py:273] Engine 000: Avg prompt throughput: 4487.5 tokens/s, Avg generation throughput: 351.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:15:35 [loggers.py:273] Engine 000: Avg prompt throughput: 3842.6 tokens/s, Avg generation throughput: 384.3 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:15:46 [loggers.py:273] Engine 000: Avg prompt throughput: 3946.7 tokens/s, Avg generation throughput: 384.7 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:15:56 [loggers.py:273] Engine 000: Avg prompt throughput: 4263.4 tokens/s, Avg generation throughput: 339.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:16:06 [loggers.py:273] Engine 000: Avg prompt throughput: 3974.7 tokens/s, Avg generation throughput: 371.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:16:16 [loggers.py:273] Engine 000: Avg prompt throughput: 3598.7 tokens/s, Avg generation throughput: 430.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:16:26 [loggers.py:273] Engine 000: Avg prompt throughput: 4424.6 tokens/s, Avg generation throughput: 319.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:16:36 [loggers.py:273] Engine 000: Avg prompt throughput: 3893.3 tokens/s, Avg generation throughput: 391.1 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:16:46 [loggers.py:273] Engine 000: Avg prompt throughput: 4142.7 tokens/s, Avg generation throughput: 365.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:16:56 [loggers.py:273] Engine 000: Avg prompt throughput: 3915.0 tokens/s, Avg generation throughput: 364.9 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:17:06 [loggers.py:273] Engine 000: Avg prompt throughput: 4126.3 tokens/s, Avg generation throughput: 365.7 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:17:16 [loggers.py:273] Engine 000: Avg prompt throughput: 3934.0 tokens/s, Avg generation throughput: 372.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:17:26 [loggers.py:273] Engine 000: Avg prompt throughput: 4004.9 tokens/s, Avg generation throughput: 377.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:17:36 [loggers.py:273] Engine 000: Avg prompt throughput: 4424.2 tokens/s, Avg generation throughput: 338.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:17:46 [loggers.py:273] Engine 000: Avg prompt throughput: 3860.5 tokens/s, Avg generation throughput: 397.8 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:17:56 [loggers.py:273] Engine 000: Avg prompt throughput: 4459.6 tokens/s, Avg generation throughput: 333.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:18:06 [loggers.py:273] Engine 000: Avg prompt throughput: 3586.2 tokens/s, Avg generation throughput: 416.7 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:18:16 [loggers.py:273] Engine 000: Avg prompt throughput: 4328.5 tokens/s, Avg generation throughput: 352.3 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:18:26 [loggers.py:273] Engine 000: Avg prompt throughput: 4119.6 tokens/s, Avg generation throughput: 352.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:18:36 [loggers.py:273] Engine 000: Avg prompt throughput: 3911.3 tokens/s, Avg generation throughput: 378.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:18:46 [loggers.py:273] Engine 000: Avg prompt throughput: 4747.8 tokens/s, Avg generation throughput: 332.3 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:18:56 [loggers.py:273] Engine 000: Avg prompt throughput: 3690.0 tokens/s, Avg generation throughput: 391.8 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:19:06 [loggers.py:273] Engine 000: Avg prompt throughput: 3584.4 tokens/s, Avg generation throughput: 392.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:19:16 [loggers.py:273] Engine 000: Avg prompt throughput: 4890.5 tokens/s, Avg generation throughput: 312.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:19:26 [loggers.py:273] Engine 000: Avg prompt throughput: 3363.1 tokens/s, Avg generation throughput: 436.4 tokens/s, Running: 62 reqs, Waiting: 1 reqs, GPU KV cache usage: 23.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:19:36 [loggers.py:273] Engine 000: Avg prompt throughput: 4504.5 tokens/s, Avg generation throughput: 339.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:19:46 [loggers.py:273] Engine 000: Avg prompt throughput: 3925.0 tokens/s, Avg generation throughput: 371.5 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:19:56 [loggers.py:273] Engine 000: Avg prompt throughput: 3428.5 tokens/s, Avg generation throughput: 417.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:20:06 [loggers.py:273] Engine 000: Avg prompt throughput: 3495.0 tokens/s, Avg generation throughput: 430.3 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:20:16 [loggers.py:273] Engine 000: Avg prompt throughput: 3807.8 tokens/s, Avg generation throughput: 385.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:20:26 [loggers.py:273] Engine 000: Avg prompt throughput: 3336.0 tokens/s, Avg generation throughput: 424.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:20:36 [loggers.py:273] Engine 000: Avg prompt throughput: 3474.3 tokens/s, Avg generation throughput: 411.3 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:20:46 [loggers.py:273] Engine 000: Avg prompt throughput: 3459.2 tokens/s, Avg generation throughput: 418.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:20:56 [loggers.py:273] Engine 000: Avg prompt throughput: 2749.7 tokens/s, Avg generation throughput: 476.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:21:06 [loggers.py:273] Engine 000: Avg prompt throughput: 3103.8 tokens/s, Avg generation throughput: 450.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:21:16 [loggers.py:273] Engine 000: Avg prompt throughput: 3831.5 tokens/s, Avg generation throughput: 378.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:21:26 [loggers.py:273] Engine 000: Avg prompt throughput: 3003.1 tokens/s, Avg generation throughput: 456.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:21:36 [loggers.py:273] Engine 000: Avg prompt throughput: 3339.7 tokens/s, Avg generation throughput: 424.5 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:21:46 [loggers.py:273] Engine 000: Avg prompt throughput: 4234.2 tokens/s, Avg generation throughput: 353.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:21:56 [loggers.py:273] Engine 000: Avg prompt throughput: 2457.5 tokens/s, Avg generation throughput: 483.7 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:22:06 [loggers.py:273] Engine 000: Avg prompt throughput: 936.1 tokens/s, Avg generation throughput: 619.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:22:16 [loggers.py:273] Engine 000: Avg prompt throughput: 2087.7 tokens/s, Avg generation throughput: 522.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:22:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1713.9 tokens/s, Avg generation throughput: 548.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:22:36 [loggers.py:273] Engine 000: Avg prompt throughput: 934.2 tokens/s, Avg generation throughput: 626.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:22:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1313.0 tokens/s, Avg generation throughput: 581.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:22:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1717.9 tokens/s, Avg generation throughput: 541.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:23:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1542.6 tokens/s, Avg generation throughput: 574.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:23:16 [loggers.py:273] Engine 000: Avg prompt throughput: 387.5 tokens/s, Avg generation throughput: 658.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:23:26 [loggers.py:273] Engine 000: Avg prompt throughput: 576.4 tokens/s, Avg generation throughput: 645.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:23:36 [loggers.py:273] Engine 000: Avg prompt throughput: 941.7 tokens/s, Avg generation throughput: 607.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:23:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1160.6 tokens/s, Avg generation throughput: 593.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:23:56 [loggers.py:273] Engine 000: Avg prompt throughput: 781.9 tokens/s, Avg generation throughput: 620.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:24:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1532.1 tokens/s, Avg generation throughput: 561.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:24:16 [loggers.py:273] Engine 000: Avg prompt throughput: 922.1 tokens/s, Avg generation throughput: 619.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:24:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1315.5 tokens/s, Avg generation throughput: 581.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:24:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1328.1 tokens/s, Avg generation throughput: 580.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:24:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1679.9 tokens/s, Avg generation throughput: 542.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:24:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1702.0 tokens/s, Avg generation throughput: 555.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:25:06 [loggers.py:273] Engine 000: Avg prompt throughput: 934.7 tokens/s, Avg generation throughput: 600.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:25:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1883.3 tokens/s, Avg generation throughput: 554.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:25:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1499.4 tokens/s, Avg generation throughput: 561.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:25:36 [loggers.py:273] Engine 000: Avg prompt throughput: 571.2 tokens/s, Avg generation throughput: 658.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:25:46 [loggers.py:273] Engine 000: Avg prompt throughput: 969.0 tokens/s, Avg generation throughput: 606.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:25:56 [loggers.py:273] Engine 000: Avg prompt throughput: 594.6 tokens/s, Avg generation throughput: 639.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:26:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1361.9 tokens/s, Avg generation throughput: 568.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:26:16 [loggers.py:273] Engine 000: Avg prompt throughput: 968.1 tokens/s, Avg generation throughput: 606.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:26:26 [loggers.py:273] Engine 000: Avg prompt throughput: 736.5 tokens/s, Avg generation throughput: 632.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:26:36 [loggers.py:273] Engine 000: Avg prompt throughput: 782.7 tokens/s, Avg generation throughput: 619.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:26:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1410.1 tokens/s, Avg generation throughput: 574.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:26:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1156.3 tokens/s, Avg generation throughput: 600.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:27:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1882.1 tokens/s, Avg generation throughput: 548.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:27:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1679.4 tokens/s, Avg generation throughput: 561.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:27:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1729.3 tokens/s, Avg generation throughput: 541.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:27:36 [loggers.py:273] Engine 000: Avg prompt throughput: 2260.8 tokens/s, Avg generation throughput: 529.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:27:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1314.9 tokens/s, Avg generation throughput: 600.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:27:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1694.2 tokens/s, Avg generation throughput: 561.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:28:06 [loggers.py:273] Engine 000: Avg prompt throughput: 960.7 tokens/s, Avg generation throughput: 619.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:28:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1548.8 tokens/s, Avg generation throughput: 567.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:28:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1765.5 tokens/s, Avg generation throughput: 561.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:28:36 [loggers.py:273] Engine 000: Avg prompt throughput: 395.7 tokens/s, Avg generation throughput: 665.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:28:46 [loggers.py:273] Engine 000: Avg prompt throughput: 774.0 tokens/s, Avg generation throughput: 639.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:28:56 [loggers.py:273] Engine 000: Avg prompt throughput: 379.1 tokens/s, Avg generation throughput: 671.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:29:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1129.3 tokens/s, Avg generation throughput: 600.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:29:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1173.3 tokens/s, Avg generation throughput: 606.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:29:26 [loggers.py:273] Engine 000: Avg prompt throughput: 957.7 tokens/s, Avg generation throughput: 619.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:29:36 [loggers.py:273] Engine 000: Avg prompt throughput: 933.3 tokens/s, Avg generation throughput: 619.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:29:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1504.5 tokens/s, Avg generation throughput: 554.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:29:56 [loggers.py:273] Engine 000: Avg prompt throughput: 2249.0 tokens/s, Avg generation throughput: 535.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:30:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1683.2 tokens/s, Avg generation throughput: 561.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:30:16 [loggers.py:273] Engine 000: Avg prompt throughput: 2060.0 tokens/s, Avg generation throughput: 535.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:30:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1711.5 tokens/s, Avg generation throughput: 548.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:30:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1877.7 tokens/s, Avg generation throughput: 555.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:30:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1682.6 tokens/s, Avg generation throughput: 561.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:30:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1335.9 tokens/s, Avg generation throughput: 587.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:31:06 [loggers.py:273] Engine 000: Avg prompt throughput: 598.0 tokens/s, Avg generation throughput: 645.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:31:16 [loggers.py:273] Engine 000: Avg prompt throughput: 574.9 tokens/s, Avg generation throughput: 652.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:31:26 [loggers.py:273] Engine 000: Avg prompt throughput: 970.7 tokens/s, Avg generation throughput: 613.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:31:36 [loggers.py:273] Engine 000: Avg prompt throughput: 778.1 tokens/s, Avg generation throughput: 613.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:31:46 [loggers.py:273] Engine 000: Avg prompt throughput: 381.9 tokens/s, Avg generation throughput: 684.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:31:56 [loggers.py:273] Engine 000: Avg prompt throughput: 780.0 tokens/s, Avg generation throughput: 632.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:32:06 [loggers.py:273] Engine 000: Avg prompt throughput: 615.1 tokens/s, Avg generation throughput: 639.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 30.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:32:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1179.2 tokens/s, Avg generation throughput: 587.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:32:26 [loggers.py:273] Engine 000: Avg prompt throughput: 2267.2 tokens/s, Avg generation throughput: 509.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:32:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1702.1 tokens/s, Avg generation throughput: 554.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:32:46 [loggers.py:273] Engine 000: Avg prompt throughput: 2222.4 tokens/s, Avg generation throughput: 515.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:32:56 [loggers.py:273] Engine 000: Avg prompt throughput: 2622.0 tokens/s, Avg generation throughput: 483.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:33:06 [loggers.py:273] Engine 000: Avg prompt throughput: 2462.0 tokens/s, Avg generation throughput: 483.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:33:16 [loggers.py:273] Engine 000: Avg prompt throughput: 2302.9 tokens/s, Avg generation throughput: 515.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:33:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1313.5 tokens/s, Avg generation throughput: 574.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:33:36 [loggers.py:273] Engine 000: Avg prompt throughput: 745.0 tokens/s, Avg generation throughput: 645.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:33:46 [loggers.py:273] Engine 000: Avg prompt throughput: 760.9 tokens/s, Avg generation throughput: 632.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:33:56 [loggers.py:273] Engine 000: Avg prompt throughput: 381.9 tokens/s, Avg generation throughput: 665.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:34:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1164.6 tokens/s, Avg generation throughput: 593.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:34:16 [loggers.py:273] Engine 000: Avg prompt throughput: 569.3 tokens/s, Avg generation throughput: 645.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:34:26 [loggers.py:273] Engine 000: Avg prompt throughput: 964.5 tokens/s, Avg generation throughput: 619.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:34:36 [loggers.py:273] Engine 000: Avg prompt throughput: 195.8 tokens/s, Avg generation throughput: 678.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:34:46 [loggers.py:273] Engine 000: Avg prompt throughput: 381.1 tokens/s, Avg generation throughput: 658.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 30.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:34:56 [loggers.py:273] Engine 000: Avg prompt throughput: 599.2 tokens/s, Avg generation throughput: 639.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 30.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:35:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1303.7 tokens/s, Avg generation throughput: 574.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:35:16 [loggers.py:273] Engine 000: Avg prompt throughput: 3029.7 tokens/s, Avg generation throughput: 457.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:35:26 [loggers.py:273] Engine 000: Avg prompt throughput: 2619.8 tokens/s, Avg generation throughput: 483.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:35:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1510.5 tokens/s, Avg generation throughput: 574.1 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:35:46 [loggers.py:273] Engine 000: Avg prompt throughput: 2474.9 tokens/s, Avg generation throughput: 490.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:35:56 [loggers.py:273] Engine 000: Avg prompt throughput: 2091.2 tokens/s, Avg generation throughput: 522.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:36:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1511.9 tokens/s, Avg generation throughput: 574.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:36:16 [loggers.py:273] Engine 000: Avg prompt throughput: 569.5 tokens/s, Avg generation throughput: 645.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:36:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1552.4 tokens/s, Avg generation throughput: 554.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:36:36 [loggers.py:273] Engine 000: Avg prompt throughput: 762.5 tokens/s, Avg generation throughput: 620.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:36:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1146.8 tokens/s, Avg generation throughput: 613.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:36:56 [loggers.py:273] Engine 000: Avg prompt throughput: 609.6 tokens/s, Avg generation throughput: 645.7 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:37:06 [loggers.py:273] Engine 000: Avg prompt throughput: 588.4 tokens/s, Avg generation throughput: 639.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:37:16 [loggers.py:273] Engine 000: Avg prompt throughput: 770.2 tokens/s, Avg generation throughput: 632.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:37:26 [loggers.py:273] Engine 000: Avg prompt throughput: 369.6 tokens/s, Avg generation throughput: 665.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 30.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:37:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1332.1 tokens/s, Avg generation throughput: 580.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:37:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1323.4 tokens/s, Avg generation throughput: 580.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:37:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1517.2 tokens/s, Avg generation throughput: 568.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:38:06 [loggers.py:273] Engine 000: Avg prompt throughput: 2251.8 tokens/s, Avg generation throughput: 496.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:38:16 [loggers.py:273] Engine 000: Avg prompt throughput: 3002.7 tokens/s, Avg generation throughput: 464.1 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:38:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1702.7 tokens/s, Avg generation throughput: 554.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:38:36 [loggers.py:273] Engine 000: Avg prompt throughput: 387.8 tokens/s, Avg generation throughput: 645.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:38:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1886.6 tokens/s, Avg generation throughput: 542.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:38:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1150.5 tokens/s, Avg generation throughput: 593.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:39:06 [loggers.py:273] Engine 000: Avg prompt throughput: 768.1 tokens/s, Avg generation throughput: 626.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:39:16 [loggers.py:273] Engine 000: Avg prompt throughput: 203.8 tokens/s, Avg generation throughput: 671.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:39:26 [loggers.py:273] Engine 000: Avg prompt throughput: 387.0 tokens/s, Avg generation throughput: 658.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:39:36 [loggers.py:273] Engine 000: Avg prompt throughput: 585.4 tokens/s, Avg generation throughput: 645.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:39:46 [loggers.py:273] Engine 000: Avg prompt throughput: 381.7 tokens/s, Avg generation throughput: 639.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 30.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:39:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1392.3 tokens/s, Avg generation throughput: 580.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:40:06 [loggers.py:273] Engine 000: Avg prompt throughput: 398.7 tokens/s, Avg generation throughput: 652.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 30.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:40:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1706.6 tokens/s, Avg generation throughput: 548.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:40:26 [loggers.py:273] Engine 000: Avg prompt throughput: 2085.4 tokens/s, Avg generation throughput: 522.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:40:36 [loggers.py:273] Engine 000: Avg prompt throughput: 2643.6 tokens/s, Avg generation throughput: 476.7 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:40:46 [loggers.py:273] Engine 000: Avg prompt throughput: 2418.5 tokens/s, Avg generation throughput: 503.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:40:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1877.0 tokens/s, Avg generation throughput: 542.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:41:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1877.9 tokens/s, Avg generation throughput: 548.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:41:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1525.4 tokens/s, Avg generation throughput: 580.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:41:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1729.6 tokens/s, Avg generation throughput: 554.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:41:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1155.5 tokens/s, Avg generation throughput: 606.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:41:46 [loggers.py:273] Engine 000: Avg prompt throughput: 570.2 tokens/s, Avg generation throughput: 652.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:41:56 [loggers.py:273] Engine 000: Avg prompt throughput: 774.2 tokens/s, Avg generation throughput: 632.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:42:06 [loggers.py:273] Engine 000: Avg prompt throughput: 195.3 tokens/s, Avg generation throughput: 684.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:42:16 [loggers.py:273] Engine 000: Avg prompt throughput: 411.7 tokens/s, Avg generation throughput: 658.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:42:26 [loggers.py:273] Engine 000: Avg prompt throughput: 404.1 tokens/s, Avg generation throughput: 658.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:42:36 [loggers.py:273] Engine 000: Avg prompt throughput: 768.0 tokens/s, Avg generation throughput: 632.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 30.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:42:46 [loggers.py:273] Engine 000: Avg prompt throughput: 948.2 tokens/s, Avg generation throughput: 619.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 30.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:42:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1695.6 tokens/s, Avg generation throughput: 554.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:43:06 [loggers.py:273] Engine 000: Avg prompt throughput: 2644.4 tokens/s, Avg generation throughput: 477.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:43:16 [loggers.py:273] Engine 000: Avg prompt throughput: 2439.9 tokens/s, Avg generation throughput: 502.7 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:43:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1695.8 tokens/s, Avg generation throughput: 561.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:43:36 [loggers.py:273] Engine 000: Avg prompt throughput: 2098.1 tokens/s, Avg generation throughput: 528.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:43:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1711.3 tokens/s, Avg generation throughput: 548.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:43:56 [loggers.py:273] Engine 000: Avg prompt throughput: 567.3 tokens/s, Avg generation throughput: 652.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:44:06 [loggers.py:273] Engine 000: Avg prompt throughput: 564.0 tokens/s, Avg generation throughput: 658.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:44:16 [loggers.py:273] Engine 000: Avg prompt throughput: 584.0 tokens/s, Avg generation throughput: 645.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:44:26 [loggers.py:273] Engine 000: Avg prompt throughput: 597.8 tokens/s, Avg generation throughput: 645.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:44:36 [loggers.py:273] Engine 000: Avg prompt throughput: 204.5 tokens/s, Avg generation throughput: 678.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:44:46 [loggers.py:273] Engine 000: Avg prompt throughput: 598.4 tokens/s, Avg generation throughput: 645.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:44:56 [loggers.py:273] Engine 000: Avg prompt throughput: 387.3 tokens/s, Avg generation throughput: 665.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 30.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:45:06 [loggers.py:273] Engine 000: Avg prompt throughput: 758.4 tokens/s, Avg generation throughput: 639.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 30.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:45:16 [loggers.py:273] Engine 000: Avg prompt throughput: 988.1 tokens/s, Avg generation throughput: 606.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 30.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:45:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1693.8 tokens/s, Avg generation throughput: 548.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:45:36 [loggers.py:273] Engine 000: Avg prompt throughput: 2165.7 tokens/s, Avg generation throughput: 522.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:45:46 [loggers.py:273] Engine 000: Avg prompt throughput: 2108.6 tokens/s, Avg generation throughput: 522.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:45:56 [loggers.py:273] Engine 000: Avg prompt throughput: 2454.1 tokens/s, Avg generation throughput: 496.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:46:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1862.9 tokens/s, Avg generation throughput: 554.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:46:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1314.1 tokens/s, Avg generation throughput: 580.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:46:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1860.7 tokens/s, Avg generation throughput: 567.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:46:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1327.6 tokens/s, Avg generation throughput: 587.2 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:46:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1346.2 tokens/s, Avg generation throughput: 587.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:46:56 [loggers.py:273] Engine 000: Avg prompt throughput: 199.8 tokens/s, Avg generation throughput: 678.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:47:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1171.4 tokens/s, Avg generation throughput: 600.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:47:16 [loggers.py:273] Engine 000: Avg prompt throughput: 376.2 tokens/s, Avg generation throughput: 671.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:47:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1127.4 tokens/s, Avg generation throughput: 606.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:47:36 [loggers.py:273] Engine 000: Avg prompt throughput: 748.4 tokens/s, Avg generation throughput: 639.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:47:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1324.8 tokens/s, Avg generation throughput: 580.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:47:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1173.5 tokens/s, Avg generation throughput: 606.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:48:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1533.0 tokens/s, Avg generation throughput: 574.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:48:16 [loggers.py:273] Engine 000: Avg prompt throughput: 365.4 tokens/s, Avg generation throughput: 665.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:48:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1217.2 tokens/s, Avg generation throughput: 593.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:48:36 [loggers.py:273] Engine 000: Avg prompt throughput: 2251.7 tokens/s, Avg generation throughput: 509.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:48:46 [loggers.py:273] Engine 000: Avg prompt throughput: 2254.6 tokens/s, Avg generation throughput: 515.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:48:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1700.4 tokens/s, Avg generation throughput: 548.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:49:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1694.4 tokens/s, Avg generation throughput: 568.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:49:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1310.8 tokens/s, Avg generation throughput: 593.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:49:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1313.5 tokens/s, Avg generation throughput: 580.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:49:36 [loggers.py:273] Engine 000: Avg prompt throughput: 562.6 tokens/s, Avg generation throughput: 665.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:49:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1159.8 tokens/s, Avg generation throughput: 593.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:49:56 [loggers.py:273] Engine 000: Avg prompt throughput: 948.0 tokens/s, Avg generation throughput: 626.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:50:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1196.7 tokens/s, Avg generation throughput: 587.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:50:16 [loggers.py:273] Engine 000: Avg prompt throughput: 976.9 tokens/s, Avg generation throughput: 613.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:50:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1349.6 tokens/s, Avg generation throughput: 580.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:50:36 [loggers.py:273] Engine 000: Avg prompt throughput: 365.2 tokens/s, Avg generation throughput: 671.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:50:46 [loggers.py:273] Engine 000: Avg prompt throughput: 931.5 tokens/s, Avg generation throughput: 619.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:50:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1333.9 tokens/s, Avg generation throughput: 587.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:51:06 [loggers.py:273] Engine 000: Avg prompt throughput: 2059.1 tokens/s, Avg generation throughput: 516.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:51:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1342.0 tokens/s, Avg generation throughput: 593.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:51:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1895.0 tokens/s, Avg generation throughput: 541.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:51:36 [loggers.py:273] Engine 000: Avg prompt throughput: 2079.8 tokens/s, Avg generation throughput: 535.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:51:46 [loggers.py:273] Engine 000: Avg prompt throughput: 2082.6 tokens/s, Avg generation throughput: 528.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:51:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1158.1 tokens/s, Avg generation throughput: 600.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:52:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1316.4 tokens/s, Avg generation throughput: 593.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:52:16 [loggers.py:273] Engine 000: Avg prompt throughput: 750.4 tokens/s, Avg generation throughput: 632.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:52:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1351.0 tokens/s, Avg generation throughput: 587.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:52:36 [loggers.py:273] Engine 000: Avg prompt throughput: 578.7 tokens/s, Avg generation throughput: 632.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:52:46 [loggers.py:273] Engine 000: Avg prompt throughput: 767.7 tokens/s, Avg generation throughput: 645.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 02:52:56 [loggers.py:273] Engine 000: Avg prompt throughput: 194.3 tokens/s, Avg generation throughput: 672.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:53:06 [loggers.py:273] Engine 000: Avg prompt throughput: 944.9 tokens/s, Avg generation throughput: 619.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:53:16 [loggers.py:273] Engine 000: Avg prompt throughput: 984.3 tokens/s, Avg generation throughput: 613.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:53:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1154.0 tokens/s, Avg generation throughput: 593.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:53:36 [loggers.py:273] Engine 000: Avg prompt throughput: 929.7 tokens/s, Avg generation throughput: 607.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:53:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1493.7 tokens/s, Avg generation throughput: 567.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:53:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1878.7 tokens/s, Avg generation throughput: 541.7 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:54:06 [loggers.py:273] Engine 000: Avg prompt throughput: 3017.5 tokens/s, Avg generation throughput: 451.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:54:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1712.0 tokens/s, Avg generation throughput: 535.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:54:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1360.3 tokens/s, Avg generation throughput: 587.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:54:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1309.5 tokens/s, Avg generation throughput: 579.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:54:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1549.3 tokens/s, Avg generation throughput: 573.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:54:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1937.3 tokens/s, Avg generation throughput: 527.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:55:06 [loggers.py:273] Engine 000: Avg prompt throughput: 2420.1 tokens/s, Avg generation throughput: 481.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:55:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1782.9 tokens/s, Avg generation throughput: 546.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:55:26 [loggers.py:273] Engine 000: Avg prompt throughput: 2169.3 tokens/s, Avg generation throughput: 520.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:55:36 [loggers.py:273] Engine 000: Avg prompt throughput: 2453.1 tokens/s, Avg generation throughput: 481.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:55:46 [loggers.py:273] Engine 000: Avg prompt throughput: 2320.7 tokens/s, Avg generation throughput: 500.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:55:56 [loggers.py:273] Engine 000: Avg prompt throughput: 2554.7 tokens/s, Avg generation throughput: 481.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:56:06 [loggers.py:273] Engine 000: Avg prompt throughput: 2560.9 tokens/s, Avg generation throughput: 474.7 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:56:16 [loggers.py:273] Engine 000: Avg prompt throughput: 2620.9 tokens/s, Avg generation throughput: 481.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:56:26 [loggers.py:273] Engine 000: Avg prompt throughput: 2456.5 tokens/s, Avg generation throughput: 487.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:56:36 [loggers.py:273] Engine 000: Avg prompt throughput: 2980.7 tokens/s, Avg generation throughput: 454.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:56:46 [loggers.py:273] Engine 000: Avg prompt throughput: 4330.2 tokens/s, Avg generation throughput: 348.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:56:56 [loggers.py:273] Engine 000: Avg prompt throughput: 4092.8 tokens/s, Avg generation throughput: 355.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:57:06 [loggers.py:273] Engine 000: Avg prompt throughput: 4274.7 tokens/s, Avg generation throughput: 362.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:57:16 [loggers.py:273] Engine 000: Avg prompt throughput: 5014.7 tokens/s, Avg generation throughput: 322.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:57:26 [loggers.py:273] Engine 000: Avg prompt throughput: 4439.3 tokens/s, Avg generation throughput: 361.9 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:57:36 [loggers.py:273] Engine 000: Avg prompt throughput: 4835.0 tokens/s, Avg generation throughput: 329.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:57:46 [loggers.py:273] Engine 000: Avg prompt throughput: 4109.8 tokens/s, Avg generation throughput: 387.9 tokens/s, Running: 60 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:57:56 [loggers.py:273] Engine 000: Avg prompt throughput: 4285.9 tokens/s, Avg generation throughput: 368.5 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:58:06 [loggers.py:273] Engine 000: Avg prompt throughput: 4400.8 tokens/s, Avg generation throughput: 374.5 tokens/s, Running: 61 reqs, Waiting: 1 reqs, GPU KV cache usage: 19.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:58:16 [loggers.py:273] Engine 000: Avg prompt throughput: 4280.6 tokens/s, Avg generation throughput: 361.8 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:58:26 [loggers.py:273] Engine 000: Avg prompt throughput: 4616.3 tokens/s, Avg generation throughput: 354.2 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:58:36 [loggers.py:273] Engine 000: Avg prompt throughput: 4228.9 tokens/s, Avg generation throughput: 387.9 tokens/s, Running: 60 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:58:46 [loggers.py:273] Engine 000: Avg prompt throughput: 4035.0 tokens/s, Avg generation throughput: 388.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:58:56 [loggers.py:273] Engine 000: Avg prompt throughput: 4211.5 tokens/s, Avg generation throughput: 374.9 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:59:06 [loggers.py:273] Engine 000: Avg prompt throughput: 4172.2 tokens/s, Avg generation throughput: 381.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:59:16 [loggers.py:273] Engine 000: Avg prompt throughput: 4158.6 tokens/s, Avg generation throughput: 368.1 tokens/s, Running: 63 reqs, Waiting: 1 reqs, GPU KV cache usage: 20.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:59:26 [loggers.py:273] Engine 000: Avg prompt throughput: 3999.3 tokens/s, Avg generation throughput: 408.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:59:36 [loggers.py:273] Engine 000: Avg prompt throughput: 4205.0 tokens/s, Avg generation throughput: 381.6 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:59:46 [loggers.py:273] Engine 000: Avg prompt throughput: 3742.2 tokens/s, Avg generation throughput: 408.2 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 02:59:56 [loggers.py:273] Engine 000: Avg prompt throughput: 3119.5 tokens/s, Avg generation throughput: 454.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:00:06 [loggers.py:273] Engine 000: Avg prompt throughput: 3432.7 tokens/s, Avg generation throughput: 447.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:00:16 [loggers.py:273] Engine 000: Avg prompt throughput: 2664.3 tokens/s, Avg generation throughput: 494.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:00:26 [loggers.py:273] Engine 000: Avg prompt throughput: 2612.5 tokens/s, Avg generation throughput: 494.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:00:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1987.5 tokens/s, Avg generation throughput: 546.7 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:00:46 [loggers.py:273] Engine 000: Avg prompt throughput: 3037.1 tokens/s, Avg generation throughput: 454.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:00:56 [loggers.py:273] Engine 000: Avg prompt throughput: 2300.1 tokens/s, Avg generation throughput: 520.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:01:06 [loggers.py:273] Engine 000: Avg prompt throughput: 2234.5 tokens/s, Avg generation throughput: 533.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:01:16 [loggers.py:273] Engine 000: Avg prompt throughput: 2398.7 tokens/s, Avg generation throughput: 500.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:01:26 [loggers.py:273] Engine 000: Avg prompt throughput: 2355.5 tokens/s, Avg generation throughput: 520.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:01:36 [loggers.py:273] Engine 000: Avg prompt throughput: 2355.9 tokens/s, Avg generation throughput: 513.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:01:46 [loggers.py:273] Engine 000: Avg prompt throughput: 2774.9 tokens/s, Avg generation throughput: 480.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:01:56 [loggers.py:273] Engine 000: Avg prompt throughput: 2462.1 tokens/s, Avg generation throughput: 487.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:02:06 [loggers.py:273] Engine 000: Avg prompt throughput: 2829.1 tokens/s, Avg generation throughput: 493.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:02:16 [loggers.py:273] Engine 000: Avg prompt throughput: 2385.5 tokens/s, Avg generation throughput: 514.1 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:02:26 [loggers.py:273] Engine 000: Avg prompt throughput: 3549.2 tokens/s, Avg generation throughput: 408.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:02:36 [loggers.py:273] Engine 000: Avg prompt throughput: 3797.5 tokens/s, Avg generation throughput: 409.5 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:02:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1585.2 tokens/s, Avg generation throughput: 573.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:02:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1383.1 tokens/s, Avg generation throughput: 580.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:03:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1100.2 tokens/s, Avg generation throughput: 619.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:03:16 [loggers.py:273] Engine 000: Avg prompt throughput: 796.3 tokens/s, Avg generation throughput: 632.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:03:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1180.2 tokens/s, Avg generation throughput: 606.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:03:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1771.5 tokens/s, Avg generation throughput: 560.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:03:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1095.1 tokens/s, Avg generation throughput: 612.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:03:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1109.0 tokens/s, Avg generation throughput: 606.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:04:06 [loggers.py:273] Engine 000: Avg prompt throughput: 427.1 tokens/s, Avg generation throughput: 664.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:04:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1365.9 tokens/s, Avg generation throughput: 580.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:04:26 [loggers.py:273] Engine 000: Avg prompt throughput: 732.8 tokens/s, Avg generation throughput: 638.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:04:36 [loggers.py:273] Engine 000: Avg prompt throughput: 928.4 tokens/s, Avg generation throughput: 625.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:04:46 [loggers.py:273] Engine 000: Avg prompt throughput: 803.4 tokens/s, Avg generation throughput: 619.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:04:56 [loggers.py:273] Engine 000: Avg prompt throughput: 2001.0 tokens/s, Avg generation throughput: 547.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:05:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1312.7 tokens/s, Avg generation throughput: 593.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:05:16 [loggers.py:273] Engine 000: Avg prompt throughput: 556.1 tokens/s, Avg generation throughput: 658.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:05:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1028.8 tokens/s, Avg generation throughput: 619.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:05:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1792.3 tokens/s, Avg generation throughput: 553.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:05:46 [loggers.py:273] Engine 000: Avg prompt throughput: 707.8 tokens/s, Avg generation throughput: 632.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:05:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1258.6 tokens/s, Avg generation throughput: 599.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:06:06 [loggers.py:273] Engine 000: Avg prompt throughput: 675.5 tokens/s, Avg generation throughput: 638.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:06:16 [loggers.py:273] Engine 000: Avg prompt throughput: 717.7 tokens/s, Avg generation throughput: 645.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:06:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1048.6 tokens/s, Avg generation throughput: 612.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:06:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1496.1 tokens/s, Avg generation throughput: 567.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:06:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1107.9 tokens/s, Avg generation throughput: 612.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:06:56 [loggers.py:273] Engine 000: Avg prompt throughput: 788.8 tokens/s, Avg generation throughput: 632.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:07:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1182.2 tokens/s, Avg generation throughput: 586.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:07:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1187.1 tokens/s, Avg generation throughput: 612.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:07:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1042.6 tokens/s, Avg generation throughput: 606.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:07:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1125.3 tokens/s, Avg generation throughput: 612.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:07:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1140.9 tokens/s, Avg generation throughput: 606.0 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:07:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1496.4 tokens/s, Avg generation throughput: 560.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:08:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1087.9 tokens/s, Avg generation throughput: 619.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:08:16 [loggers.py:273] Engine 000: Avg prompt throughput: 790.5 tokens/s, Avg generation throughput: 632.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:08:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1409.2 tokens/s, Avg generation throughput: 586.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:08:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1712.2 tokens/s, Avg generation throughput: 547.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:08:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1262.0 tokens/s, Avg generation throughput: 593.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:08:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1196.1 tokens/s, Avg generation throughput: 599.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:09:06 [loggers.py:273] Engine 000: Avg prompt throughput: 661.8 tokens/s, Avg generation throughput: 645.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:09:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1325.0 tokens/s, Avg generation throughput: 586.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:09:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1531.6 tokens/s, Avg generation throughput: 573.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:09:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1134.7 tokens/s, Avg generation throughput: 599.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:09:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1670.6 tokens/s, Avg generation throughput: 553.7 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:09:56 [loggers.py:273] Engine 000: Avg prompt throughput: 790.4 tokens/s, Avg generation throughput: 632.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:10:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1114.2 tokens/s, Avg generation throughput: 606.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:10:16 [loggers.py:273] Engine 000: Avg prompt throughput: 824.0 tokens/s, Avg generation throughput: 626.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:10:26 [loggers.py:273] Engine 000: Avg prompt throughput: 685.5 tokens/s, Avg generation throughput: 632.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:10:36 [loggers.py:273] Engine 000: Avg prompt throughput: 657.5 tokens/s, Avg generation throughput: 638.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:10:46 [loggers.py:273] Engine 000: Avg prompt throughput: 927.7 tokens/s, Avg generation throughput: 606.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:10:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1787.4 tokens/s, Avg generation throughput: 554.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:11:06 [loggers.py:273] Engine 000: Avg prompt throughput: 973.3 tokens/s, Avg generation throughput: 600.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:11:16 [loggers.py:273] Engine 000: Avg prompt throughput: 797.1 tokens/s, Avg generation throughput: 626.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:11:26 [loggers.py:273] Engine 000: Avg prompt throughput: 547.2 tokens/s, Avg generation throughput: 645.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:11:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1490.1 tokens/s, Avg generation throughput: 567.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:11:46 [loggers.py:273] Engine 000: Avg prompt throughput: 747.1 tokens/s, Avg generation throughput: 632.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:11:56 [loggers.py:273] Engine 000: Avg prompt throughput: 421.3 tokens/s, Avg generation throughput: 645.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:12:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1951.4 tokens/s, Avg generation throughput: 534.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:12:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1133.5 tokens/s, Avg generation throughput: 586.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:12:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1766.8 tokens/s, Avg generation throughput: 548.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:12:36 [loggers.py:273] Engine 000: Avg prompt throughput: 828.5 tokens/s, Avg generation throughput: 613.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:12:46 [loggers.py:273] Engine 000: Avg prompt throughput: 725.4 tokens/s, Avg generation throughput: 619.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:12:56 [loggers.py:273] Engine 000: Avg prompt throughput: 706.9 tokens/s, Avg generation throughput: 632.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:13:06 [loggers.py:273] Engine 000: Avg prompt throughput: 977.2 tokens/s, Avg generation throughput: 606.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:13:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1387.6 tokens/s, Avg generation throughput: 573.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:13:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1397.9 tokens/s, Avg generation throughput: 574.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:13:36 [loggers.py:273] Engine 000: Avg prompt throughput: 848.8 tokens/s, Avg generation throughput: 625.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:13:46 [loggers.py:273] Engine 000: Avg prompt throughput: 383.8 tokens/s, Avg generation throughput: 652.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:13:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1197.7 tokens/s, Avg generation throughput: 586.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:14:06 [loggers.py:273] Engine 000: Avg prompt throughput: 400.6 tokens/s, Avg generation throughput: 665.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:14:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1151.4 tokens/s, Avg generation throughput: 599.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:14:26 [loggers.py:273] Engine 000: Avg prompt throughput: 659.6 tokens/s, Avg generation throughput: 638.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:14:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1504.9 tokens/s, Avg generation throughput: 567.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:14:46 [loggers.py:273] Engine 000: Avg prompt throughput: 518.5 tokens/s, Avg generation throughput: 632.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:14:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1663.0 tokens/s, Avg generation throughput: 560.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:15:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1088.2 tokens/s, Avg generation throughput: 593.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:15:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1076.4 tokens/s, Avg generation throughput: 606.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:15:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1073.3 tokens/s, Avg generation throughput: 606.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:15:36 [loggers.py:273] Engine 000: Avg prompt throughput: 992.5 tokens/s, Avg generation throughput: 613.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:15:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1213.9 tokens/s, Avg generation throughput: 599.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:15:56 [loggers.py:273] Engine 000: Avg prompt throughput: 394.2 tokens/s, Avg generation throughput: 664.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:16:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1325.3 tokens/s, Avg generation throughput: 593.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:16:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1624.8 tokens/s, Avg generation throughput: 560.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:16:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1896.4 tokens/s, Avg generation throughput: 554.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:16:36 [loggers.py:273] Engine 000: Avg prompt throughput: 677.9 tokens/s, Avg generation throughput: 638.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:16:46 [loggers.py:273] Engine 000: Avg prompt throughput: 684.4 tokens/s, Avg generation throughput: 645.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:16:56 [loggers.py:273] Engine 000: Avg prompt throughput: 918.9 tokens/s, Avg generation throughput: 625.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:17:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1586.8 tokens/s, Avg generation throughput: 566.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:17:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1352.2 tokens/s, Avg generation throughput: 593.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:17:26 [loggers.py:273] Engine 000: Avg prompt throughput: 935.3 tokens/s, Avg generation throughput: 625.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:17:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1190.6 tokens/s, Avg generation throughput: 599.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:17:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1014.1 tokens/s, Avg generation throughput: 613.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:17:56 [loggers.py:273] Engine 000: Avg prompt throughput: 809.7 tokens/s, Avg generation throughput: 632.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:18:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1409.6 tokens/s, Avg generation throughput: 586.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:18:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1044.3 tokens/s, Avg generation throughput: 606.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:18:26 [loggers.py:273] Engine 000: Avg prompt throughput: 924.6 tokens/s, Avg generation throughput: 638.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:18:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1284.8 tokens/s, Avg generation throughput: 586.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:18:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1595.6 tokens/s, Avg generation throughput: 573.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:18:56 [loggers.py:273] Engine 000: Avg prompt throughput: 941.6 tokens/s, Avg generation throughput: 619.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:19:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1057.2 tokens/s, Avg generation throughput: 619.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:19:16 [loggers.py:273] Engine 000: Avg prompt throughput: 709.5 tokens/s, Avg generation throughput: 638.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:19:26 [loggers.py:273] Engine 000: Avg prompt throughput: 818.9 tokens/s, Avg generation throughput: 625.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:19:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1638.8 tokens/s, Avg generation throughput: 560.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:19:46 [loggers.py:273] Engine 000: Avg prompt throughput: 829.4 tokens/s, Avg generation throughput: 632.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:19:56 [loggers.py:273] Engine 000: Avg prompt throughput: 893.9 tokens/s, Avg generation throughput: 626.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:20:06 [loggers.py:273] Engine 000: Avg prompt throughput: 296.2 tokens/s, Avg generation throughput: 671.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:20:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1367.6 tokens/s, Avg generation throughput: 586.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:20:26 [loggers.py:273] Engine 000: Avg prompt throughput: 681.1 tokens/s, Avg generation throughput: 645.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:20:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1766.4 tokens/s, Avg generation throughput: 547.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:20:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1332.3 tokens/s, Avg generation throughput: 593.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:20:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1778.3 tokens/s, Avg generation throughput: 554.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:21:06 [loggers.py:273] Engine 000: Avg prompt throughput: 715.1 tokens/s, Avg generation throughput: 638.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:21:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1228.2 tokens/s, Avg generation throughput: 599.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:21:26 [loggers.py:273] Engine 000: Avg prompt throughput: 913.3 tokens/s, Avg generation throughput: 625.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:21:36 [loggers.py:273] Engine 000: Avg prompt throughput: 640.8 tokens/s, Avg generation throughput: 645.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:21:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1669.1 tokens/s, Avg generation throughput: 561.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:21:56 [loggers.py:273] Engine 000: Avg prompt throughput: 799.0 tokens/s, Avg generation throughput: 638.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:22:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1206.2 tokens/s, Avg generation throughput: 599.7 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:22:16 [loggers.py:273] Engine 000: Avg prompt throughput: 654.8 tokens/s, Avg generation throughput: 645.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:22:26 [loggers.py:273] Engine 000: Avg prompt throughput: 785.0 tokens/s, Avg generation throughput: 638.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:22:36 [loggers.py:273] Engine 000: Avg prompt throughput: 674.8 tokens/s, Avg generation throughput: 638.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:22:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1037.0 tokens/s, Avg generation throughput: 606.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:22:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1171.6 tokens/s, Avg generation throughput: 612.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:23:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1808.1 tokens/s, Avg generation throughput: 547.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:23:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1262.0 tokens/s, Avg generation throughput: 593.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:23:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1199.1 tokens/s, Avg generation throughput: 606.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:23:36 [loggers.py:273] Engine 000: Avg prompt throughput: 861.0 tokens/s, Avg generation throughput: 632.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:23:46 [loggers.py:273] Engine 000: Avg prompt throughput: 640.9 tokens/s, Avg generation throughput: 645.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:23:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1033.1 tokens/s, Avg generation throughput: 606.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:24:06 [loggers.py:273] Engine 000: Avg prompt throughput: 703.1 tokens/s, Avg generation throughput: 645.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:24:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1244.4 tokens/s, Avg generation throughput: 599.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:24:26 [loggers.py:273] Engine 000: Avg prompt throughput: 931.4 tokens/s, Avg generation throughput: 619.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:24:36 [loggers.py:273] Engine 000: Avg prompt throughput: 404.3 tokens/s, Avg generation throughput: 665.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:24:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1198.3 tokens/s, Avg generation throughput: 606.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:24:56 [loggers.py:273] Engine 000: Avg prompt throughput: 417.7 tokens/s, Avg generation throughput: 664.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:25:06 [loggers.py:273] Engine 000: Avg prompt throughput: 922.1 tokens/s, Avg generation throughput: 632.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:25:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1281.7 tokens/s, Avg generation throughput: 593.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:25:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1800.1 tokens/s, Avg generation throughput: 554.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:25:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1516.0 tokens/s, Avg generation throughput: 573.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:25:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1398.7 tokens/s, Avg generation throughput: 580.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:25:56 [loggers.py:273] Engine 000: Avg prompt throughput: 520.3 tokens/s, Avg generation throughput: 664.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:26:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1267.6 tokens/s, Avg generation throughput: 586.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:26:16 [loggers.py:273] Engine 000: Avg prompt throughput: 2178.9 tokens/s, Avg generation throughput: 527.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:26:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1233.8 tokens/s, Avg generation throughput: 606.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:26:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1039.7 tokens/s, Avg generation throughput: 612.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:26:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1263.6 tokens/s, Avg generation throughput: 599.4 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:26:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1344.8 tokens/s, Avg generation throughput: 593.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:27:06 [loggers.py:273] Engine 000: Avg prompt throughput: 1255.5 tokens/s, Avg generation throughput: 593.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:27:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1123.1 tokens/s, Avg generation throughput: 606.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:27:26 [loggers.py:273] Engine 000: Avg prompt throughput: 679.7 tokens/s, Avg generation throughput: 645.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:27:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1224.2 tokens/s, Avg generation throughput: 606.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:27:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1486.8 tokens/s, Avg generation throughput: 573.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:27:56 [loggers.py:273] Engine 000: Avg prompt throughput: 982.2 tokens/s, Avg generation throughput: 619.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:28:06 [loggers.py:273] Engine 000: Avg prompt throughput: 391.5 tokens/s, Avg generation throughput: 664.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:28:16 [loggers.py:273] Engine 000: Avg prompt throughput: 674.0 tokens/s, Avg generation throughput: 645.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:28:26 [loggers.py:273] Engine 000: Avg prompt throughput: 2146.0 tokens/s, Avg generation throughput: 521.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:28:36 [loggers.py:273] Engine 000: Avg prompt throughput: 939.8 tokens/s, Avg generation throughput: 619.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:28:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1384.9 tokens/s, Avg generation throughput: 580.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:28:56 [loggers.py:273] Engine 000: Avg prompt throughput: 2646.2 tokens/s, Avg generation throughput: 501.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:29:06 [loggers.py:273] Engine 000: Avg prompt throughput: 2888.6 tokens/s, Avg generation throughput: 461.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:29:16 [loggers.py:273] Engine 000: Avg prompt throughput: 3353.0 tokens/s, Avg generation throughput: 422.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:29:26 [loggers.py:273] Engine 000: Avg prompt throughput: 3260.9 tokens/s, Avg generation throughput: 454.9 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:29:36 [loggers.py:273] Engine 000: Avg prompt throughput: 3460.9 tokens/s, Avg generation throughput: 422.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:29:46 [loggers.py:273] Engine 000: Avg prompt throughput: 3829.8 tokens/s, Avg generation throughput: 382.8 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:29:56 [loggers.py:273] Engine 000: Avg prompt throughput: 4182.6 tokens/s, Avg generation throughput: 389.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:30:06 [loggers.py:273] Engine 000: Avg prompt throughput: 3791.9 tokens/s, Avg generation throughput: 402.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:30:16 [loggers.py:273] Engine 000: Avg prompt throughput: 4081.2 tokens/s, Avg generation throughput: 383.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:30:26 [loggers.py:273] Engine 000: Avg prompt throughput: 4616.3 tokens/s, Avg generation throughput: 335.8 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:30:36 [loggers.py:273] Engine 000: Avg prompt throughput: 4727.3 tokens/s, Avg generation throughput: 343.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:30:46 [loggers.py:273] Engine 000: Avg prompt throughput: 4670.4 tokens/s, Avg generation throughput: 330.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:30:56 [loggers.py:273] Engine 000: Avg prompt throughput: 5077.9 tokens/s, Avg generation throughput: 316.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:31:06 [loggers.py:273] Engine 000: Avg prompt throughput: 4659.7 tokens/s, Avg generation throughput: 328.2 tokens/s, Running: 60 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:31:16 [loggers.py:273] Engine 000: Avg prompt throughput: 5245.5 tokens/s, Avg generation throughput: 303.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:31:26 [loggers.py:273] Engine 000: Avg prompt throughput: 5140.3 tokens/s, Avg generation throughput: 295.3 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:31:36 [loggers.py:273] Engine 000: Avg prompt throughput: 5399.7 tokens/s, Avg generation throughput: 289.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:31:46 [loggers.py:273] Engine 000: Avg prompt throughput: 5039.3 tokens/s, Avg generation throughput: 320.5 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:31:56 [loggers.py:273] Engine 000: Avg prompt throughput: 5296.5 tokens/s, Avg generation throughput: 289.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:32:06 [loggers.py:273] Engine 000: Avg prompt throughput: 5272.2 tokens/s, Avg generation throughput: 302.8 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:32:16 [loggers.py:273] Engine 000: Avg prompt throughput: 5019.1 tokens/s, Avg generation throughput: 315.8 tokens/s, Running: 60 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50170 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50078 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:32:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1385.2 tokens/s, Avg generation throughput: 309.0 tokens/s, Running: 17 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:32:36 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 150.9 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:32:46 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 90.2 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:32:56 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 80.5 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:33:06 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 67.9 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 03:33:16 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 66.0 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 03:33:26 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 65.4 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 03:33:36 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 64.2 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:33:46 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 61.8 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:33:56 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 47.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:34:06 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 43.1 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:34:16 [loggers.py:273] Engine 000: Avg prompt throughput: 399.3 tokens/s, Avg generation throughput: 21.9 tokens/s, Running: 27 reqs, Waiting: 37 reqs, GPU KV cache usage: 6.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 03:34:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1022.1 tokens/s, Avg generation throughput: 610.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 03:34:36 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 678.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 03:34:46 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 684.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:34:56 [loggers.py:273] Engine 000: Avg prompt throughput: 27.3 tokens/s, Avg generation throughput: 684.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:35:06 [loggers.py:273] Engine 000: Avg prompt throughput: 10.8 tokens/s, Avg generation throughput: 684.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44780 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:35:16 [loggers.py:273] Engine 000: Avg prompt throughput: 37.1 tokens/s, Avg generation throughput: 684.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:35:26 [loggers.py:273] Engine 000: Avg prompt throughput: 34.0 tokens/s, Avg generation throughput: 684.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:35:36 [loggers.py:273] Engine 000: Avg prompt throughput: 15.5 tokens/s, Avg generation throughput: 684.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 03:35:46 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 684.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:45164 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:35:56 [loggers.py:273] Engine 000: Avg prompt throughput: 18.3 tokens/s, Avg generation throughput: 690.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 03:36:06 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 697.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44690 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44594 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:36:16 [loggers.py:273] Engine 000: Avg prompt throughput: 17.1 tokens/s, Avg generation throughput: 690.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44662 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44694 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44704 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44724 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44750 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44792 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44844 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44934 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44952 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44954 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44986 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45050 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45128 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45180 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:36:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1683.4 tokens/s, Avg generation throughput: 563.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 03:36:36 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 697.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 03:36:46 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 697.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:36:56 [loggers.py:273] Engine 000: Avg prompt throughput: 37.8 tokens/s, Avg generation throughput: 684.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:37:06 [loggers.py:273] Engine 000: Avg prompt throughput: 64.3 tokens/s, Avg generation throughput: 690.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44662 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44694 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44780 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:37:16 [loggers.py:273] Engine 000: Avg prompt throughput: 87.5 tokens/s, Avg generation throughput: 683.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:37:26 [loggers.py:273] Engine 000: Avg prompt throughput: 86.2 tokens/s, Avg generation throughput: 677.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:37:36 [loggers.py:273] Engine 000: Avg prompt throughput: 8.3 tokens/s, Avg generation throughput: 697.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44750 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:37:46 [loggers.py:273] Engine 000: Avg prompt throughput: 67.7 tokens/s, Avg generation throughput: 684.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:45164 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:37:56 [loggers.py:273] Engine 000: Avg prompt throughput: 70.7 tokens/s, Avg generation throughput: 684.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:38:06 [loggers.py:273] Engine 000: Avg prompt throughput: 23.4 tokens/s, Avg generation throughput: 697.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44690 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:38:16 [loggers.py:273] Engine 000: Avg prompt throughput: 41.7 tokens/s, Avg generation throughput: 684.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44594 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44724 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44704 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44792 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44844 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44952 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44934 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44954 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44986 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45050 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45180 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45128 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:38:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1008.0 tokens/s, Avg generation throughput: 612.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 03:38:36 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 697.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:38:46 [loggers.py:273] Engine 000: Avg prompt throughput: 20.6 tokens/s, Avg generation throughput: 690.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:38:56 [loggers.py:273] Engine 000: Avg prompt throughput: 18.8 tokens/s, Avg generation throughput: 690.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44952 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:39:06 [loggers.py:273] Engine 000: Avg prompt throughput: 64.8 tokens/s, Avg generation throughput: 684.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44792 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44662 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44694 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44780 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:39:16 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 670.6 tokens/s, Running: 58 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45180 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:39:26 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 591.9 tokens/s, Running: 52 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44954 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:39:36 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 544.4 tokens/s, Running: 49 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44844 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45050 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44750 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44952 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:39:46 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 486.7 tokens/s, Running: 42 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:45164 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:39:56 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 434.9 tokens/s, Running: 39 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:40:06 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 418.3 tokens/s, Running: 38 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44690 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44594 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44724 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:40:16 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 401.0 tokens/s, Running: 30 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44704 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44934 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44986 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45128 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:40:26 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 53.1 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 03:40:36 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 44.4 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:40:46 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 40.3 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:40:56 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:44702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:44638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:41:06 [loggers.py:273] Engine 000: Avg prompt throughput: 3671.7 tokens/s, Avg generation throughput: 54.3 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 03:41:16 [loggers.py:273] Engine 000: Avg prompt throughput: 1285.1 tokens/s, Avg generation throughput: 609.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39662 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:41:26 [loggers.py:273] Engine 000: Avg prompt throughput: 150.3 tokens/s, Avg generation throughput: 684.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39306 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39436 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39222 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:41:36 [loggers.py:273] Engine 000: Avg prompt throughput: 304.8 tokens/s, Avg generation throughput: 664.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39194 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39670 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:41:46 [loggers.py:273] Engine 000: Avg prompt throughput: 447.1 tokens/s, Avg generation throughput: 664.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:41:56 [loggers.py:273] Engine 000: Avg prompt throughput: 74.8 tokens/s, Avg generation throughput: 684.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39194 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39306 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:42:06 [loggers.py:273] Engine 000: Avg prompt throughput: 150.6 tokens/s, Avg generation throughput: 678.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39436 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:42:16 [loggers.py:273] Engine 000: Avg prompt throughput: 73.1 tokens/s, Avg generation throughput: 684.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 03:42:26 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 697.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 03:42:36 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 691.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:42:46 [loggers.py:273] Engine 000: Avg prompt throughput: 75.5 tokens/s, Avg generation throughput: 684.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 03:42:56 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 691.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39164 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39168 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39232 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39248 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39278 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39282 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39296 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:43:06 [loggers.py:273] Engine 000: Avg prompt throughput: 382.1 tokens/s, Avg generation throughput: 657.4 tokens/s, Running: 59 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39302 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39316 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39318 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39322 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39334 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39348 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39382 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39398 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39414 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39446 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39460 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39476 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39492 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39502 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39524 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39550 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39576 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39598 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39602 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39654 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39668 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:43:16 [loggers.py:273] Engine 000: Avg prompt throughput: 3696.7 tokens/s, Avg generation throughput: 425.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 03:43:26 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 697.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39662 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39248 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:43:36 [loggers.py:273] Engine 000: Avg prompt throughput: 316.1 tokens/s, Avg generation throughput: 671.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39598 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39222 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39670 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:43:46 [loggers.py:273] Engine 000: Avg prompt throughput: 455.7 tokens/s, Avg generation throughput: 657.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39446 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39654 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:43:56 [loggers.py:273] Engine 000: Avg prompt throughput: 231.3 tokens/s, Avg generation throughput: 671.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39194 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:44:06 [loggers.py:273] Engine 000: Avg prompt throughput: 72.7 tokens/s, Avg generation throughput: 690.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39306 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39282 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39348 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:44:16 [loggers.py:273] Engine 000: Avg prompt throughput: 226.4 tokens/s, Avg generation throughput: 684.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39476 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39446 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39654 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39164 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39436 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39334 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:44:26 [loggers.py:273] Engine 000: Avg prompt throughput: 502.9 tokens/s, Avg generation throughput: 651.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 03:44:36 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 704.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:44:46 [loggers.py:273] Engine 000: Avg prompt throughput: 91.9 tokens/s, Avg generation throughput: 697.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 03:44:56 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 703.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:45:06 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 703.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39168 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39232 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39278 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39296 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39302 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39316 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39318 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39322 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39382 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39398 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39414 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39460 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39492 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39502 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39524 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39550 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39576 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39602 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39668 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:45:16 [loggers.py:273] Engine 000: Avg prompt throughput: 3323.5 tokens/s, Avg generation throughput: 439.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 03:45:26 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 697.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39662 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39436 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39248 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:45:36 [loggers.py:273] Engine 000: Avg prompt throughput: 461.2 tokens/s, Avg generation throughput: 651.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39502 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39598 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39296 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39222 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:45:46 [loggers.py:273] Engine 000: Avg prompt throughput: 552.9 tokens/s, Avg generation throughput: 657.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39670 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39168 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39278 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:45:56 [loggers.py:273] Engine 000: Avg prompt throughput: 458.0 tokens/s, Avg generation throughput: 657.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:46:06 [loggers.py:273] Engine 000: Avg prompt throughput: 74.3 tokens/s, Avg generation throughput: 691.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39194 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39306 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39282 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:46:16 [loggers.py:273] Engine 000: Avg prompt throughput: 297.0 tokens/s, Avg generation throughput: 671.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39602 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39348 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39476 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39446 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39654 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39164 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39654 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:46:26 [loggers.py:273] Engine 000: Avg prompt throughput: 544.1 tokens/s, Avg generation throughput: 651.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39334 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:46:36 [loggers.py:273] Engine 000: Avg prompt throughput: 235.4 tokens/s, Avg generation throughput: 671.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39476 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:46:46 [loggers.py:273] Engine 000: Avg prompt throughput: 159.4 tokens/s, Avg generation throughput: 678.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:46:56 [loggers.py:273] Engine 000: Avg prompt throughput: 77.1 tokens/s, Avg generation throughput: 691.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39306 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:47:06 [loggers.py:273] Engine 000: Avg prompt throughput: 150.4 tokens/s, Avg generation throughput: 684.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39232 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39302 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39316 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39318 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39322 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39382 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39414 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39398 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:47:16 [loggers.py:273] Engine 000: Avg prompt throughput: 786.3 tokens/s, Avg generation throughput: 618.1 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39492 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39460 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39524 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39550 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39576 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39668 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:47:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1766.9 tokens/s, Avg generation throughput: 546.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39662 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:47:36 [loggers.py:273] Engine 000: Avg prompt throughput: 77.6 tokens/s, Avg generation throughput: 684.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39436 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39550 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39248 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39502 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39598 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39296 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:47:46 [loggers.py:273] Engine 000: Avg prompt throughput: 869.9 tokens/s, Avg generation throughput: 605.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39222 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39232 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39670 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39302 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39168 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:47:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1024.6 tokens/s, Avg generation throughput: 605.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39278 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:48:06 [loggers.py:273] Engine 000: Avg prompt throughput: 334.7 tokens/s, Avg generation throughput: 664.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39662 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39194 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:48:16 [loggers.py:273] Engine 000: Avg prompt throughput: 148.4 tokens/s, Avg generation throughput: 671.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39282 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39602 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39348 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:48:26 [loggers.py:273] Engine 000: Avg prompt throughput: 233.9 tokens/s, Avg generation throughput: 671.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39446 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39164 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39550 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39654 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39334 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:48:36 [loggers.py:273] Engine 000: Avg prompt throughput: 527.2 tokens/s, Avg generation throughput: 645.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39502 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:48:46 [loggers.py:273] Engine 000: Avg prompt throughput: 230.0 tokens/s, Avg generation throughput: 677.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39476 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:48:56 [loggers.py:273] Engine 000: Avg prompt throughput: 152.4 tokens/s, Avg generation throughput: 678.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39502 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39164 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39306 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:49:06 [loggers.py:273] Engine 000: Avg prompt throughput: 230.6 tokens/s, Avg generation throughput: 677.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39282 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39348 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:49:16 [loggers.py:273] Engine 000: Avg prompt throughput: 308.3 tokens/s, Avg generation throughput: 671.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39476 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39316 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39318 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39322 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39382 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39414 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39398 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39492 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39460 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39524 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39576 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39668 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:49:26 [loggers.py:273] Engine 000: Avg prompt throughput: 1836.6 tokens/s, Avg generation throughput: 545.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 03:49:36 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 691.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39460 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39436 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39248 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39668 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:49:46 [loggers.py:273] Engine 000: Avg prompt throughput: 448.2 tokens/s, Avg generation throughput: 657.8 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39476 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39598 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39296 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39222 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39436 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39232 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:49:56 [loggers.py:273] Engine 000: Avg prompt throughput: 1176.5 tokens/s, Avg generation throughput: 598.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39670 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39302 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39168 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39278 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:50:06 [loggers.py:273] Engine 000: Avg prompt throughput: 792.1 tokens/s, Avg generation throughput: 625.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39662 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:50:16 [loggers.py:273] Engine 000: Avg prompt throughput: 149.8 tokens/s, Avg generation throughput: 677.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39194 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39670 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:50:26 [loggers.py:273] Engine 000: Avg prompt throughput: 407.8 tokens/s, Avg generation throughput: 664.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39602 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39446 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39668 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39550 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:50:36 [loggers.py:273] Engine 000: Avg prompt throughput: 458.1 tokens/s, Avg generation throughput: 651.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39654 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39334 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:50:46 [loggers.py:273] Engine 000: Avg prompt throughput: 339.8 tokens/s, Avg generation throughput: 664.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39668 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:50:56 [loggers.py:273] Engine 000: Avg prompt throughput: 156.7 tokens/s, Avg generation throughput: 684.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:51:06 [loggers.py:273] Engine 000: Avg prompt throughput: 168.2 tokens/s, Avg generation throughput: 684.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39502 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39164 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39306 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39446 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39282 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39348 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:51:16 [loggers.py:273] Engine 000: Avg prompt throughput: 608.4 tokens/s, Avg generation throughput: 644.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39502 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39662 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:51:26 [loggers.py:273] Engine 000: Avg prompt throughput: 147.2 tokens/s, Avg generation throughput: 671.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39316 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39318 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39322 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39382 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39414 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39398 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39492 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39524 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39576 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39654 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:51:36 [loggers.py:273] Engine 000: Avg prompt throughput: 1381.5 tokens/s, Avg generation throughput: 572.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39502 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:51:46 [loggers.py:273] Engine 000: Avg prompt throughput: 161.5 tokens/s, Avg generation throughput: 671.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39460 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39492 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39248 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39476 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:51:56 [loggers.py:273] Engine 000: Avg prompt throughput: 756.0 tokens/s, Avg generation throughput: 625.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39598 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39296 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39382 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39222 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39436 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39232 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39318 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39302 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:52:06 [loggers.py:273] Engine 000: Avg prompt throughput: 931.6 tokens/s, Avg generation throughput: 611.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39168 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39278 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39414 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:52:16 [loggers.py:273] Engine 000: Avg prompt throughput: 690.3 tokens/s, Avg generation throughput: 632.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39576 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39382 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39318 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39194 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:52:26 [loggers.py:273] Engine 000: Avg prompt throughput: 403.5 tokens/s, Avg generation throughput: 658.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39302 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39414 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39670 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:52:36 [loggers.py:273] Engine 000: Avg prompt throughput: 616.6 tokens/s, Avg generation throughput: 638.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39492 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39602 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39316 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39550 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39334 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:52:46 [loggers.py:273] Engine 000: Avg prompt throughput: 410.9 tokens/s, Avg generation throughput: 645.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:52:56 [loggers.py:273] Engine 000: Avg prompt throughput: 228.6 tokens/s, Avg generation throughput: 665.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39668 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39492 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39194 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:53:06 [loggers.py:273] Engine 000: Avg prompt throughput: 384.8 tokens/s, Avg generation throughput: 658.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39670 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39164 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:53:16 [loggers.py:273] Engine 000: Avg prompt throughput: 242.8 tokens/s, Avg generation throughput: 677.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39306 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39446 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39282 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39348 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:53:26 [loggers.py:273] Engine 000: Avg prompt throughput: 396.2 tokens/s, Avg generation throughput: 658.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39662 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39322 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39398 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39524 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:53:36 [loggers.py:273] Engine 000: Avg prompt throughput: 535.5 tokens/s, Avg generation throughput: 644.3 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39654 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:53:46 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 643.0 tokens/s, Running: 59 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39502 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39460 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:53:56 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 615.0 tokens/s, Running: 55 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39248 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39476 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39598 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39296 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39222 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39436 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39232 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:54:06 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 530.2 tokens/s, Running: 44 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39524 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39168 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39278 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:54:16 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 437.3 tokens/s, Running: 37 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39576 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39382 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39318 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:54:26 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 372.8 tokens/s, Running: 31 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39670 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39302 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39414 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:54:36 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 302.3 tokens/s, Running: 24 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39602 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39316 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39550 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:54:46 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 238.7 tokens/s, Running: 20 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39334 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:54:56 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 197.9 tokens/s, Running: 17 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39668 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39492 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39194 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:55:06 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 161.6 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39164 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39306 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39446 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:55:16 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 119.6 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39282 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39348 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39662 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:55:26 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 72.4 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:39364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39322 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39398 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:39538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:55:36 [loggers.py:273] Engine 000: Avg prompt throughput: 393.6 tokens/s, Avg generation throughput: 181.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50648 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50780 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50654 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51032 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50556 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50962 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50740 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50734 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50970 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50700 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:55:46 [loggers.py:273] Engine 000: Avg prompt throughput: 140.8 tokens/s, Avg generation throughput: 648.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50556 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51032 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50518 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50670 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50734 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51004 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:55:56 [loggers.py:273] Engine 000: Avg prompt throughput: 82.6 tokens/s, Avg generation throughput: 662.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50962 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50794 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50518 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:56:06 [loggers.py:273] Engine 000: Avg prompt throughput: 84.1 tokens/s, Avg generation throughput: 662.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:51004 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51004 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50794 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50740 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:56:16 [loggers.py:273] Engine 000: Avg prompt throughput: 50.8 tokens/s, Avg generation throughput: 670.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50734 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50654 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:56:26 [loggers.py:273] Engine 000: Avg prompt throughput: 70.0 tokens/s, Avg generation throughput: 669.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50740 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50986 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50794 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51032 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:56:36 [loggers.py:273] Engine 000: Avg prompt throughput: 46.5 tokens/s, Avg generation throughput: 670.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50518 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50780 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50986 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:56:46 [loggers.py:273] Engine 000: Avg prompt throughput: 43.6 tokens/s, Avg generation throughput: 676.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50986 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51020 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50780 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50878 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51032 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51032 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:56:56 [loggers.py:273] Engine 000: Avg prompt throughput: 57.7 tokens/s, Avg generation throughput: 670.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:51058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50780 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51004 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:57:06 [loggers.py:273] Engine 000: Avg prompt throughput: 61.2 tokens/s, Avg generation throughput: 670.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51004 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51020 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:57:16 [loggers.py:273] Engine 000: Avg prompt throughput: 43.7 tokens/s, Avg generation throughput: 676.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50986 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51004 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50878 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50878 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51004 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50780 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:57:26 [loggers.py:273] Engine 000: Avg prompt throughput: 61.5 tokens/s, Avg generation throughput: 676.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51004 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50518 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50780 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50492 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50550 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50720 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50934 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51040 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:57:36 [loggers.py:273] Engine 000: Avg prompt throughput: 52.0 tokens/s, Avg generation throughput: 668.1 tokens/s, Running: 59 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50648 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50986 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51020 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50550 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:57:46 [loggers.py:273] Engine 000: Avg prompt throughput: 221.0 tokens/s, Avg generation throughput: 634.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50970 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50700 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50970 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50878 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50556 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50556 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50670 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51020 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:57:56 [loggers.py:273] Engine 000: Avg prompt throughput: 62.5 tokens/s, Avg generation throughput: 662.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50986 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50780 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51020 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50670 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50962 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50878 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51040 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50720 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51020 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50670 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:58:06 [loggers.py:273] Engine 000: Avg prompt throughput: 100.2 tokens/s, Avg generation throughput: 642.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50780 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50556 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50550 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50780 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51020 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:58:16 [loggers.py:273] Engine 000: Avg prompt throughput: 94.5 tokens/s, Avg generation throughput: 655.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:51040 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50734 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50734 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51040 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50556 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50670 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50962 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51020 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50556 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:58:26 [loggers.py:273] Engine 000: Avg prompt throughput: 132.1 tokens/s, Avg generation throughput: 648.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50700 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51020 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50556 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50654 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50740 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50654 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51040 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:58:36 [loggers.py:273] Engine 000: Avg prompt throughput: 138.1 tokens/s, Avg generation throughput: 641.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50654 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50986 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50794 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50970 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50550 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50654 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50962 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:58:46 [loggers.py:273] Engine 000: Avg prompt throughput: 177.2 tokens/s, Avg generation throughput: 634.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50740 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50654 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50986 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50970 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50654 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50878 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:58:56 [loggers.py:273] Engine 000: Avg prompt throughput: 103.2 tokens/s, Avg generation throughput: 661.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51032 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50962 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:59:06 [loggers.py:273] Engine 000: Avg prompt throughput: 67.3 tokens/s, Avg generation throughput: 669.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:59:16 [loggers.py:273] Engine 000: Avg prompt throughput: 67.2 tokens/s, Avg generation throughput: 669.6 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50962 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:59:26 [loggers.py:273] Engine 000: Avg prompt throughput: 49.7 tokens/s, Avg generation throughput: 670.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51004 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:59:36 [loggers.py:273] Engine 000: Avg prompt throughput: 69.6 tokens/s, Avg generation throughput: 663.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50518 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51032 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50492 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50934 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50648 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:59:46 [loggers.py:273] Engine 000: Avg prompt throughput: 59.9 tokens/s, Avg generation throughput: 663.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50934 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50986 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51032 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50648 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 03:59:56 [loggers.py:273] Engine 000: Avg prompt throughput: 84.9 tokens/s, Avg generation throughput: 662.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:51030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51032 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50734 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50492 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51004 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:00:06 [loggers.py:273] Engine 000: Avg prompt throughput: 121.4 tokens/s, Avg generation throughput: 655.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50720 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50492 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:00:16 [loggers.py:273] Engine 000: Avg prompt throughput: 55.8 tokens/s, Avg generation throughput: 669.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:51004 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50780 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51032 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:00:26 [loggers.py:273] Engine 000: Avg prompt throughput: 83.1 tokens/s, Avg generation throughput: 669.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50670 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50670 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50700 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51020 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50556 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50556 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:00:36 [loggers.py:273] Engine 000: Avg prompt throughput: 90.1 tokens/s, Avg generation throughput: 649.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51040 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50794 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50794 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50700 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50700 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:00:46 [loggers.py:273] Engine 000: Avg prompt throughput: 98.7 tokens/s, Avg generation throughput: 649.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50550 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50700 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50740 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50740 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50970 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50970 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50654 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:00:56 [loggers.py:273] Engine 000: Avg prompt throughput: 109.2 tokens/s, Avg generation throughput: 653.5 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50878 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50492 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:01:06 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 610.1 tokens/s, Running: 51 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50550 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51020 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:01:16 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 510.5 tokens/s, Running: 43 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50962 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:01:26 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 417.5 tokens/s, Running: 36 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51004 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:01:36 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 361.6 tokens/s, Running: 30 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50518 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:01:46 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 310.0 tokens/s, Running: 28 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50934 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50986 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50648 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:01:56 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 271.3 tokens/s, Running: 22 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50734 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:02:06 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 236.9 tokens/s, Running: 19 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50720 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:02:16 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 182.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50780 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50740 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51032 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50794 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:02:26 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 127.4 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50670 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50556 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:51040 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:02:36 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 69.0 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:50686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50700 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:50892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:02:46 [loggers.py:273] Engine 000: Avg prompt throughput: 415.6 tokens/s, Avg generation throughput: 29.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:02:56 [loggers.py:273] Engine 000: Avg prompt throughput: 118.8 tokens/s, Avg generation throughput: 691.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55328 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:03:06 [loggers.py:273] Engine 000: Avg prompt throughput: 62.7 tokens/s, Avg generation throughput: 676.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55628 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55410 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55574 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55354 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55738 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:03:16 [loggers.py:273] Engine 000: Avg prompt throughput: 194.2 tokens/s, Avg generation throughput: 641.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55426 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55412 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55396 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55708 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55464 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:03:26 [loggers.py:273] Engine 000: Avg prompt throughput: 156.6 tokens/s, Avg generation throughput: 661.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55768 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55726 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:03:36 [loggers.py:273] Engine 000: Avg prompt throughput: 112.6 tokens/s, Avg generation throughput: 663.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55482 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55354 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55426 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55628 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55494 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55464 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55412 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55574 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:03:46 [loggers.py:273] Engine 000: Avg prompt throughput: 210.2 tokens/s, Avg generation throughput: 647.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55410 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55396 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55738 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55726 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:03:56 [loggers.py:273] Engine 000: Avg prompt throughput: 156.8 tokens/s, Avg generation throughput: 656.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55628 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55708 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55482 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55354 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:04:06 [loggers.py:273] Engine 000: Avg prompt throughput: 152.0 tokens/s, Avg generation throughput: 661.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55426 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55768 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:04:16 [loggers.py:273] Engine 000: Avg prompt throughput: 99.4 tokens/s, Avg generation throughput: 663.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55494 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55574 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:04:26 [loggers.py:273] Engine 000: Avg prompt throughput: 144.2 tokens/s, Avg generation throughput: 656.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55738 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55628 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55708 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:04:36 [loggers.py:273] Engine 000: Avg prompt throughput: 113.3 tokens/s, Avg generation throughput: 669.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55396 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55482 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55768 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:04:46 [loggers.py:273] Engine 000: Avg prompt throughput: 97.8 tokens/s, Avg generation throughput: 669.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55494 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55420 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55354 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55426 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:04:56 [loggers.py:273] Engine 000: Avg prompt throughput: 164.4 tokens/s, Avg generation throughput: 648.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55628 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55738 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55482 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55396 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55328 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:05:06 [loggers.py:273] Engine 000: Avg prompt throughput: 179.5 tokens/s, Avg generation throughput: 648.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55768 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:05:16 [loggers.py:273] Engine 000: Avg prompt throughput: 82.1 tokens/s, Avg generation throughput: 670.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55494 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55426 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55708 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55412 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:05:26 [loggers.py:273] Engine 000: Avg prompt throughput: 161.1 tokens/s, Avg generation throughput: 655.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55354 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55420 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55396 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55328 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55628 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:05:36 [loggers.py:273] Engine 000: Avg prompt throughput: 151.0 tokens/s, Avg generation throughput: 655.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55482 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55412 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55328 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55494 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55768 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:05:46 [loggers.py:273] Engine 000: Avg prompt throughput: 108.0 tokens/s, Avg generation throughput: 669.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55464 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55410 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55420 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55426 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:05:56 [loggers.py:273] Engine 000: Avg prompt throughput: 184.3 tokens/s, Avg generation throughput: 654.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55726 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55738 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:06:06 [loggers.py:273] Engine 000: Avg prompt throughput: 130.2 tokens/s, Avg generation throughput: 662.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55482 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55354 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55464 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55412 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55420 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55410 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:06:16 [loggers.py:273] Engine 000: Avg prompt throughput: 147.8 tokens/s, Avg generation throughput: 655.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55494 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55328 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55396 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55574 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55628 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55738 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:06:26 [loggers.py:273] Engine 000: Avg prompt throughput: 165.2 tokens/s, Avg generation throughput: 668.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55768 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55726 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:06:36 [loggers.py:273] Engine 000: Avg prompt throughput: 180.0 tokens/s, Avg generation throughput: 661.7 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55482 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55574 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:06:46 [loggers.py:273] Engine 000: Avg prompt throughput: 120.0 tokens/s, Avg generation throughput: 662.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55412 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55410 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55420 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55464 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55354 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:06:56 [loggers.py:273] Engine 000: Avg prompt throughput: 121.8 tokens/s, Avg generation throughput: 662.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55726 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55328 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55426 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55628 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55494 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:07:06 [loggers.py:273] Engine 000: Avg prompt throughput: 142.8 tokens/s, Avg generation throughput: 655.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55354 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55412 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55768 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55420 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:07:16 [loggers.py:273] Engine 000: Avg prompt throughput: 113.6 tokens/s, Avg generation throughput: 662.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55426 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55726 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55410 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55464 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55708 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55396 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:07:26 [loggers.py:273] Engine 000: Avg prompt throughput: 205.8 tokens/s, Avg generation throughput: 648.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55574 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55354 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55768 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55738 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55412 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55328 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55482 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:07:36 [loggers.py:273] Engine 000: Avg prompt throughput: 144.9 tokens/s, Avg generation throughput: 662.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55708 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:07:46 [loggers.py:273] Engine 000: Avg prompt throughput: 126.3 tokens/s, Avg generation throughput: 662.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55726 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55426 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55574 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55410 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55396 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:07:56 [loggers.py:273] Engine 000: Avg prompt throughput: 114.9 tokens/s, Avg generation throughput: 662.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55768 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55420 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:08:06 [loggers.py:273] Engine 000: Avg prompt throughput: 105.8 tokens/s, Avg generation throughput: 663.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55482 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55426 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55412 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55708 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:08:16 [loggers.py:273] Engine 000: Avg prompt throughput: 178.4 tokens/s, Avg generation throughput: 648.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55328 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55574 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55410 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55464 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55420 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55726 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:08:26 [loggers.py:273] Engine 000: Avg prompt throughput: 119.0 tokens/s, Avg generation throughput: 656.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55768 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55482 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55412 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55628 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:08:36 [loggers.py:273] Engine 000: Avg prompt throughput: 130.0 tokens/s, Avg generation throughput: 643.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55328 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55426 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:08:46 [loggers.py:273] Engine 000: Avg prompt throughput: 97.3 tokens/s, Avg generation throughput: 663.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55574 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55738 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55482 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:08:56 [loggers.py:273] Engine 000: Avg prompt throughput: 142.4 tokens/s, Avg generation throughput: 649.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55464 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55726 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55420 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55768 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55412 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55494 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55426 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:09:06 [loggers.py:273] Engine 000: Avg prompt throughput: 152.2 tokens/s, Avg generation throughput: 649.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55768 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55738 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:09:16 [loggers.py:273] Engine 000: Avg prompt throughput: 102.2 tokens/s, Avg generation throughput: 656.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55574 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55628 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55412 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55482 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55410 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55464 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55420 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55726 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:09:26 [loggers.py:273] Engine 000: Avg prompt throughput: 198.2 tokens/s, Avg generation throughput: 641.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55426 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55354 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55328 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:09:36 [loggers.py:273] Engine 000: Avg prompt throughput: 164.1 tokens/s, Avg generation throughput: 642.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55574 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:09:46 [loggers.py:273] Engine 000: Avg prompt throughput: 120.3 tokens/s, Avg generation throughput: 656.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55412 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55420 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55726 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55328 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:09:56 [loggers.py:273] Engine 000: Avg prompt throughput: 140.9 tokens/s, Avg generation throughput: 649.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55628 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55768 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55396 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55738 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55410 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55354 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:10:06 [loggers.py:273] Engine 000: Avg prompt throughput: 165.2 tokens/s, Avg generation throughput: 648.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55574 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55494 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:10:16 [loggers.py:273] Engine 000: Avg prompt throughput: 186.6 tokens/s, Avg generation throughput: 648.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55708 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55328 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55628 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55426 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55410 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:10:26 [loggers.py:273] Engine 000: Avg prompt throughput: 150.0 tokens/s, Avg generation throughput: 655.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55768 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55574 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55494 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:10:36 [loggers.py:273] Engine 000: Avg prompt throughput: 133.4 tokens/s, Avg generation throughput: 656.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55628 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55420 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:10:46 [loggers.py:273] Engine 000: Avg prompt throughput: 118.4 tokens/s, Avg generation throughput: 656.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55426 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55628 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55410 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:10:56 [loggers.py:273] Engine 000: Avg prompt throughput: 140.4 tokens/s, Avg generation throughput: 655.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55396 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:11:06 [loggers.py:273] Engine 000: Avg prompt throughput: 94.7 tokens/s, Avg generation throughput: 663.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55420 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55574 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55708 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55410 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:11:16 [loggers.py:273] Engine 000: Avg prompt throughput: 152.3 tokens/s, Avg generation throughput: 648.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55426 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55738 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:11:26 [loggers.py:273] Engine 000: Avg prompt throughput: 93.8 tokens/s, Avg generation throughput: 663.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55574 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55482 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55464 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55628 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55494 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55396 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55708 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55328 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:11:36 [loggers.py:273] Engine 000: Avg prompt throughput: 203.2 tokens/s, Avg generation throughput: 648.1 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55410 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:11:46 [loggers.py:273] Engine 000: Avg prompt throughput: 159.6 tokens/s, Avg generation throughput: 649.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55426 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55738 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55420 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55412 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55464 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:11:56 [loggers.py:273] Engine 000: Avg prompt throughput: 87.3 tokens/s, Avg generation throughput: 670.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55574 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55726 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55494 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:12:06 [loggers.py:273] Engine 000: Avg prompt throughput: 108.5 tokens/s, Avg generation throughput: 662.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55628 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55410 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55354 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55328 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55396 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55482 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:12:16 [loggers.py:273] Engine 000: Avg prompt throughput: 157.7 tokens/s, Avg generation throughput: 649.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55412 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55726 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:12:26 [loggers.py:273] Engine 000: Avg prompt throughput: 145.2 tokens/s, Avg generation throughput: 656.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55420 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55738 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55328 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55768 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:12:36 [loggers.py:273] Engine 000: Avg prompt throughput: 142.8 tokens/s, Avg generation throughput: 662.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55628 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55396 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55494 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55412 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:12:46 [loggers.py:273] Engine 000: Avg prompt throughput: 137.7 tokens/s, Avg generation throughput: 656.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55354 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:12:56 [loggers.py:273] Engine 000: Avg prompt throughput: 111.1 tokens/s, Avg generation throughput: 662.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55410 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55726 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55328 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55628 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:13:06 [loggers.py:273] Engine 000: Avg prompt throughput: 147.3 tokens/s, Avg generation throughput: 655.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55482 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55708 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55412 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55410 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:13:16 [loggers.py:273] Engine 000: Avg prompt throughput: 135.4 tokens/s, Avg generation throughput: 669.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55354 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55396 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55726 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:13:26 [loggers.py:273] Engine 000: Avg prompt throughput: 136.6 tokens/s, Avg generation throughput: 662.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55420 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55482 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55410 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:13:36 [loggers.py:273] Engine 000: Avg prompt throughput: 98.4 tokens/s, Avg generation throughput: 662.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55412 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55628 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55768 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55328 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55396 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:13:46 [loggers.py:273] Engine 000: Avg prompt throughput: 196.8 tokens/s, Avg generation throughput: 641.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55354 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55410 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55426 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55708 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:13:56 [loggers.py:273] Engine 000: Avg prompt throughput: 96.0 tokens/s, Avg generation throughput: 663.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55464 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55574 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55482 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:14:06 [loggers.py:273] Engine 000: Avg prompt throughput: 94.8 tokens/s, Avg generation throughput: 669.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55426 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55494 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55768 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55328 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55412 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55410 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55396 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:14:16 [loggers.py:273] Engine 000: Avg prompt throughput: 210.2 tokens/s, Avg generation throughput: 641.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55574 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:14:26 [loggers.py:273] Engine 000: Avg prompt throughput: 103.1 tokens/s, Avg generation throughput: 669.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55354 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55726 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55464 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55738 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:14:36 [loggers.py:273] Engine 000: Avg prompt throughput: 127.2 tokens/s, Avg generation throughput: 663.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55396 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55494 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55768 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:14:46 [loggers.py:273] Engine 000: Avg prompt throughput: 111.0 tokens/s, Avg generation throughput: 669.6 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55482 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55328 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55464 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55426 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:14:56 [loggers.py:273] Engine 000: Avg prompt throughput: 98.2 tokens/s, Avg generation throughput: 668.7 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55574 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55494 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55396 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:15:06 [loggers.py:273] Engine 000: Avg prompt throughput: 189.4 tokens/s, Avg generation throughput: 649.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55738 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55354 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55726 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55768 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55464 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:15:16 [loggers.py:273] Engine 000: Avg prompt throughput: 133.2 tokens/s, Avg generation throughput: 669.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55482 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55328 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:15:26 [loggers.py:273] Engine 000: Avg prompt throughput: 124.0 tokens/s, Avg generation throughput: 656.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55426 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55420 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55726 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:15:36 [loggers.py:273] Engine 000: Avg prompt throughput: 119.4 tokens/s, Avg generation throughput: 656.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55574 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55354 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55738 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55768 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55494 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55396 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55628 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55426 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:15:46 [loggers.py:273] Engine 000: Avg prompt throughput: 167.3 tokens/s, Avg generation throughput: 660.7 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55482 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55328 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55708 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:15:56 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 564.9 tokens/s, Running: 43 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55412 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55464 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55420 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55738 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:16:06 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 398.2 tokens/s, Running: 29 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55574 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55726 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55628 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55494 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55354 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:16:16 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 197.1 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55410 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55768 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55568 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:16:26 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 95.1 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:55812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:16:36 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 59.4 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55364 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:16:46 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 46.5 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:16:56 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 43.8 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:17:06 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:17:16 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:17:26 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:17:36 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:55396 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:17:46 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:17:56 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:33288 - "GET /tokenizer_info HTTP/1.1" 404 Not Found (APIServer pid=1162132) INFO: 127.0.0.1:33288 - "GET /tokenizer_info HTTP/1.1" 404 Not Found (APIServer pid=1162132) INFO: 127.0.0.1:33288 - "GET /tokenizer_info HTTP/1.1" 404 Not Found (APIServer pid=1162132) INFO: 127.0.0.1:38474 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:18:46 [loggers.py:273] Engine 000: Avg prompt throughput: 1323.5 tokens/s, Avg generation throughput: 569.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:38682 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:18:56 [loggers.py:273] Engine 000: Avg prompt throughput: 11.8 tokens/s, Avg generation throughput: 691.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:19:06 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 697.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:19:16 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 697.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:19:26 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 697.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:19:36 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 697.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:19:46 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 691.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:19:56 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 697.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.5%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:20:06 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 703.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:38332 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38346 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38348 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38350 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38354 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38364 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38368 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38382 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38394 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38396 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38412 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38418 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38422 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38428 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38440 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38444 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38454 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38470 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38490 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38500 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38508 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38520 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38526 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38528 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38540 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38542 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38554 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38556 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38572 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38586 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38590 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38598 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38608 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38616 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38626 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38632 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38644 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38646 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38660 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38668 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38694 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38710 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38714 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38724 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38730 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38732 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38736 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38738 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38748 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38760 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38774 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38780 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38792 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38796 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38802 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38812 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38814 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38818 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38822 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38832 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38846 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38862 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38474 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38796 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:20:16 [loggers.py:273] Engine 000: Avg prompt throughput: 722.6 tokens/s, Avg generation throughput: 633.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:38710 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38598 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38682 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:20:26 [loggers.py:273] Engine 000: Avg prompt throughput: 32.8 tokens/s, Avg generation throughput: 690.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:20:36 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 703.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:20:46 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 703.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:20:56 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 697.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:21:06 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 697.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:21:16 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 703.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:21:26 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 697.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:21:36 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 704.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:38332 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:21:46 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 697.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:38354 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38364 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38348 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38346 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38350 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38368 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38440 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38418 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38422 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38394 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38428 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38396 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38382 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38412 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38508 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38490 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38526 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38520 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38454 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38500 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38444 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38470 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38616 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38556 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38608 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38542 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38528 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38586 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38554 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38540 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38590 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38572 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38644 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38730 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38660 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38714 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38626 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38632 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38694 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38724 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38646 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38668 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38732 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38738 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38774 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38792 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38814 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38736 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38780 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38802 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38818 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38760 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38812 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38748 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38862 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38832 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38846 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38822 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38474 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38796 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38710 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38598 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:21:56 [loggers.py:273] Engine 000: Avg prompt throughput: 235.9 tokens/s, Avg generation throughput: 369.4 tokens/s, Running: 32 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.2%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:38682 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:22:06 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 341.8 tokens/s, Running: 31 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:22:16 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 341.0 tokens/s, Running: 31 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:22:26 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 341.0 tokens/s, Running: 31 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:22:36 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 337.9 tokens/s, Running: 31 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:22:46 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 341.0 tokens/s, Running: 31 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:22:56 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 341.0 tokens/s, Running: 31 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:23:06 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 341.0 tokens/s, Running: 31 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:23:16 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 337.9 tokens/s, Running: 31 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:38332 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38354 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38364 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38348 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38346 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38350 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38368 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38440 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38422 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38418 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38394 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38428 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38396 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38382 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38412 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38490 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38526 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38520 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38454 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38500 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38444 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38470 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38508 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38616 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38556 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38608 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38542 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38528 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38586 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38554 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:38540 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:23:26 [loggers.py:273] Engine 000: Avg prompt throughput: 4212.1 tokens/s, Avg generation throughput: 181.8 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 13.9%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:45730 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45958 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45794 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45740 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45758 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45764 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45938 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45810 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45848 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45820 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45816 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45912 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46158 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45830 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45982 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46092 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45822 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46148 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45882 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45954 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46042 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46136 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46190 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45872 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46024 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46048 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45998 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46058 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45796 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45878 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46098 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46262 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45860 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46218 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46064 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45936 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46172 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45968 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45764 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46238 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46208 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46248 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46232 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46202 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46236 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46176 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45758 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46040 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46106 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46268 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45900 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45912 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45816 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:23:36 [loggers.py:273] Engine 000: Avg prompt throughput: 5405.5 tokens/s, Avg generation throughput: 289.7 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:45740 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46158 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45882 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45772 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46048 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45848 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46118 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45822 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46236 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45764 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46172 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45938 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46042 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46024 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45796 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45830 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46262 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46064 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46248 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46148 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45936 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45820 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45758 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45730 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46218 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45968 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46136 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46190 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45998 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45860 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46176 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45900 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46158 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46092 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46232 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46048 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45764 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46268 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45912 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45740 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45772 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45954 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46024 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46208 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45816 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46148 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46118 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46190 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46236 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46174 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45848 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46218 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45998 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45936 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45820 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46136 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45938 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46262 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45822 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46248 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45796 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:23:46 [loggers.py:273] Engine 000: Avg prompt throughput: 4335.1 tokens/s, Avg generation throughput: 358.5 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:45968 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45912 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46158 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46092 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46148 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45830 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46064 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46024 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45900 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45860 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45764 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46208 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46268 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46174 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45938 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45848 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45998 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46262 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45772 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46190 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46218 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46176 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46092 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46064 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45822 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45796 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45968 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45912 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46236 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45820 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45764 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46024 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46148 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45830 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45938 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45772 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46174 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46268 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46262 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45912 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46106 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45998 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46190 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46064 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45822 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:23:56 [loggers.py:273] Engine 000: Avg prompt throughput: 3139.4 tokens/s, Avg generation throughput: 438.5 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.6%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:46158 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46024 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45938 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45848 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46148 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46174 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46248 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45764 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46190 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45830 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45820 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45822 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45796 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46262 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45912 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45938 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46232 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46148 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46106 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45796 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46248 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45830 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45722 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46232 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45750 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46268 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45788 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45836 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45848 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45842 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46158 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45892 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45926 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46106 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46022 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45912 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46076 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:24:06 [loggers.py:273] Engine 000: Avg prompt throughput: 2727.9 tokens/s, Avg generation throughput: 491.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:46130 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45764 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45750 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46258 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46148 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45822 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46232 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45958 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45788 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45794 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45830 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45938 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45810 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46236 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45836 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45764 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45982 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45796 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46268 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46130 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45842 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45872 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46058 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45878 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46098 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46238 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45830 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46202 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46022 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45810 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46040 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46148 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45794 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45958 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46076 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45882 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45912 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45750 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45788 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45836 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45892 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46058 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46172 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45872 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45926 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46258 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45842 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45982 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45830 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46042 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46268 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:24:16 [loggers.py:273] Engine 000: Avg prompt throughput: 3373.5 tokens/s, Avg generation throughput: 437.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:45822 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45796 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45758 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46232 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45730 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46238 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46130 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45912 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46048 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46202 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46098 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45740 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46058 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45750 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45954 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45788 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45878 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45836 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45816 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46118 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46076 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46238 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45796 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45758 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45830 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46136 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46040 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45730 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45936 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46258 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46048 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45892 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46172 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45822 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46058 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45954 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45788 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45900 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46118 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45860 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46208 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45750 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46130 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45836 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46258 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45842 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46040 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45816 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46076 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46202 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45740 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46238 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45892 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46218 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45936 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:24:26 [loggers.py:273] Engine 000: Avg prompt throughput: 3677.1 tokens/s, Avg generation throughput: 436.8 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.1%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:45900 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46176 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45878 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46092 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45954 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45968 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46098 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45796 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46208 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45730 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46048 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45830 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46092 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45772 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46058 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46176 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45750 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45788 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46258 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45830 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45998 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46064 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46040 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45936 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46202 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45796 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46092 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46098 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46208 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46048 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46024 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46176 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45750 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45892 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46058 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46174 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45816 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46190 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46258 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45820 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45730 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45998 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46262 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45772 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45878 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45968 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46098 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:24:36 [loggers.py:273] Engine 000: Avg prompt throughput: 3106.1 tokens/s, Avg generation throughput: 464.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:45740 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45830 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46202 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45892 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45820 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46248 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45878 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46098 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45722 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46176 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45848 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46058 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45830 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46158 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45740 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45968 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46106 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46208 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45998 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45730 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45772 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45722 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45816 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45878 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46048 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46202 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45938 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46248 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46236 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45820 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45848 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46106 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45968 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45764 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45794 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46176 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46098 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45878 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:24:46 [loggers.py:273] Engine 000: Avg prompt throughput: 2492.0 tokens/s, Avg generation throughput: 510.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.7%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:45722 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46048 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46058 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46208 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46022 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45740 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45810 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46158 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46248 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46148 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46024 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45958 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46232 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46202 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45882 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46092 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45730 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45998 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46236 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45794 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45820 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45872 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45926 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46048 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46022 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45982 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46106 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46042 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46268 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46158 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45764 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46232 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46202 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45882 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45968 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45848 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45998 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45912 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46176 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46268 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45938 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45982 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45820 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46022 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46106 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45740 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45958 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46064 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45764 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45758 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46208 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46092 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46232 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46136 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46202 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46268 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46172 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46014 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46048 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:24:56 [loggers.py:273] Engine 000: Avg prompt throughput: 3119.6 tokens/s, Avg generation throughput: 453.9 tokens/s, Running: 50 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.8%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:45820 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45822 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46022 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46106 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46118 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45860 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45848 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46130 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45836 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45842 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45938 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46076 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46238 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46218 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45900 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45982 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45954 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46176 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45788 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46040 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45936 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45796 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45750 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46174 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46258 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46190 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46262 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:25:06 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 358.2 tokens/s, Running: 23 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:45892 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45830 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45772 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45816 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46098 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45878 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45722 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46058 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45810 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46248 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46148 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46024 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46236 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45730 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45794 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45872 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45926 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:46042 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:25:16 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 191.0 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO: 127.0.0.1:46158 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45882 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45968 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45998 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO: 127.0.0.1:45912 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1162132) INFO 07-24 04:25:26 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:25:36 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1162132) INFO 07-24 04:28:03 [launcher.py:100] [shutdown] API server: shutdown triggered (APIServer pid=1162132) INFO 07-24 04:28:03 [launcher.py:116] [shutdown] API server: stopping engine client mode=abort timeout=0s (APIServer pid=1162132) INFO 07-24 04:28:03 [core_client.py:655] [shutdown] MPClient: start timeout=0s (APIServer pid=1162132) INFO 07-24 04:28:03 [core_client.py:657] [shutdown] MPClient: stopping engine manager (APIServer pid=1162132) WARNING 07-24 04:28:03 [utils.py:626] [shutdown] Process manager: force killing remaining processes count=1 (EngineCore pid=1162266) INFO 07-24 04:28:03 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM (EngineCore pid=1162266) INFO 07-24 04:28:03 [core.py:1333] [shutdown] EngineCore: start mode=abort timeout=0s (EngineCore pid=1162266) INFO 07-24 04:28:03 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown (EngineCore pid=1162266) INFO 07-24 04:28:03 [core.py:1227] [shutdown] EngineCore: exiting busy loop (Worker_PP0 pid=1162380) INFO 07-24 04:28:03 [multiproc_executor.py:790] Parent process exited, terminating worker queues (EngineCore pid=1162266) INFO 07-24 04:28:03 [multiproc_executor.py:426] [shutdown] Executor: waiting for worker exit count=2 (APIServer pid=1162132) INFO 07-24 04:28:03 [core_client.py:659] [shutdown] MPClient: engine manager stopped (APIServer pid=1162132) INFO 07-24 04:28:03 [core_client.py:660] [shutdown] MPClient: cleaning up background resources (APIServer pid=1162132) INFO 07-24 04:28:03 [core_client.py:662] [shutdown] MPClient: complete (APIServer pid=1162132) INFO 07-24 04:28:03 [launcher.py:125] [shutdown] API server: engine client stopped (APIServer pid=1162132) INFO 07-24 04:28:03 [launcher.py:128] [shutdown] API server: signalling HTTP server shutdown (APIServer pid=1162132) INFO 07-24 04:28:03 [launcher.py:149] [shutdown] API server: shutting down FastAPI HTTP server (APIServer pid=1162132) INFO: Shutting down (APIServer pid=1162132) INFO: Waiting for application shutdown. (APIServer pid=1162132) INFO: Application shutdown complete. /home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 2 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' /home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked shared_memory objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '